2023-12-19T14:22:21.284 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T14:22:21.284 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --name rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi --owner scheduled_rishabh@teuthology --verbose -- /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/orig.config.yaml 2023-12-19T14:22:21.327 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T14:22:21.411 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876 branch: rishabh-mdscap-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: null first_in_suite: false job_id: '7496876' kernel: flavor: default kdb: true sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: rishabh-mdscap-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-mdscap-quincy sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4463 sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 sleep_before_teardown: 0 subset: 344/512 suite: fs suite_branch: rishabh-mdscap-quincy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi125.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_12:25:28 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-19T14:22:21.411 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa; will attempt to use it 2023-12-19T14:22:21.412 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa/tasks 2023-12-19T14:22:21.412 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T14:22:21.414 INFO:teuthology.task.internal:Checking packages... 2023-12-19T14:22:21.436 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c7312db04d305d59a3441862c84d6d4da3817eb0' 2023-12-19T14:22:21.436 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T14:22:21.436 INFO:teuthology.packaging:ref: None 2023-12-19T14:22:21.436 INFO:teuthology.packaging:tag: None 2023-12-19T14:22:21.436 INFO:teuthology.packaging:branch: rishabh-mdscap-quincy 2023-12-19T14:22:21.436 INFO:teuthology.packaging:sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T14:22:21.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=rishabh-mdscap-quincy 2023-12-19T14:22:21.584 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-141-gc7312db0-1focal 2023-12-19T14:22:21.586 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T14:22:21.594 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T14:22:21.594 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T14:22:21.600 INFO:teuthology.task.internal:Saving configuration 2023-12-19T14:22:21.617 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T14:22:21.624 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T14:22:21.649 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 14:16:21.564741', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-19T14:22:21.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 14:16:21.565958', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-19T14:22:21.670 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-19T14:22:21.676 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-12-19T14:22:21.677 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-12-19T14:22:21.677 INFO:teuthology.run_tasks:Running task console_log... 2023-12-19T14:22:21.784 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe8dccbb790>, signals=[15]) 2023-12-19T14:22:21.785 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-19T14:22:21.792 INFO:teuthology.task.internal:Opening connections... 2023-12-19T14:22:21.792 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:22:21.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:22:21.842 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:22:21.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:22:21.892 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-19T14:22:21.899 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-12-19T14:22:21.924 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-12-19T14:22:21.924 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Ubuntu" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:ID=ubuntu 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE=debian 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="20.04" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-19T14:22:21.978 INFO:teuthology.orchestra.run.smithi050.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-19T14:22:21.979 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_CODENAME=focal 2023-12-19T14:22:21.979 INFO:teuthology.orchestra.run.smithi050.stdout:UBUNTU_CODENAME=focal 2023-12-19T14:22:21.979 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-12-19T14:22:22.003 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2023-12-19T14:22:22.010 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2023-12-19T14:22:22.011 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="Ubuntu" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:ID=ubuntu 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE=debian 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="20.04" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_CODENAME=focal 2023-12-19T14:22:22.017 INFO:teuthology.orchestra.run.smithi125.stdout:UBUNTU_CODENAME=focal 2023-12-19T14:22:22.018 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-12-19T14:22:22.036 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-19T14:22:22.046 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-19T14:22:22.052 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-19T14:22:22.052 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-12-19T14:22:22.054 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2023-12-19T14:22:22.064 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-19T14:22:22.070 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-19T14:22:22.070 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-12-19T14:22:22.072 DEBUG:teuthology.orchestra.run.smithi125:> test -z $(ls -A /var/lib/ceph) 2023-12-19T14:22:22.090 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-19T14:22:22.145 INFO:teuthology.run_tasks:Running task kernel... 2023-12-19T14:22:22.159 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'} 2023-12-19T14:22:22.159 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}}, timeout 300 2023-12-19T14:22:22.159 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T14:22:22.159 DEBUG:teuthology.orchestra.run.smithi125:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T14:22:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:22:22.165 DEBUG:teuthology.task.kernel:sha1 for mon.a is cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.165 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "cd68252d7b4d7802f3cae7f5945b38e9b20dad4c"... 2023-12-19T14:22:22.165 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-19T14:22:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:22:22.166 DEBUG:teuthology.task.kernel:sha1 for mon.b is cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.166 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "cd68252d7b4d7802f3cae7f5945b38e9b20dad4c"... 2023-12-19T14:22:22.166 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2023-12-19T14:22:22.172 INFO:teuthology.orchestra.run.smithi050.stdout:5.4.0-124-generic 2023-12-19T14:22:22.173 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-124-generic vs cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.173 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-12-19T14:22:22.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.177 INFO:teuthology.orchestra.run.smithi125.stdout:5.4.0-124-generic 2023-12-19T14:22:22.177 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-124-generic vs cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.177 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-12-19T14:22:22.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:22:22.319 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2023-12-19T14:22:22.319 DEBUG:teuthology.orchestra.run.smithi050:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-12-19T14:22:22.343 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 0: cd: /lib/firmware/updates: No such file or directory 2023-12-19T14:22:22.398 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2023-12-19T14:22:22.398 DEBUG:teuthology.orchestra.run.smithi125:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-12-19T14:22:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:bash: line 0: cd: /lib/firmware/updates: No such file or directory 2023-12-19T14:22:22.468 INFO:teuthology.orchestra.run.smithi050.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-12-19T14:22:22.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-12-19T14:22:22.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-12-19T14:22:22.555 INFO:teuthology.orchestra.run.smithi125.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-12-19T14:22:22.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-12-19T14:22:22.559 DEBUG:teuthology.orchestra.run.smithi050:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-12-19T14:22:22.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-12-19T14:22:22.653 DEBUG:teuthology.orchestra.run.smithi125:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-12-19T14:23:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-12-19T14:23:23.954 INFO:teuthology.orchestra.run.smithi050.stderr: * [new branch] main -> origin/main 2023-12-19T14:23:24.014 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190312 -> 20190312 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190416 -> 20190416 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190514 -> 20190514 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190618 -> 20190618 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190717 -> 20190717 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190815 -> 20190815 2023-12-19T14:23:24.015 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20190923 -> 20190923 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20191022 -> 20191022 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20191215 -> 20191215 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200122 -> 20200122 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200316 -> 20200316 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200421 -> 20200421 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200519 -> 20200519 2023-12-19T14:23:24.017 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200619 -> 20200619 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200721 -> 20200721 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200817 -> 20200817 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20200918 -> 20200918 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20201022 -> 20201022 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20201118 -> 20201118 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20201218 -> 20201218 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210208 -> 20210208 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210315 -> 20210315 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210511 -> 20210511 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210716 -> 20210716 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210818 -> 20210818 2023-12-19T14:23:24.018 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20210919 -> 20210919 2023-12-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20211027 -> 20211027 2023-12-19T14:23:24.020 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20211216 -> 20211216 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220209 -> 20220209 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220310 -> 20220310 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220411 -> 20220411 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220509 -> 20220509 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220610 -> 20220610 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220708 -> 20220708 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220815 -> 20220815 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20220913 -> 20220913 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20221012 -> 20221012 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20221109 -> 20221109 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20221214 -> 20221214 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230117 -> 20230117 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230210 -> 20230210 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230310 -> 20230310 2023-12-19T14:23:24.021 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230404 -> 20230404 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230515 -> 20230515 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230625 -> 20230625 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230804 -> 20230804 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20230919 -> 20230919 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20231030 -> 20231030 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20231111 -> 20231111 2023-12-19T14:23:24.022 INFO:teuthology.orchestra.run.smithi050.stderr: * [new tag] 20231211 -> 20231211 2023-12-19T14:23:28.071 INFO:teuthology.orchestra.run.smithi125.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-12-19T14:23:28.071 INFO:teuthology.orchestra.run.smithi125.stderr: * [new branch] main -> origin/main 2023-12-19T14:23:28.131 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190312 -> 20190312 2023-12-19T14:23:28.131 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190416 -> 20190416 2023-12-19T14:23:28.131 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190514 -> 20190514 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190618 -> 20190618 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190717 -> 20190717 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190815 -> 20190815 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190923 -> 20190923 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20191022 -> 20191022 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20191215 -> 20191215 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200122 -> 20200122 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200316 -> 20200316 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200421 -> 20200421 2023-12-19T14:23:28.132 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200519 -> 20200519 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200619 -> 20200619 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200721 -> 20200721 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200817 -> 20200817 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200918 -> 20200918 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201022 -> 20201022 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201118 -> 20201118 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201218 -> 20201218 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210208 -> 20210208 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210315 -> 20210315 2023-12-19T14:23:28.134 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210511 -> 20210511 2023-12-19T14:23:28.135 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210716 -> 20210716 2023-12-19T14:23:28.135 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210818 -> 20210818 2023-12-19T14:23:28.137 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210919 -> 20210919 2023-12-19T14:23:28.137 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20211027 -> 20211027 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20211216 -> 20211216 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220209 -> 20220209 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220310 -> 20220310 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220411 -> 20220411 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220509 -> 20220509 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220610 -> 20220610 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220708 -> 20220708 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220815 -> 20220815 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220913 -> 20220913 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221012 -> 20221012 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221109 -> 20221109 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221214 -> 20221214 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230117 -> 20230117 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230210 -> 20230210 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230310 -> 20230310 2023-12-19T14:23:28.138 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230404 -> 20230404 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230515 -> 20230515 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230625 -> 20230625 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230804 -> 20230804 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230919 -> 20230919 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20231030 -> 20231030 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20231111 -> 20231111 2023-12-19T14:23:28.139 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20231211 -> 20231211 2023-12-19T14:23:28.176 INFO:teuthology.orchestra.run.smithi050.stderr:Updating files: 49% (1609/3256) Updating files: 50% (1628/3256) Updating files: 51% (1661/3256) Updating files: 52% (1694/3256) Updating files: 53% (1726/3256) Updating files: 54% (1759/3256) Updating files: 55% (1791/3256) Updating files: 56% (1824/3256) Updating files: 57% (1856/3256) Updating files: 58% (1889/3256) Updating files: 59% (1922/3256) Updating files: 60% (1954/3256) Updating files: 61% (1987/3256) Updating files: 62% (2019/3256) Updating files: 63% (2052/3256) Updating files: 64% (2084/3256) Updating files: 64% (2085/3256) Updating files: 65% (2117/3256) Updating files: 66% (2149/3256) Updating files: 67% (2182/3256) Updating files: 68% (2215/3256) Updating files: 69% (2247/3256) Updating files: 70% (2280/3256) Updating files: 71% (2312/3256) Updating files: 72% (2345/3256) Updating files: 73% (2377/3256) Updating files: 74% (2410/3256) Updating files: 75% (2442/3256) Updating files: 76% (2475/3256) Updating files: 77% (2508/3256) Updating files: 78% (2540/3256) Updating files: 79% (2573/3256) Updating files: 80% (2605/3256) Updating files: 81% (2638/3256) Updating files: 81% (2647/3256) Updating files: 82% (2670/3256) Updating files: 83% (2703/3256) Updating files: 84% (2736/3256) Updating files: 85% (2768/3256) Updating files: 86% (2801/3256) Updating files: 87% (2833/3256) Updating files: 88% (2866/3256) Updating files: 89% (2898/3256) Updating files: 90% (2931/3256) Updating files: 91% (2963/3256) Updating files: 92% (2996/3256) Updating files: 93% (3029/3256) Updating files: 93% (3050/3256) Updating files: 94% (3061/3256) Updating files: 95% (3094/3256) Updating files: 96% (3126/3256) Updating files: 97% (3159/3256) Updating files: 98% (3191/3256) Updating files: 99% (3224/3256) Updating files: 100% (3256/3256) Updating files: 100% (3256/3256), done. 2023-12-19T14:23:28.181 INFO:teuthology.orchestra.run.smithi050.stdout:HEAD is now at c156e6be Merge branch 'robot/pr-0-1702625169' into 'main' 2023-12-19T14:23:28.199 INFO:teuthology.task.kernel:Downloading kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.a... 2023-12-19T14:23:28.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:23:28.420 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ 2023-12-19T14:23:28.421 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ --input-file=- 2023-12-19T14:23:28.423 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi050.front.sepia.ceph.com to complete... 2023-12-19T14:23:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19 14:23:30 URL:https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb [38928782/38928782] -> "/tmp/linux-image.deb" [1] 2023-12-19T14:23:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:FINISHED --2023-12-19 14:23:30-- 2023-12-19T14:23:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:Total wall clock time: 1.5s 2023-12-19T14:23:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:Downloaded: 1 files, 37M in 1.4s (27.5 MB/s) 2023-12-19T14:23:29.983 INFO:teuthology.task.kernel:Installing kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.a... 2023-12-19T14:23:29.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo dpkg -i /tmp/linux-image.deb 2023-12-19T14:23:30.269 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-image-6.6.0-gcd68252d7b4d. 2023-12-19T14:23:31.994 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... 104373 files and directories currently installed.) 2023-12-19T14:23:31.999 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-12-19T14:23:32.081 INFO:teuthology.orchestra.run.smithi125.stderr:Updating files: 51% (1689/3256) Updating files: 52% (1694/3256) Updating files: 53% (1726/3256) Updating files: 54% (1759/3256) Updating files: 55% (1791/3256) Updating files: 56% (1824/3256) Updating files: 57% (1856/3256) Updating files: 58% (1889/3256) Updating files: 59% (1922/3256) Updating files: 60% (1954/3256) Updating files: 61% (1987/3256) Updating files: 62% (2019/3256) Updating files: 63% (2052/3256) Updating files: 64% (2084/3256) Updating files: 65% (2117/3256) Updating files: 65% (2130/3256) Updating files: 66% (2149/3256) Updating files: 67% (2182/3256) Updating files: 68% (2215/3256) Updating files: 69% (2247/3256) Updating files: 70% (2280/3256) Updating files: 71% (2312/3256) Updating files: 72% (2345/3256) Updating files: 73% (2377/3256) Updating files: 74% (2410/3256) Updating files: 75% (2442/3256) Updating files: 76% (2475/3256) Updating files: 77% (2508/3256) Updating files: 78% (2540/3256) Updating files: 79% (2573/3256) Updating files: 80% (2605/3256) Updating files: 81% (2638/3256) Updating files: 81% (2659/3256) Updating files: 82% (2670/3256) Updating files: 83% (2703/3256) Updating files: 84% (2736/3256) Updating files: 85% (2768/3256) Updating files: 86% (2801/3256) Updating files: 87% (2833/3256) Updating files: 88% (2866/3256) Updating files: 89% (2898/3256) Updating files: 90% (2931/3256) Updating files: 91% (2963/3256) Updating files: 92% (2996/3256) Updating files: 93% (3029/3256) Updating files: 94% (3061/3256) Updating files: 95% (3094/3256) Updating files: 96% (3126/3256) Updating files: 97% (3159/3256) Updating files: 98% (3191/3256) Updating files: 99% (3224/3256) Updating files: 100% (3256/3256) Updating files: 100% (3256/3256), done. 2023-12-19T14:23:32.086 INFO:teuthology.orchestra.run.smithi125.stdout:HEAD is now at c156e6be Merge branch 'robot/pr-0-1702625169' into 'main' 2023-12-19T14:23:32.102 INFO:teuthology.task.kernel:Downloading kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.b... 2023-12-19T14:23:32.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T14:23:32.156 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T14:23:32.243 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ 2023-12-19T14:23:32.243 DEBUG:teuthology.orchestra.run.smithi125:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ --input-file=- 2023-12-19T14:23:32.246 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi125.front.sepia.ceph.com to complete... 2023-12-19T14:23:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19 14:23:34 URL:https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb [38928782/38928782] -> "/tmp/linux-image.deb" [1] 2023-12-19T14:23:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:FINISHED --2023-12-19 14:23:34-- 2023-12-19T14:23:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:Total wall clock time: 2.7s 2023-12-19T14:23:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:Downloaded: 1 files, 37M in 2.5s (15.0 MB/s) 2023-12-19T14:23:34.982 INFO:teuthology.task.kernel:Installing kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.b... 2023-12-19T14:23:34.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo dpkg -i /tmp/linux-image.deb 2023-12-19T14:23:35.155 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T14:23:35.268 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-image-6.6.0-gcd68252d7b4d. 2023-12-19T14:23:35.286 INFO:teuthology.orchestra.run.smithi050.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:23:37.001 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... 104373 files and directories currently installed.) 2023-12-19T14:23:37.006 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-12-19T14:23:37.132 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T14:23:40.043 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T14:23:40.164 INFO:teuthology.orchestra.run.smithi125.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:24:01.674 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub' 2023-12-19T14:24:01.691 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T14:24:01.763 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-12-19T14:24:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:24:02.634 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T14:24:02.647 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T14:24:04.444 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-12-19T14:24:04.635 DEBUG:teuthology.orchestra.run.smithi050:> dpkg-deb -c /tmp/linux-image.deb 2023-12-19T14:24:04.645 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./ 2023-12-19T14:24:04.645 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./boot/ 2023-12-19T14:24:04.645 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 4415617 2023-12-08 17:50 ./boot/System.map-6.6.0-gcd68252d7b4d 2023-12-19T14:24:04.658 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 169214 2023-12-08 17:50 ./boot/config-6.6.0-gcd68252d7b4d 2023-12-19T14:24:04.658 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11352864 2023-12-08 17:50 ./boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postinst.d/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postrm.d/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/preinst.d/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/prerm.d/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/ 2023-12-19T14:24:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/ 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/ 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/ 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 490233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/aesni-intel.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 93113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 106729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 152241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-12-19T14:24:04.684 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 83889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-12-19T14:24:04.685 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/ 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/ 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/ 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpuid.ko 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/msr.ko 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/ 2023-12-19T14:24:04.686 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 354433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-amd.ko 2023-12-19T14:24:04.687 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 881625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-intel.ko 2023-12-19T14:24:04.687 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 2275865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/af_alg.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_hash.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_skcipher.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ansi_cprng.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/anubis.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/arc4.ko 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/ 2023-12-19T14:24:04.760 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/ 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_memcpy.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_pq.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_tx.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_xor.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/raid6test.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authenc.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authencesn.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blake2b_generic.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_common.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_generic.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/camellia_generic.ko 2023-12-19T14:24:04.761 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast5_generic.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast6_generic.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast_common.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ccm.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cmac.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crc32_generic.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cryptd.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_engine.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_simd.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_user.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cts.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/des_generic.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/dh_generic.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/echainiv.ko 2023-12-19T14:24:04.762 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/essiv.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/fcrypt.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/khazad.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lrw.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4hc.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/md4.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/michael_mic.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcbc.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcrypt.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/rmd160.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/seed.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/serpent_generic.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 149193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tcrypt.ko 2023-12-19T14:24:04.763 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tea.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_common.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_generic.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/vmac.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/wp512.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xcbc.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xor.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xts.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xxhash_generic.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_extlog.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_ipmi.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_pad.ko 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/ 2023-12-19T14:24:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/einj.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ec_sys.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/platform_profile.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbs.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbshc.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 103025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/video.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/acard-ahci.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 88865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci_platform.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 88289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci_platform.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_acpi.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ali.ko 2023-12-19T14:24:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_amd.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_artop.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atiixp.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atp867x.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd640.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd64x.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cypress.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_efar.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt366.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt37x.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x3.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it8213.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it821x.ko 2023-12-19T14:24:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_jmicron.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_legacy.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_marvell.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_mpiix.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_netcell.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ninja32.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87410.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87415.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_oldpiix.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_opti.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_optidma.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pcmcia.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc2027x.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_piccolo.ko 2023-12-19T14:24:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_radisys.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rdc.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rz1000.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sch.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_serverworks.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sil680.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sl82c105.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_triflex.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_via.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pdc_adma.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_inic162x.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 83345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_mv.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_nv.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_promise.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_qstor.ko 2023-12-19T14:24:04.768 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil24.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sis.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_svw.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sx4.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_uli.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_via.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_vsc.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ks0108.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/ 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 124473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/bcma.ko 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/ 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/ 2023-12-19T14:24:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 113665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/aoe.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/ 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 965585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/drbd.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 173225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/floppy.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/ 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 94297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 115793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/nbd.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/ 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 110385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/null_blk.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 98193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/pktcdvd.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 263769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/rbd.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/ 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 98145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/ 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/zram.ko 2023-12-19T14:24:04.770 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/applicom.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hangcheck-timer.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/ 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/amd-rng.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/intel-rng.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/via-rng.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 148601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 139313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-12-19T14:24:04.771 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/lp.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/ 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 84097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/mwave.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/nvram.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ppdev.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tlclk.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/ 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-12-19T14:24:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/ 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_class.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_core.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/ 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-palmas.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-si5351.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-twl6040.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-wm831x.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/ 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ 2023-12-19T14:24:04.773 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 243025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 438761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-aes.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-sha.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/dca.ko 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ 2023-12-19T14:24:04.774 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/ 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/dw_dmac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 122569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ioatdma.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 79953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/amd64_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/e752x_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 63617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/edac_mce_amd.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3000_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3200_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5100_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5400_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7300_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7core_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i82975x_edac.ko 2023-12-19T14:24:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ie31200_edac.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 56825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/sb_edac.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/x38_edac.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/ 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-gpio.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max14577.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max77693.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max8997.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-palmas.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-rt8973a.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-sm5502.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/ 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/dmi-sysfs.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/ 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/efi-pstore.ko 2023-12-19T14:24:04.776 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/iscsi_ibft.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/ 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-adp5520.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-amd8111.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-arizona.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9052.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9055.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-dln2.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 77849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-f7188x.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-generic.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ich.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-kempld.ko 2023-12-19T14:24:04.777 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-lp3943.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max7300.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max730x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max732x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pca953x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pcf857x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-rdc321x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch311x.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-tps65912.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl4030.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl6040.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-viperboard.ko 2023-12-19T14:24:04.778 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-vx855.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm831x.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8350.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8994.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/ 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-a4tech.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-apple.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-appleir.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-aureal.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-axff.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-belkin.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cherry.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-chicony.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cp2112.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cypress.ko 2023-12-19T14:24:04.779 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-dr.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elecom.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elo.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-emsff.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ezkey.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gaff.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-generic.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gt683r.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gyration.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtekff.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-hyperv.ko 2023-12-19T14:24:04.780 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-icade.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kensington.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-keytouch.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kye.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lcpower.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-led.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lenovo.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lg-g15.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-dj.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 80993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-magicmouse.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-microsoft.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-monterey.ko 2023-12-19T14:24:04.781 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-multitouch.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ntrig.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ortek.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-penmount.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-petalynx.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-picolcd.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-pl.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-plantronics.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-primax.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-rmi.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-common.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-isku.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kone.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-12-19T14:24:04.782 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-lua.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-savu.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-saitek.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-samsung.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sensor-hub.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sjoy.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sony.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-speedlink.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-steelseries.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sunplus.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-thrustmaster.ko 2023-12-19T14:24:04.783 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tivo.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tmff.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-topseed.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-twinhan.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 71073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-uclogic.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-waltop.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 143473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-wiimote.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-xinmo.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zpff.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zydacron.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 267825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/uhid.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/ 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbhid.ko 2023-12-19T14:24:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbkbd.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbmouse.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 217873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/wacom.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/ 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 64201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_balloon.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_utils.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 305465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_vmbus.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru3.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7414.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7418.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adc128d818.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1025.ko 2023-12-19T14:24:04.785 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 97977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1026.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1029.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1031.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm9240.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ads7828.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7410.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7411.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 74433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7462.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7470.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7475.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7x10.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/amc6821.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/applesmc.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asb100.ko 2023-12-19T14:24:04.786 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asc7621.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asus_atk0110.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/atxp1.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/coretemp.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9052-hwmon.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9055-hwmon.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 99601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dme1737.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds1621.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds620.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc1403.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc2103.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc6w201.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71805f.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 94457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71882fg.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f75375s.ko 2023-12-19T14:24:04.787 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fam15h_power.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fschmd.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g760a.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g762.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl518sm.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl520sm.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hih6130.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hwmon-vid.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5500_temp.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5k_amb.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmaem.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmpex.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina209.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina2xx.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 130449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/it87.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/jc42.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k10temp.ko 2023-12-19T14:24:04.788 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k8temp.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lineage-pem.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm63.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm73.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm75.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm77.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm78.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm80.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm83.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm85.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm87.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm90.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm92.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 108841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm93.ko 2023-12-19T14:24:04.789 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95234.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95241.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95245.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc2945.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4151.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4215.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4222.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4245.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4260.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4261.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max16065.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1619.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1668.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max197.ko 2023-12-19T14:24:04.790 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6639.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6650.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6697.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mc13783-adc.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mcp3021.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6683.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 119817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775-core.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct7802.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 82337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87360.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87427.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pcf8591.ko 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ 2023-12-19T14:24:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 82577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/powr1220.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5627.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5636.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch56xx-common.ko 2023-12-19T14:24:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht15.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht21.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/shtc1.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sis5595.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47b397.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m1.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m192.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/thmc50.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp102.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp103.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp401.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp421.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via-cputemp.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via686a.ko 2023-12-19T14:24:04.793 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt1211.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt8231.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627ehf.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627hf.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 71345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83781d.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 63225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83791d.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 71321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83792d.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83793.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 119505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83795.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l785ts.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l786ng.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/ 2023-12-19T14:24:04.794 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/ 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/ 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-12-19T14:24:04.795 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-12-19T14:24:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-via.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-mux.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-smbus.ko 2023-12-19T14:24:04.797 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-stub.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/ 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 273609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_cm.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 879969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_core.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 69273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_umad.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 324217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 97985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/iw_cm.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 282273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/ 2023-12-19T14:24:04.798 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 493073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 419737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 739025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 296625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 153697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 168969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ 2023-12-19T14:24:04.799 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 263081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 100241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 111425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 128753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 129529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/evbug.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ff-memless.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/ 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/fm801-gp.ko 2023-12-19T14:24:04.800 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/gameport.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/joydev.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/ 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lkkbd.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8323.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8333.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-12-19T14:24:04.801 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/newtonkbd.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt1070.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt2160.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/stowaway.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/sunkbd.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/xtkbd.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/matrix-keymap.ko 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/ 2023-12-19T14:24:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/appletouch.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/bcm5974.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 123681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/cyapatp.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/elan_i2c.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/gpio_mouse.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 335929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/psmouse.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/sermouse.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/ 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 109873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/rmi_core.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/altera_ps2.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/arc_ps2.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ct82c710.ko 2023-12-19T14:24:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/parkbd.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/pcips2.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ps2mult.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serio_raw.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serport.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/sparse-keymap.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/ 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/ 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/ 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-88pm860x.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-adp5520.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-bd2802.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-blinkm.ko 2023-12-19T14:24:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da903x.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da9052.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-gpio.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3530.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3533.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm355x.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3642.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp3944.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp8788.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-max8997.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-mc13783.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-menf21bmc.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca9532.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca955x.ko 2023-12-19T14:24:04.805 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca963x.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-ss4200.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-tca6507.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm831x-status.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm8350.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/ 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/ 2023-12-19T14:24:04.806 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 654969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/bcache.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bio-prison.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bufio.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache-smq.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 156009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 106897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-crypt.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-delay.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-era.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-flakey.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log-userspace.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-mirror.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-multipath.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-queue-length.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-raid.ko 2023-12-19T14:24:04.807 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-region-hash.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-round-robin.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-service-time.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 109529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-snapshot.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-switch.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 181865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-thin-pool.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-verity.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-zero.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/faulty.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/linear.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/multipath.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/ 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 199937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid0.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 113073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid1.ko 2023-12-19T14:24:04.808 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 158681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid10.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 395721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid456.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/ 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/ 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 178137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptbase.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 63385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptctl.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptfc.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptlan.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 123321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptsas.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 79041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptscsih.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptspi.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm800.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm805.ko 2023-12-19T14:24:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm80x.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona-i2c.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 123721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/bcm590xx.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/cros_ec_dev.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/dln2.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/janz-cmodio.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/kempld-core.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-core.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lp3943.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_ich.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_sch.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/max8907.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-core.ko 2023-12-19T14:24:04.810 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/menf21bmc.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-adc.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-gpio.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/retu-mfd.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 80425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/si476x-core.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/sm501.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps65010.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps6507x.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/viperboard.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/vx855.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wl1273-core.ko 2023-12-19T14:24:04.811 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wm8994.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/ 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds9802als.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds990x.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/bh1770glc.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/ 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/core.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/ 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/cb710.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ds1682.ko 2023-12-19T14:24:04.812 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/dummy-irq.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/ 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/echo.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/ 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/at24.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/max6875.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/enclosure.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/ 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 152201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hmc6352.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hpilo.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-12-19T14:24:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ics932s401.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29003.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29020.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/ 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/ 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-me.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 56521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-txe.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 310113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/phantom.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/ 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 74265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/st_drv.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_7xx1.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_core.ko 2023-12-19T14:24:04.814 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tsl2550.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_balloon.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 183609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 490129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/bonding.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 106033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6060.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/ 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 453737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-12-19T14:24:04.815 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dummy.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/eql.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/ 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 130553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 68849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/8390.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 99049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/ 2023-12-19T14:24:04.816 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 69137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 87297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/ 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 97337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-12-19T14:24:04.817 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 104673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 90257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/ 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 81569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 69729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/ 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 176761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-12-19T14:24:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1421497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-12-19T14:24:04.896 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 128569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-12-19T14:24:04.896 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/ 2023-12-19T14:24:04.896 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 132089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 312081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 323513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 193497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 347417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1044497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-12-19T14:24:04.897 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 166193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 204345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 69449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 141225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/ 2023-12-19T14:24:04.898 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dnet.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ec_bhf.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/ 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/ 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 281137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ethoc.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fealnx.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/ 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 90465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e100.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/ 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 295577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-12-19T14:24:04.899 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 584497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 246209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 925825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 302937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 514641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 93593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 734305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-12-19T14:24:04.900 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 151769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 86921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/jme.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/ 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 102289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 122649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/ 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 727601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 278937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1113681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-12-19T14:24:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 95665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 108649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 150905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/ 2023-12-19T14:24:04.902 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 136721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 227897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 87081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 672049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/ 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 83985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-12-19T14:24:04.903 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 189801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 104241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1052497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 63337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/ 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-12-19T14:24:04.904 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 68177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/ 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 487025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/ 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 103337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 166097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/niu.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/ 2023-12-19T14:24:04.905 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 86529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 80273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 83633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/defxx.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/ 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 216985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/skfp.ko 2023-12-19T14:24:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 77345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/geneve.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/ 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 235521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/ 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/fakelb.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ifb.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 83769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvlan.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvtap.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/ 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-gpio.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-regmap.ko 2023-12-19T14:24:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mii.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/netconsole.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/nlmon.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/ 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs-lynx.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/amd.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm7xxx.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm87xx.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/broadcom.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/cicada.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/davicom.ko 2023-12-19T14:24:04.908 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/et1011c.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/icplus.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/lxt.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 82049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/marvell.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 76113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/micrel.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/national.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 97857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/phylink.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/qsemi.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/realtek.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/smsc.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ste10Xp.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/vitesse.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/ 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/plip.ko 2023-12-19T14:24:04.909 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/bsd_comp.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_async.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppoe.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppox.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pptp.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sb1000.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/ 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/slip.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sungem_phy.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 64449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/tap.ko 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/ 2023-12-19T14:24:04.910 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 106169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_random.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 87017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/veth.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/ 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 144297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/ 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 297329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/vxlan.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/ 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 144585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ntb.ko 2023-12-19T14:24:04.911 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/ 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/ 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 265121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme-core.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 102209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/ 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 133841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_cs.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 84465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_pc.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_serial.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/ 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/ 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-12-19T14:24:04.912 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/pci-stub.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/xen-pcifront.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/ 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/i82092.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 145809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_core.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pd6729.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 112889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/yenta_socket.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/ 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/ 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/ 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/ 2023-12-19T14:24:04.913 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/ 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/ 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/ 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acer-wmi.ko 2023-12-19T14:24:04.914 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acerhdf.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/apple-gmux.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-laptop.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 127441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-wmi.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/hdaps.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ibm_rtl.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/ 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-12-19T14:24:04.915 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel_ips.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/msi-wmi.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-q10.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/ 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_battery.ko 2023-12-19T14:24:04.916 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq2415x_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24190_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24735-charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9030_battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9052-battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2760_battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2780_battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2781_battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2782_battery.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/gpio-charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/isp1704_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/lp8727_charger.ko 2023-12-19T14:24:04.917 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max14577_charger.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17040_battery.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17042_battery.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8903_charger.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8925_power.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/pcf50633-charger.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/sbs-battery.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/test_power.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/tps65090-charger.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_backup.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_power.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm8350_power.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/ 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_common.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/ 2023-12-19T14:24:04.918 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/ 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-gpio.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps_parport.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/pps_core.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp_kvm.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/ 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm80x.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm860x.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-bq32k.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9052.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9055.ko 2023-12-19T14:24:04.919 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9063.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1286.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 68577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1307.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1374.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1511.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1553.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1672.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1742.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds2404.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds3232.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-em3027.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-fm3130.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl12022.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl1208.ko 2023-12-19T14:24:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-lp8788.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m41t80.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t35.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t59.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t86.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max6900.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8907.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8925.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8997.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8998.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-msm6242.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-palmas.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-12-19T14:24:04.921 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf50633.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rc5t583.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8025.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8581.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-s35390a.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps6586x.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps65910.ko 2023-12-19T14:24:04.922 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm831x.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm8350.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-x1205.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 77617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-9xxx.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-sas.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-xxxx.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 135185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/BusLogic.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/a100u2w.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/ 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 272305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 100617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/advansys.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/ 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 356201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 295801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-12-19T14:24:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 189185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/am53c974.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 106881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/atp870u.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 246985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1171945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/bfa.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 237273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/ 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 115369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ch.ko 2023-12-19T14:24:04.924 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 308921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 76145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 146689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 75553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dc395x.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dmx3191d.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 212425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esp_scsi.ko 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/ 2023-12-19T14:24:04.925 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 158505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fdomain.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/ 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 263489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/fnic.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 201817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hpsa.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hptiop.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hv_storvsc.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/imm.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/initio.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 266537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ipr.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ips.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/ 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 317153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/isci.ko 2023-12-19T14:24:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_tcp.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/ 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 314865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/libfc.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 145585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/ 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 216665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/libsas.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/ 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 2197249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/ 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 310617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 84721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid.ko 2023-12-19T14:24:04.927 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/ 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 659825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/ 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 119561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvumi.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/ 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 81625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/ 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 364105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 123617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pmcraid.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ppa.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla1280.ko 2023-12-19T14:24:04.928 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/ 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1876689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 85289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/ 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 559033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qlogicfas408.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/raid_class.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 238761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_debug.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 173937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 279313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 97729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ses.ko 2023-12-19T14:24:04.929 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 115889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/st.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/stex.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/ 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 162121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/virtio_scsi.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/wd719x.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/xen-scsifront.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 129441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ssb.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/ 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/ 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 736601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/ 2023-12-19T14:24:04.930 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/tcm_loop.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_file.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_iblock.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 930257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_mod.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_pscsi.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 122521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_user.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/ 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 75593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/ 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/ 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/ 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-12-19T14:24:04.931 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-12-19T14:24:04.932 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/moxa.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 50953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/mxser.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 108601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_gsm.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_hdlc.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/nozomi.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/ 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/ 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_uart.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/arc_uart.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/ 2023-12-19T14:24:04.933 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 103073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/rp2.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/sc16is7xx.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/ 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 531265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/ 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_aec.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_cif.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_dmem_genirq.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_mf624.ko 2023-12-19T14:24:04.934 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_netx.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pci_generic.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_sercos3.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/ 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/ 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/c67x00.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 144377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/ 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 87089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-acm.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-wdm.ko 2023-12-19T14:24:04.935 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usblp.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usbtmc.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ulpi.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/ 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/ 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 448505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/ 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/ 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_ether.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_serial.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-12-19T14:24:04.936 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 92489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 122081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 85609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-12-19T14:24:04.937 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/ 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 62185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-12-19T14:24:04.938 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 161225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/libcomposite.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/ 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/ 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 157681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 69705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 56033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 76057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2272.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 75073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2280.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 73169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 65809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 81505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-12-19T14:24:04.939 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 156073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/bcma-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 80113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/isp116x-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 87353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811_cs.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ssb-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/ 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/mdc800.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/microtek.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/adutux.ko 2023-12-19T14:24:04.940 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/appledisplay.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cytherm.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ehset.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi26.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi62.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ezusb.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/idmouse.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/iowarrior.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/isight_firmware.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ldusb.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/legousbtower.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/lvstest.ko 2023-12-19T14:24:04.941 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/ 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 64841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/trancevibrator.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usb3503.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usblcd.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbsevseg.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbtest.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/uss720.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/yurex.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/ 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/usbmon.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/ 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 301697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/musb_hdrc.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/ 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-generic.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-12-19T14:24:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-isp1301.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-tahvo.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/aircable.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ark3116.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/belkin_sa.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ch341.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cp210x.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cyberjack.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cypress_m8.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/empeg.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/f81232.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 175137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/garmin_gps.ko 2023-12-19T14:24:04.943 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 137081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_edgeport.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 124833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_ti.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipaq.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipw.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ir-usb.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 107521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kobil_sct.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mct_u232.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/metro-usb.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7720.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 78217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7840.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 56337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mxuport.ko 2023-12-19T14:24:04.944 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/navman.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/omninet.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/opticon.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 191137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/option.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/oti6858.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/pl2303.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcaux.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcserial.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/quatech2.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/safe_serial.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 51625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/sierra.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/spcp8x5.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ssu100.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/symbolserial.ko 2023-12-19T14:24:04.945 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_debug.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_wwan.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usbserial.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/visor.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/whiteheat.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/wishbone-serial.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/xsens_mt.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 52913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/uas.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-alauda.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-cypress.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-datafab.ko 2023-12-19T14:24:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-freecom.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-isd200.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-karma.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-realtek.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-usbat.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 179969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/usb-storage.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/ 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-core.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-host.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 107001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-12-19T14:24:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/ 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/ 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 154297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 108737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 77609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/ 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 113785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_iotlb.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 58089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_net.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_scsi.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/ 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/ 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/aat2870_bl.ko 2023-12-19T14:24:04.948 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp5520_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8860_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8870_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/apple_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/as3711_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/bd6107.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da903x_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da9052_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/gpio_backlight.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/kb3886_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lcd.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3533_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3639_bl.ko 2023-12-19T14:24:04.949 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lv5207lp.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/max8925_bl.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pandora_bl.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/platform_lcd.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/wm831x_bl.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/ 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2482.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2490.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/matrox_w1.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/w1-gpio.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/ 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-12-19T14:24:04.950 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_smem.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 56425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_therm.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 102673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/wire.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/acquirewdt.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/advantechwdt.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-12-19T14:24:04.951 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/cpu5wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9052_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9055_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9063_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/dw_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/eurotechwdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/hpwdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/i6300esb.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ib700wdt.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ibmasr.ko 2023-12-19T14:24:04.952 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it87_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/kempld_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/machzwd.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/mena21_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/nv_tco.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pc87413_wdt.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_pci.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_usb.ko 2023-12-19T14:24:04.953 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/retu_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sc1200wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/softdog.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sp5100_tco.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/twl4030_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/via_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-12-19T14:24:04.954 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wafer5823wdt.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wdt_pci.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm831x_wdt.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm8350_wdt.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/xen_wdt.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/ 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-evtchn.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntalloc.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntdev.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/ 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 176377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-privcmd.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 71249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-scsiback.ko 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/ 2023-12-19T14:24:04.955 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/xenfs.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 166425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/9p.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/adfs.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 163593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/affs.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 932817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/kafs.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/autofs4.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 64545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/befs.ko 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/ 2023-12-19T14:24:04.956 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/bfs.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/binfmt_misc.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 3402729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/btrfs.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 306009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/cachefiles.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1333017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ceph.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 100801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/coda.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 108393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/configfs.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 29625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/cramfs.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/ 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 642513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/dlm.ko 2023-12-19T14:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/efs.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1505025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/f2fs.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/msdos.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/freevxfs.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 226481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/fscache.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/cuse.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/ 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1231529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/gfs2.ko 2023-12-19T14:24:04.958 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 141209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/hfs.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 222817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/hfsplus.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 192329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/hpfs.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 74065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/isofs.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 419497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/jfs.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 280313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/lockd.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/minix.ko 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/ 2023-12-19T14:24:04.959 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 124649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/netfs.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/ 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/ 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 81449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/ 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 59449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/ 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 117465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1055721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfs.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv2.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 120177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv3.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 2042137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv4.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/ 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/grace.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/nfs_acl.ko 2023-12-19T14:24:04.960 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/ 2023-12-19T14:24:04.961 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1516873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/nfsd.ko 2023-12-19T14:24:05.007 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/ 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 592833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/nilfs2.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/ 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-celtic.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-centeuro.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-croatian.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-cyrillic.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-gaelic.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-greek.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-iceland.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-inuit.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-roman.ko 2023-12-19T14:24:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-romanian.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-turkish.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ascii.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1250.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1251.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1255.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp737.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp775.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp850.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp852.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp855.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp857.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp860.ko 2023-12-19T14:24:05.009 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp861.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp862.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp863.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp864.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp865.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp866.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp869.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp874.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 94137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp932.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 142673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp936.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 165745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp949.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 118121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp950.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_euc-jp.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-1.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-13.ko 2023-12-19T14:24:05.010 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-14.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-15.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-2.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-3.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-4.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-5.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-6.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-7.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-9.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-r.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-ru.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-u.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ucs2_utils.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_utf8.ko 2023-12-19T14:24:05.011 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 219417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ntfs.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 244409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 440737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 2592433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/ 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/omfs.ko 2023-12-19T14:24:05.012 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 352161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/overlay.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ramoops.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/qnx4.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 46689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/qnx6.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_tree.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v1.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v2.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/ 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 532673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/reiserfs.ko 2023-12-19T14:24:05.013 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/romfs.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 2668401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/cifs.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_arc4.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_md4.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 134641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/squashfs.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 90721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/sysv.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 247737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/udf.ko 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ 2023-12-19T14:24:05.014 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 162497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ufs.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 4011017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/xfs.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 75489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/locktorture.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/test_udelay.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/torture.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc-itu-t.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc7.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc8.ko 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/ 2023-12-19T14:24:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libarc4.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libdes.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/interval_tree_test.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/libcrc32c.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lru_cache.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/ 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4_compress.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4hc_compress.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/ 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/cordic.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/memory-notifier-error-inject.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/notifier-error-inject.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/percpu_test.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/pm-notifier-error-inject.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/ 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 143433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/raid6_pq.ko 2023-12-19T14:24:05.016 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/rbtree_test.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/ 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/reed_solomon.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-kstrtox.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-string_helpers.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 4760425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_bpf.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_firmware.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_module.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_user_copy.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_bm.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_fsm.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_kmp.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/ 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/xz_dec_test.ko 2023-12-19T14:24:05.017 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/ 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/hwpoison-inject.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/ 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/6lowpan.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_dest.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_fragment.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_hop.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 6985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_ipv6.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_mobility.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_routing.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_udp.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/ 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/garp.ko 2023-12-19T14:24:05.018 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/mrp.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/p8022.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/psnap.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/stp.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/ 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 81737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/8021q.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/ 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 198337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_fd.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_rdma.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_virtio.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/ 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 649297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/batman-adv.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/ 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 63809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/br_netfilter.ko 2023-12-19T14:24:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 777689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/bridge.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_among.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_log.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-12-19T14:24:05.020 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 55929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtables.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/ 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1046681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/libceph.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/ 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/pktgen.ko 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/ 2023-12-19T14:24:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 194209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_diag.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 57881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv4.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv6.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/ 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 390473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/dsa_core.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_brcm.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_dsa.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_none.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_trailer.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/ 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 99729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/hsr.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 45233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-12-19T14:24:05.022 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 254625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154_socket.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ah4.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/esp4.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/fou.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/gre.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/inet_diag.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_gre.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_tunnel.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_vti.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipcomp.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipip.ko 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ 2023-12-19T14:24:05.023 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ip_tables.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-12-19T14:24:05.024 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_bic.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_dctcp.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_diag.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_highspeed.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_htcp.ko 2023-12-19T14:24:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_hybla.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_illinois.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_lp.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_scalable.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_vegas.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_veno.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_westwood.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_yeah.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tunnel4.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_diag.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 41665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_tunnel.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ah6.ko 2023-12-19T14:24:05.026 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/esp6.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/fou6.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 80897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_gre.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 93577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_tunnel.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_vti.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ipcomp6.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/mip6.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-12-19T14:24:05.027 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-12-19T14:24:05.028 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 74073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/sit.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/tunnel6.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/ 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 82833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/af_key.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/ 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_core.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_debugfs.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 25953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_eth.ko 2023-12-19T14:24:05.029 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip6.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_netlink.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 53241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ppp.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/ 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 47977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/lapb.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/ 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 267913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc2.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/ 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 290417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/mac802154.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/ 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/mpls_gso.ko 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ 2023-12-19T14:24:05.030 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 119585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 100297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 101441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 102817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 103417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 112849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 60329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 109593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 113129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 112585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-12-19T14:24:05.031 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 111585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 114465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 411729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-12-19T14:24:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conncount.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 408257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 118145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 107233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-12-19T14:24:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_dup_netdev.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 37129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_log_syslog.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 95353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_amanda.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_ftp.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_irc.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_sip.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_tftp.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 44065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_synproxy_core.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 615649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_tables.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 42561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink.ko 2023-12-19T14:24:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_acct.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 39785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_log.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_osf.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_queue.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_chain_nat.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_compat.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 40721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_ct.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_dup_netdev.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fib.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_hash.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_limit.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_log.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_masq.ko 2023-12-19T14:24:05.035 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_nat.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_numgen.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_osf.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_queue.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_redir.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject_inet.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_socket.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_synproxy.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_tproxy.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 89313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/x_tables.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_AUDIT.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-12-19T14:24:05.036 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 18169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CT.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_DSCP.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HL.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HMARK.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LED.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LOG.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NETMAP.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFLOG.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_RATEEST.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_REDIRECT.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 14337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_SECMARK.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPMSS.ko 2023-12-19T14:24:05.037 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TEE.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TPROXY.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TRACE.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_addrtype.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_bpf.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cgroup.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cluster.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_comment.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connbytes.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlabel.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlimit.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connmark.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_conntrack.ko 2023-12-19T14:24:05.038 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cpu.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dccp.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_devgroup.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dscp.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ecn.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_esp.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hashlimit.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_helper.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hl.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipcomp.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_iprange.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipvs.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_l2tp.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_length.ko 2023-12-19T14:24:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_limit.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mac.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mark.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_multiport.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nat.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nfacct.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_osf.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_owner.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_physdev.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_pkttype.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_policy.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_quota.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_rateest.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_realm.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_recent.ko 2023-12-19T14:24:05.040 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_sctp.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_set.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 16089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_socket.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_state.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_statistic.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_string.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 8553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpmss.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpudp.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_time.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_u32.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/ 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/netlink_diag.ko 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/ 2023-12-19T14:24:05.041 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/nsh.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/ 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 380033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/openvswitch.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-geneve.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-gre.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 11793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-vxlan.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/ 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/af_packet_diag.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/ 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 110201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/phonet.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/pn_pep.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/ 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 251481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 164937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_rdma.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 67377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_tcp.ko 2023-12-19T14:24:05.042 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/ 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 819657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/rxrpc.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/ 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_csum.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_gact.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_ipt.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_mirred.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_nat.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 43449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_pedit.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_police.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_simple.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_skbedit.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_vlan.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_basic.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_bpf.ko 2023-12-19T14:24:05.043 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_cgroup.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_flow.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_fw.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 38425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_route.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 61409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_u32.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_cmp.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 13409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_ipset.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_meta.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_nbyte.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_text.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_u32.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 21649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_choke.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_codel.ko 2023-12-19T14:24:05.044 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 34633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_drr.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 33609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq_codel.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 35433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_gred.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hfsc.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hhf.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 66393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_htb.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_ingress.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 30273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio_lib.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_multiq.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 36137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_netem.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 27497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_pie.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_plug.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_prio.ko 2023-12-19T14:24:05.045 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 48729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_qfq.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 32905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_red.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 26881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfb.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 31353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfq.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_tbf.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 28105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_teql.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/ 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 929745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 19033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp_diag.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/ 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/ 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 336529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 54737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 1520097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/sunrpc.ko 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/ 2023-12-19T14:24:05.046 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 707657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/ 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/diag.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 862665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/tipc.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/ 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/unix_diag.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/ 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 85409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 68049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 96193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 10705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_diag.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 17521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/ 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 125177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/x25.ko 2023-12-19T14:24:05.047 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/ 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 22657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_algo.ko 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 20649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 102257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_user.ko 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/ 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/ 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 15537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/irqbypass.ko 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 640081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 624107 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias.bin 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 7663 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 23428 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.alias.bin 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 9821 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.bin 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 70004 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.modinfo 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 146313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep 2023-12-19T14:24:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 232548 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep.bin 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 210 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.devname 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 72606 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.order 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 879 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.softdep 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 254438 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 309363 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols.bin 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/ 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/ 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/ 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/ 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 170 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/changelog.Debian.gz 2023-12-19T14:24:05.049 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r-- root/root 691 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/copyright 2023-12-19T14:24:05.050 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gcd68252d7b4d 2023-12-19T14:24:05.050 INFO:teuthology.task.kernel:searching for kernel 6.6.0-gcd68252d7b4d 2023-12-19T14:24:05.050 DEBUG:teuthology.orchestra.run.smithi050:> egrep '(submenu|menuentry.*6.6.0-gcd68252d7b4d).*{' /boot/grub/grub.cfg 2023-12-19T14:24:05.059 INFO:teuthology.orchestra.run.smithi050.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-recovery-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:05.060 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-12-19T14:24:05.060 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-gcd68252d7b4d 2023-12-19T14:24:05.060 DEBUG:teuthology.orchestra.run.smithi050:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-12-19T14:24:05.063 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi050.front.sepia.ceph.com to complete... 2023-12-19T14:24:05.637 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub' 2023-12-19T14:24:05.649 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T14:24:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-12-19T14:24:05.930 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub' 2023-12-19T14:24:05.936 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T14:24:05.957 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2023-12-19T14:24:05.993 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:06.005 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:24:06.281 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:06.301 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:24:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T14:24:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T14:24:06.715 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T14:24:06.728 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T14:24:07.146 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-12-19T14:24:08.534 INFO:teuthology.orchestra.run.smithi125.stderr:done 2023-12-19T14:24:08.678 DEBUG:teuthology.orchestra.run.smithi125:> dpkg-deb -c /tmp/linux-image.deb 2023-12-19T14:24:08.687 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./ 2023-12-19T14:24:08.687 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./boot/ 2023-12-19T14:24:08.687 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4415617 2023-12-08 17:50 ./boot/System.map-6.6.0-gcd68252d7b4d 2023-12-19T14:24:08.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 169214 2023-12-08 17:50 ./boot/config-6.6.0-gcd68252d7b4d 2023-12-19T14:24:08.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11352864 2023-12-08 17:50 ./boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:08.728 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postinst.d/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postrm.d/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/preinst.d/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/prerm.d/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ 2023-12-19T14:24:08.729 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 490233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/aesni-intel.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 152241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-12-19T14:24:08.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/ 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/ 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/ 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpuid.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/msr.ko 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/ 2023-12-19T14:24:08.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 354433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-amd.ko 2023-12-19T14:24:08.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 881625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-intel.ko 2023-12-19T14:24:08.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2275865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm.ko 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/af_alg.ko 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_hash.ko 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_skcipher.ko 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ansi_cprng.ko 2023-12-19T14:24:08.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/anubis.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/arc4.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/ 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/ 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_memcpy.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_pq.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_tx.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_xor.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/raid6test.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authenc.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authencesn.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blake2b_generic.ko 2023-12-19T14:24:08.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_common.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_generic.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/camellia_generic.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast5_generic.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast6_generic.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast_common.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ccm.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cmac.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crc32_generic.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cryptd.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_engine.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_simd.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_user.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cts.ko 2023-12-19T14:24:08.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/des_generic.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/dh_generic.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/echainiv.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/essiv.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/fcrypt.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/khazad.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lrw.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4hc.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/md4.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/michael_mic.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcbc.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcrypt.ko 2023-12-19T14:24:08.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/rmd160.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/seed.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/serpent_generic.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 149193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tcrypt.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tea.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_common.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_generic.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/vmac.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/wp512.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xcbc.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xor.ko 2023-12-19T14:24:08.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xts.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xxhash_generic.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_extlog.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_ipmi.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_pad.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/ 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/einj.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ec_sys.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/platform_profile.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbs.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbshc.ko 2023-12-19T14:24:08.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/video.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/acard-ahci.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 88865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci_platform.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 88289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci_platform.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_acpi.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ali.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_amd.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_artop.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atiixp.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atp867x.ko 2023-12-19T14:24:08.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd640.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd64x.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cypress.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_efar.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt366.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt37x.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x3.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it8213.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it821x.ko 2023-12-19T14:24:08.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_jmicron.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_legacy.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_marvell.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_mpiix.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_netcell.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ninja32.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87410.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87415.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_oldpiix.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_opti.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_optidma.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pcmcia.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc2027x.ko 2023-12-19T14:24:08.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_piccolo.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_radisys.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rdc.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rz1000.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sch.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_serverworks.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sil680.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sl82c105.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_triflex.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_via.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pdc_adma.ko 2023-12-19T14:24:08.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_inic162x.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_mv.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_nv.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_promise.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_qstor.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil24.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sis.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_svw.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sx4.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_uli.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_via.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_vsc.ko 2023-12-19T14:24:08.825 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ks0108.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/bcma.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/aoe.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 965585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/drbd.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 173225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/floppy.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/ 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/nbd.ko 2023-12-19T14:24:08.826 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/ 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 110385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/null_blk.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 98193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/pktcdvd.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/rbd.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/ 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 98145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/ 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/zram.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/applicom.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hangcheck-timer.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/ 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/amd-rng.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/intel-rng.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-12-19T14:24:08.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/via-rng.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 148601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 139313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/lp.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/ 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/mwave.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/nvram.ko 2023-12-19T14:24:08.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ppdev.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tlclk.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/ 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/ 2023-12-19T14:24:08.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_class.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_core.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/ 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-palmas.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-si5351.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-twl6040.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-wm831x.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/ 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ 2023-12-19T14:24:08.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 243025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 438761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-aes.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-sha.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/dca.ko 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ 2023-12-19T14:24:08.831 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/ 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/dw_dmac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ioatdma.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 79953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/amd64_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/e752x_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/edac_mce_amd.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3000_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3200_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5100_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5400_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7300_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7core_edac.ko 2023-12-19T14:24:08.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i82975x_edac.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ie31200_edac.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/sb_edac.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/x38_edac.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/ 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-gpio.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max14577.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max77693.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max8997.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-palmas.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-rt8973a.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-sm5502.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/ 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/dmi-sysfs.ko 2023-12-19T14:24:08.833 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/ 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/efi-pstore.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/iscsi_ibft.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/ 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-adp5520.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-amd8111.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-arizona.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9052.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9055.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-dln2.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-f7188x.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-generic.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ich.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-kempld.ko 2023-12-19T14:24:08.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-lp3943.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max7300.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max730x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max732x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pca953x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pcf857x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-rdc321x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch311x.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-tps65912.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl4030.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl6040.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-viperboard.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-vx855.ko 2023-12-19T14:24:08.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm831x.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8350.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8994.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/ 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-a4tech.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-apple.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-appleir.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-aureal.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-axff.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-belkin.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cherry.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-chicony.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cp2112.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cypress.ko 2023-12-19T14:24:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-dr.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elecom.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elo.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-emsff.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ezkey.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gaff.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-generic.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gt683r.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gyration.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtekff.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-hyperv.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-icade.ko 2023-12-19T14:24:08.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kensington.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-keytouch.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kye.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lcpower.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-led.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lenovo.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lg-g15.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-dj.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-magicmouse.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-microsoft.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-monterey.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-multitouch.ko 2023-12-19T14:24:08.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ntrig.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ortek.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-penmount.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-petalynx.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-picolcd.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-pl.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-plantronics.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-primax.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-rmi.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-common.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-isku.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kone.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-12-19T14:24:08.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-lua.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-savu.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-saitek.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-samsung.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sensor-hub.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sjoy.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sony.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-speedlink.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-steelseries.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sunplus.ko 2023-12-19T14:24:08.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-thrustmaster.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tivo.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tmff.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-topseed.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-twinhan.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-uclogic.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-waltop.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 143473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-wiimote.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-xinmo.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zpff.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zydacron.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 267825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid.ko 2023-12-19T14:24:08.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/uhid.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/ 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbhid.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbkbd.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbmouse.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 217873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/wacom.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/ 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_balloon.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_utils.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 305465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_vmbus.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru3.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-12-19T14:24:08.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7414.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7418.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adc128d818.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1025.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1026.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1029.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1031.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm9240.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ads7828.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7410.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7411.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7462.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7470.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7475.ko 2023-12-19T14:24:08.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7x10.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/amc6821.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/applesmc.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asb100.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asc7621.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asus_atk0110.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/atxp1.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/coretemp.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9052-hwmon.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9055-hwmon.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dme1737.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds1621.ko 2023-12-19T14:24:08.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds620.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc1403.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc2103.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc6w201.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71805f.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71882fg.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f75375s.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fam15h_power.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fschmd.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g760a.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g762.ko 2023-12-19T14:24:08.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl518sm.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl520sm.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hih6130.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hwmon-vid.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5500_temp.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5k_amb.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmaem.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmpex.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina209.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina2xx.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 130449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/it87.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/jc42.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k10temp.ko 2023-12-19T14:24:08.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k8temp.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lineage-pem.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm63.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm73.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm75.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm77.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm78.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm80.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm83.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm85.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm87.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm90.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm92.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm93.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95234.ko 2023-12-19T14:24:08.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95241.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95245.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc2945.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4151.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4215.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4222.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4245.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4260.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4261.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max16065.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1619.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1668.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max197.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6639.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6650.ko 2023-12-19T14:24:08.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6697.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mc13783-adc.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mcp3021.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6683.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775-core.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct7802.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87360.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87427.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pcf8591.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-12-19T14:24:08.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/powr1220.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5627.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5636.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch56xx-common.ko 2023-12-19T14:24:08.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht15.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht21.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/shtc1.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sis5595.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47b397.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m1.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m192.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/thmc50.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp102.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp103.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp401.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp421.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via-cputemp.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via686a.ko 2023-12-19T14:24:08.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt1211.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt8231.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627ehf.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627hf.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83781d.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83791d.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83792d.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83793.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83795.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l785ts.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l786ng.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/ 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/ 2023-12-19T14:24:08.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/ 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-12-19T14:24:08.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-12-19T14:24:08.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-via.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-mux.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-smbus.ko 2023-12-19T14:24:08.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-stub.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/ 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 273609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_cm.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 879969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_core.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_umad.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 324217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/iw_cm.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 282273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ 2023-12-19T14:24:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 493073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 419737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 739025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 296625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 153697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 168969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ 2023-12-19T14:24:08.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 111425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 128753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 129529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/evbug.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ff-memless.ko 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/ 2023-12-19T14:24:08.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/fm801-gp.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/gameport.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/joydev.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/ 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lkkbd.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8323.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8333.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-12-19T14:24:08.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/newtonkbd.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt1070.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt2160.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/stowaway.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/sunkbd.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/xtkbd.ko 2023-12-19T14:24:08.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/matrix-keymap.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/ 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/appletouch.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/bcm5974.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/cyapatp.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/elan_i2c.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/gpio_mouse.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 335929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/psmouse.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/sermouse.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/ 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/rmi_core.ko 2023-12-19T14:24:08.861 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/altera_ps2.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/arc_ps2.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ct82c710.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/parkbd.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/pcips2.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ps2mult.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serio_raw.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serport.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/sparse-keymap.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/ 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/ 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/ 2023-12-19T14:24:08.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-88pm860x.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-adp5520.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-bd2802.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-blinkm.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da903x.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da9052.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-gpio.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3530.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3533.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm355x.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3642.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp3944.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp8788.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-max8997.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-mc13783.ko 2023-12-19T14:24:08.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-menf21bmc.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca9532.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca955x.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca963x.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-ss4200.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-tca6507.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm831x-status.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm8350.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-12-19T14:24:08.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/ 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/ 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 654969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/bcache.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bio-prison.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bufio.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache-smq.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 156009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-crypt.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-delay.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-era.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-flakey.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log-userspace.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log.ko 2023-12-19T14:24:08.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-mirror.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-multipath.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-queue-length.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-raid.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-region-hash.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-round-robin.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-service-time.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-snapshot.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-switch.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 181865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-thin-pool.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-verity.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-zero.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/faulty.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/linear.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/multipath.ko 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/ 2023-12-19T14:24:08.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 199937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid0.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid1.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 158681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid10.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 395721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid456.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/ 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/ 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 178137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptbase.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptctl.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptfc.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptlan.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptsas.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 79041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptscsih.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptspi.ko 2023-12-19T14:24:08.867 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm800.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm805.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm80x.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona-i2c.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/bcm590xx.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/cros_ec_dev.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/dln2.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/janz-cmodio.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/kempld-core.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-core.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lp3943.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_ich.ko 2023-12-19T14:24:08.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_sch.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/max8907.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-core.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/menf21bmc.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-adc.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-gpio.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/retu-mfd.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/si476x-core.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/sm501.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps65010.ko 2023-12-19T14:24:08.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps6507x.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/viperboard.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/vx855.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wl1273-core.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wm8994.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/ 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds9802als.ko 2023-12-19T14:24:08.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds990x.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/bh1770glc.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/ 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/core.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/ 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/cb710.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ds1682.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/dummy-irq.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/ 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/echo.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/ 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/at24.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-12-19T14:24:08.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/max6875.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/enclosure.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/ 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 152201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hmc6352.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hpilo.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ics932s401.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29003.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29020.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/ 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/ 2023-12-19T14:24:08.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-me.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-txe.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 310113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/phantom.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/ 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/st_drv.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_7xx1.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_core.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tsl2550.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_balloon.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/ 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 183609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/ 2023-12-19T14:24:08.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 490129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/bonding.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/ 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/ 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6060.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/ 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 453737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dummy.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/eql.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/ 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 130553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-12-19T14:24:08.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/8390.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/ 2023-12-19T14:24:08.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/ 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/ 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 104673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/ 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-12-19T14:24:08.876 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 176761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1421497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 128569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 132089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 312081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 323513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-12-19T14:24:08.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 193497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 347417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1044497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/ 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 204345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-12-19T14:24:08.878 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/ 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/ 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 141225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/ 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dnet.ko 2023-12-19T14:24:08.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ec_bhf.ko 2023-12-19T14:24:08.880 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/ 2023-12-19T14:24:08.880 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/ 2023-12-19T14:24:08.880 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 281137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ethoc.ko 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fealnx.ko 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/ 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e100.ko 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/ 2023-12-19T14:24:08.964 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 295577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 584497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 246209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 925825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 302937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 514641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/ 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-12-19T14:24:08.965 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 734305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 151769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 86921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/jme.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 727601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 278937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-12-19T14:24:08.966 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1113681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 95665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/ 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-12-19T14:24:08.967 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 150905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 136721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 227897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 672049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-12-19T14:24:08.968 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 189801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 104241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1052497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/ 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-12-19T14:24:08.969 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/ 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/ 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/ 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 487025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/ 2023-12-19T14:24:08.970 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/niu.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/ 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/ 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 86529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/ 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/ 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-12-19T14:24:08.971 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/defxx.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 216985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/skfp.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/geneve.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 235521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/fakelb.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ifb.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvlan.ko 2023-12-19T14:24:08.972 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvtap.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/ 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-gpio.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-regmap.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mii.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/netconsole.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/nlmon.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/ 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs-lynx.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ 2023-12-19T14:24:08.973 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/amd.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm7xxx.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm87xx.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/broadcom.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/cicada.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/davicom.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/et1011c.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/icplus.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/lxt.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/marvell.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/micrel.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/national.ko 2023-12-19T14:24:08.974 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/phylink.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/qsemi.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/realtek.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/smsc.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ste10Xp.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/vitesse.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/ 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/plip.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/bsd_comp.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_async.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppoe.ko 2023-12-19T14:24:08.975 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppox.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pptp.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sb1000.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/ 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/slip.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sungem_phy.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/tap.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/ 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_random.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/veth.ko 2023-12-19T14:24:08.976 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 297329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/vxlan.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ntb.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 265121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme-core.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/ 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 133841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport.ko 2023-12-19T14:24:08.977 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_cs.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_pc.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_serial.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/ 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/ 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/pci-stub.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/xen-pcifront.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/ 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/i82092.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 145809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_core.ko 2023-12-19T14:24:08.978 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pd6729.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/yenta_socket.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/ 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-12-19T14:24:08.979 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acer-wmi.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acerhdf.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/apple-gmux.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-laptop.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 127441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-wmi.ko 2023-12-19T14:24:08.980 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/hdaps.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ibm_rtl.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/ 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel_ips.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/msi-wmi.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-12-19T14:24:08.981 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-q10.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/ 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_battery.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_charger.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq2415x_charger.ko 2023-12-19T14:24:08.982 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24190_charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24735-charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9030_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9052-battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2760_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2780_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2781_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2782_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/gpio-charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/isp1704_charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/lp8727_charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max14577_charger.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17040_battery.ko 2023-12-19T14:24:08.983 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17042_battery.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8903_charger.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8925_power.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/pcf50633-charger.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/sbs-battery.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/test_power.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/tps65090-charger.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_backup.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_power.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm8350_power.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/ 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_common.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-12-19T14:24:08.984 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/ 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/ 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-gpio.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps_parport.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/pps_core.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp_kvm.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/ 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm80x.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm860x.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-bq32k.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9052.ko 2023-12-19T14:24:08.985 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9055.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9063.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1286.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1307.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1374.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1511.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1553.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1672.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1742.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds2404.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds3232.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-em3027.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-fm3130.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl12022.ko 2023-12-19T14:24:08.986 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl1208.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-lp8788.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m41t80.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t35.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t59.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t86.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max6900.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8907.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8925.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8997.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8998.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-msm6242.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-palmas.ko 2023-12-19T14:24:08.987 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf50633.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rc5t583.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8025.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8581.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-s35390a.ko 2023-12-19T14:24:08.988 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps6586x.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps65910.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm831x.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm8350.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-x1205.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-9xxx.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-sas.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-xxxx.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 135185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/BusLogic.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/a100u2w.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/ 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 272305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-12-19T14:24:08.989 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/advansys.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/ 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 356201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 295801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/ 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 189185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/am53c974.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/ 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/atp870u.ko 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/ 2023-12-19T14:24:08.990 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 246985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1171945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/bfa.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 237273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ch.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 308921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-12-19T14:24:08.991 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 146689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dc395x.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dmx3191d.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/ 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 212425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esp_scsi.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/ 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 158505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fdomain.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/ 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/fnic.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 201817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hpsa.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hptiop.ko 2023-12-19T14:24:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hv_storvsc.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/imm.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/initio.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 266537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ipr.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ips.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/ 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 317153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/isci.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_tcp.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/ 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 314865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/libfc.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 145585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/ 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 216665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/libsas.ko 2023-12-19T14:24:08.993 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/ 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2197249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/ 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 310617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/ 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 659825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/ 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvumi.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/ 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-12-19T14:24:08.994 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/ 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 364105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pmcraid.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ppa.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla1280.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/ 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1876689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/ 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 559033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qlogicfas408.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/raid_class.ko 2023-12-19T14:24:08.995 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 238761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_debug.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 173937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 279313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ses.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/st.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/stex.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/ 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 162121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/virtio_scsi.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/wd719x.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/xen-scsifront.ko 2023-12-19T14:24:08.996 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 129441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ssb.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 736601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/tcm_loop.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_file.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_iblock.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 930257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_mod.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_pscsi.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_user.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/ 2023-12-19T14:24:08.997 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/ 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/ 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-12-19T14:24:08.998 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/moxa.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/mxser.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_gsm.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_hdlc.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/nozomi.ko 2023-12-19T14:24:08.999 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/ 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/ 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_uart.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/arc_uart.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/ 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/rp2.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/sc16is7xx.ko 2023-12-19T14:24:09.000 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/ 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 531265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/ 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_aec.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_cif.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_dmem_genirq.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_mf624.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_netx.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pci_generic.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-12-19T14:24:09.001 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_sercos3.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/ 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/ 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/c67x00.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/ 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-acm.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-wdm.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usblp.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usbtmc.ko 2023-12-19T14:24:09.002 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ulpi.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/ 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/ 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 448505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/ 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/ 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_ether.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_serial.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-12-19T14:24:09.003 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 92489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-12-19T14:24:09.004 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/ 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-12-19T14:24:09.005 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 161225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/libcomposite.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/ 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/ 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 157681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2272.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2280.ko 2023-12-19T14:24:09.006 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 65809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 156073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/bcma-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/isp116x-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811_cs.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ssb-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/ 2023-12-19T14:24:09.007 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/mdc800.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/microtek.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/adutux.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/appledisplay.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cytherm.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ehset.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi26.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi62.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ezusb.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/idmouse.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/iowarrior.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/isight_firmware.ko 2023-12-19T14:24:09.008 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ldusb.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/legousbtower.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/lvstest.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/ 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/trancevibrator.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usb3503.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usblcd.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbsevseg.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbtest.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/uss720.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/yurex.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/ 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/usbmon.ko 2023-12-19T14:24:09.009 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/ 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 301697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/musb_hdrc.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/ 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-generic.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-isp1301.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-tahvo.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/aircable.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ark3116.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/belkin_sa.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ch341.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cp210x.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cyberjack.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cypress_m8.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-12-19T14:24:09.010 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/empeg.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/f81232.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 175137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/garmin_gps.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 137081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_edgeport.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_ti.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipaq.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipw.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ir-usb.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kobil_sct.ko 2023-12-19T14:24:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mct_u232.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/metro-usb.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7720.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7840.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mxuport.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/navman.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/omninet.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/opticon.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 191137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/option.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/oti6858.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/pl2303.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcaux.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcserial.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/quatech2.ko 2023-12-19T14:24:09.012 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/safe_serial.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/sierra.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/spcp8x5.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ssu100.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/symbolserial.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_debug.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_wwan.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usbserial.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/visor.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/whiteheat.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/wishbone-serial.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/xsens_mt.ko 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ 2023-12-19T14:24:09.013 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/uas.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-alauda.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-cypress.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-datafab.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-freecom.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-isd200.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-karma.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-realtek.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-12-19T14:24:09.014 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-usbat.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 179969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/usb-storage.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/ 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-core.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-host.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/ 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/ 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 154297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/ 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost.ko 2023-12-19T14:24:09.015 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_iotlb.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_net.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_scsi.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/ 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/ 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/aat2870_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp5520_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8860_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8870_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/apple_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/as3711_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/bd6107.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da903x_bl.ko 2023-12-19T14:24:09.016 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da9052_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/gpio_backlight.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/kb3886_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lcd.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3533_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3639_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lv5207lp.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/max8925_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pandora_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/platform_lcd.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/wm831x_bl.ko 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/ 2023-12-19T14:24:09.017 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2482.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2490.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/matrox_w1.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/w1-gpio.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/ 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-12-19T14:24:09.018 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_smem.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_therm.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/wire.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/acquirewdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/advantechwdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/cpu5wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9052_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9055_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9063_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/dw_wdt.ko 2023-12-19T14:24:09.019 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/eurotechwdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/hpwdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/i6300esb.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ib700wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ibmasr.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it87_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/kempld_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/machzwd.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/mena21_wdt.ko 2023-12-19T14:24:09.020 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/nv_tco.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pc87413_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_pci.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_usb.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/retu_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sc1200wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-12-19T14:24:09.021 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/softdog.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sp5100_tco.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/twl4030_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/via_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wafer5823wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wdt_pci.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm831x_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm8350_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/xen_wdt.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/ 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-evtchn.ko 2023-12-19T14:24:09.022 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntalloc.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntdev.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/ 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 176377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-privcmd.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-scsiback.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/ 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/xenfs.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/ 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/9p.ko 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/ 2023-12-19T14:24:09.023 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/adfs.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/ 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 163593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/affs.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/ 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 932817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/kafs.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/ 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/autofs4.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/ 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/befs.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/ 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/bfs.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/binfmt_misc.ko 2023-12-19T14:24:09.024 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 3402729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/btrfs.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 306009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/cachefiles.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1333017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ceph.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/coda.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/configfs.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/cramfs.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 642513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/dlm.ko 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/ 2023-12-19T14:24:09.025 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/efs.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1505025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/f2fs.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/msdos.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/freevxfs.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 226481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/fscache.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/cuse.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/ 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1231529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/gfs2.ko 2023-12-19T14:24:09.026 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 141209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/hfs.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 222817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/hfsplus.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 192329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/hpfs.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/isofs.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 419497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/jfs.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 280313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/lockd.ko 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/ 2023-12-19T14:24:09.027 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/minix.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/ 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/netfs.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/ 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/ 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/ 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/ 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 117465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1055721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfs.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv2.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 120177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv3.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2042137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv4.ko 2023-12-19T14:24:09.028 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/ 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/grace.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/nfs_acl.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/ 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1516873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/nfsd.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/ 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 592833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/nilfs2.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/ 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-celtic.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-centeuro.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-croatian.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-cyrillic.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-gaelic.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-greek.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-iceland.ko 2023-12-19T14:24:09.029 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-inuit.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-roman.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-romanian.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-turkish.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ascii.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1250.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1251.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1255.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp737.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp775.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp850.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp852.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp855.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp857.ko 2023-12-19T14:24:09.030 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp860.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp861.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp862.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp863.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp864.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp865.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp866.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp869.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp874.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp932.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 142673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp936.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 165745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp949.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 118121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp950.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_euc-jp.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-1.ko 2023-12-19T14:24:09.031 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-13.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-14.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-15.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-2.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-3.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-4.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-5.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-6.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-7.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-9.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-r.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-ru.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-u.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ucs2_utils.ko 2023-12-19T14:24:09.032 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_utf8.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 219417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ntfs.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 244409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 440737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2592433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/ 2023-12-19T14:24:09.033 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/omfs.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 352161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/overlay.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ramoops.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/qnx4.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/qnx6.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_tree.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v1.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v2.ko 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/ 2023-12-19T14:24:09.034 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 532673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/reiserfs.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/romfs.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2668401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/cifs.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_arc4.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_md4.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 134641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/squashfs.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/sysv.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/ 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 247737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/udf.ko 2023-12-19T14:24:09.035 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ 2023-12-19T14:24:09.036 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 162497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ufs.ko 2023-12-19T14:24:09.036 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/ 2023-12-19T14:24:09.036 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4011017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/xfs.ko 2023-12-19T14:24:09.076 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/ 2023-12-19T14:24:09.076 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/ 2023-12-19T14:24:09.076 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/locktorture.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/ 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/test_udelay.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/torture.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc-itu-t.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc7.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc8.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/ 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libarc4.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libdes.ko 2023-12-19T14:24:09.077 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/interval_tree_test.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/libcrc32c.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lru_cache.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/ 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4_compress.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4hc_compress.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/ 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/cordic.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/memory-notifier-error-inject.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/notifier-error-inject.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/percpu_test.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/pm-notifier-error-inject.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/ 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 143433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/raid6_pq.ko 2023-12-19T14:24:09.078 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/rbtree_test.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/ 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/reed_solomon.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-kstrtox.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-string_helpers.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4760425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_bpf.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_firmware.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_module.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_user_copy.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_bm.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_fsm.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_kmp.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/ 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/xz_dec_test.ko 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/ 2023-12-19T14:24:09.079 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/hwpoison-inject.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/ 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/6lowpan.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_dest.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_fragment.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_hop.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_ipv6.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_mobility.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_routing.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_udp.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/ 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/garp.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/mrp.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/p8022.ko 2023-12-19T14:24:09.080 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/psnap.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/stp.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/ 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/8021q.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/ 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 198337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_fd.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_rdma.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_virtio.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/ 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 649297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/batman-adv.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/ 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/br_netfilter.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 777689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/bridge.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-12-19T14:24:09.081 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_among.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_log.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-12-19T14:24:09.082 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtables.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/ 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1046681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/libceph.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/ 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/pktgen.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/ 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 194209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_diag.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv4.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv6.ko 2023-12-19T14:24:09.083 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/ 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 390473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/dsa_core.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_brcm.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_dsa.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_none.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_trailer.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/ 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/hsr.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 254625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154_socket.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ah4.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/esp4.ko 2023-12-19T14:24:09.084 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/fou.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/gre.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/inet_diag.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_gre.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_tunnel.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_vti.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipcomp.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipip.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ip_tables.ko 2023-12-19T14:24:09.085 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-12-19T14:24:09.086 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_bic.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_dctcp.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_diag.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_highspeed.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_htcp.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_hybla.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_illinois.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_lp.ko 2023-12-19T14:24:09.087 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_scalable.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_vegas.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_veno.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_westwood.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_yeah.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tunnel4.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_diag.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_tunnel.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ah6.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/esp6.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/fou6.ko 2023-12-19T14:24:09.088 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_gre.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_tunnel.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_vti.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ipcomp6.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/mip6.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-12-19T14:24:09.089 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-12-19T14:24:09.090 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/sit.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/tunnel6.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/ 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/af_key.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/ 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_core.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_debugfs.ko 2023-12-19T14:24:09.091 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_eth.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip6.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_netlink.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ppp.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/ 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/lapb.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/ 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 267913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc2.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/ 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 290417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/mac802154.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/ 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/mpls_gso.ko 2023-12-19T14:24:09.092 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-12-19T14:24:09.093 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 111585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 114465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 411729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-12-19T14:24:09.094 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conncount.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 408257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 118145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-12-19T14:24:09.095 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_dup_netdev.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_log_syslog.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 95353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_amanda.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_ftp.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_irc.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_sip.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_tftp.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_synproxy_core.ko 2023-12-19T14:24:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 615649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_tables.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_acct.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_log.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_osf.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_queue.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_chain_nat.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_compat.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_ct.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_dup_netdev.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fib.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_hash.ko 2023-12-19T14:24:09.097 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_limit.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_log.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_masq.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_nat.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_numgen.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_osf.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_queue.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_redir.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject_inet.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_socket.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_synproxy.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_tproxy.ko 2023-12-19T14:24:09.098 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/x_tables.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_AUDIT.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CT.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_DSCP.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HL.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HMARK.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LED.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LOG.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NETMAP.ko 2023-12-19T14:24:09.099 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFLOG.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_RATEEST.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_REDIRECT.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_SECMARK.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPMSS.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TEE.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TPROXY.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TRACE.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_addrtype.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_bpf.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cgroup.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cluster.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_comment.ko 2023-12-19T14:24:09.100 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connbytes.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlabel.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlimit.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connmark.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_conntrack.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cpu.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dccp.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_devgroup.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dscp.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ecn.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_esp.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hashlimit.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_helper.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hl.ko 2023-12-19T14:24:09.101 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipcomp.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_iprange.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipvs.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_l2tp.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_length.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_limit.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mac.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mark.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_multiport.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nat.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nfacct.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_osf.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_owner.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_physdev.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_pkttype.ko 2023-12-19T14:24:09.102 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_policy.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_quota.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_rateest.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_realm.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_recent.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_sctp.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_set.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_socket.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_state.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_statistic.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_string.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpmss.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpudp.ko 2023-12-19T14:24:09.103 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_time.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_u32.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/ 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/netlink_diag.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/ 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/nsh.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/ 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 380033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/openvswitch.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-geneve.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-gre.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-vxlan.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/ 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/af_packet_diag.ko 2023-12-19T14:24:09.104 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/ 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 110201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/phonet.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/pn_pep.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/ 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 251481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 164937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_rdma.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_tcp.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/ 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 819657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/rxrpc.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/ 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_csum.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_gact.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_ipt.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_mirred.ko 2023-12-19T14:24:09.105 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_nat.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_pedit.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_police.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_simple.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_skbedit.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_vlan.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_basic.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_bpf.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_cgroup.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_flow.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_fw.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_route.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_u32.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_cmp.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_ipset.ko 2023-12-19T14:24:09.106 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_meta.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_nbyte.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_text.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_u32.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_choke.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_codel.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_drr.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq_codel.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_gred.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hfsc.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hhf.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_htb.ko 2023-12-19T14:24:09.107 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_ingress.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio_lib.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_multiq.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_netem.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_pie.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_plug.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_prio.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_qfq.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_red.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfb.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfq.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_tbf.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_teql.ko 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/ 2023-12-19T14:24:09.108 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 929745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp_diag.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/ 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/ 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 336529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1520097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/sunrpc.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/ 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 707657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/ 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/diag.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 862665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/tipc.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/ 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/unix_diag.ko 2023-12-19T14:24:09.109 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/ 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 96193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_diag.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/ 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/x25.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/ 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_algo.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_user.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/ 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/ 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/irqbypass.ko 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 640081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias 2023-12-19T14:24:09.110 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 624107 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias.bin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7663 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23428 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.alias.bin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9821 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.bin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70004 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.modinfo 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 146313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 232548 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep.bin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 210 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.devname 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72606 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.order 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 879 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.softdep 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 254438 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 309363 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols.bin 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/ 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/ 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/ 2023-12-19T14:24:09.111 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/ 2023-12-19T14:24:09.112 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 170 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/changelog.Debian.gz 2023-12-19T14:24:09.112 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 691 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/copyright 2023-12-19T14:24:09.112 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gcd68252d7b4d 2023-12-19T14:24:09.112 INFO:teuthology.task.kernel:searching for kernel 6.6.0-gcd68252d7b4d 2023-12-19T14:24:09.112 DEBUG:teuthology.orchestra.run.smithi125:> egrep '(submenu|menuentry.*6.6.0-gcd68252d7b4d).*{' /boot/grub/grub.cfg 2023-12-19T14:24:09.121 INFO:teuthology.orchestra.run.smithi125.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-recovery-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T14:24:09.122 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-12-19T14:24:09.122 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-gcd68252d7b4d 2023-12-19T14:24:09.122 DEBUG:teuthology.orchestra.run.smithi125:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-12-19T14:24:09.166 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi125.front.sepia.ceph.com to complete... 2023-12-19T14:24:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub' 2023-12-19T14:24:09.861 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T14:24:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2023-12-19T14:24:10.114 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T14:24:10.126 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T14:24:10.465 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T14:24:10.477 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T14:24:11.265 INFO:teuthology.orchestra.run.smithi125.stderr:done 2023-12-19T14:24:38.286 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-19T14:24:38.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:24:38.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:24:42.388 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-19T14:24:42.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:24:42.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:25:17.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-12-19T14:25:26.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:25:26.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:25:29.169 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-12-19T14:25:38.297 DEBUG:teuthology.orchestra.remote:timed out 2023-12-19T14:25:41.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:25:41.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:25:47.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:25:47.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:25:50.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-12-19T14:25:59.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-12-19T14:26:02.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:26:02.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:26:02.584 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-12-19T14:26:03.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:26:03.351 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-gcd68252d7b4d"... 2023-12-19T14:26:03.351 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-19T14:26:03.418 INFO:teuthology.orchestra.run.smithi050.stdout:6.6.0-gcd68252d7b4d 2023-12-19T14:26:03.418 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gcd68252d7b4d vs 6.6.0-gcd68252d7b4d 2023-12-19T14:26:03.418 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T14:26:03.418 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-19T14:26:04.419 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-19T14:26:04.420 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T14:26:04.533 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-12-19T14:26:04.592 DEBUG:teuthology.parallel:result is None 2023-12-19T14:26:14.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:26:14.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:26:14.957 DEBUG:teuthology.orchestra.run.smithi125:> true 2023-12-19T14:26:16.366 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:26:16.366 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-gcd68252d7b4d"... 2023-12-19T14:26:16.367 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2023-12-19T14:26:16.428 INFO:teuthology.orchestra.run.smithi125.stdout:6.6.0-gcd68252d7b4d 2023-12-19T14:26:16.429 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gcd68252d7b4d vs 6.6.0-gcd68252d7b4d 2023-12-19T14:26:16.429 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T14:26:16.429 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-19T14:26:17.430 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-19T14:26:17.430 DEBUG:teuthology.orchestra.run.smithi125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T14:26:17.697 INFO:teuthology.orchestra.run.smithi125.stdout:ttyS1 2023-12-19T14:26:17.778 DEBUG:teuthology.parallel:result is None 2023-12-19T14:26:17.778 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-19T14:26:17.785 INFO:teuthology.task.internal:Creating test directory... 2023-12-19T14:26:17.786 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T14:26:17.793 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T14:26:17.809 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-19T14:26:17.897 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-19T14:26:17.904 INFO:teuthology.task.internal:Creating archive directory... 2023-12-19T14:26:17.904 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T14:26:17.912 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T14:26:17.946 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-19T14:26:17.953 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-19T14:26:17.953 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T14:26:17.989 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T14:26:18.026 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T14:26:18.035 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T14:26:18.051 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T14:26:18.060 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T14:26:18.064 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-19T14:26:18.071 INFO:teuthology.task.internal:Configuring sudo... 2023-12-19T14:26:18.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T14:26:18.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T14:26:18.138 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-19T14:26:18.148 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-19T14:26:18.148 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T14:26:18.189 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T14:26:18.206 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T14:26:18.268 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T14:26:18.328 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:26:18.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T14:26:18.416 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T14:26:18.431 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T14:26:18.490 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:26:18.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T14:26:18.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-12-19T14:26:18.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo service rsyslog restart 2023-12-19T14:26:18.784 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-19T14:26:18.792 INFO:teuthology.task.internal:Starting timer... 2023-12-19T14:26:18.792 INFO:teuthology.run_tasks:Running task pcp... 2023-12-19T14:26:18.802 INFO:teuthology.run_tasks:Running task selinux... 2023-12-19T14:26:18.811 DEBUG:teuthology.task.selinux:Excluding smithi050: OS 'ubuntu' does not support SELinux 2023-12-19T14:26:18.811 DEBUG:teuthology.task.selinux:Excluding smithi125: OS 'ubuntu' does not support SELinux 2023-12-19T14:26:18.811 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-19T14:26:18.811 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-19T14:26:18.811 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-19T14:26:18.811 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-19T14:26:18.820 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-19T14:26:18.821 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-19T14:26:18.838 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-19T14:26:18.840 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi125.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-19T14:34:33.092 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi125.front.sepia.ceph.com')] 2023-12-19T14:34:33.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:34:33.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:34:33.202 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-12-19T14:34:33.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-19T14:34:33.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:34:33.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T14:34:33.547 DEBUG:teuthology.orchestra.run.smithi125:> true 2023-12-19T14:34:33.778 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-12-19T14:34:33.779 INFO:teuthology.run_tasks:Running task clock... 2023-12-19T14:34:33.792 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-19T14:34:33.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T14:34:33.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T14:34:33.800 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T14:34:33.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T14:34:33.868 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-19T14:34:33.868 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Command line: ntpd -gq 2023-12-19T14:34:33.869 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: proto: precision = 0.057 usec (-24) 2023-12-19T14:34:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T14:34:33.870 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T14:34:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-19T14:34:33.871 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: restrict ::: KOD does nothing without LIMITED. 2023-12-19T14:34:33.871 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen and drop on 0 v6wildcard [::]:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen normally on 2 lo 127.0.0.1:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen normally on 3 ens1f0 172.21.15.50:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen normally on 4 lo [::1]:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1584%4]:123 2023-12-19T14:34:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:33 ntpd[23358]: Listening on routing socket on fd #22 for interface updates 2023-12-19T14:34:33.897 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-19T14:34:33.897 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Command line: ntpd -gq 2023-12-19T14:34:33.897 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: proto: precision = 0.041 usec (-24) 2023-12-19T14:34:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T14:34:33.898 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T14:34:33.898 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: restrict ::: KOD does nothing without LIMITED. 2023-12-19T14:34:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-19T14:34:33.899 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen and drop on 0 v6wildcard [::]:123 2023-12-19T14:34:33.899 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-19T14:34:33.900 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen normally on 2 lo 127.0.0.1:123 2023-12-19T14:34:33.900 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen normally on 3 enp3s0f1 172.21.15.125:123 2023-12-19T14:34:33.900 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen normally on 4 lo [::1]:123 2023-12-19T14:34:33.900 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d739%5]:123 2023-12-19T14:34:33.900 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:33 ntpd[23435]: Listening on routing socket on fd #22 for interface updates 2023-12-19T14:34:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:34.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:34 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:34.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:34 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:34.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:34 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:35.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:35.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:35 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:36.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:36.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:36.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:36.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:36 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:36.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:36.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:36.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:36.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:36.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:36.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:36 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:37.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:37.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:37 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:37.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:37 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:38.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:38 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:38.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:38.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:38 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:39.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:39 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:39 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:40.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.869 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:40.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:40.870 INFO:teuthology.orchestra.run.smithi050.stdout:19 Dec 14:34:40 ntpd[23358]: ntpd: time slew -0.000198 s 2023-12-19T14:34:40.870 INFO:teuthology.orchestra.run.smithi050.stdout:ntpd: time slew -0.000198s 2023-12-19T14:34:40.870 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-19T14:34:40.871 INFO:teuthology.orchestra.run.smithi050.stderr:19 Dec 14:34:40 ntpd[23358]: can't open /var/log/ntpstats/loopstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:40.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:40.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:40 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:41 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stdout:19 Dec 14:34:42 ntpd[23435]: ntpd: time slew +0.001469 s 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stdout:ntpd: time slew +0.001469s 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-19T14:34:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:19 Dec 14:34:42 ntpd[23435]: can't open /var/log/ntpstats/loopstats.20231219: Permission denied 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================== 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:43.040 INFO:teuthology.orchestra.run.smithi125.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T14:34:43.041 INFO:teuthology.run_tasks:Running task install... 2023-12-19T14:34:43.050 DEBUG:teuthology.task.install:project ceph 2023-12-19T14:34:43.050 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}} 2023-12-19T14:34:43.050 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'} 2023-12-19T14:34:43.050 INFO:teuthology.task.install:Using flavor: default 2023-12-19T14:34:43.056 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-19T14:34:43.056 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-12-19T14:34:43.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-key list | grep Ceph 2023-12-19T14:34:43.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-key list | grep Ceph 2023-12-19T14:34:43.292 INFO:teuthology.orchestra.run.smithi125.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-19T14:34:43.318 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-19T14:34:43.438 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-19T14:34:43.438 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph.com (release key) 2023-12-19T14:34:43.440 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-19T14:34:43.440 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-12-19T14:34:43.441 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T14:34:43.474 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-19T14:34:43.474 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph.com (release key) 2023-12-19T14:34:43.474 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-19T14:34:43.474 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-12-19T14:34:43.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T14:34:43.582 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default/ 2023-12-19T14:34:43.583 INFO:teuthology.task.install.deb:Package version is 17.2.7-141-gc7312db0-1focal 2023-12-19T14:34:43.619 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default/ 2023-12-19T14:34:43.619 INFO:teuthology.task.install.deb:Package version is 17.2.7-141-gc7312db0-1focal 2023-12-19T14:34:43.750 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:34:43.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-19T14:34:43.784 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:34:43.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-19T14:34:43.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get update 2023-12-19T14:34:43.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2023-12-19T14:34:44.201 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-19T14:34:44.202 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-19T14:34:44.202 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal InRelease 2023-12-19T14:34:44.241 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-12-19T14:34:44.242 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-19T14:34:44.242 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-19T14:34:44.242 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal InRelease 2023-12-19T14:34:44.257 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-19T14:34:44.279 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release.gpg 2023-12-19T14:34:44.280 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-12-19T14:34:44.297 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-19T14:34:44.335 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main i386 Packages [3,368 B] 2023-12-19T14:34:44.336 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release.gpg 2023-12-19T14:34:44.364 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-19T14:34:44.373 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main i386 Packages [3,368 B] 2023-12-19T14:34:44.392 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-19T14:34:44.425 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-12-19T14:34:44.429 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-12-19T14:34:45.618 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 28.4 kB in 1s (19.2 kB/s) 2023-12-19T14:34:45.672 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 28.4 kB in 2s (18.8 kB/s) 2023-12-19T14:34:46.860 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-12-19T14:34:46.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-141-gc7312db0-1focal cephadm=17.2.7-141-gc7312db0-1focal ceph-mds=17.2.7-141-gc7312db0-1focal ceph-mgr=17.2.7-141-gc7312db0-1focal ceph-common=17.2.7-141-gc7312db0-1focal ceph-fuse=17.2.7-141-gc7312db0-1focal ceph-test=17.2.7-141-gc7312db0-1focal ceph-volume=17.2.7-141-gc7312db0-1focal radosgw=17.2.7-141-gc7312db0-1focal python3-rados=17.2.7-141-gc7312db0-1focal python3-rgw=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal python3-rbd=17.2.7-141-gc7312db0-1focal libcephfs2=17.2.7-141-gc7312db0-1focal libcephfs-dev=17.2.7-141-gc7312db0-1focal librados2=17.2.7-141-gc7312db0-1focal librbd1=17.2.7-141-gc7312db0-1focal rbd-fuse=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal cephfs-shell=17.2.7-141-gc7312db0-1focal cephfs-top=17.2.7-141-gc7312db0-1focal cephfs-mirror=17.2.7-141-gc7312db0-1focal 2023-12-19T14:34:46.943 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2023-12-19T14:34:46.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-141-gc7312db0-1focal cephadm=17.2.7-141-gc7312db0-1focal ceph-mds=17.2.7-141-gc7312db0-1focal ceph-mgr=17.2.7-141-gc7312db0-1focal ceph-common=17.2.7-141-gc7312db0-1focal ceph-fuse=17.2.7-141-gc7312db0-1focal ceph-test=17.2.7-141-gc7312db0-1focal ceph-volume=17.2.7-141-gc7312db0-1focal radosgw=17.2.7-141-gc7312db0-1focal python3-rados=17.2.7-141-gc7312db0-1focal python3-rgw=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal python3-rbd=17.2.7-141-gc7312db0-1focal libcephfs2=17.2.7-141-gc7312db0-1focal libcephfs-dev=17.2.7-141-gc7312db0-1focal librados2=17.2.7-141-gc7312db0-1focal librbd1=17.2.7-141-gc7312db0-1focal rbd-fuse=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal cephfs-shell=17.2.7-141-gc7312db0-1focal cephfs-top=17.2.7-141-gc7312db0-1focal cephfs-mirror=17.2.7-141-gc7312db0-1focal 2023-12-19T14:34:47.029 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-12-19T14:34:47.117 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2023-12-19T14:34:47.230 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2023-12-19T14:34:47.231 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2023-12-19T14:34:47.325 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2023-12-19T14:34:47.326 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2023-12-19T14:34:47.426 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-12-19T14:34:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-12-19T14:34:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T14:34:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: qttranslations5-l10n socat xmlstarlet 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-19T14:34:47.431 INFO:teuthology.orchestra.run.smithi125.stdout:Recommended packages: 2023-12-19T14:34:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: btrfs-tools 2023-12-19T14:34:47.525 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-12-19T14:34:47.527 INFO:teuthology.orchestra.run.smithi050.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-12-19T14:34:47.528 INFO:teuthology.orchestra.run.smithi050.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-12-19T14:34:47.528 INFO:teuthology.orchestra.run.smithi050.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T14:34:47.529 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: qttranslations5-l10n socat xmlstarlet 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-19T14:34:47.530 INFO:teuthology.orchestra.run.smithi050.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-19T14:34:47.531 INFO:teuthology.orchestra.run.smithi050.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-19T14:34:47.531 INFO:teuthology.orchestra.run.smithi050.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-19T14:34:47.531 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2023-12-19T14:34:47.531 INFO:teuthology.orchestra.run.smithi050.stdout: btrfs-tools 2023-12-19T14:34:47.636 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-12-19T14:34:47.637 INFO:teuthology.orchestra.run.smithi125.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T14:34:47.638 INFO:teuthology.orchestra.run.smithi125.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T14:34:47.639 INFO:teuthology.orchestra.run.smithi125.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-12-19T14:34:47.640 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be upgraded: 2023-12-19T14:34:47.640 INFO:teuthology.orchestra.run.smithi125.stdout: librados2 librbd1 2023-12-19T14:34:47.735 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2023-12-19T14:34:47.735 INFO:teuthology.orchestra.run.smithi050.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-12-19T14:34:47.736 INFO:teuthology.orchestra.run.smithi050.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-12-19T14:34:47.737 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2023-12-19T14:34:47.738 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 librbd1 2023-12-19T14:34:47.858 INFO:teuthology.orchestra.run.smithi125.stdout:2 upgraded, 114 newly installed, 0 to remove and 262 not upgraded. 2023-12-19T14:34:47.859 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 210 MB of archives. 2023-12-19T14:34:47.859 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 982 MB of additional disk space will be used. 2023-12-19T14:34:47.859 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-12-19T14:34:47.943 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-19T14:34:47.957 INFO:teuthology.orchestra.run.smithi050.stdout:2 upgraded, 114 newly installed, 0 to remove and 262 not upgraded. 2023-12-19T14:34:47.957 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 210 MB of archives. 2023-12-19T14:34:47.958 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 982 MB of additional disk space will be used. 2023-12-19T14:34:47.958 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-12-19T14:34:47.978 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-12-19T14:34:48.042 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-19T14:34:48.077 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-12-19T14:34:48.117 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-19T14:34:48.155 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-141-gc7312db0-1focal [3,501 kB] 2023-12-19T14:34:48.216 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-19T14:34:48.253 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-19T14:34:48.255 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-141-gc7312db0-1focal [3,501 kB] 2023-12-19T14:34:48.263 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-19T14:34:48.308 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-19T14:34:48.319 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-19T14:34:48.352 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-19T14:34:48.362 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-19T14:34:48.404 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-12-19T14:34:48.406 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-19T14:34:48.408 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-12-19T14:34:48.417 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-19T14:34:48.431 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-12-19T14:34:48.502 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-12-19T14:34:48.507 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-12-19T14:34:48.529 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-12-19T14:34:48.723 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-12-19T14:34:48.821 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-12-19T14:34:48.903 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-141-gc7312db0-1focal [3,553 kB] 2023-12-19T14:34:49.041 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-141-gc7312db0-1focal [759 kB] 2023-12-19T14:34:49.073 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-141-gc7312db0-1focal [414 kB] 2023-12-19T14:34:49.141 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-141-gc7312db0-1focal [22.6 kB] 2023-12-19T14:34:49.142 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-141-gc7312db0-1focal [200 kB] 2023-12-19T14:34:49.160 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-141-gc7312db0-1focal [47.2 kB] 2023-12-19T14:34:49.162 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-141-gc7312db0-1focal [386 kB] 2023-12-19T14:34:49.173 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-12-19T14:34:49.219 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-141-gc7312db0-1focal [5,578 kB] 2023-12-19T14:34:49.269 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-12-19T14:34:49.357 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-12-19T14:34:49.388 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-19T14:34:49.394 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-19T14:34:49.419 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-19T14:34:49.420 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-141-gc7312db0-1focal [116 kB] 2023-12-19T14:34:49.422 INFO:teuthology.orchestra.run.smithi125.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-141-gc7312db0-1focal [479 kB] 2023-12-19T14:34:49.423 INFO:teuthology.orchestra.run.smithi125.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-19T14:34:49.425 INFO:teuthology.orchestra.run.smithi125.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-19T14:34:49.437 INFO:teuthology.orchestra.run.smithi125.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-141-gc7312db0-1focal [27.8 MB] 2023-12-19T14:34:49.439 INFO:teuthology.orchestra.run.smithi125.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-12-19T14:34:49.441 INFO:teuthology.orchestra.run.smithi125.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-19T14:34:49.443 INFO:teuthology.orchestra.run.smithi125.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-19T14:34:49.452 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-12-19T14:34:49.484 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-19T14:34:49.488 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-19T14:34:49.514 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-19T14:34:49.518 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-19T14:34:49.521 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-19T14:34:49.535 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-12-19T14:34:49.536 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-19T14:34:49.538 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-19T14:34:49.570 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-141-gc7312db0-1focal [3,553 kB] 2023-12-19T14:34:49.608 INFO:teuthology.orchestra.run.smithi125.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-19T14:34:49.704 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-19T14:34:49.773 INFO:teuthology.orchestra.run.smithi125.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-19T14:34:49.870 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-19T14:34:49.927 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-141-gc7312db0-1focal [759 kB] 2023-12-19T14:34:49.937 INFO:teuthology.orchestra.run.smithi125.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-19T14:34:49.948 INFO:teuthology.orchestra.run.smithi125.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-19T14:34:49.950 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-141-gc7312db0-1focal [414 kB] 2023-12-19T14:34:49.955 INFO:teuthology.orchestra.run.smithi125.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-19T14:34:50.013 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-141-gc7312db0-1focal [22.6 kB] 2023-12-19T14:34:50.013 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-141-gc7312db0-1focal [200 kB] 2023-12-19T14:34:50.020 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-141-gc7312db0-1focal [47.2 kB] 2023-12-19T14:34:50.021 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-141-gc7312db0-1focal [386 kB] 2023-12-19T14:34:50.032 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-19T14:34:50.033 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-141-gc7312db0-1focal [5,578 kB] 2023-12-19T14:34:50.036 INFO:teuthology.orchestra.run.smithi125.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-19T14:34:50.044 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-19T14:34:50.051 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-19T14:34:50.056 INFO:teuthology.orchestra.run.smithi125.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-19T14:34:50.099 INFO:teuthology.orchestra.run.smithi125.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-19T14:34:50.126 INFO:teuthology.orchestra.run.smithi125.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-19T14:34:50.131 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-19T14:34:50.131 INFO:teuthology.orchestra.run.smithi125.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-19T14:34:50.151 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-19T14:34:50.194 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-19T14:34:50.221 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-19T14:34:50.225 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-19T14:34:50.228 INFO:teuthology.orchestra.run.smithi125.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-19T14:34:50.322 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-19T14:34:50.393 INFO:teuthology.orchestra.run.smithi125.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-19T14:34:50.408 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-141-gc7312db0-1focal [116 kB] 2023-12-19T14:34:50.410 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-141-gc7312db0-1focal [479 kB] 2023-12-19T14:34:50.458 INFO:teuthology.orchestra.run.smithi050.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-141-gc7312db0-1focal [27.8 MB] 2023-12-19T14:34:50.486 INFO:teuthology.orchestra.run.smithi050.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-19T14:34:50.579 INFO:teuthology.orchestra.run.smithi125.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-19T14:34:50.626 INFO:teuthology.orchestra.run.smithi125.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-19T14:34:50.674 INFO:teuthology.orchestra.run.smithi050.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-19T14:34:50.721 INFO:teuthology.orchestra.run.smithi050.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-19T14:34:50.789 INFO:teuthology.orchestra.run.smithi125.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-19T14:34:50.884 INFO:teuthology.orchestra.run.smithi050.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-19T14:34:50.908 INFO:teuthology.orchestra.run.smithi125.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-19T14:34:50.938 INFO:teuthology.orchestra.run.smithi125.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-19T14:34:50.942 INFO:teuthology.orchestra.run.smithi125.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-19T14:34:50.970 INFO:teuthology.orchestra.run.smithi125.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-19T14:34:50.991 INFO:teuthology.orchestra.run.smithi125.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-19T14:34:51.002 INFO:teuthology.orchestra.run.smithi050.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-19T14:34:51.007 INFO:teuthology.orchestra.run.smithi125.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-19T14:34:51.009 INFO:teuthology.orchestra.run.smithi125.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-19T14:34:51.032 INFO:teuthology.orchestra.run.smithi050.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-19T14:34:51.036 INFO:teuthology.orchestra.run.smithi050.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-19T14:34:51.064 INFO:teuthology.orchestra.run.smithi050.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-19T14:34:51.085 INFO:teuthology.orchestra.run.smithi050.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-19T14:34:51.101 INFO:teuthology.orchestra.run.smithi050.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-19T14:34:51.103 INFO:teuthology.orchestra.run.smithi050.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-19T14:34:51.178 INFO:teuthology.orchestra.run.smithi125.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-19T14:34:51.236 INFO:teuthology.orchestra.run.smithi125.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-141-gc7312db0-1focal [6,455 kB] 2023-12-19T14:34:51.272 INFO:teuthology.orchestra.run.smithi050.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-19T14:34:51.804 INFO:teuthology.orchestra.run.smithi125.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-141-gc7312db0-1focal [221 kB] 2023-12-19T14:34:51.807 INFO:teuthology.orchestra.run.smithi125.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-141-gc7312db0-1focal [152 kB] 2023-12-19T14:34:51.813 INFO:teuthology.orchestra.run.smithi125.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-141-gc7312db0-1focal [1,590 kB] 2023-12-19T14:34:51.974 INFO:teuthology.orchestra.run.smithi125.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-141-gc7312db0-1focal [7,850 kB] 2023-12-19T14:34:52.170 INFO:teuthology.orchestra.run.smithi125.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-19T14:34:52.191 INFO:teuthology.orchestra.run.smithi125.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-19T14:34:52.193 INFO:teuthology.orchestra.run.smithi125.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-19T14:34:52.264 INFO:teuthology.orchestra.run.smithi050.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-19T14:34:52.268 INFO:teuthology.orchestra.run.smithi125.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-19T14:34:52.269 INFO:teuthology.orchestra.run.smithi125.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-19T14:34:52.271 INFO:teuthology.orchestra.run.smithi125.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-19T14:34:52.273 INFO:teuthology.orchestra.run.smithi125.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-19T14:34:52.274 INFO:teuthology.orchestra.run.smithi125.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-19T14:34:52.276 INFO:teuthology.orchestra.run.smithi125.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-19T14:34:52.277 INFO:teuthology.orchestra.run.smithi125.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-19T14:34:52.292 INFO:teuthology.orchestra.run.smithi050.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-19T14:34:52.294 INFO:teuthology.orchestra.run.smithi050.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-19T14:34:52.360 INFO:teuthology.orchestra.run.smithi050.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-19T14:34:52.360 INFO:teuthology.orchestra.run.smithi050.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-19T14:34:52.361 INFO:teuthology.orchestra.run.smithi050.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-19T14:34:52.362 INFO:teuthology.orchestra.run.smithi050.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-19T14:34:52.362 INFO:teuthology.orchestra.run.smithi050.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-19T14:34:52.363 INFO:teuthology.orchestra.run.smithi050.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-19T14:34:52.364 INFO:teuthology.orchestra.run.smithi050.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-19T14:34:52.442 INFO:teuthology.orchestra.run.smithi125.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-19T14:34:52.530 INFO:teuthology.orchestra.run.smithi050.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-19T14:34:52.611 INFO:teuthology.orchestra.run.smithi125.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-19T14:34:52.625 INFO:teuthology.orchestra.run.smithi125.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-141-gc7312db0-1focal [28.7 MB] 2023-12-19T14:34:52.693 INFO:teuthology.orchestra.run.smithi125.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-19T14:34:52.699 INFO:teuthology.orchestra.run.smithi050.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-19T14:34:52.781 INFO:teuthology.orchestra.run.smithi050.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-19T14:34:52.942 INFO:teuthology.orchestra.run.smithi125.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-19T14:34:52.986 INFO:teuthology.orchestra.run.smithi125.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-19T14:34:53.015 INFO:teuthology.orchestra.run.smithi125.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-19T14:34:53.025 INFO:teuthology.orchestra.run.smithi125.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-19T14:34:53.034 INFO:teuthology.orchestra.run.smithi050.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-19T14:34:53.073 INFO:teuthology.orchestra.run.smithi125.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-19T14:34:53.078 INFO:teuthology.orchestra.run.smithi050.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-19T14:34:53.088 INFO:teuthology.orchestra.run.smithi050.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-141-gc7312db0-1focal [6,455 kB] 2023-12-19T14:34:53.103 INFO:teuthology.orchestra.run.smithi125.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-12-19T14:34:53.104 INFO:teuthology.orchestra.run.smithi125.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-12-19T14:34:53.109 INFO:teuthology.orchestra.run.smithi050.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-19T14:34:53.118 INFO:teuthology.orchestra.run.smithi050.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-19T14:34:53.168 INFO:teuthology.orchestra.run.smithi050.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-19T14:34:53.196 INFO:teuthology.orchestra.run.smithi050.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-12-19T14:34:53.198 INFO:teuthology.orchestra.run.smithi050.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-12-19T14:34:53.228 INFO:teuthology.orchestra.run.smithi125.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-19T14:34:53.321 INFO:teuthology.orchestra.run.smithi050.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-19T14:34:53.400 INFO:teuthology.orchestra.run.smithi125.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-12-19T14:34:53.494 INFO:teuthology.orchestra.run.smithi050.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-12-19T14:34:53.733 INFO:teuthology.orchestra.run.smithi050.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-141-gc7312db0-1focal [221 kB] 2023-12-19T14:34:53.976 INFO:teuthology.orchestra.run.smithi050.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-141-gc7312db0-1focal [152 kB] 2023-12-19T14:34:53.981 INFO:teuthology.orchestra.run.smithi050.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-141-gc7312db0-1focal [1,590 kB] 2023-12-19T14:34:54.063 INFO:teuthology.orchestra.run.smithi050.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-141-gc7312db0-1focal [7,850 kB] 2023-12-19T14:34:54.252 INFO:teuthology.orchestra.run.smithi125.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-12-19T14:34:54.255 INFO:teuthology.orchestra.run.smithi125.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-19T14:34:54.256 INFO:teuthology.orchestra.run.smithi125.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-19T14:34:54.258 INFO:teuthology.orchestra.run.smithi125.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-19T14:34:54.259 INFO:teuthology.orchestra.run.smithi125.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-19T14:34:54.262 INFO:teuthology.orchestra.run.smithi125.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-19T14:34:54.279 INFO:teuthology.orchestra.run.smithi125.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-19T14:34:54.279 INFO:teuthology.orchestra.run.smithi125.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-19T14:34:54.280 INFO:teuthology.orchestra.run.smithi125.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-19T14:34:54.332 INFO:teuthology.orchestra.run.smithi125.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-19T14:34:54.348 INFO:teuthology.orchestra.run.smithi050.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-12-19T14:34:54.353 INFO:teuthology.orchestra.run.smithi050.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-19T14:34:54.354 INFO:teuthology.orchestra.run.smithi050.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-19T14:34:54.356 INFO:teuthology.orchestra.run.smithi050.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-19T14:34:54.358 INFO:teuthology.orchestra.run.smithi050.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-19T14:34:54.360 INFO:teuthology.orchestra.run.smithi050.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-19T14:34:54.376 INFO:teuthology.orchestra.run.smithi050.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-19T14:34:54.377 INFO:teuthology.orchestra.run.smithi050.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-19T14:34:54.377 INFO:teuthology.orchestra.run.smithi050.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-19T14:34:54.429 INFO:teuthology.orchestra.run.smithi050.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-19T14:34:54.501 INFO:teuthology.orchestra.run.smithi125.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-19T14:34:54.598 INFO:teuthology.orchestra.run.smithi050.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-19T14:34:54.666 INFO:teuthology.orchestra.run.smithi050.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-141-gc7312db0-1focal [28.7 MB] 2023-12-19T14:34:54.794 INFO:teuthology.orchestra.run.smithi125.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-19T14:34:54.816 INFO:teuthology.orchestra.run.smithi125.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-19T14:34:54.884 INFO:teuthology.orchestra.run.smithi050.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-19T14:34:54.908 INFO:teuthology.orchestra.run.smithi050.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-19T14:34:54.968 INFO:teuthology.orchestra.run.smithi125.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-19T14:34:54.987 INFO:teuthology.orchestra.run.smithi125.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-12-19T14:34:55.065 INFO:teuthology.orchestra.run.smithi050.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-19T14:34:55.086 INFO:teuthology.orchestra.run.smithi050.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-12-19T14:34:55.533 INFO:teuthology.orchestra.run.smithi125.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-141-gc7312db0-1focal [3,940 B] 2023-12-19T14:34:55.534 INFO:teuthology.orchestra.run.smithi125.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-141-gc7312db0-1focal [868 kB] 2023-12-19T14:34:55.584 INFO:teuthology.orchestra.run.smithi125.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-141-gc7312db0-1focal [2,408 kB] 2023-12-19T14:34:55.816 INFO:teuthology.orchestra.run.smithi125.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-141-gc7312db0-1focal [79.2 kB] 2023-12-19T14:34:55.833 INFO:teuthology.orchestra.run.smithi125.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-141-gc7312db0-1focal [117 kB] 2023-12-19T14:34:55.872 INFO:teuthology.orchestra.run.smithi125.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-141-gc7312db0-1focal [1,760 kB] 2023-12-19T14:34:56.042 INFO:teuthology.orchestra.run.smithi125.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-141-gc7312db0-1focal [7,876 kB] 2023-12-19T14:34:56.824 INFO:teuthology.orchestra.run.smithi125.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-141-gc7312db0-1focal [4,056 B] 2023-12-19T14:34:56.824 INFO:teuthology.orchestra.run.smithi125.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-141-gc7312db0-1focal [56.3 MB] 2023-12-19T14:34:57.029 INFO:teuthology.orchestra.run.smithi050.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-141-gc7312db0-1focal [3,940 B] 2023-12-19T14:34:57.030 INFO:teuthology.orchestra.run.smithi050.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-141-gc7312db0-1focal [868 kB] 2023-12-19T14:34:57.085 INFO:teuthology.orchestra.run.smithi050.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-141-gc7312db0-1focal [2,408 kB] 2023-12-19T14:34:57.280 INFO:teuthology.orchestra.run.smithi050.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-141-gc7312db0-1focal [79.2 kB] 2023-12-19T14:34:57.282 INFO:teuthology.orchestra.run.smithi050.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-141-gc7312db0-1focal [117 kB] 2023-12-19T14:34:57.305 INFO:teuthology.orchestra.run.smithi050.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-141-gc7312db0-1focal [1,760 kB] 2023-12-19T14:34:57.450 INFO:teuthology.orchestra.run.smithi050.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-141-gc7312db0-1focal [7,876 kB] 2023-12-19T14:34:58.438 INFO:teuthology.orchestra.run.smithi050.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-141-gc7312db0-1focal [4,056 B] 2023-12-19T14:34:58.438 INFO:teuthology.orchestra.run.smithi050.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-141-gc7312db0-1focal [56.3 MB] 2023-12-19T14:35:02.676 INFO:teuthology.orchestra.run.smithi125.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-141-gc7312db0-1focal [229 kB] 2023-12-19T14:35:02.678 INFO:teuthology.orchestra.run.smithi125.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.7-141-gc7312db0-1focal [240 kB] 2023-12-19T14:35:02.680 INFO:teuthology.orchestra.run.smithi125.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.7-141-gc7312db0-1focal [22.0 kB] 2023-12-19T14:35:02.681 INFO:teuthology.orchestra.run.smithi125.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.7-141-gc7312db0-1focal [13.1 kB] 2023-12-19T14:35:02.681 INFO:teuthology.orchestra.run.smithi125.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-141-gc7312db0-1focal [21.9 kB] 2023-12-19T14:35:02.681 INFO:teuthology.orchestra.run.smithi125.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-141-gc7312db0-1focal [17.1 MB] 2023-12-19T14:35:04.752 INFO:teuthology.orchestra.run.smithi125.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-141-gc7312db0-1focal [79.9 kB] 2023-12-19T14:35:05.708 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 210 MB in 17s (12.3 MB/s) 2023-12-19T14:35:05.871 INFO:teuthology.orchestra.run.smithi050.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-141-gc7312db0-1focal [229 kB] 2023-12-19T14:35:05.873 INFO:teuthology.orchestra.run.smithi050.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.7-141-gc7312db0-1focal [240 kB] 2023-12-19T14:35:05.918 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-19T14:35:05.934 INFO:teuthology.orchestra.run.smithi050.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.7-141-gc7312db0-1focal [22.0 kB] 2023-12-19T14:35:05.935 INFO:teuthology.orchestra.run.smithi050.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.7-141-gc7312db0-1focal [13.1 kB] 2023-12-19T14:35:05.936 INFO:teuthology.orchestra.run.smithi050.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-141-gc7312db0-1focal [21.9 kB] 2023-12-19T14:35:05.937 INFO:teuthology.orchestra.run.smithi050.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-141-gc7312db0-1focal [17.1 MB] 2023-12-19T14:35:05.990 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 107756 files and directories currently installed.) 2023-12-19T14:35:05.993 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-12-19T14:35:06.039 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-12-19T14:35:06.417 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-19T14:35:06.438 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-19T14:35:06.483 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T14:35:06.853 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cmd2. 2023-12-19T14:35:06.871 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-12-19T14:35:06.911 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-12-19T14:35:07.356 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-19T14:35:07.375 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-19T14:35:07.456 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T14:35:07.863 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../004-librbd1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:07.993 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librbd1 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T14:35:08.132 INFO:teuthology.orchestra.run.smithi050.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-141-gc7312db0-1focal [79.9 kB] 2023-12-19T14:35:08.743 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../005-librados2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:08.856 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librados2 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T14:35:09.121 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 210 MB in 20s (10.3 MB/s) 2023-12-19T14:35:09.296 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-19T14:35:09.368 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 107756 files and directories currently installed.) 2023-12-19T14:35:09.371 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-12-19T14:35:09.434 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-12-19T14:35:09.602 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs2. 2023-12-19T14:35:09.622 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../006-libcephfs2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:09.700 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:09.737 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-19T14:35:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-19T14:35:09.804 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T14:35:10.182 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cmd2. 2023-12-19T14:35:10.202 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-12-19T14:35:10.240 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-12-19T14:35:10.312 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rados. 2023-12-19T14:35:10.332 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../007-python3-rados_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:10.370 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:10.781 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-19T14:35:10.784 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-19T14:35:10.798 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-19T14:35:10.798 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:10.839 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:10.885 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T14:35:11.126 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cephfs. 2023-12-19T14:35:11.146 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../009-python3-cephfs_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:11.183 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:11.250 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../004-librbd1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:11.397 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librbd1 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T14:35:11.595 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-19T14:35:11.617 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:11.687 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:12.081 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-prettytable. 2023-12-19T14:35:12.103 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-12-19T14:35:12.148 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-19T14:35:12.170 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../005-librados2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:12.301 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librados2 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T14:35:12.450 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rbd. 2023-12-19T14:35:12.468 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../012-python3-rbd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:12.508 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:12.888 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-19T14:35:12.906 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-19T14:35:12.946 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T14:35:13.055 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs2. 2023-12-19T14:35:13.073 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../006-libcephfs2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:13.130 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:13.307 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-19T14:35:13.326 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-19T14:35:13.365 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T14:35:13.491 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rados. 2023-12-19T14:35:13.513 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../007-python3-rados_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:13.557 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:13.760 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-19T14:35:13.780 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-19T14:35:13.818 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T14:35:13.961 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-19T14:35:13.981 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:14.255 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-19T14:35:14.276 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-12-19T14:35:14.321 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cephfs. 2023-12-19T14:35:14.338 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T14:35:14.342 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../009-python3-cephfs_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:14.538 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:14.649 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-19T14:35:14.667 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-12-19T14:35:14.740 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T14:35:14.907 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-19T14:35:14.926 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:14.965 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:15.261 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-19T14:35:15.281 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:15.339 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:15.360 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-prettytable. 2023-12-19T14:35:15.373 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-12-19T14:35:15.410 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-19T14:35:15.746 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rbd. 2023-12-19T14:35:15.766 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../012-python3-rbd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:15.804 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:15.897 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-19T14:35:15.910 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:15.955 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:16.215 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-19T14:35:16.236 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-19T14:35:16.317 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T14:35:16.392 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-19T14:35:16.412 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:16.450 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:16.735 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-19T14:35:16.756 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-19T14:35:16.793 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T14:35:16.903 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-12-19T14:35:16.915 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-12-19T14:35:16.961 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T14:35:17.180 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-19T14:35:17.200 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-19T14:35:17.238 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T14:35:17.347 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librgw2. 2023-12-19T14:35:17.370 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../022-librgw2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:17.439 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:17.691 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-19T14:35:17.713 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-12-19T14:35:17.757 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T14:35:18.152 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-19T14:35:18.173 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-12-19T14:35:18.244 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T14:35:18.275 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rgw. 2023-12-19T14:35:18.291 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../023-python3-rgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:18.324 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:18.727 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liboath0. 2023-12-19T14:35:18.746 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-19T14:35:18.747 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-19T14:35:18.768 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:18.785 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-19T14:35:18.831 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:19.155 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libradosstriper1. 2023-12-19T14:35:19.176 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../025-libradosstriper1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:19.213 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:19.392 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-19T14:35:19.412 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:19.450 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:19.800 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-common. 2023-12-19T14:35:19.822 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../026-ceph-common_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:19.866 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:19.878 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-19T14:35:19.899 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T14:35:19.928 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:35:20.381 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-12-19T14:35:20.402 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-12-19T14:35:20.447 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T14:35:21.509 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librgw2. 2023-12-19T14:35:21.530 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../022-librgw2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:22.411 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rgw. 2023-12-19T14:35:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../023-python3-rgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:22.461 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:22.738 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-base. 2023-12-19T14:35:22.754 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../027-ceph-base_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:22.821 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:22.872 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liboath0. 2023-12-19T14:35:22.894 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-19T14:35:22.964 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-19T14:35:23.300 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libradosstriper1. 2023-12-19T14:35:23.318 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../025-libradosstriper1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:23.383 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:23.892 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cryptography. 2023-12-19T14:35:23.895 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-common. 2023-12-19T14:35:23.914 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-19T14:35:23.917 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../026-ceph-common_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:23.958 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T14:35:23.961 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:24.404 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-openssl. 2023-12-19T14:35:24.426 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-19T14:35:24.470 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-19T14:35:24.781 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-19T14:35:24.800 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:24.839 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:25.311 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-19T14:35:25.328 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-19T14:35:25.369 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T14:35:25.739 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-19T14:35:25.759 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-19T14:35:25.796 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-19T14:35:26.208 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-natsort. 2023-12-19T14:35:26.229 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-12-19T14:35:26.266 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-12-19T14:35:26.603 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jwt. 2023-12-19T14:35:26.623 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-19T14:35:26.698 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T14:35:26.708 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-base. 2023-12-19T14:35:26.729 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../027-ceph-base_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:26.787 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:27.022 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-logutils. 2023-12-19T14:35:27.035 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-12-19T14:35:27.097 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-19T14:35:27.367 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-19T14:35:27.385 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-19T14:35:27.416 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-19T14:35:27.752 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-mako. 2023-12-19T14:35:27.770 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-19T14:35:27.832 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T14:35:27.880 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cryptography. 2023-12-19T14:35:27.900 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-19T14:35:27.938 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T14:35:28.236 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-19T14:35:28.255 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-19T14:35:28.305 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-19T14:35:28.391 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-openssl. 2023-12-19T14:35:28.413 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-19T14:35:28.458 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-19T14:35:28.675 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-19T14:35:28.695 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-19T14:35:28.733 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T14:35:28.777 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-19T14:35:28.799 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:28.843 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:29.061 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webob. 2023-12-19T14:35:29.081 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-19T14:35:29.110 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-19T14:35:29.299 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-19T14:35:29.323 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-19T14:35:29.365 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T14:35:29.505 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-19T14:35:29.526 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-19T14:35:29.563 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T14:35:29.743 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-19T14:35:29.765 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-19T14:35:29.818 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-19T14:35:29.908 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-bs4. 2023-12-19T14:35:29.920 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-12-19T14:35:29.966 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-19T14:35:30.205 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-natsort. 2023-12-19T14:35:30.227 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-12-19T14:35:30.271 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-12-19T14:35:30.369 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-waitress. 2023-12-19T14:35:30.387 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-19T14:35:30.434 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T14:35:30.641 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jwt. 2023-12-19T14:35:30.661 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-19T14:35:30.699 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T14:35:30.855 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-tempita. 2023-12-19T14:35:30.876 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-12-19T14:35:30.938 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-19T14:35:31.086 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-logutils. 2023-12-19T14:35:31.106 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-12-19T14:35:31.143 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-19T14:35:31.292 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-paste. 2023-12-19T14:35:31.311 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-19T14:35:31.357 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T14:35:31.497 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-19T14:35:31.514 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-19T14:35:31.671 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-19T14:35:31.736 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-19T14:35:31.758 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-19T14:35:31.802 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T14:35:32.024 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-mako. 2023-12-19T14:35:32.045 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-19T14:35:32.082 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T14:35:32.164 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-19T14:35:32.181 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-19T14:35:32.221 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-19T14:35:32.453 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-19T14:35:32.475 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-19T14:35:32.519 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-19T14:35:32.600 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webtest. 2023-12-19T14:35:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-12-19T14:35:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-19T14:35:32.864 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-19T14:35:32.883 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-19T14:35:32.922 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T14:35:33.036 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pecan. 2023-12-19T14:35:33.055 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-19T14:35:33.094 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T14:35:33.291 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webob. 2023-12-19T14:35:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-19T14:35:33.358 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-19T14:35:33.524 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-19T14:35:33.543 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-19T14:35:33.606 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T14:35:33.744 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-19T14:35:33.766 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-19T14:35:33.802 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T14:35:33.993 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-19T14:35:34.012 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:34.052 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:34.180 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bs4. 2023-12-19T14:35:34.201 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-12-19T14:35:34.263 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-19T14:35:34.471 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr. 2023-12-19T14:35:34.491 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../052-ceph-mgr_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:34.529 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:34.658 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-waitress. 2023-12-19T14:35:34.677 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-19T14:35:34.723 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T14:35:35.066 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mon. 2023-12-19T14:35:35.085 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../053-ceph-mon_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:35.103 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempita. 2023-12-19T14:35:35.123 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-12-19T14:35:35.149 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:35.161 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-19T14:35:35.497 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-paste. 2023-12-19T14:35:35.515 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-19T14:35:35.547 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T14:35:36.076 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-19T14:35:36.098 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-19T14:35:36.142 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T14:35:36.437 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-19T14:35:36.444 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-osd. 2023-12-19T14:35:36.458 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-19T14:35:36.464 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../054-ceph-osd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:36.503 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-19T14:35:36.510 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:36.906 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webtest. 2023-12-19T14:35:36.925 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-12-19T14:35:36.981 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-19T14:35:37.359 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pecan. 2023-12-19T14:35:37.377 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-19T14:35:37.417 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T14:35:37.839 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-19T14:35:37.859 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-19T14:35:37.896 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T14:35:38.308 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-19T14:35:38.328 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:38.357 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:38.761 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr. 2023-12-19T14:35:38.779 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../052-ceph-mgr_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:38.819 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:39.014 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph. 2023-12-19T14:35:39.034 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../055-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:39.080 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:39.364 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mon. 2023-12-19T14:35:39.383 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../053-ceph-mon_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:39.416 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-fuse. 2023-12-19T14:35:39.421 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:39.436 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../056-ceph-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:39.465 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:39.952 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mds. 2023-12-19T14:35:39.972 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../057-ceph-mds_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:40.010 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:40.601 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-osd. 2023-12-19T14:35:40.605 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephadm. 2023-12-19T14:35:40.619 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../054-ceph-osd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:40.625 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../058-cephadm_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:40.658 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:40.662 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:41.024 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jinja2. 2023-12-19T14:35:41.046 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-12-19T14:35:41.348 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-19T14:35:41.726 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-19T14:35:41.747 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:41.792 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:42.179 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-19T14:35:42.202 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-12-19T14:35:42.266 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-19T14:35:42.528 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjs-underscore. 2023-12-19T14:35:42.549 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-19T14:35:42.577 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T14:35:42.914 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-19T14:35:42.931 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-19T14:35:42.972 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T14:35:43.203 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph. 2023-12-19T14:35:43.217 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../055-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:43.261 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:43.333 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-routes. 2023-12-19T14:35:43.353 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-12-19T14:35:43.424 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-19T14:35:43.648 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-fuse. 2023-12-19T14:35:43.668 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../056-ceph-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:43.730 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:43.827 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-19T14:35:43.845 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:43.885 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:44.184 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mds. 2023-12-19T14:35:44.205 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../057-ceph-mds_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:44.250 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:44.481 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-decorator. 2023-12-19T14:35:44.503 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-19T14:35:44.572 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T14:35:44.853 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephadm. 2023-12-19T14:35:44.857 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-19T14:35:44.871 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../058-cephadm_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:44.872 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-19T14:35:44.911 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:44.941 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T14:35:45.277 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-scipy. 2023-12-19T14:35:45.290 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-19T14:35:45.297 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jinja2. 2023-12-19T14:35:45.318 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-12-19T14:35:45.343 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-19T14:35:45.355 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-19T14:35:45.725 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-19T14:35:45.746 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:45.783 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:46.194 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-19T14:35:46.214 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-12-19T14:35:46.252 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-19T14:35:46.563 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjs-underscore. 2023-12-19T14:35:46.584 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-19T14:35:46.629 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T14:35:46.765 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-19T14:35:46.785 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-19T14:35:46.861 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T14:35:46.966 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-19T14:35:46.986 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-19T14:35:47.023 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T14:35:47.435 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-routes. 2023-12-19T14:35:47.441 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-joblib. 2023-12-19T14:35:47.454 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-12-19T14:35:47.459 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-12-19T14:35:47.492 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-19T14:35:47.499 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-19T14:35:47.879 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-19T14:35:47.902 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:47.928 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn. 2023-12-19T14:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-19T14:35:47.953 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:48.027 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T14:35:48.499 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-decorator. 2023-12-19T14:35:48.520 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-19T14:35:48.565 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T14:35:48.903 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-19T14:35:48.921 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:48.935 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-19T14:35:48.958 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-19T14:35:48.960 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:49.001 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T14:35:49.380 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-scipy. 2023-12-19T14:35:49.399 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-19T14:35:49.487 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-19T14:35:50.181 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cachetools. 2023-12-19T14:35:50.202 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-12-19T14:35:50.247 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-19T14:35:50.608 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-19T14:35:50.629 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-19T14:35:50.708 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T14:35:51.025 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-19T14:35:51.046 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-19T14:35:51.093 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T14:35:51.128 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-19T14:35:51.149 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-19T14:35:51.194 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T14:35:51.572 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rsa. 2023-12-19T14:35:51.592 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-12-19T14:35:51.621 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-19T14:35:51.669 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-joblib. 2023-12-19T14:35:51.688 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-12-19T14:35:51.754 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-19T14:35:51.984 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-google-auth. 2023-12-19T14:35:52.004 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-12-19T14:35:52.041 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-19T14:35:52.165 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn. 2023-12-19T14:35:52.186 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-19T14:35:52.230 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T14:35:52.428 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-blinker. 2023-12-19T14:35:52.451 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-19T14:35:52.494 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T14:35:52.872 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-19T14:35:52.893 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-19T14:35:52.938 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T14:35:53.023 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-19T14:35:53.042 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:53.080 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:53.283 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-19T14:35:53.306 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-19T14:35:53.341 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T14:35:53.855 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-websocket. 2023-12-19T14:35:53.875 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-19T14:35:53.913 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T14:35:54.292 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-19T14:35:54.312 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-19T14:35:54.357 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T14:35:54.368 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cachetools. 2023-12-19T14:35:54.387 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-12-19T14:35:54.425 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-19T14:35:54.795 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-19T14:35:54.815 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-19T14:35:54.852 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T14:35:55.083 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-19T14:35:55.106 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:55.150 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:55.206 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-19T14:35:55.224 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-19T14:35:55.289 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T14:35:55.594 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-19T14:35:55.614 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-12-19T14:35:55.652 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-19T14:35:55.684 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rsa. 2023-12-19T14:35:55.706 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-12-19T14:35:55.750 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-19T14:35:56.047 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-19T14:35:56.064 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:35:56.128 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-google-auth. 2023-12-19T14:35:56.129 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:35:56.141 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-12-19T14:35:56.186 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-19T14:35:56.417 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package jq. 2023-12-19T14:35:56.437 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:35:56.482 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:35:56.581 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-blinker. 2023-12-19T14:35:56.599 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-19T14:35:56.639 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T14:35:56.777 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package socat. 2023-12-19T14:35:56.799 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-12-19T14:35:56.843 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-19T14:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-19T14:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-19T14:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T14:35:57.205 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package xmlstarlet. 2023-12-19T14:35:57.226 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-19T14:35:57.304 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-19T14:35:57.512 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-19T14:35:57.533 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-19T14:35:57.603 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T14:35:57.718 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-test. 2023-12-19T14:35:57.738 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../089-ceph-test_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:35:57.784 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:58.134 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-websocket. 2023-12-19T14:35:58.154 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-19T14:35:58.199 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T14:35:58.588 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-19T14:35:58.609 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-19T14:35:58.654 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T14:35:59.394 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-19T14:35:59.414 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:35:59.451 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:35:59.838 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-19T14:35:59.859 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-12-19T14:35:59.912 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-19T14:36:00.341 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-19T14:36:00.362 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:36:00.407 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:00.752 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package jq. 2023-12-19T14:36:00.772 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:36:00.809 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:01.087 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package socat. 2023-12-19T14:36:01.109 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-12-19T14:36:01.153 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-19T14:36:01.532 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package xmlstarlet. 2023-12-19T14:36:01.554 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-19T14:36:01.598 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-19T14:36:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-test. 2023-12-19T14:36:02.047 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../089-ceph-test_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:02.085 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:02.513 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-volume. 2023-12-19T14:36:02.533 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../090-ceph-volume_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:02.562 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:03.126 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-19T14:36:03.144 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:03.184 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:03.579 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-colorama. 2023-12-19T14:36:03.600 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-12-19T14:36:03.645 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-12-19T14:36:03.965 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-shell. 2023-12-19T14:36:03.984 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../093-cephfs-shell_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:04.023 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:04.309 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-top. 2023-12-19T14:36:04.330 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../094-cephfs-top_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:04.375 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:04.687 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package fonts-mathjax. 2023-12-19T14:36:04.707 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T14:36:04.745 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:05.299 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-19T14:36:05.320 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:05.357 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:05.835 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package nvme-cli. 2023-12-19T14:36:05.855 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-19T14:36:05.932 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T14:36:06.315 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjs-mathjax. 2023-12-19T14:36:06.334 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T14:36:06.373 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:06.873 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-volume. 2023-12-19T14:36:06.891 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../090-ceph-volume_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:06.956 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:07.436 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-19T14:36:07.458 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:07.502 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:07.906 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-colorama. 2023-12-19T14:36:07.925 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-12-19T14:36:07.964 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-12-19T14:36:08.275 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephfs-shell. 2023-12-19T14:36:08.298 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../093-cephfs-shell_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:08.341 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:08.540 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-natsort-doc. 2023-12-19T14:36:08.562 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-12-19T14:36:08.605 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-12-19T14:36:08.653 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephfs-top. 2023-12-19T14:36:08.673 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../094-cephfs-top_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T14:36:08.718 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:08.992 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-19T14:36:09.011 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-19T14:36:09.031 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package fonts-mathjax. 2023-12-19T14:36:09.048 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T14:36:09.049 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T14:36:09.097 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:09.389 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-attr. 2023-12-19T14:36:09.408 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-12-19T14:36:09.491 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-19T14:36:09.651 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-19T14:36:09.673 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:09.716 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:09.875 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webencodings. 2023-12-19T14:36:09.896 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-19T14:36:09.941 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T14:36:10.163 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package nvme-cli. 2023-12-19T14:36:10.180 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-19T14:36:10.255 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T14:36:10.286 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-html5lib. 2023-12-19T14:36:10.307 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-12-19T14:36:10.352 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-19T14:36:10.651 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjs-mathjax. 2023-12-19T14:36:10.671 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T14:36:10.717 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:10.739 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-19T14:36:10.759 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-19T14:36:10.807 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T14:36:11.327 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-packaging. 2023-12-19T14:36:11.349 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-12-19T14:36:11.418 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-19T14:36:11.796 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastescript. 2023-12-19T14:36:11.819 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-19T14:36:12.187 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-19T14:36:12.574 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pluggy. 2023-12-19T14:36:12.593 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-12-19T14:36:12.632 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-19T14:36:12.943 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-natsort-doc. 2023-12-19T14:36:12.965 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-12-19T14:36:13.009 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-12-19T14:36:13.018 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-psutil. 2023-12-19T14:36:13.031 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-19T14:36:13.076 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T14:36:13.421 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-py. 2023-12-19T14:36:13.429 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-19T14:36:13.440 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-19T14:36:13.451 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-19T14:36:13.479 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T14:36:13.520 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T14:36:13.857 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-19T14:36:13.878 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-19T14:36:13.893 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-attr. 2023-12-19T14:36:13.915 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T14:36:13.916 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-12-19T14:36:13.951 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-19T14:36:14.277 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pytest. 2023-12-19T14:36:14.297 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-12-19T14:36:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webencodings. 2023-12-19T14:36:14.342 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-19T14:36:14.360 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-19T14:36:14.404 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T14:36:14.738 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplejson. 2023-12-19T14:36:14.757 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-html5lib. 2023-12-19T14:36:14.759 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-19T14:36:14.779 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-12-19T14:36:14.804 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T14:36:14.823 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-19T14:36:15.115 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-19T14:36:15.135 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-12-19T14:36:15.173 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T14:36:15.246 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-19T14:36:15.266 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-19T14:36:15.303 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T14:36:15.840 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-packaging. 2023-12-19T14:36:15.862 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-12-19T14:36:15.898 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-19T14:36:15.944 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package radosgw. 2023-12-19T14:36:15.964 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../114-radosgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:16.002 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:16.276 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastescript. 2023-12-19T14:36:16.298 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-19T14:36:16.342 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-19T14:36:16.713 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pluggy. 2023-12-19T14:36:16.734 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-12-19T14:36:16.779 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-19T14:36:17.158 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-psutil. 2023-12-19T14:36:17.181 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-19T14:36:17.224 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T14:36:17.714 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package rbd-fuse. 2023-12-19T14:36:17.737 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../115-rbd-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:17.780 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:17.837 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-py. 2023-12-19T14:36:17.857 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-19T14:36:18.035 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T14:36:18.149 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-19T14:36:18.431 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-19T14:36:18.452 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-19T14:36:18.496 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T14:36:18.526 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T14:36:18.653 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T14:36:18.803 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T14:36:18.867 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pytest. 2023-12-19T14:36:18.891 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-12-19T14:36:18.932 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-19T14:36:19.022 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:19.353 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplejson. 2023-12-19T14:36:19.379 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-19T14:36:19.444 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T14:36:19.763 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user cephadm....done 2023-12-19T14:36:19.814 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-19T14:36:19.828 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-12-19T14:36:19.871 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T14:36:19.953 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:20.079 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T14:36:20.514 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-19T14:36:20.692 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package radosgw. 2023-12-19T14:36:20.707 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../114-radosgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:20.750 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:20.857 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:20.983 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-19T14:36:21.142 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T14:36:21.544 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-19T14:36:22.045 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T14:36:22.422 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-19T14:36:22.446 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package rbd-fuse. 2023-12-19T14:36:22.460 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../115-rbd-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T14:36:22.504 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:22.773 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-12-19T14:36:22.895 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-19T14:36:23.140 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T14:36:23.293 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T14:36:23.452 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T14:36:23.492 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T14:36:23.578 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T14:36:23.838 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:23.851 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:24.219 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T14:36:24.344 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-12-19T14:36:24.556 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user cephadm....done 2023-12-19T14:36:24.687 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-19T14:36:24.745 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:24.896 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T14:36:25.021 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T14:36:25.340 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-19T14:36:25.373 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T14:36:25.498 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T14:36:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T14:36:25.825 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-19T14:36:25.953 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T14:36:25.958 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-12-19T14:36:26.343 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T14:36:26.420 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-19T14:36:26.711 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T14:36:26.921 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T14:36:27.153 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T14:36:27.281 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-19T14:36:27.530 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-19T14:36:27.632 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-12-19T14:36:28.001 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T14:36:28.008 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-19T14:36:28.368 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T14:36:28.401 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-19T14:36:28.753 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:28.802 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T14:36:28.928 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T14:36:29.054 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T14:36:29.145 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T14:36:29.271 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-12-19T14:36:29.430 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T14:36:29.631 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-19T14:36:29.956 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-12-19T14:36:29.998 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T14:36:30.324 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T14:36:30.349 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T14:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T14:36:30.769 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T14:36:30.951 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-12-19T14:36:31.204 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-19T14:36:31.330 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:31.353 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T14:36:31.724 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:31.745 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T14:36:31.918 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-19T14:36:32.206 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T14:36:32.336 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T14:36:32.462 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-19T14:36:32.593 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-19T14:36:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-19T14:36:33.007 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T14:36:33.019 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-19T14:36:33.395 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-19T14:36:33.415 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:33.541 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T14:36:33.667 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T14:36:33.780 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T14:36:33.906 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T14:36:34.032 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T14:36:34.102 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T14:36:34.408 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T14:36:34.478 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-19T14:36:34.872 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T14:36:34.951 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-12-19T14:36:35.023 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-19T14:36:35.336 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T14:36:35.391 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-19T14:36:35.759 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T14:36:35.787 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T14:36:36.143 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T14:36:36.248 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-19T14:36:36.394 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:36.520 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-19T14:36:36.819 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:36.863 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-19T14:36:36.970 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-19T14:36:36.988 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T14:36:37.398 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-19T14:36:37.405 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T14:36:37.556 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-19T14:36:37.682 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-19T14:36:37.849 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T14:36:38.000 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T14:36:38.050 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T14:36:38.135 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-19T14:36:38.443 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:38.544 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T14:36:38.577 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T14:36:38.737 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T14:36:38.887 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librados2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:39.013 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-19T14:36:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T14:36:39.499 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-19T14:36:39.892 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T14:36:40.018 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-19T14:36:40.369 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-19T14:36:40.737 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T14:36:40.852 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:40.977 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:41.113 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T14:36:41.128 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-19T14:36:41.464 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-19T14:36:41.588 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-19T14:36:41.808 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-19T14:36:41.933 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T14:36:41.964 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-19T14:36:42.324 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:42.335 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-19T14:36:42.450 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-19T14:36:42.786 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T14:36:42.946 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T14:36:43.026 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:43.072 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-19T14:36:43.185 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T14:36:43.481 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T14:36:43.596 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:43.722 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T14:36:43.832 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librados2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:43.958 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-19T14:36:44.131 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T14:36:45.361 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librbd1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:45.512 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T14:36:45.638 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-12-19T14:36:45.772 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:45.814 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:45.923 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-19T14:36:45.940 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:46.100 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-19T14:36:46.291 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T14:36:46.426 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-19T14:36:46.576 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-19T14:36:46.904 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:46.961 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-19T14:36:47.055 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:47.318 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T14:36:47.318 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T14:36:47.337 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T14:36:47.463 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-19T14:36:47.956 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:48.099 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T14:36:48.156 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:48.282 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T14:36:48.510 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:48.658 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-19T14:36:48.927 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T14:36:49.052 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T14:36:49.345 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T14:36:49.488 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:49.613 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:49.773 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:49.899 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:50.050 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:50.184 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-19T14:36:50.527 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-19T14:36:50.667 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librbd1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:50.801 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T14:36:50.895 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:50.936 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-12-19T14:36:51.021 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:51.104 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T14:36:51.147 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:51.230 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-19T14:36:51.306 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T14:36:51.606 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T14:36:51.732 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-19T14:36:52.244 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:52.370 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:52.468 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:52.644 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T14:36:52.644 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T14:36:53.141 INFO:teuthology.orchestra.run.smithi125.stdout:Adding group ceph....done 2023-12-19T14:36:53.513 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:53.622 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user ceph....done 2023-12-19T14:36:53.639 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T14:36:53.847 INFO:teuthology.orchestra.run.smithi125.stdout:Setting system user ceph properties....done 2023-12-19T14:36:53.887 INFO:teuthology.orchestra.run.smithi125.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-19T14:36:54.041 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-19T14:36:54.081 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-19T14:36:54.492 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T14:36:54.621 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-19T14:36:54.945 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.071 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.197 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.356 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.482 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.561 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:55.633 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-19T14:36:55.820 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T14:36:55.820 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T14:36:56.051 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-19T14:36:56.428 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:56.554 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:56.638 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:56.680 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:56.763 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:56.839 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T14:36:57.080 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T14:36:57.081 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T14:36:58.026 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:58.332 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:58.644 INFO:teuthology.orchestra.run.smithi050.stdout:Adding group ceph....done 2023-12-19T14:36:58.855 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-19T14:36:59.267 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user ceph....done 2023-12-19T14:36:59.531 INFO:teuthology.orchestra.run.smithi050.stdout:Setting system user ceph properties....done 2023-12-19T14:36:59.563 INFO:teuthology.orchestra.run.smithi050.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-19T14:36:59.676 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:36:59.772 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-19T14:36:59.940 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T14:36:59.940 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T14:37:00.290 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-19T14:37:00.769 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T14:37:01.070 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T14:37:01.263 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:01.510 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T14:37:01.510 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T14:37:01.961 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:02.356 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:02.482 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:02.742 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T14:37:02.742 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T14:37:02.836 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T14:37:02.836 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T14:37:03.580 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:03.706 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:03.891 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:04.034 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:04.158 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T14:37:04.162 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T14:37:04.574 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-19T14:37:04.958 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:05.151 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:05.277 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:05.445 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:05.495 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:05.572 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T14:37:05.731 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T14:37:05.787 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T14:37:05.787 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T14:37:06.216 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T14:37:06.663 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:06.979 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T14:37:06.979 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T14:37:07.806 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:08.646 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T14:37:08.646 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T14:37:09.500 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:09.625 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:09.785 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:10.081 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T14:37:10.081 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T14:37:10.928 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:11.113 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:11.239 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:11.410 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-12-19T14:37:11.423 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T14:37:11.550 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T14:37:11.710 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T14:37:12.196 INFO:teuthology.orchestra.run.smithi125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T14:37:12.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-12-19T14:37:12.270 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T14:37:12.319 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-12-19T14:37:12.532 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2023-12-19T14:37:12.532 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:flex is already the newest version (2.6.4-6.2). 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-12-19T14:37:12.729 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T14:37:12.730 INFO:teuthology.orchestra.run.smithi125.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T14:37:12.730 INFO:teuthology.orchestra.run.smithi125.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T14:37:12.730 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2023-12-19T14:37:12.730 INFO:teuthology.orchestra.run.smithi125.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-12-19T14:37:12.731 INFO:teuthology.orchestra.run.smithi125.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-12-19T14:37:12.731 INFO:teuthology.orchestra.run.smithi125.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-12-19T14:37:12.731 INFO:teuthology.orchestra.run.smithi125.stdout: usb-modeswitch-data wpasupplicant 2023-12-19T14:37:12.732 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2023-12-19T14:37:12.732 INFO:teuthology.orchestra.run.smithi125.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-12-19T14:37:12.732 INFO:teuthology.orchestra.run.smithi125.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-12-19T14:37:12.829 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2023-12-19T14:37:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-12-19T14:37:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-12-19T14:37:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-12-19T14:37:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-19T14:37:12.831 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be upgraded: 2023-12-19T14:37:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-12-19T14:37:12.896 INFO:teuthology.orchestra.run.smithi125.stdout:6 upgraded, 23 newly installed, 0 to remove and 256 not upgraded. 2023-12-19T14:37:12.896 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 10.4 MB of archives. 2023-12-19T14:37:12.896 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-12-19T14:37:12.897 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-12-19T14:37:13.032 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-12-19T14:37:13.033 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-12-19T14:37:13.035 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-12-19T14:37:13.042 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-12-19T14:37:13.044 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.20 [1,321 kB] 2023-12-19T14:37:13.060 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-12-19T14:37:13.060 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-12-19T14:37:13.062 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-12-19T14:37:13.064 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-12-19T14:37:13.064 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.7 [60.7 kB] 2023-12-19T14:37:13.065 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-12-19T14:37:13.066 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-12-19T14:37:13.067 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-12-19T14:37:13.068 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-12-19T14:37:13.082 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-12-19T14:37:13.083 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-12-19T14:37:13.089 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-12-19T14:37:13.101 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-12-19T14:37:13.101 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.20 [1,584 kB] 2023-12-19T14:37:13.119 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-12-19T14:37:13.119 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-12-19T14:37:13.129 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-12-19T14:37:13.142 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-12-19T14:37:13.159 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-12-19T14:37:13.163 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-12-19T14:37:13.163 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-12-19T14:37:13.164 INFO:teuthology.orchestra.run.smithi125.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-12-19T14:37:13.164 INFO:teuthology.orchestra.run.smithi125.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-12-19T14:37:13.585 INFO:teuthology.orchestra.run.smithi125.stdout:Preconfiguring packages ... 2023-12-19T14:37:14.005 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 10.4 MB in 0s (32.1 MB/s) 2023-12-19T14:37:14.331 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:14.335 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:14.519 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:16.191 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:17.614 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-12-19T14:37:18.245 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:18.249 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:18.377 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T14:37:18.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-12-19T14:37:18.515 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2023-12-19T14:37:18.729 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2023-12-19T14:37:18.730 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2023-12-19T14:37:18.840 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:18.940 INFO:teuthology.orchestra.run.smithi050.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-12-19T14:37:18.940 INFO:teuthology.orchestra.run.smithi050.stdout:flex is already the newest version (2.6.4-6.2). 2023-12-19T14:37:18.940 INFO:teuthology.orchestra.run.smithi050.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-12-19T14:37:18.941 INFO:teuthology.orchestra.run.smithi050.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-12-19T14:37:18.941 INFO:teuthology.orchestra.run.smithi050.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-12-19T14:37:18.941 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T14:37:18.942 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T14:37:18.942 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout: usb-modeswitch-data wpasupplicant 2023-12-19T14:37:18.943 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2023-12-19T14:37:18.944 INFO:teuthology.orchestra.run.smithi050.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-12-19T14:37:18.944 INFO:teuthology.orchestra.run.smithi050.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-12-19T14:37:19.034 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2023-12-19T14:37:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-12-19T14:37:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-12-19T14:37:19.035 INFO:teuthology.orchestra.run.smithi050.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-12-19T14:37:19.035 INFO:teuthology.orchestra.run.smithi050.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-19T14:37:19.036 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2023-12-19T14:37:19.037 INFO:teuthology.orchestra.run.smithi050.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-12-19T14:37:19.108 INFO:teuthology.orchestra.run.smithi050.stdout:6 upgraded, 23 newly installed, 0 to remove and 256 not upgraded. 2023-12-19T14:37:19.109 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 10.4 MB of archives. 2023-12-19T14:37:19.109 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-12-19T14:37:19.109 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-12-19T14:37:19.259 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:19.263 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-12-19T14:37:19.266 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-12-19T14:37:19.269 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-12-19T14:37:19.275 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-12-19T14:37:19.276 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.20 [1,321 kB] 2023-12-19T14:37:19.295 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-12-19T14:37:19.296 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-12-19T14:37:19.297 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-12-19T14:37:19.299 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-12-19T14:37:19.299 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.7 [60.7 kB] 2023-12-19T14:37:19.302 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-12-19T14:37:19.304 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-12-19T14:37:19.305 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-12-19T14:37:19.306 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-12-19T14:37:19.321 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-12-19T14:37:19.323 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-12-19T14:37:19.334 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-12-19T14:37:19.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-12-19T14:37:19.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.20 [1,584 kB] 2023-12-19T14:37:19.362 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-12-19T14:37:19.363 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-12-19T14:37:19.391 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-12-19T14:37:19.431 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-12-19T14:37:19.473 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-12-19T14:37:19.489 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-12-19T14:37:19.495 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-12-19T14:37:19.495 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-12-19T14:37:19.495 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-12-19T14:37:19.626 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:19.926 INFO:teuthology.orchestra.run.smithi050.stdout:Preconfiguring packages ... 2023-12-19T14:37:20.099 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 10.4 MB in 0s (23.1 MB/s) 2023-12-19T14:37:20.168 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:20.172 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:20.309 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T14:37:20.448 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:20.452 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:20.773 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:20.858 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:20.998 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T14:37:21.538 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T14:37:21.702 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.20) over (1.1.1f-1ubuntu2.16) ... 2023-12-19T14:37:22.231 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-19T14:37:22.249 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-12-19T14:37:22.289 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-12-19T14:37:22.536 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:22.692 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package dump. 2023-12-19T14:37:22.717 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-12-19T14:37:22.792 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking dump (0.4b46-6) ... 2023-12-19T14:37:23.187 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package indent. 2023-12-19T14:37:23.208 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:37:23.254 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T14:37:23.624 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-19T14:37:23.645 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-12-19T14:37:23.690 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T14:37:24.085 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-19T14:37:24.107 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.7_amd64.deb ... 2023-12-19T14:37:24.152 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T14:37:24.581 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-19T14:37:24.603 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:24.672 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:24.720 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:24.723 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:24.893 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:25.076 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-19T14:37:25.095 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:25.134 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:25.396 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T14:37:25.470 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-19T14:37:25.489 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:25.528 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:25.566 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T14:37:25.915 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-19T14:37:25.934 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:25.970 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:25.972 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:26.376 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-19T14:37:26.397 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-12-19T14:37:26.442 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T14:37:26.641 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T14:37:26.645 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:26.795 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-19T14:37:26.821 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T14:37:26.861 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:26.899 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T14:37:27.314 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-19T14:37:27.337 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:27.397 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:27.436 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T14:37:27.775 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-19T14:37:27.795 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:27.833 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:28.144 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-19T14:37:28.147 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T14:37:28.167 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T14:37:28.211 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:28.284 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.20) over (1.1.1f-1ubuntu2.16) ... 2023-12-19T14:37:28.773 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-19T14:37:28.795 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-12-19T14:37:28.839 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-12-19T14:37:28.888 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-19T14:37:28.912 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-12-19T14:37:28.954 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-12-19T14:37:29.310 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package modemmanager. 2023-12-19T14:37:29.329 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:29.332 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package dump. 2023-12-19T14:37:29.351 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-12-19T14:37:29.389 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:29.390 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking dump (0.4b46-6) ... 2023-12-19T14:37:29.736 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package indent. 2023-12-19T14:37:29.762 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T14:37:29.802 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T14:37:30.082 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package wpasupplicant. 2023-12-19T14:37:30.101 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-12-19T14:37:30.148 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T14:37:30.399 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-19T14:37:30.418 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-12-19T14:37:30.490 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T14:37:30.746 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package network-manager. 2023-12-19T14:37:30.767 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T14:37:30.869 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-19T14:37:30.891 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.7_amd64.deb ... 2023-12-19T14:37:30.898 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:30.935 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T14:37:31.280 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-19T14:37:31.302 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T14:37:31.346 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:31.694 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ppp. 2023-12-19T14:37:31.716 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-12-19T14:37:31.741 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-19T14:37:31.760 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:31.799 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:32.127 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-19T14:37:32.152 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:32.193 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:32.229 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T14:37:32.580 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-19T14:37:32.602 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package pptp-linux. 2023-12-19T14:37:32.646 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-12-19T14:37:32.700 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-12-19T14:37:33.049 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-19T14:37:33.069 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-12-19T14:37:33.137 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-19T14:37:33.141 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T14:37:33.161 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-12-19T14:37:33.238 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-12-19T14:37:33.548 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-19T14:37:33.552 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-19T14:37:33.572 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-12-19T14:37:33.574 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T14:37:33.615 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-12-19T14:37:33.618 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:34.038 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-19T14:37:34.051 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-19T14:37:34.062 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:34.066 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-12-19T14:37:34.105 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:34.132 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T14:37:34.505 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:34.507 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-19T14:37:34.530 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:34.574 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:34.890 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-12-19T14:37:34.893 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-19T14:37:34.912 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T14:37:34.960 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:35.039 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T14:37:35.166 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T14:37:35.564 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-19T14:37:35.586 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-12-19T14:37:35.630 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-12-19T14:37:35.636 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T14:37:35.636 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T14:37:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package modemmanager. 2023-12-19T14:37:36.079 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T14:37:36.147 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:36.511 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:36.665 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-12-19T14:37:36.788 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:36.864 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package wpasupplicant. 2023-12-19T14:37:36.878 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-12-19T14:37:36.914 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-12-19T14:37:36.922 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T14:37:37.073 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:37.200 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T14:37:37.511 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package network-manager. 2023-12-19T14:37:37.531 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T14:37:37.619 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:38.401 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ppp. 2023-12-19T14:37:38.423 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-12-19T14:37:38.467 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-12-19T14:37:38.897 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up dump (0.4b46-6) ... 2023-12-19T14:37:38.923 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T14:37:39.004 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-19T14:37:39.073 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:39.200 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T14:37:39.341 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package pptp-linux. 2023-12-19T14:37:39.362 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-12-19T14:37:39.367 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T14:37:39.406 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-12-19T14:37:39.493 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T14:37:39.703 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T14:37:39.826 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-19T14:37:39.829 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:39.848 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-12-19T14:37:39.930 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-12-19T14:37:39.956 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:40.106 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-12-19T14:37:40.263 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-19T14:37:40.274 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:40.287 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-12-19T14:37:40.329 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-12-19T14:37:40.425 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:40.741 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-19T14:37:40.760 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-12-19T14:37:40.855 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T14:37:41.202 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout:which means they are currently configured by ifupdown: 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout:- enp3s0f1 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout:remove their configuration from /etc/network/interfaces. 2023-12-19T14:37:41.593 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:37:41.612 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-12-19T14:37:41.738 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T14:37:41.812 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-19T14:37:41.864 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T14:37:42.338 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T14:37:42.338 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T14:37:42.363 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-19T14:37:43.259 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T14:37:43.352 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-19T14:37:43.385 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-12-19T14:37:43.519 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T14:37:43.662 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-12-19T14:37:43.788 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:43.948 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T14:37:44.295 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:44.438 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-12-19T14:37:44.597 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:45.078 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:45.166 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-12-19T14:37:45.587 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up dump (0.4b46-6) ... 2023-12-19T14:37:45.632 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:45.694 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-19T14:37:45.759 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:45.763 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:45.914 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T14:37:46.032 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T14:37:46.074 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T14:37:46.074 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T14:37:46.133 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T14:37:46.343 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T14:37:46.469 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:46.628 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:46.763 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-12-19T14:37:46.974 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:47.087 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-12-19T14:37:47.106 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T14:37:47.204 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-19T14:37:47.338 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T14:37:47.548 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T14:37:48.024 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T14:37:48.313 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:37:48.313 INFO:teuthology.orchestra.run.smithi050.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-19T14:37:48.313 INFO:teuthology.orchestra.run.smithi050.stdout:which means they are currently configured by ifupdown: 2023-12-19T14:37:48.313 INFO:teuthology.orchestra.run.smithi050.stdout:- ens1f0 2023-12-19T14:37:48.314 INFO:teuthology.orchestra.run.smithi050.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-19T14:37:48.314 INFO:teuthology.orchestra.run.smithi050.stdout:remove their configuration from /etc/network/interfaces. 2023-12-19T14:37:48.314 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:37:48.553 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-19T14:37:49.112 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-19T14:37:50.065 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-19T14:37:51.268 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T14:37:51.403 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-12-19T14:37:51.604 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T14:37:51.730 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:51.856 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T14:37:52.007 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T14:37:52.335 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T14:37:52.335 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T14:37:53.343 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-12-19T14:37:53.469 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-19T14:37:53.637 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T14:37:53.796 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T14:37:54.315 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T14:37:54.918 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-12-19T14:37:55.026 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-19T14:37:56.209 INFO:teuthology.orchestra.run.smithi125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T14:37:56.216 DEBUG:teuthology.parallel:result is None 2023-12-19T14:38:01.350 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-12-19T14:38:01.459 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-19T14:38:02.678 INFO:teuthology.orchestra.run.smithi050.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T14:38:02.685 DEBUG:teuthology.parallel:result is None 2023-12-19T14:38:02.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T14:38:02.984 DEBUG:teuthology.orchestra.run.smithi050:> dpkg-query -W -f '${Version}' ceph 2023-12-19T14:38:03.026 INFO:teuthology.orchestra.run.smithi050.stdout:17.2.7-141-gc7312db0-1focal 2023-12-19T14:38:03.026 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-141-gc7312db0-1focal 2023-12-19T14:38:03.026 INFO:teuthology.task.install:The correct ceph version 17.2.7-141-gc7312db0-1focal is installed. 2023-12-19T14:38:03.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T14:38:03.170 DEBUG:teuthology.orchestra.run.smithi125:> dpkg-query -W -f '${Version}' ceph 2023-12-19T14:38:03.206 INFO:teuthology.orchestra.run.smithi125.stdout:17.2.7-141-gc7312db0-1focal 2023-12-19T14:38:03.207 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-141-gc7312db0-1focal 2023-12-19T14:38:03.207 INFO:teuthology.task.install:The correct ceph version 17.2.7-141-gc7312db0-1focal is installed. 2023-12-19T14:38:03.209 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-19T14:38:03.209 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:03.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T14:38:03.245 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:03.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T14:38:03.288 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-19T14:38:03.288 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:03.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T14:38:03.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T14:38:03.407 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:03.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T14:38:03.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T14:38:03.519 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-19T14:38:03.519 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:03.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T14:38:03.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T14:38:03.638 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:03.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T14:38:03.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T14:38:03.751 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-19T14:38:03.751 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:03.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T14:38:03.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T14:38:03.863 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:03.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T14:38:03.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T14:38:03.973 INFO:teuthology.run_tasks:Running task ceph... 2023-12-19T14:38:04.038 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-19T14:38:04.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2023-12-19T14:38:04.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /var/log/ceph 2023-12-19T14:38:04.082 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-19T14:38:04.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-19T14:38:04.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-19T14:38:04.158 INFO:tasks.ceph:Creating extra log directories... 2023-12-19T14:38:04.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-19T14:38:04.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-19T14:38:04.251 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-19T14:38:04.251 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-19T14:38:04.252 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-19T14:38:04.252 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:04.252 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-19T14:38:04.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-19T14:38:04.425 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:04.425 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-19T14:38:04.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-19T14:38:04.582 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-19T14:38:04.582 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-19T14:38:04.582 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876', 'branch': 'rishabh-mdscap-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': None, 'first_in_suite': False, 'job_id': '7496876', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'rishabh-mdscap-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-mdscap-quincy', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4463, 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'sleep_before_teardown': 0, 'subset': '344/512', 'suite': 'fs', 'suite_branch': 'rishabh-mdscap-quincy', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi125.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-19_12:25:28', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-19T14:38:04.583 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-19T14:38:04.590 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-19T14:38:04.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-12-19T14:38:04.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/run/ceph 2023-12-19T14:38:04.689 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:04.690 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T14:38:04.742 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T14:38:04.742 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2023-12-19T14:38:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T14:38:04.803 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:04.803 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 495 Links: 1 2023-12-19T14:38:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-12-19 14:32:39.247533490 +0000 2023-12-19T14:38:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-12-19 14:32:38.755544275 +0000 2023-12-19T14:38:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-12-19 14:32:38.755544275 +0000 2023-12-19T14:38:04.804 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-12-19T14:38:04.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T14:38:04.879 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-12-19T14:38:04.879 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-12-19T14:38:04.879 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000601273 s, 852 kB/s 2023-12-19T14:38:04.881 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T14:38:04.952 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 500 Links: 1 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-12-19 14:32:40.019516566 +0000 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-12-19 14:32:39.547526914 +0000 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-12-19 14:32:39.547526914 +0000 2023-12-19T14:38:05.015 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-12-19T14:38:05.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T14:38:05.091 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-12-19T14:38:05.091 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-12-19T14:38:05.091 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00062603 s, 818 kB/s 2023-12-19T14:38:05.093 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T14:38:05.163 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2023-12-19T14:38:05.225 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 507 Links: 1 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-12-19 14:32:40.319509989 +0000 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-12-19 14:32:40.315510077 +0000 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-12-19 14:32:40.315510077 +0000 2023-12-19T14:38:05.226 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-12-19T14:38:05.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T14:38:05.301 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-12-19T14:38:05.301 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-12-19T14:38:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000412097 s, 1.2 MB/s 2023-12-19T14:38:05.304 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T14:38:05.368 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2023-12-19T14:38:05.427 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 512 Links: 1 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-12-19 14:32:41.559482797 +0000 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-12-19 14:32:41.083493236 +0000 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-12-19 14:32:41.083493236 +0000 2023-12-19T14:38:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-12-19T14:38:05.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T14:38:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-12-19T14:38:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-12-19T14:38:05.502 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000423583 s, 1.2 MB/s 2023-12-19T14:38:05.504 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T14:38:05.572 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:05.572 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:05.572 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T14:38:05.590 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T14:38:05.593 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_1 2023-12-19T14:38:05.655 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 496 Links: 1 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-12-19 14:32:35.399555178 +0000 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-12-19 14:32:34.907566524 +0000 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-12-19 14:32:34.907566524 +0000 2023-12-19T14:38:05.656 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2023-12-19T14:38:05.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T14:38:05.734 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-12-19T14:38:05.735 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-12-19T14:38:05.735 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000709165 s, 722 kB/s 2023-12-19T14:38:05.737 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T14:38:05.803 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_2 2023-12-19T14:38:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T14:38:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:05.869 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 500 Links: 1 2023-12-19T14:38:05.869 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:05.870 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-12-19 14:32:36.171537374 +0000 2023-12-19T14:38:05.870 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-12-19 14:32:35.691548444 +0000 2023-12-19T14:38:05.870 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-12-19 14:32:35.691548444 +0000 2023-12-19T14:38:05.870 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2023-12-19T14:38:05.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T14:38:05.946 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-12-19T14:38:05.947 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-12-19T14:38:05.947 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.00069132 s, 741 kB/s 2023-12-19T14:38:05.950 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T14:38:06.017 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_3 2023-12-19T14:38:06.083 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T14:38:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:06.084 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 506 Links: 1 2023-12-19T14:38:06.085 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:06.085 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-12-19 14:32:36.939519663 +0000 2023-12-19T14:38:06.085 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-12-19 14:32:36.471530457 +0000 2023-12-19T14:38:06.085 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-12-19 14:32:36.471530457 +0000 2023-12-19T14:38:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2023-12-19T14:38:06.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T14:38:06.164 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-12-19T14:38:06.164 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-12-19T14:38:06.164 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000456796 s, 1.1 MB/s 2023-12-19T14:38:06.165 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T14:38:06.230 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_4 2023-12-19T14:38:06.295 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 513 Links: 1 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-12-19 14:32:37.687502414 +0000 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-12-19 14:32:37.219513207 +0000 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-12-19 14:32:37.219513207 +0000 2023-12-19T14:38:06.296 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2023-12-19T14:38:06.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T14:38:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-12-19T14:38:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-12-19T14:38:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000673626 s, 760 kB/s 2023-12-19T14:38:06.379 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T14:38:06.447 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:06.447 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-19T14:38:06.447 INFO:tasks.ceph:Generating config... 2023-12-19T14:38:06.448 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-19T14:38:06.449 INFO:tasks.ceph:[client] debug client = 20 2023-12-19T14:38:06.449 INFO:tasks.ceph:[client] debug ms = 1 2023-12-19T14:38:06.449 INFO:tasks.ceph:[client] pid file = 2023-12-19T14:38:06.449 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-19T14:38:06.449 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-19T14:38:06.449 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mgr] debug client = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-19T14:38:06.450 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-19T14:38:06.451 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-19T14:38:06.451 INFO:tasks.ceph:Setting up mon.a... 2023-12-19T14:38:06.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-19T14:38:06.511 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2023-12-19T14:38:06.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-19T14:38:06.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T14:38:06.627 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50'), ('mon.b', '172.21.15.125'), ('mon.c', '[v2:172.21.15.125:3301,v1:172.21.15.125:6790]')] 2023-12-19T14:38:06.627 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50,172.21.15.125,[v2:172.21.15.125:3301,v1:172.21.15.125:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-19T14:38:06.628 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:06.628 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T14:38:06.693 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --add b 172.21.15.125 --addv c '[v2:172.21.15.125:3301,v1:172.21.15.125:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = octopus 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T14:38:06.797 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2023-12-19T14:38:06.796569+0000 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:created 2023-12-19T14:38:06.796569+0000 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 15 (octopus) 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:1: [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] mon.b 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:2: [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] mon.c 2023-12-19T14:38:06.798 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-19T14:38:06.800 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T14:38:06.819 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d... 2023-12-19T14:38:06.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T14:38:06.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-12-19T14:38:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.50,172.21.15.125,[v2:172.21.15.125:3301,v1:172.21.15.125:6790]" 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-12-19T14:38:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-12-19T14:38:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: osd op complaint time = 180 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug client = 20 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: debug finisher = 20 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.955 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T14:38:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: mon op complaint time = 120 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-12-19T14:38:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: client mount timeout = 600 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: debug client = 20 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = "" 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: rados mon op timeout = 900 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: rados osd op timeout = 900 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.a] 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.c] 2023-12-19T14:38:06.958 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.b] 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.d] 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout:[mds] 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds = 20 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: mds debug frag = True 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: mds debug scatterstat = True 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: mds op complaint time = 180 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: mds verify scatter = True 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: osd op complaint time = 180 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: rados mon op timeout = 900 2023-12-19T14:38:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: rados osd op timeout = 900 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout:[global] 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: chdir = "" 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: auth supported = cephx 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: filestore xattr use omap = true 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: mon clock drift allowed = 1.000 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: osd crush chooseleaf type = 0 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: auth debug = true 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on old message = true 2023-12-19T14:38:06.960 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on bug = true 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon pg warn max object skew = 0 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default size = 2 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow primary affinity = true 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow pg remap = true 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on legacy crush tunables = false 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on crush straw calc version zero = false 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on no sortbitwise = false 2023-12-19T14:38:06.961 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on osd down out interval zero = false 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on too few osds = false 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon_allow_pool_size_one = true 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: osd default data pool replay window = 5 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon allow pool delete = true 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon cluster log file level = debug 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: debug asserts on shutdown = true 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon health detail to clog = false 2023-12-19T14:38:06.962 INFO:teuthology.orchestra.run.smithi125.stdout: mon host = "172.21.15.50,172.21.15.125,[v2:172.21.15.125:3301,v1:172.21.15.125:6790]" 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: fsid = 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout:[osd] 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd journal size = 100 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub load threshold = 5.0 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub max interval = 600 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd mclock profile = high_recovery_ops 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd recover clone overlap = true 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd recovery max chunk = 1048576 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug shutdown = true 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug op order = true 2023-12-19T14:38:06.963 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug verify stray on activate = true 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: osd open classes on start = true 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug pg log writeout = true 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: osd deep scrub update digest min age = 30 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: osd map max advance = 10 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: journal zero on create = true 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: filestore ondisk finisher threads = 3 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: filestore apply finisher threads = 3 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: bdev debug aio = true 2023-12-19T14:38:06.964 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug misdirected ops = true 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore block size = 96636764160 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore fsck on mount = True 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluefs = 20 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluestore = 20 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: debug osd = 20 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: debug rocksdb = 10 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd full ratio = 0.9 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd nearfull ratio = 0.8 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: osd failsafe full ratio = 0.95 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: osd objectstore = bluestore 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.965 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr] 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug mgr = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: mgr/telemetry/nag = false 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug finisher = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout:[mon] 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug paxos = 20 2023-12-19T14:38:06.966 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon data avail warn = 5 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon mgr mkfs grace = 240 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd reporter subtree level = osd 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd prime pg temp = true 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T14:38:06.967 INFO:teuthology.orchestra.run.smithi125.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: # 1m isn't quite enough 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: mon_down_mkfs_grace = 2m 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_filestore_osds = false 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: mon op complaint time = 120 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout:[client] 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: rgw cache enabled = true 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable ops log = true 2023-12-19T14:38:06.968 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable usage log = true 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: client mount timeout = 600 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = "" 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.a] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.b] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.c] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2023-12-19T14:38:06.969 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout:[mds] 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds = 20 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug frag = True 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug scatterstat = True 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: mds op complaint time = 180 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: mds verify scatter = True 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-12-19T14:38:06.970 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-12-19T14:38:06.976 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-19T14:38:06.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-19T14:38:07.092 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-19T14:38:07.092 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:07.092 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-19T14:38:07.112 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:07.112 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-19T14:38:07.175 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:38:07.176 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:07.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-19T14:38:07.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T14:38:07.284 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:07.284 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:07.343 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:38:07.343 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:07.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-19T14:38:07.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T14:38:07.408 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:07.408 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:07.468 INFO:tasks.ceph:Setting up mon nodes... 2023-12-19T14:38:07.469 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-19T14:38:07.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-19T14:38:07.553 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-19T14:38:07.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T14:38:07.650 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T14:38:07.657 INFO:tasks.ceph:Setting up mds nodes... 2023-12-19T14:38:07.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-19T14:38:07.746 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-19T14:38:07.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-19T14:38:07.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-19T14:38:07.916 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-19T14:38:07.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-19T14:38:07.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-19T14:38:08.048 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-19T14:38:08.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-19T14:38:08.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-19T14:38:08.226 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-19T14:38:08.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-19T14:38:08.266 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-19T14:38:08.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-19T14:38:08.335 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-19T14:38:08.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-12-19T14:38:08.431 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-12-19T14:38:08.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-19T14:38:08.530 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-19T14:38:08.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-12-19T14:38:08.615 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-12-19T14:38:08.643 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-19T14:38:08.643 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-19T14:38:08.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-19T14:38:08.682 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:08.682 INFO:tasks.ceph:role: osd.0 2023-12-19T14:38:08.682 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:38:08.682 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:08.796 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:08.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-12-19T14:38:08.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-19T14:38:08.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-19T14:38:08.909 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:08.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-19T14:38:08.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:08.989 INFO:tasks.ceph:role: osd.1 2023-12-19T14:38:08.990 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:38:08.990 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-19T14:38:09.106 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:09.106 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:09.106 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:09.106 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-12-19T14:38:09.106 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:09.107 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:09.107 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:09.107 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:09.107 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:09.107 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:09.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-12-19T14:38:09.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-19T14:38:09.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-19T14:38:09.230 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:09.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-19T14:38:09.314 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:09.314 INFO:tasks.ceph:role: osd.2 2023-12-19T14:38:09.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:38:09.314 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:09.431 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:09.434 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-12-19T14:38:09.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-19T14:38:09.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-19T14:38:09.550 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:09.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-19T14:38:09.629 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:09.629 INFO:tasks.ceph:role: osd.3 2023-12-19T14:38:09.629 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:38:09.629 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:09.757 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:09.758 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:09.758 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:09.758 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:09.758 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:09.761 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-12-19T14:38:09.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-19T14:38:09.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-19T14:38:09.870 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:09.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.015+0000 7f6b5bae83c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-19T14:38:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.015+0000 7f6b5bae83c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-19T14:38:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.015+0000 7f6b5bae83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T14:38:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.015+0000 7f6b5bae83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T14:38:10.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.015+0000 7f6b5bae83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T14:38:10.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.019+0000 7f6b5bae83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-19T14:38:10.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:10.591+0000 7f6b5bae83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:11.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:11.375+0000 7f6b5bae83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:12.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:12.183+0000 7f6b5bae83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:13.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-19T14:38:13.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.239+0000 7f4234eea3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-19T14:38:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.239+0000 7f4234eea3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-19T14:38:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.239+0000 7f4234eea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T14:38:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.239+0000 7f4234eea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T14:38:13.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.239+0000 7f4234eea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T14:38:13.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.243+0000 7f4234eea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-19T14:38:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:13.815+0000 7f4234eea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:14.595+0000 7f4234eea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:15.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:15.383+0000 7f4234eea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:16.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-19T14:38:16.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.447+0000 7fec1e6de3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-19T14:38:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.447+0000 7fec1e6de3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-19T14:38:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.447+0000 7fec1e6de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T14:38:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.447+0000 7fec1e6de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T14:38:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.447+0000 7fec1e6de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T14:38:16.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:16.451+0000 7fec1e6de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-19T14:38:17.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:17.019+0000 7fec1e6de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:17.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:17.811+0000 7fec1e6de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:18.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:18.599+0000 7fec1e6de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:19.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-19T14:38:19.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.647+0000 7f5279ae43c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-19T14:38:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.647+0000 7f5279ae43c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-19T14:38:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.647+0000 7f5279ae43c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T14:38:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.647+0000 7f5279ae43c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T14:38:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.647+0000 7f5279ae43c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T14:38:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:19.651+0000 7f5279ae43c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-19T14:38:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:20.223+0000 7f5279ae43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:21.010+0000 7f5279ae43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:21.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:21.798+0000 7f5279ae43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:22.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-19T14:38:22.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-19T14:38:22.774 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:22.774 INFO:tasks.ceph:role: osd.4 2023-12-19T14:38:22.774 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:38:22.774 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:23.375 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:23.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-12-19T14:38:23.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-19T14:38:23.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-19T14:38:23.525 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:23.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-19T14:38:23.609 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:23.609 INFO:tasks.ceph:role: osd.5 2023-12-19T14:38:23.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:38:23.610 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:24.164 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:24.169 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-12-19T14:38:24.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-19T14:38:24.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-19T14:38:24.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:24.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-19T14:38:24.346 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:24.346 INFO:tasks.ceph:role: osd.6 2023-12-19T14:38:24.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:38:24.347 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-19T14:38:24.907 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:24.907 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:24.907 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:24.907 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 2023-12-19T14:38:24.907 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:24.908 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:24.908 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:24.908 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:24.908 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:24.908 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:24.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-12-19T14:38:24.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-19T14:38:25.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-19T14:38:25.061 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:25.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-19T14:38:25.150 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T14:38:25.150 INFO:tasks.ceph:role: osd.7 2023-12-19T14:38:25.150 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:38:25.150 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-19T14:38:25.736 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T14:38:25.736 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T14:38:25.737 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T14:38:25.741 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-12-19T14:38:25.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-19T14:38:25.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-19T14:38:25.902 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T14:38:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:38:25.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.049+0000 7fbec36e23c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-19T14:38:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.053+0000 7fbec36e23c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-19T14:38:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.053+0000 7fbec36e23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T14:38:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.053+0000 7fbec36e23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T14:38:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.053+0000 7fbec36e23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T14:38:26.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.057+0000 7fbec36e23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-19T14:38:26.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:26.633+0000 7fbec36e23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:27.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:27.441+0000 7fbec36e23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:28.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:28.233+0000 7fbec36e23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:29.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-19T14:38:29.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:29.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.269+0000 7f0f550f63c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-19T14:38:29.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.273+0000 7f0f550f63c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-19T14:38:29.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.273+0000 7f0f550f63c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T14:38:29.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.273+0000 7f0f550f63c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T14:38:29.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.273+0000 7f0f550f63c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T14:38:29.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.273+0000 7f0f550f63c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-19T14:38:29.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:29.849+0000 7f0f550f63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:30.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:30.633+0000 7f0f550f63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:31.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:31.421+0000 7f0f550f63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:32.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-19T14:38:32.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:32.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-19T14:38:32.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-19T14:38:32.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T14:38:32.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T14:38:32.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T14:38:32.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:32.481+0000 7f2d8a4eb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-19T14:38:33.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:33.057+0000 7f2d8a4eb3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:33.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:33.849+0000 7f2d8a4eb3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:34.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:38:34.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:38:34.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:34.637+0000 7f2d8a4eb3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:35.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-19T14:38:35.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:35.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.701+0000 7f348fef73c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-19T14:38:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.705+0000 7f348fef73c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-19T14:38:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.705+0000 7f348fef73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T14:38:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.705+0000 7f348fef73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T14:38:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.705+0000 7f348fef73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T14:38:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:35.705+0000 7f348fef73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-19T14:38:36.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:36.277+0000 7f348fef73c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:37.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:37.061+0000 7f348fef73c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:37.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:37.861+0000 7f348fef73c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T14:38:38.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-19T14:38:38.785 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-19T14:38:38.786 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:38.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-19T14:38:38.821 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:38.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-19T14:38:38.902 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:38.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-19T14:38:38.984 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:38.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-19T14:38:39.059 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:39.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-19T14:38:39.141 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:39.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-19T14:38:39.222 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:39.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-19T14:38:39.311 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-19T14:38:39.346 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-19T14:38:39.424 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-19T14:38:39.503 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-19T14:38:39.582 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-19T14:38:39.659 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-19T14:38:39.735 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-19T14:38:39.817 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:39.817 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-19T14:38:39.836 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:39.836 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-12-19T14:38:39.901 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.901 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-19T14:38:39.921 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:39.921 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-12-19T14:38:39.988 INFO:tasks.ceph:Adding keys to all mons... 2023-12-19T14:38:39.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-19T14:38:39.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-19T14:38:40.037 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw== 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.a] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ== 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.c] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg== 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ== 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA== 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2023-12-19T14:38:40.038 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.b] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.d] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ== 2023-12-19T14:38:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg== 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg== 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA== 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout:[client.2] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA== 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw== 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout:[client.3] 2023-12-19T14:38:40.040 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.y] 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.0] 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.1] 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA== 2023-12-19T14:38:40.061 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.2] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.3] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.x] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.4] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.5] 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ== 2023-12-19T14:38:40.062 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.6] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg== 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.7] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg== 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA== 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout:[client.2] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA== 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout:[client.1] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw== 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout:[client.3] 2023-12-19T14:38:40.063 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA== 2023-12-19T14:38:40.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T14:38:40.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T14:38:40.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T14:38:40.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T14:38:40.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T14:38:40.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:40.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T14:38:41.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T14:38:41.592 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-19T14:38:41.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-19T14:38:41.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T14:38:41.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-19T14:38:41.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-19T14:38:41.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T14:38:41.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-19T14:38:42.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-19T14:38:42.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T14:38:42.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-19T14:38:42.261 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:42.268 DEBUG:teuthology.orchestra.run.smithi125:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-19T14:38:42.318 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-19T14:38:42.318 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-19T14:38:42.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-19T14:38:42.332 INFO:tasks.ceph.mon.a:Started 2023-12-19T14:38:42.332 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-19T14:38:42.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-19T14:38:42.368 INFO:tasks.ceph.mon.b:Started 2023-12-19T14:38:42.368 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-19T14:38:42.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-19T14:38:42.373 INFO:tasks.ceph.mon.c:Started 2023-12-19T14:38:42.373 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-19T14:38:42.373 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-19T14:38:42.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-19T14:38:42.381 INFO:tasks.ceph.mgr.y:Started 2023-12-19T14:38:42.381 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-19T14:38:42.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-19T14:38:42.385 INFO:tasks.ceph.mgr.x:Started 2023-12-19T14:38:42.386 INFO:tasks.ceph:Setting crush tunables to default 2023-12-19T14:38:42.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2023-12-19T14:38:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 Processor -- start 2023-12-19T14:38:42.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 -- start start 2023-12-19T14:38:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 0x7fc92815c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc928159b80 con 0x7fc9280c7a80 2023-12-19T14:38:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9280c8220 con 0x7fc928141e70 2023-12-19T14:38:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc92ce16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9280c7f40 con 0x7fc9280c53f0 2023-12-19T14:38:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc9277fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46952/0 (socket says 172.21.15.50:46952) 2023-12-19T14:38:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 -- 172.21.15.50:0/1863583888 learned_addr learned my addr 172.21.15.50:0/1863583888 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc927fff700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 0x7fc92815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.538+0000 7fc926ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc9280c7a80 0x7fc9280c4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33410/0 (socket says 172.21.15.50:33410) 2023-12-19T14:38:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc927fff700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 msgr2=0x7fc92815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 msgr2=0x7fc9280c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc927fff700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 msgr2=0x7fc92815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 msgr2=0x7fc9280c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc927fff700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 0x7fc92815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:38:42.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.542+0000 7fc927fff700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 0x7fc92815c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:38:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897870769 0 0) 0x7fc928159b80 con 0x7fc9280c7a80 2023-12-19T14:38:42.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc910003570 con 0x7fc9280c7a80 2023-12-19T14:38:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 898558381 0 0) 0x7fc910003570 con 0x7fc9280c7a80 2023-12-19T14:38:42.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 msgr2=0x7fc92815c970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T14:38:42.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.125:3300/0 conn(0x7fc9280c53f0 0x7fc92815c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.734+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 msgr2=0x7fc9280c7540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 --2- 172.21.15.50:0/1863583888 >> v2:172.21.15.50:3300/0 conn(0x7fc928141e70 0x7fc9280c7540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928159530 con 0x7fc9280c7a80 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fc91c001470 con 0x7fc9280c7a80 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 >> v1:172.21.15.50:6789/0 conn(0x7fc9280c7a80 legacy=0x7fc9280c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910003a60 0x7fc910003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 0x7fc910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9100051c0 0x7fc910014270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280c8220 con 0x7fc910004810 2023-12-19T14:38:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280c7f40 con 0x7fc9100051c0 2023-12-19T14:38:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc910014970 con 0x7fc910003a60 2023-12-19T14:38:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910003a60 0x7fc910003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 0x7fc910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc927fff700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9100051c0 0x7fc910014270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 -- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910003a60 msgr2=0x7fc910003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910003a60 0x7fc910003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 -- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9100051c0 msgr2=0x7fc910014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9100051c0 0x7fc910014270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc927fff700 1 -- 172.21.15.50:0/1863583888 reap_dead start 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 -- 172.21.15.50:0/1863583888 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928159530 con 0x7fc910004810 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910003a60 0x7fc910003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:42.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.738+0000 7fc926ffd700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 0x7fc910004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c390 tx=0x7fc91c007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fa1ae33e0ff5a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00cc40 con 0x7fc910004810 2023-12-19T14:38:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c00cda0 con 0x7fc910004810 2023-12-19T14:38:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc9267fc700 1 -- 172.21.15.50:0/1863583888 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c005a90 con 0x7fc910004810 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 -- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 msgr2=0x7fc910004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 0x7fc910004c80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c390 tx=0x7fc91c007910 comp rx=0 tx=0).stop 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 -- 172.21.15.50:0/1863583888 shutdown_connections 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1863583888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc910004810 0x7fc910004c80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 -- 172.21.15.50:0/1863583888 >> 172.21.15.50:0/1863583888 conn(0x7fc928067b50 msgr2=0x7fc9280cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 -- 172.21.15.50:0/1863583888 shutdown_connections 2023-12-19T14:38:42.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.742+0000 7fc92ce16700 1 -- 172.21.15.50:0/1863583888 wait complete. 2023-12-19T14:38:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 Processor -- start 2023-12-19T14:38:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 -- start start 2023-12-19T14:38:42.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 0x7fc928138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 0x7fc92813a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928158bb0 con 0x7fc9281380b0 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928138b00 con 0x7fc928139ba0 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc92ce16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc928138de0 con 0x7fc910004810 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc9277fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc926ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 0x7fc928138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc9277fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:43166/0 (socket says 172.21.15.50:43166) 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc926ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 0x7fc928138520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46990/0 (socket says 172.21.15.50:46990) 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc9277fe700 1 -- 172.21.15.50:0/1183775032 learned_addr learned my addr 172.21.15.50:0/1183775032 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:42.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.746+0000 7fc927fff700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 0x7fc92813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 -- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 msgr2=0x7fc92813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 0x7fc92813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 -- 172.21.15.50:0/1183775032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 msgr2=0x7fc928138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 0x7fc928138520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 -- 172.21.15.50:0/1183775032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9281390c0 con 0x7fc910004810 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc927fff700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 0x7fc92813a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:38:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc9277fe700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc918011940 tx=0x7fc918008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91800c070 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc918015740 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc918015ac0 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9281393a0 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280c0ec0 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc91801e6d0 con 0x7fc910004810 2023-12-19T14:38:42.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc91800f7c0 con 0x7fc910004810 2023-12-19T14:38:42.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.750+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92804be40 con 0x7fc910004810 2023-12-19T14:38:42.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.754+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc918023060 con 0x7fc910004810 2023-12-19T14:38:42.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.842+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc928159810 con 0x7fc910004810 2023-12-19T14:38:42.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc90f7fe700 1 -- 172.21.15.50:0/1183775032 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc91802c090 con 0x7fc910004810 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 msgr2=0x7fc9280c2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc918011940 tx=0x7fc918008040 comp rx=0 tx=0).stop 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 shutdown_connections 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc928139ba0 0x7fc92813a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9281380b0 0x7fc928138520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 --2- 172.21.15.50:0/1183775032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc910004810 0x7fc9280c2540 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.854+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 >> 172.21.15.50:0/1183775032 conn(0x7fc928067b50 msgr2=0x7fc92806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.858+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 shutdown_connections 2023-12-19T14:38:42.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:42.858+0000 7fc92ce16700 1 -- 172.21.15.50:0/1183775032 wait complete. 2023-12-19T14:38:42.864 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2023-12-19T14:38:42.881 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-19T14:38:42.881 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:42.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-19T14:38:42.913 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:42.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-19T14:38:42.976 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:42.970+0000 7f699a9c6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:38:42.981 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:42.973+0000 7fc13e347000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:38:42.990 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:42.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-19T14:38:43.074 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-19T14:38:43.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-19T14:38:43.158 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:43.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-19T14:38:43.193 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:43.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-19T14:38:43.265 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:43.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-19T14:38:43.342 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-12-19T14:38:43.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-19T14:38:43.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new b2790f77-b8c8-4d2d-ba35-18140a10a9ae 0 2023-12-19T14:38:43.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.601+0000 7f5d9e8f0700 1 Processor -- start 2023-12-19T14:38:43.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.601+0000 7f5d9e8f0700 1 -- start start 2023-12-19T14:38:43.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9e8f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5d980cb970 0x7f5d9806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9e8f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d980cbdf0 con 0x7f5d980cecd0 2023-12-19T14:38:43.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9e8f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d980cc0d0 con 0x7f5d98158e20 2023-12-19T14:38:43.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9e8f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d980cc3b0 con 0x7f5d980cb970 2023-12-19T14:38:43.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d97fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5d98158e20 0x7f5d980ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53260/0 (socket says 172.21.15.125:53260) 2023-12-19T14:38:43.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d97fff700 1 -- 172.21.15.125:0/339543362 learned_addr learned my addr 172.21.15.125:0/339543362 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-12-19T14:38:43.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635062144 0 0) 0x7f5d980cc0d0 con 0x7f5d98158e20 2023-12-19T14:38:43.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 --2- 172.21.15.125:0/339543362 >> v2:172.21.15.50:3300/0 conn(0x7f5d980cb970 0x7f5d9806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d88003570 con 0x7f5d98158e20 2023-12-19T14:38:43.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342144032 0 0) 0x7f5d88003570 con 0x7f5d98158e20 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 >> v2:172.21.15.50:3300/0 conn(0x7f5d980cb970 msgr2=0x7f5d9806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 --2- 172.21.15.125:0/339543362 >> v2:172.21.15.50:3300/0 conn(0x7f5d980cb970 0x7f5d9806acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 >> v1:172.21.15.50:6789/0 conn(0x7f5d980cecd0 legacy=0x7f5d9806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d980cc690 con 0x7f5d98158e20 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.1 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f5d8c001470 con 0x7f5d98158e20 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 >> v1:172.21.15.125:6789/0 conn(0x7f5d98158e20 legacy=0x7f5d980ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d88003a60 0x7f5d88003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 0x7f5d88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d880051c0 0x7f5d8800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980cc3b0 con 0x7f5d88004810 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8800c0e0 con 0x7f5d880051c0 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8800c2a0 con 0x7f5d88003a60 2023-12-19T14:38:43.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9ce8d700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d880051c0 0x7f5d8800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 0x7f5d88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d97fff700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d88003a60 0x7f5d88003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 -- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d88003a60 msgr2=0x7f5d88003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d88003a60 0x7f5d88003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 -- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d880051c0 msgr2=0x7f5d8800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d880051c0 0x7f5d8800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d977fe700 1 -- 172.21.15.125:0/339543362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88003570 con 0x7f5d88004810 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9ce8d700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d880051c0 0x7f5d8800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:43.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.605+0000 7f5d9ce8d700 1 -- 172.21.15.125:0/339543362 reap_dead start 2023-12-19T14:38:43.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d977fe700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 0x7f5d88004c80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5d9806a570 tx=0x7f5d840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ea59c5f7afa2ef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:43.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400e710 con 0x7f5d88004810 2023-12-19T14:38:43.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8400e870 con 0x7f5d88004810 2023-12-19T14:38:43.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d96ffd700 1 -- 172.21.15.125:0/339543362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400ec30 con 0x7f5d88004810 2023-12-19T14:38:43.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 msgr2=0x7f5d88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 0x7f5d88004c80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5d9806a570 tx=0x7f5d840091a0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/339543362 shutdown_connections 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/339543362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d88004810 0x7f5d88004c80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/339543362 >> 172.21.15.125:0/339543362 conn(0x7f5d98067bb0 msgr2=0x7f5d980d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/339543362 shutdown_connections 2023-12-19T14:38:43.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.609+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/339543362 wait complete. 2023-12-19T14:38:43.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 Processor -- start 2023-12-19T14:38:43.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 -- start start 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 0x7f5d9814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d980b4870 0x7f5d980b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980cd500 con 0x7f5d980b4870 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9814e5d0 con 0x7f5d88004810 2023-12-19T14:38:43.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9e8f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980b5250 con 0x7f5d980ccb50 2023-12-19T14:38:43.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d97fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 0x7f5d9814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d977fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d977fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48636/0 (socket says 172.21.15.125:48636) 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d977fe700 1 -- 172.21.15.125:0/1971912376 learned_addr learned my addr 172.21.15.125:0/1971912376 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.613+0000 7f5d9ce8d700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d980b4870 0x7f5d980b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 -- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 msgr2=0x7f5d980ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 -- 172.21.15.125:0/1971912376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d980b4870 msgr2=0x7f5d980b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d980b4870 0x7f5d980b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 -- 172.21.15.125:0/1971912376 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d981591f0 con 0x7f5d88004810 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d977fe700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:43.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d97fff700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 0x7f5d9814e090 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c007390 tx=0x7f5d8c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:43.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c018030 con 0x7f5d88004810 2023-12-19T14:38:43.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c00f660 con 0x7f5d88004810 2023-12-19T14:38:43.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c00fa00 con 0x7f5d88004810 2023-12-19T14:38:43.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d981594d0 con 0x7f5d88004810 2023-12-19T14:38:43.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d98159a90 con 0x7f5d88004810 2023-12-19T14:38:43.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5d8c01b070 con 0x7f5d88004810 2023-12-19T14:38:43.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5d8c019070 con 0x7f5d88004810 2023-12-19T14:38:43.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.617+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d98071120 con 0x7f5d88004810 2023-12-19T14:38:43.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.621+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5d8c00fdf0 con 0x7f5d88004810 2023-12-19T14:38:43.696 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:43.690+0000 7f699a9c6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:38:43.702 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:43.693+0000 7fc13e347000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:38:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.701+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b2790f77-b8c8-4d2d-ba35-18140a10a9ae", "id": 0} v 0) v1 -- 0x7f5d9806a7a0 con 0x7f5d88004810 2023-12-19T14:38:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.709+0000 7f5d83fff700 1 -- 172.21.15.125:0/1971912376 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2790f77-b8c8-4d2d-ba35-18140a10a9ae", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d8c01e830 con 0x7f5d88004810 2023-12-19T14:38:43.718 INFO:teuthology.orchestra.run.smithi125.stdout:0 2023-12-19T14:38:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 msgr2=0x7f5d9814e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 0x7f5d9814e090 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c007390 tx=0x7f5d8c001e70 comp rx=0 tx=0).stop 2023-12-19T14:38:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 shutdown_connections 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d980b4870 0x7f5d980b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5d980ccb50 0x7f5d980ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 --2- 172.21.15.125:0/1971912376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d88004810 0x7f5d9814e090 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 >> 172.21.15.125:0/1971912376 conn(0x7f5d98067bb0 msgr2=0x7f5d9814f420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 shutdown_connections 2023-12-19T14:38:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.713+0000 7f5d9e8f0700 1 -- 172.21.15.125:0/1971912376 wait complete. 2023-12-19T14:38:43.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new e0075d32-ebb0-4a3a-b24d-81059f4b0775 1 2023-12-19T14:38:43.778 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:43.770+0000 7f699a9c6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:38:43.784 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:43.777+0000 7fc13e347000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:38:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.873+0000 7f01ab59e700 1 Processor -- start 2023-12-19T14:38:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 -- start start 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01ac158fd0 0x7f01ac06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f01ac0cc610 0x7f01ac0c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac06d750 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0cbaf0 con 0x7f01ac0cc610 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01ab59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac06e120 con 0x7f01ac158fd0 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01a9d9b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01ac0cc1e0 0x7f01ac0c8420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53266/0 (socket says 172.21.15.125:53266) 2023-12-19T14:38:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01a9d9b700 1 -- 172.21.15.125:0/1590113715 learned_addr learned my addr 172.21.15.125:0/1590113715 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01aad9d700 1 --2- 172.21.15.125:0/1590113715 >> v2:172.21.15.50:3300/0 conn(0x7f01ac0cc610 0x7f01ac0c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01ac158fd0 0x7f01ac06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.877+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522580406 0 0) 0x7f01ac06d750 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0194003570 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282366474 0 0) 0x7f0194003570 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01ac158fd0 msgr2=0x7f01ac06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01ac158fd0 0x7f01ac06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 >> v2:172.21.15.50:3300/0 conn(0x7f01ac0cc610 msgr2=0x7f01ac0c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 --2- 172.21.15.125:0/1590113715 >> v2:172.21.15.50:3300/0 conn(0x7f01ac0cc610 0x7f01ac0c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac06cb40 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f01a0001470 con 0x7f01ac0cc1e0 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 >> v1:172.21.15.125:6789/0 conn(0x7f01ac0cc1e0 legacy=0x7f01ac0c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0194003a60 0x7f0194003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194004810 0x7f0194004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01940051c0 0x7f019400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0cbaf0 con 0x7f01940051c0 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac06e120 con 0x7f0194003a60 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019400bfa0 con 0x7f0194004810 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0194003a60 0x7f0194003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a9d9b700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194004810 0x7f0194004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aad9d700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01940051c0 0x7f019400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 -- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194004810 msgr2=0x7f0194004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194004810 0x7f0194004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 -- 172.21.15.125:0/1590113715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01940051c0 msgr2=0x7f019400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01940051c0 0x7f019400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 -- 172.21.15.125:0/1590113715 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0194003570 con 0x7f0194003a60 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a9d9b700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194004810 0x7f0194004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0194003a60 0x7f0194003e70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f01ac06b760 tx=0x7f019c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1066d6395d5cbf7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00e640 con 0x7f0194003a60 2023-12-19T14:38:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c00e7a0 con 0x7f0194003a60 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01a959a700 1 -- 172.21.15.125:0/1590113715 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00eb40 con 0x7f0194003a60 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01ab59e700 1 -- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0194003a60 msgr2=0x7f0194003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01ab59e700 1 --2- 172.21.15.125:0/1590113715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0194003a60 0x7f0194003e70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f01ac06b760 tx=0x7f019c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01aad9d700 1 -- 172.21.15.125:0/1590113715 reap_dead start 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01ab59e700 1 -- 172.21.15.125:0/1590113715 shutdown_connections 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01ab59e700 1 -- 172.21.15.125:0/1590113715 >> 172.21.15.125:0/1590113715 conn(0x7f01ac0cdc00 msgr2=0x7f01ac0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.881+0000 7f01ab59e700 1 -- 172.21.15.125:0/1590113715 shutdown_connections 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 -- 172.21.15.125:0/1590113715 wait complete. 2023-12-19T14:38:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 Processor -- start 2023-12-19T14:38:43.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 -- start start 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac16a370 0x7f01ac0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 0x7f01ac0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c5c80 con 0x7f01ac16a370 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0af9a0 con 0x7f01ac16a7e0 2023-12-19T14:38:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01ab59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c5ed0 con 0x7f0194003a60 2023-12-19T14:38:43.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01aad9d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 0x7f01ac0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48658/0 (socket says 172.21.15.125:48658) 2023-12-19T14:38:43.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01aad9d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 0x7f01ac0b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40268/0 (socket says 172.21.15.125:40268) 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01aa59c700 1 -- 172.21.15.125:0/1369359476 learned_addr learned my addr 172.21.15.125:0/1369359476 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.885+0000 7f01a9d9b700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac16a370 0x7f01ac0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 -- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 msgr2=0x7f01ac0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 0x7f01ac0b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 -- 172.21.15.125:0/1369359476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac16a370 msgr2=0x7f01ac0b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac16a370 0x7f01ac0b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 -- 172.21.15.125:0/1369359476 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0c61b0 con 0x7f0194003a60 2023-12-19T14:38:43.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01aa59c700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01940046d0 tx=0x7f019c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c023070 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c00e910 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c01e030 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac0c6490 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac0c6a50 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f019c0136f0 con 0x7f0194003a60 2023-12-19T14:38:43.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f019c01c810 con 0x7f0194003a60 2023-12-19T14:38:43.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.889+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac0c5ed0 con 0x7f0194003a60 2023-12-19T14:38:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.893+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f019c035060 con 0x7f0194003a60 2023-12-19T14:38:43.922 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:43.918+0000 7f699a9c6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:38:43.925 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:43.917+0000 7fc13e347000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:38:43.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.977+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e0075d32-ebb0-4a3a-b24d-81059f4b0775", "id": 1} v 0) v1 -- 0x7f01ac04be40 con 0x7f0194003a60 2023-12-19T14:38:43.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.985+0000 7f019a7fc700 1 -- 172.21.15.125:0/1369359476 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e0075d32-ebb0-4a3a-b24d-81059f4b0775", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f019c03e0b0 con 0x7f0194003a60 2023-12-19T14:38:43.993 INFO:teuthology.orchestra.run.smithi125.stdout:1 2023-12-19T14:38:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 msgr2=0x7f01ac1690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01940046d0 tx=0x7f019c01a000 comp rx=0 tx=0).stop 2023-12-19T14:38:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 shutdown_connections 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01ac16a7e0 0x7f01ac0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01ac16a370 0x7f01ac0b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 --2- 172.21.15.125:0/1369359476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0194003a60 0x7f01ac1690e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 >> 172.21.15.125:0/1369359476 conn(0x7f01ac0cdc00 msgr2=0x7f01ac0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 shutdown_connections 2023-12-19T14:38:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:43.989+0000 7f01ab59e700 1 -- 172.21.15.125:0/1369359476 wait complete. 2023-12-19T14:38:44.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new 10ce176d-07e0-4d6e-adb0-d6ea4c52b378 2 2023-12-19T14:38:44.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 Processor -- start 2023-12-19T14:38:44.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 -- start start 2023-12-19T14:38:44.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f37300cb970 0x7f37300ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300cbdf0 con 0x7f37300ca2c0 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37300cc0d0 con 0x7f37300cb970 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3737185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37300cc3b0 con 0x7f3730158e20 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3735722700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f37300ca2c0 0x7f373006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53270/0 (socket says 172.21.15.125:53270) 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3735722700 1 -- 172.21.15.125:0/4201959762 learned_addr learned my addr 172.21.15.125:0/4201959762 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f3734f21700 1 --2- 172.21.15.125:0/4201959762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.201+0000 7f372ffff700 1 --2- 172.21.15.125:0/4201959762 >> v2:172.21.15.50:3300/0 conn(0x7f37300cb970 0x7f37300ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f372f7fe700 1 -- 172.21.15.125:0/4201959762 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285447108 0 0) 0x7f37300cbdf0 con 0x7f37300ca2c0 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f372f7fe700 1 -- 172.21.15.125:0/4201959762 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3720003570 con 0x7f37300ca2c0 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3734f21700 1 -- 172.21.15.125:0/4201959762 >> v2:172.21.15.50:3300/0 conn(0x7f37300cb970 msgr2=0x7f37300ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3734f21700 1 --2- 172.21.15.125:0/4201959762 >> v2:172.21.15.50:3300/0 conn(0x7f37300cb970 0x7f37300ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3734f21700 1 -- 172.21.15.125:0/4201959762 >> v1:172.21.15.125:6789/0 conn(0x7f37300ca2c0 legacy=0x7f373006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3734f21700 1 -- 172.21.15.125:0/4201959762 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37300cc690 con 0x7f3730158e20 2023-12-19T14:38:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3734f21700 1 --2- 172.21.15.125:0/4201959762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300ce5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f37240098e0 tx=0x7f37240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d82a9cba4dc01d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f372f7fe700 1 -- 172.21.15.125:0/4201959762 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011630 con 0x7f3730158e20 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f372f7fe700 1 -- 172.21.15.125:0/4201959762 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3724011790 con 0x7f3730158e20 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f372f7fe700 1 -- 172.21.15.125:0/4201959762 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011b30 con 0x7f3730158e20 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 -- 172.21.15.125:0/4201959762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 msgr2=0x7f37300ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 --2- 172.21.15.125:0/4201959762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300ce5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f37240098e0 tx=0x7f37240091a0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 -- 172.21.15.125:0/4201959762 shutdown_connections 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 --2- 172.21.15.125:0/4201959762 >> v2:172.21.15.50:3300/0 conn(0x7f37300cb970 0x7f37300ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 --2- 172.21.15.125:0/4201959762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300ce5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 -- 172.21.15.125:0/4201959762 >> 172.21.15.125:0/4201959762 conn(0x7f3730067bb0 msgr2=0x7f37300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 -- 172.21.15.125:0/4201959762 shutdown_connections 2023-12-19T14:38:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.205+0000 7f3737185700 1 -- 172.21.15.125:0/4201959762 wait complete. 2023-12-19T14:38:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 Processor -- start 2023-12-19T14:38:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- start start 2023-12-19T14:38:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 0x7f37301492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 0x7f37300b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37300c7190 con 0x7f37300ca2c0 2023-12-19T14:38:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37300c7d10 con 0x7f3730149800 2023-12-19T14:38:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37300b5d70 con 0x7f3730158e20 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 0x7f37301492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48680/0 (socket says 172.21.15.125:48680) 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3735722700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 0x7f37300b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372ffff700 1 -- 172.21.15.125:0/1115691559 learned_addr learned my addr 172.21.15.125:0/1115691559 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 -- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 msgr2=0x7f37300b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 -- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 msgr2=0x7f37300b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 0x7f37300b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 -- 172.21.15.125:0/1115691559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724009420 con 0x7f37300ca2c0 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3735722700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 0x7f37300b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372ffff700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3734f21700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 0x7f37301492c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f37300cfd10 tx=0x7f3724017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724000bc0 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3724011900 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724000d20 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37300c73e0 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.209+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37300c79a0 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.213+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3724023030 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.213+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f3724025800 con 0x7f37300ca2c0 2023-12-19T14:38:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.213+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3730071120 con 0x7f37300ca2c0 2023-12-19T14:38:44.220 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:44.214+0000 7f699a9c6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:38:44.221 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:44.213+0000 7fc13e347000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:38:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.217+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3724015020 con 0x7f37300ca2c0 2023-12-19T14:38:44.294 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:44.290+0000 7f699a9c6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:38:44.295 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:44.285+0000 7fc13e347000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:38:44.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.297+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "10ce176d-07e0-4d6e-adb0-d6ea4c52b378", "id": 2} v 0) v1 -- 0x7f373006a7a0 con 0x7f37300ca2c0 2023-12-19T14:38:44.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.301+0000 7f372cff9700 1 -- 172.21.15.125:0/1115691559 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "10ce176d-07e0-4d6e-adb0-d6ea4c52b378", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3724026a50 con 0x7f37300ca2c0 2023-12-19T14:38:44.310 INFO:teuthology.orchestra.run.smithi125.stdout:2 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 msgr2=0x7f37301492c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 0x7f37301492c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f37300cfd10 tx=0x7f3724017000 comp rx=0 tx=0).stop 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 shutdown_connections 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730149800 0x7f37300b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3730158e20 0x7f37300b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 --2- 172.21.15.125:0/1115691559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37300ca2c0 0x7f37301492c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 >> 172.21.15.125:0/1115691559 conn(0x7f3730067bb0 msgr2=0x7f3730156700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 shutdown_connections 2023-12-19T14:38:44.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.305+0000 7f3737185700 1 -- 172.21.15.125:0/1115691559 wait complete. 2023-12-19T14:38:44.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new b7522fc3-0859-4ba8-8d9e-f60d66a999ee 3 2023-12-19T14:38:44.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.477+0000 7fa5b3f45700 1 Processor -- start 2023-12-19T14:38:44.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.481+0000 7fa5b3f45700 1 -- start start 2023-12-19T14:38:44.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5ac0ca2c0 0x7fa5ac0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0cbdf0 con 0x7fa5ac0cb970 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ac0cc0d0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac0cc3b0 con 0x7fa5ac158e20 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48696/0 (socket says 172.21.15.125:48696) 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1646222663 learned_addr learned my addr 172.21.15.125:0/1646222663 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:44.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b24e2700 1 --2- 172.21.15.125:0/1646222663 >> v2:172.21.15.125:3300/0 conn(0x7fa5ac0ca2c0 0x7fa5ac0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1646222663 >> v2:172.21.15.125:3300/0 conn(0x7fa5ac0ca2c0 msgr2=0x7fa5ac0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 --2- 172.21.15.125:0/1646222663 >> v2:172.21.15.125:3300/0 conn(0x7fa5ac0ca2c0 0x7fa5ac0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1646222663 >> v1:172.21.15.125:6789/0 conn(0x7fa5ac0cb970 legacy=0x7fa5ac06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1646222663 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0cc690 con 0x7fa5ac158e20 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b1ce1700 1 --2- 172.21.15.125:0/1646222663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80099b0 tx=0x7fa5a8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0ec13fdf2a0718 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b0cdf700 1 -- 172.21.15.125:0/1646222663 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800b860 con 0x7fa5ac158e20 2023-12-19T14:38:44.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b0cdf700 1 -- 172.21.15.125:0/1646222663 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a800b9c0 con 0x7fa5ac158e20 2023-12-19T14:38:44.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b0cdf700 1 -- 172.21.15.125:0/1646222663 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800bd60 con 0x7fa5ac158e20 2023-12-19T14:38:44.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1646222663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 msgr2=0x7fa5ac0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1646222663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80099b0 tx=0x7fa5a8009250 comp rx=0 tx=0).stop 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1646222663 shutdown_connections 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1646222663 >> v2:172.21.15.125:3300/0 conn(0x7fa5ac0ca2c0 0x7fa5ac0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1646222663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac0ce5c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1646222663 >> 172.21.15.125:0/1646222663 conn(0x7fa5ac067bb0 msgr2=0x7fa5ac0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1646222663 shutdown_connections 2023-12-19T14:38:44.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.485+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1646222663 wait complete. 2023-12-19T14:38:44.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 Processor -- start 2023-12-19T14:38:44.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 -- start start 2023-12-19T14:38:44.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 0x7fa5ac14f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac154f50 con 0x7fa5ac151340 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac14fd60 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b3f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac151b00 con 0x7fa5ac158e20 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40294/0 (socket says 172.21.15.125:40294) 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1962753179 learned_addr learned my addr 172.21.15.125:0/1962753179 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b14e0700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b24e2700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 0x7fa5ac14f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 msgr2=0x7fa5ac1553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac1553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1962753179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 msgr2=0x7fa5ac14f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 0x7fa5ac14f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 -- 172.21.15.125:0/1962753179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80094f0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b14e0700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac1553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:38:44.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b24e2700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 0x7fa5ac14f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:38:44.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.489+0000 7fa5b1ce1700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0cc4f0 tx=0x7fa5a8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8024440 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a80245c0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac154620 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8024980 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac154be0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa5a801a6a0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fa5a80233e0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.493+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac071120 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.497+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa5a8015020 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.531 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:44.525+0000 7fc13e347000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:38:44.538 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:44.534+0000 7f699a9c6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:38:44.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.613+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b7522fc3-0859-4ba8-8d9e-f60d66a999ee", "id": 3} v 0) v1 -- 0x7fa5ac06a7a0 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.624 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:44.618+0000 7f699a9c6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:38:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.621+0000 7fa5a1ffb700 1 -- 172.21.15.125:0/1962753179 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7522fc3-0859-4ba8-8d9e-f60d66a999ee", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa5a801a950 con 0x7fa5ac0ca2c0 2023-12-19T14:38:44.629 INFO:teuthology.orchestra.run.smithi125.stdout:3 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 msgr2=0x7fa5ac0b5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0cc4f0 tx=0x7fa5a8017000 comp rx=0 tx=0).stop 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 shutdown_connections 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5ac151340 0x7fa5ac14f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa5ac158e20 0x7fa5ac1553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 --2- 172.21.15.125:0/1962753179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5ac0ca2c0 0x7fa5ac0b5cf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 >> 172.21.15.125:0/1962753179 conn(0x7fa5ac067bb0 msgr2=0x7fa5ac0cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 shutdown_connections 2023-12-19T14:38:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.625+0000 7fa5b3f45700 1 -- 172.21.15.125:0/1962753179 wait complete. 2023-12-19T14:38:44.646 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:44.637+0000 7fc13e347000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:38:44.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new 2a72128a-b8f2-4dc3-994b-651ffc33b923 4 2023-12-19T14:38:44.715 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:44.710+0000 7f699a9c6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:38:44.735 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:44.729+0000 7fc13e347000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:38:44.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.801+0000 7f9d4fa6e700 1 Processor -- start 2023-12-19T14:38:44.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 -- start start 2023-12-19T14:38:44.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9d480cecd0 0x7f9d4806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480cbdf0 con 0x7f9d48158e20 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d480cc0d0 con 0x7f9d480cecd0 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4fa6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480cc3b0 con 0x7f9d480cb970 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48728/0 (socket says 172.21.15.125:48728) 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d80a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9d48158e20 0x7f9d480ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53292/0 (socket says 172.21.15.125:53292) 2023-12-19T14:38:44.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 -- 172.21.15.125:0/2458346715 learned_addr learned my addr 172.21.15.125:0/2458346715 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:44.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4e00b700 1 --2- 172.21.15.125:0/2458346715 >> v2:172.21.15.125:3300/0 conn(0x7f9d480cecd0 0x7f9d4806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4c808700 1 -- 172.21.15.125:0/2458346715 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680006865 0 0) 0x7f9d480cbdf0 con 0x7f9d48158e20 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4c808700 1 -- 172.21.15.125:0/2458346715 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34003570 con 0x7f9d48158e20 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 -- 172.21.15.125:0/2458346715 >> v2:172.21.15.125:3300/0 conn(0x7f9d480cecd0 msgr2=0x7f9d4806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 --2- 172.21.15.125:0/2458346715 >> v2:172.21.15.125:3300/0 conn(0x7f9d480cecd0 0x7f9d4806b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 -- 172.21.15.125:0/2458346715 >> v1:172.21.15.125:6789/0 conn(0x7f9d48158e20 legacy=0x7f9d480ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 -- 172.21.15.125:0/2458346715 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d480cc690 con 0x7f9d480cb970 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.805+0000 7f9d4d009700 1 --2- 172.21.15.125:0/2458346715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9d380099b0 tx=0x7f9d38009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ce489706c78fe0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4c808700 1 -- 172.21.15.125:0/2458346715 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38009ea0 con 0x7f9d480cb970 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4c808700 1 -- 172.21.15.125:0/2458346715 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3800b860 con 0x7f9d480cb970 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4c808700 1 -- 172.21.15.125:0/2458346715 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800bc00 con 0x7f9d480cb970 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/2458346715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 msgr2=0x7f9d4806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/2458346715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9d380099b0 tx=0x7f9d38009250 comp rx=0 tx=0).stop 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/2458346715 shutdown_connections 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/2458346715 >> v2:172.21.15.125:3300/0 conn(0x7f9d480cecd0 0x7f9d4806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/2458346715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d480cb970 0x7f9d4806acc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/2458346715 >> 172.21.15.125:0/2458346715 conn(0x7f9d48067bb0 msgr2=0x7f9d480d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/2458346715 shutdown_connections 2023-12-19T14:38:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/2458346715 wait complete. 2023-12-19T14:38:44.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 Processor -- start 2023-12-19T14:38:44.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- start start 2023-12-19T14:38:44.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d48158e20 0x7f9d480c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d480c9070 0x7f9d480c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:44.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c94e0 con 0x7f9d480c9070 2023-12-19T14:38:44.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c7190 con 0x7f9d480cecd0 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.809+0000 7f9d4fa6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c7ff0 con 0x7f9d48158e20 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40308/0 (socket says 172.21.15.125:40308) 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 -- 172.21.15.125:0/784774857 learned_addr learned my addr 172.21.15.125:0/784774857 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d009700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d48158e20 0x7f9d480c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4e00b700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d480c9070 0x7f9d480c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 -- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d48158e20 msgr2=0x7f9d480c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d48158e20 0x7f9d480c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 -- 172.21.15.125:0/784774857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d480c9070 msgr2=0x7f9d480c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d480c9070 0x7f9d480c53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 -- 172.21.15.125:0/784774857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380094f0 con 0x7f9d480cecd0 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4d80a700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c5c0 tx=0x7f9d44007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44005a70 con 0x7f9d480cecd0 2023-12-19T14:38:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d44005bf0 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d480c8480 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440070d0 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9d44012400 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d480c7740 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f9d440128e0 con 0x7f9d480cecd0 2023-12-19T14:38:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.813+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d48071120 con 0x7f9d480cecd0 2023-12-19T14:38:44.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.817+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9d44017070 con 0x7f9d480cecd0 2023-12-19T14:38:44.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.897+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2a72128a-b8f2-4dc3-994b-651ffc33b923", "id": 4} v 0) v1 -- 0x7f9d4806a7a0 con 0x7f9d480cecd0 2023-12-19T14:38:44.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d3dffb700 1 -- 172.21.15.125:0/784774857 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a72128a-b8f2-4dc3-994b-651ffc33b923", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9d44012aa0 con 0x7f9d480cecd0 2023-12-19T14:38:44.911 INFO:teuthology.orchestra.run.smithi125.stdout:4 2023-12-19T14:38:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 msgr2=0x7f9d4814bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c5c0 tx=0x7f9d44007e00 comp rx=0 tx=0).stop 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 shutdown_connections 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d480c9070 0x7f9d480c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d48158e20 0x7f9d480c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 --2- 172.21.15.125:0/784774857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d480cecd0 0x7f9d4814bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 >> 172.21.15.125:0/784774857 conn(0x7f9d48067bb0 msgr2=0x7f9d4814caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 shutdown_connections 2023-12-19T14:38:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:44.905+0000 7f9d4fa6e700 1 -- 172.21.15.125:0/784774857 wait complete. 2023-12-19T14:38:44.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new 23091188-4eb7-4031-acbf-f650873398da 5 2023-12-19T14:38:45.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 Processor -- start 2023-12-19T14:38:45.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 -- start start 2023-12-19T14:38:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd780cb970 0x7fbd7806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd780cbdf0 con 0x7fbd78158e20 2023-12-19T14:38:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd780cc0d0 con 0x7fbd780cb970 2023-12-19T14:38:45.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.077+0000 7fbd80145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd780cc3b0 con 0x7fbd780cecd0 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48748/0 (socket says 172.21.15.125:48748) 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/3267765720 learned_addr learned my addr 172.21.15.125:0/3267765720 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7d6e0700 1 --2- 172.21.15.125:0/3267765720 >> v2:172.21.15.50:3300/0 conn(0x7fbd780cb970 0x7fbd7806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/3267765720 >> v2:172.21.15.50:3300/0 conn(0x7fbd780cb970 msgr2=0x7fbd7806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 --2- 172.21.15.125:0/3267765720 >> v2:172.21.15.50:3300/0 conn(0x7fbd780cb970 0x7fbd7806acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/3267765720 >> v1:172.21.15.50:6789/0 conn(0x7fbd78158e20 legacy=0x7fbd780ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/3267765720 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd780cc690 con 0x7fbd780cecd0 2023-12-19T14:38:45.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7e6e2700 1 --2- 172.21.15.125:0/3267765720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009f50 tx=0x7fbd70009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be9e720a2e4cbf4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7cedf700 1 -- 172.21.15.125:0/3267765720 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7000e070 con 0x7fbd780cecd0 2023-12-19T14:38:45.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7cedf700 1 -- 172.21.15.125:0/3267765720 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7000ba70 con 0x7fbd780cecd0 2023-12-19T14:38:45.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd7cedf700 1 -- 172.21.15.125:0/3267765720 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700092e0 con 0x7fbd780cecd0 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 -- 172.21.15.125:0/3267765720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 msgr2=0x7fbd7806b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 --2- 172.21.15.125:0/3267765720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009f50 tx=0x7fbd70009840 comp rx=0 tx=0).stop 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 -- 172.21.15.125:0/3267765720 shutdown_connections 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 --2- 172.21.15.125:0/3267765720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd780cecd0 0x7fbd7806b200 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 --2- 172.21.15.125:0/3267765720 >> v2:172.21.15.50:3300/0 conn(0x7fbd780cb970 0x7fbd7806acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 -- 172.21.15.125:0/3267765720 >> 172.21.15.125:0/3267765720 conn(0x7fbd78067bb0 msgr2=0x7fbd780d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 -- 172.21.15.125:0/3267765720 shutdown_connections 2023-12-19T14:38:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.081+0000 7fbd80145700 1 -- 172.21.15.125:0/3267765720 wait complete. 2023-12-19T14:38:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 Processor -- start 2023-12-19T14:38:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 -- start start 2023-12-19T14:38:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd780cecd0 0x7fbd780b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 0x7fbd78152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd780b57b0 con 0x7fbd780cecd0 2023-12-19T14:38:45.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd780b4ba0 con 0x7fbd7814bf30 2023-12-19T14:38:45.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd80145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd780b4e80 con 0x7fbd78158e20 2023-12-19T14:38:45.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7d6e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 0x7fbd78152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40312/0 (socket says 172.21.15.125:40312) 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/913147480 learned_addr learned my addr 172.21.15.125:0/913147480 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7d6e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 0x7fbd78152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48764/0 (socket says 172.21.15.125:48764) 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7dee1700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd780cecd0 0x7fbd780b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 msgr2=0x7fbd78152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 0x7fbd78152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/913147480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd780cecd0 msgr2=0x7fbd780b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd780cecd0 0x7fbd780b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.085+0000 7fbd7e6e2700 1 -- 172.21.15.125:0/913147480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70009ae0 con 0x7fbd7814bf30 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd7e6e2700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd70012000 tx=0x7fbd70018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700032f0 con 0x7fbd7814bf30 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70008ce0 con 0x7fbd7814bf30 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd780b5160 con 0x7fbd7814bf30 2023-12-19T14:38:45.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70026690 con 0x7fbd7814bf30 2023-12-19T14:38:45.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd780b5c40 con 0x7fbd7814bf30 2023-12-19T14:38:45.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbd70003040 con 0x7fbd7814bf30 2023-12-19T14:38:45.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fbd70026d00 con 0x7fbd7814bf30 2023-12-19T14:38:45.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.089+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd780b55e0 con 0x7fbd7814bf30 2023-12-19T14:38:45.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.093+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbd70021070 con 0x7fbd7814bf30 2023-12-19T14:38:45.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.177+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "23091188-4eb7-4031-acbf-f650873398da", "id": 5} v 0) v1 -- 0x7fbd78071120 con 0x7fbd7814bf30 2023-12-19T14:38:45.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.185+0000 7fbd6dffb700 1 -- 172.21.15.125:0/913147480 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23091188-4eb7-4031-acbf-f650873398da", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbd70035020 con 0x7fbd7814bf30 2023-12-19T14:38:45.193 INFO:teuthology.orchestra.run.smithi125.stdout:5 2023-12-19T14:38:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 msgr2=0x7fbd78155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd70012000 tx=0x7fbd70018000 comp rx=0 tx=0).stop 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 shutdown_connections 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd7814bf30 0x7fbd78155590 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd78158e20 0x7fbd78152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 --2- 172.21.15.125:0/913147480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd780cecd0 0x7fbd780b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 >> 172.21.15.125:0/913147480 conn(0x7fbd78067bb0 msgr2=0x7fbd7806adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 shutdown_connections 2023-12-19T14:38:45.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.189+0000 7fbd80145700 1 -- 172.21.15.125:0/913147480 wait complete. 2023-12-19T14:38:45.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new ff2c80ac-fb63-4c00-af3d-11845bb54a35 6 2023-12-19T14:38:45.236 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:45.230+0000 7f699a9c6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:38:45.297 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:45.289+0000 7fc13e347000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:38:45.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 Processor -- start 2023-12-19T14:38:45.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 -- start start 2023-12-19T14:38:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73b015a010 0x7f73b00c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 0x7f73b00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b0159b80 con 0x7f73b0141e70 2023-12-19T14:38:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00c8220 con 0x7f73b015a010 2023-12-19T14:38:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73b4c0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00c7f40 con 0x7f73b0142690 2023-12-19T14:38:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73b015a010 0x7f73b00c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73affff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 0x7f73b00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73b015a010 0x7f73b00c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40316/0 (socket says 172.21.15.125:40316) 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 -- 172.21.15.125:0/267420166 learned_addr learned my addr 172.21.15.125:0/267420166 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 -- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 msgr2=0x7f73b00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 0x7f73b00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 -- 172.21.15.125:0/267420166 >> v1:172.21.15.125:6789/0 conn(0x7f73b0141e70 legacy=0x7f73b00c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73aeffd700 1 -- 172.21.15.125:0/267420166 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0159530 con 0x7f73b015a010 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.349+0000 7f73affff700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 0x7f73b00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73aeffd700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b00c4ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73a40099b0 tx=0x7f73a4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2fee4a36ced6e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73ae7fc700 1 -- 172.21.15.125:0/267420166 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4009ea0 con 0x7f73b015a010 2023-12-19T14:38:45.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73ae7fc700 1 -- 172.21.15.125:0/267420166 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a400b860 con 0x7f73b015a010 2023-12-19T14:38:45.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73ae7fc700 1 -- 172.21.15.125:0/267420166 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400bc00 con 0x7f73b015a010 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 msgr2=0x7f73b00c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b00c4ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73a40099b0 tx=0x7f73a4009250 comp rx=0 tx=0).stop 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- 172.21.15.125:0/267420166 shutdown_connections 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0142690 0x7f73b00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/267420166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b00c4ce0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- 172.21.15.125:0/267420166 >> 172.21.15.125:0/267420166 conn(0x7f73b0067b50 msgr2=0x7f73b00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- 172.21.15.125:0/267420166 shutdown_connections 2023-12-19T14:38:45.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- 172.21.15.125:0/267420166 wait complete. 2023-12-19T14:38:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 Processor -- start 2023-12-19T14:38:45.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.353+0000 7f73b4c0c700 1 -- start start 2023-12-19T14:38:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 0x7f73b013ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 0x7f73b00b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b013e970 con 0x7f73b0142690 2023-12-19T14:38:45.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0159080 con 0x7f73b015a010 2023-12-19T14:38:45.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b013d690 con 0x7f73b0141e70 2023-12-19T14:38:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73af7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 0x7f73b013ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73af7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 0x7f73b013ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:48784/0 (socket says 172.21.15.125:48784) 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73af7fe700 1 -- 172.21.15.125:0/3060300142 learned_addr learned my addr 172.21.15.125:0/3060300142 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40320/0 (socket says 172.21.15.125:40320) 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73aeffd700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 0x7f73b00b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 -- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 msgr2=0x7f73b013ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 0x7f73b013ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 -- 172.21.15.125:0/3060300142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 msgr2=0x7f73b00b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 0x7f73b00b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 -- 172.21.15.125:0/3060300142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a8009a20 con 0x7f73b015a010 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73aeffd700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 0x7f73b00b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73affff700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009e70 tx=0x7f73a800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800cb50 con 0x7f73b015a010 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a800f040 con 0x7f73b015a010 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a40094f0 con 0x7f73b015a010 2023-12-19T14:38:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8014480 con 0x7f73b015a010 2023-12-19T14:38:45.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b013d950 con 0x7f73b015a010 2023-12-19T14:38:45.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f73a8003020 con 0x7f73b015a010 2023-12-19T14:38:45.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f73a80139e0 con 0x7f73b015a010 2023-12-19T14:38:45.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.357+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b004be40 con 0x7f73b015a010 2023-12-19T14:38:45.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.361+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f73a8018030 con 0x7f73b015a010 2023-12-19T14:38:45.400 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:45.394+0000 7f699a9c6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:38:45.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.437+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff2c80ac-fb63-4c00-af3d-11845bb54a35", "id": 6} v 0) v1 -- 0x7f73b0159810 con 0x7f73b015a010 2023-12-19T14:38:45.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73977fe700 1 -- 172.21.15.125:0/3060300142 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff2c80ac-fb63-4c00-af3d-11845bb54a35", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f73a8013c80 con 0x7f73b015a010 2023-12-19T14:38:45.451 INFO:teuthology.orchestra.run.smithi125.stdout:6 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 msgr2=0x7f73b013e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009e70 tx=0x7f73a800b040 comp rx=0 tx=0).stop 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 shutdown_connections 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73b015a010 0x7f73b013e430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73b0142690 0x7f73b00b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 --2- 172.21.15.125:0/3060300142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73b0141e70 0x7f73b013ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 >> 172.21.15.125:0/3060300142 conn(0x7f73b0067b50 msgr2=0x7f73b006d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 shutdown_connections 2023-12-19T14:38:45.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.445+0000 7f73b4c0c700 1 -- 172.21.15.125:0/3060300142 wait complete. 2023-12-19T14:38:45.454 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:45.445+0000 7fc13e347000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:38:45.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd new fc3490c7-db4f-483d-b20d-d7bbda886522 7 2023-12-19T14:38:45.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.621+0000 7f9fb7f85700 1 Processor -- start 2023-12-19T14:38:45.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.621+0000 7f9fb7f85700 1 -- start start 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9fb00cb970 0x7f9fb006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb00cbdf0 con 0x7f9fb0158e20 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb00cc0d0 con 0x7f9fb00cb970 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00cc3b0 con 0x7f9fb00cecd0 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:36118/0 (socket says 172.21.15.125:36118) 2023-12-19T14:38:45.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 -- 172.21.15.125:0/2808480863 learned_addr learned my addr 172.21.15.125:0/2808480863 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb5520700 1 --2- 172.21.15.125:0/2808480863 >> v2:172.21.15.50:3300/0 conn(0x7f9fb00cb970 0x7f9fb006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 -- 172.21.15.125:0/2808480863 >> v2:172.21.15.50:3300/0 conn(0x7f9fb00cb970 msgr2=0x7f9fb006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:45.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 --2- 172.21.15.125:0/2808480863 >> v2:172.21.15.50:3300/0 conn(0x7f9fb00cb970 0x7f9fb006acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 -- 172.21.15.125:0/2808480863 >> v1:172.21.15.125:6789/0 conn(0x7f9fb0158e20 legacy=0x7f9fb00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:38:45.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 -- 172.21.15.125:0/2808480863 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb00cc690 con 0x7f9fb00cecd0 2023-12-19T14:38:45.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb6522700 1 --2- 172.21.15.125:0/2808480863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009f50 tx=0x7f9fa8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85483b79fd52f386 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb4d1f700 1 -- 172.21.15.125:0/2808480863 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa800e070 con 0x7f9fb00cecd0 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb4d1f700 1 -- 172.21.15.125:0/2808480863 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800ba30 con 0x7f9fb00cecd0 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb4d1f700 1 -- 172.21.15.125:0/2808480863 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80092e0 con 0x7f9fb00cecd0 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- 172.21.15.125:0/2808480863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 msgr2=0x7f9fb006b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/2808480863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009f50 tx=0x7f9fa8009840 comp rx=0 tx=0).stop 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- 172.21.15.125:0/2808480863 shutdown_connections 2023-12-19T14:38:45.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/2808480863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb006b200 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/2808480863 >> v2:172.21.15.50:3300/0 conn(0x7f9fb00cb970 0x7f9fb006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.625+0000 7f9fb7f85700 1 -- 172.21.15.125:0/2808480863 >> 172.21.15.125:0/2808480863 conn(0x7f9fb0067bb0 msgr2=0x7f9fb00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.629+0000 7f9fb7f85700 1 -- 172.21.15.125:0/2808480863 shutdown_connections 2023-12-19T14:38:45.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.629+0000 7f9fb7f85700 1 -- 172.21.15.125:0/2808480863 wait complete. 2023-12-19T14:38:45.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.629+0000 7f9fb7f85700 1 Processor -- start 2023-12-19T14:38:45.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.629+0000 7f9fb7f85700 1 -- start start 2023-12-19T14:38:45.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fb0158e20 0x7f9fb00b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 0x7f9fb00b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb015c430 con 0x7f9fb014c6c0 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00b3e20 con 0x7f9fb0158e20 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb015c920 con 0x7f9fb00cecd0 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.125:36134/0 (socket says 172.21.15.125:36134) 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 -- 172.21.15.125:0/1786852514 learned_addr learned my addr 172.21.15.125:0/1786852514 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5520700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fb0158e20 0x7f9fb00b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb6522700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 0x7f9fb00b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 -- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fb0158e20 msgr2=0x7f9fb00b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fb0158e20 0x7f9fb00b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 -- 172.21.15.125:0/1786852514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 msgr2=0x7f9fb00b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 0x7f9fb00b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 -- 172.21.15.125:0/1786852514 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8009ae0 con 0x7f9fb00cecd0 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb6522700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 0x7f9fb00b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb5d21700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00b5e0 tx=0x7f9fac009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac00f510 con 0x7f9fb00cecd0 2023-12-19T14:38:45.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac00f670 con 0x7f9fb00cecd0 2023-12-19T14:38:45.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb00b3450 con 0x7f9fb00cecd0 2023-12-19T14:38:45.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac00fa30 con 0x7f9fb00cecd0 2023-12-19T14:38:45.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb015c640 con 0x7f9fb00cecd0 2023-12-19T14:38:45.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9fac01a400 con 0x7f9fb00cecd0 2023-12-19T14:38:45.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f9fac00fe20 con 0x7f9fb00cecd0 2023-12-19T14:38:45.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.633+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb0071120 con 0x7f9fb00cecd0 2023-12-19T14:38:45.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.641+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9fac01a6b0 con 0x7f9fb00cecd0 2023-12-19T14:38:45.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.717+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fc3490c7-db4f-483d-b20d-d7bbda886522", "id": 7} v 0) v1 -- 0x7f9fb006abc0 con 0x7f9fb00cecd0 2023-12-19T14:38:45.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fa5ffb700 1 -- 172.21.15.125:0/1786852514 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc3490c7-db4f-483d-b20d-d7bbda886522", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9fac01a850 con 0x7f9fb00cecd0 2023-12-19T14:38:45.734 INFO:teuthology.orchestra.run.smithi125.stdout:7 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 msgr2=0x7f9fb014cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00b5e0 tx=0x7f9fac009120 comp rx=0 tx=0).stop 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 shutdown_connections 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fb0158e20 0x7f9fb00b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9fb014c6c0 0x7f9fb00b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 --2- 172.21.15.125:0/1786852514 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9fb00cecd0 0x7f9fb014cb50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 >> 172.21.15.125:0/1786852514 conn(0x7f9fb0067bb0 msgr2=0x7f9fb00cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 shutdown_connections 2023-12-19T14:38:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-12-19T14:38:45.729+0000 7f9fb7f85700 1 -- 172.21.15.125:0/1786852514 wait complete. 2023-12-19T14:38:45.753 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-19T14:38:45.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-19T14:38:45.760 INFO:tasks.ceph.osd.0:Started 2023-12-19T14:38:45.760 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-19T14:38:45.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-19T14:38:45.767 INFO:tasks.ceph.osd.1:Started 2023-12-19T14:38:45.767 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-19T14:38:45.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-19T14:38:45.771 INFO:tasks.ceph.osd.2:Started 2023-12-19T14:38:45.771 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-19T14:38:45.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-19T14:38:45.772 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:45.766+0000 7f699a9c6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:38:45.778 INFO:tasks.ceph.osd.3:Started 2023-12-19T14:38:45.778 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-19T14:38:45.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-19T14:38:45.783 INFO:tasks.ceph.osd.4:Started 2023-12-19T14:38:45.783 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-19T14:38:45.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-19T14:38:45.787 INFO:tasks.ceph.osd.5:Started 2023-12-19T14:38:45.788 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-19T14:38:45.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-19T14:38:45.792 INFO:tasks.ceph.osd.6:Started 2023-12-19T14:38:45.792 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-19T14:38:45.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-19T14:38:45.797 INFO:tasks.ceph.osd.7:Started 2023-12-19T14:38:45.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:38:45.834 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:45.825+0000 7fc13e347000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:38:45.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.982+0000 7feb561c5700 1 Processor -- start 2023-12-19T14:38:45.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.982+0000 7feb561c5700 1 -- start start 2023-12-19T14:38:45.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb561c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb50067c90 0x7feb50068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:45.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb561c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500c6630 con 0x7feb5014fba0 2023-12-19T14:38:45.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb561c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500c6910 con 0x7feb500c5900 2023-12-19T14:38:45.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb561c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500c6bf0 con 0x7feb50067c90 2023-12-19T14:38:45.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4f7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb50067c90 0x7feb50068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:45.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4ffff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feb500c5900 0x7feb500c5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46622/0 (socket says 172.21.15.50:46622) 2023-12-19T14:38:45.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4effd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7feb5014fba0 0x7feb50150000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33434/0 (socket says 172.21.15.50:33434) 2023-12-19T14:38:45.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4ffff700 1 -- 172.21.15.50:0/3735205845 learned_addr learned my addr 172.21.15.50:0/3735205845 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752773507 0 0) 0x7feb500c6910 con 0x7feb500c5900 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38003570 con 0x7feb500c5900 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355711517 0 0) 0x7feb500c6630 con 0x7feb5014fba0 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4f7fe700 1 -- 172.21.15.50:0/3735205845 >> v1:172.21.15.125:6789/0 conn(0x7feb500c5900 legacy=0x7feb500c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4f7fe700 1 -- 172.21.15.50:0/3735205845 >> v1:172.21.15.50:6789/0 conn(0x7feb5014fba0 legacy=0x7feb50150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.986+0000 7feb4f7fe700 1 -- 172.21.15.50:0/3735205845 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500c6ed0 con 0x7feb50067c90 2023-12-19T14:38:45.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb4f7fe700 1 --2- 172.21.15.50:0/3735205845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb50068060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb400098e0 tx=0x7feb400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5ddda9b79017ac7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:45.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000b640 con 0x7feb50067c90 2023-12-19T14:38:45.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4000b7a0 con 0x7feb50067c90 2023-12-19T14:38:45.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb4e7fc700 1 -- 172.21.15.50:0/3735205845 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000bb40 con 0x7feb50067c90 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 -- 172.21.15.50:0/3735205845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 msgr2=0x7feb50068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 --2- 172.21.15.50:0/3735205845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb50068060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feb400098e0 tx=0x7feb400091a0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 -- 172.21.15.50:0/3735205845 shutdown_connections 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 --2- 172.21.15.50:0/3735205845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb50068060 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 -- 172.21.15.50:0/3735205845 >> 172.21.15.50:0/3735205845 conn(0x7feb500cd6d0 msgr2=0x7feb5006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 -- 172.21.15.50:0/3735205845 shutdown_connections 2023-12-19T14:38:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.990+0000 7feb561c5700 1 -- 172.21.15.50:0/3735205845 wait complete. 2023-12-19T14:38:45.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.994+0000 7feb561c5700 1 Processor -- start 2023-12-19T14:38:46.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.994+0000 7feb561c5700 1 -- start start 2023-12-19T14:38:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.994+0000 7feb561c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:46.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb500c5900 0x7feb500c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 0x7feb5014ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c8470 con 0x7feb5014fba0 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c8750 con 0x7feb50067c90 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c8a30 con 0x7feb500c5900 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:36644/0 (socket says 172.21.15.50:36644) 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 -- 172.21.15.50:0/2450069842 learned_addr learned my addr 172.21.15.50:0/2450069842 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4effd700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb500c5900 0x7feb500c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:46.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4ffff700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 0x7feb5014ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:46.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 -- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb500c5900 msgr2=0x7feb500c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:46.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb500c5900 0x7feb500c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:46.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 -- 172.21.15.50:0/2450069842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 msgr2=0x7feb5014ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:46.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 0x7feb5014ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:46.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 -- 172.21.15.50:0/2450069842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009420 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4ffff700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 0x7feb5014ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb4f7fe700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feb5006a0f0 tx=0x7feb40017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4001f070 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40002f70 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000b910 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb50152190 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb500b0240 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7feb40023070 con 0x7feb50067c90 2023-12-19T14:38:46.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:45.998+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7feb4000ba70 con 0x7feb50067c90 2023-12-19T14:38:46.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.002+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb50071120 con 0x7feb50067c90 2023-12-19T14:38:46.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.006+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7feb4002a070 con 0x7feb50067c90 2023-12-19T14:38:46.051 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.046+0000 7f699a9c6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:38:46.070 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.065+0000 7fc13e347000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:38:46.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.086+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb500642a0 con 0x7feb50067c90 2023-12-19T14:38:46.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb377fe700 1 -- 172.21.15.50:0/2450069842 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7feb40015020 con 0x7feb50067c90 2023-12-19T14:38:46.096 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:38:46.096 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:45.726769+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 msgr2=0x7feb500afd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7feb5006a0f0 tx=0x7feb40017000 comp rx=0 tx=0).stop 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 shutdown_connections 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb5014fba0 0x7feb5014ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb500c5900 0x7feb500c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 --2- 172.21.15.50:0/2450069842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb50067c90 0x7feb500afd00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 >> 172.21.15.50:0/2450069842 conn(0x7feb500cd6d0 msgr2=0x7feb5006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:46.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 shutdown_connections 2023-12-19T14:38:46.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:38:46.090+0000 7feb561c5700 1 -- 172.21.15.50:0/2450069842 wait complete. 2023-12-19T14:38:46.120 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-19T14:38:46.121 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-19T14:38:46.149 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.142+0000 7f699a9c6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:38:46.156 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.149+0000 7fc13e347000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:38:46.230 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.226+0000 7f699a9c6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:38:46.235 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.229+0000 7fc13e347000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:38:46.314 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.305+0000 7fc13e347000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:38:46.323 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.318+0000 7f699a9c6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:38:46.437 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-12-19T14:38:46.430+0000 7fd9f9dc03c0 -1 Falling back to public interface 2023-12-19T14:38:46.448 INFO:tasks.ceph.osd.6.smithi125.stderr:2023-12-19T14:38:46.441+0000 7facac2fa3c0 -1 Falling back to public interface 2023-12-19T14:38:46.456 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-12-19T14:38:46.450+0000 7f3bdd6e33c0 -1 Falling back to public interface 2023-12-19T14:38:46.469 INFO:tasks.ceph.osd.5.smithi125.stderr:2023-12-19T14:38:46.461+0000 7feb40bba3c0 -1 Falling back to public interface 2023-12-19T14:38:46.469 INFO:tasks.ceph.osd.7.smithi125.stderr:2023-12-19T14:38:46.461+0000 7ff5b7f433c0 -1 Falling back to public interface 2023-12-19T14:38:46.470 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-12-19T14:38:46.466+0000 7f31ab6dc3c0 -1 Falling back to public interface 2023-12-19T14:38:46.471 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-12-19T14:38:46.466+0000 7effca1ba3c0 -1 Falling back to public interface 2023-12-19T14:38:46.485 INFO:tasks.ceph.osd.4.smithi125.stderr:2023-12-19T14:38:46.477+0000 7f3e7f0e63c0 -1 Falling back to public interface 2023-12-19T14:38:46.655 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.649+0000 7fc13e347000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:38:46.675 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.670+0000 7f699a9c6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:38:46.735 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.729+0000 7fc13e347000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:38:46.766 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.762+0000 7f699a9c6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:38:46.827 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:46.821+0000 7fc13e347000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:38:46.866 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:46.858+0000 7f699a9c6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:38:46.926 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T14:38:47.055 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:47.049+0000 7fc13e347000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:38:47.059 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 Processor -- start 2023-12-19T14:38:47.059 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 -- start start 2023-12-19T14:38:47.060 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f935006ae80 0x7f9350069690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:47.061 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93500b1820 con 0x7f9350069bd0 2023-12-19T14:38:47.061 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935006a030 con 0x7f9350156bc0 2023-12-19T14:38:47.061 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9354984700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500afc60 con 0x7f935006ae80 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9347fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f935006ae80 0x7f9350069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9347fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f935006ae80 0x7f9350069690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55004/0 (socket says 172.21.15.50:55004) 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.054+0000 7f9347fff700 1 -- 172.21.15.50:0/3487464325 learned_addr learned my addr 172.21.15.50:0/3487464325 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9347fff700 1 -- 172.21.15.50:0/3487464325 >> v1:172.21.15.125:6789/0 conn(0x7f9350156bc0 legacy=0x7f93500b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9347fff700 1 -- 172.21.15.50:0/3487464325 >> v1:172.21.15.50:6789/0 conn(0x7f9350069bd0 legacy=0x7f935006c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9347fff700 1 -- 172.21.15.50:0/3487464325 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935006efa0 con 0x7f935006ae80 2023-12-19T14:38:47.062 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f934e59c700 1 -- 172.21.15.50:0/3487464325 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179830584 0 0) 0x7f9348001490 con 0x7f9350069bd0 2023-12-19T14:38:47.063 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9347fff700 1 --2- 172.21.15.50:0/3487464325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f935006ae80 0x7f9350069690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93380099b0 tx=0x7f9338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b341cd95fd3bf46c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:47.063 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f934e59c700 1 -- 172.21.15.50:0/3487464325 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800b860 con 0x7f935006ae80 2023-12-19T14:38:47.063 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f934e59c700 1 -- 172.21.15.50:0/3487464325 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800b9c0 con 0x7f935006ae80 2023-12-19T14:38:47.063 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f934e59c700 1 -- 172.21.15.50:0/3487464325 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800bd60 con 0x7f935006ae80 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 -- 172.21.15.50:0/3487464325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f935006ae80 msgr2=0x7f9350069690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 --2- 172.21.15.50:0/3487464325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f935006ae80 0x7f9350069690 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93380099b0 tx=0x7f9338009250 comp rx=0 tx=0).stop 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 -- 172.21.15.50:0/3487464325 shutdown_connections 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 --2- 172.21.15.50:0/3487464325 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f935006ae80 0x7f9350069690 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 -- 172.21.15.50:0/3487464325 >> 172.21.15.50:0/3487464325 conn(0x7f93500c8420 msgr2=0x7f93500c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 -- 172.21.15.50:0/3487464325 shutdown_connections 2023-12-19T14:38:47.064 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.058+0000 7f9354984700 1 -- 172.21.15.50:0/3487464325 wait complete. 2023-12-19T14:38:47.066 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 Processor -- start 2023-12-19T14:38:47.066 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 -- start start 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935006ae80 0x7f93500ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 0x7f93500c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935015ba70 con 0x7f9350069bd0 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9350169a90 con 0x7f935006ae80 2023-12-19T14:38:47.068 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9354984700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935015a6f0 con 0x7f9350156bc0 2023-12-19T14:38:47.069 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:47.069 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55010/0 (socket says 172.21.15.50:55010) 2023-12-19T14:38:47.069 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 -- 172.21.15.50:0/1733324362 learned_addr learned my addr 172.21.15.50:0/1733324362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:47.069 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9347fff700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935006ae80 0x7f93500ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:47.069 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934f59e700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 0x7f93500c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:47.070 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 -- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 msgr2=0x7f93500c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.070 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 0x7f93500c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 -- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935006ae80 msgr2=0x7f93500ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935006ae80 0x7f93500ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 -- 172.21.15.50:0/1733324362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380094f0 con 0x7f9350069bd0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934f59e700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 0x7f93500c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f934ed9d700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f934000c4e0 tx=0x7f934000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.062+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340011590 con 0x7f9350069bd0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340011710 con 0x7f9350069bd0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340011ad0 con 0x7f9350069bd0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935015a9d0 con 0x7f9350069bd0 2023-12-19T14:38:47.071 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935015af90 con 0x7f9350069bd0 2023-12-19T14:38:47.072 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f934001c6c0 con 0x7f9350069bd0 2023-12-19T14:38:47.072 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f9340012b00 con 0x7f9350069bd0 2023-12-19T14:38:47.072 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.066+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935004be40 con 0x7f9350069bd0 2023-12-19T14:38:47.075 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.070+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9340017070 con 0x7f9350069bd0 2023-12-19T14:38:47.099 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:47.094+0000 7f699a9c6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:38:47.144 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.138+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f935006f520 con 0x7f9350069bd0 2023-12-19T14:38:47.145 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9346ffd700 1 -- 172.21.15.50:0/1733324362 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f935006a030 con 0x7f9350069bd0 2023-12-19T14:38:47.146 INFO:teuthology.misc.health.smithi050.stdout: 2023-12-19T14:38:47.146 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:45.726769+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:38:47.147 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 msgr2=0x7f9350169550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:47.147 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f934000c4e0 tx=0x7f934000c7f0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 shutdown_connections 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f935006ae80 0x7f93500ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9350156bc0 0x7f93500c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 --2- 172.21.15.50:0/1733324362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350069bd0 0x7f9350169550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 >> 172.21.15.50:0/1733324362 conn(0x7f93500c8420 msgr2=0x7f93500b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 shutdown_connections 2023-12-19T14:38:47.148 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:47.142+0000 7f9354984700 1 -- 172.21.15.50:0/1733324362 wait complete. 2023-12-19T14:38:47.148 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:47.141+0000 7fc13e347000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:38:47.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-19T14:38:47.198 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:47.194+0000 7f699a9c6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:38:48.101 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:48.093+0000 7fc13e347000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:38:48.156 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:38:48.150+0000 7f699a9c6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:38:49.137 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:49.129+0000 7fc12719f700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-19T14:38:49.140 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:38:49.133+0000 7fc12719f700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-19T14:38:50.376 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-12-19T14:38:50.369+0000 7f3bdd6e33c0 -1 osd.2 0 log_to_monitors true 2023-12-19T14:38:50.437 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-12-19T14:38:50.429+0000 7f31ab6dc3c0 -1 osd.1 0 log_to_monitors true 2023-12-19T14:38:50.573 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-12-19T14:38:50.565+0000 7fd9f9dc03c0 -1 osd.0 0 log_to_monitors true 2023-12-19T14:38:50.596 INFO:tasks.ceph.osd.6.smithi125.stderr:2023-12-19T14:38:50.589+0000 7facac2fa3c0 -1 osd.6 0 log_to_monitors true 2023-12-19T14:38:50.672 INFO:tasks.ceph.osd.5.smithi125.stderr:2023-12-19T14:38:50.665+0000 7feb40bba3c0 -1 osd.5 0 log_to_monitors true 2023-12-19T14:38:50.703 INFO:tasks.ceph.osd.4.smithi125.stderr:2023-12-19T14:38:50.697+0000 7f3e7f0e63c0 -1 osd.4 0 log_to_monitors true 2023-12-19T14:38:50.711 INFO:tasks.ceph.osd.7.smithi125.stderr:2023-12-19T14:38:50.705+0000 7ff5b7f433c0 -1 osd.7 0 log_to_monitors true 2023-12-19T14:38:50.723 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-12-19T14:38:50.717+0000 7effca1ba3c0 -1 osd.3 0 log_to_monitors true 2023-12-19T14:38:53.971 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T14:38:54.024 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-12-19T14:38:54.013+0000 7effc595a700 -1 osd.3 0 waiting for initial osdmap 2023-12-19T14:38:54.035 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-12-19T14:38:54.029+0000 7effbe6fe700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.137 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.129+0000 7f62f22ae700 1 Processor -- start 2023-12-19T14:38:54.138 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62f22ae700 1 -- start start 2023-12-19T14:38:54.140 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62f22ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 0x7f62ec158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:54.140 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62f22ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec15d970 con 0x7f62ec158ba0 2023-12-19T14:38:54.140 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62f22ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec06cb80 con 0x7f62ec06d010 2023-12-19T14:38:54.140 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62f22ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec06bf70 con 0x7f62ec15d1f0 2023-12-19T14:38:54.141 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62ebfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f62ec158ba0 0x7f62ec15cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36296/0 (socket says 172.21.15.50:36296) 2023-12-19T14:38:54.141 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62ebfff700 1 -- 172.21.15.50:0/4267813179 learned_addr learned my addr 172.21.15.50:0/4267813179 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:38:54.141 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62e3fff700 1 --2- 172.21.15.50:0/4267813179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 0x7f62ec158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.133+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388949888 0 0) 0x7f62ec06cb80 con 0x7f62ec06d010 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d8003570 con 0x7f62ec06d010 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596865580 0 0) 0x7f62ec15d970 con 0x7f62ec158ba0 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec06cb80 con 0x7f62ec158ba0 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62e3fff700 1 -- 172.21.15.50:0/4267813179 >> v1:172.21.15.125:6789/0 conn(0x7f62ec06d010 legacy=0x7f62ec0b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62e3fff700 1 -- 172.21.15.50:0/4267813179 >> v1:172.21.15.50:6789/0 conn(0x7f62ec158ba0 legacy=0x7f62ec15cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.142 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62e3fff700 1 -- 172.21.15.50:0/4267813179 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec06c250 con 0x7f62ec15d1f0 2023-12-19T14:38:54.143 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-12-19T14:38:54.137+0000 7fd9f5560700 -1 osd.0 0 waiting for initial osdmap 2023-12-19T14:38:54.143 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.137+0000 7f62e3fff700 1 --2- 172.21.15.50:0/4267813179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 0x7f62ec158f70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009940 tx=0x7f62d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4432f5c9ca82754c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:54.146 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4011630 con 0x7f62ec15d1f0 2023-12-19T14:38:54.146 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d4011790 con 0x7f62ec15d1f0 2023-12-19T14:38:54.146 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62eb7fe700 1 -- 172.21.15.50:0/4267813179 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4011b50 con 0x7f62ec15d1f0 2023-12-19T14:38:54.147 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 -- 172.21.15.50:0/4267813179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 msgr2=0x7f62ec158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.147 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 --2- 172.21.15.50:0/4267813179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 0x7f62ec158f70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009940 tx=0x7f62d40091a0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.149 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 -- 172.21.15.50:0/4267813179 shutdown_connections 2023-12-19T14:38:54.149 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 --2- 172.21.15.50:0/4267813179 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec15d1f0 0x7f62ec158f70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.149 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 -- 172.21.15.50:0/4267813179 >> 172.21.15.50:0/4267813179 conn(0x7f62ec06dcb0 msgr2=0x7f62ec06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:54.149 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 -- 172.21.15.50:0/4267813179 shutdown_connections 2023-12-19T14:38:54.149 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.141+0000 7f62f22ae700 1 -- 172.21.15.50:0/4267813179 wait complete. 2023-12-19T14:38:54.152 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.145+0000 7f62f22ae700 1 Processor -- start 2023-12-19T14:38:54.153 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.145+0000 7f62f22ae700 1 -- start start 2023-12-19T14:38:54.154 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 0x7f62ec152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:54.155 INFO:tasks.ceph.osd.0.smithi050.stderr:2023-12-19T14:38:54.149+0000 7fd9ee304700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.155 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 0x7f62ec0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:54.155 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62ec15d1f0 0x7f62ec156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:54.155 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0c70a0 con 0x7f62ec158ba0 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec1534d0 con 0x7f62ec15d1f0 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0c61b0 con 0x7f62ec06d010 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 0x7f62ec0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 0x7f62ec152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 0x7f62ec0b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55074/0 (socket says 172.21.15.50:55074) 2023-12-19T14:38:54.156 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e3fff700 1 -- 172.21.15.50:0/3368920697 learned_addr learned my addr 172.21.15.50:0/3368920697 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 -- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62ec15d1f0 msgr2=0x7f62ec156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62ec15d1f0 0x7f62ec156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 -- 172.21.15.50:0/3368920697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 msgr2=0x7f62ec0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 0x7f62ec0b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 -- 172.21.15.50:0/3368920697 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009420 con 0x7f62ec06d010 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62ebfff700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 0x7f62ec152f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00c5b0 tx=0x7f62dc007db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc005c00 con 0x7f62ec06d010 2023-12-19T14:38:54.157 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc005d60 con 0x7f62ec06d010 2023-12-19T14:38:54.158 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec0c6490 con 0x7f62ec06d010 2023-12-19T14:38:54.158 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec0c6a50 con 0x7f62ec06d010 2023-12-19T14:38:54.158 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.149+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0072e0 con 0x7f62ec06d010 2023-12-19T14:38:54.159 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.153+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec071120 con 0x7f62ec06d010 2023-12-19T14:38:54.161 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.153+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 4) v1 ==== 79905+0+0 (secure 0 0 0) 0x7f62dc007690 con 0x7f62ec06d010 2023-12-19T14:38:54.162 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.153+0000 7f62e8ff9700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 0x7f62cc062a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:38:54.162 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.153+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f62dc07ac70 con 0x7f62ec06d010 2023-12-19T14:38:54.164 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.157+0000 7f62e3fff700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 0x7f62cc062a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:38:54.165 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.157+0000 7f62e3fff700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 0x7f62cc062a40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0af0b0 tx=0x7f62d4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:38:54.171 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.165+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f62dc049be0 con 0x7f62ec06d010 2023-12-19T14:38:54.227 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-12-19T14:38:54.221+0000 7f31a6659700 -1 osd.1 0 waiting for initial osdmap 2023-12-19T14:38:54.239 INFO:tasks.ceph.osd.1.smithi050.stderr:2023-12-19T14:38:54.233+0000 7f319f3fd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.268 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-12-19T14:38:54.261+0000 7f3bd9dc9700 -1 osd.2 0 waiting for initial osdmap 2023-12-19T14:38:54.279 INFO:tasks.ceph.osd.2.smithi050.stderr:2023-12-19T14:38:54.273+0000 7f3bd1348700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.380 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.373+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62ec06c530 con 0x7f62ec06d010 2023-12-19T14:38:54.382 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62e8ff9700 1 -- 172.21.15.50:0/3368920697 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f62dc017090 con 0x7f62ec06d010 2023-12-19T14:38:54.382 INFO:teuthology.misc.health.smithi050.stdout: 2023-12-19T14:38:54.382 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":12,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:52.162266+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 msgr2=0x7f62cc062a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 0x7f62cc062a40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0af0b0 tx=0x7f62d4016000 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 msgr2=0x7f62ec152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 0x7f62ec152f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00c5b0 tx=0x7f62dc007db0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 shutdown_connections 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62ec15d1f0 0x7f62ec156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62cc060590 0x7f62cc062a40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62ec158ba0 0x7f62ec0b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 --2- 172.21.15.50:0/3368920697 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62ec06d010 0x7f62ec152f90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 >> 172.21.15.50:0/3368920697 conn(0x7f62ec06dcb0 msgr2=0x7f62ec0b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 shutdown_connections 2023-12-19T14:38:54.385 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:38:54.377+0000 7f62f22ae700 1 -- 172.21.15.50:0/3368920697 wait complete. 2023-12-19T14:38:54.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-19T14:38:54.696 INFO:tasks.ceph.osd.4.smithi125.stderr:2023-12-19T14:38:54.689+0000 7f3e79fa4700 -1 osd.4 0 waiting for initial osdmap 2023-12-19T14:38:54.711 INFO:tasks.ceph.osd.4.smithi125.stderr:2023-12-19T14:38:54.705+0000 7f3e72d48700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.741 INFO:tasks.ceph.osd.6.smithi125.stderr:2023-12-19T14:38:54.733+0000 7faca8ab8700 -1 osd.6 0 waiting for initial osdmap 2023-12-19T14:38:54.749 INFO:tasks.ceph.osd.7.smithi125.stderr:2023-12-19T14:38:54.741+0000 7ff5b36e3700 -1 osd.7 0 waiting for initial osdmap 2023-12-19T14:38:54.752 INFO:tasks.ceph.osd.6.smithi125.stderr:2023-12-19T14:38:54.745+0000 7faca0048700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.757 INFO:tasks.ceph.osd.7.smithi125.stderr:2023-12-19T14:38:54.749+0000 7ff5ac487700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:38:54.808 INFO:tasks.ceph.osd.5.smithi125.stderr:2023-12-19T14:38:54.801+0000 7feb3c35a700 -1 osd.5 0 waiting for initial osdmap 2023-12-19T14:38:54.820 INFO:tasks.ceph.osd.5.smithi125.stderr:2023-12-19T14:38:54.813+0000 7feb350fe700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T14:39:01.212 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:01.362 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 Processor -- start 2023-12-19T14:39:01.363 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 -- start start 2023-12-19T14:39:01.365 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad60160e90 0x7fad6015e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.365 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600697c0 con 0x7fad600c7030 2023-12-19T14:39:01.365 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6015e890 con 0x7fad6015ada0 2023-12-19T14:39:01.365 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.357+0000 7fad666e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad60161da0 con 0x7fad60160e90 2023-12-19T14:39:01.366 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad60160e90 0x7fad6015e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.366 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad60160e90 0x7fad6015e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37114/0 (socket says 172.21.15.50:37114) 2023-12-19T14:39:01.366 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 -- 172.21.15.50:0/1708527174 learned_addr learned my addr 172.21.15.50:0/1708527174 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:01.367 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad64ee0700 1 --1- 172.21.15.50:0/1708527174 >> v1:172.21.15.50:6789/0 conn(0x7fad600c7030 0x7fad60069c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49780/0 (socket says 172.21.15.50:49780) 2023-12-19T14:39:01.368 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 -- 172.21.15.50:0/1708527174 >> v1:172.21.15.125:6789/0 conn(0x7fad6015ada0 legacy=0x7fad600c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.368 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 -- 172.21.15.50:0/1708527174 >> v1:172.21.15.50:6789/0 conn(0x7fad600c7030 legacy=0x7fad60069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.368 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 -- 172.21.15.50:0/1708527174 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad60161500 con 0x7fad60160e90 2023-12-19T14:39:01.368 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad65ee2700 1 --2- 172.21.15.50:0/1708527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad6015e350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fad580099b0 tx=0x7fad58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b40897ae73c426 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.369 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad57fff700 1 -- 172.21.15.50:0/1708527174 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800b860 con 0x7fad60160e90 2023-12-19T14:39:01.369 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad57fff700 1 -- 172.21.15.50:0/1708527174 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5800b9c0 con 0x7fad60160e90 2023-12-19T14:39:01.369 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad57fff700 1 -- 172.21.15.50:0/1708527174 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800bd60 con 0x7fad60160e90 2023-12-19T14:39:01.371 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad666e3700 1 -- 172.21.15.50:0/1708527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 msgr2=0x7fad6015e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.371 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad666e3700 1 --2- 172.21.15.50:0/1708527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad6015e350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fad580099b0 tx=0x7fad58009250 comp rx=0 tx=0).stop 2023-12-19T14:39:01.371 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad666e3700 1 -- 172.21.15.50:0/1708527174 shutdown_connections 2023-12-19T14:39:01.371 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad666e3700 1 --2- 172.21.15.50:0/1708527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad6015e350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.371 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.361+0000 7fad666e3700 1 -- 172.21.15.50:0/1708527174 >> 172.21.15.50:0/1708527174 conn(0x7fad600c8420 msgr2=0x7fad600c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:01.372 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.365+0000 7fad666e3700 1 -- 172.21.15.50:0/1708527174 shutdown_connections 2023-12-19T14:39:01.372 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.365+0000 7fad666e3700 1 -- 172.21.15.50:0/1708527174 wait complete. 2023-12-19T14:39:01.373 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.365+0000 7fad666e3700 1 Processor -- start 2023-12-19T14:39:01.373 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.365+0000 7fad666e3700 1 -- start start 2023-12-19T14:39:01.377 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 0x7fad60158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.377 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6015ada0 0x7fad6016e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.377 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad601582a0 con 0x7fad60160e90 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6016e740 con 0x7fad6015ada0 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad600b15d0 con 0x7fad600c7030 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37118/0 (socket says 172.21.15.50:37118) 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 -- 172.21.15.50:0/4278569239 learned_addr learned my addr 172.21.15.50:0/4278569239 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad64ee0700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6015ada0 0x7fad6016e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad656e1700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 0x7fad60158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 -- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 msgr2=0x7fad60158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 0x7fad60158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 -- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6015ada0 msgr2=0x7fad6016e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.378 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6015ada0 0x7fad6016e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 -- 172.21.15.50:0/4278569239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580094f0 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad656e1700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 0x7fad60158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad65ee2700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fad580094c0 tx=0x7fad58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5801a630 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5801a790 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad601584f0 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5801ab50 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.369+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad60158ab0 con 0x7fad60160e90 2023-12-19T14:39:01.379 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.373+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad60158cd0 con 0x7fad60160e90 2023-12-19T14:39:01.380 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.373+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fad580233e0 con 0x7fad60160e90 2023-12-19T14:39:01.380 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.373+0000 7fad557fa700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 0x7fad44062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.381 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.373+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fad5807e0f0 con 0x7fad60160e90 2023-12-19T14:39:01.381 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.373+0000 7fad656e1700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 0x7fad44062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.382 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.377+0000 7fad656e1700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 0x7fad44062a00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007fd0 tx=0x7fad5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.392 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.385+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad5804ea30 con 0x7fad60160e90 2023-12-19T14:39:01.566 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.561+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad6004be40 con 0x7fad60160e90 2023-12-19T14:39:01.568 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.561+0000 7fad557fa700 1 -- 172.21.15.50:0/4278569239 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12127 (secure 0 0 0) 0x7fad58052050 con 0x7fad60160e90 2023-12-19T14:39:01.569 INFO:teuthology.misc.health.smithi050.stdout: 2023-12-19T14:39:01.569 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":16,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:58.489546+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:01.571 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 msgr2=0x7fad44062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.571 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 0x7fad44062a00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007fd0 tx=0x7fad5c009040 comp rx=0 tx=0).stop 2023-12-19T14:39:01.571 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 msgr2=0x7fad600b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.571 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fad580094c0 tx=0x7fad58017000 comp rx=0 tx=0).stop 2023-12-19T14:39:01.571 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 shutdown_connections 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad6015ada0 0x7fad6016e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad44060550 0x7fad44062a00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad60160e90 0x7fad600b02b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 --2- 172.21.15.50:0/4278569239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad600c7030 0x7fad60158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 >> 172.21.15.50:0/4278569239 conn(0x7fad600c8420 msgr2=0x7fad600c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 shutdown_connections 2023-12-19T14:39:01.572 INFO:teuthology.misc.health.smithi050.stderr:2023-12-19T14:39:01.565+0000 7fad666e3700 1 -- 172.21.15.50:0/4278569239 wait complete. 2023-12-19T14:39:01.589 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-19T14:39:01.589 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-19T14:39:01.589 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:39:01.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:39:01.596 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:39:01.597 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:39:01.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:39:01.604 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:39:01.604 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:39:01.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:39:01.611 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:39:01.611 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:39:01.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:39:01.618 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:39:01.619 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-19T14:39:01.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:01.705 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:39:01.706 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:39:01.725 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:39:01.733 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:39:01.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.769+0000 7f593359e700 1 Processor -- start 2023-12-19T14:39:01.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.769+0000 7f593359e700 1 -- start start 2023-12-19T14:39:01.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f593406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f59340ce810 0x7f59340ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593406b730 con 0x7f59341591f0 2023-12-19T14:39:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593406ab20 con 0x7f59340ce810 2023-12-19T14:39:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593406ae00 con 0x7f5934158e20 2023-12-19T14:39:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f59340ce810 0x7f59340ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f59340ce810 0x7f59340ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37170/0 (socket says 172.21.15.50:37170) 2023-12-19T14:39:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 -- 172.21.15.50:0/645527539 learned_addr learned my addr 172.21.15.50:0/645527539 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:01.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593259c700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f593406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 -- 172.21.15.50:0/645527539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 msgr2=0x7f593406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:01.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f593406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 -- 172.21.15.50:0/645527539 >> v1:172.21.15.125:6789/0 conn(0x7f59341591f0 legacy=0x7f59340ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 -- 172.21.15.50:0/645527539 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593406b0e0 con 0x7f59340ce810 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593259c700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f593406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f5932d9d700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340ce810 0x7f59340ca950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f592c009f20 tx=0x7f592c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d2706b0c5e236 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593159a700 1 -- 172.21.15.50:0/645527539 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c00e070 con 0x7f59340ce810 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593159a700 1 -- 172.21.15.50:0/645527539 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c00ba80 con 0x7f59340ce810 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593159a700 1 -- 172.21.15.50:0/645527539 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0092e0 con 0x7f59340ce810 2023-12-19T14:39:01.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 -- 172.21.15.50:0/645527539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340ce810 msgr2=0x7f59340ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.773+0000 7f593359e700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340ce810 0x7f59340ca950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f592c009f20 tx=0x7f592c009840 comp rx=0 tx=0).stop 2023-12-19T14:39:01.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- 172.21.15.50:0/645527539 shutdown_connections 2023-12-19T14:39:01.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340ce810 0x7f59340ca950 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 --2- 172.21.15.50:0/645527539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f593406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- 172.21.15.50:0/645527539 >> 172.21.15.50:0/645527539 conn(0x7f5934067bb0 msgr2=0x7f59340cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- 172.21.15.50:0/645527539 shutdown_connections 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- 172.21.15.50:0/645527539 wait complete. 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 Processor -- start 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- start start 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59340ce810 0x7f593413ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f59340b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5934141ff0 con 0x7f59341591f0 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59341413e0 con 0x7f59340ce810 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f593359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59341416c0 con 0x7f5934158e20 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.777+0000 7f5932d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5931d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f59340b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37184/0 (socket says 172.21.15.50:37184) 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 -- 172.21.15.50:0/3919256259 learned_addr learned my addr 172.21.15.50:0/3919256259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f593259c700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59340ce810 0x7f593413ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 -- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 msgr2=0x7f59340b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f59340b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 -- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59340ce810 msgr2=0x7f593413ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59340ce810 0x7f593413ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 -- 172.21.15.50:0/3919256259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c009ae0 con 0x7f59341591f0 2023-12-19T14:39:01.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f5932d9d700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f593406e810 tx=0x7f592c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c008d20 con 0x7f59341591f0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c008e80 con 0x7f59341591f0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59341419a0 con 0x7f59341591f0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c0266b0 con 0x7f59341591f0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.781+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5934140720 con 0x7f59341591f0 2023-12-19T14:39:01.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.785+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5934142160 con 0x7f59341591f0 2023-12-19T14:39:01.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.785+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f592c003040 con 0x7f59341591f0 2023-12-19T14:39:01.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.785+0000 7f59227fc700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 0x7f5914062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:01.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.785+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f592c07e900 con 0x7f59341591f0 2023-12-19T14:39:01.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.793+0000 7f593259c700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 0x7f5914062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:01.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.793+0000 7f593259c700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 0x7f5914062a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59341596a0 tx=0x7f5924006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:01.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.793+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f592c04f220 con 0x7f59341591f0 2023-12-19T14:39:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.977+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f593404be40 con 0x7f59341591f0 2023-12-19T14:39:01.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.977+0000 7f59227fc700 1 -- 172.21.15.50:0/3919256259 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12127 (secure 0 0 0) 0x7f592c052840 con 0x7f59341591f0 2023-12-19T14:39:01.985 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:01.985 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":16,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:58.489546+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:01.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 msgr2=0x7f5914062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 0x7f5914062a80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f59341596a0 tx=0x7f5924006f90 comp rx=0 tx=0).stop 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 msgr2=0x7f59340b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f593406e810 tx=0x7f592c018000 comp rx=0 tx=0).stop 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 shutdown_connections 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59140605d0 0x7f5914062a80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59341591f0 0x7f59340b53f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934158e20 0x7f59340b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 --2- 172.21.15.50:0/3919256259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59340ce810 0x7f593413ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 >> 172.21.15.50:0/3919256259 conn(0x7f5934067bb0 msgr2=0x7f5934152510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 shutdown_connections 2023-12-19T14:39:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:01.981+0000 7f593359e700 1 -- 172.21.15.50:0/3919256259 wait complete. 2023-12-19T14:39:02.009 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:39:02.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:02.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.177+0000 7ffb0dd45700 1 Processor -- start 2023-12-19T14:39:02.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.177+0000 7ffb0dd45700 1 -- start start 2023-12-19T14:39:02.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb0dd45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffb0806db40 0x7ffb080cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb0dd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffb0806df10 0x7ffb08159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb0dd45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb080c9db0 con 0x7ffb080c63f0 2023-12-19T14:39:02.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb0dd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080ca950 con 0x7ffb0806df10 2023-12-19T14:39:02.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb0dd45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080cb2e0 con 0x7ffb0806db40 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffb0806df10 0x7ffb08159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffb0806df10 0x7ffb08159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37194/0 (socket says 172.21.15.50:37194) 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 -- 172.21.15.50:0/1890025649 learned_addr learned my addr 172.21.15.50:0/1890025649 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb077fe700 1 --2- 172.21.15.50:0/1890025649 >> v2:172.21.15.125:3300/0 conn(0x7ffb0806db40 0x7ffb080cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 -- 172.21.15.50:0/1890025649 >> v2:172.21.15.125:3300/0 conn(0x7ffb0806db40 msgr2=0x7ffb080cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:02.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 --2- 172.21.15.50:0/1890025649 >> v2:172.21.15.125:3300/0 conn(0x7ffb0806db40 0x7ffb080cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 -- 172.21.15.50:0/1890025649 >> v1:172.21.15.125:6789/0 conn(0x7ffb080c63f0 legacy=0x7ffb080ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.181+0000 7ffb07fff700 1 -- 172.21.15.50:0/1890025649 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080cb5c0 con 0x7ffb0806df10 2023-12-19T14:39:02.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb07fff700 1 --2- 172.21.15.50:0/1890025649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806df10 0x7ffb08159900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb08067fe0 tx=0x7ffb00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb301e92bb90dcb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb067fc700 1 -- 172.21.15.50:0/1890025649 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0000e070 con 0x7ffb0806df10 2023-12-19T14:39:02.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb067fc700 1 -- 172.21.15.50:0/1890025649 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0000ba20 con 0x7ffb0806df10 2023-12-19T14:39:02.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb067fc700 1 -- 172.21.15.50:0/1890025649 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb000092e0 con 0x7ffb0806df10 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 -- 172.21.15.50:0/1890025649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806df10 msgr2=0x7ffb08159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/1890025649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806df10 0x7ffb08159900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb08067fe0 tx=0x7ffb00009840 comp rx=0 tx=0).stop 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 -- 172.21.15.50:0/1890025649 shutdown_connections 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/1890025649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806df10 0x7ffb08159900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/1890025649 >> v2:172.21.15.125:3300/0 conn(0x7ffb0806db40 0x7ffb080cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 -- 172.21.15.50:0/1890025649 >> 172.21.15.50:0/1890025649 conn(0x7ffb08067bb0 msgr2=0x7ffb080cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:02.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 -- 172.21.15.50:0/1890025649 shutdown_connections 2023-12-19T14:39:02.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.185+0000 7ffb0dd45700 1 -- 172.21.15.50:0/1890025649 wait complete. 2023-12-19T14:39:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 Processor -- start 2023-12-19T14:39:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 -- start start 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb0806df10 0x7ffb080aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 0x7ffb080b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0815c130 con 0x7ffb0806db40 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080b1a10 con 0x7ffb0806df10 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb0dd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0815bb70 con 0x7ffb080c63f0 2023-12-19T14:39:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37206/0 (socket says 172.21.15.50:37206) 2023-12-19T14:39:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb077fe700 1 -- 172.21.15.50:0/2486492379 learned_addr learned my addr 172.21.15.50:0/2486492379 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb06ffd700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb0806df10 0x7ffb080aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.189+0000 7ffb07fff700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 0x7ffb080b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 -- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 msgr2=0x7ffb080b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 0x7ffb080b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 -- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb0806df10 msgr2=0x7ffb080aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb0806df10 0x7ffb080aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 -- 172.21.15.50:0/2486492379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb00009ae0 con 0x7ffb0806db40 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb07fff700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 0x7ffb080b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb077fe700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e930 tx=0x7ffaf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8013680 con 0x7ffb0806db40 2023-12-19T14:39:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf80137e0 con 0x7ffb0806db40 2023-12-19T14:39:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0815be50 con 0x7ffb0806db40 2023-12-19T14:39:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8013b80 con 0x7ffb0806db40 2023-12-19T14:39:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb081524f0 con 0x7ffb0806db40 2023-12-19T14:39:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.193+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb08071120 con 0x7ffb0806db40 2023-12-19T14:39:02.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.197+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ffaf800c3e0 con 0x7ffb0806db40 2023-12-19T14:39:02.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.197+0000 7ffaef7fe700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 0x7ffaf4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.197+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ffaf807c4a0 con 0x7ffb0806db40 2023-12-19T14:39:02.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.197+0000 7ffb06ffd700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 0x7ffaf4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.197+0000 7ffb06ffd700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 0x7ffaf4062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffb0815baa0 tx=0x7ffafc007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.201+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffaf804cde0 con 0x7ffb0806db40 2023-12-19T14:39:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.373+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb080cb8a0 con 0x7ffb0806db40 2023-12-19T14:39:02.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.373+0000 7ffaef7fe700 1 -- 172.21.15.50:0/2486492379 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffaf8050400 con 0x7ffb0806db40 2023-12-19T14:39:02.381 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 msgr2=0x7ffaf4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 0x7ffaf4062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffb0815baa0 tx=0x7ffafc007ec0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 msgr2=0x7ffb0815b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e930 tx=0x7ffaf80091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 shutdown_connections 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb0806df10 0x7ffb080aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffaf4060550 0x7ffaf4062a00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb080c63f0 0x7ffb080b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 --2- 172.21.15.50:0/2486492379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb0806db40 0x7ffb0815b560 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 >> 172.21.15.50:0/2486492379 conn(0x7ffb08067bb0 msgr2=0x7ffb080adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 shutdown_connections 2023-12-19T14:39:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.377+0000 7ffb0dd45700 1 -- 172.21.15.50:0/2486492379 wait complete. 2023-12-19T14:39:02.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.565+0000 7f3d90a78700 1 Processor -- start 2023-12-19T14:39:02.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d90a78700 1 -- start start 2023-12-19T14:39:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d90a78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d8c067c90 0x7f3d8c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d90a78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0c6630 con 0x7f3d8c14fba0 2023-12-19T14:39:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d90a78700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0c6910 con 0x7f3d8c0c5900 2023-12-19T14:39:02.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d90a78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c0c6bf0 con 0x7f3d8c067c90 2023-12-19T14:39:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d8c067c90 0x7f3d8c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d8c067c90 0x7f3d8c068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37212/0 (socket says 172.21.15.50:37212) 2023-12-19T14:39:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 -- 172.21.15.50:0/2011626147 learned_addr learned my addr 172.21.15.50:0/2011626147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:02.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d7bfff700 1 --1- 172.21.15.50:0/2011626147 >> v1:172.21.15.50:6789/0 conn(0x7f3d8c14fba0 0x7f3d8c150000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49798/0 (socket says 172.21.15.50:49798) 2023-12-19T14:39:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 -- 172.21.15.50:0/2011626147 >> v1:172.21.15.125:6789/0 conn(0x7f3d8c0c5900 legacy=0x7f3d8c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 -- 172.21.15.50:0/2011626147 >> v1:172.21.15.50:6789/0 conn(0x7f3d8c14fba0 legacy=0x7f3d8c150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.569+0000 7f3d8a59c700 1 -- 172.21.15.50:0/2011626147 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c0c6ed0 con 0x7f3d8c067c90 2023-12-19T14:39:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d8a59c700 1 --2- 172.21.15.50:0/2011626147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c067c90 0x7f3d8c068060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009f20 tx=0x7f3d7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40a29bdde4f62da server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d89d9b700 1 -- 172.21.15.50:0/2011626147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c00e070 con 0x7f3d8c067c90 2023-12-19T14:39:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d89d9b700 1 -- 172.21.15.50:0/2011626147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c00ba80 con 0x7f3d8c067c90 2023-12-19T14:39:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d89d9b700 1 -- 172.21.15.50:0/2011626147 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0092e0 con 0x7f3d8c067c90 2023-12-19T14:39:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 -- 172.21.15.50:0/2011626147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c067c90 msgr2=0x7f3d8c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 --2- 172.21.15.50:0/2011626147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c067c90 0x7f3d8c068060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009f20 tx=0x7f3d7c009840 comp rx=0 tx=0).stop 2023-12-19T14:39:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 -- 172.21.15.50:0/2011626147 shutdown_connections 2023-12-19T14:39:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 --2- 172.21.15.50:0/2011626147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c067c90 0x7f3d8c068060 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 -- 172.21.15.50:0/2011626147 >> 172.21.15.50:0/2011626147 conn(0x7f3d8c0cd6d0 msgr2=0x7f3d8c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 -- 172.21.15.50:0/2011626147 shutdown_connections 2023-12-19T14:39:02.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.573+0000 7f3d90a78700 1 -- 172.21.15.50:0/2011626147 wait complete. 2023-12-19T14:39:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 Processor -- start 2023-12-19T14:39:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 -- start start 2023-12-19T14:39:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8c067c90 0x7f3d8c140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8c14fba0 0x7f3d8c0b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c144500 con 0x7f3d8c0c5900 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0b6020 con 0x7f3d8c14fba0 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d90a78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0b5a60 con 0x7f3d8c067c90 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37222/0 (socket says 172.21.15.50:37222) 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 -- 172.21.15.50:0/1397262422 learned_addr learned my addr 172.21.15.50:0/1397262422 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d8a59c700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8c067c90 0x7f3d8c140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d8ad9d700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8c14fba0 0x7f3d8c0b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 -- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8c067c90 msgr2=0x7f3d8c140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8c067c90 0x7f3d8c140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 -- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8c14fba0 msgr2=0x7f3d8c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8c14fba0 0x7f3d8c0b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.577+0000 7f3d7bfff700 1 -- 172.21.15.50:0/1397262422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c009ae0 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d7bfff700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c560 tx=0x7f3d80007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80005a70 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80005bd0 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c0b5d40 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800070d0 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c140880 con 0x7f3d8c0c5900 2023-12-19T14:39:02.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c1446b0 con 0x7f3d8c0c5900 2023-12-19T14:39:02.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.581+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3d80007480 con 0x7f3d8c0c5900 2023-12-19T14:39:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.585+0000 7f3d7a7fc700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 0x7f3d6c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:02.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.585+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3d8007b970 con 0x7f3d8c0c5900 2023-12-19T14:39:02.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.585+0000 7f3d8a59c700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 0x7f3d6c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:02.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.585+0000 7f3d8a59c700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 0x7f3d6c062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009880 tx=0x7f3d7c018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:02.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.589+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3d8c1446b0 con 0x7f3d8c0c5900 2023-12-19T14:39:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.801+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c071120 con 0x7f3d8c0c5900 2023-12-19T14:39:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d7a7fc700 1 -- 172.21.15.50:0/1397262422 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7f3d80017090 con 0x7f3d8c0c5900 2023-12-19T14:39:02.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:02.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 msgr2=0x7f3d6c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 0x7f3d6c062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009880 tx=0x7f3d7c018000 comp rx=0 tx=0).stop 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 msgr2=0x7f3d8c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c560 tx=0x7f3d80007f70 comp rx=0 tx=0).stop 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 shutdown_connections 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.805+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8c14fba0 0x7f3d8c0b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d6c060580 0x7f3d6c062a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d8c0c5900 0x7f3d8c0b4f30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 --2- 172.21.15.50:0/1397262422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8c067c90 0x7f3d8c140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 >> 172.21.15.50:0/1397262422 conn(0x7f3d8c0cd6d0 msgr2=0x7f3d8c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 shutdown_connections 2023-12-19T14:39:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:02.809+0000 7f3d90a78700 1 -- 172.21.15.50:0/1397262422 wait complete. 2023-12-19T14:39:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 3 2023-12-19T14:39:02.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.001+0000 7fbe9da2e700 1 Processor -- start 2023-12-19T14:39:03.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 -- start start 2023-12-19T14:39:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbe980b04e0 0x7fbe980ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbe98153290 0x7fbe98153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980ccb40 con 0x7fbe980b0110 2023-12-19T14:39:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980b0980 con 0x7fbe980b04e0 2023-12-19T14:39:03.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe9da2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980cd510 con 0x7fbe98153290 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbe980b04e0 0x7fbe980ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbe980b04e0 0x7fbe980ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37226/0 (socket says 172.21.15.50:37226) 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 -- 172.21.15.50:0/3868980476 learned_addr learned my addr 172.21.15.50:0/3868980476 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe977fe700 1 --2- 172.21.15.50:0/3868980476 >> v2:172.21.15.125:3300/0 conn(0x7fbe98153290 0x7fbe98153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 -- 172.21.15.50:0/3868980476 >> v2:172.21.15.125:3300/0 conn(0x7fbe98153290 msgr2=0x7fbe98153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 --2- 172.21.15.50:0/3868980476 >> v2:172.21.15.125:3300/0 conn(0x7fbe98153290 0x7fbe98153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 -- 172.21.15.50:0/3868980476 >> v1:172.21.15.50:6789/0 conn(0x7fbe980b0110 legacy=0x7fbe9815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.005+0000 7fbe967fc700 1 -- 172.21.15.50:0/3868980476 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe98153bd0 con 0x7fbe980b04e0 2023-12-19T14:39:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe967fc700 1 --2- 172.21.15.50:0/3868980476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe980ccfd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe980cdd50 tx=0x7fbe8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e304cf3895718ad5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe95ffb700 1 -- 172.21.15.50:0/3868980476 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c00b820 con 0x7fbe980b04e0 2023-12-19T14:39:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe95ffb700 1 -- 172.21.15.50:0/3868980476 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00b980 con 0x7fbe980b04e0 2023-12-19T14:39:03.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe95ffb700 1 -- 172.21.15.50:0/3868980476 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c00bd20 con 0x7fbe980b04e0 2023-12-19T14:39:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 -- 172.21.15.50:0/3868980476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 msgr2=0x7fbe980ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/3868980476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe980ccfd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe980cdd50 tx=0x7fbe8c009250 comp rx=0 tx=0).stop 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 -- 172.21.15.50:0/3868980476 shutdown_connections 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/3868980476 >> v2:172.21.15.125:3300/0 conn(0x7fbe98153290 0x7fbe98153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/3868980476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe980ccfd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 -- 172.21.15.50:0/3868980476 >> 172.21.15.50:0/3868980476 conn(0x7fbe9806c7c0 msgr2=0x7fbe9806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 -- 172.21.15.50:0/3868980476 shutdown_connections 2023-12-19T14:39:03.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.009+0000 7fbe9da2e700 1 -- 172.21.15.50:0/3868980476 wait complete. 2023-12-19T14:39:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 Processor -- start 2023-12-19T14:39:03.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 -- start start 2023-12-19T14:39:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe980b0110 0x7fbe980ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 0x7fbe981456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9816d720 con 0x7fbe980b04e0 2023-12-19T14:39:03.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9816cb10 con 0x7fbe98153290 2023-12-19T14:39:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe9da2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9816cdf0 con 0x7fbe980b0110 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37240/0 (socket says 172.21.15.50:37240) 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 -- 172.21.15.50:0/2387683417 learned_addr learned my addr 172.21.15.50:0/2387683417 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe977fe700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 0x7fbe981456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe96ffd700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe980b0110 0x7fbe980ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 -- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe980b0110 msgr2=0x7fbe980ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe980b0110 0x7fbe980ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 -- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 msgr2=0x7fbe981456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 0x7fbe981456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe967fc700 1 -- 172.21.15.50:0/2387683417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c0094f0 con 0x7fbe980b04e0 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.013+0000 7fbe977fe700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 0x7fbe981456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe967fc700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0094c0 tx=0x7fbe8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c024440 con 0x7fbe980b04e0 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c0245a0 con 0x7fbe980b04e0 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9816d0d0 con 0x7fbe980b04e0 2023-12-19T14:39:03.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c024940 con 0x7fbe980b04e0 2023-12-19T14:39:03.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe98151fa0 con 0x7fbe980b04e0 2023-12-19T14:39:03.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.017+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe98071120 con 0x7fbe980b04e0 2023-12-19T14:39:03.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.021+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbe8c01a420 con 0x7fbe980b04e0 2023-12-19T14:39:03.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.021+0000 7fbe7effd700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 0x7fbe84062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.021+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fbe8c07e3b0 con 0x7fbe980b04e0 2023-12-19T14:39:03.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.021+0000 7fbe96ffd700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 0x7fbe84062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.025+0000 7fbe96ffd700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 0x7fbe84062a80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe88003720 tx=0x7fbe88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.029+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbe8c04ec50 con 0x7fbe980b04e0 2023-12-19T14:39:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.205+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe9806dba0 con 0x7fbe980b04e0 2023-12-19T14:39:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.205+0000 7fbe7effd700 1 -- 172.21.15.50:0/2387683417 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12127 (secure 0 0 0) 0x7fbe8c052270 con 0x7fbe980b04e0 2023-12-19T14:39:03.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:03.213 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":16,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:38:58.489546+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 msgr2=0x7fbe84062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 0x7fbe84062a80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe88003720 tx=0x7fbe88009040 comp rx=0 tx=0).stop 2023-12-19T14:39:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 msgr2=0x7fbe9815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0094c0 tx=0x7fbe8c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 shutdown_connections 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe98153290 0x7fbe981456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe840605d0 0x7fbe84062a80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe980b04e0 0x7fbe9815b6f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 --2- 172.21.15.50:0/2387683417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe980b0110 0x7fbe980ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 >> 172.21.15.50:0/2387683417 conn(0x7fbe9806c7c0 msgr2=0x7fbe981527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 shutdown_connections 2023-12-19T14:39:03.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.209+0000 7fbe9da2e700 1 -- 172.21.15.50:0/2387683417 wait complete. 2023-12-19T14:39:03.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:39:03.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.397+0000 7f8308c2e700 1 Processor -- start 2023-12-19T14:39:03.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.397+0000 7f8308c2e700 1 -- start start 2023-12-19T14:39:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f830413cf20 0x7f83040cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 0x7f830406b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040cc0e0 con 0x7f830413d2f0 2023-12-19T14:39:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830406b9e0 con 0x7f830413cf20 2023-12-19T14:39:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040cb4d0 con 0x7f830406b060 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f830413cf20 0x7f83040cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8301d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f830413d2f0 0x7f83040c85d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49820/0 (socket says 172.21.15.50:49820) 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8301d9b700 1 -- 172.21.15.50:0/2498140366 learned_addr learned my addr 172.21.15.50:0/2498140366 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8302d9d700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 0x7f830406b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 -- 172.21.15.50:0/2498140366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 msgr2=0x7f830406b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 0x7f830406b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 -- 172.21.15.50:0/2498140366 >> v1:172.21.15.50:6789/0 conn(0x7f830413d2f0 legacy=0x7f83040c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 -- 172.21.15.50:0/2498140366 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040cb7b0 con 0x7f830413cf20 2023-12-19T14:39:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8302d9d700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 0x7f830406b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830159a700 1 -- 172.21.15.50:0/2498140366 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578622960 0 0) 0x7f83040cc0e0 con 0x7f830413d2f0 2023-12-19T14:39:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830259c700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f83040cadf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f40098e0 tx=0x7f82f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5491f3ed220426da server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830159a700 1 -- 172.21.15.50:0/2498140366 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4009e80 con 0x7f830413cf20 2023-12-19T14:39:03.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830159a700 1 -- 172.21.15.50:0/2498140366 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f400b860 con 0x7f830413cf20 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f830159a700 1 -- 172.21.15.50:0/2498140366 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f400bbe0 con 0x7f830413cf20 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 -- 172.21.15.50:0/2498140366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 msgr2=0x7f83040cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.401+0000 7f8308c2e700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f83040cadf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f40098e0 tx=0x7f82f40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- 172.21.15.50:0/2498140366 shutdown_connections 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830406b060 0x7f830406b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 --2- 172.21.15.50:0/2498140366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f83040cadf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- 172.21.15.50:0/2498140366 >> 172.21.15.50:0/2498140366 conn(0x7f83040cc580 msgr2=0x7f83040cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- 172.21.15.50:0/2498140366 shutdown_connections 2023-12-19T14:39:03.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- 172.21.15.50:0/2498140366 wait complete. 2023-12-19T14:39:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 Processor -- start 2023-12-19T14:39:03.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- start start 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f830406b060 0x7f830406d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830413d2f0 0x7f83040c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830415a3d0 con 0x7f830413cf20 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830406d6b0 con 0x7f830406b060 2023-12-19T14:39:03.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8308c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040c5ca0 con 0x7f830413d2f0 2023-12-19T14:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.405+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8302d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830413d2f0 0x7f83040c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f830259c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f830406b060 0x7f830406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37262/0 (socket says 172.21.15.50:37262) 2023-12-19T14:39:03.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 -- 172.21.15.50:0/4022306158 learned_addr learned my addr 172.21.15.50:0/4022306158 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 -- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830413d2f0 msgr2=0x7f83040c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830413d2f0 0x7f83040c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 -- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f830406b060 msgr2=0x7f830406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f830406b060 0x7f830406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 -- 172.21.15.50:0/4022306158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f4009420 con 0x7f830413cf20 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8301d9b700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c5e0 tx=0x7f82f8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8005b30 con 0x7f830413cf20 2023-12-19T14:39:03.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8005c90 con 0x7f830413cf20 2023-12-19T14:39:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83040c5f80 con 0x7f830413cf20 2023-12-19T14:39:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8007300 con 0x7f830413cf20 2023-12-19T14:39:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83040c6540 con 0x7f830413cf20 2023-12-19T14:39:03.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.409+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830415a5d0 con 0x7f830413cf20 2023-12-19T14:39:03.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.413+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f82f801b3b0 con 0x7f830413cf20 2023-12-19T14:39:03.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.413+0000 7f82f27fc700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 0x7f82e4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.413+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f82f807b890 con 0x7f830413cf20 2023-12-19T14:39:03.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.417+0000 7f830259c700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 0x7f82e4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.417+0000 7f830259c700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 0x7f82e4062a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f40091e0 tx=0x7f82f4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.421+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f830415a5d0 con 0x7f830413cf20 2023-12-19T14:39:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.593+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f830415a5a0 con 0x7f830413cf20 2023-12-19T14:39:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.593+0000 7f82f27fc700 1 -- 172.21.15.50:0/4022306158 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82f804c130 con 0x7f830413cf20 2023-12-19T14:39:03.600 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 msgr2=0x7f82e4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 0x7f82e4062a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f82f40091e0 tx=0x7f82f4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 msgr2=0x7f830413c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c5e0 tx=0x7f82f8000fe0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 shutdown_connections 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82e4060580 0x7f82e4062a30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f830413d2f0 0x7f83040c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f830413cf20 0x7f830413c060 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 --2- 172.21.15.50:0/4022306158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f830406b060 0x7f830406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 >> 172.21.15.50:0/4022306158 conn(0x7f83040cc580 msgr2=0x7f83040ada80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 shutdown_connections 2023-12-19T14:39:03.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.597+0000 7f8308c2e700 1 -- 172.21.15.50:0/4022306158 wait complete. 2023-12-19T14:39:03.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:39:03.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.781+0000 7f686a82e700 1 Processor -- start 2023-12-19T14:39:03.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.781+0000 7f686a82e700 1 -- start start 2023-12-19T14:39:03.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f686a82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f68640a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f686a82e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686406dc20 con 0x7f68640ae510 2023-12-19T14:39:03.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f686a82e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686406d940 con 0x7f68640c1d60 2023-12-19T14:39:03.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f686a82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640c2200 con 0x7f68640ae140 2023-12-19T14:39:03.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f68637fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f68640ae510 0x7f686406d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49826/0 (socket says 172.21.15.50:49826) 2023-12-19T14:39:03.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f68637fe700 1 -- 172.21.15.50:0/1881821995 learned_addr learned my addr 172.21.15.50:0/1881821995 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:03.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6863fff700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f68640a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625654688 0 0) 0x7f686406dc20 con 0x7f68640ae510 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c003570 con 0x7f68640ae510 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350127438 0 0) 0x7f686406d940 con 0x7f68640c1d60 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686406dc20 con 0x7f68640c1d60 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106335717 0 0) 0x7f684c003570 con 0x7f68640ae510 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 msgr2=0x7f68640a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f68640a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 >> v1:172.21.15.125:6789/0 conn(0x7f68640c1d60 legacy=0x7f68640680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640c6e20 con 0x7f68640ae510 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f6858001470 con 0x7f68640ae510 2023-12-19T14:39:03.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 >> v1:172.21.15.50:6789/0 conn(0x7f68640ae510 legacy=0x7f686406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f684c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f684c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640c2200 con 0x7f684c003c20 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00bb40 con 0x7f684c004d50 2023-12-19T14:39:03.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.785+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c00bd00 con 0x7f68640ae140 2023-12-19T14:39:03.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f684c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f68637fe700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f684c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6868dcb700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 -- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 msgr2=0x7f684c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640ae140 0x7f684c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 -- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f684c004d50 msgr2=0x7f684c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f684c004d50 0x7f684c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 -- 172.21.15.50:0/1881821995 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c003570 con 0x7f684c003c20 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6863fff700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f684c004010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6854009c00 tx=0x7f68540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2498b0f2b9106f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685400e710 con 0x7f684c003c20 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685400e870 con 0x7f684c003c20 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6862ffd700 1 -- 172.21.15.50:0/1881821995 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685400ec30 con 0x7f684c003c20 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 -- 172.21.15.50:0/1881821995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 msgr2=0x7f684c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 --2- 172.21.15.50:0/1881821995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f684c004010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6854009c00 tx=0x7f68540091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f6868dcb700 1 -- 172.21.15.50:0/1881821995 reap_dead start 2023-12-19T14:39:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 -- 172.21.15.50:0/1881821995 shutdown_connections 2023-12-19T14:39:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 -- 172.21.15.50:0/1881821995 >> 172.21.15.50:0/1881821995 conn(0x7f68640c8850 msgr2=0x7f68640c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 -- 172.21.15.50:0/1881821995 shutdown_connections 2023-12-19T14:39:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.789+0000 7f686a82e700 1 -- 172.21.15.50:0/1881821995 wait complete. 2023-12-19T14:39:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 Processor -- start 2023-12-19T14:39:03.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 -- start start 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6864156940 0x7f68640c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640caaf0 0x7f68640caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640cb4a0 con 0x7f684c003c20 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640c3b00 con 0x7f6864156940 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.793+0000 7f686a82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640c2f30 con 0x7f68640caaf0 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37278/0 (socket says 172.21.15.50:37278) 2023-12-19T14:39:03.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 -- 172.21.15.50:0/1243090757 learned_addr learned my addr 172.21.15.50:0/1243090757 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f68637fe700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6864156940 0x7f68640c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6868dcb700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640caaf0 0x7f68640caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 -- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640caaf0 msgr2=0x7f68640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640caaf0 0x7f68640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 -- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6864156940 msgr2=0x7f68640c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6864156940 0x7f68640c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 -- 172.21.15.50:0/1243090757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864156610 con 0x7f684c003c20 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f6863fff700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f684c004790 tx=0x7f685401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6854022070 con 0x7f684c003c20 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685400e9e0 con 0x7f684c003c20 2023-12-19T14:39:03.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685400e710 con 0x7f684c003c20 2023-12-19T14:39:03.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68640c31d0 con 0x7f684c003c20 2023-12-19T14:39:03.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.797+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68640c3790 con 0x7f684c003c20 2023-12-19T14:39:03.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.801+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864071120 con 0x7f684c003c20 2023-12-19T14:39:03.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.801+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f685401c660 con 0x7f684c003c20 2023-12-19T14:39:03.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.801+0000 7f684bfff700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 0x7f6850062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:03.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.805+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f685407dc10 con 0x7f684c003c20 2023-12-19T14:39:03.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.805+0000 7f68637fe700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 0x7f6850062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:03.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.805+0000 7f68637fe700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 0x7f6850062a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68640cb920 tx=0x7f6858001e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:03.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.809+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6854051f40 con 0x7f684c003c20 2023-12-19T14:39:03.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.981+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f68640642a0 con 0x7f684c003c20 2023-12-19T14:39:03.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.981+0000 7f684bfff700 1 -- 172.21.15.50:0/1243090757 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f685402b020 con 0x7f684c003c20 2023-12-19T14:39:03.987 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:39:03.988 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:39:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 msgr2=0x7f6850062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 0x7f6850062a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68640cb920 tx=0x7f6858001e10 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 msgr2=0x7f68641560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f684c004790 tx=0x7f685401a000 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 shutdown_connections 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6864156940 0x7f68640c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6850060580 0x7f6850062a30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68640caaf0 0x7f68640caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 --2- 172.21.15.50:0/1243090757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f684c003c20 0x7f68641560d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 >> 172.21.15.50:0/1243090757 conn(0x7f68640c8850 msgr2=0x7f68640670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 shutdown_connections 2023-12-19T14:39:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:03.985+0000 7f686a82e700 1 -- 172.21.15.50:0/1243090757 wait complete. 2023-12-19T14:39:04.012 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:39:04.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:39:04.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.181+0000 7fbe658f0700 1 Processor -- start 2023-12-19T14:39:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.181+0000 7fbe658f0700 1 -- start start 2023-12-19T14:39:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe658f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe600cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe658f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbe6015c930 0x7fbe6006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe658f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe600b0ac0 con 0x7fbe6006a6a0 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe658f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe6006b030 con 0x7fbe6015c930 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe658f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6006b210 con 0x7fbe600cb130 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbe6006a6a0 0x7fbe6015c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49840/0 (socket says 172.21.15.50:49840) 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5e7fc700 1 -- 172.21.15.50:0/97751047 learned_addr learned my addr 172.21.15.50:0/97751047 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5f7fe700 1 --2- 172.21.15.50:0/97751047 >> v2:172.21.15.125:3300/0 conn(0x7fbe6015c930 0x7fbe6006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 --2- 172.21.15.50:0/97751047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe600cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 -- 172.21.15.50:0/97751047 >> v2:172.21.15.125:3300/0 conn(0x7fbe6015c930 msgr2=0x7fbe6006aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 --2- 172.21.15.50:0/97751047 >> v2:172.21.15.125:3300/0 conn(0x7fbe6015c930 0x7fbe6006aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 -- 172.21.15.50:0/97751047 >> v1:172.21.15.50:6789/0 conn(0x7fbe6006a6a0 legacy=0x7fbe6015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:04.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 -- 172.21.15.50:0/97751047 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe600ca810 con 0x7fbe600cb130 2023-12-19T14:39:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5dffb700 1 -- 172.21.15.50:0/97751047 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494107534 0 0) 0x7fbe600b0ac0 con 0x7fbe6006a6a0 2023-12-19T14:39:04.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.185+0000 7fbe5effd700 1 --2- 172.21.15.50:0/97751047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe600cb500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098e0 tx=0x7fbe500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3450599f3d173f91 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe5dffb700 1 -- 172.21.15.50:0/97751047 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5000b640 con 0x7fbe600cb130 2023-12-19T14:39:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe5dffb700 1 -- 172.21.15.50:0/97751047 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5000b7a0 con 0x7fbe600cb130 2023-12-19T14:39:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe5dffb700 1 -- 172.21.15.50:0/97751047 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5000bb40 con 0x7fbe600cb130 2023-12-19T14:39:04.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 -- 172.21.15.50:0/97751047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 msgr2=0x7fbe600cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:04.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 --2- 172.21.15.50:0/97751047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe600cb500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098e0 tx=0x7fbe500091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 -- 172.21.15.50:0/97751047 shutdown_connections 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 --2- 172.21.15.50:0/97751047 >> v2:172.21.15.125:3300/0 conn(0x7fbe6015c930 0x7fbe6006aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 --2- 172.21.15.50:0/97751047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe600cb500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 -- 172.21.15.50:0/97751047 >> 172.21.15.50:0/97751047 conn(0x7fbe60067bb0 msgr2=0x7fbe600cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 -- 172.21.15.50:0/97751047 shutdown_connections 2023-12-19T14:39:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.193+0000 7fbe658f0700 1 -- 172.21.15.50:0/97751047 wait complete. 2023-12-19T14:39:04.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 Processor -- start 2023-12-19T14:39:04.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 -- start start 2023-12-19T14:39:04.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe6006a6a0 0x7fbe600c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe6015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 0x7fbe600b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600b46f0 con 0x7fbe6015c930 2023-12-19T14:39:04.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60158b90 con 0x7fbe6006a6a0 2023-12-19T14:39:04.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe658f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60158e30 con 0x7fbe600cb130 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe6015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 0x7fbe600b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5effd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe6006a6a0 0x7fbe600c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe6015b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:50894/0 (socket says 172.21.15.50:50894) 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5e7fc700 1 -- 172.21.15.50:0/1781777722 learned_addr learned my addr 172.21.15.50:0/1781777722 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 -- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 msgr2=0x7fbe6015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe6015b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 -- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe6006a6a0 msgr2=0x7fbe600c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe6006a6a0 0x7fbe600c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:04.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.197+0000 7fbe5f7fe700 1 -- 172.21.15.50:0/1781777722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe50009420 con 0x7fbe6015c930 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe5f7fe700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 0x7fbe600b4b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800ec90 tx=0x7fbe5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800c890 con 0x7fbe6015c930 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5800f040 con 0x7fbe6015c930 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800cc60 con 0x7fbe6015c930 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe60159110 con 0x7fbe6015c930 2023-12-19T14:39:04.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe601559d0 con 0x7fbe6015c930 2023-12-19T14:39:04.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe60155bf0 con 0x7fbe6015c930 2023-12-19T14:39:04.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.201+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbe58014690 con 0x7fbe6015c930 2023-12-19T14:39:04.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.205+0000 7fbe46ffd700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 0x7fbe4c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:04.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.205+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fbe5807cf10 con 0x7fbe6015c930 2023-12-19T14:39:04.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.205+0000 7fbe5effd700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 0x7fbe4c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:04.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.205+0000 7fbe5effd700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 0x7fbe4c062a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe500093f0 tx=0x7fbe50017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:04.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.213+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbe60155bf0 con 0x7fbe6015c930 2023-12-19T14:39:04.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:04.385+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbe60155bf0 con 0x7fbe6015c930 2023-12-19T14:39:04.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:39:04.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:39:05.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.209+0000 7fbe46ffd700 1 -- 172.21.15.50:0/1781777722 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbe60155bf0 con 0x7fbe6015c930 2023-12-19T14:39:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 msgr2=0x7fbe4c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 0x7fbe4c062a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe500093f0 tx=0x7fbe50017000 comp rx=0 tx=0).stop 2023-12-19T14:39:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 msgr2=0x7fbe600b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 0x7fbe600b4b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800ec90 tx=0x7fbe5800b040 comp rx=0 tx=0).stop 2023-12-19T14:39:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 shutdown_connections 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe4c060580 0x7fbe4c062a30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe6015c930 0x7fbe600b4b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbe600cb130 0x7fbe6015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 --2- 172.21.15.50:0/1781777722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe6006a6a0 0x7fbe600c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 >> 172.21.15.50:0/1781777722 conn(0x7fbe60067bb0 msgr2=0x7fbe600cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 shutdown_connections 2023-12-19T14:39:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.213+0000 7fbe658f0700 1 -- 172.21.15.50:0/1781777722 wait complete. 2023-12-19T14:39:05.225 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:39:05.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-12-19T14:39:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.409+0000 7f32d8e16700 1 Processor -- start 2023-12-19T14:39:05.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.409+0000 7f32d8e16700 1 -- start start 2023-12-19T14:39:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d8e16700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32d4158c90 0x7f32d40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d8e16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 0x7f32d406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d8e16700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d406c850 con 0x7f32d406aeb0 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d8e16700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d406b470 con 0x7f32d4158c90 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d8e16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d406b750 con 0x7f32d4159770 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32d4158c90 0x7f32d40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f32d4158c90 0x7f32d40caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37294/0 (socket says 172.21.15.50:37294) 2023-12-19T14:39:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 -- 172.21.15.50:0/2034722258 learned_addr learned my addr 172.21.15.50:0/2034722258 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d3fff700 1 --2- 172.21.15.50:0/2034722258 >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 0x7f32d406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 -- 172.21.15.50:0/2034722258 >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 msgr2=0x7f32d406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 --2- 172.21.15.50:0/2034722258 >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 0x7f32d406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 -- 172.21.15.50:0/2034722258 >> v1:172.21.15.125:6789/0 conn(0x7f32d406aeb0 legacy=0x7f32d4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 -- 172.21.15.50:0/2034722258 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d406ba30 con 0x7f32d4158c90 2023-12-19T14:39:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d3fff700 1 --2- 172.21.15.50:0/2034722258 >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 0x7f32d406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.413+0000 7f32d37fe700 1 --2- 172.21.15.50:0/2034722258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d4158c90 0x7f32d40caf40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32d40cc7f0 tx=0x7f32c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1618a8fb0199a1df server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d27fc700 1 -- 172.21.15.50:0/2034722258 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c400b860 con 0x7f32d4158c90 2023-12-19T14:39:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d27fc700 1 -- 172.21.15.50:0/2034722258 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c400b9c0 con 0x7f32d4158c90 2023-12-19T14:39:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d27fc700 1 -- 172.21.15.50:0/2034722258 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c400bd60 con 0x7f32d4158c90 2023-12-19T14:39:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 -- 172.21.15.50:0/2034722258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d4158c90 msgr2=0x7f32d40caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 --2- 172.21.15.50:0/2034722258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d4158c90 0x7f32d40caf40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32d40cc7f0 tx=0x7f32c4009250 comp rx=0 tx=0).stop 2023-12-19T14:39:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 -- 172.21.15.50:0/2034722258 shutdown_connections 2023-12-19T14:39:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 --2- 172.21.15.50:0/2034722258 >> v2:172.21.15.125:3300/0 conn(0x7f32d4159770 0x7f32d406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 --2- 172.21.15.50:0/2034722258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d4158c90 0x7f32d40caf40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 -- 172.21.15.50:0/2034722258 >> 172.21.15.50:0/2034722258 conn(0x7f32d4067bb0 msgr2=0x7f32d40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 -- 172.21.15.50:0/2034722258 shutdown_connections 2023-12-19T14:39:05.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.417+0000 7f32d8e16700 1 -- 172.21.15.50:0/2034722258 wait complete. 2023-12-19T14:39:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 Processor -- start 2023-12-19T14:39:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 -- start start 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32d4158c90 0x7f32d414b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32d4159770 0x7f32d40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40c5c60 con 0x7f32d406aeb0 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4157980 con 0x7f32d4158c90 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d8e16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d40c5050 con 0x7f32d4159770 2023-12-19T14:39:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37304/0 (socket says 172.21.15.50:37304) 2023-12-19T14:39:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 -- 172.21.15.50:0/1621395287 learned_addr learned my addr 172.21.15.50:0/1621395287 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d3fff700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32d4159770 0x7f32d40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d2ffd700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32d4158c90 0x7f32d414b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 -- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32d4159770 msgr2=0x7f32d40ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32d4159770 0x7f32d40ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 -- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32d4158c90 msgr2=0x7f32d414b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32d4158c90 0x7f32d414b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 -- 172.21.15.50:0/1621395287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40094f0 con 0x7f32d406aeb0 2023-12-19T14:39:05.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.421+0000 7f32d37fe700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009290 tx=0x7f32c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4011e60 con 0x7f32d406aeb0 2023-12-19T14:39:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c401f070 con 0x7f32d406aeb0 2023-12-19T14:39:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d40c5330 con 0x7f32d406aeb0 2023-12-19T14:39:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c401a870 con 0x7f32d406aeb0 2023-12-19T14:39:05.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d40c58f0 con 0x7f32d406aeb0 2023-12-19T14:39:05.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.425+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d40c85d0 con 0x7f32d406aeb0 2023-12-19T14:39:05.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.429+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f32c401abe0 con 0x7f32d406aeb0 2023-12-19T14:39:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.429+0000 7f32bb7fe700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 0x7f32c0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:05.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.429+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f32c4021070 con 0x7f32d406aeb0 2023-12-19T14:39:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.429+0000 7f32d2ffd700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 0x7f32c0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.433+0000 7f32d2ffd700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 0x7f32c0062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32d4153be0 tx=0x7f32c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:05.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.437+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f32c404f0b0 con 0x7f32d406aeb0 2023-12-19T14:39:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:05.609+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f32d404be40 con 0x7f32d406aeb0 2023-12-19T14:39:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.221+0000 7f32bb7fe700 1 -- 172.21.15.50:0/1621395287 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f32c40150d0 con 0x7f32d406aeb0 2023-12-19T14:39:06.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 msgr2=0x7f32c0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 0x7f32c0062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f32d4153be0 tx=0x7f32c8009040 comp rx=0 tx=0).stop 2023-12-19T14:39:06.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 msgr2=0x7f32d4157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009290 tx=0x7f32c4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 shutdown_connections 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32d4158c90 0x7f32d414b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f32c0060550 0x7f32c0062a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32d4159770 0x7f32d40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 --2- 172.21.15.50:0/1621395287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32d406aeb0 0x7f32d4157440 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 >> 172.21.15.50:0/1621395287 conn(0x7f32d4067bb0 msgr2=0x7f32d40afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 shutdown_connections 2023-12-19T14:39:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.229+0000 7f32d8e16700 1 -- 172.21.15.50:0/1621395287 wait complete. 2023-12-19T14:39:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:39:06.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:39:06.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.429+0000 7f1850d45700 1 Processor -- start 2023-12-19T14:39:06.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.429+0000 7f1850d45700 1 -- start start 2023-12-19T14:39:06.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1850d45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c0cb970 0x7f184c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1850d45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f184c0cecd0 0x7f184c06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1850d45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c0cbdf0 con 0x7f184c158e20 2023-12-19T14:39:06.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1850d45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c0cc0d0 con 0x7f184c0cecd0 2023-12-19T14:39:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1850d45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0cc3b0 con 0x7f184c0cb970 2023-12-19T14:39:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f184c0cecd0 0x7f184c06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f1849d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c0cb970 0x7f184c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184a59c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f184c158e20 0x7f184c0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52558/0 (socket says 172.21.15.50:52558) 2023-12-19T14:39:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184a59c700 1 -- 172.21.15.50:0/1254032796 learned_addr learned my addr 172.21.15.50:0/1254032796 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184ad9d700 1 -- 172.21.15.50:0/1254032796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c0cb970 msgr2=0x7f184c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184ad9d700 1 --2- 172.21.15.50:0/1254032796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c0cb970 0x7f184c06acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184ad9d700 1 -- 172.21.15.50:0/1254032796 >> v1:172.21.15.125:6789/0 conn(0x7f184c158e20 legacy=0x7f184c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184ad9d700 1 -- 172.21.15.50:0/1254032796 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0cc690 con 0x7f184c0cecd0 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.433+0000 7f184959a700 1 -- 172.21.15.50:0/1254032796 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113876775 0 0) 0x7f184c0cbdf0 con 0x7f184c158e20 2023-12-19T14:39:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f184ad9d700 1 --2- 172.21.15.50:0/1254032796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c0cecd0 0x7f184c06b200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e60 tx=0x7f1844009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a385b9dd32c685ce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f184959a700 1 -- 172.21.15.50:0/1254032796 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184400e070 con 0x7f184c0cecd0 2023-12-19T14:39:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f184959a700 1 -- 172.21.15.50:0/1254032796 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184400bb60 con 0x7f184c0cecd0 2023-12-19T14:39:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f184959a700 1 -- 172.21.15.50:0/1254032796 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844012430 con 0x7f184c0cecd0 2023-12-19T14:39:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 -- 172.21.15.50:0/1254032796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c0cecd0 msgr2=0x7f184c06b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 --2- 172.21.15.50:0/1254032796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c0cecd0 0x7f184c06b200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e60 tx=0x7f1844009780 comp rx=0 tx=0).stop 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 -- 172.21.15.50:0/1254032796 shutdown_connections 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 --2- 172.21.15.50:0/1254032796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c0cecd0 0x7f184c06b200 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 --2- 172.21.15.50:0/1254032796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c0cb970 0x7f184c06acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 -- 172.21.15.50:0/1254032796 >> 172.21.15.50:0/1254032796 conn(0x7f184c067bb0 msgr2=0x7f184c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 -- 172.21.15.50:0/1254032796 shutdown_connections 2023-12-19T14:39:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.437+0000 7f1850d45700 1 -- 172.21.15.50:0/1254032796 wait complete. 2023-12-19T14:39:06.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 Processor -- start 2023-12-19T14:39:06.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 -- start start 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f184c0cecd0 0x7f184c0c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c158e20 0x7f184c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c15c150 con 0x7f184c15aa50 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c15b410 con 0x7f184c0cecd0 2023-12-19T14:39:06.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1850d45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0b3af0 con 0x7f184c158e20 2023-12-19T14:39:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36012/0 (socket says 172.21.15.50:36012) 2023-12-19T14:39:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 -- 172.21.15.50:0/662853383 learned_addr learned my addr 172.21.15.50:0/662853383 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184a59c700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f184c0cecd0 0x7f184c0c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f1849d9b700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c158e20 0x7f184c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 -- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c158e20 msgr2=0x7f184c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c158e20 0x7f184c15a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 -- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f184c0cecd0 msgr2=0x7f184c0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f184c0cecd0 0x7f184c0c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.441+0000 7f184ad9d700 1 -- 172.21.15.50:0/662853383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009a20 con 0x7f184c15aa50 2023-12-19T14:39:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f184ad9d700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1844013000 tx=0x7f1844019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:06.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844012ea0 con 0x7f184c15aa50 2023-12-19T14:39:06.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1844027470 con 0x7f184c15aa50 2023-12-19T14:39:06.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c0b3dd0 con 0x7f184c15aa50 2023-12-19T14:39:06.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844027830 con 0x7f184c15aa50 2023-12-19T14:39:06.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c0b4390 con 0x7f184c15aa50 2023-12-19T14:39:06.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.445+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c0b4580 con 0x7f184c15aa50 2023-12-19T14:39:06.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.449+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1844024070 con 0x7f184c15aa50 2023-12-19T14:39:06.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.449+0000 7f183a7fc700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 0x7f182c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:06.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.449+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5605+0+0 (secure 0 0 0) 0x7f184407ee30 con 0x7f184c15aa50 2023-12-19T14:39:06.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.449+0000 7f184a59c700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 0x7f182c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:06.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.449+0000 7f184a59c700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 0x7f182c062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:06.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.457+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f184404f3e0 con 0x7f184c15aa50 2023-12-19T14:39:06.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:06.669+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f184c071120 con 0x7f184c15aa50 2023-12-19T14:39:07.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.261+0000 7f183a7fc700 1 -- 172.21.15.50:0/662853383 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f184402c0c0 con 0x7f184c15aa50 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 msgr2=0x7f182c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 0x7f182c062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c007d30 comp rx=0 tx=0).stop 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 msgr2=0x7f184c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1844013000 tx=0x7f1844019000 comp rx=0 tx=0).stop 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 shutdown_connections 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f182c060550 0x7f182c062a00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f184c15aa50 0x7f184c15aed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f184c158e20 0x7f184c15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 --2- 172.21.15.50:0/662853383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f184c0cecd0 0x7f184c0c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 >> 172.21.15.50:0/662853383 conn(0x7f184c067bb0 msgr2=0x7f184c0ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 shutdown_connections 2023-12-19T14:39:07.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.273+0000 7f1850d45700 1 -- 172.21.15.50:0/662853383 wait complete. 2023-12-19T14:39:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 2 and data pool 3 2023-12-19T14:39:07.314 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-12-19T14:39:07.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-12-19T14:39:07.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.485+0000 7f26e38ae700 1 Processor -- start 2023-12-19T14:39:07.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.485+0000 7f26e38ae700 1 -- start start 2023-12-19T14:39:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.489+0000 7f26e38ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f26dc0cb500 0x7f26dc15c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.489+0000 7f26e38ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0b0ac0 con 0x7f26dc15c760 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e38ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc06b210 con 0x7f26dc0cb130 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e38ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26dc15cbe0 con 0x7f26dc0cb500 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e164a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f26dc0cb130 0x7f26dc06a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52564/0 (socket says 172.21.15.50:52564) 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e164a700 1 -- 172.21.15.50:0/3137884650 learned_addr learned my addr 172.21.15.50:0/3137884650 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e0e49700 1 --2- 172.21.15.50:0/3137884650 >> v2:172.21.15.50:3300/0 conn(0x7f26dc0cb500 0x7f26dc15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:07.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e0e49700 1 -- 172.21.15.50:0/3137884650 >> v1:172.21.15.125:6789/0 conn(0x7f26dc0cb130 legacy=0x7f26dc06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e0e49700 1 -- 172.21.15.50:0/3137884650 >> v1:172.21.15.50:6789/0 conn(0x7f26dc15c760 legacy=0x7f26dc0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e0e49700 1 -- 172.21.15.50:0/3137884650 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0ca810 con 0x7f26dc0cb500 2023-12-19T14:39:07.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.493+0000 7f26e0e49700 1 --2- 172.21.15.50:0/3137884650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0cb500 0x7f26dc15c220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0098e0 tx=0x7f26cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c13ee5530babecc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:07.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26d3fff700 1 -- 172.21.15.50:0/3137884650 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc009e80 con 0x7f26dc0cb500 2023-12-19T14:39:07.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26d3fff700 1 -- 172.21.15.50:0/3137884650 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc00b860 con 0x7f26dc0cb500 2023-12-19T14:39:07.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26d3fff700 1 -- 172.21.15.50:0/3137884650 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00bbe0 con 0x7f26dc0cb500 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 -- 172.21.15.50:0/3137884650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0cb500 msgr2=0x7f26dc15c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 --2- 172.21.15.50:0/3137884650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0cb500 0x7f26dc15c220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0098e0 tx=0x7f26cc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 -- 172.21.15.50:0/3137884650 shutdown_connections 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 --2- 172.21.15.50:0/3137884650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0cb500 0x7f26dc15c220 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 -- 172.21.15.50:0/3137884650 >> 172.21.15.50:0/3137884650 conn(0x7f26dc067bb0 msgr2=0x7f26dc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:07.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 -- 172.21.15.50:0/3137884650 shutdown_connections 2023-12-19T14:39:07.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.497+0000 7f26e38ae700 1 -- 172.21.15.50:0/3137884650 wait complete. 2023-12-19T14:39:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 Processor -- start 2023-12-19T14:39:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 -- start start 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26dc0cb130 0x7f26dc149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26dc15c760 0x7f26dc0b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc14a280 con 0x7f26dc0b3850 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0b4e90 con 0x7f26dc0cb130 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e38ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0b3df0 con 0x7f26dc15c760 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36028/0 (socket says 172.21.15.50:36028) 2023-12-19T14:39:07.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 -- 172.21.15.50:0/4191196131 learned_addr learned my addr 172.21.15.50:0/4191196131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:07.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e164a700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26dc0cb130 0x7f26dc149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:07.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e0e49700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26dc15c760 0x7f26dc0b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 -- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26dc15c760 msgr2=0x7f26dc0b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26dc15c760 0x7f26dc0b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 -- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26dc0cb130 msgr2=0x7f26dc149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26dc0cb130 0x7f26dc149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.501+0000 7f26e1e4b700 1 -- 172.21.15.50:0/4191196131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc009420 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26e1e4b700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26d400c840 tx=0x7f26d4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4012630 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d40127b0 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc0b40d0 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4012b70 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc0b4690 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc071120 con 0x7f26dc0b3850 2023-12-19T14:39:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.505+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f26d401a3a0 con 0x7f26dc0b3850 2023-12-19T14:39:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.509+0000 7f26d17fa700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 0x7f26c0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.509+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f26d407c9e0 con 0x7f26dc0b3850 2023-12-19T14:39:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.509+0000 7f26e164a700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 0x7f26c0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.509+0000 7f26e164a700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 0x7f26c0062a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0b5f50 tx=0x7f26d8007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:07.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.517+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f26d404cf30 con 0x7f26dc0b3850 2023-12-19T14:39:07.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:07.697+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f26dc0cbe90 con 0x7f26dc0b3850 2023-12-19T14:39:07.745 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T14:39:07.737+0000 7f00653f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:39:08.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.253+0000 7f26d17fa700 1 -- 172.21.15.50:0/4191196131 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f26d4050550 con 0x7f26dc0b3850 2023-12-19T14:39:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 msgr2=0x7f26c0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 0x7f26c0062a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0b5f50 tx=0x7f26d8007eb0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 msgr2=0x7f26dc06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26d400c840 tx=0x7f26d4001670 comp rx=0 tx=0).stop 2023-12-19T14:39:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 shutdown_connections 2023-12-19T14:39:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26c0060580 0x7f26c0062a30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26dc0b3850 0x7f26dc06c7c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26dc15c760 0x7f26dc0b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 --2- 172.21.15.50:0/4191196131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26dc0cb130 0x7f26dc149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 >> 172.21.15.50:0/4191196131 conn(0x7f26dc067bb0 msgr2=0x7f26dc0cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:08.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 shutdown_connections 2023-12-19T14:39:08.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.257+0000 7f26e38ae700 1 -- 172.21.15.50:0/4191196131 wait complete. 2023-12-19T14:39:08.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:39:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.453+0000 7fca40e4c700 1 Processor -- start 2023-12-19T14:39:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.453+0000 7fca40e4c700 1 -- start start 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06b050 0x7fca3c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca3c06c7c0 0x7fca3c15d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fca3c1594b0 0x7fca3c1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c15da20 con 0x7fca3c06c7c0 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c06da40 con 0x7fca3c1594b0 2023-12-19T14:39:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca40e4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c06cc10 con 0x7fca3c06b050 2023-12-19T14:39:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca3affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca3c06c7c0 0x7fca3c15d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca3affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca3c06c7c0 0x7fca3c15d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36044/0 (socket says 172.21.15.50:36044) 2023-12-19T14:39:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca3affd700 1 -- 172.21.15.50:0/2091728065 learned_addr learned my addr 172.21.15.50:0/2091728065 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.457+0000 7fca3b7fe700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06b050 0x7fca3c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3bfff700 1 --2- 172.21.15.50:0/2091728065 >> v2:172.21.15.125:3300/0 conn(0x7fca3c1594b0 0x7fca3c1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 -- 172.21.15.50:0/2091728065 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06b050 msgr2=0x7fca3c06b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06b050 0x7fca3c06b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 -- 172.21.15.50:0/2091728065 >> v2:172.21.15.125:3300/0 conn(0x7fca3c1594b0 msgr2=0x7fca3c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 --2- 172.21.15.50:0/2091728065 >> v2:172.21.15.125:3300/0 conn(0x7fca3c1594b0 0x7fca3c1598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 -- 172.21.15.50:0/2091728065 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c06cec0 con 0x7fca3c06c7c0 2023-12-19T14:39:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3affd700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c06c7c0 0x7fca3c15d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0aee10 tx=0x7fca30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d395d3565477c407 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3a7fc700 1 -- 172.21.15.50:0/2091728065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3000b640 con 0x7fca3c06c7c0 2023-12-19T14:39:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3a7fc700 1 -- 172.21.15.50:0/2091728065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3000b7a0 con 0x7fca3c06c7c0 2023-12-19T14:39:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca3a7fc700 1 -- 172.21.15.50:0/2091728065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3000bb60 con 0x7fca3c06c7c0 2023-12-19T14:39:08.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 -- 172.21.15.50:0/2091728065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c06c7c0 msgr2=0x7fca3c15d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:08.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c06c7c0 0x7fca3c15d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0aee10 tx=0x7fca30009250 comp rx=0 tx=0).stop 2023-12-19T14:39:08.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 -- 172.21.15.50:0/2091728065 shutdown_connections 2023-12-19T14:39:08.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c06c7c0 0x7fca3c15d4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2091728065 >> v2:172.21.15.125:3300/0 conn(0x7fca3c1594b0 0x7fca3c1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2091728065 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06b050 0x7fca3c06b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 -- 172.21.15.50:0/2091728065 >> 172.21.15.50:0/2091728065 conn(0x7fca3c0c8cd0 msgr2=0x7fca3c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:08.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 -- 172.21.15.50:0/2091728065 shutdown_connections 2023-12-19T14:39:08.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.461+0000 7fca40e4c700 1 -- 172.21.15.50:0/2091728065 wait complete. 2023-12-19T14:39:08.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.465+0000 7fca40e4c700 1 Processor -- start 2023-12-19T14:39:08.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.465+0000 7fca40e4c700 1 -- start start 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 0x7fca3c0b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06c7c0 0x7fca3c0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0a8860 con 0x7fca3c1594b0 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0a9830 con 0x7fca3c06b050 2023-12-19T14:39:08.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0a8a70 con 0x7fca3c06c7c0 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36052/0 (socket says 172.21.15.50:36052) 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 -- 172.21.15.50:0/2072507419 learned_addr learned my addr 172.21.15.50:0/2072507419 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3affd700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06c7c0 0x7fca3c0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3b7fe700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 0x7fca3c0b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 -- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06c7c0 msgr2=0x7fca3c0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06c7c0 0x7fca3c0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 -- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 msgr2=0x7fca3c0b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 0x7fca3c0b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 -- 172.21.15.50:0/2072507419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300094f0 con 0x7fca3c1594b0 2023-12-19T14:39:08.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3b7fe700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 0x7fca3c0b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca3bfff700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fca34009e20 tx=0x7fca3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400cb50 con 0x7fca3c1594b0 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3400f040 con 0x7fca3c1594b0 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34014480 con 0x7fca3c1594b0 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c0a8d50 con 0x7fca3c1594b0 2023-12-19T14:39:08.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.469+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c0a9310 con 0x7fca3c1594b0 2023-12-19T14:39:08.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.473+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c0abdb0 con 0x7fca3c1594b0 2023-12-19T14:39:08.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.473+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fca34014830 con 0x7fca3c1594b0 2023-12-19T14:39:08.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.477+0000 7fca237fe700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 0x7fca28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:08.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.477+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fca3407d1f0 con 0x7fca3c1594b0 2023-12-19T14:39:08.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.477+0000 7fca3b7fe700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 0x7fca28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:08.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.477+0000 7fca3b7fe700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 0x7fca28062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009d80 tx=0x7fca2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:08.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.485+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca3c0abdb0 con 0x7fca3c1594b0 2023-12-19T14:39:08.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:08.665+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fca3c0abdb0 con 0x7fca3c1594b0 2023-12-19T14:39:10.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.509+0000 7fca237fe700 1 -- 172.21.15.50:0/2072507419 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7fca3c0abdb0 con 0x7fca3c1594b0 2023-12-19T14:39:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 msgr2=0x7fca28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 0x7fca28062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009d80 tx=0x7fca2c008040 comp rx=0 tx=0).stop 2023-12-19T14:39:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 msgr2=0x7fca3c0cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fca34009e20 tx=0x7fca3400b040 comp rx=0 tx=0).stop 2023-12-19T14:39:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 shutdown_connections 2023-12-19T14:39:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fca28060550 0x7fca28062a00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca3c1594b0 0x7fca3c0cb4d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca3c06c7c0 0x7fca3c0caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 --2- 172.21.15.50:0/2072507419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca3c06b050 0x7fca3c0b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.521+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 >> 172.21.15.50:0/2072507419 conn(0x7fca3c0c8cd0 msgr2=0x7fca3c157b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:10.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.525+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 shutdown_connections 2023-12-19T14:39:10.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.525+0000 7fca40e4c700 1 -- 172.21.15.50:0/2072507419 wait complete. 2023-12-19T14:39:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data_ec' created 2023-12-19T14:39:10.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-12-19T14:39:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.725+0000 7f9efde8e700 1 Processor -- start 2023-12-19T14:39:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.725+0000 7f9efde8e700 1 -- start start 2023-12-19T14:39:10.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efde8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ef80aedc0 0x7f9ef80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efde8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 0x7f9ef80cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efde8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef806b3d0 con 0x7f9ef806d6b0 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efde8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef80cc140 con 0x7f9ef80aedc0 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efde8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef80cb3f0 con 0x7f9ef806af50 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ef80aedc0 0x7f9ef80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ef80aedc0 0x7f9ef80af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36060/0 (socket says 172.21.15.50:36060) 2023-12-19T14:39:10.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 -- 172.21.15.50:0/3377817685 learned_addr learned my addr 172.21.15.50:0/3377817685 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 -- 172.21.15.50:0/3377817685 >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 msgr2=0x7f9ef80cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9ef7fff700 1 --2- 172.21.15.50:0/3377817685 >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 0x7f9ef80cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 --2- 172.21.15.50:0/3377817685 >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 0x7f9ef80cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 -- 172.21.15.50:0/3377817685 >> v1:172.21.15.125:6789/0 conn(0x7f9ef806d6b0 legacy=0x7f9ef80af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9efce8c700 1 -- 172.21.15.50:0/3377817685 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef806d220 con 0x7f9ef80aedc0 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.729+0000 7f9ef7fff700 1 --2- 172.21.15.50:0/3377817685 >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 0x7f9ef80cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:10.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efce8c700 1 --2- 172.21.15.50:0/3377817685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80af190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cfc50 tx=0x7f9eec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5dc7795c31a1fa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:10.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9ef77fe700 1 -- 172.21.15.50:0/3377817685 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00b820 con 0x7f9ef80aedc0 2023-12-19T14:39:10.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9ef77fe700 1 -- 172.21.15.50:0/3377817685 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec00b980 con 0x7f9ef80aedc0 2023-12-19T14:39:10.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9ef77fe700 1 -- 172.21.15.50:0/3377817685 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00bd20 con 0x7f9ef80aedc0 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 -- 172.21.15.50:0/3377817685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef80aedc0 msgr2=0x7f9ef80af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 --2- 172.21.15.50:0/3377817685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80af190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cfc50 tx=0x7f9eec009250 comp rx=0 tx=0).stop 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 -- 172.21.15.50:0/3377817685 shutdown_connections 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 --2- 172.21.15.50:0/3377817685 >> v2:172.21.15.125:3300/0 conn(0x7f9ef806af50 0x7f9ef80cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 --2- 172.21.15.50:0/3377817685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80af190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 -- 172.21.15.50:0/3377817685 >> 172.21.15.50:0/3377817685 conn(0x7f9ef80cc760 msgr2=0x7f9ef80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 -- 172.21.15.50:0/3377817685 shutdown_connections 2023-12-19T14:39:10.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.733+0000 7f9efde8e700 1 -- 172.21.15.50:0/3377817685 wait complete. 2023-12-19T14:39:10.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 Processor -- start 2023-12-19T14:39:10.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 -- start start 2023-12-19T14:39:10.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ef806af50 0x7f9ef8159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef80b3f70 con 0x7f9ef806d6b0 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef80b4250 con 0x7f9ef80aedc0 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efde8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef80b4530 con 0x7f9ef806af50 2023-12-19T14:39:10.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36072/0 (socket says 172.21.15.50:36072) 2023-12-19T14:39:10.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 -- 172.21.15.50:0/216511674 learned_addr learned my addr 172.21.15.50:0/216511674 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:10.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efce8c700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ef806af50 0x7f9ef8159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9efd68d700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 -- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ef806af50 msgr2=0x7f9ef8159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ef806af50 0x7f9ef8159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 -- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ef80aedc0 msgr2=0x7f9ef80b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 -- 172.21.15.50:0/216511674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec0094f0 con 0x7f9ef806d6b0 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.737+0000 7f9ef7fff700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400c850 tx=0x7f9ee4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4011600 con 0x7f9ef806d6b0 2023-12-19T14:39:10.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee4011760 con 0x7f9ef806d6b0 2023-12-19T14:39:10.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4011b00 con 0x7f9ef806d6b0 2023-12-19T14:39:10.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef80b4810 con 0x7f9ef806d6b0 2023-12-19T14:39:10.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef80b55b0 con 0x7f9ef806d6b0 2023-12-19T14:39:10.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.741+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef804be40 con 0x7f9ef806d6b0 2023-12-19T14:39:10.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.745+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9ee4018350 con 0x7f9ef806d6b0 2023-12-19T14:39:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.745+0000 7f9ef4ff9700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 0x7f9edc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.745+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f9ee407c980 con 0x7f9ef806d6b0 2023-12-19T14:39:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.745+0000 7f9efce8c700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 0x7f9edc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:10.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.745+0000 7f9efce8c700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 0x7f9edc062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cfc50 tx=0x7f9eec016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:10.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.753+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9ee404cc00 con 0x7f9ef806d6b0 2023-12-19T14:39:10.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:10.925+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f9ef80c9910 con 0x7f9ef806d6b0 2023-12-19T14:39:11.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.529+0000 7f9ef4ff9700 1 -- 172.21.15.50:0/216511674 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9ee4050220 con 0x7f9ef806d6b0 2023-12-19T14:39:11.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.541+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 msgr2=0x7f9edc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.541+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 0x7f9edc062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cfc50 tx=0x7f9eec016000 comp rx=0 tx=0).stop 2023-12-19T14:39:11.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.541+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 msgr2=0x7f9ef80b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.541+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400c850 tx=0x7f9ee4008040 comp rx=0 tx=0).stop 2023-12-19T14:39:11.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 shutdown_connections 2023-12-19T14:39:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ef80aedc0 0x7f9ef80b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9edc060550 0x7f9edc062a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806d6b0 0x7f9ef80b3660 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 --2- 172.21.15.50:0/216511674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ef806af50 0x7f9ef8159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 >> 172.21.15.50:0/216511674 conn(0x7f9ef80cc760 msgr2=0x7f9ef806e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:11.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.545+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 shutdown_connections 2023-12-19T14:39:11.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.549+0000 7f9efde8e700 1 -- 172.21.15.50:0/216511674 wait complete. 2023-12-19T14:39:11.565 INFO:teuthology.orchestra.run.smithi050.stderr:set pool 4 allow_ec_overwrites to true 2023-12-19T14:39:11.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-12-19T14:39:11.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 Processor -- start 2023-12-19T14:39:11.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 -- start start 2023-12-19T14:39:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f436006c7c0 0x7f436006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 0x7f436006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436006a7a0 con 0x7f43600c6d20 2023-12-19T14:39:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436006d4b0 con 0x7f436006c7c0 2023-12-19T14:39:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.745+0000 7f4364878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436006d790 con 0x7f436006ac30 2023-12-19T14:39:11.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435e59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f43600c6d20 0x7f43600c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36588/0 (socket says 172.21.15.50:36588) 2023-12-19T14:39:11.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435e59c700 1 -- 172.21.15.50:0/2967347361 learned_addr learned my addr 172.21.15.50:0/2967347361 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:11.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435dd9b700 1 --2- 172.21.15.50:0/2967347361 >> v2:172.21.15.125:3300/0 conn(0x7f436006c7c0 0x7f436006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 --2- 172.21.15.50:0/2967347361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 0x7f436006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435d59a700 1 -- 172.21.15.50:0/2967347361 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555633310 0 0) 0x7f436006a7a0 con 0x7f43600c6d20 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435d59a700 1 -- 172.21.15.50:0/2967347361 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348003570 con 0x7f43600c6d20 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 -- 172.21.15.50:0/2967347361 >> v2:172.21.15.125:3300/0 conn(0x7f436006c7c0 msgr2=0x7f436006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 --2- 172.21.15.50:0/2967347361 >> v2:172.21.15.125:3300/0 conn(0x7f436006c7c0 0x7f436006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 -- 172.21.15.50:0/2967347361 >> v1:172.21.15.50:6789/0 conn(0x7f43600c6d20 legacy=0x7f43600c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 -- 172.21.15.50:0/2967347361 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436006da70 con 0x7f436006ac30 2023-12-19T14:39:11.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435ed9d700 1 --2- 172.21.15.50:0/2967347361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 0x7f436006b050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4358009e60 tx=0x7f4358009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d1d2597b4e6387 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:11.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435d59a700 1 -- 172.21.15.50:0/2967347361 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800e070 con 0x7f436006ac30 2023-12-19T14:39:11.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435d59a700 1 -- 172.21.15.50:0/2967347361 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435800bab0 con 0x7f436006ac30 2023-12-19T14:39:11.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.749+0000 7f435d59a700 1 -- 172.21.15.50:0/2967347361 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800be70 con 0x7f436006ac30 2023-12-19T14:39:11.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 -- 172.21.15.50:0/2967347361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 msgr2=0x7f436006b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 --2- 172.21.15.50:0/2967347361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 0x7f436006b050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4358009e60 tx=0x7f4358009780 comp rx=0 tx=0).stop 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 -- 172.21.15.50:0/2967347361 shutdown_connections 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 --2- 172.21.15.50:0/2967347361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f436006ac30 0x7f436006b050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 --2- 172.21.15.50:0/2967347361 >> v2:172.21.15.125:3300/0 conn(0x7f436006c7c0 0x7f436006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 -- 172.21.15.50:0/2967347361 >> 172.21.15.50:0/2967347361 conn(0x7f4360067bb0 msgr2=0x7f43600cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 -- 172.21.15.50:0/2967347361 shutdown_connections 2023-12-19T14:39:11.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 -- 172.21.15.50:0/2967347361 wait complete. 2023-12-19T14:39:11.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.753+0000 7f4364878700 1 Processor -- start 2023-12-19T14:39:11.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 -- start start 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 0x7f4360152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43600c6d20 0x7f43600cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600ac380 con 0x7f436006c7c0 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600aa1b0 con 0x7f436006ac30 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43600c9c70 con 0x7f43600c6d20 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36086/0 (socket says 172.21.15.50:36086) 2023-12-19T14:39:11.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 -- 172.21.15.50:0/1086993111 learned_addr learned my addr 172.21.15.50:0/1086993111 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:11.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435e59c700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 0x7f4360152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435ed9d700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43600c6d20 0x7f43600cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 -- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43600c6d20 msgr2=0x7f43600cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43600c6d20 0x7f43600cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 -- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 msgr2=0x7f4360152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 0x7f4360152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 -- 172.21.15.50:0/1086993111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358009a20 con 0x7f436006c7c0 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435e59c700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 0x7f4360152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f435dd9b700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f435400ec80 tx=0x7f43540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435400c9d0 con 0x7f436006c7c0 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400cb30 con 0x7f436006c7c0 2023-12-19T14:39:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43600c9f50 con 0x7f436006c7c0 2023-12-19T14:39:11.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.757+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354013470 con 0x7f436006c7c0 2023-12-19T14:39:11.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.761+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43600ca510 con 0x7f436006c7c0 2023-12-19T14:39:11.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.761+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43600aa350 con 0x7f436006c7c0 2023-12-19T14:39:11.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.761+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4354013820 con 0x7f436006c7c0 2023-12-19T14:39:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.765+0000 7f434e7fc700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 0x7f4340062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.765+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f435407ce20 con 0x7f436006c7c0 2023-12-19T14:39:11.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.765+0000 7f435e59c700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 0x7f4340062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:11.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.765+0000 7f435e59c700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 0x7f4340062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4350001200 tx=0x7f4350009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:11.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.769+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f435404d0a0 con 0x7f436006c7c0 2023-12-19T14:39:11.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:11.977+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f4360071120 con 0x7f436006c7c0 2023-12-19T14:39:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.537+0000 7f434e7fc700 1 -- 172.21.15.50:0/1086993111 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7f43540506c0 con 0x7f436006c7c0 2023-12-19T14:39:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 msgr2=0x7f4340062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 0x7f4340062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4350001200 tx=0x7f4350009040 comp rx=0 tx=0).stop 2023-12-19T14:39:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 msgr2=0x7f43600abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f435400ec80 tx=0x7f43540091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 shutdown_connections 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4340060580 0x7f4340062a30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43600c6d20 0x7f43600cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f436006c7c0 0x7f43600abe40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 --2- 172.21.15.50:0/1086993111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f436006ac30 0x7f4360152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 >> 172.21.15.50:0/1086993111 conn(0x7f4360067bb0 msgr2=0x7f43600c2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 shutdown_connections 2023-12-19T14:39:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.541+0000 7f4364878700 1 -- 172.21.15.50:0/1086993111 wait complete. 2023-12-19T14:39:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:added data pool 4 to fsmap 2023-12-19T14:39:12.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 Processor -- start 2023-12-19T14:39:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 -- start start 2023-12-19T14:39:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8170158e70 0x7f81700afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f81700cc470 0x7f81700cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700ca5a0 con 0x7f817006a540 2023-12-19T14:39:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817006dac0 con 0x7f81700cc470 2023-12-19T14:39:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8176616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81700b0260 con 0x7f8170158e70 2023-12-19T14:39:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8175e15700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f81700cc470 0x7f81700cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8175614700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8170158e70 0x7f81700afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8174e13700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f817006a540 0x7f817006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52584/0 (socket says 172.21.15.50:52584) 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.729+0000 7f8174e13700 1 -- 172.21.15.50:0/2558636310 learned_addr learned my addr 172.21.15.50:0/2558636310 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8175e15700 1 -- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8170158e70 msgr2=0x7f81700afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8175e15700 1 --2- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8170158e70 0x7f81700afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8175e15700 1 -- 172.21.15.50:0/2558636310 >> v1:172.21.15.125:6789/0 conn(0x7f817006a540 legacy=0x7f817006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8175e15700 1 -- 172.21.15.50:0/2558636310 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817006d220 con 0x7f81700cc470 2023-12-19T14:39:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8175e15700 1 --2- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81700cc470 0x7f81700cc8b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e60 tx=0x7f8168009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f349c973956e2193 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8167fff700 1 -- 172.21.15.50:0/2558636310 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800e070 con 0x7f81700cc470 2023-12-19T14:39:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8167fff700 1 -- 172.21.15.50:0/2558636310 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816800bb60 con 0x7f81700cc470 2023-12-19T14:39:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8167fff700 1 -- 172.21.15.50:0/2558636310 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168012430 con 0x7f81700cc470 2023-12-19T14:39:12.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 -- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81700cc470 msgr2=0x7f81700cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 --2- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81700cc470 0x7f81700cc8b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8168009e60 tx=0x7f8168009780 comp rx=0 tx=0).stop 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 -- 172.21.15.50:0/2558636310 shutdown_connections 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 --2- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81700cc470 0x7f81700cc8b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 --2- 172.21.15.50:0/2558636310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8170158e70 0x7f81700afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 -- 172.21.15.50:0/2558636310 >> 172.21.15.50:0/2558636310 conn(0x7f81700670e0 msgr2=0x7f81700674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 -- 172.21.15.50:0/2558636310 shutdown_connections 2023-12-19T14:39:12.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.733+0000 7f8176616700 1 -- 172.21.15.50:0/2558636310 wait complete. 2023-12-19T14:39:12.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.737+0000 7f8176616700 1 Processor -- start 2023-12-19T14:39:12.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.737+0000 7f8176616700 1 -- start start 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f81700cc470 0x7f81700b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 0x7f8170156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c96b0 con 0x7f817006a540 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700b3b60 con 0x7f8170158e70 2023-12-19T14:39:12.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8170159ec0 con 0x7f81700cc470 2023-12-19T14:39:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175614700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175614700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36094/0 (socket says 172.21.15.50:36094) 2023-12-19T14:39:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175614700 1 -- 172.21.15.50:0/136109847 learned_addr learned my addr 172.21.15.50:0/136109847 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 0x7f8170156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8174e13700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f81700cc470 0x7f81700b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 -- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f81700cc470 msgr2=0x7f81700b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f81700cc470 0x7f81700b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 -- 172.21.15.50:0/136109847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 msgr2=0x7f81700c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 -- 172.21.15.50:0/136109847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009a20 con 0x7f8170158e70 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175614700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:12.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8175e15700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 0x7f8170156210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8168016000 tx=0x7f8168019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:12.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168012e60 con 0x7f8170158e70 2023-12-19T14:39:12.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8168027470 con 0x7f8170158e70 2023-12-19T14:39:12.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81700b4620 con 0x7f8170158e70 2023-12-19T14:39:12.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.741+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168027810 con 0x7f8170158e70 2023-12-19T14:39:12.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.745+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81700b4060 con 0x7f8170158e70 2023-12-19T14:39:12.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.745+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817015a330 con 0x7f8170158e70 2023-12-19T14:39:12.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.745+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8168024070 con 0x7f8170158e70 2023-12-19T14:39:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.745+0000 7f81657fa700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 0x7f8154062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:12.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.745+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f816807fb90 con 0x7f8170158e70 2023-12-19T14:39:12.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.749+0000 7f8175614700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 0x7f8154062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.749+0000 7f8175614700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 0x7f8154062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f816c009fd0 tx=0x7f816c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.753+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f816804fdf0 con 0x7f8170158e70 2023-12-19T14:39:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.969+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f817015a330 con 0x7f8170158e70 2023-12-19T14:39:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.973+0000 7f81657fa700 1 -- 172.21.15.50:0/136109847 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f817015a330 con 0x7f8170158e70 2023-12-19T14:39:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:12.979 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:12.534167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T14:39:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 msgr2=0x7f8154062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 0x7f8154062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f816c009fd0 tx=0x7f816c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 msgr2=0x7f8170156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:12.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 0x7f8170156210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8168016000 tx=0x7f8168019000 comp rx=0 tx=0).stop 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 shutdown_connections 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8170158e70 0x7f8170156210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8154060550 0x7f8154062a00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f81700cc470 0x7f81700b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 --2- 172.21.15.50:0/136109847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f817006a540 0x7f81700c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 >> 172.21.15.50:0/136109847 conn(0x7f81700670e0 msgr2=0x7f81700cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 shutdown_connections 2023-12-19T14:39:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:12.977+0000 7f8176616700 1 -- 172.21.15.50:0/136109847 wait complete. 2023-12-19T14:39:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 6 2023-12-19T14:39:13.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:13.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.153+0000 7f386542e700 1 Processor -- start 2023-12-19T14:39:13.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.153+0000 7f386542e700 1 -- start start 2023-12-19T14:39:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f386542e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38601502b0 0x7f3860068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f386542e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38600c5900 0x7f38600c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f386542e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600c6630 con 0x7f3860067c90 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f386542e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38600c6910 con 0x7f38601502b0 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f386542e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38600c6bf0 con 0x7f38600c5900 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38601502b0 0x7f3860068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38601502b0 0x7f3860068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36110/0 (socket says 172.21.15.50:36110) 2023-12-19T14:39:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 -- 172.21.15.50:0/2074382628 learned_addr learned my addr 172.21.15.50:0/2074382628 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:13.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385f7fe700 1 --2- 172.21.15.50:0/2074382628 >> v2:172.21.15.125:3300/0 conn(0x7f38600c5900 0x7f38600c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 -- 172.21.15.50:0/2074382628 >> v2:172.21.15.125:3300/0 conn(0x7f38600c5900 msgr2=0x7f38600c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 --2- 172.21.15.50:0/2074382628 >> v2:172.21.15.125:3300/0 conn(0x7f38600c5900 0x7f38600c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 -- 172.21.15.50:0/2074382628 >> v1:172.21.15.125:6789/0 conn(0x7f3860067c90 legacy=0x7f386014fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 -- 172.21.15.50:0/2074382628 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38600c6ed0 con 0x7f38601502b0 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385e7fc700 1 --2- 172.21.15.50:0/2074382628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f3860068060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3860068a60 tx=0x7f3854009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ba2b1155dccdb6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385dffb700 1 -- 172.21.15.50:0/2074382628 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400b860 con 0x7f38601502b0 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385dffb700 1 -- 172.21.15.50:0/2074382628 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385400b9c0 con 0x7f38601502b0 2023-12-19T14:39:13.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.157+0000 7f385dffb700 1 -- 172.21.15.50:0/2074382628 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400bd60 con 0x7f38601502b0 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- 172.21.15.50:0/2074382628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 msgr2=0x7f3860068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 --2- 172.21.15.50:0/2074382628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f3860068060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3860068a60 tx=0x7f3854009250 comp rx=0 tx=0).stop 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- 172.21.15.50:0/2074382628 shutdown_connections 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 --2- 172.21.15.50:0/2074382628 >> v2:172.21.15.125:3300/0 conn(0x7f38600c5900 0x7f38600c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 --2- 172.21.15.50:0/2074382628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f3860068060 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- 172.21.15.50:0/2074382628 >> 172.21.15.50:0/2074382628 conn(0x7f38600cd6d0 msgr2=0x7f386006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- 172.21.15.50:0/2074382628 shutdown_connections 2023-12-19T14:39:13.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- 172.21.15.50:0/2074382628 wait complete. 2023-12-19T14:39:13.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 Processor -- start 2023-12-19T14:39:13.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.161+0000 7f386542e700 1 -- start start 2023-12-19T14:39:13.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3860067c90 0x7f38600c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 0x7f386016c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b5020 con 0x7f38601502b0 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b4410 con 0x7f3860067c90 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b46f0 con 0x7f38600c5900 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36124/0 (socket says 172.21.15.50:36124) 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 -- 172.21.15.50:0/1399386085 learned_addr learned my addr 172.21.15.50:0/1399386085 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:13.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385effd700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3860067c90 0x7f38600c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385e7fc700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 0x7f386016c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 -- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 msgr2=0x7f386016c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 0x7f386016c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 -- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3860067c90 msgr2=0x7f38600c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3860067c90 0x7f38600c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 -- 172.21.15.50:0/1399386085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540094f0 con 0x7f38601502b0 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385e7fc700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 0x7f386016c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f385f7fe700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3858009fa0 tx=0x7f3858009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385800ed20 con 0x7f38601502b0 2023-12-19T14:39:13.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385800ee80 con 0x7f38601502b0 2023-12-19T14:39:13.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3858014670 con 0x7f38601502b0 2023-12-19T14:39:13.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38600b49d0 con 0x7f38601502b0 2023-12-19T14:39:13.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38600b40c0 con 0x7f38601502b0 2023-12-19T14:39:13.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.165+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38600b5190 con 0x7f38601502b0 2023-12-19T14:39:13.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.169+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f38580149e0 con 0x7f38601502b0 2023-12-19T14:39:13.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.169+0000 7f3846ffd700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 0x7f384c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.169+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f385807d290 con 0x7f38601502b0 2023-12-19T14:39:13.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.169+0000 7f385effd700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 0x7f384c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.169+0000 7f385effd700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 0x7f384c062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3850003860 tx=0x7f3850009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.173+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f385804d4f0 con 0x7f38601502b0 2023-12-19T14:39:13.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.345+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3860071120 con 0x7f38601502b0 2023-12-19T14:39:13.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.345+0000 7f3846ffd700 1 -- 172.21.15.50:0/1399386085 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f3858050b10 con 0x7f38601502b0 2023-12-19T14:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:13.353 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 msgr2=0x7f384c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 0x7f384c062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3850003860 tx=0x7f3850009040 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 msgr2=0x7f38600c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3858009fa0 tx=0x7f3858009900 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 shutdown_connections 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f384c060580 0x7f384c062a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38601502b0 0x7f38600c72a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38600c5900 0x7f386016c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 --2- 172.21.15.50:0/1399386085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3860067c90 0x7f38600c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 >> 172.21.15.50:0/1399386085 conn(0x7f38600cd6d0 msgr2=0x7f38600ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 shutdown_connections 2023-12-19T14:39:13.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.349+0000 7f386542e700 1 -- 172.21.15.50:0/1399386085 wait complete. 2023-12-19T14:39:13.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:13.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.533+0000 7fd1263c5700 1 Processor -- start 2023-12-19T14:39:13.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 -- start start 2023-12-19T14:39:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1201502b0 0x7fd120068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1200c6630 con 0x7fd120067c90 2023-12-19T14:39:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1200c6910 con 0x7fd1201502b0 2023-12-19T14:39:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd1263c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200c6bf0 con 0x7fd1200c5900 2023-12-19T14:39:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1201502b0 0x7fd120068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd1201502b0 0x7fd120068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36136/0 (socket says 172.21.15.50:36136) 2023-12-19T14:39:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 -- 172.21.15.50:0/757867294 learned_addr learned my addr 172.21.15.50:0/757867294 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd124962700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 -- 172.21.15.50:0/757867294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 msgr2=0x7fd1200c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 -- 172.21.15.50:0/757867294 >> v1:172.21.15.50:6789/0 conn(0x7fd120067c90 legacy=0x7fd12014fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 -- 172.21.15.50:0/757867294 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1200c6ed0 con 0x7fd1201502b0 2023-12-19T14:39:13.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.537+0000 7fd117fff700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd120068060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd1080099b0 tx=0x7fd108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65a1888709c4ce8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.541+0000 7fd11f7fe700 1 -- 172.21.15.50:0/757867294 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800b860 con 0x7fd1201502b0 2023-12-19T14:39:13.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.541+0000 7fd11f7fe700 1 -- 172.21.15.50:0/757867294 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10800b9c0 con 0x7fd1201502b0 2023-12-19T14:39:13.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.541+0000 7fd11f7fe700 1 -- 172.21.15.50:0/757867294 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800bd60 con 0x7fd1201502b0 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 -- 172.21.15.50:0/757867294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 msgr2=0x7fd120068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd120068060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd1080099b0 tx=0x7fd108009250 comp rx=0 tx=0).stop 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 -- 172.21.15.50:0/757867294 shutdown_connections 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 --2- 172.21.15.50:0/757867294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd120068060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 -- 172.21.15.50:0/757867294 >> 172.21.15.50:0/757867294 conn(0x7fd1200cd6d0 msgr2=0x7fd12006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 -- 172.21.15.50:0/757867294 shutdown_connections 2023-12-19T14:39:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 -- 172.21.15.50:0/757867294 wait complete. 2023-12-19T14:39:13.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.545+0000 7fd1263c5700 1 Processor -- start 2023-12-19T14:39:13.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 -- start start 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd120067c90 0x7fd1200b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200b5920 con 0x7fd1201502b0 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200b5490 con 0x7fd120067c90 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200b4880 con 0x7fd1200c5900 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36138/0 (socket says 172.21.15.50:36138) 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 -- 172.21.15.50:0/232664071 learned_addr learned my addr 172.21.15.50:0/232664071 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd117fff700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd11ffff700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd120067c90 0x7fd1200b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 -- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 msgr2=0x7fd1200c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 -- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd120067c90 msgr2=0x7fd1200b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd120067c90 0x7fd1200b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 -- 172.21.15.50:0/232664071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1080094f0 con 0x7fd1201502b0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd124962700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd118009fa0 tx=0x7fd118009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11800ebc0 con 0x7fd1201502b0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11800ed20 con 0x7fd1201502b0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118014670 con 0x7fd1201502b0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.549+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1200b4b60 con 0x7fd1201502b0 2023-12-19T14:39:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.553+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1200b5120 con 0x7fd1201502b0 2023-12-19T14:39:13.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.553+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1200b5620 con 0x7fd1201502b0 2023-12-19T14:39:13.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.553+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd1180149e0 con 0x7fd1201502b0 2023-12-19T14:39:13.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.553+0000 7fd11cff9700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 0x7fd100062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.553+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd11807d020 con 0x7fd1201502b0 2023-12-19T14:39:13.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.557+0000 7fd11ffff700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 0x7fd100062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.557+0000 7fd11ffff700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 0x7fd100062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd110003a80 tx=0x7fd110007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.561+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd11804d320 con 0x7fd1201502b0 2023-12-19T14:39:13.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.721+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd120071120 con 0x7fd1201502b0 2023-12-19T14:39:13.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.725+0000 7fd11cff9700 1 -- 172.21.15.50:0/232664071 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fd118050940 con 0x7fd1201502b0 2023-12-19T14:39:13.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:13.731 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 msgr2=0x7fd100062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 0x7fd100062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd110003a80 tx=0x7fd110007ec0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 msgr2=0x7fd12016c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd118009fa0 tx=0x7fd118009900 comp rx=0 tx=0).stop 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 shutdown_connections 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd100060550 0x7fd100062a00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd1201502b0 0x7fd12016c790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd1200c5900 0x7fd1200c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 --2- 172.21.15.50:0/232664071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd120067c90 0x7fd1200b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 >> 172.21.15.50:0/232664071 conn(0x7fd1200cd6d0 msgr2=0x7fd1200cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 shutdown_connections 2023-12-19T14:39:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.729+0000 7fd1263c5700 1 -- 172.21.15.50:0/232664071 wait complete. 2023-12-19T14:39:13.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.925+0000 7f394e616700 1 Processor -- start 2023-12-19T14:39:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.925+0000 7f394e616700 1 -- start start 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394e616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f39480c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394e616700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f39480c51b0 0x7f394806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394e616700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480c5610 con 0x7f3948158d60 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394e616700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394806c7c0 con 0x7f39480c51b0 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394e616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394806caa0 con 0x7f394806b7a0 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394d614700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3948158d60 0x7f394815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36600/0 (socket says 172.21.15.50:36600) 2023-12-19T14:39:13.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394d614700 1 -- 172.21.15.50:0/1007504861 learned_addr learned my addr 172.21.15.50:0/1007504861 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217492034 0 0) 0x7f3944001490 con 0x7f3948158d60 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3934003570 con 0x7f3948158d60 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394ce13700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f39480c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f394de15700 1 --2- 172.21.15.50:0/1007504861 >> v2:172.21.15.125:3300/0 conn(0x7f39480c51b0 0x7f394806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1799558849 0 0) 0x7f3934003570 con 0x7f3948158d60 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 msgr2=0x7f39480c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f39480c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 >> v2:172.21.15.125:3300/0 conn(0x7f39480c51b0 msgr2=0x7f394806cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 --2- 172.21.15.50:0/1007504861 >> v2:172.21.15.125:3300/0 conn(0x7f39480c51b0 0x7f394806cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394815d130 con 0x7f3948158d60 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f3944001980 con 0x7f3948158d60 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.929+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 >> v1:172.21.15.50:6789/0 conn(0x7f3948158d60 legacy=0x7f394815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f3934003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f3934004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934004d50 0x7f39340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394806c7c0 con 0x7f3934004d50 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394806caa0 con 0x7f3934003a60 2023-12-19T14:39:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393400bc70 con 0x7f394806b7a0 2023-12-19T14:39:13.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f3934003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394ce13700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f3934004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394de15700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934004d50 0x7f39340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 -- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 msgr2=0x7f3934004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394806b7a0 0x7f3934004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 -- 172.21.15.50:0/1007504861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934004d50 msgr2=0x7f39340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934004d50 0x7f39340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 -- 172.21.15.50:0/1007504861 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934003570 con 0x7f3934003a60 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394de15700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3934004d50 0x7f39340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394d614700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f3934003e70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f394400ab00 tx=0x7f394400a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bcfa3f34aed6a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394400a490 con 0x7f3934003a60 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394400c5d0 con 0x7f3934003a60 2023-12-19T14:39:13.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f393ffff700 1 -- 172.21.15.50:0/1007504861 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394400c9d0 con 0x7f3934003a60 2023-12-19T14:39:13.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394e616700 1 -- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 msgr2=0x7f3934003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394e616700 1 --2- 172.21.15.50:0/1007504861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f3934003e70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f394400ab00 tx=0x7f394400a1a0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394de15700 1 -- 172.21.15.50:0/1007504861 reap_dead start 2023-12-19T14:39:13.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394e616700 1 -- 172.21.15.50:0/1007504861 shutdown_connections 2023-12-19T14:39:13.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.933+0000 7f394e616700 1 -- 172.21.15.50:0/1007504861 >> 172.21.15.50:0/1007504861 conn(0x7f3948067bb0 msgr2=0x7f39480cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:13.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- 172.21.15.50:0/1007504861 shutdown_connections 2023-12-19T14:39:13.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- 172.21.15.50:0/1007504861 wait complete. 2023-12-19T14:39:13.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 Processor -- start 2023-12-19T14:39:13.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- start start 2023-12-19T14:39:13.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f394815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 0x7f3948169330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394816a560 con 0x7f3948169870 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394816a220 con 0x7f3934003a60 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394e616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394815bac0 con 0x7f394815b560 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394de15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394de15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36148/0 (socket says 172.21.15.50:36148) 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394de15700 1 -- 172.21.15.50:0/4028678985 learned_addr learned my addr 172.21.15.50:0/4028678985 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.937+0000 7f394d614700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f394815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394ce13700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 0x7f3948169330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 -- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 msgr2=0x7f3948169330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 0x7f3948169330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 -- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 msgr2=0x7f394815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f394815b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 -- 172.21.15.50:0/4028678985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394815bda0 con 0x7f3948169870 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394ce13700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 0x7f3948169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394de15700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f394806bcf0 tx=0x7f3940008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940014460 con 0x7f3948169870 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39400145c0 con 0x7f3948169870 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394816a7b0 con 0x7f3948169870 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940014980 con 0x7f3948169870 2023-12-19T14:39:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39481581f0 con 0x7f3948169870 2023-12-19T14:39:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.941+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394816acb0 con 0x7f3948169870 2023-12-19T14:39:13.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.945+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3940014cf0 con 0x7f3948169870 2023-12-19T14:39:13.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.945+0000 7f393d7fa700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 0x7f392c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:13.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.945+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f394007cce0 con 0x7f3948169870 2023-12-19T14:39:13.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.945+0000 7f394d614700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 0x7f392c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:13.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.945+0000 7f394d614700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 0x7f392c062a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f394400afd0 tx=0x7f3944016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:13.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:13.949+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f394004cf40 con 0x7f3948169870 2023-12-19T14:39:14.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.157+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f394804be40 con 0x7f3948169870 2023-12-19T14:39:14.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.157+0000 7f393d7fa700 1 -- 172.21.15.50:0/4028678985 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f3940000e60 con 0x7f3948169870 2023-12-19T14:39:14.165 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:14.166 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:12.534167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T14:39:14.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 msgr2=0x7f392c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 0x7f392c062a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f394400afd0 tx=0x7f3944016000 comp rx=0 tx=0).stop 2023-12-19T14:39:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 msgr2=0x7f3948169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f394806bcf0 tx=0x7f3940008040 comp rx=0 tx=0).stop 2023-12-19T14:39:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 shutdown_connections 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f392c060580 0x7f392c062a30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3948169870 0x7f3948169ce0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f394815b560 0x7f3948169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 --2- 172.21.15.50:0/4028678985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3934003a60 0x7f394815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 >> 172.21.15.50:0/4028678985 conn(0x7f3948067bb0 msgr2=0x7f394806a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 shutdown_connections 2023-12-19T14:39:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.161+0000 7f394e616700 1 -- 172.21.15.50:0/4028678985 wait complete. 2023-12-19T14:39:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 6 2023-12-19T14:39:14.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.353+0000 7fc8ec2f0700 1 Processor -- start 2023-12-19T14:39:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.353+0000 7fc8ec2f0700 1 -- start start 2023-12-19T14:39:14.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8ec2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8e40abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8ec2f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e40cbf50 con 0x7fc8e40aba90 2023-12-19T14:39:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8ec2f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e40c8730 con 0x7fc8e40cbb20 2023-12-19T14:39:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8ec2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40c7e70 con 0x7fc8e40cb750 2023-12-19T14:39:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e988b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc8e40aba90 0x7fc8e40c1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36604/0 (socket says 172.21.15.50:36604) 2023-12-19T14:39:14.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e988b700 1 -- 172.21.15.50:0/659305747 learned_addr learned my addr 172.21.15.50:0/659305747 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8ea08c700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8e40abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956674549 0 0) 0x7fc8d4001110 con 0x7fc8e40aba90 2023-12-19T14:39:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d0003570 con 0x7fc8e40aba90 2023-12-19T14:39:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353768448 0 0) 0x7fc8e40c8730 con 0x7fc8e40cbb20 2023-12-19T14:39:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d4001110 con 0x7fc8e40cbb20 2023-12-19T14:39:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103662320 0 0) 0x7fc8d0003570 con 0x7fc8e40aba90 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 msgr2=0x7fc8e40abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8e40abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 >> v1:172.21.15.125:6789/0 conn(0x7fc8e40cbb20 legacy=0x7fc8e40c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40c23a0 con 0x7fc8e40aba90 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.357+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fc8d4001600 con 0x7fc8e40aba90 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 >> v1:172.21.15.50:6789/0 conn(0x7fc8e40aba90 legacy=0x7fc8e40c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8d0003c20 0x7fc8d0003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40c7e70 con 0x7fc8d0004d50 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000bb70 con 0x7fc8d0003c20 2023-12-19T14:39:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000bd30 con 0x7fc8e40cb750 2023-12-19T14:39:14.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea08c700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8d0003c20 0x7fc8d0003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e988b700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 -- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 msgr2=0x7fc8d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 -- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8d0003c20 msgr2=0x7fc8d0003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8d0003c20 0x7fc8d0003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 -- 172.21.15.50:0/659305747 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d0003570 con 0x7fc8d0004d50 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e988b700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40cb750 0x7fc8d0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8d00051c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc001a10 tx=0x7fc8dc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fd3cf61f693a07 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00c5d0 con 0x7fc8d0004d50 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc00c730 con 0x7fc8d0004d50 2023-12-19T14:39:14.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8e908a700 1 -- 172.21.15.50:0/659305747 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00cad0 con 0x7fc8d0004d50 2023-12-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/659305747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 msgr2=0x7fc8d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/659305747 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8d00051c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc001a10 tx=0x7fc8dc007d30 comp rx=0 tx=0).stop 2023-12-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.361+0000 7fc8ea88d700 1 -- 172.21.15.50:0/659305747 reap_dead start 2023-12-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.365+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/659305747 shutdown_connections 2023-12-19T14:39:14.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.365+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/659305747 >> 172.21.15.50:0/659305747 conn(0x7fc8e40cd9c0 msgr2=0x7fc8e40c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.365+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/659305747 shutdown_connections 2023-12-19T14:39:14.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.365+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/659305747 wait complete. 2023-12-19T14:39:14.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 Processor -- start 2023-12-19T14:39:14.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 -- start start 2023-12-19T14:39:14.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e40aac60 0x7fc8e40a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40a9e00 0x7fc8e41563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40b15d0 con 0x7fc8d0004d50 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40b1870 con 0x7fc8e40aac60 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e40b1b50 con 0x7fc8e40a9e00 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36160/0 (socket says 172.21.15.50:36160) 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 -- 172.21.15.50:0/2261068846 learned_addr learned my addr 172.21.15.50:0/2261068846 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8e988b700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e40aac60 0x7fc8e40a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea88d700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40a9e00 0x7fc8e41563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 -- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40a9e00 msgr2=0x7fc8e41563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40a9e00 0x7fc8e41563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 -- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e40aac60 msgr2=0x7fc8e40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e40aac60 0x7fc8e40a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 -- 172.21.15.50:0/2261068846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40b1e30 con 0x7fc8d0004d50 2023-12-19T14:39:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ea08c700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0011f00 tx=0x7fc8e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e000e040 con 0x7fc8d0004d50 2023-12-19T14:39:14.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e000f040 con 0x7fc8d0004d50 2023-12-19T14:39:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0015430 con 0x7fc8d0004d50 2023-12-19T14:39:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.369+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e40b2110 con 0x7fc8d0004d50 2023-12-19T14:39:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.373+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e40a8af0 con 0x7fc8d0004d50 2023-12-19T14:39:14.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.373+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e4071120 con 0x7fc8d0004d50 2023-12-19T14:39:14.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.373+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc8e0015860 con 0x7fc8d0004d50 2023-12-19T14:39:14.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.373+0000 7fc8da7fc700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 0x7fc8c8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.377+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc8e007d850 con 0x7fc8d0004d50 2023-12-19T14:39:14.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.377+0000 7fc8e988b700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 0x7fc8c8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:14.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.377+0000 7fc8e988b700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 0x7fc8c8062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000e50 tx=0x7fc8d400a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:14.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.381+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc8e004dab0 con 0x7fc8d0004d50 2023-12-19T14:39:14.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.553+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8e4158d30 con 0x7fc8d0004d50 2023-12-19T14:39:14.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.557+0000 7fc8da7fc700 1 -- 172.21.15.50:0/2261068846 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fc8e00510d0 con 0x7fc8d0004d50 2023-12-19T14:39:14.563 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:14.563 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 msgr2=0x7fc8c8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 0x7fc8c8062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4000e50 tx=0x7fc8d400a040 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 msgr2=0x7fc8e40b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0011f00 tx=0x7fc8e000b040 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 shutdown_connections 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8e40aac60 0x7fc8e40a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc8c8060550 0x7fc8c8062a00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc8e40a9e00 0x7fc8e41563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 --2- 172.21.15.50:0/2261068846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc8d0004d50 0x7fc8e40b2630 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 >> 172.21.15.50:0/2261068846 conn(0x7fc8e40cd9c0 msgr2=0x7fc8e40c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 shutdown_connections 2023-12-19T14:39:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:14.561+0000 7fc8ec2f0700 1 -- 172.21.15.50:0/2261068846 wait complete. 2023-12-19T14:39:14.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:39:06.675595+0000', 'modified': '2023-12-19T14:39:12.534167+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/3977471945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6833', 'nonce': 3977471945}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 3977471945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-12-19T14:39:14.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:39:14.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T14:39:15.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.157+0000 7fc300478700 1 Processor -- start 2023-12-19T14:39:15.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.157+0000 7fc300478700 1 -- start start 2023-12-19T14:39:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc300478700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2f8158e50 0x7fc2f806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc300478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8159220 0x7fc2f806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc300478700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f806e160 con 0x7fc2f80a98d0 2023-12-19T14:39:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc300478700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f806d0c0 con 0x7fc2f8158e50 2023-12-19T14:39:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc300478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f806d3a0 con 0x7fc2f8159220 2023-12-19T14:39:15.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc2fe214700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2f8158e50 0x7fc2f806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc2fe214700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2f8158e50 0x7fc2f806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36166/0 (socket says 172.21.15.50:36166) 2023-12-19T14:39:15.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc2fe214700 1 -- 172.21.15.50:0/1591272990 learned_addr learned my addr 172.21.15.50:0/1591272990 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.161+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8159220 0x7fc2f806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fe214700 1 -- 172.21.15.50:0/1591272990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8159220 msgr2=0x7fc2f806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8159220 0x7fc2f806a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fe214700 1 -- 172.21.15.50:0/1591272990 >> v1:172.21.15.125:6789/0 conn(0x7fc2f80a98d0 legacy=0x7fc2f80a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fe214700 1 -- 172.21.15.50:0/1591272990 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f806d680 con 0x7fc2f8158e50 2023-12-19T14:39:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8158e50 0x7fc2f806c7c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40099b0 tx=0x7fc2f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cb102715f9fba6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fd212700 1 -- 172.21.15.50:0/1591272990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4009ea0 con 0x7fc2f8158e50 2023-12-19T14:39:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fd212700 1 -- 172.21.15.50:0/1591272990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f400b8f0 con 0x7fc2f8158e50 2023-12-19T14:39:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc2fd212700 1 -- 172.21.15.50:0/1591272990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400bc90 con 0x7fc2f8158e50 2023-12-19T14:39:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 -- 172.21.15.50:0/1591272990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8158e50 msgr2=0x7fc2f806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8158e50 0x7fc2f806c7c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40099b0 tx=0x7fc2f4009250 comp rx=0 tx=0).stop 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 -- 172.21.15.50:0/1591272990 shutdown_connections 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8159220 0x7fc2f806a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 --2- 172.21.15.50:0/1591272990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8158e50 0x7fc2f806c7c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 -- 172.21.15.50:0/1591272990 >> 172.21.15.50:0/1591272990 conn(0x7fc2f8067b60 msgr2=0x7fc2f80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 -- 172.21.15.50:0/1591272990 shutdown_connections 2023-12-19T14:39:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.165+0000 7fc300478700 1 -- 172.21.15.50:0/1591272990 wait complete. 2023-12-19T14:39:15.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 Processor -- start 2023-12-19T14:39:15.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 -- start start 2023-12-19T14:39:15.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 0x7fc2f8141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 0x7fc2f81692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f816a2e0 con 0x7fc2f8159220 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8156690 con 0x7fc2f80a98d0 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc300478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8169d20 con 0x7fc2f8158e50 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc2fea15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc2fea15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36174/0 (socket says 172.21.15.50:36174) 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc2fea15700 1 -- 172.21.15.50:0/91027371 learned_addr learned my addr 172.21.15.50:0/91027371 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc2fe214700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 0x7fc2f8141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.169+0000 7fc2fda13700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 0x7fc2f81692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 -- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 msgr2=0x7fc2f81692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 0x7fc2f81692d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 -- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 msgr2=0x7fc2f8141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 0x7fc2f8141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 -- 172.21.15.50:0/91027371 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f40094f0 con 0x7fc2f8159220 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fe214700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 0x7fc2f8141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fda13700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 0x7fc2f81692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2fea15700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c8e0 tx=0x7fc2f0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2ee7fc700 1 -- 172.21.15.50:0/91027371 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000ce80 con 0x7fc2f8159220 2023-12-19T14:39:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2ee7fc700 1 -- 172.21.15.50:0/91027371 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f000a7e0 con 0x7fc2f8159220 2023-12-19T14:39:15.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f816a000 con 0x7fc2f8159220 2023-12-19T14:39:15.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc2ee7fc700 1 -- 172.21.15.50:0/91027371 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000ab80 con 0x7fc2f8159220 2023-12-19T14:39:15.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.173+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f816a4a0 con 0x7fc2f8159220 2023-12-19T14:39:15.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.177+0000 7fc2ee7fc700 1 -- 172.21.15.50:0/91027371 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc2f0012420 con 0x7fc2f8159220 2023-12-19T14:39:15.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.177+0000 7fc2ee7fc700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 0x7fc2dc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.177+0000 7fc2ee7fc700 1 -- 172.21.15.50:0/91027371 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc2f007ce70 con 0x7fc2f8159220 2023-12-19T14:39:15.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc2fe214700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 0x7fc2dc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc2fe214700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 0x7fc2dc062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40099b0 tx=0x7fc2f4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 Processor -- start 2023-12-19T14:39:15.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 -- start start 2023-12-19T14:39:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1cd3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c00f30 con 0x1aa0790 2023-12-19T14:39:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bfced0 con 0x1a55710 2023-12-19T14:39:15.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bc30b0 con 0x1a55ae0 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.181+0000 7fc2fea15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36190/0 (socket says 172.21.15.50:36190) 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fe214700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 -- 172.21.15.50:0/3380744128 learned_addr learned my addr 172.21.15.50:0/3380744128 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fda13700 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1cd3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 -- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 msgr2=0x1cd2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 -- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 msgr2=0x1cd3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1cd3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 -- 172.21.15.50:0/3380744128 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1aa19e0 con 0x1aa0790 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fda13700 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1cd3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2fea15700 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2f007afd0 tx=0x7fc2f007af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be87eefb229897d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2db7fe700 1 -- 172.21.15.50:0/3380744128 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0050d90 con 0x1aa0790 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2db7fe700 1 -- 172.21.15.50:0/3380744128 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f004eed0 con 0x1aa0790 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc2db7fe700 1 -- 172.21.15.50:0/3380744128 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0066100 con 0x1aa0790 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 -- 172.21.15.50:0/3380744128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 msgr2=0x1cd3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2f007afd0 tx=0x7fc2f007af60 comp rx=0 tx=0).stop 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 -- 172.21.15.50:0/3380744128 shutdown_connections 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1cd3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1aa0790 0x1cd3980 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 --2- 172.21.15.50:0/3380744128 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 -- 172.21.15.50:0/3380744128 >> 172.21.15.50:0/3380744128 conn(0x1a55eb0 msgr2=0x1c8f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 -- 172.21.15.50:0/3380744128 shutdown_connections 2023-12-19T14:39:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.185+0000 7fc30276d740 1 -- 172.21.15.50:0/3380744128 wait complete. 2023-12-19T14:39:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 -- start start 2023-12-19T14:39:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 Processor -- start 2023-12-19T14:39:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1aa0790 0x1cb8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bc2d50 con 0x1a55710 2023-12-19T14:39:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b9b970 con 0x1aa0790 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bfc6e0 con 0x1a55ae0 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc30276d740 10 client.0 _ll_register_callbacks cb 0x1c466a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc2fe214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc2fe214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36206/0 (socket says 172.21.15.50:36206) 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc2fe214700 1 -- 172.21.15.50:0/1781532440 learned_addr learned my addr 172.21.15.50:0/1781532440 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc2fea15700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1aa0790 0x1cb8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.189+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 -- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 msgr2=0x1cd7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 -- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1aa0790 msgr2=0x1cb8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1aa0790 0x1cb8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 -- 172.21.15.50:0/1781532440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f816a4a0 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43d86e415d37692 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f402f070 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1b9bcc0 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc2f4026430 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2f4020070 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 10 client.4410 fetch_fsmap learned FSMap version 6 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x1bcb370 con 0x1a55710 2023-12-19T14:39:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1b9bcc0 con 0x1a55710 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7fc2f4026800 con 0x1a55710 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 10 client.4410 fetch_fsmap finished waiting for FSMap version 6 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 10 client.4410 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 20 client.4410 populate_metadata read hostname 'smithi050' 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1cba370 0x1ce26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 4 client.4410 mds_command: new command op to 4275 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:39:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 --> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1b9f3e0 con 0x1cba370 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.193+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1cba370 0x1ce26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1cba370 0x1ce26b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7f36d86fd6d2c06 server_cookie=2b416fba7d176b14 in_seq=0 out_seq=0 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc2d8ff9700 10 client.4410 ms_handle_connect on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/1781532440 <== mds.0 v2:172.21.15.50:6833/3977471945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x1b9f3e0 con 0x1cba370 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc2d8ff9700 10 client.4410 handle_command_reply: tid=0 2023-12-19T14:39:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 client.4410 shutdown 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 msgr2=0x1cd7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4020000 comp rx=0 tx=0).stop 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 shutdown_connections 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1cba370 0x1ce26b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1aa0790 0x1cb8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1a55ae0 0x1cd7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 --2- 172.21.15.50:0/1781532440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55710 0x1cd7510 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc2d8ff9700 0 client.4410 ms_handle_reset on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 >> 172.21.15.50:0/1781532440 conn(0x1a55eb0 msgr2=0x1c87f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 shutdown_connections 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 1 -- 172.21.15.50:0/1781532440 wait complete. 2023-12-19T14:39:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.197+0000 7fc30276d740 20 client.4410 trim_cache size 0 max 16384 2023-12-19T14:39:15.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 Processor -- start 2023-12-19T14:39:15.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 -- start start 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c89ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1cb6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bd0f40 con 0x1a55ae0 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c01ef0 con 0x1a55710 2023-12-19T14:39:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1babc90 con 0x1aa0790 2023-12-19T14:39:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36214/0 (socket says 172.21.15.50:36214) 2023-12-19T14:39:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 -- 172.21.15.50:0/1703967652 learned_addr learned my addr 172.21.15.50:0/1703967652 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c89ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fea15700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1cb6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 -- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 msgr2=0x1cb6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1cb6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 -- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 msgr2=0x1c89ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c89ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fda13700 1 -- 172.21.15.50:0/1703967652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1bb1d80 con 0x1a55ae0 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.221+0000 7fc2fe214700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c89ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc2fda13700 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8000c00 tx=0x7fc2e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12079ddb9bfb4818 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc2dbfff700 1 -- 172.21.15.50:0/1703967652 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8019d10 con 0x1a55ae0 2023-12-19T14:39:15.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc2dbfff700 1 -- 172.21.15.50:0/1703967652 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8019e70 con 0x1a55ae0 2023-12-19T14:39:15.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc2dbfff700 1 -- 172.21.15.50:0/1703967652 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80190d0 con 0x1a55ae0 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- 172.21.15.50:0/1703967652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 msgr2=0x1c8a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8000c00 tx=0x7fc2e8008040 comp rx=0 tx=0).stop 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- 172.21.15.50:0/1703967652 shutdown_connections 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1cb6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8a010 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 --2- 172.21.15.50:0/1703967652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c89ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- 172.21.15.50:0/1703967652 >> 172.21.15.50:0/1703967652 conn(0x1a55eb0 msgr2=0x1cd2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- 172.21.15.50:0/1703967652 shutdown_connections 2023-12-19T14:39:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- 172.21.15.50:0/1703967652 wait complete. 2023-12-19T14:39:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.225+0000 7fc30276d740 1 -- start start 2023-12-19T14:39:15.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 Processor -- start 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c88cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1c8ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bc6400 con 0x1a55ae0 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c009d0 con 0x1a55710 2023-12-19T14:39:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c01810 con 0x1aa0790 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc30276d740 10 client.0 _ll_register_callbacks cb 0x1a219d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc2fda13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc2fda13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36220/0 (socket says 172.21.15.50:36220) 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc2fda13700 1 -- 172.21.15.50:0/355928938 learned_addr learned my addr 172.21.15.50:0/355928938 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc2fea15700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1c8ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.229+0000 7fc2fe214700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c88cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 -- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 msgr2=0x1c8ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1c8ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 -- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 msgr2=0x1c88cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c88cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 -- 172.21.15.50:0/355928938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1bcb370 con 0x1a55ae0 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fe214700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c88cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fda13700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x1a76510 tx=0x7fc2e8026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197b4d93f6578492 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8017720 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc2e801c070 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2f004eed0 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2e8017a90 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 10 client.4416 fetch_fsmap learned FSMap version 6 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x7fc2e80096e0 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f004eed0 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7fc2e8017bf0 con 0x1a55ae0 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 10 client.4416 fetch_fsmap finished waiting for FSMap version 6 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 10 client.4416 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 20 client.4416 populate_metadata read hostname 'smithi050' 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1c8cf40 0x1d34b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 4 client.4416 mds_command: new command op to 4275 tid=0[{"prefix": "status"}] 2023-12-19T14:39:15.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 --> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1bd1410 con 0x1c8cf40 2023-12-19T14:39:15.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fe214700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1c8cf40 0x1d34b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2d8ff9700 10 client.4416 ms_handle_connect on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.233+0000 7fc2fe214700 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1c8cf40 0x1d34b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9de7c7433ff99187 server_cookie=f14155ac2bbd9152 in_seq=0 out_seq=0 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc2d8ff9700 1 -- 172.21.15.50:0/355928938 <== mds.0 v2:172.21.15.50:6833/3977471945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x1bd1410 con 0x1c8cf40 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc2d8ff9700 10 client.4416 handle_command_reply: tid=0 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 client.4416 shutdown 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 msgr2=0x1c8c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x1a76510 tx=0x7fc2e8026000 comp rx=0 tx=0).stop 2023-12-19T14:39:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 shutdown_connections 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1c8cf40 0x1d34b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1aa0790 0x1c8ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1a55ae0 0x1c8c4c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc2d8ff9700 0 client.4416 ms_handle_reset on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 --2- 172.21.15.50:0/355928938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1a55710 0x1c88cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 >> 172.21.15.50:0/355928938 conn(0x1a55eb0 msgr2=0x1cb6780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 shutdown_connections 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 1 -- 172.21.15.50:0/355928938 wait complete. 2023-12-19T14:39:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc30276d740 20 client.4416 trim_cache size 0 max 16384 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 4275, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 7.977351745, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 6, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 24, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 24, 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 13.530368026 2023-12-19T14:39:15.247 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 msgr2=0x7fc2dc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 0x7fc2dc062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40099b0 tx=0x7fc2f4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 msgr2=0x7fc2f80ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c8e0 tx=0x7fc2f0001710 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 shutdown_connections 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2dc060550 0x7fc2dc062a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2f8159220 0x7fc2f80ca850 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2f8158e50 0x7fc2f81692d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 --2- 172.21.15.50:0/91027371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2f80a98d0 0x7fc2f8141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 >> 172.21.15.50:0/91027371 conn(0x7fc2f8067b60 msgr2=0x7fc2f81543e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 shutdown_connections 2023-12-19T14:39:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.237+0000 7fc300478700 1 -- 172.21.15.50:0/91027371 wait complete. 2023-12-19T14:39:15.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:39:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.429+0000 7f6b235c5700 1 Processor -- start 2023-12-19T14:39:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.429+0000 7f6b235c5700 1 -- start start 2023-12-19T14:39:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b235c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6b1c1552f0 0x7f6b1c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b235c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c06c7c0 0x7f6b1c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b235c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c06d140 con 0x7f6b1c0ae520 2023-12-19T14:39:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b235c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c8710 con 0x7f6b1c1552f0 2023-12-19T14:39:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b235c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c1b90 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6b1c1552f0 0x7f6b1c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6b1c1552f0 0x7f6b1c1556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36230/0 (socket says 172.21.15.50:36230) 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 -- 172.21.15.50:0/3351740446 learned_addr learned my addr 172.21.15.50:0/3351740446 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21b62700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c06c7c0 0x7f6b1c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 -- 172.21.15.50:0/3351740446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c06c7c0 msgr2=0x7f6b1c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c06c7c0 0x7f6b1c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 -- 172.21.15.50:0/3351740446 >> v1:172.21.15.125:6789/0 conn(0x7f6b1c0ae520 legacy=0x7f6b1c0c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 -- 172.21.15.50:0/3351740446 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c06a9d0 con 0x7f6b1c1552f0 2023-12-19T14:39:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b21361700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1556c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b180099b0 tx=0x7f6b18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f75dbdba7c6ac7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b13fff700 1 -- 172.21.15.50:0/3351740446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18009ea0 con 0x7f6b1c1552f0 2023-12-19T14:39:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b13fff700 1 -- 172.21.15.50:0/3351740446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800b860 con 0x7f6b1c1552f0 2023-12-19T14:39:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.433+0000 7f6b13fff700 1 -- 172.21.15.50:0/3351740446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800bc00 con 0x7f6b1c1552f0 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- 172.21.15.50:0/3351740446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c1552f0 msgr2=0x7f6b1c1556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1556c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b180099b0 tx=0x7f6b18009250 comp rx=0 tx=0).stop 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- 172.21.15.50:0/3351740446 shutdown_connections 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c06c7c0 0x7f6b1c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 --2- 172.21.15.50:0/3351740446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1556c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- 172.21.15.50:0/3351740446 >> 172.21.15.50:0/3351740446 conn(0x7f6b1c0c8850 msgr2=0x7f6b1c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- 172.21.15.50:0/3351740446 shutdown_connections 2023-12-19T14:39:15.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- 172.21.15.50:0/3351740446 wait complete. 2023-12-19T14:39:15.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 Processor -- start 2023-12-19T14:39:15.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.437+0000 7f6b235c5700 1 -- start start 2023-12-19T14:39:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c0ae520 0x7f6b1c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1680c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c31d0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0b0610 con 0x7f6b1c1552f0 2023-12-19T14:39:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b235c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c158620 con 0x7f6b1c0ae520 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36244/0 (socket says 172.21.15.50:36244) 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 -- 172.21.15.50:0/1174008919 learned_addr learned my addr 172.21.15.50:0/1174008919 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21b62700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b20b60700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c0ae520 0x7f6b1c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 -- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c0ae520 msgr2=0x7f6b1c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c0ae520 0x7f6b1c0c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 -- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c1552f0 msgr2=0x7f6b1c1680c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1680c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 -- 172.21.15.50:0/1174008919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180094f0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b21361700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6b180094c0 tx=0x7f6b18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1801e070 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.441+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800bbe0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.445+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c158890 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.445+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b180106e0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.445+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c1563f0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.445+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c071120 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.445+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6b1802b3e0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.449+0000 7f6b117fa700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 0x7f6b00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.449+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6b18014070 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.449+0000 7f6b20b60700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 0x7f6b00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.449+0000 7f6b20b60700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 0x7f6b00062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c168020 tx=0x7f6b0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.453+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6b1804e230 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.665+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6b1c0642a0 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.665+0000 7f6b117fa700 1 -- 172.21.15.50:0/1174008919 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6b18051850 con 0x7f6b1c06c7c0 2023-12-19T14:39:15.672 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T14:39:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 msgr2=0x7f6b00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 0x7f6b00062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c168020 tx=0x7f6b0c009040 comp rx=0 tx=0).stop 2023-12-19T14:39:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 msgr2=0x7f6b1c145750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6b180094c0 tx=0x7f6b18019000 comp rx=0 tx=0).stop 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 shutdown_connections 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c1552f0 0x7f6b1c1680c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060550 0x7f6b00062a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c0ae520 0x7f6b1c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 --2- 172.21.15.50:0/1174008919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c06c7c0 0x7f6b1c145750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 >> 172.21.15.50:0/1174008919 conn(0x7f6b1c0c8850 msgr2=0x7f6b1c0cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:15.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 shutdown_connections 2023-12-19T14:39:15.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:15.669+0000 7f6b235c5700 1 -- 172.21.15.50:0/1174008919 wait complete. 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:39:15.698 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:39:15.699 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-12-19T14:39:15.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:15.699 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:15.715 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 42394sec preferred_lft 42394sec 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:15.716 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:15.717 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:39:15.717 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:39:15.717 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:15.717 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:39:15.717 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:15.718 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T14:39:15.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:15.718 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:15.718 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:15.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add name ceph-brx type bridge 2023-12-19T14:39:15.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr flush dev ceph-brx 2023-12-19T14:39:15.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set ceph-brx up 2023-12-19T14:39:15.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T14:39:15.719 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:15.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:15.884 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:39:15.921 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:39:15.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:15.924 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:39:16.010 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:39:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:39:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:39:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:39:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:39:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:39:16.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.012 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:16.012 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:16.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:39:16.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:39:16.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:39:16.013 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:16.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.158 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:39:16.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.175 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list-id 2023-12-19T14:39:16.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.237 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:16.238 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:16.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-12-19T14:39:16.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-12-19T14:39:16.238 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:16.361 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-12-19T14:39:16.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-12-19T14:39:16.362 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:16.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:16.526 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:16.526 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:16.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:39:16.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T14:39:16.526 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:16.600 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-12-19T14:39:16.600 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:16.600 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:16.619 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-12-19T14:39:16.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:16.620 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:16.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:16.768 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:16.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:16.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:16.854 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:16.917 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:39:16.917 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-12-19T14:39:16.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:17.004 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:17.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:17.075 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:17.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:17.077 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:17.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi050.stderr:2023-12-19T14:39:17.101+0000 7fcda25c3380 -1 init, newargv = 0x559afd072fe0 newargc=15 2023-12-19T14:39:17.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi050.stderr:ceph-fuse[38566]: starting ceph client 2023-12-19T14:39:17.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi050.stderr:ceph-fuse[38566]: starting fuse 2023-12-19T14:39:18.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:18.265 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:18.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:18.339 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:18.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:18.343 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:18.409 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:39:18.409 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:39:18.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:39:18.409 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:39:18.409 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:39:18.409 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:39:18.410 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.admin")) 2023-12-19T14:39:18.411 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:39:18.583 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.admin.38566.asok 2023-12-19T14:39:18.765 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.38566.asok 2023-12-19T14:39:18.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:18.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.38566.asok status 2023-12-19T14:39:18.956 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:39:18.956 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:39:18.956 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:39:18.956 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:39:18.956 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "admin", 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "38566", 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 4428, 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4428 2023-12-19T14:39:18.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 717253005 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 717253005 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.4428 192.168.0.1:0/717253005", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/717253005", 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:39:18.958 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 6, 2023-12-19T14:39:18.959 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 24, 2023-12-19T14:39:18.959 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:39:18.959 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:39:18.959 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:39:18.959 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:39:18.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:18.970 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:39:18.971 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:18.993 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:39:18.993 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:18.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:18.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:19.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:19.078 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-12-19T14:39:19.191 INFO:teuthology.orchestra.run.smithi050.stdout:# file: . 2023-12-19T14:39:19.191 INFO:teuthology.orchestra.run.smithi050.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-12-19T14:39:19.191 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:19.193 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:39:19.193 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:39:19.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-12-19T14:39:19.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi050.stderr:ceph-fuse[38566]: fuse finished with error 0 and tester_r 0 2023-12-19T14:39:19.234 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T14:39:19.298 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:39:19.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:19.299 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-12-19T14:39:19.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.525+0000 7f8bbec0c700 1 Processor -- start 2023-12-19T14:39:19.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.525+0000 7f8bbec0c700 1 -- start start 2023-12-19T14:39:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbec0c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8bb8158d60 0x7f8bb815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbec0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8bb806b7a0 0x7f8bb80c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbec0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb806c7c0 con 0x7f8bb80c51b0 2023-12-19T14:39:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbec0c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb806caa0 con 0x7f8bb8158d60 2023-12-19T14:39:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbec0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80c55a0 con 0x7f8bb806b7a0 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbdc0a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8bb8158d60 0x7f8bb815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbe40b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8bb80c51b0 0x7f8bb806a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37326/0 (socket says 172.21.15.50:37326) 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbe40b700 1 -- 172.21.15.50:0/2201694369 learned_addr learned my addr 172.21.15.50:0/2201694369 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbd409700 1 --2- 172.21.15.50:0/2201694369 >> v2:172.21.15.125:3300/0 conn(0x7f8bb806b7a0 0x7f8bb80c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/2201694369 >> v2:172.21.15.125:3300/0 conn(0x7f8bb806b7a0 msgr2=0x7f8bb80c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbdc0a700 1 --2- 172.21.15.50:0/2201694369 >> v2:172.21.15.125:3300/0 conn(0x7f8bb806b7a0 0x7f8bb80c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/2201694369 >> v1:172.21.15.50:6789/0 conn(0x7f8bb80c51b0 legacy=0x7f8bb806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.529+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/2201694369 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb815ce60 con 0x7f8bb8158d60 2023-12-19T14:39:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbdc0a700 1 --2- 172.21.15.50:0/2201694369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb8158d60 0x7f8bb815c920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8bb815c860 tx=0x7f8bb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33baea939a3ac7a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbcc08700 1 -- 172.21.15.50:0/2201694369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4011630 con 0x7f8bb8158d60 2023-12-19T14:39:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbcc08700 1 -- 172.21.15.50:0/2201694369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb4011790 con 0x7f8bb8158d60 2023-12-19T14:39:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbcc08700 1 -- 172.21.15.50:0/2201694369 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4011b30 con 0x7f8bb8158d60 2023-12-19T14:39:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 -- 172.21.15.50:0/2201694369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb8158d60 msgr2=0x7f8bb815c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/2201694369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb8158d60 0x7f8bb815c920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8bb815c860 tx=0x7f8bb40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 -- 172.21.15.50:0/2201694369 shutdown_connections 2023-12-19T14:39:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/2201694369 >> v2:172.21.15.125:3300/0 conn(0x7f8bb806b7a0 0x7f8bb80c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/2201694369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb8158d60 0x7f8bb815c920 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 -- 172.21.15.50:0/2201694369 >> 172.21.15.50:0/2201694369 conn(0x7f8bb8067bb0 msgr2=0x7f8bb80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 -- 172.21.15.50:0/2201694369 shutdown_connections 2023-12-19T14:39:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.533+0000 7f8bbec0c700 1 -- 172.21.15.50:0/2201694369 wait complete. 2023-12-19T14:39:19.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 Processor -- start 2023-12-19T14:39:19.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 -- start start 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 0x7f8bb81578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 0x7f8bb8157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80ae370 con 0x7f8bb806b7a0 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815aea0 con 0x7f8bb80c51b0 2023-12-19T14:39:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbec0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815b180 con 0x7f8bb8158d60 2023-12-19T14:39:19.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53516/0 (socket says 172.21.15.50:53516) 2023-12-19T14:39:19.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/1137314743 learned_addr learned my addr 172.21.15.50:0/1137314743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbd409700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 0x7f8bb81578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbe40b700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 0x7f8bb8157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 msgr2=0x7f8bb8157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 0x7f8bb8157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 msgr2=0x7f8bb81578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 0x7f8bb81578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 -- 172.21.15.50:0/1137314743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb4009420 con 0x7f8bb806b7a0 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbd409700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 0x7f8bb81578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbe40b700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 0x7f8bb8157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.537+0000 7f8bbdc0a700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8064040 tx=0x7f8bb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4011b10 con 0x7f8bb806b7a0 2023-12-19T14:39:19.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb40253f0 con 0x7f8bb806b7a0 2023-12-19T14:39:19.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb815b460 con 0x7f8bb806b7a0 2023-12-19T14:39:19.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4025790 con 0x7f8bb806b7a0 2023-12-19T14:39:19.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb815a170 con 0x7f8bb806b7a0 2023-12-19T14:39:19.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb804be40 con 0x7f8bb806b7a0 2023-12-19T14:39:19.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.541+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8bb4025b40 con 0x7f8bb806b7a0 2023-12-19T14:39:19.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.545+0000 7f8badffb700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 0x7f8b9c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.545+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8bb407e3c0 con 0x7f8bb806b7a0 2023-12-19T14:39:19.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.545+0000 7f8bbd409700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 0x7f8b9c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.545+0000 7f8bbd409700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 0x7f8b9c062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009cf0 tx=0x7f8ba8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.549+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8bb404e620 con 0x7f8bb806b7a0 2023-12-19T14:39:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.729+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb815d130 con 0x7f8bb806b7a0 2023-12-19T14:39:19.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8badffb700 1 -- 172.21.15.50:0/1137314743 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f8bb4051c40 con 0x7f8bb806b7a0 2023-12-19T14:39:19.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:19.739 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 msgr2=0x7f8b9c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 0x7f8b9c062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009cf0 tx=0x7f8ba8008040 comp rx=0 tx=0).stop 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 msgr2=0x7f8bb80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8064040 tx=0x7f8bb4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.733+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 shutdown_connections 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bb80c51b0 0x7f8bb81578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b9c060550 0x7f8b9c062a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8bb8158d60 0x7f8bb8157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 --2- 172.21.15.50:0/1137314743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bb806b7a0 0x7f8bb80ade30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 >> 172.21.15.50:0/1137314743 conn(0x7f8bb8067bb0 msgr2=0x7f8bb80cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 shutdown_connections 2023-12-19T14:39:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.737+0000 7f8bbec0c700 1 -- 172.21.15.50:0/1137314743 wait complete. 2023-12-19T14:39:19.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:19.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.925+0000 7fae99f85700 1 Processor -- start 2023-12-19T14:39:19.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.925+0000 7fae99f85700 1 -- start start 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fae94140710 0x7fae940c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 0x7fae940680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae94068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9406e150 con 0x7fae94140710 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9406d540 con 0x7fae94140ae0 2023-12-19T14:39:19.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae99f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9406d820 con 0x7fae940c75e0 2023-12-19T14:39:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fae94140710 0x7fae940c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fae94140710 0x7fae940c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53530/0 (socket says 172.21.15.50:53530) 2023-12-19T14:39:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 -- 172.21.15.50:0/812508791 learned_addr learned my addr 172.21.15.50:0/812508791 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae93fff700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae94068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae8affd700 1 --2- 172.21.15.50:0/812508791 >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 0x7fae940680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 -- 172.21.15.50:0/812508791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 msgr2=0x7fae94068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:19.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae94068630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 -- 172.21.15.50:0/812508791 >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 msgr2=0x7fae940680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 --2- 172.21.15.50:0/812508791 >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 0x7fae940680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 -- 172.21.15.50:0/812508791 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9406db00 con 0x7fae94140710 2023-12-19T14:39:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae93fff700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae94068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae8affd700 1 --2- 172.21.15.50:0/812508791 >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 0x7fae940680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.929+0000 7fae937fe700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae940c70a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae940c9960 tx=0x7fae84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557c99ca78b87252 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae92ffd700 1 -- 172.21.15.50:0/812508791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400b640 con 0x7fae94140710 2023-12-19T14:39:19.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae92ffd700 1 -- 172.21.15.50:0/812508791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8400b7a0 con 0x7fae94140710 2023-12-19T14:39:19.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae92ffd700 1 -- 172.21.15.50:0/812508791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400bb60 con 0x7fae94140710 2023-12-19T14:39:19.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 -- 172.21.15.50:0/812508791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 msgr2=0x7fae940c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae940c70a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae940c9960 tx=0x7fae84009250 comp rx=0 tx=0).stop 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 -- 172.21.15.50:0/812508791 shutdown_connections 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae94068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 --2- 172.21.15.50:0/812508791 >> v2:172.21.15.125:3300/0 conn(0x7fae94140ae0 0x7fae940680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 --2- 172.21.15.50:0/812508791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae940c70a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 -- 172.21.15.50:0/812508791 >> 172.21.15.50:0/812508791 conn(0x7fae940c8850 msgr2=0x7fae940c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 -- 172.21.15.50:0/812508791 shutdown_connections 2023-12-19T14:39:19.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 -- 172.21.15.50:0/812508791 wait complete. 2023-12-19T14:39:19.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.933+0000 7fae99f85700 1 Processor -- start 2023-12-19T14:39:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 -- start start 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae940c1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae94140ae0 0x7fae9415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940c4590 con 0x7fae94140710 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940c3980 con 0x7fae94140ae0 2023-12-19T14:39:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae99f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940c3c60 con 0x7fae940c75e0 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae937fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae940c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae937fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae940c1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46054/0 (socket says 172.21.15.50:46054) 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae937fe700 1 -- 172.21.15.50:0/2119529272 learned_addr learned my addr 172.21.15.50:0/2119529272 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53544/0 (socket says 172.21.15.50:53544) 2023-12-19T14:39:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae93fff700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae94140ae0 0x7fae9415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 -- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 msgr2=0x7fae940c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae940c1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 -- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae94140ae0 msgr2=0x7fae9415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae94140ae0 0x7fae9415c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 -- 172.21.15.50:0/2119529272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c009a20 con 0x7fae94140710 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.937+0000 7fae8affd700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ec80 tx=0x7fae7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00c850 con 0x7fae94140710 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c00c9b0 con 0x7fae94140710 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c009420 con 0x7fae94140710 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00cd50 con 0x7fae94140710 2023-12-19T14:39:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae840094f0 con 0x7fae94140710 2023-12-19T14:39:19.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.941+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae94071120 con 0x7fae94140710 2023-12-19T14:39:19.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.945+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fae7c013680 con 0x7fae94140710 2023-12-19T14:39:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.945+0000 7fae8bfff700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 0x7fae74062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.945+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fae7c07ccb0 con 0x7fae94140710 2023-12-19T14:39:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.945+0000 7fae937fe700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 0x7fae74062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.945+0000 7fae937fe700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 0x7fae74062a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b770 tx=0x7fae84017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:19.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:19.949+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fae7c04cfd0 con 0x7fae94140710 2023-12-19T14:39:20.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.129+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae940c42e0 con 0x7fae94140710 2023-12-19T14:39:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.129+0000 7fae8bfff700 1 -- 172.21.15.50:0/2119529272 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fae7c0505f0 con 0x7fae94140710 2023-12-19T14:39:20.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:20.137 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:20.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 msgr2=0x7fae74062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 0x7fae74062a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b770 tx=0x7fae84017000 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 msgr2=0x7fae9413f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00ec80 tx=0x7fae7c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 shutdown_connections 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae94140ae0 0x7fae9415c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fae74060580 0x7fae74062a30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae94140710 0x7fae9413f6f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 --2- 172.21.15.50:0/2119529272 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fae940c75e0 0x7fae940c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 >> 172.21.15.50:0/2119529272 conn(0x7fae940c8850 msgr2=0x7fae940ad040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 shutdown_connections 2023-12-19T14:39:20.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.133+0000 7fae99f85700 1 -- 172.21.15.50:0/2119529272 wait complete. 2023-12-19T14:39:20.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:39:20.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.325+0000 7fad88e2e700 1 Processor -- start 2023-12-19T14:39:20.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.325+0000 7fad88e2e700 1 -- start start 2023-12-19T14:39:20.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad88e2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad8413d2f0 0x7fad840cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad88e2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad8406b060 0x7fad8406b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad88e2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad840cc0e0 con 0x7fad8413cf20 2023-12-19T14:39:20.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad88e2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8406b9e0 con 0x7fad8413d2f0 2023-12-19T14:39:20.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad88e2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840cb4d0 con 0x7fad8406b060 2023-12-19T14:39:20.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fad8413cf20 0x7fad840c85d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37342/0 (socket says 172.21.15.50:37342) 2023-12-19T14:39:20.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8259c700 1 -- 172.21.15.50:0/1777291145 learned_addr learned my addr 172.21.15.50:0/1777291145 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:20.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 --2- 172.21.15.50:0/1777291145 >> v2:172.21.15.50:3300/0 conn(0x7fad8413d2f0 0x7fad840cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad82d9d700 1 --2- 172.21.15.50:0/1777291145 >> v2:172.21.15.125:3300/0 conn(0x7fad8406b060 0x7fad8406b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8159a700 1 -- 172.21.15.50:0/1777291145 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44999779 0 0) 0x7fad840cc0e0 con 0x7fad8413cf20 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8159a700 1 -- 172.21.15.50:0/1777291145 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad6c003570 con 0x7fad8413cf20 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 -- 172.21.15.50:0/1777291145 >> v2:172.21.15.125:3300/0 conn(0x7fad8406b060 msgr2=0x7fad8406b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 --2- 172.21.15.50:0/1777291145 >> v2:172.21.15.125:3300/0 conn(0x7fad8406b060 0x7fad8406b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 -- 172.21.15.50:0/1777291145 >> v1:172.21.15.50:6789/0 conn(0x7fad8413cf20 legacy=0x7fad840c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 -- 172.21.15.50:0/1777291145 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad840cb7b0 con 0x7fad8413d2f0 2023-12-19T14:39:20.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad81d9b700 1 --2- 172.21.15.50:0/1777291145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8413d2f0 0x7fad840cadf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad840cc050 tx=0x7fad780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fabd6941687390 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8159a700 1 -- 172.21.15.50:0/1777291145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad78011630 con 0x7fad8413d2f0 2023-12-19T14:39:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8159a700 1 -- 172.21.15.50:0/1777291145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad780117b0 con 0x7fad8413d2f0 2023-12-19T14:39:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.329+0000 7fad8159a700 1 -- 172.21.15.50:0/1777291145 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad78011b70 con 0x7fad8413d2f0 2023-12-19T14:39:20.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- 172.21.15.50:0/1777291145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8413d2f0 msgr2=0x7fad840cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 --2- 172.21.15.50:0/1777291145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8413d2f0 0x7fad840cadf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad840cc050 tx=0x7fad780091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- 172.21.15.50:0/1777291145 shutdown_connections 2023-12-19T14:39:20.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 --2- 172.21.15.50:0/1777291145 >> v2:172.21.15.125:3300/0 conn(0x7fad8406b060 0x7fad8406b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 --2- 172.21.15.50:0/1777291145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8413d2f0 0x7fad840cadf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- 172.21.15.50:0/1777291145 >> 172.21.15.50:0/1777291145 conn(0x7fad840cc580 msgr2=0x7fad840cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:20.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- 172.21.15.50:0/1777291145 shutdown_connections 2023-12-19T14:39:20.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- 172.21.15.50:0/1777291145 wait complete. 2023-12-19T14:39:20.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 Processor -- start 2023-12-19T14:39:20.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.333+0000 7fad88e2e700 1 -- start start 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 0x7fad84154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 0x7fad840b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8406d620 con 0x7fad8406b060 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8406c7c0 con 0x7fad8413cf20 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8406e760 con 0x7fad8413d2f0 2023-12-19T14:39:20.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53552/0 (socket says 172.21.15.50:53552) 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 -- 172.21.15.50:0/2581613890 learned_addr learned my addr 172.21.15.50:0/2581613890 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad82d9d700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 0x7fad840b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad81d9b700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 0x7fad84154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 -- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 msgr2=0x7fad840b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 0x7fad840b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 -- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 msgr2=0x7fad84154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 0x7fad84154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 -- 172.21.15.50:0/2581613890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78009420 con 0x7fad8406b060 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad81d9b700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 0x7fad84154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad82d9d700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 0x7fad840b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad8259c700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c7c0 tx=0x7fad74007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74005b30 con 0x7fad8406b060 2023-12-19T14:39:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74005c90 con 0x7fad8406b060 2023-12-19T14:39:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74007340 con 0x7fad8406b060 2023-12-19T14:39:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8406ca10 con 0x7fad8406b060 2023-12-19T14:39:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.337+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8406cfd0 con 0x7fad8406b060 2023-12-19T14:39:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.341+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84071120 con 0x7fad8406b060 2023-12-19T14:39:20.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.341+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fad7401b400 con 0x7fad8406b060 2023-12-19T14:39:20.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.345+0000 7fad727fc700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 0x7fad64062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.345+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fad7407c5f0 con 0x7fad8406b060 2023-12-19T14:39:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.345+0000 7fad81d9b700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 0x7fad64062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.345+0000 7fad81d9b700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 0x7fad64062ad0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad8406ae30 tx=0x7fad78017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.349+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad7404c950 con 0x7fad8406b060 2023-12-19T14:39:20.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.561+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fad840cba90 con 0x7fad8406b060 2023-12-19T14:39:20.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.589+0000 7fad727fc700 1 -- 172.21.15.50:0/2581613890 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fad7404ff70 con 0x7fad8406b060 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 msgr2=0x7fad64062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 0x7fad64062ad0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad8406ae30 tx=0x7fad78017000 comp rx=0 tx=0).stop 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 msgr2=0x7fad8406e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad7400c7c0 tx=0x7fad74007e00 comp rx=0 tx=0).stop 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 shutdown_connections 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8413cf20 0x7fad84154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fad64060620 0x7fad64062ad0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad8413d2f0 0x7fad840b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 --2- 172.21.15.50:0/2581613890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad8406b060 0x7fad8406e220 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 >> 172.21.15.50:0/2581613890 conn(0x7fad840cc580 msgr2=0x7fad84156560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:20.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 shutdown_connections 2023-12-19T14:39:20.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.593+0000 7fad88e2e700 1 -- 172.21.15.50:0/2581613890 wait complete. 2023-12-19T14:39:20.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:39:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.781+0000 7f38f0ecd700 1 Processor -- start 2023-12-19T14:39:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.781+0000 7f38f0ecd700 1 -- start start 2023-12-19T14:39:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38f0ecd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38f0ecd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38ec0cb970 0x7f38ec0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38f0ecd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ec0cbdf0 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38f0ecd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ec0cc0d0 con 0x7f38ec0cb970 2023-12-19T14:39:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38f0ecd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0cc3b0 con 0x7f38ec158e20 2023-12-19T14:39:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38ea59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38e9d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38ec0cb970 0x7f38ec0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38ead9d700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f38ec0ca2c0 0x7f38ec06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46908/0 (socket says 172.21.15.50:46908) 2023-12-19T14:39:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38ea59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec0ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46080/0 (socket says 172.21.15.50:46080) 2023-12-19T14:39:20.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38ead9d700 1 -- 172.21.15.50:0/2267255479 learned_addr learned my addr 172.21.15.50:0/2267255479 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38e9d9b700 1 -- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 msgr2=0x7f38ec0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38e9d9b700 1 -- 172.21.15.50:0/2267255479 >> v1:172.21.15.125:6789/0 conn(0x7f38ec0ca2c0 legacy=0x7f38ec06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.785+0000 7f38e9d9b700 1 -- 172.21.15.50:0/2267255479 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec0cc690 con 0x7f38ec0cb970 2023-12-19T14:39:20.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec0cb970 0x7f38ec0ceb00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38e00098e0 tx=0x7f38e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=760411dbb2527ee5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38e959a700 1 -- 172.21.15.50:0/2267255479 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e000b640 con 0x7f38ec0cb970 2023-12-19T14:39:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38e959a700 1 -- 172.21.15.50:0/2267255479 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e000b7a0 con 0x7f38ec0cb970 2023-12-19T14:39:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38e959a700 1 -- 172.21.15.50:0/2267255479 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e000bb40 con 0x7f38ec0cb970 2023-12-19T14:39:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 -- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec0cb970 msgr2=0x7f38ec0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec0cb970 0x7f38ec0ceb00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38e00098e0 tx=0x7f38e00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 -- 172.21.15.50:0/2267255479 shutdown_connections 2023-12-19T14:39:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec0cb970 0x7f38ec0ceb00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/2267255479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 -- 172.21.15.50:0/2267255479 >> 172.21.15.50:0/2267255479 conn(0x7f38ec067bb0 msgr2=0x7f38ec0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 -- 172.21.15.50:0/2267255479 shutdown_connections 2023-12-19T14:39:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.789+0000 7f38f0ecd700 1 -- 172.21.15.50:0/2267255479 wait complete. 2023-12-19T14:39:20.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.793+0000 7f38f0ecd700 1 Processor -- start 2023-12-19T14:39:20.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.793+0000 7f38f0ecd700 1 -- start start 2023-12-19T14:39:20.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.793+0000 7f38f0ecd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.793+0000 7f38f0ecd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec15abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.793+0000 7f38f0ecd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec15b100 0x7f38ec15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38f0ecd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c7c40 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38f0ecd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c7f20 con 0x7f38ec15b100 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38f0ecd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c8200 con 0x7f38ec158e20 2023-12-19T14:39:20.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53554/0 (socket says 172.21.15.50:53554) 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 -- 172.21.15.50:0/326838789 learned_addr learned my addr 172.21.15.50:0/326838789 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec15abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ead9d700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec15b100 0x7f38ec15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 -- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 msgr2=0x7f38ec15abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec15abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 -- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec15b100 msgr2=0x7f38ec15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:20.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec15b100 0x7f38ec15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:20.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 -- 172.21.15.50:0/326838789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e0009420 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec15abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:20.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38ea59c700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00e930 tx=0x7f38dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc013680 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0137e0 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc013b80 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ec0c84e0 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.797+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec0c9f50 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.801+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec071120 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.801+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f38dc00c3e0 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.801+0000 7f38da7fc700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 0x7f38cc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:20.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.801+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f38dc07cb50 con 0x7f38ec0ca2c0 2023-12-19T14:39:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.805+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 0x7f38cc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.805+0000 7f38e9d9b700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 0x7f38cc062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f38e00093f0 tx=0x7f38e0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:20.809+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38dc04ce50 con 0x7f38ec0ca2c0 2023-12-19T14:39:21.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.017+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f38ec06a7a0 con 0x7f38ec0ca2c0 2023-12-19T14:39:21.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.601+0000 7f38da7fc700 1 -- 172.21.15.50:0/326838789 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f38dc0170c0 con 0x7f38ec0ca2c0 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 msgr2=0x7f38cc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 0x7f38cc062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f38e00093f0 tx=0x7f38e0017000 comp rx=0 tx=0).stop 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 msgr2=0x7f38ec14c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00e930 tx=0x7f38dc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 shutdown_connections 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38ec15b100 0x7f38ec15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f38cc060550 0x7f38cc062a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38ec158e20 0x7f38ec15abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 --2- 172.21.15.50:0/326838789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38ec0ca2c0 0x7f38ec14c7f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 >> 172.21.15.50:0/326838789 conn(0x7f38ec067bb0 msgr2=0x7f38ec06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 shutdown_connections 2023-12-19T14:39:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.605+0000 7f38f0ecd700 1 -- 172.21.15.50:0/326838789 wait complete. 2023-12-19T14:39:21.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:21.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.797+0000 7efd0e2f0700 1 Processor -- start 2023-12-19T14:39:21.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.797+0000 7efd0e2f0700 1 -- start start 2023-12-19T14:39:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd0815d320 0x7efd0815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd080c15b0 0x7efd080aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080c7190 con 0x7efd0815d320 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080c1b50 con 0x7efd080c15b0 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0e2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd080c1e30 con 0x7efd0806d690 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd0815d320 0x7efd0815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd0815d320 0x7efd0815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53570/0 (socket says 172.21.15.50:53570) 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 -- 172.21.15.50:0/1165913712 learned_addr learned my addr 172.21.15.50:0/1165913712 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efcff7fe700 1 --2- 172.21.15.50:0/1165913712 >> v2:172.21.15.125:3300/0 conn(0x7efd080c15b0 0x7efd080aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd0c88d700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 -- 172.21.15.50:0/1165913712 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 msgr2=0x7efd080680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 -- 172.21.15.50:0/1165913712 >> v2:172.21.15.125:3300/0 conn(0x7efd080c15b0 msgr2=0x7efd080aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 --2- 172.21.15.50:0/1165913712 >> v2:172.21.15.125:3300/0 conn(0x7efd080c15b0 0x7efd080aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.801+0000 7efd07fff700 1 -- 172.21.15.50:0/1165913712 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080c6e20 con 0x7efd0815d320 2023-12-19T14:39:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd07fff700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815d6f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0806f530 tx=0x7efcf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e950202771a2cd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd077fe700 1 -- 172.21.15.50:0/1165913712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800b640 con 0x7efd0815d320 2023-12-19T14:39:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd077fe700 1 -- 172.21.15.50:0/1165913712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf800b7a0 con 0x7efd0815d320 2023-12-19T14:39:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd077fe700 1 -- 172.21.15.50:0/1165913712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800bb60 con 0x7efd0815d320 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 -- 172.21.15.50:0/1165913712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 msgr2=0x7efd0815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815d6f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0806f530 tx=0x7efcf8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 -- 172.21.15.50:0/1165913712 shutdown_connections 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/1165913712 >> v2:172.21.15.125:3300/0 conn(0x7efd080c15b0 0x7efd080aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/1165913712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815d6f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 -- 172.21.15.50:0/1165913712 >> 172.21.15.50:0/1165913712 conn(0x7efd080c8850 msgr2=0x7efd080c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 -- 172.21.15.50:0/1165913712 shutdown_connections 2023-12-19T14:39:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.805+0000 7efd0e2f0700 1 -- 172.21.15.50:0/1165913712 wait complete. 2023-12-19T14:39:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 Processor -- start 2023-12-19T14:39:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 -- start start 2023-12-19T14:39:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd080c15b0 0x7efd08146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd080b0830 con 0x7efd0815d320 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd080c3ea0 con 0x7efd080c15b0 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0e2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd080afcc0 con 0x7efd0806d690 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53582/0 (socket says 172.21.15.50:53582) 2023-12-19T14:39:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 -- 172.21.15.50:0/3116672990 learned_addr learned my addr 172.21.15.50:0/3116672990 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efcff7fe700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd080c15b0 0x7efd08146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd07fff700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 -- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 msgr2=0x7efd080af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 -- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd080c15b0 msgr2=0x7efd08146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd080c15b0 0x7efd08146df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 -- 172.21.15.50:0/3116672990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80094f0 con 0x7efd0815d320 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.809+0000 7efd0c88d700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd00009e20 tx=0x7efd0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0000cc20 con 0x7efd0815d320 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0000f040 con 0x7efd0815d320 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd080aff90 con 0x7efd0815d320 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00014480 con 0x7efd0815d320 2023-12-19T14:39:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd080b0550 con 0x7efd0815d320 2023-12-19T14:39:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.813+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd080c4030 con 0x7efd0815d320 2023-12-19T14:39:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.817+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efd00014830 con 0x7efd0815d320 2023-12-19T14:39:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.817+0000 7efd04ff9700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 0x7efce8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.817+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7efd0007d6e0 con 0x7efd0815d320 2023-12-19T14:39:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.817+0000 7efd07fff700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 0x7efce8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.817+0000 7efd07fff700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 0x7efce8062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd0806f530 tx=0x7efcf8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:21.825+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd0004d9e0 con 0x7efd0815d320 2023-12-19T14:39:22.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.029+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd08071120 con 0x7efd0815d320 2023-12-19T14:39:22.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.029+0000 7efd04ff9700 1 -- 172.21.15.50:0/3116672990 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7efd00051000 con 0x7efd0815d320 2023-12-19T14:39:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:22.037 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:39:22.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 msgr2=0x7efce8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 0x7efce8062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd0806f530 tx=0x7efcf8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 msgr2=0x7efd0815fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd00009e20 tx=0x7efd0000b040 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 shutdown_connections 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd080c15b0 0x7efd08146df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efce8060550 0x7efce8062a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd0815d320 0x7efd0815fa90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 --2- 172.21.15.50:0/3116672990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd0806d690 0x7efd080af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 >> 172.21.15.50:0/3116672990 conn(0x7efd080c8850 msgr2=0x7efd0815b360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:22.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 shutdown_connections 2023-12-19T14:39:22.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.033+0000 7efd0e2f0700 1 -- 172.21.15.50:0/3116672990 wait complete. 2023-12-19T14:39:22.043 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:39:22.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:22.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 Processor -- start 2023-12-19T14:39:22.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 -- start start 2023-12-19T14:39:22.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc180c6900 0x7fbc180c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc1806da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1806b470 con 0x7fbc1815cc20 2023-12-19T14:39:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc1815d7b0 con 0x7fbc180c6900 2023-12-19T14:39:22.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.221+0000 7fbc1da8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc18161d10 con 0x7fbc1806b900 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc1d28d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbc1815cc20 0x7fbc1815d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37352/0 (socket says 172.21.15.50:37352) 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc1d28d700 1 -- 172.21.15.50:0/3593498952 learned_addr learned my addr 172.21.15.50:0/3593498952 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 --2- 172.21.15.50:0/3593498952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc1806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc1ca8c700 1 --2- 172.21.15.50:0/3593498952 >> v2:172.21.15.125:3300/0 conn(0x7fbc180c6900 0x7fbc180c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc177fe700 1 -- 172.21.15.50:0/3593498952 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784457837 0 0) 0x7fbc1806b470 con 0x7fbc1815cc20 2023-12-19T14:39:22.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc177fe700 1 -- 172.21.15.50:0/3593498952 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc08003570 con 0x7fbc1815cc20 2023-12-19T14:39:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 -- 172.21.15.50:0/3593498952 >> v2:172.21.15.125:3300/0 conn(0x7fbc180c6900 msgr2=0x7fbc180c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 --2- 172.21.15.50:0/3593498952 >> v2:172.21.15.125:3300/0 conn(0x7fbc180c6900 0x7fbc180c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 -- 172.21.15.50:0/3593498952 >> v1:172.21.15.50:6789/0 conn(0x7fbc1815cc20 legacy=0x7fbc1815d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 -- 172.21.15.50:0/3593498952 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1806bd70 con 0x7fbc1806b900 2023-12-19T14:39:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc17fff700 1 --2- 172.21.15.50:0/3593498952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc1806da00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009910 tx=0x7fbc040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d75b73780cd395 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc177fe700 1 -- 172.21.15.50:0/3593498952 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc04011630 con 0x7fbc1806b900 2023-12-19T14:39:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc177fe700 1 -- 172.21.15.50:0/3593498952 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc04011790 con 0x7fbc1806b900 2023-12-19T14:39:22.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc177fe700 1 -- 172.21.15.50:0/3593498952 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc04011b30 con 0x7fbc1806b900 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3593498952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 msgr2=0x7fbc1806da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.225+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3593498952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc1806da00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009910 tx=0x7fbc040091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3593498952 shutdown_connections 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3593498952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc1806da00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3593498952 >> v2:172.21.15.125:3300/0 conn(0x7fbc180c6900 0x7fbc180c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3593498952 >> 172.21.15.50:0/3593498952 conn(0x7fbc18067bb0 msgr2=0x7fbc180cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3593498952 shutdown_connections 2023-12-19T14:39:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3593498952 wait complete. 2023-12-19T14:39:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 Processor -- start 2023-12-19T14:39:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- start start 2023-12-19T14:39:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc18159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 0x7fbc180ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180b0b50 con 0x7fbc1815cc20 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180aff40 con 0x7fbc180c6900 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.229+0000 7fbc1da8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180b0220 con 0x7fbc1806b900 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53584/0 (socket says 172.21.15.50:53584) 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 -- 172.21.15.50:0/3814552630 learned_addr learned my addr 172.21.15.50:0/3814552630 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1ca8c700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc18159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc17fff700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 0x7fbc180ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 -- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 msgr2=0x7fbc18159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc18159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 -- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 msgr2=0x7fbc180ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 0x7fbc180ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 -- 172.21.15.50:0/3814552630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04009420 con 0x7fbc1815cc20 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc17fff700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 0x7fbc180ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1d28d700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc10001460 tx=0x7fbc10001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000a8a0 con 0x7fbc1815cc20 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1000aa00 con 0x7fbc1815cc20 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000ad80 con 0x7fbc1815cc20 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc180b0500 con 0x7fbc1815cc20 2023-12-19T14:39:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.233+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1815a360 con 0x7fbc1815cc20 2023-12-19T14:39:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.237+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1804be40 con 0x7fbc1815cc20 2023-12-19T14:39:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.237+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbc10007300 con 0x7fbc1815cc20 2023-12-19T14:39:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.237+0000 7fbc14ff9700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 0x7fbbfc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.237+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbc1007ce90 con 0x7fbc1815cc20 2023-12-19T14:39:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.237+0000 7fbc1ca8c700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 0x7fbbfc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.241+0000 7fbc1ca8c700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 0x7fbbfc062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009db0 tx=0x7fbc0c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.245+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbc1004d210 con 0x7fbc1815cc20 2023-12-19T14:39:22.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.417+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc1806a580 con 0x7fbc1815cc20 2023-12-19T14:39:22.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.417+0000 7fbc14ff9700 1 -- 172.21.15.50:0/3814552630 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7fbc10050830 con 0x7fbc1815cc20 2023-12-19T14:39:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:22.424 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 msgr2=0x7fbbfc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 0x7fbbfc062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009db0 tx=0x7fbc0c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 msgr2=0x7fbc180ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc10001460 tx=0x7fbc10001a40 comp rx=0 tx=0).stop 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 shutdown_connections 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc180c6900 0x7fbc180ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbfc060550 0x7fbbfc062a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc1815cc20 0x7fbc180ad9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 --2- 172.21.15.50:0/3814552630 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc1806b900 0x7fbc18159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 >> 172.21.15.50:0/3814552630 conn(0x7fbc18067bb0 msgr2=0x7fbc1815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 shutdown_connections 2023-12-19T14:39:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.421+0000 7fbc1da8e700 1 -- 172.21.15.50:0/3814552630 wait complete. 2023-12-19T14:39:22.451 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:39:22.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:39:22.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 Processor -- start 2023-12-19T14:39:22.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 -- start start 2023-12-19T14:39:22.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 0x7fb0ac06d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac06c7c0 con 0x7fb0ac0b1410 2023-12-19T14:39:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac06cd80 con 0x7fb0ac06a580 2023-12-19T14:39:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.609+0000 7fb0b0dcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06b8f0 con 0x7fb0ac06aa00 2023-12-19T14:39:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0aaffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb0ac06a580 0x7fb0ac06d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46910/0 (socket says 172.21.15.50:46910) 2023-12-19T14:39:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0aaffd700 1 -- 172.21.15.50:0/32792854 learned_addr learned my addr 172.21.15.50:0/32792854 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:22.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0ab7fe700 1 --1- 172.21.15.50:0/32792854 >> v1:172.21.15.50:6789/0 conn(0x7fb0ac0b1410 0x7fb0ac0ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37358/0 (socket says 172.21.15.50:37358) 2023-12-19T14:39:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0abfff700 1 --2- 172.21.15.50:0/32792854 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 0x7fb0ac06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0abfff700 1 -- 172.21.15.50:0/32792854 >> v1:172.21.15.125:6789/0 conn(0x7fb0ac06a580 legacy=0x7fb0ac06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0abfff700 1 -- 172.21.15.50:0/32792854 >> v1:172.21.15.50:6789/0 conn(0x7fb0ac0b1410 legacy=0x7fb0ac0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0abfff700 1 -- 172.21.15.50:0/32792854 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0ac1e0 con 0x7fb0ac06aa00 2023-12-19T14:39:22.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0aa7fc700 1 -- 172.21.15.50:0/32792854 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499550413 0 0) 0x7fb0ac06c7c0 con 0x7fb0ac0b1410 2023-12-19T14:39:22.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.613+0000 7fb0abfff700 1 --2- 172.21.15.50:0/32792854 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 0x7fb0ac06d920 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009e90 tx=0x7fb0a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a720218e407815aa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.617+0000 7fb0aa7fc700 1 -- 172.21.15.50:0/32792854 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400e070 con 0x7fb0ac06aa00 2023-12-19T14:39:22.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.617+0000 7fb0aa7fc700 1 -- 172.21.15.50:0/32792854 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a400bc30 con 0x7fb0ac06aa00 2023-12-19T14:39:22.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.617+0000 7fb0aa7fc700 1 -- 172.21.15.50:0/32792854 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4012430 con 0x7fb0ac06aa00 2023-12-19T14:39:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/32792854 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 msgr2=0x7fb0ac06d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/32792854 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 0x7fb0ac06d920 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009e90 tx=0x7fb0a4009780 comp rx=0 tx=0).stop 2023-12-19T14:39:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/32792854 shutdown_connections 2023-12-19T14:39:22.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/32792854 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06aa00 0x7fb0ac06d920 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/32792854 >> 172.21.15.50:0/32792854 conn(0x7fb0ac067bb0 msgr2=0x7fb0ac0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/32792854 shutdown_connections 2023-12-19T14:39:22.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/32792854 wait complete. 2023-12-19T14:39:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 Processor -- start 2023-12-19T14:39:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.621+0000 7fb0b0dcc700 1 -- start start 2023-12-19T14:39:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06a580 0x7fb0ac14c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0ac0b1410 0x7fb0ac0c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac14cb80 con 0x7fb0ac06aa00 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac158dc0 con 0x7fb0ac0b1410 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0b0310 con 0x7fb0ac06a580 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53586/0 (socket says 172.21.15.50:53586) 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 -- 172.21.15.50:0/2602571486 learned_addr learned my addr 172.21.15.50:0/2602571486 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:22.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0ab7fe700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06a580 0x7fb0ac14c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0abfff700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0ac0b1410 0x7fb0ac0c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 -- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06a580 msgr2=0x7fb0ac14c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06a580 0x7fb0ac14c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 -- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0ac0b1410 msgr2=0x7fb0ac0c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0ac0b1410 0x7fb0ac0c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 -- 172.21.15.50:0/2602571486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4009a20 con 0x7fb0ac06aa00 2023-12-19T14:39:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0aaffd700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000c7f0 tx=0x7fb0a0007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0005b30 con 0x7fb0ac06aa00 2023-12-19T14:39:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a0005cb0 con 0x7fb0ac06aa00 2023-12-19T14:39:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac0af9e0 con 0x7fb0ac06aa00 2023-12-19T14:39:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0007320 con 0x7fb0ac06aa00 2023-12-19T14:39:22.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.625+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac0affa0 con 0x7fb0ac06aa00 2023-12-19T14:39:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.629+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac0b0160 con 0x7fb0ac06aa00 2023-12-19T14:39:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.629+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb0a0007690 con 0x7fb0ac06aa00 2023-12-19T14:39:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.633+0000 7fb0937fe700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 0x7fb098062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:22.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.633+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb0a007be10 con 0x7fb0ac06aa00 2023-12-19T14:39:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.633+0000 7fb0ab7fe700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 0x7fb098062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.633+0000 7fb0ab7fe700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 0x7fb098062a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb09c007180 tx=0x7fb09c007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.641+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb0a007f030 con 0x7fb0ac06aa00 2023-12-19T14:39:22.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.809+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0ac04be40 con 0x7fb098060580 2023-12-19T14:39:22.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.833+0000 7fb0937fe700 1 -- 172.21.15.50:0/2602571486 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345091 (secure 0 0 0) 0x7fb0ac04be40 con 0x7fb098060580 2023-12-19T14:39:22.840 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 msgr2=0x7fb098062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 0x7fb098062a30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb09c007180 tx=0x7fb09c007ef0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 msgr2=0x7fb0ac0c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000c7f0 tx=0x7fb0a0007f30 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 shutdown_connections 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0ac0b1410 0x7fb0ac0c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb098060580 0x7fb098062a30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0ac06aa00 0x7fb0ac0c4c70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 --2- 172.21.15.50:0/2602571486 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0ac06a580 0x7fb0ac14c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 >> 172.21.15.50:0/2602571486 conn(0x7fb0ac067bb0 msgr2=0x7fb0ac156320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:22.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 shutdown_connections 2023-12-19T14:39:22.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:22.837+0000 7fb0b0dcc700 1 -- 172.21.15.50:0/2602571486 wait complete. 2023-12-19T14:39:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:39:22.867 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-12-19T14:39:22.132391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2393792,"kb_used_data":3200,"kb_used_omap":0,"kb_used_meta":2390528,"kb_avail":752580928,"statfs":{"total":773094113280,"available":770642870272,"internally_reserved":0,"allocated":3276800,"data_stored":1983584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2447900672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003428"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562203+0000","last_change":"2023-12-19T14:39:06.261176+0000","last_active":"2023-12-19T14:39:11.562203+0000","last_peered":"2023-12-19T14:39:11.562203+0000","last_clean":"2023-12-19T14:39:11.562203+0000","last_became_active":"2023-12-19T14:39:06.259857+0000","last_became_peered":"2023-12-19T14:39:06.259857+0000","last_unstale":"2023-12-19T14:39:11.562203+0000","last_undegraded":"2023-12-19T14:39:11.562203+0000","last_fullsized":"2023-12-19T14:39:11.562203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:32:43.075778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545898+0000","last_change":"2023-12-19T14:39:07.280964+0000","last_active":"2023-12-19T14:39:11.545898+0000","last_peered":"2023-12-19T14:39:11.545898+0000","last_clean":"2023-12-19T14:39:11.545898+0000","last_became_active":"2023-12-19T14:39:07.280410+0000","last_became_peered":"2023-12-19T14:39:07.280410+0000","last_unstale":"2023-12-19T14:39:11.545898+0000","last_undegraded":"2023-12-19T14:39:11.545898+0000","last_fullsized":"2023-12-19T14:39:11.545898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588669+0000","last_change":"2023-12-19T14:39:11.580119+0000","last_active":"2023-12-19T14:39:11.588669+0000","last_peered":"2023-12-19T14:39:11.588669+0000","last_clean":"2023-12-19T14:39:11.588669+0000","last_became_active":"2023-12-19T14:39:11.579405+0000","last_became_peered":"2023-12-19T14:39:11.579405+0000","last_unstale":"2023-12-19T14:39:11.588669+0000","last_undegraded":"2023-12-19T14:39:11.588669+0000","last_fullsized":"2023-12-19T14:39:11.588669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546035+0000","last_change":"2023-12-19T14:39:06.252186+0000","last_active":"2023-12-19T14:39:11.546035+0000","last_peered":"2023-12-19T14:39:11.546035+0000","last_clean":"2023-12-19T14:39:11.546035+0000","last_became_active":"2023-12-19T14:39:06.251869+0000","last_became_peered":"2023-12-19T14:39:06.251869+0000","last_unstale":"2023-12-19T14:39:11.546035+0000","last_undegraded":"2023-12-19T14:39:11.546035+0000","last_fullsized":"2023-12-19T14:39:11.546035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561557+0000","last_change":"2023-12-19T14:39:07.289142+0000","last_active":"2023-12-19T14:39:11.561557+0000","last_peered":"2023-12-19T14:39:11.561557+0000","last_clean":"2023-12-19T14:39:11.561557+0000","last_became_active":"2023-12-19T14:39:07.288822+0000","last_became_peered":"2023-12-19T14:39:07.288822+0000","last_unstale":"2023-12-19T14:39:11.561557+0000","last_undegraded":"2023-12-19T14:39:11.561557+0000","last_fullsized":"2023-12-19T14:39:11.561557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582458+0000","last_change":"2023-12-19T14:39:11.573848+0000","last_active":"2023-12-19T14:39:11.582458+0000","last_peered":"2023-12-19T14:39:11.582458+0000","last_clean":"2023-12-19T14:39:11.582458+0000","last_became_active":"2023-12-19T14:39:11.573088+0000","last_became_peered":"2023-12-19T14:39:11.573088+0000","last_unstale":"2023-12-19T14:39:11.582458+0000","last_undegraded":"2023-12-19T14:39:11.582458+0000","last_fullsized":"2023-12-19T14:39:11.582458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:29.985849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570627+0000","last_change":"2023-12-19T14:39:06.262908+0000","last_active":"2023-12-19T14:39:11.570627+0000","last_peered":"2023-12-19T14:39:11.570627+0000","last_clean":"2023-12-19T14:39:11.570627+0000","last_became_active":"2023-12-19T14:39:06.260114+0000","last_became_peered":"2023-12-19T14:39:06.260114+0000","last_unstale":"2023-12-19T14:39:11.570627+0000","last_undegraded":"2023-12-19T14:39:11.570627+0000","last_fullsized":"2023-12-19T14:39:11.570627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545839+0000","last_change":"2023-12-19T14:39:07.279790+0000","last_active":"2023-12-19T14:39:11.545839+0000","last_peered":"2023-12-19T14:39:11.545839+0000","last_clean":"2023-12-19T14:39:11.545839+0000","last_became_active":"2023-12-19T14:39:07.279497+0000","last_became_peered":"2023-12-19T14:39:07.279497+0000","last_unstale":"2023-12-19T14:39:11.545839+0000","last_undegraded":"2023-12-19T14:39:11.545839+0000","last_fullsized":"2023-12-19T14:39:11.545839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.574367+0000","last_change":"2023-12-19T14:39:11.571054+0000","last_active":"2023-12-19T14:39:11.574367+0000","last_peered":"2023-12-19T14:39:11.574367+0000","last_clean":"2023-12-19T14:39:11.574367+0000","last_became_active":"2023-12-19T14:39:11.570620+0000","last_became_peered":"2023-12-19T14:39:11.570620+0000","last_unstale":"2023-12-19T14:39:11.574367+0000","last_undegraded":"2023-12-19T14:39:11.574367+0000","last_fullsized":"2023-12-19T14:39:11.574367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:06:34.827613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570831+0000","last_change":"2023-12-19T14:39:06.260028+0000","last_active":"2023-12-19T14:39:11.570831+0000","last_peered":"2023-12-19T14:39:11.570831+0000","last_clean":"2023-12-19T14:39:11.570831+0000","last_became_active":"2023-12-19T14:39:06.259269+0000","last_became_peered":"2023-12-19T14:39:06.259269+0000","last_unstale":"2023-12-19T14:39:11.570831+0000","last_undegraded":"2023-12-19T14:39:11.570831+0000","last_fullsized":"2023-12-19T14:39:11.570831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:29.156364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548594+0000","last_change":"2023-12-19T14:39:07.282662+0000","last_active":"2023-12-19T14:39:11.548594+0000","last_peered":"2023-12-19T14:39:11.548594+0000","last_clean":"2023-12-19T14:39:11.548594+0000","last_became_active":"2023-12-19T14:39:07.282105+0000","last_became_peered":"2023-12-19T14:39:07.282105+0000","last_unstale":"2023-12-19T14:39:11.548594+0000","last_undegraded":"2023-12-19T14:39:11.548594+0000","last_fullsized":"2023-12-19T14:39:11.548594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583605+0000","last_change":"2023-12-19T14:39:11.578372+0000","last_active":"2023-12-19T14:39:11.583605+0000","last_peered":"2023-12-19T14:39:11.583605+0000","last_clean":"2023-12-19T14:39:11.583605+0000","last_became_active":"2023-12-19T14:39:11.575437+0000","last_became_peered":"2023-12-19T14:39:11.575437+0000","last_unstale":"2023-12-19T14:39:11.583605+0000","last_undegraded":"2023-12-19T14:39:11.583605+0000","last_fullsized":"2023-12-19T14:39:11.583605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:01.783711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550051+0000","last_change":"2023-12-19T14:39:07.274147+0000","last_active":"2023-12-19T14:39:11.550051+0000","last_peered":"2023-12-19T14:39:11.550051+0000","last_clean":"2023-12-19T14:39:11.550051+0000","last_became_active":"2023-12-19T14:39:07.273776+0000","last_became_peered":"2023-12-19T14:39:07.273776+0000","last_unstale":"2023-12-19T14:39:11.550051+0000","last_undegraded":"2023-12-19T14:39:11.550051+0000","last_fullsized":"2023-12-19T14:39:11.550051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570793+0000","last_change":"2023-12-19T14:39:06.253750+0000","last_active":"2023-12-19T14:39:11.570793+0000","last_peered":"2023-12-19T14:39:11.570793+0000","last_clean":"2023-12-19T14:39:11.570793+0000","last_became_active":"2023-12-19T14:39:06.253479+0000","last_became_peered":"2023-12-19T14:39:06.253479+0000","last_unstale":"2023-12-19T14:39:11.570793+0000","last_undegraded":"2023-12-19T14:39:11.570793+0000","last_fullsized":"2023-12-19T14:39:11.570793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:22.901851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564091+0000","last_change":"2023-12-19T14:39:11.562226+0000","last_active":"2023-12-19T14:39:11.564091+0000","last_peered":"2023-12-19T14:39:11.564091+0000","last_clean":"2023-12-19T14:39:11.564091+0000","last_became_active":"2023-12-19T14:39:11.561874+0000","last_became_peered":"2023-12-19T14:39:11.561874+0000","last_unstale":"2023-12-19T14:39:11.564091+0000","last_undegraded":"2023-12-19T14:39:11.564091+0000","last_fullsized":"2023-12-19T14:39:11.564091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543918+0000","last_change":"2023-12-19T14:39:07.297728+0000","last_active":"2023-12-19T14:39:11.543918+0000","last_peered":"2023-12-19T14:39:11.543918+0000","last_clean":"2023-12-19T14:39:11.543918+0000","last_became_active":"2023-12-19T14:39:07.297419+0000","last_became_peered":"2023-12-19T14:39:07.297419+0000","last_unstale":"2023-12-19T14:39:11.543918+0000","last_undegraded":"2023-12-19T14:39:11.543918+0000","last_fullsized":"2023-12-19T14:39:11.543918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540603+0000","last_change":"2023-12-19T14:39:06.250154+0000","last_active":"2023-12-19T14:39:11.540603+0000","last_peered":"2023-12-19T14:39:11.540603+0000","last_clean":"2023-12-19T14:39:11.540603+0000","last_became_active":"2023-12-19T14:39:06.249851+0000","last_became_peered":"2023-12-19T14:39:06.249851+0000","last_unstale":"2023-12-19T14:39:11.540603+0000","last_undegraded":"2023-12-19T14:39:11.540603+0000","last_fullsized":"2023-12-19T14:39:11.540603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583288+0000","last_change":"2023-12-19T14:39:11.576656+0000","last_active":"2023-12-19T14:39:11.583288+0000","last_peered":"2023-12-19T14:39:11.583288+0000","last_clean":"2023-12-19T14:39:11.583288+0000","last_became_active":"2023-12-19T14:39:11.576165+0000","last_became_peered":"2023-12-19T14:39:11.576165+0000","last_unstale":"2023-12-19T14:39:11.583288+0000","last_undegraded":"2023-12-19T14:39:11.583288+0000","last_fullsized":"2023-12-19T14:39:11.583288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545215+0000","last_change":"2023-12-19T14:39:07.274124+0000","last_active":"2023-12-19T14:39:11.545215+0000","last_peered":"2023-12-19T14:39:11.545215+0000","last_clean":"2023-12-19T14:39:11.545215+0000","last_became_active":"2023-12-19T14:39:07.273792+0000","last_became_peered":"2023-12-19T14:39:07.273792+0000","last_unstale":"2023-12-19T14:39:11.545215+0000","last_undegraded":"2023-12-19T14:39:11.545215+0000","last_fullsized":"2023-12-19T14:39:11.545215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540233+0000","last_change":"2023-12-19T14:39:06.244868+0000","last_active":"2023-12-19T14:39:11.540233+0000","last_peered":"2023-12-19T14:39:11.540233+0000","last_clean":"2023-12-19T14:39:11.540233+0000","last_became_active":"2023-12-19T14:39:06.244567+0000","last_became_peered":"2023-12-19T14:39:06.244567+0000","last_unstale":"2023-12-19T14:39:11.540233+0000","last_undegraded":"2023-12-19T14:39:11.540233+0000","last_fullsized":"2023-12-19T14:39:11.540233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.587736+0000","last_change":"2023-12-19T14:39:11.585814+0000","last_active":"2023-12-19T14:39:11.587736+0000","last_peered":"2023-12-19T14:39:11.587736+0000","last_clean":"2023-12-19T14:39:11.587736+0000","last_became_active":"2023-12-19T14:39:11.581875+0000","last_became_peered":"2023-12-19T14:39:11.581875+0000","last_unstale":"2023-12-19T14:39:11.587736+0000","last_undegraded":"2023-12-19T14:39:11.587736+0000","last_fullsized":"2023-12-19T14:39:11.587736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547078+0000","last_change":"2023-12-19T14:39:07.276433+0000","last_active":"2023-12-19T14:39:11.547078+0000","last_peered":"2023-12-19T14:39:11.547078+0000","last_clean":"2023-12-19T14:39:11.547078+0000","last_became_active":"2023-12-19T14:39:07.274815+0000","last_became_peered":"2023-12-19T14:39:07.274815+0000","last_unstale":"2023-12-19T14:39:11.547078+0000","last_undegraded":"2023-12-19T14:39:11.547078+0000","last_fullsized":"2023-12-19T14:39:11.547078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:57.360165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569740+0000","last_change":"2023-12-19T14:39:06.262949+0000","last_active":"2023-12-19T14:39:11.569740+0000","last_peered":"2023-12-19T14:39:11.569740+0000","last_clean":"2023-12-19T14:39:11.569740+0000","last_became_active":"2023-12-19T14:39:06.262630+0000","last_became_peered":"2023-12-19T14:39:06.262630+0000","last_unstale":"2023-12-19T14:39:11.569740+0000","last_undegraded":"2023-12-19T14:39:11.569740+0000","last_fullsized":"2023-12-19T14:39:11.569740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580388+0000","last_change":"2023-12-19T14:39:11.572679+0000","last_active":"2023-12-19T14:39:11.580388+0000","last_peered":"2023-12-19T14:39:11.580388+0000","last_clean":"2023-12-19T14:39:11.580388+0000","last_became_active":"2023-12-19T14:39:11.572334+0000","last_became_peered":"2023-12-19T14:39:11.572334+0000","last_unstale":"2023-12-19T14:39:11.580388+0000","last_undegraded":"2023-12-19T14:39:11.580388+0000","last_fullsized":"2023-12-19T14:39:11.580388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:00:02.733099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562034+0000","last_change":"2023-12-19T14:39:07.287397+0000","last_active":"2023-12-19T14:39:11.562034+0000","last_peered":"2023-12-19T14:39:11.562034+0000","last_clean":"2023-12-19T14:39:11.562034+0000","last_became_active":"2023-12-19T14:39:07.287079+0000","last_became_peered":"2023-12-19T14:39:07.287079+0000","last_unstale":"2023-12-19T14:39:11.562034+0000","last_undegraded":"2023-12-19T14:39:11.562034+0000","last_fullsized":"2023-12-19T14:39:11.562034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551941+0000","last_change":"2023-12-19T14:39:06.259853+0000","last_active":"2023-12-19T14:39:11.551941+0000","last_peered":"2023-12-19T14:39:11.551941+0000","last_clean":"2023-12-19T14:39:11.551941+0000","last_became_active":"2023-12-19T14:39:06.259573+0000","last_became_peered":"2023-12-19T14:39:06.259573+0000","last_unstale":"2023-12-19T14:39:11.551941+0000","last_undegraded":"2023-12-19T14:39:11.551941+0000","last_fullsized":"2023-12-19T14:39:11.551941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583774+0000","last_change":"2023-12-19T14:39:11.578284+0000","last_active":"2023-12-19T14:39:11.583774+0000","last_peered":"2023-12-19T14:39:11.583774+0000","last_clean":"2023-12-19T14:39:11.583774+0000","last_became_active":"2023-12-19T14:39:11.577805+0000","last_became_peered":"2023-12-19T14:39:11.577805+0000","last_unstale":"2023-12-19T14:39:11.583774+0000","last_undegraded":"2023-12-19T14:39:11.583774+0000","last_fullsized":"2023-12-19T14:39:11.583774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564518+0000","last_change":"2023-12-19T14:39:07.288511+0000","last_active":"2023-12-19T14:39:11.564518+0000","last_peered":"2023-12-19T14:39:11.564518+0000","last_clean":"2023-12-19T14:39:11.564518+0000","last_became_active":"2023-12-19T14:39:07.288221+0000","last_became_peered":"2023-12-19T14:39:07.288221+0000","last_unstale":"2023-12-19T14:39:11.564518+0000","last_undegraded":"2023-12-19T14:39:11.564518+0000","last_fullsized":"2023-12-19T14:39:11.564518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.728724+0000","last_change":"2023-12-19T14:39:06.260565+0000","last_active":"2023-12-19T14:39:11.728724+0000","last_peered":"2023-12-19T14:39:11.728724+0000","last_clean":"2023-12-19T14:39:11.728724+0000","last_became_active":"2023-12-19T14:39:06.260244+0000","last_became_peered":"2023-12-19T14:39:06.260244+0000","last_unstale":"2023-12-19T14:39:11.728724+0000","last_undegraded":"2023-12-19T14:39:11.728724+0000","last_fullsized":"2023-12-19T14:39:11.728724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588612+0000","last_change":"2023-12-19T14:39:11.583334+0000","last_active":"2023-12-19T14:39:11.588612+0000","last_peered":"2023-12-19T14:39:11.588612+0000","last_clean":"2023-12-19T14:39:11.588612+0000","last_became_active":"2023-12-19T14:39:11.582906+0000","last_became_peered":"2023-12-19T14:39:11.582906+0000","last_unstale":"2023-12-19T14:39:11.588612+0000","last_undegraded":"2023-12-19T14:39:11.588612+0000","last_fullsized":"2023-12-19T14:39:11.588612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:50.106844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540825+0000","last_change":"2023-12-19T14:39:07.267435+0000","last_active":"2023-12-19T14:39:11.540825+0000","last_peered":"2023-12-19T14:39:11.540825+0000","last_clean":"2023-12-19T14:39:11.540825+0000","last_became_active":"2023-12-19T14:39:07.267113+0000","last_became_peered":"2023-12-19T14:39:07.267113+0000","last_unstale":"2023-12-19T14:39:11.540825+0000","last_undegraded":"2023-12-19T14:39:11.540825+0000","last_fullsized":"2023-12-19T14:39:11.540825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570841+0000","last_change":"2023-12-19T14:39:06.262808+0000","last_active":"2023-12-19T14:39:11.570841+0000","last_peered":"2023-12-19T14:39:11.570841+0000","last_clean":"2023-12-19T14:39:11.570841+0000","last_became_active":"2023-12-19T14:39:06.262439+0000","last_became_peered":"2023-12-19T14:39:06.262439+0000","last_unstale":"2023-12-19T14:39:11.570841+0000","last_undegraded":"2023-12-19T14:39:11.570841+0000","last_fullsized":"2023-12-19T14:39:11.570841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583520+0000","last_change":"2023-12-19T14:39:11.575768+0000","last_active":"2023-12-19T14:39:11.583520+0000","last_peered":"2023-12-19T14:39:11.583520+0000","last_clean":"2023-12-19T14:39:11.583520+0000","last_became_active":"2023-12-19T14:39:11.575222+0000","last_became_peered":"2023-12-19T14:39:11.575222+0000","last_unstale":"2023-12-19T14:39:11.583520+0000","last_undegraded":"2023-12-19T14:39:11.583520+0000","last_fullsized":"2023-12-19T14:39:11.583520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548697+0000","last_change":"2023-12-19T14:39:07.291010+0000","last_active":"2023-12-19T14:39:11.548697+0000","last_peered":"2023-12-19T14:39:11.548697+0000","last_clean":"2023-12-19T14:39:11.548697+0000","last_became_active":"2023-12-19T14:39:07.290719+0000","last_became_peered":"2023-12-19T14:39:07.290719+0000","last_unstale":"2023-12-19T14:39:11.548697+0000","last_undegraded":"2023-12-19T14:39:11.548697+0000","last_fullsized":"2023-12-19T14:39:11.548697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540846+0000","last_change":"2023-12-19T14:39:06.252011+0000","last_active":"2023-12-19T14:39:11.540846+0000","last_peered":"2023-12-19T14:39:11.540846+0000","last_clean":"2023-12-19T14:39:11.540846+0000","last_became_active":"2023-12-19T14:39:06.251724+0000","last_became_peered":"2023-12-19T14:39:06.251724+0000","last_unstale":"2023-12-19T14:39:11.540846+0000","last_undegraded":"2023-12-19T14:39:11.540846+0000","last_fullsized":"2023-12-19T14:39:11.540846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573312+0000","last_change":"2023-12-19T14:39:11.562285+0000","last_active":"2023-12-19T14:39:11.573312+0000","last_peered":"2023-12-19T14:39:11.573312+0000","last_clean":"2023-12-19T14:39:11.573312+0000","last_became_active":"2023-12-19T14:39:11.561826+0000","last_became_peered":"2023-12-19T14:39:11.561826+0000","last_unstale":"2023-12-19T14:39:11.573312+0000","last_undegraded":"2023-12-19T14:39:11.573312+0000","last_fullsized":"2023-12-19T14:39:11.573312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546810+0000","last_change":"2023-12-19T14:39:07.289173+0000","last_active":"2023-12-19T14:39:11.546810+0000","last_peered":"2023-12-19T14:39:11.546810+0000","last_clean":"2023-12-19T14:39:11.546810+0000","last_became_active":"2023-12-19T14:39:07.288745+0000","last_became_peered":"2023-12-19T14:39:07.288745+0000","last_unstale":"2023-12-19T14:39:11.546810+0000","last_undegraded":"2023-12-19T14:39:11.546810+0000","last_fullsized":"2023-12-19T14:39:11.546810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545170+0000","last_change":"2023-12-19T14:39:06.258745+0000","last_active":"2023-12-19T14:39:11.545170+0000","last_peered":"2023-12-19T14:39:11.545170+0000","last_clean":"2023-12-19T14:39:11.545170+0000","last_became_active":"2023-12-19T14:39:06.257007+0000","last_became_peered":"2023-12-19T14:39:06.257007+0000","last_unstale":"2023-12-19T14:39:11.545170+0000","last_undegraded":"2023-12-19T14:39:11.545170+0000","last_fullsized":"2023-12-19T14:39:11.545170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571904+0000","last_change":"2023-12-19T14:39:11.568661+0000","last_active":"2023-12-19T14:39:11.571904+0000","last_peered":"2023-12-19T14:39:11.571904+0000","last_clean":"2023-12-19T14:39:11.571904+0000","last_became_active":"2023-12-19T14:39:11.567940+0000","last_became_peered":"2023-12-19T14:39:11.567940+0000","last_unstale":"2023-12-19T14:39:11.571904+0000","last_undegraded":"2023-12-19T14:39:11.571904+0000","last_fullsized":"2023-12-19T14:39:11.571904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539181+0000","last_change":"2023-12-19T14:39:07.269639+0000","last_active":"2023-12-19T14:39:11.539181+0000","last_peered":"2023-12-19T14:39:11.539181+0000","last_clean":"2023-12-19T14:39:11.539181+0000","last_became_active":"2023-12-19T14:39:07.269276+0000","last_became_peered":"2023-12-19T14:39:07.269276+0000","last_unstale":"2023-12-19T14:39:11.539181+0000","last_undegraded":"2023-12-19T14:39:11.539181+0000","last_fullsized":"2023-12-19T14:39:11.539181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553025+0000","last_change":"2023-12-19T14:39:06.254091+0000","last_active":"2023-12-19T14:39:11.553025+0000","last_peered":"2023-12-19T14:39:11.553025+0000","last_clean":"2023-12-19T14:39:11.553025+0000","last_became_active":"2023-12-19T14:39:06.253444+0000","last_became_peered":"2023-12-19T14:39:06.253444+0000","last_unstale":"2023-12-19T14:39:11.553025+0000","last_undegraded":"2023-12-19T14:39:11.553025+0000","last_fullsized":"2023-12-19T14:39:11.553025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584042+0000","last_change":"2023-12-19T14:39:11.574535+0000","last_active":"2023-12-19T14:39:11.584042+0000","last_peered":"2023-12-19T14:39:11.584042+0000","last_clean":"2023-12-19T14:39:11.584042+0000","last_became_active":"2023-12-19T14:39:11.574060+0000","last_became_peered":"2023-12-19T14:39:11.574060+0000","last_unstale":"2023-12-19T14:39:11.584042+0000","last_undegraded":"2023-12-19T14:39:11.584042+0000","last_fullsized":"2023-12-19T14:39:11.584042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.785232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556888+0000","last_change":"2023-12-19T14:39:07.287233+0000","last_active":"2023-12-19T14:39:11.556888+0000","last_peered":"2023-12-19T14:39:11.556888+0000","last_clean":"2023-12-19T14:39:11.556888+0000","last_became_active":"2023-12-19T14:39:07.286769+0000","last_became_peered":"2023-12-19T14:39:07.286769+0000","last_unstale":"2023-12-19T14:39:11.556888+0000","last_undegraded":"2023-12-19T14:39:11.556888+0000","last_fullsized":"2023-12-19T14:39:11.556888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569853+0000","last_change":"2023-12-19T14:39:06.251774+0000","last_active":"2023-12-19T14:39:11.569853+0000","last_peered":"2023-12-19T14:39:11.569853+0000","last_clean":"2023-12-19T14:39:11.569853+0000","last_became_active":"2023-12-19T14:39:06.251528+0000","last_became_peered":"2023-12-19T14:39:06.251528+0000","last_unstale":"2023-12-19T14:39:11.569853+0000","last_undegraded":"2023-12-19T14:39:11.569853+0000","last_fullsized":"2023-12-19T14:39:11.569853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581699+0000","last_change":"2023-12-19T14:39:11.575777+0000","last_active":"2023-12-19T14:39:11.581699+0000","last_peered":"2023-12-19T14:39:11.581699+0000","last_clean":"2023-12-19T14:39:11.581699+0000","last_became_active":"2023-12-19T14:39:11.575283+0000","last_became_peered":"2023-12-19T14:39:11.575283+0000","last_unstale":"2023-12-19T14:39:11.581699+0000","last_undegraded":"2023-12-19T14:39:11.581699+0000","last_fullsized":"2023-12-19T14:39:11.581699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:54.006908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543534+0000","last_change":"2023-12-19T14:39:07.294894+0000","last_active":"2023-12-19T14:39:11.543534+0000","last_peered":"2023-12-19T14:39:11.543534+0000","last_clean":"2023-12-19T14:39:11.543534+0000","last_became_active":"2023-12-19T14:39:07.294640+0000","last_became_peered":"2023-12-19T14:39:07.294640+0000","last_unstale":"2023-12-19T14:39:11.543534+0000","last_undegraded":"2023-12-19T14:39:11.543534+0000","last_fullsized":"2023-12-19T14:39:11.543534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545405+0000","last_change":"2023-12-19T14:39:06.256452+0000","last_active":"2023-12-19T14:39:11.545405+0000","last_peered":"2023-12-19T14:39:11.545405+0000","last_clean":"2023-12-19T14:39:11.545405+0000","last_became_active":"2023-12-19T14:39:06.255327+0000","last_became_peered":"2023-12-19T14:39:06.255327+0000","last_unstale":"2023-12-19T14:39:11.545405+0000","last_undegraded":"2023-12-19T14:39:11.545405+0000","last_fullsized":"2023-12-19T14:39:11.545405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569956+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:39:11.569956+0000","last_peered":"2023-12-19T14:39:11.569956+0000","last_clean":"2023-12-19T14:39:11.569956+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:39:11.569956+0000","last_undegraded":"2023-12-19T14:39:11.569956+0000","last_fullsized":"2023-12-19T14:39:11.569956+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582739+0000","last_change":"2023-12-19T14:39:11.578205+0000","last_active":"2023-12-19T14:39:11.582739+0000","last_peered":"2023-12-19T14:39:11.582739+0000","last_clean":"2023-12-19T14:39:11.582739+0000","last_became_active":"2023-12-19T14:39:11.577809+0000","last_became_peered":"2023-12-19T14:39:11.577809+0000","last_unstale":"2023-12-19T14:39:11.582739+0000","last_undegraded":"2023-12-19T14:39:11.582739+0000","last_fullsized":"2023-12-19T14:39:11.582739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:10.870899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570477+0000","last_change":"2023-12-19T14:39:07.283187+0000","last_active":"2023-12-19T14:39:11.570477+0000","last_peered":"2023-12-19T14:39:11.570477+0000","last_clean":"2023-12-19T14:39:11.570477+0000","last_became_active":"2023-12-19T14:39:07.282884+0000","last_became_peered":"2023-12-19T14:39:07.282884+0000","last_unstale":"2023-12-19T14:39:11.570477+0000","last_undegraded":"2023-12-19T14:39:11.570477+0000","last_fullsized":"2023-12-19T14:39:11.570477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553094+0000","last_change":"2023-12-19T14:39:06.260560+0000","last_active":"2023-12-19T14:39:11.553094+0000","last_peered":"2023-12-19T14:39:11.553094+0000","last_clean":"2023-12-19T14:39:11.553094+0000","last_became_active":"2023-12-19T14:39:06.260257+0000","last_became_peered":"2023-12-19T14:39:06.260257+0000","last_unstale":"2023-12-19T14:39:11.553094+0000","last_undegraded":"2023-12-19T14:39:11.553094+0000","last_fullsized":"2023-12-19T14:39:11.553094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580038+0000","last_change":"2023-12-19T14:39:11.575041+0000","last_active":"2023-12-19T14:39:11.580038+0000","last_peered":"2023-12-19T14:39:11.580038+0000","last_clean":"2023-12-19T14:39:11.580038+0000","last_became_active":"2023-12-19T14:39:11.574653+0000","last_became_peered":"2023-12-19T14:39:11.574653+0000","last_unstale":"2023-12-19T14:39:11.580038+0000","last_undegraded":"2023-12-19T14:39:11.580038+0000","last_fullsized":"2023-12-19T14:39:11.580038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545946+0000","last_change":"2023-12-19T14:39:07.291614+0000","last_active":"2023-12-19T14:39:11.545946+0000","last_peered":"2023-12-19T14:39:11.545946+0000","last_clean":"2023-12-19T14:39:11.545946+0000","last_became_active":"2023-12-19T14:39:07.291096+0000","last_became_peered":"2023-12-19T14:39:07.291096+0000","last_unstale":"2023-12-19T14:39:11.545946+0000","last_undegraded":"2023-12-19T14:39:11.545946+0000","last_fullsized":"2023-12-19T14:39:11.545946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540658+0000","last_change":"2023-12-19T14:39:06.245861+0000","last_active":"2023-12-19T14:39:11.540658+0000","last_peered":"2023-12-19T14:39:11.540658+0000","last_clean":"2023-12-19T14:39:11.540658+0000","last_became_active":"2023-12-19T14:39:06.245543+0000","last_became_peered":"2023-12-19T14:39:06.245543+0000","last_unstale":"2023-12-19T14:39:11.540658+0000","last_undegraded":"2023-12-19T14:39:11.540658+0000","last_fullsized":"2023-12-19T14:39:11.540658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.575504+0000","last_change":"2023-12-19T14:39:11.572562+0000","last_active":"2023-12-19T14:39:11.575504+0000","last_peered":"2023-12-19T14:39:11.575504+0000","last_clean":"2023-12-19T14:39:11.575504+0000","last_became_active":"2023-12-19T14:39:11.572159+0000","last_became_peered":"2023-12-19T14:39:11.572159+0000","last_unstale":"2023-12-19T14:39:11.575504+0000","last_undegraded":"2023-12-19T14:39:11.575504+0000","last_fullsized":"2023-12-19T14:39:11.575504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571036+0000","last_change":"2023-12-19T14:39:07.284702+0000","last_active":"2023-12-19T14:39:11.571036+0000","last_peered":"2023-12-19T14:39:11.571036+0000","last_clean":"2023-12-19T14:39:11.571036+0000","last_became_active":"2023-12-19T14:39:07.284258+0000","last_became_peered":"2023-12-19T14:39:07.284258+0000","last_unstale":"2023-12-19T14:39:11.571036+0000","last_undegraded":"2023-12-19T14:39:11.571036+0000","last_fullsized":"2023-12-19T14:39:11.571036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555155+0000","last_change":"2023-12-19T14:39:06.256120+0000","last_active":"2023-12-19T14:39:11.555155+0000","last_peered":"2023-12-19T14:39:11.555155+0000","last_clean":"2023-12-19T14:39:11.555155+0000","last_became_active":"2023-12-19T14:39:06.255187+0000","last_became_peered":"2023-12-19T14:39:06.255187+0000","last_unstale":"2023-12-19T14:39:11.555155+0000","last_undegraded":"2023-12-19T14:39:11.555155+0000","last_fullsized":"2023-12-19T14:39:11.555155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584352+0000","last_change":"2023-12-19T14:39:11.579495+0000","last_active":"2023-12-19T14:39:11.584352+0000","last_peered":"2023-12-19T14:39:11.584352+0000","last_clean":"2023-12-19T14:39:11.584352+0000","last_became_active":"2023-12-19T14:39:11.579259+0000","last_became_peered":"2023-12-19T14:39:11.579259+0000","last_unstale":"2023-12-19T14:39:11.584352+0000","last_undegraded":"2023-12-19T14:39:11.584352+0000","last_fullsized":"2023-12-19T14:39:11.584352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573192+0000","last_change":"2023-12-19T14:39:07.279364+0000","last_active":"2023-12-19T14:39:11.573192+0000","last_peered":"2023-12-19T14:39:11.573192+0000","last_clean":"2023-12-19T14:39:11.573192+0000","last_became_active":"2023-12-19T14:39:07.278999+0000","last_became_peered":"2023-12-19T14:39:07.278999+0000","last_unstale":"2023-12-19T14:39:11.573192+0000","last_undegraded":"2023-12-19T14:39:11.573192+0000","last_fullsized":"2023-12-19T14:39:11.573192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547974+0000","last_change":"2023-12-19T14:39:06.262399+0000","last_active":"2023-12-19T14:39:11.547974+0000","last_peered":"2023-12-19T14:39:11.547974+0000","last_clean":"2023-12-19T14:39:11.547974+0000","last_became_active":"2023-12-19T14:39:06.261918+0000","last_became_peered":"2023-12-19T14:39:06.261918+0000","last_unstale":"2023-12-19T14:39:11.547974+0000","last_undegraded":"2023-12-19T14:39:11.547974+0000","last_fullsized":"2023-12-19T14:39:11.547974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:48:08.535588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583755+0000","last_change":"2023-12-19T14:39:11.577720+0000","last_active":"2023-12-19T14:39:11.583755+0000","last_peered":"2023-12-19T14:39:11.583755+0000","last_clean":"2023-12-19T14:39:11.583755+0000","last_became_active":"2023-12-19T14:39:11.577297+0000","last_became_peered":"2023-12-19T14:39:11.577297+0000","last_unstale":"2023-12-19T14:39:11.583755+0000","last_undegraded":"2023-12-19T14:39:11.583755+0000","last_fullsized":"2023-12-19T14:39:11.583755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570176+0000","last_change":"2023-12-19T14:39:07.291311+0000","last_active":"2023-12-19T14:39:11.570176+0000","last_peered":"2023-12-19T14:39:11.570176+0000","last_clean":"2023-12-19T14:39:11.570176+0000","last_became_active":"2023-12-19T14:39:07.290587+0000","last_became_peered":"2023-12-19T14:39:07.290587+0000","last_unstale":"2023-12-19T14:39:11.570176+0000","last_undegraded":"2023-12-19T14:39:11.570176+0000","last_fullsized":"2023-12-19T14:39:11.570176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548490+0000","last_change":"2023-12-19T14:39:06.248952+0000","last_active":"2023-12-19T14:39:11.548490+0000","last_peered":"2023-12-19T14:39:11.548490+0000","last_clean":"2023-12-19T14:39:11.548490+0000","last_became_active":"2023-12-19T14:39:06.248504+0000","last_became_peered":"2023-12-19T14:39:06.248504+0000","last_unstale":"2023-12-19T14:39:11.548490+0000","last_undegraded":"2023-12-19T14:39:11.548490+0000","last_fullsized":"2023-12-19T14:39:11.548490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581722+0000","last_change":"2023-12-19T14:39:11.579028+0000","last_active":"2023-12-19T14:39:11.581722+0000","last_peered":"2023-12-19T14:39:11.581722+0000","last_clean":"2023-12-19T14:39:11.581722+0000","last_became_active":"2023-12-19T14:39:11.578386+0000","last_became_peered":"2023-12-19T14:39:11.578386+0000","last_unstale":"2023-12-19T14:39:11.581722+0000","last_undegraded":"2023-12-19T14:39:11.581722+0000","last_fullsized":"2023-12-19T14:39:11.581722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551739+0000","last_change":"2023-12-19T14:39:07.289550+0000","last_active":"2023-12-19T14:39:11.551739+0000","last_peered":"2023-12-19T14:39:11.551739+0000","last_clean":"2023-12-19T14:39:11.551739+0000","last_became_active":"2023-12-19T14:39:07.289242+0000","last_became_peered":"2023-12-19T14:39:07.289242+0000","last_unstale":"2023-12-19T14:39:11.551739+0000","last_undegraded":"2023-12-19T14:39:11.551739+0000","last_fullsized":"2023-12-19T14:39:11.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549347+0000","last_change":"2023-12-19T14:39:06.250752+0000","last_active":"2023-12-19T14:39:11.549347+0000","last_peered":"2023-12-19T14:39:11.549347+0000","last_clean":"2023-12-19T14:39:11.549347+0000","last_became_active":"2023-12-19T14:39:06.249438+0000","last_became_peered":"2023-12-19T14:39:06.249438+0000","last_unstale":"2023-12-19T14:39:11.549347+0000","last_undegraded":"2023-12-19T14:39:11.549347+0000","last_fullsized":"2023-12-19T14:39:11.549347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588513+0000","last_change":"2023-12-19T14:39:11.581267+0000","last_active":"2023-12-19T14:39:11.588513+0000","last_peered":"2023-12-19T14:39:11.588513+0000","last_clean":"2023-12-19T14:39:11.588513+0000","last_became_active":"2023-12-19T14:39:11.577498+0000","last_became_peered":"2023-12-19T14:39:11.577498+0000","last_unstale":"2023-12-19T14:39:11.588513+0000","last_undegraded":"2023-12-19T14:39:11.588513+0000","last_fullsized":"2023-12-19T14:39:11.588513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:16:19.999833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570611+0000","last_change":"2023-12-19T14:39:07.283321+0000","last_active":"2023-12-19T14:39:11.570611+0000","last_peered":"2023-12-19T14:39:11.570611+0000","last_clean":"2023-12-19T14:39:11.570611+0000","last_became_active":"2023-12-19T14:39:07.283017+0000","last_became_peered":"2023-12-19T14:39:07.283017+0000","last_unstale":"2023-12-19T14:39:11.570611+0000","last_undegraded":"2023-12-19T14:39:11.570611+0000","last_fullsized":"2023-12-19T14:39:11.570611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545060+0000","last_change":"2023-12-19T14:39:06.259282+0000","last_active":"2023-12-19T14:39:11.545060+0000","last_peered":"2023-12-19T14:39:11.545060+0000","last_clean":"2023-12-19T14:39:11.545060+0000","last_became_active":"2023-12-19T14:39:06.257316+0000","last_became_peered":"2023-12-19T14:39:06.257316+0000","last_unstale":"2023-12-19T14:39:11.545060+0000","last_undegraded":"2023-12-19T14:39:11.545060+0000","last_fullsized":"2023-12-19T14:39:11.545060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588628+0000","last_change":"2023-12-19T14:39:11.577972+0000","last_active":"2023-12-19T14:39:11.588628+0000","last_peered":"2023-12-19T14:39:11.588628+0000","last_clean":"2023-12-19T14:39:11.588628+0000","last_became_active":"2023-12-19T14:39:11.577529+0000","last_became_peered":"2023-12-19T14:39:11.577529+0000","last_unstale":"2023-12-19T14:39:11.588628+0000","last_undegraded":"2023-12-19T14:39:11.588628+0000","last_fullsized":"2023-12-19T14:39:11.588628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:19:59.360333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.557043+0000","last_change":"2023-12-19T14:39:07.289405+0000","last_active":"2023-12-19T14:39:11.557043+0000","last_peered":"2023-12-19T14:39:11.557043+0000","last_clean":"2023-12-19T14:39:11.557043+0000","last_became_active":"2023-12-19T14:39:07.288885+0000","last_became_peered":"2023-12-19T14:39:07.288885+0000","last_unstale":"2023-12-19T14:39:11.557043+0000","last_undegraded":"2023-12-19T14:39:11.557043+0000","last_fullsized":"2023-12-19T14:39:11.557043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559942+0000","last_change":"2023-12-19T14:39:06.256658+0000","last_active":"2023-12-19T14:39:11.559942+0000","last_peered":"2023-12-19T14:39:11.559942+0000","last_clean":"2023-12-19T14:39:11.559942+0000","last_became_active":"2023-12-19T14:39:06.255561+0000","last_became_peered":"2023-12-19T14:39:06.255561+0000","last_unstale":"2023-12-19T14:39:11.559942+0000","last_undegraded":"2023-12-19T14:39:11.559942+0000","last_fullsized":"2023-12-19T14:39:11.559942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584461+0000","last_change":"2023-12-19T14:39:11.579961+0000","last_active":"2023-12-19T14:39:11.584461+0000","last_peered":"2023-12-19T14:39:11.584461+0000","last_clean":"2023-12-19T14:39:11.584461+0000","last_became_active":"2023-12-19T14:39:11.578069+0000","last_became_peered":"2023-12-19T14:39:11.578069+0000","last_unstale":"2023-12-19T14:39:11.584461+0000","last_undegraded":"2023-12-19T14:39:11.584461+0000","last_fullsized":"2023-12-19T14:39:11.584461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570370+0000","last_change":"2023-12-19T14:39:07.288175+0000","last_active":"2023-12-19T14:39:11.570370+0000","last_peered":"2023-12-19T14:39:11.570370+0000","last_clean":"2023-12-19T14:39:11.570370+0000","last_became_active":"2023-12-19T14:39:07.287333+0000","last_became_peered":"2023-12-19T14:39:07.287333+0000","last_unstale":"2023-12-19T14:39:11.570370+0000","last_undegraded":"2023-12-19T14:39:11.570370+0000","last_fullsized":"2023-12-19T14:39:11.570370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T14:39:19.178278+0000","last_change":"2023-12-19T14:39:06.256219+0000","last_active":"2023-12-19T14:39:19.178278+0000","last_peered":"2023-12-19T14:39:19.178278+0000","last_clean":"2023-12-19T14:39:19.178278+0000","last_became_active":"2023-12-19T14:39:06.255840+0000","last_became_peered":"2023-12-19T14:39:06.255840+0000","last_unstale":"2023-12-19T14:39:19.178278+0000","last_undegraded":"2023-12-19T14:39:19.178278+0000","last_fullsized":"2023-12-19T14:39:19.178278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573334+0000","last_change":"2023-12-19T14:39:11.569520+0000","last_active":"2023-12-19T14:39:11.573334+0000","last_peered":"2023-12-19T14:39:11.573334+0000","last_clean":"2023-12-19T14:39:11.573334+0000","last_became_active":"2023-12-19T14:39:11.563543+0000","last_became_peered":"2023-12-19T14:39:11.563543+0000","last_unstale":"2023-12-19T14:39:11.573334+0000","last_undegraded":"2023-12-19T14:39:11.573334+0000","last_fullsized":"2023-12-19T14:39:11.573334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546941+0000","last_change":"2023-12-19T14:39:07.276126+0000","last_active":"2023-12-19T14:39:11.546941+0000","last_peered":"2023-12-19T14:39:11.546941+0000","last_clean":"2023-12-19T14:39:11.546941+0000","last_became_active":"2023-12-19T14:39:07.275186+0000","last_became_peered":"2023-12-19T14:39:07.275186+0000","last_unstale":"2023-12-19T14:39:11.546941+0000","last_undegraded":"2023-12-19T14:39:11.546941+0000","last_fullsized":"2023-12-19T14:39:11.546941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:40.362295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540550+0000","last_change":"2023-12-19T14:39:06.259793+0000","last_active":"2023-12-19T14:39:11.540550+0000","last_peered":"2023-12-19T14:39:11.540550+0000","last_clean":"2023-12-19T14:39:11.540550+0000","last_became_active":"2023-12-19T14:39:06.259474+0000","last_became_peered":"2023-12-19T14:39:06.259474+0000","last_unstale":"2023-12-19T14:39:11.540550+0000","last_undegraded":"2023-12-19T14:39:11.540550+0000","last_fullsized":"2023-12-19T14:39:11.540550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573690+0000","last_change":"2023-12-19T14:39:11.567330+0000","last_active":"2023-12-19T14:39:11.573690+0000","last_peered":"2023-12-19T14:39:11.573690+0000","last_clean":"2023-12-19T14:39:11.573690+0000","last_became_active":"2023-12-19T14:39:11.566909+0000","last_became_peered":"2023-12-19T14:39:11.566909+0000","last_unstale":"2023-12-19T14:39:11.573690+0000","last_undegraded":"2023-12-19T14:39:11.573690+0000","last_fullsized":"2023-12-19T14:39:11.573690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:41:18.238432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561959+0000","last_change":"2023-12-19T14:39:07.290356+0000","last_active":"2023-12-19T14:39:11.561959+0000","last_peered":"2023-12-19T14:39:11.561959+0000","last_clean":"2023-12-19T14:39:11.561959+0000","last_became_active":"2023-12-19T14:39:07.288476+0000","last_became_peered":"2023-12-19T14:39:07.288476+0000","last_unstale":"2023-12-19T14:39:11.561959+0000","last_undegraded":"2023-12-19T14:39:11.561959+0000","last_fullsized":"2023-12-19T14:39:11.561959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549225+0000","last_change":"2023-12-19T14:39:06.263437+0000","last_active":"2023-12-19T14:39:11.549225+0000","last_peered":"2023-12-19T14:39:11.549225+0000","last_clean":"2023-12-19T14:39:11.549225+0000","last_became_active":"2023-12-19T14:39:06.263146+0000","last_became_peered":"2023-12-19T14:39:06.263146+0000","last_unstale":"2023-12-19T14:39:11.549225+0000","last_undegraded":"2023-12-19T14:39:11.549225+0000","last_fullsized":"2023-12-19T14:39:11.549225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582385+0000","last_change":"2023-12-19T14:39:11.569957+0000","last_active":"2023-12-19T14:39:11.582385+0000","last_peered":"2023-12-19T14:39:11.582385+0000","last_clean":"2023-12-19T14:39:11.582385+0000","last_became_active":"2023-12-19T14:39:11.568502+0000","last_became_peered":"2023-12-19T14:39:11.568502+0000","last_unstale":"2023-12-19T14:39:11.582385+0000","last_undegraded":"2023-12-19T14:39:11.582385+0000","last_fullsized":"2023-12-19T14:39:11.582385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:02:34.258771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549124+0000","last_change":"2023-12-19T14:39:07.284419+0000","last_active":"2023-12-19T14:39:11.549124+0000","last_peered":"2023-12-19T14:39:11.549124+0000","last_clean":"2023-12-19T14:39:11.549124+0000","last_became_active":"2023-12-19T14:39:07.283507+0000","last_became_peered":"2023-12-19T14:39:07.283507+0000","last_unstale":"2023-12-19T14:39:11.549124+0000","last_undegraded":"2023-12-19T14:39:11.549124+0000","last_fullsized":"2023-12-19T14:39:11.549124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551787+0000","last_change":"2023-12-19T14:39:06.261357+0000","last_active":"2023-12-19T14:39:11.551787+0000","last_peered":"2023-12-19T14:39:11.551787+0000","last_clean":"2023-12-19T14:39:11.551787+0000","last_became_active":"2023-12-19T14:39:06.260833+0000","last_became_peered":"2023-12-19T14:39:06.260833+0000","last_unstale":"2023-12-19T14:39:11.551787+0000","last_undegraded":"2023-12-19T14:39:11.551787+0000","last_fullsized":"2023-12-19T14:39:11.551787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588646+0000","last_change":"2023-12-19T14:39:11.579827+0000","last_active":"2023-12-19T14:39:11.588646+0000","last_peered":"2023-12-19T14:39:11.588646+0000","last_clean":"2023-12-19T14:39:11.588646+0000","last_became_active":"2023-12-19T14:39:11.578792+0000","last_became_peered":"2023-12-19T14:39:11.578792+0000","last_unstale":"2023-12-19T14:39:11.588646+0000","last_undegraded":"2023-12-19T14:39:11.588646+0000","last_fullsized":"2023-12-19T14:39:11.588646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583746+0000","last_change":"2023-12-19T14:39:11.581965+0000","last_active":"2023-12-19T14:39:11.583746+0000","last_peered":"2023-12-19T14:39:11.583746+0000","last_clean":"2023-12-19T14:39:11.583746+0000","last_became_active":"2023-12-19T14:39:11.581412+0000","last_became_peered":"2023-12-19T14:39:11.581412+0000","last_unstale":"2023-12-19T14:39:11.583746+0000","last_undegraded":"2023-12-19T14:39:11.583746+0000","last_fullsized":"2023-12-19T14:39:11.583746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539409+0000","last_change":"2023-12-19T14:39:07.283081+0000","last_active":"2023-12-19T14:39:11.539409+0000","last_peered":"2023-12-19T14:39:11.539409+0000","last_clean":"2023-12-19T14:39:11.539409+0000","last_became_active":"2023-12-19T14:39:07.276260+0000","last_became_peered":"2023-12-19T14:39:07.276260+0000","last_unstale":"2023-12-19T14:39:11.539409+0000","last_undegraded":"2023-12-19T14:39:11.539409+0000","last_fullsized":"2023-12-19T14:39:11.539409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543780+0000","last_change":"2023-12-19T14:39:06.257897+0000","last_active":"2023-12-19T14:39:11.543780+0000","last_peered":"2023-12-19T14:39:11.543780+0000","last_clean":"2023-12-19T14:39:11.543780+0000","last_became_active":"2023-12-19T14:39:06.257281+0000","last_became_peered":"2023-12-19T14:39:06.257281+0000","last_unstale":"2023-12-19T14:39:11.543780+0000","last_undegraded":"2023-12-19T14:39:11.543780+0000","last_fullsized":"2023-12-19T14:39:11.543780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571142+0000","last_change":"2023-12-19T14:39:11.566055+0000","last_active":"2023-12-19T14:39:11.571142+0000","last_peered":"2023-12-19T14:39:11.571142+0000","last_clean":"2023-12-19T14:39:11.571142+0000","last_became_active":"2023-12-19T14:39:11.565416+0000","last_became_peered":"2023-12-19T14:39:11.565416+0000","last_unstale":"2023-12-19T14:39:11.571142+0000","last_undegraded":"2023-12-19T14:39:11.571142+0000","last_fullsized":"2023-12-19T14:39:11.571142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:12:19.857166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545049+0000","last_change":"2023-12-19T14:39:07.290776+0000","last_active":"2023-12-19T14:39:11.545049+0000","last_peered":"2023-12-19T14:39:11.545049+0000","last_clean":"2023-12-19T14:39:11.545049+0000","last_became_active":"2023-12-19T14:39:07.290465+0000","last_became_peered":"2023-12-19T14:39:07.290465+0000","last_unstale":"2023-12-19T14:39:11.545049+0000","last_undegraded":"2023-12-19T14:39:11.545049+0000","last_fullsized":"2023-12-19T14:39:11.545049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543823+0000","last_change":"2023-12-19T14:39:06.259345+0000","last_active":"2023-12-19T14:39:11.543823+0000","last_peered":"2023-12-19T14:39:11.543823+0000","last_clean":"2023-12-19T14:39:11.543823+0000","last_became_active":"2023-12-19T14:39:06.259067+0000","last_became_peered":"2023-12-19T14:39:06.259067+0000","last_unstale":"2023-12-19T14:39:11.543823+0000","last_undegraded":"2023-12-19T14:39:11.543823+0000","last_fullsized":"2023-12-19T14:39:11.543823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550184+0000","last_change":"2023-12-19T14:39:06.260437+0000","last_active":"2023-12-19T14:39:11.550184+0000","last_peered":"2023-12-19T14:39:11.550184+0000","last_clean":"2023-12-19T14:39:11.550184+0000","last_became_active":"2023-12-19T14:39:06.260008+0000","last_became_peered":"2023-12-19T14:39:06.260008+0000","last_unstale":"2023-12-19T14:39:11.550184+0000","last_undegraded":"2023-12-19T14:39:11.550184+0000","last_fullsized":"2023-12-19T14:39:11.550184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559895+0000","last_change":"2023-12-19T14:39:07.288421+0000","last_active":"2023-12-19T14:39:11.559895+0000","last_peered":"2023-12-19T14:39:11.559895+0000","last_clean":"2023-12-19T14:39:11.559895+0000","last_became_active":"2023-12-19T14:39:07.287403+0000","last_became_peered":"2023-12-19T14:39:07.287403+0000","last_unstale":"2023-12-19T14:39:11.559895+0000","last_undegraded":"2023-12-19T14:39:11.559895+0000","last_fullsized":"2023-12-19T14:39:11.559895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581278+0000","last_change":"2023-12-19T14:39:11.577684+0000","last_active":"2023-12-19T14:39:11.581278+0000","last_peered":"2023-12-19T14:39:11.581278+0000","last_clean":"2023-12-19T14:39:11.581278+0000","last_became_active":"2023-12-19T14:39:11.577068+0000","last_became_peered":"2023-12-19T14:39:11.577068+0000","last_unstale":"2023-12-19T14:39:11.581278+0000","last_undegraded":"2023-12-19T14:39:11.581278+0000","last_fullsized":"2023-12-19T14:39:11.581278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:42.373695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544048+0000","last_change":"2023-12-19T14:39:06.247049+0000","last_active":"2023-12-19T14:39:11.544048+0000","last_peered":"2023-12-19T14:39:11.544048+0000","last_clean":"2023-12-19T14:39:11.544048+0000","last_became_active":"2023-12-19T14:39:06.246716+0000","last_became_peered":"2023-12-19T14:39:06.246716+0000","last_unstale":"2023-12-19T14:39:11.544048+0000","last_undegraded":"2023-12-19T14:39:11.544048+0000","last_fullsized":"2023-12-19T14:39:11.544048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546858+0000","last_change":"2023-12-19T14:39:07.266860+0000","last_active":"2023-12-19T14:39:11.546858+0000","last_peered":"2023-12-19T14:39:11.546858+0000","last_clean":"2023-12-19T14:39:11.546858+0000","last_became_active":"2023-12-19T14:39:07.266544+0000","last_became_peered":"2023-12-19T14:39:07.266544+0000","last_unstale":"2023-12-19T14:39:11.546858+0000","last_undegraded":"2023-12-19T14:39:11.546858+0000","last_fullsized":"2023-12-19T14:39:11.546858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:47.972055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.585215+0000","last_change":"2023-12-19T14:39:11.567865+0000","last_active":"2023-12-19T14:39:11.585215+0000","last_peered":"2023-12-19T14:39:11.585215+0000","last_clean":"2023-12-19T14:39:11.585215+0000","last_became_active":"2023-12-19T14:39:11.566466+0000","last_became_peered":"2023-12-19T14:39:11.566466+0000","last_unstale":"2023-12-19T14:39:11.585215+0000","last_undegraded":"2023-12-19T14:39:11.585215+0000","last_fullsized":"2023-12-19T14:39:11.585215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:11.949761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551612+0000","last_change":"2023-12-19T14:39:07.289708+0000","last_active":"2023-12-19T14:39:11.551612+0000","last_peered":"2023-12-19T14:39:11.551612+0000","last_clean":"2023-12-19T14:39:11.551612+0000","last_became_active":"2023-12-19T14:39:07.289420+0000","last_became_peered":"2023-12-19T14:39:07.289420+0000","last_unstale":"2023-12-19T14:39:11.551612+0000","last_undegraded":"2023-12-19T14:39:11.551612+0000","last_fullsized":"2023-12-19T14:39:11.551612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.552204+0000","last_change":"2023-12-19T14:39:07.291613+0000","last_active":"2023-12-19T14:39:11.552204+0000","last_peered":"2023-12-19T14:39:11.552204+0000","last_clean":"2023-12-19T14:39:11.552204+0000","last_became_active":"2023-12-19T14:39:07.291277+0000","last_became_peered":"2023-12-19T14:39:07.291277+0000","last_unstale":"2023-12-19T14:39:11.552204+0000","last_undegraded":"2023-12-19T14:39:11.552204+0000","last_fullsized":"2023-12-19T14:39:11.552204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:23.914755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544957+0000","last_change":"2023-12-19T14:39:07.294351+0000","last_active":"2023-12-19T14:39:11.544957+0000","last_peered":"2023-12-19T14:39:11.544957+0000","last_clean":"2023-12-19T14:39:11.544957+0000","last_became_active":"2023-12-19T14:39:07.293414+0000","last_became_peered":"2023-12-19T14:39:07.293414+0000","last_unstale":"2023-12-19T14:39:11.544957+0000","last_undegraded":"2023-12-19T14:39:11.544957+0000","last_fullsized":"2023-12-19T14:39:11.544957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559855+0000","last_change":"2023-12-19T14:39:07.287045+0000","last_active":"2023-12-19T14:39:11.559855+0000","last_peered":"2023-12-19T14:39:11.559855+0000","last_clean":"2023-12-19T14:39:11.559855+0000","last_became_active":"2023-12-19T14:39:07.286711+0000","last_became_peered":"2023-12-19T14:39:07.286711+0000","last_unstale":"2023-12-19T14:39:11.559855+0000","last_undegraded":"2023-12-19T14:39:11.559855+0000","last_fullsized":"2023-12-19T14:39:11.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545892+0000","last_change":"2023-12-19T14:39:07.284424+0000","last_active":"2023-12-19T14:39:11.545892+0000","last_peered":"2023-12-19T14:39:11.545892+0000","last_clean":"2023-12-19T14:39:11.545892+0000","last_became_active":"2023-12-19T14:39:07.284121+0000","last_became_peered":"2023-12-19T14:39:07.284121+0000","last_unstale":"2023-12-19T14:39:11.545892+0000","last_undegraded":"2023-12-19T14:39:11.545892+0000","last_fullsized":"2023-12-19T14:39:11.545892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540559+0000","last_change":"2023-12-19T14:39:07.280745+0000","last_active":"2023-12-19T14:39:11.540559+0000","last_peered":"2023-12-19T14:39:11.540559+0000","last_clean":"2023-12-19T14:39:11.540559+0000","last_became_active":"2023-12-19T14:39:07.280443+0000","last_became_peered":"2023-12-19T14:39:07.280443+0000","last_unstale":"2023-12-19T14:39:11.540559+0000","last_undegraded":"2023-12-19T14:39:11.540559+0000","last_fullsized":"2023-12-19T14:39:11.540559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550784+0000","last_change":"2023-12-19T14:39:07.288049+0000","last_active":"2023-12-19T14:39:11.550784+0000","last_peered":"2023-12-19T14:39:11.550784+0000","last_clean":"2023-12-19T14:39:11.550784+0000","last_became_active":"2023-12-19T14:39:07.287758+0000","last_became_peered":"2023-12-19T14:39:07.287758+0000","last_unstale":"2023-12-19T14:39:11.550784+0000","last_undegraded":"2023-12-19T14:39:11.550784+0000","last_fullsized":"2023-12-19T14:39:11.550784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.558076+0000","last_change":"2023-12-19T14:39:07.284681+0000","last_active":"2023-12-19T14:39:11.558076+0000","last_peered":"2023-12-19T14:39:11.558076+0000","last_clean":"2023-12-19T14:39:11.558076+0000","last_became_active":"2023-12-19T14:39:07.283596+0000","last_became_peered":"2023-12-19T14:39:07.283596+0000","last_unstale":"2023-12-19T14:39:11.558076+0000","last_undegraded":"2023-12-19T14:39:11.558076+0000","last_fullsized":"2023-12-19T14:39:11.558076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545014+0000","last_change":"2023-12-19T14:39:07.293485+0000","last_active":"2023-12-19T14:39:11.545014+0000","last_peered":"2023-12-19T14:39:11.545014+0000","last_clean":"2023-12-19T14:39:11.545014+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.545014+0000","last_undegraded":"2023-12-19T14:39:11.545014+0000","last_fullsized":"2023-12-19T14:39:11.545014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543694+0000","last_change":"2023-12-19T14:39:07.283853+0000","last_active":"2023-12-19T14:39:11.543694+0000","last_peered":"2023-12-19T14:39:11.543694+0000","last_clean":"2023-12-19T14:39:11.543694+0000","last_became_active":"2023-12-19T14:39:07.283616+0000","last_became_peered":"2023-12-19T14:39:07.283616+0000","last_unstale":"2023-12-19T14:39:11.543694+0000","last_undegraded":"2023-12-19T14:39:11.543694+0000","last_fullsized":"2023-12-19T14:39:11.543694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544936+0000","last_change":"2023-12-19T14:39:07.294504+0000","last_active":"2023-12-19T14:39:11.544936+0000","last_peered":"2023-12-19T14:39:11.544936+0000","last_clean":"2023-12-19T14:39:11.544936+0000","last_became_active":"2023-12-19T14:39:07.293953+0000","last_became_peered":"2023-12-19T14:39:07.293953+0000","last_unstale":"2023-12-19T14:39:11.544936+0000","last_undegraded":"2023-12-19T14:39:11.544936+0000","last_fullsized":"2023-12-19T14:39:11.544936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559798+0000","last_change":"2023-12-19T14:39:07.296762+0000","last_active":"2023-12-19T14:39:11.559798+0000","last_peered":"2023-12-19T14:39:11.559798+0000","last_clean":"2023-12-19T14:39:11.559798+0000","last_became_active":"2023-12-19T14:39:07.295928+0000","last_became_peered":"2023-12-19T14:39:07.295928+0000","last_unstale":"2023-12-19T14:39:11.559798+0000","last_undegraded":"2023-12-19T14:39:11.559798+0000","last_fullsized":"2023-12-19T14:39:11.559798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549583+0000","last_change":"2023-12-19T14:39:07.280149+0000","last_active":"2023-12-19T14:39:11.549583+0000","last_peered":"2023-12-19T14:39:11.549583+0000","last_clean":"2023-12-19T14:39:11.549583+0000","last_became_active":"2023-12-19T14:39:07.279877+0000","last_became_peered":"2023-12-19T14:39:07.279877+0000","last_unstale":"2023-12-19T14:39:11.549583+0000","last_undegraded":"2023-12-19T14:39:11.549583+0000","last_fullsized":"2023-12-19T14:39:11.549583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:59:49.003736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570730+0000","last_change":"2023-12-19T14:39:07.288397+0000","last_active":"2023-12-19T14:39:11.570730+0000","last_peered":"2023-12-19T14:39:11.570730+0000","last_clean":"2023-12-19T14:39:11.570730+0000","last_became_active":"2023-12-19T14:39:07.287749+0000","last_became_peered":"2023-12-19T14:39:07.287749+0000","last_unstale":"2023-12-19T14:39:11.570730+0000","last_undegraded":"2023-12-19T14:39:11.570730+0000","last_fullsized":"2023-12-19T14:39:11.570730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555145+0000","last_change":"2023-12-19T14:39:07.276259+0000","last_active":"2023-12-19T14:39:11.555145+0000","last_peered":"2023-12-19T14:39:11.555145+0000","last_clean":"2023-12-19T14:39:11.555145+0000","last_became_active":"2023-12-19T14:39:07.275951+0000","last_became_peered":"2023-12-19T14:39:07.275951+0000","last_unstale":"2023-12-19T14:39:11.555145+0000","last_undegraded":"2023-12-19T14:39:11.555145+0000","last_fullsized":"2023-12-19T14:39:11.555145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549564+0000","last_change":"2023-12-19T14:39:07.295581+0000","last_active":"2023-12-19T14:39:11.549564+0000","last_peered":"2023-12-19T14:39:11.549564+0000","last_clean":"2023-12-19T14:39:11.549564+0000","last_became_active":"2023-12-19T14:39:07.293567+0000","last_became_peered":"2023-12-19T14:39:07.293567+0000","last_unstale":"2023-12-19T14:39:11.549564+0000","last_undegraded":"2023-12-19T14:39:11.549564+0000","last_fullsized":"2023-12-19T14:39:11.549564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543139+0000","last_change":"2023-12-19T14:39:07.293875+0000","last_active":"2023-12-19T14:39:11.543139+0000","last_peered":"2023-12-19T14:39:11.543139+0000","last_clean":"2023-12-19T14:39:11.543139+0000","last_became_active":"2023-12-19T14:39:07.293381+0000","last_became_peered":"2023-12-19T14:39:07.293381+0000","last_unstale":"2023-12-19T14:39:11.543139+0000","last_undegraded":"2023-12-19T14:39:11.543139+0000","last_fullsized":"2023-12-19T14:39:11.543139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540124+0000","last_change":"2023-12-19T14:39:07.261761+0000","last_active":"2023-12-19T14:39:11.540124+0000","last_peered":"2023-12-19T14:39:11.540124+0000","last_clean":"2023-12-19T14:39:11.540124+0000","last_became_active":"2023-12-19T14:39:07.261446+0000","last_became_peered":"2023-12-19T14:39:07.261446+0000","last_unstale":"2023-12-19T14:39:11.540124+0000","last_undegraded":"2023-12-19T14:39:11.540124+0000","last_fullsized":"2023-12-19T14:39:11.540124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556947+0000","last_change":"2023-12-19T14:39:07.290139+0000","last_active":"2023-12-19T14:39:11.556947+0000","last_peered":"2023-12-19T14:39:11.556947+0000","last_clean":"2023-12-19T14:39:11.556947+0000","last_became_active":"2023-12-19T14:39:07.289843+0000","last_became_peered":"2023-12-19T14:39:07.289843+0000","last_unstale":"2023-12-19T14:39:11.556947+0000","last_undegraded":"2023-12-19T14:39:11.556947+0000","last_fullsized":"2023-12-19T14:39:11.556947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543821+0000","last_change":"2023-12-19T14:39:07.281738+0000","last_active":"2023-12-19T14:39:11.543821+0000","last_peered":"2023-12-19T14:39:11.543821+0000","last_clean":"2023-12-19T14:39:11.543821+0000","last_became_active":"2023-12-19T14:39:07.281443+0000","last_became_peered":"2023-12-19T14:39:07.281443+0000","last_unstale":"2023-12-19T14:39:11.543821+0000","last_undegraded":"2023-12-19T14:39:11.543821+0000","last_fullsized":"2023-12-19T14:39:11.543821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546820+0000","last_change":"2023-12-19T14:39:07.276674+0000","last_active":"2023-12-19T14:39:11.546820+0000","last_peered":"2023-12-19T14:39:11.546820+0000","last_clean":"2023-12-19T14:39:11.546820+0000","last_became_active":"2023-12-19T14:39:07.276368+0000","last_became_peered":"2023-12-19T14:39:07.276368+0000","last_unstale":"2023-12-19T14:39:11.546820+0000","last_undegraded":"2023-12-19T14:39:11.546820+0000","last_fullsized":"2023-12-19T14:39:11.546820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:35.210805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549881+0000","last_change":"2023-12-19T14:39:07.281026+0000","last_active":"2023-12-19T14:39:11.549881+0000","last_peered":"2023-12-19T14:39:11.549881+0000","last_clean":"2023-12-19T14:39:11.549881+0000","last_became_active":"2023-12-19T14:39:07.280689+0000","last_became_peered":"2023-12-19T14:39:07.280689+0000","last_unstale":"2023-12-19T14:39:11.549881+0000","last_undegraded":"2023-12-19T14:39:11.549881+0000","last_fullsized":"2023-12-19T14:39:11.549881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550701+0000","last_change":"2023-12-19T14:39:07.280698+0000","last_active":"2023-12-19T14:39:11.550701+0000","last_peered":"2023-12-19T14:39:11.550701+0000","last_clean":"2023-12-19T14:39:11.550701+0000","last_became_active":"2023-12-19T14:39:07.280406+0000","last_became_peered":"2023-12-19T14:39:07.280406+0000","last_unstale":"2023-12-19T14:39:11.550701+0000","last_undegraded":"2023-12-19T14:39:11.550701+0000","last_fullsized":"2023-12-19T14:39:11.550701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570926+0000","last_change":"2023-12-19T14:39:07.284712+0000","last_active":"2023-12-19T14:39:11.570926+0000","last_peered":"2023-12-19T14:39:11.570926+0000","last_clean":"2023-12-19T14:39:11.570926+0000","last_became_active":"2023-12-19T14:39:07.283598+0000","last_became_peered":"2023-12-19T14:39:07.283598+0000","last_unstale":"2023-12-19T14:39:11.570926+0000","last_undegraded":"2023-12-19T14:39:11.570926+0000","last_fullsized":"2023-12-19T14:39:11.570926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544886+0000","last_change":"2023-12-19T14:39:07.294169+0000","last_active":"2023-12-19T14:39:11.544886+0000","last_peered":"2023-12-19T14:39:11.544886+0000","last_clean":"2023-12-19T14:39:11.544886+0000","last_became_active":"2023-12-19T14:39:07.293746+0000","last_became_peered":"2023-12-19T14:39:07.293746+0000","last_unstale":"2023-12-19T14:39:11.544886+0000","last_undegraded":"2023-12-19T14:39:11.544886+0000","last_fullsized":"2023-12-19T14:39:11.544886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570207+0000","last_change":"2023-12-19T14:39:07.291661+0000","last_active":"2023-12-19T14:39:11.570207+0000","last_peered":"2023-12-19T14:39:11.570207+0000","last_clean":"2023-12-19T14:39:11.570207+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.570207+0000","last_undegraded":"2023-12-19T14:39:11.570207+0000","last_fullsized":"2023-12-19T14:39:11.570207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:34:24.416865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544859+0000","last_change":"2023-12-19T14:39:07.297368+0000","last_active":"2023-12-19T14:39:11.544859+0000","last_peered":"2023-12-19T14:39:11.544859+0000","last_clean":"2023-12-19T14:39:11.544859+0000","last_became_active":"2023-12-19T14:39:07.297082+0000","last_became_peered":"2023-12-19T14:39:07.297082+0000","last_unstale":"2023-12-19T14:39:11.544859+0000","last_undegraded":"2023-12-19T14:39:11.544859+0000","last_fullsized":"2023-12-19T14:39:11.544859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570469+0000","last_change":"2023-12-19T14:39:07.287921+0000","last_active":"2023-12-19T14:39:11.570469+0000","last_peered":"2023-12-19T14:39:11.570469+0000","last_clean":"2023-12-19T14:39:11.570469+0000","last_became_active":"2023-12-19T14:39:07.287493+0000","last_became_peered":"2023-12-19T14:39:07.287493+0000","last_unstale":"2023-12-19T14:39:11.570469+0000","last_undegraded":"2023-12-19T14:39:11.570469+0000","last_fullsized":"2023-12-19T14:39:11.570469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545732+0000","last_change":"2023-12-19T14:39:07.290935+0000","last_active":"2023-12-19T14:39:11.545732+0000","last_peered":"2023-12-19T14:39:11.545732+0000","last_clean":"2023-12-19T14:39:11.545732+0000","last_became_active":"2023-12-19T14:39:07.290614+0000","last_became_peered":"2023-12-19T14:39:07.290614+0000","last_unstale":"2023-12-19T14:39:11.545732+0000","last_undegraded":"2023-12-19T14:39:11.545732+0000","last_fullsized":"2023-12-19T14:39:11.545732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.560160+0000","last_change":"2023-12-19T14:39:07.287280+0000","last_active":"2023-12-19T14:39:11.560160+0000","last_peered":"2023-12-19T14:39:11.560160+0000","last_clean":"2023-12-19T14:39:11.560160+0000","last_became_active":"2023-12-19T14:39:07.286949+0000","last_became_peered":"2023-12-19T14:39:07.286949+0000","last_unstale":"2023-12-19T14:39:11.560160+0000","last_undegraded":"2023-12-19T14:39:11.560160+0000","last_fullsized":"2023-12-19T14:39:11.560160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540585+0000","last_change":"2023-12-19T14:39:07.295118+0000","last_active":"2023-12-19T14:39:11.540585+0000","last_peered":"2023-12-19T14:39:11.540585+0000","last_clean":"2023-12-19T14:39:11.540585+0000","last_became_active":"2023-12-19T14:39:07.294282+0000","last_became_peered":"2023-12-19T14:39:07.294282+0000","last_unstale":"2023-12-19T14:39:11.540585+0000","last_undegraded":"2023-12-19T14:39:11.540585+0000","last_fullsized":"2023-12-19T14:39:11.540585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570810+0000","last_change":"2023-12-19T14:39:07.283979+0000","last_active":"2023-12-19T14:39:11.570810+0000","last_peered":"2023-12-19T14:39:11.570810+0000","last_clean":"2023-12-19T14:39:11.570810+0000","last_became_active":"2023-12-19T14:39:07.280235+0000","last_became_peered":"2023-12-19T14:39:07.280235+0000","last_unstale":"2023-12-19T14:39:11.570810+0000","last_undegraded":"2023-12-19T14:39:11.570810+0000","last_fullsized":"2023-12-19T14:39:11.570810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9910000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301628,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070212,"statfs":{"total":96636764160,"available":96327897088,"internally_reserved":0,"allocated":249856,"data_stored":98477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302220,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069620,"statfs":{"total":96636764160,"available":96327290880,"internally_reserved":0,"allocated":856064,"data_stored":689455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.298}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3730000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:39:22.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:39:23.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 Processor -- start 2023-12-19T14:39:23.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 -- start start 2023-12-19T14:39:23.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8eb80b1410 0x7f8eb80ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 0x7f8eb806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb815a860 0x7f8eb815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb806af40 con 0x7f8eb80b1410 2023-12-19T14:39:23.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb806c7c0 con 0x7f8eb806a580 2023-12-19T14:39:23.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb806cd80 con 0x7f8eb815a860 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8eb80b1410 0x7f8eb80ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8eb80b1410 0x7f8eb80ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53592/0 (socket says 172.21.15.50:53592) 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 -- 172.21.15.50:0/1450538755 learned_addr learned my addr 172.21.15.50:0/1450538755 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb5d9b700 1 --2- 172.21.15.50:0/1450538755 >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 0x7f8eb806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb6d9d700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb815a860 0x7f8eb815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 -- 172.21.15.50:0/1450538755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb815a860 msgr2=0x7f8eb815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb815a860 0x7f8eb815ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 -- 172.21.15.50:0/1450538755 >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 msgr2=0x7f8eb806aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 --2- 172.21.15.50:0/1450538755 >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 0x7f8eb806aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 -- 172.21.15.50:0/1450538755 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb806b8f0 con 0x7f8eb80b1410 2023-12-19T14:39:23.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb5d9b700 1 --2- 172.21.15.50:0/1450538755 >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 0x7f8eb806aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb659c700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb80b1410 0x7f8eb80ab440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8068040 tx=0x7f8ea8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4d7ccfab490ca server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.073+0000 7f8eb559a700 1 -- 172.21.15.50:0/1450538755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800b860 con 0x7f8eb80b1410 2023-12-19T14:39:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb559a700 1 -- 172.21.15.50:0/1450538755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea800b9c0 con 0x7f8eb80b1410 2023-12-19T14:39:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb559a700 1 -- 172.21.15.50:0/1450538755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800bd60 con 0x7f8eb80b1410 2023-12-19T14:39:23.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 -- 172.21.15.50:0/1450538755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb80b1410 msgr2=0x7f8eb80ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb80b1410 0x7f8eb80ab440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8068040 tx=0x7f8ea8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:23.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 -- 172.21.15.50:0/1450538755 shutdown_connections 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb815a860 0x7f8eb815ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 --2- 172.21.15.50:0/1450538755 >> v2:172.21.15.125:3300/0 conn(0x7f8eb806a580 0x7f8eb806aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 --2- 172.21.15.50:0/1450538755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb80b1410 0x7f8eb80ab440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 -- 172.21.15.50:0/1450538755 >> 172.21.15.50:0/1450538755 conn(0x7f8eb8067bb0 msgr2=0x7f8eb80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 -- 172.21.15.50:0/1450538755 shutdown_connections 2023-12-19T14:39:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.077+0000 7f8eb759e700 1 -- 172.21.15.50:0/1450538755 wait complete. 2023-12-19T14:39:23.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 Processor -- start 2023-12-19T14:39:23.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 -- start start 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8eb806a580 0x7f8eb8146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 0x7f8eb815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80c8710 con 0x7f8eb815a860 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8146cb0 con 0x7f8eb806a580 2023-12-19T14:39:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb81563b0 con 0x7f8eb80b1410 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53602/0 (socket says 172.21.15.50:53602) 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 -- 172.21.15.50:0/3757415320 learned_addr learned my addr 172.21.15.50:0/3757415320 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb5d9b700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 0x7f8eb815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb659c700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8eb806a580 0x7f8eb8146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 -- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 msgr2=0x7f8eb815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 0x7f8eb815b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 -- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8eb806a580 msgr2=0x7f8eb8146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8eb806a580 0x7f8eb8146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 -- 172.21.15.50:0/3757415320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea80094f0 con 0x7f8eb815a860 2023-12-19T14:39:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb5d9b700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 0x7f8eb815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.081+0000 7f8eb6d9d700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009fd0 tx=0x7f8eb0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000eca0 con 0x7f8eb815a860 2023-12-19T14:39:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb000ee00 con 0x7f8eb815a860 2023-12-19T14:39:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb80c7b00 con 0x7f8eb815a860 2023-12-19T14:39:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0014670 con 0x7f8eb815a860 2023-12-19T14:39:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb80c80c0 con 0x7f8eb815a860 2023-12-19T14:39:23.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb804be40 con 0x7f8eb815a860 2023-12-19T14:39:23.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.085+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8eb00149e0 con 0x7f8eb815a860 2023-12-19T14:39:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.089+0000 7f8ea67fc700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 0x7f8e98062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.089+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8eb007d240 con 0x7f8eb815a860 2023-12-19T14:39:23.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.089+0000 7f8eb659c700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 0x7f8e98062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.093+0000 7f8eb659c700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 0x7f8e98062a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8068040 tx=0x7f8ea8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.101+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8eb004d4a0 con 0x7f8eb815a860 2023-12-19T14:39:23.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.265+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8eb80ab220 con 0x7f8e98060550 2023-12-19T14:39:23.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.285+0000 7f8ea67fc700 1 -- 172.21.15.50:0/3757415320 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345091 (secure 0 0 0) 0x7f8eb80ab220 con 0x7f8e98060550 2023-12-19T14:39:23.298 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:23.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 msgr2=0x7f8e98062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 0x7f8e98062a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8068040 tx=0x7f8ea8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:23.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 msgr2=0x7f8eb8157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009fd0 tx=0x7f8eb0009900 comp rx=0 tx=0).stop 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 shutdown_connections 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e98060550 0x7f8e98062a00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8eb815a860 0x7f8eb8157320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8eb80b1410 0x7f8eb815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 --2- 172.21.15.50:0/3757415320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8eb806a580 0x7f8eb8146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 >> 172.21.15.50:0/3757415320 conn(0x7f8eb8067bb0 msgr2=0x7f8eb80adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 shutdown_connections 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.293+0000 7f8eb759e700 1 -- 172.21.15.50:0/3757415320 wait complete. 2023-12-19T14:39:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:39:23.321 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-12-19T14:39:22.132391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2393792,"kb_used_data":3200,"kb_used_omap":0,"kb_used_meta":2390528,"kb_avail":752580928,"statfs":{"total":773094113280,"available":770642870272,"internally_reserved":0,"allocated":3276800,"data_stored":1983584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2447900672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003428"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562203+0000","last_change":"2023-12-19T14:39:06.261176+0000","last_active":"2023-12-19T14:39:11.562203+0000","last_peered":"2023-12-19T14:39:11.562203+0000","last_clean":"2023-12-19T14:39:11.562203+0000","last_became_active":"2023-12-19T14:39:06.259857+0000","last_became_peered":"2023-12-19T14:39:06.259857+0000","last_unstale":"2023-12-19T14:39:11.562203+0000","last_undegraded":"2023-12-19T14:39:11.562203+0000","last_fullsized":"2023-12-19T14:39:11.562203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:32:43.075778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545898+0000","last_change":"2023-12-19T14:39:07.280964+0000","last_active":"2023-12-19T14:39:11.545898+0000","last_peered":"2023-12-19T14:39:11.545898+0000","last_clean":"2023-12-19T14:39:11.545898+0000","last_became_active":"2023-12-19T14:39:07.280410+0000","last_became_peered":"2023-12-19T14:39:07.280410+0000","last_unstale":"2023-12-19T14:39:11.545898+0000","last_undegraded":"2023-12-19T14:39:11.545898+0000","last_fullsized":"2023-12-19T14:39:11.545898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588669+0000","last_change":"2023-12-19T14:39:11.580119+0000","last_active":"2023-12-19T14:39:11.588669+0000","last_peered":"2023-12-19T14:39:11.588669+0000","last_clean":"2023-12-19T14:39:11.588669+0000","last_became_active":"2023-12-19T14:39:11.579405+0000","last_became_peered":"2023-12-19T14:39:11.579405+0000","last_unstale":"2023-12-19T14:39:11.588669+0000","last_undegraded":"2023-12-19T14:39:11.588669+0000","last_fullsized":"2023-12-19T14:39:11.588669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546035+0000","last_change":"2023-12-19T14:39:06.252186+0000","last_active":"2023-12-19T14:39:11.546035+0000","last_peered":"2023-12-19T14:39:11.546035+0000","last_clean":"2023-12-19T14:39:11.546035+0000","last_became_active":"2023-12-19T14:39:06.251869+0000","last_became_peered":"2023-12-19T14:39:06.251869+0000","last_unstale":"2023-12-19T14:39:11.546035+0000","last_undegraded":"2023-12-19T14:39:11.546035+0000","last_fullsized":"2023-12-19T14:39:11.546035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561557+0000","last_change":"2023-12-19T14:39:07.289142+0000","last_active":"2023-12-19T14:39:11.561557+0000","last_peered":"2023-12-19T14:39:11.561557+0000","last_clean":"2023-12-19T14:39:11.561557+0000","last_became_active":"2023-12-19T14:39:07.288822+0000","last_became_peered":"2023-12-19T14:39:07.288822+0000","last_unstale":"2023-12-19T14:39:11.561557+0000","last_undegraded":"2023-12-19T14:39:11.561557+0000","last_fullsized":"2023-12-19T14:39:11.561557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582458+0000","last_change":"2023-12-19T14:39:11.573848+0000","last_active":"2023-12-19T14:39:11.582458+0000","last_peered":"2023-12-19T14:39:11.582458+0000","last_clean":"2023-12-19T14:39:11.582458+0000","last_became_active":"2023-12-19T14:39:11.573088+0000","last_became_peered":"2023-12-19T14:39:11.573088+0000","last_unstale":"2023-12-19T14:39:11.582458+0000","last_undegraded":"2023-12-19T14:39:11.582458+0000","last_fullsized":"2023-12-19T14:39:11.582458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:29.985849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570627+0000","last_change":"2023-12-19T14:39:06.262908+0000","last_active":"2023-12-19T14:39:11.570627+0000","last_peered":"2023-12-19T14:39:11.570627+0000","last_clean":"2023-12-19T14:39:11.570627+0000","last_became_active":"2023-12-19T14:39:06.260114+0000","last_became_peered":"2023-12-19T14:39:06.260114+0000","last_unstale":"2023-12-19T14:39:11.570627+0000","last_undegraded":"2023-12-19T14:39:11.570627+0000","last_fullsized":"2023-12-19T14:39:11.570627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545839+0000","last_change":"2023-12-19T14:39:07.279790+0000","last_active":"2023-12-19T14:39:11.545839+0000","last_peered":"2023-12-19T14:39:11.545839+0000","last_clean":"2023-12-19T14:39:11.545839+0000","last_became_active":"2023-12-19T14:39:07.279497+0000","last_became_peered":"2023-12-19T14:39:07.279497+0000","last_unstale":"2023-12-19T14:39:11.545839+0000","last_undegraded":"2023-12-19T14:39:11.545839+0000","last_fullsized":"2023-12-19T14:39:11.545839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.574367+0000","last_change":"2023-12-19T14:39:11.571054+0000","last_active":"2023-12-19T14:39:11.574367+0000","last_peered":"2023-12-19T14:39:11.574367+0000","last_clean":"2023-12-19T14:39:11.574367+0000","last_became_active":"2023-12-19T14:39:11.570620+0000","last_became_peered":"2023-12-19T14:39:11.570620+0000","last_unstale":"2023-12-19T14:39:11.574367+0000","last_undegraded":"2023-12-19T14:39:11.574367+0000","last_fullsized":"2023-12-19T14:39:11.574367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:06:34.827613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570831+0000","last_change":"2023-12-19T14:39:06.260028+0000","last_active":"2023-12-19T14:39:11.570831+0000","last_peered":"2023-12-19T14:39:11.570831+0000","last_clean":"2023-12-19T14:39:11.570831+0000","last_became_active":"2023-12-19T14:39:06.259269+0000","last_became_peered":"2023-12-19T14:39:06.259269+0000","last_unstale":"2023-12-19T14:39:11.570831+0000","last_undegraded":"2023-12-19T14:39:11.570831+0000","last_fullsized":"2023-12-19T14:39:11.570831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:29.156364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548594+0000","last_change":"2023-12-19T14:39:07.282662+0000","last_active":"2023-12-19T14:39:11.548594+0000","last_peered":"2023-12-19T14:39:11.548594+0000","last_clean":"2023-12-19T14:39:11.548594+0000","last_became_active":"2023-12-19T14:39:07.282105+0000","last_became_peered":"2023-12-19T14:39:07.282105+0000","last_unstale":"2023-12-19T14:39:11.548594+0000","last_undegraded":"2023-12-19T14:39:11.548594+0000","last_fullsized":"2023-12-19T14:39:11.548594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583605+0000","last_change":"2023-12-19T14:39:11.578372+0000","last_active":"2023-12-19T14:39:11.583605+0000","last_peered":"2023-12-19T14:39:11.583605+0000","last_clean":"2023-12-19T14:39:11.583605+0000","last_became_active":"2023-12-19T14:39:11.575437+0000","last_became_peered":"2023-12-19T14:39:11.575437+0000","last_unstale":"2023-12-19T14:39:11.583605+0000","last_undegraded":"2023-12-19T14:39:11.583605+0000","last_fullsized":"2023-12-19T14:39:11.583605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:01.783711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550051+0000","last_change":"2023-12-19T14:39:07.274147+0000","last_active":"2023-12-19T14:39:11.550051+0000","last_peered":"2023-12-19T14:39:11.550051+0000","last_clean":"2023-12-19T14:39:11.550051+0000","last_became_active":"2023-12-19T14:39:07.273776+0000","last_became_peered":"2023-12-19T14:39:07.273776+0000","last_unstale":"2023-12-19T14:39:11.550051+0000","last_undegraded":"2023-12-19T14:39:11.550051+0000","last_fullsized":"2023-12-19T14:39:11.550051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570793+0000","last_change":"2023-12-19T14:39:06.253750+0000","last_active":"2023-12-19T14:39:11.570793+0000","last_peered":"2023-12-19T14:39:11.570793+0000","last_clean":"2023-12-19T14:39:11.570793+0000","last_became_active":"2023-12-19T14:39:06.253479+0000","last_became_peered":"2023-12-19T14:39:06.253479+0000","last_unstale":"2023-12-19T14:39:11.570793+0000","last_undegraded":"2023-12-19T14:39:11.570793+0000","last_fullsized":"2023-12-19T14:39:11.570793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:22.901851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564091+0000","last_change":"2023-12-19T14:39:11.562226+0000","last_active":"2023-12-19T14:39:11.564091+0000","last_peered":"2023-12-19T14:39:11.564091+0000","last_clean":"2023-12-19T14:39:11.564091+0000","last_became_active":"2023-12-19T14:39:11.561874+0000","last_became_peered":"2023-12-19T14:39:11.561874+0000","last_unstale":"2023-12-19T14:39:11.564091+0000","last_undegraded":"2023-12-19T14:39:11.564091+0000","last_fullsized":"2023-12-19T14:39:11.564091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543918+0000","last_change":"2023-12-19T14:39:07.297728+0000","last_active":"2023-12-19T14:39:11.543918+0000","last_peered":"2023-12-19T14:39:11.543918+0000","last_clean":"2023-12-19T14:39:11.543918+0000","last_became_active":"2023-12-19T14:39:07.297419+0000","last_became_peered":"2023-12-19T14:39:07.297419+0000","last_unstale":"2023-12-19T14:39:11.543918+0000","last_undegraded":"2023-12-19T14:39:11.543918+0000","last_fullsized":"2023-12-19T14:39:11.543918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540603+0000","last_change":"2023-12-19T14:39:06.250154+0000","last_active":"2023-12-19T14:39:11.540603+0000","last_peered":"2023-12-19T14:39:11.540603+0000","last_clean":"2023-12-19T14:39:11.540603+0000","last_became_active":"2023-12-19T14:39:06.249851+0000","last_became_peered":"2023-12-19T14:39:06.249851+0000","last_unstale":"2023-12-19T14:39:11.540603+0000","last_undegraded":"2023-12-19T14:39:11.540603+0000","last_fullsized":"2023-12-19T14:39:11.540603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583288+0000","last_change":"2023-12-19T14:39:11.576656+0000","last_active":"2023-12-19T14:39:11.583288+0000","last_peered":"2023-12-19T14:39:11.583288+0000","last_clean":"2023-12-19T14:39:11.583288+0000","last_became_active":"2023-12-19T14:39:11.576165+0000","last_became_peered":"2023-12-19T14:39:11.576165+0000","last_unstale":"2023-12-19T14:39:11.583288+0000","last_undegraded":"2023-12-19T14:39:11.583288+0000","last_fullsized":"2023-12-19T14:39:11.583288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545215+0000","last_change":"2023-12-19T14:39:07.274124+0000","last_active":"2023-12-19T14:39:11.545215+0000","last_peered":"2023-12-19T14:39:11.545215+0000","last_clean":"2023-12-19T14:39:11.545215+0000","last_became_active":"2023-12-19T14:39:07.273792+0000","last_became_peered":"2023-12-19T14:39:07.273792+0000","last_unstale":"2023-12-19T14:39:11.545215+0000","last_undegraded":"2023-12-19T14:39:11.545215+0000","last_fullsized":"2023-12-19T14:39:11.545215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540233+0000","last_change":"2023-12-19T14:39:06.244868+0000","last_active":"2023-12-19T14:39:11.540233+0000","last_peered":"2023-12-19T14:39:11.540233+0000","last_clean":"2023-12-19T14:39:11.540233+0000","last_became_active":"2023-12-19T14:39:06.244567+0000","last_became_peered":"2023-12-19T14:39:06.244567+0000","last_unstale":"2023-12-19T14:39:11.540233+0000","last_undegraded":"2023-12-19T14:39:11.540233+0000","last_fullsized":"2023-12-19T14:39:11.540233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.587736+0000","last_change":"2023-12-19T14:39:11.585814+0000","last_active":"2023-12-19T14:39:11.587736+0000","last_peered":"2023-12-19T14:39:11.587736+0000","last_clean":"2023-12-19T14:39:11.587736+0000","last_became_active":"2023-12-19T14:39:11.581875+0000","last_became_peered":"2023-12-19T14:39:11.581875+0000","last_unstale":"2023-12-19T14:39:11.587736+0000","last_undegraded":"2023-12-19T14:39:11.587736+0000","last_fullsized":"2023-12-19T14:39:11.587736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547078+0000","last_change":"2023-12-19T14:39:07.276433+0000","last_active":"2023-12-19T14:39:11.547078+0000","last_peered":"2023-12-19T14:39:11.547078+0000","last_clean":"2023-12-19T14:39:11.547078+0000","last_became_active":"2023-12-19T14:39:07.274815+0000","last_became_peered":"2023-12-19T14:39:07.274815+0000","last_unstale":"2023-12-19T14:39:11.547078+0000","last_undegraded":"2023-12-19T14:39:11.547078+0000","last_fullsized":"2023-12-19T14:39:11.547078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:57.360165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569740+0000","last_change":"2023-12-19T14:39:06.262949+0000","last_active":"2023-12-19T14:39:11.569740+0000","last_peered":"2023-12-19T14:39:11.569740+0000","last_clean":"2023-12-19T14:39:11.569740+0000","last_became_active":"2023-12-19T14:39:06.262630+0000","last_became_peered":"2023-12-19T14:39:06.262630+0000","last_unstale":"2023-12-19T14:39:11.569740+0000","last_undegraded":"2023-12-19T14:39:11.569740+0000","last_fullsized":"2023-12-19T14:39:11.569740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580388+0000","last_change":"2023-12-19T14:39:11.572679+0000","last_active":"2023-12-19T14:39:11.580388+0000","last_peered":"2023-12-19T14:39:11.580388+0000","last_clean":"2023-12-19T14:39:11.580388+0000","last_became_active":"2023-12-19T14:39:11.572334+0000","last_became_peered":"2023-12-19T14:39:11.572334+0000","last_unstale":"2023-12-19T14:39:11.580388+0000","last_undegraded":"2023-12-19T14:39:11.580388+0000","last_fullsized":"2023-12-19T14:39:11.580388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:00:02.733099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562034+0000","last_change":"2023-12-19T14:39:07.287397+0000","last_active":"2023-12-19T14:39:11.562034+0000","last_peered":"2023-12-19T14:39:11.562034+0000","last_clean":"2023-12-19T14:39:11.562034+0000","last_became_active":"2023-12-19T14:39:07.287079+0000","last_became_peered":"2023-12-19T14:39:07.287079+0000","last_unstale":"2023-12-19T14:39:11.562034+0000","last_undegraded":"2023-12-19T14:39:11.562034+0000","last_fullsized":"2023-12-19T14:39:11.562034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551941+0000","last_change":"2023-12-19T14:39:06.259853+0000","last_active":"2023-12-19T14:39:11.551941+0000","last_peered":"2023-12-19T14:39:11.551941+0000","last_clean":"2023-12-19T14:39:11.551941+0000","last_became_active":"2023-12-19T14:39:06.259573+0000","last_became_peered":"2023-12-19T14:39:06.259573+0000","last_unstale":"2023-12-19T14:39:11.551941+0000","last_undegraded":"2023-12-19T14:39:11.551941+0000","last_fullsized":"2023-12-19T14:39:11.551941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583774+0000","last_change":"2023-12-19T14:39:11.578284+0000","last_active":"2023-12-19T14:39:11.583774+0000","last_peered":"2023-12-19T14:39:11.583774+0000","last_clean":"2023-12-19T14:39:11.583774+0000","last_became_active":"2023-12-19T14:39:11.577805+0000","last_became_peered":"2023-12-19T14:39:11.577805+0000","last_unstale":"2023-12-19T14:39:11.583774+0000","last_undegraded":"2023-12-19T14:39:11.583774+0000","last_fullsized":"2023-12-19T14:39:11.583774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564518+0000","last_change":"2023-12-19T14:39:07.288511+0000","last_active":"2023-12-19T14:39:11.564518+0000","last_peered":"2023-12-19T14:39:11.564518+0000","last_clean":"2023-12-19T14:39:11.564518+0000","last_became_active":"2023-12-19T14:39:07.288221+0000","last_became_peered":"2023-12-19T14:39:07.288221+0000","last_unstale":"2023-12-19T14:39:11.564518+0000","last_undegraded":"2023-12-19T14:39:11.564518+0000","last_fullsized":"2023-12-19T14:39:11.564518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.728724+0000","last_change":"2023-12-19T14:39:06.260565+0000","last_active":"2023-12-19T14:39:11.728724+0000","last_peered":"2023-12-19T14:39:11.728724+0000","last_clean":"2023-12-19T14:39:11.728724+0000","last_became_active":"2023-12-19T14:39:06.260244+0000","last_became_peered":"2023-12-19T14:39:06.260244+0000","last_unstale":"2023-12-19T14:39:11.728724+0000","last_undegraded":"2023-12-19T14:39:11.728724+0000","last_fullsized":"2023-12-19T14:39:11.728724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588612+0000","last_change":"2023-12-19T14:39:11.583334+0000","last_active":"2023-12-19T14:39:11.588612+0000","last_peered":"2023-12-19T14:39:11.588612+0000","last_clean":"2023-12-19T14:39:11.588612+0000","last_became_active":"2023-12-19T14:39:11.582906+0000","last_became_peered":"2023-12-19T14:39:11.582906+0000","last_unstale":"2023-12-19T14:39:11.588612+0000","last_undegraded":"2023-12-19T14:39:11.588612+0000","last_fullsized":"2023-12-19T14:39:11.588612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:50.106844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540825+0000","last_change":"2023-12-19T14:39:07.267435+0000","last_active":"2023-12-19T14:39:11.540825+0000","last_peered":"2023-12-19T14:39:11.540825+0000","last_clean":"2023-12-19T14:39:11.540825+0000","last_became_active":"2023-12-19T14:39:07.267113+0000","last_became_peered":"2023-12-19T14:39:07.267113+0000","last_unstale":"2023-12-19T14:39:11.540825+0000","last_undegraded":"2023-12-19T14:39:11.540825+0000","last_fullsized":"2023-12-19T14:39:11.540825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570841+0000","last_change":"2023-12-19T14:39:06.262808+0000","last_active":"2023-12-19T14:39:11.570841+0000","last_peered":"2023-12-19T14:39:11.570841+0000","last_clean":"2023-12-19T14:39:11.570841+0000","last_became_active":"2023-12-19T14:39:06.262439+0000","last_became_peered":"2023-12-19T14:39:06.262439+0000","last_unstale":"2023-12-19T14:39:11.570841+0000","last_undegraded":"2023-12-19T14:39:11.570841+0000","last_fullsized":"2023-12-19T14:39:11.570841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583520+0000","last_change":"2023-12-19T14:39:11.575768+0000","last_active":"2023-12-19T14:39:11.583520+0000","last_peered":"2023-12-19T14:39:11.583520+0000","last_clean":"2023-12-19T14:39:11.583520+0000","last_became_active":"2023-12-19T14:39:11.575222+0000","last_became_peered":"2023-12-19T14:39:11.575222+0000","last_unstale":"2023-12-19T14:39:11.583520+0000","last_undegraded":"2023-12-19T14:39:11.583520+0000","last_fullsized":"2023-12-19T14:39:11.583520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548697+0000","last_change":"2023-12-19T14:39:07.291010+0000","last_active":"2023-12-19T14:39:11.548697+0000","last_peered":"2023-12-19T14:39:11.548697+0000","last_clean":"2023-12-19T14:39:11.548697+0000","last_became_active":"2023-12-19T14:39:07.290719+0000","last_became_peered":"2023-12-19T14:39:07.290719+0000","last_unstale":"2023-12-19T14:39:11.548697+0000","last_undegraded":"2023-12-19T14:39:11.548697+0000","last_fullsized":"2023-12-19T14:39:11.548697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540846+0000","last_change":"2023-12-19T14:39:06.252011+0000","last_active":"2023-12-19T14:39:11.540846+0000","last_peered":"2023-12-19T14:39:11.540846+0000","last_clean":"2023-12-19T14:39:11.540846+0000","last_became_active":"2023-12-19T14:39:06.251724+0000","last_became_peered":"2023-12-19T14:39:06.251724+0000","last_unstale":"2023-12-19T14:39:11.540846+0000","last_undegraded":"2023-12-19T14:39:11.540846+0000","last_fullsized":"2023-12-19T14:39:11.540846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573312+0000","last_change":"2023-12-19T14:39:11.562285+0000","last_active":"2023-12-19T14:39:11.573312+0000","last_peered":"2023-12-19T14:39:11.573312+0000","last_clean":"2023-12-19T14:39:11.573312+0000","last_became_active":"2023-12-19T14:39:11.561826+0000","last_became_peered":"2023-12-19T14:39:11.561826+0000","last_unstale":"2023-12-19T14:39:11.573312+0000","last_undegraded":"2023-12-19T14:39:11.573312+0000","last_fullsized":"2023-12-19T14:39:11.573312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546810+0000","last_change":"2023-12-19T14:39:07.289173+0000","last_active":"2023-12-19T14:39:11.546810+0000","last_peered":"2023-12-19T14:39:11.546810+0000","last_clean":"2023-12-19T14:39:11.546810+0000","last_became_active":"2023-12-19T14:39:07.288745+0000","last_became_peered":"2023-12-19T14:39:07.288745+0000","last_unstale":"2023-12-19T14:39:11.546810+0000","last_undegraded":"2023-12-19T14:39:11.546810+0000","last_fullsized":"2023-12-19T14:39:11.546810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545170+0000","last_change":"2023-12-19T14:39:06.258745+0000","last_active":"2023-12-19T14:39:11.545170+0000","last_peered":"2023-12-19T14:39:11.545170+0000","last_clean":"2023-12-19T14:39:11.545170+0000","last_became_active":"2023-12-19T14:39:06.257007+0000","last_became_peered":"2023-12-19T14:39:06.257007+0000","last_unstale":"2023-12-19T14:39:11.545170+0000","last_undegraded":"2023-12-19T14:39:11.545170+0000","last_fullsized":"2023-12-19T14:39:11.545170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571904+0000","last_change":"2023-12-19T14:39:11.568661+0000","last_active":"2023-12-19T14:39:11.571904+0000","last_peered":"2023-12-19T14:39:11.571904+0000","last_clean":"2023-12-19T14:39:11.571904+0000","last_became_active":"2023-12-19T14:39:11.567940+0000","last_became_peered":"2023-12-19T14:39:11.567940+0000","last_unstale":"2023-12-19T14:39:11.571904+0000","last_undegraded":"2023-12-19T14:39:11.571904+0000","last_fullsized":"2023-12-19T14:39:11.571904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539181+0000","last_change":"2023-12-19T14:39:07.269639+0000","last_active":"2023-12-19T14:39:11.539181+0000","last_peered":"2023-12-19T14:39:11.539181+0000","last_clean":"2023-12-19T14:39:11.539181+0000","last_became_active":"2023-12-19T14:39:07.269276+0000","last_became_peered":"2023-12-19T14:39:07.269276+0000","last_unstale":"2023-12-19T14:39:11.539181+0000","last_undegraded":"2023-12-19T14:39:11.539181+0000","last_fullsized":"2023-12-19T14:39:11.539181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553025+0000","last_change":"2023-12-19T14:39:06.254091+0000","last_active":"2023-12-19T14:39:11.553025+0000","last_peered":"2023-12-19T14:39:11.553025+0000","last_clean":"2023-12-19T14:39:11.553025+0000","last_became_active":"2023-12-19T14:39:06.253444+0000","last_became_peered":"2023-12-19T14:39:06.253444+0000","last_unstale":"2023-12-19T14:39:11.553025+0000","last_undegraded":"2023-12-19T14:39:11.553025+0000","last_fullsized":"2023-12-19T14:39:11.553025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584042+0000","last_change":"2023-12-19T14:39:11.574535+0000","last_active":"2023-12-19T14:39:11.584042+0000","last_peered":"2023-12-19T14:39:11.584042+0000","last_clean":"2023-12-19T14:39:11.584042+0000","last_became_active":"2023-12-19T14:39:11.574060+0000","last_became_peered":"2023-12-19T14:39:11.574060+0000","last_unstale":"2023-12-19T14:39:11.584042+0000","last_undegraded":"2023-12-19T14:39:11.584042+0000","last_fullsized":"2023-12-19T14:39:11.584042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.785232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556888+0000","last_change":"2023-12-19T14:39:07.287233+0000","last_active":"2023-12-19T14:39:11.556888+0000","last_peered":"2023-12-19T14:39:11.556888+0000","last_clean":"2023-12-19T14:39:11.556888+0000","last_became_active":"2023-12-19T14:39:07.286769+0000","last_became_peered":"2023-12-19T14:39:07.286769+0000","last_unstale":"2023-12-19T14:39:11.556888+0000","last_undegraded":"2023-12-19T14:39:11.556888+0000","last_fullsized":"2023-12-19T14:39:11.556888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569853+0000","last_change":"2023-12-19T14:39:06.251774+0000","last_active":"2023-12-19T14:39:11.569853+0000","last_peered":"2023-12-19T14:39:11.569853+0000","last_clean":"2023-12-19T14:39:11.569853+0000","last_became_active":"2023-12-19T14:39:06.251528+0000","last_became_peered":"2023-12-19T14:39:06.251528+0000","last_unstale":"2023-12-19T14:39:11.569853+0000","last_undegraded":"2023-12-19T14:39:11.569853+0000","last_fullsized":"2023-12-19T14:39:11.569853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581699+0000","last_change":"2023-12-19T14:39:11.575777+0000","last_active":"2023-12-19T14:39:11.581699+0000","last_peered":"2023-12-19T14:39:11.581699+0000","last_clean":"2023-12-19T14:39:11.581699+0000","last_became_active":"2023-12-19T14:39:11.575283+0000","last_became_peered":"2023-12-19T14:39:11.575283+0000","last_unstale":"2023-12-19T14:39:11.581699+0000","last_undegraded":"2023-12-19T14:39:11.581699+0000","last_fullsized":"2023-12-19T14:39:11.581699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:54.006908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543534+0000","last_change":"2023-12-19T14:39:07.294894+0000","last_active":"2023-12-19T14:39:11.543534+0000","last_peered":"2023-12-19T14:39:11.543534+0000","last_clean":"2023-12-19T14:39:11.543534+0000","last_became_active":"2023-12-19T14:39:07.294640+0000","last_became_peered":"2023-12-19T14:39:07.294640+0000","last_unstale":"2023-12-19T14:39:11.543534+0000","last_undegraded":"2023-12-19T14:39:11.543534+0000","last_fullsized":"2023-12-19T14:39:11.543534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545405+0000","last_change":"2023-12-19T14:39:06.256452+0000","last_active":"2023-12-19T14:39:11.545405+0000","last_peered":"2023-12-19T14:39:11.545405+0000","last_clean":"2023-12-19T14:39:11.545405+0000","last_became_active":"2023-12-19T14:39:06.255327+0000","last_became_peered":"2023-12-19T14:39:06.255327+0000","last_unstale":"2023-12-19T14:39:11.545405+0000","last_undegraded":"2023-12-19T14:39:11.545405+0000","last_fullsized":"2023-12-19T14:39:11.545405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569956+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:39:11.569956+0000","last_peered":"2023-12-19T14:39:11.569956+0000","last_clean":"2023-12-19T14:39:11.569956+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:39:11.569956+0000","last_undegraded":"2023-12-19T14:39:11.569956+0000","last_fullsized":"2023-12-19T14:39:11.569956+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582739+0000","last_change":"2023-12-19T14:39:11.578205+0000","last_active":"2023-12-19T14:39:11.582739+0000","last_peered":"2023-12-19T14:39:11.582739+0000","last_clean":"2023-12-19T14:39:11.582739+0000","last_became_active":"2023-12-19T14:39:11.577809+0000","last_became_peered":"2023-12-19T14:39:11.577809+0000","last_unstale":"2023-12-19T14:39:11.582739+0000","last_undegraded":"2023-12-19T14:39:11.582739+0000","last_fullsized":"2023-12-19T14:39:11.582739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:10.870899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570477+0000","last_change":"2023-12-19T14:39:07.283187+0000","last_active":"2023-12-19T14:39:11.570477+0000","last_peered":"2023-12-19T14:39:11.570477+0000","last_clean":"2023-12-19T14:39:11.570477+0000","last_became_active":"2023-12-19T14:39:07.282884+0000","last_became_peered":"2023-12-19T14:39:07.282884+0000","last_unstale":"2023-12-19T14:39:11.570477+0000","last_undegraded":"2023-12-19T14:39:11.570477+0000","last_fullsized":"2023-12-19T14:39:11.570477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553094+0000","last_change":"2023-12-19T14:39:06.260560+0000","last_active":"2023-12-19T14:39:11.553094+0000","last_peered":"2023-12-19T14:39:11.553094+0000","last_clean":"2023-12-19T14:39:11.553094+0000","last_became_active":"2023-12-19T14:39:06.260257+0000","last_became_peered":"2023-12-19T14:39:06.260257+0000","last_unstale":"2023-12-19T14:39:11.553094+0000","last_undegraded":"2023-12-19T14:39:11.553094+0000","last_fullsized":"2023-12-19T14:39:11.553094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580038+0000","last_change":"2023-12-19T14:39:11.575041+0000","last_active":"2023-12-19T14:39:11.580038+0000","last_peered":"2023-12-19T14:39:11.580038+0000","last_clean":"2023-12-19T14:39:11.580038+0000","last_became_active":"2023-12-19T14:39:11.574653+0000","last_became_peered":"2023-12-19T14:39:11.574653+0000","last_unstale":"2023-12-19T14:39:11.580038+0000","last_undegraded":"2023-12-19T14:39:11.580038+0000","last_fullsized":"2023-12-19T14:39:11.580038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545946+0000","last_change":"2023-12-19T14:39:07.291614+0000","last_active":"2023-12-19T14:39:11.545946+0000","last_peered":"2023-12-19T14:39:11.545946+0000","last_clean":"2023-12-19T14:39:11.545946+0000","last_became_active":"2023-12-19T14:39:07.291096+0000","last_became_peered":"2023-12-19T14:39:07.291096+0000","last_unstale":"2023-12-19T14:39:11.545946+0000","last_undegraded":"2023-12-19T14:39:11.545946+0000","last_fullsized":"2023-12-19T14:39:11.545946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540658+0000","last_change":"2023-12-19T14:39:06.245861+0000","last_active":"2023-12-19T14:39:11.540658+0000","last_peered":"2023-12-19T14:39:11.540658+0000","last_clean":"2023-12-19T14:39:11.540658+0000","last_became_active":"2023-12-19T14:39:06.245543+0000","last_became_peered":"2023-12-19T14:39:06.245543+0000","last_unstale":"2023-12-19T14:39:11.540658+0000","last_undegraded":"2023-12-19T14:39:11.540658+0000","last_fullsized":"2023-12-19T14:39:11.540658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.575504+0000","last_change":"2023-12-19T14:39:11.572562+0000","last_active":"2023-12-19T14:39:11.575504+0000","last_peered":"2023-12-19T14:39:11.575504+0000","last_clean":"2023-12-19T14:39:11.575504+0000","last_became_active":"2023-12-19T14:39:11.572159+0000","last_became_peered":"2023-12-19T14:39:11.572159+0000","last_unstale":"2023-12-19T14:39:11.575504+0000","last_undegraded":"2023-12-19T14:39:11.575504+0000","last_fullsized":"2023-12-19T14:39:11.575504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571036+0000","last_change":"2023-12-19T14:39:07.284702+0000","last_active":"2023-12-19T14:39:11.571036+0000","last_peered":"2023-12-19T14:39:11.571036+0000","last_clean":"2023-12-19T14:39:11.571036+0000","last_became_active":"2023-12-19T14:39:07.284258+0000","last_became_peered":"2023-12-19T14:39:07.284258+0000","last_unstale":"2023-12-19T14:39:11.571036+0000","last_undegraded":"2023-12-19T14:39:11.571036+0000","last_fullsized":"2023-12-19T14:39:11.571036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555155+0000","last_change":"2023-12-19T14:39:06.256120+0000","last_active":"2023-12-19T14:39:11.555155+0000","last_peered":"2023-12-19T14:39:11.555155+0000","last_clean":"2023-12-19T14:39:11.555155+0000","last_became_active":"2023-12-19T14:39:06.255187+0000","last_became_peered":"2023-12-19T14:39:06.255187+0000","last_unstale":"2023-12-19T14:39:11.555155+0000","last_undegraded":"2023-12-19T14:39:11.555155+0000","last_fullsized":"2023-12-19T14:39:11.555155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584352+0000","last_change":"2023-12-19T14:39:11.579495+0000","last_active":"2023-12-19T14:39:11.584352+0000","last_peered":"2023-12-19T14:39:11.584352+0000","last_clean":"2023-12-19T14:39:11.584352+0000","last_became_active":"2023-12-19T14:39:11.579259+0000","last_became_peered":"2023-12-19T14:39:11.579259+0000","last_unstale":"2023-12-19T14:39:11.584352+0000","last_undegraded":"2023-12-19T14:39:11.584352+0000","last_fullsized":"2023-12-19T14:39:11.584352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573192+0000","last_change":"2023-12-19T14:39:07.279364+0000","last_active":"2023-12-19T14:39:11.573192+0000","last_peered":"2023-12-19T14:39:11.573192+0000","last_clean":"2023-12-19T14:39:11.573192+0000","last_became_active":"2023-12-19T14:39:07.278999+0000","last_became_peered":"2023-12-19T14:39:07.278999+0000","last_unstale":"2023-12-19T14:39:11.573192+0000","last_undegraded":"2023-12-19T14:39:11.573192+0000","last_fullsized":"2023-12-19T14:39:11.573192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547974+0000","last_change":"2023-12-19T14:39:06.262399+0000","last_active":"2023-12-19T14:39:11.547974+0000","last_peered":"2023-12-19T14:39:11.547974+0000","last_clean":"2023-12-19T14:39:11.547974+0000","last_became_active":"2023-12-19T14:39:06.261918+0000","last_became_peered":"2023-12-19T14:39:06.261918+0000","last_unstale":"2023-12-19T14:39:11.547974+0000","last_undegraded":"2023-12-19T14:39:11.547974+0000","last_fullsized":"2023-12-19T14:39:11.547974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:48:08.535588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583755+0000","last_change":"2023-12-19T14:39:11.577720+0000","last_active":"2023-12-19T14:39:11.583755+0000","last_peered":"2023-12-19T14:39:11.583755+0000","last_clean":"2023-12-19T14:39:11.583755+0000","last_became_active":"2023-12-19T14:39:11.577297+0000","last_became_peered":"2023-12-19T14:39:11.577297+0000","last_unstale":"2023-12-19T14:39:11.583755+0000","last_undegraded":"2023-12-19T14:39:11.583755+0000","last_fullsized":"2023-12-19T14:39:11.583755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570176+0000","last_change":"2023-12-19T14:39:07.291311+0000","last_active":"2023-12-19T14:39:11.570176+0000","last_peered":"2023-12-19T14:39:11.570176+0000","last_clean":"2023-12-19T14:39:11.570176+0000","last_became_active":"2023-12-19T14:39:07.290587+0000","last_became_peered":"2023-12-19T14:39:07.290587+0000","last_unstale":"2023-12-19T14:39:11.570176+0000","last_undegraded":"2023-12-19T14:39:11.570176+0000","last_fullsized":"2023-12-19T14:39:11.570176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548490+0000","last_change":"2023-12-19T14:39:06.248952+0000","last_active":"2023-12-19T14:39:11.548490+0000","last_peered":"2023-12-19T14:39:11.548490+0000","last_clean":"2023-12-19T14:39:11.548490+0000","last_became_active":"2023-12-19T14:39:06.248504+0000","last_became_peered":"2023-12-19T14:39:06.248504+0000","last_unstale":"2023-12-19T14:39:11.548490+0000","last_undegraded":"2023-12-19T14:39:11.548490+0000","last_fullsized":"2023-12-19T14:39:11.548490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581722+0000","last_change":"2023-12-19T14:39:11.579028+0000","last_active":"2023-12-19T14:39:11.581722+0000","last_peered":"2023-12-19T14:39:11.581722+0000","last_clean":"2023-12-19T14:39:11.581722+0000","last_became_active":"2023-12-19T14:39:11.578386+0000","last_became_peered":"2023-12-19T14:39:11.578386+0000","last_unstale":"2023-12-19T14:39:11.581722+0000","last_undegraded":"2023-12-19T14:39:11.581722+0000","last_fullsized":"2023-12-19T14:39:11.581722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551739+0000","last_change":"2023-12-19T14:39:07.289550+0000","last_active":"2023-12-19T14:39:11.551739+0000","last_peered":"2023-12-19T14:39:11.551739+0000","last_clean":"2023-12-19T14:39:11.551739+0000","last_became_active":"2023-12-19T14:39:07.289242+0000","last_became_peered":"2023-12-19T14:39:07.289242+0000","last_unstale":"2023-12-19T14:39:11.551739+0000","last_undegraded":"2023-12-19T14:39:11.551739+0000","last_fullsized":"2023-12-19T14:39:11.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549347+0000","last_change":"2023-12-19T14:39:06.250752+0000","last_active":"2023-12-19T14:39:11.549347+0000","last_peered":"2023-12-19T14:39:11.549347+0000","last_clean":"2023-12-19T14:39:11.549347+0000","last_became_active":"2023-12-19T14:39:06.249438+0000","last_became_peered":"2023-12-19T14:39:06.249438+0000","last_unstale":"2023-12-19T14:39:11.549347+0000","last_undegraded":"2023-12-19T14:39:11.549347+0000","last_fullsized":"2023-12-19T14:39:11.549347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588513+0000","last_change":"2023-12-19T14:39:11.581267+0000","last_active":"2023-12-19T14:39:11.588513+0000","last_peered":"2023-12-19T14:39:11.588513+0000","last_clean":"2023-12-19T14:39:11.588513+0000","last_became_active":"2023-12-19T14:39:11.577498+0000","last_became_peered":"2023-12-19T14:39:11.577498+0000","last_unstale":"2023-12-19T14:39:11.588513+0000","last_undegraded":"2023-12-19T14:39:11.588513+0000","last_fullsized":"2023-12-19T14:39:11.588513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:16:19.999833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570611+0000","last_change":"2023-12-19T14:39:07.283321+0000","last_active":"2023-12-19T14:39:11.570611+0000","last_peered":"2023-12-19T14:39:11.570611+0000","last_clean":"2023-12-19T14:39:11.570611+0000","last_became_active":"2023-12-19T14:39:07.283017+0000","last_became_peered":"2023-12-19T14:39:07.283017+0000","last_unstale":"2023-12-19T14:39:11.570611+0000","last_undegraded":"2023-12-19T14:39:11.570611+0000","last_fullsized":"2023-12-19T14:39:11.570611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545060+0000","last_change":"2023-12-19T14:39:06.259282+0000","last_active":"2023-12-19T14:39:11.545060+0000","last_peered":"2023-12-19T14:39:11.545060+0000","last_clean":"2023-12-19T14:39:11.545060+0000","last_became_active":"2023-12-19T14:39:06.257316+0000","last_became_peered":"2023-12-19T14:39:06.257316+0000","last_unstale":"2023-12-19T14:39:11.545060+0000","last_undegraded":"2023-12-19T14:39:11.545060+0000","last_fullsized":"2023-12-19T14:39:11.545060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588628+0000","last_change":"2023-12-19T14:39:11.577972+0000","last_active":"2023-12-19T14:39:11.588628+0000","last_peered":"2023-12-19T14:39:11.588628+0000","last_clean":"2023-12-19T14:39:11.588628+0000","last_became_active":"2023-12-19T14:39:11.577529+0000","last_became_peered":"2023-12-19T14:39:11.577529+0000","last_unstale":"2023-12-19T14:39:11.588628+0000","last_undegraded":"2023-12-19T14:39:11.588628+0000","last_fullsized":"2023-12-19T14:39:11.588628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:19:59.360333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.557043+0000","last_change":"2023-12-19T14:39:07.289405+0000","last_active":"2023-12-19T14:39:11.557043+0000","last_peered":"2023-12-19T14:39:11.557043+0000","last_clean":"2023-12-19T14:39:11.557043+0000","last_became_active":"2023-12-19T14:39:07.288885+0000","last_became_peered":"2023-12-19T14:39:07.288885+0000","last_unstale":"2023-12-19T14:39:11.557043+0000","last_undegraded":"2023-12-19T14:39:11.557043+0000","last_fullsized":"2023-12-19T14:39:11.557043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559942+0000","last_change":"2023-12-19T14:39:06.256658+0000","last_active":"2023-12-19T14:39:11.559942+0000","last_peered":"2023-12-19T14:39:11.559942+0000","last_clean":"2023-12-19T14:39:11.559942+0000","last_became_active":"2023-12-19T14:39:06.255561+0000","last_became_peered":"2023-12-19T14:39:06.255561+0000","last_unstale":"2023-12-19T14:39:11.559942+0000","last_undegraded":"2023-12-19T14:39:11.559942+0000","last_fullsized":"2023-12-19T14:39:11.559942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584461+0000","last_change":"2023-12-19T14:39:11.579961+0000","last_active":"2023-12-19T14:39:11.584461+0000","last_peered":"2023-12-19T14:39:11.584461+0000","last_clean":"2023-12-19T14:39:11.584461+0000","last_became_active":"2023-12-19T14:39:11.578069+0000","last_became_peered":"2023-12-19T14:39:11.578069+0000","last_unstale":"2023-12-19T14:39:11.584461+0000","last_undegraded":"2023-12-19T14:39:11.584461+0000","last_fullsized":"2023-12-19T14:39:11.584461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570370+0000","last_change":"2023-12-19T14:39:07.288175+0000","last_active":"2023-12-19T14:39:11.570370+0000","last_peered":"2023-12-19T14:39:11.570370+0000","last_clean":"2023-12-19T14:39:11.570370+0000","last_became_active":"2023-12-19T14:39:07.287333+0000","last_became_peered":"2023-12-19T14:39:07.287333+0000","last_unstale":"2023-12-19T14:39:11.570370+0000","last_undegraded":"2023-12-19T14:39:11.570370+0000","last_fullsized":"2023-12-19T14:39:11.570370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T14:39:19.178278+0000","last_change":"2023-12-19T14:39:06.256219+0000","last_active":"2023-12-19T14:39:19.178278+0000","last_peered":"2023-12-19T14:39:19.178278+0000","last_clean":"2023-12-19T14:39:19.178278+0000","last_became_active":"2023-12-19T14:39:06.255840+0000","last_became_peered":"2023-12-19T14:39:06.255840+0000","last_unstale":"2023-12-19T14:39:19.178278+0000","last_undegraded":"2023-12-19T14:39:19.178278+0000","last_fullsized":"2023-12-19T14:39:19.178278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573334+0000","last_change":"2023-12-19T14:39:11.569520+0000","last_active":"2023-12-19T14:39:11.573334+0000","last_peered":"2023-12-19T14:39:11.573334+0000","last_clean":"2023-12-19T14:39:11.573334+0000","last_became_active":"2023-12-19T14:39:11.563543+0000","last_became_peered":"2023-12-19T14:39:11.563543+0000","last_unstale":"2023-12-19T14:39:11.573334+0000","last_undegraded":"2023-12-19T14:39:11.573334+0000","last_fullsized":"2023-12-19T14:39:11.573334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546941+0000","last_change":"2023-12-19T14:39:07.276126+0000","last_active":"2023-12-19T14:39:11.546941+0000","last_peered":"2023-12-19T14:39:11.546941+0000","last_clean":"2023-12-19T14:39:11.546941+0000","last_became_active":"2023-12-19T14:39:07.275186+0000","last_became_peered":"2023-12-19T14:39:07.275186+0000","last_unstale":"2023-12-19T14:39:11.546941+0000","last_undegraded":"2023-12-19T14:39:11.546941+0000","last_fullsized":"2023-12-19T14:39:11.546941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:40.362295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540550+0000","last_change":"2023-12-19T14:39:06.259793+0000","last_active":"2023-12-19T14:39:11.540550+0000","last_peered":"2023-12-19T14:39:11.540550+0000","last_clean":"2023-12-19T14:39:11.540550+0000","last_became_active":"2023-12-19T14:39:06.259474+0000","last_became_peered":"2023-12-19T14:39:06.259474+0000","last_unstale":"2023-12-19T14:39:11.540550+0000","last_undegraded":"2023-12-19T14:39:11.540550+0000","last_fullsized":"2023-12-19T14:39:11.540550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573690+0000","last_change":"2023-12-19T14:39:11.567330+0000","last_active":"2023-12-19T14:39:11.573690+0000","last_peered":"2023-12-19T14:39:11.573690+0000","last_clean":"2023-12-19T14:39:11.573690+0000","last_became_active":"2023-12-19T14:39:11.566909+0000","last_became_peered":"2023-12-19T14:39:11.566909+0000","last_unstale":"2023-12-19T14:39:11.573690+0000","last_undegraded":"2023-12-19T14:39:11.573690+0000","last_fullsized":"2023-12-19T14:39:11.573690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:41:18.238432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561959+0000","last_change":"2023-12-19T14:39:07.290356+0000","last_active":"2023-12-19T14:39:11.561959+0000","last_peered":"2023-12-19T14:39:11.561959+0000","last_clean":"2023-12-19T14:39:11.561959+0000","last_became_active":"2023-12-19T14:39:07.288476+0000","last_became_peered":"2023-12-19T14:39:07.288476+0000","last_unstale":"2023-12-19T14:39:11.561959+0000","last_undegraded":"2023-12-19T14:39:11.561959+0000","last_fullsized":"2023-12-19T14:39:11.561959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549225+0000","last_change":"2023-12-19T14:39:06.263437+0000","last_active":"2023-12-19T14:39:11.549225+0000","last_peered":"2023-12-19T14:39:11.549225+0000","last_clean":"2023-12-19T14:39:11.549225+0000","last_became_active":"2023-12-19T14:39:06.263146+0000","last_became_peered":"2023-12-19T14:39:06.263146+0000","last_unstale":"2023-12-19T14:39:11.549225+0000","last_undegraded":"2023-12-19T14:39:11.549225+0000","last_fullsized":"2023-12-19T14:39:11.549225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582385+0000","last_change":"2023-12-19T14:39:11.569957+0000","last_active":"2023-12-19T14:39:11.582385+0000","last_peered":"2023-12-19T14:39:11.582385+0000","last_clean":"2023-12-19T14:39:11.582385+0000","last_became_active":"2023-12-19T14:39:11.568502+0000","last_became_peered":"2023-12-19T14:39:11.568502+0000","last_unstale":"2023-12-19T14:39:11.582385+0000","last_undegraded":"2023-12-19T14:39:11.582385+0000","last_fullsized":"2023-12-19T14:39:11.582385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:02:34.258771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549124+0000","last_change":"2023-12-19T14:39:07.284419+0000","last_active":"2023-12-19T14:39:11.549124+0000","last_peered":"2023-12-19T14:39:11.549124+0000","last_clean":"2023-12-19T14:39:11.549124+0000","last_became_active":"2023-12-19T14:39:07.283507+0000","last_became_peered":"2023-12-19T14:39:07.283507+0000","last_unstale":"2023-12-19T14:39:11.549124+0000","last_undegraded":"2023-12-19T14:39:11.549124+0000","last_fullsized":"2023-12-19T14:39:11.549124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551787+0000","last_change":"2023-12-19T14:39:06.261357+0000","last_active":"2023-12-19T14:39:11.551787+0000","last_peered":"2023-12-19T14:39:11.551787+0000","last_clean":"2023-12-19T14:39:11.551787+0000","last_became_active":"2023-12-19T14:39:06.260833+0000","last_became_peered":"2023-12-19T14:39:06.260833+0000","last_unstale":"2023-12-19T14:39:11.551787+0000","last_undegraded":"2023-12-19T14:39:11.551787+0000","last_fullsized":"2023-12-19T14:39:11.551787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588646+0000","last_change":"2023-12-19T14:39:11.579827+0000","last_active":"2023-12-19T14:39:11.588646+0000","last_peered":"2023-12-19T14:39:11.588646+0000","last_clean":"2023-12-19T14:39:11.588646+0000","last_became_active":"2023-12-19T14:39:11.578792+0000","last_became_peered":"2023-12-19T14:39:11.578792+0000","last_unstale":"2023-12-19T14:39:11.588646+0000","last_undegraded":"2023-12-19T14:39:11.588646+0000","last_fullsized":"2023-12-19T14:39:11.588646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583746+0000","last_change":"2023-12-19T14:39:11.581965+0000","last_active":"2023-12-19T14:39:11.583746+0000","last_peered":"2023-12-19T14:39:11.583746+0000","last_clean":"2023-12-19T14:39:11.583746+0000","last_became_active":"2023-12-19T14:39:11.581412+0000","last_became_peered":"2023-12-19T14:39:11.581412+0000","last_unstale":"2023-12-19T14:39:11.583746+0000","last_undegraded":"2023-12-19T14:39:11.583746+0000","last_fullsized":"2023-12-19T14:39:11.583746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539409+0000","last_change":"2023-12-19T14:39:07.283081+0000","last_active":"2023-12-19T14:39:11.539409+0000","last_peered":"2023-12-19T14:39:11.539409+0000","last_clean":"2023-12-19T14:39:11.539409+0000","last_became_active":"2023-12-19T14:39:07.276260+0000","last_became_peered":"2023-12-19T14:39:07.276260+0000","last_unstale":"2023-12-19T14:39:11.539409+0000","last_undegraded":"2023-12-19T14:39:11.539409+0000","last_fullsized":"2023-12-19T14:39:11.539409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543780+0000","last_change":"2023-12-19T14:39:06.257897+0000","last_active":"2023-12-19T14:39:11.543780+0000","last_peered":"2023-12-19T14:39:11.543780+0000","last_clean":"2023-12-19T14:39:11.543780+0000","last_became_active":"2023-12-19T14:39:06.257281+0000","last_became_peered":"2023-12-19T14:39:06.257281+0000","last_unstale":"2023-12-19T14:39:11.543780+0000","last_undegraded":"2023-12-19T14:39:11.543780+0000","last_fullsized":"2023-12-19T14:39:11.543780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571142+0000","last_change":"2023-12-19T14:39:11.566055+0000","last_active":"2023-12-19T14:39:11.571142+0000","last_peered":"2023-12-19T14:39:11.571142+0000","last_clean":"2023-12-19T14:39:11.571142+0000","last_became_active":"2023-12-19T14:39:11.565416+0000","last_became_peered":"2023-12-19T14:39:11.565416+0000","last_unstale":"2023-12-19T14:39:11.571142+0000","last_undegraded":"2023-12-19T14:39:11.571142+0000","last_fullsized":"2023-12-19T14:39:11.571142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:12:19.857166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545049+0000","last_change":"2023-12-19T14:39:07.290776+0000","last_active":"2023-12-19T14:39:11.545049+0000","last_peered":"2023-12-19T14:39:11.545049+0000","last_clean":"2023-12-19T14:39:11.545049+0000","last_became_active":"2023-12-19T14:39:07.290465+0000","last_became_peered":"2023-12-19T14:39:07.290465+0000","last_unstale":"2023-12-19T14:39:11.545049+0000","last_undegraded":"2023-12-19T14:39:11.545049+0000","last_fullsized":"2023-12-19T14:39:11.545049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543823+0000","last_change":"2023-12-19T14:39:06.259345+0000","last_active":"2023-12-19T14:39:11.543823+0000","last_peered":"2023-12-19T14:39:11.543823+0000","last_clean":"2023-12-19T14:39:11.543823+0000","last_became_active":"2023-12-19T14:39:06.259067+0000","last_became_peered":"2023-12-19T14:39:06.259067+0000","last_unstale":"2023-12-19T14:39:11.543823+0000","last_undegraded":"2023-12-19T14:39:11.543823+0000","last_fullsized":"2023-12-19T14:39:11.543823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550184+0000","last_change":"2023-12-19T14:39:06.260437+0000","last_active":"2023-12-19T14:39:11.550184+0000","last_peered":"2023-12-19T14:39:11.550184+0000","last_clean":"2023-12-19T14:39:11.550184+0000","last_became_active":"2023-12-19T14:39:06.260008+0000","last_became_peered":"2023-12-19T14:39:06.260008+0000","last_unstale":"2023-12-19T14:39:11.550184+0000","last_undegraded":"2023-12-19T14:39:11.550184+0000","last_fullsized":"2023-12-19T14:39:11.550184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559895+0000","last_change":"2023-12-19T14:39:07.288421+0000","last_active":"2023-12-19T14:39:11.559895+0000","last_peered":"2023-12-19T14:39:11.559895+0000","last_clean":"2023-12-19T14:39:11.559895+0000","last_became_active":"2023-12-19T14:39:07.287403+0000","last_became_peered":"2023-12-19T14:39:07.287403+0000","last_unstale":"2023-12-19T14:39:11.559895+0000","last_undegraded":"2023-12-19T14:39:11.559895+0000","last_fullsized":"2023-12-19T14:39:11.559895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581278+0000","last_change":"2023-12-19T14:39:11.577684+0000","last_active":"2023-12-19T14:39:11.581278+0000","last_peered":"2023-12-19T14:39:11.581278+0000","last_clean":"2023-12-19T14:39:11.581278+0000","last_became_active":"2023-12-19T14:39:11.577068+0000","last_became_peered":"2023-12-19T14:39:11.577068+0000","last_unstale":"2023-12-19T14:39:11.581278+0000","last_undegraded":"2023-12-19T14:39:11.581278+0000","last_fullsized":"2023-12-19T14:39:11.581278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:42.373695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544048+0000","last_change":"2023-12-19T14:39:06.247049+0000","last_active":"2023-12-19T14:39:11.544048+0000","last_peered":"2023-12-19T14:39:11.544048+0000","last_clean":"2023-12-19T14:39:11.544048+0000","last_became_active":"2023-12-19T14:39:06.246716+0000","last_became_peered":"2023-12-19T14:39:06.246716+0000","last_unstale":"2023-12-19T14:39:11.544048+0000","last_undegraded":"2023-12-19T14:39:11.544048+0000","last_fullsized":"2023-12-19T14:39:11.544048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546858+0000","last_change":"2023-12-19T14:39:07.266860+0000","last_active":"2023-12-19T14:39:11.546858+0000","last_peered":"2023-12-19T14:39:11.546858+0000","last_clean":"2023-12-19T14:39:11.546858+0000","last_became_active":"2023-12-19T14:39:07.266544+0000","last_became_peered":"2023-12-19T14:39:07.266544+0000","last_unstale":"2023-12-19T14:39:11.546858+0000","last_undegraded":"2023-12-19T14:39:11.546858+0000","last_fullsized":"2023-12-19T14:39:11.546858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:47.972055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.585215+0000","last_change":"2023-12-19T14:39:11.567865+0000","last_active":"2023-12-19T14:39:11.585215+0000","last_peered":"2023-12-19T14:39:11.585215+0000","last_clean":"2023-12-19T14:39:11.585215+0000","last_became_active":"2023-12-19T14:39:11.566466+0000","last_became_peered":"2023-12-19T14:39:11.566466+0000","last_unstale":"2023-12-19T14:39:11.585215+0000","last_undegraded":"2023-12-19T14:39:11.585215+0000","last_fullsized":"2023-12-19T14:39:11.585215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:11.949761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551612+0000","last_change":"2023-12-19T14:39:07.289708+0000","last_active":"2023-12-19T14:39:11.551612+0000","last_peered":"2023-12-19T14:39:11.551612+0000","last_clean":"2023-12-19T14:39:11.551612+0000","last_became_active":"2023-12-19T14:39:07.289420+0000","last_became_peered":"2023-12-19T14:39:07.289420+0000","last_unstale":"2023-12-19T14:39:11.551612+0000","last_undegraded":"2023-12-19T14:39:11.551612+0000","last_fullsized":"2023-12-19T14:39:11.551612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.552204+0000","last_change":"2023-12-19T14:39:07.291613+0000","last_active":"2023-12-19T14:39:11.552204+0000","last_peered":"2023-12-19T14:39:11.552204+0000","last_clean":"2023-12-19T14:39:11.552204+0000","last_became_active":"2023-12-19T14:39:07.291277+0000","last_became_peered":"2023-12-19T14:39:07.291277+0000","last_unstale":"2023-12-19T14:39:11.552204+0000","last_undegraded":"2023-12-19T14:39:11.552204+0000","last_fullsized":"2023-12-19T14:39:11.552204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:23.914755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544957+0000","last_change":"2023-12-19T14:39:07.294351+0000","last_active":"2023-12-19T14:39:11.544957+0000","last_peered":"2023-12-19T14:39:11.544957+0000","last_clean":"2023-12-19T14:39:11.544957+0000","last_became_active":"2023-12-19T14:39:07.293414+0000","last_became_peered":"2023-12-19T14:39:07.293414+0000","last_unstale":"2023-12-19T14:39:11.544957+0000","last_undegraded":"2023-12-19T14:39:11.544957+0000","last_fullsized":"2023-12-19T14:39:11.544957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559855+0000","last_change":"2023-12-19T14:39:07.287045+0000","last_active":"2023-12-19T14:39:11.559855+0000","last_peered":"2023-12-19T14:39:11.559855+0000","last_clean":"2023-12-19T14:39:11.559855+0000","last_became_active":"2023-12-19T14:39:07.286711+0000","last_became_peered":"2023-12-19T14:39:07.286711+0000","last_unstale":"2023-12-19T14:39:11.559855+0000","last_undegraded":"2023-12-19T14:39:11.559855+0000","last_fullsized":"2023-12-19T14:39:11.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545892+0000","last_change":"2023-12-19T14:39:07.284424+0000","last_active":"2023-12-19T14:39:11.545892+0000","last_peered":"2023-12-19T14:39:11.545892+0000","last_clean":"2023-12-19T14:39:11.545892+0000","last_became_active":"2023-12-19T14:39:07.284121+0000","last_became_peered":"2023-12-19T14:39:07.284121+0000","last_unstale":"2023-12-19T14:39:11.545892+0000","last_undegraded":"2023-12-19T14:39:11.545892+0000","last_fullsized":"2023-12-19T14:39:11.545892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540559+0000","last_change":"2023-12-19T14:39:07.280745+0000","last_active":"2023-12-19T14:39:11.540559+0000","last_peered":"2023-12-19T14:39:11.540559+0000","last_clean":"2023-12-19T14:39:11.540559+0000","last_became_active":"2023-12-19T14:39:07.280443+0000","last_became_peered":"2023-12-19T14:39:07.280443+0000","last_unstale":"2023-12-19T14:39:11.540559+0000","last_undegraded":"2023-12-19T14:39:11.540559+0000","last_fullsized":"2023-12-19T14:39:11.540559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550784+0000","last_change":"2023-12-19T14:39:07.288049+0000","last_active":"2023-12-19T14:39:11.550784+0000","last_peered":"2023-12-19T14:39:11.550784+0000","last_clean":"2023-12-19T14:39:11.550784+0000","last_became_active":"2023-12-19T14:39:07.287758+0000","last_became_peered":"2023-12-19T14:39:07.287758+0000","last_unstale":"2023-12-19T14:39:11.550784+0000","last_undegraded":"2023-12-19T14:39:11.550784+0000","last_fullsized":"2023-12-19T14:39:11.550784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.558076+0000","last_change":"2023-12-19T14:39:07.284681+0000","last_active":"2023-12-19T14:39:11.558076+0000","last_peered":"2023-12-19T14:39:11.558076+0000","last_clean":"2023-12-19T14:39:11.558076+0000","last_became_active":"2023-12-19T14:39:07.283596+0000","last_became_peered":"2023-12-19T14:39:07.283596+0000","last_unstale":"2023-12-19T14:39:11.558076+0000","last_undegraded":"2023-12-19T14:39:11.558076+0000","last_fullsized":"2023-12-19T14:39:11.558076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545014+0000","last_change":"2023-12-19T14:39:07.293485+0000","last_active":"2023-12-19T14:39:11.545014+0000","last_peered":"2023-12-19T14:39:11.545014+0000","last_clean":"2023-12-19T14:39:11.545014+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.545014+0000","last_undegraded":"2023-12-19T14:39:11.545014+0000","last_fullsized":"2023-12-19T14:39:11.545014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543694+0000","last_change":"2023-12-19T14:39:07.283853+0000","last_active":"2023-12-19T14:39:11.543694+0000","last_peered":"2023-12-19T14:39:11.543694+0000","last_clean":"2023-12-19T14:39:11.543694+0000","last_became_active":"2023-12-19T14:39:07.283616+0000","last_became_peered":"2023-12-19T14:39:07.283616+0000","last_unstale":"2023-12-19T14:39:11.543694+0000","last_undegraded":"2023-12-19T14:39:11.543694+0000","last_fullsized":"2023-12-19T14:39:11.543694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544936+0000","last_change":"2023-12-19T14:39:07.294504+0000","last_active":"2023-12-19T14:39:11.544936+0000","last_peered":"2023-12-19T14:39:11.544936+0000","last_clean":"2023-12-19T14:39:11.544936+0000","last_became_active":"2023-12-19T14:39:07.293953+0000","last_became_peered":"2023-12-19T14:39:07.293953+0000","last_unstale":"2023-12-19T14:39:11.544936+0000","last_undegraded":"2023-12-19T14:39:11.544936+0000","last_fullsized":"2023-12-19T14:39:11.544936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559798+0000","last_change":"2023-12-19T14:39:07.296762+0000","last_active":"2023-12-19T14:39:11.559798+0000","last_peered":"2023-12-19T14:39:11.559798+0000","last_clean":"2023-12-19T14:39:11.559798+0000","last_became_active":"2023-12-19T14:39:07.295928+0000","last_became_peered":"2023-12-19T14:39:07.295928+0000","last_unstale":"2023-12-19T14:39:11.559798+0000","last_undegraded":"2023-12-19T14:39:11.559798+0000","last_fullsized":"2023-12-19T14:39:11.559798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549583+0000","last_change":"2023-12-19T14:39:07.280149+0000","last_active":"2023-12-19T14:39:11.549583+0000","last_peered":"2023-12-19T14:39:11.549583+0000","last_clean":"2023-12-19T14:39:11.549583+0000","last_became_active":"2023-12-19T14:39:07.279877+0000","last_became_peered":"2023-12-19T14:39:07.279877+0000","last_unstale":"2023-12-19T14:39:11.549583+0000","last_undegraded":"2023-12-19T14:39:11.549583+0000","last_fullsized":"2023-12-19T14:39:11.549583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:59:49.003736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570730+0000","last_change":"2023-12-19T14:39:07.288397+0000","last_active":"2023-12-19T14:39:11.570730+0000","last_peered":"2023-12-19T14:39:11.570730+0000","last_clean":"2023-12-19T14:39:11.570730+0000","last_became_active":"2023-12-19T14:39:07.287749+0000","last_became_peered":"2023-12-19T14:39:07.287749+0000","last_unstale":"2023-12-19T14:39:11.570730+0000","last_undegraded":"2023-12-19T14:39:11.570730+0000","last_fullsized":"2023-12-19T14:39:11.570730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555145+0000","last_change":"2023-12-19T14:39:07.276259+0000","last_active":"2023-12-19T14:39:11.555145+0000","last_peered":"2023-12-19T14:39:11.555145+0000","last_clean":"2023-12-19T14:39:11.555145+0000","last_became_active":"2023-12-19T14:39:07.275951+0000","last_became_peered":"2023-12-19T14:39:07.275951+0000","last_unstale":"2023-12-19T14:39:11.555145+0000","last_undegraded":"2023-12-19T14:39:11.555145+0000","last_fullsized":"2023-12-19T14:39:11.555145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549564+0000","last_change":"2023-12-19T14:39:07.295581+0000","last_active":"2023-12-19T14:39:11.549564+0000","last_peered":"2023-12-19T14:39:11.549564+0000","last_clean":"2023-12-19T14:39:11.549564+0000","last_became_active":"2023-12-19T14:39:07.293567+0000","last_became_peered":"2023-12-19T14:39:07.293567+0000","last_unstale":"2023-12-19T14:39:11.549564+0000","last_undegraded":"2023-12-19T14:39:11.549564+0000","last_fullsized":"2023-12-19T14:39:11.549564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543139+0000","last_change":"2023-12-19T14:39:07.293875+0000","last_active":"2023-12-19T14:39:11.543139+0000","last_peered":"2023-12-19T14:39:11.543139+0000","last_clean":"2023-12-19T14:39:11.543139+0000","last_became_active":"2023-12-19T14:39:07.293381+0000","last_became_peered":"2023-12-19T14:39:07.293381+0000","last_unstale":"2023-12-19T14:39:11.543139+0000","last_undegraded":"2023-12-19T14:39:11.543139+0000","last_fullsized":"2023-12-19T14:39:11.543139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540124+0000","last_change":"2023-12-19T14:39:07.261761+0000","last_active":"2023-12-19T14:39:11.540124+0000","last_peered":"2023-12-19T14:39:11.540124+0000","last_clean":"2023-12-19T14:39:11.540124+0000","last_became_active":"2023-12-19T14:39:07.261446+0000","last_became_peered":"2023-12-19T14:39:07.261446+0000","last_unstale":"2023-12-19T14:39:11.540124+0000","last_undegraded":"2023-12-19T14:39:11.540124+0000","last_fullsized":"2023-12-19T14:39:11.540124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556947+0000","last_change":"2023-12-19T14:39:07.290139+0000","last_active":"2023-12-19T14:39:11.556947+0000","last_peered":"2023-12-19T14:39:11.556947+0000","last_clean":"2023-12-19T14:39:11.556947+0000","last_became_active":"2023-12-19T14:39:07.289843+0000","last_became_peered":"2023-12-19T14:39:07.289843+0000","last_unstale":"2023-12-19T14:39:11.556947+0000","last_undegraded":"2023-12-19T14:39:11.556947+0000","last_fullsized":"2023-12-19T14:39:11.556947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543821+0000","last_change":"2023-12-19T14:39:07.281738+0000","last_active":"2023-12-19T14:39:11.543821+0000","last_peered":"2023-12-19T14:39:11.543821+0000","last_clean":"2023-12-19T14:39:11.543821+0000","last_became_active":"2023-12-19T14:39:07.281443+0000","last_became_peered":"2023-12-19T14:39:07.281443+0000","last_unstale":"2023-12-19T14:39:11.543821+0000","last_undegraded":"2023-12-19T14:39:11.543821+0000","last_fullsized":"2023-12-19T14:39:11.543821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546820+0000","last_change":"2023-12-19T14:39:07.276674+0000","last_active":"2023-12-19T14:39:11.546820+0000","last_peered":"2023-12-19T14:39:11.546820+0000","last_clean":"2023-12-19T14:39:11.546820+0000","last_became_active":"2023-12-19T14:39:07.276368+0000","last_became_peered":"2023-12-19T14:39:07.276368+0000","last_unstale":"2023-12-19T14:39:11.546820+0000","last_undegraded":"2023-12-19T14:39:11.546820+0000","last_fullsized":"2023-12-19T14:39:11.546820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:35.210805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549881+0000","last_change":"2023-12-19T14:39:07.281026+0000","last_active":"2023-12-19T14:39:11.549881+0000","last_peered":"2023-12-19T14:39:11.549881+0000","last_clean":"2023-12-19T14:39:11.549881+0000","last_became_active":"2023-12-19T14:39:07.280689+0000","last_became_peered":"2023-12-19T14:39:07.280689+0000","last_unstale":"2023-12-19T14:39:11.549881+0000","last_undegraded":"2023-12-19T14:39:11.549881+0000","last_fullsized":"2023-12-19T14:39:11.549881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550701+0000","last_change":"2023-12-19T14:39:07.280698+0000","last_active":"2023-12-19T14:39:11.550701+0000","last_peered":"2023-12-19T14:39:11.550701+0000","last_clean":"2023-12-19T14:39:11.550701+0000","last_became_active":"2023-12-19T14:39:07.280406+0000","last_became_peered":"2023-12-19T14:39:07.280406+0000","last_unstale":"2023-12-19T14:39:11.550701+0000","last_undegraded":"2023-12-19T14:39:11.550701+0000","last_fullsized":"2023-12-19T14:39:11.550701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570926+0000","last_change":"2023-12-19T14:39:07.284712+0000","last_active":"2023-12-19T14:39:11.570926+0000","last_peered":"2023-12-19T14:39:11.570926+0000","last_clean":"2023-12-19T14:39:11.570926+0000","last_became_active":"2023-12-19T14:39:07.283598+0000","last_became_peered":"2023-12-19T14:39:07.283598+0000","last_unstale":"2023-12-19T14:39:11.570926+0000","last_undegraded":"2023-12-19T14:39:11.570926+0000","last_fullsized":"2023-12-19T14:39:11.570926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544886+0000","last_change":"2023-12-19T14:39:07.294169+0000","last_active":"2023-12-19T14:39:11.544886+0000","last_peered":"2023-12-19T14:39:11.544886+0000","last_clean":"2023-12-19T14:39:11.544886+0000","last_became_active":"2023-12-19T14:39:07.293746+0000","last_became_peered":"2023-12-19T14:39:07.293746+0000","last_unstale":"2023-12-19T14:39:11.544886+0000","last_undegraded":"2023-12-19T14:39:11.544886+0000","last_fullsized":"2023-12-19T14:39:11.544886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570207+0000","last_change":"2023-12-19T14:39:07.291661+0000","last_active":"2023-12-19T14:39:11.570207+0000","last_peered":"2023-12-19T14:39:11.570207+0000","last_clean":"2023-12-19T14:39:11.570207+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.570207+0000","last_undegraded":"2023-12-19T14:39:11.570207+0000","last_fullsized":"2023-12-19T14:39:11.570207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:34:24.416865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544859+0000","last_change":"2023-12-19T14:39:07.297368+0000","last_active":"2023-12-19T14:39:11.544859+0000","last_peered":"2023-12-19T14:39:11.544859+0000","last_clean":"2023-12-19T14:39:11.544859+0000","last_became_active":"2023-12-19T14:39:07.297082+0000","last_became_peered":"2023-12-19T14:39:07.297082+0000","last_unstale":"2023-12-19T14:39:11.544859+0000","last_undegraded":"2023-12-19T14:39:11.544859+0000","last_fullsized":"2023-12-19T14:39:11.544859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570469+0000","last_change":"2023-12-19T14:39:07.287921+0000","last_active":"2023-12-19T14:39:11.570469+0000","last_peered":"2023-12-19T14:39:11.570469+0000","last_clean":"2023-12-19T14:39:11.570469+0000","last_became_active":"2023-12-19T14:39:07.287493+0000","last_became_peered":"2023-12-19T14:39:07.287493+0000","last_unstale":"2023-12-19T14:39:11.570469+0000","last_undegraded":"2023-12-19T14:39:11.570469+0000","last_fullsized":"2023-12-19T14:39:11.570469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545732+0000","last_change":"2023-12-19T14:39:07.290935+0000","last_active":"2023-12-19T14:39:11.545732+0000","last_peered":"2023-12-19T14:39:11.545732+0000","last_clean":"2023-12-19T14:39:11.545732+0000","last_became_active":"2023-12-19T14:39:07.290614+0000","last_became_peered":"2023-12-19T14:39:07.290614+0000","last_unstale":"2023-12-19T14:39:11.545732+0000","last_undegraded":"2023-12-19T14:39:11.545732+0000","last_fullsized":"2023-12-19T14:39:11.545732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.560160+0000","last_change":"2023-12-19T14:39:07.287280+0000","last_active":"2023-12-19T14:39:11.560160+0000","last_peered":"2023-12-19T14:39:11.560160+0000","last_clean":"2023-12-19T14:39:11.560160+0000","last_became_active":"2023-12-19T14:39:07.286949+0000","last_became_peered":"2023-12-19T14:39:07.286949+0000","last_unstale":"2023-12-19T14:39:11.560160+0000","last_undegraded":"2023-12-19T14:39:11.560160+0000","last_fullsized":"2023-12-19T14:39:11.560160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540585+0000","last_change":"2023-12-19T14:39:07.295118+0000","last_active":"2023-12-19T14:39:11.540585+0000","last_peered":"2023-12-19T14:39:11.540585+0000","last_clean":"2023-12-19T14:39:11.540585+0000","last_became_active":"2023-12-19T14:39:07.294282+0000","last_became_peered":"2023-12-19T14:39:07.294282+0000","last_unstale":"2023-12-19T14:39:11.540585+0000","last_undegraded":"2023-12-19T14:39:11.540585+0000","last_fullsized":"2023-12-19T14:39:11.540585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570810+0000","last_change":"2023-12-19T14:39:07.283979+0000","last_active":"2023-12-19T14:39:11.570810+0000","last_peered":"2023-12-19T14:39:11.570810+0000","last_clean":"2023-12-19T14:39:11.570810+0000","last_became_active":"2023-12-19T14:39:07.280235+0000","last_became_peered":"2023-12-19T14:39:07.280235+0000","last_unstale":"2023-12-19T14:39:11.570810+0000","last_undegraded":"2023-12-19T14:39:11.570810+0000","last_fullsized":"2023-12-19T14:39:11.570810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2889999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9910000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301628,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070212,"statfs":{"total":96636764160,"available":96327897088,"internally_reserved":0,"allocated":249856,"data_stored":98477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302220,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069620,"statfs":{"total":96636764160,"available":96327290880,"internally_reserved":0,"allocated":856064,"data_stored":689455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.298}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3730000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5129999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:39:23.326 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:39:23.328 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-19T14:39:23.329 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-19T14:39:23.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-19T14:39:23.329 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-19T14:39:23.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 Processor -- start 2023-12-19T14:39:23.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 -- start start 2023-12-19T14:39:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12a0069100 0x7f12a00cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a0068c70 con 0x7f12a0159340 2023-12-19T14:39:23.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a0068060 con 0x7f12a0069100 2023-12-19T14:39:23.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a76ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0068340 con 0x7f12a0158f70 2023-12-19T14:39:23.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a544a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a4c49700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12a0069100 0x7f12a00cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a544a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46174/0 (socket says 172.21.15.50:46174) 2023-12-19T14:39:23.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.529+0000 7f12a544a700 1 -- 172.21.15.50:0/2826135091 learned_addr learned my addr 172.21.15.50:0/2826135091 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a4c49700 1 -- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 msgr2=0x7f12a0067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a4c49700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a4c49700 1 -- 172.21.15.50:0/2826135091 >> v1:172.21.15.125:6789/0 conn(0x7f12a0159340 legacy=0x7f12a00cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a4c49700 1 -- 172.21.15.50:0/2826135091 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0068620 con 0x7f12a0069100 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a544a700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:23.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a4c49700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0069100 0x7f12a00cf7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12900098e0 tx=0x7f12900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9a68fd745c3545b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f1297fff700 1 -- 172.21.15.50:0/2826135091 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129000b640 con 0x7f12a0069100 2023-12-19T14:39:23.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f1297fff700 1 -- 172.21.15.50:0/2826135091 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129000b7a0 con 0x7f12a0069100 2023-12-19T14:39:23.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f1297fff700 1 -- 172.21.15.50:0/2826135091 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129000bb40 con 0x7f12a0069100 2023-12-19T14:39:23.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 -- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0069100 msgr2=0x7f12a00cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0069100 0x7f12a00cf7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12900098e0 tx=0x7f12900091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 -- 172.21.15.50:0/2826135091 shutdown_connections 2023-12-19T14:39:23.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0069100 0x7f12a00cf7e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 --2- 172.21.15.50:0/2826135091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0158f70 0x7f12a0067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 -- 172.21.15.50:0/2826135091 >> 172.21.15.50:0/2826135091 conn(0x7f12a00ccc40 msgr2=0x7f12a00cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:23.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 -- 172.21.15.50:0/2826135091 shutdown_connections 2023-12-19T14:39:23.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.533+0000 7f12a76ae700 1 -- 172.21.15.50:0/2826135091 wait complete. 2023-12-19T14:39:23.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.537+0000 7f12a76ae700 1 Processor -- start 2023-12-19T14:39:23.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.537+0000 7f12a76ae700 1 -- start start 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0069100 0x7f12a0169f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0159340 0x7f12a0157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a016aec0 con 0x7f12a0158f70 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a015ca60 con 0x7f12a0159340 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a016a620 con 0x7f12a0069100 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53612/0 (socket says 172.21.15.50:53612) 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 -- 172.21.15.50:0/3629432074 learned_addr learned my addr 172.21.15.50:0/3629432074 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a544a700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0069100 0x7f12a0169f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a5c4b700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0159340 0x7f12a0157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 -- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0069100 msgr2=0x7f12a0169f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0069100 0x7f12a0169f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 -- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0159340 msgr2=0x7f12a0157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0159340 0x7f12a0157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 -- 172.21.15.50:0/3629432074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0094f0 con 0x7f12a0158f70 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a4c49700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f12a006b980 tx=0x7f1290017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129001bc40 con 0x7f12a0158f70 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129001bde0 con 0x7f12a0158f70 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290011670 con 0x7f12a0158f70 2023-12-19T14:39:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1290009420 con 0x7f12a0158f70 2023-12-19T14:39:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a016abe0 con 0x7f12a0158f70 2023-12-19T14:39:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a0071120 con 0x7f12a0158f70 2023-12-19T14:39:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.545+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f12900119e0 con 0x7f12a0158f70 2023-12-19T14:39:23.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.549+0000 7f12957fa700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 0x7f1284062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.549+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f129007e170 con 0x7f12a0158f70 2023-12-19T14:39:23.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.549+0000 7f12a544a700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 0x7f1284062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.549+0000 7f12a544a700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 0x7f1284062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f129c006d20 tx=0x7f129c009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.557+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f129004e3d0 con 0x7f12a0158f70 2023-12-19T14:39:23.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.765+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f12a0068900 con 0x7f12a0158f70 2023-12-19T14:39:23.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.769+0000 7f12957fa700 1 -- 172.21.15.50:0/3629432074 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+154139 (secure 0 0 0) 0x7f12900519f0 con 0x7f12a0158f70 2023-12-19T14:39:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 msgr2=0x7f1284062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 0x7f1284062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f129c006d20 tx=0x7f129c009250 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 msgr2=0x7f12a015c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f12a006b980 tx=0x7f1290017000 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 shutdown_connections 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12a0159340 0x7f12a0157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1284060550 0x7f1284062a00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12a0158f70 0x7f12a015c520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 --2- 172.21.15.50:0/3629432074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12a0069100 0x7f12a0169f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 >> 172.21.15.50:0/3629432074 conn(0x7f12a00ccc40 msgr2=0x7f12a00c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 shutdown_connections 2023-12-19T14:39:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.777+0000 7f12a76ae700 1 -- 172.21.15.50:0/3629432074 wait complete. 2023-12-19T14:39:23.803 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":33827},{"type":"v1","addr":"172.21.15.125:6833","nonce":33827}]},"active_addr":"172.21.15.125:6833/33827","active_change":"2023-12-19T14:38:48.104907+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":4107501911}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":3449382498}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":1210582553}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":751498889}]}]} 2023-12-19T14:39:23.806 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-19T14:39:23.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-19T14:39:23.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:23.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.969+0000 7f610fef0700 1 Processor -- start 2023-12-19T14:39:23.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.969+0000 7f610fef0700 1 -- start start 2023-12-19T14:39:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610fef0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6108067c90 0x7f6108068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610fef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 0x7f61080c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610fef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61080c6630 con 0x7f610814fba0 2023-12-19T14:39:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610fef0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080c6910 con 0x7f6108067c90 2023-12-19T14:39:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610fef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61080c6bf0 con 0x7f61080c5900 2023-12-19T14:39:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6108067c90 0x7f6108068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6108067c90 0x7f6108068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53614/0 (socket says 172.21.15.50:53614) 2023-12-19T14:39:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 -- 172.21.15.50:0/499950637 learned_addr learned my addr 172.21.15.50:0/499950637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610e48d700 1 --2- 172.21.15.50:0/499950637 >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 0x7f61080c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 -- 172.21.15.50:0/499950637 >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 msgr2=0x7f61080c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 --2- 172.21.15.50:0/499950637 >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 0x7f61080c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 -- 172.21.15.50:0/499950637 >> v1:172.21.15.125:6789/0 conn(0x7f610814fba0 legacy=0x7f6108150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610dc8c700 1 -- 172.21.15.50:0/499950637 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080c6ed0 con 0x7f6108067c90 2023-12-19T14:39:23.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.973+0000 7f610e48d700 1 --2- 172.21.15.50:0/499950637 >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 0x7f61080c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610dc8c700 1 --2- 172.21.15.50:0/499950637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f6108068060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6108068a60 tx=0x7f6104009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827c08b5b051b65a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610cc8a700 1 -- 172.21.15.50:0/499950637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400b860 con 0x7f6108067c90 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610cc8a700 1 -- 172.21.15.50:0/499950637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610400b9c0 con 0x7f6108067c90 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610cc8a700 1 -- 172.21.15.50:0/499950637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400bd60 con 0x7f6108067c90 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 -- 172.21.15.50:0/499950637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 msgr2=0x7f6108068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 --2- 172.21.15.50:0/499950637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f6108068060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6108068a60 tx=0x7f6104009250 comp rx=0 tx=0).stop 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 -- 172.21.15.50:0/499950637 shutdown_connections 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 --2- 172.21.15.50:0/499950637 >> v2:172.21.15.125:3300/0 conn(0x7f61080c5900 0x7f61080c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 --2- 172.21.15.50:0/499950637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f6108068060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 -- 172.21.15.50:0/499950637 >> 172.21.15.50:0/499950637 conn(0x7f61080cd6d0 msgr2=0x7f610806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 -- 172.21.15.50:0/499950637 shutdown_connections 2023-12-19T14:39:23.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.977+0000 7f610fef0700 1 -- 172.21.15.50:0/499950637 wait complete. 2023-12-19T14:39:23.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 Processor -- start 2023-12-19T14:39:23.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 -- start start 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61080c5900 0x7f6108150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f610814fba0 0x7f6108150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080c9cb0 con 0x7f6108067c90 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080c9130 con 0x7f610814fba0 2023-12-19T14:39:23.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610fef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080c9410 con 0x7f61080c5900 2023-12-19T14:39:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53630/0 (socket says 172.21.15.50:53630) 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 -- 172.21.15.50:0/3379114031 learned_addr learned my addr 172.21.15.50:0/3379114031 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610e48d700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f610814fba0 0x7f6108150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610d48b700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61080c5900 0x7f6108150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 -- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61080c5900 msgr2=0x7f6108150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61080c5900 0x7f6108150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 -- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f610814fba0 msgr2=0x7f6108150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f610814fba0 0x7f6108150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 -- 172.21.15.50:0/3379114031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040094f0 con 0x7f6108067c90 2023-12-19T14:39:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.981+0000 7f610dc8c700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6104009290 tx=0x7f6104017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104011e60 con 0x7f6108067c90 2023-12-19T14:39:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610401a660 con 0x7f6108067c90 2023-12-19T14:39:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61080c96f0 con 0x7f6108067c90 2023-12-19T14:39:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610401aa20 con 0x7f6108067c90 2023-12-19T14:39:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6108140930 con 0x7f6108067c90 2023-12-19T14:39:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6108071120 con 0x7f6108067c90 2023-12-19T14:39:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.985+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f610401b430 con 0x7f6108067c90 2023-12-19T14:39:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.989+0000 7f60fdffb700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 0x7f60ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.989+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f610407e270 con 0x7f6108067c90 2023-12-19T14:39:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.989+0000 7f610d48b700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 0x7f60ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.993+0000 7f610d48b700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 0x7f60ec062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61080b3260 tx=0x7f60f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:23.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:23.997+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f610404e570 con 0x7f6108067c90 2023-12-19T14:39:24.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.169+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61080642a0 con 0x7f6108067c90 2023-12-19T14:39:24.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f60fdffb700 1 -- 172.21.15.50:0/3379114031 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f6104051b90 con 0x7f6108067c90 2023-12-19T14:39:24.175 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:24.175 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 msgr2=0x7f60ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 0x7f60ec062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f61080b3260 tx=0x7f60f8009040 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 msgr2=0x7f61081403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6104009290 tx=0x7f6104017000 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 shutdown_connections 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f610814fba0 0x7f6108150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60ec060550 0x7f60ec062a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61080c5900 0x7f6108150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 --2- 172.21.15.50:0/3379114031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6108067c90 0x7f61081403f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.173+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 >> 172.21.15.50:0/3379114031 conn(0x7f61080cd6d0 msgr2=0x7f61080cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.177+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 shutdown_connections 2023-12-19T14:39:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.177+0000 7f610fef0700 1 -- 172.21.15.50:0/3379114031 wait complete. 2023-12-19T14:39:24.195 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-19T14:39:24.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.357+0000 7f950af85700 1 Processor -- start 2023-12-19T14:39:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 -- start start 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95040ae510 0x7f95040c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950406dc20 con 0x7f950406d180 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950406d940 con 0x7f95040ae140 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f950af85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95040c6e20 con 0x7f95040ae510 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9508d21700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9508d21700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53644/0 (socket says 172.21.15.50:53644) 2023-12-19T14:39:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9508d21700 1 -- 172.21.15.50:0/376128 learned_addr learned my addr 172.21.15.50:0/376128 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9503fff700 1 --2- 172.21.15.50:0/376128 >> v2:172.21.15.125:3300/0 conn(0x7f95040ae510 0x7f95040c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9503fff700 1 -- 172.21.15.50:0/376128 >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 msgr2=0x7f95040a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9503fff700 1 --2- 172.21.15.50:0/376128 >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9503fff700 1 -- 172.21.15.50:0/376128 >> v1:172.21.15.125:6789/0 conn(0x7f950406d180 legacy=0x7f95040680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9503fff700 1 -- 172.21.15.50:0/376128 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040c7100 con 0x7f95040ae510 2023-12-19T14:39:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.361+0000 7f9508d21700 1 --2- 172.21.15.50:0/376128 >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f9503fff700 1 --2- 172.21.15.50:0/376128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040ae510 0x7f95040c1d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f950406a0a0 tx=0x7f94f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f13e6eff2edc5c4d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f95037fe700 1 -- 172.21.15.50:0/376128 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0011630 con 0x7f95040ae510 2023-12-19T14:39:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f95037fe700 1 -- 172.21.15.50:0/376128 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f0011790 con 0x7f95040ae510 2023-12-19T14:39:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f95037fe700 1 -- 172.21.15.50:0/376128 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0011b30 con 0x7f95040ae510 2023-12-19T14:39:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 -- 172.21.15.50:0/376128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040ae510 msgr2=0x7f95040c1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 --2- 172.21.15.50:0/376128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040ae510 0x7f95040c1d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f950406a0a0 tx=0x7f94f00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 -- 172.21.15.50:0/376128 shutdown_connections 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 --2- 172.21.15.50:0/376128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040ae510 0x7f95040c1d60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 --2- 172.21.15.50:0/376128 >> v2:172.21.15.50:3300/0 conn(0x7f95040ae140 0x7f95040a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 -- 172.21.15.50:0/376128 >> 172.21.15.50:0/376128 conn(0x7f95040c8850 msgr2=0x7f95040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 -- 172.21.15.50:0/376128 shutdown_connections 2023-12-19T14:39:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.365+0000 7f950af85700 1 -- 172.21.15.50:0/376128 wait complete. 2023-12-19T14:39:24.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 Processor -- start 2023-12-19T14:39:24.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 -- start start 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95040ae140 0x7f9504150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95040ae510 0x7f950415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95041565f0 con 0x7f95040ae510 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95041568d0 con 0x7f950406d180 2023-12-19T14:39:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f950af85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9504156bb0 con 0x7f95040ae140 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:47978/0 (socket says 172.21.15.50:47978) 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 -- 172.21.15.50:0/3833246811 learned_addr learned my addr 172.21.15.50:0/3833246811 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9509522700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95040ae510 0x7f950415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9503fff700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95040ae140 0x7f9504150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 -- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95040ae140 msgr2=0x7f9504150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95040ae140 0x7f9504150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 -- 172.21.15.50:0/3833246811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95040ae510 msgr2=0x7f950415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95040ae510 0x7f950415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.369+0000 7f9508d21700 1 -- 172.21.15.50:0/3833246811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80094f0 con 0x7f950406d180 2023-12-19T14:39:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f9508d21700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94f800ec50 tx=0x7f94f8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f800c850 con 0x7f950406d180 2023-12-19T14:39:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f800c9b0 con 0x7f950406d180 2023-12-19T14:39:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0009420 con 0x7f950406d180 2023-12-19T14:39:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f800cd50 con 0x7f950406d180 2023-12-19T14:39:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9504157170 con 0x7f950406d180 2023-12-19T14:39:24.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.373+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9504071120 con 0x7f950406d180 2023-12-19T14:39:24.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.377+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f94f8013680 con 0x7f950406d180 2023-12-19T14:39:24.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.377+0000 7f9500ff9700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 0x7f94e8062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.377+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f94f807cb40 con 0x7f950406d180 2023-12-19T14:39:24.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.377+0000 7f9503fff700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 0x7f94e8062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.381+0000 7f9503fff700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 0x7f94e8062ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f950406a0a0 tx=0x7f94f0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.385+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f94f804ce20 con 0x7f950406d180 2023-12-19T14:39:24.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.561+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95040642a0 con 0x7f950406d180 2023-12-19T14:39:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f9500ff9700 1 -- 172.21.15.50:0/3833246811 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f94f8017090 con 0x7f950406d180 2023-12-19T14:39:24.567 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:24.567 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:24.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 msgr2=0x7f94e8062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 0x7f94e8062ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f950406a0a0 tx=0x7f94f0017000 comp rx=0 tx=0).stop 2023-12-19T14:39:24.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 msgr2=0x7f9504157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94f800ec50 tx=0x7f94f8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 shutdown_connections 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f94e8060620 0x7f94e8062ad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95040ae510 0x7f950415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95040ae140 0x7f9504150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 --2- 172.21.15.50:0/3833246811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f950406d180 0x7f9504157690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 >> 172.21.15.50:0/3833246811 conn(0x7f95040c8850 msgr2=0x7f950406a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.565+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 shutdown_connections 2023-12-19T14:39:24.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.569+0000 7f950af85700 1 -- 172.21.15.50:0/3833246811 wait complete. 2023-12-19T14:39:24.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-19T14:39:24.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-19T14:39:24.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-19T14:39:24.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-19T14:39:24.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-19T14:39:24.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-19T14:39:24.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-19T14:39:24.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-19T14:39:24.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 Processor -- start 2023-12-19T14:39:24.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.809+0000 7f749de43700 1 Processor -- start 2023-12-19T14:39:24.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 -- start start 2023-12-19T14:39:24.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 0x7f749815d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 -- start start 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74980afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f74980ca720 0x7f749806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749806a580 con 0x7f74980ca720 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749806ae20 con 0x7f749815cf40 2023-12-19T14:39:24.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749de43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749806df50 con 0x7f749806d9e0 2023-12-19T14:39:24.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2d00ea760 0x7fb2d00501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00eabe0 con 0x7fb2d00ea390 2023-12-19T14:39:24.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d0054250 con 0x7fb2d00e67c0 2023-12-19T14:39:24.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d7fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00506f0 con 0x7fb2d00ea760 2023-12-19T14:39:24.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f7497fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74980afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749ce41700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 0x7f749815d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f7497fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74980afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46212/0 (socket says 172.21.15.50:46212) 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749ce41700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 0x7f749815d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:47990/0 (socket says 172.21.15.50:47990) 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7f749ce41700 1 -- 172.21.15.50:0/1670726948 learned_addr learned my addr 172.21.15.50:0/1670726948 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d77fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2d00ea760 0x7fb2d00501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 --2- 172.21.15.50:0/1670726948 >> v2:172.21.15.50:3300/0 conn(0x7f74980ca720 0x7f749806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d77fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2d00ea760 0x7fb2d00501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53674/0 (socket says 172.21.15.50:53674) 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.813+0000 7fb2d77fe700 1 -- 172.21.15.50:0/1331057854 learned_addr learned my addr 172.21.15.50:0/1331057854 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 -- 172.21.15.50:0/1670726948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 msgr2=0x7f74980afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 --2- 172.21.15.50:0/1670726948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74980afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 -- 172.21.15.50:0/1670726948 >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 msgr2=0x7f749815d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 --2- 172.21.15.50:0/1670726948 >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 0x7f749815d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 -- 172.21.15.50:0/1670726948 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749806a7d0 con 0x7f74980ca720 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d5ffb700 1 -- 172.21.15.50:0/1331057854 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83279630 0 0) 0x7fb2d00eabe0 con 0x7fb2d00ea390 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d5ffb700 1 -- 172.21.15.50:0/1331057854 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0003570 con 0x7fb2d00ea390 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d77fe700 1 -- 172.21.15.50:0/1331057854 >> v1:172.21.15.125:6789/0 conn(0x7fb2d00e67c0 legacy=0x7fb2d0034a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d77fe700 1 -- 172.21.15.50:0/1331057854 >> v1:172.21.15.50:6789/0 conn(0x7fb2d00ea390 legacy=0x7fb2d00e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d77fe700 1 -- 172.21.15.50:0/1331057854 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0053640 con 0x7fb2d00ea760 2023-12-19T14:39:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749d642700 1 --2- 172.21.15.50:0/1670726948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74980ca720 0x7f749806c7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f749806edd0 tx=0x7f74880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d0512f13c33d08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d77fe700 1 --2- 172.21.15.50:0/1331057854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 0x7fb2d00501b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80658c0 tx=0x7fb2d8066cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7716dbf51f8f78 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f74977fe700 1 -- 172.21.15.50:0/1670726948 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488011630 con 0x7f74980ca720 2023-12-19T14:39:24.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d5ffb700 1 -- 172.21.15.50:0/1331057854 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d804be40 con 0x7fb2d00ea760 2023-12-19T14:39:24.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f74977fe700 1 -- 172.21.15.50:0/1670726948 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74880117b0 con 0x7f74980ca720 2023-12-19T14:39:24.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f74977fe700 1 -- 172.21.15.50:0/1670726948 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488011b70 con 0x7f74980ca720 2023-12-19T14:39:24.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d5ffb700 1 -- 172.21.15.50:0/1331057854 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d80008d0 con 0x7fb2d00ea760 2023-12-19T14:39:24.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d5ffb700 1 -- 172.21.15.50:0/1331057854 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8070460 con 0x7fb2d00ea760 2023-12-19T14:39:24.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f27db785700 1 Processor -- start 2023-12-19T14:39:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f27db785700 1 -- start start 2023-12-19T14:39:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749de43700 1 -- 172.21.15.50:0/1670726948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74980ca720 msgr2=0x7f749806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d7fff700 1 -- 172.21.15.50:0/1331057854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 msgr2=0x7fb2d00501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7fb2d7fff700 1 --2- 172.21.15.50:0/1331057854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 0x7fb2d00501b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80658c0 tx=0x7fb2d8066cd0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749de43700 1 --2- 172.21.15.50:0/1670726948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74980ca720 0x7f749806c7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f749806edd0 tx=0x7f74880091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749de43700 1 -- 172.21.15.50:0/1670726948 shutdown_connections 2023-12-19T14:39:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.817+0000 7f749de43700 1 --2- 172.21.15.50:0/1670726948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74980ca720 0x7f749806c7c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f749de43700 1 --2- 172.21.15.50:0/1670726948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74980afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f749de43700 1 --2- 172.21.15.50:0/1670726948 >> v2:172.21.15.125:3300/0 conn(0x7f749815cf40 0x7f749815d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7fb2d7fff700 1 -- 172.21.15.50:0/1331057854 shutdown_connections 2023-12-19T14:39:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f749de43700 1 -- 172.21.15.50:0/1670726948 >> 172.21.15.50:0/1670726948 conn(0x7f7498067bb0 msgr2=0x7f74980cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7fb2d7fff700 1 --2- 172.21.15.50:0/1331057854 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 0x7fb2d00501b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7fb2d7fff700 1 -- 172.21.15.50:0/1331057854 >> 172.21.15.50:0/1331057854 conn(0x7fb2d0054da0 msgr2=0x7fb2d00551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f749de43700 1 -- 172.21.15.50:0/1670726948 shutdown_connections 2023-12-19T14:39:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27db785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f27d4154e50 0x7f27d40ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27db785700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40addf0 con 0x7f27d406b300 2023-12-19T14:39:24.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f749de43700 1 -- 172.21.15.50:0/1670726948 wait complete. 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27d9521700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f27d40ca370 0x7f27d40ad6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46950/0 (socket says 172.21.15.50:46950) 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27d9521700 1 -- 172.21.15.50:0/1590365849 learned_addr learned my addr 172.21.15.50:0/1590365849 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27d9d22700 1 --2- 172.21.15.50:0/1590365849 >> v2:172.21.15.50:3300/0 conn(0x7f27d4154e50 0x7f27d40ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40ae330 con 0x7f27d40ca370 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40ae050 con 0x7f27d4154e50 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7fb2d7fff700 1 -- 172.21.15.50:0/1331057854 shutdown_connections 2023-12-19T14:39:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27cbfff700 1 -- 172.21.15.50:0/1590365849 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332405306 0 0) 0x7f27d40ae330 con 0x7f27d40ca370 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27cbfff700 1 -- 172.21.15.50:0/1590365849 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c0003570 con 0x7f27d40ca370 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f08f9acd700 1 Processor -- start 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7f08f9acd700 1 -- start start 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27d9d22700 1 -- 172.21.15.50:0/1590365849 >> v1:172.21.15.125:6789/0 conn(0x7f27d40ca370 legacy=0x7f27d40ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27d9d22700 1 -- 172.21.15.50:0/1590365849 >> v1:172.21.15.50:6789/0 conn(0x7f27d406b300 legacy=0x7f27d4154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27d9d22700 1 -- 172.21.15.50:0/1590365849 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d406c860 con 0x7f27d4154e50 2023-12-19T14:39:24.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f749de43700 1 Processor -- start 2023-12-19T14:39:24.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.821+0000 7fb2d7fff700 1 -- 172.21.15.50:0/1331057854 wait complete. 2023-12-19T14:39:24.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7fb2d7fff700 1 Processor -- start 2023-12-19T14:39:24.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27d9d22700 1 --2- 172.21.15.50:0/1590365849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d4154e50 0x7f27d40ca740 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098f0 tx=0x7f27c4009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a293d8902690fd94 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f749de43700 1 -- start start 2023-12-19T14:39:24.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7fb2d7fff700 1 -- start start 2023-12-19T14:39:24.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27cbfff700 1 -- 172.21.15.50:0/1590365849 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4017070 con 0x7f27d4154e50 2023-12-19T14:39:24.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f9acd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f08f40b4270 0x7f08f4167f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f37fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f08f415d1d0 0x7f08f406b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37378/0 (socket says 172.21.15.50:37378) 2023-12-19T14:39:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f27cbfff700 1 -- 172.21.15.50:0/1590365849 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400d040 con 0x7f27d4154e50 2023-12-19T14:39:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f2ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f08f415d5a0 0x7f08f40b3b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46952/0 (socket says 172.21.15.50:46952) 2023-12-19T14:39:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d4154e50 msgr2=0x7f27d40ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 --2- 172.21.15.50:0/1590365849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d4154e50 0x7f27d40ca740 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27c40098f0 tx=0x7f27c4009190 comp rx=0 tx=0).stop 2023-12-19T14:39:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f37fe700 1 -- 172.21.15.50:0/2701083975 learned_addr learned my addr 172.21.15.50:0/2701083975 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f40cc8a0 con 0x7f08f415d1d0 2023-12-19T14:39:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f406b9a0 con 0x7f08f415d5a0 2023-12-19T14:39:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f919bc2e700 1 Processor -- start 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f406bc80 con 0x7f08f40b4270 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.825+0000 7f08f3fff700 1 --2- 172.21.15.50:0/2701083975 >> v2:172.21.15.50:3300/0 conn(0x7f08f40b4270 0x7f08f4167f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135015570 0 0) 0x7f08f406b9a0 con 0x7f08f415d5a0 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc003570 con 0x7f08f415d5a0 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224044807 0 0) 0x7f08f40cc8a0 con 0x7f08f415d1d0 2023-12-19T14:39:24.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc003890 con 0x7f08f415d1d0 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74981480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74980ca720 0x7f749815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749815cf40 0x7f749815de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749815e360 con 0x7f749815cf40 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498160510 con 0x7f74980ca720 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74981607f0 con 0x7f749806d9e0 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f3fff700 1 -- 172.21.15.50:0/2701083975 >> v1:172.21.15.125:6789/0 conn(0x7f08f415d5a0 legacy=0x7f08f40b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f3fff700 1 -- 172.21.15.50:0/2701083975 >> v1:172.21.15.50:6789/0 conn(0x7f08f415d1d0 legacy=0x7f08f406b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f3fff700 1 -- 172.21.15.50:0/2701083975 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f406c940 con 0x7f08f40b4270 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74981480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74981480e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46222/0 (socket says 172.21.15.50:46222) 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 -- 172.21.15.50:0/2812844081 learned_addr learned my addr 172.21.15.50:0/2812844081 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f7497fff700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74980ca720 0x7f749815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2d00e67c0 0x7fb2d00fead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f919bc2e700 1 -- start start 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 0x7fb2d004e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 0x7fb2d00e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00dd930 con 0x7fb2d00ea760 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00ff010 con 0x7fb2d00ea390 2023-12-19T14:39:24.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d7fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0038710 con 0x7fb2d00e67c0 2023-12-19T14:39:24.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 -- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74980ca720 msgr2=0x7f749815d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74980ca720 0x7f749815d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 -- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749815cf40 msgr2=0x7f749815de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749815cf40 0x7f749815de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7488009420 con 0x7f749806d9e0 2023-12-19T14:39:24.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 0x7fb2d004e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 0x7fb2d004e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:47998/0 (socket says 172.21.15.50:47998) 2023-12-19T14:39:24.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 -- 172.21.15.50:0/3884863007 learned_addr learned my addr 172.21.15.50:0/3884863007 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d6ffd700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2d00e67c0 0x7fb2d00fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f3fff700 1 --2- 172.21.15.50:0/2701083975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f40b4270 0x7f08f4167f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f08e40098e0 tx=0x7f08e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412b6bac8ea7aecf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 shutdown_connections 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 --2- 172.21.15.50:0/1590365849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d4154e50 0x7f27d40ca740 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 >> 172.21.15.50:0/1590365849 conn(0x7f27d40670e0 msgr2=0x7f27d40674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 -- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2d00e67c0 msgr2=0x7fb2d00fead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2d00e67c0 0x7fb2d00fead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 -- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 msgr2=0x7fb2d00e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2d00ea760 0x7fb2d00e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7fb2d67fc700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0037b40 con 0x7fb2d00ea390 2023-12-19T14:39:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749ce41700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74981480e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e980 tx=0x7f748c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f919bc2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f919bc2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91940b04c0 0x7f919406d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 shutdown_connections 2023-12-19T14:39:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f27db785700 1 -- 172.21.15.50:0/1590365849 wait complete. 2023-12-19T14:39:24.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4011630 con 0x7f08f40b4270 2023-12-19T14:39:24.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c00ee70 con 0x7f749806d9e0 2023-12-19T14:39:24.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7498160ad0 con 0x7f749806d9e0 2023-12-19T14:39:24.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7498161090 con 0x7f749806d9e0 2023-12-19T14:39:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f919bc2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940b0900 con 0x7f91941550a0 2023-12-19T14:39:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c00c850 con 0x7f749806d9e0 2023-12-19T14:39:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f919bc2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91940c9300 con 0x7f91940b04c0 2023-12-19T14:39:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f919bc2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9194159d50 con 0x7f9194155470 2023-12-19T14:39:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2d67fc700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 0x7fb2d004e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c4e0 tx=0x7fb2cc00c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f919a1cb700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91940b04c0 0x7f919406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46232/0 (socket says 172.21.15.50:46232) 2023-12-19T14:39:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 -- 172.21.15.50:0/3505726784 learned_addr learned my addr 172.21.15.50:0/3505726784 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c00cbd0 con 0x7f749806d9e0 2023-12-19T14:39:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e4011790 con 0x7f08f40b4270 2023-12-19T14:39:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.829+0000 7f08f27fc700 1 -- 172.21.15.50:0/2701083975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4011b30 con 0x7f08f40b4270 2023-12-19T14:39:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f40b4270 msgr2=0x7f08f4167f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 --2- 172.21.15.50:0/2701083975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f40b4270 0x7f08f4167f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f08e40098e0 tx=0x7f08e40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011590 con 0x7fb2d00ea390 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc011710 con 0x7fb2d00ea390 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 -- 172.21.15.50:0/3505726784 >> v2:172.21.15.125:3300/0 conn(0x7f91940b04c0 msgr2=0x7f919406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 --2- 172.21.15.50:0/3505726784 >> v2:172.21.15.125:3300/0 conn(0x7f91940b04c0 0x7f919406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 -- 172.21.15.50:0/3505726784 >> v1:172.21.15.50:6789/0 conn(0x7f91941550a0 legacy=0x7f91940c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 -- 172.21.15.50:0/3505726784 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919406c860 con 0x7f9194155470 2023-12-19T14:39:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7478000fd0 con 0x7f749806d9e0 2023-12-19T14:39:24.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc011ad0 con 0x7fb2d00ea390 2023-12-19T14:39:24.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d0037de0 con 0x7fb2d00ea390 2023-12-19T14:39:24.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d0038420 con 0x7fb2d00ea390 2023-12-19T14:39:24.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91991c9700 1 --2- 172.21.15.50:0/3505726784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91840099b0 tx=0x7f9184009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f8ee077dc2c055d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f27db785700 1 Processor -- start 2023-12-19T14:39:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91989c8700 1 -- 172.21.15.50:0/3505726784 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400b860 con 0x7f9194155470 2023-12-19T14:39:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2d0005280 con 0x7fb2d00ea390 2023-12-19T14:39:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f27db785700 1 -- start start 2023-12-19T14:39:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 shutdown_connections 2023-12-19T14:39:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 --2- 172.21.15.50:0/2701083975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f40b4270 0x7f08f4167f70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 >> 172.21.15.50:0/2701083975 conn(0x7f08f4067bb0 msgr2=0x7f08f40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f91989c8700 1 -- 172.21.15.50:0/3505726784 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918400b9c0 con 0x7f9194155470 2023-12-19T14:39:24.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 shutdown_connections 2023-12-19T14:39:24.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.833+0000 7f08f9acd700 1 -- 172.21.15.50:0/2701083975 wait complete. 2023-12-19T14:39:24.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f748c013470 con 0x7f749806d9e0 2023-12-19T14:39:24.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 -- 172.21.15.50:0/3505726784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 msgr2=0x7f91940aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 --2- 172.21.15.50:0/3505726784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91840099b0 tx=0x7f9184009250 comp rx=0 tx=0).stop 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 Processor -- start 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 -- 172.21.15.50:0/3505726784 shutdown_connections 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 --2- 172.21.15.50:0/3505726784 >> v2:172.21.15.125:3300/0 conn(0x7f91940b04c0 0x7f919406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 --2- 172.21.15.50:0/3505726784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9194155470 0x7f91940aff80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 -- 172.21.15.50:0/3505726784 >> 172.21.15.50:0/3505726784 conn(0x7f91940670e0 msgr2=0x7f91940674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d406b300 0x7f27d40c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 0x7f27d4170a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f27d4154e50 0x7f27d4170fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40b17c0 con 0x7f27d406b300 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d416d360 con 0x7f27d40ca370 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27db785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4146760 con 0x7f27d4154e50 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 -- 172.21.15.50:0/3505726784 shutdown_connections 2023-12-19T14:39:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f919bc2e700 1 -- 172.21.15.50:0/3505726784 wait complete. 2023-12-19T14:39:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7484060550 0x7f7484062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 -- start start 2023-12-19T14:39:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 Processor -- start 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f748c07ccf0 con 0x7f749806d9e0 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f7484069b80 0x7f748406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748406c680 con 0x7f7484069b80 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f748c082240 con 0x7f749806d9e0 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f7497fff700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7484060550 0x7f7484062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb2cc01b3b0 con 0x7fb2d00ea390 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 -- start start 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f08f40b4270 0x7f08f414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f415d1d0 0x7f08f416fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f415d5a0 0x7f08f416df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f415a720 con 0x7f08f415d1d0 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f415aa00 con 0x7f08f415d5a0 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f08f9acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f415ace0 con 0x7f08f40b4270 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc0c6900 0x7fb3fc0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3fc15cc20 0x7fb3fc15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc06b470 con 0x7fb3fc06b900 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc06e110 con 0x7fb3fc15cc20 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb4013cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc161d10 con 0x7fb3fc0c6900 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7fb3fbfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc0c6900 0x7fb3fc0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f749d642700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f7484069b80 0x7f748406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27d8d20700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 0x7f27d4170a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27d8d20700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 0x7f27d4170a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48002/0 (socket says 172.21.15.50:48002) 2023-12-19T14:39:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27d8d20700 1 -- 172.21.15.50:0/398507136 learned_addr learned my addr 172.21.15.50:0/398507136 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.837+0000 7f27d9521700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d406b300 0x7f27d40c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9d22700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f27d4154e50 0x7f27d4170fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2c40605a0 0x7fb2c4062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f415d5a0 0x7f08f416df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f919bc2e700 1 Processor -- start 2023-12-19T14:39:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb3fb7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb3fc06b900 0x7fb3fc06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37392/0 (socket says 172.21.15.50:37392) 2023-12-19T14:39:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb3fb7fe700 1 -- 172.21.15.50:0/3163036344 learned_addr learned my addr 172.21.15.50:0/3163036344 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb2cc07c5e0 con 0x7fb2d00ea390 2023-12-19T14:39:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7fb2c4069a60 0x7fb2c406be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f415d1d0 0x7f08f416fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f415d1d0 0x7f08f416fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53730/0 (socket says 172.21.15.50:53730) 2023-12-19T14:39:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f2ffd700 1 -- 172.21.15.50:0/4067457584 learned_addr learned my addr 172.21.15.50:0/4067457584 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f08f40b4270 0x7f08f414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 --2- 172.21.15.50:0/3163036344 >> v2:172.21.15.125:3300/0 conn(0x7fb3fc15cc20 0x7fb3fc15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2c406c560 con 0x7fb2c4069a60 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2cc07c9b0 con 0x7fb2d00ea390 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9521700 1 -- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f27d4154e50 msgr2=0x7f27d4170fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 -- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f415d5a0 msgr2=0x7f08f416df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08f415d5a0 0x7f08f416df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 -- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f415d1d0 msgr2=0x7f08f416fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08f415d1d0 0x7f08f416fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9521700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f27d4154e50 0x7f27d4170fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4009420 con 0x7f08f40b4270 2023-12-19T14:39:24.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2d6ffd700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2c40605a0 0x7fb2c4062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9521700 1 -- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 msgr2=0x7f27d4170a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2d77fe700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7fb2c4069a60 0x7fb2c406be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f749d642700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f7484069b80 0x7f748406bfb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f37fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f08f40b4270 0x7f08f414d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00cc20 tx=0x7f08ec001a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 -- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc0c6900 msgr2=0x7fb3fc0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 --2- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc0c6900 0x7fb3fc0c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f7497fff700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7484060550 0x7f7484062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74980b33a0 tx=0x7f749000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9521700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 0x7f27d4170a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9521700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4009430 con 0x7f27d406b300 2023-12-19T14:39:24.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d8d20700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d40ca370 0x7f27d4170a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f27d9d22700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f27d4154e50 0x7f27d4170fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2d77fe700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7fb2c4069a60 0x7fb2c406be90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2d6ffd700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2c40605a0 0x7fb2c4062a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007fd0 tx=0x7fb2c8007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f919bc2e700 1 -- start start 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 -- 172.21.15.50:0/3163036344 >> v1:172.21.15.50:6789/0 conn(0x7fb3fc06b900 legacy=0x7fb3fc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 -- 172.21.15.50:0/3163036344 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc06bd70 con 0x7fb3fc15cc20 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb400bcb700 1 --2- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc15cc20 0x7fb3fc15d0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e90 tx=0x7fb3f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68beef33fbd29dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb3faffd700 1 -- 172.21.15.50:0/3163036344 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400e070 con 0x7fb3fc15cc20 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec00a9c0 con 0x7f08f40b4270 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec00ab20 con 0x7f08f40b4270 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0070f0 con 0x7f08f40b4270 2023-12-19T14:39:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f415afc0 con 0x7f08f40b4270 2023-12-19T14:39:24.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f416da00 con 0x7f08f40b4270 2023-12-19T14:39:24.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== osd.2 v2:172.21.15.50:6808/33838 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb2c406c560 con 0x7fb2c4069a60 2023-12-19T14:39:24.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.841+0000 7fb3faffd700 1 -- 172.21.15.50:0/3163036344 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400bbf0 con 0x7fb3fc15cc20 2023-12-19T14:39:24.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27d9521700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d406b300 0x7f27d40c5310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00c940 tx=0x7f27cc006420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== osd.0 v2:172.21.15.50:6800/33839 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f748406c680 con 0x7f7484069b80 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc007260 con 0x7f27d406b300 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc0073c0 con 0x7f27d406b300 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc006700 con 0x7f27d406b300 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91940b04c0 0x7f919414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f91941550a0 0x7f91940abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194155470 0x7f91941576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194157260 con 0x7f9194155470 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91999ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91940b04c0 0x7f919414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91999ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91940b04c0 0x7f919414ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48026/0 (socket says 172.21.15.50:48026) 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91999ca700 1 -- 172.21.15.50:0/230285584 learned_addr learned my addr 172.21.15.50:0/230285584 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb3faffd700 1 -- 172.21.15.50:0/3163036344 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4012430 con 0x7fb3fc15cc20 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 -- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc15cc20 msgr2=0x7fb3fc15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 --2- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc15cc20 0x7fb3fc15d0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009e90 tx=0x7fb3f4009780 comp rx=0 tx=0).stop 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919414ef80 con 0x7f91940b04c0 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91940ac710 con 0x7f91941550a0 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f91941550a0 0x7f91940abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f08f415a890 con 0x7f08f40b4270 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d4146a40 con 0x7f27d406b300 2023-12-19T14:39:24.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d4147000 con 0x7f27d406b300 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 -- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91940b04c0 msgr2=0x7f919414ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91940b04c0 0x7f919414ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 -- 172.21.15.50:0/230285584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194155470 msgr2=0x7f91941576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194155470 0x7f91941576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840094f0 con 0x7f91941550a0 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 -- 172.21.15.50:0/3163036344 shutdown_connections 2023-12-19T14:39:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 --2- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc15cc20 0x7fb3fc15d0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f91991c9700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f91941550a0 0x7f91940abd30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f919406a7e0 tx=0x7f9184019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 --2- 172.21.15.50:0/3163036344 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc0c6900 0x7fb3fc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7fb4013cc700 1 -- 172.21.15.50:0/3163036344 >> 172.21.15.50:0/3163036344 conn(0x7fb3fc067bb0 msgr2=0x7fb3fc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184022640 con 0x7f91941550a0 2023-12-19T14:39:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f08ec035b50 con 0x7f08f40b4270 2023-12-19T14:39:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91840227e0 con 0x7f91941550a0 2023-12-19T14:39:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.845+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184022ba0 con 0x7f91941550a0 2023-12-19T14:39:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9194156c10 con 0x7f91941550a0 2023-12-19T14:39:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27d404be40 con 0x7f27d406b300 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f08db7fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08e00606a0 0x7f08e0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f08ec07d0d0 con 0x7f08f40b4270 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f08db7fe700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7f08e0069cd0 0x7f08e006c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08e006c7d0 con 0x7f08e0069cd0 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08ec07d400 con 0x7f08f40b4270 2023-12-19T14:39:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91941584d0 con 0x7f91941550a0 2023-12-19T14:39:24.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f2d4cf45700 1 Processor -- start 2023-12-19T14:39:24.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7efed4cae700 1 Processor -- start 2023-12-19T14:39:24.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9194071120 con 0x7f91941550a0 2023-12-19T14:39:24.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7fb4013cc700 1 -- 172.21.15.50:0/3163036344 shutdown_connections 2023-12-19T14:39:24.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7fb4013cc700 1 -- 172.21.15.50:0/3163036344 wait complete. 2023-12-19T14:39:24.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9184017420 con 0x7f91941550a0 2023-12-19T14:39:24.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f08f2ffd700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08e00606a0 0x7f08e0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f27cc006b30 con 0x7f27d406b300 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27c97fa700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f27b80606a0 0x7f27b8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f27cc07c9d0 con 0x7f27d406b300 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27c97fa700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7f27b8069cd0 0x7f27b806c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27b806c7d0 con 0x7f27b8069cd0 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27cc081220 con 0x7f27d406b300 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f08f2ffd700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08e00606a0 0x7f08e0062b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005fa0 tx=0x7f08e8007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f08f3fff700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7f08e0069cd0 0x7f08e006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9178060620 0x7f9178062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27d8d20700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f27b80606a0 0x7f27b8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9184013070 con 0x7f91941550a0 2023-12-19T14:39:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7f2d4cf45700 1 -- start start 2023-12-19T14:39:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4cf45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2d48158cc0 0x7f2d4806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.849+0000 7efed4cae700 1 -- start start 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efed4cae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efed0068110 0x7efed00ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efed4cae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed00adb50 con 0x7efed00c16c0 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efed4cae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0068610 con 0x7efed00c12f0 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efed4cae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed00688f0 con 0x7efed0068110 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7f917806a8a0 0x7f917806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f91999ca700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9178060620 0x7f9178062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f917806d400 con 0x7f917806a8a0 2023-12-19T14:39:24.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f918400b860 con 0x7f91941550a0 2023-12-19T14:39:24.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27d9d22700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7f27b8069cd0 0x7f27b806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7fb4013cc700 1 Processor -- start 2023-12-19T14:39:24.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4cf45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d480b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4cf45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4806b0d0 con 0x7f2d4806dc00 2023-12-19T14:39:24.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4cf45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d480b4790 con 0x7f2d48158cc0 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4cf45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4806d710 con 0x7f2d480b3870 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efece59c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efed00c12f0 0x7efed00c8f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46968/0 (socket says 172.21.15.50:46968) 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efeced9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efed00c16c0 0x7efed00ae570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37404/0 (socket says 172.21.15.50:37404) 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efeced9d700 1 -- 172.21.15.50:0/1776508318 learned_addr learned my addr 172.21.15.50:0/1776508318 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f27d8d20700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f27b80606a0 0x7f27b8062b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27d40c1600 tx=0x7f27d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f919a1cb700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7f917806a8a0 0x7f917806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4659c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2d48158cc0 0x7f2d4806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4659c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2d48158cc0 0x7f2d4806d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48036/0 (socket says 172.21.15.50:48036) 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d4659c700 1 -- 172.21.15.50:0/596118002 learned_addr learned my addr 172.21.15.50:0/596118002 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7f2d46d9d700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d480b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7efecd59a700 1 -- 172.21.15.50:0/1776508318 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229361860 0 0) 0x7efed0068610 con 0x7efed00c12f0 2023-12-19T14:39:24.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f27d9d22700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7f27b8069cd0 0x7f27b806c100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecd59a700 1 -- 172.21.15.50:0/1776508318 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003570 con 0x7efed00c12f0 2023-12-19T14:39:24.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4659c700 1 -- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 msgr2=0x7f2d480b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4659c700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d480b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f91999ca700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9178060620 0x7f9178062ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f91940ae8d0 tx=0x7f9190009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4659c700 1 -- 172.21.15.50:0/596118002 >> v1:172.21.15.50:6789/0 conn(0x7f2d4806dc00 legacy=0x7f2d4806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:24.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4659c700 1 -- 172.21.15.50:0/596118002 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4806a7e0 con 0x7f2d48158cc0 2023-12-19T14:39:24.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.853+0000 7fb4013cc700 1 -- start start 2023-12-19T14:39:24.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f919a1cb700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7f917806a8a0 0x7f917806cd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecdd9b700 1 --2- 172.21.15.50:0/1776508318 >> v2:172.21.15.50:3300/0 conn(0x7efed0068110 0x7efed00ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f08f3fff700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7f08e0069cd0 0x7f08e006c100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4659c700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d4806d1d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009f20 tx=0x7f2d38009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12547a4786fe439f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d45d9b700 1 -- 172.21.15.50:0/596118002 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3800e070 con 0x7f2d48158cc0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d45d9b700 1 -- 172.21.15.50:0/596118002 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3800b990 con 0x7f2d48158cc0 2023-12-19T14:39:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d45d9b700 1 -- 172.21.15.50:0/596118002 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d380092e0 con 0x7f2d48158cc0 2023-12-19T14:39:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecdd9b700 1 -- 172.21.15.50:0/1776508318 >> v1:172.21.15.125:6789/0 conn(0x7efed00c12f0 legacy=0x7efed00c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecdd9b700 1 -- 172.21.15.50:0/1776508318 >> v1:172.21.15.50:6789/0 conn(0x7efed00c16c0 legacy=0x7efed00ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecdd9b700 1 -- 172.21.15.50:0/1776508318 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0068bd0 con 0x7efed0068110 2023-12-19T14:39:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== osd.4 v2:172.21.15.125:6824/34166 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f917806d400 con 0x7f917806a8a0 2023-12-19T14:39:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecdd9b700 1 --2- 172.21.15.50:0/1776508318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed00ae030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efec00098e0 tx=0x7efec00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed1f49698dcc993 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecd59a700 1 -- 172.21.15.50:0/1776508318 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000b6d0 con 0x7efed0068110 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc06b900 0x7fb3fc1505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3fc0c6900 0x7fb3fc153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc15cc20 0x7fb3fc0ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1578c0 con 0x7fb3fc0c6900 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc152a30 con 0x7fb3fc06b900 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb4013cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc151eb0 con 0x7fb3fc15cc20 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3fc0c6900 0x7fb3fc153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3fc0c6900 0x7fb3fc153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53756/0 (socket says 172.21.15.50:53756) 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fb7fe700 1 -- 172.21.15.50:0/524513100 learned_addr learned my addr 172.21.15.50:0/524513100 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb400bcb700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc15cc20 0x7fb3fc0ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc06b900 0x7fb3fc1505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 -- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 msgr2=0x7f2d4806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d4806d1d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009f20 tx=0x7f2d38009840 comp rx=0 tx=0).stop 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecd59a700 1 -- 172.21.15.50:0/1776508318 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec000b830 con 0x7efed0068110 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7efecd59a700 1 -- 172.21.15.50:0/1776508318 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000bbf0 con 0x7efed0068110 2023-12-19T14:39:24.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== osd.3 v2:172.21.15.50:6816/33840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f08e006c7d0 con 0x7f08e0069cd0 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 -- 172.21.15.50:0/596118002 shutdown_connections 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d480b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/596118002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d4806d1d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 -- 172.21.15.50:0/596118002 >> 172.21.15.50:0/596118002 conn(0x7f2d48067bb0 msgr2=0x7f2d480cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 -- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc15cc20 msgr2=0x7fb3fc0ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb3fc15cc20 0x7fb3fc0ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 -- 172.21.15.50:0/596118002 shutdown_connections 2023-12-19T14:39:24.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 -- 172.21.15.50:0/524513100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3fc0c6900 msgr2=0x7fb3fc153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb3fc0c6900 0x7fb3fc153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7fb3fbfff700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009a20 con 0x7fb3fc06b900 2023-12-19T14:39:24.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== osd.1 v2:172.21.15.50:6817/33841 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f27b806c7d0 con 0x7f27b8069cd0 2023-12-19T14:39:24.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.857+0000 7f2d4cf45700 1 -- 172.21.15.50:0/596118002 wait complete. 2023-12-19T14:39:24.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb3fbfff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc06b900 0x7fb3fc1505e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c8b0 tx=0x7fb3ec008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7efed4cae700 1 -- 172.21.15.50:0/1776508318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 msgr2=0x7efed00ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7efed4cae700 1 --2- 172.21.15.50:0/1776508318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed00ae030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efec00098e0 tx=0x7efec00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec011600 con 0x7fb3fc06b900 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec011780 con 0x7fb3fc06b900 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec011b40 con 0x7fb3fc06b900 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc152100 con 0x7fb3fc06b900 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1526c0 con 0x7fb3fc06b900 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7f2d4cf45700 1 Processor -- start 2023-12-19T14:39:24.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7f2d4cf45700 1 -- start start 2023-12-19T14:39:24.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 --> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2d0037f70 con 0x7fb2c4069a60 2023-12-19T14:39:24.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb2beffd700 1 -- 172.21.15.50:0/3884863007 <== osd.2 v2:172.21.15.50:6808/33838 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb2d0037f70 con 0x7fb2c4069a60 2023-12-19T14:39:24.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.861+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3fc04be40 con 0x7fb3fc06b900 2023-12-19T14:39:24.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 -- 172.21.15.50:0/1776508318 shutdown_connections 2023-12-19T14:39:24.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 --2- 172.21.15.50:0/1776508318 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed00ae030 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 -- 172.21.15.50:0/1776508318 >> 172.21.15.50:0/1776508318 conn(0x7efed0069d80 msgr2=0x7efed00ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 -- 172.21.15.50:0/1776508318 shutdown_connections 2023-12-19T14:39:24.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 -- 172.21.15.50:0/1776508318 wait complete. 2023-12-19T14:39:24.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb3ec019400 con 0x7fb3fc06b900 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7fb2c4069a60 msgr2=0x7fb2c406be90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7fb2c4069a60 0x7fb2c406be90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2c40605a0 msgr2=0x7fb2c4062a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2c40605a0 0x7fb2c4062a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007fd0 tx=0x7fb2c8007d30 comp rx=0 tx=0).stop 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 msgr2=0x7fb2d004e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 --2- 172.21.15.50:0/3884863007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2d00ea390 0x7fb2d004e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c4e0 tx=0x7fb2cc00c7f0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d77fe700 1 -- 172.21.15.50:0/3884863007 reap_dead start 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 shutdown_connections 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 >> 172.21.15.50:0/3884863007 conn(0x7fb2d0054da0 msgr2=0x7fb2d004aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 shutdown_connections 2023-12-19T14:39:24.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb2d7fff700 1 -- 172.21.15.50:0/3884863007 wait complete. 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d4806dc00 0x7f2d480afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d48148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d48148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d481490b0 con 0x7f2d4806dc00 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480ab3c0 con 0x7f2d48158cc0 2023-12-19T14:39:24.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4cf45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480ab6a0 con 0x7f2d480b3870 2023-12-19T14:39:24.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb3e80606a0 0x7fb3e8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 Processor -- start 2023-12-19T14:39:24.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb3ec07c610 con 0x7fb3fc06b900 2023-12-19T14:39:24.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d46d9d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d48148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d46d9d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d48148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48050/0 (socket says 172.21.15.50:48050) 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d46d9d700 1 -- 172.21.15.50:0/931658999 learned_addr learned my addr 172.21.15.50:0/931658999 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7fb3e8069cd0 0x7fb3e806c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3e806c7d0 con 0x7fb3e8069cd0 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3ec07ebb0 con 0x7fb3fc06b900 2023-12-19T14:39:24.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7efed4cae700 1 -- start start 2023-12-19T14:39:24.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7f2d4659c700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d4806dc00 0x7f2d480afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.865+0000 7fb3fb7fe700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb3e80606a0 0x7fb3e8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7fb400bcb700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7fb3e8069cd0 0x7fb3e806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d3dd9b700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d48148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 -- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 msgr2=0x7f2d48148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2d480b3870 0x7f2d48148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 -- 172.21.15.50:0/931658999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d4806dc00 msgr2=0x7f2d480afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2d4806dc00 0x7f2d480afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d38009ae0 con 0x7f2d48158cc0 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 0x7efed015c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7fb400bcb700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7fb3e8069cd0 0x7fb3e806c100 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d46d9d700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d48148b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000ea50 tx=0x7f2d400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efed00c16c0 0x7efed015cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed01618a0 con 0x7efed0068110 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed006dbb0 con 0x7efed00c16c0 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efed4cae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0161250 con 0x7efed00c12f0 2023-12-19T14:39:24.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7fb3fb7fe700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb3e80606a0 0x7fb3e8062b50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc15b350 tx=0x7fb3f0007f70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000c920 con 0x7f2d48158cc0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d480ab980 con 0x7f2d48158cc0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d480b0390 con 0x7f2d48158cc0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4000caa0 con 0x7f2d48158cc0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000ce60 con 0x7f2d48158cc0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 0x7efed015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 0x7efed015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46286/0 (socket says 172.21.15.50:46286) 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 -- 172.21.15.50:0/3110615743 learned_addr learned my addr 172.21.15.50:0/3110615743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efece59c700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efeced9d700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efed00c16c0 0x7efed015cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 -- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efed00c16c0 msgr2=0x7efed015cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efed00c16c0 0x7efed015cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 -- 172.21.15.50:0/3110615743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 msgr2=0x7efed006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed006d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0009420 con 0x7efed00c12f0 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efece59c700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efed0068110 0x7efed006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:24.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d2c000fd0 con 0x7f2d48158cc0 2023-12-19T14:39:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7efecdd9b700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 0x7efed015c800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec00093f0 tx=0x7efec0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.869+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== osd.5 v2:172.21.15.125:6808/34168 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb3e806c7d0 con 0x7fb3e8069cd0 2023-12-19T14:39:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.873+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000b9a0 con 0x7efed00c12f0 2023-12-19T14:39:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.873+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec000bb40 con 0x7efed00c12f0 2023-12-19T14:39:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.873+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0026680 con 0x7efed00c12f0 2023-12-19T14:39:24.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.873+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2d40013680 con 0x7f2d48158cc0 2023-12-19T14:39:24.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed0161530 con 0x7efed00c12f0 2023-12-19T14:39:24.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed01538b0 con 0x7efed00c12f0 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3effd700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2d28060550 0x7f2d28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2d4007d370 con 0x7f2d48158cc0 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3effd700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f2d28069b80 0x7f2d2806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d2806c680 con 0x7f2d28069b80 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d4007d660 con 0x7f2d48158cc0 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efed00b3c90 con 0x7efed00c12f0 2023-12-19T14:39:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 --> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7478002d30 con 0x7f7484069b80 2023-12-19T14:39:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f2d3dd9b700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f2d28069b80 0x7f2d2806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f7494ff9700 1 -- 172.21.15.50:0/2812844081 <== osd.0 v2:172.21.15.50:6800/33839 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7478002d30 con 0x7f7484069b80 2023-12-19T14:39:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f7484069b80 msgr2=0x7f748406bfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.877+0000 7f749de43700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f7484069b80 0x7f748406bfb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f2d4659c700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2d28060550 0x7f2d28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7484060550 msgr2=0x7f7484062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7484060550 0x7f7484062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74980b33a0 tx=0x7f749000b040 comp rx=0 tx=0).stop 2023-12-19T14:39:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 msgr2=0x7f74981480e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 --2- 172.21.15.50:0/2812844081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749806d9e0 0x7f74981480e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e980 tx=0x7f748c009250 comp rx=0 tx=0).stop 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749d642700 1 -- 172.21.15.50:0/2812844081 reap_dead start 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 shutdown_connections 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 >> 172.21.15.50:0/2812844081 conn(0x7f7498067bb0 msgr2=0x7f74980cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f2d3dd9b700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f2d28069b80 0x7f2d2806bfb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 shutdown_connections 2023-12-19T14:39:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efec0023030 con 0x7efed00c12f0 2023-12-19T14:39:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f2d4659c700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2d28060550 0x7f2d28062a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009880 tx=0x7f2d38017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f749de43700 1 -- 172.21.15.50:0/2812844081 wait complete. 2023-12-19T14:39:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.881+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== osd.6 v2:172.21.15.125:6800/34165 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2d2806c680 con 0x7f2d28069b80 2023-12-19T14:39:24.911 INFO:teuthology.orchestra.run.smithi050.stdout:55834574857 2023-12-19T14:39:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efebe7fc700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb00606a0 0x7efeb0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7efec007e580 con 0x7efed00c12f0 2023-12-19T14:39:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efebe7fc700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7efeb0069cd0 0x7efeb006c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:24.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T14:39:24.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 --> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f27d406cb40 con 0x7f27b8069cd0 2023-12-19T14:39:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efece59c700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb00606a0 0x7efeb0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efeb006c7d0 con 0x7efeb0069cd0 2023-12-19T14:39:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efec007e8b0 con 0x7efed00c12f0 2023-12-19T14:39:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efeced9d700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7efeb0069cd0 0x7efeb006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7f27c97fa700 1 -- 172.21.15.50:0/398507136 <== osd.1 v2:172.21.15.50:6817/33841 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f27d406cb40 con 0x7f27b8069cd0 2023-12-19T14:39:24.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efeced9d700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7efeb0069cd0 0x7efeb006c100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7efece59c700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb00606a0 0x7efeb0062b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efec4005f00 tx=0x7efec4007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7f27b8069cd0 msgr2=0x7f27b806c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.885+0000 7f27db785700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7f27b8069cd0 0x7f27b806c100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f27b80606a0 msgr2=0x7f27b8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f27b80606a0 0x7f27b8062b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27d40c1600 tx=0x7f27d0009040 comp rx=0 tx=0).stop 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d406b300 msgr2=0x7f27d40c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 --2- 172.21.15.50:0/398507136 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27d406b300 0x7f27d40c5310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00c940 tx=0x7f27cc006420 comp rx=0 tx=0).stop 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27d9d22700 1 -- 172.21.15.50:0/398507136 reap_dead start 2023-12-19T14:39:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== osd.7 v2:172.21.15.125:6809/34167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7efeb006c7d0 con 0x7efeb0069cd0 2023-12-19T14:39:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 shutdown_connections 2023-12-19T14:39:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 >> 172.21.15.50:0/398507136 conn(0x7f27d40670e0 msgr2=0x7f27d414f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 shutdown_connections 2023-12-19T14:39:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f27db785700 1 -- 172.21.15.50:0/398507136 wait complete. 2023-12-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 --> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f08f416dce0 con 0x7f08e0069cd0 2023-12-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08db7fe700 1 -- 172.21.15.50:0/4067457584 <== osd.3 v2:172.21.15.50:6816/33840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f08e006c7d0 con 0x7f08e0069cd0 2023-12-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7f08e0069cd0 msgr2=0x7f08e006c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7f08e0069cd0 0x7f08e006c100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08e00606a0 msgr2=0x7f08e0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08e00606a0 0x7f08e0062b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08e8005fa0 tx=0x7f08e8007d00 comp rx=0 tx=0).stop 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f08f40b4270 msgr2=0x7f08f414d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 --2- 172.21.15.50:0/4067457584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f08f40b4270 0x7f08f414d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00cc20 tx=0x7f08ec001a60 comp rx=0 tx=0).stop 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f3fff700 1 -- 172.21.15.50:0/4067457584 reap_dead start 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 shutdown_connections 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.889+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 >> 172.21.15.50:0/4067457584 conn(0x7f08f4067bb0 msgr2=0x7f08f414f430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 --> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f919406cb40 con 0x7f917806a8a0 2023-12-19T14:39:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f9189ffb700 1 -- 172.21.15.50:0/230285584 <== osd.4 v2:172.21.15.125:6824/34166 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f919406cb40 con 0x7f917806a8a0 2023-12-19T14:39:24.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7f917806a8a0 msgr2=0x7f917806cd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7f917806a8a0 0x7f917806cd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9178060620 msgr2=0x7f9178062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9178060620 0x7f9178062ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f91940ae8d0 tx=0x7f9190009040 comp rx=0 tx=0).stop 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f91941550a0 msgr2=0x7f91940abd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 --2- 172.21.15.50:0/230285584 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f91941550a0 0x7f91940abd30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f919406a7e0 tx=0x7f9184019000 comp rx=0 tx=0).stop 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919a1cb700 1 -- 172.21.15.50:0/230285584 reap_dead start 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 shutdown_connections 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 >> 172.21.15.50:0/230285584 conn(0x7f91940670e0 msgr2=0x7f91940674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 shutdown_connections 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.893+0000 7f919bc2e700 1 -- 172.21.15.50:0/230285584 wait complete. 2023-12-19T14:39:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.897+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 shutdown_connections 2023-12-19T14:39:24.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.897+0000 7f08f9acd700 1 -- 172.21.15.50:0/4067457584 wait complete. 2023-12-19T14:39:24.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.901+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 --> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb3fc06a580 con 0x7fb3e8069cd0 2023-12-19T14:39:24.921 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:24.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-19T14:39:24.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.901+0000 7fb3e3fff700 1 -- 172.21.15.50:0/524513100 <== osd.5 v2:172.21.15.125:6808/34168 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb3fc06a580 con 0x7fb3e8069cd0 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.901+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7fb3e8069cd0 msgr2=0x7fb3e806c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.901+0000 7fb4013cc700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7fb3e8069cd0 0x7fb3e806c100 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb3e80606a0 msgr2=0x7fb3e8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb3e80606a0 0x7fb3e8062b50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc15b350 tx=0x7fb3f0007f70 comp rx=0 tx=0).stop 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc06b900 msgr2=0x7fb3fc1505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 --2- 172.21.15.50:0/524513100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3fc06b900 0x7fb3fc1505e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c8b0 tx=0x7fb3ec008040 comp rx=0 tx=0).stop 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb400bcb700 1 -- 172.21.15.50:0/524513100 reap_dead start 2023-12-19T14:39:24.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 shutdown_connections 2023-12-19T14:39:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 >> 172.21.15.50:0/524513100 conn(0x7fb3fc067bb0 msgr2=0x7fb3fc0ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 shutdown_connections 2023-12-19T14:39:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7fb4013cc700 1 -- 172.21.15.50:0/524513100 wait complete. 2023-12-19T14:39:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.905+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 --> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2d2c002d30 con 0x7f2d28069b80 2023-12-19T14:39:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d3effd700 1 -- 172.21.15.50:0/931658999 <== osd.6 v2:172.21.15.125:6800/34165 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2d2c002d30 con 0x7f2d28069b80 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f2d28069b80 msgr2=0x7f2d2806bfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f2d28069b80 0x7f2d2806bfb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2d28060550 msgr2=0x7f2d28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2d28060550 0x7f2d28062a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009880 tx=0x7f2d38017000 comp rx=0 tx=0).stop 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 msgr2=0x7f2d48148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 --2- 172.21.15.50:0/931658999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d48158cc0 0x7f2d48148b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000ea50 tx=0x7f2d400091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d46d9d700 1 -- 172.21.15.50:0/931658999 reap_dead start 2023-12-19T14:39:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 shutdown_connections 2023-12-19T14:39:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 >> 172.21.15.50:0/931658999 conn(0x7f2d48067bb0 msgr2=0x7f2d480a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 shutdown_connections 2023-12-19T14:39:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.909+0000 7f2d4cf45700 1 -- 172.21.15.50:0/931658999 wait complete. 2023-12-19T14:39:24.927 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:24.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-19T14:39:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 --> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efed01616c0 con 0x7efeb0069cd0 2023-12-19T14:39:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efebe7fc700 1 -- 172.21.15.50:0/3110615743 <== osd.7 v2:172.21.15.125:6809/34167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efed01616c0 con 0x7efeb0069cd0 2023-12-19T14:39:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7efeb0069cd0 msgr2=0x7efeb006c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7efeb0069cd0 0x7efeb006c100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb00606a0 msgr2=0x7efeb0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb00606a0 0x7efeb0062b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efec4005f00 tx=0x7efec4007c60 comp rx=0 tx=0).stop 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 msgr2=0x7efed015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 --2- 172.21.15.50:0/3110615743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efed00c12f0 0x7efed015c800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec00093f0 tx=0x7efec0017000 comp rx=0 tx=0).stop 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efeced9d700 1 -- 172.21.15.50:0/3110615743 reap_dead start 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 shutdown_connections 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 >> 172.21.15.50:0/3110615743 conn(0x7efed0069d80 msgr2=0x7efed00af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 shutdown_connections 2023-12-19T14:39:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:24.913+0000 7efed4cae700 1 -- 172.21.15.50:0/3110615743 wait complete. 2023-12-19T14:39:24.930 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:24.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-19T14:39:24.931 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:24.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-19T14:39:24.932 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:24.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-19T14:39:24.934 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:24.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-19T14:39:24.935 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:24.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-19T14:39:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.117+0000 7f7cfb185700 1 Processor -- start 2023-12-19T14:39:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 -- start start 2023-12-19T14:39:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 0x7f7cf40adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7cf40cad40 0x7f7cf406c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf4159a80 con 0x7f7cf406a580 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf40c6850 con 0x7f7cf40cad40 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cf8f21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 0x7f7cf40adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cfb185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf4158e70 con 0x7f7cf4152ba0 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cf8f21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 0x7f7cf40adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46300/0 (socket says 172.21.15.50:46300) 2023-12-19T14:39:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.121+0000 7f7cf8f21700 1 -- 172.21.15.50:0/3241119917 learned_addr learned my addr 172.21.15.50:0/3241119917 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 --2- 172.21.15.50:0/3241119917 >> v2:172.21.15.50:3300/0 conn(0x7f7cf40cad40 0x7f7cf406c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240731896 0 0) 0x7f7cf4159a80 con 0x7f7cf406a580 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cdc003570 con 0x7f7cf406a580 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 -- 172.21.15.50:0/3241119917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 msgr2=0x7f7cf40adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 --2- 172.21.15.50:0/3241119917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 0x7f7cf40adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1936827993 0 0) 0x7f7cdc003570 con 0x7f7cf406a580 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 -- 172.21.15.50:0/3241119917 >> v1:172.21.15.50:6789/0 conn(0x7f7cf406a580 legacy=0x7f7cf40ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 -- 172.21.15.50:0/3241119917 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4159150 con 0x7f7cf40cad40 2023-12-19T14:39:25.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf9722700 1 --2- 172.21.15.50:0/3241119917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf40cad40 0x7f7cf406c8a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40098e0 tx=0x7f7ce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f1ee467cde0cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8001450 con 0x7f7cf40cad40 2023-12-19T14:39:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce4010340 con 0x7f7cf40cad40 2023-12-19T14:39:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cf37fe700 1 -- 172.21.15.50:0/3241119917 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8001450 con 0x7f7cf40cad40 2023-12-19T14:39:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cfb185700 1 -- 172.21.15.50:0/3241119917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf40cad40 msgr2=0x7f7cf406c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.125+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3241119917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf40cad40 0x7f7cf406c8a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40098e0 tx=0x7f7ce40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 -- 172.21.15.50:0/3241119917 shutdown_connections 2023-12-19T14:39:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3241119917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf40cad40 0x7f7cf406c8a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3241119917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf4152ba0 0x7f7cf40adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 -- 172.21.15.50:0/3241119917 >> 172.21.15.50:0/3241119917 conn(0x7f7cf40670e0 msgr2=0x7f7cf40674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 -- 172.21.15.50:0/3241119917 shutdown_connections 2023-12-19T14:39:25.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.129+0000 7f7cfb185700 1 -- 172.21.15.50:0/3241119917 wait complete. 2023-12-19T14:39:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.133+0000 7f7cfb185700 1 Processor -- start 2023-12-19T14:39:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.133+0000 7f7cfb185700 1 -- start start 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf40cad40 0x7f7cf4153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf4152ba0 0x7f7cf41543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40b1410 con 0x7f7cf4152ba0 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40b0800 con 0x7f7cf406a580 2023-12-19T14:39:25.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf40b0ae0 con 0x7f7cf40cad40 2023-12-19T14:39:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48070/0 (socket says 172.21.15.50:48070) 2023-12-19T14:39:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 -- 172.21.15.50:0/3535431778 learned_addr learned my addr 172.21.15.50:0/3535431778 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf3fff700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf40cad40 0x7f7cf4153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 -- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf40cad40 msgr2=0x7f7cf4153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf40cad40 0x7f7cf4153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 -- 172.21.15.50:0/3535431778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf4152ba0 msgr2=0x7f7cf41543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf4152ba0 0x7f7cf41543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 -- 172.21.15.50:0/3535431778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4009420 con 0x7f7cf406a580 2023-12-19T14:39:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf8f21700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec00b120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec010070 con 0x7f7cf406a580 2023-12-19T14:39:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf40b0dc0 con 0x7f7cf406a580 2023-12-19T14:39:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf40b0450 con 0x7f7cf406a580 2023-12-19T14:39:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec008ac0 con 0x7f7cf406a580 2023-12-19T14:39:25.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.137+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec008e60 con 0x7f7cf406a580 2023-12-19T14:39:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 Processor -- start 2023-12-19T14:39:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf404be40 con 0x7f7cf406a580 2023-12-19T14:39:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 -- start start 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 0x7ff74c06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff74c06ced0 0x7ff74c0a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74c06bc30 con 0x7ff74c06d370 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.141+0000 7ff75402e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74c06b020 con 0x7ff74c06ced0 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff75402e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74c06b300 con 0x7ff74c158e50 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7cec014070 con 0x7f7cf406a580 2023-12-19T14:39:25.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 Processor -- start 2023-12-19T14:39:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff74c06ced0 0x7ff74c0a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff74c06ced0 0x7ff74c0a8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53820/0 (socket says 172.21.15.50:53820) 2023-12-19T14:39:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 -- 172.21.15.50:0/119927518 learned_addr learned my addr 172.21.15.50:0/119927518 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff751dca700 1 --2- 172.21.15.50:0/119927518 >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 0x7ff74c06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 -- 172.21.15.50:0/119927518 >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 msgr2=0x7ff74c06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 --2- 172.21.15.50:0/119927518 >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 0x7ff74c06a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 -- 172.21.15.50:0/119927518 >> v1:172.21.15.50:6789/0 conn(0x7ff74c06d370 legacy=0x7ff74c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 -- 172.21.15.50:0/119927518 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c06b5e0 con 0x7ff74c06ced0 2023-12-19T14:39:25.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff751dca700 1 --2- 172.21.15.50:0/119927518 >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 0x7ff74c06a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 -- start start 2023-12-19T14:39:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7f7cf0ff9700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 0x7f7ce0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7cec07c5a0 con 0x7f7cf406a580 2023-12-19T14:39:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7f7cf3fff700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 0x7f7ce0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff7525cb700 1 --2- 172.21.15.50:0/119927518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0a8af0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff74c069b90 tx=0x7ff744009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5150afa24b34467b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7face40ce810 0x7face40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face406b730 con 0x7face41591f0 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face406ab20 con 0x7face40ce810 2023-12-19T14:39:25.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7face948e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face406ae00 con 0x7face4158e20 2023-12-19T14:39:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7f7cf3fff700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 0x7f7ce0062a00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8001360 tx=0x7f7ce800a320 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff750dc8700 1 -- 172.21.15.50:0/119927518 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74400e070 con 0x7ff74c06ced0 2023-12-19T14:39:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff750dc8700 1 -- 172.21.15.50:0/119927518 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74400b9e0 con 0x7ff74c06ced0 2023-12-19T14:39:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.145+0000 7ff750dc8700 1 -- 172.21.15.50:0/119927518 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7440092e0 con 0x7ff74c06ced0 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- 172.21.15.50:0/119927518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 msgr2=0x7ff74c0a8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 --2- 172.21.15.50:0/119927518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0a8af0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff74c069b90 tx=0x7ff744009840 comp rx=0 tx=0).stop 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- 172.21.15.50:0/119927518 shutdown_connections 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 --2- 172.21.15.50:0/119927518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0a8af0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 --2- 172.21.15.50:0/119927518 >> v2:172.21.15.125:3300/0 conn(0x7ff74c158e50 0x7ff74c06a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- 172.21.15.50:0/119927518 >> 172.21.15.50:0/119927518 conn(0x7ff74c067bb0 msgr2=0x7ff74c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46306/0 (socket says 172.21.15.50:46306) 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 -- 172.21.15.50:0/1272331260 learned_addr learned my addr 172.21.15.50:0/1272331260 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face8c8d700 1 --2- 172.21.15.50:0/1272331260 >> v2:172.21.15.50:3300/0 conn(0x7face40ce810 0x7face40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- 172.21.15.50:0/119927518 shutdown_connections 2023-12-19T14:39:25.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- 172.21.15.50:0/119927518 wait complete. 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 -- 172.21.15.50:0/1272331260 >> v2:172.21.15.50:3300/0 conn(0x7face40ce810 msgr2=0x7face40ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 --2- 172.21.15.50:0/1272331260 >> v2:172.21.15.50:3300/0 conn(0x7face40ce810 0x7face40ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 -- 172.21.15.50:0/1272331260 >> v1:172.21.15.125:6789/0 conn(0x7face41591f0 legacy=0x7face40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 -- 172.21.15.50:0/1272331260 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face406b0e0 con 0x7face4158e20 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face3fff700 1 --2- 172.21.15.50:0/1272331260 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facd40099b0 tx=0x7facd4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ef0062ac4ba20 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face2ffd700 1 -- 172.21.15.50:0/1272331260 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400b820 con 0x7face4158e20 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face2ffd700 1 -- 172.21.15.50:0/1272331260 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400b980 con 0x7face4158e20 2023-12-19T14:39:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7face2ffd700 1 -- 172.21.15.50:0/1272331260 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400bd20 con 0x7face4158e20 2023-12-19T14:39:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7fbf57fff700 1 Processor -- start 2023-12-19T14:39:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 Processor -- start 2023-12-19T14:39:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7fbf57fff700 1 -- start start 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 -- 172.21.15.50:0/1272331260 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 msgr2=0x7face406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 --2- 172.21.15.50:0/1272331260 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facd40099b0 tx=0x7facd4009250 comp rx=0 tx=0).stop 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 -- 172.21.15.50:0/1272331260 shutdown_connections 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 --2- 172.21.15.50:0/1272331260 >> v2:172.21.15.50:3300/0 conn(0x7face40ce810 0x7face40ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 --2- 172.21.15.50:0/1272331260 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face4158e20 0x7face406a540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 -- 172.21.15.50:0/1272331260 >> 172.21.15.50:0/1272331260 conn(0x7face4067bb0 msgr2=0x7face40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.149+0000 7ff75402e700 1 -- start start 2023-12-19T14:39:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 Processor -- start 2023-12-19T14:39:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 -- 172.21.15.50:0/1272331260 shutdown_connections 2023-12-19T14:39:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf57fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf580cc0a0 0x7fbf580abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf57fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 0x7ff74c14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c171170 con 0x7ff74c06ced0 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0acfb0 con 0x7ff74c06d370 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff75402e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0aca30 con 0x7ff74c158e50 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7face948e700 1 -- 172.21.15.50:0/1272331260 wait complete. 2023-12-19T14:39:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff7515c9700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf57fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5806d3a0 con 0x7fbf5806a580 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf57fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf580c5c70 con 0x7fbf580cc0a0 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf57fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5806d680 con 0x7fbf581549f0 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf4e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf4e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46316/0 (socket says 172.21.15.50:46316) 2023-12-19T14:39:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf4e7fc700 1 -- 172.21.15.50:0/1944439147 learned_addr learned my addr 172.21.15.50:0/1944439147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff7515c9700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48092/0 (socket says 172.21.15.50:48092) 2023-12-19T14:39:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff7515c9700 1 -- 172.21.15.50:0/740471084 learned_addr learned my addr 172.21.15.50:0/740471084 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7cec04c950 con 0x7f7cf406a580 2023-12-19T14:39:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fbf56ffd700 1 --2- 172.21.15.50:0/1944439147 >> v2:172.21.15.50:3300/0 conn(0x7fbf580cc0a0 0x7fbf580abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7ff751dca700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 -- start start 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fec8415ae40 0x7fec84161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8406ca60 0x7fec8406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8406d790 con 0x7fec84160f30 2023-12-19T14:39:25.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8406da70 con 0x7fec8415ae40 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.153+0000 7fec89545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406dd50 con 0x7fec8406ca60 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf4e7fc700 1 -- 172.21.15.50:0/1944439147 >> v2:172.21.15.50:3300/0 conn(0x7fbf580cc0a0 msgr2=0x7fbf580abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf4e7fc700 1 --2- 172.21.15.50:0/1944439147 >> v2:172.21.15.50:3300/0 conn(0x7fbf580cc0a0 0x7fbf580abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf4e7fc700 1 -- 172.21.15.50:0/1944439147 >> v1:172.21.15.50:6789/0 conn(0x7fbf5806a580 legacy=0x7fbf5806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf4e7fc700 1 -- 172.21.15.50:0/1944439147 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580c6220 con 0x7fbf581549f0 2023-12-19T14:39:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff7525cb700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 0x7ff74c14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7face948e700 1 Processor -- start 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 -- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 msgr2=0x7ff74c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 0x7ff74c14f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 -- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 msgr2=0x7ff74c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 -- 172.21.15.50:0/740471084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740009420 con 0x7ff74c06ced0 2023-12-19T14:39:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec82ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fec84160f30 0x7fec8415a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37458/0 (socket says 172.21.15.50:37458) 2023-12-19T14:39:25.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec82ffd700 1 -- 172.21.15.50:0/258366687 learned_addr learned my addr 172.21.15.50:0/258366687 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:25.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff7525cb700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 0x7ff74c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:25.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff7515c9700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:25.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf4e7fc700 1 --2- 172.21.15.50:0/1944439147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf480099b0 tx=0x7fbf48009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e716eb0586e77f44 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7face948e700 1 -- start start 2023-12-19T14:39:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf567fc700 1 -- 172.21.15.50:0/1944439147 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48009ea0 con 0x7fbf581549f0 2023-12-19T14:39:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf567fc700 1 -- 172.21.15.50:0/1944439147 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4800b8b0 con 0x7fbf581549f0 2023-12-19T14:39:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38170937 0 0) 0x7fec8406d790 con 0x7fec84160f30 2023-12-19T14:39:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec70003570 con 0x7fec84160f30 2023-12-19T14:39:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff751dca700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0ac4f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff74000ecb0 tx=0x7ff7400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf567fc700 1 -- 172.21.15.50:0/1944439147 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800bc30 con 0x7fbf581549f0 2023-12-19T14:39:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec837fe700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8406ca60 0x7fec8406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 -- 172.21.15.50:0/1944439147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 msgr2=0x7fbf58154e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1944439147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf480099b0 tx=0x7fbf48009250 comp rx=0 tx=0).stop 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876633682 0 0) 0x7fec70003570 con 0x7fec84160f30 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8406ca60 msgr2=0x7fec8406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> v2:172.21.15.125:3300/0 conn(0x7fec8415ae40 0x7fec84161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8406ca60 0x7fec8406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 >> v2:172.21.15.125:3300/0 conn(0x7fec8415ae40 msgr2=0x7fec84161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 --2- 172.21.15.50:0/258366687 >> v2:172.21.15.125:3300/0 conn(0x7fec8415ae40 0x7fec84161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8406e030 con 0x7fec84160f30 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fec740017f0 con 0x7fec84160f30 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74000c850 con 0x7ff74c06ced0 2023-12-19T14:39:25.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74000c9b0 con 0x7ff74c06ced0 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74000cd50 con 0x7ff74c06ced0 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff744009ae0 con 0x7ff74c06ced0 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c170b20 con 0x7ff74c06ced0 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7face948e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 0x7face41496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7face948e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4158e20 0x7face414a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7face948e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46344/0 (socket says 172.21.15.50:46344) 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 -- 172.21.15.50:0/1857665045 learned_addr learned my addr 172.21.15.50:0/1857665045 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face3fff700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 0x7face41496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40c84d0 con 0x7face4158e20 2023-12-19T14:39:25.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4149be0 con 0x7face40ce810 2023-12-19T14:39:25.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face416d250 con 0x7face41591f0 2023-12-19T14:39:25.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 -- 172.21.15.50:0/1944439147 shutdown_connections 2023-12-19T14:39:25.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1944439147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf581549f0 0x7fbf58154e40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1944439147 >> v2:172.21.15.50:3300/0 conn(0x7fbf580cc0a0 0x7fbf580abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fbf57fff700 1 -- 172.21.15.50:0/1944439147 >> 172.21.15.50:0/1944439147 conn(0x7fbf58067bb0 msgr2=0x7fbf580cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fbf57fff700 1 -- 172.21.15.50:0/1944439147 shutdown_connections 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 >> v1:172.21.15.50:6789/0 conn(0x7fec84160f30 legacy=0x7fec8415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec70003a60 0x7fec70003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec70004810 0x7fec70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec70005290 0x7fec7000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406da70 con 0x7fec70003a60 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8406dd50 con 0x7fec70005290 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7000c2b0 con 0x7fec70004810 2023-12-19T14:39:25.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.157+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> v2:172.21.15.125:3300/0 conn(0x7fec8415ae40 0x7fec84161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:25.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec70004810 0x7fec70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face37fe700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4158e20 0x7face414a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec837fe700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8406ca60 0x7fec8406ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:25.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 -- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 msgr2=0x7face41496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 0x7face41496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 -- 172.21.15.50:0/1857665045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4158e20 msgr2=0x7face414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4158e20 0x7face414a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40094f0 con 0x7face41591f0 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face3fff700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 0x7face41496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face8c8d700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facdc007f60 tx=0x7facdc0098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec837fe700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec70005290 0x7fec7000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 -- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec70005290 msgr2=0x7fec7000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fbf57fff700 1 -- 172.21.15.50:0/1944439147 wait complete. 2023-12-19T14:39:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fbf57fff700 1 Processor -- start 2023-12-19T14:39:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 Processor -- start 2023-12-19T14:39:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 -- start start 2023-12-19T14:39:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f9f0b145700 1 Processor -- start 2023-12-19T14:39:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec70005290 0x7fec7000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec82ffd700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec70003a60 0x7fec70003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 -- 172.21.15.50:0/258366687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec70003a60 msgr2=0x7fec70003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec70003a60 0x7fec70003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00ec40 con 0x7face41591f0 2023-12-19T14:39:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00eda0 con 0x7face41591f0 2023-12-19T14:39:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fec7a7fc700 1 -- 172.21.15.50:0/258366687 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70003570 con 0x7fec70004810 2023-12-19T14:39:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc014670 con 0x7face41591f0 2023-12-19T14:39:25.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7fbf57fff700 1 -- start start 2023-12-19T14:39:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f209806dce0 0x7f209806cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209806d640 con 0x7f20980ce720 2023-12-19T14:39:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980cb500 con 0x7f209806c9f0 2023-12-19T14:39:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f209d20c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980cee70 con 0x7f209806dce0 2023-12-19T14:39:25.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7f9f0b145700 1 -- start start 2023-12-19T14:39:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c071120 con 0x7ff74c06ced0 2023-12-19T14:39:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face416d530 con 0x7face41591f0 2023-12-19T14:39:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.161+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face416daf0 con 0x7face41591f0 2023-12-19T14:39:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec837fe700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec70005290 0x7fec7000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec82ffd700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec70003a60 0x7fec70003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fc732985700 1 Processor -- start 2023-12-19T14:39:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f208f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f209806dce0 0x7f209806cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec70004810 0x7fec70004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00abc0 tx=0x7fec6c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976fc285f6db339d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f208f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f209806dce0 0x7f209806cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53866/0 (socket says 172.21.15.50:53866) 2023-12-19T14:39:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f208f7fe700 1 -- 172.21.15.50:0/3110421722 learned_addr learned my addr 172.21.15.50:0/3110421722 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f2097fff700 1 --1- 172.21.15.50:0/3110421722 >> v1:172.21.15.125:6789/0 conn(0x7f209806c9f0 0x7f20980cb990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46974/0 (socket says 172.21.15.50:46974) 2023-12-19T14:39:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc40052b0 con 0x7face41591f0 2023-12-19T14:39:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00a530 con 0x7fec70004810 2023-12-19T14:39:25.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c00c600 con 0x7fec70004810 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fbf57fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 0x7fbf580b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fbf57fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf580cc0a0 0x7fbf581517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fbf57fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 0x7fbf581749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955075273 0 0) 0x7f209806d640 con 0x7f20980ce720 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2084003570 con 0x7f20980ce720 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853515151 0 0) 0x7f20980cb500 con 0x7f209806c9f0 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209806d640 con 0x7f209806c9f0 2023-12-19T14:39:25.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 908800780 0 0) 0x7f209806d640 con 0x7f209806c9f0 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 >> v2:172.21.15.50:3300/0 conn(0x7f209806dce0 msgr2=0x7f209806cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 --2- 172.21.15.50:0/3110421722 >> v2:172.21.15.50:3300/0 conn(0x7f209806dce0 0x7f209806cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 >> v1:172.21.15.50:6789/0 conn(0x7f20980ce720 legacy=0x7f209815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980ca950 con 0x7f209806c9f0 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f20880017f0 con 0x7f209806c9f0 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 >> v1:172.21.15.125:6789/0 conn(0x7f209806c9f0 legacy=0x7f20980cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084004c50 0x7f2084005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 0x7f2084003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f0b145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0406a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f0b145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f04160e60 0x7f9f040af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2084004160 0x7f208400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980cee70 con 0x7f2084004c50 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208400c130 con 0x7f20840037b0 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208400c2c0 con 0x7f2084004160 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f0b145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0406ba50 con 0x7f9f040afd20 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f0b145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0406bee0 con 0x7f9f04160e60 2023-12-19T14:39:25.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f0b145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040b01a0 con 0x7f9f0415d050 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f03fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f04160e60 0x7f9f040af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f03fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f04160e60 0x7f9f040af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48134/0 (socket says 172.21.15.50:48134) 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f03fff700 1 -- 172.21.15.50:0/1221064513 learned_addr learned my addr 172.21.15.50:0/1221064513 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fbf56ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 0x7fbf580b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fbf56ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 0x7fbf580b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46350/0 (socket says 172.21.15.50:46350) 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 -- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec70004810 msgr2=0x7fec70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec827fc700 1 -- 172.21.15.50:0/258366687 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00a530 con 0x7fec70004810 2023-12-19T14:39:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 --2- 172.21.15.50:0/258366687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec70004810 0x7fec70004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00abc0 tx=0x7fec6c008040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec837fe700 1 -- 172.21.15.50:0/258366687 reap_dead start 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 -- 172.21.15.50:0/258366687 shutdown_connections 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 -- 172.21.15.50:0/258366687 >> 172.21.15.50:0/258366687 conn(0x7fec84067bb0 msgr2=0x7fec840cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f208f7fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 0x7f2084003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 -- 172.21.15.50:0/258366687 shutdown_connections 2023-12-19T14:39:25.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff740013680 con 0x7ff74c06ced0 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f2097fff700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084004c50 0x7f2084005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f209ca0b700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2084004160 0x7f208400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7f9f08ee1700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f03fff700 1 -- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 msgr2=0x7f9f0406a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f03fff700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0406a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f03fff700 1 -- 172.21.15.50:0/1221064513 >> v1:172.21.15.50:6789/0 conn(0x7f9f040afd20 legacy=0x7f9f0406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fc732985700 1 -- start start 2023-12-19T14:39:25.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f03fff700 1 -- 172.21.15.50:0/1221064513 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040af600 con 0x7f9f04160e60 2023-12-19T14:39:25.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.165+0000 7fec89545700 1 -- 172.21.15.50:0/258366687 wait complete. 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf56ffd700 1 -- 172.21.15.50:0/1388047393 learned_addr learned my addr 172.21.15.50:0/1388047393 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580af5a0 con 0x7fbf580cc0a0 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58179120 con 0x7fbf581549f0 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58179400 con 0x7fbf5806a580 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f208f7fe700 1 -- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2084004160 msgr2=0x7f208400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f208f7fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2084004160 0x7f208400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 0x7fbf581749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f208f7fe700 1 -- 172.21.15.50:0/3110421722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084004c50 msgr2=0x7f2084005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf4e7fc700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf580cc0a0 0x7fbf581517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f208f7fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084004c50 0x7f2084005040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f208f7fe700 1 -- 172.21.15.50:0/3110421722 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980ca950 con 0x7f20840037b0 2023-12-19T14:39:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7facdc038b50 con 0x7face41591f0 2023-12-19T14:39:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc732985700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc724053000 0x7fc7240ea670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f03fff700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f040af7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40099b0 tx=0x7f9ef4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=955262b9d2581c7a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f209ca0b700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2084004160 0x7f208400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 -- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 msgr2=0x7fbf580b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 0x7fbf580b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 -- 172.21.15.50:0/1388047393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf580cc0a0 msgr2=0x7fbf581517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf580cc0a0 0x7fbf581517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480094f0 con 0x7fbf581549f0 2023-12-19T14:39:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f209ca0b700 1 -- 172.21.15.50:0/3110421722 reap_dead start 2023-12-19T14:39:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc72bfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc7240ea2a0 0x7fc724054680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:46976/0 (socket says 172.21.15.50:46976) 2023-12-19T14:39:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc72bfff700 1 -- 172.21.15.50:0/2301030581 learned_addr learned my addr 172.21.15.50:0/2301030581 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7faccbfff700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 0x7facd0062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7facdc07d340 con 0x7face41591f0 2023-12-19T14:39:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240535e0 con 0x7fc724007410 2023-12-19T14:39:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf577fe700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 0x7fbf581749f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c810 tx=0x7fbf50002620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7face3fff700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 0x7facd0062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240538c0 con 0x7fc7240ea2a0 2023-12-19T14:39:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fec89545700 1 Processor -- start 2023-12-19T14:39:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc724053ba0 con 0x7fc724053000 2023-12-19T14:39:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50012630 con 0x7fbf581549f0 2023-12-19T14:39:25.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf500127b0 con 0x7fbf581549f0 2023-12-19T14:39:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f037fe700 1 -- 172.21.15.50:0/1221064513 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4009ea0 con 0x7f9f04160e60 2023-12-19T14:39:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f037fe700 1 -- 172.21.15.50:0/1221064513 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400b820 con 0x7f9f04160e60 2023-12-19T14:39:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf581796e0 con 0x7fbf581549f0 2023-12-19T14:39:25.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7f9f037fe700 1 -- 172.21.15.50:0/1221064513 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef400bbc0 con 0x7f9f04160e60 2023-12-19T14:39:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf580b16b0 con 0x7fbf581549f0 2023-12-19T14:39:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50012b70 con 0x7fbf581549f0 2023-12-19T14:39:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fec89545700 1 -- start start 2023-12-19T14:39:25.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7face3fff700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 0x7facd0062ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7facd40099b0 tx=0x7facd4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc72b7fe700 1 --2- 172.21.15.50:0/2301030581 >> v2:172.21.15.50:3300/0 conn(0x7fc724053000 0x7fc7240ea670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494940527 0 0) 0x7fc7240538c0 con 0x7fc7240ea2a0 2023-12-19T14:39:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc71c003570 con 0x7fc7240ea2a0 2023-12-19T14:39:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f208f7fe700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 0x7f2084003c20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2080006fa0 tx=0x7f2080009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f96bbb1a3b9963 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 -- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 msgr2=0x7f9f040af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f040af7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40099b0 tx=0x7f9ef4009250 comp rx=0 tx=0).stop 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186961046 0 0) 0x7fc71c003570 con 0x7fc7240ea2a0 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 >> v2:172.21.15.50:3300/0 conn(0x7fc724053000 msgr2=0x7fc7240ea670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 --2- 172.21.15.50:0/2301030581 >> v2:172.21.15.50:3300/0 conn(0x7fc724053000 0x7fc7240ea670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 >> v1:172.21.15.50:6789/0 conn(0x7fc724007410 legacy=0x7fc7240e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000e740 con 0x7f20840037b0 2023-12-19T14:39:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000e8a0 con 0x7f20840037b0 2023-12-19T14:39:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724053e80 con 0x7fc7240ea2a0 2023-12-19T14:39:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 <== mon.1 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fc720001470 con 0x7fc7240ea2a0 2023-12-19T14:39:25.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f20977fe700 1 -- 172.21.15.50:0/3110421722 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000ec60 con 0x7f20840037b0 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec8413b480 0x7fec8406ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 0x7fec8406b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840afdb0 con 0x7fec841383a0 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8415f600 con 0x7fec8413b480 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec89545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84138950 con 0x7fec8413b0b0 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7ff73dffb700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 0x7ff730062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff74007cdf0 con 0x7ff74c06ced0 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.169+0000 7ff7515c9700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 0x7ff730062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7ff7515c9700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 0x7ff730062a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0afdc0 tx=0x7ff748007f30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 -- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 msgr2=0x7f2084003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 0x7f2084003c20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2080006fa0 tx=0x7f2080009250 comp rx=0 tx=0).stop 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 >> v1:172.21.15.125:6789/0 conn(0x7fc7240ea2a0 legacy=0x7fc724054680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c003ae0 0x7fc71c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc71c004810 0x7fc71c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 -- 172.21.15.50:0/1221064513 shutdown_connections 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f040af7e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 --2- 172.21.15.50:0/1221064513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0406a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 -- 172.21.15.50:0/1221064513 >> 172.21.15.50:0/1221064513 conn(0x7f9f04067b60 msgr2=0x7f9f040ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 -- 172.21.15.50:0/1221064513 shutdown_connections 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f9f0b145700 1 -- 172.21.15.50:0/1221064513 wait complete. 2023-12-19T14:39:25.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c0052b0 con 0x7fbf581549f0 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46376/0 (socket says 172.21.15.50:46376) 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 -- 172.21.15.50:0/3190862307 learned_addr learned my addr 172.21.15.50:0/3190862307 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec8413b480 0x7fec8406ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec837fe700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 0x7fec8406b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 -- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec8413b480 msgr2=0x7fec8406ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec8413b480 0x7fec8406ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 -- 172.21.15.50:0/3190862307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 msgr2=0x7fec8406b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 0x7fec8406b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 -- 172.21.15.50:0/3190862307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84138c30 con 0x7fec8413b0b0 2023-12-19T14:39:25.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 -- 172.21.15.50:0/3110421722 shutdown_connections 2023-12-19T14:39:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 --2- 172.21.15.50:0/3110421722 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20840037b0 0x7f2084003c20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 -- 172.21.15.50:0/3110421722 >> 172.21.15.50:0/3110421722 conn(0x7f2098067bb0 msgr2=0x7f20980cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 0x7fc71c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec82ffd700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec7400b250 tx=0x7fec74009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 -- 172.21.15.50:0/3110421722 shutdown_connections 2023-12-19T14:39:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7f209d20c700 1 -- 172.21.15.50:0/3110421722 wait complete. 2023-12-19T14:39:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc724053ba0 con 0x7fc71c003ae0 2023-12-19T14:39:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fec837fe700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 0x7fec8406b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fc72bfff700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c003ae0 0x7fc71c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.173+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbf5001a3b0 con 0x7fbf581549f0 2023-12-19T14:39:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 Processor -- start 2023-12-19T14:39:25.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 Processor -- start 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74007150 con 0x7fec8413b0b0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec740072b0 con 0x7fec8413b0b0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74007630 con 0x7fec8413b0b0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec84138f10 con 0x7fec8413b0b0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8415ef60 con 0x7fec8413b0b0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c00c210 con 0x7fc71c004810 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 0x7fc71c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc72b7fe700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc71c004810 0x7fc71c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c00c3d0 con 0x7fc71c0052e0 2023-12-19T14:39:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 -- start start 2023-12-19T14:39:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 -- start start 2023-12-19T14:39:25.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f040afd20 0x7f9f04148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0415eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f209815d270 0x7f2098086f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04149390 con 0x7f9f040afd20 2023-12-19T14:39:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04170d20 con 0x7f9f04160e60 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f0b145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f041715c0 con 0x7f9f0415d050 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 -- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc71c004810 msgr2=0x7fc71c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc71c004810 0x7fc71c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 -- 172.21.15.50:0/2301030581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c003ae0 msgr2=0x7fc71c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c003ae0 0x7fc71c003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 -- 172.21.15.50:0/2301030581 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c003570 con 0x7fc71c0052e0 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20980874b0 0x7f209815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209815ba90 con 0x7f209815d270 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098089f60 con 0x7f20980874b0 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f209d20c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980896a0 con 0x7f20840037b0 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 -- 172.21.15.50:0/2301030581 reap_dead start 2023-12-19T14:39:25.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc72bfff700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c003ae0 0x7fc71c003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7facdc04d6a0 con 0x7face41591f0 2023-12-19T14:39:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46390/0 (socket says 172.21.15.50:46390) 2023-12-19T14:39:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 -- 172.21.15.50:0/1939240932 learned_addr learned my addr 172.21.15.50:0/1939240932 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc731723700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 0x7fc71c00b9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc72c060f70 tx=0x7fc72c070c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c672ee35ad219803 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f096e2700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f096e2700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48184/0 (socket says 172.21.15.50:48184) 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f9f096e2700 1 -- 172.21.15.50:0/2122271615 learned_addr learned my addr 172.21.15.50:0/2122271615 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8415f790 con 0x7fec8413b0b0 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 -- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20980874b0 msgr2=0x7f209815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20980874b0 0x7f209815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 -- 172.21.15.50:0/1939240932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f209815d270 msgr2=0x7f2098086f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f209815d270 0x7f2098086f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7f2097fff700 1 -- 172.21.15.50:0/1939240932 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098089910 con 0x7f20840037b0 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0711f0 con 0x7fc71c0052e0 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fbf4f7fe700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 0x7fbf38062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.177+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbf5007c6e0 con 0x7fbf581549f0 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fbf56ffd700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 0x7fbf38062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f2097fff700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f208800b380 tx=0x7f2088009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc72affd700 1 -- 172.21.15.50:0/2301030581 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c070250 con 0x7fc71c0052e0 2023-12-19T14:39:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff74004d050 con 0x7ff74c06ced0 2023-12-19T14:39:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088004dc0 con 0x7f20840037b0 2023-12-19T14:39:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208800d050 con 0x7f20840037b0 2023-12-19T14:39:25.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880073c0 con 0x7f20840037b0 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 msgr2=0x7fc71c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 0x7fc71c00b9d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc72c060f70 tx=0x7fc72c070c50 comp rx=0 tx=0).stop 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 -- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 msgr2=0x7f9f0415eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0415eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 -- 172.21.15.50:0/2122271615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f040afd20 msgr2=0x7f9f04148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f040afd20 0x7f9f04148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fbf56ffd700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 0x7fbf38062b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e00 tx=0x7fbf40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fec740079e0 con 0x7fec8413b0b0 2023-12-19T14:39:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2098089bf0 con 0x7f20840037b0 2023-12-19T14:39:25.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209815b380 con 0x7f20840037b0 2023-12-19T14:39:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 -- 172.21.15.50:0/2122271615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef40094f0 con 0x7f9f04160e60 2023-12-19T14:39:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 shutdown_connections 2023-12-19T14:39:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 --2- 172.21.15.50:0/2301030581 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc71c0052e0 0x7fc71c00b9d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 >> 172.21.15.50:0/2301030581 conn(0x7fc724054e30 msgr2=0x7fc724055230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f9f096e2700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00bc30 tx=0x7f9efc009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 shutdown_connections 2023-12-19T14:39:25.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fc732985700 1 -- 172.21.15.50:0/2301030581 wait complete. 2023-12-19T14:39:25.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fec7b7fe700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 0x7fec64062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fec7407d1b0 con 0x7fec8413b0b0 2023-12-19T14:39:25.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209804be40 con 0x7f20840037b0 2023-12-19T14:39:25.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.181+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 0x7fec64062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc00c9b0 con 0x7f9f04160e60 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0416e470 con 0x7f9f04160e60 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f04171250 con 0x7f9f04160e60 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc00cb50 con 0x7f9f04160e60 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc014410 con 0x7f9f04160e60 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf5004ca00 con 0x7fbf581549f0 2023-12-19T14:39:25.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7fec7a7fc700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 0x7fec64062b00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00abc0 tx=0x7fec6c01a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7fc732985700 1 Processor -- start 2023-12-19T14:39:25.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7fc732985700 1 -- start start 2023-12-19T14:39:25.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.185+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9efc014800 con 0x7f9f04160e60 2023-12-19T14:39:25.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c0052e0 0x7fc7240e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee40052b0 con 0x7f9f04160e60 2023-12-19T14:39:25.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 0x7fc7240e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72404fe30 con 0x7fc71c0052e0 2023-12-19T14:39:25.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72404f870 con 0x7fc7240d4370 2023-12-19T14:39:25.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7f9f00ff9700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 0x7f9ef0062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc732985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72404fb50 con 0x7fc7240d3f00 2023-12-19T14:39:25.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:46408/0 (socket says 172.21.15.50:46408) 2023-12-19T14:39:25.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 -- 172.21.15.50:0/2583529505 learned_addr learned my addr 172.21.15.50:0/2583529505 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:25.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9efc07cbf0 con 0x7f9f04160e60 2023-12-19T14:39:25.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc731723700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 0x7fc7240e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72bfff700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c0052e0 0x7fc7240e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 -- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 msgr2=0x7fc7240e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 0x7fc7240e4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 -- 172.21.15.50:0/2583529505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c0052e0 msgr2=0x7fc7240e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c0052e0 0x7fc7240e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 -- 172.21.15.50:0/2583529505 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72404c010 con 0x7fc7240d3f00 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc731723700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 0x7fc7240e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7f9f08ee1700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 0x7f9ef0062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.189+0000 7fc72b7fe700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc7180109f0 tx=0x7fc71800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.193+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec7404bbe0 con 0x7fec8413b0b0 2023-12-19T14:39:25.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.193+0000 7f9f08ee1700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 0x7f9ef0062a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f040c3600 tx=0x7f9ef80091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.193+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2088007730 con 0x7f20840037b0 2023-12-19T14:39:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.193+0000 7f2094ff9700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 0x7f2078062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2088054a50 con 0x7f20840037b0 2023-12-19T14:39:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f208804e340 con 0x7f20840037b0 2023-12-19T14:39:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7f208f7fe700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 0x7f2078062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71800e8d0 con 0x7fc7240d3f00 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71800ea30 con 0x7fc7240d3f00 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71800ee30 con 0x7fc7240d3f00 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72404c2f0 con 0x7fc7240d3f00 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72404ed20 con 0x7fc7240d3f00 2023-12-19T14:39:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.197+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9efc04abe0 con 0x7f9f04160e60 2023-12-19T14:39:25.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.201+0000 7f208f7fe700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 0x7f2078062ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f209815c1b0 tx=0x7f208001a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.205+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc724005530 con 0x7fc7240d3f00 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.205+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc718014640 con 0x7fc7240d3f00 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.205+0000 7fc717fff700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 0x7fc710062b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.205+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc71807d180 con 0x7fc7240d3f00 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.205+0000 7fc72bfff700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 0x7fc710062b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.209+0000 7fc72bfff700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 0x7fc710062b80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc720001c70 tx=0x7fc72000a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:25.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.209+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc71804d460 con 0x7fc7240d3f00 2023-12-19T14:39:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.497+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7cf4159430 con 0x7f7cf406a580 2023-12-19T14:39:25.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.501+0000 7f7cf0ff9700 1 -- 172.21.15.50:0/3535431778 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7cec04ff70 con 0x7f7cf406a580 2023-12-19T14:39:25.509 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:25.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 msgr2=0x7f7ce0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 0x7f7ce0062a00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8001360 tx=0x7f7ce800a320 comp rx=0 tx=0).stop 2023-12-19T14:39:25.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 msgr2=0x7f7cf414d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec00b120 comp rx=0 tx=0).stop 2023-12-19T14:39:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 shutdown_connections 2023-12-19T14:39:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ce0060550 0x7f7ce0062a00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7cf4152ba0 0x7f7cf41543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7cf40cad40 0x7f7cf4153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 --2- 172.21.15.50:0/3535431778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cf406a580 0x7f7cf414d780 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009d90 tx=0x7f7cec00b120 comp rx=0 tx=0).stop 2023-12-19T14:39:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 >> 172.21.15.50:0/3535431778 conn(0x7f7cf40670e0 msgr2=0x7f7cf40c0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 shutdown_connections 2023-12-19T14:39:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7f7cfb185700 1 -- 172.21.15.50:0/3535431778 wait complete. 2023-12-19T14:39:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7facc4005830 con 0x7face41591f0 2023-12-19T14:39:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.509+0000 7faccbfff700 1 -- 172.21.15.50:0/1857665045 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facdc050cc0 con 0x7face41591f0 2023-12-19T14:39:25.514 INFO:teuthology.orchestra.run.smithi050.stdout:55834574854 2023-12-19T14:39:25.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.513+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff74c1550a0 con 0x7ff74c06ced0 2023-12-19T14:39:25.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 msgr2=0x7facd0062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 0x7facd0062ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7facd40099b0 tx=0x7facd4016000 comp rx=0 tx=0).stop 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 msgr2=0x7face40b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facdc007f60 tx=0x7facdc0098e0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 shutdown_connections 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facd0060620 0x7facd0062ad0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7face41591f0 0x7face40b4390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7face4158e20 0x7face414a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 --2- 172.21.15.50:0/1857665045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7face40ce810 0x7face41496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 >> 172.21.15.50:0/1857665045 conn(0x7face4067bb0 msgr2=0x7face41533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7ff73dffb700 1 -- 172.21.15.50:0/740471084 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff74004d050 con 0x7ff74c06ced0 2023-12-19T14:39:25.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 shutdown_connections 2023-12-19T14:39:25.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.517+0000 7face948e700 1 -- 172.21.15.50:0/1857665045 wait complete. 2023-12-19T14:39:25.521 INFO:teuthology.orchestra.run.smithi050.stdout:55834574854 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 msgr2=0x7ff730062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 0x7ff730062a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0afdc0 tx=0x7ff748007f30 comp rx=0 tx=0).stop 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 msgr2=0x7ff74c0ac4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0ac4f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff74000ecb0 tx=0x7ff7400091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 shutdown_connections 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff74c06d370 0x7ff74c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7300605d0 0x7ff730062a80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbf3c005560 con 0x7fbf581549f0 2023-12-19T14:39:25.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff74c158e50 0x7ff74c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 --2- 172.21.15.50:0/740471084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff74c06ced0 0x7ff74c0ac4f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 >> 172.21.15.50:0/740471084 conn(0x7ff74c067bb0 msgr2=0x7ff74c14c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 shutdown_connections 2023-12-19T14:39:25.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7ff75402e700 1 -- 172.21.15.50:0/740471084 wait complete. 2023-12-19T14:39:25.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.521+0000 7fbf4f7fe700 1 -- 172.21.15.50:0/1388047393 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf50050020 con 0x7fbf581549f0 2023-12-19T14:39:25.529 INFO:teuthology.orchestra.run.smithi050.stdout:55834574854 2023-12-19T14:39:25.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fec8404be40 con 0x7fec8413b0b0 2023-12-19T14:39:25.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fec7b7fe700 1 -- 172.21.15.50:0/3190862307 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec7404d5c0 con 0x7fec8413b0b0 2023-12-19T14:39:25.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9ee4005560 con 0x7f9f04160e60 2023-12-19T14:39:25.530 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:25.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7f9f00ff9700 1 -- 172.21.15.50:0/2122271615 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9efc04ce50 con 0x7f9f04160e60 2023-12-19T14:39:25.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 msgr2=0x7fbf38062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 0x7fbf38062b50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e00 tx=0x7fbf40008040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 msgr2=0x7fbf581749f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.525+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 0x7fbf581749f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf5000c810 tx=0x7fbf50002620 comp rx=0 tx=0).stop 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 shutdown_connections 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf581549f0 0x7fbf581749f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf380606a0 0x7fbf38062b50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf580cc0a0 0x7fbf581517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 --2- 172.21.15.50:0/1388047393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf5806a580 0x7fbf580b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 >> 172.21.15.50:0/1388047393 conn(0x7fbf58067bb0 msgr2=0x7fbf5806a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 shutdown_connections 2023-12-19T14:39:25.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fbf57fff700 1 -- 172.21.15.50:0/1388047393 wait complete. 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 msgr2=0x7f9ef0062a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 0x7f9ef0062a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f040c3600 tx=0x7f9ef80091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 msgr2=0x7f9f0415f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00bc30 tx=0x7f9efc009820 comp rx=0 tx=0).stop 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 msgr2=0x7fec64062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 0x7fec64062b00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00abc0 tx=0x7fec6c01a000 comp rx=0 tx=0).stop 2023-12-19T14:39:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 msgr2=0x7fec840af870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec7400b250 tx=0x7fec74009040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 shutdown_connections 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec8413b480 0x7fec8406ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec64060650 0x7fec64062b00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec841383a0 0x7fec8406b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 --2- 172.21.15.50:0/3190862307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec8413b0b0 0x7fec840af870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.529+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 >> 172.21.15.50:0/3190862307 conn(0x7fec84067bb0 msgr2=0x7fec8414d010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 shutdown_connections 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 shutdown_connections 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f04160e60 0x7f9f0415f410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9ef00605a0 0x7f9ef0062a50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f0415d050 0x7f9f0415eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 --2- 172.21.15.50:0/2122271615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f040afd20 0x7f9f04148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 >> 172.21.15.50:0/2122271615 conn(0x7f9f04067b60 msgr2=0x7f9f040aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7fec89545700 1 -- 172.21.15.50:0/3190862307 wait complete. 2023-12-19T14:39:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 shutdown_connections 2023-12-19T14:39:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f20980cabd0 con 0x7f20840037b0 2023-12-19T14:39:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f9f0b145700 1 -- 172.21.15.50:0/2122271615 wait complete. 2023-12-19T14:39:25.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7f2094ff9700 1 -- 172.21.15.50:0/1939240932 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2088018030 con 0x7f20840037b0 2023-12-19T14:39:25.540 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2023-12-19T14:39:25.541 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:25.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.533+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc724054160 con 0x7fc7240d3f00 2023-12-19T14:39:25.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7fc717fff700 1 -- 172.21.15.50:0/2583529505 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc718050a80 con 0x7fc7240d3f00 2023-12-19T14:39:25.542 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2023-12-19T14:39:25.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 msgr2=0x7f2078062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 0x7f2078062ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f209815c1b0 tx=0x7f208001a000 comp rx=0 tx=0).stop 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 msgr2=0x7f209815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f208800b380 tx=0x7f2088009040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 shutdown_connections 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20980874b0 0x7f209815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2078060620 0x7f2078062ad0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f209815d270 0x7f2098086f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 --2- 172.21.15.50:0/1939240932 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20840037b0 0x7f209815ae40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 >> 172.21.15.50:0/1939240932 conn(0x7f2098067bb0 msgr2=0x7f209806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 shutdown_connections 2023-12-19T14:39:25.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.537+0000 7f209d20c700 1 -- 172.21.15.50:0/1939240932 wait complete. 2023-12-19T14:39:25.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 msgr2=0x7fc710062b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 0x7fc710062b80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc720001c70 tx=0x7fc72000a040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 msgr2=0x7fc7240e4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc7180109f0 tx=0x7fc71800b040 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 shutdown_connections 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7240d4370 0x7fc7240e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7100606d0 0x7fc710062b80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7240d3f00 0x7fc7240e4500 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 --2- 172.21.15.50:0/2583529505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc71c0052e0 0x7fc7240e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.541+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 >> 172.21.15.50:0/2583529505 conn(0x7fc724054e30 msgr2=0x7fc724036520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.545+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 shutdown_connections 2023-12-19T14:39:25.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:25.545+0000 7fc732985700 1 -- 172.21.15.50:0/2583529505 wait complete. 2023-12-19T14:39:25.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2023-12-19T14:39:25.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-12-19T14:39:25.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2023-12-19T14:39:25.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2023-12-19T14:39:25.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-12-19T14:39:25.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2023-12-19T14:39:25.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2023-12-19T14:39:26.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T14:39:26.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-19T14:39:26.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-19T14:39:26.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-19T14:39:26.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-19T14:39:26.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-19T14:39:26.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-19T14:39:26.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-19T14:39:26.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.757+0000 7f7a6cf05700 1 Processor -- start 2023-12-19T14:39:26.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.757+0000 7f7a6cf05700 1 -- start start 2023-12-19T14:39:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.757+0000 7f7a6cf05700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 0x7f7a68154de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6cf05700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 0x7f7a680c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6cf05700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7a680ad6e0 0x7f7a680ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6cf05700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680c6dc0 con 0x7f7a6806b300 2023-12-19T14:39:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6cf05700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6806ceb0 con 0x7f7a680ad6e0 2023-12-19T14:39:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6cf05700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ae330 con 0x7f7a68154a10 2023-12-19T14:39:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a66d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7a680ad6e0 0x7f7a680ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a66d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7a680ad6e0 0x7f7a680ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60092/0 (socket says 172.21.15.50:60092) 2023-12-19T14:39:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a66d9d700 1 -- 172.21.15.50:0/2503571720 learned_addr learned my addr 172.21.15.50:0/2503571720 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a65d9b700 1 --2- 172.21.15.50:0/2503571720 >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 0x7f7a680c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.761+0000 7f7a6659c700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 0x7f7a68154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 -- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 msgr2=0x7f7a68154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 0x7f7a68154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 -- 172.21.15.50:0/2503571720 >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 msgr2=0x7f7a680c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/2503571720 >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 0x7f7a680c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 -- 172.21.15.50:0/2503571720 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680ae050 con 0x7f7a680ad6e0 2023-12-19T14:39:26.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a65d9b700 1 --2- 172.21.15.50:0/2503571720 >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 0x7f7a680c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:26.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a6659c700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 0x7f7a68154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:26.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a680ca140 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a6806ce20 tx=0x7f7a60009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fadbf967144bd55 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a6559a700 1 -- 172.21.15.50:0/2503571720 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a600092e0 con 0x7f7a680ad6e0 2023-12-19T14:39:26.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a6559a700 1 -- 172.21.15.50:0/2503571720 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6000ca60 con 0x7f7a680ad6e0 2023-12-19T14:39:26.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.765+0000 7f7a6559a700 1 -- 172.21.15.50:0/2503571720 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a600092e0 con 0x7f7a680ad6e0 2023-12-19T14:39:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 -- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 msgr2=0x7f7a680ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a680ca140 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a6806ce20 tx=0x7f7a60009840 comp rx=0 tx=0).stop 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 -- 172.21.15.50:0/2503571720 shutdown_connections 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/2503571720 >> v2:172.21.15.50:3300/0 conn(0x7f7a6806b300 0x7f7a680c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a680ca140 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/2503571720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a68154a10 0x7f7a68154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 -- 172.21.15.50:0/2503571720 >> 172.21.15.50:0/2503571720 conn(0x7f7a680670e0 msgr2=0x7f7a680674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 -- 172.21.15.50:0/2503571720 shutdown_connections 2023-12-19T14:39:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.769+0000 7f7a6cf05700 1 -- 172.21.15.50:0/2503571720 wait complete. 2023-12-19T14:39:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 Processor -- start 2023-12-19T14:39:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 -- start start 2023-12-19T14:39:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a6806b300 0x7f7a68152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a68153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680c3ae0 con 0x7f7a68154a10 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a681537b0 con 0x7f7a680ad6e0 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6cf05700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a68153a90 con 0x7f7a6806b300 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a66d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a65d9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a68153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a66d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58956/0 (socket says 172.21.15.50:58956) 2023-12-19T14:39:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a66d9d700 1 -- 172.21.15.50:0/425764810 learned_addr learned my addr 172.21.15.50:0/425764810 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.773+0000 7f7a6659c700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a6806b300 0x7f7a68152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 -- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a6806b300 msgr2=0x7f7a68152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a6806b300 0x7f7a68152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 -- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 msgr2=0x7f7a68153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a68153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 -- 172.21.15.50:0/425764810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a58009420 con 0x7f7a68154a10 2023-12-19T14:39:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a66d9d700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a6806cff0 tx=0x7f7a60019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a600092e0 con 0x7f7a68154a10 2023-12-19T14:39:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a600094e0 con 0x7f7a68154a10 2023-12-19T14:39:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60023680 con 0x7f7a68154a10 2023-12-19T14:39:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6000d040 con 0x7f7a68154a10 2023-12-19T14:39:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a68154050 con 0x7f7a68154a10 2023-12-19T14:39:26.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.777+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a680c3c90 con 0x7f7a68154a10 2023-12-19T14:39:26.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.789+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7a60003050 con 0x7f7a68154a10 2023-12-19T14:39:26.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.789+0000 7f7a567fc700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 0x7f7a48062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.789+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7a6007e630 con 0x7f7a68154a10 2023-12-19T14:39:26.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.789+0000 7f7a6659c700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 0x7f7a48062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.797+0000 7f7a6659c700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 0x7f7a48062ad0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009cd0 tx=0x7f7a5c0070e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.797+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7a6004e990 con 0x7f7a68154a10 2023-12-19T14:39:26.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 Processor -- start 2023-12-19T14:39:26.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 -- start start 2023-12-19T14:39:26.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f202400dfd0 0x7f20240568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f0db9e6e700 1 Processor -- start 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f20240dfb20 0x7f20240dff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2024056460 con 0x7f202400dc00 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2024055850 con 0x7f202400dfd0 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203188e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2024055b30 con 0x7f20240dfb20 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203088c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f202400dc00 0x7f2024051ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44586/0 (socket says 172.21.15.50:44586) 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f203088c700 1 -- 172.21.15.50:0/1149166078 learned_addr learned my addr 172.21.15.50:0/1149166078 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508061981 0 0) 0x7f2024056460 con 0x7f202400dc00 2023-12-19T14:39:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c003570 con 0x7f202400dc00 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202bfff700 1 --2- 172.21.15.50:0/1149166078 >> v2:172.21.15.50:3300/0 conn(0x7f202400dfd0 0x7f20240568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.809+0000 7f0db9e6e700 1 -- start start 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879687898 0 0) 0x7f201c003570 con 0x7f202400dc00 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 >> v2:172.21.15.125:3300/0 conn(0x7f20240dfb20 msgr2=0x7f20240dff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 --2- 172.21.15.50:0/1149166078 >> v2:172.21.15.125:3300/0 conn(0x7f20240dfb20 0x7f20240dff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 >> v2:172.21.15.50:3300/0 conn(0x7f202400dfd0 msgr2=0x7f20240568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 Processor -- start 2023-12-19T14:39:26.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db9e6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db406dc00 0x7f0db406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db9e6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0db406baa0 0x7f0db40b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db9e6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db406b0d0 con 0x7f0db4158cc0 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db9e6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db406bec0 con 0x7f0db406dc00 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db9e6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db40b4790 con 0x7f0db406baa0 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 --2- 172.21.15.50:0/1149166078 >> v2:172.21.15.50:3300/0 conn(0x7f202400dfd0 0x7f20240568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 -- start start 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024055e10 con 0x7f202400dc00 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f2020001470 con 0x7f202400dc00 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 >> v1:172.21.15.50:6789/0 conn(0x7f202400dc00 legacy=0x7f2024051ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201c003ae0 0x7f201c003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db406dc00 0x7f0db406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0db406dc00 0x7f0db406b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58970/0 (socket says 172.21.15.50:58970) 2023-12-19T14:39:26.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0db3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0db406baa0 0x7f0db40b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 -- 172.21.15.50:0/3145266293 learned_addr learned my addr 172.21.15.50:0/3145266293 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c004810 0x7f201c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 0x7f201c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024055850 con 0x7f201c003ae0 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2024055b30 con 0x7f201c004810 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c00c280 con 0x7f201c005230 2023-12-19T14:39:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f202bfff700 1 --2- 172.21.15.50:0/1149166078 >> v2:172.21.15.50:3300/0 conn(0x7f202400dfd0 0x7f20240568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f603c158e20 0x7f603c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c06b730 con 0x7f603c1591f0 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f203108d700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 0x7f201c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f202bfff700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c004810 0x7f201c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 -- 172.21.15.50:0/3145266293 >> v2:172.21.15.125:3300/0 conn(0x7f0db406baa0 msgr2=0x7f0db40b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 --2- 172.21.15.50:0/3145266293 >> v2:172.21.15.125:3300/0 conn(0x7f0db406baa0 0x7f0db40b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 -- 172.21.15.50:0/3145266293 >> v1:172.21.15.50:6789/0 conn(0x7f0db4158cc0 legacy=0x7f0db406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f0daaffd700 1 -- 172.21.15.50:0/3145266293 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db406a7e0 con 0x7f0db406dc00 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0daaffd700 1 --2- 172.21.15.50:0/3145266293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db406b560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0db406b040 tx=0x7f0da4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cf481737ceef91 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7fc002e78700 1 Processor -- start 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c06ab20 con 0x7f603c0ce810 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.813+0000 7f604186b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c06ae00 con 0x7f603c158e20 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f6040869700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f603c158e20 0x7f603c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f604106a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f603c0ce810 0x7f603c0ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40934/0 (socket says 172.21.15.50:40934) 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f604106a700 1 -- 172.21.15.50:0/319867147 learned_addr learned my addr 172.21.15.50:0/319867147 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 -- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c004810 msgr2=0x7f201c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c004810 0x7f201c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 -- 172.21.15.50:0/1149166078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201c003ae0 msgr2=0x7f201c003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201c003ae0 0x7f201c003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 -- 172.21.15.50:0/1149166078 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024055e10 con 0x7f201c005230 2023-12-19T14:39:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 -- 172.21.15.50:0/1149166078 reap_dead start 2023-12-19T14:39:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203108d700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 0x7f201c00ba60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f202c060f70 tx=0x7f202c070c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f055d421e460807d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db2ffd700 1 -- 172.21.15.50:0/3145266293 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da400e070 con 0x7f0db406dc00 2023-12-19T14:39:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db2ffd700 1 -- 172.21.15.50:0/3145266293 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da400b9e0 con 0x7f0db406dc00 2023-12-19T14:39:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db2ffd700 1 -- 172.21.15.50:0/3145266293 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da40092e0 con 0x7f0db406dc00 2023-12-19T14:39:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7fc002e78700 1 -- start start 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f6040869700 1 -- 172.21.15.50:0/319867147 >> v1:172.21.15.125:6789/0 conn(0x7f603c0ce810 legacy=0x7f603c0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f60337fe700 1 -- 172.21.15.50:0/319867147 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750241728 0 0) 0x7f603c06ab20 con 0x7f603c0ce810 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f6040869700 1 -- 172.21.15.50:0/319867147 >> v1:172.21.15.50:6789/0 conn(0x7f603c1591f0 legacy=0x7f603c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f6040869700 1 -- 172.21.15.50:0/319867147 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c06b0e0 con 0x7f603c158e20 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0711f0 con 0x7f201c005230 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c070250 con 0x7f201c005230 2023-12-19T14:39:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f202b7fe700 1 -- 172.21.15.50:0/1149166078 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c070400 con 0x7f201c005230 2023-12-19T14:39:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3145266293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 msgr2=0x7f0db406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3145266293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db406b560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0db406b040 tx=0x7f0da4009840 comp rx=0 tx=0).stop 2023-12-19T14:39:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3145266293 shutdown_connections 2023-12-19T14:39:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f6040869700 1 --2- 172.21.15.50:0/319867147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c158e20 0x7f603c06a540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f60380098e0 tx=0x7f60380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56fef6619dc87c84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3145266293 >> v2:172.21.15.125:3300/0 conn(0x7f0db406baa0 0x7f0db40b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3145266293 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db406b560 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3145266293 >> 172.21.15.50:0/3145266293 conn(0x7f0db4067bb0 msgr2=0x7f0db40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3145266293 shutdown_connections 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3145266293 wait complete. 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f60337fe700 1 -- 172.21.15.50:0/319867147 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038009e80 con 0x7f603c158e20 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f60337fe700 1 -- 172.21.15.50:0/319867147 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603800b760 con 0x7f603c158e20 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f60337fe700 1 -- 172.21.15.50:0/319867147 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603800bae0 con 0x7f603c158e20 2023-12-19T14:39:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203188e700 1 -- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 msgr2=0x7f201c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f203188e700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 0x7f201c00ba60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f202c060f70 tx=0x7f202c070c50 comp rx=0 tx=0).stop 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f604186b700 1 -- 172.21.15.50:0/319867147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c158e20 msgr2=0x7f603c06a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.817+0000 7f604186b700 1 --2- 172.21.15.50:0/319867147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c158e20 0x7f603c06a540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f60380098e0 tx=0x7f60380091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 -- 172.21.15.50:0/319867147 shutdown_connections 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 --2- 172.21.15.50:0/319867147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c158e20 0x7f603c06a540 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 -- 172.21.15.50:0/319867147 >> 172.21.15.50:0/319867147 conn(0x7f603c067bb0 msgr2=0x7f603c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc002e78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbff404f470 0x7fbff40533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 -- 172.21.15.50:0/319867147 shutdown_connections 2023-12-19T14:39:26.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 Processor -- start 2023-12-19T14:39:26.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc002e78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff404f970 con 0x7fbff40e43d0 2023-12-19T14:39:26.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 -- 172.21.15.50:0/319867147 wait complete. 2023-12-19T14:39:26.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 -- 172.21.15.50:0/1149166078 shutdown_connections 2023-12-19T14:39:26.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 --2- 172.21.15.50:0/1149166078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201c005230 0x7f201c00ba60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 -- 172.21.15.50:0/1149166078 >> 172.21.15.50:0/1149166078 conn(0x7f202400cd70 msgr2=0x7f2024056e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 -- 172.21.15.50:0/1149166078 shutdown_connections 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc000c14700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbff4053000 0x7fbff40e3cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40938/0 (socket says 172.21.15.50:40938) 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc000c14700 1 -- 172.21.15.50:0/3990148378 learned_addr learned my addr 172.21.15.50:0/3990148378 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff404fc50 con 0x7fbff4053000 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3990148378 >> v2:172.21.15.50:3300/0 conn(0x7fbff404f470 0x7fbff40533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff4054b50 con 0x7fbff404f470 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248739646 0 0) 0x7fbff404fc50 con 0x7fbff4053000 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfec003570 con 0x7fbff4053000 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 -- 172.21.15.50:0/1149166078 wait complete. 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 -- start start 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db4152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0db4158cc0 0x7f0db415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db41515e0 con 0x7f0db406dc00 2023-12-19T14:39:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4150000 con 0x7f0db406baa0 2023-12-19T14:39:26.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db9e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db41502e0 con 0x7f0db4158cc0 2023-12-19T14:39:26.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f203188e700 1 Processor -- start 2023-12-19T14:39:26.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f149e44c700 1 Processor -- start 2023-12-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db37fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db37fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60132/0 (socket says 172.21.15.50:60132) 2023-12-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db37fe700 1 -- 172.21.15.50:0/3627146037 learned_addr learned my addr 172.21.15.50:0/3627146037 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f604186b700 1 Processor -- start 2023-12-19T14:39:26.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db3fff700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0db4158cc0 0x7f0db415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124291293 0 0) 0x7fbff404f970 con 0x7fbff40e43d0 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff404fc50 con 0x7fbff40e43d0 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407430555 0 0) 0x7fbfec003570 con 0x7fbff4053000 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 >> v2:172.21.15.50:3300/0 conn(0x7fbff404f470 msgr2=0x7fbff40533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 --2- 172.21.15.50:0/3990148378 >> v2:172.21.15.50:3300/0 conn(0x7fbff404f470 0x7fbff40533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 >> v1:172.21.15.50:6789/0 conn(0x7fbff40e43d0 legacy=0x7fbff40063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff404ff30 con 0x7fbff4053000 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fbff0001470 con 0x7fbff4053000 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 >> v1:172.21.15.125:6789/0 conn(0x7fbff4053000 legacy=0x7fbff40e3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfec003bf0 0x7fbfec003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbfec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfec0051c0 0x7fbfec00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4054b50 con 0x7fbfec003bf0 2023-12-19T14:39:26.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec00bfa0 con 0x7fbfec004810 2023-12-19T14:39:26.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec01d170 con 0x7fbfec0051c0 2023-12-19T14:39:26.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f604186b700 1 -- start start 2023-12-19T14:39:26.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbfec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f0db37fe700 1 -- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0db4158cc0 msgr2=0x7f0db415a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db37fe700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0db4158cc0 0x7f0db415a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db37fe700 1 -- 172.21.15.50:0/3627146037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 msgr2=0x7f0db4152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db37fe700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db4152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db37fe700 1 -- 172.21.15.50:0/3627146037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4009ae0 con 0x7f0db406baa0 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc001415700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfec0051c0 0x7fbfec00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc000c14700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfec003bf0 0x7fbfec003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db37fe700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00b600 tx=0x7f0d9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 -- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfec0051c0 msgr2=0x7fbfec00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfec0051c0 0x7fbfec00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 -- 172.21.15.50:0/3990148378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfec003bf0 msgr2=0x7fbfec003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfec003bf0 0x7fbfec003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 -- 172.21.15.50:0/3990148378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec003570 con 0x7fbfec004810 2023-12-19T14:39:26.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00f510 con 0x7f0db406baa0 2023-12-19T14:39:26.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.821+0000 7f149e44c700 1 -- start start 2023-12-19T14:39:26.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbfec004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006f00 tx=0x7fbfe80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f891dbd97dadaa7a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc001415700 1 -- 172.21.15.50:0/3990148378 reap_dead start 2023-12-19T14:39:26.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe800e710 con 0x7fbfec004810 2023-12-19T14:39:26.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f149e44c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1498154ff0 0x7f149806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe800e870 con 0x7fbfec004810 2023-12-19T14:39:26.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f149e44c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f149806cd00 0x7f149806a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c00f670 con 0x7f0db406baa0 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00fa30 con 0x7f0db406baa0 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db41505c0 con 0x7f0db406baa0 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db4150c00 con 0x7f0db406baa0 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f149d44a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1498154c20 0x7f14980abca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44634/0 (socket says 172.21.15.50:44634) 2023-12-19T14:39:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f149d44a700 1 -- 172.21.15.50:0/3189422610 learned_addr learned my addr 172.21.15.50:0/3189422610 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:26.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fbffb7fe700 1 -- 172.21.15.50:0/3990148378 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe800ebf0 con 0x7fbfec004810 2023-12-19T14:39:26.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 msgr2=0x7fbfec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc002e78700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbfec004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006f00 tx=0x7fbfe80091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7f203188e700 1 -- start start 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 shutdown_connections 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc002e78700 1 --2- 172.21.15.50:0/3990148378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbfec004c80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.825+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 >> 172.21.15.50:0/3990148378 conn(0x7fbff4054e30 msgr2=0x7fbff4055230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f603c0ce810 0x7f603c151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f603c158e20 0x7f603c1595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c158210 con 0x7f603c158e20 2023-12-19T14:39:26.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c152310 con 0x7f603c1591f0 2023-12-19T14:39:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15a1d0 con 0x7f603c0ce810 2023-12-19T14:39:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149cc49700 1 --2- 172.21.15.50:0/3189422610 >> v2:172.21.15.125:3300/0 conn(0x7f1498154ff0 0x7f149806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149dc4b700 1 --2- 172.21.15.50:0/3189422610 >> v2:172.21.15.50:3300/0 conn(0x7f149806cd00 0x7f149806a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149806d160 con 0x7f1498154c20 2023-12-19T14:39:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4150750 con 0x7f0db406baa0 2023-12-19T14:39:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980c5c70 con 0x7f149806cd00 2023-12-19T14:39:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 shutdown_connections 2023-12-19T14:39:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60150/0 (socket says 172.21.15.50:60150) 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 -- 172.21.15.50:0/3679358889 learned_addr learned my addr 172.21.15.50:0/3679358889 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f6040869700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f603c0ce810 0x7f603c151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 -- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f603c0ce810 msgr2=0x7f603c151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980ac3b0 con 0x7f1498154ff0 2023-12-19T14:39:26.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f2f6cc2e700 1 Processor -- start 2023-12-19T14:39:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f603c0ce810 0x7f603c151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 -- 172.21.15.50:0/3679358889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f603c158e20 msgr2=0x7f603c1595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f603c158e20 0x7f603c1595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 -- 172.21.15.50:0/3679358889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038009420 con 0x7f603c1591f0 2023-12-19T14:39:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f2f6cc2e700 1 -- start start 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7fc002e78700 1 -- 172.21.15.50:0/3990148378 wait complete. 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7fc002e78700 1 Processor -- start 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f203188e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 0x7f20240d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f203188e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 0x7f2024039c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f203188e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7fc002e78700 1 -- start start 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604106a700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f603400ccc0 tx=0x7f6034001a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603400ce70 con 0x7f603c1591f0 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603400ab70 con 0x7f603c1591f0 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60340070f0 con 0x7f603c1591f0 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149cc49700 1 -- 172.21.15.50:0/3189422610 >> v2:172.21.15.50:3300/0 conn(0x7f149806cd00 msgr2=0x7f149806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149cc49700 1 --2- 172.21.15.50:0/3189422610 >> v2:172.21.15.50:3300/0 conn(0x7f149806cd00 0x7f149806a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149cc49700 1 -- 172.21.15.50:0/3189422610 >> v1:172.21.15.50:6789/0 conn(0x7f1498154c20 legacy=0x7f14980abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f149cc49700 1 -- 172.21.15.50:0/3189422610 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980c5fd0 con 0x7f1498154ff0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0d9c01a400 con 0x7f0db406baa0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c15a4b0 con 0x7f603c1591f0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.829+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c15aa70 con 0x7f603c1591f0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203088c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 0x7f20240d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f202bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 0x7f2024039c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203088c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 0x7f20240d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60154/0 (socket says 172.21.15.50:60154) 2023-12-19T14:39:26.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f202bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 0x7f2024039c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54272/0 (socket says 172.21.15.50:54272) 2023-12-19T14:39:26.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6cc2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f68158cc0 0x7f2f6806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59018/0 (socket says 172.21.15.50:59018) 2023-12-19T14:39:26.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6cc2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2f680b3870 0x7f2f680b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f149cc49700 1 --2- 172.21.15.50:0/3189422610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1498154ff0 0x7f149806c7c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f149806a030 tx=0x7f148c00abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f9ecc2e6679c42 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f174144c700 1 Processor -- start 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6cc2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6806b0d0 con 0x7f2f6806dc00 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6659c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f68158cc0 0x7f2f6806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f65d9b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2f6806dc00 0x7f2f6806b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40944/0 (socket says 172.21.15.50:40944) 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f65d9b700 1 -- 172.21.15.50:0/429402109 learned_addr learned my addr 172.21.15.50:0/429402109 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f0dabfff700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 0x7f0d94062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f174144c700 1 -- start start 2023-12-19T14:39:26.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f66d9d700 1 --2- 172.21.15.50:0/429402109 >> v2:172.21.15.50:3300/0 conn(0x7f2f680b3870 0x7f2f680b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f680b4790 con 0x7f2f680b3870 2023-12-19T14:39:26.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6806d710 con 0x7f2f68158cc0 2023-12-19T14:39:26.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0d9c07c5b0 con 0x7f0db406baa0 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f0daaffd700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 0x7f0d94062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c0c6e70 con 0x7f603c1591f0 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203088c700 1 -- 172.21.15.50:0/3865306280 learned_addr learned my addr 172.21.15.50:0/3865306280 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240f66a0 con 0x7f20240f5770 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240d3590 con 0x7f201c005230 2023-12-19T14:39:26.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20240d2980 con 0x7f20240d24f0 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 -- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 msgr2=0x7f2024039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 0x7f2024039c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 -- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 msgr2=0x7f20240d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 0x7f20240d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240d2c60 con 0x7f20240f5770 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff404b9e0 0x7fbff404a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 0x7fbff4036560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40da0e0 con 0x7fbff40e1ba0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40da3c0 con 0x7fbfec004810 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4034f40 con 0x7fbff404b9e0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc000c14700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc000c14700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60164/0 (socket says 172.21.15.50:60164) 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc000c14700 1 -- 172.21.15.50:0/3597022389 learned_addr learned my addr 172.21.15.50:0/3597022389 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 0x7fbff4036560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fbffbfff700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff404b9e0 0x7fbff404a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 -- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff404b9e0 msgr2=0x7fbff404a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff404b9e0 0x7fbff404a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 -- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 msgr2=0x7fbff4034a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 -- 172.21.15.50:0/3597022389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff404bee0 con 0x7fbff40e1ba0 2023-12-19T14:39:26.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc000c14700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6659c700 1 -- 172.21.15.50:0/429402109 >> v2:172.21.15.50:3300/0 conn(0x7f2f680b3870 msgr2=0x7f2f680b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6659c700 1 --2- 172.21.15.50:0/429402109 >> v2:172.21.15.50:3300/0 conn(0x7f2f680b3870 0x7f2f680b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6659c700 1 -- 172.21.15.50:0/429402109 >> v1:172.21.15.125:6789/0 conn(0x7f2f6806dc00 legacy=0x7f2f6806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f2f6659c700 1 -- 172.21.15.50:0/429402109 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f680b41d0 con 0x7f2f68158cc0 2023-12-19T14:39:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc001415700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 0x7fbff4036560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbffc060be0 tx=0x7fbffc066cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f174144c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f173c158e20 0x7f173c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f203108d700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f202c065640 tx=0x7f202c06a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc08b070 con 0x7fbff40e1ba0 2023-12-19T14:39:26.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc064160 con 0x7fbff40e1ba0 2023-12-19T14:39:26.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc06d940 con 0x7fbff40e1ba0 2023-12-19T14:39:26.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7f174144c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c06b730 con 0x7f173c1591f0 2023-12-19T14:39:26.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f0daaffd700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 0x7f0d94062b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0db415b1e0 tx=0x7f0da4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff404c1c0 con 0x7fbff40e1ba0 2023-12-19T14:39:26.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.833+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff404aad0 con 0x7fbff40e1ba0 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f2f6659c700 1 --2- 172.21.15.50:0/429402109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f68158cc0 0x7f2f6806d1d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2f680d0830 tx=0x7f2f500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691bc6e018cdd0cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f173c158e20 0x7f173c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f174144c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c06ab20 con 0x7f173c0ce810 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f174144c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173c06ae00 con 0x7f173c158e20 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f173c158e20 0x7f173c06a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59048/0 (socket says 172.21.15.50:59048) 2023-12-19T14:39:26.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c094070 con 0x7f20240f5770 2023-12-19T14:39:26.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173bfff700 1 -- 172.21.15.50:0/3998721244 learned_addr learned my addr 172.21.15.50:0/3998721244 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20240d2f40 con 0x7f20240f5770 2023-12-19T14:39:26.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20240df380 con 0x7f20240f5770 2023-12-19T14:39:26.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0705d0 con 0x7f20240f5770 2023-12-19T14:39:26.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff40da530 con 0x7fbff40e1ba0 2023-12-19T14:39:26.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262568919 0 0) 0x7f173c06b730 con 0x7f173c1591f0 2023-12-19T14:39:26.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1728003570 con 0x7f173c1591f0 2023-12-19T14:39:26.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c094070 con 0x7f20240f5770 2023-12-19T14:39:26.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20240df570 con 0x7f20240f5770 2023-12-19T14:39:26.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbffc06dd50 con 0x7fbff40e1ba0 2023-12-19T14:39:26.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f202c06db50 con 0x7f20240f5770 2023-12-19T14:39:26.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690535383 0 0) 0x7f1728003570 con 0x7f173c1591f0 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 >> v2:172.21.15.50:3300/0 conn(0x7f173c158e20 msgr2=0x7f173c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 --2- 172.21.15.50:0/3998721244 >> v2:172.21.15.50:3300/0 conn(0x7f173c158e20 0x7f173c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 >> v1:172.21.15.125:6789/0 conn(0x7f173c0ce810 legacy=0x7f173c0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c06b0e0 con 0x7f173c1591f0 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f1724001470 con 0x7f173c1591f0 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 >> v1:172.21.15.50:6789/0 conn(0x7f173c1591f0 legacy=0x7f173c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 0x7f1728003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1728004810 0x7f1728004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280051c0 0x7f172800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c06ae00 con 0x7f1728004810 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172800c0d0 con 0x7f17280051c0 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172800c290 con 0x7f1728003a60 2023-12-19T14:39:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f1740c4b700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280051c0 0x7f172800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.837+0000 7f173bfff700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 0x7f1728003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173b7fe700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1728004810 0x7f1728004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7fbff8ff9700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 0x7fbfe0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbffc0f28f0 con 0x7fbff40e1ba0 2023-12-19T14:39:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 -- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280051c0 msgr2=0x7f172800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280051c0 0x7f172800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 -- 172.21.15.50:0/3998721244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1728004810 msgr2=0x7f1728004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1728004810 0x7f1728004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 -- 172.21.15.50:0/3998721244 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1728003570 con 0x7f1728003a60 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f1740c4b700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17280051c0 0x7f172800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f1740c4b700 1 -- 172.21.15.50:0/3998721244 reap_dead start 2023-12-19T14:39:26.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7fc000c14700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 0x7fbfe0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2028ff9700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 0x7f2010062a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f202c0f43f0 con 0x7f20240f5770 2023-12-19T14:39:26.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f203088c700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 0x7f2010062a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173bfff700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 0x7f1728003e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1734006fd0 tx=0x7f1734009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e76fb64510ad21b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173400e740 con 0x7f1728003a60 2023-12-19T14:39:26.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7fc000c14700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 0x7fbfe0062a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000e50 tx=0x7fbff0001d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f173affd700 1 -- 172.21.15.50:0/3998721244 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173400e8a0 con 0x7f1728003a60 2023-12-19T14:39:26.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f203088c700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 0x7f2010062a60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2020001120 tx=0x7f20200015a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d9c04c910 con 0x7f0db406baa0 2023-12-19T14:39:26.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f1497fff700 1 -- 172.21.15.50:0/3189422610 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00a380 con 0x7f1498154ff0 2023-12-19T14:39:26.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6559a700 1 -- 172.21.15.50:0/429402109 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50011630 con 0x7f2f68158cc0 2023-12-19T14:39:26.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6559a700 1 -- 172.21.15.50:0/429402109 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50011790 con 0x7f2f68158cc0 2023-12-19T14:39:26.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6559a700 1 -- 172.21.15.50:0/429402109 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50011b30 con 0x7f2f68158cc0 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 -- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 msgr2=0x7f1728003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 0x7f1728003e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1734006fd0 tx=0x7f1734009250 comp rx=0 tx=0).stop 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 -- 172.21.15.50:0/3998721244 shutdown_connections 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 --2- 172.21.15.50:0/3998721244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1728003a60 0x7f1728003e70 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1734006fd0 tx=0x7f1734009250 comp rx=0 tx=0).stop 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 -- 172.21.15.50:0/3998721244 >> 172.21.15.50:0/3998721244 conn(0x7f173c067bb0 msgr2=0x7f173c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 -- 172.21.15.50:0/3998721244 shutdown_connections 2023-12-19T14:39:26.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f68158cc0 msgr2=0x7f2f6806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/429402109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f68158cc0 0x7f2f6806d1d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2f680d0830 tx=0x7f2f500091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f174144c700 1 -- 172.21.15.50:0/3998721244 wait complete. 2023-12-19T14:39:26.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f1497fff700 1 -- 172.21.15.50:0/3189422610 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c00a4e0 con 0x7f1498154ff0 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 shutdown_connections 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/429402109 >> v2:172.21.15.50:3300/0 conn(0x7f2f680b3870 0x7f2f680b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/429402109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f68158cc0 0x7f2f6806d1d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.841+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 >> 172.21.15.50:0/429402109 conn(0x7f2f68067bb0 msgr2=0x7f2f680cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 shutdown_connections 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/429402109 wait complete. 2023-12-19T14:39:26.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1498154ff0 msgr2=0x7f149806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 --2- 172.21.15.50:0/3189422610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1498154ff0 0x7f149806c7c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f149806a030 tx=0x7f148c00abc0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 shutdown_connections 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 --2- 172.21.15.50:0/3189422610 >> v2:172.21.15.50:3300/0 conn(0x7f149806cd00 0x7f149806a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 --2- 172.21.15.50:0/3189422610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1498154ff0 0x7f149806c7c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 >> 172.21.15.50:0/3189422610 conn(0x7f1498067bb0 msgr2=0x7f14980cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 shutdown_connections 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6034007460 con 0x7f603c1591f0 2023-12-19T14:39:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f149e44c700 1 -- 172.21.15.50:0/3189422610 wait complete. 2023-12-19T14:39:26.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f174144c700 1 Processor -- start 2023-12-19T14:39:26.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f174144c700 1 -- start start 2023-12-19T14:39:26.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f6030ff9700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 0x7f6028062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f2f6cc2e700 1 Processor -- start 2023-12-19T14:39:26.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f2f6cc2e700 1 -- start start 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f174144c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f173c15f2f0 0x7f173c16a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f174144c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f173c159660 0x7f173c0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ca1a0 con 0x7f173c159660 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0c5630 con 0x7f173c15f2f0 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c159f70 con 0x7f173c0c4c70 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54290/0 (socket says 172.21.15.50:54290) 2023-12-19T14:39:26.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 -- 172.21.15.50:0/2328976239 learned_addr learned my addr 172.21.15.50:0/2328976239 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 0x7f2f68161440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 -- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f173c15f2f0 msgr2=0x7f173c16a3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f173c15f2f0 0x7f173c16a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 -- 172.21.15.50:0/2328976239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f173c159660 msgr2=0x7f173c0c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f173c159660 0x7f173c0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 -- 172.21.15.50:0/2328976239 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c169f20 con 0x7f173c0c4c70 2023-12-19T14:39:26.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 0x7f2f6808a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1740c4b700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f173c06b6a0 tx=0x7f172c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f20240df570 con 0x7f20240f5770 2023-12-19T14:39:26.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f149e44c700 1 Processor -- start 2023-12-19T14:39:26.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6034052010 con 0x7f603c1591f0 2023-12-19T14:39:26.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.845+0000 7f6040869700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 0x7f6028062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f603404e3a0 con 0x7f603c1591f0 2023-12-19T14:39:26.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f6040869700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 0x7f6028062ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f60380030c0 tx=0x7f6038016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60198/0 (socket says 172.21.15.50:60198) 2023-12-19T14:39:26.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f66d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 0x7f2f6808a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f66d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 0x7f2f6808a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54296/0 (socket says 172.21.15.50:54296) 2023-12-19T14:39:26.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c00d970 con 0x7f173c0c4c70 2023-12-19T14:39:26.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 -- 172.21.15.50:0/834382967 learned_addr learned my addr 172.21.15.50:0/834382967 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c00dad0 con 0x7f173c0c4c70 2023-12-19T14:39:26.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c00de70 con 0x7f173c0c4c70 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6808ac20 con 0x7f2f6806dc00 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68087a40 con 0x7f2f680b3870 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68161980 con 0x7f2f68158cc0 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f6659c700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 0x7f2f68161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 -- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 msgr2=0x7f2f6808a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 0x7f2f6808a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f2f65d9b700 1 -- 172.21.15.50:0/834382967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 msgr2=0x7f2f68161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c15a1c0 con 0x7f173c0c4c70 2023-12-19T14:39:26.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.849+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c169bb0 con 0x7f173c0c4c70 2023-12-19T14:39:26.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17200052b0 con 0x7f173c0c4c70 2023-12-19T14:39:26.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f65d9b700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 0x7f2f68161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f65d9b700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50009420 con 0x7f2f680b3870 2023-12-19T14:39:26.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f6659c700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 0x7f2f68161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f65d9b700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c560 tx=0x7f2f58001020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005b30 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f58005cb0 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58007320 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f68156e20 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f680873f0 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f68071120 con 0x7f2f680b3870 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f149e44c700 1 -- start start 2023-12-19T14:39:26.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f172c003030 con 0x7f173c0c4c70 2023-12-19T14:39:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.853+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbffc0c2b50 con 0x7fbff40e1ba0 2023-12-19T14:39:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149806cd00 0x7f149813e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 0x7f14981553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f1733fff700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 0x7f171c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f172c07d300 con 0x7f173c0c4c70 2023-12-19T14:39:26.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f173bfff700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 0x7f171c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1498164f10 con 0x7f1498154c20 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1498152240 con 0x7f149806cd00 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149e44c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1498152520 con 0x7f14981582c0 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54306/0 (socket says 172.21.15.50:54306) 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 -- 172.21.15.50:0/3874223562 learned_addr learned my addr 172.21.15.50:0/3874223562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149cc49700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 0x7f14981553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 -- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149806cd00 msgr2=0x7f149813e2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149806cd00 0x7f149813e2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 -- 172.21.15.50:0/3874223562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 msgr2=0x7f14981553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 0x7f14981553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 -- 172.21.15.50:0/3874223562 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980aeb10 con 0x7f14981582c0 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149cc49700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 0x7f14981553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f173bfff700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 0x7f171c062a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f173c06e590 tx=0x7f173401a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f149dc4b700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f148800c880 tx=0x7f1488008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.857+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f172c04d5e0 con 0x7f173c0c4c70 2023-12-19T14:39:26.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.861+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2f58007690 con 0x7f2f680b3870 2023-12-19T14:39:26.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.861+0000 7f2f5e7fc700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 0x7f2f48062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.861+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2f5807bec0 con 0x7f2f680b3870 2023-12-19T14:39:26.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.861+0000 7f2f6659c700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 0x7f2f48062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.861+0000 7f2f6659c700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 0x7f2f48062a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f50003080 tx=0x7f2f50017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488011600 con 0x7f14981582c0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488011760 con 0x7f14981582c0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488011b00 con 0x7f14981582c0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14980aedf0 con 0x7f14981582c0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149813e7e0 con 0x7f14981582c0 2023-12-19T14:39:26.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.869+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1488018360 con 0x7f14981582c0 2023-12-19T14:39:26.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.873+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f5804c1c0 con 0x7f2f680b3870 2023-12-19T14:39:26.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.873+0000 7f14957fa700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:26.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.873+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f148807c7c0 con 0x7f14981582c0 2023-12-19T14:39:26.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.873+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14780052b0 con 0x7f14981582c0 2023-12-19T14:39:26.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.873+0000 7f149d44a700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:26.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.877+0000 7f149d44a700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1490005c50 tx=0x7f1490001620 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:26.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:26.881+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f148801e080 con 0x7f14981582c0 2023-12-19T14:39:27.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.117+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7a6804be40 con 0x7f7a68154a10 2023-12-19T14:39:27.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.121+0000 7f7a567fc700 1 -- 172.21.15.50:0/425764810 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a6004e990 con 0x7f7a68154a10 2023-12-19T14:39:27.122 INFO:teuthology.orchestra.run.smithi050.stdout:55834574858 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.121+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 msgr2=0x7f7a48062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.121+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 0x7f7a48062ad0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009cd0 tx=0x7f7a5c0070e0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.121+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 msgr2=0x7f7a68178fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.121+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a6806cff0 tx=0x7f7a60019000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 shutdown_connections 2023-12-19T14:39:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a680ad6e0 0x7f7a68153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a48060620 0x7f7a48062ad0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a68154a10 0x7f7a68178fc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 --2- 172.21.15.50:0/425764810 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a6806b300 0x7f7a68152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 >> 172.21.15.50:0/425764810 conn(0x7f7a680670e0 msgr2=0x7f7a680ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 shutdown_connections 2023-12-19T14:39:27.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.125+0000 7f7a6cf05700 1 -- 172.21.15.50:0/425764810 wait complete. 2023-12-19T14:39:27.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.2 2023-12-19T14:39:27.156 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f20240df570 con 0x7f20240f5770 2023-12-19T14:39:27.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f2028ff9700 1 -- 172.21.15.50:0/3865306280 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f20240df570 con 0x7f20240f5770 2023-12-19T14:39:27.163 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 msgr2=0x7f2010062a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 0x7f2010062a60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2020001120 tx=0x7f20200015a0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 msgr2=0x7f202403a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f202c065640 tx=0x7f202c06a000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 shutdown_connections 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20100605b0 0x7f2010062a60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20240f5770 0x7f202403a170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20240d24f0 0x7f2024039c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 --2- 172.21.15.50:0/3865306280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201c005230 0x7f20240d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 >> 172.21.15.50:0/3865306280 conn(0x7f202400cd70 msgr2=0x7f2024051ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 shutdown_connections 2023-12-19T14:39:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.161+0000 7f203188e700 1 -- 172.21.15.50:0/3865306280 wait complete. 2023-12-19T14:39:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.177+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0db4071120 con 0x7f0db406baa0 2023-12-19T14:39:27.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.177+0000 7f0dabfff700 1 -- 172.21.15.50:0/3627146037 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0d9c04ff30 con 0x7f0db406baa0 2023-12-19T14:39:27.182 INFO:teuthology.orchestra.run.smithi050.stdout:55834574857 2023-12-19T14:39:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.181+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 msgr2=0x7f0d94062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.181+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 0x7f0d94062b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0db415b1e0 tx=0x7f0da4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.181+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 msgr2=0x7f0db41510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.181+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00b600 tx=0x7f0d9c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 shutdown_connections 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d940606a0 0x7f0d94062b50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0db4158cc0 0x7f0db415a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0db406dc00 0x7f0db4152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 --2- 172.21.15.50:0/3627146037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0db406baa0 0x7f0db41510a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 >> 172.21.15.50:0/3627146037 conn(0x7f0db4067bb0 msgr2=0x7f0db40b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 shutdown_connections 2023-12-19T14:39:27.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7f0db9e6e700 1 -- 172.21.15.50:0/3627146037 wait complete. 2023-12-19T14:39:27.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbff4005530 con 0x7fbff40e1ba0 2023-12-19T14:39:27.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.185+0000 7fbff8ff9700 1 -- 172.21.15.50:0/3597022389 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbffc0c6170 con 0x7fbff40e1ba0 2023-12-19T14:39:27.188 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:27.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1720005830 con 0x7f173c0c4c70 2023-12-19T14:39:27.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 msgr2=0x7fbfe0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 0x7fbfe0062a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000e50 tx=0x7fbff0001d80 comp rx=0 tx=0).stop 2023-12-19T14:39:27.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 msgr2=0x7fbff4036560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 0x7fbff4036560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbffc060be0 tx=0x7fbffc066cd0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.192 INFO:teuthology.orchestra.run.smithi050.stdout:55834574857 2023-12-19T14:39:27.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f1733fff700 1 -- 172.21.15.50:0/2328976239 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f172c050c00 con 0x7f173c0c4c70 2023-12-19T14:39:27.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.0 2023-12-19T14:39:27.193 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 shutdown_connections 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfe0060550 0x7fbfe0062a00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff40e1ba0 0x7fbff4036560 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff404b9e0 0x7fbff404a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 --2- 172.21.15.50:0/3597022389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfec004810 0x7fbff4034a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 >> 172.21.15.50:0/3597022389 conn(0x7fbff4054e30 msgr2=0x7fbff4006c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 shutdown_connections 2023-12-19T14:39:27.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7fc002e78700 1 -- 172.21.15.50:0/3597022389 wait complete. 2023-12-19T14:39:27.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 msgr2=0x7f171c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 0x7f171c062a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f173c06e590 tx=0x7f173401a000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 msgr2=0x7f173c0c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f173c06b6a0 tx=0x7f172c009040 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 shutdown_connections 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f171c060580 0x7f171c062a30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f173c0c4c70 0x7f173c0c50f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f173c159660 0x7f173c0c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 --2- 172.21.15.50:0/2328976239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f173c15f2f0 0x7f173c16a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.189+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 >> 172.21.15.50:0/2328976239 conn(0x7f173c067bb0 msgr2=0x7f173c0cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.193+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 shutdown_connections 2023-12-19T14:39:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.193+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f603c0c6e70 con 0x7f603c1591f0 2023-12-19T14:39:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.193+0000 7f174144c700 1 -- 172.21.15.50:0/2328976239 wait complete. 2023-12-19T14:39:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.193+0000 7f6030ff9700 1 -- 172.21.15.50:0/3679358889 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f603c0c6e70 con 0x7f603c1591f0 2023-12-19T14:39:27.196 INFO:teuthology.orchestra.run.smithi050.stdout:55834574857 2023-12-19T14:39:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.193+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2f6815de60 con 0x7f2f680b3870 2023-12-19T14:39:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f5e7fc700 1 -- 172.21.15.50:0/834382967 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f5804f7e0 con 0x7f2f680b3870 2023-12-19T14:39:27.198 INFO:teuthology.orchestra.run.smithi050.stdout:55834574856 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 msgr2=0x7f6028062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 0x7f6028062ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f60380030c0 tx=0x7f6038016000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 msgr2=0x7f603c159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f603400ccc0 tx=0x7f6034001a10 comp rx=0 tx=0).stop 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 shutdown_connections 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f603c1591f0 0x7f603c159b30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6028060620 0x7f6028062ad0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f603c158e20 0x7f603c1595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 --2- 172.21.15.50:0/3679358889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f603c0ce810 0x7f603c151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 >> 172.21.15.50:0/3679358889 conn(0x7f603c067bb0 msgr2=0x7f603c15ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 shutdown_connections 2023-12-19T14:39:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f604186b700 1 -- 172.21.15.50:0/3679358889 wait complete. 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 msgr2=0x7f2f48062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 0x7f2f48062a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f50003080 tx=0x7f2f50017000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 msgr2=0x7f2f681568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c560 tx=0x7f2f58001020 comp rx=0 tx=0).stop 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 shutdown_connections 2023-12-19T14:39:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f680b3870 0x7f2f681568e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f48060550 0x7f2f48062a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f68158cc0 0x7f2f6808a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 --2- 172.21.15.50:0/834382967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6806dc00 0x7f2f68161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 >> 172.21.15.50:0/834382967 conn(0x7f2f68067bb0 msgr2=0x7f2f680a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 shutdown_connections 2023-12-19T14:39:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.197+0000 7f2f6cc2e700 1 -- 172.21.15.50:0/834382967 wait complete. 2023-12-19T14:39:27.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.7 2023-12-19T14:39:27.208 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.213+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1478005560 con 0x7f14981582c0 2023-12-19T14:39:27.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f14957fa700 1 -- 172.21.15.50:0/3874223562 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1488015020 con 0x7f14981582c0 2023-12-19T14:39:27.218 INFO:teuthology.orchestra.run.smithi050.stdout:55834574857 2023-12-19T14:39:27.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.1 2023-12-19T14:39:27.220 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.3 2023-12-19T14:39:27.220 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 msgr2=0x7f1484062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1490005c50 tx=0x7f1490001620 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 msgr2=0x7f14980af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f148800c880 tx=0x7f1488008040 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 shutdown_connections 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14981582c0 0x7f14980af400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1498154c20 0x7f14981553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 --2- 172.21.15.50:0/3874223562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149806cd00 0x7f149813e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 >> 172.21.15.50:0/3874223562 conn(0x7f1498067bb0 msgr2=0x7f149806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.217+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 shutdown_connections 2023-12-19T14:39:27.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.221+0000 7f149e44c700 1 -- 172.21.15.50:0/3874223562 wait complete. 2023-12-19T14:39:27.223 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.4 2023-12-19T14:39:27.224 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.5 2023-12-19T14:39:27.224 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.238 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.6 2023-12-19T14:39:27.238 DEBUG:teuthology.parallel:result is None 2023-12-19T14:39:27.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-19T14:39:27.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:39:27.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 Processor -- start 2023-12-19T14:39:27.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 -- start start 2023-12-19T14:39:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10d4140ae0 0x7f10d40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d406e150 con 0x7f10d40c75e0 2023-12-19T14:39:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d406d540 con 0x7f10d4140ae0 2023-12-19T14:39:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d9945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d406d820 con 0x7f10d4140710 2023-12-19T14:39:27.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10d4140ae0 0x7f10d40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10d4140ae0 0x7f10d40680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59102/0 (socket says 172.21.15.50:59102) 2023-12-19T14:39:27.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 -- 172.21.15.50:0/3912893861 learned_addr learned my addr 172.21.15.50:0/3912893861 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:27.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d2ffd700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 -- 172.21.15.50:0/3912893861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 msgr2=0x7f10d40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d40c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 -- 172.21.15.50:0/3912893861 >> v1:172.21.15.50:6789/0 conn(0x7f10d40c75e0 legacy=0x7f10d406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.437+0000 7f10d27fc700 1 -- 172.21.15.50:0/3912893861 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d406db00 con 0x7f10d4140ae0 2023-12-19T14:39:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d27fc700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d4140ae0 0x7f10d40680f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10c80099b0 tx=0x7f10c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cf99c671b3c57a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d1ffb700 1 -- 172.21.15.50:0/3912893861 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c800b860 con 0x7f10d4140ae0 2023-12-19T14:39:27.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d1ffb700 1 -- 172.21.15.50:0/3912893861 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c800b9c0 con 0x7f10d4140ae0 2023-12-19T14:39:27.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d1ffb700 1 -- 172.21.15.50:0/3912893861 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c800bd60 con 0x7f10d4140ae0 2023-12-19T14:39:27.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 -- 172.21.15.50:0/3912893861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d4140ae0 msgr2=0x7f10d40680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d4140ae0 0x7f10d40680f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10c80099b0 tx=0x7f10c8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 -- 172.21.15.50:0/3912893861 shutdown_connections 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d4140ae0 0x7f10d40680f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 --2- 172.21.15.50:0/3912893861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 -- 172.21.15.50:0/3912893861 >> 172.21.15.50:0/3912893861 conn(0x7f10d40c8850 msgr2=0x7f10d40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 -- 172.21.15.50:0/3912893861 shutdown_connections 2023-12-19T14:39:27.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.441+0000 7f10d9945700 1 -- 172.21.15.50:0/3912893861 wait complete. 2023-12-19T14:39:27.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 Processor -- start 2023-12-19T14:39:27.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 -- start start 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d4159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d4140ae0 0x7f10d4157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40aa930 con 0x7f10d40c75e0 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40a9d20 con 0x7f10d4140ae0 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d40aa000 con 0x7f10d4140710 2023-12-19T14:39:27.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59106/0 (socket says 172.21.15.50:59106) 2023-12-19T14:39:27.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 -- 172.21.15.50:0/1145577882 learned_addr learned my addr 172.21.15.50:0/1145577882 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:27.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d27fc700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d4159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d37fe700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d4140ae0 0x7f10d4157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 -- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 msgr2=0x7f10d4159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d4159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 -- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d4140ae0 msgr2=0x7f10d4157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d4140ae0 0x7f10d4157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 -- 172.21.15.50:0/1145577882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c80094f0 con 0x7f10d40c75e0 2023-12-19T14:39:27.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d2ffd700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d890 tx=0x7f10c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c40125f0 con 0x7f10d40c75e0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c4012770 con 0x7f10d40c75e0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4012b30 con 0x7f10d40c75e0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.445+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d40aa2e0 con 0x7f10d40c75e0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.449+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d415a480 con 0x7f10d40c75e0 2023-12-19T14:39:27.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.449+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d415a670 con 0x7f10d40c75e0 2023-12-19T14:39:27.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.449+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f10c401e460 con 0x7f10d40c75e0 2023-12-19T14:39:27.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.453+0000 7f10baffd700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 0x7f10c0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.453+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f10c407d1e0 con 0x7f10d40c75e0 2023-12-19T14:39:27.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.453+0000 7f10d27fc700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 0x7f10c0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.453+0000 7f10d27fc700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 0x7f10c0062a00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10c80099b0 tx=0x7f10c8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.457+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10c404d4e0 con 0x7f10d40c75e0 2023-12-19T14:39:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.625+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10d40aaa70 con 0x7f10c0060550 2023-12-19T14:39:27.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.649+0000 7f10baffd700 1 -- 172.21.15.50:0/1145577882 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345081 (secure 0 0 0) 0x7f10c800bb30 con 0x7f10c0060550 2023-12-19T14:39:27.657 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 msgr2=0x7f10c0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 0x7f10c0062a00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10c80099b0 tx=0x7f10c8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 msgr2=0x7f10d40aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d890 tx=0x7f10c4008040 comp rx=0 tx=0).stop 2023-12-19T14:39:27.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 shutdown_connections 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d4140ae0 0x7f10d4157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10c0060550 0x7f10c0062a00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10d4140710 0x7f10d4159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 --2- 172.21.15.50:0/1145577882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10d40c75e0 0x7f10d40aadc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 >> 172.21.15.50:0/1145577882 conn(0x7f10d40c8850 msgr2=0x7f10d40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 shutdown_connections 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.657+0000 7f10d9945700 1 -- 172.21.15.50:0/1145577882 wait complete. 2023-12-19T14:39:27.661 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:39:27.681 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-19T14:39:26.134085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2393792,"kb_used_data":3200,"kb_used_omap":0,"kb_used_meta":2390528,"kb_avail":752580928,"statfs":{"total":773094113280,"available":770642870272,"internally_reserved":0,"allocated":3276800,"data_stored":1983584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2447900672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004506"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562203+0000","last_change":"2023-12-19T14:39:06.261176+0000","last_active":"2023-12-19T14:39:11.562203+0000","last_peered":"2023-12-19T14:39:11.562203+0000","last_clean":"2023-12-19T14:39:11.562203+0000","last_became_active":"2023-12-19T14:39:06.259857+0000","last_became_peered":"2023-12-19T14:39:06.259857+0000","last_unstale":"2023-12-19T14:39:11.562203+0000","last_undegraded":"2023-12-19T14:39:11.562203+0000","last_fullsized":"2023-12-19T14:39:11.562203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:32:43.075778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545898+0000","last_change":"2023-12-19T14:39:07.280964+0000","last_active":"2023-12-19T14:39:11.545898+0000","last_peered":"2023-12-19T14:39:11.545898+0000","last_clean":"2023-12-19T14:39:11.545898+0000","last_became_active":"2023-12-19T14:39:07.280410+0000","last_became_peered":"2023-12-19T14:39:07.280410+0000","last_unstale":"2023-12-19T14:39:11.545898+0000","last_undegraded":"2023-12-19T14:39:11.545898+0000","last_fullsized":"2023-12-19T14:39:11.545898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588669+0000","last_change":"2023-12-19T14:39:11.580119+0000","last_active":"2023-12-19T14:39:11.588669+0000","last_peered":"2023-12-19T14:39:11.588669+0000","last_clean":"2023-12-19T14:39:11.588669+0000","last_became_active":"2023-12-19T14:39:11.579405+0000","last_became_peered":"2023-12-19T14:39:11.579405+0000","last_unstale":"2023-12-19T14:39:11.588669+0000","last_undegraded":"2023-12-19T14:39:11.588669+0000","last_fullsized":"2023-12-19T14:39:11.588669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546035+0000","last_change":"2023-12-19T14:39:06.252186+0000","last_active":"2023-12-19T14:39:11.546035+0000","last_peered":"2023-12-19T14:39:11.546035+0000","last_clean":"2023-12-19T14:39:11.546035+0000","last_became_active":"2023-12-19T14:39:06.251869+0000","last_became_peered":"2023-12-19T14:39:06.251869+0000","last_unstale":"2023-12-19T14:39:11.546035+0000","last_undegraded":"2023-12-19T14:39:11.546035+0000","last_fullsized":"2023-12-19T14:39:11.546035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561557+0000","last_change":"2023-12-19T14:39:07.289142+0000","last_active":"2023-12-19T14:39:11.561557+0000","last_peered":"2023-12-19T14:39:11.561557+0000","last_clean":"2023-12-19T14:39:11.561557+0000","last_became_active":"2023-12-19T14:39:07.288822+0000","last_became_peered":"2023-12-19T14:39:07.288822+0000","last_unstale":"2023-12-19T14:39:11.561557+0000","last_undegraded":"2023-12-19T14:39:11.561557+0000","last_fullsized":"2023-12-19T14:39:11.561557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582458+0000","last_change":"2023-12-19T14:39:11.573848+0000","last_active":"2023-12-19T14:39:11.582458+0000","last_peered":"2023-12-19T14:39:11.582458+0000","last_clean":"2023-12-19T14:39:11.582458+0000","last_became_active":"2023-12-19T14:39:11.573088+0000","last_became_peered":"2023-12-19T14:39:11.573088+0000","last_unstale":"2023-12-19T14:39:11.582458+0000","last_undegraded":"2023-12-19T14:39:11.582458+0000","last_fullsized":"2023-12-19T14:39:11.582458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:29.985849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570627+0000","last_change":"2023-12-19T14:39:06.262908+0000","last_active":"2023-12-19T14:39:11.570627+0000","last_peered":"2023-12-19T14:39:11.570627+0000","last_clean":"2023-12-19T14:39:11.570627+0000","last_became_active":"2023-12-19T14:39:06.260114+0000","last_became_peered":"2023-12-19T14:39:06.260114+0000","last_unstale":"2023-12-19T14:39:11.570627+0000","last_undegraded":"2023-12-19T14:39:11.570627+0000","last_fullsized":"2023-12-19T14:39:11.570627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545839+0000","last_change":"2023-12-19T14:39:07.279790+0000","last_active":"2023-12-19T14:39:11.545839+0000","last_peered":"2023-12-19T14:39:11.545839+0000","last_clean":"2023-12-19T14:39:11.545839+0000","last_became_active":"2023-12-19T14:39:07.279497+0000","last_became_peered":"2023-12-19T14:39:07.279497+0000","last_unstale":"2023-12-19T14:39:11.545839+0000","last_undegraded":"2023-12-19T14:39:11.545839+0000","last_fullsized":"2023-12-19T14:39:11.545839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.574367+0000","last_change":"2023-12-19T14:39:11.571054+0000","last_active":"2023-12-19T14:39:11.574367+0000","last_peered":"2023-12-19T14:39:11.574367+0000","last_clean":"2023-12-19T14:39:11.574367+0000","last_became_active":"2023-12-19T14:39:11.570620+0000","last_became_peered":"2023-12-19T14:39:11.570620+0000","last_unstale":"2023-12-19T14:39:11.574367+0000","last_undegraded":"2023-12-19T14:39:11.574367+0000","last_fullsized":"2023-12-19T14:39:11.574367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:06:34.827613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570831+0000","last_change":"2023-12-19T14:39:06.260028+0000","last_active":"2023-12-19T14:39:11.570831+0000","last_peered":"2023-12-19T14:39:11.570831+0000","last_clean":"2023-12-19T14:39:11.570831+0000","last_became_active":"2023-12-19T14:39:06.259269+0000","last_became_peered":"2023-12-19T14:39:06.259269+0000","last_unstale":"2023-12-19T14:39:11.570831+0000","last_undegraded":"2023-12-19T14:39:11.570831+0000","last_fullsized":"2023-12-19T14:39:11.570831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:29.156364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548594+0000","last_change":"2023-12-19T14:39:07.282662+0000","last_active":"2023-12-19T14:39:11.548594+0000","last_peered":"2023-12-19T14:39:11.548594+0000","last_clean":"2023-12-19T14:39:11.548594+0000","last_became_active":"2023-12-19T14:39:07.282105+0000","last_became_peered":"2023-12-19T14:39:07.282105+0000","last_unstale":"2023-12-19T14:39:11.548594+0000","last_undegraded":"2023-12-19T14:39:11.548594+0000","last_fullsized":"2023-12-19T14:39:11.548594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583605+0000","last_change":"2023-12-19T14:39:11.578372+0000","last_active":"2023-12-19T14:39:11.583605+0000","last_peered":"2023-12-19T14:39:11.583605+0000","last_clean":"2023-12-19T14:39:11.583605+0000","last_became_active":"2023-12-19T14:39:11.575437+0000","last_became_peered":"2023-12-19T14:39:11.575437+0000","last_unstale":"2023-12-19T14:39:11.583605+0000","last_undegraded":"2023-12-19T14:39:11.583605+0000","last_fullsized":"2023-12-19T14:39:11.583605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:01.783711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550051+0000","last_change":"2023-12-19T14:39:07.274147+0000","last_active":"2023-12-19T14:39:11.550051+0000","last_peered":"2023-12-19T14:39:11.550051+0000","last_clean":"2023-12-19T14:39:11.550051+0000","last_became_active":"2023-12-19T14:39:07.273776+0000","last_became_peered":"2023-12-19T14:39:07.273776+0000","last_unstale":"2023-12-19T14:39:11.550051+0000","last_undegraded":"2023-12-19T14:39:11.550051+0000","last_fullsized":"2023-12-19T14:39:11.550051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570793+0000","last_change":"2023-12-19T14:39:06.253750+0000","last_active":"2023-12-19T14:39:11.570793+0000","last_peered":"2023-12-19T14:39:11.570793+0000","last_clean":"2023-12-19T14:39:11.570793+0000","last_became_active":"2023-12-19T14:39:06.253479+0000","last_became_peered":"2023-12-19T14:39:06.253479+0000","last_unstale":"2023-12-19T14:39:11.570793+0000","last_undegraded":"2023-12-19T14:39:11.570793+0000","last_fullsized":"2023-12-19T14:39:11.570793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:22.901851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564091+0000","last_change":"2023-12-19T14:39:11.562226+0000","last_active":"2023-12-19T14:39:11.564091+0000","last_peered":"2023-12-19T14:39:11.564091+0000","last_clean":"2023-12-19T14:39:11.564091+0000","last_became_active":"2023-12-19T14:39:11.561874+0000","last_became_peered":"2023-12-19T14:39:11.561874+0000","last_unstale":"2023-12-19T14:39:11.564091+0000","last_undegraded":"2023-12-19T14:39:11.564091+0000","last_fullsized":"2023-12-19T14:39:11.564091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543918+0000","last_change":"2023-12-19T14:39:07.297728+0000","last_active":"2023-12-19T14:39:11.543918+0000","last_peered":"2023-12-19T14:39:11.543918+0000","last_clean":"2023-12-19T14:39:11.543918+0000","last_became_active":"2023-12-19T14:39:07.297419+0000","last_became_peered":"2023-12-19T14:39:07.297419+0000","last_unstale":"2023-12-19T14:39:11.543918+0000","last_undegraded":"2023-12-19T14:39:11.543918+0000","last_fullsized":"2023-12-19T14:39:11.543918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540603+0000","last_change":"2023-12-19T14:39:06.250154+0000","last_active":"2023-12-19T14:39:11.540603+0000","last_peered":"2023-12-19T14:39:11.540603+0000","last_clean":"2023-12-19T14:39:11.540603+0000","last_became_active":"2023-12-19T14:39:06.249851+0000","last_became_peered":"2023-12-19T14:39:06.249851+0000","last_unstale":"2023-12-19T14:39:11.540603+0000","last_undegraded":"2023-12-19T14:39:11.540603+0000","last_fullsized":"2023-12-19T14:39:11.540603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583288+0000","last_change":"2023-12-19T14:39:11.576656+0000","last_active":"2023-12-19T14:39:11.583288+0000","last_peered":"2023-12-19T14:39:11.583288+0000","last_clean":"2023-12-19T14:39:11.583288+0000","last_became_active":"2023-12-19T14:39:11.576165+0000","last_became_peered":"2023-12-19T14:39:11.576165+0000","last_unstale":"2023-12-19T14:39:11.583288+0000","last_undegraded":"2023-12-19T14:39:11.583288+0000","last_fullsized":"2023-12-19T14:39:11.583288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545215+0000","last_change":"2023-12-19T14:39:07.274124+0000","last_active":"2023-12-19T14:39:11.545215+0000","last_peered":"2023-12-19T14:39:11.545215+0000","last_clean":"2023-12-19T14:39:11.545215+0000","last_became_active":"2023-12-19T14:39:07.273792+0000","last_became_peered":"2023-12-19T14:39:07.273792+0000","last_unstale":"2023-12-19T14:39:11.545215+0000","last_undegraded":"2023-12-19T14:39:11.545215+0000","last_fullsized":"2023-12-19T14:39:11.545215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540233+0000","last_change":"2023-12-19T14:39:06.244868+0000","last_active":"2023-12-19T14:39:11.540233+0000","last_peered":"2023-12-19T14:39:11.540233+0000","last_clean":"2023-12-19T14:39:11.540233+0000","last_became_active":"2023-12-19T14:39:06.244567+0000","last_became_peered":"2023-12-19T14:39:06.244567+0000","last_unstale":"2023-12-19T14:39:11.540233+0000","last_undegraded":"2023-12-19T14:39:11.540233+0000","last_fullsized":"2023-12-19T14:39:11.540233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.587736+0000","last_change":"2023-12-19T14:39:11.585814+0000","last_active":"2023-12-19T14:39:11.587736+0000","last_peered":"2023-12-19T14:39:11.587736+0000","last_clean":"2023-12-19T14:39:11.587736+0000","last_became_active":"2023-12-19T14:39:11.581875+0000","last_became_peered":"2023-12-19T14:39:11.581875+0000","last_unstale":"2023-12-19T14:39:11.587736+0000","last_undegraded":"2023-12-19T14:39:11.587736+0000","last_fullsized":"2023-12-19T14:39:11.587736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547078+0000","last_change":"2023-12-19T14:39:07.276433+0000","last_active":"2023-12-19T14:39:11.547078+0000","last_peered":"2023-12-19T14:39:11.547078+0000","last_clean":"2023-12-19T14:39:11.547078+0000","last_became_active":"2023-12-19T14:39:07.274815+0000","last_became_peered":"2023-12-19T14:39:07.274815+0000","last_unstale":"2023-12-19T14:39:11.547078+0000","last_undegraded":"2023-12-19T14:39:11.547078+0000","last_fullsized":"2023-12-19T14:39:11.547078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:57.360165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569740+0000","last_change":"2023-12-19T14:39:06.262949+0000","last_active":"2023-12-19T14:39:11.569740+0000","last_peered":"2023-12-19T14:39:11.569740+0000","last_clean":"2023-12-19T14:39:11.569740+0000","last_became_active":"2023-12-19T14:39:06.262630+0000","last_became_peered":"2023-12-19T14:39:06.262630+0000","last_unstale":"2023-12-19T14:39:11.569740+0000","last_undegraded":"2023-12-19T14:39:11.569740+0000","last_fullsized":"2023-12-19T14:39:11.569740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580388+0000","last_change":"2023-12-19T14:39:11.572679+0000","last_active":"2023-12-19T14:39:11.580388+0000","last_peered":"2023-12-19T14:39:11.580388+0000","last_clean":"2023-12-19T14:39:11.580388+0000","last_became_active":"2023-12-19T14:39:11.572334+0000","last_became_peered":"2023-12-19T14:39:11.572334+0000","last_unstale":"2023-12-19T14:39:11.580388+0000","last_undegraded":"2023-12-19T14:39:11.580388+0000","last_fullsized":"2023-12-19T14:39:11.580388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:00:02.733099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562034+0000","last_change":"2023-12-19T14:39:07.287397+0000","last_active":"2023-12-19T14:39:11.562034+0000","last_peered":"2023-12-19T14:39:11.562034+0000","last_clean":"2023-12-19T14:39:11.562034+0000","last_became_active":"2023-12-19T14:39:07.287079+0000","last_became_peered":"2023-12-19T14:39:07.287079+0000","last_unstale":"2023-12-19T14:39:11.562034+0000","last_undegraded":"2023-12-19T14:39:11.562034+0000","last_fullsized":"2023-12-19T14:39:11.562034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551941+0000","last_change":"2023-12-19T14:39:06.259853+0000","last_active":"2023-12-19T14:39:11.551941+0000","last_peered":"2023-12-19T14:39:11.551941+0000","last_clean":"2023-12-19T14:39:11.551941+0000","last_became_active":"2023-12-19T14:39:06.259573+0000","last_became_peered":"2023-12-19T14:39:06.259573+0000","last_unstale":"2023-12-19T14:39:11.551941+0000","last_undegraded":"2023-12-19T14:39:11.551941+0000","last_fullsized":"2023-12-19T14:39:11.551941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583774+0000","last_change":"2023-12-19T14:39:11.578284+0000","last_active":"2023-12-19T14:39:11.583774+0000","last_peered":"2023-12-19T14:39:11.583774+0000","last_clean":"2023-12-19T14:39:11.583774+0000","last_became_active":"2023-12-19T14:39:11.577805+0000","last_became_peered":"2023-12-19T14:39:11.577805+0000","last_unstale":"2023-12-19T14:39:11.583774+0000","last_undegraded":"2023-12-19T14:39:11.583774+0000","last_fullsized":"2023-12-19T14:39:11.583774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564518+0000","last_change":"2023-12-19T14:39:07.288511+0000","last_active":"2023-12-19T14:39:11.564518+0000","last_peered":"2023-12-19T14:39:11.564518+0000","last_clean":"2023-12-19T14:39:11.564518+0000","last_became_active":"2023-12-19T14:39:07.288221+0000","last_became_peered":"2023-12-19T14:39:07.288221+0000","last_unstale":"2023-12-19T14:39:11.564518+0000","last_undegraded":"2023-12-19T14:39:11.564518+0000","last_fullsized":"2023-12-19T14:39:11.564518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.728724+0000","last_change":"2023-12-19T14:39:06.260565+0000","last_active":"2023-12-19T14:39:11.728724+0000","last_peered":"2023-12-19T14:39:11.728724+0000","last_clean":"2023-12-19T14:39:11.728724+0000","last_became_active":"2023-12-19T14:39:06.260244+0000","last_became_peered":"2023-12-19T14:39:06.260244+0000","last_unstale":"2023-12-19T14:39:11.728724+0000","last_undegraded":"2023-12-19T14:39:11.728724+0000","last_fullsized":"2023-12-19T14:39:11.728724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588612+0000","last_change":"2023-12-19T14:39:11.583334+0000","last_active":"2023-12-19T14:39:11.588612+0000","last_peered":"2023-12-19T14:39:11.588612+0000","last_clean":"2023-12-19T14:39:11.588612+0000","last_became_active":"2023-12-19T14:39:11.582906+0000","last_became_peered":"2023-12-19T14:39:11.582906+0000","last_unstale":"2023-12-19T14:39:11.588612+0000","last_undegraded":"2023-12-19T14:39:11.588612+0000","last_fullsized":"2023-12-19T14:39:11.588612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:50.106844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540825+0000","last_change":"2023-12-19T14:39:07.267435+0000","last_active":"2023-12-19T14:39:11.540825+0000","last_peered":"2023-12-19T14:39:11.540825+0000","last_clean":"2023-12-19T14:39:11.540825+0000","last_became_active":"2023-12-19T14:39:07.267113+0000","last_became_peered":"2023-12-19T14:39:07.267113+0000","last_unstale":"2023-12-19T14:39:11.540825+0000","last_undegraded":"2023-12-19T14:39:11.540825+0000","last_fullsized":"2023-12-19T14:39:11.540825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570841+0000","last_change":"2023-12-19T14:39:06.262808+0000","last_active":"2023-12-19T14:39:11.570841+0000","last_peered":"2023-12-19T14:39:11.570841+0000","last_clean":"2023-12-19T14:39:11.570841+0000","last_became_active":"2023-12-19T14:39:06.262439+0000","last_became_peered":"2023-12-19T14:39:06.262439+0000","last_unstale":"2023-12-19T14:39:11.570841+0000","last_undegraded":"2023-12-19T14:39:11.570841+0000","last_fullsized":"2023-12-19T14:39:11.570841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583520+0000","last_change":"2023-12-19T14:39:11.575768+0000","last_active":"2023-12-19T14:39:11.583520+0000","last_peered":"2023-12-19T14:39:11.583520+0000","last_clean":"2023-12-19T14:39:11.583520+0000","last_became_active":"2023-12-19T14:39:11.575222+0000","last_became_peered":"2023-12-19T14:39:11.575222+0000","last_unstale":"2023-12-19T14:39:11.583520+0000","last_undegraded":"2023-12-19T14:39:11.583520+0000","last_fullsized":"2023-12-19T14:39:11.583520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548697+0000","last_change":"2023-12-19T14:39:07.291010+0000","last_active":"2023-12-19T14:39:11.548697+0000","last_peered":"2023-12-19T14:39:11.548697+0000","last_clean":"2023-12-19T14:39:11.548697+0000","last_became_active":"2023-12-19T14:39:07.290719+0000","last_became_peered":"2023-12-19T14:39:07.290719+0000","last_unstale":"2023-12-19T14:39:11.548697+0000","last_undegraded":"2023-12-19T14:39:11.548697+0000","last_fullsized":"2023-12-19T14:39:11.548697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540846+0000","last_change":"2023-12-19T14:39:06.252011+0000","last_active":"2023-12-19T14:39:11.540846+0000","last_peered":"2023-12-19T14:39:11.540846+0000","last_clean":"2023-12-19T14:39:11.540846+0000","last_became_active":"2023-12-19T14:39:06.251724+0000","last_became_peered":"2023-12-19T14:39:06.251724+0000","last_unstale":"2023-12-19T14:39:11.540846+0000","last_undegraded":"2023-12-19T14:39:11.540846+0000","last_fullsized":"2023-12-19T14:39:11.540846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573312+0000","last_change":"2023-12-19T14:39:11.562285+0000","last_active":"2023-12-19T14:39:11.573312+0000","last_peered":"2023-12-19T14:39:11.573312+0000","last_clean":"2023-12-19T14:39:11.573312+0000","last_became_active":"2023-12-19T14:39:11.561826+0000","last_became_peered":"2023-12-19T14:39:11.561826+0000","last_unstale":"2023-12-19T14:39:11.573312+0000","last_undegraded":"2023-12-19T14:39:11.573312+0000","last_fullsized":"2023-12-19T14:39:11.573312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546810+0000","last_change":"2023-12-19T14:39:07.289173+0000","last_active":"2023-12-19T14:39:11.546810+0000","last_peered":"2023-12-19T14:39:11.546810+0000","last_clean":"2023-12-19T14:39:11.546810+0000","last_became_active":"2023-12-19T14:39:07.288745+0000","last_became_peered":"2023-12-19T14:39:07.288745+0000","last_unstale":"2023-12-19T14:39:11.546810+0000","last_undegraded":"2023-12-19T14:39:11.546810+0000","last_fullsized":"2023-12-19T14:39:11.546810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545170+0000","last_change":"2023-12-19T14:39:06.258745+0000","last_active":"2023-12-19T14:39:11.545170+0000","last_peered":"2023-12-19T14:39:11.545170+0000","last_clean":"2023-12-19T14:39:11.545170+0000","last_became_active":"2023-12-19T14:39:06.257007+0000","last_became_peered":"2023-12-19T14:39:06.257007+0000","last_unstale":"2023-12-19T14:39:11.545170+0000","last_undegraded":"2023-12-19T14:39:11.545170+0000","last_fullsized":"2023-12-19T14:39:11.545170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571904+0000","last_change":"2023-12-19T14:39:11.568661+0000","last_active":"2023-12-19T14:39:11.571904+0000","last_peered":"2023-12-19T14:39:11.571904+0000","last_clean":"2023-12-19T14:39:11.571904+0000","last_became_active":"2023-12-19T14:39:11.567940+0000","last_became_peered":"2023-12-19T14:39:11.567940+0000","last_unstale":"2023-12-19T14:39:11.571904+0000","last_undegraded":"2023-12-19T14:39:11.571904+0000","last_fullsized":"2023-12-19T14:39:11.571904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539181+0000","last_change":"2023-12-19T14:39:07.269639+0000","last_active":"2023-12-19T14:39:11.539181+0000","last_peered":"2023-12-19T14:39:11.539181+0000","last_clean":"2023-12-19T14:39:11.539181+0000","last_became_active":"2023-12-19T14:39:07.269276+0000","last_became_peered":"2023-12-19T14:39:07.269276+0000","last_unstale":"2023-12-19T14:39:11.539181+0000","last_undegraded":"2023-12-19T14:39:11.539181+0000","last_fullsized":"2023-12-19T14:39:11.539181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553025+0000","last_change":"2023-12-19T14:39:06.254091+0000","last_active":"2023-12-19T14:39:11.553025+0000","last_peered":"2023-12-19T14:39:11.553025+0000","last_clean":"2023-12-19T14:39:11.553025+0000","last_became_active":"2023-12-19T14:39:06.253444+0000","last_became_peered":"2023-12-19T14:39:06.253444+0000","last_unstale":"2023-12-19T14:39:11.553025+0000","last_undegraded":"2023-12-19T14:39:11.553025+0000","last_fullsized":"2023-12-19T14:39:11.553025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584042+0000","last_change":"2023-12-19T14:39:11.574535+0000","last_active":"2023-12-19T14:39:11.584042+0000","last_peered":"2023-12-19T14:39:11.584042+0000","last_clean":"2023-12-19T14:39:11.584042+0000","last_became_active":"2023-12-19T14:39:11.574060+0000","last_became_peered":"2023-12-19T14:39:11.574060+0000","last_unstale":"2023-12-19T14:39:11.584042+0000","last_undegraded":"2023-12-19T14:39:11.584042+0000","last_fullsized":"2023-12-19T14:39:11.584042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.785232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556888+0000","last_change":"2023-12-19T14:39:07.287233+0000","last_active":"2023-12-19T14:39:11.556888+0000","last_peered":"2023-12-19T14:39:11.556888+0000","last_clean":"2023-12-19T14:39:11.556888+0000","last_became_active":"2023-12-19T14:39:07.286769+0000","last_became_peered":"2023-12-19T14:39:07.286769+0000","last_unstale":"2023-12-19T14:39:11.556888+0000","last_undegraded":"2023-12-19T14:39:11.556888+0000","last_fullsized":"2023-12-19T14:39:11.556888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569853+0000","last_change":"2023-12-19T14:39:06.251774+0000","last_active":"2023-12-19T14:39:11.569853+0000","last_peered":"2023-12-19T14:39:11.569853+0000","last_clean":"2023-12-19T14:39:11.569853+0000","last_became_active":"2023-12-19T14:39:06.251528+0000","last_became_peered":"2023-12-19T14:39:06.251528+0000","last_unstale":"2023-12-19T14:39:11.569853+0000","last_undegraded":"2023-12-19T14:39:11.569853+0000","last_fullsized":"2023-12-19T14:39:11.569853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581699+0000","last_change":"2023-12-19T14:39:11.575777+0000","last_active":"2023-12-19T14:39:11.581699+0000","last_peered":"2023-12-19T14:39:11.581699+0000","last_clean":"2023-12-19T14:39:11.581699+0000","last_became_active":"2023-12-19T14:39:11.575283+0000","last_became_peered":"2023-12-19T14:39:11.575283+0000","last_unstale":"2023-12-19T14:39:11.581699+0000","last_undegraded":"2023-12-19T14:39:11.581699+0000","last_fullsized":"2023-12-19T14:39:11.581699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:54.006908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543534+0000","last_change":"2023-12-19T14:39:07.294894+0000","last_active":"2023-12-19T14:39:11.543534+0000","last_peered":"2023-12-19T14:39:11.543534+0000","last_clean":"2023-12-19T14:39:11.543534+0000","last_became_active":"2023-12-19T14:39:07.294640+0000","last_became_peered":"2023-12-19T14:39:07.294640+0000","last_unstale":"2023-12-19T14:39:11.543534+0000","last_undegraded":"2023-12-19T14:39:11.543534+0000","last_fullsized":"2023-12-19T14:39:11.543534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545405+0000","last_change":"2023-12-19T14:39:06.256452+0000","last_active":"2023-12-19T14:39:11.545405+0000","last_peered":"2023-12-19T14:39:11.545405+0000","last_clean":"2023-12-19T14:39:11.545405+0000","last_became_active":"2023-12-19T14:39:06.255327+0000","last_became_peered":"2023-12-19T14:39:06.255327+0000","last_unstale":"2023-12-19T14:39:11.545405+0000","last_undegraded":"2023-12-19T14:39:11.545405+0000","last_fullsized":"2023-12-19T14:39:11.545405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569956+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:39:11.569956+0000","last_peered":"2023-12-19T14:39:11.569956+0000","last_clean":"2023-12-19T14:39:11.569956+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:39:11.569956+0000","last_undegraded":"2023-12-19T14:39:11.569956+0000","last_fullsized":"2023-12-19T14:39:11.569956+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582739+0000","last_change":"2023-12-19T14:39:11.578205+0000","last_active":"2023-12-19T14:39:11.582739+0000","last_peered":"2023-12-19T14:39:11.582739+0000","last_clean":"2023-12-19T14:39:11.582739+0000","last_became_active":"2023-12-19T14:39:11.577809+0000","last_became_peered":"2023-12-19T14:39:11.577809+0000","last_unstale":"2023-12-19T14:39:11.582739+0000","last_undegraded":"2023-12-19T14:39:11.582739+0000","last_fullsized":"2023-12-19T14:39:11.582739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:10.870899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570477+0000","last_change":"2023-12-19T14:39:07.283187+0000","last_active":"2023-12-19T14:39:11.570477+0000","last_peered":"2023-12-19T14:39:11.570477+0000","last_clean":"2023-12-19T14:39:11.570477+0000","last_became_active":"2023-12-19T14:39:07.282884+0000","last_became_peered":"2023-12-19T14:39:07.282884+0000","last_unstale":"2023-12-19T14:39:11.570477+0000","last_undegraded":"2023-12-19T14:39:11.570477+0000","last_fullsized":"2023-12-19T14:39:11.570477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553094+0000","last_change":"2023-12-19T14:39:06.260560+0000","last_active":"2023-12-19T14:39:11.553094+0000","last_peered":"2023-12-19T14:39:11.553094+0000","last_clean":"2023-12-19T14:39:11.553094+0000","last_became_active":"2023-12-19T14:39:06.260257+0000","last_became_peered":"2023-12-19T14:39:06.260257+0000","last_unstale":"2023-12-19T14:39:11.553094+0000","last_undegraded":"2023-12-19T14:39:11.553094+0000","last_fullsized":"2023-12-19T14:39:11.553094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580038+0000","last_change":"2023-12-19T14:39:11.575041+0000","last_active":"2023-12-19T14:39:11.580038+0000","last_peered":"2023-12-19T14:39:11.580038+0000","last_clean":"2023-12-19T14:39:11.580038+0000","last_became_active":"2023-12-19T14:39:11.574653+0000","last_became_peered":"2023-12-19T14:39:11.574653+0000","last_unstale":"2023-12-19T14:39:11.580038+0000","last_undegraded":"2023-12-19T14:39:11.580038+0000","last_fullsized":"2023-12-19T14:39:11.580038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545946+0000","last_change":"2023-12-19T14:39:07.291614+0000","last_active":"2023-12-19T14:39:11.545946+0000","last_peered":"2023-12-19T14:39:11.545946+0000","last_clean":"2023-12-19T14:39:11.545946+0000","last_became_active":"2023-12-19T14:39:07.291096+0000","last_became_peered":"2023-12-19T14:39:07.291096+0000","last_unstale":"2023-12-19T14:39:11.545946+0000","last_undegraded":"2023-12-19T14:39:11.545946+0000","last_fullsized":"2023-12-19T14:39:11.545946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540658+0000","last_change":"2023-12-19T14:39:06.245861+0000","last_active":"2023-12-19T14:39:11.540658+0000","last_peered":"2023-12-19T14:39:11.540658+0000","last_clean":"2023-12-19T14:39:11.540658+0000","last_became_active":"2023-12-19T14:39:06.245543+0000","last_became_peered":"2023-12-19T14:39:06.245543+0000","last_unstale":"2023-12-19T14:39:11.540658+0000","last_undegraded":"2023-12-19T14:39:11.540658+0000","last_fullsized":"2023-12-19T14:39:11.540658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.575504+0000","last_change":"2023-12-19T14:39:11.572562+0000","last_active":"2023-12-19T14:39:11.575504+0000","last_peered":"2023-12-19T14:39:11.575504+0000","last_clean":"2023-12-19T14:39:11.575504+0000","last_became_active":"2023-12-19T14:39:11.572159+0000","last_became_peered":"2023-12-19T14:39:11.572159+0000","last_unstale":"2023-12-19T14:39:11.575504+0000","last_undegraded":"2023-12-19T14:39:11.575504+0000","last_fullsized":"2023-12-19T14:39:11.575504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571036+0000","last_change":"2023-12-19T14:39:07.284702+0000","last_active":"2023-12-19T14:39:11.571036+0000","last_peered":"2023-12-19T14:39:11.571036+0000","last_clean":"2023-12-19T14:39:11.571036+0000","last_became_active":"2023-12-19T14:39:07.284258+0000","last_became_peered":"2023-12-19T14:39:07.284258+0000","last_unstale":"2023-12-19T14:39:11.571036+0000","last_undegraded":"2023-12-19T14:39:11.571036+0000","last_fullsized":"2023-12-19T14:39:11.571036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555155+0000","last_change":"2023-12-19T14:39:06.256120+0000","last_active":"2023-12-19T14:39:11.555155+0000","last_peered":"2023-12-19T14:39:11.555155+0000","last_clean":"2023-12-19T14:39:11.555155+0000","last_became_active":"2023-12-19T14:39:06.255187+0000","last_became_peered":"2023-12-19T14:39:06.255187+0000","last_unstale":"2023-12-19T14:39:11.555155+0000","last_undegraded":"2023-12-19T14:39:11.555155+0000","last_fullsized":"2023-12-19T14:39:11.555155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584352+0000","last_change":"2023-12-19T14:39:11.579495+0000","last_active":"2023-12-19T14:39:11.584352+0000","last_peered":"2023-12-19T14:39:11.584352+0000","last_clean":"2023-12-19T14:39:11.584352+0000","last_became_active":"2023-12-19T14:39:11.579259+0000","last_became_peered":"2023-12-19T14:39:11.579259+0000","last_unstale":"2023-12-19T14:39:11.584352+0000","last_undegraded":"2023-12-19T14:39:11.584352+0000","last_fullsized":"2023-12-19T14:39:11.584352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573192+0000","last_change":"2023-12-19T14:39:07.279364+0000","last_active":"2023-12-19T14:39:11.573192+0000","last_peered":"2023-12-19T14:39:11.573192+0000","last_clean":"2023-12-19T14:39:11.573192+0000","last_became_active":"2023-12-19T14:39:07.278999+0000","last_became_peered":"2023-12-19T14:39:07.278999+0000","last_unstale":"2023-12-19T14:39:11.573192+0000","last_undegraded":"2023-12-19T14:39:11.573192+0000","last_fullsized":"2023-12-19T14:39:11.573192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547974+0000","last_change":"2023-12-19T14:39:06.262399+0000","last_active":"2023-12-19T14:39:11.547974+0000","last_peered":"2023-12-19T14:39:11.547974+0000","last_clean":"2023-12-19T14:39:11.547974+0000","last_became_active":"2023-12-19T14:39:06.261918+0000","last_became_peered":"2023-12-19T14:39:06.261918+0000","last_unstale":"2023-12-19T14:39:11.547974+0000","last_undegraded":"2023-12-19T14:39:11.547974+0000","last_fullsized":"2023-12-19T14:39:11.547974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:48:08.535588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583755+0000","last_change":"2023-12-19T14:39:11.577720+0000","last_active":"2023-12-19T14:39:11.583755+0000","last_peered":"2023-12-19T14:39:11.583755+0000","last_clean":"2023-12-19T14:39:11.583755+0000","last_became_active":"2023-12-19T14:39:11.577297+0000","last_became_peered":"2023-12-19T14:39:11.577297+0000","last_unstale":"2023-12-19T14:39:11.583755+0000","last_undegraded":"2023-12-19T14:39:11.583755+0000","last_fullsized":"2023-12-19T14:39:11.583755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570176+0000","last_change":"2023-12-19T14:39:07.291311+0000","last_active":"2023-12-19T14:39:11.570176+0000","last_peered":"2023-12-19T14:39:11.570176+0000","last_clean":"2023-12-19T14:39:11.570176+0000","last_became_active":"2023-12-19T14:39:07.290587+0000","last_became_peered":"2023-12-19T14:39:07.290587+0000","last_unstale":"2023-12-19T14:39:11.570176+0000","last_undegraded":"2023-12-19T14:39:11.570176+0000","last_fullsized":"2023-12-19T14:39:11.570176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548490+0000","last_change":"2023-12-19T14:39:06.248952+0000","last_active":"2023-12-19T14:39:11.548490+0000","last_peered":"2023-12-19T14:39:11.548490+0000","last_clean":"2023-12-19T14:39:11.548490+0000","last_became_active":"2023-12-19T14:39:06.248504+0000","last_became_peered":"2023-12-19T14:39:06.248504+0000","last_unstale":"2023-12-19T14:39:11.548490+0000","last_undegraded":"2023-12-19T14:39:11.548490+0000","last_fullsized":"2023-12-19T14:39:11.548490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581722+0000","last_change":"2023-12-19T14:39:11.579028+0000","last_active":"2023-12-19T14:39:11.581722+0000","last_peered":"2023-12-19T14:39:11.581722+0000","last_clean":"2023-12-19T14:39:11.581722+0000","last_became_active":"2023-12-19T14:39:11.578386+0000","last_became_peered":"2023-12-19T14:39:11.578386+0000","last_unstale":"2023-12-19T14:39:11.581722+0000","last_undegraded":"2023-12-19T14:39:11.581722+0000","last_fullsized":"2023-12-19T14:39:11.581722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551739+0000","last_change":"2023-12-19T14:39:07.289550+0000","last_active":"2023-12-19T14:39:11.551739+0000","last_peered":"2023-12-19T14:39:11.551739+0000","last_clean":"2023-12-19T14:39:11.551739+0000","last_became_active":"2023-12-19T14:39:07.289242+0000","last_became_peered":"2023-12-19T14:39:07.289242+0000","last_unstale":"2023-12-19T14:39:11.551739+0000","last_undegraded":"2023-12-19T14:39:11.551739+0000","last_fullsized":"2023-12-19T14:39:11.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549347+0000","last_change":"2023-12-19T14:39:06.250752+0000","last_active":"2023-12-19T14:39:11.549347+0000","last_peered":"2023-12-19T14:39:11.549347+0000","last_clean":"2023-12-19T14:39:11.549347+0000","last_became_active":"2023-12-19T14:39:06.249438+0000","last_became_peered":"2023-12-19T14:39:06.249438+0000","last_unstale":"2023-12-19T14:39:11.549347+0000","last_undegraded":"2023-12-19T14:39:11.549347+0000","last_fullsized":"2023-12-19T14:39:11.549347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588513+0000","last_change":"2023-12-19T14:39:11.581267+0000","last_active":"2023-12-19T14:39:11.588513+0000","last_peered":"2023-12-19T14:39:11.588513+0000","last_clean":"2023-12-19T14:39:11.588513+0000","last_became_active":"2023-12-19T14:39:11.577498+0000","last_became_peered":"2023-12-19T14:39:11.577498+0000","last_unstale":"2023-12-19T14:39:11.588513+0000","last_undegraded":"2023-12-19T14:39:11.588513+0000","last_fullsized":"2023-12-19T14:39:11.588513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:16:19.999833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570611+0000","last_change":"2023-12-19T14:39:07.283321+0000","last_active":"2023-12-19T14:39:11.570611+0000","last_peered":"2023-12-19T14:39:11.570611+0000","last_clean":"2023-12-19T14:39:11.570611+0000","last_became_active":"2023-12-19T14:39:07.283017+0000","last_became_peered":"2023-12-19T14:39:07.283017+0000","last_unstale":"2023-12-19T14:39:11.570611+0000","last_undegraded":"2023-12-19T14:39:11.570611+0000","last_fullsized":"2023-12-19T14:39:11.570611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545060+0000","last_change":"2023-12-19T14:39:06.259282+0000","last_active":"2023-12-19T14:39:11.545060+0000","last_peered":"2023-12-19T14:39:11.545060+0000","last_clean":"2023-12-19T14:39:11.545060+0000","last_became_active":"2023-12-19T14:39:06.257316+0000","last_became_peered":"2023-12-19T14:39:06.257316+0000","last_unstale":"2023-12-19T14:39:11.545060+0000","last_undegraded":"2023-12-19T14:39:11.545060+0000","last_fullsized":"2023-12-19T14:39:11.545060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588628+0000","last_change":"2023-12-19T14:39:11.577972+0000","last_active":"2023-12-19T14:39:11.588628+0000","last_peered":"2023-12-19T14:39:11.588628+0000","last_clean":"2023-12-19T14:39:11.588628+0000","last_became_active":"2023-12-19T14:39:11.577529+0000","last_became_peered":"2023-12-19T14:39:11.577529+0000","last_unstale":"2023-12-19T14:39:11.588628+0000","last_undegraded":"2023-12-19T14:39:11.588628+0000","last_fullsized":"2023-12-19T14:39:11.588628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:19:59.360333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.557043+0000","last_change":"2023-12-19T14:39:07.289405+0000","last_active":"2023-12-19T14:39:11.557043+0000","last_peered":"2023-12-19T14:39:11.557043+0000","last_clean":"2023-12-19T14:39:11.557043+0000","last_became_active":"2023-12-19T14:39:07.288885+0000","last_became_peered":"2023-12-19T14:39:07.288885+0000","last_unstale":"2023-12-19T14:39:11.557043+0000","last_undegraded":"2023-12-19T14:39:11.557043+0000","last_fullsized":"2023-12-19T14:39:11.557043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559942+0000","last_change":"2023-12-19T14:39:06.256658+0000","last_active":"2023-12-19T14:39:11.559942+0000","last_peered":"2023-12-19T14:39:11.559942+0000","last_clean":"2023-12-19T14:39:11.559942+0000","last_became_active":"2023-12-19T14:39:06.255561+0000","last_became_peered":"2023-12-19T14:39:06.255561+0000","last_unstale":"2023-12-19T14:39:11.559942+0000","last_undegraded":"2023-12-19T14:39:11.559942+0000","last_fullsized":"2023-12-19T14:39:11.559942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584461+0000","last_change":"2023-12-19T14:39:11.579961+0000","last_active":"2023-12-19T14:39:11.584461+0000","last_peered":"2023-12-19T14:39:11.584461+0000","last_clean":"2023-12-19T14:39:11.584461+0000","last_became_active":"2023-12-19T14:39:11.578069+0000","last_became_peered":"2023-12-19T14:39:11.578069+0000","last_unstale":"2023-12-19T14:39:11.584461+0000","last_undegraded":"2023-12-19T14:39:11.584461+0000","last_fullsized":"2023-12-19T14:39:11.584461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570370+0000","last_change":"2023-12-19T14:39:07.288175+0000","last_active":"2023-12-19T14:39:11.570370+0000","last_peered":"2023-12-19T14:39:11.570370+0000","last_clean":"2023-12-19T14:39:11.570370+0000","last_became_active":"2023-12-19T14:39:07.287333+0000","last_became_peered":"2023-12-19T14:39:07.287333+0000","last_unstale":"2023-12-19T14:39:11.570370+0000","last_undegraded":"2023-12-19T14:39:11.570370+0000","last_fullsized":"2023-12-19T14:39:11.570370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T14:39:19.178278+0000","last_change":"2023-12-19T14:39:06.256219+0000","last_active":"2023-12-19T14:39:19.178278+0000","last_peered":"2023-12-19T14:39:19.178278+0000","last_clean":"2023-12-19T14:39:19.178278+0000","last_became_active":"2023-12-19T14:39:06.255840+0000","last_became_peered":"2023-12-19T14:39:06.255840+0000","last_unstale":"2023-12-19T14:39:19.178278+0000","last_undegraded":"2023-12-19T14:39:19.178278+0000","last_fullsized":"2023-12-19T14:39:19.178278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573334+0000","last_change":"2023-12-19T14:39:11.569520+0000","last_active":"2023-12-19T14:39:11.573334+0000","last_peered":"2023-12-19T14:39:11.573334+0000","last_clean":"2023-12-19T14:39:11.573334+0000","last_became_active":"2023-12-19T14:39:11.563543+0000","last_became_peered":"2023-12-19T14:39:11.563543+0000","last_unstale":"2023-12-19T14:39:11.573334+0000","last_undegraded":"2023-12-19T14:39:11.573334+0000","last_fullsized":"2023-12-19T14:39:11.573334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546941+0000","last_change":"2023-12-19T14:39:07.276126+0000","last_active":"2023-12-19T14:39:11.546941+0000","last_peered":"2023-12-19T14:39:11.546941+0000","last_clean":"2023-12-19T14:39:11.546941+0000","last_became_active":"2023-12-19T14:39:07.275186+0000","last_became_peered":"2023-12-19T14:39:07.275186+0000","last_unstale":"2023-12-19T14:39:11.546941+0000","last_undegraded":"2023-12-19T14:39:11.546941+0000","last_fullsized":"2023-12-19T14:39:11.546941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:40.362295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540550+0000","last_change":"2023-12-19T14:39:06.259793+0000","last_active":"2023-12-19T14:39:11.540550+0000","last_peered":"2023-12-19T14:39:11.540550+0000","last_clean":"2023-12-19T14:39:11.540550+0000","last_became_active":"2023-12-19T14:39:06.259474+0000","last_became_peered":"2023-12-19T14:39:06.259474+0000","last_unstale":"2023-12-19T14:39:11.540550+0000","last_undegraded":"2023-12-19T14:39:11.540550+0000","last_fullsized":"2023-12-19T14:39:11.540550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573690+0000","last_change":"2023-12-19T14:39:11.567330+0000","last_active":"2023-12-19T14:39:11.573690+0000","last_peered":"2023-12-19T14:39:11.573690+0000","last_clean":"2023-12-19T14:39:11.573690+0000","last_became_active":"2023-12-19T14:39:11.566909+0000","last_became_peered":"2023-12-19T14:39:11.566909+0000","last_unstale":"2023-12-19T14:39:11.573690+0000","last_undegraded":"2023-12-19T14:39:11.573690+0000","last_fullsized":"2023-12-19T14:39:11.573690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:41:18.238432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561959+0000","last_change":"2023-12-19T14:39:07.290356+0000","last_active":"2023-12-19T14:39:11.561959+0000","last_peered":"2023-12-19T14:39:11.561959+0000","last_clean":"2023-12-19T14:39:11.561959+0000","last_became_active":"2023-12-19T14:39:07.288476+0000","last_became_peered":"2023-12-19T14:39:07.288476+0000","last_unstale":"2023-12-19T14:39:11.561959+0000","last_undegraded":"2023-12-19T14:39:11.561959+0000","last_fullsized":"2023-12-19T14:39:11.561959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549225+0000","last_change":"2023-12-19T14:39:06.263437+0000","last_active":"2023-12-19T14:39:11.549225+0000","last_peered":"2023-12-19T14:39:11.549225+0000","last_clean":"2023-12-19T14:39:11.549225+0000","last_became_active":"2023-12-19T14:39:06.263146+0000","last_became_peered":"2023-12-19T14:39:06.263146+0000","last_unstale":"2023-12-19T14:39:11.549225+0000","last_undegraded":"2023-12-19T14:39:11.549225+0000","last_fullsized":"2023-12-19T14:39:11.549225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582385+0000","last_change":"2023-12-19T14:39:11.569957+0000","last_active":"2023-12-19T14:39:11.582385+0000","last_peered":"2023-12-19T14:39:11.582385+0000","last_clean":"2023-12-19T14:39:11.582385+0000","last_became_active":"2023-12-19T14:39:11.568502+0000","last_became_peered":"2023-12-19T14:39:11.568502+0000","last_unstale":"2023-12-19T14:39:11.582385+0000","last_undegraded":"2023-12-19T14:39:11.582385+0000","last_fullsized":"2023-12-19T14:39:11.582385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:02:34.258771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549124+0000","last_change":"2023-12-19T14:39:07.284419+0000","last_active":"2023-12-19T14:39:11.549124+0000","last_peered":"2023-12-19T14:39:11.549124+0000","last_clean":"2023-12-19T14:39:11.549124+0000","last_became_active":"2023-12-19T14:39:07.283507+0000","last_became_peered":"2023-12-19T14:39:07.283507+0000","last_unstale":"2023-12-19T14:39:11.549124+0000","last_undegraded":"2023-12-19T14:39:11.549124+0000","last_fullsized":"2023-12-19T14:39:11.549124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551787+0000","last_change":"2023-12-19T14:39:06.261357+0000","last_active":"2023-12-19T14:39:11.551787+0000","last_peered":"2023-12-19T14:39:11.551787+0000","last_clean":"2023-12-19T14:39:11.551787+0000","last_became_active":"2023-12-19T14:39:06.260833+0000","last_became_peered":"2023-12-19T14:39:06.260833+0000","last_unstale":"2023-12-19T14:39:11.551787+0000","last_undegraded":"2023-12-19T14:39:11.551787+0000","last_fullsized":"2023-12-19T14:39:11.551787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588646+0000","last_change":"2023-12-19T14:39:11.579827+0000","last_active":"2023-12-19T14:39:11.588646+0000","last_peered":"2023-12-19T14:39:11.588646+0000","last_clean":"2023-12-19T14:39:11.588646+0000","last_became_active":"2023-12-19T14:39:11.578792+0000","last_became_peered":"2023-12-19T14:39:11.578792+0000","last_unstale":"2023-12-19T14:39:11.588646+0000","last_undegraded":"2023-12-19T14:39:11.588646+0000","last_fullsized":"2023-12-19T14:39:11.588646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583746+0000","last_change":"2023-12-19T14:39:11.581965+0000","last_active":"2023-12-19T14:39:11.583746+0000","last_peered":"2023-12-19T14:39:11.583746+0000","last_clean":"2023-12-19T14:39:11.583746+0000","last_became_active":"2023-12-19T14:39:11.581412+0000","last_became_peered":"2023-12-19T14:39:11.581412+0000","last_unstale":"2023-12-19T14:39:11.583746+0000","last_undegraded":"2023-12-19T14:39:11.583746+0000","last_fullsized":"2023-12-19T14:39:11.583746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539409+0000","last_change":"2023-12-19T14:39:07.283081+0000","last_active":"2023-12-19T14:39:11.539409+0000","last_peered":"2023-12-19T14:39:11.539409+0000","last_clean":"2023-12-19T14:39:11.539409+0000","last_became_active":"2023-12-19T14:39:07.276260+0000","last_became_peered":"2023-12-19T14:39:07.276260+0000","last_unstale":"2023-12-19T14:39:11.539409+0000","last_undegraded":"2023-12-19T14:39:11.539409+0000","last_fullsized":"2023-12-19T14:39:11.539409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543780+0000","last_change":"2023-12-19T14:39:06.257897+0000","last_active":"2023-12-19T14:39:11.543780+0000","last_peered":"2023-12-19T14:39:11.543780+0000","last_clean":"2023-12-19T14:39:11.543780+0000","last_became_active":"2023-12-19T14:39:06.257281+0000","last_became_peered":"2023-12-19T14:39:06.257281+0000","last_unstale":"2023-12-19T14:39:11.543780+0000","last_undegraded":"2023-12-19T14:39:11.543780+0000","last_fullsized":"2023-12-19T14:39:11.543780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571142+0000","last_change":"2023-12-19T14:39:11.566055+0000","last_active":"2023-12-19T14:39:11.571142+0000","last_peered":"2023-12-19T14:39:11.571142+0000","last_clean":"2023-12-19T14:39:11.571142+0000","last_became_active":"2023-12-19T14:39:11.565416+0000","last_became_peered":"2023-12-19T14:39:11.565416+0000","last_unstale":"2023-12-19T14:39:11.571142+0000","last_undegraded":"2023-12-19T14:39:11.571142+0000","last_fullsized":"2023-12-19T14:39:11.571142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:12:19.857166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545049+0000","last_change":"2023-12-19T14:39:07.290776+0000","last_active":"2023-12-19T14:39:11.545049+0000","last_peered":"2023-12-19T14:39:11.545049+0000","last_clean":"2023-12-19T14:39:11.545049+0000","last_became_active":"2023-12-19T14:39:07.290465+0000","last_became_peered":"2023-12-19T14:39:07.290465+0000","last_unstale":"2023-12-19T14:39:11.545049+0000","last_undegraded":"2023-12-19T14:39:11.545049+0000","last_fullsized":"2023-12-19T14:39:11.545049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543823+0000","last_change":"2023-12-19T14:39:06.259345+0000","last_active":"2023-12-19T14:39:11.543823+0000","last_peered":"2023-12-19T14:39:11.543823+0000","last_clean":"2023-12-19T14:39:11.543823+0000","last_became_active":"2023-12-19T14:39:06.259067+0000","last_became_peered":"2023-12-19T14:39:06.259067+0000","last_unstale":"2023-12-19T14:39:11.543823+0000","last_undegraded":"2023-12-19T14:39:11.543823+0000","last_fullsized":"2023-12-19T14:39:11.543823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550184+0000","last_change":"2023-12-19T14:39:06.260437+0000","last_active":"2023-12-19T14:39:11.550184+0000","last_peered":"2023-12-19T14:39:11.550184+0000","last_clean":"2023-12-19T14:39:11.550184+0000","last_became_active":"2023-12-19T14:39:06.260008+0000","last_became_peered":"2023-12-19T14:39:06.260008+0000","last_unstale":"2023-12-19T14:39:11.550184+0000","last_undegraded":"2023-12-19T14:39:11.550184+0000","last_fullsized":"2023-12-19T14:39:11.550184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559895+0000","last_change":"2023-12-19T14:39:07.288421+0000","last_active":"2023-12-19T14:39:11.559895+0000","last_peered":"2023-12-19T14:39:11.559895+0000","last_clean":"2023-12-19T14:39:11.559895+0000","last_became_active":"2023-12-19T14:39:07.287403+0000","last_became_peered":"2023-12-19T14:39:07.287403+0000","last_unstale":"2023-12-19T14:39:11.559895+0000","last_undegraded":"2023-12-19T14:39:11.559895+0000","last_fullsized":"2023-12-19T14:39:11.559895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581278+0000","last_change":"2023-12-19T14:39:11.577684+0000","last_active":"2023-12-19T14:39:11.581278+0000","last_peered":"2023-12-19T14:39:11.581278+0000","last_clean":"2023-12-19T14:39:11.581278+0000","last_became_active":"2023-12-19T14:39:11.577068+0000","last_became_peered":"2023-12-19T14:39:11.577068+0000","last_unstale":"2023-12-19T14:39:11.581278+0000","last_undegraded":"2023-12-19T14:39:11.581278+0000","last_fullsized":"2023-12-19T14:39:11.581278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:42.373695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544048+0000","last_change":"2023-12-19T14:39:06.247049+0000","last_active":"2023-12-19T14:39:11.544048+0000","last_peered":"2023-12-19T14:39:11.544048+0000","last_clean":"2023-12-19T14:39:11.544048+0000","last_became_active":"2023-12-19T14:39:06.246716+0000","last_became_peered":"2023-12-19T14:39:06.246716+0000","last_unstale":"2023-12-19T14:39:11.544048+0000","last_undegraded":"2023-12-19T14:39:11.544048+0000","last_fullsized":"2023-12-19T14:39:11.544048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546858+0000","last_change":"2023-12-19T14:39:07.266860+0000","last_active":"2023-12-19T14:39:11.546858+0000","last_peered":"2023-12-19T14:39:11.546858+0000","last_clean":"2023-12-19T14:39:11.546858+0000","last_became_active":"2023-12-19T14:39:07.266544+0000","last_became_peered":"2023-12-19T14:39:07.266544+0000","last_unstale":"2023-12-19T14:39:11.546858+0000","last_undegraded":"2023-12-19T14:39:11.546858+0000","last_fullsized":"2023-12-19T14:39:11.546858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:47.972055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.585215+0000","last_change":"2023-12-19T14:39:11.567865+0000","last_active":"2023-12-19T14:39:11.585215+0000","last_peered":"2023-12-19T14:39:11.585215+0000","last_clean":"2023-12-19T14:39:11.585215+0000","last_became_active":"2023-12-19T14:39:11.566466+0000","last_became_peered":"2023-12-19T14:39:11.566466+0000","last_unstale":"2023-12-19T14:39:11.585215+0000","last_undegraded":"2023-12-19T14:39:11.585215+0000","last_fullsized":"2023-12-19T14:39:11.585215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:11.949761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551612+0000","last_change":"2023-12-19T14:39:07.289708+0000","last_active":"2023-12-19T14:39:11.551612+0000","last_peered":"2023-12-19T14:39:11.551612+0000","last_clean":"2023-12-19T14:39:11.551612+0000","last_became_active":"2023-12-19T14:39:07.289420+0000","last_became_peered":"2023-12-19T14:39:07.289420+0000","last_unstale":"2023-12-19T14:39:11.551612+0000","last_undegraded":"2023-12-19T14:39:11.551612+0000","last_fullsized":"2023-12-19T14:39:11.551612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.552204+0000","last_change":"2023-12-19T14:39:07.291613+0000","last_active":"2023-12-19T14:39:11.552204+0000","last_peered":"2023-12-19T14:39:11.552204+0000","last_clean":"2023-12-19T14:39:11.552204+0000","last_became_active":"2023-12-19T14:39:07.291277+0000","last_became_peered":"2023-12-19T14:39:07.291277+0000","last_unstale":"2023-12-19T14:39:11.552204+0000","last_undegraded":"2023-12-19T14:39:11.552204+0000","last_fullsized":"2023-12-19T14:39:11.552204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:23.914755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544957+0000","last_change":"2023-12-19T14:39:07.294351+0000","last_active":"2023-12-19T14:39:11.544957+0000","last_peered":"2023-12-19T14:39:11.544957+0000","last_clean":"2023-12-19T14:39:11.544957+0000","last_became_active":"2023-12-19T14:39:07.293414+0000","last_became_peered":"2023-12-19T14:39:07.293414+0000","last_unstale":"2023-12-19T14:39:11.544957+0000","last_undegraded":"2023-12-19T14:39:11.544957+0000","last_fullsized":"2023-12-19T14:39:11.544957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559855+0000","last_change":"2023-12-19T14:39:07.287045+0000","last_active":"2023-12-19T14:39:11.559855+0000","last_peered":"2023-12-19T14:39:11.559855+0000","last_clean":"2023-12-19T14:39:11.559855+0000","last_became_active":"2023-12-19T14:39:07.286711+0000","last_became_peered":"2023-12-19T14:39:07.286711+0000","last_unstale":"2023-12-19T14:39:11.559855+0000","last_undegraded":"2023-12-19T14:39:11.559855+0000","last_fullsized":"2023-12-19T14:39:11.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545892+0000","last_change":"2023-12-19T14:39:07.284424+0000","last_active":"2023-12-19T14:39:11.545892+0000","last_peered":"2023-12-19T14:39:11.545892+0000","last_clean":"2023-12-19T14:39:11.545892+0000","last_became_active":"2023-12-19T14:39:07.284121+0000","last_became_peered":"2023-12-19T14:39:07.284121+0000","last_unstale":"2023-12-19T14:39:11.545892+0000","last_undegraded":"2023-12-19T14:39:11.545892+0000","last_fullsized":"2023-12-19T14:39:11.545892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540559+0000","last_change":"2023-12-19T14:39:07.280745+0000","last_active":"2023-12-19T14:39:11.540559+0000","last_peered":"2023-12-19T14:39:11.540559+0000","last_clean":"2023-12-19T14:39:11.540559+0000","last_became_active":"2023-12-19T14:39:07.280443+0000","last_became_peered":"2023-12-19T14:39:07.280443+0000","last_unstale":"2023-12-19T14:39:11.540559+0000","last_undegraded":"2023-12-19T14:39:11.540559+0000","last_fullsized":"2023-12-19T14:39:11.540559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550784+0000","last_change":"2023-12-19T14:39:07.288049+0000","last_active":"2023-12-19T14:39:11.550784+0000","last_peered":"2023-12-19T14:39:11.550784+0000","last_clean":"2023-12-19T14:39:11.550784+0000","last_became_active":"2023-12-19T14:39:07.287758+0000","last_became_peered":"2023-12-19T14:39:07.287758+0000","last_unstale":"2023-12-19T14:39:11.550784+0000","last_undegraded":"2023-12-19T14:39:11.550784+0000","last_fullsized":"2023-12-19T14:39:11.550784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.558076+0000","last_change":"2023-12-19T14:39:07.284681+0000","last_active":"2023-12-19T14:39:11.558076+0000","last_peered":"2023-12-19T14:39:11.558076+0000","last_clean":"2023-12-19T14:39:11.558076+0000","last_became_active":"2023-12-19T14:39:07.283596+0000","last_became_peered":"2023-12-19T14:39:07.283596+0000","last_unstale":"2023-12-19T14:39:11.558076+0000","last_undegraded":"2023-12-19T14:39:11.558076+0000","last_fullsized":"2023-12-19T14:39:11.558076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545014+0000","last_change":"2023-12-19T14:39:07.293485+0000","last_active":"2023-12-19T14:39:11.545014+0000","last_peered":"2023-12-19T14:39:11.545014+0000","last_clean":"2023-12-19T14:39:11.545014+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.545014+0000","last_undegraded":"2023-12-19T14:39:11.545014+0000","last_fullsized":"2023-12-19T14:39:11.545014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543694+0000","last_change":"2023-12-19T14:39:07.283853+0000","last_active":"2023-12-19T14:39:11.543694+0000","last_peered":"2023-12-19T14:39:11.543694+0000","last_clean":"2023-12-19T14:39:11.543694+0000","last_became_active":"2023-12-19T14:39:07.283616+0000","last_became_peered":"2023-12-19T14:39:07.283616+0000","last_unstale":"2023-12-19T14:39:11.543694+0000","last_undegraded":"2023-12-19T14:39:11.543694+0000","last_fullsized":"2023-12-19T14:39:11.543694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544936+0000","last_change":"2023-12-19T14:39:07.294504+0000","last_active":"2023-12-19T14:39:11.544936+0000","last_peered":"2023-12-19T14:39:11.544936+0000","last_clean":"2023-12-19T14:39:11.544936+0000","last_became_active":"2023-12-19T14:39:07.293953+0000","last_became_peered":"2023-12-19T14:39:07.293953+0000","last_unstale":"2023-12-19T14:39:11.544936+0000","last_undegraded":"2023-12-19T14:39:11.544936+0000","last_fullsized":"2023-12-19T14:39:11.544936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559798+0000","last_change":"2023-12-19T14:39:07.296762+0000","last_active":"2023-12-19T14:39:11.559798+0000","last_peered":"2023-12-19T14:39:11.559798+0000","last_clean":"2023-12-19T14:39:11.559798+0000","last_became_active":"2023-12-19T14:39:07.295928+0000","last_became_peered":"2023-12-19T14:39:07.295928+0000","last_unstale":"2023-12-19T14:39:11.559798+0000","last_undegraded":"2023-12-19T14:39:11.559798+0000","last_fullsized":"2023-12-19T14:39:11.559798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549583+0000","last_change":"2023-12-19T14:39:07.280149+0000","last_active":"2023-12-19T14:39:11.549583+0000","last_peered":"2023-12-19T14:39:11.549583+0000","last_clean":"2023-12-19T14:39:11.549583+0000","last_became_active":"2023-12-19T14:39:07.279877+0000","last_became_peered":"2023-12-19T14:39:07.279877+0000","last_unstale":"2023-12-19T14:39:11.549583+0000","last_undegraded":"2023-12-19T14:39:11.549583+0000","last_fullsized":"2023-12-19T14:39:11.549583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:59:49.003736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570730+0000","last_change":"2023-12-19T14:39:07.288397+0000","last_active":"2023-12-19T14:39:11.570730+0000","last_peered":"2023-12-19T14:39:11.570730+0000","last_clean":"2023-12-19T14:39:11.570730+0000","last_became_active":"2023-12-19T14:39:07.287749+0000","last_became_peered":"2023-12-19T14:39:07.287749+0000","last_unstale":"2023-12-19T14:39:11.570730+0000","last_undegraded":"2023-12-19T14:39:11.570730+0000","last_fullsized":"2023-12-19T14:39:11.570730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555145+0000","last_change":"2023-12-19T14:39:07.276259+0000","last_active":"2023-12-19T14:39:11.555145+0000","last_peered":"2023-12-19T14:39:11.555145+0000","last_clean":"2023-12-19T14:39:11.555145+0000","last_became_active":"2023-12-19T14:39:07.275951+0000","last_became_peered":"2023-12-19T14:39:07.275951+0000","last_unstale":"2023-12-19T14:39:11.555145+0000","last_undegraded":"2023-12-19T14:39:11.555145+0000","last_fullsized":"2023-12-19T14:39:11.555145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549564+0000","last_change":"2023-12-19T14:39:07.295581+0000","last_active":"2023-12-19T14:39:11.549564+0000","last_peered":"2023-12-19T14:39:11.549564+0000","last_clean":"2023-12-19T14:39:11.549564+0000","last_became_active":"2023-12-19T14:39:07.293567+0000","last_became_peered":"2023-12-19T14:39:07.293567+0000","last_unstale":"2023-12-19T14:39:11.549564+0000","last_undegraded":"2023-12-19T14:39:11.549564+0000","last_fullsized":"2023-12-19T14:39:11.549564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543139+0000","last_change":"2023-12-19T14:39:07.293875+0000","last_active":"2023-12-19T14:39:11.543139+0000","last_peered":"2023-12-19T14:39:11.543139+0000","last_clean":"2023-12-19T14:39:11.543139+0000","last_became_active":"2023-12-19T14:39:07.293381+0000","last_became_peered":"2023-12-19T14:39:07.293381+0000","last_unstale":"2023-12-19T14:39:11.543139+0000","last_undegraded":"2023-12-19T14:39:11.543139+0000","last_fullsized":"2023-12-19T14:39:11.543139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540124+0000","last_change":"2023-12-19T14:39:07.261761+0000","last_active":"2023-12-19T14:39:11.540124+0000","last_peered":"2023-12-19T14:39:11.540124+0000","last_clean":"2023-12-19T14:39:11.540124+0000","last_became_active":"2023-12-19T14:39:07.261446+0000","last_became_peered":"2023-12-19T14:39:07.261446+0000","last_unstale":"2023-12-19T14:39:11.540124+0000","last_undegraded":"2023-12-19T14:39:11.540124+0000","last_fullsized":"2023-12-19T14:39:11.540124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556947+0000","last_change":"2023-12-19T14:39:07.290139+0000","last_active":"2023-12-19T14:39:11.556947+0000","last_peered":"2023-12-19T14:39:11.556947+0000","last_clean":"2023-12-19T14:39:11.556947+0000","last_became_active":"2023-12-19T14:39:07.289843+0000","last_became_peered":"2023-12-19T14:39:07.289843+0000","last_unstale":"2023-12-19T14:39:11.556947+0000","last_undegraded":"2023-12-19T14:39:11.556947+0000","last_fullsized":"2023-12-19T14:39:11.556947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543821+0000","last_change":"2023-12-19T14:39:07.281738+0000","last_active":"2023-12-19T14:39:11.543821+0000","last_peered":"2023-12-19T14:39:11.543821+0000","last_clean":"2023-12-19T14:39:11.543821+0000","last_became_active":"2023-12-19T14:39:07.281443+0000","last_became_peered":"2023-12-19T14:39:07.281443+0000","last_unstale":"2023-12-19T14:39:11.543821+0000","last_undegraded":"2023-12-19T14:39:11.543821+0000","last_fullsized":"2023-12-19T14:39:11.543821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546820+0000","last_change":"2023-12-19T14:39:07.276674+0000","last_active":"2023-12-19T14:39:11.546820+0000","last_peered":"2023-12-19T14:39:11.546820+0000","last_clean":"2023-12-19T14:39:11.546820+0000","last_became_active":"2023-12-19T14:39:07.276368+0000","last_became_peered":"2023-12-19T14:39:07.276368+0000","last_unstale":"2023-12-19T14:39:11.546820+0000","last_undegraded":"2023-12-19T14:39:11.546820+0000","last_fullsized":"2023-12-19T14:39:11.546820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:35.210805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549881+0000","last_change":"2023-12-19T14:39:07.281026+0000","last_active":"2023-12-19T14:39:11.549881+0000","last_peered":"2023-12-19T14:39:11.549881+0000","last_clean":"2023-12-19T14:39:11.549881+0000","last_became_active":"2023-12-19T14:39:07.280689+0000","last_became_peered":"2023-12-19T14:39:07.280689+0000","last_unstale":"2023-12-19T14:39:11.549881+0000","last_undegraded":"2023-12-19T14:39:11.549881+0000","last_fullsized":"2023-12-19T14:39:11.549881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550701+0000","last_change":"2023-12-19T14:39:07.280698+0000","last_active":"2023-12-19T14:39:11.550701+0000","last_peered":"2023-12-19T14:39:11.550701+0000","last_clean":"2023-12-19T14:39:11.550701+0000","last_became_active":"2023-12-19T14:39:07.280406+0000","last_became_peered":"2023-12-19T14:39:07.280406+0000","last_unstale":"2023-12-19T14:39:11.550701+0000","last_undegraded":"2023-12-19T14:39:11.550701+0000","last_fullsized":"2023-12-19T14:39:11.550701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570926+0000","last_change":"2023-12-19T14:39:07.284712+0000","last_active":"2023-12-19T14:39:11.570926+0000","last_peered":"2023-12-19T14:39:11.570926+0000","last_clean":"2023-12-19T14:39:11.570926+0000","last_became_active":"2023-12-19T14:39:07.283598+0000","last_became_peered":"2023-12-19T14:39:07.283598+0000","last_unstale":"2023-12-19T14:39:11.570926+0000","last_undegraded":"2023-12-19T14:39:11.570926+0000","last_fullsized":"2023-12-19T14:39:11.570926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544886+0000","last_change":"2023-12-19T14:39:07.294169+0000","last_active":"2023-12-19T14:39:11.544886+0000","last_peered":"2023-12-19T14:39:11.544886+0000","last_clean":"2023-12-19T14:39:11.544886+0000","last_became_active":"2023-12-19T14:39:07.293746+0000","last_became_peered":"2023-12-19T14:39:07.293746+0000","last_unstale":"2023-12-19T14:39:11.544886+0000","last_undegraded":"2023-12-19T14:39:11.544886+0000","last_fullsized":"2023-12-19T14:39:11.544886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570207+0000","last_change":"2023-12-19T14:39:07.291661+0000","last_active":"2023-12-19T14:39:11.570207+0000","last_peered":"2023-12-19T14:39:11.570207+0000","last_clean":"2023-12-19T14:39:11.570207+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.570207+0000","last_undegraded":"2023-12-19T14:39:11.570207+0000","last_fullsized":"2023-12-19T14:39:11.570207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:34:24.416865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544859+0000","last_change":"2023-12-19T14:39:07.297368+0000","last_active":"2023-12-19T14:39:11.544859+0000","last_peered":"2023-12-19T14:39:11.544859+0000","last_clean":"2023-12-19T14:39:11.544859+0000","last_became_active":"2023-12-19T14:39:07.297082+0000","last_became_peered":"2023-12-19T14:39:07.297082+0000","last_unstale":"2023-12-19T14:39:11.544859+0000","last_undegraded":"2023-12-19T14:39:11.544859+0000","last_fullsized":"2023-12-19T14:39:11.544859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570469+0000","last_change":"2023-12-19T14:39:07.287921+0000","last_active":"2023-12-19T14:39:11.570469+0000","last_peered":"2023-12-19T14:39:11.570469+0000","last_clean":"2023-12-19T14:39:11.570469+0000","last_became_active":"2023-12-19T14:39:07.287493+0000","last_became_peered":"2023-12-19T14:39:07.287493+0000","last_unstale":"2023-12-19T14:39:11.570469+0000","last_undegraded":"2023-12-19T14:39:11.570469+0000","last_fullsized":"2023-12-19T14:39:11.570469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545732+0000","last_change":"2023-12-19T14:39:07.290935+0000","last_active":"2023-12-19T14:39:11.545732+0000","last_peered":"2023-12-19T14:39:11.545732+0000","last_clean":"2023-12-19T14:39:11.545732+0000","last_became_active":"2023-12-19T14:39:07.290614+0000","last_became_peered":"2023-12-19T14:39:07.290614+0000","last_unstale":"2023-12-19T14:39:11.545732+0000","last_undegraded":"2023-12-19T14:39:11.545732+0000","last_fullsized":"2023-12-19T14:39:11.545732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.560160+0000","last_change":"2023-12-19T14:39:07.287280+0000","last_active":"2023-12-19T14:39:11.560160+0000","last_peered":"2023-12-19T14:39:11.560160+0000","last_clean":"2023-12-19T14:39:11.560160+0000","last_became_active":"2023-12-19T14:39:07.286949+0000","last_became_peered":"2023-12-19T14:39:07.286949+0000","last_unstale":"2023-12-19T14:39:11.560160+0000","last_undegraded":"2023-12-19T14:39:11.560160+0000","last_fullsized":"2023-12-19T14:39:11.560160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540585+0000","last_change":"2023-12-19T14:39:07.295118+0000","last_active":"2023-12-19T14:39:11.540585+0000","last_peered":"2023-12-19T14:39:11.540585+0000","last_clean":"2023-12-19T14:39:11.540585+0000","last_became_active":"2023-12-19T14:39:07.294282+0000","last_became_peered":"2023-12-19T14:39:07.294282+0000","last_unstale":"2023-12-19T14:39:11.540585+0000","last_undegraded":"2023-12-19T14:39:11.540585+0000","last_fullsized":"2023-12-19T14:39:11.540585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570810+0000","last_change":"2023-12-19T14:39:07.283979+0000","last_active":"2023-12-19T14:39:11.570810+0000","last_peered":"2023-12-19T14:39:11.570810+0000","last_clean":"2023-12-19T14:39:11.570810+0000","last_became_active":"2023-12-19T14:39:07.280235+0000","last_became_peered":"2023-12-19T14:39:07.280235+0000","last_unstale":"2023-12-19T14:39:11.570810+0000","last_undegraded":"2023-12-19T14:39:11.570810+0000","last_fullsized":"2023-12-19T14:39:11.570810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301628,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070212,"statfs":{"total":96636764160,"available":96327897088,"internally_reserved":0,"allocated":249856,"data_stored":98477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302220,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069620,"statfs":{"total":96636764160,"available":96327290880,"internally_reserved":0,"allocated":856064,"data_stored":689455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:39:27.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:39:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 Processor -- start 2023-12-19T14:39:27.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 -- start start 2023-12-19T14:39:27.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 0x7fce34159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340cd650 con 0x7fce340cc4b0 2023-12-19T14:39:27.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340cca40 con 0x7fce340d0150 2023-12-19T14:39:27.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.897+0000 7fce39745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340ccd20 con 0x7fce34158c90 2023-12-19T14:39:27.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce2a7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fce340cc4b0 0x7fce340cdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44660/0 (socket says 172.21.15.50:44660) 2023-12-19T14:39:27.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce2a7fc700 1 -- 172.21.15.50:0/2506104639 learned_addr learned my addr 172.21.15.50:0/2506104639 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:27.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce32ffd700 1 --2- 172.21.15.50:0/2506104639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 0x7fce34159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce33fff700 1 --1- 172.21.15.50:0/2506104639 >> v1:172.21.15.125:6789/0 conn(0x7fce340d0150 0x7fce3406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40952/0 (socket says 172.21.15.50:40952) 2023-12-19T14:39:27.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079702254 0 0) 0x7fce340cd650 con 0x7fce340cc4b0 2023-12-19T14:39:27.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003570 con 0x7fce340cc4b0 2023-12-19T14:39:27.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294272394 0 0) 0x7fce340cca40 con 0x7fce340d0150 2023-12-19T14:39:27.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340cd650 con 0x7fce340d0150 2023-12-19T14:39:27.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce32ffd700 1 -- 172.21.15.50:0/2506104639 >> v1:172.21.15.125:6789/0 conn(0x7fce340d0150 legacy=0x7fce3406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce32ffd700 1 -- 172.21.15.50:0/2506104639 >> v1:172.21.15.50:6789/0 conn(0x7fce340cc4b0 legacy=0x7fce340cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce32ffd700 1 -- 172.21.15.50:0/2506104639 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340cd000 con 0x7fce34158c90 2023-12-19T14:39:27.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce32ffd700 1 --2- 172.21.15.50:0/2506104639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 0x7fce34159060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fce24009910 tx=0x7fce240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb8aa4bb4862b053 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24011630 con 0x7fce34158c90 2023-12-19T14:39:27.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24011790 con 0x7fce34158c90 2023-12-19T14:39:27.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce327fc700 1 -- 172.21.15.50:0/2506104639 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24011b30 con 0x7fce34158c90 2023-12-19T14:39:27.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce39745700 1 -- 172.21.15.50:0/2506104639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 msgr2=0x7fce34159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.901+0000 7fce39745700 1 --2- 172.21.15.50:0/2506104639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 0x7fce34159060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fce24009910 tx=0x7fce240091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 -- 172.21.15.50:0/2506104639 shutdown_connections 2023-12-19T14:39:27.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 --2- 172.21.15.50:0/2506104639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce34158c90 0x7fce34159060 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 -- 172.21.15.50:0/2506104639 >> 172.21.15.50:0/2506104639 conn(0x7fce340ce320 msgr2=0x7fce340ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:27.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 -- 172.21.15.50:0/2506104639 shutdown_connections 2023-12-19T14:39:27.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 -- 172.21.15.50:0/2506104639 wait complete. 2023-12-19T14:39:27.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 Processor -- start 2023-12-19T14:39:27.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.905+0000 7fce39745700 1 -- start start 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce340cc4b0 0x7fce340cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce34158c90 0x7fce3416a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340b6630 con 0x7fce340d0150 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340b5a20 con 0x7fce34158c90 2023-12-19T14:39:27.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340b5d00 con 0x7fce340cc4b0 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59122/0 (socket says 172.21.15.50:59122) 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 -- 172.21.15.50:0/1444033513 learned_addr learned my addr 172.21.15.50:0/1444033513 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce32ffd700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce340cc4b0 0x7fce340cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce33fff700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce34158c90 0x7fce3416a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 -- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce340cc4b0 msgr2=0x7fce340cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce340cc4b0 0x7fce340cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 -- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce34158c90 msgr2=0x7fce3416a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:27.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce34158c90 0x7fce3416a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:27.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 -- 172.21.15.50:0/1444033513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24009420 con 0x7fce340d0150 2023-12-19T14:39:27.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2a7fc700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c790 tx=0x7fce1c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c005b30 con 0x7fce340d0150 2023-12-19T14:39:27.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c005c90 con 0x7fce340d0150 2023-12-19T14:39:27.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce340b5fe0 con 0x7fce340d0150 2023-12-19T14:39:27.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c007300 con 0x7fce340d0150 2023-12-19T14:39:27.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.909+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3414c4e0 con 0x7fce340d0150 2023-12-19T14:39:27.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.913+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce34071120 con 0x7fce340d0150 2023-12-19T14:39:27.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.913+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fce1c01b3b0 con 0x7fce340d0150 2023-12-19T14:39:27.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.917+0000 7fce2b7fe700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 0x7fce14062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:27.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.917+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fce1c07bff0 con 0x7fce340d0150 2023-12-19T14:39:27.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.917+0000 7fce32ffd700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 0x7fce14062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:27.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.917+0000 7fce32ffd700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 0x7fce14062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce340cba70 tx=0x7fce24017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:27.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:27.925+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce1c04fce0 con 0x7fce340d0150 2023-12-19T14:39:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.081+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce340cd2e0 con 0x7fce14060550 2023-12-19T14:39:28.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.109+0000 7fce2b7fe700 1 -- 172.21.15.50:0/1444033513 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345081 (secure 0 0 0) 0x7fce340cd2e0 con 0x7fce14060550 2023-12-19T14:39:28.117 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:28.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 msgr2=0x7fce14062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 0x7fce14062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce340cba70 tx=0x7fce24017000 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 msgr2=0x7fce340b6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c790 tx=0x7fce1c007ea0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 shutdown_connections 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce34158c90 0x7fce3416a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce14060550 0x7fce14062a00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce340d0150 0x7fce340b6ac0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 --2- 172.21.15.50:0/1444033513 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce340cc4b0 0x7fce340cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 >> 172.21.15.50:0/1444033513 conn(0x7fce340ce320 msgr2=0x7fce340687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 shutdown_connections 2023-12-19T14:39:28.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.117+0000 7fce39745700 1 -- 172.21.15.50:0/1444033513 wait complete. 2023-12-19T14:39:28.123 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:39:28.141 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-19T14:39:26.134085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2393792,"kb_used_data":3200,"kb_used_omap":0,"kb_used_meta":2390528,"kb_avail":752580928,"statfs":{"total":773094113280,"available":770642870272,"internally_reserved":0,"allocated":3276800,"data_stored":1983584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2447900672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004506"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562203+0000","last_change":"2023-12-19T14:39:06.261176+0000","last_active":"2023-12-19T14:39:11.562203+0000","last_peered":"2023-12-19T14:39:11.562203+0000","last_clean":"2023-12-19T14:39:11.562203+0000","last_became_active":"2023-12-19T14:39:06.259857+0000","last_became_peered":"2023-12-19T14:39:06.259857+0000","last_unstale":"2023-12-19T14:39:11.562203+0000","last_undegraded":"2023-12-19T14:39:11.562203+0000","last_fullsized":"2023-12-19T14:39:11.562203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:32:43.075778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545898+0000","last_change":"2023-12-19T14:39:07.280964+0000","last_active":"2023-12-19T14:39:11.545898+0000","last_peered":"2023-12-19T14:39:11.545898+0000","last_clean":"2023-12-19T14:39:11.545898+0000","last_became_active":"2023-12-19T14:39:07.280410+0000","last_became_peered":"2023-12-19T14:39:07.280410+0000","last_unstale":"2023-12-19T14:39:11.545898+0000","last_undegraded":"2023-12-19T14:39:11.545898+0000","last_fullsized":"2023-12-19T14:39:11.545898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588669+0000","last_change":"2023-12-19T14:39:11.580119+0000","last_active":"2023-12-19T14:39:11.588669+0000","last_peered":"2023-12-19T14:39:11.588669+0000","last_clean":"2023-12-19T14:39:11.588669+0000","last_became_active":"2023-12-19T14:39:11.579405+0000","last_became_peered":"2023-12-19T14:39:11.579405+0000","last_unstale":"2023-12-19T14:39:11.588669+0000","last_undegraded":"2023-12-19T14:39:11.588669+0000","last_fullsized":"2023-12-19T14:39:11.588669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546035+0000","last_change":"2023-12-19T14:39:06.252186+0000","last_active":"2023-12-19T14:39:11.546035+0000","last_peered":"2023-12-19T14:39:11.546035+0000","last_clean":"2023-12-19T14:39:11.546035+0000","last_became_active":"2023-12-19T14:39:06.251869+0000","last_became_peered":"2023-12-19T14:39:06.251869+0000","last_unstale":"2023-12-19T14:39:11.546035+0000","last_undegraded":"2023-12-19T14:39:11.546035+0000","last_fullsized":"2023-12-19T14:39:11.546035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561557+0000","last_change":"2023-12-19T14:39:07.289142+0000","last_active":"2023-12-19T14:39:11.561557+0000","last_peered":"2023-12-19T14:39:11.561557+0000","last_clean":"2023-12-19T14:39:11.561557+0000","last_became_active":"2023-12-19T14:39:07.288822+0000","last_became_peered":"2023-12-19T14:39:07.288822+0000","last_unstale":"2023-12-19T14:39:11.561557+0000","last_undegraded":"2023-12-19T14:39:11.561557+0000","last_fullsized":"2023-12-19T14:39:11.561557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582458+0000","last_change":"2023-12-19T14:39:11.573848+0000","last_active":"2023-12-19T14:39:11.582458+0000","last_peered":"2023-12-19T14:39:11.582458+0000","last_clean":"2023-12-19T14:39:11.582458+0000","last_became_active":"2023-12-19T14:39:11.573088+0000","last_became_peered":"2023-12-19T14:39:11.573088+0000","last_unstale":"2023-12-19T14:39:11.582458+0000","last_undegraded":"2023-12-19T14:39:11.582458+0000","last_fullsized":"2023-12-19T14:39:11.582458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:29.985849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570627+0000","last_change":"2023-12-19T14:39:06.262908+0000","last_active":"2023-12-19T14:39:11.570627+0000","last_peered":"2023-12-19T14:39:11.570627+0000","last_clean":"2023-12-19T14:39:11.570627+0000","last_became_active":"2023-12-19T14:39:06.260114+0000","last_became_peered":"2023-12-19T14:39:06.260114+0000","last_unstale":"2023-12-19T14:39:11.570627+0000","last_undegraded":"2023-12-19T14:39:11.570627+0000","last_fullsized":"2023-12-19T14:39:11.570627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545839+0000","last_change":"2023-12-19T14:39:07.279790+0000","last_active":"2023-12-19T14:39:11.545839+0000","last_peered":"2023-12-19T14:39:11.545839+0000","last_clean":"2023-12-19T14:39:11.545839+0000","last_became_active":"2023-12-19T14:39:07.279497+0000","last_became_peered":"2023-12-19T14:39:07.279497+0000","last_unstale":"2023-12-19T14:39:11.545839+0000","last_undegraded":"2023-12-19T14:39:11.545839+0000","last_fullsized":"2023-12-19T14:39:11.545839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.574367+0000","last_change":"2023-12-19T14:39:11.571054+0000","last_active":"2023-12-19T14:39:11.574367+0000","last_peered":"2023-12-19T14:39:11.574367+0000","last_clean":"2023-12-19T14:39:11.574367+0000","last_became_active":"2023-12-19T14:39:11.570620+0000","last_became_peered":"2023-12-19T14:39:11.570620+0000","last_unstale":"2023-12-19T14:39:11.574367+0000","last_undegraded":"2023-12-19T14:39:11.574367+0000","last_fullsized":"2023-12-19T14:39:11.574367+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:06:34.827613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570831+0000","last_change":"2023-12-19T14:39:06.260028+0000","last_active":"2023-12-19T14:39:11.570831+0000","last_peered":"2023-12-19T14:39:11.570831+0000","last_clean":"2023-12-19T14:39:11.570831+0000","last_became_active":"2023-12-19T14:39:06.259269+0000","last_became_peered":"2023-12-19T14:39:06.259269+0000","last_unstale":"2023-12-19T14:39:11.570831+0000","last_undegraded":"2023-12-19T14:39:11.570831+0000","last_fullsized":"2023-12-19T14:39:11.570831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:29.156364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548594+0000","last_change":"2023-12-19T14:39:07.282662+0000","last_active":"2023-12-19T14:39:11.548594+0000","last_peered":"2023-12-19T14:39:11.548594+0000","last_clean":"2023-12-19T14:39:11.548594+0000","last_became_active":"2023-12-19T14:39:07.282105+0000","last_became_peered":"2023-12-19T14:39:07.282105+0000","last_unstale":"2023-12-19T14:39:11.548594+0000","last_undegraded":"2023-12-19T14:39:11.548594+0000","last_fullsized":"2023-12-19T14:39:11.548594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583605+0000","last_change":"2023-12-19T14:39:11.578372+0000","last_active":"2023-12-19T14:39:11.583605+0000","last_peered":"2023-12-19T14:39:11.583605+0000","last_clean":"2023-12-19T14:39:11.583605+0000","last_became_active":"2023-12-19T14:39:11.575437+0000","last_became_peered":"2023-12-19T14:39:11.575437+0000","last_unstale":"2023-12-19T14:39:11.583605+0000","last_undegraded":"2023-12-19T14:39:11.583605+0000","last_fullsized":"2023-12-19T14:39:11.583605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:01.783711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550051+0000","last_change":"2023-12-19T14:39:07.274147+0000","last_active":"2023-12-19T14:39:11.550051+0000","last_peered":"2023-12-19T14:39:11.550051+0000","last_clean":"2023-12-19T14:39:11.550051+0000","last_became_active":"2023-12-19T14:39:07.273776+0000","last_became_peered":"2023-12-19T14:39:07.273776+0000","last_unstale":"2023-12-19T14:39:11.550051+0000","last_undegraded":"2023-12-19T14:39:11.550051+0000","last_fullsized":"2023-12-19T14:39:11.550051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570793+0000","last_change":"2023-12-19T14:39:06.253750+0000","last_active":"2023-12-19T14:39:11.570793+0000","last_peered":"2023-12-19T14:39:11.570793+0000","last_clean":"2023-12-19T14:39:11.570793+0000","last_became_active":"2023-12-19T14:39:06.253479+0000","last_became_peered":"2023-12-19T14:39:06.253479+0000","last_unstale":"2023-12-19T14:39:11.570793+0000","last_undegraded":"2023-12-19T14:39:11.570793+0000","last_fullsized":"2023-12-19T14:39:11.570793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:22.901851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564091+0000","last_change":"2023-12-19T14:39:11.562226+0000","last_active":"2023-12-19T14:39:11.564091+0000","last_peered":"2023-12-19T14:39:11.564091+0000","last_clean":"2023-12-19T14:39:11.564091+0000","last_became_active":"2023-12-19T14:39:11.561874+0000","last_became_peered":"2023-12-19T14:39:11.561874+0000","last_unstale":"2023-12-19T14:39:11.564091+0000","last_undegraded":"2023-12-19T14:39:11.564091+0000","last_fullsized":"2023-12-19T14:39:11.564091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543918+0000","last_change":"2023-12-19T14:39:07.297728+0000","last_active":"2023-12-19T14:39:11.543918+0000","last_peered":"2023-12-19T14:39:11.543918+0000","last_clean":"2023-12-19T14:39:11.543918+0000","last_became_active":"2023-12-19T14:39:07.297419+0000","last_became_peered":"2023-12-19T14:39:07.297419+0000","last_unstale":"2023-12-19T14:39:11.543918+0000","last_undegraded":"2023-12-19T14:39:11.543918+0000","last_fullsized":"2023-12-19T14:39:11.543918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540603+0000","last_change":"2023-12-19T14:39:06.250154+0000","last_active":"2023-12-19T14:39:11.540603+0000","last_peered":"2023-12-19T14:39:11.540603+0000","last_clean":"2023-12-19T14:39:11.540603+0000","last_became_active":"2023-12-19T14:39:06.249851+0000","last_became_peered":"2023-12-19T14:39:06.249851+0000","last_unstale":"2023-12-19T14:39:11.540603+0000","last_undegraded":"2023-12-19T14:39:11.540603+0000","last_fullsized":"2023-12-19T14:39:11.540603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583288+0000","last_change":"2023-12-19T14:39:11.576656+0000","last_active":"2023-12-19T14:39:11.583288+0000","last_peered":"2023-12-19T14:39:11.583288+0000","last_clean":"2023-12-19T14:39:11.583288+0000","last_became_active":"2023-12-19T14:39:11.576165+0000","last_became_peered":"2023-12-19T14:39:11.576165+0000","last_unstale":"2023-12-19T14:39:11.583288+0000","last_undegraded":"2023-12-19T14:39:11.583288+0000","last_fullsized":"2023-12-19T14:39:11.583288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545215+0000","last_change":"2023-12-19T14:39:07.274124+0000","last_active":"2023-12-19T14:39:11.545215+0000","last_peered":"2023-12-19T14:39:11.545215+0000","last_clean":"2023-12-19T14:39:11.545215+0000","last_became_active":"2023-12-19T14:39:07.273792+0000","last_became_peered":"2023-12-19T14:39:07.273792+0000","last_unstale":"2023-12-19T14:39:11.545215+0000","last_undegraded":"2023-12-19T14:39:11.545215+0000","last_fullsized":"2023-12-19T14:39:11.545215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540233+0000","last_change":"2023-12-19T14:39:06.244868+0000","last_active":"2023-12-19T14:39:11.540233+0000","last_peered":"2023-12-19T14:39:11.540233+0000","last_clean":"2023-12-19T14:39:11.540233+0000","last_became_active":"2023-12-19T14:39:06.244567+0000","last_became_peered":"2023-12-19T14:39:06.244567+0000","last_unstale":"2023-12-19T14:39:11.540233+0000","last_undegraded":"2023-12-19T14:39:11.540233+0000","last_fullsized":"2023-12-19T14:39:11.540233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.587736+0000","last_change":"2023-12-19T14:39:11.585814+0000","last_active":"2023-12-19T14:39:11.587736+0000","last_peered":"2023-12-19T14:39:11.587736+0000","last_clean":"2023-12-19T14:39:11.587736+0000","last_became_active":"2023-12-19T14:39:11.581875+0000","last_became_peered":"2023-12-19T14:39:11.581875+0000","last_unstale":"2023-12-19T14:39:11.587736+0000","last_undegraded":"2023-12-19T14:39:11.587736+0000","last_fullsized":"2023-12-19T14:39:11.587736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547078+0000","last_change":"2023-12-19T14:39:07.276433+0000","last_active":"2023-12-19T14:39:11.547078+0000","last_peered":"2023-12-19T14:39:11.547078+0000","last_clean":"2023-12-19T14:39:11.547078+0000","last_became_active":"2023-12-19T14:39:07.274815+0000","last_became_peered":"2023-12-19T14:39:07.274815+0000","last_unstale":"2023-12-19T14:39:11.547078+0000","last_undegraded":"2023-12-19T14:39:11.547078+0000","last_fullsized":"2023-12-19T14:39:11.547078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:57.360165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569740+0000","last_change":"2023-12-19T14:39:06.262949+0000","last_active":"2023-12-19T14:39:11.569740+0000","last_peered":"2023-12-19T14:39:11.569740+0000","last_clean":"2023-12-19T14:39:11.569740+0000","last_became_active":"2023-12-19T14:39:06.262630+0000","last_became_peered":"2023-12-19T14:39:06.262630+0000","last_unstale":"2023-12-19T14:39:11.569740+0000","last_undegraded":"2023-12-19T14:39:11.569740+0000","last_fullsized":"2023-12-19T14:39:11.569740+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580388+0000","last_change":"2023-12-19T14:39:11.572679+0000","last_active":"2023-12-19T14:39:11.580388+0000","last_peered":"2023-12-19T14:39:11.580388+0000","last_clean":"2023-12-19T14:39:11.580388+0000","last_became_active":"2023-12-19T14:39:11.572334+0000","last_became_peered":"2023-12-19T14:39:11.572334+0000","last_unstale":"2023-12-19T14:39:11.580388+0000","last_undegraded":"2023-12-19T14:39:11.580388+0000","last_fullsized":"2023-12-19T14:39:11.580388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:00:02.733099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.562034+0000","last_change":"2023-12-19T14:39:07.287397+0000","last_active":"2023-12-19T14:39:11.562034+0000","last_peered":"2023-12-19T14:39:11.562034+0000","last_clean":"2023-12-19T14:39:11.562034+0000","last_became_active":"2023-12-19T14:39:07.287079+0000","last_became_peered":"2023-12-19T14:39:07.287079+0000","last_unstale":"2023-12-19T14:39:11.562034+0000","last_undegraded":"2023-12-19T14:39:11.562034+0000","last_fullsized":"2023-12-19T14:39:11.562034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551941+0000","last_change":"2023-12-19T14:39:06.259853+0000","last_active":"2023-12-19T14:39:11.551941+0000","last_peered":"2023-12-19T14:39:11.551941+0000","last_clean":"2023-12-19T14:39:11.551941+0000","last_became_active":"2023-12-19T14:39:06.259573+0000","last_became_peered":"2023-12-19T14:39:06.259573+0000","last_unstale":"2023-12-19T14:39:11.551941+0000","last_undegraded":"2023-12-19T14:39:11.551941+0000","last_fullsized":"2023-12-19T14:39:11.551941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583774+0000","last_change":"2023-12-19T14:39:11.578284+0000","last_active":"2023-12-19T14:39:11.583774+0000","last_peered":"2023-12-19T14:39:11.583774+0000","last_clean":"2023-12-19T14:39:11.583774+0000","last_became_active":"2023-12-19T14:39:11.577805+0000","last_became_peered":"2023-12-19T14:39:11.577805+0000","last_unstale":"2023-12-19T14:39:11.583774+0000","last_undegraded":"2023-12-19T14:39:11.583774+0000","last_fullsized":"2023-12-19T14:39:11.583774+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.564518+0000","last_change":"2023-12-19T14:39:07.288511+0000","last_active":"2023-12-19T14:39:11.564518+0000","last_peered":"2023-12-19T14:39:11.564518+0000","last_clean":"2023-12-19T14:39:11.564518+0000","last_became_active":"2023-12-19T14:39:07.288221+0000","last_became_peered":"2023-12-19T14:39:07.288221+0000","last_unstale":"2023-12-19T14:39:11.564518+0000","last_undegraded":"2023-12-19T14:39:11.564518+0000","last_fullsized":"2023-12-19T14:39:11.564518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.728724+0000","last_change":"2023-12-19T14:39:06.260565+0000","last_active":"2023-12-19T14:39:11.728724+0000","last_peered":"2023-12-19T14:39:11.728724+0000","last_clean":"2023-12-19T14:39:11.728724+0000","last_became_active":"2023-12-19T14:39:06.260244+0000","last_became_peered":"2023-12-19T14:39:06.260244+0000","last_unstale":"2023-12-19T14:39:11.728724+0000","last_undegraded":"2023-12-19T14:39:11.728724+0000","last_fullsized":"2023-12-19T14:39:11.728724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588612+0000","last_change":"2023-12-19T14:39:11.583334+0000","last_active":"2023-12-19T14:39:11.588612+0000","last_peered":"2023-12-19T14:39:11.588612+0000","last_clean":"2023-12-19T14:39:11.588612+0000","last_became_active":"2023-12-19T14:39:11.582906+0000","last_became_peered":"2023-12-19T14:39:11.582906+0000","last_unstale":"2023-12-19T14:39:11.588612+0000","last_undegraded":"2023-12-19T14:39:11.588612+0000","last_fullsized":"2023-12-19T14:39:11.588612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:50.106844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540825+0000","last_change":"2023-12-19T14:39:07.267435+0000","last_active":"2023-12-19T14:39:11.540825+0000","last_peered":"2023-12-19T14:39:11.540825+0000","last_clean":"2023-12-19T14:39:11.540825+0000","last_became_active":"2023-12-19T14:39:07.267113+0000","last_became_peered":"2023-12-19T14:39:07.267113+0000","last_unstale":"2023-12-19T14:39:11.540825+0000","last_undegraded":"2023-12-19T14:39:11.540825+0000","last_fullsized":"2023-12-19T14:39:11.540825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570841+0000","last_change":"2023-12-19T14:39:06.262808+0000","last_active":"2023-12-19T14:39:11.570841+0000","last_peered":"2023-12-19T14:39:11.570841+0000","last_clean":"2023-12-19T14:39:11.570841+0000","last_became_active":"2023-12-19T14:39:06.262439+0000","last_became_peered":"2023-12-19T14:39:06.262439+0000","last_unstale":"2023-12-19T14:39:11.570841+0000","last_undegraded":"2023-12-19T14:39:11.570841+0000","last_fullsized":"2023-12-19T14:39:11.570841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583520+0000","last_change":"2023-12-19T14:39:11.575768+0000","last_active":"2023-12-19T14:39:11.583520+0000","last_peered":"2023-12-19T14:39:11.583520+0000","last_clean":"2023-12-19T14:39:11.583520+0000","last_became_active":"2023-12-19T14:39:11.575222+0000","last_became_peered":"2023-12-19T14:39:11.575222+0000","last_unstale":"2023-12-19T14:39:11.583520+0000","last_undegraded":"2023-12-19T14:39:11.583520+0000","last_fullsized":"2023-12-19T14:39:11.583520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548697+0000","last_change":"2023-12-19T14:39:07.291010+0000","last_active":"2023-12-19T14:39:11.548697+0000","last_peered":"2023-12-19T14:39:11.548697+0000","last_clean":"2023-12-19T14:39:11.548697+0000","last_became_active":"2023-12-19T14:39:07.290719+0000","last_became_peered":"2023-12-19T14:39:07.290719+0000","last_unstale":"2023-12-19T14:39:11.548697+0000","last_undegraded":"2023-12-19T14:39:11.548697+0000","last_fullsized":"2023-12-19T14:39:11.548697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540846+0000","last_change":"2023-12-19T14:39:06.252011+0000","last_active":"2023-12-19T14:39:11.540846+0000","last_peered":"2023-12-19T14:39:11.540846+0000","last_clean":"2023-12-19T14:39:11.540846+0000","last_became_active":"2023-12-19T14:39:06.251724+0000","last_became_peered":"2023-12-19T14:39:06.251724+0000","last_unstale":"2023-12-19T14:39:11.540846+0000","last_undegraded":"2023-12-19T14:39:11.540846+0000","last_fullsized":"2023-12-19T14:39:11.540846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573312+0000","last_change":"2023-12-19T14:39:11.562285+0000","last_active":"2023-12-19T14:39:11.573312+0000","last_peered":"2023-12-19T14:39:11.573312+0000","last_clean":"2023-12-19T14:39:11.573312+0000","last_became_active":"2023-12-19T14:39:11.561826+0000","last_became_peered":"2023-12-19T14:39:11.561826+0000","last_unstale":"2023-12-19T14:39:11.573312+0000","last_undegraded":"2023-12-19T14:39:11.573312+0000","last_fullsized":"2023-12-19T14:39:11.573312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:29.623763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546810+0000","last_change":"2023-12-19T14:39:07.289173+0000","last_active":"2023-12-19T14:39:11.546810+0000","last_peered":"2023-12-19T14:39:11.546810+0000","last_clean":"2023-12-19T14:39:11.546810+0000","last_became_active":"2023-12-19T14:39:07.288745+0000","last_became_peered":"2023-12-19T14:39:07.288745+0000","last_unstale":"2023-12-19T14:39:11.546810+0000","last_undegraded":"2023-12-19T14:39:11.546810+0000","last_fullsized":"2023-12-19T14:39:11.546810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545170+0000","last_change":"2023-12-19T14:39:06.258745+0000","last_active":"2023-12-19T14:39:11.545170+0000","last_peered":"2023-12-19T14:39:11.545170+0000","last_clean":"2023-12-19T14:39:11.545170+0000","last_became_active":"2023-12-19T14:39:06.257007+0000","last_became_peered":"2023-12-19T14:39:06.257007+0000","last_unstale":"2023-12-19T14:39:11.545170+0000","last_undegraded":"2023-12-19T14:39:11.545170+0000","last_fullsized":"2023-12-19T14:39:11.545170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571904+0000","last_change":"2023-12-19T14:39:11.568661+0000","last_active":"2023-12-19T14:39:11.571904+0000","last_peered":"2023-12-19T14:39:11.571904+0000","last_clean":"2023-12-19T14:39:11.571904+0000","last_became_active":"2023-12-19T14:39:11.567940+0000","last_became_peered":"2023-12-19T14:39:11.567940+0000","last_unstale":"2023-12-19T14:39:11.571904+0000","last_undegraded":"2023-12-19T14:39:11.571904+0000","last_fullsized":"2023-12-19T14:39:11.571904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539181+0000","last_change":"2023-12-19T14:39:07.269639+0000","last_active":"2023-12-19T14:39:11.539181+0000","last_peered":"2023-12-19T14:39:11.539181+0000","last_clean":"2023-12-19T14:39:11.539181+0000","last_became_active":"2023-12-19T14:39:07.269276+0000","last_became_peered":"2023-12-19T14:39:07.269276+0000","last_unstale":"2023-12-19T14:39:11.539181+0000","last_undegraded":"2023-12-19T14:39:11.539181+0000","last_fullsized":"2023-12-19T14:39:11.539181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553025+0000","last_change":"2023-12-19T14:39:06.254091+0000","last_active":"2023-12-19T14:39:11.553025+0000","last_peered":"2023-12-19T14:39:11.553025+0000","last_clean":"2023-12-19T14:39:11.553025+0000","last_became_active":"2023-12-19T14:39:06.253444+0000","last_became_peered":"2023-12-19T14:39:06.253444+0000","last_unstale":"2023-12-19T14:39:11.553025+0000","last_undegraded":"2023-12-19T14:39:11.553025+0000","last_fullsized":"2023-12-19T14:39:11.553025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584042+0000","last_change":"2023-12-19T14:39:11.574535+0000","last_active":"2023-12-19T14:39:11.584042+0000","last_peered":"2023-12-19T14:39:11.584042+0000","last_clean":"2023-12-19T14:39:11.584042+0000","last_became_active":"2023-12-19T14:39:11.574060+0000","last_became_peered":"2023-12-19T14:39:11.574060+0000","last_unstale":"2023-12-19T14:39:11.584042+0000","last_undegraded":"2023-12-19T14:39:11.584042+0000","last_fullsized":"2023-12-19T14:39:11.584042+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.785232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556888+0000","last_change":"2023-12-19T14:39:07.287233+0000","last_active":"2023-12-19T14:39:11.556888+0000","last_peered":"2023-12-19T14:39:11.556888+0000","last_clean":"2023-12-19T14:39:11.556888+0000","last_became_active":"2023-12-19T14:39:07.286769+0000","last_became_peered":"2023-12-19T14:39:07.286769+0000","last_unstale":"2023-12-19T14:39:11.556888+0000","last_undegraded":"2023-12-19T14:39:11.556888+0000","last_fullsized":"2023-12-19T14:39:11.556888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569853+0000","last_change":"2023-12-19T14:39:06.251774+0000","last_active":"2023-12-19T14:39:11.569853+0000","last_peered":"2023-12-19T14:39:11.569853+0000","last_clean":"2023-12-19T14:39:11.569853+0000","last_became_active":"2023-12-19T14:39:06.251528+0000","last_became_peered":"2023-12-19T14:39:06.251528+0000","last_unstale":"2023-12-19T14:39:11.569853+0000","last_undegraded":"2023-12-19T14:39:11.569853+0000","last_fullsized":"2023-12-19T14:39:11.569853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581699+0000","last_change":"2023-12-19T14:39:11.575777+0000","last_active":"2023-12-19T14:39:11.581699+0000","last_peered":"2023-12-19T14:39:11.581699+0000","last_clean":"2023-12-19T14:39:11.581699+0000","last_became_active":"2023-12-19T14:39:11.575283+0000","last_became_peered":"2023-12-19T14:39:11.575283+0000","last_unstale":"2023-12-19T14:39:11.581699+0000","last_undegraded":"2023-12-19T14:39:11.581699+0000","last_fullsized":"2023-12-19T14:39:11.581699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:54.006908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543534+0000","last_change":"2023-12-19T14:39:07.294894+0000","last_active":"2023-12-19T14:39:11.543534+0000","last_peered":"2023-12-19T14:39:11.543534+0000","last_clean":"2023-12-19T14:39:11.543534+0000","last_became_active":"2023-12-19T14:39:07.294640+0000","last_became_peered":"2023-12-19T14:39:07.294640+0000","last_unstale":"2023-12-19T14:39:11.543534+0000","last_undegraded":"2023-12-19T14:39:11.543534+0000","last_fullsized":"2023-12-19T14:39:11.543534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545405+0000","last_change":"2023-12-19T14:39:06.256452+0000","last_active":"2023-12-19T14:39:11.545405+0000","last_peered":"2023-12-19T14:39:11.545405+0000","last_clean":"2023-12-19T14:39:11.545405+0000","last_became_active":"2023-12-19T14:39:06.255327+0000","last_became_peered":"2023-12-19T14:39:06.255327+0000","last_unstale":"2023-12-19T14:39:11.545405+0000","last_undegraded":"2023-12-19T14:39:11.545405+0000","last_fullsized":"2023-12-19T14:39:11.545405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:34.197901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.569956+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:39:11.569956+0000","last_peered":"2023-12-19T14:39:11.569956+0000","last_clean":"2023-12-19T14:39:11.569956+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:39:11.569956+0000","last_undegraded":"2023-12-19T14:39:11.569956+0000","last_fullsized":"2023-12-19T14:39:11.569956+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582739+0000","last_change":"2023-12-19T14:39:11.578205+0000","last_active":"2023-12-19T14:39:11.582739+0000","last_peered":"2023-12-19T14:39:11.582739+0000","last_clean":"2023-12-19T14:39:11.582739+0000","last_became_active":"2023-12-19T14:39:11.577809+0000","last_became_peered":"2023-12-19T14:39:11.577809+0000","last_unstale":"2023-12-19T14:39:11.582739+0000","last_undegraded":"2023-12-19T14:39:11.582739+0000","last_fullsized":"2023-12-19T14:39:11.582739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:10.870899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570477+0000","last_change":"2023-12-19T14:39:07.283187+0000","last_active":"2023-12-19T14:39:11.570477+0000","last_peered":"2023-12-19T14:39:11.570477+0000","last_clean":"2023-12-19T14:39:11.570477+0000","last_became_active":"2023-12-19T14:39:07.282884+0000","last_became_peered":"2023-12-19T14:39:07.282884+0000","last_unstale":"2023-12-19T14:39:11.570477+0000","last_undegraded":"2023-12-19T14:39:11.570477+0000","last_fullsized":"2023-12-19T14:39:11.570477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.553094+0000","last_change":"2023-12-19T14:39:06.260560+0000","last_active":"2023-12-19T14:39:11.553094+0000","last_peered":"2023-12-19T14:39:11.553094+0000","last_clean":"2023-12-19T14:39:11.553094+0000","last_became_active":"2023-12-19T14:39:06.260257+0000","last_became_peered":"2023-12-19T14:39:06.260257+0000","last_unstale":"2023-12-19T14:39:11.553094+0000","last_undegraded":"2023-12-19T14:39:11.553094+0000","last_fullsized":"2023-12-19T14:39:11.553094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.580038+0000","last_change":"2023-12-19T14:39:11.575041+0000","last_active":"2023-12-19T14:39:11.580038+0000","last_peered":"2023-12-19T14:39:11.580038+0000","last_clean":"2023-12-19T14:39:11.580038+0000","last_became_active":"2023-12-19T14:39:11.574653+0000","last_became_peered":"2023-12-19T14:39:11.574653+0000","last_unstale":"2023-12-19T14:39:11.580038+0000","last_undegraded":"2023-12-19T14:39:11.580038+0000","last_fullsized":"2023-12-19T14:39:11.580038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:42:12.283648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545946+0000","last_change":"2023-12-19T14:39:07.291614+0000","last_active":"2023-12-19T14:39:11.545946+0000","last_peered":"2023-12-19T14:39:11.545946+0000","last_clean":"2023-12-19T14:39:11.545946+0000","last_became_active":"2023-12-19T14:39:07.291096+0000","last_became_peered":"2023-12-19T14:39:07.291096+0000","last_unstale":"2023-12-19T14:39:11.545946+0000","last_undegraded":"2023-12-19T14:39:11.545946+0000","last_fullsized":"2023-12-19T14:39:11.545946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540658+0000","last_change":"2023-12-19T14:39:06.245861+0000","last_active":"2023-12-19T14:39:11.540658+0000","last_peered":"2023-12-19T14:39:11.540658+0000","last_clean":"2023-12-19T14:39:11.540658+0000","last_became_active":"2023-12-19T14:39:06.245543+0000","last_became_peered":"2023-12-19T14:39:06.245543+0000","last_unstale":"2023-12-19T14:39:11.540658+0000","last_undegraded":"2023-12-19T14:39:11.540658+0000","last_fullsized":"2023-12-19T14:39:11.540658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:39.349391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.575504+0000","last_change":"2023-12-19T14:39:11.572562+0000","last_active":"2023-12-19T14:39:11.575504+0000","last_peered":"2023-12-19T14:39:11.575504+0000","last_clean":"2023-12-19T14:39:11.575504+0000","last_became_active":"2023-12-19T14:39:11.572159+0000","last_became_peered":"2023-12-19T14:39:11.572159+0000","last_unstale":"2023-12-19T14:39:11.575504+0000","last_undegraded":"2023-12-19T14:39:11.575504+0000","last_fullsized":"2023-12-19T14:39:11.575504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:51:12.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571036+0000","last_change":"2023-12-19T14:39:07.284702+0000","last_active":"2023-12-19T14:39:11.571036+0000","last_peered":"2023-12-19T14:39:11.571036+0000","last_clean":"2023-12-19T14:39:11.571036+0000","last_became_active":"2023-12-19T14:39:07.284258+0000","last_became_peered":"2023-12-19T14:39:07.284258+0000","last_unstale":"2023-12-19T14:39:11.571036+0000","last_undegraded":"2023-12-19T14:39:11.571036+0000","last_fullsized":"2023-12-19T14:39:11.571036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:45:18.801982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555155+0000","last_change":"2023-12-19T14:39:06.256120+0000","last_active":"2023-12-19T14:39:11.555155+0000","last_peered":"2023-12-19T14:39:11.555155+0000","last_clean":"2023-12-19T14:39:11.555155+0000","last_became_active":"2023-12-19T14:39:06.255187+0000","last_became_peered":"2023-12-19T14:39:06.255187+0000","last_unstale":"2023-12-19T14:39:11.555155+0000","last_undegraded":"2023-12-19T14:39:11.555155+0000","last_fullsized":"2023-12-19T14:39:11.555155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584352+0000","last_change":"2023-12-19T14:39:11.579495+0000","last_active":"2023-12-19T14:39:11.584352+0000","last_peered":"2023-12-19T14:39:11.584352+0000","last_clean":"2023-12-19T14:39:11.584352+0000","last_became_active":"2023-12-19T14:39:11.579259+0000","last_became_peered":"2023-12-19T14:39:11.579259+0000","last_unstale":"2023-12-19T14:39:11.584352+0000","last_undegraded":"2023-12-19T14:39:11.584352+0000","last_fullsized":"2023-12-19T14:39:11.584352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:27.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573192+0000","last_change":"2023-12-19T14:39:07.279364+0000","last_active":"2023-12-19T14:39:11.573192+0000","last_peered":"2023-12-19T14:39:11.573192+0000","last_clean":"2023-12-19T14:39:11.573192+0000","last_became_active":"2023-12-19T14:39:07.278999+0000","last_became_peered":"2023-12-19T14:39:07.278999+0000","last_unstale":"2023-12-19T14:39:11.573192+0000","last_undegraded":"2023-12-19T14:39:11.573192+0000","last_fullsized":"2023-12-19T14:39:11.573192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.547974+0000","last_change":"2023-12-19T14:39:06.262399+0000","last_active":"2023-12-19T14:39:11.547974+0000","last_peered":"2023-12-19T14:39:11.547974+0000","last_clean":"2023-12-19T14:39:11.547974+0000","last_became_active":"2023-12-19T14:39:06.261918+0000","last_became_peered":"2023-12-19T14:39:06.261918+0000","last_unstale":"2023-12-19T14:39:11.547974+0000","last_undegraded":"2023-12-19T14:39:11.547974+0000","last_fullsized":"2023-12-19T14:39:11.547974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:48:08.535588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583755+0000","last_change":"2023-12-19T14:39:11.577720+0000","last_active":"2023-12-19T14:39:11.583755+0000","last_peered":"2023-12-19T14:39:11.583755+0000","last_clean":"2023-12-19T14:39:11.583755+0000","last_became_active":"2023-12-19T14:39:11.577297+0000","last_became_peered":"2023-12-19T14:39:11.577297+0000","last_unstale":"2023-12-19T14:39:11.583755+0000","last_undegraded":"2023-12-19T14:39:11.583755+0000","last_fullsized":"2023-12-19T14:39:11.583755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:47.074994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570176+0000","last_change":"2023-12-19T14:39:07.291311+0000","last_active":"2023-12-19T14:39:11.570176+0000","last_peered":"2023-12-19T14:39:11.570176+0000","last_clean":"2023-12-19T14:39:11.570176+0000","last_became_active":"2023-12-19T14:39:07.290587+0000","last_became_peered":"2023-12-19T14:39:07.290587+0000","last_unstale":"2023-12-19T14:39:11.570176+0000","last_undegraded":"2023-12-19T14:39:11.570176+0000","last_fullsized":"2023-12-19T14:39:11.570176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.548490+0000","last_change":"2023-12-19T14:39:06.248952+0000","last_active":"2023-12-19T14:39:11.548490+0000","last_peered":"2023-12-19T14:39:11.548490+0000","last_clean":"2023-12-19T14:39:11.548490+0000","last_became_active":"2023-12-19T14:39:06.248504+0000","last_became_peered":"2023-12-19T14:39:06.248504+0000","last_unstale":"2023-12-19T14:39:11.548490+0000","last_undegraded":"2023-12-19T14:39:11.548490+0000","last_fullsized":"2023-12-19T14:39:11.548490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581722+0000","last_change":"2023-12-19T14:39:11.579028+0000","last_active":"2023-12-19T14:39:11.581722+0000","last_peered":"2023-12-19T14:39:11.581722+0000","last_clean":"2023-12-19T14:39:11.581722+0000","last_became_active":"2023-12-19T14:39:11.578386+0000","last_became_peered":"2023-12-19T14:39:11.578386+0000","last_unstale":"2023-12-19T14:39:11.581722+0000","last_undegraded":"2023-12-19T14:39:11.581722+0000","last_fullsized":"2023-12-19T14:39:11.581722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551739+0000","last_change":"2023-12-19T14:39:07.289550+0000","last_active":"2023-12-19T14:39:11.551739+0000","last_peered":"2023-12-19T14:39:11.551739+0000","last_clean":"2023-12-19T14:39:11.551739+0000","last_became_active":"2023-12-19T14:39:07.289242+0000","last_became_peered":"2023-12-19T14:39:07.289242+0000","last_unstale":"2023-12-19T14:39:11.551739+0000","last_undegraded":"2023-12-19T14:39:11.551739+0000","last_fullsized":"2023-12-19T14:39:11.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549347+0000","last_change":"2023-12-19T14:39:06.250752+0000","last_active":"2023-12-19T14:39:11.549347+0000","last_peered":"2023-12-19T14:39:11.549347+0000","last_clean":"2023-12-19T14:39:11.549347+0000","last_became_active":"2023-12-19T14:39:06.249438+0000","last_became_peered":"2023-12-19T14:39:06.249438+0000","last_unstale":"2023-12-19T14:39:11.549347+0000","last_undegraded":"2023-12-19T14:39:11.549347+0000","last_fullsized":"2023-12-19T14:39:11.549347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588513+0000","last_change":"2023-12-19T14:39:11.581267+0000","last_active":"2023-12-19T14:39:11.588513+0000","last_peered":"2023-12-19T14:39:11.588513+0000","last_clean":"2023-12-19T14:39:11.588513+0000","last_became_active":"2023-12-19T14:39:11.577498+0000","last_became_peered":"2023-12-19T14:39:11.577498+0000","last_unstale":"2023-12-19T14:39:11.588513+0000","last_undegraded":"2023-12-19T14:39:11.588513+0000","last_fullsized":"2023-12-19T14:39:11.588513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:16:19.999833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570611+0000","last_change":"2023-12-19T14:39:07.283321+0000","last_active":"2023-12-19T14:39:11.570611+0000","last_peered":"2023-12-19T14:39:11.570611+0000","last_clean":"2023-12-19T14:39:11.570611+0000","last_became_active":"2023-12-19T14:39:07.283017+0000","last_became_peered":"2023-12-19T14:39:07.283017+0000","last_unstale":"2023-12-19T14:39:11.570611+0000","last_undegraded":"2023-12-19T14:39:11.570611+0000","last_fullsized":"2023-12-19T14:39:11.570611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545060+0000","last_change":"2023-12-19T14:39:06.259282+0000","last_active":"2023-12-19T14:39:11.545060+0000","last_peered":"2023-12-19T14:39:11.545060+0000","last_clean":"2023-12-19T14:39:11.545060+0000","last_became_active":"2023-12-19T14:39:06.257316+0000","last_became_peered":"2023-12-19T14:39:06.257316+0000","last_unstale":"2023-12-19T14:39:11.545060+0000","last_undegraded":"2023-12-19T14:39:11.545060+0000","last_fullsized":"2023-12-19T14:39:11.545060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588628+0000","last_change":"2023-12-19T14:39:11.577972+0000","last_active":"2023-12-19T14:39:11.588628+0000","last_peered":"2023-12-19T14:39:11.588628+0000","last_clean":"2023-12-19T14:39:11.588628+0000","last_became_active":"2023-12-19T14:39:11.577529+0000","last_became_peered":"2023-12-19T14:39:11.577529+0000","last_unstale":"2023-12-19T14:39:11.588628+0000","last_undegraded":"2023-12-19T14:39:11.588628+0000","last_fullsized":"2023-12-19T14:39:11.588628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:19:59.360333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.557043+0000","last_change":"2023-12-19T14:39:07.289405+0000","last_active":"2023-12-19T14:39:11.557043+0000","last_peered":"2023-12-19T14:39:11.557043+0000","last_clean":"2023-12-19T14:39:11.557043+0000","last_became_active":"2023-12-19T14:39:07.288885+0000","last_became_peered":"2023-12-19T14:39:07.288885+0000","last_unstale":"2023-12-19T14:39:11.557043+0000","last_undegraded":"2023-12-19T14:39:11.557043+0000","last_fullsized":"2023-12-19T14:39:11.557043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559942+0000","last_change":"2023-12-19T14:39:06.256658+0000","last_active":"2023-12-19T14:39:11.559942+0000","last_peered":"2023-12-19T14:39:11.559942+0000","last_clean":"2023-12-19T14:39:11.559942+0000","last_became_active":"2023-12-19T14:39:06.255561+0000","last_became_peered":"2023-12-19T14:39:06.255561+0000","last_unstale":"2023-12-19T14:39:11.559942+0000","last_undegraded":"2023-12-19T14:39:11.559942+0000","last_fullsized":"2023-12-19T14:39:11.559942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.584461+0000","last_change":"2023-12-19T14:39:11.579961+0000","last_active":"2023-12-19T14:39:11.584461+0000","last_peered":"2023-12-19T14:39:11.584461+0000","last_clean":"2023-12-19T14:39:11.584461+0000","last_became_active":"2023-12-19T14:39:11.578069+0000","last_became_peered":"2023-12-19T14:39:11.578069+0000","last_unstale":"2023-12-19T14:39:11.584461+0000","last_undegraded":"2023-12-19T14:39:11.584461+0000","last_fullsized":"2023-12-19T14:39:11.584461+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570370+0000","last_change":"2023-12-19T14:39:07.288175+0000","last_active":"2023-12-19T14:39:11.570370+0000","last_peered":"2023-12-19T14:39:11.570370+0000","last_clean":"2023-12-19T14:39:11.570370+0000","last_became_active":"2023-12-19T14:39:07.287333+0000","last_became_peered":"2023-12-19T14:39:07.287333+0000","last_unstale":"2023-12-19T14:39:11.570370+0000","last_undegraded":"2023-12-19T14:39:11.570370+0000","last_fullsized":"2023-12-19T14:39:11.570370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T14:39:19.178278+0000","last_change":"2023-12-19T14:39:06.256219+0000","last_active":"2023-12-19T14:39:19.178278+0000","last_peered":"2023-12-19T14:39:19.178278+0000","last_clean":"2023-12-19T14:39:19.178278+0000","last_became_active":"2023-12-19T14:39:06.255840+0000","last_became_peered":"2023-12-19T14:39:06.255840+0000","last_unstale":"2023-12-19T14:39:19.178278+0000","last_undegraded":"2023-12-19T14:39:19.178278+0000","last_fullsized":"2023-12-19T14:39:19.178278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:26:16.254553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573334+0000","last_change":"2023-12-19T14:39:11.569520+0000","last_active":"2023-12-19T14:39:11.573334+0000","last_peered":"2023-12-19T14:39:11.573334+0000","last_clean":"2023-12-19T14:39:11.573334+0000","last_became_active":"2023-12-19T14:39:11.563543+0000","last_became_peered":"2023-12-19T14:39:11.563543+0000","last_unstale":"2023-12-19T14:39:11.573334+0000","last_undegraded":"2023-12-19T14:39:11.573334+0000","last_fullsized":"2023-12-19T14:39:11.573334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:23.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546941+0000","last_change":"2023-12-19T14:39:07.276126+0000","last_active":"2023-12-19T14:39:11.546941+0000","last_peered":"2023-12-19T14:39:11.546941+0000","last_clean":"2023-12-19T14:39:11.546941+0000","last_became_active":"2023-12-19T14:39:07.275186+0000","last_became_peered":"2023-12-19T14:39:07.275186+0000","last_unstale":"2023-12-19T14:39:11.546941+0000","last_undegraded":"2023-12-19T14:39:11.546941+0000","last_fullsized":"2023-12-19T14:39:11.546941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:40.362295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540550+0000","last_change":"2023-12-19T14:39:06.259793+0000","last_active":"2023-12-19T14:39:11.540550+0000","last_peered":"2023-12-19T14:39:11.540550+0000","last_clean":"2023-12-19T14:39:11.540550+0000","last_became_active":"2023-12-19T14:39:06.259474+0000","last_became_peered":"2023-12-19T14:39:06.259474+0000","last_unstale":"2023-12-19T14:39:11.540550+0000","last_undegraded":"2023-12-19T14:39:11.540550+0000","last_fullsized":"2023-12-19T14:39:11.540550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.573690+0000","last_change":"2023-12-19T14:39:11.567330+0000","last_active":"2023-12-19T14:39:11.573690+0000","last_peered":"2023-12-19T14:39:11.573690+0000","last_clean":"2023-12-19T14:39:11.573690+0000","last_became_active":"2023-12-19T14:39:11.566909+0000","last_became_peered":"2023-12-19T14:39:11.566909+0000","last_unstale":"2023-12-19T14:39:11.573690+0000","last_undegraded":"2023-12-19T14:39:11.573690+0000","last_fullsized":"2023-12-19T14:39:11.573690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:41:18.238432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.561959+0000","last_change":"2023-12-19T14:39:07.290356+0000","last_active":"2023-12-19T14:39:11.561959+0000","last_peered":"2023-12-19T14:39:11.561959+0000","last_clean":"2023-12-19T14:39:11.561959+0000","last_became_active":"2023-12-19T14:39:07.288476+0000","last_became_peered":"2023-12-19T14:39:07.288476+0000","last_unstale":"2023-12-19T14:39:11.561959+0000","last_undegraded":"2023-12-19T14:39:11.561959+0000","last_fullsized":"2023-12-19T14:39:11.561959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549225+0000","last_change":"2023-12-19T14:39:06.263437+0000","last_active":"2023-12-19T14:39:11.549225+0000","last_peered":"2023-12-19T14:39:11.549225+0000","last_clean":"2023-12-19T14:39:11.549225+0000","last_became_active":"2023-12-19T14:39:06.263146+0000","last_became_peered":"2023-12-19T14:39:06.263146+0000","last_unstale":"2023-12-19T14:39:11.549225+0000","last_undegraded":"2023-12-19T14:39:11.549225+0000","last_fullsized":"2023-12-19T14:39:11.549225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.582385+0000","last_change":"2023-12-19T14:39:11.569957+0000","last_active":"2023-12-19T14:39:11.582385+0000","last_peered":"2023-12-19T14:39:11.582385+0000","last_clean":"2023-12-19T14:39:11.582385+0000","last_became_active":"2023-12-19T14:39:11.568502+0000","last_became_peered":"2023-12-19T14:39:11.568502+0000","last_unstale":"2023-12-19T14:39:11.582385+0000","last_undegraded":"2023-12-19T14:39:11.582385+0000","last_fullsized":"2023-12-19T14:39:11.582385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:02:34.258771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549124+0000","last_change":"2023-12-19T14:39:07.284419+0000","last_active":"2023-12-19T14:39:11.549124+0000","last_peered":"2023-12-19T14:39:11.549124+0000","last_clean":"2023-12-19T14:39:11.549124+0000","last_became_active":"2023-12-19T14:39:07.283507+0000","last_became_peered":"2023-12-19T14:39:07.283507+0000","last_unstale":"2023-12-19T14:39:11.549124+0000","last_undegraded":"2023-12-19T14:39:11.549124+0000","last_fullsized":"2023-12-19T14:39:11.549124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:05.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551787+0000","last_change":"2023-12-19T14:39:06.261357+0000","last_active":"2023-12-19T14:39:11.551787+0000","last_peered":"2023-12-19T14:39:11.551787+0000","last_clean":"2023-12-19T14:39:11.551787+0000","last_became_active":"2023-12-19T14:39:06.260833+0000","last_became_peered":"2023-12-19T14:39:06.260833+0000","last_unstale":"2023-12-19T14:39:11.551787+0000","last_undegraded":"2023-12-19T14:39:11.551787+0000","last_fullsized":"2023-12-19T14:39:11.551787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:19.793375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.588646+0000","last_change":"2023-12-19T14:39:11.579827+0000","last_active":"2023-12-19T14:39:11.588646+0000","last_peered":"2023-12-19T14:39:11.588646+0000","last_clean":"2023-12-19T14:39:11.588646+0000","last_became_active":"2023-12-19T14:39:11.578792+0000","last_became_peered":"2023-12-19T14:39:11.578792+0000","last_unstale":"2023-12-19T14:39:11.588646+0000","last_undegraded":"2023-12-19T14:39:11.588646+0000","last_fullsized":"2023-12-19T14:39:11.588646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:14.098319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.583746+0000","last_change":"2023-12-19T14:39:11.581965+0000","last_active":"2023-12-19T14:39:11.583746+0000","last_peered":"2023-12-19T14:39:11.583746+0000","last_clean":"2023-12-19T14:39:11.583746+0000","last_became_active":"2023-12-19T14:39:11.581412+0000","last_became_peered":"2023-12-19T14:39:11.581412+0000","last_unstale":"2023-12-19T14:39:11.583746+0000","last_undegraded":"2023-12-19T14:39:11.583746+0000","last_fullsized":"2023-12-19T14:39:11.583746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:24.004297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.539409+0000","last_change":"2023-12-19T14:39:07.283081+0000","last_active":"2023-12-19T14:39:11.539409+0000","last_peered":"2023-12-19T14:39:11.539409+0000","last_clean":"2023-12-19T14:39:11.539409+0000","last_became_active":"2023-12-19T14:39:07.276260+0000","last_became_peered":"2023-12-19T14:39:07.276260+0000","last_unstale":"2023-12-19T14:39:11.539409+0000","last_undegraded":"2023-12-19T14:39:11.539409+0000","last_fullsized":"2023-12-19T14:39:11.539409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543780+0000","last_change":"2023-12-19T14:39:06.257897+0000","last_active":"2023-12-19T14:39:11.543780+0000","last_peered":"2023-12-19T14:39:11.543780+0000","last_clean":"2023-12-19T14:39:11.543780+0000","last_became_active":"2023-12-19T14:39:06.257281+0000","last_became_peered":"2023-12-19T14:39:06.257281+0000","last_unstale":"2023-12-19T14:39:11.543780+0000","last_undegraded":"2023-12-19T14:39:11.543780+0000","last_fullsized":"2023-12-19T14:39:11.543780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:46.959151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.571142+0000","last_change":"2023-12-19T14:39:11.566055+0000","last_active":"2023-12-19T14:39:11.571142+0000","last_peered":"2023-12-19T14:39:11.571142+0000","last_clean":"2023-12-19T14:39:11.571142+0000","last_became_active":"2023-12-19T14:39:11.565416+0000","last_became_peered":"2023-12-19T14:39:11.565416+0000","last_unstale":"2023-12-19T14:39:11.571142+0000","last_undegraded":"2023-12-19T14:39:11.571142+0000","last_fullsized":"2023-12-19T14:39:11.571142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:12:19.857166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545049+0000","last_change":"2023-12-19T14:39:07.290776+0000","last_active":"2023-12-19T14:39:11.545049+0000","last_peered":"2023-12-19T14:39:11.545049+0000","last_clean":"2023-12-19T14:39:11.545049+0000","last_became_active":"2023-12-19T14:39:07.290465+0000","last_became_peered":"2023-12-19T14:39:07.290465+0000","last_unstale":"2023-12-19T14:39:11.545049+0000","last_undegraded":"2023-12-19T14:39:11.545049+0000","last_fullsized":"2023-12-19T14:39:11.545049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543823+0000","last_change":"2023-12-19T14:39:06.259345+0000","last_active":"2023-12-19T14:39:11.543823+0000","last_peered":"2023-12-19T14:39:11.543823+0000","last_clean":"2023-12-19T14:39:11.543823+0000","last_became_active":"2023-12-19T14:39:06.259067+0000","last_became_peered":"2023-12-19T14:39:06.259067+0000","last_unstale":"2023-12-19T14:39:11.543823+0000","last_undegraded":"2023-12-19T14:39:11.543823+0000","last_fullsized":"2023-12-19T14:39:11.543823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:10.796650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550184+0000","last_change":"2023-12-19T14:39:06.260437+0000","last_active":"2023-12-19T14:39:11.550184+0000","last_peered":"2023-12-19T14:39:11.550184+0000","last_clean":"2023-12-19T14:39:11.550184+0000","last_became_active":"2023-12-19T14:39:06.260008+0000","last_became_peered":"2023-12-19T14:39:06.260008+0000","last_unstale":"2023-12-19T14:39:11.550184+0000","last_undegraded":"2023-12-19T14:39:11.550184+0000","last_fullsized":"2023-12-19T14:39:11.550184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:16:56.347261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559895+0000","last_change":"2023-12-19T14:39:07.288421+0000","last_active":"2023-12-19T14:39:11.559895+0000","last_peered":"2023-12-19T14:39:11.559895+0000","last_clean":"2023-12-19T14:39:11.559895+0000","last_became_active":"2023-12-19T14:39:07.287403+0000","last_became_peered":"2023-12-19T14:39:07.287403+0000","last_unstale":"2023-12-19T14:39:11.559895+0000","last_undegraded":"2023-12-19T14:39:11.559895+0000","last_fullsized":"2023-12-19T14:39:11.559895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.581278+0000","last_change":"2023-12-19T14:39:11.577684+0000","last_active":"2023-12-19T14:39:11.581278+0000","last_peered":"2023-12-19T14:39:11.581278+0000","last_clean":"2023-12-19T14:39:11.581278+0000","last_became_active":"2023-12-19T14:39:11.577068+0000","last_became_peered":"2023-12-19T14:39:11.577068+0000","last_unstale":"2023-12-19T14:39:11.581278+0000","last_undegraded":"2023-12-19T14:39:11.581278+0000","last_fullsized":"2023-12-19T14:39:11.581278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:42.373695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544048+0000","last_change":"2023-12-19T14:39:06.247049+0000","last_active":"2023-12-19T14:39:11.544048+0000","last_peered":"2023-12-19T14:39:11.544048+0000","last_clean":"2023-12-19T14:39:11.544048+0000","last_became_active":"2023-12-19T14:39:06.246716+0000","last_became_peered":"2023-12-19T14:39:06.246716+0000","last_unstale":"2023-12-19T14:39:11.544048+0000","last_undegraded":"2023-12-19T14:39:11.544048+0000","last_fullsized":"2023-12-19T14:39:11.544048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:05.199650+0000","last_clean_scrub_stamp":"2023-12-19T14:39:05.199650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:24.313811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546858+0000","last_change":"2023-12-19T14:39:07.266860+0000","last_active":"2023-12-19T14:39:11.546858+0000","last_peered":"2023-12-19T14:39:11.546858+0000","last_clean":"2023-12-19T14:39:11.546858+0000","last_became_active":"2023-12-19T14:39:07.266544+0000","last_became_peered":"2023-12-19T14:39:07.266544+0000","last_unstale":"2023-12-19T14:39:11.546858+0000","last_undegraded":"2023-12-19T14:39:11.546858+0000","last_fullsized":"2023-12-19T14:39:11.546858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:47.972055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.585215+0000","last_change":"2023-12-19T14:39:11.567865+0000","last_active":"2023-12-19T14:39:11.585215+0000","last_peered":"2023-12-19T14:39:11.585215+0000","last_clean":"2023-12-19T14:39:11.585215+0000","last_became_active":"2023-12-19T14:39:11.566466+0000","last_became_peered":"2023-12-19T14:39:11.566466+0000","last_unstale":"2023-12-19T14:39:11.585215+0000","last_undegraded":"2023-12-19T14:39:11.585215+0000","last_fullsized":"2023-12-19T14:39:11.585215+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:10.505263+0000","last_clean_scrub_stamp":"2023-12-19T14:39:10.505263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:11.949761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.551612+0000","last_change":"2023-12-19T14:39:07.289708+0000","last_active":"2023-12-19T14:39:11.551612+0000","last_peered":"2023-12-19T14:39:11.551612+0000","last_clean":"2023-12-19T14:39:11.551612+0000","last_became_active":"2023-12-19T14:39:07.289420+0000","last_became_peered":"2023-12-19T14:39:07.289420+0000","last_unstale":"2023-12-19T14:39:11.551612+0000","last_undegraded":"2023-12-19T14:39:11.551612+0000","last_fullsized":"2023-12-19T14:39:11.551612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.552204+0000","last_change":"2023-12-19T14:39:07.291613+0000","last_active":"2023-12-19T14:39:11.552204+0000","last_peered":"2023-12-19T14:39:11.552204+0000","last_clean":"2023-12-19T14:39:11.552204+0000","last_became_active":"2023-12-19T14:39:07.291277+0000","last_became_peered":"2023-12-19T14:39:07.291277+0000","last_unstale":"2023-12-19T14:39:11.552204+0000","last_undegraded":"2023-12-19T14:39:11.552204+0000","last_fullsized":"2023-12-19T14:39:11.552204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:23.914755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544957+0000","last_change":"2023-12-19T14:39:07.294351+0000","last_active":"2023-12-19T14:39:11.544957+0000","last_peered":"2023-12-19T14:39:11.544957+0000","last_clean":"2023-12-19T14:39:11.544957+0000","last_became_active":"2023-12-19T14:39:07.293414+0000","last_became_peered":"2023-12-19T14:39:07.293414+0000","last_unstale":"2023-12-19T14:39:11.544957+0000","last_undegraded":"2023-12-19T14:39:11.544957+0000","last_fullsized":"2023-12-19T14:39:11.544957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559855+0000","last_change":"2023-12-19T14:39:07.287045+0000","last_active":"2023-12-19T14:39:11.559855+0000","last_peered":"2023-12-19T14:39:11.559855+0000","last_clean":"2023-12-19T14:39:11.559855+0000","last_became_active":"2023-12-19T14:39:07.286711+0000","last_became_peered":"2023-12-19T14:39:07.286711+0000","last_unstale":"2023-12-19T14:39:11.559855+0000","last_undegraded":"2023-12-19T14:39:11.559855+0000","last_fullsized":"2023-12-19T14:39:11.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:30.169268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545892+0000","last_change":"2023-12-19T14:39:07.284424+0000","last_active":"2023-12-19T14:39:11.545892+0000","last_peered":"2023-12-19T14:39:11.545892+0000","last_clean":"2023-12-19T14:39:11.545892+0000","last_became_active":"2023-12-19T14:39:07.284121+0000","last_became_peered":"2023-12-19T14:39:07.284121+0000","last_unstale":"2023-12-19T14:39:11.545892+0000","last_undegraded":"2023-12-19T14:39:11.545892+0000","last_fullsized":"2023-12-19T14:39:11.545892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540559+0000","last_change":"2023-12-19T14:39:07.280745+0000","last_active":"2023-12-19T14:39:11.540559+0000","last_peered":"2023-12-19T14:39:11.540559+0000","last_clean":"2023-12-19T14:39:11.540559+0000","last_became_active":"2023-12-19T14:39:07.280443+0000","last_became_peered":"2023-12-19T14:39:07.280443+0000","last_unstale":"2023-12-19T14:39:11.540559+0000","last_undegraded":"2023-12-19T14:39:11.540559+0000","last_fullsized":"2023-12-19T14:39:11.540559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:23.663785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550784+0000","last_change":"2023-12-19T14:39:07.288049+0000","last_active":"2023-12-19T14:39:11.550784+0000","last_peered":"2023-12-19T14:39:11.550784+0000","last_clean":"2023-12-19T14:39:11.550784+0000","last_became_active":"2023-12-19T14:39:07.287758+0000","last_became_peered":"2023-12-19T14:39:07.287758+0000","last_unstale":"2023-12-19T14:39:11.550784+0000","last_undegraded":"2023-12-19T14:39:11.550784+0000","last_fullsized":"2023-12-19T14:39:11.550784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.558076+0000","last_change":"2023-12-19T14:39:07.284681+0000","last_active":"2023-12-19T14:39:11.558076+0000","last_peered":"2023-12-19T14:39:11.558076+0000","last_clean":"2023-12-19T14:39:11.558076+0000","last_became_active":"2023-12-19T14:39:07.283596+0000","last_became_peered":"2023-12-19T14:39:07.283596+0000","last_unstale":"2023-12-19T14:39:11.558076+0000","last_undegraded":"2023-12-19T14:39:11.558076+0000","last_fullsized":"2023-12-19T14:39:11.558076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545014+0000","last_change":"2023-12-19T14:39:07.293485+0000","last_active":"2023-12-19T14:39:11.545014+0000","last_peered":"2023-12-19T14:39:11.545014+0000","last_clean":"2023-12-19T14:39:11.545014+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.545014+0000","last_undegraded":"2023-12-19T14:39:11.545014+0000","last_fullsized":"2023-12-19T14:39:11.545014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543694+0000","last_change":"2023-12-19T14:39:07.283853+0000","last_active":"2023-12-19T14:39:11.543694+0000","last_peered":"2023-12-19T14:39:11.543694+0000","last_clean":"2023-12-19T14:39:11.543694+0000","last_became_active":"2023-12-19T14:39:07.283616+0000","last_became_peered":"2023-12-19T14:39:07.283616+0000","last_unstale":"2023-12-19T14:39:11.543694+0000","last_undegraded":"2023-12-19T14:39:11.543694+0000","last_fullsized":"2023-12-19T14:39:11.543694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544936+0000","last_change":"2023-12-19T14:39:07.294504+0000","last_active":"2023-12-19T14:39:11.544936+0000","last_peered":"2023-12-19T14:39:11.544936+0000","last_clean":"2023-12-19T14:39:11.544936+0000","last_became_active":"2023-12-19T14:39:07.293953+0000","last_became_peered":"2023-12-19T14:39:07.293953+0000","last_unstale":"2023-12-19T14:39:11.544936+0000","last_undegraded":"2023-12-19T14:39:11.544936+0000","last_fullsized":"2023-12-19T14:39:11.544936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:24.398982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.559798+0000","last_change":"2023-12-19T14:39:07.296762+0000","last_active":"2023-12-19T14:39:11.559798+0000","last_peered":"2023-12-19T14:39:11.559798+0000","last_clean":"2023-12-19T14:39:11.559798+0000","last_became_active":"2023-12-19T14:39:07.295928+0000","last_became_peered":"2023-12-19T14:39:07.295928+0000","last_unstale":"2023-12-19T14:39:11.559798+0000","last_undegraded":"2023-12-19T14:39:11.559798+0000","last_fullsized":"2023-12-19T14:39:11.559798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:57.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549583+0000","last_change":"2023-12-19T14:39:07.280149+0000","last_active":"2023-12-19T14:39:11.549583+0000","last_peered":"2023-12-19T14:39:11.549583+0000","last_clean":"2023-12-19T14:39:11.549583+0000","last_became_active":"2023-12-19T14:39:07.279877+0000","last_became_peered":"2023-12-19T14:39:07.279877+0000","last_unstale":"2023-12-19T14:39:11.549583+0000","last_undegraded":"2023-12-19T14:39:11.549583+0000","last_fullsized":"2023-12-19T14:39:11.549583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:59:49.003736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570730+0000","last_change":"2023-12-19T14:39:07.288397+0000","last_active":"2023-12-19T14:39:11.570730+0000","last_peered":"2023-12-19T14:39:11.570730+0000","last_clean":"2023-12-19T14:39:11.570730+0000","last_became_active":"2023-12-19T14:39:07.287749+0000","last_became_peered":"2023-12-19T14:39:07.287749+0000","last_unstale":"2023-12-19T14:39:11.570730+0000","last_undegraded":"2023-12-19T14:39:11.570730+0000","last_fullsized":"2023-12-19T14:39:11.570730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:38.080986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.555145+0000","last_change":"2023-12-19T14:39:07.276259+0000","last_active":"2023-12-19T14:39:11.555145+0000","last_peered":"2023-12-19T14:39:11.555145+0000","last_clean":"2023-12-19T14:39:11.555145+0000","last_became_active":"2023-12-19T14:39:07.275951+0000","last_became_peered":"2023-12-19T14:39:07.275951+0000","last_unstale":"2023-12-19T14:39:11.555145+0000","last_undegraded":"2023-12-19T14:39:11.555145+0000","last_fullsized":"2023-12-19T14:39:11.555145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:43:43.990879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549564+0000","last_change":"2023-12-19T14:39:07.295581+0000","last_active":"2023-12-19T14:39:11.549564+0000","last_peered":"2023-12-19T14:39:11.549564+0000","last_clean":"2023-12-19T14:39:11.549564+0000","last_became_active":"2023-12-19T14:39:07.293567+0000","last_became_peered":"2023-12-19T14:39:07.293567+0000","last_unstale":"2023-12-19T14:39:11.549564+0000","last_undegraded":"2023-12-19T14:39:11.549564+0000","last_fullsized":"2023-12-19T14:39:11.549564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:13.763805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543139+0000","last_change":"2023-12-19T14:39:07.293875+0000","last_active":"2023-12-19T14:39:11.543139+0000","last_peered":"2023-12-19T14:39:11.543139+0000","last_clean":"2023-12-19T14:39:11.543139+0000","last_became_active":"2023-12-19T14:39:07.293381+0000","last_became_peered":"2023-12-19T14:39:07.293381+0000","last_unstale":"2023-12-19T14:39:11.543139+0000","last_undegraded":"2023-12-19T14:39:11.543139+0000","last_fullsized":"2023-12-19T14:39:11.543139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540124+0000","last_change":"2023-12-19T14:39:07.261761+0000","last_active":"2023-12-19T14:39:11.540124+0000","last_peered":"2023-12-19T14:39:11.540124+0000","last_clean":"2023-12-19T14:39:11.540124+0000","last_became_active":"2023-12-19T14:39:07.261446+0000","last_became_peered":"2023-12-19T14:39:07.261446+0000","last_unstale":"2023-12-19T14:39:11.540124+0000","last_undegraded":"2023-12-19T14:39:11.540124+0000","last_fullsized":"2023-12-19T14:39:11.540124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.556947+0000","last_change":"2023-12-19T14:39:07.290139+0000","last_active":"2023-12-19T14:39:11.556947+0000","last_peered":"2023-12-19T14:39:11.556947+0000","last_clean":"2023-12-19T14:39:11.556947+0000","last_became_active":"2023-12-19T14:39:07.289843+0000","last_became_peered":"2023-12-19T14:39:07.289843+0000","last_unstale":"2023-12-19T14:39:11.556947+0000","last_undegraded":"2023-12-19T14:39:11.556947+0000","last_fullsized":"2023-12-19T14:39:11.556947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:02.957567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.543821+0000","last_change":"2023-12-19T14:39:07.281738+0000","last_active":"2023-12-19T14:39:11.543821+0000","last_peered":"2023-12-19T14:39:11.543821+0000","last_clean":"2023-12-19T14:39:11.543821+0000","last_became_active":"2023-12-19T14:39:07.281443+0000","last_became_peered":"2023-12-19T14:39:07.281443+0000","last_unstale":"2023-12-19T14:39:11.543821+0000","last_undegraded":"2023-12-19T14:39:11.543821+0000","last_fullsized":"2023-12-19T14:39:11.543821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:15.466043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.546820+0000","last_change":"2023-12-19T14:39:07.276674+0000","last_active":"2023-12-19T14:39:11.546820+0000","last_peered":"2023-12-19T14:39:11.546820+0000","last_clean":"2023-12-19T14:39:11.546820+0000","last_became_active":"2023-12-19T14:39:07.276368+0000","last_became_peered":"2023-12-19T14:39:07.276368+0000","last_unstale":"2023-12-19T14:39:11.546820+0000","last_undegraded":"2023-12-19T14:39:11.546820+0000","last_fullsized":"2023-12-19T14:39:11.546820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:35.210805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.549881+0000","last_change":"2023-12-19T14:39:07.281026+0000","last_active":"2023-12-19T14:39:11.549881+0000","last_peered":"2023-12-19T14:39:11.549881+0000","last_clean":"2023-12-19T14:39:11.549881+0000","last_became_active":"2023-12-19T14:39:07.280689+0000","last_became_peered":"2023-12-19T14:39:07.280689+0000","last_unstale":"2023-12-19T14:39:11.549881+0000","last_undegraded":"2023-12-19T14:39:11.549881+0000","last_fullsized":"2023-12-19T14:39:11.549881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:44.941356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.550701+0000","last_change":"2023-12-19T14:39:07.280698+0000","last_active":"2023-12-19T14:39:11.550701+0000","last_peered":"2023-12-19T14:39:11.550701+0000","last_clean":"2023-12-19T14:39:11.550701+0000","last_became_active":"2023-12-19T14:39:07.280406+0000","last_became_peered":"2023-12-19T14:39:07.280406+0000","last_unstale":"2023-12-19T14:39:11.550701+0000","last_undegraded":"2023-12-19T14:39:11.550701+0000","last_fullsized":"2023-12-19T14:39:11.550701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:05.263165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570926+0000","last_change":"2023-12-19T14:39:07.284712+0000","last_active":"2023-12-19T14:39:11.570926+0000","last_peered":"2023-12-19T14:39:11.570926+0000","last_clean":"2023-12-19T14:39:11.570926+0000","last_became_active":"2023-12-19T14:39:07.283598+0000","last_became_peered":"2023-12-19T14:39:07.283598+0000","last_unstale":"2023-12-19T14:39:11.570926+0000","last_undegraded":"2023-12-19T14:39:11.570926+0000","last_fullsized":"2023-12-19T14:39:11.570926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544886+0000","last_change":"2023-12-19T14:39:07.294169+0000","last_active":"2023-12-19T14:39:11.544886+0000","last_peered":"2023-12-19T14:39:11.544886+0000","last_clean":"2023-12-19T14:39:11.544886+0000","last_became_active":"2023-12-19T14:39:07.293746+0000","last_became_peered":"2023-12-19T14:39:07.293746+0000","last_unstale":"2023-12-19T14:39:11.544886+0000","last_undegraded":"2023-12-19T14:39:11.544886+0000","last_fullsized":"2023-12-19T14:39:11.544886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:01.301277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570207+0000","last_change":"2023-12-19T14:39:07.291661+0000","last_active":"2023-12-19T14:39:11.570207+0000","last_peered":"2023-12-19T14:39:11.570207+0000","last_clean":"2023-12-19T14:39:11.570207+0000","last_became_active":"2023-12-19T14:39:07.290941+0000","last_became_peered":"2023-12-19T14:39:07.290941+0000","last_unstale":"2023-12-19T14:39:11.570207+0000","last_undegraded":"2023-12-19T14:39:11.570207+0000","last_fullsized":"2023-12-19T14:39:11.570207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:34:24.416865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.544859+0000","last_change":"2023-12-19T14:39:07.297368+0000","last_active":"2023-12-19T14:39:11.544859+0000","last_peered":"2023-12-19T14:39:11.544859+0000","last_clean":"2023-12-19T14:39:11.544859+0000","last_became_active":"2023-12-19T14:39:07.297082+0000","last_became_peered":"2023-12-19T14:39:07.297082+0000","last_unstale":"2023-12-19T14:39:11.544859+0000","last_undegraded":"2023-12-19T14:39:11.544859+0000","last_fullsized":"2023-12-19T14:39:11.544859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:30:35.887696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570469+0000","last_change":"2023-12-19T14:39:07.287921+0000","last_active":"2023-12-19T14:39:11.570469+0000","last_peered":"2023-12-19T14:39:11.570469+0000","last_clean":"2023-12-19T14:39:11.570469+0000","last_became_active":"2023-12-19T14:39:07.287493+0000","last_became_peered":"2023-12-19T14:39:07.287493+0000","last_unstale":"2023-12-19T14:39:11.570469+0000","last_undegraded":"2023-12-19T14:39:11.570469+0000","last_fullsized":"2023-12-19T14:39:11.570469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:41.402499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.545732+0000","last_change":"2023-12-19T14:39:07.290935+0000","last_active":"2023-12-19T14:39:11.545732+0000","last_peered":"2023-12-19T14:39:11.545732+0000","last_clean":"2023-12-19T14:39:11.545732+0000","last_became_active":"2023-12-19T14:39:07.290614+0000","last_became_peered":"2023-12-19T14:39:07.290614+0000","last_unstale":"2023-12-19T14:39:11.545732+0000","last_undegraded":"2023-12-19T14:39:11.545732+0000","last_fullsized":"2023-12-19T14:39:11.545732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:20.471512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.560160+0000","last_change":"2023-12-19T14:39:07.287280+0000","last_active":"2023-12-19T14:39:11.560160+0000","last_peered":"2023-12-19T14:39:11.560160+0000","last_clean":"2023-12-19T14:39:11.560160+0000","last_became_active":"2023-12-19T14:39:07.286949+0000","last_became_peered":"2023-12-19T14:39:07.286949+0000","last_unstale":"2023-12-19T14:39:11.560160+0000","last_undegraded":"2023-12-19T14:39:11.560160+0000","last_fullsized":"2023-12-19T14:39:11.560160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:23.059305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.540585+0000","last_change":"2023-12-19T14:39:07.295118+0000","last_active":"2023-12-19T14:39:11.540585+0000","last_peered":"2023-12-19T14:39:11.540585+0000","last_clean":"2023-12-19T14:39:11.540585+0000","last_became_active":"2023-12-19T14:39:07.294282+0000","last_became_peered":"2023-12-19T14:39:07.294282+0000","last_unstale":"2023-12-19T14:39:11.540585+0000","last_undegraded":"2023-12-19T14:39:11.540585+0000","last_fullsized":"2023-12-19T14:39:11.540585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:10.352666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-19T14:39:11.570810+0000","last_change":"2023-12-19T14:39:07.283979+0000","last_active":"2023-12-19T14:39:11.570810+0000","last_peered":"2023-12-19T14:39:11.570810+0000","last_clean":"2023-12-19T14:39:11.570810+0000","last_became_active":"2023-12-19T14:39:07.280235+0000","last_became_peered":"2023-12-19T14:39:07.280235+0000","last_unstale":"2023-12-19T14:39:11.570810+0000","last_undegraded":"2023-12-19T14:39:11.570810+0000","last_fullsized":"2023-12-19T14:39:11.570810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:39:06.212554+0000","last_clean_scrub_stamp":"2023-12-19T14:39:06.212554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:44:24.428676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298120,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073720,"statfs":{"total":96636764160,"available":96331489280,"internally_reserved":0,"allocated":851968,"data_stored":690410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":258048,"data_stored":99578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":99614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":103201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301628,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070212,"statfs":{"total":96636764160,"available":96327897088,"internally_reserved":0,"allocated":249856,"data_stored":98477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":262144,"data_stored":99648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302220,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069620,"statfs":{"total":96636764160,"available":96327290880,"internally_reserved":0,"allocated":856064,"data_stored":689455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:39:28.146 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-19T14:39:28.146 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-19T14:39:28.146 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-19T14:39:28.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-19T14:39:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 Processor -- start 2023-12-19T14:39:28.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 -- start start 2023-12-19T14:39:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000c15b0 0x7f2a0015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a000aeba0 0x7f2a000680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a000c7190 con 0x7f2a0015d320 2023-12-19T14:39:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a000c1b50 con 0x7f2a000aeba0 2023-12-19T14:39:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f2a05acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000c1e30 con 0x7f2a000c15b0 2023-12-19T14:39:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f29fffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a000aeba0 0x7f2a000680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f29ff7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2a0015d320 0x7f2a0006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44664/0 (socket says 172.21.15.50:44664) 2023-12-19T14:39:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f29fffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a000aeba0 0x7f2a000680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59124/0 (socket says 172.21.15.50:59124) 2023-12-19T14:39:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.345+0000 7f29ff7fe700 1 -- 172.21.15.50:0/1693539343 learned_addr learned my addr 172.21.15.50:0/1693539343 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29f7fff700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000c15b0 0x7f2a0015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29fffff700 1 -- 172.21.15.50:0/1693539343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000c15b0 msgr2=0x7f2a0015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29fffff700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000c15b0 0x7f2a0015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29fffff700 1 -- 172.21.15.50:0/1693539343 >> v1:172.21.15.50:6789/0 conn(0x7f2a0015d320 legacy=0x7f2a0006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29fffff700 1 -- 172.21.15.50:0/1693539343 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a000c6e20 con 0x7f2a000aeba0 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29fffff700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000aeba0 0x7f2a000680f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009e60 tx=0x7f29f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376e1792dbe3c5a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29feffd700 1 -- 172.21.15.50:0/1693539343 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800e070 con 0x7f2a000aeba0 2023-12-19T14:39:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29feffd700 1 -- 172.21.15.50:0/1693539343 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800bc30 con 0x7f2a000aeba0 2023-12-19T14:39:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f29feffd700 1 -- 172.21.15.50:0/1693539343 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8012430 con 0x7f2a000aeba0 2023-12-19T14:39:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 -- 172.21.15.50:0/1693539343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000aeba0 msgr2=0x7f2a000680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000aeba0 0x7f2a000680f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009e60 tx=0x7f29f8009780 comp rx=0 tx=0).stop 2023-12-19T14:39:28.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 -- 172.21.15.50:0/1693539343 shutdown_connections 2023-12-19T14:39:28.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000aeba0 0x7f2a000680f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 --2- 172.21.15.50:0/1693539343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000c15b0 0x7f2a0015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 -- 172.21.15.50:0/1693539343 >> 172.21.15.50:0/1693539343 conn(0x7f2a000c8850 msgr2=0x7f2a000c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:28.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 -- 172.21.15.50:0/1693539343 shutdown_connections 2023-12-19T14:39:28.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.349+0000 7f2a05acd700 1 -- 172.21.15.50:0/1693539343 wait complete. 2023-12-19T14:39:28.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 Processor -- start 2023-12-19T14:39:28.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 -- start start 2023-12-19T14:39:28.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 0x7f2a00150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0015d320 0x7f2a0015fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00150d40 con 0x7f2a000c15b0 2023-12-19T14:39:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0015da20 con 0x7f2a0015d320 2023-12-19T14:39:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f2a05acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000afaf0 con 0x7f2a000aeba0 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59134/0 (socket says 172.21.15.50:59134) 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29f7fff700 1 -- 172.21.15.50:0/3448475081 learned_addr learned my addr 172.21.15.50:0/3448475081 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29ff7fe700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 0x7f2a00150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29fffff700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0015d320 0x7f2a0015fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29f7fff700 1 -- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 msgr2=0x7f2a00150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.353+0000 7f29f7fff700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 0x7f2a00150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f7fff700 1 -- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0015d320 msgr2=0x7f2a0015fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f7fff700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0015d320 0x7f2a0015fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f7fff700 1 -- 172.21.15.50:0/3448475081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f8009a20 con 0x7f2a000c15b0 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29ff7fe700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 0x7f2a00150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f7fff700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d890 tx=0x7f29e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e800b600 con 0x7f2a000c15b0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e800b760 con 0x7f2a000c15b0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e800bae0 con 0x7f2a000c15b0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00160610 con 0x7f2a000c15b0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000af4a0 con 0x7f2a000c15b0 2023-12-19T14:39:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.357+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a000afc80 con 0x7f2a000c15b0 2023-12-19T14:39:28.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.361+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f29e801e460 con 0x7f2a000c15b0 2023-12-19T14:39:28.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.361+0000 7f29f77fe700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 0x7f29e0062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.361+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f29e807d1b0 con 0x7f2a000c15b0 2023-12-19T14:39:28.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.361+0000 7f29ff7fe700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 0x7f29e0062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.365+0000 7f29ff7fe700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 0x7f29e0062ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29f0001820 tx=0x7f29f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.365+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f29e804d490 con 0x7f2a000c15b0 2023-12-19T14:39:28.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.573+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2a00071120 con 0x7f2a000c15b0 2023-12-19T14:39:28.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.573+0000 7f29f77fe700 1 -- 172.21.15.50:0/3448475081 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f29e8050ab0 con 0x7f2a000c15b0 2023-12-19T14:39:28.575 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:28.576 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-19T14:39:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 msgr2=0x7f29e0062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 0x7f29e0062ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29f0001820 tx=0x7f29f0009040 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 msgr2=0x7f2a000c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29e800d890 tx=0x7f29e8008040 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 shutdown_connections 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0015d320 0x7f2a0015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f29e0060620 0x7f29e0062ad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a000c15b0 0x7f2a000c2e90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 --2- 172.21.15.50:0/3448475081 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a000aeba0 0x7f2a00150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 >> 172.21.15.50:0/3448475081 conn(0x7f2a000c8850 msgr2=0x7f2a0006d620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 shutdown_connections 2023-12-19T14:39:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.577+0000 7f2a05acd700 1 -- 172.21.15.50:0/3448475081 wait complete. 2023-12-19T14:39:28.599 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-19T14:39:28.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:28.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.773+0000 7f025f185700 1 Processor -- start 2023-12-19T14:39:28.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.773+0000 7f025f185700 1 -- start start 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025f185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 0x7f02580ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025f185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580b0950 con 0x7f02580b0110 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025f185700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580ccb40 con 0x7f0258153290 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025f185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02580cd510 con 0x7f02580b04e0 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025cf21700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f02580b0110 0x7f025815a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44676/0 (socket says 172.21.15.50:44676) 2023-12-19T14:39:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f025cf21700 1 -- 172.21.15.50:0/3356614445 learned_addr learned my addr 172.21.15.50:0/3356614445 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f0257fff700 1 --2- 172.21.15.50:0/3356614445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 0x7f02580ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f02577fe700 1 -- 172.21.15.50:0/3356614445 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841133485 0 0) 0x7f02580b0950 con 0x7f02580b0110 2023-12-19T14:39:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f02577fe700 1 -- 172.21.15.50:0/3356614445 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0248003570 con 0x7f02580b0110 2023-12-19T14:39:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f0257fff700 1 -- 172.21.15.50:0/3356614445 >> v1:172.21.15.125:6789/0 conn(0x7f0258153290 legacy=0x7f02581536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f0257fff700 1 -- 172.21.15.50:0/3356614445 >> v1:172.21.15.50:6789/0 conn(0x7f02580b0110 legacy=0x7f025815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f0257fff700 1 -- 172.21.15.50:0/3356614445 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580cbf30 con 0x7f02580b04e0 2023-12-19T14:39:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f0257fff700 1 --2- 172.21.15.50:0/3356614445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 0x7f02580ccfd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0244009910 tx=0x7f02440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c5c9bf716962162 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f02577fe700 1 -- 172.21.15.50:0/3356614445 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024400b680 con 0x7f02580b04e0 2023-12-19T14:39:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f02577fe700 1 -- 172.21.15.50:0/3356614445 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024400b7e0 con 0x7f02580b04e0 2023-12-19T14:39:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.777+0000 7f02577fe700 1 -- 172.21.15.50:0/3356614445 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024400bba0 con 0x7f02580b04e0 2023-12-19T14:39:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 -- 172.21.15.50:0/3356614445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 msgr2=0x7f02580ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 --2- 172.21.15.50:0/3356614445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 0x7f02580ccfd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0244009910 tx=0x7f02440091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 -- 172.21.15.50:0/3356614445 shutdown_connections 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 --2- 172.21.15.50:0/3356614445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02580b04e0 0x7f02580ccfd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 -- 172.21.15.50:0/3356614445 >> 172.21.15.50:0/3356614445 conn(0x7f025806c7c0 msgr2=0x7f025806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 -- 172.21.15.50:0/3356614445 shutdown_connections 2023-12-19T14:39:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.781+0000 7f025f185700 1 -- 172.21.15.50:0/3356614445 wait complete. 2023-12-19T14:39:28.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 Processor -- start 2023-12-19T14:39:28.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 -- start start 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02580b04e0 0x7f02580b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0258153290 0x7f02580b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580b2f60 con 0x7f02580b0110 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580b3240 con 0x7f02580b04e0 2023-12-19T14:39:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025f185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0258159740 con 0x7f0258153290 2023-12-19T14:39:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59138/0 (socket says 172.21.15.50:59138) 2023-12-19T14:39:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 -- 172.21.15.50:0/3701892873 learned_addr learned my addr 172.21.15.50:0/3701892873 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f0257fff700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02580b04e0 0x7f02580b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025d722700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0258153290 0x7f02580b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 -- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0258153290 msgr2=0x7f02580b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0258153290 0x7f02580b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 -- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02580b04e0 msgr2=0x7f02580b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02580b04e0 0x7f02580b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.785+0000 7f025cf21700 1 -- 172.21.15.50:0/3701892873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244009420 con 0x7f02580b0110 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f025cf21700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f024c00c7c0 tx=0x7f024c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c005cd0 con 0x7f02580b0110 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c005e30 con 0x7f02580b0110 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c007320 con 0x7f02580b0110 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0258159a20 con 0x7f02580b0110 2023-12-19T14:39:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0258152f80 con 0x7f02580b0110 2023-12-19T14:39:28.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.789+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0258159f20 con 0x7f02580b0110 2023-12-19T14:39:28.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.793+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f024c01b3b0 con 0x7f02580b0110 2023-12-19T14:39:28.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.793+0000 7f0254ff9700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 0x7f023c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:28.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.793+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f024c07c1f0 con 0x7f02580b0110 2023-12-19T14:39:28.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.793+0000 7f0257fff700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 0x7f023c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:28.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.797+0000 7f0257fff700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 0x7f023c062a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02440093f0 tx=0x7f0244017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:28.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.801+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f024c04c450 con 0x7f02580b0110 2023-12-19T14:39:28.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.976+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02580b30a0 con 0x7f02580b0110 2023-12-19T14:39:28.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.976+0000 7f0254ff9700 1 -- 172.21.15.50:0/3701892873 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f02580b30a0 con 0x7f02580b0110 2023-12-19T14:39:28.981 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:28.981 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 msgr2=0x7f023c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 0x7f023c062a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02440093f0 tx=0x7f0244017000 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 msgr2=0x7f0258152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f024c00c7c0 tx=0x7f024c007e00 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 shutdown_connections 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02580b04e0 0x7f02580b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f023c060550 0x7f023c062a00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0258153290 0x7f02580b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 --2- 172.21.15.50:0/3701892873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02580b0110 0x7f0258152a40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 >> 172.21.15.50:0/3701892873 conn(0x7f025806c7c0 msgr2=0x7f02580c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 shutdown_connections 2023-12-19T14:39:28.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:28.980+0000 7f025f185700 1 -- 172.21.15.50:0/3701892873 wait complete. 2023-12-19T14:39:29.003 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:29.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.168+0000 7f18b880c700 1 Processor -- start 2023-12-19T14:39:29.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 -- start start 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f18b41595a0 0x7f18b4159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b406c850 con 0x7f18b4158c90 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b40cb650 con 0x7f18b41595a0 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b880c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b406b470 con 0x7f18b406aeb0 2023-12-19T14:39:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f18b41595a0 0x7f18b4159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b37fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f18b4158c90 0x7f18b40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40970/0 (socket says 172.21.15.50:40970) 2023-12-19T14:39:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b37fe700 1 -- 172.21.15.50:0/2009793713 learned_addr learned my addr 172.21.15.50:0/2009793713 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.172+0000 7f18b2ffd700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b27fc700 1 -- 172.21.15.50:0/2009793713 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806294934 0 0) 0x7f18b406c850 con 0x7f18b4158c90 2023-12-19T14:39:29.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b27fc700 1 -- 172.21.15.50:0/2009793713 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f189c003570 con 0x7f18b4158c90 2023-12-19T14:39:29.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b3fff700 1 -- 172.21.15.50:0/2009793713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 msgr2=0x7f18b4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:29.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b3fff700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b3fff700 1 -- 172.21.15.50:0/2009793713 >> v1:172.21.15.125:6789/0 conn(0x7f18b4158c90 legacy=0x7f18b40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b3fff700 1 -- 172.21.15.50:0/2009793713 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b406b750 con 0x7f18b41595a0 2023-12-19T14:39:29.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b3fff700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4159a20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0098e0 tx=0x7f18ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3c418628f4df26 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b27fc700 1 -- 172.21.15.50:0/2009793713 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac011630 con 0x7f18b41595a0 2023-12-19T14:39:29.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b27fc700 1 -- 172.21.15.50:0/2009793713 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac011790 con 0x7f18b41595a0 2023-12-19T14:39:29.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b27fc700 1 -- 172.21.15.50:0/2009793713 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac011b10 con 0x7f18b41595a0 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 -- 172.21.15.50:0/2009793713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 msgr2=0x7f18b4159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4159a20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0098e0 tx=0x7f18ac0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 -- 172.21.15.50:0/2009793713 shutdown_connections 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4159a20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 --2- 172.21.15.50:0/2009793713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 -- 172.21.15.50:0/2009793713 >> 172.21.15.50:0/2009793713 conn(0x7f18b4067bb0 msgr2=0x7f18b40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:29.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 -- 172.21.15.50:0/2009793713 shutdown_connections 2023-12-19T14:39:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.176+0000 7f18b880c700 1 -- 172.21.15.50:0/2009793713 wait complete. 2023-12-19T14:39:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.180+0000 7f18b880c700 1 Processor -- start 2023-12-19T14:39:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.180+0000 7f18b880c700 1 -- start start 2023-12-19T14:39:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18b4158c90 0x7f18b41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4155c70 con 0x7f18b41595a0 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4154dc0 con 0x7f18b4158c90 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4155060 con 0x7f18b406aeb0 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59156/0 (socket says 172.21.15.50:59156) 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 -- 172.21.15.50:0/2662337386 learned_addr learned my addr 172.21.15.50:0/2662337386 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b37fe700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b2ffd700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18b4158c90 0x7f18b41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 -- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 msgr2=0x7f18b4156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 -- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18b4158c90 msgr2=0x7f18b41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18b4158c90 0x7f18b41571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 -- 172.21.15.50:0/2662337386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac009420 con 0x7f18b41595a0 2023-12-19T14:39:29.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b3fff700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18b4069b70 tx=0x7f18ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac01a410 con 0x7f18b41595a0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac01a590 con 0x7f18b41595a0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b4155340 con 0x7f18b41595a0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac01a950 con 0x7f18b41595a0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.184+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b4155900 con 0x7f18b41595a0 2023-12-19T14:39:29.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.188+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4155de0 con 0x7f18b41595a0 2023-12-19T14:39:29.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.188+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f18ac01ad00 con 0x7f18b41595a0 2023-12-19T14:39:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.188+0000 7f189b7fe700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 0x7f18a0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.188+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f18ac07e4c0 con 0x7f18b41595a0 2023-12-19T14:39:29.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.192+0000 7f18b37fe700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 0x7f18a0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.192+0000 7f18b37fe700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 0x7f18a0062a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18a4005c50 tx=0x7f18a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.196+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18ac04e7a0 con 0x7f18b41595a0 2023-12-19T14:39:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.364+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18b404be40 con 0x7f18b41595a0 2023-12-19T14:39:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.364+0000 7f189b7fe700 1 -- 172.21.15.50:0/2662337386 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f18ac051dc0 con 0x7f18b41595a0 2023-12-19T14:39:29.369 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 msgr2=0x7f18a0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 0x7f18a0062a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18a4005c50 tx=0x7f18a4009040 comp rx=0 tx=0).stop 2023-12-19T14:39:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 msgr2=0x7f18b4154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18b4069b70 tx=0x7f18ac017000 comp rx=0 tx=0).stop 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 shutdown_connections 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18b4158c90 0x7f18b41571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a0060580 0x7f18a0062a30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18b41595a0 0x7f18b4154880 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 --2- 172.21.15.50:0/2662337386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18b406aeb0 0x7f18b4156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 >> 172.21.15.50:0/2662337386 conn(0x7f18b4067bb0 msgr2=0x7f18b41599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 shutdown_connections 2023-12-19T14:39:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.368+0000 7f18b880c700 1 -- 172.21.15.50:0/2662337386 wait complete. 2023-12-19T14:39:29.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:29.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.556+0000 7f0e044ae700 1 Processor -- start 2023-12-19T14:39:29.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.556+0000 7f0e044ae700 1 -- start start 2023-12-19T14:39:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e044ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc155420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e044ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0dfc06b090 0x7f0dfc06b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e044ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc0aff50 con 0x7f0dfc06cb90 2023-12-19T14:39:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e044ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc0c95b0 con 0x7f0dfc06b090 2023-12-19T14:39:29.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e044ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc06ba50 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e01a49700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0dfc06cb90 0x7f0dfc06da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44688/0 (socket says 172.21.15.50:44688) 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e01a49700 1 -- 172.21.15.50:0/1730890724 learned_addr learned my addr 172.21.15.50:0/1730890724 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 --2- 172.21.15.50:0/1730890724 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e02a4b700 1 --2- 172.21.15.50:0/1730890724 >> v2:172.21.15.125:3300/0 conn(0x7f0dfc06b090 0x7f0dfc06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803450362 0 0) 0x7f0dfc0aff50 con 0x7f0dfc06cb90 2023-12-19T14:39:29.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de8003570 con 0x7f0dfc06cb90 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 -- 172.21.15.50:0/1730890724 >> v2:172.21.15.125:3300/0 conn(0x7f0dfc06b090 msgr2=0x7f0dfc06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 --2- 172.21.15.50:0/1730890724 >> v2:172.21.15.125:3300/0 conn(0x7f0dfc06b090 0x7f0dfc06b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268509304 0 0) 0x7f0de8003570 con 0x7f0dfc06cb90 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 -- 172.21.15.50:0/1730890724 >> v1:172.21.15.50:6789/0 conn(0x7f0dfc06cb90 legacy=0x7f0dfc06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 -- 172.21.15.50:0/1730890724 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc0af380 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.560+0000 7f0e0224a700 1 --2- 172.21.15.50:0/1730890724 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc155420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0df80098e0 tx=0x7f0df80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab409f81928dea1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8011630 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df8011790 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e01248700 1 -- 172.21.15.50:0/1730890724 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df8011b30 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 -- 172.21.15.50:0/1730890724 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 msgr2=0x7f0dfc155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 --2- 172.21.15.50:0/1730890724 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc155420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0df80098e0 tx=0x7f0df80091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 -- 172.21.15.50:0/1730890724 shutdown_connections 2023-12-19T14:39:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 --2- 172.21.15.50:0/1730890724 >> v2:172.21.15.125:3300/0 conn(0x7f0dfc06b090 0x7f0dfc06b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 --2- 172.21.15.50:0/1730890724 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc155420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 -- 172.21.15.50:0/1730890724 >> 172.21.15.50:0/1730890724 conn(0x7f0dfc0cc580 msgr2=0x7f0dfc0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 -- 172.21.15.50:0/1730890724 shutdown_connections 2023-12-19T14:39:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.564+0000 7f0e044ae700 1 -- 172.21.15.50:0/1730890724 wait complete. 2023-12-19T14:39:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 Processor -- start 2023-12-19T14:39:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 -- start start 2023-12-19T14:39:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 0x7f0dfc0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc16c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc157dd0 con 0x7f0dfc06cb90 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc157af0 con 0x7f0dfc06b090 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.568+0000 7f0e044ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0a8aa0 con 0x7f0dfc06c7c0 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59158/0 (socket says 172.21.15.50:59158) 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 -- 172.21.15.50:0/3927936887 learned_addr learned my addr 172.21.15.50:0/3927936887 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e01a49700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc16c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e0224a700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 0x7f0dfc0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 -- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 msgr2=0x7f0dfc16c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc16c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 -- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 msgr2=0x7f0dfc0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 0x7f0dfc0c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 -- 172.21.15.50:0/3927936887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8009420 con 0x7f0dfc06cb90 2023-12-19T14:39:29.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e0224a700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 0x7f0dfc0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e01a49700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc16c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e02a4b700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009e90 tx=0x7f0df400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df400cc20 con 0x7f0dfc06cb90 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df400f040 con 0x7f0dfc06cb90 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4014480 con 0x7f0dfc06cb90 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc0a8d80 con 0x7f0dfc06cb90 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc156da0 con 0x7f0dfc06cb90 2023-12-19T14:39:29.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.572+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc0a8900 con 0x7f0dfc06cb90 2023-12-19T14:39:29.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.576+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0df40147f0 con 0x7f0dfc06cb90 2023-12-19T14:39:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.576+0000 7f0df27fc700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 0x7f0de0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.576+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0df407d7f0 con 0x7f0dfc06cb90 2023-12-19T14:39:29.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.576+0000 7f0e0224a700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 0x7f0de0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.580+0000 7f0e0224a700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 0x7f0de0062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0df8011760 tx=0x7f0df8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.584+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0dfc0a8900 con 0x7f0dfc06cb90 2023-12-19T14:39:29.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.764+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0dfc0a8900 con 0x7f0dfc06cb90 2023-12-19T14:39:29.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.764+0000 7f0df27fc700 1 -- 172.21.15.50:0/3927936887 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0dfc0a8900 con 0x7f0dfc06cb90 2023-12-19T14:39:29.769 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:29.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 msgr2=0x7f0de0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 0x7f0de0062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0df8011760 tx=0x7f0df8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:29.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 msgr2=0x7f0dfc16ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009e90 tx=0x7f0df400b040 comp rx=0 tx=0).stop 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 shutdown_connections 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0de0060550 0x7f0de0062a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0dfc06cb90 0x7f0dfc16ce30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0dfc06c7c0 0x7f0dfc16c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 --2- 172.21.15.50:0/3927936887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0dfc06b090 0x7f0dfc0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 >> 172.21.15.50:0/3927936887 conn(0x7f0dfc0cc580 msgr2=0x7f0dfc0c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 shutdown_connections 2023-12-19T14:39:29.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.768+0000 7f0e044ae700 1 -- 172.21.15.50:0/3927936887 wait complete. 2023-12-19T14:39:29.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:29.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.968+0000 7f3288d85700 1 Processor -- start 2023-12-19T14:39:29.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.968+0000 7f3288d85700 1 -- start start 2023-12-19T14:39:29.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3284158e20 0x7f32840ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 0x7f32840ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32840ca2c0 0x7f328406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840cbdf0 con 0x7f3284158e20 2023-12-19T14:39:29.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840cc0d0 con 0x7f32840cb970 2023-12-19T14:39:29.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3288d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840cc3b0 con 0x7f32840ca2c0 2023-12-19T14:39:29.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3284158e20 0x7f32840ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3284158e20 0x7f32840ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59164/0 (socket says 172.21.15.50:59164) 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 -- 172.21.15.50:0/1531682218 learned_addr learned my addr 172.21.15.50:0/1531682218 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3281d9b700 1 --2- 172.21.15.50:0/1531682218 >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 0x7f32840ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3282d9d700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32840ca2c0 0x7f328406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 -- 172.21.15.50:0/1531682218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32840ca2c0 msgr2=0x7f328406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32840ca2c0 0x7f328406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 -- 172.21.15.50:0/1531682218 >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 msgr2=0x7f32840ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 --2- 172.21.15.50:0/1531682218 >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 0x7f32840ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f328259c700 1 -- 172.21.15.50:0/1531682218 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840cc690 con 0x7f3284158e20 2023-12-19T14:39:29.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.972+0000 7f3281d9b700 1 --2- 172.21.15.50:0/1531682218 >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 0x7f32840ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:29.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f328259c700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f32840ce5c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f32740099b0 tx=0x7f3274009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dea7ed384b140c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f328159a700 1 -- 172.21.15.50:0/1531682218 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274009ea0 con 0x7f3284158e20 2023-12-19T14:39:29.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f328159a700 1 -- 172.21.15.50:0/1531682218 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327400b820 con 0x7f3284158e20 2023-12-19T14:39:29.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f328159a700 1 -- 172.21.15.50:0/1531682218 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400bbc0 con 0x7f3284158e20 2023-12-19T14:39:29.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 -- 172.21.15.50:0/1531682218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 msgr2=0x7f32840ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f32840ce5c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f32740099b0 tx=0x7f3274009250 comp rx=0 tx=0).stop 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 -- 172.21.15.50:0/1531682218 shutdown_connections 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32840ca2c0 0x7f328406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 --2- 172.21.15.50:0/1531682218 >> v2:172.21.15.125:3300/0 conn(0x7f32840cb970 0x7f32840ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 --2- 172.21.15.50:0/1531682218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f32840ce5c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 -- 172.21.15.50:0/1531682218 >> 172.21.15.50:0/1531682218 conn(0x7f3284067bb0 msgr2=0x7f32840d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:29.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 -- 172.21.15.50:0/1531682218 shutdown_connections 2023-12-19T14:39:29.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.976+0000 7f3288d85700 1 -- 172.21.15.50:0/1531682218 wait complete. 2023-12-19T14:39:29.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 Processor -- start 2023-12-19T14:39:29.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 -- start start 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32840ca2c0 0x7f328414dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 0x7f32840c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c9d10 con 0x7f3284158e20 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284158140 con 0x7f32840ca2c0 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3288d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b49e0 con 0x7f3284157cc0 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59180/0 (socket says 172.21.15.50:59180) 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 -- 172.21.15.50:0/2264020098 learned_addr learned my addr 172.21.15.50:0/2264020098 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3282d9d700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 0x7f32840c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f328259c700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32840ca2c0 0x7f328414dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 -- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 msgr2=0x7f32840c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 0x7f32840c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 -- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32840ca2c0 msgr2=0x7f328414dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32840ca2c0 0x7f328414dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3281d9b700 1 -- 172.21.15.50:0/2264020098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32740094f0 con 0x7f3284158e20 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.980+0000 7f3282d9d700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 0x7f32840c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f3281d9b700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f327800e8d0 tx=0x7f3278008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278013410 con 0x7f3284158e20 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278013570 con 0x7f3284158e20 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278013930 con 0x7f3284158e20 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32840b5990 con 0x7f3284158e20 2023-12-19T14:39:29.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32840b4390 con 0x7f3284158e20 2023-12-19T14:39:29.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32840b4b50 con 0x7f3284158e20 2023-12-19T14:39:29.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.984+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3278013ca0 con 0x7f3284158e20 2023-12-19T14:39:29.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.988+0000 7f32727fc700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 0x7f3264062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:29.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.988+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f327807c820 con 0x7f3284158e20 2023-12-19T14:39:29.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.988+0000 7f328259c700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 0x7f3264062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:29.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.988+0000 7f328259c700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 0x7f3264062a30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32740099b0 tx=0x7f3274017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:29.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:29.992+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3278050510 con 0x7f3284158e20 2023-12-19T14:39:30.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.164+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f32840b4730 con 0x7f3284158e20 2023-12-19T14:39:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.164+0000 7f32727fc700 1 -- 172.21.15.50:0/2264020098 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f32780500a0 con 0x7f3284158e20 2023-12-19T14:39:30.169 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 msgr2=0x7f3264062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 0x7f3264062a30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32740099b0 tx=0x7f3274017000 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 msgr2=0x7f3284157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f327800e8d0 tx=0x7f3278008040 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 shutdown_connections 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3264060580 0x7f3264062a30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3284157cc0 0x7f32840c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3284158e20 0x7f3284157780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 --2- 172.21.15.50:0/2264020098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32840ca2c0 0x7f328414dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.168+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 >> 172.21.15.50:0/2264020098 conn(0x7f3284067bb0 msgr2=0x7f328406b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.172+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 shutdown_connections 2023-12-19T14:39:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.172+0000 7f3288d85700 1 -- 172.21.15.50:0/2264020098 wait complete. 2023-12-19T14:39:30.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 Processor -- start 2023-12-19T14:39:30.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 -- start start 2023-12-19T14:39:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 0x7f3b340ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b340cac40 0x7f3b3406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b34159bc0 con 0x7f3b340c8bc0 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340cb130 con 0x7f3b340cac40 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.368+0000 7f3b39763700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340cb410 con 0x7f3b340c8f90 2023-12-19T14:39:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b33fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 0x7f3b340ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b38f62700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b340cac40 0x7f3b3406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b33fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59192/0 (socket says 172.21.15.50:59192) 2023-12-19T14:39:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b33fff700 1 -- 172.21.15.50:0/2097414787 learned_addr learned my addr 172.21.15.50:0/2097414787 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 -- 172.21.15.50:0/2097414787 >> v2:172.21.15.125:3300/0 conn(0x7f3b340cac40 msgr2=0x7f3b3406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 --2- 172.21.15.50:0/2097414787 >> v2:172.21.15.125:3300/0 conn(0x7f3b340cac40 0x7f3b3406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 -- 172.21.15.50:0/2097414787 >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 msgr2=0x7f3b3415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 --2- 172.21.15.50:0/2097414787 >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 -- 172.21.15.50:0/2097414787 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34159850 con 0x7f3b340c8f90 2023-12-19T14:39:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b33fff700 1 --2- 172.21.15.50:0/2097414787 >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b337fe700 1 --2- 172.21.15.50:0/2097414787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 0x7f3b340ca700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c3b90 tx=0x7f3b280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cf9916dd5911c3c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b32ffd700 1 -- 172.21.15.50:0/2097414787 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28011630 con 0x7f3b340c8f90 2023-12-19T14:39:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b32ffd700 1 -- 172.21.15.50:0/2097414787 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b280117b0 con 0x7f3b340c8f90 2023-12-19T14:39:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.372+0000 7f3b32ffd700 1 -- 172.21.15.50:0/2097414787 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28011b70 con 0x7f3b340c8f90 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- 172.21.15.50:0/2097414787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 msgr2=0x7f3b340ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 --2- 172.21.15.50:0/2097414787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 0x7f3b340ca700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c3b90 tx=0x7f3b280091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- 172.21.15.50:0/2097414787 shutdown_connections 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 --2- 172.21.15.50:0/2097414787 >> v2:172.21.15.125:3300/0 conn(0x7f3b340cac40 0x7f3b3406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 --2- 172.21.15.50:0/2097414787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340c8f90 0x7f3b340ca700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 --2- 172.21.15.50:0/2097414787 >> v2:172.21.15.50:3300/0 conn(0x7f3b340c8bc0 0x7f3b3415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- 172.21.15.50:0/2097414787 >> 172.21.15.50:0/2097414787 conn(0x7f3b340ccb30 msgr2=0x7f3b340ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- 172.21.15.50:0/2097414787 shutdown_connections 2023-12-19T14:39:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- 172.21.15.50:0/2097414787 wait complete. 2023-12-19T14:39:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 Processor -- start 2023-12-19T14:39:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.376+0000 7f3b39763700 1 -- start start 2023-12-19T14:39:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340c8f90 0x7f3b340a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 0x7f3b340aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340a9830 con 0x7f3b340c8bc0 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340ab4f0 con 0x7f3b340c8f90 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340ab980 con 0x7f3b340cac40 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59196/0 (socket says 172.21.15.50:59196) 2023-12-19T14:39:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 -- 172.21.15.50:0/3113316497 learned_addr learned my addr 172.21.15.50:0/3113316497 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b337fe700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340c8f90 0x7f3b340a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b38f62700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 0x7f3b340aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 -- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 msgr2=0x7f3b340aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 0x7f3b340aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 -- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340c8f90 msgr2=0x7f3b340a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340c8f90 0x7f3b340a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 -- 172.21.15.50:0/3113316497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240094f0 con 0x7f3b340c8bc0 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b38f62700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 0x7f3b340aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b33fff700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400ec50 tx=0x7f3b24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400c9d0 con 0x7f3b340c8bc0 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2400cb30 con 0x7f3b340c8bc0 2023-12-19T14:39:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28009420 con 0x7f3b340c8bc0 2023-12-19T14:39:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24013470 con 0x7f3b340c8bc0 2023-12-19T14:39:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.380+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b340aaea0 con 0x7f3b340c8bc0 2023-12-19T14:39:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.384+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3404be40 con 0x7f3b340c8bc0 2023-12-19T14:39:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.384+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3b24013860 con 0x7f3b340c8bc0 2023-12-19T14:39:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.384+0000 7f3b1bfff700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 0x7f3b20062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.388+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3b2407d290 con 0x7f3b340c8bc0 2023-12-19T14:39:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.388+0000 7f3b337fe700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 0x7f3b20062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.388+0000 7f3b337fe700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 0x7f3b20062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c3b90 tx=0x7f3b28017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.392+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3b2404d4f0 con 0x7f3b340c8bc0 2023-12-19T14:39:30.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.568+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3b34159520 con 0x7f3b340c8bc0 2023-12-19T14:39:30.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.568+0000 7f3b1bfff700 1 -- 172.21.15.50:0/3113316497 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3b24050b10 con 0x7f3b340c8bc0 2023-12-19T14:39:30.573 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 msgr2=0x7f3b20062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 0x7f3b20062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c3b90 tx=0x7f3b28017000 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 msgr2=0x7f3b34144b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400ec50 tx=0x7f3b24009250 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 shutdown_connections 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b340c8f90 0x7f3b340a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3b20060580 0x7f3b20062a30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3b340cac40 0x7f3b340aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 --2- 172.21.15.50:0/3113316497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b340c8bc0 0x7f3b34144b60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.572+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 >> 172.21.15.50:0/3113316497 conn(0x7f3b340ccb30 msgr2=0x7f3b340ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:30.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.576+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 shutdown_connections 2023-12-19T14:39:30.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.576+0000 7f3b39763700 1 -- 172.21.15.50:0/3113316497 wait complete. 2023-12-19T14:39:30.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.772+0000 7ff8eb523700 1 Processor -- start 2023-12-19T14:39:30.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8eb523700 1 -- start start 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8eb523700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff8e406a580 0x7ff8e406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8eb523700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e406c7c0 con 0x7ff8e406d210 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8eb523700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e406cd80 con 0x7ff8e40b1410 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8eb523700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e406af40 con 0x7ff8e406a580 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8ead22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff8e406d210 0x7ff8e406d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44692/0 (socket says 172.21.15.50:44692) 2023-12-19T14:39:30.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8ead22700 1 -- 172.21.15.50:0/824401782 learned_addr learned my addr 172.21.15.50:0/824401782 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8e9d20700 1 --2- 172.21.15.50:0/824401782 >> v2:172.21.15.125:3300/0 conn(0x7ff8e406a580 0x7ff8e406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659818087 0 0) 0x7ff8e406c7c0 con 0x7ff8e406d210 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d0003570 con 0x7ff8e406d210 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85545595 0 0) 0x7ff8e406cd80 con 0x7ff8e40b1410 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.776+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e406c7c0 con 0x7ff8e40b1410 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e9d20700 1 -- 172.21.15.50:0/824401782 >> v1:172.21.15.125:6789/0 conn(0x7ff8e40b1410 legacy=0x7ff8e40ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e9d20700 1 -- 172.21.15.50:0/824401782 >> v1:172.21.15.50:6789/0 conn(0x7ff8e406d210 legacy=0x7ff8e406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958342583 0 0) 0x7ff8d0003570 con 0x7ff8e406d210 2023-12-19T14:39:30.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e9d20700 1 -- 172.21.15.50:0/824401782 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e406b8f0 con 0x7ff8e406a580 2023-12-19T14:39:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e9d20700 1 --2- 172.21.15.50:0/824401782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e406a580 0x7ff8e406aa00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40098e0 tx=0x7ff8d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e4cbcf706e6448 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0017f0 con 0x7ff8e406a580 2023-12-19T14:39:30.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d4011420 con 0x7ff8e406a580 2023-12-19T14:39:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8e951f700 1 -- 172.21.15.50:0/824401782 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40117c0 con 0x7ff8e406a580 2023-12-19T14:39:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 -- 172.21.15.50:0/824401782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e406a580 msgr2=0x7ff8e406aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 --2- 172.21.15.50:0/824401782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e406a580 0x7ff8e406aa00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40098e0 tx=0x7ff8d40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 -- 172.21.15.50:0/824401782 shutdown_connections 2023-12-19T14:39:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 --2- 172.21.15.50:0/824401782 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e406a580 0x7ff8e406aa00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 -- 172.21.15.50:0/824401782 >> 172.21.15.50:0/824401782 conn(0x7ff8e4067bb0 msgr2=0x7ff8e40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 -- 172.21.15.50:0/824401782 shutdown_connections 2023-12-19T14:39:30.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.780+0000 7ff8eb523700 1 -- 172.21.15.50:0/824401782 wait complete. 2023-12-19T14:39:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 Processor -- start 2023-12-19T14:39:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 -- start start 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8e406a580 0x7ff8e41483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 0x7ff8e415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4158060 con 0x7ff8e406d210 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4157450 con 0x7ff8e40b1410 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8eb523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4157730 con 0x7ff8e406a580 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8e9d20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8e9d20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59198/0 (socket says 172.21.15.50:59198) 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8e9d20700 1 -- 172.21.15.50:0/3280914860 learned_addr learned my addr 172.21.15.50:0/3280914860 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:30.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8ead22700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 0x7ff8e415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.784+0000 7ff8ea521700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8e406a580 0x7ff8e41483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 -- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8e406a580 msgr2=0x7ff8e41483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8e406a580 0x7ff8e41483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 -- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 msgr2=0x7ff8e415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 0x7ff8e415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 -- 172.21.15.50:0/3280914860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4009420 con 0x7ff8e406d210 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8ead22700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 0x7ff8e415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8e9d20700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4064040 tx=0x7ff8d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d401cbe0 con 0x7ff8e406d210 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d401cd40 con 0x7ff8e406d210 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d401a6a0 con 0x7ff8e406d210 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e4157a10 con 0x7ff8e406d210 2023-12-19T14:39:30.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e4148910 con 0x7ff8e406d210 2023-12-19T14:39:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.788+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e404be40 con 0x7ff8e406d210 2023-12-19T14:39:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.792+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff8d401aa90 con 0x7ff8e406d210 2023-12-19T14:39:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.792+0000 7ff8da7fc700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 0x7ff8c8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.792+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff8d407e480 con 0x7ff8e406d210 2023-12-19T14:39:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.792+0000 7ff8ea521700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 0x7ff8c8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.796+0000 7ff8ea521700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 0x7ff8c8062a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0007fd0 tx=0x7ff8e0007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:30.800+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff8d404e6e0 con 0x7ff8e406d210 2023-12-19T14:39:31.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.008+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e415a3e0 con 0x7ff8e406d210 2023-12-19T14:39:31.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.008+0000 7ff8da7fc700 1 -- 172.21.15.50:0/3280914860 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7ff8d4051d00 con 0x7ff8e406d210 2023-12-19T14:39:31.012 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:31.012 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 msgr2=0x7ff8c8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 0x7ff8c8062a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0007fd0 tx=0x7ff8e0007d50 comp rx=0 tx=0).stop 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 msgr2=0x7ff8e415b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4064040 tx=0x7ff8d4016000 comp rx=0 tx=0).stop 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 shutdown_connections 2023-12-19T14:39:31.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e40b1410 0x7ff8e415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff8c8060550 0x7ff8c8062a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e406d210 0x7ff8e415b930 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 --2- 172.21.15.50:0/3280914860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8e406a580 0x7ff8e41483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 >> 172.21.15.50:0/3280914860 conn(0x7ff8e4067bb0 msgr2=0x7ff8e4156290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 shutdown_connections 2023-12-19T14:39:31.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.012+0000 7ff8eb523700 1 -- 172.21.15.50:0/3280914860 wait complete. 2023-12-19T14:39:31.017 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:39:31.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:31.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.200+0000 7f84efd45700 1 Processor -- start 2023-12-19T14:39:31.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.200+0000 7f84efd45700 1 -- start start 2023-12-19T14:39:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84efd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f84e8140ae0 0x7f84e80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84efd45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f84e8068630 0x7f84e80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84efd45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e806e150 con 0x7f84e8140710 2023-12-19T14:39:31.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84efd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e8068a70 con 0x7f84e8140ae0 2023-12-19T14:39:31.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84efd45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e806d540 con 0x7f84e8068630 2023-12-19T14:39:31.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f84e8140ae0 0x7f84e80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ee2e2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f84e8068630 0x7f84e80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f84e8140ae0 0x7f84e80680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59206/0 (socket says 172.21.15.50:59206) 2023-12-19T14:39:31.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 -- 172.21.15.50:0/2974465783 learned_addr learned my addr 172.21.15.50:0/2974465783 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:31.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84edae1700 1 --1- 172.21.15.50:0/2974465783 >> v1:172.21.15.125:6789/0 conn(0x7f84e8140710 0x7f84e80c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:40998/0 (socket says 172.21.15.50:40998) 2023-12-19T14:39:31.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 -- 172.21.15.50:0/2974465783 >> v2:172.21.15.125:3300/0 conn(0x7f84e8068630 msgr2=0x7f84e80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:31.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 --2- 172.21.15.50:0/2974465783 >> v2:172.21.15.125:3300/0 conn(0x7f84e8068630 0x7f84e80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 -- 172.21.15.50:0/2974465783 >> v1:172.21.15.125:6789/0 conn(0x7f84e8140710 legacy=0x7f84e80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 -- 172.21.15.50:0/2974465783 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e806d820 con 0x7f84e8140ae0 2023-12-19T14:39:31.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ed2e0700 1 --2- 172.21.15.50:0/2974465783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140ae0 0x7f84e80680f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84e806bfc0 tx=0x7f84d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8777e44d827dfd67 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ecadf700 1 -- 172.21.15.50:0/2974465783 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8009e70 con 0x7f84e8140ae0 2023-12-19T14:39:31.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ecadf700 1 -- 172.21.15.50:0/2974465783 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d800b860 con 0x7f84e8140ae0 2023-12-19T14:39:31.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.204+0000 7f84ecadf700 1 -- 172.21.15.50:0/2974465783 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800bc00 con 0x7f84e8140ae0 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 -- 172.21.15.50:0/2974465783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140ae0 msgr2=0x7f84e80680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 --2- 172.21.15.50:0/2974465783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140ae0 0x7f84e80680f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84e806bfc0 tx=0x7f84d8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 -- 172.21.15.50:0/2974465783 shutdown_connections 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 --2- 172.21.15.50:0/2974465783 >> v2:172.21.15.125:3300/0 conn(0x7f84e8068630 0x7f84e80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 --2- 172.21.15.50:0/2974465783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140ae0 0x7f84e80680f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 -- 172.21.15.50:0/2974465783 >> 172.21.15.50:0/2974465783 conn(0x7f84e80c8850 msgr2=0x7f84e80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:31.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 -- 172.21.15.50:0/2974465783 shutdown_connections 2023-12-19T14:39:31.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.208+0000 7f84efd45700 1 -- 172.21.15.50:0/2974465783 wait complete. 2023-12-19T14:39:31.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 Processor -- start 2023-12-19T14:39:31.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 -- start start 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e8068630 0x7f84e80aae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e8140ae0 0x7f84e80670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e81560b0 con 0x7f84e8140710 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80b07b0 con 0x7f84e8140ae0 2023-12-19T14:39:31.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84efd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80afec0 con 0x7f84e8068630 2023-12-19T14:39:31.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84edae1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e8068630 0x7f84e80aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59218/0 (socket says 172.21.15.50:59218) 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 -- 172.21.15.50:0/3117377710 learned_addr learned my addr 172.21.15.50:0/3117377710 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ee2e2700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e8140ae0 0x7f84e80670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 -- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e8068630 msgr2=0x7f84e80aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e8068630 0x7f84e80aae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 -- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e8140ae0 msgr2=0x7f84e80670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e8140ae0 0x7f84e80670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.212+0000 7f84ed2e0700 1 -- 172.21.15.50:0/3117377710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80094f0 con 0x7f84e8140710 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84ed2e0700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f84e806a280 tx=0x7f84d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800bbe0 con 0x7f84e8140710 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d8011420 con 0x7f84e8140710 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80117c0 con 0x7f84e8140710 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e80a99c0 con 0x7f84e8140710 2023-12-19T14:39:31.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80b0440 con 0x7f84e8140710 2023-12-19T14:39:31.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.216+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80b0960 con 0x7f84e8140710 2023-12-19T14:39:31.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.220+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f84d8011bf0 con 0x7f84e8140710 2023-12-19T14:39:31.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.220+0000 7f84ddffb700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 0x7f84cc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.220+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f84d807df60 con 0x7f84e8140710 2023-12-19T14:39:31.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.220+0000 7f84edae1700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 0x7f84cc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.224+0000 7f84edae1700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 0x7f84cc062a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84e4004fd0 tx=0x7f84e4007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.228+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f84e80b0960 con 0x7f84e8140710 2023-12-19T14:39:31.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.404+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84e80b0920 con 0x7f84e8140710 2023-12-19T14:39:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.408+0000 7f84ddffb700 1 -- 172.21.15.50:0/3117377710 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f84d804e260 con 0x7f84e8140710 2023-12-19T14:39:31.412 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:31.412 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 msgr2=0x7f84cc062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 0x7f84cc062a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84e4004fd0 tx=0x7f84e4007eb0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 msgr2=0x7f84e8155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f84e806a280 tx=0x7f84d8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 shutdown_connections 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e8140ae0 0x7f84e80670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84cc060580 0x7f84cc062a30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e8140710 0x7f84e8155b70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 --2- 172.21.15.50:0/3117377710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e8068630 0x7f84e80aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 >> 172.21.15.50:0/3117377710 conn(0x7f84e80c8850 msgr2=0x7f84e81568b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 shutdown_connections 2023-12-19T14:39:31.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.412+0000 7f84efd45700 1 -- 172.21.15.50:0/3117377710 wait complete. 2023-12-19T14:39:31.435 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:31.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:31.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.616+0000 7f1a7a68e700 1 Processor -- start 2023-12-19T14:39:31.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.616+0000 7f1a7a68e700 1 -- start start 2023-12-19T14:39:31.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7a68e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7a68e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a741595a0 0x7f1a74159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7a68e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7406c850 con 0x7f1a74158c90 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7a68e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a740cb650 con 0x7f1a741595a0 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7a68e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7406b470 con 0x7f1a7406aeb0 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7968c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1a74158c90 0x7f1a740caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44698/0 (socket says 172.21.15.50:44698) 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a741595a0 0x7f1a74159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a7968c700 1 -- 172.21.15.50:0/2195587978 learned_addr learned my addr 172.21.15.50:0/2195587978 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a78e8b700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2195587978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 msgr2=0x7f1a74159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2195587978 >> v1:172.21.15.50:6789/0 conn(0x7f1a74158c90 legacy=0x7f1a740caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2195587978 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7406b750 con 0x7f1a741595a0 2023-12-19T14:39:31.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a79e8d700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a74159a20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009e60 tx=0x7f1a6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c07cbc677b4eb0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.620+0000 7f1a6bfff700 1 -- 172.21.15.50:0/2195587978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c00e070 con 0x7f1a741595a0 2023-12-19T14:39:31.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a6bfff700 1 -- 172.21.15.50:0/2195587978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c00bc30 con 0x7f1a741595a0 2023-12-19T14:39:31.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a6bfff700 1 -- 172.21.15.50:0/2195587978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c012430 con 0x7f1a741595a0 2023-12-19T14:39:31.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2195587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 msgr2=0x7f1a74159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a74159a20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009e60 tx=0x7f1a6c009780 comp rx=0 tx=0).stop 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2195587978 shutdown_connections 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a74159a20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2195587978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2195587978 >> 172.21.15.50:0/2195587978 conn(0x7f1a74067bb0 msgr2=0x7f1a740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2195587978 shutdown_connections 2023-12-19T14:39:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.624+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2195587978 wait complete. 2023-12-19T14:39:31.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 Processor -- start 2023-12-19T14:39:31.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 -- start start 2023-12-19T14:39:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74169d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a74158c90 0x7f1a7406cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7416a500 con 0x7f1a741595a0 2023-12-19T14:39:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7416b360 con 0x7f1a74158c90 2023-12-19T14:39:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7a68e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7416a2a0 con 0x7f1a7406aeb0 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a79e8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a79e8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59232/0 (socket says 172.21.15.50:59232) 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2030997737 learned_addr learned my addr 172.21.15.50:0/2030997737 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a7968c700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74169d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.628+0000 7f1a78e8b700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a74158c90 0x7f1a7406cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 msgr2=0x7f1a74169d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74169d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a74158c90 msgr2=0x7f1a7406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a74158c90 0x7f1a7406cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 -- 172.21.15.50:0/2030997737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c009a20 con 0x7f1a741595a0 2023-12-19T14:39:31.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a7968c700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74169d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a79e8d700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a74069b70 tx=0x7f1a6c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c027470 con 0x7f1a741595a0 2023-12-19T14:39:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c0275f0 con 0x7f1a741595a0 2023-12-19T14:39:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0279b0 con 0x7f1a741595a0 2023-12-19T14:39:31.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7416a750 con 0x7f1a741595a0 2023-12-19T14:39:31.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7416aae0 con 0x7f1a741595a0 2023-12-19T14:39:31.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.632+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7406a080 con 0x7f1a741595a0 2023-12-19T14:39:31.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.636+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1a6c024070 con 0x7f1a741595a0 2023-12-19T14:39:31.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.636+0000 7f1a697fa700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 0x7f1a58062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:31.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.636+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1a6c07f560 con 0x7f1a741595a0 2023-12-19T14:39:31.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.636+0000 7f1a7968c700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 0x7f1a58062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:31.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.640+0000 7f1a7968c700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 0x7f1a58062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a700017e0 tx=0x7f1a70007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:31.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.644+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a7406a080 con 0x7f1a741595a0 2023-12-19T14:39:31.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.824+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a7406a080 con 0x7f1a741595a0 2023-12-19T14:39:31.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a697fa700 1 -- 172.21.15.50:0/2030997737 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a7406a080 con 0x7f1a741595a0 2023-12-19T14:39:31.831 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:31.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 msgr2=0x7f1a58062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 0x7f1a58062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a700017e0 tx=0x7f1a70007eb0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 msgr2=0x7f1a741687c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:31.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a74069b70 tx=0x7f1a6c019000 comp rx=0 tx=0).stop 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 shutdown_connections 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a74158c90 0x7f1a7406cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a58060550 0x7f1a58062a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a741595a0 0x7f1a741687c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 --2- 172.21.15.50:0/2030997737 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a7406aeb0 0x7f1a74169d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.828+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 >> 172.21.15.50:0/2030997737 conn(0x7f1a74067bb0 msgr2=0x7f1a740cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.832+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 shutdown_connections 2023-12-19T14:39:31.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:31.832+0000 7f1a7a68e700 1 -- 172.21.15.50:0/2030997737 wait complete. 2023-12-19T14:39:31.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 Processor -- start 2023-12-19T14:39:32.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 -- start start 2023-12-19T14:39:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb000c6d20 0x7fcb000c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006ac30 0x7fcb0006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0006a7a0 con 0x7fcb0006c7c0 2023-12-19T14:39:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb000c7630 con 0x7fcb000c6d20 2023-12-19T14:39:32.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcb048f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0006d4b0 con 0x7fcb0006ac30 2023-12-19T14:39:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafdd9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcb0006c7c0 0x7fcb0006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44710/0 (socket says 172.21.15.50:44710) 2023-12-19T14:39:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafdd9b700 1 -- 172.21.15.50:0/4074858120 learned_addr learned my addr 172.21.15.50:0/4074858120 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafe59c700 1 --2- 172.21.15.50:0/4074858120 >> v2:172.21.15.125:3300/0 conn(0x7fcb000c6d20 0x7fcb000c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafed9d700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006ac30 0x7fcb0006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafd59a700 1 -- 172.21.15.50:0/4074858120 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903027650 0 0) 0x7fcb0006a7a0 con 0x7fcb0006c7c0 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.028+0000 7fcafd59a700 1 -- 172.21.15.50:0/4074858120 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae8003570 con 0x7fcb0006c7c0 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafe59c700 1 -- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006ac30 msgr2=0x7fcb0006b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafe59c700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006ac30 0x7fcb0006b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafe59c700 1 -- 172.21.15.50:0/4074858120 >> v1:172.21.15.50:6789/0 conn(0x7fcb0006c7c0 legacy=0x7fcb0006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafe59c700 1 -- 172.21.15.50:0/4074858120 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0006d790 con 0x7fcb000c6d20 2023-12-19T14:39:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafe59c700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000c70f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00098e0 tx=0x7fcaf00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0fce366d17ad7a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafd59a700 1 -- 172.21.15.50:0/4074858120 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0011630 con 0x7fcb000c6d20 2023-12-19T14:39:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafd59a700 1 -- 172.21.15.50:0/4074858120 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf0011790 con 0x7fcb000c6d20 2023-12-19T14:39:32.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcafd59a700 1 -- 172.21.15.50:0/4074858120 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0011b30 con 0x7fcb000c6d20 2023-12-19T14:39:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 -- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 msgr2=0x7fcb000c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000c70f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00098e0 tx=0x7fcaf00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 -- 172.21.15.50:0/4074858120 shutdown_connections 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006ac30 0x7fcb0006b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 --2- 172.21.15.50:0/4074858120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000c70f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 -- 172.21.15.50:0/4074858120 >> 172.21.15.50:0/4074858120 conn(0x7fcb00067bb0 msgr2=0x7fcb000cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 -- 172.21.15.50:0/4074858120 shutdown_connections 2023-12-19T14:39:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.032+0000 7fcb048f0700 1 -- 172.21.15.50:0/4074858120 wait complete. 2023-12-19T14:39:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 Processor -- start 2023-12-19T14:39:32.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 -- start start 2023-12-19T14:39:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006c7c0 0x7fcb0016ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb00142020 con 0x7fcb0006ac30 2023-12-19T14:39:32.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb001515d0 con 0x7fcb000c6d20 2023-12-19T14:39:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.036+0000 7fcb048f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb001512a0 con 0x7fcb0006c7c0 2023-12-19T14:39:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59244/0 (socket says 172.21.15.50:59244) 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 -- 172.21.15.50:0/1228279511 learned_addr learned my addr 172.21.15.50:0/1228279511 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafdd9b700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006c7c0 0x7fcb0016ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafed9d700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 -- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006c7c0 msgr2=0x7fcb0016ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006c7c0 0x7fcb0016ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 -- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 msgr2=0x7fcb000adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 -- 172.21.15.50:0/1228279511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0009420 con 0x7fcb0006ac30 2023-12-19T14:39:32.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafed9d700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcafe59c700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcb00069e00 tx=0x7fcaf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0011b10 con 0x7fcb0006ac30 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf00253f0 con 0x7fcb0006ac30 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0016b150 con 0x7fcb0006ac30 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf00257b0 con 0x7fcb0006ac30 2023-12-19T14:39:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0016b6d0 con 0x7fcb0006ac30 2023-12-19T14:39:32.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.040+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb00071120 con 0x7fcb0006ac30 2023-12-19T14:39:32.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.044+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fcaf0025ba0 con 0x7fcb0006ac30 2023-12-19T14:39:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.044+0000 7fcaee7fc700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 0x7fcae0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.044+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcaf007e240 con 0x7fcb0006ac30 2023-12-19T14:39:32.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.044+0000 7fcafdd9b700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 0x7fcae0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.048+0000 7fcafdd9b700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 0x7fcae0062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb000c20d0 tx=0x7fcaf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.052+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcaf004e540 con 0x7fcb0006ac30 2023-12-19T14:39:32.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb0006e100 con 0x7fcb0006ac30 2023-12-19T14:39:32.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcaee7fc700 1 -- 172.21.15.50:0/1228279511 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcaf0051b60 con 0x7fcb0006ac30 2023-12-19T14:39:32.231 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 msgr2=0x7fcae0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 0x7fcae0062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb000c20d0 tx=0x7fcaf4009040 comp rx=0 tx=0).stop 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 msgr2=0x7fcb00168160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.228+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcb00069e00 tx=0x7fcaf0017000 comp rx=0 tx=0).stop 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 shutdown_connections 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb000c6d20 0x7fcb000adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcae0060550 0x7fcae0062a00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb0006c7c0 0x7fcb0016ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 --2- 172.21.15.50:0/1228279511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb0006ac30 0x7fcb00168160 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 >> 172.21.15.50:0/1228279511 conn(0x7fcb00067bb0 msgr2=0x7fcb0006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 shutdown_connections 2023-12-19T14:39:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.232+0000 7fcb048f0700 1 -- 172.21.15.50:0/1228279511 wait complete. 2023-12-19T14:39:32.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:32.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.424+0000 7f449adc3700 1 Processor -- start 2023-12-19T14:39:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.424+0000 7f449adc3700 1 -- start start 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449adc3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f44940ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449adc3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44940c7780 0x7f44940c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449adc3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44940adc60 con 0x7f449413f6a0 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449adc3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449406d7d0 con 0x7f44940c7780 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449adc3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44940c8140 con 0x7f449413f2d0 2023-12-19T14:39:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449a5c2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44940c7780 0x7f44940c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f44995c0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f449413f6a0 0x7f449415a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44716/0 (socket says 172.21.15.50:44716) 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f44995c0700 1 -- 172.21.15.50:0/40445926 learned_addr learned my addr 172.21.15.50:0/40445926 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f4499dc1700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f44940ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449a5c2700 1 -- 172.21.15.50:0/40445926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 msgr2=0x7f44940ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f4498dbf700 1 -- 172.21.15.50:0/40445926 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374144630 0 0) 0x7f44940adc60 con 0x7f449413f6a0 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449a5c2700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f44940ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449a5c2700 1 -- 172.21.15.50:0/40445926 >> v1:172.21.15.50:6789/0 conn(0x7f449413f6a0 legacy=0x7f449415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.428+0000 7f449a5c2700 1 -- 172.21.15.50:0/40445926 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44940ad050 con 0x7f44940c7780 2023-12-19T14:39:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449a5c2700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44940c7780 0x7f44940c7c00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f448c009e90 tx=0x7f448c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c41950d7a3bbf82 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f4498dbf700 1 -- 172.21.15.50:0/40445926 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c00e070 con 0x7f44940c7780 2023-12-19T14:39:32.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f4498dbf700 1 -- 172.21.15.50:0/40445926 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c00bbf0 con 0x7f44940c7780 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f4498dbf700 1 -- 172.21.15.50:0/40445926 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c012430 con 0x7f44940c7780 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- 172.21.15.50:0/40445926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44940c7780 msgr2=0x7f44940c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44940c7780 0x7f44940c7c00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f448c009e90 tx=0x7f448c009780 comp rx=0 tx=0).stop 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- 172.21.15.50:0/40445926 shutdown_connections 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44940c7780 0x7f44940c7c00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 --2- 172.21.15.50:0/40445926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f44940ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- 172.21.15.50:0/40445926 >> 172.21.15.50:0/40445926 conn(0x7f4494067b50 msgr2=0x7f44940cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- 172.21.15.50:0/40445926 shutdown_connections 2023-12-19T14:39:32.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- 172.21.15.50:0/40445926 wait complete. 2023-12-19T14:39:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 Processor -- start 2023-12-19T14:39:32.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.432+0000 7f449adc3700 1 -- start start 2023-12-19T14:39:32.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 0x7f449415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f449415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449416ac20 con 0x7f449413f6a0 2023-12-19T14:39:32.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494169fe0 con 0x7f44940c7780 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449416a2f0 con 0x7f449413f2d0 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59260/0 (socket says 172.21.15.50:59260) 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 -- 172.21.15.50:0/2986405127 learned_addr learned my addr 172.21.15.50:0/2986405127 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f4499dc1700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 0x7f449415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 -- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 msgr2=0x7f449415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f449415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 -- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 msgr2=0x7f449415cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 0x7f449415cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 -- 172.21.15.50:0/2986405127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c009a20 con 0x7f449413f6a0 2023-12-19T14:39:32.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f4499dc1700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 0x7f449415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:32.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449a5c2700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f449406a0d0 tx=0x7f448c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c01d630 con 0x7f449413f6a0 2023-12-19T14:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c01d7b0 con 0x7f449413f6a0 2023-12-19T14:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c01db70 con 0x7f449413f6a0 2023-12-19T14:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449416a5d0 con 0x7f449413f6a0 2023-12-19T14:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.436+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4494155710 con 0x7f449413f6a0 2023-12-19T14:39:32.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.440+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449416add0 con 0x7f449413f6a0 2023-12-19T14:39:32.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.440+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f448c024070 con 0x7f449413f6a0 2023-12-19T14:39:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.444+0000 7f4489ffb700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 0x7f4478062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.444+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f448c07f2a0 con 0x7f449413f6a0 2023-12-19T14:39:32.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.444+0000 7f4499dc1700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 0x7f4478062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.444+0000 7f4499dc1700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 0x7f4478062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4490009990 tx=0x7f4490008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.448+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f449416add0 con 0x7f449413f6a0 2023-12-19T14:39:32.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.624+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f449404be40 con 0x7f449413f6a0 2023-12-19T14:39:32.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.624+0000 7f4489ffb700 1 -- 172.21.15.50:0/2986405127 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f448c04f5a0 con 0x7f449413f6a0 2023-12-19T14:39:32.663 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 msgr2=0x7f4478062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 0x7f4478062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4490009990 tx=0x7f4490008040 comp rx=0 tx=0).stop 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 msgr2=0x7f44941692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f449406a0d0 tx=0x7f448c019000 comp rx=0 tx=0).stop 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 shutdown_connections 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4478060550 0x7f4478062a00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f449413f6a0 0x7f44941692b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f449413f2d0 0x7f449415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 --2- 172.21.15.50:0/2986405127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44940c7780 0x7f449415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 >> 172.21.15.50:0/2986405127 conn(0x7f4494067b50 msgr2=0x7f44940c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 shutdown_connections 2023-12-19T14:39:32.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.628+0000 7f449adc3700 1 -- 172.21.15.50:0/2986405127 wait complete. 2023-12-19T14:39:32.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:32.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.840+0000 7fd871cf0700 1 Processor -- start 2023-12-19T14:39:32.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.840+0000 7fd871cf0700 1 -- start start 2023-12-19T14:39:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.840+0000 7fd871cf0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd86c15d410 0x7fd86c0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 0x7fd86c06b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0c7450 con 0x7fd86c0ca400 2023-12-19T14:39:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c06bea0 con 0x7fd86c06b1b0 2023-12-19T14:39:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0c6e00 con 0x7fd86c15d410 2023-12-19T14:39:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 0x7fd86c06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd86c15d410 0x7fd86c0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 0x7fd86c06b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59266/0 (socket says 172.21.15.50:59266) 2023-12-19T14:39:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd86c15d410 0x7fd86c0ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60390/0 (socket says 172.21.15.50:60390) 2023-12-19T14:39:32.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86bfff700 1 -- 172.21.15.50:0/1303277193 learned_addr learned my addr 172.21.15.50:0/1303277193 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 -- 172.21.15.50:0/1303277193 >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 msgr2=0x7fd86c06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/1303277193 >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 0x7fd86c06b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 -- 172.21.15.50:0/1303277193 >> v1:172.21.15.50:6789/0 conn(0x7fd86c0ca400 legacy=0x7fd86c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 -- 172.21.15.50:0/1303277193 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0c70e0 con 0x7fd86c15d410 2023-12-19T14:39:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/1303277193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c0ab6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd86c068030 tx=0x7fd85c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a93642982dc5c2b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86a7fc700 1 -- 172.21.15.50:0/1303277193 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c011630 con 0x7fd86c15d410 2023-12-19T14:39:32.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86a7fc700 1 -- 172.21.15.50:0/1303277193 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c011790 con 0x7fd86c15d410 2023-12-19T14:39:32.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd86a7fc700 1 -- 172.21.15.50:0/1303277193 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c011b30 con 0x7fd86c15d410 2023-12-19T14:39:32.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 -- 172.21.15.50:0/1303277193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 msgr2=0x7fd86c0ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.844+0000 7fd871cf0700 1 --2- 172.21.15.50:0/1303277193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c0ab6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd86c068030 tx=0x7fd85c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 -- 172.21.15.50:0/1303277193 shutdown_connections 2023-12-19T14:39:32.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 --2- 172.21.15.50:0/1303277193 >> v2:172.21.15.50:3300/0 conn(0x7fd86c06b1b0 0x7fd86c06b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 --2- 172.21.15.50:0/1303277193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c0ab6c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 -- 172.21.15.50:0/1303277193 >> 172.21.15.50:0/1303277193 conn(0x7fd86c067bb0 msgr2=0x7fd86c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:32.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 -- 172.21.15.50:0/1303277193 shutdown_connections 2023-12-19T14:39:32.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 -- 172.21.15.50:0/1303277193 wait complete. 2023-12-19T14:39:32.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 Processor -- start 2023-12-19T14:39:32.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.848+0000 7fd871cf0700 1 -- start start 2023-12-19T14:39:32.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 0x7fd86c15a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c161630 con 0x7fd86c0ca400 2023-12-19T14:39:32.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c15f170 con 0x7fd86c15d410 2023-12-19T14:39:32.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c15a610 con 0x7fd86c06b1b0 2023-12-19T14:39:32.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59280/0 (socket says 172.21.15.50:59280) 2023-12-19T14:39:32.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86affd700 1 -- 172.21.15.50:0/3849518148 learned_addr learned my addr 172.21.15.50:0/3849518148 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:32.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 0x7fd86c15a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86bfff700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 -- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 msgr2=0x7fd86c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c15fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 -- 172.21.15.50:0/3849518148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 msgr2=0x7fd86c0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 -- 172.21.15.50:0/3849518148 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c009420 con 0x7fd86c06b1b0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86affd700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd86b7fe700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 0x7fd86c15a0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd85c016000 tx=0x7fd85c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c000d10 con 0x7fd86c06b1b0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c011b10 con 0x7fd86c06b1b0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c15f6d0 con 0x7fd86c06b1b0 2023-12-19T14:39:32.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c025640 con 0x7fd86c06b1b0 2023-12-19T14:39:32.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.852+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1610f0 con 0x7fd86c06b1b0 2023-12-19T14:39:32.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.856+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c071120 con 0x7fd86c06b1b0 2023-12-19T14:39:32.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.856+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd85c0193b0 con 0x7fd86c06b1b0 2023-12-19T14:39:32.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.856+0000 7fd8537fe700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 0x7fd858062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:32.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.856+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd85c07e420 con 0x7fd86c06b1b0 2023-12-19T14:39:32.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.856+0000 7fd86affd700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 0x7fd858062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:32.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.860+0000 7fd86affd700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 0x7fd858062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0b0b50 tx=0x7fd860009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:32.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:32.868+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd85c04e680 con 0x7fd86c06b1b0 2023-12-19T14:39:33.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.048+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd86c0ae340 con 0x7fd86c06b1b0 2023-12-19T14:39:33.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.052+0000 7fd8537fe700 1 -- 172.21.15.50:0/3849518148 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd85c051ca0 con 0x7fd86c06b1b0 2023-12-19T14:39:33.054 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 msgr2=0x7fd858062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 0x7fd858062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0b0b50 tx=0x7fd860009040 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 msgr2=0x7fd86c15a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 0x7fd86c15a0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd85c016000 tx=0x7fd85c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 shutdown_connections 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd86c15d410 0x7fd86c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd858060550 0x7fd858062a00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c0ca400 0x7fd86c0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 --2- 172.21.15.50:0/3849518148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd86c06b1b0 0x7fd86c15a0d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 >> 172.21.15.50:0/3849518148 conn(0x7fd86c067bb0 msgr2=0x7fd86c06a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 shutdown_connections 2023-12-19T14:39:33.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.056+0000 7fd871cf0700 1 -- 172.21.15.50:0/3849518148 wait complete. 2023-12-19T14:39:33.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.248+0000 7f73fa2e3700 1 Processor -- start 2023-12-19T14:39:33.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.248+0000 7f73fa2e3700 1 -- start start 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.248+0000 7f73fa2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f40c6a50 0x7f73f40c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.248+0000 7f73fa2e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73f406dc20 0x7f73f406b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73fa2e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f406d790 con 0x7f73f406bb30 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73fa2e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f40c7360 con 0x7f73f406dc20 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73fa2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f406cb80 con 0x7f73f40c6a50 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73f406dc20 0x7f73f406b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f9ae2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f73f406bb30 0x7f73f4159b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44740/0 (socket says 172.21.15.50:44740) 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f9ae2700 1 -- 172.21.15.50:0/4043290857 learned_addr learned my addr 172.21.15.50:0/4043290857 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f92e1700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f40c6a50 0x7f73f40c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 -- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f40c6a50 msgr2=0x7f73f40c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f40c6a50 0x7f73f40c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 -- 172.21.15.50:0/4043290857 >> v1:172.21.15.50:6789/0 conn(0x7f73f406bb30 legacy=0x7f73f4159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 -- 172.21.15.50:0/4043290857 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f406ce60 con 0x7f73f406dc20 2023-12-19T14:39:33.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73f8ae0700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f406dc20 0x7f73f406b5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73e40098d0 tx=0x7f73e4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67115def2e18260d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73ebfff700 1 -- 172.21.15.50:0/4043290857 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4005c10 con 0x7f73f406dc20 2023-12-19T14:39:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73ebfff700 1 -- 172.21.15.50:0/4043290857 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e4005d70 con 0x7f73f406dc20 2023-12-19T14:39:33.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.252+0000 7f73ebfff700 1 -- 172.21.15.50:0/4043290857 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4006280 con 0x7f73f406dc20 2023-12-19T14:39:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f406dc20 msgr2=0x7f73f406b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f406dc20 0x7f73f406b5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73e40098d0 tx=0x7f73e4009120 comp rx=0 tx=0).stop 2023-12-19T14:39:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- 172.21.15.50:0/4043290857 shutdown_connections 2023-12-19T14:39:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f406dc20 0x7f73f406b5f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/4043290857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f40c6a50 0x7f73f40c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- 172.21.15.50:0/4043290857 >> 172.21.15.50:0/4043290857 conn(0x7f73f4067bb0 msgr2=0x7f73f40cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- 172.21.15.50:0/4043290857 shutdown_connections 2023-12-19T14:39:33.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- 172.21.15.50:0/4043290857 wait complete. 2023-12-19T14:39:33.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 Processor -- start 2023-12-19T14:39:33.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.256+0000 7f73fa2e3700 1 -- start start 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f406dc20 0x7f73f4169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f40c6a50 0x7f73f40a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4169390 con 0x7f73f406bb30 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f40b11c0 con 0x7f73f40c6a50 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f40b14a0 con 0x7f73f406dc20 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59290/0 (socket says 172.21.15.50:59290) 2023-12-19T14:39:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 -- 172.21.15.50:0/3001403008 learned_addr learned my addr 172.21.15.50:0/3001403008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f9ae2700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f40c6a50 0x7f73f40a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f8ae0700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f406dc20 0x7f73f4169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 -- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f406dc20 msgr2=0x7f73f4169820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f406dc20 0x7f73f4169820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 -- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f40c6a50 msgr2=0x7f73f40a9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f40c6a50 0x7f73f40a9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 -- 172.21.15.50:0/3001403008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40093c0 con 0x7f73f406bb30 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73f92e1700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e930 tx=0x7f73f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000eed0 con 0x7f73f406bb30 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f000c850 con 0x7f73f406bb30 2023-12-19T14:39:33.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f40b1780 con 0x7f73f406bb30 2023-12-19T14:39:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000cbf0 con 0x7f73f406bb30 2023-12-19T14:39:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.260+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f4168fd0 con 0x7f73f406bb30 2023-12-19T14:39:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.264+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f404be40 con 0x7f73f406bb30 2023-12-19T14:39:33.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.264+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f73f0013470 con 0x7f73f406bb30 2023-12-19T14:39:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.268+0000 7f73e97fa700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 0x7f73d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.268+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f73f007cd50 con 0x7f73f406bb30 2023-12-19T14:39:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.268+0000 7f73f8ae0700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 0x7f73d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.268+0000 7f73f8ae0700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 0x7f73d8062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009390 tx=0x7f73e4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.272+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f73f004cfb0 con 0x7f73f406bb30 2023-12-19T14:39:33.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.480+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73f406d140 con 0x7f73f406bb30 2023-12-19T14:39:33.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73e97fa700 1 -- 172.21.15.50:0/3001403008 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f73f00505d0 con 0x7f73f406bb30 2023-12-19T14:39:33.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:33.487 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 msgr2=0x7f73d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 0x7f73d8062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009390 tx=0x7f73e4016000 comp rx=0 tx=0).stop 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 msgr2=0x7f73f41522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e930 tx=0x7f73f00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 shutdown_connections 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73f40c6a50 0x7f73f40a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73d8060550 0x7f73d8062a00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73f406dc20 0x7f73f4169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 --2- 172.21.15.50:0/3001403008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73f406bb30 0x7f73f41522a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 >> 172.21.15.50:0/3001403008 conn(0x7f73f4067bb0 msgr2=0x7f73f4152fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 shutdown_connections 2023-12-19T14:39:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.484+0000 7f73fa2e3700 1 -- 172.21.15.50:0/3001403008 wait complete. 2023-12-19T14:39:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:39:33.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.664+0000 7f2b894f0700 1 Processor -- start 2023-12-19T14:39:33.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.664+0000 7f2b894f0700 1 -- start start 2023-12-19T14:39:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b894f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b841557e0 0x7f2b8415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b894f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b894f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b84159d20 con 0x7f2b84155bb0 2023-12-19T14:39:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b894f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b840c81f0 con 0x7f2b841557e0 2023-12-19T14:39:33.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b894f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c7f10 con 0x7f2b8406dcb0 2023-12-19T14:39:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b82ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b841557e0 0x7f2b8415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54526/0 (socket says 172.21.15.50:54526) 2023-12-19T14:39:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 -- 172.21.15.50:0/3144685650 learned_addr learned my addr 172.21.15.50:0/3144685650 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:33.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 -- 172.21.15.50:0/3144685650 >> v2:172.21.15.125:3300/0 conn(0x7f2b841557e0 msgr2=0x7f2b8415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 --2- 172.21.15.50:0/3144685650 >> v2:172.21.15.125:3300/0 conn(0x7f2b841557e0 0x7f2b8415a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 -- 172.21.15.50:0/3144685650 >> v1:172.21.15.125:6789/0 conn(0x7f2b84155bb0 legacy=0x7f2b840cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 -- 172.21.15.50:0/3144685650 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b841596d0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.668+0000 7f2b7a7fc700 1 --2- 172.21.15.50:0/3144685650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009f20 tx=0x7f2b6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e991cd932081199 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b827fc700 1 -- 172.21.15.50:0/3144685650 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00e070 con 0x7f2b8406dcb0 2023-12-19T14:39:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b827fc700 1 -- 172.21.15.50:0/3144685650 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6c00b9d0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b827fc700 1 -- 172.21.15.50:0/3144685650 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c0092e0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 -- 172.21.15.50:0/3144685650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 msgr2=0x7f2b840c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 --2- 172.21.15.50:0/3144685650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009f20 tx=0x7f2b6c009840 comp rx=0 tx=0).stop 2023-12-19T14:39:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 -- 172.21.15.50:0/3144685650 shutdown_connections 2023-12-19T14:39:33.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 --2- 172.21.15.50:0/3144685650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b8406dcb0 0x7f2b840c7630 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 --2- 172.21.15.50:0/3144685650 >> v2:172.21.15.125:3300/0 conn(0x7f2b841557e0 0x7f2b8415a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 -- 172.21.15.50:0/3144685650 >> 172.21.15.50:0/3144685650 conn(0x7f2b84067bb0 msgr2=0x7f2b840cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 -- 172.21.15.50:0/3144685650 shutdown_connections 2023-12-19T14:39:33.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.672+0000 7f2b894f0700 1 -- 172.21.15.50:0/3144685650 wait complete. 2023-12-19T14:39:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 Processor -- start 2023-12-19T14:39:33.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 -- start start 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b841557e0 0x7f2b8416d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 0x7f2b8414e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c18f0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c0ce0 con 0x7f2b84155bb0 2023-12-19T14:39:33.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b894f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c0fc0 con 0x7f2b841557e0 2023-12-19T14:39:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59296/0 (socket says 172.21.15.50:59296) 2023-12-19T14:39:33.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b82ffd700 1 -- 172.21.15.50:0/460972855 learned_addr learned my addr 172.21.15.50:0/460972855 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.676+0000 7f2b7a7fc700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b841557e0 0x7f2b8416d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b837fe700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 0x7f2b8414e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 -- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b841557e0 msgr2=0x7f2b8416d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b841557e0 0x7f2b8416d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 -- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 msgr2=0x7f2b8414e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 0x7f2b8414e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 -- 172.21.15.50:0/460972855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c009ae0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b837fe700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 0x7f2b8414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b82ffd700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ec80 tx=0x7f2b740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7400c850 con 0x7f2b8406dcb0 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7400c9b0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7400cd50 con 0x7f2b8406dcb0 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b840c12a0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b841502b0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.680+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8404be40 con 0x7f2b8406dcb0 2023-12-19T14:39:33.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.684+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2b74013680 con 0x7f2b8406dcb0 2023-12-19T14:39:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.684+0000 7f2b7b7fe700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 0x7f2b64062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:33.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.684+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2b7407cdc0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.684+0000 7f2b7a7fc700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 0x7f2b64062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:33.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.688+0000 7f2b7a7fc700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 0x7f2b64062a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009880 tx=0x7f2b6c018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:33.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.692+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2b7404d020 con 0x7f2b8406dcb0 2023-12-19T14:39:33.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.864+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b841599b0 con 0x7f2b8406dcb0 2023-12-19T14:39:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b7b7fe700 1 -- 172.21.15.50:0/460972855 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f2b74050640 con 0x7f2b8406dcb0 2023-12-19T14:39:33.872 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:33.872 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 msgr2=0x7f2b64062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 0x7f2b64062a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c009880 tx=0x7f2b6c018000 comp rx=0 tx=0).stop 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 msgr2=0x7f2b8416cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ec80 tx=0x7f2b740091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 shutdown_connections 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b84155bb0 0x7f2b8414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2b64060580 0x7f2b64062a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2b841557e0 0x7f2b8416d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 --2- 172.21.15.50:0/460972855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b8406dcb0 0x7f2b8416cd60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.868+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 >> 172.21.15.50:0/460972855 conn(0x7f2b84067bb0 msgr2=0x7f2b840ae7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.872+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 shutdown_connections 2023-12-19T14:39:33.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:33.872+0000 7f2b894f0700 1 -- 172.21.15.50:0/460972855 wait complete. 2023-12-19T14:39:33.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:34.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.052+0000 7f62724f0700 1 Processor -- start 2023-12-19T14:39:34.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.052+0000 7f62724f0700 1 -- start start 2023-12-19T14:39:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f62724f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c15d320 0x7f626c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f62724f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f626c0c15b0 0x7f626c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f62724f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0c7190 con 0x7f626c06d690 2023-12-19T14:39:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f62724f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0c1b50 con 0x7f626c0c15b0 2023-12-19T14:39:34.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f62724f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c1e30 con 0x7f626c15d320 2023-12-19T14:39:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f626c0c15b0 0x7f626c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c15d320 0x7f626c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f626c0c15b0 0x7f626c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59302/0 (socket says 172.21.15.50:59302) 2023-12-19T14:39:34.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 -- 172.21.15.50:0/2217089939 learned_addr learned my addr 172.21.15.50:0/2217089939 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 -- 172.21.15.50:0/2217089939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c15d320 msgr2=0x7f626c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 --2- 172.21.15.50:0/2217089939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c15d320 0x7f626c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 -- 172.21.15.50:0/2217089939 >> v1:172.21.15.125:6789/0 conn(0x7f626c06d690 legacy=0x7f626c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 -- 172.21.15.50:0/2217089939 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0c6e20 con 0x7f626c0c15b0 2023-12-19T14:39:34.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626b7fe700 1 --2- 172.21.15.50:0/2217089939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c0c15b0 0x7f626c0aeba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62600099b0 tx=0x7f6260009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeb07f86d05cac7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626affd700 1 -- 172.21.15.50:0/2217089939 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6260009ea0 con 0x7f626c0c15b0 2023-12-19T14:39:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626affd700 1 -- 172.21.15.50:0/2217089939 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626000b860 con 0x7f626c0c15b0 2023-12-19T14:39:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.056+0000 7f626affd700 1 -- 172.21.15.50:0/2217089939 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626000bc00 con 0x7f626c0c15b0 2023-12-19T14:39:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- 172.21.15.50:0/2217089939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c0c15b0 msgr2=0x7f626c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 --2- 172.21.15.50:0/2217089939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c0c15b0 0x7f626c0aeba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62600099b0 tx=0x7f6260009250 comp rx=0 tx=0).stop 2023-12-19T14:39:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- 172.21.15.50:0/2217089939 shutdown_connections 2023-12-19T14:39:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 --2- 172.21.15.50:0/2217089939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c0c15b0 0x7f626c0aeba0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 --2- 172.21.15.50:0/2217089939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c15d320 0x7f626c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- 172.21.15.50:0/2217089939 >> 172.21.15.50:0/2217089939 conn(0x7f626c0c8850 msgr2=0x7f626c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- 172.21.15.50:0/2217089939 shutdown_connections 2023-12-19T14:39:34.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- 172.21.15.50:0/2217089939 wait complete. 2023-12-19T14:39:34.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 Processor -- start 2023-12-19T14:39:34.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.060+0000 7f62724f0700 1 -- start start 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f626c06d690 0x7f626c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c0c15b0 0x7f626c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c160f90 con 0x7f626c15d320 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0c4de0 con 0x7f626c06d690 2023-12-19T14:39:34.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f62724f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c160cb0 con 0x7f626c0c15b0 2023-12-19T14:39:34.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59308/0 (socket says 172.21.15.50:59308) 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 -- 172.21.15.50:0/819915755 learned_addr learned my addr 172.21.15.50:0/819915755 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f626b7fe700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c0c15b0 0x7f626c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f626bfff700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f626c06d690 0x7f626c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 -- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c0c15b0 msgr2=0x7f626c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c0c15b0 0x7f626c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 -- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f626c06d690 msgr2=0x7f626c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f626c06d690 0x7f626c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 -- 172.21.15.50:0/819915755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62600094f0 con 0x7f626c15d320 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.064+0000 7f6270a8d700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f626400c910 tx=0x7f6264001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264012630 con 0x7f626c15d320 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6264012790 con 0x7f626c15d320 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c0c4790 con 0x7f626c15d320 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264012b50 con 0x7f626c15d320 2023-12-19T14:39:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c1545a0 con 0x7f626c15d320 2023-12-19T14:39:34.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.068+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c0c45f0 con 0x7f626c15d320 2023-12-19T14:39:34.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.072+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f626401a3a0 con 0x7f626c15d320 2023-12-19T14:39:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.072+0000 7f6253fff700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 0x7f6258062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.072+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f626407ccb0 con 0x7f626c15d320 2023-12-19T14:39:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.072+0000 7f626bfff700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 0x7f6258062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.072+0000 7f626bfff700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 0x7f6258062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f625c009d10 tx=0x7f625c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.076+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f626c0c45f0 con 0x7f626c15d320 2023-12-19T14:39:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.288+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f626c0c45f0 con 0x7f626c15d320 2023-12-19T14:39:34.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.292+0000 7f6253fff700 1 -- 172.21.15.50:0/819915755 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f626c0c45f0 con 0x7f626c15d320 2023-12-19T14:39:34.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:34.295 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:39:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 msgr2=0x7f6258062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 0x7f6258062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f625c009d10 tx=0x7f625c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 msgr2=0x7f626c154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f626400c910 tx=0x7f6264001710 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 shutdown_connections 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6258060550 0x7f6258062a00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f626c15d320 0x7f626c154060 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f626c0c15b0 0x7f626c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 --2- 172.21.15.50:0/819915755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f626c06d690 0x7f626c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 >> 172.21.15.50:0/819915755 conn(0x7f626c0c8850 msgr2=0x7f626c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 shutdown_connections 2023-12-19T14:39:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.296+0000 7f62724f0700 1 -- 172.21.15.50:0/819915755 wait complete. 2023-12-19T14:39:34.303 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:39:34.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:39:06.675595+0000', 'modified': '2023-12-19T14:39:21.592876+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/3977471945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6833', 'nonce': 3977471945}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 3977471945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:39:34.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:39:34.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T14:39:34.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:39:34.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:39:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 Processor -- start 2023-12-19T14:39:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 -- start start 2023-12-19T14:39:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe63806b7a0 0x7fe6380c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806c7c0 con 0x7fe6380c51b0 2023-12-19T14:39:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806caa0 con 0x7fe638158d60 2023-12-19T14:39:34.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.896+0000 7fe63c816700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6380c55a0 con 0x7fe63806b7a0 2023-12-19T14:39:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe637fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe6380c51b0 0x7fe63806cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44748/0 (socket says 172.21.15.50:44748) 2023-12-19T14:39:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe637fff700 1 -- 172.21.15.50:0/3954805339 learned_addr learned my addr 172.21.15.50:0/3954805339 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe62effd700 1 --2- 172.21.15.50:0/3954805339 >> v2:172.21.15.125:3300/0 conn(0x7fe63806b7a0 0x7fe6380c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153668502 0 0) 0x7fe63806c7c0 con 0x7fe6380c51b0 2023-12-19T14:39:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe624003570 con 0x7fe6380c51b0 2023-12-19T14:39:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472175626 0 0) 0x7fe63806caa0 con 0x7fe638158d60 2023-12-19T14:39:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63806c7c0 con 0x7fe638158d60 2023-12-19T14:39:34.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe62effd700 1 -- 172.21.15.50:0/3954805339 >> v1:172.21.15.125:6789/0 conn(0x7fe638158d60 legacy=0x7fe63815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe62effd700 1 -- 172.21.15.50:0/3954805339 >> v1:172.21.15.50:6789/0 conn(0x7fe6380c51b0 legacy=0x7fe63806cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe62effd700 1 -- 172.21.15.50:0/3954805339 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63815d130 con 0x7fe63806b7a0 2023-12-19T14:39:34.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe62effd700 1 --2- 172.21.15.50:0/3954805339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806b7a0 0x7fe6380c4c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe620009e60 tx=0x7fe620009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa940e7fb23b842b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62000e070 con 0x7fe63806b7a0 2023-12-19T14:39:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62000bab0 con 0x7fe63806b7a0 2023-12-19T14:39:34.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe636ffd700 1 -- 172.21.15.50:0/3954805339 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62000be70 con 0x7fe63806b7a0 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe63c816700 1 -- 172.21.15.50:0/3954805339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806b7a0 msgr2=0x7fe6380c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe63c816700 1 --2- 172.21.15.50:0/3954805339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806b7a0 0x7fe6380c4c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe620009e60 tx=0x7fe620009780 comp rx=0 tx=0).stop 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe63c816700 1 -- 172.21.15.50:0/3954805339 shutdown_connections 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe63c816700 1 --2- 172.21.15.50:0/3954805339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806b7a0 0x7fe6380c4c70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.900+0000 7fe63c816700 1 -- 172.21.15.50:0/3954805339 >> 172.21.15.50:0/3954805339 conn(0x7fe638067bb0 msgr2=0x7fe6380cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- 172.21.15.50:0/3954805339 shutdown_connections 2023-12-19T14:39:34.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- 172.21.15.50:0/3954805339 wait complete. 2023-12-19T14:39:34.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 Processor -- start 2023-12-19T14:39:34.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- start start 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 0x7fe638150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6380c51b0 0x7fe6380afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381594d0 con 0x7fe63806b7a0 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381591d0 con 0x7fe638158d60 2023-12-19T14:39:34.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe63c816700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63814f140 con 0x7fe6380c51b0 2023-12-19T14:39:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe6377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 0x7fe638150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe637fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.904+0000 7fe62effd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6380c51b0 0x7fe6380afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe637fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60474/0 (socket says 172.21.15.50:60474) 2023-12-19T14:39:34.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe637fff700 1 -- 172.21.15.50:0/701086289 learned_addr learned my addr 172.21.15.50:0/701086289 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 -- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6380c51b0 msgr2=0x7fe6380afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6380c51b0 0x7fe6380afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 -- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 msgr2=0x7fe6380b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 -- 172.21.15.50:0/701086289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620009a20 con 0x7fe63806b7a0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe637fff700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe6377fe700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 0x7fe638150360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c4e0 tx=0x7fe628007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe62ffff700 1 -- 172.21.15.50:0/701086289 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62800ce50 con 0x7fe63806b7a0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe62ffff700 1 -- 172.21.15.50:0/701086289 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628011590 con 0x7fe63806b7a0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63814f3f0 con 0x7fe63806b7a0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe62ffff700 1 -- 172.21.15.50:0/701086289 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628011950 con 0x7fe63806b7a0 2023-12-19T14:39:34.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.908+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6381508a0 con 0x7fe63806b7a0 2023-12-19T14:39:34.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe62ffff700 1 -- 172.21.15.50:0/701086289 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe628011d40 con 0x7fe63806b7a0 2023-12-19T14:39:34.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe62ffff700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 0x7fe618062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe63fd6a740 1 Processor -- start 2023-12-19T14:39:34.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe62ffff700 1 -- 172.21.15.50:0/701086289 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe62807c670 con 0x7fe63806b7a0 2023-12-19T14:39:34.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe63fd6a740 1 -- start start 2023-12-19T14:39:34.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.912+0000 7fe62effd700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 0x7fe618062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe62effd700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 0x7fe618062a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe620009460 tx=0x7fe620019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x199b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x18c8f30 con 0x1768790 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x18c4ed0 con 0x171d710 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x188b0b0 con 0x171dae0 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe637fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe62effd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x199b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54580/0 (socket says 172.21.15.50:54580) 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 -- 172.21.15.50:0/3547477370 learned_addr learned my addr 172.21.15.50:0/3547477370 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 -- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 msgr2=0x199b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x199b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 -- 172.21.15.50:0/3547477370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 msgr2=0x199b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 -- 172.21.15.50:0/3547477370 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x17699e0 con 0x171dae0 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe637fff700 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe6377fe700 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe62800cfd0 tx=0x7fe62807a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bee3f783d59c31b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe62cff9700 1 -- 172.21.15.50:0/3547477370 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628057570 con 0x171dae0 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe62cff9700 1 -- 172.21.15.50:0/3547477370 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628050590 con 0x171dae0 2023-12-19T14:39:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe62cff9700 1 -- 172.21.15.50:0/3547477370 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62804e930 con 0x171dae0 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 msgr2=0x199af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe62800cfd0 tx=0x7fe62807a760 comp rx=0 tx=0).stop 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- 172.21.15.50:0/3547477370 shutdown_connections 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x199b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/3547477370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x199af00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.916+0000 7fe63fd6a740 1 -- 172.21.15.50:0/3547477370 >> 172.21.15.50:0/3547477370 conn(0x171deb0 msgr2=0x1957930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- 172.21.15.50:0/3547477370 shutdown_connections 2023-12-19T14:39:34.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- 172.21.15.50:0/3547477370 wait complete. 2023-12-19T14:39:34.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- start start 2023-12-19T14:39:34.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 Processor -- start 2023-12-19T14:39:34.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171d710 0x199f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1768790 0x1980cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x188ad50 con 0x171dae0 2023-12-19T14:39:34.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1863970 con 0x1768790 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x186e510 con 0x171d710 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe63fd6a740 10 client.0 _ll_register_callbacks cb 0x190e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59320/0 (socket says 172.21.15.50:59320) 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 -- 172.21.15.50:0/1703698340 learned_addr learned my addr 172.21.15.50:0/1703698340 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe6377fe700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171d710 0x199f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe637fff700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1768790 0x1980cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 -- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171d710 msgr2=0x199f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171d710 0x199f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 -- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1768790 msgr2=0x1980cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1768790 0x1980cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 -- 172.21.15.50:0/1703698340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe624003570 con 0x171dae0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe62effd700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe620009c20 tx=0x7fe62001f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c61a9562de4a275 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.920+0000 7fe615ffb700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62000bed0 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1863cc0 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe620027e40 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe62001c400 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 10 client.4746 fetch_fsmap learned FSMap version 8 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x7fe6381508a0 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1863cc0 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7fe62001c560 con 0x171dae0 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 10 client.4746 fetch_fsmap finished waiting for FSMap version 8 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 10 client.4746 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 20 client.4746 populate_metadata read hostname 'smithi050' 2023-12-19T14:39:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1982280 0x19aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 4 client.4746 mds_command: new command op to 4275 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:39:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 --> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x18673e0 con 0x1982280 2023-12-19T14:39:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe6377fe700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1982280 0x19aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe6377fe700 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1982280 0x19aa6b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8276168823aebb9 server_cookie=3f004fd5def259c8 in_seq=0 out_seq=0 2023-12-19T14:39:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 10 client.4746 ms_handle_connect on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 1 -- 172.21.15.50:0/1703698340 <== mds.0 v2:172.21.15.50:6833/3977471945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x18673e0 con 0x1982280 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 10 client.4746 handle_command_reply: tid=0 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 client.4746 shutdown 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 msgr2=0x199fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe620009c20 tx=0x7fe62001f070 comp rx=0 tx=0).stop 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 shutdown_connections 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1982280 0x19aa6b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1768790 0x1980cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171dae0 0x199fa50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/1703698340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171d710 0x199f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe615ffb700 0 client.4746 ms_handle_reset on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.924+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 >> 172.21.15.50:0/1703698340 conn(0x171deb0 msgr2=0x194ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.928+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 shutdown_connections 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.928+0000 7fe63fd6a740 1 -- 172.21.15.50:0/1703698340 wait complete. 2023-12-19T14:39:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.928+0000 7fe63fd6a740 20 client.4746 trim_cache size 0 max 16384 2023-12-19T14:39:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 Processor -- start 2023-12-19T14:39:34.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- start start 2023-12-19T14:39:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x1951ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x1952010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x19810b0 con 0x1768790 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1898040 con 0x171d710 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x18c2890 con 0x171dae0 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59328/0 (socket says 172.21.15.50:59328) 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 -- 172.21.15.50:0/2927531202 learned_addr learned my addr 172.21.15.50:0/2927531202 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 -- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 msgr2=0x1952010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x1952010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 -- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 msgr2=0x1951ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x1951ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 -- 172.21.15.50:0/2927531202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1893370 con 0x1768790 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe637fff700 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe630001ea0 tx=0x7fe630009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc692c9148392b9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe62d7fa700 1 -- 172.21.15.50:0/2927531202 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630005e70 con 0x1768790 2023-12-19T14:39:34.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe62d7fa700 1 -- 172.21.15.50:0/2927531202 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6300076a0 con 0x1768790 2023-12-19T14:39:34.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe62d7fa700 1 -- 172.21.15.50:0/2927531202 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630016630 con 0x1768790 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- 172.21.15.50:0/2927531202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 msgr2=0x199a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe630001ea0 tx=0x7fe630009040 comp rx=0 tx=0).stop 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- 172.21.15.50:0/2927531202 shutdown_connections 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1768790 0x199a710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x171dae0 0x1952010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/2927531202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171d710 0x1951ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- 172.21.15.50:0/2927531202 >> 172.21.15.50:0/2927531202 conn(0x171deb0 msgr2=0x19511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- 172.21.15.50:0/2927531202 shutdown_connections 2023-12-19T14:39:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.948+0000 7fe63fd6a740 1 -- 172.21.15.50:0/2927531202 wait complete. 2023-12-19T14:39:34.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 -- start start 2023-12-19T14:39:34.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 Processor -- start 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 0x19544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1768790 0x1954a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x189a550 con 0x171d710 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x189cd60 con 0x171dae0 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.952+0000 7fe63fd6a740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1768ea0 con 0x1768790 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 10 client.0 _ll_register_callbacks cb 0x16e99d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59330/0 (socket says 172.21.15.50:59330) 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 -- 172.21.15.50:0/120171124 learned_addr learned my addr 172.21.15.50:0/120171124 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:34.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 -- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1768790 msgr2=0x1954a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1768790 0x1954a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 -- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 msgr2=0x19544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe62effd700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 0x19544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 0x19544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 -- 172.21.15.50:0/120171124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630001c50 con 0x171d710 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe62effd700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 0x19544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe6377fe700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe628007fa0 tx=0x7fe62807a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df6911bf588ff22 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62804e930 con 0x171d710 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe628011ab0 con 0x171d710 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe628050590 con 0x171d710 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe62804fc90 con 0x171d710 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 10 client.4752 fetch_fsmap learned FSMap version 8 2023-12-19T14:39:34.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x1879d80 con 0x171d710 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628050590 con 0x171d710 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7fe6280570e0 con 0x171d710 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 10 client.4752 fetch_fsmap finished waiting for FSMap version 8 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 10 client.4752 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 20 client.4752 populate_metadata read hostname 'smithi050' 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1954f40 0x19f4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 4 client.4752 mds_command: new command op to 4275 tid=0[{"prefix": "status"}] 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 --> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1899410 con 0x1954f40 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe62effd700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1954f40 0x19f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe615ffb700 10 client.4752 ms_handle_connect on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.956+0000 7fe62effd700 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1954f40 0x19f4740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b210ab158060388e server_cookie=7f33c4a24297c932 in_seq=0 out_seq=0 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe615ffb700 1 -- 172.21.15.50:0/120171124 <== mds.0 v2:172.21.15.50:6833/3977471945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x1899410 con 0x1954f40 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe615ffb700 10 client.4752 handle_command_reply: tid=0 2023-12-19T14:39:34.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 client.4752 shutdown 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 msgr2=0x1950cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe628007fa0 tx=0x7fe62807a250 comp rx=0 tx=0).stop 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 shutdown_connections 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:6833/3977471945,v1:172.21.15.50:6835/3977471945] conn(0x1954f40 0x19f4740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x171dae0 0x19544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1768790 0x1954a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 --2- 172.21.15.50:0/120171124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x171d710 0x1950cf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe615ffb700 0 client.4752 ms_handle_reset on v2:172.21.15.50:6833/3977471945 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 >> 172.21.15.50:0/120171124 conn(0x171deb0 msgr2=0x1952570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 shutdown_connections 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 1 -- 172.21.15.50:0/120171124 wait complete. 2023-12-19T14:39:34.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63fd6a740 20 client.4752 trim_cache size 0 max 16384 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 4275, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 27.697357145000002, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 8, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 24, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 24, 2023-12-19T14:39:34.968 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 33.250376103999997 2023-12-19T14:39:34.969 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:39:34.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 msgr2=0x7fe618062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 0x7fe618062a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe620009460 tx=0x7fe620019000 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 msgr2=0x7fe638150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 0x7fe638150360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c4e0 tx=0x7fe628007d50 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 shutdown_connections 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe638158d60 0x7fe6380b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe618060580 0x7fe618062a30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6380c51b0 0x7fe6380afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 --2- 172.21.15.50:0/701086289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe63806b7a0 0x7fe638150360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 >> 172.21.15.50:0/701086289 conn(0x7fe638067bb0 msgr2=0x7fe6380cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 shutdown_connections 2023-12-19T14:39:34.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:34.960+0000 7fe63c816700 1 -- 172.21.15.50:0/701086289 wait complete. 2023-12-19T14:39:34.985 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-19T14:39:34.995 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-19T14:39:34.995 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-12-19T14:39:34.995 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-19T14:39:34.996 INFO:tasks.ceph_fuse:client.2 config is {} 2023-12-19T14:39:34.996 INFO:tasks.ceph_fuse:client.1 config is {} 2023-12-19T14:39:34.996 INFO:tasks.ceph_fuse:client.3 config is {} 2023-12-19T14:39:34.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:34.996 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-12-19T14:39:35.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2023-12-19T14:39:35.100 INFO:teuthology.orchestra.run.smithi125.stderr:Cannot find device "ceph-brx" 2023-12-19T14:39:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:39:35.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.104 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:39:35.124 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-12-19T14:39:35.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-12-19T14:39:35.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link delete ceph-brx 2023-12-19T14:39:35.386 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-19T14:39:35.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:39:35.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.540+0000 7f8f284f0700 1 Processor -- start 2023-12-19T14:39:35.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.540+0000 7f8f284f0700 1 -- start start 2023-12-19T14:39:35.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f284f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f20140710 0x7f8f200c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:35.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f284f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2006e150 con 0x7f8f20068800 2023-12-19T14:39:35.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f284f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2006d540 con 0x7f8f20140ae0 2023-12-19T14:39:35.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f284f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f2006d820 con 0x7f8f20140710 2023-12-19T14:39:35.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f20140710 0x7f8f200c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:35.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f20140710 0x7f8f200c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59340/0 (socket says 172.21.15.50:59340) 2023-12-19T14:39:35.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 -- 172.21.15.50:0/2776776561 learned_addr learned my addr 172.21.15.50:0/2776776561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:35.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 -- 172.21.15.50:0/2776776561 >> v1:172.21.15.125:6789/0 conn(0x7f8f20140ae0 legacy=0x7f8f200680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:35.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 -- 172.21.15.50:0/2776776561 >> v1:172.21.15.50:6789/0 conn(0x7f8f20068800 legacy=0x7f8f200ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2528a700 1 -- 172.21.15.50:0/2776776561 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443880534 0 0) 0x7f8f2006e150 con 0x7f8f20068800 2023-12-19T14:39:35.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 -- 172.21.15.50:0/2776776561 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2006db00 con 0x7f8f20140710 2023-12-19T14:39:35.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2628c700 1 --2- 172.21.15.50:0/2776776561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f200c70a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009f20 tx=0x7f8f1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76643ccf844dd48 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:35.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2528a700 1 -- 172.21.15.50:0/2776776561 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c00e070 con 0x7f8f20140710 2023-12-19T14:39:35.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2528a700 1 -- 172.21.15.50:0/2776776561 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c00ba40 con 0x7f8f20140710 2023-12-19T14:39:35.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.544+0000 7f8f2528a700 1 -- 172.21.15.50:0/2776776561 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0092e0 con 0x7f8f20140710 2023-12-19T14:39:35.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 -- 172.21.15.50:0/2776776561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 msgr2=0x7f8f200c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2776776561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f200c70a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009f20 tx=0x7f8f1c009840 comp rx=0 tx=0).stop 2023-12-19T14:39:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 -- 172.21.15.50:0/2776776561 shutdown_connections 2023-12-19T14:39:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2776776561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f200c70a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 -- 172.21.15.50:0/2776776561 >> 172.21.15.50:0/2776776561 conn(0x7f8f200c8850 msgr2=0x7f8f200c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 -- 172.21.15.50:0/2776776561 shutdown_connections 2023-12-19T14:39:35.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.548+0000 7f8f284f0700 1 -- 172.21.15.50:0/2776776561 wait complete. 2023-12-19T14:39:35.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 Processor -- start 2023-12-19T14:39:35.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 -- start start 2023-12-19T14:39:35.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 0x7f8f200c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:35.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:35.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f20140ae0 0x7f8f2013f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:35.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2013fde0 con 0x7f8f20140710 2023-12-19T14:39:35.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f200c2f30 con 0x7f8f20068800 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f284f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f200c3210 con 0x7f8f20140ae0 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f25a8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f25a8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59356/0 (socket says 172.21.15.50:59356) 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f25a8b700 1 -- 172.21.15.50:0/2132835152 learned_addr learned my addr 172.21.15.50:0/2132835152 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f2628c700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 0x7f8f200c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.552+0000 7f8f26a8d700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f20140ae0 0x7f8f2013f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 -- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f20140ae0 msgr2=0x7f8f2013f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f20140ae0 0x7f8f2013f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 -- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 msgr2=0x7f8f200c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 0x7f8f200c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 -- 172.21.15.50:0/2132835152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c009ae0 con 0x7f8f20140710 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f2628c700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 0x7f8f200c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f25a8b700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c5d0 tx=0x7f8f10007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:35.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10011590 con 0x7f8f20140710 2023-12-19T14:39:35.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10011710 con 0x7f8f20140710 2023-12-19T14:39:35.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f200c34f0 con 0x7f8f20140710 2023-12-19T14:39:35.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10011ad0 con 0x7f8f20140710 2023-12-19T14:39:35.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f200c3b30 con 0x7f8f20140710 2023-12-19T14:39:35.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.556+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f200c3cc0 con 0x7f8f20140710 2023-12-19T14:39:35.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.560+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8f1001c460 con 0x7f8f20140710 2023-12-19T14:39:35.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.560+0000 7f8f167fc700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 0x7f8f04062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:35.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.560+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8f1007c8c0 con 0x7f8f20140710 2023-12-19T14:39:35.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.560+0000 7f8f2628c700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 0x7f8f04062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:35.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.564+0000 7f8f2628c700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 0x7f8f04062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009ab0 tx=0x7f8f1c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:35.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.568+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f1004cb20 con 0x7f8f20140710 2023-12-19T14:39:35.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.780+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8f20071120 con 0x7f8f20140710 2023-12-19T14:39:35.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.780+0000 7f8f167fc700 1 -- 172.21.15.50:0/2132835152 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f8f10050140 con 0x7f8f20140710 2023-12-19T14:39:35.784 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 msgr2=0x7f8f04062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 0x7f8f04062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009ab0 tx=0x7f8f1c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 msgr2=0x7f8f2013f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c5d0 tx=0x7f8f10007f60 comp rx=0 tx=0).stop 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 shutdown_connections 2023-12-19T14:39:35.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f04060550 0x7f8f04062a00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f20140ae0 0x7f8f2013f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f20140710 0x7f8f2013f360 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 --2- 172.21.15.50:0/2132835152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f20068800 0x7f8f200c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 >> 172.21.15.50:0/2132835152 conn(0x7f8f200c8850 msgr2=0x7f8f200ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 shutdown_connections 2023-12-19T14:39:35.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:35.784+0000 7f8f284f0700 1 -- 172.21.15.50:0/2132835152 wait complete. 2023-12-19T14:39:35.807 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:39:35.807 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:39:35.807 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:39:35.808 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:39:35.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.808 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:39:35.830 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 42374sec preferred_lft 42374sec 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:39:35.832 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:39:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:39:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:35.833 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T14:39:35.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.833 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:35.833 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:35.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add name ceph-brx type bridge 2023-12-19T14:39:35.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr flush dev ceph-brx 2023-12-19T14:39:35.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set ceph-brx up 2023-12-19T14:39:35.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T14:39:35.834 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:35.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:35.983 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:39:36.019 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:39:36.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.022 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:39:36.089 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:39:36.090 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:39:36.090 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:39:36.090 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:39:36.090 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:39:36.090 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:39:36.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:39:36.091 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:36.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.230 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:39:36.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.251 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list-id 2023-12-19T14:39:36.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.314 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:36.314 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:36.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:39:36.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-19T14:39:36.314 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:36.443 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-19T14:39:36.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.443 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:36.443 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:36.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-19T14:39:36.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T14:39:36.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-19T14:39:36.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-19T14:39:36.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-19T14:39:36.444 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:36.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:36.596 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:36.596 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:36.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:39:36.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T14:39:36.596 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:36.668 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:39:36.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:36.668 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:36.687 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:39:36.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:36.688 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:36.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:36.833 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:36.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:36.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:36.913 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:36.976 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:39:36.976 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:39:37.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:37.063 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:37.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:37.141 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:37.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:37.145 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:37.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:39:37.156+0000 7fcbdb4dc380 -1 init, newargv = 0x56106599efc0 newargc=15 2023-12-19T14:39:37.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[40214]: starting ceph client 2023-12-19T14:39:37.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[40214]: starting fuse 2023-12-19T14:39:37.176 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:39:37.177 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:39:37.177 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:39:37.178 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:39:37.179 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:37.179 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:39:37.179 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:39:37.341 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.40214.asok 2023-12-19T14:39:37.523 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40214.asok 2023-12-19T14:39:37.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:37.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40214.asok status 2023-12-19T14:39:37.707 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "40214", 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 4699, 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:39:37.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4699 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 447976103 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 447976103 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:37.709 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.4699 172.21.15.50:0/447976103", 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "172.21.15.50:0/447976103", 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 8, 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 24, 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:39:37.710 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:39:37.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:39:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.884+0000 7fe960af0700 1 Processor -- start 2023-12-19T14:39:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe960af0700 1 -- start start 2023-12-19T14:39:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe960af0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe95c1592b0 0x7fe95c06d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe960af0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c0688d0 con 0x7fe95c06d590 2023-12-19T14:39:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe960af0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c0b1510 con 0x7fe95c158ee0 2023-12-19T14:39:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe960af0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95c06d9b0 con 0x7fe95c1592b0 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe953fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe95c1592b0 0x7fe95c06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe953fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe95c1592b0 0x7fe95c06d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34100/0 (socket says 172.21.15.50:34100) 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe953fff700 1 -- 172.21.15.50:0/702795004 learned_addr learned my addr 172.21.15.50:0/702795004 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe953fff700 1 -- 172.21.15.50:0/702795004 >> v1:172.21.15.125:6789/0 conn(0x7fe95c158ee0 legacy=0x7fe95c067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.888+0000 7fe953fff700 1 -- 172.21.15.50:0/702795004 >> v1:172.21.15.50:6789/0 conn(0x7fe95c06d590 legacy=0x7fe95c0b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe953fff700 1 -- 172.21.15.50:0/702795004 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c06a180 con 0x7fe95c1592b0 2023-12-19T14:39:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe953fff700 1 --2- 172.21.15.50:0/702795004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c1592b0 0x7fe95c06d050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9440099b0 tx=0x7fe944009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af435f842382dd2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe959d9b700 1 -- 172.21.15.50:0/702795004 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400b860 con 0x7fe95c1592b0 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe959d9b700 1 -- 172.21.15.50:0/702795004 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94400b9c0 con 0x7fe95c1592b0 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe959d9b700 1 -- 172.21.15.50:0/702795004 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400bd60 con 0x7fe95c1592b0 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 -- 172.21.15.50:0/702795004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c1592b0 msgr2=0x7fe95c06d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 --2- 172.21.15.50:0/702795004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c1592b0 0x7fe95c06d050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9440099b0 tx=0x7fe944009250 comp rx=0 tx=0).stop 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 -- 172.21.15.50:0/702795004 shutdown_connections 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 --2- 172.21.15.50:0/702795004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c1592b0 0x7fe95c06d050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 -- 172.21.15.50:0/702795004 >> 172.21.15.50:0/702795004 conn(0x7fe95c0cd990 msgr2=0x7fe95c0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:37.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 -- 172.21.15.50:0/702795004 shutdown_connections 2023-12-19T14:39:37.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.892+0000 7fe960af0700 1 -- 172.21.15.50:0/702795004 wait complete. 2023-12-19T14:39:37.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 Processor -- start 2023-12-19T14:39:37.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- start start 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe95c158ee0 0x7fe95c0a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 0x7fe95c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0a5a60 con 0x7fe95c06d590 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c150230 con 0x7fe95c158ee0 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0a51c0 con 0x7fe95c1592b0 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34106/0 (socket says 172.21.15.50:34106) 2023-12-19T14:39:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 -- 172.21.15.50:0/311905710 learned_addr learned my addr 172.21.15.50:0/311905710 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe953fff700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe95c158ee0 0x7fe95c0a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95ad9d700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 0x7fe95c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:37.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 -- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 msgr2=0x7fe95c0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:37.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 0x7fe95c0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:37.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 -- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe95c158ee0 msgr2=0x7fe95c0a69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:37.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe95c158ee0 0x7fe95c0a69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 -- 172.21.15.50:0/311905710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9440094f0 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95ad9d700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 0x7fe95c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe95a59c700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c5c0 tx=0x7fe94c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c011590 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c011710 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95c0a54a0 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c011ad0 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.896+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95c0a4e00 con 0x7fe95c06d590 2023-12-19T14:39:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.900+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95c150000 con 0x7fe95c06d590 2023-12-19T14:39:37.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.900+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe94c01b3b0 con 0x7fe95c06d590 2023-12-19T14:39:37.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.900+0000 7fe9527fc700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 0x7fe93c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:37.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.904+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe94c07bff0 con 0x7fe95c06d590 2023-12-19T14:39:37.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.904+0000 7fe953fff700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 0x7fe93c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:37.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.904+0000 7fe953fff700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 0x7fe93c062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9440099b0 tx=0x7fe944017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:37.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:37.908+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe95c150000 con 0x7fe95c06d590 2023-12-19T14:39:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.120+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe95c158b10 con 0x7fe95c06d590 2023-12-19T14:39:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe9527fc700 1 -- 172.21.15.50:0/311905710 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe95c158b10 con 0x7fe95c06d590 2023-12-19T14:39:38.126 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T14:39:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 msgr2=0x7fe93c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 0x7fe93c062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9440099b0 tx=0x7fe944017000 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 msgr2=0x7fe95c145d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c5c0 tx=0x7fe94c00c8d0 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 shutdown_connections 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe95c158ee0 0x7fe95c0a69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe93c060550 0x7fe93c062a00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe95c1592b0 0x7fe95c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 --2- 172.21.15.50:0/311905710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe95c06d590 0x7fe95c145d10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 >> 172.21.15.50:0/311905710 conn(0x7fe95c0cd990 msgr2=0x7fe95c0ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 shutdown_connections 2023-12-19T14:39:38.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:38.124+0000 7fe960af0700 1 -- 172.21.15.50:0/311905710 wait complete. 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:38.151 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:39:38.152 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:39:38.152 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:39:38.152 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-12-19T14:39:38.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.152 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:39:38.175 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 42372sec preferred_lft 42372sec 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:39:38.176 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:39:38.177 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:39:38.178 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link tentative 2023-12-19T14:39:38.178 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.178 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:39:38.260 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:39:38.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.263 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:39:38.332 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:39:38.332 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:39:38.332 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:39:38.332 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:39:38.333 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:39:38.333 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:39:38.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:39:38.334 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:38.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.475 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:39:38.492 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:39:38.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.493 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list-id 2023-12-19T14:39:38.556 INFO:teuthology.orchestra.run.smithi050.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-12-19T14:39:38.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.557 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:38.557 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:38.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:39:38.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-12-19T14:39:38.557 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:38.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-19T14:39:38.712 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.712 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 2a:ad:44:70:79:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::28ad:44ff:fe70:794c/64 scope link 2023-12-19T14:39:38.713 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-19T14:39:38.807 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 2a:ad:44:70:79:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T14:39:38.808 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T14:39:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::28ad:44ff:fe70:794c/64 scope link 2023-12-19T14:39:38.809 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:38.813 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-12-19T14:39:38.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:38.813 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:38.813 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:38.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-12-19T14:39:38.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-19T14:39:38.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-12-19T14:39:38.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-12-19T14:39:38.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-12-19T14:39:38.814 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:39.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:39.016 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:39:39.017 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:39:39.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.1 up 2023-12-19T14:39:39.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set dev brx.1 master ceph-brx 2023-12-19T14:39:39.017 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:39:39.091 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-12-19T14:39:39.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:39.091 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:39.105 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-12-19T14:39:39.106 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:39.106 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:39.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:39.258 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:39.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:39.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:39.338 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:39:39.403 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [49] 2023-12-19T14:39:39.403 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-12-19T14:39:39.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:39:39.498 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:39.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:39.573 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:39.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:39.576 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:39:39.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi050.stderr:2023-12-19T14:39:39.592+0000 7fcea8bc4380 -1 init, newargv = 0x562304c1af40 newargc=15 2023-12-19T14:39:39.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi050.stderr:ceph-fuse[40350]: starting ceph client 2023-12-19T14:39:39.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi050.stderr:ceph-fuse[40350]: starting fuse 2023-12-19T14:39:39.614 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:39:39.614 INFO:teuthology.orchestra.run.smithi050.stdout:50 2023-12-19T14:39:39.614 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49, 50] 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:39.614 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:39:39.615 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:39:39.616 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:39:39.616 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:39:39.616 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.2")) 2023-12-19T14:39:39.616 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:39:39.780 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.2.40350.asok 2023-12-19T14:39:39.964 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.40350.asok 2023-12-19T14:39:39.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:39.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.40350.asok status 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "2", 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:39:40.157 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "40350", 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 4779, 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:39:40.158 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4779 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.2:0", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2129337042 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.2:0", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2129337042 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.4779 192.168.0.2:0/2129337042", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.2:0/2129337042", 2023-12-19T14:39:40.159 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 8, 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 24, 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:39:40.160 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:39:40.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:39:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.328+0000 7fe018a6e700 1 Processor -- start 2023-12-19T14:39:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.328+0000 7fe018a6e700 1 -- start start 2023-12-19T14:39:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe018a6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0140ae510 0x7fe0140a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe018a6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140c1d60 0x7fe0140680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe018a6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01406dc20 con 0x7fe0140ae140 2023-12-19T14:39:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe018a6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0140c2200 con 0x7fe0140ae510 2023-12-19T14:39:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe018a6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01406d940 con 0x7fe0140c1d60 2023-12-19T14:39:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe01259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe0140ae140 0x7fe01406d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44356/0 (socket says 172.21.15.50:44356) 2023-12-19T14:39:40.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe01259c700 1 -- 172.21.15.50:0/14228336 learned_addr learned my addr 172.21.15.50:0/14228336 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 --2- 172.21.15.50:0/14228336 >> v2:172.21.15.50:3300/0 conn(0x7fe0140ae510 0x7fe0140a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe012d9d700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140c1d60 0x7fe0140680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe011d9b700 1 -- 172.21.15.50:0/14228336 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617545516 0 0) 0x7fe01406dc20 con 0x7fe0140ae140 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe011d9b700 1 -- 172.21.15.50:0/14228336 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe000003570 con 0x7fe0140ae140 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 -- 172.21.15.50:0/14228336 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140c1d60 msgr2=0x7fe0140680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140c1d60 0x7fe0140680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 -- 172.21.15.50:0/14228336 >> v1:172.21.15.50:6789/0 conn(0x7fe0140ae140 legacy=0x7fe01406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 -- 172.21.15.50:0/14228336 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140c6e20 con 0x7fe0140ae510 2023-12-19T14:39:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.332+0000 7fe009d9b700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140ae510 0x7fe0140a8850 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818de95a1170d273 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe011d9b700 1 -- 172.21.15.50:0/14228336 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc011630 con 0x7fe0140ae510 2023-12-19T14:39:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe011d9b700 1 -- 172.21.15.50:0/14228336 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc011790 con 0x7fe0140ae510 2023-12-19T14:39:40.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe011d9b700 1 -- 172.21.15.50:0/14228336 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc011b30 con 0x7fe0140ae510 2023-12-19T14:39:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 -- 172.21.15.50:0/14228336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140ae510 msgr2=0x7fe0140a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140ae510 0x7fe0140a8850 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 -- 172.21.15.50:0/14228336 shutdown_connections 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140c1d60 0x7fe0140680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 --2- 172.21.15.50:0/14228336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140ae510 0x7fe0140a8850 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 -- 172.21.15.50:0/14228336 >> 172.21.15.50:0/14228336 conn(0x7fe0140c8850 msgr2=0x7fe0140c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 -- 172.21.15.50:0/14228336 shutdown_connections 2023-12-19T14:39:40.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.336+0000 7fe018a6e700 1 -- 172.21.15.50:0/14228336 wait complete. 2023-12-19T14:39:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 Processor -- start 2023-12-19T14:39:40.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 -- start start 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 0x7fe014169e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 0x7fe0140c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c9ce0 con 0x7fe0140c1d60 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0141666a0 con 0x7fe0140ae510 2023-12-19T14:39:40.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe018a6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c9f30 con 0x7fe0140ae140 2023-12-19T14:39:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe012d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe012d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34130/0 (socket says 172.21.15.50:34130) 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.340+0000 7fe012d9d700 1 -- 172.21.15.50:0/1351953871 learned_addr learned my addr 172.21.15.50:0/1351953871 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe01259c700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 0x7fe014169e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe009d9b700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 0x7fe0140c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 -- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 msgr2=0x7fe014169e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 0x7fe014169e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 -- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 msgr2=0x7fe0140c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 0x7fe0140c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 -- 172.21.15.50:0/1351953871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc009420 con 0x7fe0140c1d60 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe009d9b700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 0x7fe0140c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe01259c700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 0x7fe014169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe012d9d700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009e90 tx=0x7fe00c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c00cc20 con 0x7fe0140c1d60 2023-12-19T14:39:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c00f040 con 0x7fe0140c1d60 2023-12-19T14:39:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0140ca210 con 0x7fe0140c1d60 2023-12-19T14:39:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c014480 con 0x7fe0140c1d60 2023-12-19T14:39:40.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0140ca7d0 con 0x7fe0140c1d60 2023-12-19T14:39:40.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.344+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe014071120 con 0x7fe0140c1d60 2023-12-19T14:39:40.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.348+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe00c014830 con 0x7fe0140c1d60 2023-12-19T14:39:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.348+0000 7fe00affd700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 0x7fdff4062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.348+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe00c07d4e0 con 0x7fe0140c1d60 2023-12-19T14:39:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.348+0000 7fe01259c700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 0x7fdff4062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:40.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.352+0000 7fe01259c700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 0x7fdff4062ad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0140cbd10 tx=0x7fe004009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.356+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe00c051250 con 0x7fe0140c1d60 2023-12-19T14:39:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.564+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe0140642a0 con 0x7fe0140c1d60 2023-12-19T14:39:40.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.568+0000 7fe00affd700 1 -- 172.21.15.50:0/1351953871 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe00c050de0 con 0x7fe0140c1d60 2023-12-19T14:39:40.570 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 msgr2=0x7fdff4062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 0x7fdff4062ad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe0140cbd10 tx=0x7fe004009040 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 msgr2=0x7fe0140cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009e90 tx=0x7fe00c00b040 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 shutdown_connections 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0140ae510 0x7fe0140c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdff4060620 0x7fdff4062ad0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0140c1d60 0x7fe0140cacf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 --2- 172.21.15.50:0/1351953871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0140ae140 0x7fe014169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 >> 172.21.15.50:0/1351953871 conn(0x7fe0140c8850 msgr2=0x7fe0141572b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 shutdown_connections 2023-12-19T14:39:40.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:40.572+0000 7fe018a6e700 1 -- 172.21.15.50:0/1351953871 wait complete. 2023-12-19T14:39:40.596 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:39:40.596 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:39:40.597 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-19T14:39:40.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:40.597 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2023-12-19T14:39:40.619 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:40.619 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:39:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42390sec preferred_lft 42390sec 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 02:42:bc:fb:08:ac brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:39:40.621 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:40.622 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T14:39:40.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:40.622 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:40.622 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:40.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add name ceph-brx type bridge 2023-12-19T14:39:40.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr flush dev ceph-brx 2023-12-19T14:39:40.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set ceph-brx up 2023-12-19T14:39:40.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T14:39:40.623 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:40.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:40.775 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:39:40.805 INFO:teuthology.orchestra.run.smithi125.stdout:1 2023-12-19T14:39:40.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:40.808 DEBUG:teuthology.orchestra.run.smithi125:> route 2023-12-19T14:39:40.890 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2023-12-19T14:39:40.890 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:39:40.890 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T14:39:40.890 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T14:39:40.891 INFO:teuthology.orchestra.run.smithi125.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:39:40.891 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T14:39:40.891 INFO:teuthology.orchestra.run.smithi125.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T14:39:40.891 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:39:40.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:40.891 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:40.891 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:40.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T14:39:40.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T14:39:40.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T14:39:40.892 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:41.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:41.027 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-12-19T14:39:41.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:41.044 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list-id 2023-12-19T14:39:41.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:41.113 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:41.113 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:41.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T14:39:41.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-19T14:39:41.113 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:41.226 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-19T14:39:41.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-19T14:39:41.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-19T14:39:41.227 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:41.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:41.404 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:41.405 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:41.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2023-12-19T14:39:41.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T14:39:41.405 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:41.474 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-12-19T14:39:41.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:41.474 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:41.491 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-19T14:39:41.491 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:41.492 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:41.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2023-12-19T14:39:41.632 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:41.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:41.714 INFO:teuthology.orchestra.run.smithi125.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:41.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:41.718 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2023-12-19T14:39:41.781 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:39:41.781 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-12-19T14:39:41.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2023-12-19T14:39:41.859 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:41.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:41.931 INFO:teuthology.orchestra.run.smithi125.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:41.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:41.933 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2023-12-19T14:39:41.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi125.stderr:2023-12-19T14:39:41.956+0000 7f7f19f7d380 -1 init, newargv = 0x5592113eb080 newargc=15 2023-12-19T14:39:41.961 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi125.stderr:ceph-fuse[39518]: starting ceph client 2023-12-19T14:39:41.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi125.stderr:ceph-fuse[39518]: starting fuse 2023-12-19T14:39:43.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2023-12-19T14:39:43.112 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:43.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:43.192 INFO:teuthology.orchestra.run.smithi125.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:43.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:43.195 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2023-12-19T14:39:43.264 INFO:teuthology.orchestra.run.smithi125.stdout:49 2023-12-19T14:39:43.264 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> import re 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> import os 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2023-12-19T14:39:43.265 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2023-12-19T14:39:43.266 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:39:43.267 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> return f 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.1")) 2023-12-19T14:39:43.268 DEBUG:teuthology.orchestra.run.smithi125:> ' 2023-12-19T14:39:43.430 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.1.39518.asok 2023-12-19T14:39:43.611 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.39518.asok 2023-12-19T14:39:43.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:43.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.39518.asok status 2023-12-19T14:39:43.803 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-12-19T14:39:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2023-12-19T14:39:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:39:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "1", 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "39518", 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2023-12-19T14:39:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4619, 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 4619 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2826671898 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: } 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2826671898 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.4619 192.168.0.1:0/2826671898", 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.1:0/2826671898", 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 8, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 24, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2023-12-19T14:39:43.806 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-12-19T14:39:43.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:39:43.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.980+0000 7f6d1f42e700 1 Processor -- start 2023-12-19T14:39:43.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 -- start start 2023-12-19T14:39:43.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18158f70 0x7f6d18067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:43.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6d18159340 0x7f6d180cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:43.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d18068c70 con 0x7f6d18069100 2023-12-19T14:39:43.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d18068060 con 0x7f6d18159340 2023-12-19T14:39:43.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18068340 con 0x7f6d18158f70 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1c9c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6d18069100 0x7f6d180cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44362/0 (socket says 172.21.15.50:44362) 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1c9c9700 1 -- 172.21.15.50:0/655789165 learned_addr learned my addr 172.21.15.50:0/655789165 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1d9cb700 1 --2- 172.21.15.50:0/655789165 >> v2:172.21.15.50:3300/0 conn(0x7f6d18159340 0x7f6d180cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d1d1ca700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18158f70 0x7f6d18067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.984+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436521292 0 0) 0x7f6d18068c70 con 0x7f6d18069100 2023-12-19T14:39:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04003570 con 0x7f6d18069100 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1d9cb700 1 -- 172.21.15.50:0/655789165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18158f70 msgr2=0x7f6d18067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1d9cb700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18158f70 0x7f6d18067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1d9cb700 1 -- 172.21.15.50:0/655789165 >> v1:172.21.15.50:6789/0 conn(0x7f6d18069100 legacy=0x7f6d180cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775482916 0 0) 0x7f6d04003570 con 0x7f6d18069100 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1d9cb700 1 -- 172.21.15.50:0/655789165 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18068620 con 0x7f6d18159340 2023-12-19T14:39:43.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1d9cb700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18159340 0x7f6d180cb3e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098e0 tx=0x7f6d100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59dffa0f97b599c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:43.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08001470 con 0x7f6d18159340 2023-12-19T14:39:43.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d10011420 con 0x7f6d18159340 2023-12-19T14:39:43.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d0ffff700 1 -- 172.21.15.50:0/655789165 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100117c0 con 0x7f6d18159340 2023-12-19T14:39:43.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 -- 172.21.15.50:0/655789165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18159340 msgr2=0x7f6d180cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:43.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18159340 0x7f6d180cb3e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098e0 tx=0x7f6d100091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 -- 172.21.15.50:0/655789165 shutdown_connections 2023-12-19T14:39:43.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18159340 0x7f6d180cb3e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/655789165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18158f70 0x7f6d18067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 -- 172.21.15.50:0/655789165 >> 172.21.15.50:0/655789165 conn(0x7f6d180ccc40 msgr2=0x7f6d180cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:43.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 -- 172.21.15.50:0/655789165 shutdown_connections 2023-12-19T14:39:43.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.988+0000 7f6d1f42e700 1 -- 172.21.15.50:0/655789165 wait complete. 2023-12-19T14:39:43.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 Processor -- start 2023-12-19T14:39:43.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 -- start start 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 0x7f6d18144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 0x7f6d18152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181528b0 con 0x7f6d18069100 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18151230 con 0x7f6d18158f70 2023-12-19T14:39:43.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18151ca0 con 0x7f6d18159340 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1d1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1d1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34142/0 (socket says 172.21.15.50:34142) 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.992+0000 7f6d1d1ca700 1 -- 172.21.15.50:0/3197533619 learned_addr learned my addr 172.21.15.50:0/3197533619 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1c9c9700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 0x7f6d18144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d9cb700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 0x7f6d18152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 -- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 msgr2=0x7f6d18152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 0x7f6d18152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 -- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 msgr2=0x7f6d18144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 0x7f6d18144bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 -- 172.21.15.50:0/3197533619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10009420 con 0x7f6d18069100 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1c9c9700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 0x7f6d18144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d9cb700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 0x7f6d18152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:43.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1d1ca700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009f10 tx=0x7f6d1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:44.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1400cc80 con 0x7f6d18069100 2023-12-19T14:39:44.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1400f040 con 0x7f6d18069100 2023-12-19T14:39:44.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d18151f80 con 0x7f6d18069100 2023-12-19T14:39:44.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14014480 con 0x7f6d18069100 2023-12-19T14:39:44.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d18152540 con 0x7f6d18069100 2023-12-19T14:39:44.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:43.996+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d18071120 con 0x7f6d18069100 2023-12-19T14:39:44.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.000+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6d14014830 con 0x7f6d18069100 2023-12-19T14:39:44.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.000+0000 7f6d0d7fa700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 0x7f6cfc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:44.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.000+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6d1407d620 con 0x7f6d18069100 2023-12-19T14:39:44.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.000+0000 7f6d1c9c9700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 0x7f6cfc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:44.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.004+0000 7f6d1c9c9700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 0x7f6cfc062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d080011e0 tx=0x7f6d08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:44.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.008+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6d1404d880 con 0x7f6d18069100 2023-12-19T14:39:44.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.220+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6d18068900 con 0x7f6d18069100 2023-12-19T14:39:44.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d0d7fa700 1 -- 172.21.15.50:0/3197533619 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6d14050ea0 con 0x7f6d18069100 2023-12-19T14:39:44.227 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T14:39:44.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 msgr2=0x7f6cfc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:44.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 0x7f6cfc062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d080011e0 tx=0x7f6d08009040 comp rx=0 tx=0).stop 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 msgr2=0x7f6d180c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009f10 tx=0x7f6d1400b040 comp rx=0 tx=0).stop 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 shutdown_connections 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d18158f70 0x7f6d18144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6cfc060550 0x7f6cfc062a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:44.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6d18159340 0x7f6d18152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:44.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 --2- 172.21.15.50:0/3197533619 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d18069100 0x7f6d180c9690 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:44.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 >> 172.21.15.50:0/3197533619 conn(0x7f6d180ccc40 msgr2=0x7f6d180ca780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:44.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 shutdown_connections 2023-12-19T14:39:44.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:44.224+0000 7f6d1f42e700 1 -- 172.21.15.50:0/3197533619 wait complete. 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:39:44.249 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:39:44.250 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-12-19T14:39:44.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.250 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2023-12-19T14:39:44.269 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:44.269 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.270 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42386sec preferred_lft 42386sec 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 02:42:bc:fb:08:ac brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 5e:92:c3:e9:b1:4a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::2031:18ff:fe06:b23a/64 scope link 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:39:44.271 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 5e:92:c3:e9:b1:4a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T14:39:44.272 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::5c92:c3ff:fee9:b14a/64 scope link 2023-12-19T14:39:44.272 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.272 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:39:44.352 INFO:teuthology.orchestra.run.smithi125.stdout:1 2023-12-19T14:39:44.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.356 DEBUG:teuthology.orchestra.run.smithi125:> route 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T14:39:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:39:44.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T14:39:44.424 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:44.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.562 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-12-19T14:39:44.579 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-19T14:39:44.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.581 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list-id 2023-12-19T14:39:44.638 INFO:teuthology.orchestra.run.smithi125.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-12-19T14:39:44.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.639 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:44.639 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:44.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-19T14:39:44.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-12-19T14:39:44.640 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:44.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-19T14:39:44.789 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 2a:d4:6f:33:a2:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.790 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::28d4:6fff:fe33:a216/64 scope link 2023-12-19T14:39:44.791 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2023-12-19T14:39:44.874 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 2a:d4:6f:33:a2:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::28d4:6fff:fe33:a216/64 scope link 2023-12-19T14:39:44.875 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:39:44.877 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-12-19T14:39:44.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:44.877 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:44.877 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:44.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-12-19T14:39:44.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-19T14:39:44.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-12-19T14:39:44.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-12-19T14:39:44.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-12-19T14:39:44.878 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:45.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:45.068 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-12-19T14:39:45.069 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-12-19T14:39:45.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.1 up 2023-12-19T14:39:45.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set dev brx.1 master ceph-brx 2023-12-19T14:39:45.069 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-12-19T14:39:45.141 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-12-19T14:39:45.141 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:45.141 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:45.157 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-12-19T14:39:45.158 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:39:45.158 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:45.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2023-12-19T14:39:45.300 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:45.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:45.381 INFO:teuthology.orchestra.run.smithi125.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:45.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:45.384 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2023-12-19T14:39:45.441 INFO:teuthology.orchestra.run.smithi125.stdout:49 2023-12-19T14:39:45.441 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [49] 2023-12-19T14:39:45.441 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-12-19T14:39:45.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo modprobe fuse 2023-12-19T14:39:45.521 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:39:45.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:39:45.593 INFO:teuthology.orchestra.run.smithi125.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:39:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:39:45.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:45.596 DEBUG:teuthology.orchestra.run.smithi125:> ls /sys/fs/fuse/connections 2023-12-19T14:39:45.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi125.stderr:2023-12-19T14:39:45.616+0000 7fe4648ee380 -1 init, newargv = 0x56028d3ba2a0 newargc=15 2023-12-19T14:39:45.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi125.stderr:ceph-fuse[39632]: starting ceph client 2023-12-19T14:39:45.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi125.stderr:ceph-fuse[39632]: starting fuse 2023-12-19T14:39:45.635 INFO:teuthology.orchestra.run.smithi125.stdout:49 2023-12-19T14:39:45.635 INFO:teuthology.orchestra.run.smithi125.stdout:50 2023-12-19T14:39:45.636 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49, 50] 2023-12-19T14:39:45.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:39:45.636 DEBUG:teuthology.orchestra.run.smithi125:> import glob 2023-12-19T14:39:45.636 DEBUG:teuthology.orchestra.run.smithi125:> import re 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> import os 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> import subprocess 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> def _find_admin_socket(client_name): 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> files = glob.glob(asok_path) 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> # Given a non-glob path, it better be there 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> if "*" not in asok_path: 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> assert(len(files) == 1) 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> return files[0] 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> for f in files: 2023-12-19T14:39:45.637 DEBUG:teuthology.orchestra.run.smithi125:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> contents = proc_f.read() 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> if mountpoint in contents: 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> return f 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> print(_find_admin_socket("client.3")) 2023-12-19T14:39:45.638 DEBUG:teuthology.orchestra.run.smithi125:> ' 2023-12-19T14:39:45.799 INFO:teuthology.orchestra.run.smithi125.stdout:/var/run/ceph/ceph-client.3.39632.asok 2023-12-19T14:39:45.982 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.39632.asok 2023-12-19T14:39:45.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:45.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.39632.asok status 2023-12-19T14:39:46.169 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-12-19T14:39:46.169 INFO:teuthology.orchestra.run.smithi125.stdout: "metadata": { 2023-12-19T14:39:46.169 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:39:46.169 INFO:teuthology.orchestra.run.smithi125.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:39:46.169 INFO:teuthology.orchestra.run.smithi125.stdout: "entity_id": "3", 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "hostname": "smithi125", 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "pid": "39632", 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "root": "/" 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_count": 0, 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "dentry_pinned_count": 0, 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4732, 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "inst": { 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "name": { 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "client", 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "num": 4732 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:46.170 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.2:0", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1948267778 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": { 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "192.168.0.2:0", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1948267778 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "inst_str": "client.4732 192.168.0.2:0/1948267778", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "addr_str": "192.168.0.2:0/1948267778", 2023-12-19T14:39:46.171 INFO:teuthology.orchestra.run.smithi125.stdout: "inode_count": 1, 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_epoch": 8, 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch": 24, 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklisted": false, 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs" 2023-12-19T14:39:46.172 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-12-19T14:39:46.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.183 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:39:46.183 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:46.204 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:39:46.204 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:46.204 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:46.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.282 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:39:46.282 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:46.349 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:39:46.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:46.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:46.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.430 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2023-12-19T14:39:46.430 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:46.450 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2023-12-19T14:39:46.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:46.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:46.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.532 DEBUG:teuthology.orchestra.run.smithi125:> stat --file-system '--printf=%T 2023-12-19T14:39:46.532 DEBUG:teuthology.orchestra.run.smithi125:> ' -- /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:46.597 INFO:teuthology.orchestra.run.smithi125.stdout:fuseblk 2023-12-19T14:39:46.597 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:46.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:39:46.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:46.675 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-12-19T14:39:46.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.852+0000 7f7e2c42e700 1 Processor -- start 2023-12-19T14:39:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 -- start start 2023-12-19T14:39:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7e240ae520 0x7f7e240c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7e24068530 0x7f7e2406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2406a9d0 con 0x7f7e240680f0 2023-12-19T14:39:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2406acb0 con 0x7f7e24068530 2023-12-19T14:39:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.856+0000 7f7e2c42e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2406af90 con 0x7f7e240ae520 2023-12-19T14:39:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7e24068530 0x7f7e2406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7e24068530 0x7f7e2406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57756/0 (socket says 172.21.15.50:57756) 2023-12-19T14:39:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 -- 172.21.15.50:0/2807011532 learned_addr learned my addr 172.21.15.50:0/2807011532 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a1ca700 1 --2- 172.21.15.50:0/2807011532 >> v2:172.21.15.125:3300/0 conn(0x7f7e240ae520 0x7f7e240c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 -- 172.21.15.50:0/2807011532 >> v2:172.21.15.125:3300/0 conn(0x7f7e240ae520 msgr2=0x7f7e240c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 --2- 172.21.15.50:0/2807011532 >> v2:172.21.15.125:3300/0 conn(0x7f7e240ae520 0x7f7e240c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 -- 172.21.15.50:0/2807011532 >> v1:172.21.15.125:6789/0 conn(0x7f7e240680f0 legacy=0x7f7e241552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 -- 172.21.15.50:0/2807011532 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2406b270 con 0x7f7e24068530 2023-12-19T14:39:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e291c8700 1 -- 172.21.15.50:0/2807011532 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814971702 0 0) 0x7f7e2406a9d0 con 0x7f7e240680f0 2023-12-19T14:39:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2a9cb700 1 --2- 172.21.15.50:0/2807011532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e2406c7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e24069700 tx=0x7f7e1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9118629f6e874f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:46.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e291c8700 1 -- 172.21.15.50:0/2807011532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00e070 con 0x7f7e24068530 2023-12-19T14:39:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e291c8700 1 -- 172.21.15.50:0/2807011532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c00ba20 con 0x7f7e24068530 2023-12-19T14:39:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e291c8700 1 -- 172.21.15.50:0/2807011532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0092e0 con 0x7f7e24068530 2023-12-19T14:39:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2c42e700 1 -- 172.21.15.50:0/2807011532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 msgr2=0x7f7e2406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:46.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.860+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/2807011532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e2406c7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e24069700 tx=0x7f7e1c009840 comp rx=0 tx=0).stop 2023-12-19T14:39:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 -- 172.21.15.50:0/2807011532 shutdown_connections 2023-12-19T14:39:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/2807011532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e2406c7c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/2807011532 >> v2:172.21.15.125:3300/0 conn(0x7f7e240ae520 0x7f7e240c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 -- 172.21.15.50:0/2807011532 >> 172.21.15.50:0/2807011532 conn(0x7f7e240c8850 msgr2=0x7f7e240c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:46.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 -- 172.21.15.50:0/2807011532 shutdown_connections 2023-12-19T14:39:46.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 -- 172.21.15.50:0/2807011532 wait complete. 2023-12-19T14:39:46.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 Processor -- start 2023-12-19T14:39:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.864+0000 7f7e2c42e700 1 -- start start 2023-12-19T14:39:46.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 0x7f7e240b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e240c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e240ae520 0x7f7e2414eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240aef10 con 0x7f7e24068530 2023-12-19T14:39:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240c5740 con 0x7f7e240680f0 2023-12-19T14:39:46.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240c5a20 con 0x7f7e240ae520 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e299c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e240c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e299c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e240c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57768/0 (socket says 172.21.15.50:57768) 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e299c9700 1 -- 172.21.15.50:0/1568468975 learned_addr learned my addr 172.21.15.50:0/1568468975 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 0x7f7e240b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a9cb700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e240ae520 0x7f7e2414eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 -- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e240ae520 msgr2=0x7f7e2414eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e240ae520 0x7f7e2414eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 -- 172.21.15.50:0/1568468975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 msgr2=0x7f7e240c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e240c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 -- 172.21.15.50:0/1568468975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c009ae0 con 0x7f7e240680f0 2023-12-19T14:39:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2a1ca700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 0x7f7e240b0ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000ea50 tx=0x7f7e200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:46.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000ee90 con 0x7f7e240680f0 2023-12-19T14:39:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2000c7f0 con 0x7f7e240680f0 2023-12-19T14:39:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240c5d00 con 0x7f7e240680f0 2023-12-19T14:39:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000cb90 con 0x7f7e240680f0 2023-12-19T14:39:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.868+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e240b10e0 con 0x7f7e240680f0 2023-12-19T14:39:46.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.872+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e24071120 con 0x7f7e240680f0 2023-12-19T14:39:46.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.872+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7e20005a00 con 0x7f7e240680f0 2023-12-19T14:39:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.876+0000 7f7e1a7fc700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 0x7f7e08062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.876+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7e2007d3b0 con 0x7f7e240680f0 2023-12-19T14:39:46.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.876+0000 7f7e299c9700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 0x7f7e08062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.876+0000 7f7e299c9700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 0x7f7e08062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e14003ad0 tx=0x7f7e14007f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:46.880+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7e2004d690 con 0x7f7e240680f0 2023-12-19T14:39:47.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.056+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e240642a0 con 0x7f7e240680f0 2023-12-19T14:39:47.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.060+0000 7f7e1a7fc700 1 -- 172.21.15.50:0/1568468975 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17191 (secure 0 0 0) 0x7f7e20017090 con 0x7f7e240680f0 2023-12-19T14:39:47.065 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:47.065 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:11.985475+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:47.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 msgr2=0x7f7e08062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 0x7f7e08062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e14003ad0 tx=0x7f7e14007f50 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 msgr2=0x7f7e240b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 0x7f7e240b0ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000ea50 tx=0x7f7e200091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 shutdown_connections 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7e08060550 0x7f7e08062a00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e240ae520 0x7f7e2414eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e24068530 0x7f7e240c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 --2- 172.21.15.50:0/1568468975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e240680f0 0x7f7e240b0ba0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 >> 172.21.15.50:0/1568468975 conn(0x7f7e240c8850 msgr2=0x7f7e240ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:47.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 shutdown_connections 2023-12-19T14:39:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.064+0000 7f7e2c42e700 1 -- 172.21.15.50:0/1568468975 wait complete. 2023-12-19T14:39:47.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:47.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:47.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.264+0000 7ff789c25700 1 Processor -- start 2023-12-19T14:39:47.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.264+0000 7ff789c25700 1 -- start start 2023-12-19T14:39:47.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff789c25700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff78406db40 0x7ff7840cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff789c25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff7840c63f0 0x7ff7840ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff789c25700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7840c9db0 con 0x7ff78406df10 2023-12-19T14:39:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff789c25700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7840cb2e0 con 0x7ff78406db40 2023-12-19T14:39:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff789c25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840cb5c0 con 0x7ff7840c63f0 2023-12-19T14:39:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff783fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff78406df10 0x7ff784159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35022/0 (socket says 172.21.15.50:35022) 2023-12-19T14:39:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff783fff700 1 -- 172.21.15.50:0/3760311805 learned_addr learned my addr 172.21.15.50:0/3760311805 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 --2- 172.21.15.50:0/3760311805 >> v2:172.21.15.125:3300/0 conn(0x7ff78406db40 0x7ff7840cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff77bfff700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff7840c63f0 0x7ff7840ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff782ffd700 1 -- 172.21.15.50:0/3760311805 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101242198 0 0) 0x7ff7840c9db0 con 0x7ff78406df10 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff782ffd700 1 -- 172.21.15.50:0/3760311805 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff770003570 con 0x7ff78406df10 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 -- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff7840c63f0 msgr2=0x7ff7840ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff7840c63f0 0x7ff7840ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 -- 172.21.15.50:0/3760311805 >> v1:172.21.15.50:6789/0 conn(0x7ff78406df10 legacy=0x7ff784159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 -- 172.21.15.50:0/3760311805 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7840cb8a0 con 0x7ff78406db40 2023-12-19T14:39:47.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff7837fe700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406db40 0x7ff7840cc3e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7740098e0 tx=0x7ff7740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f472153e89dcec71 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff782ffd700 1 -- 172.21.15.50:0/3760311805 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011630 con 0x7ff78406db40 2023-12-19T14:39:47.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff782ffd700 1 -- 172.21.15.50:0/3760311805 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774011790 con 0x7ff78406db40 2023-12-19T14:39:47.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.268+0000 7ff782ffd700 1 -- 172.21.15.50:0/3760311805 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011b30 con 0x7ff78406db40 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 -- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406db40 msgr2=0x7ff7840cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406db40 0x7ff7840cc3e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7740098e0 tx=0x7ff7740091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 -- 172.21.15.50:0/3760311805 shutdown_connections 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff7840c63f0 0x7ff7840ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 --2- 172.21.15.50:0/3760311805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406db40 0x7ff7840cc3e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 -- 172.21.15.50:0/3760311805 >> 172.21.15.50:0/3760311805 conn(0x7ff784067bb0 msgr2=0x7ff7840cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:47.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 -- 172.21.15.50:0/3760311805 shutdown_connections 2023-12-19T14:39:47.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.272+0000 7ff789c25700 1 -- 172.21.15.50:0/3760311805 wait complete. 2023-12-19T14:39:47.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 Processor -- start 2023-12-19T14:39:47.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 -- start start 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff78406db40 0x7ff7840c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406df10 0x7ff7841447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840afe20 con 0x7ff7840c63f0 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff784144d30 con 0x7ff78406df10 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff789c25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840c4b20 con 0x7ff78406db40 2023-12-19T14:39:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff7837fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff78406db40 0x7ff7840c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406df10 0x7ff7841447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57776/0 (socket says 172.21.15.50:57776) 2023-12-19T14:39:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 -- 172.21.15.50:0/3375613971 learned_addr learned my addr 172.21.15.50:0/3375613971 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 -- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff78406db40 msgr2=0x7ff7840c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff78406db40 0x7ff7840c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 -- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406df10 msgr2=0x7ff7841447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406df10 0x7ff7841447f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.276+0000 7ff783fff700 1 -- 172.21.15.50:0/3375613971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff774009420 con 0x7ff7840c63f0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff783fff700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff77c001460 tx=0x7ff77c001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c00a8b0 con 0x7ff7840c63f0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c00aa10 con 0x7ff7840c63f0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c00ad90 con 0x7ff7840c63f0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7840af210 con 0x7ff7840c63f0 2023-12-19T14:39:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7840af7d0 con 0x7ff7840c63f0 2023-12-19T14:39:47.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff784071120 con 0x7ff7840c63f0 2023-12-19T14:39:47.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.280+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff77c007300 con 0x7ff7840c63f0 2023-12-19T14:39:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.284+0000 7ff77b7fe700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 0x7ff764062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.284+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff77c07cea0 con 0x7ff7840c63f0 2023-12-19T14:39:47.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.284+0000 7ff7837fe700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 0x7ff764062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.284+0000 7ff7837fe700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 0x7ff764062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff7740091e0 tx=0x7ff774017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.292+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff77c04d180 con 0x7ff7840c63f0 2023-12-19T14:39:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.464+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7840c61d0 con 0x7ff7840c63f0 2023-12-19T14:39:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.464+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3375613971 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff77c0507a0 con 0x7ff7840c63f0 2023-12-19T14:39:47.469 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 msgr2=0x7ff764062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 0x7ff764062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff7740091e0 tx=0x7ff774017000 comp rx=0 tx=0).stop 2023-12-19T14:39:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 msgr2=0x7ff7840b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff77c001460 tx=0x7ff77c001a80 comp rx=0 tx=0).stop 2023-12-19T14:39:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 shutdown_connections 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff78406df10 0x7ff7841447f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff764060550 0x7ff764062a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7840c63f0 0x7ff7840b02b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 --2- 172.21.15.50:0/3375613971 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff78406db40 0x7ff7840c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 >> 172.21.15.50:0/3375613971 conn(0x7ff784067bb0 msgr2=0x7ff784157f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 shutdown_connections 2023-12-19T14:39:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.468+0000 7ff789c25700 1 -- 172.21.15.50:0/3375613971 wait complete. 2023-12-19T14:39:47.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.660+0000 7f5631d85700 1 Processor -- start 2023-12-19T14:39:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.660+0000 7f5631d85700 1 -- start start 2023-12-19T14:39:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f5631d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f5631d85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f562c0ca2c0 0x7f562c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f5631d85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c0cbdf0 con 0x7f562c0cb970 2023-12-19T14:39:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f5631d85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c0cc0d0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f5631d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0cc3b0 con 0x7f562c158e20 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f562c0ca2c0 0x7f562c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f562c0ca2c0 0x7f562c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57786/0 (socket says 172.21.15.50:57786) 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 -- 172.21.15.50:0/3713633859 learned_addr learned my addr 172.21.15.50:0/3713633859 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562affd700 1 --1- 172.21.15.50:0/3713633859 >> v1:172.21.15.50:6789/0 conn(0x7f562c0cb970 0x7f562c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35034/0 (socket says 172.21.15.50:35034) 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562b7fe700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 -- 172.21.15.50:0/3713633859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 msgr2=0x7f562c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 -- 172.21.15.50:0/3713633859 >> v1:172.21.15.50:6789/0 conn(0x7f562c0cb970 legacy=0x7f562c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 -- 172.21.15.50:0/3713633859 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0cc690 con 0x7f562c0ca2c0 2023-12-19T14:39:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562bfff700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c0ceb00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5624009f20 tx=0x7f5624009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d84fe1836786ced server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562a7fc700 1 -- 172.21.15.50:0/3713633859 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562400e070 con 0x7f562c0ca2c0 2023-12-19T14:39:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562a7fc700 1 -- 172.21.15.50:0/3713633859 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562400ba80 con 0x7f562c0ca2c0 2023-12-19T14:39:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.664+0000 7f562a7fc700 1 -- 172.21.15.50:0/3713633859 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240092e0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 -- 172.21.15.50:0/3713633859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 msgr2=0x7f562c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c0ceb00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5624009f20 tx=0x7f5624009840 comp rx=0 tx=0).stop 2023-12-19T14:39:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 -- 172.21.15.50:0/3713633859 shutdown_connections 2023-12-19T14:39:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c0ceb00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 --2- 172.21.15.50:0/3713633859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 -- 172.21.15.50:0/3713633859 >> 172.21.15.50:0/3713633859 conn(0x7f562c067bb0 msgr2=0x7f562c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 -- 172.21.15.50:0/3713633859 shutdown_connections 2023-12-19T14:39:47.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.668+0000 7f5631d85700 1 -- 172.21.15.50:0/3713633859 wait complete. 2023-12-19T14:39:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 Processor -- start 2023-12-19T14:39:47.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 -- start start 2023-12-19T14:39:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c1453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f562c145c20 0x7f562c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c16cad0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0c98c0 con 0x7f562c145c20 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f5631d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0c9300 con 0x7f562c158e20 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57800/0 (socket says 172.21.15.50:57800) 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 -- 172.21.15.50:0/3123288285 learned_addr learned my addr 172.21.15.50:0/3123288285 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:47.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562bfff700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f562c145c20 0x7f562c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 -- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 msgr2=0x7f562c1453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562affd700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c1453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c1453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 -- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f562c145c20 msgr2=0x7f562c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f562c145c20 0x7f562c0cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562b7fe700 1 -- 172.21.15.50:0/3123288285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624009ae0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.672+0000 7f562affd700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c1453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:47.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f562b7fe700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f561c00d860 tx=0x7f561c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c0125f0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c012750 con 0x7f562c0ca2c0 2023-12-19T14:39:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c0c95e0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c012b10 con 0x7f562c0ca2c0 2023-12-19T14:39:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c16d4a0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c071120 con 0x7f562c0ca2c0 2023-12-19T14:39:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.676+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f561c01e460 con 0x7f562c0ca2c0 2023-12-19T14:39:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.680+0000 7f56137fe700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 0x7f5618062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.680+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f561c07cfc0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.680+0000 7f562affd700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 0x7f5618062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:47.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.680+0000 7f562affd700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 0x7f5618062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f562c14cc70 tx=0x7f5620009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:47.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.688+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f561c0acbe0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.868+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f562c06a7a0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.868+0000 7f56137fe700 1 -- 172.21.15.50:0/3123288285 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f561c04d2c0 con 0x7f562c0ca2c0 2023-12-19T14:39:47.871 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:47.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 msgr2=0x7f5618062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 0x7f5618062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f562c14cc70 tx=0x7f5620009040 comp rx=0 tx=0).stop 2023-12-19T14:39:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 msgr2=0x7f562c16cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f561c00d860 tx=0x7f561c008040 comp rx=0 tx=0).stop 2023-12-19T14:39:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 shutdown_connections 2023-12-19T14:39:47.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f562c145c20 0x7f562c0cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5618060550 0x7f5618062a00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f562c158e20 0x7f562c1453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 --2- 172.21.15.50:0/3123288285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f562c0ca2c0 0x7f562c16cf60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 >> 172.21.15.50:0/3123288285 conn(0x7f562c067bb0 msgr2=0x7f562c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 shutdown_connections 2023-12-19T14:39:47.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:47.872+0000 7f5631d85700 1 -- 172.21.15.50:0/3123288285 wait complete. 2023-12-19T14:39:47.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:48.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.076+0000 7fe40f04c700 1 Processor -- start 2023-12-19T14:39:48.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.076+0000 7fe40f04c700 1 -- start start 2023-12-19T14:39:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe408158c90 0x7fe4080caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 0x7fe408159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40806c850 con 0x7fe4081595a0 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40806b470 con 0x7fe408158c90 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40806b750 con 0x7fe40806aeb0 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe408158c90 0x7fe4080caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe408158c90 0x7fe4080caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57808/0 (socket says 172.21.15.50:57808) 2023-12-19T14:39:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 -- 172.21.15.50:0/3100294310 learned_addr learned my addr 172.21.15.50:0/3100294310 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 -- 172.21.15.50:0/3100294310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 msgr2=0x7fe408159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40d849700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 0x7fe408159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 0x7fe408159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 -- 172.21.15.50:0/3100294310 >> v1:172.21.15.125:6789/0 conn(0x7fe4081595a0 legacy=0x7fe40815c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 -- 172.21.15.50:0/3100294310 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40806ba30 con 0x7fe408158c90 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40d849700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 0x7fe408159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40e04a700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe408158c90 0x7fe4080caf40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4040099b0 tx=0x7fe404009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d767f58af6b6889f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40d048700 1 -- 172.21.15.50:0/3100294310 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400b820 con 0x7fe408158c90 2023-12-19T14:39:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40d048700 1 -- 172.21.15.50:0/3100294310 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400b980 con 0x7fe408158c90 2023-12-19T14:39:48.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40d048700 1 -- 172.21.15.50:0/3100294310 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400bd20 con 0x7fe408158c90 2023-12-19T14:39:48.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- 172.21.15.50:0/3100294310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe408158c90 msgr2=0x7fe4080caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe408158c90 0x7fe4080caf40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe4040099b0 tx=0x7fe404009250 comp rx=0 tx=0).stop 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- 172.21.15.50:0/3100294310 shutdown_connections 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40806aeb0 0x7fe408159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3100294310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe408158c90 0x7fe4080caf40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.080+0000 7fe40f04c700 1 -- 172.21.15.50:0/3100294310 >> 172.21.15.50:0/3100294310 conn(0x7fe408067bb0 msgr2=0x7fe4080cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- 172.21.15.50:0/3100294310 shutdown_connections 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- 172.21.15.50:0/3100294310 wait complete. 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 Processor -- start 2023-12-19T14:39:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- start start 2023-12-19T14:39:48.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe408158c90 0x7fe408153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4081595a0 0x7fe4080c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c69a0 con 0x7fe40806aeb0 2023-12-19T14:39:48.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c5530 con 0x7fe4081595a0 2023-12-19T14:39:48.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.084+0000 7fe40f04c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c5810 con 0x7fe408158c90 2023-12-19T14:39:48.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57818/0 (socket says 172.21.15.50:57818) 2023-12-19T14:39:48.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 -- 172.21.15.50:0/3235750913 learned_addr learned my addr 172.21.15.50:0/3235750913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e84b700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4081595a0 0x7fe4080c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40d849700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe408158c90 0x7fe408153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 -- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe408158c90 msgr2=0x7fe408153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe408158c90 0x7fe408153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 -- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4081595a0 msgr2=0x7fe4080c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4081595a0 0x7fe4080c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 -- 172.21.15.50:0/3235750913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040094f0 con 0x7fe40806aeb0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40e04a700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe404009290 tx=0x7fe404016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404011c10 con 0x7fe40806aeb0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404020070 con 0x7fe40806aeb0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4080c5af0 con 0x7fe40806aeb0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40401b450 con 0x7fe40806aeb0 2023-12-19T14:39:48.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.088+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4080c5220 con 0x7fe40806aeb0 2023-12-19T14:39:48.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.092+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4080c67d0 con 0x7fe40806aeb0 2023-12-19T14:39:48.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.092+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe404022030 con 0x7fe40806aeb0 2023-12-19T14:39:48.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.092+0000 7fe3fe7fc700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 0x7fe3ec062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.092+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe40407e970 con 0x7fe40806aeb0 2023-12-19T14:39:48.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.096+0000 7fe40d849700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 0x7fe3ec062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.096+0000 7fe40d849700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 0x7fe3ec062ad0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4080c62b0 tx=0x7fe3f8006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.100+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe40404ec50 con 0x7fe40806aeb0 2023-12-19T14:39:48.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.272+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4080c67d0 con 0x7fe40806aeb0 2023-12-19T14:39:48.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.272+0000 7fe3fe7fc700 1 -- 172.21.15.50:0/3235750913 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe404052270 con 0x7fe40806aeb0 2023-12-19T14:39:48.277 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 msgr2=0x7fe3ec062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 0x7fe3ec062ad0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4080c62b0 tx=0x7fe3f8006f90 comp rx=0 tx=0).stop 2023-12-19T14:39:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 msgr2=0x7fe408156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe404009290 tx=0x7fe404016000 comp rx=0 tx=0).stop 2023-12-19T14:39:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 shutdown_connections 2023-12-19T14:39:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4081595a0 0x7fe4080c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3ec060620 0x7fe3ec062ad0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe408158c90 0x7fe408153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 --2- 172.21.15.50:0/3235750913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe40806aeb0 0x7fe408156650 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 >> 172.21.15.50:0/3235750913 conn(0x7fe408067bb0 msgr2=0x7fe4080b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:48.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 shutdown_connections 2023-12-19T14:39:48.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.276+0000 7fe40f04c700 1 -- 172.21.15.50:0/3235750913 wait complete. 2023-12-19T14:39:48.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.468+0000 7f2cb49c5700 1 Processor -- start 2023-12-19T14:39:48.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.468+0000 7f2cb49c5700 1 -- start start 2023-12-19T14:39:48.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cb0158e20 0x7f2cb00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cb00cb970 0x7f2cb00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb00ca2c0 0x7f2cb006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00cbdf0 con 0x7f2cb00cb970 2023-12-19T14:39:48.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00cc0d0 con 0x7f2cb0158e20 2023-12-19T14:39:48.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cb49c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00cc3b0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cb00cb970 0x7f2cb00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2cb00cb970 0x7f2cb00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57826/0 (socket says 172.21.15.50:57826) 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 -- 172.21.15.50:0/208508219 learned_addr learned my addr 172.21.15.50:0/208508219 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cae59c700 1 --2- 172.21.15.50:0/208508219 >> v2:172.21.15.125:3300/0 conn(0x7f2cb0158e20 0x7f2cb00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2caed9d700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb00ca2c0 0x7f2cb006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 -- 172.21.15.50:0/208508219 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb00ca2c0 msgr2=0x7f2cb006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb00ca2c0 0x7f2cb006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 -- 172.21.15.50:0/208508219 >> v2:172.21.15.125:3300/0 conn(0x7f2cb0158e20 msgr2=0x7f2cb00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/208508219 >> v2:172.21.15.125:3300/0 conn(0x7f2cb0158e20 0x7f2cb00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 -- 172.21.15.50:0/208508219 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00cc690 con 0x7f2cb00cb970 2023-12-19T14:39:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00cb970 0x7f2cb00ceb00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40099b0 tx=0x7f2ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3629b2cafec5bbd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cad59a700 1 -- 172.21.15.50:0/208508219 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4009ea0 con 0x7f2cb00cb970 2023-12-19T14:39:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cad59a700 1 -- 172.21.15.50:0/208508219 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca400b820 con 0x7f2cb00cb970 2023-12-19T14:39:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.472+0000 7f2cad59a700 1 -- 172.21.15.50:0/208508219 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400bbc0 con 0x7f2cb00cb970 2023-12-19T14:39:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- 172.21.15.50:0/208508219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00cb970 msgr2=0x7f2cb00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00cb970 0x7f2cb00ceb00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40099b0 tx=0x7f2ca4009250 comp rx=0 tx=0).stop 2023-12-19T14:39:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- 172.21.15.50:0/208508219 shutdown_connections 2023-12-19T14:39:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00cb970 0x7f2cb00ceb00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/208508219 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb00ca2c0 0x7f2cb006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/208508219 >> v2:172.21.15.125:3300/0 conn(0x7f2cb0158e20 0x7f2cb00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- 172.21.15.50:0/208508219 >> 172.21.15.50:0/208508219 conn(0x7f2cb0067bb0 msgr2=0x7f2cb00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- 172.21.15.50:0/208508219 shutdown_connections 2023-12-19T14:39:48.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- 172.21.15.50:0/208508219 wait complete. 2023-12-19T14:39:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 Processor -- start 2023-12-19T14:39:48.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.476+0000 7f2cb49c5700 1 -- start start 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 0x7f2cb00b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb014c5f0 0x7f2cb015b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b5ad0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b4a30 con 0x7f2cb0158e20 2023-12-19T14:39:48.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b4d10 con 0x7f2cb014c5f0 2023-12-19T14:39:48.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57830/0 (socket says 172.21.15.50:57830) 2023-12-19T14:39:48.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 -- 172.21.15.50:0/2911172323 learned_addr learned my addr 172.21.15.50:0/2911172323 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 0x7f2cb00b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2caed9d700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb014c5f0 0x7f2cb015b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 -- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb014c5f0 msgr2=0x7f2cb015b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb014c5f0 0x7f2cb015b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 -- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 msgr2=0x7f2cb00b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 0x7f2cb00b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 -- 172.21.15.50:0/2911172323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40094f0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 0x7f2cb00b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cae59c700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000e930 tx=0x7f2ca00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000eed0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca000c850 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb00b4ff0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000cbf0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.480+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb00b55b0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.484+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb0071120 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.484+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2ca0013470 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.484+0000 7f2c9e7fc700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 0x7f2c90062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.484+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2ca007cd20 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.488+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 0x7f2c90062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.488+0000 7f2cadd9b700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 0x7f2c90062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40099b0 tx=0x7f2ca4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.492+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ca004cf80 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.672+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2cb006a7a0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2c9e7fc700 1 -- 172.21.15.50:0/2911172323 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2ca00505a0 con 0x7f2cb00ca2c0 2023-12-19T14:39:48.679 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 msgr2=0x7f2c90062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 0x7f2c90062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40099b0 tx=0x7f2ca4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 msgr2=0x7f2cb014bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000e930 tx=0x7f2ca00091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 shutdown_connections 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cb0158e20 0x7f2cb00b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2c90060550 0x7f2c90062a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2cb014c5f0 0x7f2cb015b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 --2- 172.21.15.50:0/2911172323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2cb00ca2c0 0x7f2cb014bfd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 >> 172.21.15.50:0/2911172323 conn(0x7f2cb0067bb0 msgr2=0x7f2cb00ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 shutdown_connections 2023-12-19T14:39:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.676+0000 7f2cb49c5700 1 -- 172.21.15.50:0/2911172323 wait complete. 2023-12-19T14:39:48.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.864+0000 7f46d4c16700 1 Processor -- start 2023-12-19T14:39:48.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.864+0000 7f46d4c16700 1 -- start start 2023-12-19T14:39:48.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46d4c16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46d4c16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46d006a540 0x7f46d006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46d4c16700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d00ca5a0 con 0x7f46d00cc470 2023-12-19T14:39:48.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46d4c16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d006dac0 con 0x7f46d006a540 2023-12-19T14:39:48.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46d4c16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b0260 con 0x7f46d0158e70 2023-12-19T14:39:48.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46cf7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ceffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46d006a540 0x7f46d006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46cf7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:42224/0 (socket says 172.21.15.50:42224) 2023-12-19T14:39:48.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46cffff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f46d00cc470 0x7f46d00cc8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60872/0 (socket says 172.21.15.50:60872) 2023-12-19T14:39:48.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46cf7fe700 1 -- 172.21.15.50:0/4239661801 learned_addr learned my addr 172.21.15.50:0/4239661801 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ceffd700 1 -- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 msgr2=0x7f46d00afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ceffd700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ceffd700 1 -- 172.21.15.50:0/4239661801 >> v1:172.21.15.125:6789/0 conn(0x7f46d00cc470 legacy=0x7f46d00cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ceffd700 1 -- 172.21.15.50:0/4239661801 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d006d220 con 0x7f46d006a540 2023-12-19T14:39:48.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46cf7fe700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.868+0000 7f46ce7fc700 1 -- 172.21.15.50:0/4239661801 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321565063 0 0) 0x7f46d00ca5a0 con 0x7f46d00cc470 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46ceffd700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d006c9d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46c40098e0 tx=0x7f46c40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c826475ef1999c41 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46ce7fc700 1 -- 172.21.15.50:0/4239661801 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400b640 con 0x7f46d006a540 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46ce7fc700 1 -- 172.21.15.50:0/4239661801 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400b7a0 con 0x7f46d006a540 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46ce7fc700 1 -- 172.21.15.50:0/4239661801 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400bb40 con 0x7f46d006a540 2023-12-19T14:39:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 -- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 msgr2=0x7f46d006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d006c9d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46c40098e0 tx=0x7f46c40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 -- 172.21.15.50:0/4239661801 shutdown_connections 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d006c9d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 --2- 172.21.15.50:0/4239661801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d0158e70 0x7f46d00afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 -- 172.21.15.50:0/4239661801 >> 172.21.15.50:0/4239661801 conn(0x7f46d00670e0 msgr2=0x7f46d00674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 -- 172.21.15.50:0/4239661801 shutdown_connections 2023-12-19T14:39:48.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.872+0000 7f46d4c16700 1 -- 172.21.15.50:0/4239661801 wait complete. 2023-12-19T14:39:48.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 Processor -- start 2023-12-19T14:39:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 -- start start 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d00cc470 0x7f46d00b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b4b10 con 0x7f46d0158e70 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0159500 con 0x7f46d006a540 2023-12-19T14:39:48.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.876+0000 7f46d4c16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b3f00 con 0x7f46d00cc470 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57840/0 (socket says 172.21.15.50:57840) 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cffff700 1 -- 172.21.15.50:0/2309223483 learned_addr learned my addr 172.21.15.50:0/2309223483 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46ceffd700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d00cc470 0x7f46d00b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 -- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d00cc470 msgr2=0x7f46d00b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d00cc470 0x7f46d00b4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 -- 172.21.15.50:0/2309223483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 msgr2=0x7f46d00b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 -- 172.21.15.50:0/2309223483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00094f0 con 0x7f46d006a540 2023-12-19T14:39:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cffff700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46cf7fe700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d00c4760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000c6a0 con 0x7f46d006a540 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c000c800 con 0x7f46d006a540 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c4009420 con 0x7f46d006a540 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000cb80 con 0x7f46d006a540 2023-12-19T14:39:48.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.880+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d00b44c0 con 0x7f46d006a540 2023-12-19T14:39:48.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.884+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d004be40 con 0x7f46d006a540 2023-12-19T14:39:48.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.884+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f46c0013470 con 0x7f46d006a540 2023-12-19T14:39:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.884+0000 7f46b77fe700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 0x7f46bc062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:48.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.884+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f46c007cc60 con 0x7f46d006a540 2023-12-19T14:39:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.888+0000 7f46ceffd700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 0x7f46bc062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:48.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.888+0000 7f46ceffd700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 0x7f46bc062a80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46c40093f0 tx=0x7f46c4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:48.896+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f46c004cdf0 con 0x7f46d006a540 2023-12-19T14:39:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.072+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d00af100 con 0x7f46d006a540 2023-12-19T14:39:49.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.076+0000 7f46b77fe700 1 -- 172.21.15.50:0/2309223483 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17519 (secure 0 0 0) 0x7f46c0020090 con 0x7f46d006a540 2023-12-19T14:39:49.079 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:49.079 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:48.536721+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 msgr2=0x7f46bc062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 0x7f46bc062a80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46c40093f0 tx=0x7f46c4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 msgr2=0x7f46d00c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d00c4760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0009250 comp rx=0 tx=0).stop 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 shutdown_connections 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46bc0605d0 0x7f46bc062a80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46d0158e70 0x7f46d00b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46d00cc470 0x7f46d00b4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 --2- 172.21.15.50:0/2309223483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d006a540 0x7f46d00c4760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 >> 172.21.15.50:0/2309223483 conn(0x7f46d00670e0 msgr2=0x7f46d0157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 shutdown_connections 2023-12-19T14:39:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.080+0000 7f46d4c16700 1 -- 172.21.15.50:0/2309223483 wait complete. 2023-12-19T14:39:49.106 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:49.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.288+0000 7f393a2ae700 1 Processor -- start 2023-12-19T14:39:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.288+0000 7f393a2ae700 1 -- start start 2023-12-19T14:39:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393a2ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3934159790 0x7f39340684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393a2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f39340b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393a2ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934067790 con 0x7f39340b1d00 2023-12-19T14:39:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393a2ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39340b2180 con 0x7f3934159790 2023-12-19T14:39:49.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393a2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3934068030 con 0x7f39340b0cb0 2023-12-19T14:39:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393884b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f39340b1d00 0x7f3934159b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35040/0 (socket says 172.21.15.50:35040) 2023-12-19T14:39:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f393884b700 1 -- 172.21.15.50:0/1679533053 learned_addr learned my addr 172.21.15.50:0/1679533053 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f3933fff700 1 --2- 172.21.15.50:0/1679533053 >> v2:172.21.15.125:3300/0 conn(0x7f3934159790 0x7f39340684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f392b7fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f39340b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145388348 0 0) 0x7f3934067790 con 0x7f39340b1d00 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3914003570 con 0x7f39340b1d00 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101203186 0 0) 0x7f3914003570 con 0x7f39340b1d00 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 msgr2=0x7f39340b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f39340b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 >> v2:172.21.15.125:3300/0 conn(0x7f3934159790 msgr2=0x7f39340684c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 --2- 172.21.15.50:0/1679533053 >> v2:172.21.15.125:3300/0 conn(0x7f3934159790 0x7f39340684c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340b1200 con 0x7f39340b1d00 2023-12-19T14:39:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f392c0017f0 con 0x7f39340b1d00 2023-12-19T14:39:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 >> v1:172.21.15.50:6789/0 conn(0x7f39340b1d00 legacy=0x7f3934159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3914003a60 0x7f3914003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f3914004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914004d50 0x7f39140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340b2180 con 0x7f3914003a60 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3934068030 con 0x7f3914004d50 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.292+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391400ba60 con 0x7f39340b0cb0 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3914003a60 0x7f3914003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f392b7fe700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f3914004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393884b700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914004d50 0x7f39140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 -- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 msgr2=0x7f3914004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39340b0cb0 0x7f3914004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 -- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914004d50 msgr2=0x7f39140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914004d50 0x7f39140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 -- 172.21.15.50:0/1679533053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914003570 con 0x7f3914003a60 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f3933fff700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3914003a60 0x7f3914003e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3924009c00 tx=0x7f39240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee776db20bc785e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392400e6b0 con 0x7f3914003a60 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392400e810 con 0x7f3914003a60 2023-12-19T14:39:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f39337fe700 1 -- 172.21.15.50:0/1679533053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392400ebd0 con 0x7f3914003a60 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 -- 172.21.15.50:0/1679533053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3914003a60 msgr2=0x7f3914003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 --2- 172.21.15.50:0/1679533053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3914003a60 0x7f3914003e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3924009c00 tx=0x7f39240091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393884b700 1 -- 172.21.15.50:0/1679533053 reap_dead start 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 -- 172.21.15.50:0/1679533053 shutdown_connections 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 -- 172.21.15.50:0/1679533053 >> 172.21.15.50:0/1679533053 conn(0x7f39340cdef0 msgr2=0x7f39340ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 -- 172.21.15.50:0/1679533053 shutdown_connections 2023-12-19T14:39:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.296+0000 7f393a2ae700 1 -- 172.21.15.50:0/1679533053 wait complete. 2023-12-19T14:39:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 Processor -- start 2023-12-19T14:39:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 -- start start 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 0x7f39341428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f393415b110 0x7f393415b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39340c9e30 con 0x7f39340c9480 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393415c290 con 0x7f3914003a60 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.300+0000 7f393a2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393415ce10 con 0x7f393415b110 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57862/0 (socket says 172.21.15.50:57862) 2023-12-19T14:39:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 -- 172.21.15.50:0/2920654144 learned_addr learned my addr 172.21.15.50:0/2920654144 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f3933fff700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 0x7f39341428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f393884b700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f393415b110 0x7f393415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 -- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f393415b110 msgr2=0x7f393415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f393415b110 0x7f393415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 -- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 msgr2=0x7f39341428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 0x7f39341428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 -- 172.21.15.50:0/2920654144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393415c4e0 con 0x7f39340c9480 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f3933fff700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 0x7f39341428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f392b7fe700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f391c011d70 tx=0x7f391c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c009e20 con 0x7f39340c9480 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c00fcd0 con 0x7f39340c9480 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c014650 con 0x7f39340c9480 2023-12-19T14:39:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393415c7c0 con 0x7f39340c9480 2023-12-19T14:39:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934142df0 con 0x7f39340c9480 2023-12-19T14:39:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.304+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393415cc40 con 0x7f39340c9480 2023-12-19T14:39:49.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.308+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f391c014a80 con 0x7f39340c9480 2023-12-19T14:39:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.308+0000 7f3930ff9700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 0x7f3918062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.308+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f391c018030 con 0x7f39340c9480 2023-12-19T14:39:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.308+0000 7f3933fff700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 0x7f3918062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.308+0000 7f3933fff700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 0x7f3918062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f39240091e0 tx=0x7f392401a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.316+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f391c04d970 con 0x7f39340c9480 2023-12-19T14:39:49.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.488+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3934071120 con 0x7f39340c9480 2023-12-19T14:39:49.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.488+0000 7f3930ff9700 1 -- 172.21.15.50:0/2920654144 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f391c050f90 con 0x7f39340c9480 2023-12-19T14:39:49.494 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:49.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 msgr2=0x7f3918062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 0x7f3918062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f39240091e0 tx=0x7f392401a000 comp rx=0 tx=0).stop 2023-12-19T14:39:49.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 msgr2=0x7f39340c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f391c011d70 tx=0x7f391c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 shutdown_connections 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3918060550 0x7f3918062a00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f393415b110 0x7f393415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39340c9480 0x7f39340c98f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 --2- 172.21.15.50:0/2920654144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3914003a60 0x7f39341428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 >> 172.21.15.50:0/2920654144 conn(0x7f39340cdef0 msgr2=0x7f3934153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:49.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 shutdown_connections 2023-12-19T14:39:49.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.492+0000 7f393a2ae700 1 -- 172.21.15.50:0/2920654144 wait complete. 2023-12-19T14:39:49.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:49.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.684+0000 7f4903ef0700 1 Processor -- start 2023-12-19T14:39:49.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.684+0000 7f4903ef0700 1 -- start start 2023-12-19T14:39:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4903ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48fc06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4903ef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0cbdf0 con 0x7f48fc158e20 2023-12-19T14:39:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4903ef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0cc0d0 con 0x7f48fc0cb970 2023-12-19T14:39:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4903ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0cc3b0 con 0x7f48fc0cecd0 2023-12-19T14:39:49.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4901c8c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f48fc158e20 0x7f48fc0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35054/0 (socket says 172.21.15.50:35054) 2023-12-19T14:39:49.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4901c8c700 1 -- 172.21.15.50:0/1262260226 learned_addr learned my addr 172.21.15.50:0/1262260226 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f490248d700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48fc06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212038269 0 0) 0x7f48fc0cbdf0 con 0x7f48fc158e20 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e8003570 con 0x7f48fc158e20 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458332551 0 0) 0x7f48fc0cc0d0 con 0x7f48fc0cb970 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc0cbdf0 con 0x7f48fc0cb970 2023-12-19T14:39:49.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8922624 0 0) 0x7f48e8003570 con 0x7f48fc158e20 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 msgr2=0x7f48fc06b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48fc06b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 >> v1:172.21.15.125:6789/0 conn(0x7f48fc0cb970 legacy=0x7f48fc06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0cc690 con 0x7f48fc158e20 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f48f8001470 con 0x7f48fc158e20 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 >> v1:172.21.15.50:6789/0 conn(0x7f48fc158e20 legacy=0x7f48fc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48e8003c20 0x7f48e8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e8004d50 0x7f48e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.688+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0cc3b0 con 0x7f48e8004d50 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e800bbf0 con 0x7f48e8003c20 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e800bd60 con 0x7f48fc0cecd0 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e8004d50 0x7f48e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4901c8c700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48e8003c20 0x7f48e8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490148b700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 -- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 msgr2=0x7f48e8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48fc0cecd0 0x7f48e8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 -- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48e8003c20 msgr2=0x7f48e8004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:49.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48e8003c20 0x7f48e8004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 -- 172.21.15.50:0/1262260226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8003570 con 0x7f48e8004d50 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e8004d50 0x7f48e80051c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f48f40094d0 tx=0x7f48f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec5a1f9e7061e42 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f400ea60 con 0x7f48e8004d50 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f400f040 con 0x7f48e8004d50 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4900c8a700 1 -- 172.21.15.50:0/1262260226 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f400edf0 con 0x7f48e8004d50 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 -- 172.21.15.50:0/1262260226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e8004d50 msgr2=0x7f48e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 --2- 172.21.15.50:0/1262260226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e8004d50 0x7f48e80051c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f48f40094d0 tx=0x7f48f400b040 comp rx=0 tx=0).stop 2023-12-19T14:39:49.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f490248d700 1 -- 172.21.15.50:0/1262260226 reap_dead start 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 -- 172.21.15.50:0/1262260226 shutdown_connections 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 -- 172.21.15.50:0/1262260226 >> 172.21.15.50:0/1262260226 conn(0x7f48fc067bb0 msgr2=0x7f48fc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 -- 172.21.15.50:0/1262260226 shutdown_connections 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.692+0000 7f4903ef0700 1 -- 172.21.15.50:0/1262260226 wait complete. 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 Processor -- start 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 -- start start 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48e8004d50 0x7f48fc16d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48fc16e510 0x7f48fc170000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0c8540 con 0x7f48fc086f70 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc16d880 con 0x7f48fc16e510 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4903ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc16db20 con 0x7f48e8004d50 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57888/0 (socket says 172.21.15.50:57888) 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 -- 172.21.15.50:0/2110286764 learned_addr learned my addr 172.21.15.50:0/2110286764 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f4901c8c700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48e8004d50 0x7f48fc16d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490148b700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48fc16e510 0x7f48fc170000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 -- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48e8004d50 msgr2=0x7f48fc16d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48e8004d50 0x7f48fc16d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 -- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48fc16e510 msgr2=0x7f48fc170000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.696+0000 7f490248d700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48fc16e510 0x7f48fc170000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f490248d700 1 -- 172.21.15.50:0/2110286764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc16de00 con 0x7f48fc086f70 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f490248d700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f48f4016000 tx=0x7f48f401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f401d630 con 0x7f48fc086f70 2023-12-19T14:39:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f401d790 con 0x7f48fc086f70 2023-12-19T14:39:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f401db10 con 0x7f48fc086f70 2023-12-19T14:39:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc16e0e0 con 0x7f48fc086f70 2023-12-19T14:39:49.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc1706e0 con 0x7f48fc086f70 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.700+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc0c8310 con 0x7f48fc086f70 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.704+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f48f4003040 con 0x7f48fc086f70 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.704+0000 7f48f1ffb700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 0x7f48e0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.704+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f48f407e390 con 0x7f48fc086f70 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.704+0000 7f4901c8c700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 0x7f48e0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:49.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.704+0000 7f4901c8c700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 0x7f48e0062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48f8001120 tx=0x7f48f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:49.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.712+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f48fc0c8310 con 0x7f48fc086f70 2023-12-19T14:39:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.884+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f48fc0c8310 con 0x7f48fc086f70 2023-12-19T14:39:49.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.888+0000 7f48f1ffb700 1 -- 172.21.15.50:0/2110286764 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f48fc0c8310 con 0x7f48fc086f70 2023-12-19T14:39:49.890 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 msgr2=0x7f48e0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 0x7f48e0062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48f8001120 tx=0x7f48f8009040 comp rx=0 tx=0).stop 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 msgr2=0x7f48fc0cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f48f4016000 tx=0x7f48f401a000 comp rx=0 tx=0).stop 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 shutdown_connections 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48fc16e510 0x7f48fc170000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48e0060550 0x7f48e0062a00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48fc086f70 0x7f48fc0cfff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 --2- 172.21.15.50:0/2110286764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48e8004d50 0x7f48fc16d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:49.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 >> 172.21.15.50:0/2110286764 conn(0x7f48fc067bb0 msgr2=0x7f48fc0cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:49.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 shutdown_connections 2023-12-19T14:39:49.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:49.892+0000 7f4903ef0700 1 -- 172.21.15.50:0/2110286764 wait complete. 2023-12-19T14:39:49.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:50.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.104+0000 7f423da4c700 1 Processor -- start 2023-12-19T14:39:50.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423da4c700 1 -- start start 2023-12-19T14:39:50.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423da4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 0x7f42380c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423da4c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423806d970 con 0x7f423806c7c0 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423da4c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423806cd60 con 0x7f42380c69a0 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423da4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423806d040 con 0x7f423806de00 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423d24b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f423806c7c0 0x7f423815cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35056/0 (socket says 172.21.15.50:35056) 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f423d24b700 1 -- 172.21.15.50:0/2553926709 learned_addr learned my addr 172.21.15.50:0/2553926709 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:50.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.108+0000 7f422ffff700 1 --2- 172.21.15.50:0/2553926709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 0x7f42380c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722225157 0 0) 0x7f423806d970 con 0x7f423806c7c0 2023-12-19T14:39:50.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003570 con 0x7f423806c7c0 2023-12-19T14:39:50.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523980982 0 0) 0x7f423806cd60 con 0x7f42380c69a0 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423806d970 con 0x7f42380c69a0 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422ffff700 1 -- 172.21.15.50:0/2553926709 >> v1:172.21.15.125:6789/0 conn(0x7f42380c69a0 legacy=0x7f423806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422ffff700 1 -- 172.21.15.50:0/2553926709 >> v1:172.21.15.50:6789/0 conn(0x7f423806c7c0 legacy=0x7f423815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422ffff700 1 -- 172.21.15.50:0/2553926709 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423806d320 con 0x7f423806de00 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422ffff700 1 --2- 172.21.15.50:0/2553926709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 0x7f42380c6d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4228009910 tx=0x7f42280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90fa2af1ec7f250 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228011630 con 0x7f423806de00 2023-12-19T14:39:50.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228011790 con 0x7f423806de00 2023-12-19T14:39:50.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f422f7fe700 1 -- 172.21.15.50:0/2553926709 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228011b30 con 0x7f423806de00 2023-12-19T14:39:50.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 -- 172.21.15.50:0/2553926709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 msgr2=0x7f42380c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 --2- 172.21.15.50:0/2553926709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 0x7f42380c6d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4228009910 tx=0x7f42280091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 -- 172.21.15.50:0/2553926709 shutdown_connections 2023-12-19T14:39:50.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 --2- 172.21.15.50:0/2553926709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806de00 0x7f42380c6d70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 -- 172.21.15.50:0/2553926709 >> 172.21.15.50:0/2553926709 conn(0x7f4238067b50 msgr2=0x7f42380cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:50.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 -- 172.21.15.50:0/2553926709 shutdown_connections 2023-12-19T14:39:50.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.112+0000 7f423da4c700 1 -- 172.21.15.50:0/2553926709 wait complete. 2023-12-19T14:39:50.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 Processor -- start 2023-12-19T14:39:50.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 -- start start 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806c7c0 0x7f423815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 0x7f42380afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380c58e0 con 0x7f423806de00 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380b18c0 con 0x7f42380c69a0 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423da4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423815c970 con 0x7f423806c7c0 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57890/0 (socket says 172.21.15.50:57890) 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 -- 172.21.15.50:0/2802971301 learned_addr learned my addr 172.21.15.50:0/2802971301 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423d24b700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 0x7f42380afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423ca4a700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806c7c0 0x7f423815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 -- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806c7c0 msgr2=0x7f423815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806c7c0 0x7f423815dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 -- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 msgr2=0x7f42380afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 0x7f42380afcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f422ffff700 1 -- 172.21.15.50:0/2802971301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009420 con 0x7f423806de00 2023-12-19T14:39:50.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.116+0000 7f423d24b700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 0x7f42380afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:50.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f422ffff700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4228009f00 tx=0x7f4228017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422801f070 con 0x7f423806de00 2023-12-19T14:39:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228011900 con 0x7f423806de00 2023-12-19T14:39:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423815d7d0 con 0x7f423806de00 2023-12-19T14:39:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228023410 con 0x7f423806de00 2023-12-19T14:39:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423815cea0 con 0x7f423806de00 2023-12-19T14:39:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423815d960 con 0x7f423806de00 2023-12-19T14:39:50.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.120+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4228023800 con 0x7f423806de00 2023-12-19T14:39:50.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.124+0000 7f422cff9700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 0x7f4224062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.124+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f422807dd50 con 0x7f423806de00 2023-12-19T14:39:50.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.124+0000 7f423ca4a700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 0x7f4224062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.124+0000 7f423ca4a700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 0x7f4224062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4234007fd0 tx=0x7f4234009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.128+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4228081030 con 0x7f423806de00 2023-12-19T14:39:50.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.312+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f423804be40 con 0x7f423806de00 2023-12-19T14:39:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.316+0000 7f422cff9700 1 -- 172.21.15.50:0/2802971301 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4228015020 con 0x7f423806de00 2023-12-19T14:39:50.319 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:50.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 msgr2=0x7f4224062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 0x7f4224062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4234007fd0 tx=0x7f4234009040 comp rx=0 tx=0).stop 2023-12-19T14:39:50.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 msgr2=0x7f42380b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4228009f00 tx=0x7f4228017000 comp rx=0 tx=0).stop 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 shutdown_connections 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42380c69a0 0x7f42380afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4224060550 0x7f4224062a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423806de00 0x7f42380b1380 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 --2- 172.21.15.50:0/2802971301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f423806c7c0 0x7f423815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 >> 172.21.15.50:0/2802971301 conn(0x7f4238067b50 msgr2=0x7f42380cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 shutdown_connections 2023-12-19T14:39:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.320+0000 7f423da4c700 1 -- 172.21.15.50:0/2802971301 wait complete. 2023-12-19T14:39:50.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.508+0000 7f323af45700 1 Processor -- start 2023-12-19T14:39:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.508+0000 7f323af45700 1 -- start start 2023-12-19T14:39:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 0x7f32340ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32340cb970 0x7f32340ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32340ca2c0 0x7f323406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340cbdf0 con 0x7f3234158e20 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340cc0d0 con 0x7f32340cb970 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f323af45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32340cc3b0 con 0x7f32340ca2c0 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.512+0000 7f3238ce1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 0x7f32340ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32340cb970 0x7f32340ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32340cb970 0x7f32340ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:54554/0 (socket says 172.21.15.50:54554) 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 -- 172.21.15.50:0/1999753271 learned_addr learned my addr 172.21.15.50:0/1999753271 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f32394e2700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32340ca2c0 0x7f323406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 -- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32340ca2c0 msgr2=0x7f323406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32340ca2c0 0x7f323406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 -- 172.21.15.50:0/1999753271 >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 msgr2=0x7f32340ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 --2- 172.21.15.50:0/1999753271 >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 0x7f32340ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 -- 172.21.15.50:0/1999753271 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32340cc690 con 0x7f32340cb970 2023-12-19T14:39:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3238ce1700 1 --2- 172.21.15.50:0/1999753271 >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 0x7f32340ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:50.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f3233fff700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340cb970 0x7f32340ceb00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f32200099b0 tx=0x7f3220009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f9eb837fc27a2ed server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f32337fe700 1 -- 172.21.15.50:0/1999753271 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322000b640 con 0x7f32340cb970 2023-12-19T14:39:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f32337fe700 1 -- 172.21.15.50:0/1999753271 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322000b7a0 con 0x7f32340cb970 2023-12-19T14:39:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f32337fe700 1 -- 172.21.15.50:0/1999753271 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322000bb40 con 0x7f32340cb970 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f323af45700 1 -- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340cb970 msgr2=0x7f32340ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.516+0000 7f323af45700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340cb970 0x7f32340ceb00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f32200099b0 tx=0x7f3220009250 comp rx=0 tx=0).stop 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- 172.21.15.50:0/1999753271 shutdown_connections 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f32340ca2c0 0x7f323406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- 172.21.15.50:0/1999753271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340cb970 0x7f32340ceb00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- 172.21.15.50:0/1999753271 >> v2:172.21.15.50:3300/0 conn(0x7f3234158e20 0x7f32340ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- 172.21.15.50:0/1999753271 >> 172.21.15.50:0/1999753271 conn(0x7f3234067bb0 msgr2=0x7f32340d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:50.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- 172.21.15.50:0/1999753271 shutdown_connections 2023-12-19T14:39:50.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- 172.21.15.50:0/1999753271 wait complete. 2023-12-19T14:39:50.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 Processor -- start 2023-12-19T14:39:50.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- start start 2023-12-19T14:39:50.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 0x7f32340c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340c9e70 0x7f323414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414d640 con 0x7f32340ca2c0 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32340c7570 con 0x7f32340c9e70 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.520+0000 7f323af45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32340c7850 con 0x7f3234158e20 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3238ce1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3238ce1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57912/0 (socket says 172.21.15.50:57912) 2023-12-19T14:39:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3238ce1700 1 -- 172.21.15.50:0/2385628130 learned_addr learned my addr 172.21.15.50:0/2385628130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 0x7f32340c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f32394e2700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340c9e70 0x7f323414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 -- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340c9e70 msgr2=0x7f323414f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340c9e70 0x7f323414f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 -- 172.21.15.50:0/2385628130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 msgr2=0x7f32340c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 -- 172.21.15.50:0/2385628130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228009420 con 0x7f3234158e20 2023-12-19T14:39:50.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3238ce1700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:50.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3233fff700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 0x7f32340c50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32340cdc10 tx=0x7f3220017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322000bb20 con 0x7f3234158e20 2023-12-19T14:39:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3220000c90 con 0x7f3234158e20 2023-12-19T14:39:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220011690 con 0x7f3234158e20 2023-12-19T14:39:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32200094f0 con 0x7f3234158e20 2023-12-19T14:39:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.524+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323414dbe0 con 0x7f3234158e20 2023-12-19T14:39:50.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.528+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32340c7a10 con 0x7f3234158e20 2023-12-19T14:39:50.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.528+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3220011a80 con 0x7f3234158e20 2023-12-19T14:39:50.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.528+0000 7f3230ff9700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 0x7f3218062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.528+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f322007e3d0 con 0x7f3234158e20 2023-12-19T14:39:50.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.532+0000 7f3238ce1700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 0x7f3218062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.532+0000 7f3238ce1700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 0x7f3218062a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3228011640 tx=0x7f32280091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.540+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f322004e560 con 0x7f3234158e20 2023-12-19T14:39:50.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.716+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f323406a7a0 con 0x7f3234158e20 2023-12-19T14:39:50.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f3230ff9700 1 -- 172.21.15.50:0/2385628130 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3220051b80 con 0x7f3234158e20 2023-12-19T14:39:50.723 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 msgr2=0x7f3218062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 0x7f3218062a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3228011640 tx=0x7f32280091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 msgr2=0x7f32340c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 0x7f32340c50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32340cdc10 tx=0x7f3220017000 comp rx=0 tx=0).stop 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 shutdown_connections 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32340c9e70 0x7f323414f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3218060580 0x7f3218062a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3234158e20 0x7f32340c50d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 --2- 172.21.15.50:0/2385628130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32340ca2c0 0x7f32340c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.720+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 >> 172.21.15.50:0/2385628130 conn(0x7f3234067bb0 msgr2=0x7f32340c6770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:50.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.724+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 shutdown_connections 2023-12-19T14:39:50.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.724+0000 7f323af45700 1 -- 172.21.15.50:0/2385628130 wait complete. 2023-12-19T14:39:50.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:50.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.912+0000 7fb6a986b700 1 Processor -- start 2023-12-19T14:39:50.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.912+0000 7fb6a986b700 1 -- start start 2023-12-19T14:39:50.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a986b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6a40c6900 0x7fb6a40c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a986b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a406da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a986b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a406b470 con 0x7fb6a415cc20 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a986b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a415d7b0 con 0x7fb6a40c6900 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a986b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4161d10 con 0x7fb6a406b900 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6a40c6900 0x7fb6a40c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6a40c6900 0x7fb6a40c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57914/0 (socket says 172.21.15.50:57914) 2023-12-19T14:39:50.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 -- 172.21.15.50:0/3318514467 learned_addr learned my addr 172.21.15.50:0/3318514467 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:50.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a3fff700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a37fe700 1 -- 172.21.15.50:0/3318514467 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034029865 0 0) 0x7fb6a406b470 con 0x7fb6a415cc20 2023-12-19T14:39:50.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a37fe700 1 -- 172.21.15.50:0/3318514467 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb694003570 con 0x7fb6a415cc20 2023-12-19T14:39:50.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 -- 172.21.15.50:0/3318514467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 msgr2=0x7fb6a406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a406da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 -- 172.21.15.50:0/3318514467 >> v1:172.21.15.125:6789/0 conn(0x7fb6a415cc20 legacy=0x7fb6a415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 -- 172.21.15.50:0/3318514467 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a406bd70 con 0x7fb6a40c6900 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a3fff700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a406da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a8869700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a40c6900 0x7fb6a40c6cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb6980099e0 tx=0x7fb698009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadbf73ef31e7841 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a37fe700 1 -- 172.21.15.50:0/3318514467 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69800b640 con 0x7fb6a40c6900 2023-12-19T14:39:50.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a37fe700 1 -- 172.21.15.50:0/3318514467 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69800b7a0 con 0x7fb6a40c6900 2023-12-19T14:39:50.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.916+0000 7fb6a37fe700 1 -- 172.21.15.50:0/3318514467 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69800bb40 con 0x7fb6a40c6900 2023-12-19T14:39:50.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- 172.21.15.50:0/3318514467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a40c6900 msgr2=0x7fb6a40c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a40c6900 0x7fb6a40c6cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb6980099e0 tx=0x7fb698009250 comp rx=0 tx=0).stop 2023-12-19T14:39:50.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- 172.21.15.50:0/3318514467 shutdown_connections 2023-12-19T14:39:50.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a406da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 --2- 172.21.15.50:0/3318514467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a40c6900 0x7fb6a40c6cd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- 172.21.15.50:0/3318514467 >> 172.21.15.50:0/3318514467 conn(0x7fb6a4067bb0 msgr2=0x7fb6a40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:50.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- 172.21.15.50:0/3318514467 shutdown_connections 2023-12-19T14:39:50.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- 172.21.15.50:0/3318514467 wait complete. 2023-12-19T14:39:50.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 Processor -- start 2023-12-19T14:39:50.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.920+0000 7fb6a986b700 1 -- start start 2023-12-19T14:39:50.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a4160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a40c6900 0x7fb6a40b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415f2c0 con 0x7fb6a415cc20 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415f5a0 con 0x7fb6a40c6900 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415f880 con 0x7fb6a406b900 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57930/0 (socket says 172.21.15.50:57930) 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 -- 172.21.15.50:0/4250192465 learned_addr learned my addr 172.21.15.50:0/4250192465 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a8869700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a4160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a3fff700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a40c6900 0x7fb6a40b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 -- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 msgr2=0x7fb6a4160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a4160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 -- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a40c6900 msgr2=0x7fb6a40b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a40c6900 0x7fb6a40b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 -- 172.21.15.50:0/4250192465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690009420 con 0x7fb6a415cc20 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a8869700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a4160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a906a700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c860 tx=0x7fb69c002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c002030 con 0x7fb6a415cc20 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c0021b0 con 0x7fb6a415cc20 2023-12-19T14:39:50.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6980094f0 con 0x7fb6a415cc20 2023-12-19T14:39:50.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c012630 con 0x7fb6a415cc20 2023-12-19T14:39:50.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.924+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a415fe40 con 0x7fb6a415cc20 2023-12-19T14:39:50.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.928+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4160060 con 0x7fb6a415cc20 2023-12-19T14:39:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.928+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb69c012a20 con 0x7fb6a415cc20 2023-12-19T14:39:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.932+0000 7fb6a0ff9700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 0x7fb688062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.932+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fb69c07cae0 con 0x7fb6a415cc20 2023-12-19T14:39:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.932+0000 7fb6a8869700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 0x7fb688062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.932+0000 7fb6a8869700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 0x7fb688062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb6980099e0 tx=0x7fb698017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:50.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:50.936+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb69c04cd10 con 0x7fb6a415cc20 2023-12-19T14:39:51.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.108+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6a404be40 con 0x7fb6a415cc20 2023-12-19T14:39:51.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a0ff9700 1 -- 172.21.15.50:0/4250192465 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7fb69c050330 con 0x7fb6a415cc20 2023-12-19T14:39:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:51.115 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:51.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 msgr2=0x7fb688062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 0x7fb688062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb6980099e0 tx=0x7fb698017000 comp rx=0 tx=0).stop 2023-12-19T14:39:51.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 msgr2=0x7fb6a414f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c860 tx=0x7fb69c002700 comp rx=0 tx=0).stop 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 shutdown_connections 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6a40c6900 0x7fb6a40b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb688060550 0x7fb688062a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6a415cc20 0x7fb6a414f900 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 --2- 172.21.15.50:0/4250192465 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6a406b900 0x7fb6a4160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.112+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 >> 172.21.15.50:0/4250192465 conn(0x7fb6a4067bb0 msgr2=0x7fb6a4159ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.116+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 shutdown_connections 2023-12-19T14:39:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.116+0000 7fb6a986b700 1 -- 172.21.15.50:0/4250192465 wait complete. 2023-12-19T14:39:51.141 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:51.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 Processor -- start 2023-12-19T14:39:51.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 -- start start 2023-12-19T14:39:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0c38158e20 0x7f0c380ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c380ca2c0 0x7f0c380ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380cbdf0 con 0x7f0c380cb970 2023-12-19T14:39:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c380cc0d0 con 0x7f0c38158e20 2023-12-19T14:39:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c3d3c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380cc3b0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c367fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0c380cb970 0x7f0c3806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35070/0 (socket says 172.21.15.50:35070) 2023-12-19T14:39:51.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c367fc700 1 -- 172.21.15.50:0/66383031 learned_addr learned my addr 172.21.15.50:0/66383031 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:51.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.328+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/66383031 >> v2:172.21.15.125:3300/0 conn(0x7f0c38158e20 0x7f0c380ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c377fe700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c380ca2c0 0x7f0c380ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c36ffd700 1 -- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c380ca2c0 msgr2=0x7f0c380ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c380ca2c0 0x7f0c380ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c36ffd700 1 -- 172.21.15.50:0/66383031 >> v1:172.21.15.50:6789/0 conn(0x7f0c380cb970 legacy=0x7f0c3806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c36ffd700 1 -- 172.21.15.50:0/66383031 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380cc690 con 0x7f0c38158e20 2023-12-19T14:39:51.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c35ffb700 1 -- 172.21.15.50:0/66383031 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388962538 0 0) 0x7f0c380cbdf0 con 0x7f0c380cb970 2023-12-19T14:39:51.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38158e20 0x7f0c380ce5c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8141b849fe81f85 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c35ffb700 1 -- 172.21.15.50:0/66383031 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800b640 con 0x7f0c38158e20 2023-12-19T14:39:51.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c35ffb700 1 -- 172.21.15.50:0/66383031 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2800b7a0 con 0x7f0c38158e20 2023-12-19T14:39:51.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c35ffb700 1 -- 172.21.15.50:0/66383031 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800bb60 con 0x7f0c38158e20 2023-12-19T14:39:51.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38158e20 msgr2=0x7f0c380ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38158e20 0x7f0c380ce5c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/66383031 shutdown_connections 2023-12-19T14:39:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c380ca2c0 0x7f0c380ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/66383031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38158e20 0x7f0c380ce5c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/66383031 >> 172.21.15.50:0/66383031 conn(0x7f0c38067bb0 msgr2=0x7f0c380d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/66383031 shutdown_connections 2023-12-19T14:39:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.332+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/66383031 wait complete. 2023-12-19T14:39:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.336+0000 7f0c3d3c5700 1 Processor -- start 2023-12-19T14:39:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.336+0000 7f0c3d3c5700 1 -- start start 2023-12-19T14:39:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c38158e20 0x7f0c3814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3814fef0 0x7f0c3815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380b2fb0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3815b330 con 0x7f0c3814fef0 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3815b610 con 0x7f0c38158e20 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57942/0 (socket says 172.21.15.50:57942) 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 -- 172.21.15.50:0/3707823757 learned_addr learned my addr 172.21.15.50:0/3707823757 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c377fe700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3814fef0 0x7f0c3815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c367fc700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c38158e20 0x7f0c3814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 -- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c38158e20 msgr2=0x7f0c3814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c38158e20 0x7f0c3814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 -- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3814fef0 msgr2=0x7f0c3815a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3814fef0 0x7f0c3815a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 -- 172.21.15.50:0/3707823757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009420 con 0x7f0c380ca2c0 2023-12-19T14:39:51.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c36ffd700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009fb0 tx=0x7f0c28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2801f070 con 0x7f0c380ca2c0 2023-12-19T14:39:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28000bc0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800bb60 con 0x7f0c380ca2c0 2023-12-19T14:39:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c380b3200 con 0x7f0c380ca2c0 2023-12-19T14:39:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.340+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3814f6e0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.344+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38071120 con 0x7f0c380ca2c0 2023-12-19T14:39:51.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.344+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0c28023070 con 0x7f0c380ca2c0 2023-12-19T14:39:51.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.344+0000 7f0c1effd700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 0x7f0c24062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.344+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f0c2807dca0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.348+0000 7f0c367fc700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 0x7f0c24062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.348+0000 7f0c367fc700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 0x7f0c24062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0c380b6210 tx=0x7f0c2c007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.352+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0c2804deb0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.528+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c3806a7a0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.528+0000 7f0c1effd700 1 -- 172.21.15.50:0/3707823757 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c280150a0 con 0x7f0c380ca2c0 2023-12-19T14:39:51.532 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 msgr2=0x7f0c24062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 0x7f0c24062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0c380b6210 tx=0x7f0c2c007d30 comp rx=0 tx=0).stop 2023-12-19T14:39:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 msgr2=0x7f0c3814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009fb0 tx=0x7f0c28017000 comp rx=0 tx=0).stop 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 shutdown_connections 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3814fef0 0x7f0c3815a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c24060550 0x7f0c24062a00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c38158e20 0x7f0c3814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 --2- 172.21.15.50:0/3707823757 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c380ca2c0 0x7f0c3814f1a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 >> 172.21.15.50:0/3707823757 conn(0x7f0c38067bb0 msgr2=0x7f0c381568d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 shutdown_connections 2023-12-19T14:39:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.532+0000 7f0c3d3c5700 1 -- 172.21.15.50:0/3707823757 wait complete. 2023-12-19T14:39:51.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:51.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.724+0000 7f84e8e8e700 1 Processor -- start 2023-12-19T14:39:51.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.724+0000 7f84e8e8e700 1 -- start start 2023-12-19T14:39:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e8e8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 0x7f84e40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e8e8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f84e4159770 0x7f84e406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e8e8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e406c850 con 0x7f84e406aeb0 2023-12-19T14:39:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e8e8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e406b470 con 0x7f84e4159770 2023-12-19T14:39:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e8e8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e406b750 con 0x7f84e4158c90 2023-12-19T14:39:51.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 0x7f84e40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f84e4159770 0x7f84e406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e2ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f84e406aeb0 0x7f84e4159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60918/0 (socket says 172.21.15.50:60918) 2023-12-19T14:39:51.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e2ffd700 1 -- 172.21.15.50:0/2561304777 learned_addr learned my addr 172.21.15.50:0/2561304777 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e37fe700 1 -- 172.21.15.50:0/2561304777 >> v2:172.21.15.125:3300/0 conn(0x7f84e4159770 msgr2=0x7f84e406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e37fe700 1 --2- 172.21.15.50:0/2561304777 >> v2:172.21.15.125:3300/0 conn(0x7f84e4159770 0x7f84e406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e37fe700 1 -- 172.21.15.50:0/2561304777 >> v1:172.21.15.125:6789/0 conn(0x7f84e406aeb0 legacy=0x7f84e4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.728+0000 7f84e37fe700 1 -- 172.21.15.50:0/2561304777 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e406ba30 con 0x7f84e4158c90 2023-12-19T14:39:51.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e37fe700 1 --2- 172.21.15.50:0/2561304777 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 0x7f84e40caf40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f84d40098e0 tx=0x7f84d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e942623e6593c7d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e27fc700 1 -- 172.21.15.50:0/2561304777 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d400b640 con 0x7f84e4158c90 2023-12-19T14:39:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e27fc700 1 -- 172.21.15.50:0/2561304777 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d400b7a0 con 0x7f84e4158c90 2023-12-19T14:39:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e27fc700 1 -- 172.21.15.50:0/2561304777 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d400bb40 con 0x7f84e4158c90 2023-12-19T14:39:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 -- 172.21.15.50:0/2561304777 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 msgr2=0x7f84e40caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/2561304777 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 0x7f84e40caf40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f84d40098e0 tx=0x7f84d40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 -- 172.21.15.50:0/2561304777 shutdown_connections 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/2561304777 >> v2:172.21.15.125:3300/0 conn(0x7f84e4159770 0x7f84e406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/2561304777 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4158c90 0x7f84e40caf40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 -- 172.21.15.50:0/2561304777 >> 172.21.15.50:0/2561304777 conn(0x7f84e4067bb0 msgr2=0x7f84e40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 -- 172.21.15.50:0/2561304777 shutdown_connections 2023-12-19T14:39:51.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.732+0000 7f84e8e8e700 1 -- 172.21.15.50:0/2561304777 wait complete. 2023-12-19T14:39:51.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 Processor -- start 2023-12-19T14:39:51.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 -- start start 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 0x7f84e4148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4159770 0x7f84e40c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40c9c00 con 0x7f84e406aeb0 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40c9770 con 0x7f84e4158c90 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e8e8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40c88c0 con 0x7f84e4159770 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57952/0 (socket says 172.21.15.50:57952) 2023-12-19T14:39:51.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e37fe700 1 -- 172.21.15.50:0/475396147 learned_addr learned my addr 172.21.15.50:0/475396147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:51.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e3fff700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4159770 0x7f84e40c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.736+0000 7f84e2ffd700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 0x7f84e4148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 -- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4159770 msgr2=0x7f84e40c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4159770 0x7f84e40c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 -- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 msgr2=0x7f84e4148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 0x7f84e4148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 -- 172.21.15.50:0/475396147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d4009420 con 0x7f84e406aeb0 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e2ffd700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 0x7f84e4148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:51.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e37fe700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009f80 tx=0x7f84d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d401bce0 con 0x7f84e406aeb0 2023-12-19T14:39:51.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d401be80 con 0x7f84e406aeb0 2023-12-19T14:39:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e40c8b60 con 0x7f84e406aeb0 2023-12-19T14:39:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4011670 con 0x7f84e406aeb0 2023-12-19T14:39:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e40c8ef0 con 0x7f84e406aeb0 2023-12-19T14:39:51.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.740+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e40c9520 con 0x7f84e406aeb0 2023-12-19T14:39:51.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.744+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f84d40119e0 con 0x7f84e406aeb0 2023-12-19T14:39:51.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.744+0000 7f84cb7fe700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 0x7f84d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:51.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.744+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f84d407e2c0 con 0x7f84e406aeb0 2023-12-19T14:39:51.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.744+0000 7f84e2ffd700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 0x7f84d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:51.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.744+0000 7f84e2ffd700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 0x7f84d0062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f84e40c6620 tx=0x7f84d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:51.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.752+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f84d404e450 con 0x7f84e406aeb0 2023-12-19T14:39:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.936+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84e404be40 con 0x7f84e406aeb0 2023-12-19T14:39:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.936+0000 7f84cb7fe700 1 -- 172.21.15.50:0/475396147 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84d4051a70 con 0x7f84e406aeb0 2023-12-19T14:39:51.940 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 msgr2=0x7f84d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 0x7f84d0062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f84e40c6620 tx=0x7f84d8009040 comp rx=0 tx=0).stop 2023-12-19T14:39:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 msgr2=0x7f84e4157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009f80 tx=0x7f84d4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 shutdown_connections 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4158c90 0x7f84e4148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84d0060550 0x7f84d0062a00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84e4159770 0x7f84e40c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 --2- 172.21.15.50:0/475396147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84e406aeb0 0x7f84e4157dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 >> 172.21.15.50:0/475396147 conn(0x7f84e4067bb0 msgr2=0x7f84e40cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 shutdown_connections 2023-12-19T14:39:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:51.940+0000 7f84e8e8e700 1 -- 172.21.15.50:0/475396147 wait complete. 2023-12-19T14:39:51.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.140+0000 7f9f9d478700 1 Processor -- start 2023-12-19T14:39:52.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.140+0000 7f9f9d478700 1 -- start start 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f98158e20 0x7f9f980ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f980cb970 0x7f9f980ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 0x7f9f9806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f980cbdf0 con 0x7f9f980cb970 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f980cc0d0 con 0x7f9f98158e20 2023-12-19T14:39:52.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980cc3b0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f980cb970 0x7f9f980ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9f980cb970 0x7f9f980ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57966/0 (socket says 172.21.15.50:57966) 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 -- 172.21.15.50:0/4176349976 learned_addr learned my addr 172.21.15.50:0/4176349976 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f96ffd700 1 --2- 172.21.15.50:0/4176349976 >> v2:172.21.15.125:3300/0 conn(0x7f9f98158e20 0x7f9f980ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f977fe700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 0x7f9f9806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 -- 172.21.15.50:0/4176349976 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 msgr2=0x7f9f9806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 0x7f9f9806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 -- 172.21.15.50:0/4176349976 >> v2:172.21.15.125:3300/0 conn(0x7f9f98158e20 msgr2=0x7f9f980ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 --2- 172.21.15.50:0/4176349976 >> v2:172.21.15.125:3300/0 conn(0x7f9f98158e20 0x7f9f980ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 -- 172.21.15.50:0/4176349976 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980cc690 con 0x7f9f980cb970 2023-12-19T14:39:52.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f977fe700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 0x7f9f9806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f967fc700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980cb970 0x7f9f980ceb00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0099b0 tx=0x7f9f8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225de8d7fd4c7cfd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f95ffb700 1 -- 172.21.15.50:0/4176349976 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c009ea0 con 0x7f9f980cb970 2023-12-19T14:39:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f95ffb700 1 -- 172.21.15.50:0/4176349976 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c00b820 con 0x7f9f980cb970 2023-12-19T14:39:52.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f95ffb700 1 -- 172.21.15.50:0/4176349976 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00bbc0 con 0x7f9f980cb970 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 -- 172.21.15.50:0/4176349976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980cb970 msgr2=0x7f9f980ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.144+0000 7f9f9d478700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980cb970 0x7f9f980ceb00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0099b0 tx=0x7f9f8c009250 comp rx=0 tx=0).stop 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- 172.21.15.50:0/4176349976 shutdown_connections 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980cb970 0x7f9f980ceb00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- 172.21.15.50:0/4176349976 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f980ca2c0 0x7f9f9806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- 172.21.15.50:0/4176349976 >> v2:172.21.15.125:3300/0 conn(0x7f9f98158e20 0x7f9f980ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- 172.21.15.50:0/4176349976 >> 172.21.15.50:0/4176349976 conn(0x7f9f98067bb0 msgr2=0x7f9f980d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- 172.21.15.50:0/4176349976 shutdown_connections 2023-12-19T14:39:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- 172.21.15.50:0/4176349976 wait complete. 2023-12-19T14:39:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 Processor -- start 2023-12-19T14:39:52.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- start start 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f98158e20 0x7f9f980c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f980c7d40 0x7f9f980c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980c9210 con 0x7f9f980ca2c0 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980b3390 con 0x7f9f980c7d40 2023-12-19T14:39:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f9d478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f98153420 con 0x7f9f98158e20 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f96ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f96ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57980/0 (socket says 172.21.15.50:57980) 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f96ffd700 1 -- 172.21.15.50:0/2035563941 learned_addr learned my addr 172.21.15.50:0/2035563941 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f967fc700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f98158e20 0x7f9f980c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.148+0000 7f9f977fe700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f980c7d40 0x7f9f980c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 -- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f98158e20 msgr2=0x7f9f980c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f98158e20 0x7f9f980c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 -- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f980c7d40 msgr2=0x7f9f980c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f980c7d40 0x7f9f980c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 -- 172.21.15.50:0/2035563941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0094f0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f96ffd700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800e930 tx=0x7f9f880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88013680 con 0x7f9f980ca2c0 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f880137e0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88013b80 con 0x7f9f980ca2c0 2023-12-19T14:39:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f98152af0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f981530b0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.152+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f98071120 con 0x7f9f980ca2c0 2023-12-19T14:39:52.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.156+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9f8800c3e0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.156+0000 7f9f7effd700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 0x7f9f84062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.156+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f9f8807ccb0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.156+0000 7f9f967fc700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 0x7f9f84062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.160+0000 7f9f967fc700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 0x7f9f84062a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0099b0 tx=0x7f9f8c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.164+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9f8804ce40 con 0x7f9f980ca2c0 2023-12-19T14:39:52.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.340+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f9806a7a0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.340+0000 7f9f7effd700 1 -- 172.21.15.50:0/2035563941 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9f880210a0 con 0x7f9f980ca2c0 2023-12-19T14:39:52.344 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:52.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 msgr2=0x7f9f84062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 0x7f9f84062a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0099b0 tx=0x7f9f8c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:52.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 msgr2=0x7f9f980c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800e930 tx=0x7f9f880091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 shutdown_connections 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f980c7d40 0x7f9f980c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9f84060580 0x7f9f84062a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9f98158e20 0x7f9f980c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 --2- 172.21.15.50:0/2035563941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f980ca2c0 0x7f9f980c4f30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 >> 172.21.15.50:0/2035563941 conn(0x7f9f98067bb0 msgr2=0x7f9f9814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 shutdown_connections 2023-12-19T14:39:52.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.344+0000 7f9f9d478700 1 -- 172.21.15.50:0/2035563941 wait complete. 2023-12-19T14:39:52.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:52.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.540+0000 7f58940cd700 1 Processor -- start 2023-12-19T14:39:52.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.540+0000 7f58940cd700 1 -- start start 2023-12-19T14:39:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f58940cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f588c158e20 0x7f588c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f58940cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f58940cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0cbdf0 con 0x7f588c0cb970 2023-12-19T14:39:52.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f58940cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c0cc0d0 con 0x7f588c158e20 2023-12-19T14:39:52.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f58940cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0cc3b0 con 0x7f588c0ca2c0 2023-12-19T14:39:52.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f5891668700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f588c0cb970 0x7f588c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35074/0 (socket says 172.21.15.50:35074) 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f5891668700 1 -- 172.21.15.50:0/3721377244 learned_addr learned my addr 172.21.15.50:0/3721377244 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f5891e69700 1 --2- 172.21.15.50:0/3721377244 >> v2:172.21.15.125:3300/0 conn(0x7f588c158e20 0x7f588c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.544+0000 7f589266a700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5891e69700 1 -- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 msgr2=0x7f588c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5890e67700 1 -- 172.21.15.50:0/3721377244 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116745344 0 0) 0x7f588c0cbdf0 con 0x7f588c0cb970 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5891e69700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5891e69700 1 -- 172.21.15.50:0/3721377244 >> v1:172.21.15.50:6789/0 conn(0x7f588c0cb970 legacy=0x7f588c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5891e69700 1 -- 172.21.15.50:0/3721377244 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0cc690 con 0x7f588c158e20 2023-12-19T14:39:52.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5891e69700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c158e20 0x7f588c0ce5c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5888009910 tx=0x7f58880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d692ebb5353e1b5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5890e67700 1 -- 172.21.15.50:0/3721377244 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800b640 con 0x7f588c158e20 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5890e67700 1 -- 172.21.15.50:0/3721377244 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800b7a0 con 0x7f588c158e20 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f5890e67700 1 -- 172.21.15.50:0/3721377244 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800bb60 con 0x7f588c158e20 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 -- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c158e20 msgr2=0x7f588c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c158e20 0x7f588c0ce5c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5888009910 tx=0x7f58880091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 -- 172.21.15.50:0/3721377244 shutdown_connections 2023-12-19T14:39:52.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 --2- 172.21.15.50:0/3721377244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c158e20 0x7f588c0ce5c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 -- 172.21.15.50:0/3721377244 >> 172.21.15.50:0/3721377244 conn(0x7f588c067bb0 msgr2=0x7f588c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:52.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 -- 172.21.15.50:0/3721377244 shutdown_connections 2023-12-19T14:39:52.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.548+0000 7f58940cd700 1 -- 172.21.15.50:0/3721377244 wait complete. 2023-12-19T14:39:52.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.552+0000 7f58940cd700 1 Processor -- start 2023-12-19T14:39:52.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.552+0000 7f58940cd700 1 -- start start 2023-12-19T14:39:52.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 0x7f588c169db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0c9fd0 con 0x7f588c158e20 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c15bda0 con 0x7f588c169940 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c16ab10 con 0x7f588c0ca2c0 2023-12-19T14:39:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5891668700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5891668700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57984/0 (socket says 172.21.15.50:57984) 2023-12-19T14:39:52.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5891668700 1 -- 172.21.15.50:0/1897109120 learned_addr learned my addr 172.21.15.50:0/1897109120 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:52.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5891e69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 0x7f588c169db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 -- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 msgr2=0x7f588c159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 -- 172.21.15.50:0/1897109120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 msgr2=0x7f588c15a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 -- 172.21.15.50:0/1897109120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888009420 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5891668700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f589266a700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 0x7f588c169db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5884009e50 tx=0x7f588400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400caa0 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400f040 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400ce50 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c15a770 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.556+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c0b4550 con 0x7f588c169940 2023-12-19T14:39:52.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.560+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c071120 con 0x7f588c169940 2023-12-19T14:39:52.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.560+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5884014690 con 0x7f588c169940 2023-12-19T14:39:52.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.564+0000 7f5881ffb700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 0x7f5870062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.564+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f588407d760 con 0x7f588c169940 2023-12-19T14:39:52.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.564+0000 7f5891e69700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 0x7f5870062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.564+0000 7f5891e69700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 0x7f5870062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f58880091e0 tx=0x7f5888017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.572+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5884051380 con 0x7f588c169940 2023-12-19T14:39:52.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.748+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f588c06a7a0 con 0x7f588c169940 2023-12-19T14:39:52.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.748+0000 7f5881ffb700 1 -- 172.21.15.50:0/1897109120 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5884050f10 con 0x7f588c169940 2023-12-19T14:39:52.753 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:52.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 msgr2=0x7f5870062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 0x7f5870062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f58880091e0 tx=0x7f5888017000 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 msgr2=0x7f588c169db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 0x7f588c169db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5884009e50 tx=0x7f588400b040 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 shutdown_connections 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f588c169940 0x7f588c169db0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5870060550 0x7f5870062a00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f588c158e20 0x7f588c15a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 --2- 172.21.15.50:0/1897109120 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f588c0ca2c0 0x7f588c159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 >> 172.21.15.50:0/1897109120 conn(0x7f588c067bb0 msgr2=0x7f588c0cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 shutdown_connections 2023-12-19T14:39:52.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.752+0000 7f58940cd700 1 -- 172.21.15.50:0/1897109120 wait complete. 2023-12-19T14:39:52.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.940+0000 7fb20dd45700 1 Processor -- start 2023-12-19T14:39:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.940+0000 7fb20dd45700 1 -- start start 2023-12-19T14:39:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb20dd45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2080c15b0 0x7fb20815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb20dd45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080c7190 con 0x7fb2080aeba0 2023-12-19T14:39:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb20dd45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080c1b50 con 0x7fb20815d320 2023-12-19T14:39:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb20dd45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2080aefd0 con 0x7fb2080c15b0 2023-12-19T14:39:52.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb207fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb2080aeba0 0x7fb2080680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35086/0 (socket says 172.21.15.50:35086) 2023-12-19T14:39:52.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb207fff700 1 -- 172.21.15.50:0/2448797299 learned_addr learned my addr 172.21.15.50:0/2448797299 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb206ffd700 1 --2- 172.21.15.50:0/2448797299 >> v2:172.21.15.125:3300/0 conn(0x7fb2080c15b0 0x7fb20815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325715321 0 0) 0x7fb2080c7190 con 0x7fb2080aeba0 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f0003570 con 0x7fb2080aeba0 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679697518 0 0) 0x7fb1f0003570 con 0x7fb2080aeba0 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 >> v2:172.21.15.125:3300/0 conn(0x7fb2080c15b0 msgr2=0x7fb20815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 --2- 172.21.15.50:0/2448797299 >> v2:172.21.15.125:3300/0 conn(0x7fb2080c15b0 0x7fb20815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 >> v1:172.21.15.125:6789/0 conn(0x7fb20815d320 legacy=0x7fb20806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080c1e30 con 0x7fb2080aeba0 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fb2000017f0 con 0x7fb2080aeba0 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 >> v1:172.21.15.50:6789/0 conn(0x7fb2080aeba0 legacy=0x7fb2080680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f0003a60 0x7fb1f0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb1f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1f00051c0 0x7fb1f000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f000c050 con 0x7fb1f0004810 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f000c190 con 0x7fb1f0003a60 2023-12-19T14:39:52.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f000c350 con 0x7fb1f00051c0 2023-12-19T14:39:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.944+0000 7fb206ffd700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb1f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb207fff700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1f00051c0 0x7fb1f000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb2077fe700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f0003a60 0x7fb1f0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 -- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1f00051c0 msgr2=0x7fb1f000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1f00051c0 0x7fb1f000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 -- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f0003a60 msgr2=0x7fb1f0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f0003a60 0x7fb1f0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 -- 172.21.15.50:0/2448797299 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f0003570 con 0x7fb1f0004810 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb207fff700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1f00051c0 0x7fb1f000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb207fff700 1 -- 172.21.15.50:0/2448797299 reap_dead start 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb206ffd700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb1f0004c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009c00 tx=0x7fb1fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e8c1bd2e6d0904 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00e710 con 0x7fb1f0004810 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc00e870 con 0x7fb1f0004810 2023-12-19T14:39:52.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb2067fc700 1 -- 172.21.15.50:0/2448797299 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00ec30 con 0x7fb1f0004810 2023-12-19T14:39:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 -- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 msgr2=0x7fb1f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb1f0004c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009c00 tx=0x7fb1fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 -- 172.21.15.50:0/2448797299 shutdown_connections 2023-12-19T14:39:52.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 --2- 172.21.15.50:0/2448797299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb1f0004c80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 -- 172.21.15.50:0/2448797299 >> 172.21.15.50:0/2448797299 conn(0x7fb2080c8850 msgr2=0x7fb2080c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 -- 172.21.15.50:0/2448797299 shutdown_connections 2023-12-19T14:39:52.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.948+0000 7fb20dd45700 1 -- 172.21.15.50:0/2448797299 wait complete. 2023-12-19T14:39:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 Processor -- start 2023-12-19T14:39:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 -- start start 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20815fe40 0x7fb20815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb20815dc30 0x7fb20815e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20815e5a0 con 0x7fb1f0004810 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2080c3bd0 con 0x7fb20815fe40 2023-12-19T14:39:52.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb20dd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2081602b0 con 0x7fb20815dc30 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb2077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb2077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58000/0 (socket says 172.21.15.50:58000) 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb2077fe700 1 -- 172.21.15.50:0/1791873571 learned_addr learned my addr 172.21.15.50:0/1791873571 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.952+0000 7fb206ffd700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20815fe40 0x7fb20815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb207fff700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb20815dc30 0x7fb20815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 -- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb20815dc30 msgr2=0x7fb20815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb20815dc30 0x7fb20815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 -- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20815fe40 msgr2=0x7fb20815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20815fe40 0x7fb20815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 -- 172.21.15.50:0/1791873571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080c2fc0 con 0x7fb1f0004810 2023-12-19T14:39:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb2077fe700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8002b00 tx=0x7fb1f8007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800f8d0 con 0x7fb1f0004810 2023-12-19T14:39:52.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f800fa50 con 0x7fb1f0004810 2023-12-19T14:39:52.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2080c32a0 con 0x7fb1f0004810 2023-12-19T14:39:52.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800fe10 con 0x7fb1f0004810 2023-12-19T14:39:52.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2080c3860 con 0x7fb1f0004810 2023-12-19T14:39:52.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb208071120 con 0x7fb1f0004810 2023-12-19T14:39:52.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.956+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb1f8013600 con 0x7fb1f0004810 2023-12-19T14:39:52.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.960+0000 7fb1ef7fe700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 0x7fb1f4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.960+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fb1f807d290 con 0x7fb1f0004810 2023-12-19T14:39:52.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.960+0000 7fb206ffd700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 0x7fb1f4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:52.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.964+0000 7fb206ffd700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 0x7fb1f4062a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0091e0 tx=0x7fb1fc01a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:52.968+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb1f804d420 con 0x7fb1f0004810 2023-12-19T14:39:53.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.140+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2080642a0 con 0x7fb1f0004810 2023-12-19T14:39:53.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.140+0000 7fb1ef7fe700 1 -- 172.21.15.50:0/1791873571 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7fb1f8050a40 con 0x7fb1f0004810 2023-12-19T14:39:53.145 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:53.145 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 msgr2=0x7fb1f4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 0x7fb1f4062a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0091e0 tx=0x7fb1fc01a000 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 msgr2=0x7fb20815f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8002b00 tx=0x7fb1f8007d10 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 shutdown_connections 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20815fe40 0x7fb20815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb1f4060580 0x7fb1f4062a30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb20815dc30 0x7fb20815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 --2- 172.21.15.50:0/1791873571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f0004810 0x7fb20815f900 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 >> 172.21.15.50:0/1791873571 conn(0x7fb2080c8850 msgr2=0x7fb2080670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 shutdown_connections 2023-12-19T14:39:53.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.144+0000 7fb20dd45700 1 -- 172.21.15.50:0/1791873571 wait complete. 2023-12-19T14:39:53.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:53.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.344+0000 7f0992e16700 1 Processor -- start 2023-12-19T14:39:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.344+0000 7f0992e16700 1 -- start start 2023-12-19T14:39:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992e16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f098c0ace10 0x7f098c0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992e16700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f098c06daa0 0x7f098c0c8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992e16700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098c0c8e80 con 0x7f098c0c84e0 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992e16700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098c157110 con 0x7f098c06daa0 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992e16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098c0ae110 con 0x7f098c0ace10 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f098c06daa0 0x7f098c0c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f098c06daa0 0x7f098c0c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58014/0 (socket says 172.21.15.50:58014) 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 -- 172.21.15.50:0/3109392430 learned_addr learned my addr 172.21.15.50:0/3109392430 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0991613700 1 --1- 172.21.15.50:0/3109392430 >> v1:172.21.15.50:6789/0 conn(0x7f098c0c84e0 0x7f098c06d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35102/0 (socket says 172.21.15.50:35102) 2023-12-19T14:39:53.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0991e14700 1 --2- 172.21.15.50:0/3109392430 >> v2:172.21.15.125:3300/0 conn(0x7f098c0ace10 0x7f098c0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 -- 172.21.15.50:0/3109392430 >> v2:172.21.15.125:3300/0 conn(0x7f098c0ace10 msgr2=0x7f098c0ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 --2- 172.21.15.50:0/3109392430 >> v2:172.21.15.125:3300/0 conn(0x7f098c0ace10 0x7f098c0ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 -- 172.21.15.50:0/3109392430 >> v1:172.21.15.50:6789/0 conn(0x7f098c0c84e0 legacy=0x7f098c06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 -- 172.21.15.50:0/3109392430 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c06e0c0 con 0x7f098c06daa0 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0992615700 1 --2- 172.21.15.50:0/3109392430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c06daa0 0x7f098c0c8940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f098c0cb710 tx=0x7f0984009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8375652e92b85d54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0990e12700 1 -- 172.21.15.50:0/3109392430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098400e070 con 0x7f098c06daa0 2023-12-19T14:39:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0990e12700 1 -- 172.21.15.50:0/3109392430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098400ba50 con 0x7f098c06daa0 2023-12-19T14:39:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.348+0000 7f0990e12700 1 -- 172.21.15.50:0/3109392430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09840092e0 con 0x7f098c06daa0 2023-12-19T14:39:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- 172.21.15.50:0/3109392430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c06daa0 msgr2=0x7f098c0c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- 172.21.15.50:0/3109392430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c06daa0 0x7f098c0c8940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f098c0cb710 tx=0x7f0984009840 comp rx=0 tx=0).stop 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- 172.21.15.50:0/3109392430 shutdown_connections 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- 172.21.15.50:0/3109392430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c06daa0 0x7f098c0c8940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- 172.21.15.50:0/3109392430 >> v2:172.21.15.125:3300/0 conn(0x7f098c0ace10 0x7f098c0ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- 172.21.15.50:0/3109392430 >> 172.21.15.50:0/3109392430 conn(0x7f098c0cdc00 msgr2=0x7f098c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- 172.21.15.50:0/3109392430 shutdown_connections 2023-12-19T14:39:53.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- 172.21.15.50:0/3109392430 wait complete. 2023-12-19T14:39:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 Processor -- start 2023-12-19T14:39:53.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- start start 2023-12-19T14:39:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f098c06daa0 0x7f098c156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f098c0c84e0 0x7f098c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c156dd0 con 0x7f098c0ace10 2023-12-19T14:39:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c1695d0 con 0x7f098c0c84e0 2023-12-19T14:39:53.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.352+0000 7f0992e16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c15ae20 con 0x7f098c06daa0 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58028/0 (socket says 172.21.15.50:58028) 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 -- 172.21.15.50:0/3461120969 learned_addr learned my addr 172.21.15.50:0/3461120969 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991e14700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f098c06daa0 0x7f098c156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0992615700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f098c0c84e0 0x7f098c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 -- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f098c06daa0 msgr2=0x7f098c156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f098c06daa0 0x7f098c156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 -- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f098c0c84e0 msgr2=0x7f098c159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f098c0c84e0 0x7f098c159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 -- 172.21.15.50:0/3461120969 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0984009ae0 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0991613700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c7c0 tx=0x7f097c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c005a70 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c005bd0 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c15b100 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c0070d0 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.356+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c169250 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.360+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c04be40 con 0x7f098c0ace10 2023-12-19T14:39:53.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.360+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f097c007480 con 0x7f098c0ace10 2023-12-19T14:39:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.360+0000 7f0981ffb700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 0x7f0970062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.360+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f097c07be50 con 0x7f098c0ace10 2023-12-19T14:39:53.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.364+0000 7f0991e14700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 0x7f0970062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.364+0000 7f0991e14700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 0x7f0970062a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.368+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f097c04bfe0 con 0x7f098c0ace10 2023-12-19T14:39:53.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.544+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f098c0c8e80 con 0x7f098c0ace10 2023-12-19T14:39:53.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.544+0000 7f0981ffb700 1 -- 172.21.15.50:0/3461120969 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f098c0c8e80 con 0x7f098c0ace10 2023-12-19T14:39:53.549 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 msgr2=0x7f0970062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 0x7f0970062a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988008040 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 msgr2=0x7f098c15c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f097c00c7c0 tx=0x7f097c007f70 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 shutdown_connections 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f098c0c84e0 0x7f098c159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0970060580 0x7f0970062a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f098c0ace10 0x7f098c15c8f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 --2- 172.21.15.50:0/3461120969 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f098c06daa0 0x7f098c156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 >> 172.21.15.50:0/3461120969 conn(0x7f098c0cdc00 msgr2=0x7f098c0c29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 shutdown_connections 2023-12-19T14:39:53.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.552+0000 7f0992e16700 1 -- 172.21.15.50:0/3461120969 wait complete. 2023-12-19T14:39:53.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 Processor -- start 2023-12-19T14:39:53.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 -- start start 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ac80c69a0 0x7f5ac80c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 0x7f5ac806b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac806d970 con 0x7f5ac806de00 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac806cd60 con 0x7f5ac80c69a0 2023-12-19T14:39:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acea4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac806d040 con 0x7f5ac806c7c0 2023-12-19T14:39:53.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.744+0000 7f5acd249700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 0x7f5ac806b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:42452/0 (socket says 172.21.15.50:42452) 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acd249700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 0x7f5ac806b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58036/0 (socket says 172.21.15.50:58036) 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 -- 172.21.15.50:0/2294041757 learned_addr learned my addr 172.21.15.50:0/2294041757 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acda4a700 1 --2- 172.21.15.50:0/2294041757 >> v2:172.21.15.125:3300/0 conn(0x7f5ac80c69a0 0x7f5ac80c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 -- 172.21.15.50:0/2294041757 >> v2:172.21.15.125:3300/0 conn(0x7f5ac80c69a0 msgr2=0x7f5ac80c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 --2- 172.21.15.50:0/2294041757 >> v2:172.21.15.125:3300/0 conn(0x7f5ac80c69a0 0x7f5ac80c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 -- 172.21.15.50:0/2294041757 >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 msgr2=0x7f5ac806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 --2- 172.21.15.50:0/2294041757 >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 0x7f5ac806b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 -- 172.21.15.50:0/2294041757 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac806d320 con 0x7f5ac806c7c0 2023-12-19T14:39:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5ace24b700 1 --2- 172.21.15.50:0/2294041757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80c8390 tx=0x7f5ac0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d988988c478f6dbd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acca48700 1 -- 172.21.15.50:0/2294041757 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000e070 con 0x7f5ac806c7c0 2023-12-19T14:39:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acca48700 1 -- 172.21.15.50:0/2294041757 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac000b970 con 0x7f5ac806c7c0 2023-12-19T14:39:53.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acca48700 1 -- 172.21.15.50:0/2294041757 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00092e0 con 0x7f5ac806c7c0 2023-12-19T14:39:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 -- 172.21.15.50:0/2294041757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 msgr2=0x7f5ac815cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 --2- 172.21.15.50:0/2294041757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80c8390 tx=0x7f5ac0009840 comp rx=0 tx=0).stop 2023-12-19T14:39:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 -- 172.21.15.50:0/2294041757 shutdown_connections 2023-12-19T14:39:53.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 --2- 172.21.15.50:0/2294041757 >> v2:172.21.15.50:3300/0 conn(0x7f5ac806de00 0x7f5ac806b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 --2- 172.21.15.50:0/2294041757 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac806c7c0 0x7f5ac815cba0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 --2- 172.21.15.50:0/2294041757 >> v2:172.21.15.125:3300/0 conn(0x7f5ac80c69a0 0x7f5ac80c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 -- 172.21.15.50:0/2294041757 >> 172.21.15.50:0/2294041757 conn(0x7f5ac8067b50 msgr2=0x7f5ac80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 -- 172.21.15.50:0/2294041757 shutdown_connections 2023-12-19T14:39:53.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.748+0000 7f5acea4c700 1 -- 172.21.15.50:0/2294041757 wait complete. 2023-12-19T14:39:53.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.752+0000 7f5acea4c700 1 Processor -- start 2023-12-19T14:39:53.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.752+0000 7f5acea4c700 1 -- start start 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 0x7f5ac815e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac80c69a0 0x7f5ac80c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80c3f50 con 0x7f5ac806de00 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac815c310 con 0x7f5ac806c7c0 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80c0e10 con 0x7f5ac80c69a0 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58052/0 (socket says 172.21.15.50:58052) 2023-12-19T14:39:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 -- 172.21.15.50:0/3040051899 learned_addr learned my addr 172.21.15.50:0/3040051899 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acda4a700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 0x7f5ac815e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5ace24b700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac80c69a0 0x7f5ac80c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 -- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac80c69a0 msgr2=0x7f5ac80c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac80c69a0 0x7f5ac80c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 -- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 msgr2=0x7f5ac815e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 0x7f5ac815e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 -- 172.21.15.50:0/3040051899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac0009ae0 con 0x7f5ac806de00 2023-12-19T14:39:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acda4a700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 0x7f5ac815e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acd249700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec80 tx=0x7f5ab80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800c9d0 con 0x7f5ac806de00 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab800cb30 con 0x7f5ac806de00 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8013470 con 0x7f5ac806de00 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac80c10e0 con 0x7f5ac806de00 2023-12-19T14:39:53.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.756+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac80c16a0 con 0x7f5ac806de00 2023-12-19T14:39:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.760+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac80c18c0 con 0x7f5ac806de00 2023-12-19T14:39:53.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.760+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5ab8013820 con 0x7f5ac806de00 2023-12-19T14:39:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.760+0000 7f5abdffb700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 0x7f5aac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.760+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f5ab807cf70 con 0x7f5ac806de00 2023-12-19T14:39:53.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.764+0000 7f5acda4a700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 0x7f5aac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.764+0000 7f5acda4a700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 0x7f5aac062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009ce0 tx=0x7f5ac4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:53.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.768+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ab804d100 con 0x7f5ac806de00 2023-12-19T14:39:53.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.944+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ac804be40 con 0x7f5ac806de00 2023-12-19T14:39:53.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5abdffb700 1 -- 172.21.15.50:0/3040051899 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ab8050720 con 0x7f5ac806de00 2023-12-19T14:39:53.951 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 msgr2=0x7f5aac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 0x7f5aac062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009ce0 tx=0x7f5ac4008040 comp rx=0 tx=0).stop 2023-12-19T14:39:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 msgr2=0x7f5ac815bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec80 tx=0x7f5ab80091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 shutdown_connections 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5aac060550 0x7f5aac062a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac80c69a0 0x7f5ac80c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806de00 0x7f5ac815bdd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 --2- 172.21.15.50:0/3040051899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac806c7c0 0x7f5ac815e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 >> 172.21.15.50:0/3040051899 conn(0x7f5ac8067b50 msgr2=0x7f5ac80acb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.948+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 shutdown_connections 2023-12-19T14:39:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:53.952+0000 7f5acea4c700 1 -- 172.21.15.50:0/3040051899 wait complete. 2023-12-19T14:39:53.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.140+0000 7fdfa202e700 1 Processor -- start 2023-12-19T14:39:54.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.140+0000 7fdfa202e700 1 -- start start 2023-12-19T14:39:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdfa202e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf9c0cb970 0x7fdf9c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdfa202e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdfa202e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0cbdf0 con 0x7fdf9c158e20 2023-12-19T14:39:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdfa202e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cc0d0 con 0x7fdf9c0cb970 2023-12-19T14:39:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdfa202e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0cc3b0 con 0x7fdf9c0cecd0 2023-12-19T14:39:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf9c0cb970 0x7fdf9c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf9c0cb970 0x7fdf9c06acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58058/0 (socket says 172.21.15.50:58058) 2023-12-19T14:39:54.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 -- 172.21.15.50:0/1597334107 learned_addr learned my addr 172.21.15.50:0/1597334107 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf9bfff700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 -- 172.21.15.50:0/1597334107 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 msgr2=0x7fdf9c06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c06b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 -- 172.21.15.50:0/1597334107 >> v1:172.21.15.125:6789/0 conn(0x7fdf9c158e20 legacy=0x7fdf9c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf93fff700 1 -- 172.21.15.50:0/1597334107 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0cc690 con 0x7fdf9c0cb970 2023-12-19T14:39:54.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.144+0000 7fdf9bfff700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c06b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdf93fff700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0cb970 0x7fdf9c06acc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdf840099e0 tx=0x7fdf84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a90a3211174ada server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdf9affd700 1 -- 172.21.15.50:0/1597334107 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8400b820 con 0x7fdf9c0cb970 2023-12-19T14:39:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdf9affd700 1 -- 172.21.15.50:0/1597334107 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8400b980 con 0x7fdf9c0cb970 2023-12-19T14:39:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdf9affd700 1 -- 172.21.15.50:0/1597334107 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8400bd20 con 0x7fdf9c0cb970 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 -- 172.21.15.50:0/1597334107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0cb970 msgr2=0x7fdf9c06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0cb970 0x7fdf9c06acc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdf840099e0 tx=0x7fdf84009250 comp rx=0 tx=0).stop 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 -- 172.21.15.50:0/1597334107 shutdown_connections 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c06b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1597334107 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0cb970 0x7fdf9c06acc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 -- 172.21.15.50:0/1597334107 >> 172.21.15.50:0/1597334107 conn(0x7fdf9c067bb0 msgr2=0x7fdf9c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 -- 172.21.15.50:0/1597334107 shutdown_connections 2023-12-19T14:39:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.148+0000 7fdfa202e700 1 -- 172.21.15.50:0/1597334107 wait complete. 2023-12-19T14:39:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 Processor -- start 2023-12-19T14:39:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 -- start start 2023-12-19T14:39:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf9c158e20 0x7fdf9c0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0c9ef0 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c14ad10 con 0x7fdf9c158e20 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0c8180 con 0x7fdf9c0cecd0 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58060/0 (socket says 172.21.15.50:58060) 2023-12-19T14:39:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 -- 172.21.15.50:0/1658722341 learned_addr learned my addr 172.21.15.50:0/1658722341 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf93fff700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf9c158e20 0x7fdf9c0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9b7fe700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 -- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 msgr2=0x7fdf9c14d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c14d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 -- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf9c158e20 msgr2=0x7fdf9c0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf9c158e20 0x7fdf9c0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 -- 172.21.15.50:0/1658722341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840094f0 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf9bfff700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400d150 tx=0x7fdf94009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9400dd40 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9400f040 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c0c8460 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.152+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94015690 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.156+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c14c350 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.156+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c14ae80 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.156+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdf940143a0 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.160+0000 7fdf937fe700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 0x7fdf7c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.160+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fdf9407cde0 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.160+0000 7fdf9b7fe700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 0x7fdf7c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.160+0000 7fdf9b7fe700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 0x7fdf7c062a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0c55a0 tx=0x7fdf8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.164+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdf9404cf70 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.344+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf9c071120 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.344+0000 7fdf937fe700 1 -- 172.21.15.50:0/1658722341 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf9401a020 con 0x7fdf9c0c9a70 2023-12-19T14:39:54.350 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 msgr2=0x7fdf7c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 0x7fdf7c062a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0c55a0 tx=0x7fdf8c009040 comp rx=0 tx=0).stop 2023-12-19T14:39:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 msgr2=0x7fdf9c149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400d150 tx=0x7fdf94009a50 comp rx=0 tx=0).stop 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 shutdown_connections 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf9c158e20 0x7fdf9c0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf7c060580 0x7fdf7c062a30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf9c0c9a70 0x7fdf9c149e20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 --2- 172.21.15.50:0/1658722341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf9c0cecd0 0x7fdf9c14d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 >> 172.21.15.50:0/1658722341 conn(0x7fdf9c067bb0 msgr2=0x7fdf9c14c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 shutdown_connections 2023-12-19T14:39:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.348+0000 7fdfa202e700 1 -- 172.21.15.50:0/1658722341 wait complete. 2023-12-19T14:39:54.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.552+0000 7f02bc9c5700 1 Processor -- start 2023-12-19T14:39:54.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.552+0000 7f02bc9c5700 1 -- start start 2023-12-19T14:39:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02bc9c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b815c8f0 0x7f02b80a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02bc9c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02b80ca240 0x7f02b80ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02bc9c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80c6ab0 con 0x7f02b815c520 2023-12-19T14:39:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02bc9c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b806b520 con 0x7f02b815c8f0 2023-12-19T14:39:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02bc9c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b80cb910 con 0x7f02b80ca240 2023-12-19T14:39:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b815c8f0 0x7f02b80a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b815c8f0 0x7f02b80a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58074/0 (socket says 172.21.15.50:58074) 2023-12-19T14:39:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b6d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02b80ca240 0x7f02b80ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 -- 172.21.15.50:0/3551549036 learned_addr learned my addr 172.21.15.50:0/3551549036 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:54.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b659c700 1 --1- 172.21.15.50:0/3551549036 >> v1:172.21.15.125:6789/0 conn(0x7f02b815c520 0x7f02b806a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60946/0 (socket says 172.21.15.50:60946) 2023-12-19T14:39:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 -- 172.21.15.50:0/3551549036 >> v2:172.21.15.125:3300/0 conn(0x7f02b80ca240 msgr2=0x7f02b80ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 --2- 172.21.15.50:0/3551549036 >> v2:172.21.15.125:3300/0 conn(0x7f02b80ca240 0x7f02b80ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 -- 172.21.15.50:0/3551549036 >> v1:172.21.15.125:6789/0 conn(0x7f02b815c520 legacy=0x7f02b806a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.556+0000 7f02b5d9b700 1 -- 172.21.15.50:0/3551549036 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80cab70 con 0x7f02b815c8f0 2023-12-19T14:39:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02b5d9b700 1 --2- 172.21.15.50:0/3551549036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c8f0 0x7f02b80a8850 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02b80635a0 tx=0x7f02ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dcdbd382578b52 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02b559a700 1 -- 172.21.15.50:0/3551549036 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac009e70 con 0x7f02b815c8f0 2023-12-19T14:39:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02b559a700 1 -- 172.21.15.50:0/3551549036 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac00b8f0 con 0x7f02b815c8f0 2023-12-19T14:39:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02b559a700 1 -- 172.21.15.50:0/3551549036 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac00bc90 con 0x7f02b815c8f0 2023-12-19T14:39:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 -- 172.21.15.50:0/3551549036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c8f0 msgr2=0x7f02b80a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/3551549036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c8f0 0x7f02b80a8850 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02b80635a0 tx=0x7f02ac009250 comp rx=0 tx=0).stop 2023-12-19T14:39:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 -- 172.21.15.50:0/3551549036 shutdown_connections 2023-12-19T14:39:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/3551549036 >> v2:172.21.15.125:3300/0 conn(0x7f02b80ca240 0x7f02b80ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/3551549036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c8f0 0x7f02b80a8850 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 -- 172.21.15.50:0/3551549036 >> 172.21.15.50:0/3551549036 conn(0x7f02b8067bb0 msgr2=0x7f02b80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 -- 172.21.15.50:0/3551549036 shutdown_connections 2023-12-19T14:39:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.560+0000 7f02bc9c5700 1 -- 172.21.15.50:0/3551549036 wait complete. 2023-12-19T14:39:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 Processor -- start 2023-12-19T14:39:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 -- start start 2023-12-19T14:39:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 0x7f02b815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b815c8f0 0x7f02b8153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b815e170 con 0x7f02b815c520 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b815d5f0 con 0x7f02b80ca240 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02bc9c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b815d8d0 con 0x7f02b815c8f0 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58082/0 (socket says 172.21.15.50:58082) 2023-12-19T14:39:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 -- 172.21.15.50:0/2818580417 learned_addr learned my addr 172.21.15.50:0/2818580417 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b659c700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 0x7f02b815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b6d9d700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b815c8f0 0x7f02b8153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 -- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b815c8f0 msgr2=0x7f02b8153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b815c8f0 0x7f02b8153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 -- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 msgr2=0x7f02b815ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 0x7f02b815ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 -- 172.21.15.50:0/2818580417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0094f0 con 0x7f02b815c520 2023-12-19T14:39:54.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b659c700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 0x7f02b815ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:54.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.564+0000 7f02b5d9b700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02b806a040 tx=0x7f02ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac024440 con 0x7f02b815c520 2023-12-19T14:39:54.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac0245a0 con 0x7f02b815c520 2023-12-19T14:39:54.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac024960 con 0x7f02b815c520 2023-12-19T14:39:54.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b815dbb0 con 0x7f02b815c520 2023-12-19T14:39:54.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b815d230 con 0x7f02b815c520 2023-12-19T14:39:54.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b815e940 con 0x7f02b815c520 2023-12-19T14:39:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.568+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f02ac01a420 con 0x7f02b815c520 2023-12-19T14:39:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.572+0000 7f02a67fc700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 0x7f0298062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.572+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f02ac07e8b0 con 0x7f02b815c520 2023-12-19T14:39:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.572+0000 7f02b659c700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 0x7f0298062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.572+0000 7f02b659c700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 0x7f0298062ad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02a8007fd0 tx=0x7f02a8007e10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.580+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02b815e940 con 0x7f02b815c520 2023-12-19T14:39:54.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.744+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f02b815e940 con 0x7f02b815c520 2023-12-19T14:39:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.748+0000 7f02a67fc700 1 -- 172.21.15.50:0/2818580417 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f02b815e940 con 0x7f02b815c520 2023-12-19T14:39:54.750 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.748+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 msgr2=0x7f0298062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.748+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 0x7f0298062ad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f02a8007fd0 tx=0x7f02a8007e10 comp rx=0 tx=0).stop 2023-12-19T14:39:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 msgr2=0x7f02b815b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02b806a040 tx=0x7f02ac017000 comp rx=0 tx=0).stop 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 shutdown_connections 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0298060620 0x7f0298062ad0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b815c8f0 0x7f02b8153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b815c520 0x7f02b815b200 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 --2- 172.21.15.50:0/2818580417 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b80ca240 0x7f02b815ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 >> 172.21.15.50:0/2818580417 conn(0x7f02b8067bb0 msgr2=0x7f02b80c0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 shutdown_connections 2023-12-19T14:39:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.752+0000 7f02bc9c5700 1 -- 172.21.15.50:0/2818580417 wait complete. 2023-12-19T14:39:54.785 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-12-19T14:39:54.785 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:39:54.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.956+0000 7fb111745700 1 Processor -- start 2023-12-19T14:39:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.956+0000 7fb111745700 1 -- start start 2023-12-19T14:39:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb111745700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 0x7fb10c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb111745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 0x7fb10c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb111745700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c0c6630 con 0x7fb10c067c90 2023-12-19T14:39:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb111745700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c0c6910 con 0x7fb10c1502b0 2023-12-19T14:39:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb111745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0c6bf0 con 0x7fb10c0c5900 2023-12-19T14:39:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10affd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb10c067c90 0x7fb10c14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60954/0 (socket says 172.21.15.50:60954) 2023-12-19T14:39:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10affd700 1 -- 172.21.15.50:0/986171428 learned_addr learned my addr 172.21.15.50:0/986171428 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/986171428 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 0x7fb10c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10a7fc700 1 --2- 172.21.15.50:0/986171428 >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 0x7fb10c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb109ffb700 1 -- 172.21.15.50:0/986171428 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569429219 0 0) 0x7fb10c0c6630 con 0x7fb10c067c90 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb109ffb700 1 -- 172.21.15.50:0/986171428 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4003570 con 0x7fb10c067c90 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 -- 172.21.15.50:0/986171428 >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 msgr2=0x7fb10c068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/986171428 >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 0x7fb10c068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 -- 172.21.15.50:0/986171428 >> v1:172.21.15.125:6789/0 conn(0x7fb10c067c90 legacy=0x7fb10c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 -- 172.21.15.50:0/986171428 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0c6ed0 con 0x7fb10c0c5900 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10a7fc700 1 --2- 172.21.15.50:0/986171428 >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 0x7fb10c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:54.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.960+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/986171428 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 0x7fb10c0c5d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb104009e60 tx=0x7fb104009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7c2e6071a21f79 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb109ffb700 1 -- 172.21.15.50:0/986171428 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10400e070 con 0x7fb10c0c5900 2023-12-19T14:39:54.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb109ffb700 1 -- 172.21.15.50:0/986171428 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400bab0 con 0x7fb10c0c5900 2023-12-19T14:39:54.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb109ffb700 1 -- 172.21.15.50:0/986171428 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10400be70 con 0x7fb10c0c5900 2023-12-19T14:39:54.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 -- 172.21.15.50:0/986171428 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 msgr2=0x7fb10c0c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 --2- 172.21.15.50:0/986171428 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 0x7fb10c0c5d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb104009e60 tx=0x7fb104009780 comp rx=0 tx=0).stop 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 -- 172.21.15.50:0/986171428 shutdown_connections 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 --2- 172.21.15.50:0/986171428 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c0c5900 0x7fb10c0c5d60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 --2- 172.21.15.50:0/986171428 >> v2:172.21.15.125:3300/0 conn(0x7fb10c1502b0 0x7fb10c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 -- 172.21.15.50:0/986171428 >> 172.21.15.50:0/986171428 conn(0x7fb10c0cd6d0 msgr2=0x7fb10c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 -- 172.21.15.50:0/986171428 shutdown_connections 2023-12-19T14:39:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.964+0000 7fb111745700 1 -- 172.21.15.50:0/986171428 wait complete. 2023-12-19T14:39:54.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 Processor -- start 2023-12-19T14:39:54.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 -- start start 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 0x7fb10c0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb10c0c5900 0x7fb10c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0afad0 con 0x7fb10c1502b0 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0c97a0 con 0x7fb10c0c5900 2023-12-19T14:39:54.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb111745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0c9a80 con 0x7fb10c067c90 2023-12-19T14:39:54.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb10b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58096/0 (socket says 172.21.15.50:58096) 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb10b7fe700 1 -- 172.21.15.50:0/176663545 learned_addr learned my addr 172.21.15.50:0/176663545 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.968+0000 7fb10affd700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 0x7fb10c0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10a7fc700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb10c0c5900 0x7fb10c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 -- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 msgr2=0x7fb10c0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 0x7fb10c0c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 -- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb10c0c5900 msgr2=0x7fb10c0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb10c0c5900 0x7fb10c0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 -- 172.21.15.50:0/176663545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb100009420 con 0x7fb10c1502b0 2023-12-19T14:39:54.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10affd700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 0x7fb10c0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb10b7fe700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0c6d30 tx=0x7fb104019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104012e10 con 0x7fb10c1502b0 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb104027470 con 0x7fb10c1502b0 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104027830 con 0x7fb10c1502b0 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb104009a20 con 0x7fb10c1502b0 2023-12-19T14:39:54.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.972+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c0ca040 con 0x7fb10c1502b0 2023-12-19T14:39:54.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb104024070 con 0x7fb10c1502b0 2023-12-19T14:39:54.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c0afc40 con 0x7fb10c1502b0 2023-12-19T14:39:54.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb0f2ffd700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 0x7fb0f8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:54.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fb10407f490 con 0x7fb10c1502b0 2023-12-19T14:39:54.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb10affd700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 0x7fb0f8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:54.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.976+0000 7fb10affd700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 0x7fb0f8062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc007fd0 tx=0x7fb0fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:54.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:54.980+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb10c0afc40 con 0x7fb10c1502b0 2023-12-19T14:39:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.168+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb10c0afc40 con 0x7fb10c1502b0 2023-12-19T14:39:55.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.168+0000 7fb0f2ffd700 1 -- 172.21.15.50:0/176663545 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7fb10c0afc40 con 0x7fb10c1502b0 2023-12-19T14:39:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:55.174 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 msgr2=0x7fb0f8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 0x7fb0f8062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc007fd0 tx=0x7fb0fc009040 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 msgr2=0x7fb10c14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0c6d30 tx=0x7fb104019000 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 shutdown_connections 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb10c0c5900 0x7fb10c0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0f8060550 0x7fb0f8062a00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb10c1502b0 0x7fb10c14e800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 --2- 172.21.15.50:0/176663545 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb10c067c90 0x7fb10c0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 >> 172.21.15.50:0/176663545 conn(0x7fb10c0cd6d0 msgr2=0x7fb10c0ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 shutdown_connections 2023-12-19T14:39:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.176+0000 7fb111745700 1 -- 172.21.15.50:0/176663545 wait complete. 2023-12-19T14:39:55.201 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:55.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:55.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.364+0000 7f24a2523700 1 Processor -- start 2023-12-19T14:39:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.364+0000 7f24a2523700 1 -- start start 2023-12-19T14:39:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 0x7f249c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c15ae30 con 0x7f249c0adab0 2023-12-19T14:39:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0abf70 con 0x7f249c154780 2023-12-19T14:39:55.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c154b50 con 0x7f249c06baa0 2023-12-19T14:39:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a1d22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f249c0adab0 0x7f249c0adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35106/0 (socket says 172.21.15.50:35106) 2023-12-19T14:39:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a1d22700 1 -- 172.21.15.50:0/1158220320 learned_addr learned my addr 172.21.15.50:0/1158220320 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:55.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a0d20700 1 --2- 172.21.15.50:0/1158220320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 0x7f249c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901231551 0 0) 0x7f249c15ae30 con 0x7f249c0adab0 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2488003570 con 0x7f249c0adab0 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000124762 0 0) 0x7f249c0abf70 con 0x7f249c154780 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c15ae30 con 0x7f249c154780 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a0d20700 1 -- 172.21.15.50:0/1158220320 >> v1:172.21.15.125:6789/0 conn(0x7f249c154780 legacy=0x7f249c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a0d20700 1 -- 172.21.15.50:0/1158220320 >> v1:172.21.15.50:6789/0 conn(0x7f249c0adab0 legacy=0x7f249c0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a0d20700 1 -- 172.21.15.50:0/1158220320 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0ac210 con 0x7f249c06baa0 2023-12-19T14:39:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a0d20700 1 --2- 172.21.15.50:0/1158220320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 0x7f249c15b2c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f248c009910 tx=0x7f248c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9536d604319d147 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c011630 con 0x7f249c06baa0 2023-12-19T14:39:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c011790 con 0x7f249c06baa0 2023-12-19T14:39:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f2493fff700 1 -- 172.21.15.50:0/1158220320 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c011b30 con 0x7f249c06baa0 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 -- 172.21.15.50:0/1158220320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 msgr2=0x7f249c15b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.368+0000 7f24a2523700 1 --2- 172.21.15.50:0/1158220320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 0x7f249c15b2c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f248c009910 tx=0x7f248c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- 172.21.15.50:0/1158220320 shutdown_connections 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 --2- 172.21.15.50:0/1158220320 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c06baa0 0x7f249c15b2c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- 172.21.15.50:0/1158220320 >> 172.21.15.50:0/1158220320 conn(0x7f249c067b50 msgr2=0x7f249c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- 172.21.15.50:0/1158220320 shutdown_connections 2023-12-19T14:39:55.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- 172.21.15.50:0/1158220320 wait complete. 2023-12-19T14:39:55.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 Processor -- start 2023-12-19T14:39:55.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- start start 2023-12-19T14:39:55.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 0x7f249c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 0x7f249c0c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0ae950 con 0x7f249c06baa0 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0af1f0 con 0x7f249c154780 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.372+0000 7f24a2523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c148a60 con 0x7f249c0adab0 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58104/0 (socket says 172.21.15.50:58104) 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 -- 172.21.15.50:0/3243519069 learned_addr learned my addr 172.21.15.50:0/3243519069 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1d22700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 0x7f249c0c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a0d20700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 0x7f249c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 -- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 msgr2=0x7f249c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 0x7f249c0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 -- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 msgr2=0x7f249c0c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 0x7f249c0c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 -- 172.21.15.50:0/3243519069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009420 con 0x7f249c06baa0 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a0d20700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 0x7f249c0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1d22700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 0x7f249c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a1521700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f249800c7c0 tx=0x7f2498007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498010510 con 0x7f249c06baa0 2023-12-19T14:39:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2498010690 con 0x7f249c06baa0 2023-12-19T14:39:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c1681d0 con 0x7f249c06baa0 2023-12-19T14:39:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498010a50 con 0x7f249c06baa0 2023-12-19T14:39:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c0aee80 con 0x7f249c06baa0 2023-12-19T14:39:55.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.376+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c04be40 con 0x7f249c06baa0 2023-12-19T14:39:55.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.380+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f249801b400 con 0x7f249c06baa0 2023-12-19T14:39:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.380+0000 7f24917fa700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 0x7f2480062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.380+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f249807c8f0 con 0x7f249c06baa0 2023-12-19T14:39:55.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.384+0000 7f24a0d20700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 0x7f2480062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.384+0000 7f24a0d20700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 0x7f2480062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f249c06be70 tx=0x7f248c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.388+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f249804cb30 con 0x7f249c06baa0 2023-12-19T14:39:55.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.564+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f249c0ac5b0 con 0x7f249c06baa0 2023-12-19T14:39:55.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.564+0000 7f24917fa700 1 -- 172.21.15.50:0/3243519069 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2498050150 con 0x7f249c06baa0 2023-12-19T14:39:55.569 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:55.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 msgr2=0x7f2480062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 0x7f2480062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f249c06be70 tx=0x7f248c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:55.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 msgr2=0x7f249c148520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f249800c7c0 tx=0x7f2498007e10 comp rx=0 tx=0).stop 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 shutdown_connections 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f249c154780 0x7f249c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060550 0x7f2480062a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249c0adab0 0x7f249c0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 --2- 172.21.15.50:0/3243519069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249c06baa0 0x7f249c148520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 >> 172.21.15.50:0/3243519069 conn(0x7f249c067b50 msgr2=0x7f249c152620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 shutdown_connections 2023-12-19T14:39:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.568+0000 7f24a2523700 1 -- 172.21.15.50:0/3243519069 wait complete. 2023-12-19T14:39:55.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:55.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.756+0000 7f3da5385700 1 Processor -- start 2023-12-19T14:39:55.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.756+0000 7f3da5385700 1 -- start start 2023-12-19T14:39:55.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3da5385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3da0158c60 0x7f3da00afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3da5385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 0x7f3da006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3da5385700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00cb550 con 0x7f3da00b0060 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3da5385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da006cf40 con 0x7f3da0158c60 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3da5385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da00cb270 con 0x7f3da00677f0 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3da0158c60 0x7f3da00afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3da0158c60 0x7f3da00afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58110/0 (socket says 172.21.15.50:58110) 2023-12-19T14:39:55.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 -- 172.21.15.50:0/3048400933 learned_addr learned my addr 172.21.15.50:0/3048400933 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 -- 172.21.15.50:0/3048400933 >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 msgr2=0x7f3da006d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9f7fe700 1 --2- 172.21.15.50:0/3048400933 >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 0x7f3da006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 --2- 172.21.15.50:0/3048400933 >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 0x7f3da006d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 -- 172.21.15.50:0/3048400933 >> v1:172.21.15.125:6789/0 conn(0x7f3da00b0060 legacy=0x7f3da00670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 -- 172.21.15.50:0/3048400933 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da006c8f0 con 0x7f3da0158c60 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9f7fe700 1 --2- 172.21.15.50:0/3048400933 >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 0x7f3da006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9effd700 1 --2- 172.21.15.50:0/3048400933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da0158c60 0x7f3da00afb20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3da0067fd0 tx=0x7f3d90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d48e7c085cfab7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.760+0000 7f3d9dffb700 1 -- 172.21.15.50:0/3048400933 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000b860 con 0x7f3da0158c60 2023-12-19T14:39:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3d9dffb700 1 -- 172.21.15.50:0/3048400933 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9000b9c0 con 0x7f3da0158c60 2023-12-19T14:39:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3d9dffb700 1 -- 172.21.15.50:0/3048400933 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000bd60 con 0x7f3da0158c60 2023-12-19T14:39:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 -- 172.21.15.50:0/3048400933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da0158c60 msgr2=0x7f3da00afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 --2- 172.21.15.50:0/3048400933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da0158c60 0x7f3da00afb20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3da0067fd0 tx=0x7f3d90009250 comp rx=0 tx=0).stop 2023-12-19T14:39:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 -- 172.21.15.50:0/3048400933 shutdown_connections 2023-12-19T14:39:55.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 --2- 172.21.15.50:0/3048400933 >> v2:172.21.15.125:3300/0 conn(0x7f3da00677f0 0x7f3da006d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 --2- 172.21.15.50:0/3048400933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da0158c60 0x7f3da00afb20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 -- 172.21.15.50:0/3048400933 >> 172.21.15.50:0/3048400933 conn(0x7f3da00cfde0 msgr2=0x7f3da00d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 -- 172.21.15.50:0/3048400933 shutdown_connections 2023-12-19T14:39:55.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.764+0000 7f3da5385700 1 -- 172.21.15.50:0/3048400933 wait complete. 2023-12-19T14:39:55.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 Processor -- start 2023-12-19T14:39:55.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 -- start start 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3da00677f0 0x7f3da00c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3da0158c60 0x7f3da015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da015a7f0 con 0x7f3da00b0060 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014b750 con 0x7f3da00677f0 2023-12-19T14:39:55.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3da5385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da01693e0 con 0x7f3da0158c60 2023-12-19T14:39:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58114/0 (socket says 172.21.15.50:58114) 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 -- 172.21.15.50:0/245376470 learned_addr learned my addr 172.21.15.50:0/245376470 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9f7fe700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3da0158c60 0x7f3da015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9effd700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3da00677f0 0x7f3da00c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 -- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3da0158c60 msgr2=0x7f3da015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3da0158c60 0x7f3da015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 -- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3da00677f0 msgr2=0x7f3da00c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3da00677f0 0x7f3da00c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.768+0000 7f3d9e7fc700 1 -- 172.21.15.50:0/245376470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900094f0 con 0x7f3da00b0060 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3d9e7fc700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c5c0 tx=0x7f3d9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94011590 con 0x7f3da00b0060 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d94011710 con 0x7f3da00b0060 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da01696c0 con 0x7f3da00b0060 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94011ad0 con 0x7f3da00b0060 2023-12-19T14:39:55.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da0169c80 con 0x7f3da00b0060 2023-12-19T14:39:55.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.772+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da004be40 con 0x7f3da00b0060 2023-12-19T14:39:55.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.776+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3d9401b3b0 con 0x7f3da00b0060 2023-12-19T14:39:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.776+0000 7f3d86ffd700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 0x7f3d8c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.776+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f3d9407bff0 con 0x7f3da00b0060 2023-12-19T14:39:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.776+0000 7f3d9effd700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 0x7f3d8c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:55.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.780+0000 7f3d9effd700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 0x7f3d8c062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3da0067fd0 tx=0x7f3d90017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.784+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3d9404c180 con 0x7f3da00b0060 2023-12-19T14:39:55.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.952+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3da006cbd0 con 0x7f3da00b0060 2023-12-19T14:39:55.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3d86ffd700 1 -- 172.21.15.50:0/245376470 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d9404f7a0 con 0x7f3da00b0060 2023-12-19T14:39:55.959 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:55.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 msgr2=0x7f3d8c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 0x7f3d8c062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3da0067fd0 tx=0x7f3d90017000 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 msgr2=0x7f3da014b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c5c0 tx=0x7f3d9400c8d0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 shutdown_connections 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3d8c060550 0x7f3d8c062a00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3da0158c60 0x7f3da015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3da00b0060 0x7f3da014b210 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 --2- 172.21.15.50:0/245376470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3da00677f0 0x7f3da00c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 >> 172.21.15.50:0/245376470 conn(0x7f3da00cfde0 msgr2=0x7f3da0157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 shutdown_connections 2023-12-19T14:39:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:55.956+0000 7f3da5385700 1 -- 172.21.15.50:0/245376470 wait complete. 2023-12-19T14:39:55.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:39:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.144+0000 7fa69fb45700 1 Processor -- start 2023-12-19T14:39:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.144+0000 7fa69fb45700 1 -- start start 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.144+0000 7fa69fb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa6980ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.144+0000 7fa69fb45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6980c7450 con 0x7fa69806b1b0 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69fb45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806bea0 con 0x7fa69815d410 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69fb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6980c6e00 con 0x7fa6980cab10 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa69806b1b0 0x7fa69806b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35116/0 (socket says 172.21.15.50:35116) 2023-12-19T14:39:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 -- 172.21.15.50:0/2268765540 learned_addr learned my addr 172.21.15.50:0/2268765540 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69d0e0700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa6980ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561322342 0 0) 0x7fa6980c7450 con 0x7fa69806b1b0 2023-12-19T14:39:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa684003570 con 0x7fa69806b1b0 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719751866 0 0) 0x7fa69806bea0 con 0x7fa69815d410 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6980c7450 con 0x7fa69815d410 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612083936 0 0) 0x7fa684003570 con 0x7fa69806b1b0 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 msgr2=0x7fa6980ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa6980ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 >> v1:172.21.15.125:6789/0 conn(0x7fa69815d410 legacy=0x7fa6980ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6980c70e0 con 0x7fa69806b1b0 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fa6900017f0 con 0x7fa69806b1b0 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 >> v1:172.21.15.50:6789/0 conn(0x7fa69806b1b0 legacy=0x7fa69806b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa684003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa684004180 0x7fa684004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684004d50 0x7fa6840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6980c6e00 con 0x7fa684004d50 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68400bba0 con 0x7fa684004180 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68400bd60 con 0x7fa6980cab10 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684004d50 0x7fa6840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa684003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69d0e0700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa684004180 0x7fa684004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 -- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 msgr2=0x7fa684003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa684003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 -- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa684004180 msgr2=0x7fa684004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa684004180 0x7fa684004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69e0e2700 1 -- 172.21.15.50:0/2268765540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684003570 con 0x7fa684004d50 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.148+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6980cab10 0x7fa684003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69e0e2700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684004d50 0x7fa6840051c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa69000a8d0 tx=0x7fa690002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e76945a9c4b063 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6900070d0 con 0x7fa684004d50 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa690007230 con 0x7fa684004d50 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69c8df700 1 -- 172.21.15.50:0/2268765540 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6900075f0 con 0x7fa684004d50 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 -- 172.21.15.50:0/2268765540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684004d50 msgr2=0x7fa6840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 --2- 172.21.15.50:0/2268765540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa684004d50 0x7fa6840051c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa69000a8d0 tx=0x7fa690002190 comp rx=0 tx=0).stop 2023-12-19T14:39:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69e0e2700 1 -- 172.21.15.50:0/2268765540 reap_dead start 2023-12-19T14:39:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 -- 172.21.15.50:0/2268765540 shutdown_connections 2023-12-19T14:39:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 -- 172.21.15.50:0/2268765540 >> 172.21.15.50:0/2268765540 conn(0x7fa698067bb0 msgr2=0x7fa6980cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 -- 172.21.15.50:0/2268765540 shutdown_connections 2023-12-19T14:39:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.152+0000 7fa69fb45700 1 -- 172.21.15.50:0/2268765540 wait complete. 2023-12-19T14:39:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 Processor -- start 2023-12-19T14:39:56.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 -- start start 2023-12-19T14:39:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 0x7fa698170fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6980c48d0 0x7fa6980c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6980c5280 con 0x7fa6980c9530 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69815db80 con 0x7fa6980c48d0 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69fb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69815d880 con 0x7fa684004d50 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58132/0 (socket says 172.21.15.50:58132) 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 -- 172.21.15.50:0/3323126346 learned_addr learned my addr 172.21.15.50:0/3323126346 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 0x7fa698170fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69e0e2700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6980c48d0 0x7fa6980c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 -- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 msgr2=0x7fa698170fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 0x7fa698170fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 -- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6980c48d0 msgr2=0x7fa6980c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6980c48d0 0x7fa6980c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d0e0700 1 -- 172.21.15.50:0/3323126346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698171820 con 0x7fa6980c9530 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.156+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 0x7fa698170fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa69d0e0700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa688011b10 tx=0x7fa6880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68800f9f0 con 0x7fa6980c9530 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68800fb50 con 0x7fa6980c9530 2023-12-19T14:39:56.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa698171b00 con 0x7fa6980c9530 2023-12-19T14:39:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68800fed0 con 0x7fa6980c9530 2023-12-19T14:39:56.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69814b400 con 0x7fa6980c9530 2023-12-19T14:39:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69815dd30 con 0x7fa6980c9530 2023-12-19T14:39:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.160+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa688014650 con 0x7fa6980c9530 2023-12-19T14:39:56.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.164+0000 7fa68dffb700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 0x7fa67c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.164+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fa688018030 con 0x7fa6980c9530 2023-12-19T14:39:56.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.164+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 0x7fa67c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.164+0000 7fa69d8e1700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 0x7fa67c062a30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa694000f60 tx=0x7fa694009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.168+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa69815dd30 con 0x7fa6980c9530 2023-12-19T14:39:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.344+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa69815dd30 con 0x7fa6980c9530 2023-12-19T14:39:56.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.344+0000 7fa68dffb700 1 -- 172.21.15.50:0/3323126346 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa69815dd30 con 0x7fa6980c9530 2023-12-19T14:39:56.349 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 msgr2=0x7fa67c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 0x7fa67c062a30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa694000f60 tx=0x7fa694009040 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 msgr2=0x7fa69814aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa688011b10 tx=0x7fa6880091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 shutdown_connections 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6980c48d0 0x7fa6980c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa67c060580 0x7fa67c062a30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6980c9530 0x7fa69814aec0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 --2- 172.21.15.50:0/3323126346 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa684004d50 0x7fa698170fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 >> 172.21.15.50:0/3323126346 conn(0x7fa698067bb0 msgr2=0x7fa69815ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:56.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 shutdown_connections 2023-12-19T14:39:56.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.348+0000 7fa69fb45700 1 -- 172.21.15.50:0/3323126346 wait complete. 2023-12-19T14:39:56.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:39:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.544+0000 7fa07aa25700 1 Processor -- start 2023-12-19T14:39:56.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.544+0000 7fa07aa25700 1 -- start start 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa07aa25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 0x7fa074159a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa07aa25700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07406db20 con 0x7fa07406e530 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa07aa25700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07406e9d0 con 0x7fa0741554d0 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa07aa25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740cc310 con 0x7fa07406b770 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa078fc2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa07406e530 0x7fa0740af340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36632/0 (socket says 172.21.15.50:36632) 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa078fc2700 1 -- 172.21.15.50:0/385683713 learned_addr learned my addr 172.21.15.50:0/385683713 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa0737fe700 1 --2- 172.21.15.50:0/385683713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 0x7fa074159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495695494 0 0) 0x7fa07406db20 con 0x7fa07406e530 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c004680 con 0x7fa07406e530 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414040297 0 0) 0x7fa07406e9d0 con 0x7fa0741554d0 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07406db20 con 0x7fa0741554d0 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa0737fe700 1 -- 172.21.15.50:0/385683713 >> v1:172.21.15.125:6789/0 conn(0x7fa0741554d0 legacy=0x7fa07406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa0737fe700 1 -- 172.21.15.50:0/385683713 >> v1:172.21.15.50:6789/0 conn(0x7fa07406e530 legacy=0x7fa0740af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.548+0000 7fa0737fe700 1 -- 172.21.15.50:0/385683713 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07406dfb0 con 0x7fa07406b770 2023-12-19T14:39:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa0737fe700 1 --2- 172.21.15.50:0/385683713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 0x7fa074159a30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa068009910 tx=0x7fa0680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a32ac2480bed9d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068011630 con 0x7fa07406b770 2023-12-19T14:39:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa068011790 con 0x7fa07406b770 2023-12-19T14:39:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa072ffd700 1 -- 172.21.15.50:0/385683713 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068011b30 con 0x7fa07406b770 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 -- 172.21.15.50:0/385683713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 msgr2=0x7fa074159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 --2- 172.21.15.50:0/385683713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 0x7fa074159a30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa068009910 tx=0x7fa0680091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 -- 172.21.15.50:0/385683713 shutdown_connections 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 --2- 172.21.15.50:0/385683713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406b770 0x7fa074159a30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 -- 172.21.15.50:0/385683713 >> 172.21.15.50:0/385683713 conn(0x7fa0740cc580 msgr2=0x7fa0740cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 -- 172.21.15.50:0/385683713 shutdown_connections 2023-12-19T14:39:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.552+0000 7fa07aa25700 1 -- 172.21.15.50:0/385683713 wait complete. 2023-12-19T14:39:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 Processor -- start 2023-12-19T14:39:56.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 -- start start 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406e530 0x7fa074143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 0x7fa0741532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0741569e0 con 0x7fa07406b770 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa074156cb0 con 0x7fa0741554d0 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa07aa25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa074156f90 con 0x7fa07406e530 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa073fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa073fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34328/0 (socket says 172.21.15.50:34328) 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa073fff700 1 -- 172.21.15.50:0/3159241376 learned_addr learned my addr 172.21.15.50:0/3159241376 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa0737fe700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406e530 0x7fa074143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.556+0000 7fa078fc2700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 0x7fa0741532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 -- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406e530 msgr2=0x7fa074143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406e530 0x7fa074143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 -- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 msgr2=0x7fa0741532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 0x7fa0741532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 -- 172.21.15.50:0/3159241376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009420 con 0x7fa07406b770 2023-12-19T14:39:56.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa078fc2700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 0x7fa0741532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa073fff700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c790 tx=0x7fa064007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064010510 con 0x7fa07406b770 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa064010670 con 0x7fa07406b770 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa074157270 con 0x7fa07406b770 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064010a30 con 0x7fa07406b770 2023-12-19T14:39:56.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa074157830 con 0x7fa07406b770 2023-12-19T14:39:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.560+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa074157a50 con 0x7fa07406b770 2023-12-19T14:39:56.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.564+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa06401b3b0 con 0x7fa07406b770 2023-12-19T14:39:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.564+0000 7fa05bfff700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.564+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fa06407c420 con 0x7fa07406b770 2023-12-19T14:39:56.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.564+0000 7fa0737fe700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.564+0000 7fa0737fe700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 0x7fa060062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa068009860 tx=0x7fa068017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.572+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa06404c650 con 0x7fa07406b770 2023-12-19T14:39:56.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.748+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa074071120 con 0x7fa07406b770 2023-12-19T14:39:56.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.748+0000 7fa05bfff700 1 -- 172.21.15.50:0/3159241376 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa074071120 con 0x7fa07406b770 2023-12-19T14:39:56.751 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 msgr2=0x7fa060062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 0x7fa060062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa068009860 tx=0x7fa068017000 comp rx=0 tx=0).stop 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 msgr2=0x7fa0741564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c790 tx=0x7fa064007e10 comp rx=0 tx=0).stop 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 shutdown_connections 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0741554d0 0x7fa0741532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa07406e530 0x7fa074143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 --2- 172.21.15.50:0/3159241376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa07406b770 0x7fa0741564a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 >> 172.21.15.50:0/3159241376 conn(0x7fa0740cc580 msgr2=0x7fa0740ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:56.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 shutdown_connections 2023-12-19T14:39:56.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.752+0000 7fa07aa25700 1 -- 172.21.15.50:0/3159241376 wait complete. 2023-12-19T14:39:56.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-12-19T14:39:56.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.940+0000 7f048359e700 1 Processor -- start 2023-12-19T14:39:56.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048359e700 1 -- start start 2023-12-19T14:39:56.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0484155960 0x7f048415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048406d990 con 0x7f04840c7f00 2023-12-19T14:39:56.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048415a570 con 0x7f048406de20 2023-12-19T14:39:56.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048359e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f048406cd80 con 0x7f0484155960 2023-12-19T14:39:56.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f0482d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f04840c7f00 0x7f04840aca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36636/0 (socket says 172.21.15.50:36636) 2023-12-19T14:39:56.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f0482d9d700 1 -- 172.21.15.50:0/1638476950 learned_addr learned my addr 172.21.15.50:0/1638476950 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:56.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048259c700 1 --2- 172.21.15.50:0/1638476950 >> v2:172.21.15.50:3300/0 conn(0x7f0484155960 0x7f048415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243904953 0 0) 0x7f048406d990 con 0x7f04840c7f00 2023-12-19T14:39:56.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.944+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003570 con 0x7f04840c7f00 2023-12-19T14:39:56.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748350155 0 0) 0x7f048415a570 con 0x7f048406de20 2023-12-19T14:39:56.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048406d990 con 0x7f048406de20 2023-12-19T14:39:56.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048259c700 1 -- 172.21.15.50:0/1638476950 >> v1:172.21.15.125:6789/0 conn(0x7f048406de20 legacy=0x7f04840c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048259c700 1 -- 172.21.15.50:0/1638476950 >> v1:172.21.15.50:6789/0 conn(0x7f04840c7f00 legacy=0x7f04840aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048259c700 1 -- 172.21.15.50:0/1638476950 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048406d060 con 0x7f0484155960 2023-12-19T14:39:56.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048259c700 1 --2- 172.21.15.50:0/1638476950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f048415a030 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f04740098e0 tx=0x7f04740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e909d1db633d39e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474011630 con 0x7f0484155960 2023-12-19T14:39:56.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0474011790 con 0x7f0484155960 2023-12-19T14:39:56.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048159a700 1 -- 172.21.15.50:0/1638476950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474011b30 con 0x7f0484155960 2023-12-19T14:39:56.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 -- 172.21.15.50:0/1638476950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 msgr2=0x7f048415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 --2- 172.21.15.50:0/1638476950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f048415a030 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f04740098e0 tx=0x7f04740091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 -- 172.21.15.50:0/1638476950 shutdown_connections 2023-12-19T14:39:56.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 --2- 172.21.15.50:0/1638476950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f048415a030 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 -- 172.21.15.50:0/1638476950 >> 172.21.15.50:0/1638476950 conn(0x7f0484067bb0 msgr2=0x7f04840cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:56.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 -- 172.21.15.50:0/1638476950 shutdown_connections 2023-12-19T14:39:56.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.948+0000 7f048359e700 1 -- 172.21.15.50:0/1638476950 wait complete. 2023-12-19T14:39:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 Processor -- start 2023-12-19T14:39:56.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 -- start start 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048406de20 0x7f0484154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04840c7f00 0x7f048414aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04840af9e0 con 0x7f0484155960 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048414b750 con 0x7f048406de20 2023-12-19T14:39:56.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f048359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048414ba30 con 0x7f04840c7f00 2023-12-19T14:39:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.952+0000 7f0482d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34346/0 (socket says 172.21.15.50:34346) 2023-12-19T14:39:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 -- 172.21.15.50:0/387091273 learned_addr learned my addr 172.21.15.50:0/387091273 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f048259c700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048406de20 0x7f0484154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0481d9b700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04840c7f00 0x7f048414aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 -- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04840c7f00 msgr2=0x7f048414aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04840c7f00 0x7f048414aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 -- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048406de20 msgr2=0x7f0484154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048406de20 0x7f0484154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 -- 172.21.15.50:0/387091273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009420 con 0x7f0484155960 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f0482d9d700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f047c00cc60 tx=0x7f047c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c00a8b0 con 0x7f0484155960 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c00aa30 con 0x7f0484155960 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c00adf0 con 0x7f0484155960 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048414bd10 con 0x7f0484155960 2023-12-19T14:39:56.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.956+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048414b420 con 0x7f0484155960 2023-12-19T14:39:56.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.960+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04840afb90 con 0x7f0484155960 2023-12-19T14:39:56.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.960+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f047c007300 con 0x7f0484155960 2023-12-19T14:39:56.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.960+0000 7f04727fc700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 0x7f0464062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.960+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f047c07cf10 con 0x7f0484155960 2023-12-19T14:39:56.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.960+0000 7f048259c700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 0x7f0464062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:56.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.964+0000 7f048259c700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 0x7f0464062a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04740093f0 tx=0x7f0474017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:56.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:56.968+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f04840afb90 con 0x7f0484155960 2023-12-19T14:39:57.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.200+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f048404be40 con 0x7f0484155960 2023-12-19T14:39:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.620+0000 7f04727fc700 1 -- 172.21.15.50:0/387091273 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f047c050920 con 0x7f0484155960 2023-12-19T14:39:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 msgr2=0x7f0464062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 0x7f0464062a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04740093f0 tx=0x7f0474017000 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 msgr2=0x7f04841498a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f047c00cc60 tx=0x7f047c001ae0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 shutdown_connections 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0464060580 0x7f0464062a30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0484155960 0x7f04841498a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04840c7f00 0x7f048414aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 --2- 172.21.15.50:0/387091273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f048406de20 0x7f0484154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 >> 172.21.15.50:0/387091273 conn(0x7f0484067bb0 msgr2=0x7f04840c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 shutdown_connections 2023-12-19T14:39:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.624+0000 7f048359e700 1 -- 172.21.15.50:0/387091273 wait complete. 2023-12-19T14:39:57.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:39:57.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.812+0000 7f9b80963700 1 Processor -- start 2023-12-19T14:39:57.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.812+0000 7f9b80963700 1 -- start start 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b80963700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b80963700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9b7c0caa10 0x7f9b7c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b80963700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c0cbbb0 con 0x7f9b7c0cc040 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b80963700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b7c15d960 con 0x7f9b7c0caa10 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b80963700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c06dc90 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7bfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9b7c0cc040 0x7f9b7c158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:37284/0 (socket says 172.21.15.50:37284) 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7bfff700 1 -- 172.21.15.50:0/2911929064 learned_addr learned my addr 172.21.15.50:0/2911929064 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b73fff700 1 --2- 172.21.15.50:0/2911929064 >> v2:172.21.15.50:3300/0 conn(0x7f9b7c0caa10 0x7f9b7c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/2911929064 >> v2:172.21.15.50:3300/0 conn(0x7f9b7c0caa10 msgr2=0x7f9b7c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 --2- 172.21.15.50:0/2911929064 >> v2:172.21.15.50:3300/0 conn(0x7f9b7c0caa10 0x7f9b7c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/2911929064 >> v1:172.21.15.125:6789/0 conn(0x7f9b7c0cc040 legacy=0x7f9b7c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/2911929064 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c0cafa0 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7affd700 1 -- 172.21.15.50:0/2911929064 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282961658 0 0) 0x7f9b7c0cbbb0 con 0x7f9b7c0cc040 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.816+0000 7f9b7b7fe700 1 --2- 172.21.15.50:0/2911929064 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0ce6a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0098e0 tx=0x7f9b6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df893a13566639eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b7affd700 1 -- 172.21.15.50:0/2911929064 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00b640 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b7affd700 1 -- 172.21.15.50:0/2911929064 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c00b7a0 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b7affd700 1 -- 172.21.15.50:0/2911929064 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00bb40 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- 172.21.15.50:0/2911929064 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 msgr2=0x7f9b7c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 --2- 172.21.15.50:0/2911929064 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0ce6a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0098e0 tx=0x7f9b6c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- 172.21.15.50:0/2911929064 shutdown_connections 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 --2- 172.21.15.50:0/2911929064 >> v2:172.21.15.50:3300/0 conn(0x7f9b7c0caa10 0x7f9b7c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 --2- 172.21.15.50:0/2911929064 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0ce6a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- 172.21.15.50:0/2911929064 >> 172.21.15.50:0/2911929064 conn(0x7f9b7c067bb0 msgr2=0x7f9b7c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- 172.21.15.50:0/2911929064 shutdown_connections 2023-12-19T14:39:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- 172.21.15.50:0/2911929064 wait complete. 2023-12-19T14:39:57.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 Processor -- start 2023-12-19T14:39:57.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.820+0000 7f9b80963700 1 -- start start 2023-12-19T14:39:57.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b7c0cc040 0x7f9b7c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c154be0 con 0x7f9b7c0caa10 2023-12-19T14:39:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0c7750 con 0x7f9b7c0cc040 2023-12-19T14:39:57.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0c7a30 con 0x7f9b7c0ce2d0 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34354/0 (socket says 172.21.15.50:34354) 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/73716716 learned_addr learned my addr 172.21.15.50:0/73716716 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b73fff700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b7c0cc040 0x7f9b7c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7bfff700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 msgr2=0x7f9b7c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b7c0cc040 msgr2=0x7f9b7c0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b7c0cc040 0x7f9b7c0c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 -- 172.21.15.50:0/73716716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009420 con 0x7f9b7c0caa10 2023-12-19T14:39:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b7b7fe700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0093f0 tx=0x7f9b6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00bb20 con 0x7f9b7c0caa10 2023-12-19T14:39:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c011420 con 0x7f9b7c0caa10 2023-12-19T14:39:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c0c7d10 con 0x7f9b7c0caa10 2023-12-19T14:39:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0117e0 con 0x7f9b7c0caa10 2023-12-19T14:39:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.824+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c0b3780 con 0x7f9b7c0caa10 2023-12-19T14:39:57.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.828+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c04be40 con 0x7f9b7c0caa10 2023-12-19T14:39:57.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.828+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9b6c011bd0 con 0x7f9b7c0caa10 2023-12-19T14:39:57.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.828+0000 7f9b737fe700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 0x7f9b5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.828+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f9b6c07e160 con 0x7f9b7c0caa10 2023-12-19T14:39:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.832+0000 7f9b73fff700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 0x7f9b5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:57.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.832+0000 7f9b73fff700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 0x7f9b5c062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0b5990 tx=0x7f9b64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:57.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:57.836+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9b6c04e370 con 0x7f9b7c0caa10 2023-12-19T14:39:58.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.004+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9b7c0cb280 con 0x7f9b7c0caa10 2023-12-19T14:39:58.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.012+0000 7f9b737fe700 1 -- 172.21.15.50:0/73716716 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9b6c051990 con 0x7f9b7c0caa10 2023-12-19T14:39:58.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 msgr2=0x7f9b5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 0x7f9b5c062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0b5990 tx=0x7f9b64008040 comp rx=0 tx=0).stop 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 msgr2=0x7f9b7c14f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0093f0 tx=0x7f9b6c017000 comp rx=0 tx=0).stop 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 shutdown_connections 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b7c0cc040 0x7f9b7c0c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9b5c060550 0x7f9b5c062a00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9b7c0ce2d0 0x7f9b7c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 --2- 172.21.15.50:0/73716716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b7c0caa10 0x7f9b7c14f900 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 >> 172.21.15.50:0/73716716 conn(0x7f9b7c067bb0 msgr2=0x7f9b7c155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 shutdown_connections 2023-12-19T14:39:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.016+0000 7f9b80963700 1 -- 172.21.15.50:0/73716716 wait complete. 2023-12-19T14:39:58.041 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:39:58.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:39:58.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:39:58.126 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:39:58.126 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:39:58.126 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:39:58.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:39:58.163 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T14:39:58.163 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:39:58.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:58.164 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:39:58.225 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2023-12-19T14:39:58.225 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:39:58.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-12-19T14:39:58.272 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T14:39:58.272 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:39:58.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:58.272 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:39:58.345 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:39:58.345 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:39:58.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-12-19T14:39:58.383 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T14:39:58.383 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:39:58.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:58.383 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:39:58.447 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi125.front.sepia.ceph.com... 2023-12-19T14:39:58.448 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:39:58.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-12-19T14:39:58.482 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T14:39:58.482 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:39:58.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:39:58.483 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:39:58.546 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:39:58.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:39:58.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.708+0000 7f7988af0700 1 Processor -- start 2023-12-19T14:39:58.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.708+0000 7f7988af0700 1 -- start start 2023-12-19T14:39:58.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7988af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7988af0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 0x7f79840ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7988af0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798406e150 con 0x7f7984140710 2023-12-19T14:39:58.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7988af0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7984068ab0 con 0x7f7984068630 2023-12-19T14:39:58.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7988af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798406d540 con 0x7f7984140ae0 2023-12-19T14:39:58.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f798259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7984140710 0x7f79840c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36640/0 (socket says 172.21.15.50:36640) 2023-12-19T14:39:58.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f798259c700 1 -- 172.21.15.50:0/3320680759 learned_addr learned my addr 172.21.15.50:0/3320680759 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:58.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3320680759 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7982d9d700 1 --2- 172.21.15.50:0/3320680759 >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 0x7f79840ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693247353 0 0) 0x7f798406e150 con 0x7f7984140710 2023-12-19T14:39:58.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f796c003570 con 0x7f7984140710 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 -- 172.21.15.50:0/3320680759 >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 msgr2=0x7f79840ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2296350799 0 0) 0x7f796c003570 con 0x7f7984140710 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3320680759 >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 0x7f79840ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 -- 172.21.15.50:0/3320680759 >> v1:172.21.15.50:6789/0 conn(0x7f7984140710 legacy=0x7f79840c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 -- 172.21.15.50:0/3320680759 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798406d820 con 0x7f7984140ae0 2023-12-19T14:39:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7982d9d700 1 --2- 172.21.15.50:0/3320680759 >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 0x7f79840ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:39:58.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.712+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3320680759 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840680f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79780098e0 tx=0x7f79780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52580ccf5c033d8b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:58.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974001470 con 0x7f7984140ae0 2023-12-19T14:39:58.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978011420 con 0x7f7984140ae0 2023-12-19T14:39:58.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f798159a700 1 -- 172.21.15.50:0/3320680759 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780117c0 con 0x7f7984140ae0 2023-12-19T14:39:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 -- 172.21.15.50:0/3320680759 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 msgr2=0x7f79840680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 --2- 172.21.15.50:0/3320680759 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840680f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79780098e0 tx=0x7f79780091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 -- 172.21.15.50:0/3320680759 shutdown_connections 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 --2- 172.21.15.50:0/3320680759 >> v2:172.21.15.125:3300/0 conn(0x7f7984068630 0x7f79840ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 --2- 172.21.15.50:0/3320680759 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840680f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 -- 172.21.15.50:0/3320680759 >> 172.21.15.50:0/3320680759 conn(0x7f79840c8850 msgr2=0x7f79840c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 -- 172.21.15.50:0/3320680759 shutdown_connections 2023-12-19T14:39:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.716+0000 7f7988af0700 1 -- 172.21.15.50:0/3320680759 wait complete. 2023-12-19T14:39:58.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 Processor -- start 2023-12-19T14:39:58.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 -- start start 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7984140710 0x7f798413bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798413c150 con 0x7f7984068630 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840c2f30 con 0x7f7984140710 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7988af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840c3210 con 0x7f7984140ae0 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f798259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f798259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34356/0 (socket says 172.21.15.50:34356) 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f798259c700 1 -- 172.21.15.50:0/3792885422 learned_addr learned my addr 172.21.15.50:0/3792885422 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.720+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7984140710 0x7f798413bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f7982d9d700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 -- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 msgr2=0x7f79840af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 -- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7984140710 msgr2=0x7f798413bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7984140710 0x7f798413bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 -- 172.21.15.50:0/3792885422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c009a20 con 0x7f7984068630 2023-12-19T14:39:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f798259c700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f797400cb90 tx=0x7f7974001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974001d40 con 0x7f7984068630 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974007360 con 0x7f7984068630 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978009420 con 0x7f7984068630 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974004830 con 0x7f7984068630 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79840c37d0 con 0x7f7984068630 2023-12-19T14:39:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.724+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7984071120 con 0x7f7984068630 2023-12-19T14:39:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.728+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7974004ba0 con 0x7f7984068630 2023-12-19T14:39:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.728+0000 7f79727fc700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 0x7f7964062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.728+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f797407c340 con 0x7f7984068630 2023-12-19T14:39:58.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.728+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 0x7f7964062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:58.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.732+0000 7f7981d9b700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 0x7f7964062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f79780093f0 tx=0x7f7978016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:58.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.736+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f797404c610 con 0x7f7984068630 2023-12-19T14:39:58.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.940+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79840642a0 con 0x7f7984068630 2023-12-19T14:39:58.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.940+0000 7f79727fc700 1 -- 172.21.15.50:0/3792885422 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f797404fc30 con 0x7f7984068630 2023-12-19T14:39:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:58.944 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:39:58.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 msgr2=0x7f7964062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 0x7f7964062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f79780093f0 tx=0x7f7978016000 comp rx=0 tx=0).stop 2023-12-19T14:39:58.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 msgr2=0x7f7984157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:58.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f797400cb90 tx=0x7f7974001ab0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 shutdown_connections 2023-12-19T14:39:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7984140710 0x7f798413bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7964060550 0x7f7964062a00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7984140ae0 0x7f79840af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 --2- 172.21.15.50:0/3792885422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7984068630 0x7f7984157b10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:58.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 >> 172.21.15.50:0/3792885422 conn(0x7f79840c8850 msgr2=0x7f79840aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 shutdown_connections 2023-12-19T14:39:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:58.944+0000 7f7988af0700 1 -- 172.21.15.50:0/3792885422 wait complete. 2023-12-19T14:39:58.949 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:39:58.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:39:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 Processor -- start 2023-12-19T14:39:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 -- start start 2023-12-19T14:39:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f80c6cd0 0x7f40f806b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f806b470 con 0x7f40f815cc20 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f806e110 con 0x7f40f80c6900 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40ff563700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f806be40 con 0x7f40f80c6cd0 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40fdd60700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f80c6cd0 0x7f40f806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40fdd60700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f80c6cd0 0x7f40f806b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34362/0 (socket says 172.21.15.50:34362) 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40fdd60700 1 -- 172.21.15.50:0/2408864402 learned_addr learned my addr 172.21.15.50:0/2408864402 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:59.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.120+0000 7f40fed62700 1 --1- 172.21.15.50:0/2408864402 >> v1:172.21.15.50:6789/0 conn(0x7f40f815cc20 0x7f40f815d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36652/0 (socket says 172.21.15.50:36652) 2023-12-19T14:39:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fdd60700 1 -- 172.21.15.50:0/2408864402 >> v1:172.21.15.125:6789/0 conn(0x7f40f80c6900 legacy=0x7f40f806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fdd60700 1 -- 172.21.15.50:0/2408864402 >> v1:172.21.15.50:6789/0 conn(0x7f40f815cc20 legacy=0x7f40f815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fdd60700 1 -- 172.21.15.50:0/2408864402 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8161d10 con 0x7f40f80c6cd0 2023-12-19T14:39:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fdd60700 1 --2- 172.21.15.50:0/2408864402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f806b900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f40e80099b0 tx=0x7f40e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195e59dd1d42ae34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fd55f700 1 -- 172.21.15.50:0/2408864402 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e800b860 con 0x7f40f80c6cd0 2023-12-19T14:39:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fd55f700 1 -- 172.21.15.50:0/2408864402 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e800b9c0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40fd55f700 1 -- 172.21.15.50:0/2408864402 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e800bd60 con 0x7f40f80c6cd0 2023-12-19T14:39:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 -- 172.21.15.50:0/2408864402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 msgr2=0x7f40f806b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 --2- 172.21.15.50:0/2408864402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f806b900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f40e80099b0 tx=0x7f40e8009250 comp rx=0 tx=0).stop 2023-12-19T14:39:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 -- 172.21.15.50:0/2408864402 shutdown_connections 2023-12-19T14:39:59.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 --2- 172.21.15.50:0/2408864402 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f806b900 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 -- 172.21.15.50:0/2408864402 >> 172.21.15.50:0/2408864402 conn(0x7f40f8067bb0 msgr2=0x7f40f80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 -- 172.21.15.50:0/2408864402 shutdown_connections 2023-12-19T14:39:59.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.124+0000 7f40ff563700 1 -- 172.21.15.50:0/2408864402 wait complete. 2023-12-19T14:39:59.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 Processor -- start 2023-12-19T14:39:59.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 -- start start 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 0x7f40f816da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 0x7f40f80b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f816e9f0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8159a50 con 0x7f40f815cc20 2023-12-19T14:39:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40ff563700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f816e150 con 0x7f40f80c6900 2023-12-19T14:39:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34376/0 (socket says 172.21.15.50:34376) 2023-12-19T14:39:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 -- 172.21.15.50:0/709829413 learned_addr learned my addr 172.21.15.50:0/709829413 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fed62700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 0x7f40f80b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fe561700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 0x7f40f816da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 -- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 msgr2=0x7f40f816da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 0x7f40f816da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 -- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 msgr2=0x7f40f80b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 0x7f40f80b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 -- 172.21.15.50:0/709829413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e80094f0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fed62700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 0x7f40f80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fe561700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 0x7f40f816da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:39:59.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.128+0000 7f40fdd60700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40f8064040 tx=0x7f40e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8011ea0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e801a630 con 0x7f40f80c6cd0 2023-12-19T14:39:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f816e430 con 0x7f40f80c6cd0 2023-12-19T14:39:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e801a9b0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f816ed20 con 0x7f40f80c6cd0 2023-12-19T14:39:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f816eb60 con 0x7f40f80c6cd0 2023-12-19T14:39:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.132+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f40e802d440 con 0x7f40f80c6cd0 2023-12-19T14:39:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.136+0000 7f40ee7fc700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 0x7f40dc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.136+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f40e807e7f0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.136+0000 7f40fe561700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 0x7f40dc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.136+0000 7f40fe561700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 0x7f40dc062a30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f40f80c5800 tx=0x7f40f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.140+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f40e804e980 con 0x7f40f80c6cd0 2023-12-19T14:39:59.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.308+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f804be40 con 0x7f40f80c6cd0 2023-12-19T14:39:59.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ee7fc700 1 -- 172.21.15.50:0/709829413 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7f40e8051fa0 con 0x7f40f80c6cd0 2023-12-19T14:39:59.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:39:59.315 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 msgr2=0x7f40dc062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 0x7f40dc062a30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f40f80c5800 tx=0x7f40f4009040 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 msgr2=0x7f40f80c0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40f8064040 tx=0x7f40e8017000 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 shutdown_connections 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f815cc20 0x7f40f80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40dc060580 0x7f40dc062a30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f80c6cd0 0x7f40f80c0f80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 --2- 172.21.15.50:0/709829413 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f80c6900 0x7f40f816da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 >> 172.21.15.50:0/709829413 conn(0x7f40f8067bb0 msgr2=0x7f40f8160d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:59.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.312+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 shutdown_connections 2023-12-19T14:39:59.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.316+0000 7f40ff563700 1 -- 172.21.15.50:0/709829413 wait complete. 2023-12-19T14:39:59.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:39:04.445520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:39:05.676716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T14:39:09.568551+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:39:59.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:39:59.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.512+0000 7fe305af0700 1 Processor -- start 2023-12-19T14:39:59.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.512+0000 7fe305af0700 1 -- start start 2023-12-19T14:39:59.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe305af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 0x7fe30006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe305af0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe3000cecd0 0x7fe30006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe305af0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000cbdf0 con 0x7fe300158e20 2023-12-19T14:39:59.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe305af0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3000cc0d0 con 0x7fe3000cecd0 2023-12-19T14:39:59.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe305af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000cc3b0 con 0x7fe3000cb970 2023-12-19T14:39:59.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2ff7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe300158e20 0x7fe3000ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36666/0 (socket says 172.21.15.50:36666) 2023-12-19T14:39:59.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2ff7fe700 1 -- 172.21.15.50:0/2906861487 learned_addr learned my addr 172.21.15.50:0/2906861487 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:39:59.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2906861487 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 0x7fe30006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2fffff700 1 --2- 172.21.15.50:0/2906861487 >> v2:172.21.15.125:3300/0 conn(0x7fe3000cecd0 0x7fe30006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2fe7fc700 1 -- 172.21.15.50:0/2906861487 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157342631 0 0) 0x7fe3000cbdf0 con 0x7fe300158e20 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2fe7fc700 1 -- 172.21.15.50:0/2906861487 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e8003570 con 0x7fe300158e20 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2feffd700 1 -- 172.21.15.50:0/2906861487 >> v2:172.21.15.125:3300/0 conn(0x7fe3000cecd0 msgr2=0x7fe30006b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2906861487 >> v2:172.21.15.125:3300/0 conn(0x7fe3000cecd0 0x7fe30006b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2feffd700 1 -- 172.21.15.50:0/2906861487 >> v1:172.21.15.50:6789/0 conn(0x7fe300158e20 legacy=0x7fe3000ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.516+0000 7fe2feffd700 1 -- 172.21.15.50:0/2906861487 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000cc690 con 0x7fe3000cb970 2023-12-19T14:39:59.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2906861487 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 0x7fe30006acc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40098e0 tx=0x7fe2f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1325fccbf957cc43 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe2fe7fc700 1 -- 172.21.15.50:0/2906861487 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4011630 con 0x7fe3000cb970 2023-12-19T14:39:59.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe2fe7fc700 1 -- 172.21.15.50:0/2906861487 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4011790 con 0x7fe3000cb970 2023-12-19T14:39:59.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe2fe7fc700 1 -- 172.21.15.50:0/2906861487 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4011b30 con 0x7fe3000cb970 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 -- 172.21.15.50:0/2906861487 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 msgr2=0x7fe30006acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 --2- 172.21.15.50:0/2906861487 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 0x7fe30006acc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40098e0 tx=0x7fe2f40091a0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 -- 172.21.15.50:0/2906861487 shutdown_connections 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 --2- 172.21.15.50:0/2906861487 >> v2:172.21.15.125:3300/0 conn(0x7fe3000cecd0 0x7fe30006b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 --2- 172.21.15.50:0/2906861487 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3000cb970 0x7fe30006acc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 -- 172.21.15.50:0/2906861487 >> 172.21.15.50:0/2906861487 conn(0x7fe300067bb0 msgr2=0x7fe3000d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 -- 172.21.15.50:0/2906861487 shutdown_connections 2023-12-19T14:39:59.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.520+0000 7fe305af0700 1 -- 172.21.15.50:0/2906861487 wait complete. 2023-12-19T14:39:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 Processor -- start 2023-12-19T14:39:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 -- start start 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe300158e20 0x7fe30014dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe30015c5d0 0x7fe30015cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300159bc0 con 0x7fe3000cecd0 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30014e5d0 con 0x7fe300158e20 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30014d160 con 0x7fe30015c5d0 2023-12-19T14:39:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34384/0 (socket says 172.21.15.50:34384) 2023-12-19T14:39:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 -- 172.21.15.50:0/2516273455 learned_addr learned my addr 172.21.15.50:0/2516273455 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe300158e20 0x7fe30014dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2fffff700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe30015c5d0 0x7fe30015cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 -- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe30015c5d0 msgr2=0x7fe30015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe30015c5d0 0x7fe30015cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 -- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe300158e20 msgr2=0x7fe30014dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe300158e20 0x7fe30014dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 -- 172.21.15.50:0/2516273455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4009420 con 0x7fe3000cecd0 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2ff7fe700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c7c0 tx=0x7fe2f0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0005cd0 con 0x7fe3000cecd0 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f0005e30 con 0x7fe3000cecd0 2023-12-19T14:39:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.524+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300159290 con 0x7fe3000cecd0 2023-12-19T14:39:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.528+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0007320 con 0x7fe3000cecd0 2023-12-19T14:39:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.528+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe300159850 con 0x7fe3000cecd0 2023-12-19T14:39:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.528+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300071120 con 0x7fe3000cecd0 2023-12-19T14:39:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.528+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe2f001b3b0 con 0x7fe3000cecd0 2023-12-19T14:39:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.532+0000 7fe2e77fe700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 0x7fe2ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.532+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fe2f007c2c0 con 0x7fe3000cecd0 2023-12-19T14:39:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.532+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 0x7fe2ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.532+0000 7fe2feffd700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 0x7fe2ec062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40091e0 tx=0x7fe2f4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.536+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2f004c450 con 0x7fe3000cecd0 2023-12-19T14:39:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.720+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe30006a7a0 con 0x7fe3000cecd0 2023-12-19T14:39:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.720+0000 7fe2e77fe700 1 -- 172.21.15.50:0/2516273455 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe2f0017020 con 0x7fe3000cecd0 2023-12-19T14:39:59.726 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.724+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 msgr2=0x7fe2ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.724+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 0x7fe2ec062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40091e0 tx=0x7fe2f4017000 comp rx=0 tx=0).stop 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.724+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 msgr2=0x7fe30014cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.724+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c7c0 tx=0x7fe2f0007e00 comp rx=0 tx=0).stop 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 shutdown_connections 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe300158e20 0x7fe30014dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2ec060550 0x7fe2ec062a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe30015c5d0 0x7fe30015cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 --2- 172.21.15.50:0/2516273455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3000cecd0 0x7fe30014cc20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 >> 172.21.15.50:0/2516273455 conn(0x7fe300067bb0 msgr2=0x7fe3000cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 shutdown_connections 2023-12-19T14:39:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.728+0000 7fe305af0700 1 -- 172.21.15.50:0/2516273455 wait complete. 2023-12-19T14:39:59.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:39:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.912+0000 7efd4d68e700 1 Processor -- start 2023-12-19T14:39:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.912+0000 7efd4d68e700 1 -- start start 2023-12-19T14:39:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4d68e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd48158c90 0x7efd480caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4d68e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd48159770 0x7efd4806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4d68e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4806c850 con 0x7efd4806aeb0 2023-12-19T14:39:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4d68e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4806b470 con 0x7efd48159770 2023-12-19T14:39:59.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4d68e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4806b750 con 0x7efd48158c90 2023-12-19T14:39:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd48159770 0x7efd4806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efd48159770 0x7efd4806cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34388/0 (socket says 172.21.15.50:34388) 2023-12-19T14:39:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 -- 172.21.15.50:0/1282210807 learned_addr learned my addr 172.21.15.50:0/1282210807 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:59.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd47fff700 1 --2- 172.21.15.50:0/1282210807 >> v2:172.21.15.125:3300/0 conn(0x7efd48158c90 0x7efd480caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 -- 172.21.15.50:0/1282210807 >> v2:172.21.15.125:3300/0 conn(0x7efd48158c90 msgr2=0x7efd480caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 --2- 172.21.15.50:0/1282210807 >> v2:172.21.15.125:3300/0 conn(0x7efd48158c90 0x7efd480caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 -- 172.21.15.50:0/1282210807 >> v1:172.21.15.125:6789/0 conn(0x7efd4806aeb0 legacy=0x7efd48159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 -- 172.21.15.50:0/1282210807 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4806ba30 con 0x7efd48159770 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.916+0000 7efd4ce8d700 1 --2- 172.21.15.50:0/1282210807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd48159770 0x7efd4806cce0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efd480698f0 tx=0x7efd40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddfc3de43ecfe4e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd46ffd700 1 -- 172.21.15.50:0/1282210807 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000e070 con 0x7efd48159770 2023-12-19T14:39:59.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd46ffd700 1 -- 172.21.15.50:0/1282210807 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4000ba10 con 0x7efd48159770 2023-12-19T14:39:59.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd46ffd700 1 -- 172.21.15.50:0/1282210807 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400092e0 con 0x7efd48159770 2023-12-19T14:39:59.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- 172.21.15.50:0/1282210807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd48159770 msgr2=0x7efd4806cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 --2- 172.21.15.50:0/1282210807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd48159770 0x7efd4806cce0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efd480698f0 tx=0x7efd40009840 comp rx=0 tx=0).stop 2023-12-19T14:39:59.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- 172.21.15.50:0/1282210807 shutdown_connections 2023-12-19T14:39:59.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 --2- 172.21.15.50:0/1282210807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd48159770 0x7efd4806cce0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 --2- 172.21.15.50:0/1282210807 >> v2:172.21.15.125:3300/0 conn(0x7efd48158c90 0x7efd480caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- 172.21.15.50:0/1282210807 >> 172.21.15.50:0/1282210807 conn(0x7efd48067bb0 msgr2=0x7efd480cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:39:59.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- 172.21.15.50:0/1282210807 shutdown_connections 2023-12-19T14:39:59.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- 172.21.15.50:0/1282210807 wait complete. 2023-12-19T14:39:59.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 Processor -- start 2023-12-19T14:39:59.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.920+0000 7efd4d68e700 1 -- start start 2023-12-19T14:39:59.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd48158c90 0x7efd4808a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd48159770 0x7efd481525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd48153750 con 0x7efd4806aeb0 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd48170e10 con 0x7efd48158c90 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd481716b0 con 0x7efd48159770 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34400/0 (socket says 172.21.15.50:34400) 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 -- 172.21.15.50:0/2554610572 learned_addr learned my addr 172.21.15.50:0/2554610572 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd477fe700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd48158c90 0x7efd4808a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4ce8d700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd48159770 0x7efd481525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 -- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd48159770 msgr2=0x7efd481525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd48159770 0x7efd481525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 -- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd48158c90 msgr2=0x7efd4808a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd48158c90 0x7efd4808a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 -- 172.21.15.50:0/2554610572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40009ae0 con 0x7efd4806aeb0 2023-12-19T14:39:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd47fff700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efd3800d8c0 tx=0x7efd38008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3800de60 con 0x7efd4806aeb0 2023-12-19T14:39:59.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3800b780 con 0x7efd4806aeb0 2023-12-19T14:39:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3800bb20 con 0x7efd4806aeb0 2023-12-19T14:39:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4808b030 con 0x7efd4806aeb0 2023-12-19T14:39:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.924+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd48171340 con 0x7efd4806aeb0 2023-12-19T14:39:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.928+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4804be40 con 0x7efd4806aeb0 2023-12-19T14:39:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.928+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efd3801e460 con 0x7efd4806aeb0 2023-12-19T14:39:59.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.928+0000 7efd2ffff700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 0x7efd34062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:39:59.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.928+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7efd3807d2b0 con 0x7efd4806aeb0 2023-12-19T14:39:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.932+0000 7efd477fe700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 0x7efd34062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:39:59.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.932+0000 7efd477fe700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 0x7efd34062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd480cf900 tx=0x7efd3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:39:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:39:59.936+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd3804d440 con 0x7efd4806aeb0 2023-12-19T14:40:00.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.108+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efd4806bd10 con 0x7efd4806aeb0 2023-12-19T14:40:00.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.108+0000 7efd2ffff700 1 -- 172.21.15.50:0/2554610572 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7efd38050a60 con 0x7efd4806aeb0 2023-12-19T14:40:00.111 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.108+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 msgr2=0x7efd34062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.108+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 0x7efd34062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd480cf900 tx=0x7efd3c009040 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.108+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 msgr2=0x7efd481580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efd3800d8c0 tx=0x7efd38008040 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 shutdown_connections 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd48158c90 0x7efd4808a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efd34060550 0x7efd34062a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efd48159770 0x7efd481525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 --2- 172.21.15.50:0/2554610572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efd4806aeb0 0x7efd481580f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 >> 172.21.15.50:0/2554610572 conn(0x7efd48067bb0 msgr2=0x7efd4806a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:00.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 shutdown_connections 2023-12-19T14:40:00.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.112+0000 7efd4d68e700 1 -- 172.21.15.50:0/2554610572 wait complete. 2023-12-19T14:40:00.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:40:00.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.300+0000 7f0ab359e700 1 Processor -- start 2023-12-19T14:40:00.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.300+0000 7f0ab359e700 1 -- start start 2023-12-19T14:40:00.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ab40cc040 0x7f0ab4159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab40cbbb0 con 0x7f0ab40caa10 2023-12-19T14:40:00.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab406dc90 con 0x7f0ab40cc040 2023-12-19T14:40:00.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40cafa0 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab2d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ab40cc040 0x7f0ab4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44358/0 (socket says 172.21.15.50:44358) 2023-12-19T14:40:00.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 -- 172.21.15.50:0/242390899 learned_addr learned my addr 172.21.15.50:0/242390899 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:00.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 -- 172.21.15.50:0/242390899 >> v2:172.21.15.125:3300/0 conn(0x7f0ab40cc040 msgr2=0x7f0ab4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:00.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 --2- 172.21.15.50:0/242390899 >> v2:172.21.15.125:3300/0 conn(0x7f0ab40cc040 0x7f0ab4159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 -- 172.21.15.50:0/242390899 >> v1:172.21.15.125:6789/0 conn(0x7f0ab40caa10 legacy=0x7f0ab4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 -- 172.21.15.50:0/242390899 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab40cb280 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.304+0000 7f0ab259c700 1 --2- 172.21.15.50:0/242390899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40099b0 tx=0x7f0aa4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ffe07e1bd774fcc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab159a700 1 -- 172.21.15.50:0/242390899 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4009ea0 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab159a700 1 -- 172.21.15.50:0/242390899 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400b860 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab159a700 1 -- 172.21.15.50:0/242390899 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400bc00 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 -- 172.21.15.50:0/242390899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 msgr2=0x7f0ab40ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 --2- 172.21.15.50:0/242390899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40099b0 tx=0x7f0aa4009250 comp rx=0 tx=0).stop 2023-12-19T14:40:00.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 -- 172.21.15.50:0/242390899 shutdown_connections 2023-12-19T14:40:00.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 --2- 172.21.15.50:0/242390899 >> v2:172.21.15.125:3300/0 conn(0x7f0ab40cc040 0x7f0ab4159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 --2- 172.21.15.50:0/242390899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab40ce6a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 -- 172.21.15.50:0/242390899 >> 172.21.15.50:0/242390899 conn(0x7f0ab4067bb0 msgr2=0x7f0ab40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:00.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 -- 172.21.15.50:0/242390899 shutdown_connections 2023-12-19T14:40:00.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.308+0000 7f0ab359e700 1 -- 172.21.15.50:0/242390899 wait complete. 2023-12-19T14:40:00.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 Processor -- start 2023-12-19T14:40:00.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 -- start start 2023-12-19T14:40:00.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 0x7f0ab416a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab415a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab415a560 con 0x7f0ab40caa10 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab4159c70 con 0x7f0ab40cc040 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40c8ca0 con 0x7f0ab40ce2d0 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34408/0 (socket says 172.21.15.50:34408) 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab259c700 1 -- 172.21.15.50:0/3918788547 learned_addr learned my addr 172.21.15.50:0/3918788547 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab1d9b700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 0x7f0ab416a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.312+0000 7f0ab2d9d700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab415a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 -- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 msgr2=0x7f0ab415a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab415a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 -- 172.21.15.50:0/3918788547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 msgr2=0x7f0ab414ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 -- 172.21.15.50:0/3918788547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40094f0 con 0x7f0ab40cc040 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab259c700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab1d9b700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 0x7f0ab416a590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c590 tx=0x7f0aa8007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8011590 con 0x7f0ab40cc040 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8011710 con 0x7f0ab40cc040 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8011ad0 con 0x7f0ab40cc040 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab4159f10 con 0x7f0ab40cc040 2023-12-19T14:40:00.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab416aad0 con 0x7f0ab40cc040 2023-12-19T14:40:00.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.316+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab415a6d0 con 0x7f0ab40cc040 2023-12-19T14:40:00.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.320+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0aa801b3b0 con 0x7f0ab40cc040 2023-12-19T14:40:00.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.320+0000 7f0aa27fc700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 0x7f0a94062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.320+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f0aa807bf60 con 0x7f0ab40cc040 2023-12-19T14:40:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.320+0000 7f0ab259c700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 0x7f0a94062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.324+0000 7f0ab259c700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 0x7f0a94062ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400c010 tx=0x7f0aa4017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.328+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0aa804c170 con 0x7f0ab40cc040 2023-12-19T14:40:00.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.500+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ab416acf0 con 0x7f0ab40cc040 2023-12-19T14:40:00.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.504+0000 7f0aa27fc700 1 -- 172.21.15.50:0/3918788547 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0aa804f790 con 0x7f0ab40cc040 2023-12-19T14:40:00.506 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:40:00.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 msgr2=0x7f0a94062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 0x7f0a94062ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400c010 tx=0x7f0aa4017000 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 msgr2=0x7f0ab416a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 0x7f0ab416a590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c590 tx=0x7f0aa8007ec0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 shutdown_connections 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40cc040 0x7f0ab416a590 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a94060620 0x7f0a94062ad0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40ce2d0 0x7f0ab415a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 --2- 172.21.15.50:0/3918788547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40caa10 0x7f0ab414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 >> 172.21.15.50:0/3918788547 conn(0x7f0ab4067bb0 msgr2=0x7f0ab415bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 shutdown_connections 2023-12-19T14:40:00.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.508+0000 7f0ab359e700 1 -- 172.21.15.50:0/3918788547 wait complete. 2023-12-19T14:40:00.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T14:40:00.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.700+0000 7ff0ea185700 1 Processor -- start 2023-12-19T14:40:00.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.700+0000 7ff0ea185700 1 -- start start 2023-12-19T14:40:00.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.700+0000 7ff0ea185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0ea185700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0e406ac30 0x7ff0e406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0ea185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e406a7a0 con 0x7ff0e40c6d20 2023-12-19T14:40:00.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0ea185700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e406d4b0 con 0x7ff0e406ac30 2023-12-19T14:40:00.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0ea185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e406d790 con 0x7ff0e406c7c0 2023-12-19T14:40:00.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e37fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff0e40c6d20 0x7ff0e40c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36668/0 (socket says 172.21.15.50:36668) 2023-12-19T14:40:00.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e37fe700 1 -- 172.21.15.50:0/1807051169 learned_addr learned my addr 172.21.15.50:0/1807051169 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/1807051169 >> v2:172.21.15.125:3300/0 conn(0x7ff0e406ac30 0x7ff0e406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e2ffd700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265636314 0 0) 0x7ff0e406a7a0 con 0x7ff0e40c6d20 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0cc003570 con 0x7ff0e40c6d20 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963271149 0 0) 0x7ff0cc003570 con 0x7ff0e40c6d20 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 -- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 msgr2=0x7ff0e406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 -- 172.21.15.50:0/1807051169 >> v1:172.21.15.50:6789/0 conn(0x7ff0e40c6d20 legacy=0x7ff0e40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 -- 172.21.15.50:0/1807051169 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e406da70 con 0x7ff0e406ac30 2023-12-19T14:40:00.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e406b050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009e60 tx=0x7ff0dc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a96a3d13d5f95d1a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc00e070 con 0x7ff0e406ac30 2023-12-19T14:40:00.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc00b7a0 con 0x7ff0e406ac30 2023-12-19T14:40:00.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.704+0000 7ff0e27fc700 1 -- 172.21.15.50:0/1807051169 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4001470 con 0x7ff0e406ac30 2023-12-19T14:40:00.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 -- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 msgr2=0x7ff0e406b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e406b050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc009e60 tx=0x7ff0dc009780 comp rx=0 tx=0).stop 2023-12-19T14:40:00.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 -- 172.21.15.50:0/1807051169 shutdown_connections 2023-12-19T14:40:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e406b050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 --2- 172.21.15.50:0/1807051169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 -- 172.21.15.50:0/1807051169 >> 172.21.15.50:0/1807051169 conn(0x7ff0e4067bb0 msgr2=0x7ff0e40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 -- 172.21.15.50:0/1807051169 shutdown_connections 2023-12-19T14:40:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 -- 172.21.15.50:0/1807051169 wait complete. 2023-12-19T14:40:00.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.708+0000 7ff0ea185700 1 Processor -- start 2023-12-19T14:40:00.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 -- start start 2023-12-19T14:40:00.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e41537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40ca950 con 0x7ff0e40c6d20 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e415ad40 con 0x7ff0e406ac30 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0ea185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40ca390 con 0x7ff0e406c7c0 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e2ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e37fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e41537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e2ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44390/0 (socket says 172.21.15.50:44390) 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e2ffd700 1 -- 172.21.15.50:0/2666153576 learned_addr learned my addr 172.21.15.50:0/2666153576 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:00.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34420/0 (socket says 172.21.15.50:34420) 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 -- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 msgr2=0x7ff0e40aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 -- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 msgr2=0x7ff0e41537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e41537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 -- 172.21.15.50:0/2666153576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc009a20 con 0x7ff0e40c6d20 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e2ffd700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.712+0000 7ff0e3fff700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4068030 tx=0x7ff0dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc012b80 con 0x7ff0e40c6d20 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc012d20 con 0x7ff0e40c6d20 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e40ca670 con 0x7ff0e40c6d20 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc01d680 con 0x7ff0e40c6d20 2023-12-19T14:40:00.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e4153d20 con 0x7ff0e40c6d20 2023-12-19T14:40:00.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e415a830 con 0x7ff0e40c6d20 2023-12-19T14:40:00.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.716+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff0dc024070 con 0x7ff0e40c6d20 2023-12-19T14:40:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.720+0000 7ff0cb7fe700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 0x7ff0d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:00.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.720+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7ff0dc07ecc0 con 0x7ff0e40c6d20 2023-12-19T14:40:00.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.720+0000 7ff0e37fe700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 0x7ff0d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:00.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.720+0000 7ff0e37fe700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 0x7ff0d0062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40011e0 tx=0x7ff0d4001a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:00.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.724+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff0e415a830 con 0x7ff0e40c6d20 2023-12-19T14:40:00.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.904+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff0e415a830 con 0x7ff0e40c6d20 2023-12-19T14:40:00.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.904+0000 7ff0cb7fe700 1 -- 172.21.15.50:0/2666153576 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff0e415a830 con 0x7ff0e40c6d20 2023-12-19T14:40:00.907 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:40:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.904+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 msgr2=0x7ff0d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.904+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 0x7ff0d0062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40011e0 tx=0x7ff0d4001a00 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 msgr2=0x7ff0e415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4068030 tx=0x7ff0dc018000 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 shutdown_connections 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0d0060550 0x7ff0d0062a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0e40c6d20 0x7ff0e415b1d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0e406c7c0 0x7ff0e40aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 --2- 172.21.15.50:0/2666153576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0e406ac30 0x7ff0e41537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 >> 172.21.15.50:0/2666153576 conn(0x7ff0e4067bb0 msgr2=0x7ff0e40c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 shutdown_connections 2023-12-19T14:40:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:00.908+0000 7ff0ea185700 1 -- 172.21.15.50:0/2666153576 wait complete. 2023-12-19T14:40:00.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:40:01.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 Processor -- start 2023-12-19T14:40:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 -- start start 2023-12-19T14:40:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fed0c0c15b0 0x7fed0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0c7190 con 0x7fed0c15d320 2023-12-19T14:40:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c0aefd0 con 0x7fed0c0c15b0 2023-12-19T14:40:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed1222e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c1b50 con 0x7fed0c0aeba0 2023-12-19T14:40:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed0bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed0bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44404/0 (socket says 172.21.15.50:44404) 2023-12-19T14:40:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed0bfff700 1 -- 172.21.15.50:0/3082706682 learned_addr learned my addr 172.21.15.50:0/3082706682 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.096+0000 7fed0affd700 1 --2- 172.21.15.50:0/3082706682 >> v2:172.21.15.125:3300/0 conn(0x7fed0c0c15b0 0x7fed0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0bfff700 1 -- 172.21.15.50:0/3082706682 >> v2:172.21.15.125:3300/0 conn(0x7fed0c0c15b0 msgr2=0x7fed0c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0bfff700 1 --2- 172.21.15.50:0/3082706682 >> v2:172.21.15.125:3300/0 conn(0x7fed0c0c15b0 0x7fed0c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0bfff700 1 -- 172.21.15.50:0/3082706682 >> v1:172.21.15.125:6789/0 conn(0x7fed0c15d320 legacy=0x7fed0c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0bfff700 1 -- 172.21.15.50:0/3082706682 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0c1e30 con 0x7fed0c0aeba0 2023-12-19T14:40:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0bfff700 1 --2- 172.21.15.50:0/3082706682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed04009f20 tx=0x7fed04009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850fa48577b8331f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0a7fc700 1 -- 172.21.15.50:0/3082706682 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400e070 con 0x7fed0c0aeba0 2023-12-19T14:40:01.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0a7fc700 1 -- 172.21.15.50:0/3082706682 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0400b9d0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed0a7fc700 1 -- 172.21.15.50:0/3082706682 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040092e0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 -- 172.21.15.50:0/3082706682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 msgr2=0x7fed0c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 --2- 172.21.15.50:0/3082706682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed04009f20 tx=0x7fed04009840 comp rx=0 tx=0).stop 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 -- 172.21.15.50:0/3082706682 shutdown_connections 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 --2- 172.21.15.50:0/3082706682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c0aeba0 0x7fed0c0680f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 --2- 172.21.15.50:0/3082706682 >> v2:172.21.15.125:3300/0 conn(0x7fed0c0c15b0 0x7fed0c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 -- 172.21.15.50:0/3082706682 >> 172.21.15.50:0/3082706682 conn(0x7fed0c0c8850 msgr2=0x7fed0c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 -- 172.21.15.50:0/3082706682 shutdown_connections 2023-12-19T14:40:01.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.100+0000 7fed1222e700 1 -- 172.21.15.50:0/3082706682 wait complete. 2023-12-19T14:40:01.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 Processor -- start 2023-12-19T14:40:01.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 -- start start 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed0c0c15b0 0x7fed0c14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 0x7fed0c14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c14f180 con 0x7fed0c0aeba0 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c153680 con 0x7fed0c0c15b0 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed1222e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c153960 con 0x7fed0c15d320 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34426/0 (socket says 172.21.15.50:34426) 2023-12-19T14:40:01.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed0b7fe700 1 -- 172.21.15.50:0/325901924 learned_addr learned my addr 172.21.15.50:0/325901924 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed0affd700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed0c0c15b0 0x7fed0c14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.104+0000 7fed0bfff700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 0x7fed0c14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 -- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 msgr2=0x7fed0c14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 0x7fed0c14e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 -- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed0c0c15b0 msgr2=0x7fed0c14ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed0c0c15b0 0x7fed0c14ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 -- 172.21.15.50:0/325901924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04009ae0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0bfff700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 0x7fed0c14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed0b7fe700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c590 tx=0x7fecfc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011590 con 0x7fed0c0aeba0 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc011710 con 0x7fed0c0aeba0 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c152160 con 0x7fed0c0aeba0 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011ad0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c14e870 con 0x7fed0c0aeba0 2023-12-19T14:40:01.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.108+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c14ea60 con 0x7fed0c0aeba0 2023-12-19T14:40:01.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.112+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fecfc01b3b0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.112+0000 7fecf37fe700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 0x7fecf8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.112+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fecfc07c170 con 0x7fed0c0aeba0 2023-12-19T14:40:01.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.112+0000 7fed0affd700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 0x7fecf8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.116+0000 7fed0affd700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 0x7fecf8062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed0c170810 tx=0x7fed000091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.120+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fecfc04c300 con 0x7fed0c0aeba0 2023-12-19T14:40:01.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.324+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c1525e0 con 0x7fed0c0aeba0 2023-12-19T14:40:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.328+0000 7fecf37fe700 1 -- 172.21.15.50:0/325901924 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7fecfc04f920 con 0x7fed0c0aeba0 2023-12-19T14:40:01.331 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:01.331 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 msgr2=0x7fecf8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 0x7fecf8062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed0c170810 tx=0x7fed000091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 msgr2=0x7fed0c151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c590 tx=0x7fecfc00c8a0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 shutdown_connections 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed0c0c15b0 0x7fed0c14ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fecf8060550 0x7fecf8062a00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed0c15d320 0x7fed0c14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 --2- 172.21.15.50:0/325901924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed0c0aeba0 0x7fed0c151720 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 >> 172.21.15.50:0/325901924 conn(0x7fed0c0c8850 msgr2=0x7fed0c0ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:01.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 shutdown_connections 2023-12-19T14:40:01.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.332+0000 7fed1222e700 1 -- 172.21.15.50:0/325901924 wait complete. 2023-12-19T14:40:01.338 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:40:01.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:01.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.520+0000 7f0c52985700 1 Processor -- start 2023-12-19T14:40:01.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.520+0000 7f0c52985700 1 -- start start 2023-12-19T14:40:01.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c52985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c4c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c52985700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c06e150 con 0x7f0c4c140710 2023-12-19T14:40:01.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c52985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c06d540 con 0x7f0c4c068630 2023-12-19T14:40:01.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c52985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c068ab0 con 0x7f0c4c140ae0 2023-12-19T14:40:01.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0c4c140710 0x7f0c4c0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36670/0 (socket says 172.21.15.50:36670) 2023-12-19T14:40:01.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4bfff700 1 -- 172.21.15.50:0/1167906695 learned_addr learned my addr 172.21.15.50:0/1167906695 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:01.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c437fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c4c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943189709 0 0) 0x7f0c4c06e150 con 0x7f0c4c140710 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c38003570 con 0x7f0c4c140710 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559960341 0 0) 0x7f0c4c06d540 con 0x7f0c4c068630 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c4c06e150 con 0x7f0c4c068630 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829959014 0 0) 0x7f0c38003570 con 0x7f0c4c140710 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 msgr2=0x7f0c4c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:01.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c4c0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 >> v1:172.21.15.125:6789/0 conn(0x7f0c4c068630 legacy=0x7f0c4c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c06d820 con 0x7f0c4c140710 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f0c3c0017f0 con 0x7f0c4c140710 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 >> v1:172.21.15.50:6789/0 conn(0x7f0c4c140710 legacy=0x7f0c4c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38003c20 0x7f0c38004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c38004810 0x7f0c38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c38005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c068ab0 con 0x7f0c38004810 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3800bbf0 con 0x7f0c38003c20 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.524+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3800bdb0 con 0x7f0c4c140ae0 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c38004810 0x7f0c38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c4bfff700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38003c20 0x7f0c38004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c50f22700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c38005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 -- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 msgr2=0x7f0c38005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c140ae0 0x7f0c38005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 -- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38003c20 msgr2=0x7f0c38004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38003c20 0x7f0c38004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 -- 172.21.15.50:0/1167906695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c38003570 con 0x7f0c38004810 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c437fe700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c38004810 0x7f0c38004c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009c00 tx=0x7f0c340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834c15b7e31d7fc3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400e710 con 0x7f0c38004810 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3400e870 con 0x7f0c38004810 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c4b7fe700 1 -- 172.21.15.50:0/1167906695 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400ec30 con 0x7f0c38004810 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 -- 172.21.15.50:0/1167906695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c38004810 msgr2=0x7f0c38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 --2- 172.21.15.50:0/1167906695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c38004810 0x7f0c38004c80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009c00 tx=0x7f0c340091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c50f22700 1 -- 172.21.15.50:0/1167906695 reap_dead start 2023-12-19T14:40:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 -- 172.21.15.50:0/1167906695 shutdown_connections 2023-12-19T14:40:01.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 -- 172.21.15.50:0/1167906695 >> 172.21.15.50:0/1167906695 conn(0x7f0c4c0c8850 msgr2=0x7f0c4c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:01.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 -- 172.21.15.50:0/1167906695 shutdown_connections 2023-12-19T14:40:01.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.528+0000 7f0c52985700 1 -- 172.21.15.50:0/1167906695 wait complete. 2023-12-19T14:40:01.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 Processor -- start 2023-12-19T14:40:01.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 -- start start 2023-12-19T14:40:01.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 0x7f0c4c0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 0x7f0c4c0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0b1a20 con 0x7f0c4c13f790 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c1584b0 con 0x7f0c38004810 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.532+0000 7f0c52985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c1581a0 con 0x7f0c4c0b1070 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34430/0 (socket says 172.21.15.50:34430) 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 -- 172.21.15.50:0/2175740633 learned_addr learned my addr 172.21.15.50:0/2175740633 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c4bfff700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 0x7f0c4c0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c50f22700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 0x7f0c4c0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 -- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 msgr2=0x7f0c4c0b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 0x7f0c4c0b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 -- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 msgr2=0x7f0c4c0c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 0x7f0c4c0c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 -- 172.21.15.50:0/2175740633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c0afe00 con 0x7f0c4c13f790 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c50f22700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 0x7f0c4c0b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c4bfff700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 0x7f0c4c0c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c437fe700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e4b0 tx=0x7f0c3401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34022070 con 0x7f0c4c13f790 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3400ebf0 con 0x7f0c4c13f790 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3401c6c0 con 0x7f0c4c13f790 2023-12-19T14:40:01.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c0b00e0 con 0x7f0c4c13f790 2023-12-19T14:40:01.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.536+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c0afa90 con 0x7f0c4c13f790 2023-12-19T14:40:01.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.540+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c0c80c0 con 0x7f0c4c13f790 2023-12-19T14:40:01.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.540+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0c3401cab0 con 0x7f0c4c13f790 2023-12-19T14:40:01.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.540+0000 7f0c48ff9700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 0x7f0c2c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.540+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f0c3407e4a0 con 0x7f0c4c13f790 2023-12-19T14:40:01.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.540+0000 7f0c4bfff700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 0x7f0c2c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.544+0000 7f0c4bfff700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 0x7f0c2c062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001920 tx=0x7f0c3c006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.548+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0c4c0c80c0 con 0x7f0c4c13f790 2023-12-19T14:40:01.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.720+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c4c0c80c0 con 0x7f0c4c13f790 2023-12-19T14:40:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.724+0000 7f0c48ff9700 1 -- 172.21.15.50:0/2175740633 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7f0c4c0c80c0 con 0x7f0c4c13f790 2023-12-19T14:40:01.727 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:01.727 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:01.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 msgr2=0x7f0c2c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 0x7f0c2c062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c001920 tx=0x7f0c3c006040 comp rx=0 tx=0).stop 2023-12-19T14:40:01.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 msgr2=0x7f0c4c0af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e4b0 tx=0x7f0c3401a000 comp rx=0 tx=0).stop 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 shutdown_connections 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c2c060550 0x7f0c2c062a00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c4c0b1070 0x7f0c4c0b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c4c13f790 0x7f0c4c0af550 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 --2- 172.21.15.50:0/2175740633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c38004810 0x7f0c4c0c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 >> 172.21.15.50:0/2175740633 conn(0x7f0c4c0c8850 msgr2=0x7f0c4c0ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 shutdown_connections 2023-12-19T14:40:01.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.728+0000 7f0c52985700 1 -- 172.21.15.50:0/2175740633 wait complete. 2023-12-19T14:40:01.754 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:40:01.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:40:01.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 Processor -- start 2023-12-19T14:40:01.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 -- start start 2023-12-19T14:40:01.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe8840cc610 0x7fe8840c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840c6630 con 0x7fe884067c90 2023-12-19T14:40:01.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840c6910 con 0x7fe8841502b0 2023-12-19T14:40:01.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe88a345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840c6bf0 con 0x7fe8840cc610 2023-12-19T14:40:01.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe883fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe884067c90 0x7fe88414fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36672/0 (socket says 172.21.15.50:36672) 2023-12-19T14:40:01.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe883fff700 1 -- 172.21.15.50:0/4008714100 learned_addr learned my addr 172.21.15.50:0/4008714100 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:01.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe8888e2700 1 --2- 172.21.15.50:0/4008714100 >> v2:172.21.15.50:3300/0 conn(0x7fe8840cc610 0x7fe8840c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.920+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557194072 0 0) 0x7fe8840c6630 con 0x7fe884067c90 2023-12-19T14:40:01.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe870003570 con 0x7fe884067c90 2023-12-19T14:40:01.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281440089 0 0) 0x7fe8840c6910 con 0x7fe8841502b0 2023-12-19T14:40:01.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8840c6630 con 0x7fe8841502b0 2023-12-19T14:40:01.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 828122804 0 0) 0x7fe870003570 con 0x7fe884067c90 2023-12-19T14:40:01.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8888e2700 1 -- 172.21.15.50:0/4008714100 >> v1:172.21.15.125:6789/0 conn(0x7fe8841502b0 legacy=0x7fe8840c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8888e2700 1 -- 172.21.15.50:0/4008714100 >> v1:172.21.15.50:6789/0 conn(0x7fe884067c90 legacy=0x7fe88414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8888e2700 1 -- 172.21.15.50:0/4008714100 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840c6ed0 con 0x7fe8840cc610 2023-12-19T14:40:01.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8888e2700 1 --2- 172.21.15.50:0/4008714100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8840cc610 0x7fe8840c6010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e60 tx=0x7fe87c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c8d5bd5d6fd25a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00e070 con 0x7fe8840cc610 2023-12-19T14:40:01.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00b7a0 con 0x7fe8840cc610 2023-12-19T14:40:01.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe8837fe700 1 -- 172.21.15.50:0/4008714100 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe874001470 con 0x7fe8840cc610 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 -- 172.21.15.50:0/4008714100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8840cc610 msgr2=0x7fe8840c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 --2- 172.21.15.50:0/4008714100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8840cc610 0x7fe8840c6010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e60 tx=0x7fe87c009780 comp rx=0 tx=0).stop 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 -- 172.21.15.50:0/4008714100 shutdown_connections 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 --2- 172.21.15.50:0/4008714100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8840cc610 0x7fe8840c6010 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 -- 172.21.15.50:0/4008714100 >> 172.21.15.50:0/4008714100 conn(0x7fe8840cd6d0 msgr2=0x7fe88406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 -- 172.21.15.50:0/4008714100 shutdown_connections 2023-12-19T14:40:01.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.924+0000 7fe88a345700 1 -- 172.21.15.50:0/4008714100 wait complete. 2023-12-19T14:40:01.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 Processor -- start 2023-12-19T14:40:01.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 -- start start 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe884067c90 0x7fe88414c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8840cc610 0x7fe884152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c73b0 con 0x7fe8841502b0 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe884151180 con 0x7fe8840cc610 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe88a345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe884151d00 con 0x7fe884067c90 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe8888e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe8888e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34450/0 (socket says 172.21.15.50:34450) 2023-12-19T14:40:01.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe8888e2700 1 -- 172.21.15.50:0/3354840681 learned_addr learned my addr 172.21.15.50:0/3354840681 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe883fff700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe884067c90 0x7fe88414c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.928+0000 7fe87b7fe700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8840cc610 0x7fe884152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 -- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe884067c90 msgr2=0x7fe88414c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe884067c90 0x7fe88414c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 -- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8840cc610 msgr2=0x7fe884152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8840cc610 0x7fe884152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 -- 172.21.15.50:0/3354840681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009a20 con 0x7fe8841502b0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe8888e2700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00b610 tx=0x7fe87c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c012c30 con 0x7fe8841502b0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c012d90 con 0x7fe8841502b0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88414c6e0 con 0x7fe8841502b0 2023-12-19T14:40:01.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c01e680 con 0x7fe8841502b0 2023-12-19T14:40:01.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841516b0 con 0x7fe8841502b0 2023-12-19T14:40:01.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.932+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe884071120 con 0x7fe8841502b0 2023-12-19T14:40:01.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.936+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe87c01e9f0 con 0x7fe8841502b0 2023-12-19T14:40:01.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.936+0000 7fe880ff9700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 0x7fe864062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:01.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.936+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7fe87c07edb0 con 0x7fe8841502b0 2023-12-19T14:40:01.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.936+0000 7fe883fff700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 0x7fe864062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:01.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.940+0000 7fe883fff700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 0x7fe864062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe8740015a0 tx=0x7fe874000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:01.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:01.944+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe87c083070 con 0x7fe8841502b0 2023-12-19T14:40:02.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.148+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe884151e40 con 0x7fe8841502b0 2023-12-19T14:40:02.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe880ff9700 1 -- 172.21.15.50:0/3354840681 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7fe87c04ef40 con 0x7fe8841502b0 2023-12-19T14:40:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:02.155 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:40:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 msgr2=0x7fe864062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 0x7fe864062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe8740015a0 tx=0x7fe874000bc0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 msgr2=0x7fe88414c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00b610 tx=0x7fe87c017000 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 shutdown_connections 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8840cc610 0x7fe884152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe864060550 0x7fe864062a00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8841502b0 0x7fe88414c1a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 --2- 172.21.15.50:0/3354840681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe884067c90 0x7fe88414c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.152+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 >> 172.21.15.50:0/3354840681 conn(0x7fe8840cd6d0 msgr2=0x7fe88414ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:02.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.156+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 shutdown_connections 2023-12-19T14:40:02.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.156+0000 7fe88a345700 1 -- 172.21.15.50:0/3354840681 wait complete. 2023-12-19T14:40:02.160 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:40:02.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:40:02.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.356+0000 7f459ba25700 1 Processor -- start 2023-12-19T14:40:02.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 -- start start 2023-12-19T14:40:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4594140710 0x7f45940c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594068800 0x7f45940ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459406e150 con 0x7f4594140ae0 2023-12-19T14:40:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459406d540 con 0x7f4594140710 2023-12-19T14:40:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f459ba25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459406d820 con 0x7f4594068800 2023-12-19T14:40:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f45997c1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4594140710 0x7f45940c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f45997c1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4594140710 0x7f45940c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34452/0 (socket says 172.21.15.50:34452) 2023-12-19T14:40:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f45997c1700 1 -- 172.21.15.50:0/1847565577 learned_addr learned my addr 172.21.15.50:0/1847565577 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f4599fc2700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594068800 0x7f45940ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f458bfff700 1 -- 172.21.15.50:0/1847565577 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240244336 0 0) 0x7f459406e150 con 0x7f4594140ae0 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.360+0000 7f458bfff700 1 -- 172.21.15.50:0/1847565577 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580003570 con 0x7f4594140ae0 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f45997c1700 1 -- 172.21.15.50:0/1847565577 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594068800 msgr2=0x7f45940ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f45997c1700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594068800 0x7f45940ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f45997c1700 1 -- 172.21.15.50:0/1847565577 >> v1:172.21.15.125:6789/0 conn(0x7f4594140ae0 legacy=0x7f45940680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f45997c1700 1 -- 172.21.15.50:0/1847565577 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459406db00 con 0x7f4594140710 2023-12-19T14:40:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f45997c1700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140710 0x7f45940c70a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f45900099b0 tx=0x7f4590009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d461ed1400a32e1f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f458bfff700 1 -- 172.21.15.50:0/1847565577 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000b680 con 0x7f4594140710 2023-12-19T14:40:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f458bfff700 1 -- 172.21.15.50:0/1847565577 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459000b7e0 con 0x7f4594140710 2023-12-19T14:40:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f458bfff700 1 -- 172.21.15.50:0/1847565577 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000bb80 con 0x7f4594140710 2023-12-19T14:40:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 -- 172.21.15.50:0/1847565577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140710 msgr2=0x7f45940c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140710 0x7f45940c70a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f45900099b0 tx=0x7f4590009250 comp rx=0 tx=0).stop 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 -- 172.21.15.50:0/1847565577 shutdown_connections 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594068800 0x7f45940ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 --2- 172.21.15.50:0/1847565577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140710 0x7f45940c70a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 -- 172.21.15.50:0/1847565577 >> 172.21.15.50:0/1847565577 conn(0x7f45940c8850 msgr2=0x7f45940c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 -- 172.21.15.50:0/1847565577 shutdown_connections 2023-12-19T14:40:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.364+0000 7f459ba25700 1 -- 172.21.15.50:0/1847565577 wait complete. 2023-12-19T14:40:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.368+0000 7f459ba25700 1 Processor -- start 2023-12-19T14:40:02.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.368+0000 7f459ba25700 1 -- start start 2023-12-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4594068800 0x7f459415d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594140710 0x7f45940c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45940c43f0 con 0x7f4594140ae0 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45940c4f40 con 0x7f4594068800 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459413d3b0 con 0x7f4594140710 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34466/0 (socket says 172.21.15.50:34466) 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 -- 172.21.15.50:0/1088909426 learned_addr learned my addr 172.21.15.50:0/1088909426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f45997c1700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4594068800 0x7f459415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4598fc0700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594140710 0x7f45940c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 -- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594140710 msgr2=0x7f45940c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594140710 0x7f45940c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 -- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4594068800 msgr2=0x7f459415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4594068800 0x7f459415d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 -- 172.21.15.50:0/1088909426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45900094f0 con 0x7f4594140ae0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f4599fc2700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ec0 tx=0x7f458c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c00cb70 con 0x7f4594140ae0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c00f040 con 0x7f4594140ae0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459415dab0 con 0x7f4594140ae0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c014480 con 0x7f4594140ae0 2023-12-19T14:40:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.372+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45940c4110 con 0x7f4594140ae0 2023-12-19T14:40:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.376+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45940c50f0 con 0x7f4594140ae0 2023-12-19T14:40:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.376+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f458c0147f0 con 0x7f4594140ae0 2023-12-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.376+0000 7f45897fa700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 0x7f4578062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.376+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f458c07d7b0 con 0x7f4594140ae0 2023-12-19T14:40:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.380+0000 7f45997c1700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 0x7f4578062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.380+0000 7f45997c1700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 0x7f4578062a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45900099b0 tx=0x7f4590017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.384+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f45940c50f0 con 0x7f4594140ae0 2023-12-19T14:40:02.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.596+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45940c50f0 con 0x7f4594140ae0 2023-12-19T14:40:02.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.596+0000 7f45897fa700 1 -- 172.21.15.50:0/1088909426 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f45940c50f0 con 0x7f4594140ae0 2023-12-19T14:40:02.600 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:02.600 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/3677473885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3677473885},{"type":"v1","addr":"172.21.15.125:6835","nonce":3677473885}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4255,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/931810497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":931810497},{"type":"v1","addr":"172.21.15.125:6837","nonce":931810497}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/1431538639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1431538639},{"type":"v1","addr":"172.21.15.50:6834","nonce":1431538639}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:39:06.675595+0000","modified":"2023-12-19T14:39:21.592876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3977471945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3977471945},{"type":"v1","addr":"172.21.15.50:6835","nonce":3977471945}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T14:40:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 msgr2=0x7f4578062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 0x7f4578062a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45900099b0 tx=0x7f4590017000 comp rx=0 tx=0).stop 2023-12-19T14:40:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 msgr2=0x7f459415e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ec0 tx=0x7f458c00b040 comp rx=0 tx=0).stop 2023-12-19T14:40:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 shutdown_connections 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4578060580 0x7f4578062a30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4594140ae0 0x7f459415e130 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4594140710 0x7f45940c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 --2- 172.21.15.50:0/1088909426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4594068800 0x7f459415d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 >> 172.21.15.50:0/1088909426 conn(0x7f45940c8850 msgr2=0x7f45940c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 shutdown_connections 2023-12-19T14:40:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.600+0000 7f459ba25700 1 -- 172.21.15.50:0/1088909426 wait complete. 2023-12-19T14:40:02.607 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 8 2023-12-19T14:40:02.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.792+0000 7f112084c700 1 Processor -- start 2023-12-19T14:40:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.792+0000 7f112084c700 1 -- start start 2023-12-19T14:40:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f112084c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f112084c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f111c06b7a0 0x7f111c0c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f112084c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0c5610 con 0x7f111c0c51b0 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f112084c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c06c7c0 con 0x7f111c06b7a0 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f112084c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c06caa0 con 0x7f111c158d60 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f1112ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f111c06b7a0 0x7f111c0c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f111bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f111c0c51b0 0x7f111c06a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36688/0 (socket says 172.21.15.50:36688) 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f111bfff700 1 -- 172.21.15.50:0/3785966968 learned_addr learned my addr 172.21.15.50:0/3785966968 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.796+0000 7f111b7fe700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f1112ffd700 1 -- 172.21.15.50:0/3785966968 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 msgr2=0x7f111c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f1112ffd700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f1112ffd700 1 -- 172.21.15.50:0/3785966968 >> v1:172.21.15.50:6789/0 conn(0x7f111c0c51b0 legacy=0x7f111c06a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f1112ffd700 1 -- 172.21.15.50:0/3785966968 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c15ce60 con 0x7f111c06b7a0 2023-12-19T14:40:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f1112ffd700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c0c4c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f11040098e0 tx=0x7f11040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4822c0d4a52d4e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f111affd700 1 -- 172.21.15.50:0/3785966968 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104009e80 con 0x7f111c06b7a0 2023-12-19T14:40:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f111affd700 1 -- 172.21.15.50:0/3785966968 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110400b860 con 0x7f111c06b7a0 2023-12-19T14:40:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f111affd700 1 -- 172.21.15.50:0/3785966968 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400bbe0 con 0x7f111c06b7a0 2023-12-19T14:40:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 -- 172.21.15.50:0/3785966968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 msgr2=0x7f111c0c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c0c4c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f11040098e0 tx=0x7f11040091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 -- 172.21.15.50:0/3785966968 shutdown_connections 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c0c4c70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 --2- 172.21.15.50:0/3785966968 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 -- 172.21.15.50:0/3785966968 >> 172.21.15.50:0/3785966968 conn(0x7f111c067bb0 msgr2=0x7f111c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 -- 172.21.15.50:0/3785966968 shutdown_connections 2023-12-19T14:40:02.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.800+0000 7f112084c700 1 -- 172.21.15.50:0/3785966968 wait complete. 2023-12-19T14:40:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 Processor -- start 2023-12-19T14:40:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 -- start start 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111c0c51b0 0x7f111c144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c15c8b0 con 0x7f111c06b7a0 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c15cac0 con 0x7f111c0c51b0 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f112084c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c15bc90 con 0x7f111c158d60 2023-12-19T14:40:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34484/0 (socket says 172.21.15.50:34484) 2023-12-19T14:40:02.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 -- 172.21.15.50:0/3104716508 learned_addr learned my addr 172.21.15.50:0/3104716508 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f1112ffd700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111c0c51b0 0x7f111c144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111bfff700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 -- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 msgr2=0x7f111c0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c0b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 -- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111c0c51b0 msgr2=0x7f111c144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111c0c51b0 0x7f111c144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 -- 172.21.15.50:0/3104716508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1104009420 con 0x7f111c06b7a0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111bfff700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.804+0000 7f111b7fe700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d890 tx=0x7f110c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c009610 con 0x7f111c06b7a0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c009770 con 0x7f111c06b7a0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c009b30 con 0x7f111c06b7a0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c0af560 con 0x7f111c06b7a0 2023-12-19T14:40:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c0afae0 con 0x7f111c06b7a0 2023-12-19T14:40:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.808+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c04be40 con 0x7f111c06b7a0 2023-12-19T14:40:02.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.812+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f110c01e460 con 0x7f111c06b7a0 2023-12-19T14:40:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.812+0000 7f1113fff700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 0x7f10fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:02.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.812+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f110c07d1b0 con 0x7f111c06b7a0 2023-12-19T14:40:02.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.812+0000 7f1112ffd700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 0x7f10fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.812+0000 7f1112ffd700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 0x7f10fc062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f111c0a58d0 tx=0x7f1104017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:02.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.820+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f110c04d340 con 0x7f111c06b7a0 2023-12-19T14:40:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.992+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f111c15d130 con 0x7f111c06b7a0 2023-12-19T14:40:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:02.996+0000 7f1113fff700 1 -- 172.21.15.50:0/3104716508 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17519 (secure 0 0 0) 0x7f110c050960 con 0x7f111c06b7a0 2023-12-19T14:40:02.999 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:02.999 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":26,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:39:49.548433+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:39:04.445520+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T14:39:05.676716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T14:39:09.568551+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 msgr2=0x7f10fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 0x7f10fc062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f111c0a58d0 tx=0x7f1104017000 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 msgr2=0x7f111c15c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d890 tx=0x7f110c008040 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 shutdown_connections 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f111c0c51b0 0x7f111c144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10fc060550 0x7f10fc062a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f111c158d60 0x7f111c0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 --2- 172.21.15.50:0/3104716508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f111c06b7a0 0x7f111c15c370 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 >> 172.21.15.50:0/3104716508 conn(0x7f111c067bb0 msgr2=0x7f111c0cb650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 shutdown_connections 2023-12-19T14:40:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.000+0000 7f112084c700 1 -- 172.21.15.50:0/3104716508 wait complete. 2023-12-19T14:40:03.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:40:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.188+0000 7f8e7d9c5700 1 Processor -- start 2023-12-19T14:40:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e7d9c5700 1 -- start start 2023-12-19T14:40:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e7d9c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 0x7f8e7815b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e7d9c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7806d7d0 con 0x7f8e7806c7c0 2023-12-19T14:40:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e7d9c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7806cc50 con 0x7f8e780b13e0 2023-12-19T14:40:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e7d9c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e780ae280 con 0x7f8e78140420 2023-12-19T14:40:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e6e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8e7806c7c0 0x7f8e780b0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36702/0 (socket says 172.21.15.50:36702) 2023-12-19T14:40:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.192+0000 7f8e6e7fc700 1 -- 172.21.15.50:0/29857785 learned_addr learned my addr 172.21.15.50:0/29857785 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e76ffd700 1 --2- 172.21.15.50:0/29857785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 0x7f8e7815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962183053 0 0) 0x7f8e7806d7d0 con 0x7f8e7806c7c0 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e64003570 con 0x7f8e7806c7c0 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660755513 0 0) 0x7f8e7806cc50 con 0x7f8e780b13e0 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7806d7d0 con 0x7f8e780b13e0 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e76ffd700 1 -- 172.21.15.50:0/29857785 >> v1:172.21.15.125:6789/0 conn(0x7f8e780b13e0 legacy=0x7f8e780ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e76ffd700 1 -- 172.21.15.50:0/29857785 >> v1:172.21.15.50:6789/0 conn(0x7f8e7806c7c0 legacy=0x7f8e780b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e76ffd700 1 -- 172.21.15.50:0/29857785 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7806cf30 con 0x7f8e78140420 2023-12-19T14:40:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e76ffd700 1 --2- 172.21.15.50:0/29857785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 0x7f8e7815b220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009e90 tx=0x7f8e68009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19119df242eb1af7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800e070 con 0x7f8e78140420 2023-12-19T14:40:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6800bb80 con 0x7f8e78140420 2023-12-19T14:40:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e767fc700 1 -- 172.21.15.50:0/29857785 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68012430 con 0x7f8e78140420 2023-12-19T14:40:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/29857785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 msgr2=0x7f8e7815b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/29857785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 0x7f8e7815b220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009e90 tx=0x7f8e68009780 comp rx=0 tx=0).stop 2023-12-19T14:40:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/29857785 shutdown_connections 2023-12-19T14:40:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/29857785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e78140420 0x7f8e7815b220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.196+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/29857785 >> 172.21.15.50:0/29857785 conn(0x7f8e78067b60 msgr2=0x7f8e780ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/29857785 shutdown_connections 2023-12-19T14:40:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/29857785 wait complete. 2023-12-19T14:40:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 Processor -- start 2023-12-19T14:40:03.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- start start 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e7806c7c0 0x7f8e781691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e780b13e0 0x7f8e781696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78166de0 con 0x7f8e78140420 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7816af80 con 0x7f8e780b13e0 2023-12-19T14:40:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e7d9c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7816a370 con 0x7f8e7806c7c0 2023-12-19T14:40:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34494/0 (socket says 172.21.15.50:34494) 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.200+0000 7f8e777fe700 1 -- 172.21.15.50:0/3648363855 learned_addr learned my addr 172.21.15.50:0/3648363855 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e6e7fc700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e780b13e0 0x7f8e781696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e76ffd700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e7806c7c0 0x7f8e781691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 -- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e7806c7c0 msgr2=0x7f8e781691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e7806c7c0 0x7f8e781691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 -- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e780b13e0 msgr2=0x7f8e781696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e780b13e0 0x7f8e781696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 -- 172.21.15.50:0/3648363855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e68009a20 con 0x7f8e78140420 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e777fe700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c560 tx=0x7f8e70007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70011590 con 0x7f8e78140420 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e700116f0 con 0x7f8e78140420 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70011a90 con 0x7f8e78140420 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7816a650 con 0x7f8e78140420 2023-12-19T14:40:03.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7816ac10 con 0x7f8e78140420 2023-12-19T14:40:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.204+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7816b0f0 con 0x7f8e78140420 2023-12-19T14:40:03.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.208+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8e7001c460 con 0x7f8e78140420 2023-12-19T14:40:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.208+0000 7f8e6f7fe700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 0x7f8e58062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.208+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6659+0+0 (secure 0 0 0) 0x7f8e7007c7f0 con 0x7f8e78140420 2023-12-19T14:40:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.208+0000 7f8e76ffd700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 0x7f8e58062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.212+0000 7f8e76ffd700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 0x7f8e58062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009de0 tx=0x7f8e68019000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.216+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e7004c980 con 0x7f8e78140420 2023-12-19T14:40:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.432+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8e78071120 con 0x7f8e78140420 2023-12-19T14:40:03.437 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:40:03.432+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:40:03.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.444+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/3648363855 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8e7004ffa0 con 0x7f8e78140420 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 msgr2=0x7f8e58062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 0x7f8e58062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009de0 tx=0x7f8e68019000 comp rx=0 tx=0).stop 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 msgr2=0x7f8e78169c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c560 tx=0x7f8e70007d50 comp rx=0 tx=0).stop 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 shutdown_connections 2023-12-19T14:40:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e780b13e0 0x7f8e781696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e58060550 0x7f8e58062a00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e78140420 0x7f8e78169c20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 --2- 172.21.15.50:0/3648363855 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e7806c7c0 0x7f8e781691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 >> 172.21.15.50:0/3648363855 conn(0x7f8e78067b60 msgr2=0x7f8e780cc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 shutdown_connections 2023-12-19T14:40:03.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.448+0000 7f8e7d9c5700 1 -- 172.21.15.50:0/3648363855 wait complete. 2023-12-19T14:40:03.455 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:40:03.459 INFO:tasks.ceph.mds.c.smithi050.stderr: -3677> 2023-12-19T14:39:07.737+0000 7f00653f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:40:03.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:40:03.515 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:40:03.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.636+0000 7f13c046e700 1 Processor -- start 2023-12-19T14:40:03.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.636+0000 7f13c046e700 1 -- start start 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13c046e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13b80c6d20 0x7f13b80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13c046e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13b806c7c0 0x7f13b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13c046e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b806a7a0 con 0x7f13b815aa30 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13c046e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b80c7630 con 0x7f13b806c7c0 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13c046e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b806d4b0 con 0x7f13b80c6d20 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bea0b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f13b815aa30 0x7f13b815ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36712/0 (socket says 172.21.15.50:36712) 2023-12-19T14:40:03.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bea0b700 1 -- 172.21.15.50:0/3683902628 learned_addr learned my addr 172.21.15.50:0/3683902628 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bda09700 1 --2- 172.21.15.50:0/3683902628 >> v2:172.21.15.50:3300/0 conn(0x7f13b806c7c0 0x7f13b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13be20a700 1 --2- 172.21.15.50:0/3683902628 >> v2:172.21.15.125:3300/0 conn(0x7f13b80c6d20 0x7f13b80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bd208700 1 -- 172.21.15.50:0/3683902628 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172090366 0 0) 0x7f13b806a7a0 con 0x7f13b815aa30 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bda09700 1 -- 172.21.15.50:0/3683902628 >> v2:172.21.15.125:3300/0 conn(0x7f13b80c6d20 msgr2=0x7f13b80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bda09700 1 --2- 172.21.15.50:0/3683902628 >> v2:172.21.15.125:3300/0 conn(0x7f13b80c6d20 0x7f13b80c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bda09700 1 -- 172.21.15.50:0/3683902628 >> v1:172.21.15.50:6789/0 conn(0x7f13b815aa30 legacy=0x7f13b815ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.640+0000 7f13bda09700 1 -- 172.21.15.50:0/3683902628 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b806d790 con 0x7f13b806c7c0 2023-12-19T14:40:03.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13bda09700 1 --2- 172.21.15.50:0/3683902628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b806cc60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13b8068030 tx=0x7f13a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb41251717072ef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13bd208700 1 -- 172.21.15.50:0/3683902628 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8011630 con 0x7f13b806c7c0 2023-12-19T14:40:03.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13bd208700 1 -- 172.21.15.50:0/3683902628 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8011790 con 0x7f13b806c7c0 2023-12-19T14:40:03.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13bd208700 1 -- 172.21.15.50:0/3683902628 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8011b30 con 0x7f13b806c7c0 2023-12-19T14:40:03.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 -- 172.21.15.50:0/3683902628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 msgr2=0x7f13b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 --2- 172.21.15.50:0/3683902628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b806cc60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13b8068030 tx=0x7f13a80091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 -- 172.21.15.50:0/3683902628 shutdown_connections 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 --2- 172.21.15.50:0/3683902628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b806cc60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 --2- 172.21.15.50:0/3683902628 >> v2:172.21.15.125:3300/0 conn(0x7f13b80c6d20 0x7f13b80c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 -- 172.21.15.50:0/3683902628 >> 172.21.15.50:0/3683902628 conn(0x7f13b8067bb0 msgr2=0x7f13b80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 -- 172.21.15.50:0/3683902628 shutdown_connections 2023-12-19T14:40:03.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.644+0000 7f13c046e700 1 -- 172.21.15.50:0/3683902628 wait complete. 2023-12-19T14:40:03.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 Processor -- start 2023-12-19T14:40:03.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 -- start start 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13b80c6d20 0x7f13b80c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b815aa30 0x7f13b8156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80c9bd0 con 0x7f13b806c7c0 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80c9810 con 0x7f13b815aa30 2023-12-19T14:40:03.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13c046e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b81699d0 con 0x7f13b80c6d20 2023-12-19T14:40:03.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34526/0 (socket says 172.21.15.50:34526) 2023-12-19T14:40:03.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 -- 172.21.15.50:0/1356002142 learned_addr learned my addr 172.21.15.50:0/1356002142 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:03.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13bda09700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13b80c6d20 0x7f13b80c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13bea0b700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b815aa30 0x7f13b8156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 -- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13b80c6d20 msgr2=0x7f13b80c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13b80c6d20 0x7f13b80c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 -- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b815aa30 msgr2=0x7f13b8156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b815aa30 0x7f13b8156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.648+0000 7f13be20a700 1 -- 172.21.15.50:0/1356002142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8009420 con 0x7f13b806c7c0 2023-12-19T14:40:03.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13be20a700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f13b400ec80 tx=0x7f13b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400c850 con 0x7f13b806c7c0 2023-12-19T14:40:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b8169cb0 con 0x7f13b806c7c0 2023-12-19T14:40:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400c9b0 con 0x7f13b806c7c0 2023-12-19T14:40:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400cd50 con 0x7f13b806c7c0 2023-12-19T14:40:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.652+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b81696c0 con 0x7f13b806c7c0 2023-12-19T14:40:03.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.656+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b80c9e40 con 0x7f13b806c7c0 2023-12-19T14:40:03.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.656+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f13b4013680 con 0x7f13b806c7c0 2023-12-19T14:40:03.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.660+0000 7f13ae7fc700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 0x7f139c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:03.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.660+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f13b407ceb0 con 0x7f13b806c7c0 2023-12-19T14:40:03.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.660+0000 7f13bda09700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 0x7f139c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:03.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.660+0000 7f13bda09700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 0x7f139c062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13a80093f0 tx=0x7f13a8017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:03.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.664+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f13b404d090 con 0x7f13b806c7c0 2023-12-19T14:40:03.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:03.876+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f13b8071120 con 0x7f13b806c7c0 2023-12-19T14:40:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.448+0000 7f13ae7fc700 1 -- 172.21.15.50:0/1356002142 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f13b40506b0 con 0x7f13b806c7c0 2023-12-19T14:40:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 msgr2=0x7f139c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 0x7f139c062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13a80093f0 tx=0x7f13a8017000 comp rx=0 tx=0).stop 2023-12-19T14:40:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 msgr2=0x7f13b8169180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f13b400ec80 tx=0x7f13b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 shutdown_connections 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13b815aa30 0x7f13b8156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f139c060550 0x7f139c062a00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13b80c6d20 0x7f13b80c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 --2- 172.21.15.50:0/1356002142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13b806c7c0 0x7f13b8169180 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 >> 172.21.15.50:0/1356002142 conn(0x7f13b8067bb0 msgr2=0x7f13b80c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 shutdown_connections 2023-12-19T14:40:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.452+0000 7f13c046e700 1 -- 172.21.15.50:0/1356002142 wait complete. 2023-12-19T14:40:04.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:40:04.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.696+0000 7f10b0945700 1 Processor -- start 2023-12-19T14:40:04.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.696+0000 7f10b0945700 1 -- start start 2023-12-19T14:40:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10b0945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10b0945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10ac0ac480 0x7f10ac0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10b0945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ac0ac8e0 con 0x7f10ac156e70 2023-12-19T14:40:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10b0945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ac06d7b0 con 0x7f10ac0ac480 2023-12-19T14:40:04.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10b0945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0ae570 con 0x7f10ac06dc40 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10ac0ac480 0x7f10ac0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aa59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f10ac156e70 0x7f10ac06b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36720/0 (socket says 172.21.15.50:36720) 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aa59c700 1 -- 172.21.15.50:0/2277869728 learned_addr learned my addr 172.21.15.50:0/2277869728 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a9d9b700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a959a700 1 -- 172.21.15.50:0/2277869728 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106185362 0 0) 0x7f109c001110 con 0x7f10ac156e70 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a959a700 1 -- 172.21.15.50:0/2277869728 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1094003570 con 0x7f10ac156e70 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 -- 172.21.15.50:0/2277869728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 msgr2=0x7f10ac0abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac0abf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 -- 172.21.15.50:0/2277869728 >> v1:172.21.15.50:6789/0 conn(0x7f10ac156e70 legacy=0x7f10ac06b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 -- 172.21.15.50:0/2277869728 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac06cba0 con 0x7f10ac0ac480 2023-12-19T14:40:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a9d9b700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10aad9d700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac0ac480 0x7f10ac0aea30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009e60 tx=0x7f10a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a78411b1014f5ca server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a959a700 1 -- 172.21.15.50:0/2277869728 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a400e070 con 0x7f10ac0ac480 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a959a700 1 -- 172.21.15.50:0/2277869728 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a400bab0 con 0x7f10ac0ac480 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.700+0000 7f10a959a700 1 -- 172.21.15.50:0/2277869728 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a400be70 con 0x7f10ac0ac480 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- 172.21.15.50:0/2277869728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac0ac480 msgr2=0x7f10ac0aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac0ac480 0x7f10ac0aea30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009e60 tx=0x7f10a4009780 comp rx=0 tx=0).stop 2023-12-19T14:40:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- 172.21.15.50:0/2277869728 shutdown_connections 2023-12-19T14:40:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac0ac480 0x7f10ac0aea30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 --2- 172.21.15.50:0/2277869728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- 172.21.15.50:0/2277869728 >> 172.21.15.50:0/2277869728 conn(0x7f10ac067bb0 msgr2=0x7f10ac0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- 172.21.15.50:0/2277869728 shutdown_connections 2023-12-19T14:40:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- 172.21.15.50:0/2277869728 wait complete. 2023-12-19T14:40:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 Processor -- start 2023-12-19T14:40:04.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.704+0000 7f10b0945700 1 -- start start 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 0x7f10ac0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0c7590 con 0x7f10ac156e70 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0c6980 con 0x7f10ac0ac480 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0c6c60 con 0x7f10ac06dc40 2023-12-19T14:40:04.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34534/0 (socket says 172.21.15.50:34534) 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 -- 172.21.15.50:0/2598714572 learned_addr learned my addr 172.21.15.50:0/2598714572 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aa59c700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10a9d9b700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 0x7f10ac0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 -- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 msgr2=0x7f10ac1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 -- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 msgr2=0x7f10ac0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 0x7f10ac0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 -- 172.21.15.50:0/2598714572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0009420 con 0x7f10ac156e70 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10a9d9b700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 0x7f10ac0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aa59c700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10aad9d700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f10ac068360 tx=0x7f10a4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:04.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4027470 con 0x7f10ac156e70 2023-12-19T14:40:04.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a40275f0 con 0x7f10ac156e70 2023-12-19T14:40:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40279b0 con 0x7f10ac156e70 2023-12-19T14:40:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a4009a20 con 0x7f10ac156e70 2023-12-19T14:40:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac0c7220 con 0x7f10ac156e70 2023-12-19T14:40:04.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.708+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac0c7740 con 0x7f10ac156e70 2023-12-19T14:40:04.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.712+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f10a4024070 con 0x7f10ac156e70 2023-12-19T14:40:04.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.712+0000 7f109a7fc700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 0x7f108c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:04.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.712+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f10a407f580 con 0x7f10ac156e70 2023-12-19T14:40:04.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.712+0000 7f10aa59c700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 0x7f108c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:04.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.716+0000 7f10aa59c700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 0x7f108c062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f109c0058d0 tx=0x7f109c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:04.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.720+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10ac0c7740 con 0x7f10ac156e70 2023-12-19T14:40:04.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:40:04.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:40:04.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:04.892+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10ac0c76d0 con 0x7f10ac156e70 2023-12-19T14:40:05.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.672+0000 7f109a7fc700 1 -- 172.21.15.50:0/2598714572 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f10a404f6c0 con 0x7f10ac156e70 2023-12-19T14:40:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.676+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 msgr2=0x7f108c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.676+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 0x7f108c062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f109c0058d0 tx=0x7f109c009040 comp rx=0 tx=0).stop 2023-12-19T14:40:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.676+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 msgr2=0x7f10ac0afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.676+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f10ac068360 tx=0x7f10a4019000 comp rx=0 tx=0).stop 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 shutdown_connections 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10ac0ac480 0x7f10ac0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f108c060550 0x7f108c062a00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10ac156e70 0x7f10ac0afda0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 --2- 172.21.15.50:0/2598714572 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10ac06dc40 0x7f10ac1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 >> 172.21.15.50:0/2598714572 conn(0x7f10ac067bb0 msgr2=0x7f10ac152040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:05.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 shutdown_connections 2023-12-19T14:40:05.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.680+0000 7f10b0945700 1 -- 172.21.15.50:0/2598714572 wait complete. 2023-12-19T14:40:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:40:05.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:40:05.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.880+0000 7f3ddd6e3700 1 Processor -- start 2023-12-19T14:40:05.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.880+0000 7f3ddd6e3700 1 -- start start 2023-12-19T14:40:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddd6e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 0x7f3dd80c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddd6e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3dd815b140 0x7f3dd80b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddd6e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd806da90 con 0x7f3dd80ad9f0 2023-12-19T14:40:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddd6e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd806a620 con 0x7f3dd815b140 2023-12-19T14:40:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddd6e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd806dd30 con 0x7f3dd80addc0 2023-12-19T14:40:05.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3dd815b140 0x7f3dd80b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3dd815b140 0x7f3dd80b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34548/0 (socket says 172.21.15.50:34548) 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 -- 172.21.15.50:0/494795642 learned_addr learned my addr 172.21.15.50:0/494795642 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3dd7fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3dd80ad9f0 0x7f3dd806ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36728/0 (socket says 172.21.15.50:36728) 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3dcf7fe700 1 --2- 172.21.15.50:0/494795642 >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 0x7f3dd80c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 -- 172.21.15.50:0/494795642 >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 msgr2=0x7f3dd80c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 --2- 172.21.15.50:0/494795642 >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 0x7f3dd80c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 -- 172.21.15.50:0/494795642 >> v1:172.21.15.50:6789/0 conn(0x7f3dd80ad9f0 legacy=0x7f3dd806ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3ddcee2700 1 -- 172.21.15.50:0/494795642 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd806a8c0 con 0x7f3dd815b140 2023-12-19T14:40:05.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.884+0000 7f3dcf7fe700 1 --2- 172.21.15.50:0/494795642 >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 0x7f3dd80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddcee2700 1 --2- 172.21.15.50:0/494795642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd815b140 0x7f3dd80b0340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8068a90 tx=0x7f3dd0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3e75d1ea59eaf2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3dd77fe700 1 -- 172.21.15.50:0/494795642 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd000e070 con 0x7f3dd815b140 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3dd77fe700 1 -- 172.21.15.50:0/494795642 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd000ba50 con 0x7f3dd815b140 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3dd77fe700 1 -- 172.21.15.50:0/494795642 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00092e0 con 0x7f3dd815b140 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/494795642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd815b140 msgr2=0x7f3dd80b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/494795642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd815b140 0x7f3dd80b0340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8068a90 tx=0x7f3dd0009840 comp rx=0 tx=0).stop 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/494795642 shutdown_connections 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/494795642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd815b140 0x7f3dd80b0340 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/494795642 >> v2:172.21.15.125:3300/0 conn(0x7f3dd80addc0 0x7f3dd80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/494795642 >> 172.21.15.50:0/494795642 conn(0x7f3dd8067bb0 msgr2=0x7f3dd80c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/494795642 shutdown_connections 2023-12-19T14:40:05.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.888+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/494795642 wait complete. 2023-12-19T14:40:05.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 Processor -- start 2023-12-19T14:40:05.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 -- start start 2023-12-19T14:40:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3dd80ad9f0 0x7f3dd813bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 0x7f3dd80c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd816a4c0 con 0x7f3dd80addc0 2023-12-19T14:40:05.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80c55c0 con 0x7f3dd815b140 2023-12-19T14:40:05.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddd6e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd816aa10 con 0x7f3dd80ad9f0 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34552/0 (socket says 172.21.15.50:34552) 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 -- 172.21.15.50:0/3685257407 learned_addr learned my addr 172.21.15.50:0/3685257407 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3ddcee2700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 0x7f3dd80c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dd7fff700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3dd80ad9f0 0x7f3dd813bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 -- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3dd80ad9f0 msgr2=0x7f3dd813bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3dd80ad9f0 0x7f3dd813bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 -- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 msgr2=0x7f3dd80c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 0x7f3dd80c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.892+0000 7f3dcf7fe700 1 -- 172.21.15.50:0/3685257407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd0009ae0 con 0x7f3dd80addc0 2023-12-19T14:40:05.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3ddcee2700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 0x7f3dd80c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:05.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3dcf7fe700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000c860 tx=0x7f3dc0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:05.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011600 con 0x7f3dd80addc0 2023-12-19T14:40:05.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc0011760 con 0x7f3dd80addc0 2023-12-19T14:40:05.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011b00 con 0x7f3dd80addc0 2023-12-19T14:40:05.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd80c5000 con 0x7f3dd80addc0 2023-12-19T14:40:05.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd816a720 con 0x7f3dd80addc0 2023-12-19T14:40:05.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.896+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd813c140 con 0x7f3dd80addc0 2023-12-19T14:40:05.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.900+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3dc00193b0 con 0x7f3dd80addc0 2023-12-19T14:40:05.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.900+0000 7f3dd4ff9700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 0x7f3db8062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:05.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.900+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6307+0+0 (secure 0 0 0) 0x7f3dc007ca20 con 0x7f3dd80addc0 2023-12-19T14:40:05.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.900+0000 7f3dd7fff700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 0x7f3db8062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:05.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.904+0000 7f3dd7fff700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 0x7f3db8062b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007fd0 tx=0x7f3dc8007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:05.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:05.908+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3dc004cd90 con 0x7f3dd80addc0 2023-12-19T14:40:06.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.096+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3dd804be40 con 0x7f3dd80addc0 2023-12-19T14:40:06.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.692+0000 7f3dd4ff9700 1 -- 172.21.15.50:0/3685257407 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3dc00150d0 con 0x7f3dd80addc0 2023-12-19T14:40:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 msgr2=0x7f3db8062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 0x7f3db8062b00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8007fd0 tx=0x7f3dc8007d70 comp rx=0 tx=0).stop 2023-12-19T14:40:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 msgr2=0x7f3dd80c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:06.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000c860 tx=0x7f3dc0008040 comp rx=0 tx=0).stop 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 shutdown_connections 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd815b140 0x7f3dd80c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3db8060650 0x7f3db8062b00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3dd80addc0 0x7f3dd80c4700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 --2- 172.21.15.50:0/3685257407 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3dd80ad9f0 0x7f3dd813bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 >> 172.21.15.50:0/3685257407 conn(0x7f3dd8067bb0 msgr2=0x7f3dd80c7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 shutdown_connections 2023-12-19T14:40:06.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.700+0000 7f3ddd6e3700 1 -- 172.21.15.50:0/3685257407 wait complete. 2023-12-19T14:40:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:40:06.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-12-19T14:40:06.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.896+0000 7fea268ae700 1 Processor -- start 2023-12-19T14:40:06.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.896+0000 7fea268ae700 1 -- start start 2023-12-19T14:40:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea20156e70 0x7fea2006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea200ac650 0x7fea200aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2006d7b0 con 0x7fea2006b060 2023-12-19T14:40:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2006e180 con 0x7fea200ac650 2023-12-19T14:40:06.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea200ae570 con 0x7fea20156e70 2023-12-19T14:40:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea200ac650 0x7fea200aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fea200ac650 0x7fea200aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56516/0 (socket says 172.21.15.50:56516) 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 -- 172.21.15.50:0/1967903373 learned_addr learned my addr 172.21.15.50:0/1967903373 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea1ffff700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea20156e70 0x7fea2006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 -- 172.21.15.50:0/1967903373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea20156e70 msgr2=0x7fea2006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea20156e70 0x7fea2006dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 -- 172.21.15.50:0/1967903373 >> v1:172.21.15.125:6789/0 conn(0x7fea2006b060 legacy=0x7fea200abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 -- 172.21.15.50:0/1967903373 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2006cba0 con 0x7fea200ac650 2023-12-19T14:40:06.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea24e4b700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea200ac650 0x7fea200aea30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea18009f20 tx=0x7fea18009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6671d2ee5bd8741f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea1f7fe700 1 -- 172.21.15.50:0/1967903373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1800e070 con 0x7fea200ac650 2023-12-19T14:40:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea1f7fe700 1 -- 172.21.15.50:0/1967903373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1800ba40 con 0x7fea200ac650 2023-12-19T14:40:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea1f7fe700 1 -- 172.21.15.50:0/1967903373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea180092e0 con 0x7fea200ac650 2023-12-19T14:40:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- 172.21.15.50:0/1967903373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea200ac650 msgr2=0x7fea200aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea200ac650 0x7fea200aea30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea18009f20 tx=0x7fea18009840 comp rx=0 tx=0).stop 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- 172.21.15.50:0/1967903373 shutdown_connections 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea200ac650 0x7fea200aea30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 --2- 172.21.15.50:0/1967903373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea20156e70 0x7fea2006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- 172.21.15.50:0/1967903373 >> 172.21.15.50:0/1967903373 conn(0x7fea20067bb0 msgr2=0x7fea200cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- 172.21.15.50:0/1967903373 shutdown_connections 2023-12-19T14:40:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.900+0000 7fea268ae700 1 -- 172.21.15.50:0/1967903373 wait complete. 2023-12-19T14:40:06.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 Processor -- start 2023-12-19T14:40:06.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 -- start start 2023-12-19T14:40:06.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea200ac650 0x7fea2014d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea20156e70 0x7fea2014da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20153ad0 con 0x7fea2006b060 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20153db0 con 0x7fea20156e70 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea268ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea200c7080 con 0x7fea200ac650 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea1ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea1ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56530/0 (socket says 172.21.15.50:56530) 2023-12-19T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.904+0000 7fea1ffff700 1 -- 172.21.15.50:0/1488521873 learned_addr learned my addr 172.21.15.50:0/1488521873 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea17fff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea200ac650 0x7fea2014d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea24e4b700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea20156e70 0x7fea2014da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 -- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea200ac650 msgr2=0x7fea2014d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea200ac650 0x7fea2014d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 -- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea20156e70 msgr2=0x7fea2014da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea20156e70 0x7fea2014da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 -- 172.21.15.50:0/1488521873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18009ae0 con 0x7fea2006b060 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1ffff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea1000d890 tx=0x7fea10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10009610 con 0x7fea2006b060 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea10009770 con 0x7fea2006b060 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea200c7360 con 0x7fea2006b060 2023-12-19T14:40:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10009b30 con 0x7fea2006b060 2023-12-19T14:40:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea200c76f0 con 0x7fea2006b060 2023-12-19T14:40:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.908+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea20071120 con 0x7fea2006b060 2023-12-19T14:40:06.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.912+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fea1001e460 con 0x7fea2006b060 2023-12-19T14:40:06.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.912+0000 7fea1cff9700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 0x7fea00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:06.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.912+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5777+0+0 (secure 0 0 0) 0x7fea1007cf10 con 0x7fea2006b060 2023-12-19T14:40:06.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.912+0000 7fea17fff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 0x7fea00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:06.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.916+0000 7fea17fff700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 0x7fea00062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea200c90d0 tx=0x7fea08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:06.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:06.920+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fea1004d410 con 0x7fea2006b060 2023-12-19T14:40:07.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.092+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea2006ce80 con 0x7fea2006b060 2023-12-19T14:40:07.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.704+0000 7fea1cff9700 1 -- 172.21.15.50:0/1488521873 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7fea10050a30 con 0x7fea2006b060 2023-12-19T14:40:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 msgr2=0x7fea00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 0x7fea00062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea200c90d0 tx=0x7fea08009040 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 msgr2=0x7fea200c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea1000d890 tx=0x7fea10008040 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 shutdown_connections 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea20156e70 0x7fea2014da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fea00060550 0x7fea00062a00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea200ac650 0x7fea2014d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 --2- 172.21.15.50:0/1488521873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea2006b060 0x7fea200c6770 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 >> 172.21.15.50:0/1488521873 conn(0x7fea20067bb0 msgr2=0x7fea200c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 shutdown_connections 2023-12-19T14:40:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.708+0000 7fea268ae700 1 -- 172.21.15.50:0/1488521873 wait complete. 2023-12-19T14:40:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data_ec' removed 2023-12-19T14:40:07.739 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:40:07.739 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:40:07.739 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:40:07.740 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:40:07.740 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:40:07.740 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:40:07.740 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:40:07.740 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:40:07.740 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:40:07.740 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:40:07.741 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:40:07.741 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:40:07.741 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:40:07.741 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:40:07.741 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:40:07.741 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:40:07.761 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:40:07.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:40:07.763 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:40:07.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:40:07.763 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:40:07.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:40:07.764 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:40:07.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:40:07.815 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:40:07.815 DEBUG:teuthology.parallel:result is None 2023-12-19T14:40:07.818 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:40:07.818 DEBUG:teuthology.parallel:result is None 2023-12-19T14:40:07.820 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:40:07.820 DEBUG:teuthology.parallel:result is None 2023-12-19T14:40:07.821 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:40:07.822 DEBUG:teuthology.parallel:result is None 2023-12-19T14:40:07.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:40:07.932 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:40:07.937 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:40:07.941 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:40:07.951 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:40:07.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.972+0000 7f0ffabcc700 1 Processor -- start 2023-12-19T14:40:07.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.972+0000 7f0ffabcc700 1 -- start start 2023-12-19T14:40:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffabcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ff4158df0 0x7f0ff406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffabcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4159cd0 con 0x7f0ff41598d0 2023-12-19T14:40:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffabcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4159f90 con 0x7f0ff406d590 2023-12-19T14:40:07.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffabcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff406da10 con 0x7f0ff4158df0 2023-12-19T14:40:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff9bca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ff4158df0 0x7f0ff406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffa3cb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0ff41598d0 0x7f0ff40c8130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39342/0 (socket says 172.21.15.50:39342) 2023-12-19T14:40:07.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff93c9700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0ff406d590 0x7f0ff41591c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:37542/0 (socket says 172.21.15.50:37542) 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ffa3cb700 1 -- 172.21.15.50:0/1817187531 learned_addr learned my addr 172.21.15.50:0/1817187531 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160703277 0 0) 0x7f0ff4159cd0 con 0x7f0ff41598d0 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe0003570 con 0x7f0ff41598d0 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024532392 0 0) 0x7f0ff4159f90 con 0x7f0ff406d590 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff9bca700 1 -- 172.21.15.50:0/1817187531 >> v1:172.21.15.125:6789/0 conn(0x7f0ff406d590 legacy=0x7f0ff41591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff9bca700 1 -- 172.21.15.50:0/1817187531 >> v1:172.21.15.50:6789/0 conn(0x7f0ff41598d0 legacy=0x7f0ff40c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.976+0000 7f0ff9bca700 1 -- 172.21.15.50:0/1817187531 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff415a270 con 0x7f0ff4158df0 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ff9bca700 1 --2- 172.21.15.50:0/1817187531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff4158df0 0x7f0ff406d050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00098e0 tx=0x7f0ff00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee40c9b006153eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000b640 con 0x7f0ff4158df0 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff000b7a0 con 0x7f0ff4158df0 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ff8bc8700 1 -- 172.21.15.50:0/1817187531 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000bb20 con 0x7f0ff4158df0 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1817187531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff4158df0 msgr2=0x7f0ff406d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1817187531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff4158df0 0x7f0ff406d050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00098e0 tx=0x7f0ff00091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1817187531 shutdown_connections 2023-12-19T14:40:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1817187531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff4158df0 0x7f0ff406d050 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1817187531 >> 172.21.15.50:0/1817187531 conn(0x7f0ff406c080 msgr2=0x7f0ff40ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1817187531 shutdown_connections 2023-12-19T14:40:07.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.980+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1817187531 wait complete. 2023-12-19T14:40:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 Processor -- start 2023-12-19T14:40:07.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 -- start start 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ff406d590 0x7f0ff4155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ff4158df0 0x7f0ff40c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40c60a0 con 0x7f0ff41598d0 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4156020 con 0x7f0ff4158df0 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffabcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4155cb0 con 0x7f0ff406d590 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffa3cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffa3cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56576/0 (socket says 172.21.15.50:56576) 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ff93c9700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ff4158df0 0x7f0ff40c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ffa3cb700 1 -- 172.21.15.50:0/1911842424 learned_addr learned my addr 172.21.15.50:0/1911842424 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.984+0000 7f0ff9bca700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ff406d590 0x7f0ff4155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 -- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ff406d590 msgr2=0x7f0ff4155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ff406d590 0x7f0ff4155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 -- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ff4158df0 msgr2=0x7f0ff40c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ff4158df0 0x7f0ff40c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 -- 172.21.15.50:0/1911842424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0009420 con 0x7f0ff41598d0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffa3cb700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0fec001460 tx=0x7f0fec001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00a890 con 0x7f0ff41598d0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec00a9f0 con 0x7f0ff41598d0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40b1660 con 0x7f0ff41598d0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00ad70 con 0x7f0ff41598d0 2023-12-19T14:40:07.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff40b1c20 con 0x7f0ff41598d0 2023-12-19T14:40:07.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.988+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff404be40 con 0x7f0ff41598d0 2023-12-19T14:40:07.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.992+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0fec007300 con 0x7f0ff41598d0 2023-12-19T14:40:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.996+0000 7f0fe9ffb700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 0x7f0fd8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.996+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5259+0+0 (secure 0 0 0) 0x7f0fec07ca80 con 0x7f0ff41598d0 2023-12-19T14:40:07.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.996+0000 7f0ff9bca700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 0x7f0fd8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:07.996+0000 7f0ff9bca700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 0x7f0fd8062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00093f0 tx=0x7f0ff0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.000+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0fec04d180 con 0x7f0ff41598d0 2023-12-19T14:40:08.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.184+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0ff40cd1f0 con 0x7f0ff41598d0 2023-12-19T14:40:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.728+0000 7f0fe9ffb700 1 -- 172.21.15.50:0/1911842424 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0fec0507a0 con 0x7f0ff41598d0 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.728+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 msgr2=0x7f0fd8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.728+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 0x7f0fd8062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00093f0 tx=0x7f0ff0017000 comp rx=0 tx=0).stop 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.728+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 msgr2=0x7f0ff40ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.728+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0fec001460 tx=0x7f0fec001ae0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 shutdown_connections 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ff4158df0 0x7f0ff40c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0fd8060550 0x7f0fd8062a00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ff41598d0 0x7f0ff40ae3b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 --2- 172.21.15.50:0/1911842424 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ff406d590 0x7f0ff4155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 >> 172.21.15.50:0/1911842424 conn(0x7f0ff406c080 msgr2=0x7f0ff41591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 shutdown_connections 2023-12-19T14:40:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.732+0000 7f0ffabcc700 1 -- 172.21.15.50:0/1911842424 wait complete. 2023-12-19T14:40:08.736 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:40:08.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:40:08.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.932+0000 7f4764b85700 1 Processor -- start 2023-12-19T14:40:08.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.932+0000 7f4764b85700 1 -- start start 2023-12-19T14:40:08.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f4764b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f47600aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f4764b85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476006c9f0 con 0x7f476013f890 2023-12-19T14:40:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f4764b85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600c7f20 con 0x7f47600680f0 2023-12-19T14:40:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f4764b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47600c1c50 con 0x7f47600c2980 2023-12-19T14:40:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475e59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f476013f890 0x7f47600c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39348/0 (socket says 172.21.15.50:39348) 2023-12-19T14:40:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475e59c700 1 -- 172.21.15.50:0/3537993454 learned_addr learned my addr 172.21.15.50:0/3537993454 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475dd9b700 1 --2- 172.21.15.50:0/3537993454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f47600aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241625182 0 0) 0x7f476006c9f0 con 0x7f476013f890 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4748003570 con 0x7f476013f890 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210706726 0 0) 0x7f47600c7f20 con 0x7f47600680f0 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f476006c9f0 con 0x7f47600680f0 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475dd9b700 1 -- 172.21.15.50:0/3537993454 >> v1:172.21.15.125:6789/0 conn(0x7f47600680f0 legacy=0x7f476006ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605181307 0 0) 0x7f4748003570 con 0x7f476013f890 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475dd9b700 1 -- 172.21.15.50:0/3537993454 >> v1:172.21.15.50:6789/0 conn(0x7f476013f890 legacy=0x7f47600c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.936+0000 7f475dd9b700 1 -- 172.21.15.50:0/3537993454 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47600c8190 con 0x7f47600c2980 2023-12-19T14:40:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f475dd9b700 1 --2- 172.21.15.50:0/3537993454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f47600aeba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4754009910 tx=0x7f47540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=128f78ffc8644eb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750001470 con 0x7f47600c2980 2023-12-19T14:40:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4754011420 con 0x7f47600c2980 2023-12-19T14:40:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f475d59a700 1 -- 172.21.15.50:0/3537993454 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47540117c0 con 0x7f47600c2980 2023-12-19T14:40:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 -- 172.21.15.50:0/3537993454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 msgr2=0x7f47600aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 --2- 172.21.15.50:0/3537993454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f47600aeba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4754009910 tx=0x7f47540091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 -- 172.21.15.50:0/3537993454 shutdown_connections 2023-12-19T14:40:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 --2- 172.21.15.50:0/3537993454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f47600aeba0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 -- 172.21.15.50:0/3537993454 >> 172.21.15.50:0/3537993454 conn(0x7f47600c8850 msgr2=0x7f47600c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 -- 172.21.15.50:0/3537993454 shutdown_connections 2023-12-19T14:40:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.940+0000 7f4764b85700 1 -- 172.21.15.50:0/3537993454 wait complete. 2023-12-19T14:40:08.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 Processor -- start 2023-12-19T14:40:08.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 -- start start 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f476013d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f476013f890 0x7f476013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760139980 con 0x7f47600680f0 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760138d70 con 0x7f476013f890 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f4764b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4760139050 con 0x7f47600c2980 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56586/0 (socket says 172.21.15.50:56586) 2023-12-19T14:40:08.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 -- 172.21.15.50:0/906087615 learned_addr learned my addr 172.21.15.50:0/906087615 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475dd9b700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f476013d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475ed9d700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f476013f890 0x7f476013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 -- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 msgr2=0x7f476013d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f476013d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 -- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f476013f890 msgr2=0x7f476013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f476013f890 0x7f476013d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 -- 172.21.15.50:0/906087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754009420 con 0x7f47600680f0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f475e59c700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f475000c7e0 tx=0x7f4750001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.944+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750007360 con 0x7f47600680f0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500074e0 con 0x7f47600680f0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4760139330 con 0x7f47600680f0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750004830 con 0x7f47600680f0 2023-12-19T14:40:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47601389e0 con 0x7f47600680f0 2023-12-19T14:40:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47600c84e0 con 0x7f47600680f0 2023-12-19T14:40:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.948+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4750004ba0 con 0x7f47600680f0 2023-12-19T14:40:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.952+0000 7f474e7fc700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 0x7f4740062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.952+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f475007bc90 con 0x7f47600680f0 2023-12-19T14:40:08.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.952+0000 7f475dd9b700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 0x7f4740062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:08.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.952+0000 7f475dd9b700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 0x7f4740062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f47600c2dc0 tx=0x7f4754016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:08.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:08.960+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f47600c84e0 con 0x7f47600680f0 2023-12-19T14:40:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.192+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f47600c84e0 con 0x7f47600680f0 2023-12-19T14:40:09.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f474e7fc700 1 -- 172.21.15.50:0/906087615 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f47600c84e0 con 0x7f47600680f0 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:40:09.199 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:40:09.200 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:40:09.201 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.202 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.203 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.204 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:40:09.209 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-mds", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFl8MefKhAAlSLqOPdY0krcfJKoJFIkYg==", 2023-12-19T14:40:09.210 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-mds" 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-mgr", 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFlaNafKhAArorToex2lfUvYySsaSTdGA==", 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-mgr" 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-osd", 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFlweSfKhAAwiSeURed3DaxxzKgytQaRg==", 2023-12-19T14:40:09.211 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-osd" 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-rbd", 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFln/SfKhAAzhVRBGYhocchLNfOb4hZHA==", 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-rbd" 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFlawGgKhAABAsrtovuuGV25LWeBDfuUw==", 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.bootstrap-rgw", 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDyqoFlZA6gKhAAHV1MiqakuZVpu5cDSbyzqQ==", 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile bootstrap-rgw" 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:40:09.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:40:09.214 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:40:09.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 msgr2=0x7f4740062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 0x7f4740062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f47600c2dc0 tx=0x7f4754016000 comp rx=0 tx=0).stop 2023-12-19T14:40:09.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 msgr2=0x7f47600c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f475000c7e0 tx=0x7f4750001a30 comp rx=0 tx=0).stop 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 shutdown_connections 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f476013f890 0x7f476013d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4740060550 0x7f4740062a00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47600c2980 0x7f476013d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 --2- 172.21.15.50:0/906087615 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47600680f0 0x7f47600c2e90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 >> 172.21.15.50:0/906087615 conn(0x7f47600c8850 msgr2=0x7f47600cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 shutdown_connections 2023-12-19T14:40:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.196+0000 7f4764b85700 1 -- 172.21.15.50:0/906087615 wait complete. 2023-12-19T14:40:09.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-12-19T14:40:09.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.388+0000 7f1ae87c5700 1 Processor -- start 2023-12-19T14:40:09.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.388+0000 7f1ae87c5700 1 -- start start 2023-12-19T14:40:09.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae87c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ae0140420 0x7f1ae015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae87c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae00b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae87c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae006d7d0 con 0x7f1ae01407f0 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae87c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae00ae280 con 0x7f1ae0140420 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae87c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00cb8f0 con 0x7f1ae006c7c0 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ae0140420 0x7f1ae015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ae0140420 0x7f1ae015b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56598/0 (socket says 172.21.15.50:56598) 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 -- 172.21.15.50:0/1157267546 learned_addr learned my addr 172.21.15.50:0/1157267546 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae5d60700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae00b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 -- 172.21.15.50:0/1157267546 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 msgr2=0x7f1ae00b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae00b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 -- 172.21.15.50:0/1157267546 >> v1:172.21.15.125:6789/0 conn(0x7f1ae01407f0 legacy=0x7f1ae006a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.392+0000 7f1ae6561700 1 -- 172.21.15.50:0/1157267546 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae006cc50 con 0x7f1ae0140420 2023-12-19T14:40:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae6561700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae015b220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0099b0 tx=0x7f1adc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fe794561bc56 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae555f700 1 -- 172.21.15.50:0/1157267546 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00b860 con 0x7f1ae0140420 2023-12-19T14:40:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae555f700 1 -- 172.21.15.50:0/1157267546 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc00b9c0 con 0x7f1ae0140420 2023-12-19T14:40:09.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae555f700 1 -- 172.21.15.50:0/1157267546 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00bd60 con 0x7f1ae0140420 2023-12-19T14:40:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 -- 172.21.15.50:0/1157267546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 msgr2=0x7f1ae015b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae015b220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0099b0 tx=0x7f1adc009250 comp rx=0 tx=0).stop 2023-12-19T14:40:09.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 -- 172.21.15.50:0/1157267546 shutdown_connections 2023-12-19T14:40:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae00b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/1157267546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae015b220 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 -- 172.21.15.50:0/1157267546 >> 172.21.15.50:0/1157267546 conn(0x7f1ae0067b60 msgr2=0x7f1ae00ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 -- 172.21.15.50:0/1157267546 shutdown_connections 2023-12-19T14:40:09.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.396+0000 7f1ae87c5700 1 -- 172.21.15.50:0/1157267546 wait complete. 2023-12-19T14:40:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 Processor -- start 2023-12-19T14:40:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 -- start start 2023-12-19T14:40:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae013e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 0x7f1ae0167390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae013ed00 con 0x7f1ae0140420 2023-12-19T14:40:09.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae013dd70 con 0x7f1ae01407f0 2023-12-19T14:40:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae87c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00c84f0 con 0x7f1ae006c7c0 2023-12-19T14:40:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56612/0 (socket says 172.21.15.50:56612) 2023-12-19T14:40:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 -- 172.21.15.50:0/3017498454 learned_addr learned my addr 172.21.15.50:0/3017498454 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae6561700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae013e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae6d62700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 0x7f1ae0167390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 -- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 msgr2=0x7f1ae013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae013e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 -- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 msgr2=0x7f1ae0167390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 0x7f1ae0167390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae5d60700 1 -- 172.21.15.50:0/3017498454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc0094f0 con 0x7f1ae0140420 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.400+0000 7f1ae6d62700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 0x7f1ae0167390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:09.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ae5d60700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d890 tx=0x7f1ad0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00125f0 con 0x7f1ae0140420 2023-12-19T14:40:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0012750 con 0x7f1ae0140420 2023-12-19T14:40:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0012ad0 con 0x7f1ae0140420 2023-12-19T14:40:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae0167ec0 con 0x7f1ae0140420 2023-12-19T14:40:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae013dfc0 con 0x7f1ae0140420 2023-12-19T14:40:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.404+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0071120 con 0x7f1ae0140420 2023-12-19T14:40:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.408+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1ad001e460 con 0x7f1ae0140420 2023-12-19T14:40:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.408+0000 7f1ad67fc700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 0x7f1ac4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.408+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f1ad007cca0 con 0x7f1ae0140420 2023-12-19T14:40:09.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.408+0000 7f1ae6561700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 0x7f1ac4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.408+0000 7f1ae6561700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 0x7f1ac4062a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0099b0 tx=0x7f1adc017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.412+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1ad004d400 con 0x7f1ae0140420 2023-12-19T14:40:09.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.640+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1ae006cf30 con 0x7f1ae0140420 2023-12-19T14:40:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.648+0000 7f1ad67fc700 1 -- 172.21.15.50:0/3017498454 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1ad0050a20 con 0x7f1ae0140420 2023-12-19T14:40:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 msgr2=0x7f1ac4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 0x7f1ac4062a00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0099b0 tx=0x7f1adc017000 comp rx=0 tx=0).stop 2023-12-19T14:40:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 msgr2=0x7f1ae00c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d890 tx=0x7f1ad0008040 comp rx=0 tx=0).stop 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 shutdown_connections 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ae01407f0 0x7f1ae0167390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1ac4060550 0x7f1ac4062a00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ae0140420 0x7f1ae00c7fb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 --2- 172.21.15.50:0/3017498454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ae006c7c0 0x7f1ae013e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 >> 172.21.15.50:0/3017498454 conn(0x7f1ae0067b60 msgr2=0x7f1ae0158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 shutdown_connections 2023-12-19T14:40:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.652+0000 7f1ae87c5700 1 -- 172.21.15.50:0/3017498454 wait complete. 2023-12-19T14:40:09.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-12-19T14:40:09.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.844+0000 7f9c88fcc700 1 Processor -- start 2023-12-19T14:40:09.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 -- start start 2023-12-19T14:40:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c84158e20 0x7f9c8406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840ce810 0x7f9c840ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8406b730 con 0x7f9c841591f0 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c8406ab20 con 0x7f9c840ce810 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c88fcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8406ae00 con 0x7f9c84158e20 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c83fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840ce810 0x7f9c840ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c83fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840ce810 0x7f9c840ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56618/0 (socket says 172.21.15.50:56618) 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c83fff700 1 -- 172.21.15.50:0/2694868461 learned_addr learned my addr 172.21.15.50:0/2694868461 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.848+0000 7f9c837fe700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c84158e20 0x7f9c8406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c83fff700 1 -- 172.21.15.50:0/2694868461 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c84158e20 msgr2=0x7f9c8406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c83fff700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c84158e20 0x7f9c8406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c83fff700 1 -- 172.21.15.50:0/2694868461 >> v1:172.21.15.50:6789/0 conn(0x7f9c841591f0 legacy=0x7f9c840ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c83fff700 1 -- 172.21.15.50:0/2694868461 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8406b0e0 con 0x7f9c840ce810 2023-12-19T14:40:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c83fff700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ce810 0x7f9c840ca950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009f20 tx=0x7f9c7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53dcb0362c1ee8d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c827fc700 1 -- 172.21.15.50:0/2694868461 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00e070 con 0x7f9c840ce810 2023-12-19T14:40:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c827fc700 1 -- 172.21.15.50:0/2694868461 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c00ba80 con 0x7f9c840ce810 2023-12-19T14:40:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c827fc700 1 -- 172.21.15.50:0/2694868461 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0092e0 con 0x7f9c840ce810 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 -- 172.21.15.50:0/2694868461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ce810 msgr2=0x7f9c840ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ce810 0x7f9c840ca950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009f20 tx=0x7f9c7c009840 comp rx=0 tx=0).stop 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 -- 172.21.15.50:0/2694868461 shutdown_connections 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ce810 0x7f9c840ca950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/2694868461 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c84158e20 0x7f9c8406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 -- 172.21.15.50:0/2694868461 >> 172.21.15.50:0/2694868461 conn(0x7f9c84067bb0 msgr2=0x7f9c840cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 -- 172.21.15.50:0/2694868461 shutdown_connections 2023-12-19T14:40:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.852+0000 7f9c88fcc700 1 -- 172.21.15.50:0/2694868461 wait complete. 2023-12-19T14:40:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 Processor -- start 2023-12-19T14:40:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 -- start start 2023-12-19T14:40:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840ce810 0x7f9c840b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c841591f0 0x7f9c84151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84157200 con 0x7f9c84158e20 2023-12-19T14:40:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841565f0 con 0x7f9c840ce810 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c841568d0 con 0x7f9c841591f0 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56628/0 (socket says 172.21.15.50:56628) 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 -- 172.21.15.50:0/3034520036 learned_addr learned my addr 172.21.15.50:0/3034520036 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c83fff700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c841591f0 0x7f9c84151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c837fe700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840ce810 0x7f9c840b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 -- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c841591f0 msgr2=0x7f9c84151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c841591f0 0x7f9c84151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 -- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840ce810 msgr2=0x7f9c840b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840ce810 0x7f9c840b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 -- 172.21.15.50:0/3034520036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c009ae0 con 0x7f9c84158e20 2023-12-19T14:40:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c82ffd700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800c590 tx=0x7f9c78007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78011590 con 0x7f9c84158e20 2023-12-19T14:40:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c78011710 con 0x7f9c84158e20 2023-12-19T14:40:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c84156bb0 con 0x7f9c84158e20 2023-12-19T14:40:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.856+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78011ad0 con 0x7f9c84158e20 2023-12-19T14:40:09.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.860+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c840b5620 con 0x7f9c84158e20 2023-12-19T14:40:09.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.860+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c84157030 con 0x7f9c84158e20 2023-12-19T14:40:09.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.860+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9c7801b3b0 con 0x7f9c84158e20 2023-12-19T14:40:09.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.860+0000 7f9c6b7fe700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 0x7f9c70062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:09.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.864+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f9c7807b8e0 con 0x7f9c84158e20 2023-12-19T14:40:09.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.864+0000 7f9c837fe700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 0x7f9c70062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:09.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.864+0000 7f9c837fe700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 0x7f9c70062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c74006fd0 tx=0x7f9c74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:09.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:09.868+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c7804c0e0 con 0x7f9c84158e20 2023-12-19T14:40:10.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.104+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f9c8404be40 con 0x7f9c84158e20 2023-12-19T14:40:10.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c6b7fe700 1 -- 172.21.15.50:0/3034520036 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9c7804f700 con 0x7f9c84158e20 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 msgr2=0x7f9c70062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 0x7f9c70062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c74006fd0 tx=0x7f9c74008040 comp rx=0 tx=0).stop 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 msgr2=0x7f9c84151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800c590 tx=0x7f9c78007f50 comp rx=0 tx=0).stop 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 shutdown_connections 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060550 0x7f9c70062a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c841591f0 0x7f9c84151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c84158e20 0x7f9c84151650 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 --2- 172.21.15.50:0/3034520036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840ce810 0x7f9c840b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.112+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 >> 172.21.15.50:0/3034520036 conn(0x7f9c84067bb0 msgr2=0x7f9c840c58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.116+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 shutdown_connections 2023-12-19T14:40:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.116+0000 7f9c88fcc700 1 -- 172.21.15.50:0/3034520036 wait complete. 2023-12-19T14:40:10.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-12-19T14:40:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.300+0000 7fe369ee3700 1 Processor -- start 2023-12-19T14:40:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.300+0000 7fe369ee3700 1 -- start start 2023-12-19T14:40:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe369ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe369ee3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe36406a540 0x7fe36406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe369ee3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640ca5a0 con 0x7fe3640cc470 2023-12-19T14:40:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe369ee3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36406dac0 con 0x7fe36406a540 2023-12-19T14:40:10.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe369ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3640b0260 con 0x7fe364158e70 2023-12-19T14:40:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe35bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe36406a540 0x7fe36406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39258/0 (socket says 172.21.15.50:39258) 2023-12-19T14:40:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 -- 172.21.15.50:0/1055912078 learned_addr learned my addr 172.21.15.50:0/1055912078 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:10.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 -- 172.21.15.50:0/1055912078 >> v2:172.21.15.50:3300/0 conn(0x7fe36406a540 msgr2=0x7fe36406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 --2- 172.21.15.50:0/1055912078 >> v2:172.21.15.50:3300/0 conn(0x7fe36406a540 0x7fe36406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 -- 172.21.15.50:0/1055912078 >> v1:172.21.15.125:6789/0 conn(0x7fe3640cc470 legacy=0x7fe3640cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe368ee1700 1 -- 172.21.15.50:0/1055912078 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36406d220 con 0x7fe364158e70 2023-12-19T14:40:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.304+0000 7fe35b7fe700 1 -- 172.21.15.50:0/1055912078 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057899887 0 0) 0x7fe3640ca5a0 con 0x7fe3640cc470 2023-12-19T14:40:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe368ee1700 1 --2- 172.21.15.50:0/1055912078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe360009f20 tx=0x7fe360009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f919b0482a8bf0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe35b7fe700 1 -- 172.21.15.50:0/1055912078 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36000e070 con 0x7fe364158e70 2023-12-19T14:40:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe35b7fe700 1 -- 172.21.15.50:0/1055912078 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36000b9d0 con 0x7fe364158e70 2023-12-19T14:40:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe35b7fe700 1 -- 172.21.15.50:0/1055912078 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3600092e0 con 0x7fe364158e70 2023-12-19T14:40:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 -- 172.21.15.50:0/1055912078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 msgr2=0x7fe3640afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 --2- 172.21.15.50:0/1055912078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe360009f20 tx=0x7fe360009840 comp rx=0 tx=0).stop 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 -- 172.21.15.50:0/1055912078 shutdown_connections 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 --2- 172.21.15.50:0/1055912078 >> v2:172.21.15.50:3300/0 conn(0x7fe36406a540 0x7fe36406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 --2- 172.21.15.50:0/1055912078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe364158e70 0x7fe3640afd20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 -- 172.21.15.50:0/1055912078 >> 172.21.15.50:0/1055912078 conn(0x7fe3640670e0 msgr2=0x7fe3640674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 -- 172.21.15.50:0/1055912078 shutdown_connections 2023-12-19T14:40:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.308+0000 7fe369ee3700 1 -- 172.21.15.50:0/1055912078 wait complete. 2023-12-19T14:40:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.312+0000 7fe369ee3700 1 Processor -- start 2023-12-19T14:40:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.312+0000 7fe369ee3700 1 -- start start 2023-12-19T14:40:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe36406a540 0x7fe3640b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3640cc470 0x7fe3640c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe364169fd0 con 0x7fe364158e70 2023-12-19T14:40:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe364169400 con 0x7fe36406a540 2023-12-19T14:40:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3641696a0 con 0x7fe3640cc470 2023-12-19T14:40:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56648/0 (socket says 172.21.15.50:56648) 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe35bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3640cc470 0x7fe3640c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 -- 172.21.15.50:0/849558301 learned_addr learned my addr 172.21.15.50:0/849558301 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe368ee1700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe36406a540 0x7fe3640b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 -- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3640cc470 msgr2=0x7fe3640c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3640cc470 0x7fe3640c38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 -- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe36406a540 msgr2=0x7fe3640b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe36406a540 0x7fe3640b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 -- 172.21.15.50:0/849558301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360009ae0 con 0x7fe364158e70 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe3696e2700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00c5d0 tx=0x7fe35c007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c011590 con 0x7fe364158e70 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c0116f0 con 0x7fe364158e70 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe364169980 con 0x7fe364158e70 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c011ab0 con 0x7fe364158e70 2023-12-19T14:40:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.316+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3640b59d0 con 0x7fe364158e70 2023-12-19T14:40:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.320+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3640b5bf0 con 0x7fe364158e70 2023-12-19T14:40:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.320+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe35c01b3b0 con 0x7fe364158e70 2023-12-19T14:40:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.320+0000 7fe358ff9700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 0x7fe350062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.324+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fe35c07baa0 con 0x7fe364158e70 2023-12-19T14:40:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.324+0000 7fe368ee1700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 0x7fe350062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.324+0000 7fe368ee1700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 0x7fe350062a00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe360009f20 tx=0x7fe360018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.328+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe35c04c200 con 0x7fe364158e70 2023-12-19T14:40:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.556+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fe3640b2350 con 0x7fe364158e70 2023-12-19T14:40:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.564+0000 7fe358ff9700 1 -- 172.21.15.50:0/849558301 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe3640b2350 con 0x7fe364158e70 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 msgr2=0x7fe350062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 0x7fe350062a00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe360009f20 tx=0x7fe360018000 comp rx=0 tx=0).stop 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 msgr2=0x7fe3640c3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00c5d0 tx=0x7fe35c007d70 comp rx=0 tx=0).stop 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 shutdown_connections 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe350060550 0x7fe350062a00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe364158e70 0x7fe3640c3e00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3640cc470 0x7fe3640c38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 --2- 172.21.15.50:0/849558301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe36406a540 0x7fe3640b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 >> 172.21.15.50:0/849558301 conn(0x7fe3640670e0 msgr2=0x7fe364158210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:10.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 shutdown_connections 2023-12-19T14:40:10.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.568+0000 7fe369ee3700 1 -- 172.21.15.50:0/849558301 wait complete. 2023-12-19T14:40:10.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-12-19T14:40:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.755+0000 7f1aba345700 1 Processor -- start 2023-12-19T14:40:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.755+0000 7f1aba345700 1 -- start start 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1aba345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ab40b3870 0x7f1ab40b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1aba345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab406b0d0 con 0x7f1ab406dc00 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1aba345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab40b4790 con 0x7f1ab4158cc0 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1aba345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40b41d0 con 0x7f1ab40b3870 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ab40b3870 0x7f1ab40b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1ab40b3870 0x7f1ab40b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56652/0 (socket says 172.21.15.50:56652) 2023-12-19T14:40:10.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 -- 172.21.15.50:0/1199460015 learned_addr learned my addr 172.21.15.50:0/1199460015 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:10.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab37fe700 1 -- 172.21.15.50:0/1199460015 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055064921 0 0) 0x7f1ab406b0d0 con 0x7f1ab406dc00 2023-12-19T14:40:10.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab37fe700 1 -- 172.21.15.50:0/1199460015 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa0003570 con 0x7f1ab406dc00 2023-12-19T14:40:10.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 -- 172.21.15.50:0/1199460015 >> v1:172.21.15.125:6789/0 conn(0x7f1ab4158cc0 legacy=0x7f1ab406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 -- 172.21.15.50:0/1199460015 >> v1:172.21.15.50:6789/0 conn(0x7f1ab406dc00 legacy=0x7f1ab406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 -- 172.21.15.50:0/1199460015 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab406a7e0 con 0x7f1ab40b3870 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab88e2700 1 --2- 172.21.15.50:0/1199460015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab40b3c90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009f20 tx=0x7f1aac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc3cbe34bf0c7a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab37fe700 1 -- 172.21.15.50:0/1199460015 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00e070 con 0x7f1ab40b3870 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab37fe700 1 -- 172.21.15.50:0/1199460015 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00b9d0 con 0x7f1ab40b3870 2023-12-19T14:40:10.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.759+0000 7f1ab37fe700 1 -- 172.21.15.50:0/1199460015 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac0092e0 con 0x7f1ab40b3870 2023-12-19T14:40:10.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- 172.21.15.50:0/1199460015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 msgr2=0x7f1ab40b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 --2- 172.21.15.50:0/1199460015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab40b3c90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009f20 tx=0x7f1aac009840 comp rx=0 tx=0).stop 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- 172.21.15.50:0/1199460015 shutdown_connections 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 --2- 172.21.15.50:0/1199460015 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab40b3c90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- 172.21.15.50:0/1199460015 >> 172.21.15.50:0/1199460015 conn(0x7f1ab4067bb0 msgr2=0x7f1ab40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- 172.21.15.50:0/1199460015 shutdown_connections 2023-12-19T14:40:10.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- 172.21.15.50:0/1199460015 wait complete. 2023-12-19T14:40:10.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 Processor -- start 2023-12-19T14:40:10.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.763+0000 7f1aba345700 1 -- start start 2023-12-19T14:40:10.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ab406dc00 0x7f1ab40a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ab4158cc0 0x7f1ab41544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab415cbb0 con 0x7f1ab40b3870 2023-12-19T14:40:10.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab415c8b0 con 0x7f1ab4158cc0 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40af3a0 con 0x7f1ab406dc00 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56658/0 (socket says 172.21.15.50:56658) 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 -- 172.21.15.50:0/1340199980 learned_addr learned my addr 172.21.15.50:0/1340199980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1ab88e2700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ab4158cc0 0x7f1ab41544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1ab3fff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ab406dc00 0x7f1ab40a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 -- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ab406dc00 msgr2=0x7f1ab40a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ab406dc00 0x7f1ab40a9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 -- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ab4158cc0 msgr2=0x7f1ab41544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ab4158cc0 0x7f1ab41544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 -- 172.21.15.50:0/1340199980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac009ae0 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aabfff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c7c0 tx=0x7f1a9c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c010510 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c010670 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c010a10 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab415bf80 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.767+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab415c540 con 0x7f1ab40b3870 2023-12-19T14:40:10.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.771+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab415cd60 con 0x7f1ab40b3870 2023-12-19T14:40:10.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.771+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1a9c01b3b0 con 0x7f1ab40b3870 2023-12-19T14:40:10.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.775+0000 7f1ab0ff9700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 0x7f1a94062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:10.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.775+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f1a9c07bdb0 con 0x7f1ab40b3870 2023-12-19T14:40:10.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.775+0000 7f1ab3fff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 0x7f1a94062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:10.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.775+0000 7f1ab3fff700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 0x7f1a94062a00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40039e0 tx=0x7f1aa4007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:10.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:10.779+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a9c04c5b0 con 0x7f1ab40b3870 2023-12-19T14:40:11.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.015+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f1ab4071120 con 0x7f1ab40b3870 2023-12-19T14:40:11.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.023+0000 7f1ab0ff9700 1 -- 172.21.15.50:0/1340199980 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1a9c017070 con 0x7f1ab40b3870 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 msgr2=0x7f1a94062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 0x7f1a94062a00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40039e0 tx=0x7f1aa4007ec0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 msgr2=0x7f1ab4153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00c7c0 tx=0x7f1a9c007e10 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 shutdown_connections 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ab4158cc0 0x7f1ab41544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a94060550 0x7f1a94062a00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ab40b3870 0x7f1ab4153fa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 --2- 172.21.15.50:0/1340199980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1ab406dc00 0x7f1ab40a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 >> 172.21.15.50:0/1340199980 conn(0x7f1ab4067bb0 msgr2=0x7f1ab4156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:11.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 shutdown_connections 2023-12-19T14:40:11.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.027+0000 7f1aba345700 1 -- 172.21.15.50:0/1340199980 wait complete. 2023-12-19T14:40:11.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-12-19T14:40:11.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.223+0000 7fedcda2e700 1 Processor -- start 2023-12-19T14:40:11.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.223+0000 7fedcda2e700 1 -- start start 2023-12-19T14:40:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedcda2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fedc80b3870 0x7fedc80b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedcda2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc806b0d0 con 0x7fedc8158cc0 2023-12-19T14:40:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedcda2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc80b4790 con 0x7fedc806dc00 2023-12-19T14:40:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedcda2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc80b41d0 con 0x7fedc80b3870 2023-12-19T14:40:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fedc8158cc0 0x7fedc806d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39374/0 (socket says 172.21.15.50:39374) 2023-12-19T14:40:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc6ffd700 1 -- 172.21.15.50:0/1217831611 learned_addr learned my addr 172.21.15.50:0/1217831611 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc77fe700 1 --2- 172.21.15.50:0/1217831611 >> v2:172.21.15.50:3300/0 conn(0x7fedc80b3870 0x7fedc80b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc77fe700 1 -- 172.21.15.50:0/1217831611 >> v1:172.21.15.125:6789/0 conn(0x7fedc806dc00 legacy=0x7fedc806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc5ffb700 1 -- 172.21.15.50:0/1217831611 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488459827 0 0) 0x7fedc80b4790 con 0x7fedc806dc00 2023-12-19T14:40:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc77fe700 1 -- 172.21.15.50:0/1217831611 >> v1:172.21.15.50:6789/0 conn(0x7fedc8158cc0 legacy=0x7fedc806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc77fe700 1 -- 172.21.15.50:0/1217831611 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc806a7e0 con 0x7fedc80b3870 2023-12-19T14:40:11.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc77fe700 1 --2- 172.21.15.50:0/1217831611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc80b3870 0x7fedc80b3c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009e60 tx=0x7fedc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3856051c5130e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc5ffb700 1 -- 172.21.15.50:0/1217831611 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc000e070 con 0x7fedc80b3870 2023-12-19T14:40:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc5ffb700 1 -- 172.21.15.50:0/1217831611 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc000bc30 con 0x7fedc80b3870 2023-12-19T14:40:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.227+0000 7fedc5ffb700 1 -- 172.21.15.50:0/1217831611 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0012430 con 0x7fedc80b3870 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 -- 172.21.15.50:0/1217831611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc80b3870 msgr2=0x7fedc80b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 --2- 172.21.15.50:0/1217831611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc80b3870 0x7fedc80b3c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009e60 tx=0x7fedc0009780 comp rx=0 tx=0).stop 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 -- 172.21.15.50:0/1217831611 shutdown_connections 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 --2- 172.21.15.50:0/1217831611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc80b3870 0x7fedc80b3c90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 -- 172.21.15.50:0/1217831611 >> 172.21.15.50:0/1217831611 conn(0x7fedc8067bb0 msgr2=0x7fedc80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 -- 172.21.15.50:0/1217831611 shutdown_connections 2023-12-19T14:40:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.231+0000 7fedcda2e700 1 -- 172.21.15.50:0/1217831611 wait complete. 2023-12-19T14:40:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 Processor -- start 2023-12-19T14:40:11.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 -- start start 2023-12-19T14:40:11.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 0x7fedc80afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc8158cc0 0x7fedc8154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc80b0230 con 0x7fedc806dc00 2023-12-19T14:40:11.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc815c490 con 0x7fedc8158cc0 2023-12-19T14:40:11.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedcda2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8155250 con 0x7fedc80b3870 2023-12-19T14:40:11.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedc6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedc6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56680/0 (socket says 172.21.15.50:56680) 2023-12-19T14:40:11.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.235+0000 7fedc6ffd700 1 -- 172.21.15.50:0/3167262518 learned_addr learned my addr 172.21.15.50:0/3167262518 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc67fc700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 0x7fedc80afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc77fe700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc8158cc0 0x7fedc8154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 -- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 msgr2=0x7fedc80afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 0x7fedc80afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 -- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc8158cc0 msgr2=0x7fedc8154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc8158cc0 0x7fedc8154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 -- 172.21.15.50:0/3167262518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc0009a20 con 0x7fedc806dc00 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc67fc700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 0x7fedc80afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedc6ffd700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fedb800c560 tx=0x7fedb8001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8005b30 con 0x7fedc806dc00 2023-12-19T14:40:11.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb8005cb0 con 0x7fedc806dc00 2023-12-19T14:40:11.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc815b880 con 0x7fedc806dc00 2023-12-19T14:40:11.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8007320 con 0x7fedc806dc00 2023-12-19T14:40:11.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc815c120 con 0x7fedc806dc00 2023-12-19T14:40:11.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.239+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc815bc70 con 0x7fedc806dc00 2023-12-19T14:40:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.243+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fedb8007690 con 0x7fedc806dc00 2023-12-19T14:40:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.243+0000 7fedaeffd700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 0x7fedb4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.243+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fedb807b9e0 con 0x7fedc806dc00 2023-12-19T14:40:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.243+0000 7fedc67fc700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 0x7fedb4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.247+0000 7fedc67fc700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 0x7fedb4062a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedc815d110 tx=0x7fedbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.247+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fedb804c1e0 con 0x7fedc806dc00 2023-12-19T14:40:11.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.483+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fedc8071120 con 0x7fedc806dc00 2023-12-19T14:40:11.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.491+0000 7fedaeffd700 1 -- 172.21.15.50:0/3167262518 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fedb804f800 con 0x7fedc806dc00 2023-12-19T14:40:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 msgr2=0x7fedb4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 0x7fedb4062a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedc815d110 tx=0x7fedbc009040 comp rx=0 tx=0).stop 2023-12-19T14:40:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 msgr2=0x7fedc815c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fedb800c560 tx=0x7fedb8001020 comp rx=0 tx=0).stop 2023-12-19T14:40:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 shutdown_connections 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fedc8158cc0 0x7fedc8154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fedb4060550 0x7fedb4062a00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fedc80b3870 0x7fedc80afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 --2- 172.21.15.50:0/3167262518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fedc806dc00 0x7fedc815c920 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 >> 172.21.15.50:0/3167262518 conn(0x7fedc8067bb0 msgr2=0x7fedc8157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 shutdown_connections 2023-12-19T14:40:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.495+0000 7fedcda2e700 1 -- 172.21.15.50:0/3167262518 wait complete. 2023-12-19T14:40:11.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-12-19T14:40:11.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.687+0000 7f14a88cd700 1 Processor -- start 2023-12-19T14:40:11.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.687+0000 7f14a88cd700 1 -- start start 2023-12-19T14:40:11.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a88cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f14a406a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a88cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14a40a98d0 0x7f14a406ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a88cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a406e160 con 0x7f14a4158e50 2023-12-19T14:40:11.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a88cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a406ced0 con 0x7f14a40a98d0 2023-12-19T14:40:11.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a88cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a406d0c0 con 0x7f14a4159220 2023-12-19T14:40:11.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f14a4158e50 0x7f14a406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39380/0 (socket says 172.21.15.50:39380) 2023-12-19T14:40:11.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a259c700 1 -- 172.21.15.50:0/3221745936 learned_addr learned my addr 172.21.15.50:0/3221745936 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:11.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f14a406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a2d9d700 1 --2- 172.21.15.50:0/3221745936 >> v2:172.21.15.125:3300/0 conn(0x7f14a40a98d0 0x7f14a406ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543161512 0 0) 0x7f14a406e160 con 0x7f14a4158e50 2023-12-19T14:40:11.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1490003570 con 0x7f14a4158e50 2023-12-19T14:40:11.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1011478127 0 0) 0x7f1490003570 con 0x7f14a4158e50 2023-12-19T14:40:11.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 msgr2=0x7f14a406a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:11.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f14a406a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 >> v2:172.21.15.125:3300/0 conn(0x7f14a40a98d0 msgr2=0x7f14a406ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 --2- 172.21.15.50:0/3221745936 >> v2:172.21.15.125:3300/0 conn(0x7f14a40a98d0 0x7f14a406ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a406d3a0 con 0x7f14a4158e50 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f1494001470 con 0x7f14a4158e50 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f14a406a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.691+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 >> v1:172.21.15.50:6789/0 conn(0x7f14a4158e50 legacy=0x7f14a406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f1490003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1490003fa0 0x7f1490004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490004d50 0x7f14900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a406d0c0 con 0x7f1490003fa0 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149000ba60 con 0x7f1490004d50 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149000bba0 con 0x7f14a4159220 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1490003fa0 0x7f1490004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a259c700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f1490003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a2d9d700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490004d50 0x7f14900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 -- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 msgr2=0x7f1490003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f1490003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 -- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490004d50 msgr2=0x7f14900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490004d50 0x7f14900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 -- 172.21.15.50:0/3221745936 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c009420 con 0x7f1490003fa0 2023-12-19T14:40:11.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a259c700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a4159220 0x7f1490003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f1499d9b700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1490003fa0 0x7f1490004810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f148c009d90 tx=0x7f148c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0d445e39d3e85c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c009740 con 0x7f1490003fa0 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c00e920 con 0x7f1490003fa0 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a1d9b700 1 -- 172.21.15.50:0/3221745936 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00ecc0 con 0x7f1490003fa0 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 -- 172.21.15.50:0/3221745936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1490003fa0 msgr2=0x7f1490004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 --2- 172.21.15.50:0/3221745936 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1490003fa0 0x7f1490004810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f148c009d90 tx=0x7f148c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a2d9d700 1 -- 172.21.15.50:0/3221745936 reap_dead start 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 -- 172.21.15.50:0/3221745936 shutdown_connections 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 -- 172.21.15.50:0/3221745936 >> 172.21.15.50:0/3221745936 conn(0x7f14a4067b60 msgr2=0x7f14a40ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 -- 172.21.15.50:0/3221745936 shutdown_connections 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.695+0000 7f14a88cd700 1 -- 172.21.15.50:0/3221745936 wait complete. 2023-12-19T14:40:11.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 Processor -- start 2023-12-19T14:40:11.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 -- start start 2023-12-19T14:40:11.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490003fa0 0x7f14a40accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a40ad230 0x7f14a415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415aae0 con 0x7f14a415ce90 2023-12-19T14:40:11.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415ad30 con 0x7f1490003fa0 2023-12-19T14:40:11.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.699+0000 7f14a88cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415b010 con 0x7f14a40ad230 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56710/0 (socket says 172.21.15.50:56710) 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 -- 172.21.15.50:0/1578406789 learned_addr learned my addr 172.21.15.50:0/1578406789 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a259c700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490003fa0 0x7f14a40accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f1499d9b700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a40ad230 0x7f14a415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 -- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a40ad230 msgr2=0x7f14a415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a40ad230 0x7f14a415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 -- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490003fa0 msgr2=0x7f14a40accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490003fa0 0x7f14a40accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 -- 172.21.15.50:0/1578406789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a415b2f0 con 0x7f14a415ce90 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a2d9d700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f14a406d200 tx=0x7f149c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00e070 con 0x7f14a415ce90 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c011c30 con 0x7f14a415ce90 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c016470 con 0x7f14a415ce90 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a415b5d0 con 0x7f14a415ce90 2023-12-19T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a415c250 con 0x7f14a415ce90 2023-12-19T14:40:11.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.703+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a4071120 con 0x7f14a415ce90 2023-12-19T14:40:11.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.707+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f149c016860 con 0x7f14a415ce90 2023-12-19T14:40:11.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.711+0000 7f149affd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:11.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.711+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f149c07d240 con 0x7f14a415ce90 2023-12-19T14:40:11.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.711+0000 7f14a259c700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:11.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.711+0000 7f14a259c700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1494001ba0 tx=0x7f1494001ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:11.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.715+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f149c0514f0 con 0x7f14a415ce90 2023-12-19T14:40:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.947+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f14a406d680 con 0x7f14a415ce90 2023-12-19T14:40:11.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.955+0000 7f149affd700 1 -- 172.21.15.50:0/1578406789 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f149c051080 con 0x7f14a415ce90 2023-12-19T14:40:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 msgr2=0x7f1484062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1494001ba0 tx=0x7f1494001ca0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 msgr2=0x7f14a415baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f14a406d200 tx=0x7f149c00b040 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 shutdown_connections 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1484060550 0x7f1484062a00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14a415ce90 0x7f14a415baf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f14a40ad230 0x7f14a415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 --2- 172.21.15.50:0/1578406789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1490003fa0 0x7f14a40accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 >> 172.21.15.50:0/1578406789 conn(0x7f14a4067b60 msgr2=0x7f14a406a740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 shutdown_connections 2023-12-19T14:40:11.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:11.959+0000 7f14a88cd700 1 -- 172.21.15.50:0/1578406789 wait complete. 2023-12-19T14:40:11.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:12.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 Processor -- start 2023-12-19T14:40:12.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 -- start start 2023-12-19T14:40:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c0ccb40 con 0x7f308c153290 2023-12-19T14:40:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c0cbf30 con 0x7f308c0ccfd0 2023-12-19T14:40:12.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3090ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0cc210 con 0x7f308c0b0110 2023-12-19T14:40:12.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3089d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3089d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:45700/0 (socket says 172.21.15.50:45700) 2023-12-19T14:40:12.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f3089d9b700 1 -- 172.21.15.50:0/3646263018 learned_addr learned my addr 172.21.15.50:0/3646263018 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f308a59c700 1 --2- 172.21.15.50:0/3646263018 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.155+0000 7f308ad9d700 1 --1- 172.21.15.50:0/3646263018 >> v1:172.21.15.125:6789/0 conn(0x7f308c153290 0x7f308c1536d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:37602/0 (socket says 172.21.15.50:37602) 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308959a700 1 -- 172.21.15.50:0/3646263018 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443334629 0 0) 0x7f308c0ccb40 con 0x7f308c153290 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308a59c700 1 -- 172.21.15.50:0/3646263018 >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 msgr2=0x7f308c15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308a59c700 1 --2- 172.21.15.50:0/3646263018 >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308a59c700 1 -- 172.21.15.50:0/3646263018 >> v1:172.21.15.125:6789/0 conn(0x7f308c153290 legacy=0x7f308c1536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308a59c700 1 -- 172.21.15.50:0/3646263018 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0cc4f0 con 0x7f308c0b0110 2023-12-19T14:40:12.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3089d9b700 1 --2- 172.21.15.50:0/3646263018 >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:12.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308a59c700 1 --2- 172.21.15.50:0/3646263018 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0b04e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f307c0098e0 tx=0x7f307c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b254953dd4d4a209 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308959a700 1 -- 172.21.15.50:0/3646263018 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c00b640 con 0x7f308c0b0110 2023-12-19T14:40:12.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308959a700 1 -- 172.21.15.50:0/3646263018 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c00b7a0 con 0x7f308c0b0110 2023-12-19T14:40:12.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f308959a700 1 -- 172.21.15.50:0/3646263018 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c00bb40 con 0x7f308c0b0110 2023-12-19T14:40:12.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 -- 172.21.15.50:0/3646263018 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 msgr2=0x7f308c0b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 --2- 172.21.15.50:0/3646263018 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0b04e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f307c0098e0 tx=0x7f307c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 -- 172.21.15.50:0/3646263018 shutdown_connections 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 --2- 172.21.15.50:0/3646263018 >> v2:172.21.15.125:3300/0 conn(0x7f308c0ccfd0 0x7f308c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 --2- 172.21.15.50:0/3646263018 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0b04e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 -- 172.21.15.50:0/3646263018 >> 172.21.15.50:0/3646263018 conn(0x7f308c06c7c0 msgr2=0x7f308c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 -- 172.21.15.50:0/3646263018 shutdown_connections 2023-12-19T14:40:12.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.159+0000 7f3090ef0700 1 -- 172.21.15.50:0/3646263018 wait complete. 2023-12-19T14:40:12.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 Processor -- start 2023-12-19T14:40:12.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 -- start start 2023-12-19T14:40:12.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 0x7f308c16cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c068be0 con 0x7f308c0ccfd0 2023-12-19T14:40:12.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c068dd0 con 0x7f308c153290 2023-12-19T14:40:12.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.163+0000 7f3090ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c069070 con 0x7f308c0b0110 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56722/0 (socket says 172.21.15.50:56722) 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f308a59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 -- 172.21.15.50:0/2681451150 learned_addr learned my addr 172.21.15.50:0/2681451150 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f308ad9d700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 0x7f308c16cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 -- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 msgr2=0x7f308c0c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 -- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 msgr2=0x7f308c16cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 0x7f308c16cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 -- 172.21.15.50:0/2681451150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30800094f0 con 0x7f308c0ccfd0 2023-12-19T14:40:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f308ad9d700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 0x7f308c16cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3089d9b700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f308000ec50 tx=0x7f3080009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308000c920 con 0x7f308c0ccfd0 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308000ca80 con 0x7f308c0ccfd0 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c009420 con 0x7f308c0ccfd0 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308000ce00 con 0x7f308c0ccfd0 2023-12-19T14:40:12.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c069630 con 0x7f308c0ccfd0 2023-12-19T14:40:12.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.167+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c071120 con 0x7f308c0ccfd0 2023-12-19T14:40:12.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.171+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3080013680 con 0x7f308c0ccfd0 2023-12-19T14:40:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.171+0000 7f307a7fc700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 0x7f306c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.171+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f308007c9c0 con 0x7f308c0ccfd0 2023-12-19T14:40:12.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.175+0000 7f308a59c700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 0x7f306c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.175+0000 7f308a59c700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 0x7f306c062a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f307c0093f0 tx=0x7f307c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.179+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f308004d120 con 0x7f308c0ccfd0 2023-12-19T14:40:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.351+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f308c0c8700 con 0x7f308c0ccfd0 2023-12-19T14:40:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.355+0000 7f307a7fc700 1 -- 172.21.15.50:0/2681451150 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+12339 (secure 0 0 0) 0x7f3080050740 con 0x7f308c0ccfd0 2023-12-19T14:40:12.360 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:12.360 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":34,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:40:08.735029+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:12.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 msgr2=0x7f306c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 0x7f306c062a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f307c0093f0 tx=0x7f307c017000 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 msgr2=0x7f308c16c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f308000ec50 tx=0x7f3080009250 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 shutdown_connections 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308c153290 0x7f308c16cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f306c060550 0x7f306c062a00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f308c0ccfd0 0x7f308c16c800 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 --2- 172.21.15.50:0/2681451150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308c0b0110 0x7f308c0c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 >> 172.21.15.50:0/2681451150 conn(0x7f308c06c7c0 msgr2=0x7f308c0c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 shutdown_connections 2023-12-19T14:40:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.359+0000 7f3090ef0700 1 -- 172.21.15.50:0/2681451150 wait complete. 2023-12-19T14:40:12.385 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:40:12.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:40:12.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.555+0000 7ff272ca3700 1 Processor -- start 2023-12-19T14:40:12.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.555+0000 7ff272ca3700 1 -- start start 2023-12-19T14:40:12.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff26c06aeb0 0x7ff26c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 0x7ff26c159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c06c850 con 0x7ff26c06aeb0 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c06b470 con 0x7ff26c1595a0 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c06b750 con 0x7ff26c158c90 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff26c06aeb0 0x7ff26c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff26c06aeb0 0x7ff26c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56728/0 (socket says 172.21.15.50:56728) 2023-12-19T14:40:12.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 -- 172.21.15.50:0/2532091651 learned_addr learned my addr 172.21.15.50:0/2532091651 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff271ca1700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 -- 172.21.15.50:0/2532091651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 msgr2=0x7ff26c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:12.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 -- 172.21.15.50:0/2532091651 >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 msgr2=0x7ff26c159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2724a2700 1 --2- 172.21.15.50:0/2532091651 >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 0x7ff26c159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 --2- 172.21.15.50:0/2532091651 >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 0x7ff26c159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 -- 172.21.15.50:0/2532091651 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c06ba30 con 0x7ff26c06aeb0 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff271ca1700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2724a2700 1 --2- 172.21.15.50:0/2532091651 >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 0x7ff26c159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff2714a0700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c159060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0ca300 tx=0x7ff25c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955fff89cd85e0a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff270c9f700 1 -- 172.21.15.50:0/2532091651 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00b860 con 0x7ff26c06aeb0 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff270c9f700 1 -- 172.21.15.50:0/2532091651 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c00b9c0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff270c9f700 1 -- 172.21.15.50:0/2532091651 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00bd60 con 0x7ff26c06aeb0 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- 172.21.15.50:0/2532091651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 msgr2=0x7ff26c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c159060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0ca300 tx=0x7ff25c009250 comp rx=0 tx=0).stop 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- 172.21.15.50:0/2532091651 shutdown_connections 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c159060 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- 172.21.15.50:0/2532091651 >> v2:172.21.15.125:3300/0 conn(0x7ff26c1595a0 0x7ff26c159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 --2- 172.21.15.50:0/2532091651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- 172.21.15.50:0/2532091651 >> 172.21.15.50:0/2532091651 conn(0x7ff26c067bb0 msgr2=0x7ff26c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- 172.21.15.50:0/2532091651 shutdown_connections 2023-12-19T14:40:12.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.559+0000 7ff272ca3700 1 -- 172.21.15.50:0/2532091651 wait complete. 2023-12-19T14:40:12.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 Processor -- start 2023-12-19T14:40:12.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 -- start start 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff26c1595a0 0x7ff26c0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c087ed0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c1503e0 con 0x7ff26c1595a0 2023-12-19T14:40:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.563+0000 7ff272ca3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c0872c0 con 0x7ff26c158c90 2023-12-19T14:40:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56738/0 (socket says 172.21.15.50:56738) 2023-12-19T14:40:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 -- 172.21.15.50:0/1008351133 learned_addr learned my addr 172.21.15.50:0/1008351133 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff2714a0700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff2724a2700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff26c1595a0 0x7ff26c0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 -- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 msgr2=0x7ff26c14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c14fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 -- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff26c1595a0 msgr2=0x7ff26c0b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff26c1595a0 0x7ff26c0b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 -- 172.21.15.50:0/1008351133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0094f0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff271ca1700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff26800c880 tx=0x7ff268008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268011600 con 0x7ff26c06aeb0 2023-12-19T14:40:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268011760 con 0x7ff26c06aeb0 2023-12-19T14:40:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268011ae0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c0875a0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c087b60 con 0x7ff26c06aeb0 2023-12-19T14:40:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.567+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c04be40 con 0x7ff26c06aeb0 2023-12-19T14:40:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.571+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff268010350 con 0x7ff26c06aeb0 2023-12-19T14:40:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.571+0000 7ff261ffb700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 0x7ff250062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.571+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7ff26807c550 con 0x7ff26c06aeb0 2023-12-19T14:40:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.571+0000 7ff2714a0700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 0x7ff250062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.571+0000 7ff2714a0700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 0x7ff250062a00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0ca300 tx=0x7ff25c017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.579+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff2680507e0 con 0x7ff26c06aeb0 2023-12-19T14:40:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.751+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff26c06bd10 con 0x7ff26c06aeb0 2023-12-19T14:40:12.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.751+0000 7ff261ffb700 1 -- 172.21.15.50:0/1008351133 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff268050370 con 0x7ff26c06aeb0 2023-12-19T14:40:12.756 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:40:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 msgr2=0x7ff250062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 0x7ff250062a00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0ca300 tx=0x7ff25c017000 comp rx=0 tx=0).stop 2023-12-19T14:40:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 msgr2=0x7ff26c08ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff26800c880 tx=0x7ff268008040 comp rx=0 tx=0).stop 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 shutdown_connections 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff26c1595a0 0x7ff26c0b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff250060550 0x7ff250062a00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff26c158c90 0x7ff26c14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 --2- 172.21.15.50:0/1008351133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff26c06aeb0 0x7ff26c08ad90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 >> 172.21.15.50:0/1008351133 conn(0x7ff26c067bb0 msgr2=0x7ff26c0ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 shutdown_connections 2023-12-19T14:40:12.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.755+0000 7ff272ca3700 1 -- 172.21.15.50:0/1008351133 wait complete. 2023-12-19T14:40:12.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:40:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 Processor -- start 2023-12-19T14:40:12.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 -- start start 2023-12-19T14:40:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0660140ae0 0x7f06600680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660068630 0x7f06600ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066006e150 con 0x7f0660140710 2023-12-19T14:40:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0660068ab0 con 0x7f0660140ae0 2023-12-19T14:40:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.955+0000 7f0666185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066006d540 con 0x7f0660068630 2023-12-19T14:40:12.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0660140ae0 0x7f06600680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0660140ae0 0x7f06600680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56752/0 (socket says 172.21.15.50:56752) 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 -- 172.21.15.50:0/1455145303 learned_addr learned my addr 172.21.15.50:0/1455145303 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f065ffff700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660068630 0x7f06600ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 -- 172.21.15.50:0/1455145303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660068630 msgr2=0x7f06600ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660068630 0x7f06600ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 -- 172.21.15.50:0/1455145303 >> v1:172.21.15.50:6789/0 conn(0x7f0660140710 legacy=0x7f06600c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 -- 172.21.15.50:0/1455145303 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066006d820 con 0x7f0660140ae0 2023-12-19T14:40:12.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f0657fff700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660140ae0 0x7f06600680f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f06480099b0 tx=0x7f0648009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0a66b9239508b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f065effd700 1 -- 172.21.15.50:0/1455145303 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064800b860 con 0x7f0660140ae0 2023-12-19T14:40:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f065effd700 1 -- 172.21.15.50:0/1455145303 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064800b9c0 con 0x7f0660140ae0 2023-12-19T14:40:12.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.959+0000 7f065effd700 1 -- 172.21.15.50:0/1455145303 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064800bd60 con 0x7f0660140ae0 2023-12-19T14:40:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- 172.21.15.50:0/1455145303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660140ae0 msgr2=0x7f06600680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660140ae0 0x7f06600680f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f06480099b0 tx=0x7f0648009250 comp rx=0 tx=0).stop 2023-12-19T14:40:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- 172.21.15.50:0/1455145303 shutdown_connections 2023-12-19T14:40:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660068630 0x7f06600ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 --2- 172.21.15.50:0/1455145303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660140ae0 0x7f06600680f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- 172.21.15.50:0/1455145303 >> 172.21.15.50:0/1455145303 conn(0x7f06600c8850 msgr2=0x7f06600c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- 172.21.15.50:0/1455145303 shutdown_connections 2023-12-19T14:40:12.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- 172.21.15.50:0/1455145303 wait complete. 2023-12-19T14:40:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 Processor -- start 2023-12-19T14:40:12.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.963+0000 7f0666185700 1 -- start start 2023-12-19T14:40:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660140710 0x7f06600cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0660140ae0 0x7f06600c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06601330c0 con 0x7f0660068630 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06600af850 con 0x7f0660140ae0 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0666185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06600aec40 con 0x7f0660140710 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56762/0 (socket says 172.21.15.50:56762) 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 -- 172.21.15.50:0/4156303728 learned_addr learned my addr 172.21.15.50:0/4156303728 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065ffff700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0660140ae0 0x7f06600c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f0657fff700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660140710 0x7f06600cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 -- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660140710 msgr2=0x7f06600cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660140710 0x7f06600cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 -- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0660140ae0 msgr2=0x7f06600c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0660140ae0 0x7f06600c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:12.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 -- 172.21.15.50:0/4156303728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06480094f0 con 0x7f0660068630 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f065f7fe700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f065000c4e0 tx=0x7f065000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650011590 con 0x7f0660068630 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0650011710 con 0x7f0660068630 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.967+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650011ad0 con 0x7f0660068630 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.971+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06600aef20 con 0x7f0660068630 2023-12-19T14:40:12.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.971+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06600af4e0 con 0x7f0660068630 2023-12-19T14:40:12.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.971+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660071120 con 0x7f0660068630 2023-12-19T14:40:12.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.971+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f065001b3b0 con 0x7f0660068630 2023-12-19T14:40:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.975+0000 7f06577fe700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 0x7f0640062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:12.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.975+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f065007b9a0 con 0x7f0660068630 2023-12-19T14:40:12.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.975+0000 7f0657fff700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 0x7f0640062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.975+0000 7f0657fff700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 0x7f0640062a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06480099b0 tx=0x7f0648017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:12.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:12.979+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f065004c100 con 0x7f0660068630 2023-12-19T14:40:13.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.151+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f06600642a0 con 0x7f0660068630 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.155+0000 7f06577fe700 1 -- 172.21.15.50:0/4156303728 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+65 (secure 0 0 0) 0x7f065004f720 con 0x7f0660068630 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:40:13.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:13.159 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 msgr2=0x7f0640062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 0x7f0640062a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06480099b0 tx=0x7f0648017000 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 msgr2=0x7f06600cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f065000c4e0 tx=0x7f065000c7f0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 shutdown_connections 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0660140ae0 0x7f06600c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0640060550 0x7f0640062a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0660140710 0x7f06600cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 --2- 172.21.15.50:0/4156303728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0660068630 0x7f06600cae20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 >> 172.21.15.50:0/4156303728 conn(0x7f06600c8850 msgr2=0x7f06600ad2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:13.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 shutdown_connections 2023-12-19T14:40:13.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.159+0000 7f0666185700 1 -- 172.21.15.50:0/4156303728 wait complete. 2023-12-19T14:40:13.185 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:40:13.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:40:13.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.355+0000 7ff9776cd700 1 Processor -- start 2023-12-19T14:40:13.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.355+0000 7ff9776cd700 1 -- start start 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.359+0000 7ff9776cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 0x7ff97006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.359+0000 7ff9776cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9700add70 con 0x7ff97015cce0 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.359+0000 7ff9776cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff970159fa0 con 0x7ff97015c490 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.359+0000 7ff9776cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff970159dc0 con 0x7ff97015c860 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff975c6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff97015cce0 0x7ff9700ae200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39394/0 (socket says 172.21.15.50:39394) 2023-12-19T14:40:13.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff975c6a700 1 -- 172.21.15.50:0/1059470571 learned_addr learned my addr 172.21.15.50:0/1059470571 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:13.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff974c68700 1 --2- 172.21.15.50:0/1059470571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 0x7ff97006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:13.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231837080 0 0) 0x7ff9700add70 con 0x7ff97015cce0 2023-12-19T14:40:13.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff95c003570 con 0x7ff97015cce0 2023-12-19T14:40:13.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34429409 0 0) 0x7ff970159fa0 con 0x7ff97015c490 2023-12-19T14:40:13.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9700add70 con 0x7ff97015c490 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff974c68700 1 -- 172.21.15.50:0/1059470571 >> v1:172.21.15.125:6789/0 conn(0x7ff97015c490 legacy=0x7ff9701596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff974c68700 1 -- 172.21.15.50:0/1059470571 >> v1:172.21.15.50:6789/0 conn(0x7ff97015cce0 legacy=0x7ff9700ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff974c68700 1 -- 172.21.15.50:0/1059470571 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97015a280 con 0x7ff97015c860 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff974c68700 1 --2- 172.21.15.50:0/1059470571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 0x7ff97006d050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff960009910 tx=0x7ff9600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27eeaf292ea1491c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960011630 con 0x7ff97015c860 2023-12-19T14:40:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff960011790 con 0x7ff97015c860 2023-12-19T14:40:13.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff967fff700 1 -- 172.21.15.50:0/1059470571 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960011b30 con 0x7ff97015c860 2023-12-19T14:40:13.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff9776cd700 1 -- 172.21.15.50:0/1059470571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 msgr2=0x7ff97006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1059470571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 0x7ff97006d050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff960009910 tx=0x7ff9600091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff9776cd700 1 -- 172.21.15.50:0/1059470571 shutdown_connections 2023-12-19T14:40:13.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1059470571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015c860 0x7ff97006d050 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.363+0000 7ff9776cd700 1 -- 172.21.15.50:0/1059470571 >> 172.21.15.50:0/1059470571 conn(0x7ff9700ccc20 msgr2=0x7ff9700cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:13.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.367+0000 7ff9776cd700 1 -- 172.21.15.50:0/1059470571 shutdown_connections 2023-12-19T14:40:13.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.367+0000 7ff9776cd700 1 -- 172.21.15.50:0/1059470571 wait complete. 2023-12-19T14:40:13.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.367+0000 7ff9776cd700 1 Processor -- start 2023-12-19T14:40:13.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.367+0000 7ff9776cd700 1 -- start start 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 0x7ff970169970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 0x7ff9700b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700a89f0 con 0x7ff97015c860 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700af420 con 0x7ff97015c490 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff9776cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700aeb80 con 0x7ff97015cce0 2023-12-19T14:40:13.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56770/0 (socket says 172.21.15.50:56770) 2023-12-19T14:40:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 -- 172.21.15.50:0/1940232400 learned_addr learned my addr 172.21.15.50:0/1940232400 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff975469700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 0x7ff970169970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff975c6a700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 0x7ff9700b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:13.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 -- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 msgr2=0x7ff9700b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 0x7ff9700b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 -- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 msgr2=0x7ff970169970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 0x7ff970169970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff974c68700 1 -- 172.21.15.50:0/1940232400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960009420 con 0x7ff97015c860 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff975469700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 0x7ff970169970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.371+0000 7ff975c6a700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 0x7ff9700b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff974c68700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff960009fd0 tx=0x7ff960017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96001f070 con 0x7ff97015c860 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff960000bc0 con 0x7ff97015c860 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9700aee60 con 0x7ff97015c860 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960025420 con 0x7ff97015c860 2023-12-19T14:40:13.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff970156d70 con 0x7ff97015c860 2023-12-19T14:40:13.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.375+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970157020 con 0x7ff97015c860 2023-12-19T14:40:13.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.379+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff960023070 con 0x7ff97015c860 2023-12-19T14:40:13.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.379+0000 7ff9657fa700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 0x7ff954062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:13.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.379+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5173+0+0 (secure 0 0 0) 0x7ff96007da30 con 0x7ff97015c860 2023-12-19T14:40:13.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.379+0000 7ff975469700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 0x7ff954062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:13.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.379+0000 7ff975469700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 0x7ff954062a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9700c8c40 tx=0x7ff96c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:13.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.383+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff96004e190 con 0x7ff97015c860 2023-12-19T14:40:13.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.563+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff9700a8ba0 con 0x7ff97015c860 2023-12-19T14:40:13.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.963+0000 7ff9657fa700 1 -- 172.21.15.50:0/1940232400 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff9700a8ba0 con 0x7ff97015c860 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 msgr2=0x7ff954062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 0x7ff954062a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9700c8c40 tx=0x7ff96c009040 comp rx=0 tx=0).stop 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 msgr2=0x7ff970156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff960009fd0 tx=0x7ff960017000 comp rx=0 tx=0).stop 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 shutdown_connections 2023-12-19T14:40:13.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff954060580 0x7ff954062a30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff97015cce0 0x7ff9700b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff97015c860 0x7ff970156830 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 --2- 172.21.15.50:0/1940232400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff97015c490 0x7ff970169970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 >> 172.21.15.50:0/1940232400 conn(0x7ff9700ccc20 msgr2=0x7ff9700690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 shutdown_connections 2023-12-19T14:40:13.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:13.967+0000 7ff9776cd700 1 -- 172.21.15.50:0/1940232400 wait complete. 2023-12-19T14:40:13.976 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:40:13.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:40:14.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.159+0000 7fa1d9a0c700 1 Processor -- start 2023-12-19T14:40:14.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.159+0000 7fa1d9a0c700 1 -- start start 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d9a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1d406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d9a0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40cc7e0 con 0x7fa1d4142680 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d9a0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40cbbf0 con 0x7fa1d40ad650 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d9a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40cbeb0 con 0x7fa1d4142a50 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d8a0a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa1d4142680 0x7fa1d40c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39404/0 (socket says 172.21.15.50:39404) 2023-12-19T14:40:14.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d8a0a700 1 -- 172.21.15.50:0/461037719 learned_addr learned my addr 172.21.15.50:0/461037719 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:14.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1d406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868703555 0 0) 0x7fa1d40cc7e0 con 0x7fa1d4142680 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c4003570 con 0x7fa1d4142680 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457296971 0 0) 0x7fa1c4003570 con 0x7fa1d4142680 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 msgr2=0x7fa1d406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1d406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:14.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 >> v1:172.21.15.125:6789/0 conn(0x7fa1d40ad650 legacy=0x7fa1d40ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40cc190 con 0x7fa1d4142680 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fa1c80017f0 con 0x7fa1d4142680 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 >> v1:172.21.15.50:6789/0 conn(0x7fa1d4142680 legacy=0x7fa1d40c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c4003a90 0x7fa1c4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40cbeb0 con 0x7fa1c4004d50 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c400bb40 con 0x7fa1c4003a90 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c400bd00 con 0x7fa1d4142a50 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d8a0a700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c4003a90 0x7fa1c4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d920b700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.163+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 -- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 msgr2=0x7fa1c4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d4142a50 0x7fa1c4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 -- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c4003a90 msgr2=0x7fa1c4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c4003a90 0x7fa1c4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 -- 172.21.15.50:0/461037719 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c4003570 con 0x7fa1c4004d50 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1c40051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0019a0 tx=0x7fa1cc007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b4272dc45d594 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc00c5d0 con 0x7fa1c4004d50 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc00c750 con 0x7fa1c4004d50 2023-12-19T14:40:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d37fe700 1 -- 172.21.15.50:0/461037719 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc00cb10 con 0x7fa1c4004d50 2023-12-19T14:40:14.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/461037719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 msgr2=0x7fa1c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:14.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/461037719 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1c40051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0019a0 tx=0x7fa1cc007d10 comp rx=0 tx=0).stop 2023-12-19T14:40:14.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d920b700 1 -- 172.21.15.50:0/461037719 reap_dead start 2023-12-19T14:40:14.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/461037719 shutdown_connections 2023-12-19T14:40:14.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/461037719 >> 172.21.15.50:0/461037719 conn(0x7fa1d40ccb30 msgr2=0x7fa1d40ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:14.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/461037719 shutdown_connections 2023-12-19T14:40:14.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.167+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/461037719 wait complete. 2023-12-19T14:40:14.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 Processor -- start 2023-12-19T14:40:14.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 -- start start 2023-12-19T14:40:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 0x7fa1d40c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 0x7fa1d40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40aeda0 con 0x7fa1c4004d50 2023-12-19T14:40:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40c6b60 con 0x7fa1d40ae3f0 2023-12-19T14:40:14.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d9a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40c4050 con 0x7fa1d40c3be0 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.171+0000 7fa1d8a0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56796/0 (socket says 172.21.15.50:56796) 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 -- 172.21.15.50:0/829334576 learned_addr learned my addr 172.21.15.50:0/829334576 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 0x7fa1d40c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d920b700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 0x7fa1d40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 -- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 msgr2=0x7fa1d40c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 0x7fa1d40c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 -- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 msgr2=0x7fa1d40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 0x7fa1d40ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 -- 172.21.15.50:0/829334576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40c42e0 con 0x7fa1c4004d50 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 0x7fa1d40c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d920b700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 0x7fa1d40ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d8a0a700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8007ac0 tx=0x7fa1d4159580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c800ae20 con 0x7fa1c4004d50 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c80133f0 con 0x7fa1c4004d50 2023-12-19T14:40:14.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c80137b0 con 0x7fa1c4004d50 2023-12-19T14:40:14.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d40c45c0 con 0x7fa1c4004d50 2023-12-19T14:40:14.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d4067d10 con 0x7fa1c4004d50 2023-12-19T14:40:14.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.175+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d4067f30 con 0x7fa1c4004d50 2023-12-19T14:40:14.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.179+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa1c8013ba0 con 0x7fa1c4004d50 2023-12-19T14:40:14.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.179+0000 7fa1d0ff9700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 0x7fa1b8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:14.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.179+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5543+0+0 (secure 0 0 0) 0x7fa1c807d570 con 0x7fa1c4004d50 2023-12-19T14:40:14.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.179+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 0x7fa1b8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:14.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.183+0000 7fa1d3fff700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 0x7fa1b8062a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1d415b5f0 tx=0x7fa1c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:14.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.187+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa1c804dc00 con 0x7fa1c4004d50 2023-12-19T14:40:14.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:14.359+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1d404be40 con 0x7fa1c4004d50 2023-12-19T14:40:15.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.027+0000 7fa1d0ff9700 1 -- 172.21.15.50:0/829334576 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa1c8051220 con 0x7fa1c4004d50 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.031+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 msgr2=0x7fa1b8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 0x7fa1b8062a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1d415b5f0 tx=0x7fa1c0008040 comp rx=0 tx=0).stop 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 msgr2=0x7fa1d4069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8007ac0 tx=0x7fa1d4159580 comp rx=0 tx=0).stop 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 shutdown_connections 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1d40ae3f0 0x7fa1d40ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1b8060580 0x7fa1b8062a30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa1d40c3be0 0x7fa1d40c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 --2- 172.21.15.50:0/829334576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1c4004d50 0x7fa1d4069b40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 >> 172.21.15.50:0/829334576 conn(0x7fa1d40ccb30 msgr2=0x7fa1d406d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 shutdown_connections 2023-12-19T14:40:15.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.035+0000 7fa1d9a0c700 1 -- 172.21.15.50:0/829334576 wait complete. 2023-12-19T14:40:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:40:15.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:40:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 Processor -- start 2023-12-19T14:40:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 -- start start 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 0x7f8b90159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b901595a0 0x7f8b901599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9006c850 con 0x7f8b90158c90 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b900cb650 con 0x7f8b901595a0 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96923700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9006b470 con 0x7f8b9006aeb0 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b901595a0 0x7f8b901599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b901595a0 0x7f8b901599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56810/0 (socket says 172.21.15.50:56810) 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 -- 172.21.15.50:0/1657047518 learned_addr learned my addr 172.21.15.50:0/1657047518 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b95921700 1 --1- 172.21.15.50:0/1657047518 >> v1:172.21.15.50:6789/0 conn(0x7f8b90158c90 0x7f8b900caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39412/0 (socket says 172.21.15.50:39412) 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b95120700 1 --2- 172.21.15.50:0/1657047518 >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 0x7f8b90159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 -- 172.21.15.50:0/1657047518 >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 msgr2=0x7f8b90159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 --2- 172.21.15.50:0/1657047518 >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 0x7f8b90159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 -- 172.21.15.50:0/1657047518 >> v1:172.21.15.50:6789/0 conn(0x7f8b90158c90 legacy=0x7f8b900caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 -- 172.21.15.50:0/1657047518 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9006b750 con 0x7f8b901595a0 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b95120700 1 --2- 172.21.15.50:0/1657047518 >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 0x7f8b90159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.199+0000 7f8b96122700 1 --2- 172.21.15.50:0/1657047518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b901595a0 0x7f8b901599c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8b90068000 tx=0x7f8b88009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c49a4865c7cce3c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b9491f700 1 -- 172.21.15.50:0/1657047518 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8800e070 con 0x7f8b901595a0 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b9491f700 1 -- 172.21.15.50:0/1657047518 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8800ba50 con 0x7f8b901595a0 2023-12-19T14:40:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b9491f700 1 -- 172.21.15.50:0/1657047518 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880092e0 con 0x7f8b901595a0 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- 172.21.15.50:0/1657047518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b901595a0 msgr2=0x7f8b901599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- 172.21.15.50:0/1657047518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b901595a0 0x7f8b901599c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8b90068000 tx=0x7f8b88009840 comp rx=0 tx=0).stop 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- 172.21.15.50:0/1657047518 shutdown_connections 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- 172.21.15.50:0/1657047518 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b901595a0 0x7f8b901599c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- 172.21.15.50:0/1657047518 >> v2:172.21.15.125:3300/0 conn(0x7f8b9006aeb0 0x7f8b90159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- 172.21.15.50:0/1657047518 >> 172.21.15.50:0/1657047518 conn(0x7f8b90067bb0 msgr2=0x7f8b900cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- 172.21.15.50:0/1657047518 shutdown_connections 2023-12-19T14:40:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- 172.21.15.50:0/1657047518 wait complete. 2023-12-19T14:40:15.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 Processor -- start 2023-12-19T14:40:15.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- start start 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 0x7f8b90153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 0x7f8b90155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901536d0 con 0x7f8b90158c90 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900c8420 con 0x7f8b901595a0 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.203+0000 7f8b96923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900c8700 con 0x7f8b9006aeb0 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56812/0 (socket says 172.21.15.50:56812) 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 -- 172.21.15.50:0/3047733656 learned_addr learned my addr 172.21.15.50:0/3047733656 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95921700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 0x7f8b90153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b96122700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 0x7f8b90155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 -- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 msgr2=0x7f8b90153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 0x7f8b90153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 -- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 msgr2=0x7f8b90155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 0x7f8b90155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 -- 172.21.15.50:0/3047733656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88009ae0 con 0x7f8b90158c90 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b96122700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 0x7f8b90155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95921700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 0x7f8b90153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b95120700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000e8d0 tx=0x7f8b80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8000ee70 con 0x7f8b90158c90 2023-12-19T14:40:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8000c5e0 con 0x7f8b90158c90 2023-12-19T14:40:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b900c89e0 con 0x7f8b90158c90 2023-12-19T14:40:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8000c980 con 0x7f8b90158c90 2023-12-19T14:40:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b900c8fa0 con 0x7f8b90158c90 2023-12-19T14:40:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.207+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9004be40 con 0x7f8b90158c90 2023-12-19T14:40:15.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.211+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8b80013410 con 0x7f8b90158c90 2023-12-19T14:40:15.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.211+0000 7f8b85ffb700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 0x7f8b74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:15.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.211+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5925+0+0 (secure 0 0 0) 0x7f8b8007c770 con 0x7f8b90158c90 2023-12-19T14:40:15.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.211+0000 7f8b95921700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 0x7f8b74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:15.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.211+0000 7f8b95921700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 0x7f8b74062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001820 tx=0x7f8b8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:15.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.215+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8b8004cbe0 con 0x7f8b90158c90 2023-12-19T14:40:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.415+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8b9006ba30 con 0x7f8b90158c90 2023-12-19T14:40:15.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.983+0000 7f8b85ffb700 1 -- 172.21.15.50:0/3047733656 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8b80050200 con 0x7f8b90158c90 2023-12-19T14:40:15.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.991+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 msgr2=0x7f8b74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.991+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 0x7f8b74062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001820 tx=0x7f8b8c009040 comp rx=0 tx=0).stop 2023-12-19T14:40:15.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.991+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 msgr2=0x7f8b900c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:15.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.991+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000e8d0 tx=0x7f8b80008040 comp rx=0 tx=0).stop 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 shutdown_connections 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b901595a0 0x7f8b90155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b74060550 0x7f8b74062a00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b90158c90 0x7f8b900c94c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 --2- 172.21.15.50:0/3047733656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b9006aeb0 0x7f8b90153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 >> 172.21.15.50:0/3047733656 conn(0x7f8b90067bb0 msgr2=0x7f8b900cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 shutdown_connections 2023-12-19T14:40:16.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:15.995+0000 7f8b96923700 1 -- 172.21.15.50:0/3047733656 wait complete. 2023-12-19T14:40:16.002 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 5 and data pool 6 2023-12-19T14:40:16.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:16.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 Processor -- start 2023-12-19T14:40:16.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 -- start start 2023-12-19T14:40:16.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0b9c0ae140 0x7f0b9c0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c0ae510 0x7f0b9c0c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c06dc20 con 0x7f0b9c06d180 2023-12-19T14:40:16.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c06d940 con 0x7f0b9c0ae140 2023-12-19T14:40:16.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba2345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c6e20 con 0x7f0b9c0ae510 2023-12-19T14:40:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba08e2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0b9c06d180 0x7f0b9c0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39418/0 (socket says 172.21.15.50:39418) 2023-12-19T14:40:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0ba08e2700 1 -- 172.21.15.50:0/1983570433 learned_addr learned my addr 172.21.15.50:0/1983570433 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.191+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1983570433 >> v2:172.21.15.50:3300/0 conn(0x7f0b9c0ae140 0x7f0b9c0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b937fe700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c0ae510 0x7f0b9c0c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9b7fe700 1 -- 172.21.15.50:0/1983570433 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944693326 0 0) 0x7f0b9c06dc20 con 0x7f0b9c06d180 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9b7fe700 1 -- 172.21.15.50:0/1983570433 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88003570 con 0x7f0b9c06d180 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9bfff700 1 -- 172.21.15.50:0/1983570433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c0ae510 msgr2=0x7f0b9c0c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c0ae510 0x7f0b9c0c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9bfff700 1 -- 172.21.15.50:0/1983570433 >> v1:172.21.15.50:6789/0 conn(0x7f0b9c06d180 legacy=0x7f0b9c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9bfff700 1 -- 172.21.15.50:0/1983570433 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0c7100 con 0x7f0b9c0ae140 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c0a8850 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c06de10 tx=0x7f0b8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedd0f182aa36c16 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9b7fe700 1 -- 172.21.15.50:0/1983570433 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c011630 con 0x7f0b9c0ae140 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9b7fe700 1 -- 172.21.15.50:0/1983570433 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c011790 con 0x7f0b9c0ae140 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0b9b7fe700 1 -- 172.21.15.50:0/1983570433 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c011b50 con 0x7f0b9c0ae140 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 -- 172.21.15.50:0/1983570433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae140 msgr2=0x7f0b9c0a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c0a8850 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c06de10 tx=0x7f0b8c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 -- 172.21.15.50:0/1983570433 shutdown_connections 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c0ae510 0x7f0b9c0c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1983570433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c0a8850 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 -- 172.21.15.50:0/1983570433 >> 172.21.15.50:0/1983570433 conn(0x7f0b9c0c8850 msgr2=0x7f0b9c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 -- 172.21.15.50:0/1983570433 shutdown_connections 2023-12-19T14:40:16.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.195+0000 7f0ba2345700 1 -- 172.21.15.50:0/1983570433 wait complete. 2023-12-19T14:40:16.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.199+0000 7f0ba2345700 1 Processor -- start 2023-12-19T14:40:16.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.199+0000 7f0ba2345700 1 -- start start 2023-12-19T14:40:16.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c06d180 0x7f0b9c0af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c15b8f0 con 0x7f0b9c0ae510 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c15ace0 con 0x7f0b9c0ae140 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c15afc0 con 0x7f0b9c06d180 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56828/0 (socket says 172.21.15.50:56828) 2023-12-19T14:40:16.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 -- 172.21.15.50:0/1695281016 learned_addr learned my addr 172.21.15.50:0/1695281016 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c06d180 0x7f0b9c0af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0b937fe700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 -- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c06d180 msgr2=0x7f0b9c0af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c06d180 0x7f0b9c0af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 -- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b9c0ae140 msgr2=0x7f0b9c15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 -- 172.21.15.50:0/1695281016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c009420 con 0x7f0b9c0ae510 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba08e2700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b94001460 tx=0x7f0b94001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9400a890 con 0x7f0b9c0ae510 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9400a9f0 con 0x7f0b9c0ae510 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c15b2a0 con 0x7f0b9c0ae510 2023-12-19T14:40:16.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9400ad70 con 0x7f0b9c0ae510 2023-12-19T14:40:16.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.203+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c15a920 con 0x7f0b9c0ae510 2023-12-19T14:40:16.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.207+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c15ba60 con 0x7f0b9c0ae510 2023-12-19T14:40:16.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.207+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0b94007300 con 0x7f0b9c0ae510 2023-12-19T14:40:16.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.211+0000 7f0b98ff9700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 0x7f0b7c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.211+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f0b9407cdf0 con 0x7f0b9c0ae510 2023-12-19T14:40:16.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.211+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 0x7f0b7c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.211+0000 7f0b9bfff700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 0x7f0b7c062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c06de10 tx=0x7f0b8c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.215+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0b9404d1f0 con 0x7f0b9c0ae510 2023-12-19T14:40:16.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.387+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b9c071120 con 0x7f0b9c0ae510 2023-12-19T14:40:16.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.391+0000 7f0b98ff9700 1 -- 172.21.15.50:0/1695281016 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+15568 (secure 0 0 0) 0x7f0b94050810 con 0x7f0b9c0ae510 2023-12-19T14:40:16.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:16.395 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":37,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:40:15.420255+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:16.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 msgr2=0x7f0b7c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 0x7f0b7c062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c06de10 tx=0x7f0b8c016000 comp rx=0 tx=0).stop 2023-12-19T14:40:16.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 msgr2=0x7f0b9c0aa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b94001460 tx=0x7f0b94001ae0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 shutdown_connections 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b9c0ae140 0x7f0b9c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0b7c060550 0x7f0b7c062a00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b9c0ae510 0x7f0b9c0aa700 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 --2- 172.21.15.50:0/1695281016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0b9c06d180 0x7f0b9c0af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 >> 172.21.15.50:0/1695281016 conn(0x7f0b9c0c8850 msgr2=0x7f0b9c157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 shutdown_connections 2023-12-19T14:40:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.395+0000 7f0ba2345700 1 -- 172.21.15.50:0/1695281016 wait complete. 2023-12-19T14:40:16.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:16.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.575+0000 7fe406bcc700 1 Processor -- start 2023-12-19T14:40:16.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe406bcc700 1 -- start start 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe406bcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe4001609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe406bcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40006d990 con 0x7fe40006de20 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe406bcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40006cd80 con 0x7fe400155960 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe406bcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40006d060 con 0x7fe4000c7f00 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe4053c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe40006de20 0x7fe4000c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39940/0 (socket says 172.21.15.50:39940) 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe4053c9700 1 -- 172.21.15.50:0/4284314373 learned_addr learned my addr 172.21.15.50:0/4284314373 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:16.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe4063cb700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe4001609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375161054 0 0) 0x7fe40006d990 con 0x7fe40006de20 2023-12-19T14:40:16.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ec005780 con 0x7fe40006de20 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334684298 0 0) 0x7fe3ec005780 con 0x7fe40006de20 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 msgr2=0x7fe4001609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe4001609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 >> v1:172.21.15.125:6789/0 conn(0x7fe400155960 legacy=0x7fe40015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40006d340 con 0x7fe40006de20 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fe3f0001470 con 0x7fe40006de20 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.579+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 >> v1:172.21.15.50:6789/0 conn(0x7fe40006de20 legacy=0x7fe4000c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3ec005ca0 0x7fe3ec006070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe3ec006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe3ec0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40006d060 con 0x7fe3ec006a20 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec00dd50 con 0x7fe3ec005ca0 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec00df10 con 0x7fe4000c7f00 2023-12-19T14:40:16.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe3ec006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe405bca700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3ec005ca0 0x7fe3ec006070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4063cb700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe3ec0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 -- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 msgr2=0x7fe3ec0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe3ec0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 -- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3ec005ca0 msgr2=0x7fe3ec006070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3ec005ca0 0x7fe3ec006070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 -- 172.21.15.50:0/4284314373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec005780 con 0x7fe3ec006a20 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe405bca700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3ec005ca0 0x7fe3ec006070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4063cb700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4000c7f00 0x7fe3ec0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4053c9700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe3ec006e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c3c0 tx=0x7fe3f0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f133d52d3f19485 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000cc90 con 0x7fe3ec006a20 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f000cdf0 con 0x7fe3ec006a20 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe404bc8700 1 -- 172.21.15.50:0/4284314373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0005b70 con 0x7fe3ec006a20 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 -- 172.21.15.50:0/4284314373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 msgr2=0x7fe3ec006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 --2- 172.21.15.50:0/4284314373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe3ec006e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c3c0 tx=0x7fe3f0001e70 comp rx=0 tx=0).stop 2023-12-19T14:40:16.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe4063cb700 1 -- 172.21.15.50:0/4284314373 reap_dead start 2023-12-19T14:40:16.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 -- 172.21.15.50:0/4284314373 shutdown_connections 2023-12-19T14:40:16.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 -- 172.21.15.50:0/4284314373 >> 172.21.15.50:0/4284314373 conn(0x7fe400067bb0 msgr2=0x7fe4000cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:16.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 -- 172.21.15.50:0/4284314373 shutdown_connections 2023-12-19T14:40:16.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.583+0000 7fe406bcc700 1 -- 172.21.15.50:0/4284314373 wait complete. 2023-12-19T14:40:16.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 Processor -- start 2023-12-19T14:40:16.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 -- start start 2023-12-19T14:40:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 0x7fe40014ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40014f170 0x7fe4001647d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40014e7a0 con 0x7fe3ec006a20 2023-12-19T14:40:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40014c660 con 0x7fe40014f170 2023-12-19T14:40:16.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe406bcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40014db90 con 0x7fe40014d620 2023-12-19T14:40:16.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe405bca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe405bca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39652/0 (socket says 172.21.15.50:39652) 2023-12-19T14:40:16.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.587+0000 7fe405bca700 1 -- 172.21.15.50:0/1994283883 learned_addr learned my addr 172.21.15.50:0/1994283883 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:16.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe4063cb700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40014f170 0x7fe4001647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe4053c9700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 0x7fe40014ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 -- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 msgr2=0x7fe40014ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 0x7fe40014ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 -- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40014f170 msgr2=0x7fe4001647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40014f170 0x7fe4001647d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 -- 172.21.15.50:0/1994283883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40014de70 con 0x7fe3ec006a20 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe4053c9700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 0x7fe40014ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe405bca700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00d8b0 tx=0x7fe3fc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc006820 con 0x7fe3ec006a20 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc006980 con 0x7fe3ec006a20 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40014e150 con 0x7fe3ec006a20 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc006d40 con 0x7fe3ec006a20 2023-12-19T14:40:16.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe400164d10 con 0x7fe3ec006a20 2023-12-19T14:40:16.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.591+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4000abdc0 con 0x7fe3ec006a20 2023-12-19T14:40:16.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.595+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe3fc007240 con 0x7fe3ec006a20 2023-12-19T14:40:16.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.595+0000 7fe3f5ffb700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 0x7fe3e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:16.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.595+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe3fc07c8c0 con 0x7fe3ec006a20 2023-12-19T14:40:16.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.595+0000 7fe4053c9700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 0x7fe3e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:16.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.595+0000 7fe4053c9700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 0x7fe3e4062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe4000c89c0 tx=0x7fe3f0015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.599+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe4000abdc0 con 0x7fe3ec006a20 2023-12-19T14:40:16.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.775+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe40014e5d0 con 0x7fe3ec006a20 2023-12-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.779+0000 7fe3f5ffb700 1 -- 172.21.15.50:0/1994283883 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+15568 (secure 0 0 0) 0x7fe3fc01f020 con 0x7fe3ec006a20 2023-12-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:16.783 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":37,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:40:15.420255+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.779+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 msgr2=0x7fe3e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.779+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 0x7fe3e4062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe4000c89c0 tx=0x7fe3f0015000 comp rx=0 tx=0).stop 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 msgr2=0x7fe40014c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00d8b0 tx=0x7fe3fc009040 comp rx=0 tx=0).stop 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 shutdown_connections 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40014f170 0x7fe4001647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3e4060550 0x7fe3e4062a00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe40014d620 0x7fe40014ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 --2- 172.21.15.50:0/1994283883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3ec006a20 0x7fe40014c120 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 >> 172.21.15.50:0/1994283883 conn(0x7fe400067bb0 msgr2=0x7fe4000ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 shutdown_connections 2023-12-19T14:40:16.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.783+0000 7fe406bcc700 1 -- 172.21.15.50:0/1994283883 wait complete. 2023-12-19T14:40:16.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:40:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.991+0000 7fdc8ab23700 1 Processor -- start 2023-12-19T14:40:16.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.991+0000 7fdc8ab23700 1 -- start start 2023-12-19T14:40:17.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc84158c90 0x7fdc840caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 0x7fdc84159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8406c850 con 0x7fdc84158c90 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8406b470 con 0x7fdc8406aeb0 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8ab23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8406b750 con 0x7fdc841595a0 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc84158c90 0x7fdc840caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc84158c90 0x7fdc840caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39660/0 (socket says 172.21.15.50:39660) 2023-12-19T14:40:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 -- 172.21.15.50:0/1427201256 learned_addr learned my addr 172.21.15.50:0/1427201256 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89320700 1 --2- 172.21.15.50:0/1427201256 >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 0x7fdc84159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 -- 172.21.15.50:0/1427201256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 msgr2=0x7fdc84159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8a322700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 -- 172.21.15.50:0/1427201256 >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 msgr2=0x7fdc84159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 --2- 172.21.15.50:0/1427201256 >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 0x7fdc84159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89b21700 1 -- 172.21.15.50:0/1427201256 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8406ba30 con 0x7fdc84158c90 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc8a322700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.995+0000 7fdc89320700 1 --2- 172.21.15.50:0/1427201256 >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 0x7fdc84159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc89b21700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84158c90 0x7fdc840caf40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdc84068000 tx=0x7fdc80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a440a05a8d361b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc88b1f700 1 -- 172.21.15.50:0/1427201256 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000b860 con 0x7fdc84158c90 2023-12-19T14:40:17.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc88b1f700 1 -- 172.21.15.50:0/1427201256 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8000b9c0 con 0x7fdc84158c90 2023-12-19T14:40:17.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc88b1f700 1 -- 172.21.15.50:0/1427201256 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000bd60 con 0x7fdc84158c90 2023-12-19T14:40:17.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 -- 172.21.15.50:0/1427201256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84158c90 msgr2=0x7fdc840caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84158c90 0x7fdc840caf40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdc84068000 tx=0x7fdc80009250 comp rx=0 tx=0).stop 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 -- 172.21.15.50:0/1427201256 shutdown_connections 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/1427201256 >> v2:172.21.15.125:3300/0 conn(0x7fdc8406aeb0 0x7fdc84159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/1427201256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc84158c90 0x7fdc840caf40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 -- 172.21.15.50:0/1427201256 >> 172.21.15.50:0/1427201256 conn(0x7fdc84067bb0 msgr2=0x7fdc840cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:17.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 -- 172.21.15.50:0/1427201256 shutdown_connections 2023-12-19T14:40:17.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:16.999+0000 7fdc8ab23700 1 -- 172.21.15.50:0/1427201256 wait complete. 2023-12-19T14:40:17.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 Processor -- start 2023-12-19T14:40:17.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 -- start start 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc84158c90 0x7fdc8414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840c5c30 con 0x7fdc8406aeb0 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84157260 con 0x7fdc84158c90 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8ab23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840c8bc0 con 0x7fdc841595a0 2023-12-19T14:40:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39666/0 (socket says 172.21.15.50:39666) 2023-12-19T14:40:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 -- 172.21.15.50:0/3457921405 learned_addr learned my addr 172.21.15.50:0/3457921405 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89320700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc84158c90 0x7fdc8414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc8a322700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 -- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 msgr2=0x7fdc84154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:17.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 -- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc84158c90 msgr2=0x7fdc8414b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:17.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc84158c90 0x7fdc8414b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:17.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.003+0000 7fdc89b21700 1 -- 172.21.15.50:0/3457921405 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc800094f0 con 0x7fdc8406aeb0 2023-12-19T14:40:17.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc89b21700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009290 tx=0x7fdc80017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80026480 con 0x7fdc8406aeb0 2023-12-19T14:40:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8001f070 con 0x7fdc8406aeb0 2023-12-19T14:40:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80026830 con 0x7fdc8406aeb0 2023-12-19T14:40:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc840c5e40 con 0x7fdc8406aeb0 2023-12-19T14:40:17.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.007+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc840c8850 con 0x7fdc8406aeb0 2023-12-19T14:40:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.011+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8404be40 con 0x7fdc8406aeb0 2023-12-19T14:40:17.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.011+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdc80026ba0 con 0x7fdc8406aeb0 2023-12-19T14:40:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.011+0000 7fdc79ffb700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 0x7fdc68062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.011+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fdc80021070 con 0x7fdc8406aeb0 2023-12-19T14:40:17.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.011+0000 7fdc89320700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 0x7fdc68062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.015+0000 7fdc89320700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 0x7fdc68062a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdc74006f00 tx=0x7fdc74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.015+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc8004ebe0 con 0x7fdc8406aeb0 2023-12-19T14:40:17.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:17.223+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdc8406bd10 con 0x7fdc8406aeb0 2023-12-19T14:40:18.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.019+0000 7fdc79ffb700 1 -- 172.21.15.50:0/3457921405 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdc80052200 con 0x7fdc8406aeb0 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 msgr2=0x7fdc68062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 0x7fdc68062a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdc74006f00 tx=0x7fdc74008040 comp rx=0 tx=0).stop 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 msgr2=0x7fdc840c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009290 tx=0x7fdc80017000 comp rx=0 tx=0).stop 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 shutdown_connections 2023-12-19T14:40:18.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc84158c90 0x7fdc8414b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc68060580 0x7fdc68062a30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc841595a0 0x7fdc84154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 --2- 172.21.15.50:0/3457921405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc8406aeb0 0x7fdc840c56f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 >> 172.21.15.50:0/3457921405 conn(0x7fdc84067bb0 msgr2=0x7fdc840afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 shutdown_connections 2023-12-19T14:40:18.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.023+0000 7fdc8ab23700 1 -- 172.21.15.50:0/3457921405 wait complete. 2023-12-19T14:40:18.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:40:18.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.223+0000 7f3411e2e700 1 Processor -- start 2023-12-19T14:40:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.223+0000 7f3411e2e700 1 -- start start 2023-12-19T14:40:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f3411e2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f340c06df10 0x7f340c0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f3411e2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0c9db0 con 0x7f340c0c63f0 2023-12-19T14:40:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f3411e2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340c0cb2e0 con 0x7f340c06db40 2023-12-19T14:40:18.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f3411e2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f340c0ca950 con 0x7f340c06df10 2023-12-19T14:40:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f340c06df10 0x7f340c0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f340c06df10 0x7f340c0cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39678/0 (socket says 172.21.15.50:39678) 2023-12-19T14:40:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 -- 172.21.15.50:0/292967072 learned_addr learned my addr 172.21.15.50:0/292967072 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:18.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f3402ffd700 1 --1- 172.21.15.50:0/292967072 >> v1:172.21.15.50:6789/0 conn(0x7f340c0c63f0 0x7f340c159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39952/0 (socket says 172.21.15.50:39952) 2023-12-19T14:40:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 -- 172.21.15.50:0/292967072 >> v1:172.21.15.125:6789/0 conn(0x7f340c06db40 legacy=0x7f340c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 -- 172.21.15.50:0/292967072 >> v1:172.21.15.50:6789/0 conn(0x7f340c0c63f0 legacy=0x7f340c159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.227+0000 7f340bfff700 1 -- 172.21.15.50:0/292967072 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0cb5c0 con 0x7f340c06df10 2023-12-19T14:40:18.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f340bfff700 1 --2- 172.21.15.50:0/292967072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c06df10 0x7f340c0cc3e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3404009f20 tx=0x7f3404009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d16d4185723a123 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f340affd700 1 -- 172.21.15.50:0/292967072 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340400e070 con 0x7f340c06df10 2023-12-19T14:40:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f340affd700 1 -- 172.21.15.50:0/292967072 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340400ba80 con 0x7f340c06df10 2023-12-19T14:40:18.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f340affd700 1 -- 172.21.15.50:0/292967072 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34040092e0 con 0x7f340c06df10 2023-12-19T14:40:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 -- 172.21.15.50:0/292967072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c06df10 msgr2=0x7f340c0cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 --2- 172.21.15.50:0/292967072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c06df10 0x7f340c0cc3e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3404009f20 tx=0x7f3404009840 comp rx=0 tx=0).stop 2023-12-19T14:40:18.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 -- 172.21.15.50:0/292967072 shutdown_connections 2023-12-19T14:40:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 --2- 172.21.15.50:0/292967072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c06df10 0x7f340c0cc3e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 -- 172.21.15.50:0/292967072 >> 172.21.15.50:0/292967072 conn(0x7f340c067bb0 msgr2=0x7f340c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 -- 172.21.15.50:0/292967072 shutdown_connections 2023-12-19T14:40:18.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.231+0000 7f3411e2e700 1 -- 172.21.15.50:0/292967072 wait complete. 2023-12-19T14:40:18.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 Processor -- start 2023-12-19T14:40:18.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 -- start start 2023-12-19T14:40:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f340c06db40 0x7f340c141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f340c06df10 0x7f340c0acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1418b0 con 0x7f340c0c63f0 2023-12-19T14:40:18.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c16e490 con 0x7f340c06df10 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.235+0000 7f3411e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340c0aeba0 con 0x7f340c06db40 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39680/0 (socket says 172.21.15.50:39680) 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 -- 172.21.15.50:0/4162544627 learned_addr learned my addr 172.21.15.50:0/4162544627 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3402ffd700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f340c06df10 0x7f340c0acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:18.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340b7fe700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f340c06db40 0x7f340c141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 -- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f340c06db40 msgr2=0x7f340c141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f340c06db40 0x7f340c141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 -- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f340c06df10 msgr2=0x7f340c0acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f340c06df10 0x7f340c0acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 -- 172.21.15.50:0/4162544627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404009ae0 con 0x7f340c0c63f0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f340bfff700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3404012000 tx=0x7f3404018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404008dd0 con 0x7f340c0c63f0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3404026460 con 0x7f340c0c63f0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c0aee80 con 0x7f340c0c63f0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404026820 con 0x7f340c0c63f0 2023-12-19T14:40:18.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.239+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c0af440 con 0x7f340c0c63f0 2023-12-19T14:40:18.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.243+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340c0af630 con 0x7f340c0c63f0 2023-12-19T14:40:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.243+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3404003040 con 0x7f340c0c63f0 2023-12-19T14:40:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.243+0000 7f3403fff700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 0x7f33ec062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.243+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f340407eac0 con 0x7f340c0c63f0 2023-12-19T14:40:18.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.243+0000 7f340b7fe700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 0x7f33ec062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:18.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.247+0000 7f340b7fe700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 0x7f33ec062a80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f33fc003860 tx=0x7f33fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:18.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.251+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f340404eed0 con 0x7f340c0c63f0 2023-12-19T14:40:18.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:18.463+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f340c071120 con 0x7f340c0c63f0 2023-12-19T14:40:19.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.031+0000 7f3403fff700 1 -- 172.21.15.50:0/4162544627 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3404008f30 con 0x7f340c0c63f0 2023-12-19T14:40:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 msgr2=0x7f33ec062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 0x7f33ec062a80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f33fc003860 tx=0x7f33fc009040 comp rx=0 tx=0).stop 2023-12-19T14:40:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 msgr2=0x7f340c0ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3404012000 tx=0x7f3404018000 comp rx=0 tx=0).stop 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 shutdown_connections 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f340c06df10 0x7f340c0acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f33ec0605d0 0x7f33ec062a80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f340c0c63f0 0x7f340c0ad1c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 --2- 172.21.15.50:0/4162544627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f340c06db40 0x7f340c141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 >> 172.21.15.50:0/4162544627 conn(0x7f340c067bb0 msgr2=0x7f340c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 shutdown_connections 2023-12-19T14:40:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.035+0000 7f3411e2e700 1 -- 172.21.15.50:0/4162544627 wait complete. 2023-12-19T14:40:19.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:40:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.219+0000 7f412042e700 1 Processor -- start 2023-12-19T14:40:19.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.219+0000 7f412042e700 1 -- start start 2023-12-19T14:40:19.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f412042e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 0x7f41180a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f412042e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411806dc20 con 0x7f41180c1d60 2023-12-19T14:40:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f412042e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411806d940 con 0x7f41180ae510 2023-12-19T14:40:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f412042e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180c2200 con 0x7f41180ae140 2023-12-19T14:40:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411e9cb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f41180c1d60 0x7f41180680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39966/0 (socket says 172.21.15.50:39966) 2023-12-19T14:40:19.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411e9cb700 1 -- 172.21.15.50:0/3265837326 learned_addr learned my addr 172.21.15.50:0/3265837326 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411e1ca700 1 --2- 172.21.15.50:0/3265837326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 0x7f41180a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108201140 0 0) 0x7f411806dc20 con 0x7f41180c1d60 2023-12-19T14:40:19.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4104003570 con 0x7f41180c1d60 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808427432 0 0) 0x7f411806d940 con 0x7f41180ae510 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.223+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411806dc20 con 0x7f41180ae510 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411e1ca700 1 -- 172.21.15.50:0/3265837326 >> v1:172.21.15.125:6789/0 conn(0x7f41180ae510 legacy=0x7f411806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218238981 0 0) 0x7f4104003570 con 0x7f41180c1d60 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411e1ca700 1 -- 172.21.15.50:0/3265837326 >> v1:172.21.15.50:6789/0 conn(0x7f41180c1d60 legacy=0x7f41180680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411e1ca700 1 -- 172.21.15.50:0/3265837326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180c6e20 con 0x7f41180ae140 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411e1ca700 1 --2- 172.21.15.50:0/3265837326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 0x7f41180a8850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4114009910 tx=0x7f41140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e0f7c6ddd7a9f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41100017f0 con 0x7f41180ae140 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4114011420 con 0x7f41180ae140 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f411d1c8700 1 -- 172.21.15.50:0/3265837326 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41140117c0 con 0x7f41180ae140 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 -- 172.21.15.50:0/3265837326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 msgr2=0x7f41180a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 --2- 172.21.15.50:0/3265837326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 0x7f41180a8850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4114009910 tx=0x7f41140091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 -- 172.21.15.50:0/3265837326 shutdown_connections 2023-12-19T14:40:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 --2- 172.21.15.50:0/3265837326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae140 0x7f41180a8850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 -- 172.21.15.50:0/3265837326 >> 172.21.15.50:0/3265837326 conn(0x7f41180c8850 msgr2=0x7f41180c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:19.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 -- 172.21.15.50:0/3265837326 shutdown_connections 2023-12-19T14:40:19.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.227+0000 7f412042e700 1 -- 172.21.15.50:0/3265837326 wait complete. 2023-12-19T14:40:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 Processor -- start 2023-12-19T14:40:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 -- start start 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 0x7f4118156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41180c1d60 0x7f411815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118157850 con 0x7f41180ae140 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118150200 con 0x7f41180c1d60 2023-12-19T14:40:19.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f412042e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41180af1d0 con 0x7f41180ae510 2023-12-19T14:40:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f411e1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f411e1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39694/0 (socket says 172.21.15.50:39694) 2023-12-19T14:40:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f411e1ca700 1 -- 172.21.15.50:0/3515725422 learned_addr learned my addr 172.21.15.50:0/3515725422 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f411e9cb700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41180c1d60 0x7f411815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.231+0000 7f411d9c9700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 0x7f4118156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 -- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 msgr2=0x7f4118156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 0x7f4118156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 -- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41180c1d60 msgr2=0x7f411815ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41180c1d60 0x7f411815ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 -- 172.21.15.50:0/3515725422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114009420 con 0x7f41180ae140 2023-12-19T14:40:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411d9c9700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 0x7f4118156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f411e1ca700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f41140091e0 tx=0x7f4114016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411401cbe0 con 0x7f41180ae140 2023-12-19T14:40:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411401cd60 con 0x7f41180ae140 2023-12-19T14:40:19.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41180cabb0 con 0x7f41180ae140 2023-12-19T14:40:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411401a6a0 con 0x7f41180ae140 2023-12-19T14:40:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41180cb170 con 0x7f41180ae140 2023-12-19T14:40:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.235+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41180af390 con 0x7f41180ae140 2023-12-19T14:40:19.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.239+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f411401aa90 con 0x7f41180ae140 2023-12-19T14:40:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.239+0000 7f410e7fc700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 0x7f40fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.239+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f411407e750 con 0x7f41180ae140 2023-12-19T14:40:19.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.239+0000 7f411d9c9700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 0x7f40fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.243+0000 7f411d9c9700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 0x7f40fc062a00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41180a92d0 tx=0x7f4108009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.247+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f411404eb60 con 0x7f41180ae140 2023-12-19T14:40:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.455+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4118071120 con 0x7f41180ae140 2023-12-19T14:40:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.455+0000 7f410e7fc700 1 -- 172.21.15.50:0/3515725422 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f4114052180 con 0x7f41180ae140 2023-12-19T14:40:19.461 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:19.461 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:40:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 msgr2=0x7f40fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 0x7f40fc062a00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41180a92d0 tx=0x7f4108009040 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 msgr2=0x7f411814fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f41140091e0 tx=0x7f4114016000 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 shutdown_connections 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41180c1d60 0x7f411815ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40fc060550 0x7f40fc062a00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41180ae510 0x7f4118156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 --2- 172.21.15.50:0/3515725422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41180ae140 0x7f411814fcc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 >> 172.21.15.50:0/3515725422 conn(0x7f41180c8850 msgr2=0x7f4118067760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 shutdown_connections 2023-12-19T14:40:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.459+0000 7f412042e700 1 -- 172.21.15.50:0/3515725422 wait complete. 2023-12-19T14:40:19.466 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:40:19.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:40:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 Processor -- start 2023-12-19T14:40:19.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 -- start start 2023-12-19T14:40:19.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f14981502b0 0x7f1498068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14980c5900 0x7f14980c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14980c6630 con 0x7f1498067c90 2023-12-19T14:40:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980c6910 con 0x7f14981502b0 2023-12-19T14:40:19.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149e2cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980c6bf0 con 0x7f14980c5900 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f148f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f14981502b0 0x7f1498068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f148f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f14981502b0 0x7f1498068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39706/0 (socket says 172.21.15.50:39706) 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f148f7fe700 1 -- 172.21.15.50:0/3182561526 learned_addr learned my addr 172.21.15.50:0/3182561526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.643+0000 7f149c86a700 1 --2- 172.21.15.50:0/3182561526 >> v2:172.21.15.125:3300/0 conn(0x7f14980c5900 0x7f14980c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f148f7fe700 1 -- 172.21.15.50:0/3182561526 >> v2:172.21.15.125:3300/0 conn(0x7f14980c5900 msgr2=0x7f14980c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f148f7fe700 1 --2- 172.21.15.50:0/3182561526 >> v2:172.21.15.125:3300/0 conn(0x7f14980c5900 0x7f14980c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f148f7fe700 1 -- 172.21.15.50:0/3182561526 >> v1:172.21.15.125:6789/0 conn(0x7f1498067c90 legacy=0x7f149814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:19.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f148f7fe700 1 -- 172.21.15.50:0/3182561526 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980c6ed0 con 0x7f14981502b0 2023-12-19T14:40:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f148f7fe700 1 --2- 172.21.15.50:0/3182561526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14981502b0 0x7f1498068060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1498068a60 tx=0x7f1480009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43862ef52f0a251 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f14977fe700 1 -- 172.21.15.50:0/3182561526 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000b860 con 0x7f14981502b0 2023-12-19T14:40:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f14977fe700 1 -- 172.21.15.50:0/3182561526 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000b9c0 con 0x7f14981502b0 2023-12-19T14:40:19.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f14977fe700 1 -- 172.21.15.50:0/3182561526 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000bd60 con 0x7f14981502b0 2023-12-19T14:40:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 -- 172.21.15.50:0/3182561526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14981502b0 msgr2=0x7f1498068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3182561526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14981502b0 0x7f1498068060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1498068a60 tx=0x7f1480009250 comp rx=0 tx=0).stop 2023-12-19T14:40:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 -- 172.21.15.50:0/3182561526 shutdown_connections 2023-12-19T14:40:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3182561526 >> v2:172.21.15.125:3300/0 conn(0x7f14980c5900 0x7f14980c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3182561526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14981502b0 0x7f1498068060 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 -- 172.21.15.50:0/3182561526 >> 172.21.15.50:0/3182561526 conn(0x7f14980cd6d0 msgr2=0x7f149806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:19.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 -- 172.21.15.50:0/3182561526 shutdown_connections 2023-12-19T14:40:19.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.647+0000 7f149e2cd700 1 -- 172.21.15.50:0/3182561526 wait complete. 2023-12-19T14:40:19.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 Processor -- start 2023-12-19T14:40:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 -- start start 2023-12-19T14:40:19.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1498067c90 0x7f1498140230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14981502b0 0x7f14980c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1498151fc0 con 0x7f14980c5900 2023-12-19T14:40:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149816a3a0 con 0x7f14981502b0 2023-12-19T14:40:19.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149e2cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980c9e80 con 0x7f1498067c90 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f148f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f148f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39714/0 (socket says 172.21.15.50:39714) 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f148f7fe700 1 -- 172.21.15.50:0/3608356886 learned_addr learned my addr 172.21.15.50:0/3608356886 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f1497fff700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1498067c90 0x7f1498140230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.651+0000 7f149c86a700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14981502b0 0x7f14980c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 -- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1498067c90 msgr2=0x7f1498140230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1498067c90 0x7f1498140230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 -- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14981502b0 msgr2=0x7f14980c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14981502b0 0x7f14980c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 -- 172.21.15.50:0/3608356886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14800094f0 con 0x7f14980c5900 2023-12-19T14:40:19.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f148f7fe700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f149806a0f0 tx=0x7f1480017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480011e00 con 0x7f14980c5900 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480026450 con 0x7f14980c5900 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14980ca160 con 0x7f14980c5900 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480026810 con 0x7f14980c5900 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1498151c50 con 0x7f14980c5900 2023-12-19T14:40:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.655+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1498152170 con 0x7f14980c5900 2023-12-19T14:40:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.659+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1480026c40 con 0x7f14980c5900 2023-12-19T14:40:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.659+0000 7f1494ff9700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 0x7f1478062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.659+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f148007e4d0 con 0x7f14980c5900 2023-12-19T14:40:19.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.659+0000 7f1497fff700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 0x7f1478062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:19.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.659+0000 7f1497fff700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 0x7f1478062a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1488007fd0 tx=0x7f1488000f40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:19.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.667+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1498152170 con 0x7f14980c5900 2023-12-19T14:40:19.744 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T14:40:19.739+0000 7fe7b05d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:40:19.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.839+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1498152170 con 0x7f14980c5900 2023-12-19T14:40:19.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.839+0000 7f1494ff9700 1 -- 172.21.15.50:0/3608356886 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+15555 (secure 0 0 0) 0x7f1498152170 con 0x7f14980c5900 2023-12-19T14:40:19.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:19.845 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":38,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:40:16.993743+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 msgr2=0x7f1478062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 0x7f1478062a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1488007fd0 tx=0x7f1488000f40 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 msgr2=0x7f14980c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f149806a0f0 tx=0x7f1480017000 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 shutdown_connections 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14981502b0 0x7f14980c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060550 0x7f1478062a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f14980c5900 0x7f14980c9420 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 --2- 172.21.15.50:0/3608356886 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1498067c90 0x7f1498140230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:19.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 >> 172.21.15.50:0/3608356886 conn(0x7f14980cd6d0 msgr2=0x7f14980b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 shutdown_connections 2023-12-19T14:40:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:19.843+0000 7f149e2cd700 1 -- 172.21.15.50:0/3608356886 wait complete. 2023-12-19T14:40:19.869 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:40:19.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:40:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.035+0000 7f48890ae700 1 Processor -- start 2023-12-19T14:40:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.035+0000 7f48890ae700 1 -- start start 2023-12-19T14:40:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f48890ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4884158ee0 0x7f4884067af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f48890ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f488406d760 0x7f48840b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f48890ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840688d0 con 0x7f48841592b0 2023-12-19T14:40:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f48890ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48840b1510 con 0x7f488406d760 2023-12-19T14:40:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f48890ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488406a180 con 0x7f4884158ee0 2023-12-19T14:40:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f488406d760 0x7f48840b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f488406d760 0x7f48840b19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39718/0 (socket says 172.21.15.50:39718) 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 -- 172.21.15.50:0/2631975714 learned_addr learned my addr 172.21.15.50:0/2631975714 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f4882d9d700 1 --2- 172.21.15.50:0/2631975714 >> v2:172.21.15.125:3300/0 conn(0x7f4884158ee0 0x7f4884067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 -- 172.21.15.50:0/2631975714 >> v2:172.21.15.125:3300/0 conn(0x7f4884158ee0 msgr2=0x7f4884067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 --2- 172.21.15.50:0/2631975714 >> v2:172.21.15.125:3300/0 conn(0x7f4884158ee0 0x7f4884067af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 -- 172.21.15.50:0/2631975714 >> v1:172.21.15.125:6789/0 conn(0x7f48841592b0 legacy=0x7f488406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 -- 172.21.15.50:0/2631975714 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840685f0 con 0x7f488406d760 2023-12-19T14:40:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f488359e700 1 --2- 172.21.15.50:0/2631975714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f488406d760 0x7f48840b19a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48840cdfd0 tx=0x7f487c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3605dc38a0f78f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f4881d9b700 1 -- 172.21.15.50:0/2631975714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00e070 con 0x7f488406d760 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f4881d9b700 1 -- 172.21.15.50:0/2631975714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c00ba50 con 0x7f488406d760 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.039+0000 7f4881d9b700 1 -- 172.21.15.50:0/2631975714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c0092e0 con 0x7f488406d760 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- 172.21.15.50:0/2631975714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f488406d760 msgr2=0x7f48840b19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 --2- 172.21.15.50:0/2631975714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f488406d760 0x7f48840b19a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48840cdfd0 tx=0x7f487c009840 comp rx=0 tx=0).stop 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- 172.21.15.50:0/2631975714 shutdown_connections 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 --2- 172.21.15.50:0/2631975714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f488406d760 0x7f48840b19a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 --2- 172.21.15.50:0/2631975714 >> v2:172.21.15.125:3300/0 conn(0x7f4884158ee0 0x7f4884067af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- 172.21.15.50:0/2631975714 >> 172.21.15.50:0/2631975714 conn(0x7f48840cd990 msgr2=0x7f48840ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- 172.21.15.50:0/2631975714 shutdown_connections 2023-12-19T14:40:20.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- 172.21.15.50:0/2631975714 wait complete. 2023-12-19T14:40:20.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 Processor -- start 2023-12-19T14:40:20.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 -- start start 2023-12-19T14:40:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.043+0000 7f48890ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f488406d760 0x7f4884152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4884158ee0 0x7f48841449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840a6cc0 con 0x7f48841592b0 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884144ef0 con 0x7f4884158ee0 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48841529b0 con 0x7f488406d760 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39730/0 (socket says 172.21.15.50:39730) 2023-12-19T14:40:20.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 -- 172.21.15.50:0/2404444380 learned_addr learned my addr 172.21.15.50:0/2404444380 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f4882d9d700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f488406d760 0x7f4884152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488259c700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4884158ee0 0x7f48841449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 -- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f488406d760 msgr2=0x7f4884152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f488406d760 0x7f4884152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 -- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4884158ee0 msgr2=0x7f48841449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4884158ee0 0x7f48841449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 -- 172.21.15.50:0/2404444380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c009ae0 con 0x7f48841592b0 2023-12-19T14:40:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f488359e700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f488406a6e0 tx=0x7f487c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00be60 con 0x7f48841592b0 2023-12-19T14:40:20.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c008ce0 con 0x7f48841592b0 2023-12-19T14:40:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4884151ca0 con 0x7f48841592b0 2023-12-19T14:40:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c0266b0 con 0x7f48841592b0 2023-12-19T14:40:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.047+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4884152360 con 0x7f48841592b0 2023-12-19T14:40:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.051+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4884071120 con 0x7f48841592b0 2023-12-19T14:40:20.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.051+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f487c003040 con 0x7f48841592b0 2023-12-19T14:40:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.051+0000 7f486affd700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 0x7f4870062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.051+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f487c07e800 con 0x7f48841592b0 2023-12-19T14:40:20.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.055+0000 7f4882d9d700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 0x7f4870062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.055+0000 7f4882d9d700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 0x7f4870062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4874006fd0 tx=0x7f4874008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.059+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f487c04ec10 con 0x7f48841592b0 2023-12-19T14:40:20.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.235+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48840b0ec0 con 0x7f4870060550 2023-12-19T14:40:20.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.251+0000 7f486affd700 1 -- 172.21.15.50:0/2404444380 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283484 (secure 0 0 0) 0x7f48840b0ec0 con 0x7f4870060550 2023-12-19T14:40:20.266 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:20.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 msgr2=0x7f4870062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 0x7f4870062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4874006fd0 tx=0x7f4874008040 comp rx=0 tx=0).stop 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 msgr2=0x7f48840a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f488406a6e0 tx=0x7f487c018000 comp rx=0 tx=0).stop 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 shutdown_connections 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4884158ee0 0x7f48841449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4870060550 0x7f4870062a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48841592b0 0x7f48840a6150 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 --2- 172.21.15.50:0/2404444380 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f488406d760 0x7f4884152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 >> 172.21.15.50:0/2404444380 conn(0x7f48840cd990 msgr2=0x7f48840ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 shutdown_connections 2023-12-19T14:40:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.263+0000 7f48890ae700 1 -- 172.21.15.50:0/2404444380 wait complete. 2023-12-19T14:40:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:40:20.290 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-12-19T14:40:20.153579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399000,"kb_used_data":4312,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575720,"statfs":{"total":773094113280,"available":770637537280,"internally_reserved":0,"allocated":4415488,"data_stored":2533784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001747"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011324+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.015299+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010786+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032050+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032514+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994572+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984445+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012508+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996425+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996909+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013771+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991932+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989734+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994060+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991210+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995599+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981853+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:07.226632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032685+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.980787+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:12.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991379+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.985539+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989176+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031534+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.982597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:03.243160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031182+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981065+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:50.763905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992991+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981044+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:20.673959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981068+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:16.111460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988543+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030864+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.998033+0000","last_change":"2023-12-19T14:40:14.998033+0000","last_active":"2023-12-19T14:40:14.998033+0000","last_peered":"2023-12-19T14:40:14.998033+0000","last_clean":"2023-12-19T14:40:14.998033+0000","last_became_active":"2023-12-19T14:40:14.997769+0000","last_became_peered":"2023-12-19T14:40:14.997769+0000","last_unstale":"2023-12-19T14:40:14.998033+0000","last_undegraded":"2023-12-19T14:40:14.998033+0000","last_fullsized":"2023-12-19T14:40:14.998033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:05.375208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031880+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022907+0000","last_change":"2023-12-19T14:40:15.022907+0000","last_active":"2023-12-19T14:40:15.022907+0000","last_peered":"2023-12-19T14:40:15.022907+0000","last_clean":"2023-12-19T14:40:15.022907+0000","last_became_active":"2023-12-19T14:40:15.022603+0000","last_became_peered":"2023-12-19T14:40:15.022603+0000","last_unstale":"2023-12-19T14:40:15.022907+0000","last_undegraded":"2023-12-19T14:40:15.022907+0000","last_fullsized":"2023-12-19T14:40:15.022907+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987338+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021930+0000","last_change":"2023-12-19T14:40:15.021930+0000","last_active":"2023-12-19T14:40:15.021930+0000","last_peered":"2023-12-19T14:40:15.021930+0000","last_clean":"2023-12-19T14:40:15.021930+0000","last_became_active":"2023-12-19T14:40:15.021586+0000","last_became_peered":"2023-12-19T14:40:15.021586+0000","last_unstale":"2023-12-19T14:40:15.021930+0000","last_undegraded":"2023-12-19T14:40:15.021930+0000","last_fullsized":"2023-12-19T14:40:15.021930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011878+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026312+0000","last_change":"2023-12-19T14:40:15.026312+0000","last_active":"2023-12-19T14:40:15.026312+0000","last_peered":"2023-12-19T14:40:15.026312+0000","last_clean":"2023-12-19T14:40:15.026312+0000","last_became_active":"2023-12-19T14:40:15.025957+0000","last_became_peered":"2023-12-19T14:40:15.025957+0000","last_unstale":"2023-12-19T14:40:15.026312+0000","last_undegraded":"2023-12-19T14:40:15.026312+0000","last_fullsized":"2023-12-19T14:40:15.026312+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014928+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025292+0000","last_change":"2023-12-19T14:40:15.025292+0000","last_active":"2023-12-19T14:40:15.025292+0000","last_peered":"2023-12-19T14:40:15.025292+0000","last_clean":"2023-12-19T14:40:15.025292+0000","last_became_active":"2023-12-19T14:40:15.024103+0000","last_became_peered":"2023-12-19T14:40:15.024103+0000","last_unstale":"2023-12-19T14:40:15.025292+0000","last_undegraded":"2023-12-19T14:40:15.025292+0000","last_fullsized":"2023-12-19T14:40:15.025292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:59.010045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992953+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.004455+0000","last_change":"2023-12-19T14:40:15.004455+0000","last_active":"2023-12-19T14:40:15.004455+0000","last_peered":"2023-12-19T14:40:15.004455+0000","last_clean":"2023-12-19T14:40:15.004455+0000","last_became_active":"2023-12-19T14:40:15.004022+0000","last_became_peered":"2023-12-19T14:40:15.004022+0000","last_unstale":"2023-12-19T14:40:15.004455+0000","last_undegraded":"2023-12-19T14:40:15.004455+0000","last_fullsized":"2023-12-19T14:40:15.004455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.986757+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.015560+0000","last_change":"2023-12-19T14:40:15.015560+0000","last_active":"2023-12-19T14:40:15.015560+0000","last_peered":"2023-12-19T14:40:15.015560+0000","last_clean":"2023-12-19T14:40:15.015560+0000","last_became_active":"2023-12-19T14:40:15.015177+0000","last_became_peered":"2023-12-19T14:40:15.015177+0000","last_unstale":"2023-12-19T14:40:15.015560+0000","last_undegraded":"2023-12-19T14:40:15.015560+0000","last_fullsized":"2023-12-19T14:40:15.015560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.994802+0000","last_change":"2023-12-19T14:40:14.994802+0000","last_active":"2023-12-19T14:40:14.994802+0000","last_peered":"2023-12-19T14:40:14.994802+0000","last_clean":"2023-12-19T14:40:14.994802+0000","last_became_active":"2023-12-19T14:40:14.994524+0000","last_became_peered":"2023-12-19T14:40:14.994524+0000","last_unstale":"2023-12-19T14:40:14.994802+0000","last_undegraded":"2023-12-19T14:40:14.994802+0000","last_fullsized":"2023-12-19T14:40:14.994802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031430+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024050+0000","last_change":"2023-12-19T14:40:15.024050+0000","last_active":"2023-12-19T14:40:15.024050+0000","last_peered":"2023-12-19T14:40:15.024050+0000","last_clean":"2023-12-19T14:40:15.024050+0000","last_became_active":"2023-12-19T14:40:15.023710+0000","last_became_peered":"2023-12-19T14:40:15.023710+0000","last_unstale":"2023-12-19T14:40:15.024050+0000","last_undegraded":"2023-12-19T14:40:15.024050+0000","last_fullsized":"2023-12-19T14:40:15.024050+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990860+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025954+0000","last_change":"2023-12-19T14:40:15.025954+0000","last_active":"2023-12-19T14:40:15.025954+0000","last_peered":"2023-12-19T14:40:15.025954+0000","last_clean":"2023-12-19T14:40:15.025954+0000","last_became_active":"2023-12-19T14:40:15.025658+0000","last_became_peered":"2023-12-19T14:40:15.025658+0000","last_unstale":"2023-12-19T14:40:15.025954+0000","last_undegraded":"2023-12-19T14:40:15.025954+0000","last_fullsized":"2023-12-19T14:40:15.025954+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010376+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023326+0000","last_change":"2023-12-19T14:40:15.023326+0000","last_active":"2023-12-19T14:40:15.023326+0000","last_peered":"2023-12-19T14:40:15.023326+0000","last_clean":"2023-12-19T14:40:15.023326+0000","last_became_active":"2023-12-19T14:40:15.022920+0000","last_became_peered":"2023-12-19T14:40:15.022920+0000","last_unstale":"2023-12-19T14:40:15.023326+0000","last_undegraded":"2023-12-19T14:40:15.023326+0000","last_fullsized":"2023-12-19T14:40:15.023326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:44.100285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987923+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023075+0000","last_change":"2023-12-19T14:40:15.023075+0000","last_active":"2023-12-19T14:40:15.023075+0000","last_peered":"2023-12-19T14:40:15.023075+0000","last_clean":"2023-12-19T14:40:15.023075+0000","last_became_active":"2023-12-19T14:40:15.022756+0000","last_became_peered":"2023-12-19T14:40:15.022756+0000","last_unstale":"2023-12-19T14:40:15.023075+0000","last_undegraded":"2023-12-19T14:40:15.023075+0000","last_fullsized":"2023-12-19T14:40:15.023075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014351+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026805+0000","last_change":"2023-12-19T14:40:15.026805+0000","last_active":"2023-12-19T14:40:15.026805+0000","last_peered":"2023-12-19T14:40:15.026805+0000","last_clean":"2023-12-19T14:40:15.026805+0000","last_became_active":"2023-12-19T14:40:15.025038+0000","last_became_peered":"2023-12-19T14:40:15.025038+0000","last_unstale":"2023-12-19T14:40:15.026805+0000","last_undegraded":"2023-12-19T14:40:15.026805+0000","last_fullsized":"2023-12-19T14:40:15.026805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984566+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.017622+0000","last_change":"2023-12-19T14:40:15.017622+0000","last_active":"2023-12-19T14:40:15.017622+0000","last_peered":"2023-12-19T14:40:15.017622+0000","last_clean":"2023-12-19T14:40:15.017622+0000","last_became_active":"2023-12-19T14:40:15.017316+0000","last_became_peered":"2023-12-19T14:40:15.017316+0000","last_unstale":"2023-12-19T14:40:15.017622+0000","last_undegraded":"2023-12-19T14:40:15.017622+0000","last_fullsized":"2023-12-19T14:40:15.017622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989643+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001820+0000","last_change":"2023-12-19T14:40:15.001820+0000","last_active":"2023-12-19T14:40:15.001820+0000","last_peered":"2023-12-19T14:40:15.001820+0000","last_clean":"2023-12-19T14:40:15.001820+0000","last_became_active":"2023-12-19T14:40:15.001305+0000","last_became_peered":"2023-12-19T14:40:15.001305+0000","last_unstale":"2023-12-19T14:40:15.001820+0000","last_undegraded":"2023-12-19T14:40:15.001820+0000","last_fullsized":"2023-12-19T14:40:15.001820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:52.732501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013809+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003285+0000","last_change":"2023-12-19T14:40:15.003285+0000","last_active":"2023-12-19T14:40:15.003285+0000","last_peered":"2023-12-19T14:40:15.003285+0000","last_clean":"2023-12-19T14:40:15.003285+0000","last_became_active":"2023-12-19T14:40:15.002941+0000","last_became_peered":"2023-12-19T14:40:15.002941+0000","last_unstale":"2023-12-19T14:40:15.003285+0000","last_undegraded":"2023-12-19T14:40:15.003285+0000","last_fullsized":"2023-12-19T14:40:15.003285+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012459+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003942+0000","last_change":"2023-12-19T14:40:15.003942+0000","last_active":"2023-12-19T14:40:15.003942+0000","last_peered":"2023-12-19T14:40:15.003942+0000","last_clean":"2023-12-19T14:40:15.003942+0000","last_became_active":"2023-12-19T14:40:15.003631+0000","last_became_peered":"2023-12-19T14:40:15.003631+0000","last_unstale":"2023-12-19T14:40:15.003942+0000","last_undegraded":"2023-12-19T14:40:15.003942+0000","last_fullsized":"2023-12-19T14:40:15.003942+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:22.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995831+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.002317+0000","last_change":"2023-12-19T14:40:15.002317+0000","last_active":"2023-12-19T14:40:15.002317+0000","last_peered":"2023-12-19T14:40:15.002317+0000","last_clean":"2023-12-19T14:40:15.002317+0000","last_became_active":"2023-12-19T14:40:15.002056+0000","last_became_peered":"2023-12-19T14:40:15.002056+0000","last_unstale":"2023-12-19T14:40:15.002317+0000","last_undegraded":"2023-12-19T14:40:15.002317+0000","last_fullsized":"2023-12-19T14:40:15.002317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:38.560751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988174+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.010764+0000","last_change":"2023-12-19T14:40:15.010764+0000","last_active":"2023-12-19T14:40:15.010764+0000","last_peered":"2023-12-19T14:40:15.010764+0000","last_clean":"2023-12-19T14:40:15.010764+0000","last_became_active":"2023-12-19T14:40:15.010427+0000","last_became_peered":"2023-12-19T14:40:15.010427+0000","last_unstale":"2023-12-19T14:40:15.010764+0000","last_undegraded":"2023-12-19T14:40:15.010764+0000","last_fullsized":"2023-12-19T14:40:15.010764+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:11.000395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031490+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021360+0000","last_change":"2023-12-19T14:40:15.021360+0000","last_active":"2023-12-19T14:40:15.021360+0000","last_peered":"2023-12-19T14:40:15.021360+0000","last_clean":"2023-12-19T14:40:15.021360+0000","last_became_active":"2023-12-19T14:40:15.020201+0000","last_became_peered":"2023-12-19T14:40:15.020201+0000","last_unstale":"2023-12-19T14:40:15.021360+0000","last_undegraded":"2023-12-19T14:40:15.021360+0000","last_fullsized":"2023-12-19T14:40:15.021360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989347+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026807+0000","last_change":"2023-12-19T14:40:15.026807+0000","last_active":"2023-12-19T14:40:15.026807+0000","last_peered":"2023-12-19T14:40:15.026807+0000","last_clean":"2023-12-19T14:40:15.026807+0000","last_became_active":"2023-12-19T14:40:15.026425+0000","last_became_peered":"2023-12-19T14:40:15.026425+0000","last_unstale":"2023-12-19T14:40:15.026807+0000","last_undegraded":"2023-12-19T14:40:15.026807+0000","last_fullsized":"2023-12-19T14:40:15.026807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011013+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.013522+0000","last_change":"2023-12-19T14:40:15.013522+0000","last_active":"2023-12-19T14:40:15.013522+0000","last_peered":"2023-12-19T14:40:15.013522+0000","last_clean":"2023-12-19T14:40:15.013522+0000","last_became_active":"2023-12-19T14:40:15.013241+0000","last_became_peered":"2023-12-19T14:40:15.013241+0000","last_unstale":"2023-12-19T14:40:15.013522+0000","last_undegraded":"2023-12-19T14:40:15.013522+0000","last_fullsized":"2023-12-19T14:40:15.013522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:49.758106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996537+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001340+0000","last_change":"2023-12-19T14:40:15.001340+0000","last_active":"2023-12-19T14:40:15.001340+0000","last_peered":"2023-12-19T14:40:15.001340+0000","last_clean":"2023-12-19T14:40:15.001340+0000","last_became_active":"2023-12-19T14:40:15.001076+0000","last_became_peered":"2023-12-19T14:40:15.001076+0000","last_unstale":"2023-12-19T14:40:15.001340+0000","last_undegraded":"2023-12-19T14:40:15.001340+0000","last_fullsized":"2023-12-19T14:40:15.001340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984661+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024120+0000","last_change":"2023-12-19T14:40:15.024120+0000","last_active":"2023-12-19T14:40:15.024120+0000","last_peered":"2023-12-19T14:40:15.024120+0000","last_clean":"2023-12-19T14:40:15.024120+0000","last_became_active":"2023-12-19T14:40:15.023840+0000","last_became_peered":"2023-12-19T14:40:15.023840+0000","last_unstale":"2023-12-19T14:40:15.024120+0000","last_undegraded":"2023-12-19T14:40:15.024120+0000","last_fullsized":"2023-12-19T14:40:15.024120+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030693+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022435+0000","last_change":"2023-12-19T14:40:15.022435+0000","last_active":"2023-12-19T14:40:15.022435+0000","last_peered":"2023-12-19T14:40:15.022435+0000","last_clean":"2023-12-19T14:40:15.022435+0000","last_became_active":"2023-12-19T14:40:15.022169+0000","last_became_peered":"2023-12-19T14:40:15.022169+0000","last_unstale":"2023-12-19T14:40:15.022435+0000","last_undegraded":"2023-12-19T14:40:15.022435+0000","last_fullsized":"2023-12-19T14:40:15.022435+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:43.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990365+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.012638+0000","last_change":"2023-12-19T14:40:15.012638+0000","last_active":"2023-12-19T14:40:15.012638+0000","last_peered":"2023-12-19T14:40:15.012638+0000","last_clean":"2023-12-19T14:40:15.012638+0000","last_became_active":"2023-12-19T14:40:15.012351+0000","last_became_peered":"2023-12-19T14:40:15.012351+0000","last_unstale":"2023-12-19T14:40:15.012638+0000","last_undegraded":"2023-12-19T14:40:15.012638+0000","last_fullsized":"2023-12-19T14:40:15.012638+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:15.105662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013206+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.029026+0000","last_change":"2023-12-19T14:40:15.029026+0000","last_active":"2023-12-19T14:40:15.029026+0000","last_peered":"2023-12-19T14:40:15.029026+0000","last_clean":"2023-12-19T14:40:15.029026+0000","last_became_active":"2023-12-19T14:40:15.028694+0000","last_became_peered":"2023-12-19T14:40:15.028694+0000","last_unstale":"2023-12-19T14:40:15.029026+0000","last_undegraded":"2023-12-19T14:40:15.029026+0000","last_fullsized":"2023-12-19T14:40:15.029026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013783+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023303+0000","last_change":"2023-12-19T14:40:15.023303+0000","last_active":"2023-12-19T14:40:15.023303+0000","last_peered":"2023-12-19T14:40:15.023303+0000","last_clean":"2023-12-19T14:40:15.023303+0000","last_became_active":"2023-12-19T14:40:15.022812+0000","last_became_peered":"2023-12-19T14:40:15.022812+0000","last_unstale":"2023-12-19T14:40:15.023303+0000","last_undegraded":"2023-12-19T14:40:15.023303+0000","last_fullsized":"2023-12-19T14:40:15.023303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030633+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024761+0000","last_change":"2023-12-19T14:40:15.024761+0000","last_active":"2023-12-19T14:40:15.024761+0000","last_peered":"2023-12-19T14:40:15.024761+0000","last_clean":"2023-12-19T14:40:15.024761+0000","last_became_active":"2023-12-19T14:40:15.024406+0000","last_became_peered":"2023-12-19T14:40:15.024406+0000","last_unstale":"2023-12-19T14:40:15.024761+0000","last_undegraded":"2023-12-19T14:40:15.024761+0000","last_fullsized":"2023-12-19T14:40:15.024761+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:36.341963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026100+0000","last_change":"2023-12-19T14:40:15.026100+0000","last_active":"2023-12-19T14:40:15.026100+0000","last_peered":"2023-12-19T14:40:15.026100+0000","last_clean":"2023-12-19T14:40:15.026100+0000","last_became_active":"2023-12-19T14:40:15.025647+0000","last_became_peered":"2023-12-19T14:40:15.025647+0000","last_unstale":"2023-12-19T14:40:15.026100+0000","last_undegraded":"2023-12-19T14:40:15.026100+0000","last_fullsized":"2023-12-19T14:40:15.026100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024302+0000","last_change":"2023-12-19T14:40:15.024302+0000","last_active":"2023-12-19T14:40:15.024302+0000","last_peered":"2023-12-19T14:40:15.024302+0000","last_clean":"2023-12-19T14:40:15.024302+0000","last_became_active":"2023-12-19T14:40:15.024017+0000","last_became_peered":"2023-12-19T14:40:15.024017+0000","last_unstale":"2023-12-19T14:40:15.024302+0000","last_undegraded":"2023-12-19T14:40:15.024302+0000","last_fullsized":"2023-12-19T14:40:15.024302+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996301+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021965+0000","last_change":"2023-12-19T14:40:15.021965+0000","last_active":"2023-12-19T14:40:15.021965+0000","last_peered":"2023-12-19T14:40:15.021965+0000","last_clean":"2023-12-19T14:40:15.021965+0000","last_became_active":"2023-12-19T14:40:15.021467+0000","last_became_peered":"2023-12-19T14:40:15.021467+0000","last_unstale":"2023-12-19T14:40:15.021965+0000","last_undegraded":"2023-12-19T14:40:15.021965+0000","last_fullsized":"2023-12-19T14:40:15.021965+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-19T14:40:13.980266+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:40:13.980266+0000","last_peered":"2023-12-19T14:40:13.980266+0000","last_clean":"2023-12-19T14:40:13.980266+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:40:13.980266+0000","last_undegraded":"2023-12-19T14:40:13.980266+0000","last_fullsized":"2023-12-19T14:40:13.980266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298260,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073580,"statfs":{"total":96636764160,"available":96331345920,"internally_reserved":0,"allocated":995328,"data_stored":758106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.90500000000000003}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.494}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.161}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.55},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.522}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.91700000000000004}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.266},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":3.0990000000000002},"last":1.125}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.222},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.1579999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":1.254},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":3.194,"5min":3.194,"15min":3.194},"last":1.321}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0150000000000001,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3600000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.367,"5min":3.367,"15min":3.367},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.51}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.143}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.1850000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.419},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2709999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.173},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.2430000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.478},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.121}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.514},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.702}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.966},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.8300000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":2.0840000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.742,"5min":2.742,"15min":2.742},"last":1.917}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":5.2060000000000004,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":2.02},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":2.5289999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":4.476,"5min":4.476,"15min":4.476},"last":0.94499999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.212}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":1.347},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.621}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.867,"5min":1.867,"15min":1.867},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.5980000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301772,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070068,"statfs":{"total":96636764160,"available":96327749632,"internally_reserved":0,"allocated":397312,"data_stored":166727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.871},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2050000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.151},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.629}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.351}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":1.306},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":1.548}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.403},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":3.0619999999999998,"5min":3.0619999999999998,"15min":3.0619999999999998},"last":1.123}]},{"osd":3,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":1.236}]},{"osd":5,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.591},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.266}]},{"osd":6,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.49},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":1.329},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.7669999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069488,"statfs":{"total":96636764160,"available":96327155712,"internally_reserved":0,"allocated":991232,"data_stored":757095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.83799999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.413}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.482},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":1.1819999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.143}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.077,"5min":1.077,"15min":1.077},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:40:20.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:40:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.495+0000 7f4f08c2e700 1 Processor -- start 2023-12-19T14:40:20.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.499+0000 7f4f08c2e700 1 -- start start 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.499+0000 7f4f08c2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4f040ae520 0x7f4f041556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.499+0000 7f4f08c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f0406c7c0 0x7f4f0406cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0406d140 con 0x7f4f041552f0 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f040c8710 con 0x7f4f040ae520 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040c1b90 con 0x7f4f0406c7c0 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f0259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4f041552f0 0x7f4f040c8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39978/0 (socket says 172.21.15.50:39978) 2023-12-19T14:40:20.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f0259c700 1 -- 172.21.15.50:0/2556434076 learned_addr learned my addr 172.21.15.50:0/2556434076 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:20.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 --2- 172.21.15.50:0/2556434076 >> v2:172.21.15.50:3300/0 conn(0x7f4f040ae520 0x7f4f041556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f02d9d700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f0406c7c0 0x7f4f0406cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451128765 0 0) 0x7f4f0406d140 con 0x7f4f041552f0 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef0003570 con 0x7f4f041552f0 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 -- 172.21.15.50:0/2556434076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f0406c7c0 msgr2=0x7f4f0406cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f0406c7c0 0x7f4f0406cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 -- 172.21.15.50:0/2556434076 >> v1:172.21.15.50:6789/0 conn(0x7f4f041552f0 legacy=0x7f4f040c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191856510 0 0) 0x7f4ef0003570 con 0x7f4f041552f0 2023-12-19T14:40:20.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 -- 172.21.15.50:0/2556434076 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0406a9d0 con 0x7f4f040ae520 2023-12-19T14:40:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4efbfff700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f041556c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0098e0 tx=0x7f4eec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3d33b50728468a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4001490 con 0x7f4f040ae520 2023-12-19T14:40:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec011420 con 0x7f4f040ae520 2023-12-19T14:40:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f01d9b700 1 -- 172.21.15.50:0/2556434076 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0117c0 con 0x7f4f040ae520 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 -- 172.21.15.50:0/2556434076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 msgr2=0x7f4f041556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f041556c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0098e0 tx=0x7f4eec0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 -- 172.21.15.50:0/2556434076 shutdown_connections 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f0406c7c0 0x7f4f0406cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.503+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/2556434076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f041556c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 -- 172.21.15.50:0/2556434076 >> 172.21.15.50:0/2556434076 conn(0x7f4f040c8850 msgr2=0x7f4f040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 -- 172.21.15.50:0/2556434076 shutdown_connections 2023-12-19T14:40:20.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 -- 172.21.15.50:0/2556434076 wait complete. 2023-12-19T14:40:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 Processor -- start 2023-12-19T14:40:20.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 -- start start 2023-12-19T14:40:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f0406c7c0 0x7f4f040ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.507+0000 7f4f08c2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f041552f0 0x7f4f040a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0416a1d0 con 0x7f4f040ae520 2023-12-19T14:40:20.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040ad6e0 con 0x7f4f0406c7c0 2023-12-19T14:40:20.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040b0f90 con 0x7f4f041552f0 2023-12-19T14:40:20.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39746/0 (socket says 172.21.15.50:39746) 2023-12-19T14:40:20.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 -- 172.21.15.50:0/713527700 learned_addr learned my addr 172.21.15.50:0/713527700 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:20.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f02d9d700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f041552f0 0x7f4f040a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f0259c700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f0406c7c0 0x7f4f040ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 -- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f041552f0 msgr2=0x7f4f040a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f041552f0 0x7f4f040a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 -- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f0406c7c0 msgr2=0x7f4f040ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f0406c7c0 0x7f4f040ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 -- 172.21.15.50:0/713527700 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec009420 con 0x7f4f040ae520 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efbfff700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0030c0 tx=0x7f4eec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec01cbe0 con 0x7f4f040ae520 2023-12-19T14:40:20.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec01cd40 con 0x7f4f040ae520 2023-12-19T14:40:20.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f040b1270 con 0x7f4f040ae520 2023-12-19T14:40:20.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec01a6a0 con 0x7f4f040ae520 2023-12-19T14:40:20.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.511+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f040b1830 con 0x7f4f040ae520 2023-12-19T14:40:20.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.515+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0416a380 con 0x7f4f040ae520 2023-12-19T14:40:20.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.515+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4eec01aad0 con 0x7f4f040ae520 2023-12-19T14:40:20.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.515+0000 7f4efa7fc700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 0x7f4ee4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.515+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4eec07e1f0 con 0x7f4f040ae520 2023-12-19T14:40:20.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.515+0000 7f4f0259c700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 0x7f4ee4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.519+0000 7f4f0259c700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 0x7f4ee4062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4001200 tx=0x7f4ef4001520 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.523+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4f0416a380 con 0x7f4f040ae520 2023-12-19T14:40:20.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.691+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f04071120 con 0x7f4ee4060550 2023-12-19T14:40:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.707+0000 7f4efa7fc700 1 -- 172.21.15.50:0/713527700 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283484 (secure 0 0 0) 0x7f4f04071120 con 0x7f4ee4060550 2023-12-19T14:40:20.717 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 msgr2=0x7f4ee4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 0x7f4ee4062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4001200 tx=0x7f4ef4001520 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 msgr2=0x7f4f040a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4eec0030c0 tx=0x7f4eec016000 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 shutdown_connections 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ee4060550 0x7f4ee4062a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4f041552f0 0x7f4f040a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f040ae520 0x7f4f040a8850 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 --2- 172.21.15.50:0/713527700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f0406c7c0 0x7f4f040ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 >> 172.21.15.50:0/713527700 conn(0x7f4f040c8850 msgr2=0x7f4f040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:20.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 shutdown_connections 2023-12-19T14:40:20.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.715+0000 7f4f08c2e700 1 -- 172.21.15.50:0/713527700 wait complete. 2023-12-19T14:40:20.722 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:40:20.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-12-19T14:40:20.153579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399000,"kb_used_data":4312,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575720,"statfs":{"total":773094113280,"available":770637537280,"internally_reserved":0,"allocated":4415488,"data_stored":2533784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001747"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011324+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.015299+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010786+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032050+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032514+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994572+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984445+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012508+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996425+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996909+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013771+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991932+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989734+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994060+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991210+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995599+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981853+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:07.226632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032685+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.980787+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:12.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991379+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.985539+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989176+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031534+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.982597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:03.243160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031182+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981065+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:50.763905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992991+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981044+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:20.673959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981068+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:16.111460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988543+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030864+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.998033+0000","last_change":"2023-12-19T14:40:14.998033+0000","last_active":"2023-12-19T14:40:14.998033+0000","last_peered":"2023-12-19T14:40:14.998033+0000","last_clean":"2023-12-19T14:40:14.998033+0000","last_became_active":"2023-12-19T14:40:14.997769+0000","last_became_peered":"2023-12-19T14:40:14.997769+0000","last_unstale":"2023-12-19T14:40:14.998033+0000","last_undegraded":"2023-12-19T14:40:14.998033+0000","last_fullsized":"2023-12-19T14:40:14.998033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:05.375208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031880+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022907+0000","last_change":"2023-12-19T14:40:15.022907+0000","last_active":"2023-12-19T14:40:15.022907+0000","last_peered":"2023-12-19T14:40:15.022907+0000","last_clean":"2023-12-19T14:40:15.022907+0000","last_became_active":"2023-12-19T14:40:15.022603+0000","last_became_peered":"2023-12-19T14:40:15.022603+0000","last_unstale":"2023-12-19T14:40:15.022907+0000","last_undegraded":"2023-12-19T14:40:15.022907+0000","last_fullsized":"2023-12-19T14:40:15.022907+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987338+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021930+0000","last_change":"2023-12-19T14:40:15.021930+0000","last_active":"2023-12-19T14:40:15.021930+0000","last_peered":"2023-12-19T14:40:15.021930+0000","last_clean":"2023-12-19T14:40:15.021930+0000","last_became_active":"2023-12-19T14:40:15.021586+0000","last_became_peered":"2023-12-19T14:40:15.021586+0000","last_unstale":"2023-12-19T14:40:15.021930+0000","last_undegraded":"2023-12-19T14:40:15.021930+0000","last_fullsized":"2023-12-19T14:40:15.021930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011878+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026312+0000","last_change":"2023-12-19T14:40:15.026312+0000","last_active":"2023-12-19T14:40:15.026312+0000","last_peered":"2023-12-19T14:40:15.026312+0000","last_clean":"2023-12-19T14:40:15.026312+0000","last_became_active":"2023-12-19T14:40:15.025957+0000","last_became_peered":"2023-12-19T14:40:15.025957+0000","last_unstale":"2023-12-19T14:40:15.026312+0000","last_undegraded":"2023-12-19T14:40:15.026312+0000","last_fullsized":"2023-12-19T14:40:15.026312+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014928+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025292+0000","last_change":"2023-12-19T14:40:15.025292+0000","last_active":"2023-12-19T14:40:15.025292+0000","last_peered":"2023-12-19T14:40:15.025292+0000","last_clean":"2023-12-19T14:40:15.025292+0000","last_became_active":"2023-12-19T14:40:15.024103+0000","last_became_peered":"2023-12-19T14:40:15.024103+0000","last_unstale":"2023-12-19T14:40:15.025292+0000","last_undegraded":"2023-12-19T14:40:15.025292+0000","last_fullsized":"2023-12-19T14:40:15.025292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:59.010045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992953+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.004455+0000","last_change":"2023-12-19T14:40:15.004455+0000","last_active":"2023-12-19T14:40:15.004455+0000","last_peered":"2023-12-19T14:40:15.004455+0000","last_clean":"2023-12-19T14:40:15.004455+0000","last_became_active":"2023-12-19T14:40:15.004022+0000","last_became_peered":"2023-12-19T14:40:15.004022+0000","last_unstale":"2023-12-19T14:40:15.004455+0000","last_undegraded":"2023-12-19T14:40:15.004455+0000","last_fullsized":"2023-12-19T14:40:15.004455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.986757+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.015560+0000","last_change":"2023-12-19T14:40:15.015560+0000","last_active":"2023-12-19T14:40:15.015560+0000","last_peered":"2023-12-19T14:40:15.015560+0000","last_clean":"2023-12-19T14:40:15.015560+0000","last_became_active":"2023-12-19T14:40:15.015177+0000","last_became_peered":"2023-12-19T14:40:15.015177+0000","last_unstale":"2023-12-19T14:40:15.015560+0000","last_undegraded":"2023-12-19T14:40:15.015560+0000","last_fullsized":"2023-12-19T14:40:15.015560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.994802+0000","last_change":"2023-12-19T14:40:14.994802+0000","last_active":"2023-12-19T14:40:14.994802+0000","last_peered":"2023-12-19T14:40:14.994802+0000","last_clean":"2023-12-19T14:40:14.994802+0000","last_became_active":"2023-12-19T14:40:14.994524+0000","last_became_peered":"2023-12-19T14:40:14.994524+0000","last_unstale":"2023-12-19T14:40:14.994802+0000","last_undegraded":"2023-12-19T14:40:14.994802+0000","last_fullsized":"2023-12-19T14:40:14.994802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031430+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024050+0000","last_change":"2023-12-19T14:40:15.024050+0000","last_active":"2023-12-19T14:40:15.024050+0000","last_peered":"2023-12-19T14:40:15.024050+0000","last_clean":"2023-12-19T14:40:15.024050+0000","last_became_active":"2023-12-19T14:40:15.023710+0000","last_became_peered":"2023-12-19T14:40:15.023710+0000","last_unstale":"2023-12-19T14:40:15.024050+0000","last_undegraded":"2023-12-19T14:40:15.024050+0000","last_fullsized":"2023-12-19T14:40:15.024050+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990860+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025954+0000","last_change":"2023-12-19T14:40:15.025954+0000","last_active":"2023-12-19T14:40:15.025954+0000","last_peered":"2023-12-19T14:40:15.025954+0000","last_clean":"2023-12-19T14:40:15.025954+0000","last_became_active":"2023-12-19T14:40:15.025658+0000","last_became_peered":"2023-12-19T14:40:15.025658+0000","last_unstale":"2023-12-19T14:40:15.025954+0000","last_undegraded":"2023-12-19T14:40:15.025954+0000","last_fullsized":"2023-12-19T14:40:15.025954+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010376+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023326+0000","last_change":"2023-12-19T14:40:15.023326+0000","last_active":"2023-12-19T14:40:15.023326+0000","last_peered":"2023-12-19T14:40:15.023326+0000","last_clean":"2023-12-19T14:40:15.023326+0000","last_became_active":"2023-12-19T14:40:15.022920+0000","last_became_peered":"2023-12-19T14:40:15.022920+0000","last_unstale":"2023-12-19T14:40:15.023326+0000","last_undegraded":"2023-12-19T14:40:15.023326+0000","last_fullsized":"2023-12-19T14:40:15.023326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:44.100285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987923+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023075+0000","last_change":"2023-12-19T14:40:15.023075+0000","last_active":"2023-12-19T14:40:15.023075+0000","last_peered":"2023-12-19T14:40:15.023075+0000","last_clean":"2023-12-19T14:40:15.023075+0000","last_became_active":"2023-12-19T14:40:15.022756+0000","last_became_peered":"2023-12-19T14:40:15.022756+0000","last_unstale":"2023-12-19T14:40:15.023075+0000","last_undegraded":"2023-12-19T14:40:15.023075+0000","last_fullsized":"2023-12-19T14:40:15.023075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014351+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026805+0000","last_change":"2023-12-19T14:40:15.026805+0000","last_active":"2023-12-19T14:40:15.026805+0000","last_peered":"2023-12-19T14:40:15.026805+0000","last_clean":"2023-12-19T14:40:15.026805+0000","last_became_active":"2023-12-19T14:40:15.025038+0000","last_became_peered":"2023-12-19T14:40:15.025038+0000","last_unstale":"2023-12-19T14:40:15.026805+0000","last_undegraded":"2023-12-19T14:40:15.026805+0000","last_fullsized":"2023-12-19T14:40:15.026805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984566+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.017622+0000","last_change":"2023-12-19T14:40:15.017622+0000","last_active":"2023-12-19T14:40:15.017622+0000","last_peered":"2023-12-19T14:40:15.017622+0000","last_clean":"2023-12-19T14:40:15.017622+0000","last_became_active":"2023-12-19T14:40:15.017316+0000","last_became_peered":"2023-12-19T14:40:15.017316+0000","last_unstale":"2023-12-19T14:40:15.017622+0000","last_undegraded":"2023-12-19T14:40:15.017622+0000","last_fullsized":"2023-12-19T14:40:15.017622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989643+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001820+0000","last_change":"2023-12-19T14:40:15.001820+0000","last_active":"2023-12-19T14:40:15.001820+0000","last_peered":"2023-12-19T14:40:15.001820+0000","last_clean":"2023-12-19T14:40:15.001820+0000","last_became_active":"2023-12-19T14:40:15.001305+0000","last_became_peered":"2023-12-19T14:40:15.001305+0000","last_unstale":"2023-12-19T14:40:15.001820+0000","last_undegraded":"2023-12-19T14:40:15.001820+0000","last_fullsized":"2023-12-19T14:40:15.001820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:52.732501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013809+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003285+0000","last_change":"2023-12-19T14:40:15.003285+0000","last_active":"2023-12-19T14:40:15.003285+0000","last_peered":"2023-12-19T14:40:15.003285+0000","last_clean":"2023-12-19T14:40:15.003285+0000","last_became_active":"2023-12-19T14:40:15.002941+0000","last_became_peered":"2023-12-19T14:40:15.002941+0000","last_unstale":"2023-12-19T14:40:15.003285+0000","last_undegraded":"2023-12-19T14:40:15.003285+0000","last_fullsized":"2023-12-19T14:40:15.003285+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012459+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003942+0000","last_change":"2023-12-19T14:40:15.003942+0000","last_active":"2023-12-19T14:40:15.003942+0000","last_peered":"2023-12-19T14:40:15.003942+0000","last_clean":"2023-12-19T14:40:15.003942+0000","last_became_active":"2023-12-19T14:40:15.003631+0000","last_became_peered":"2023-12-19T14:40:15.003631+0000","last_unstale":"2023-12-19T14:40:15.003942+0000","last_undegraded":"2023-12-19T14:40:15.003942+0000","last_fullsized":"2023-12-19T14:40:15.003942+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:22.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995831+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.002317+0000","last_change":"2023-12-19T14:40:15.002317+0000","last_active":"2023-12-19T14:40:15.002317+0000","last_peered":"2023-12-19T14:40:15.002317+0000","last_clean":"2023-12-19T14:40:15.002317+0000","last_became_active":"2023-12-19T14:40:15.002056+0000","last_became_peered":"2023-12-19T14:40:15.002056+0000","last_unstale":"2023-12-19T14:40:15.002317+0000","last_undegraded":"2023-12-19T14:40:15.002317+0000","last_fullsized":"2023-12-19T14:40:15.002317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:38.560751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988174+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.010764+0000","last_change":"2023-12-19T14:40:15.010764+0000","last_active":"2023-12-19T14:40:15.010764+0000","last_peered":"2023-12-19T14:40:15.010764+0000","last_clean":"2023-12-19T14:40:15.010764+0000","last_became_active":"2023-12-19T14:40:15.010427+0000","last_became_peered":"2023-12-19T14:40:15.010427+0000","last_unstale":"2023-12-19T14:40:15.010764+0000","last_undegraded":"2023-12-19T14:40:15.010764+0000","last_fullsized":"2023-12-19T14:40:15.010764+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:11.000395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031490+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021360+0000","last_change":"2023-12-19T14:40:15.021360+0000","last_active":"2023-12-19T14:40:15.021360+0000","last_peered":"2023-12-19T14:40:15.021360+0000","last_clean":"2023-12-19T14:40:15.021360+0000","last_became_active":"2023-12-19T14:40:15.020201+0000","last_became_peered":"2023-12-19T14:40:15.020201+0000","last_unstale":"2023-12-19T14:40:15.021360+0000","last_undegraded":"2023-12-19T14:40:15.021360+0000","last_fullsized":"2023-12-19T14:40:15.021360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989347+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026807+0000","last_change":"2023-12-19T14:40:15.026807+0000","last_active":"2023-12-19T14:40:15.026807+0000","last_peered":"2023-12-19T14:40:15.026807+0000","last_clean":"2023-12-19T14:40:15.026807+0000","last_became_active":"2023-12-19T14:40:15.026425+0000","last_became_peered":"2023-12-19T14:40:15.026425+0000","last_unstale":"2023-12-19T14:40:15.026807+0000","last_undegraded":"2023-12-19T14:40:15.026807+0000","last_fullsized":"2023-12-19T14:40:15.026807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011013+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.013522+0000","last_change":"2023-12-19T14:40:15.013522+0000","last_active":"2023-12-19T14:40:15.013522+0000","last_peered":"2023-12-19T14:40:15.013522+0000","last_clean":"2023-12-19T14:40:15.013522+0000","last_became_active":"2023-12-19T14:40:15.013241+0000","last_became_peered":"2023-12-19T14:40:15.013241+0000","last_unstale":"2023-12-19T14:40:15.013522+0000","last_undegraded":"2023-12-19T14:40:15.013522+0000","last_fullsized":"2023-12-19T14:40:15.013522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:49.758106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996537+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001340+0000","last_change":"2023-12-19T14:40:15.001340+0000","last_active":"2023-12-19T14:40:15.001340+0000","last_peered":"2023-12-19T14:40:15.001340+0000","last_clean":"2023-12-19T14:40:15.001340+0000","last_became_active":"2023-12-19T14:40:15.001076+0000","last_became_peered":"2023-12-19T14:40:15.001076+0000","last_unstale":"2023-12-19T14:40:15.001340+0000","last_undegraded":"2023-12-19T14:40:15.001340+0000","last_fullsized":"2023-12-19T14:40:15.001340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984661+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024120+0000","last_change":"2023-12-19T14:40:15.024120+0000","last_active":"2023-12-19T14:40:15.024120+0000","last_peered":"2023-12-19T14:40:15.024120+0000","last_clean":"2023-12-19T14:40:15.024120+0000","last_became_active":"2023-12-19T14:40:15.023840+0000","last_became_peered":"2023-12-19T14:40:15.023840+0000","last_unstale":"2023-12-19T14:40:15.024120+0000","last_undegraded":"2023-12-19T14:40:15.024120+0000","last_fullsized":"2023-12-19T14:40:15.024120+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030693+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022435+0000","last_change":"2023-12-19T14:40:15.022435+0000","last_active":"2023-12-19T14:40:15.022435+0000","last_peered":"2023-12-19T14:40:15.022435+0000","last_clean":"2023-12-19T14:40:15.022435+0000","last_became_active":"2023-12-19T14:40:15.022169+0000","last_became_peered":"2023-12-19T14:40:15.022169+0000","last_unstale":"2023-12-19T14:40:15.022435+0000","last_undegraded":"2023-12-19T14:40:15.022435+0000","last_fullsized":"2023-12-19T14:40:15.022435+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:43.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990365+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.012638+0000","last_change":"2023-12-19T14:40:15.012638+0000","last_active":"2023-12-19T14:40:15.012638+0000","last_peered":"2023-12-19T14:40:15.012638+0000","last_clean":"2023-12-19T14:40:15.012638+0000","last_became_active":"2023-12-19T14:40:15.012351+0000","last_became_peered":"2023-12-19T14:40:15.012351+0000","last_unstale":"2023-12-19T14:40:15.012638+0000","last_undegraded":"2023-12-19T14:40:15.012638+0000","last_fullsized":"2023-12-19T14:40:15.012638+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:15.105662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013206+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.029026+0000","last_change":"2023-12-19T14:40:15.029026+0000","last_active":"2023-12-19T14:40:15.029026+0000","last_peered":"2023-12-19T14:40:15.029026+0000","last_clean":"2023-12-19T14:40:15.029026+0000","last_became_active":"2023-12-19T14:40:15.028694+0000","last_became_peered":"2023-12-19T14:40:15.028694+0000","last_unstale":"2023-12-19T14:40:15.029026+0000","last_undegraded":"2023-12-19T14:40:15.029026+0000","last_fullsized":"2023-12-19T14:40:15.029026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013783+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023303+0000","last_change":"2023-12-19T14:40:15.023303+0000","last_active":"2023-12-19T14:40:15.023303+0000","last_peered":"2023-12-19T14:40:15.023303+0000","last_clean":"2023-12-19T14:40:15.023303+0000","last_became_active":"2023-12-19T14:40:15.022812+0000","last_became_peered":"2023-12-19T14:40:15.022812+0000","last_unstale":"2023-12-19T14:40:15.023303+0000","last_undegraded":"2023-12-19T14:40:15.023303+0000","last_fullsized":"2023-12-19T14:40:15.023303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030633+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024761+0000","last_change":"2023-12-19T14:40:15.024761+0000","last_active":"2023-12-19T14:40:15.024761+0000","last_peered":"2023-12-19T14:40:15.024761+0000","last_clean":"2023-12-19T14:40:15.024761+0000","last_became_active":"2023-12-19T14:40:15.024406+0000","last_became_peered":"2023-12-19T14:40:15.024406+0000","last_unstale":"2023-12-19T14:40:15.024761+0000","last_undegraded":"2023-12-19T14:40:15.024761+0000","last_fullsized":"2023-12-19T14:40:15.024761+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:36.341963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026100+0000","last_change":"2023-12-19T14:40:15.026100+0000","last_active":"2023-12-19T14:40:15.026100+0000","last_peered":"2023-12-19T14:40:15.026100+0000","last_clean":"2023-12-19T14:40:15.026100+0000","last_became_active":"2023-12-19T14:40:15.025647+0000","last_became_peered":"2023-12-19T14:40:15.025647+0000","last_unstale":"2023-12-19T14:40:15.026100+0000","last_undegraded":"2023-12-19T14:40:15.026100+0000","last_fullsized":"2023-12-19T14:40:15.026100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024302+0000","last_change":"2023-12-19T14:40:15.024302+0000","last_active":"2023-12-19T14:40:15.024302+0000","last_peered":"2023-12-19T14:40:15.024302+0000","last_clean":"2023-12-19T14:40:15.024302+0000","last_became_active":"2023-12-19T14:40:15.024017+0000","last_became_peered":"2023-12-19T14:40:15.024017+0000","last_unstale":"2023-12-19T14:40:15.024302+0000","last_undegraded":"2023-12-19T14:40:15.024302+0000","last_fullsized":"2023-12-19T14:40:15.024302+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996301+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021965+0000","last_change":"2023-12-19T14:40:15.021965+0000","last_active":"2023-12-19T14:40:15.021965+0000","last_peered":"2023-12-19T14:40:15.021965+0000","last_clean":"2023-12-19T14:40:15.021965+0000","last_became_active":"2023-12-19T14:40:15.021467+0000","last_became_peered":"2023-12-19T14:40:15.021467+0000","last_unstale":"2023-12-19T14:40:15.021965+0000","last_undegraded":"2023-12-19T14:40:15.021965+0000","last_fullsized":"2023-12-19T14:40:15.021965+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-19T14:40:13.980266+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:40:13.980266+0000","last_peered":"2023-12-19T14:40:13.980266+0000","last_clean":"2023-12-19T14:40:13.980266+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:40:13.980266+0000","last_undegraded":"2023-12-19T14:40:13.980266+0000","last_fullsized":"2023-12-19T14:40:13.980266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298260,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073580,"statfs":{"total":96636764160,"available":96331345920,"internally_reserved":0,"allocated":995328,"data_stored":758106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.90500000000000003}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.494}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.161}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.55},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.522}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.91700000000000004}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.266},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":3.0990000000000002},"last":1.125}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.222},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.1579999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":1.254},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":3.194,"5min":3.194,"15min":3.194},"last":1.321}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0150000000000001,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3600000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.367,"5min":3.367,"15min":3.367},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.51}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.143}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.1850000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.419},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2709999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.173},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.2430000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.478},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.121}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.514},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.702}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.966},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.8300000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":2.0840000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.742,"5min":2.742,"15min":2.742},"last":1.917}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":5.2060000000000004,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":2.02},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":2.5289999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":4.476,"5min":4.476,"15min":4.476},"last":0.94499999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.212}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":1.347},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.621}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.867,"5min":1.867,"15min":1.867},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.5980000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301772,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070068,"statfs":{"total":96636764160,"available":96327749632,"internally_reserved":0,"allocated":397312,"data_stored":166727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.871},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2050000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.151},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.629}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.351}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":1.306},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":1.548}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.403},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":3.0619999999999998,"5min":3.0619999999999998,"15min":3.0619999999999998},"last":1.123}]},{"osd":3,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":1.236}]},{"osd":5,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.591},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.266}]},{"osd":6,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.49},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":1.329},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.7669999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069488,"statfs":{"total":96636764160,"available":96327155712,"internally_reserved":0,"allocated":991232,"data_stored":757095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.83799999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.413}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.482},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":1.1819999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.143}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.077,"5min":1.077,"15min":1.077},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:40:20.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:40:20.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.947+0000 7f5dabaf0700 1 Processor -- start 2023-12-19T14:40:20.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.947+0000 7f5dabaf0700 1 -- start start 2023-12-19T14:40:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.947+0000 7f5dabaf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4156c90 0x7f5da40cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5dabaf0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5da4157060 0x7f5da415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5dabaf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40cd5a0 con 0x7f5da40cda30 2023-12-19T14:40:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5dabaf0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40cc990 con 0x7f5da4157060 2023-12-19T14:40:20.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5dabaf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40ccc70 con 0x7f5da4156c90 2023-12-19T14:40:20.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da908b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5da40cda30 0x7f5da40c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39988/0 (socket says 172.21.15.50:39988) 2023-12-19T14:40:20.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da908b700 1 -- 172.21.15.50:0/465351557 learned_addr learned my addr 172.21.15.50:0/465351557 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:20.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 --2- 172.21.15.50:0/465351557 >> v2:172.21.15.125:3300/0 conn(0x7f5da4157060 0x7f5da415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da988c700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4156c90 0x7f5da40cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da888a700 1 -- 172.21.15.50:0/465351557 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520452335 0 0) 0x7f5da40cd5a0 con 0x7f5da40cda30 2023-12-19T14:40:20.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da888a700 1 -- 172.21.15.50:0/465351557 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d90003570 con 0x7f5da40cda30 2023-12-19T14:40:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 -- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4156c90 msgr2=0x7f5da40cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4156c90 0x7f5da40cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 -- 172.21.15.50:0/465351557 >> v1:172.21.15.50:6789/0 conn(0x7f5da40cda30 legacy=0x7f5da40c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 -- 172.21.15.50:0/465351557 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40ccf50 con 0x7f5da4157060 2023-12-19T14:40:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5daa08d700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4157060 0x7f5da415ab20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009e60 tx=0x7f5d9c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=342fb2a0fe32be5e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da888a700 1 -- 172.21.15.50:0/465351557 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00e070 con 0x7f5da4157060 2023-12-19T14:40:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da888a700 1 -- 172.21.15.50:0/465351557 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c00bab0 con 0x7f5da4157060 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.951+0000 7f5da888a700 1 -- 172.21.15.50:0/465351557 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00be70 con 0x7f5da4157060 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4157060 msgr2=0x7f5da415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4157060 0x7f5da415ab20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009e60 tx=0x7f5d9c009780 comp rx=0 tx=0).stop 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- 172.21.15.50:0/465351557 shutdown_connections 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4157060 0x7f5da415ab20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/465351557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4156c90 0x7f5da40cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- 172.21.15.50:0/465351557 >> 172.21.15.50:0/465351557 conn(0x7f5da406c7c0 msgr2=0x7f5da406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:20.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- 172.21.15.50:0/465351557 shutdown_connections 2023-12-19T14:40:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- 172.21.15.50:0/465351557 wait complete. 2023-12-19T14:40:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 Processor -- start 2023-12-19T14:40:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.955+0000 7f5dabaf0700 1 -- start start 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 0x7f5da4159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4157060 0x7f5da40af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4159e50 con 0x7f5da40cda30 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4158580 con 0x7f5da4156c90 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4157970 con 0x7f5da4157060 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39750/0 (socket says 172.21.15.50:39750) 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 -- 172.21.15.50:0/1298995998 learned_addr learned my addr 172.21.15.50:0/1298995998 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da908b700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 0x7f5da4159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5daa08d700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4157060 0x7f5da40af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 -- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4157060 msgr2=0x7f5da40af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4157060 0x7f5da40af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 -- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 msgr2=0x7f5da4159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 0x7f5da4159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 -- 172.21.15.50:0/1298995998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c009a20 con 0x7f5da40cda30 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da908b700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 0x7f5da4159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5da988c700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5da000ec80 tx=0x7f5da00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da000c850 con 0x7f5da40cda30 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da000c9b0 con 0x7f5da40cda30 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da4157c50 con 0x7f5da40cda30 2023-12-19T14:40:20.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da000cd50 con 0x7f5da40cda30 2023-12-19T14:40:20.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.959+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4158210 con 0x7f5da40cda30 2023-12-19T14:40:20.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.963+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4071120 con 0x7f5da40cda30 2023-12-19T14:40:20.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.963+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5da0013680 con 0x7f5da40cda30 2023-12-19T14:40:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.963+0000 7f5d99ffb700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 0x7f5d88062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.963+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5da007ccc0 con 0x7f5da40cda30 2023-12-19T14:40:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.967+0000 7f5da908b700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 0x7f5d88062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:20.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.967+0000 7f5da908b700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 0x7f5d88062a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5da415a590 tx=0x7f5d94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:20.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:20.971+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5da0083080 con 0x7f5da40cda30 2023-12-19T14:40:21.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.143+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5da40cbd00 con 0x7f5d88060550 2023-12-19T14:40:21.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.163+0000 7f5d99ffb700 1 -- 172.21.15.50:0/1298995998 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283484 (secure 0 0 0) 0x7f5da40cbd00 con 0x7f5d88060550 2023-12-19T14:40:21.174 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.167+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 msgr2=0x7f5d88062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.167+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 0x7f5d88062a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5da415a590 tx=0x7f5d94009040 comp rx=0 tx=0).stop 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.167+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 msgr2=0x7f5da41434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.167+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5da000ec80 tx=0x7f5da00091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 shutdown_connections 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5da4156c90 0x7f5da4159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5d88060550 0x7f5d88062a00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5da4157060 0x7f5da40af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 --2- 172.21.15.50:0/1298995998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5da40cda30 0x7f5da41434c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 >> 172.21.15.50:0/1298995998 conn(0x7f5da406c7c0 msgr2=0x7f5da40c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 shutdown_connections 2023-12-19T14:40:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.171+0000 7f5dabaf0700 1 -- 172.21.15.50:0/1298995998 wait complete. 2023-12-19T14:40:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:40:21.193 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-12-19T14:40:20.153579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399000,"kb_used_data":4312,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575720,"statfs":{"total":773094113280,"available":770637537280,"internally_reserved":0,"allocated":4415488,"data_stored":2533784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001747"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011324+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.015299+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010786+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032050+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032514+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994572+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984445+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012508+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996425+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996909+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013771+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991932+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989734+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994060+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991210+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995599+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981853+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:07.226632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032685+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.980787+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:12.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991379+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.985539+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989176+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031534+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.982597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:03.243160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031182+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981065+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:50.763905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992991+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981044+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:20.673959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981068+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:16.111460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988543+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030864+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.998033+0000","last_change":"2023-12-19T14:40:14.998033+0000","last_active":"2023-12-19T14:40:14.998033+0000","last_peered":"2023-12-19T14:40:14.998033+0000","last_clean":"2023-12-19T14:40:14.998033+0000","last_became_active":"2023-12-19T14:40:14.997769+0000","last_became_peered":"2023-12-19T14:40:14.997769+0000","last_unstale":"2023-12-19T14:40:14.998033+0000","last_undegraded":"2023-12-19T14:40:14.998033+0000","last_fullsized":"2023-12-19T14:40:14.998033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:05.375208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031880+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022907+0000","last_change":"2023-12-19T14:40:15.022907+0000","last_active":"2023-12-19T14:40:15.022907+0000","last_peered":"2023-12-19T14:40:15.022907+0000","last_clean":"2023-12-19T14:40:15.022907+0000","last_became_active":"2023-12-19T14:40:15.022603+0000","last_became_peered":"2023-12-19T14:40:15.022603+0000","last_unstale":"2023-12-19T14:40:15.022907+0000","last_undegraded":"2023-12-19T14:40:15.022907+0000","last_fullsized":"2023-12-19T14:40:15.022907+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987338+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021930+0000","last_change":"2023-12-19T14:40:15.021930+0000","last_active":"2023-12-19T14:40:15.021930+0000","last_peered":"2023-12-19T14:40:15.021930+0000","last_clean":"2023-12-19T14:40:15.021930+0000","last_became_active":"2023-12-19T14:40:15.021586+0000","last_became_peered":"2023-12-19T14:40:15.021586+0000","last_unstale":"2023-12-19T14:40:15.021930+0000","last_undegraded":"2023-12-19T14:40:15.021930+0000","last_fullsized":"2023-12-19T14:40:15.021930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011878+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026312+0000","last_change":"2023-12-19T14:40:15.026312+0000","last_active":"2023-12-19T14:40:15.026312+0000","last_peered":"2023-12-19T14:40:15.026312+0000","last_clean":"2023-12-19T14:40:15.026312+0000","last_became_active":"2023-12-19T14:40:15.025957+0000","last_became_peered":"2023-12-19T14:40:15.025957+0000","last_unstale":"2023-12-19T14:40:15.026312+0000","last_undegraded":"2023-12-19T14:40:15.026312+0000","last_fullsized":"2023-12-19T14:40:15.026312+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014928+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025292+0000","last_change":"2023-12-19T14:40:15.025292+0000","last_active":"2023-12-19T14:40:15.025292+0000","last_peered":"2023-12-19T14:40:15.025292+0000","last_clean":"2023-12-19T14:40:15.025292+0000","last_became_active":"2023-12-19T14:40:15.024103+0000","last_became_peered":"2023-12-19T14:40:15.024103+0000","last_unstale":"2023-12-19T14:40:15.025292+0000","last_undegraded":"2023-12-19T14:40:15.025292+0000","last_fullsized":"2023-12-19T14:40:15.025292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:59.010045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992953+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.004455+0000","last_change":"2023-12-19T14:40:15.004455+0000","last_active":"2023-12-19T14:40:15.004455+0000","last_peered":"2023-12-19T14:40:15.004455+0000","last_clean":"2023-12-19T14:40:15.004455+0000","last_became_active":"2023-12-19T14:40:15.004022+0000","last_became_peered":"2023-12-19T14:40:15.004022+0000","last_unstale":"2023-12-19T14:40:15.004455+0000","last_undegraded":"2023-12-19T14:40:15.004455+0000","last_fullsized":"2023-12-19T14:40:15.004455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.986757+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.015560+0000","last_change":"2023-12-19T14:40:15.015560+0000","last_active":"2023-12-19T14:40:15.015560+0000","last_peered":"2023-12-19T14:40:15.015560+0000","last_clean":"2023-12-19T14:40:15.015560+0000","last_became_active":"2023-12-19T14:40:15.015177+0000","last_became_peered":"2023-12-19T14:40:15.015177+0000","last_unstale":"2023-12-19T14:40:15.015560+0000","last_undegraded":"2023-12-19T14:40:15.015560+0000","last_fullsized":"2023-12-19T14:40:15.015560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.994802+0000","last_change":"2023-12-19T14:40:14.994802+0000","last_active":"2023-12-19T14:40:14.994802+0000","last_peered":"2023-12-19T14:40:14.994802+0000","last_clean":"2023-12-19T14:40:14.994802+0000","last_became_active":"2023-12-19T14:40:14.994524+0000","last_became_peered":"2023-12-19T14:40:14.994524+0000","last_unstale":"2023-12-19T14:40:14.994802+0000","last_undegraded":"2023-12-19T14:40:14.994802+0000","last_fullsized":"2023-12-19T14:40:14.994802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031430+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024050+0000","last_change":"2023-12-19T14:40:15.024050+0000","last_active":"2023-12-19T14:40:15.024050+0000","last_peered":"2023-12-19T14:40:15.024050+0000","last_clean":"2023-12-19T14:40:15.024050+0000","last_became_active":"2023-12-19T14:40:15.023710+0000","last_became_peered":"2023-12-19T14:40:15.023710+0000","last_unstale":"2023-12-19T14:40:15.024050+0000","last_undegraded":"2023-12-19T14:40:15.024050+0000","last_fullsized":"2023-12-19T14:40:15.024050+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990860+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025954+0000","last_change":"2023-12-19T14:40:15.025954+0000","last_active":"2023-12-19T14:40:15.025954+0000","last_peered":"2023-12-19T14:40:15.025954+0000","last_clean":"2023-12-19T14:40:15.025954+0000","last_became_active":"2023-12-19T14:40:15.025658+0000","last_became_peered":"2023-12-19T14:40:15.025658+0000","last_unstale":"2023-12-19T14:40:15.025954+0000","last_undegraded":"2023-12-19T14:40:15.025954+0000","last_fullsized":"2023-12-19T14:40:15.025954+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010376+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023326+0000","last_change":"2023-12-19T14:40:15.023326+0000","last_active":"2023-12-19T14:40:15.023326+0000","last_peered":"2023-12-19T14:40:15.023326+0000","last_clean":"2023-12-19T14:40:15.023326+0000","last_became_active":"2023-12-19T14:40:15.022920+0000","last_became_peered":"2023-12-19T14:40:15.022920+0000","last_unstale":"2023-12-19T14:40:15.023326+0000","last_undegraded":"2023-12-19T14:40:15.023326+0000","last_fullsized":"2023-12-19T14:40:15.023326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:44.100285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987923+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023075+0000","last_change":"2023-12-19T14:40:15.023075+0000","last_active":"2023-12-19T14:40:15.023075+0000","last_peered":"2023-12-19T14:40:15.023075+0000","last_clean":"2023-12-19T14:40:15.023075+0000","last_became_active":"2023-12-19T14:40:15.022756+0000","last_became_peered":"2023-12-19T14:40:15.022756+0000","last_unstale":"2023-12-19T14:40:15.023075+0000","last_undegraded":"2023-12-19T14:40:15.023075+0000","last_fullsized":"2023-12-19T14:40:15.023075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014351+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026805+0000","last_change":"2023-12-19T14:40:15.026805+0000","last_active":"2023-12-19T14:40:15.026805+0000","last_peered":"2023-12-19T14:40:15.026805+0000","last_clean":"2023-12-19T14:40:15.026805+0000","last_became_active":"2023-12-19T14:40:15.025038+0000","last_became_peered":"2023-12-19T14:40:15.025038+0000","last_unstale":"2023-12-19T14:40:15.026805+0000","last_undegraded":"2023-12-19T14:40:15.026805+0000","last_fullsized":"2023-12-19T14:40:15.026805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984566+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.017622+0000","last_change":"2023-12-19T14:40:15.017622+0000","last_active":"2023-12-19T14:40:15.017622+0000","last_peered":"2023-12-19T14:40:15.017622+0000","last_clean":"2023-12-19T14:40:15.017622+0000","last_became_active":"2023-12-19T14:40:15.017316+0000","last_became_peered":"2023-12-19T14:40:15.017316+0000","last_unstale":"2023-12-19T14:40:15.017622+0000","last_undegraded":"2023-12-19T14:40:15.017622+0000","last_fullsized":"2023-12-19T14:40:15.017622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989643+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001820+0000","last_change":"2023-12-19T14:40:15.001820+0000","last_active":"2023-12-19T14:40:15.001820+0000","last_peered":"2023-12-19T14:40:15.001820+0000","last_clean":"2023-12-19T14:40:15.001820+0000","last_became_active":"2023-12-19T14:40:15.001305+0000","last_became_peered":"2023-12-19T14:40:15.001305+0000","last_unstale":"2023-12-19T14:40:15.001820+0000","last_undegraded":"2023-12-19T14:40:15.001820+0000","last_fullsized":"2023-12-19T14:40:15.001820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:52.732501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013809+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003285+0000","last_change":"2023-12-19T14:40:15.003285+0000","last_active":"2023-12-19T14:40:15.003285+0000","last_peered":"2023-12-19T14:40:15.003285+0000","last_clean":"2023-12-19T14:40:15.003285+0000","last_became_active":"2023-12-19T14:40:15.002941+0000","last_became_peered":"2023-12-19T14:40:15.002941+0000","last_unstale":"2023-12-19T14:40:15.003285+0000","last_undegraded":"2023-12-19T14:40:15.003285+0000","last_fullsized":"2023-12-19T14:40:15.003285+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012459+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003942+0000","last_change":"2023-12-19T14:40:15.003942+0000","last_active":"2023-12-19T14:40:15.003942+0000","last_peered":"2023-12-19T14:40:15.003942+0000","last_clean":"2023-12-19T14:40:15.003942+0000","last_became_active":"2023-12-19T14:40:15.003631+0000","last_became_peered":"2023-12-19T14:40:15.003631+0000","last_unstale":"2023-12-19T14:40:15.003942+0000","last_undegraded":"2023-12-19T14:40:15.003942+0000","last_fullsized":"2023-12-19T14:40:15.003942+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:22.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995831+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.002317+0000","last_change":"2023-12-19T14:40:15.002317+0000","last_active":"2023-12-19T14:40:15.002317+0000","last_peered":"2023-12-19T14:40:15.002317+0000","last_clean":"2023-12-19T14:40:15.002317+0000","last_became_active":"2023-12-19T14:40:15.002056+0000","last_became_peered":"2023-12-19T14:40:15.002056+0000","last_unstale":"2023-12-19T14:40:15.002317+0000","last_undegraded":"2023-12-19T14:40:15.002317+0000","last_fullsized":"2023-12-19T14:40:15.002317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:38.560751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988174+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.010764+0000","last_change":"2023-12-19T14:40:15.010764+0000","last_active":"2023-12-19T14:40:15.010764+0000","last_peered":"2023-12-19T14:40:15.010764+0000","last_clean":"2023-12-19T14:40:15.010764+0000","last_became_active":"2023-12-19T14:40:15.010427+0000","last_became_peered":"2023-12-19T14:40:15.010427+0000","last_unstale":"2023-12-19T14:40:15.010764+0000","last_undegraded":"2023-12-19T14:40:15.010764+0000","last_fullsized":"2023-12-19T14:40:15.010764+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:11.000395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031490+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021360+0000","last_change":"2023-12-19T14:40:15.021360+0000","last_active":"2023-12-19T14:40:15.021360+0000","last_peered":"2023-12-19T14:40:15.021360+0000","last_clean":"2023-12-19T14:40:15.021360+0000","last_became_active":"2023-12-19T14:40:15.020201+0000","last_became_peered":"2023-12-19T14:40:15.020201+0000","last_unstale":"2023-12-19T14:40:15.021360+0000","last_undegraded":"2023-12-19T14:40:15.021360+0000","last_fullsized":"2023-12-19T14:40:15.021360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989347+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026807+0000","last_change":"2023-12-19T14:40:15.026807+0000","last_active":"2023-12-19T14:40:15.026807+0000","last_peered":"2023-12-19T14:40:15.026807+0000","last_clean":"2023-12-19T14:40:15.026807+0000","last_became_active":"2023-12-19T14:40:15.026425+0000","last_became_peered":"2023-12-19T14:40:15.026425+0000","last_unstale":"2023-12-19T14:40:15.026807+0000","last_undegraded":"2023-12-19T14:40:15.026807+0000","last_fullsized":"2023-12-19T14:40:15.026807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011013+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.013522+0000","last_change":"2023-12-19T14:40:15.013522+0000","last_active":"2023-12-19T14:40:15.013522+0000","last_peered":"2023-12-19T14:40:15.013522+0000","last_clean":"2023-12-19T14:40:15.013522+0000","last_became_active":"2023-12-19T14:40:15.013241+0000","last_became_peered":"2023-12-19T14:40:15.013241+0000","last_unstale":"2023-12-19T14:40:15.013522+0000","last_undegraded":"2023-12-19T14:40:15.013522+0000","last_fullsized":"2023-12-19T14:40:15.013522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:49.758106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996537+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001340+0000","last_change":"2023-12-19T14:40:15.001340+0000","last_active":"2023-12-19T14:40:15.001340+0000","last_peered":"2023-12-19T14:40:15.001340+0000","last_clean":"2023-12-19T14:40:15.001340+0000","last_became_active":"2023-12-19T14:40:15.001076+0000","last_became_peered":"2023-12-19T14:40:15.001076+0000","last_unstale":"2023-12-19T14:40:15.001340+0000","last_undegraded":"2023-12-19T14:40:15.001340+0000","last_fullsized":"2023-12-19T14:40:15.001340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984661+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024120+0000","last_change":"2023-12-19T14:40:15.024120+0000","last_active":"2023-12-19T14:40:15.024120+0000","last_peered":"2023-12-19T14:40:15.024120+0000","last_clean":"2023-12-19T14:40:15.024120+0000","last_became_active":"2023-12-19T14:40:15.023840+0000","last_became_peered":"2023-12-19T14:40:15.023840+0000","last_unstale":"2023-12-19T14:40:15.024120+0000","last_undegraded":"2023-12-19T14:40:15.024120+0000","last_fullsized":"2023-12-19T14:40:15.024120+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030693+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022435+0000","last_change":"2023-12-19T14:40:15.022435+0000","last_active":"2023-12-19T14:40:15.022435+0000","last_peered":"2023-12-19T14:40:15.022435+0000","last_clean":"2023-12-19T14:40:15.022435+0000","last_became_active":"2023-12-19T14:40:15.022169+0000","last_became_peered":"2023-12-19T14:40:15.022169+0000","last_unstale":"2023-12-19T14:40:15.022435+0000","last_undegraded":"2023-12-19T14:40:15.022435+0000","last_fullsized":"2023-12-19T14:40:15.022435+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:43.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990365+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.012638+0000","last_change":"2023-12-19T14:40:15.012638+0000","last_active":"2023-12-19T14:40:15.012638+0000","last_peered":"2023-12-19T14:40:15.012638+0000","last_clean":"2023-12-19T14:40:15.012638+0000","last_became_active":"2023-12-19T14:40:15.012351+0000","last_became_peered":"2023-12-19T14:40:15.012351+0000","last_unstale":"2023-12-19T14:40:15.012638+0000","last_undegraded":"2023-12-19T14:40:15.012638+0000","last_fullsized":"2023-12-19T14:40:15.012638+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:15.105662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013206+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.029026+0000","last_change":"2023-12-19T14:40:15.029026+0000","last_active":"2023-12-19T14:40:15.029026+0000","last_peered":"2023-12-19T14:40:15.029026+0000","last_clean":"2023-12-19T14:40:15.029026+0000","last_became_active":"2023-12-19T14:40:15.028694+0000","last_became_peered":"2023-12-19T14:40:15.028694+0000","last_unstale":"2023-12-19T14:40:15.029026+0000","last_undegraded":"2023-12-19T14:40:15.029026+0000","last_fullsized":"2023-12-19T14:40:15.029026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013783+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023303+0000","last_change":"2023-12-19T14:40:15.023303+0000","last_active":"2023-12-19T14:40:15.023303+0000","last_peered":"2023-12-19T14:40:15.023303+0000","last_clean":"2023-12-19T14:40:15.023303+0000","last_became_active":"2023-12-19T14:40:15.022812+0000","last_became_peered":"2023-12-19T14:40:15.022812+0000","last_unstale":"2023-12-19T14:40:15.023303+0000","last_undegraded":"2023-12-19T14:40:15.023303+0000","last_fullsized":"2023-12-19T14:40:15.023303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030633+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024761+0000","last_change":"2023-12-19T14:40:15.024761+0000","last_active":"2023-12-19T14:40:15.024761+0000","last_peered":"2023-12-19T14:40:15.024761+0000","last_clean":"2023-12-19T14:40:15.024761+0000","last_became_active":"2023-12-19T14:40:15.024406+0000","last_became_peered":"2023-12-19T14:40:15.024406+0000","last_unstale":"2023-12-19T14:40:15.024761+0000","last_undegraded":"2023-12-19T14:40:15.024761+0000","last_fullsized":"2023-12-19T14:40:15.024761+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:36.341963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026100+0000","last_change":"2023-12-19T14:40:15.026100+0000","last_active":"2023-12-19T14:40:15.026100+0000","last_peered":"2023-12-19T14:40:15.026100+0000","last_clean":"2023-12-19T14:40:15.026100+0000","last_became_active":"2023-12-19T14:40:15.025647+0000","last_became_peered":"2023-12-19T14:40:15.025647+0000","last_unstale":"2023-12-19T14:40:15.026100+0000","last_undegraded":"2023-12-19T14:40:15.026100+0000","last_fullsized":"2023-12-19T14:40:15.026100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024302+0000","last_change":"2023-12-19T14:40:15.024302+0000","last_active":"2023-12-19T14:40:15.024302+0000","last_peered":"2023-12-19T14:40:15.024302+0000","last_clean":"2023-12-19T14:40:15.024302+0000","last_became_active":"2023-12-19T14:40:15.024017+0000","last_became_peered":"2023-12-19T14:40:15.024017+0000","last_unstale":"2023-12-19T14:40:15.024302+0000","last_undegraded":"2023-12-19T14:40:15.024302+0000","last_fullsized":"2023-12-19T14:40:15.024302+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996301+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021965+0000","last_change":"2023-12-19T14:40:15.021965+0000","last_active":"2023-12-19T14:40:15.021965+0000","last_peered":"2023-12-19T14:40:15.021965+0000","last_clean":"2023-12-19T14:40:15.021965+0000","last_became_active":"2023-12-19T14:40:15.021467+0000","last_became_peered":"2023-12-19T14:40:15.021467+0000","last_unstale":"2023-12-19T14:40:15.021965+0000","last_undegraded":"2023-12-19T14:40:15.021965+0000","last_fullsized":"2023-12-19T14:40:15.021965+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-19T14:40:13.980266+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:40:13.980266+0000","last_peered":"2023-12-19T14:40:13.980266+0000","last_clean":"2023-12-19T14:40:13.980266+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:40:13.980266+0000","last_undegraded":"2023-12-19T14:40:13.980266+0000","last_fullsized":"2023-12-19T14:40:13.980266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298260,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073580,"statfs":{"total":96636764160,"available":96331345920,"internally_reserved":0,"allocated":995328,"data_stored":758106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.90500000000000003}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.494}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.161}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.55},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.522}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.91700000000000004}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.266},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":3.0990000000000002},"last":1.125}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.222},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.1579999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":1.254},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":3.194,"5min":3.194,"15min":3.194},"last":1.321}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0150000000000001,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3600000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.367,"5min":3.367,"15min":3.367},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.51}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.143}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.1850000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.419},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2709999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.173},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.2430000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.478},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.121}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.514},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.702}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.966},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.8300000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":2.0840000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.742,"5min":2.742,"15min":2.742},"last":1.917}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":5.2060000000000004,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":2.02},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":2.5289999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":4.476,"5min":4.476,"15min":4.476},"last":0.94499999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.212}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":1.347},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.621}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.867,"5min":1.867,"15min":1.867},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.5980000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301772,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070068,"statfs":{"total":96636764160,"available":96327749632,"internally_reserved":0,"allocated":397312,"data_stored":166727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.871},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2050000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.151},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.629}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.351}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":1.306},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":1.548}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.403},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":3.0619999999999998,"5min":3.0619999999999998,"15min":3.0619999999999998},"last":1.123}]},{"osd":3,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":1.236}]},{"osd":5,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.591},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.266}]},{"osd":6,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.49},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":1.329},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.7669999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069488,"statfs":{"total":96636764160,"available":96327155712,"internally_reserved":0,"allocated":991232,"data_stored":757095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.83799999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.413}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.482},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":1.1819999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.143}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.077,"5min":1.077,"15min":1.077},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:40:21.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.011324+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.015299+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.010786+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.032050+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2023-12-19T14:40:21.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031938+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.032514+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.994572+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.984445+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.012508+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-12-19T14:40:21.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.996425+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.996909+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.013771+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.991932+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989734+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.994060+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.991210+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.995599+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.981853+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T00:31:07.226632+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.032685+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.980787+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T19:40:12.006194+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.991379+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.985539+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989176+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031534+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.982597+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T15:35:03.243160+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031182+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-12-19T14:40:21.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.981065+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T15:46:50.763905+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.992991+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.981044+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:36:20.673959+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.981068+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T00:07:16.111460+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.988543+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.030864+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-12-19T14:40:21.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989352+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031880+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.987338+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.011878+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.014928+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.992953+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.986757+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.014352+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031430+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.990860+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.010376+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-12-19T14:40:21.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.987923+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.014351+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.984566+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989643+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.013809+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.012459+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-12-19T14:40:21.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.995831+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.988174+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.031490+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989347+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.011013+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.996537+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-12-19T14:40:21.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.984661+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.030693+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.990365+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.013206+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.013783+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:15.030633+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-19T14:40:21.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-12-19T14:40:21.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.988938+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-19T14:40:21.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-12-19T14:40:21.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.989597+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-19T14:40:21.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-12-19T14:40:21.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-12-19T14:40:14.964555+0000', 'last_change': '2023-12-19T14:40:14.996301+0000', 'last_active': '2023-12-19T14:40:14.964555+0000', 'last_peered': '2023-12-19T14:40:14.964555+0000', 'last_clean': '2023-12-19T14:40:14.964555+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-19T14:40:14.964555+0000', 'last_undegraded': '2023-12-19T14:40:14.964555+0000', 'last_fullsized': '2023-12-19T14:40:14.964555+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'last_clean_scrub_stamp': '2023-12-19T14:40:14.964555+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-19T14:40:21.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-12-19T14:40:21.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.395+0000 7f0f7ba2e700 1 Processor -- start 2023-12-19T14:40:21.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.395+0000 7f0f7ba2e700 1 -- start start 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f7ba2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f7415d320 0x7f0f7415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f7ba2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f740c7190 con 0x7f0f740c15b0 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f7ba2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f740c1b50 con 0x7f0f740aeba0 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f7ba2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f740aefd0 con 0x7f0f7415d320 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f78fc9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0f740c15b0 0x7f0f7406d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39998/0 (socket says 172.21.15.50:39998) 2023-12-19T14:40:21.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f78fc9700 1 -- 172.21.15.50:0/3515140717 learned_addr learned my addr 172.21.15.50:0/3515140717 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f797ca700 1 --2- 172.21.15.50:0/3515140717 >> v2:172.21.15.125:3300/0 conn(0x7f0f7415d320 0x7f0f7415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117598426 0 0) 0x7f0f740c7190 con 0x7f0f740c15b0 2023-12-19T14:40:21.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60003570 con 0x7f0f740c15b0 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602804149 0 0) 0x7f0f740c1b50 con 0x7f0f740aeba0 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f740c7190 con 0x7f0f740aeba0 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f797ca700 1 -- 172.21.15.50:0/3515140717 >> v1:172.21.15.125:6789/0 conn(0x7f0f740aeba0 legacy=0x7f0f740680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f797ca700 1 -- 172.21.15.50:0/3515140717 >> v1:172.21.15.50:6789/0 conn(0x7f0f740c15b0 legacy=0x7f0f7406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024153886 0 0) 0x7f0f60003570 con 0x7f0f740c15b0 2023-12-19T14:40:21.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.399+0000 7f0f797ca700 1 -- 172.21.15.50:0/3515140717 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f740c1e30 con 0x7f0f7415d320 2023-12-19T14:40:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f797ca700 1 --2- 172.21.15.50:0/3515140717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f7415d6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36746ba1d0bbbdfc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000e070 con 0x7f0f7415d320 2023-12-19T14:40:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7000b7a0 con 0x7f0f7415d320 2023-12-19T14:40:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f6bfff700 1 -- 172.21.15.50:0/3515140717 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64001470 con 0x7f0f7415d320 2023-12-19T14:40:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/3515140717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 msgr2=0x7f0f7415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/3515140717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f7415d6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).stop 2023-12-19T14:40:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/3515140717 shutdown_connections 2023-12-19T14:40:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/3515140717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f7415d6f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/3515140717 >> 172.21.15.50:0/3515140717 conn(0x7f0f740c8850 msgr2=0x7f0f740c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/3515140717 shutdown_connections 2023-12-19T14:40:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.403+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/3515140717 wait complete. 2023-12-19T14:40:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 Processor -- start 2023-12-19T14:40:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 -- start start 2023-12-19T14:40:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f740aeba0 0x7f0f741714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f740670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f740c5c20 con 0x7f0f740c15b0 2023-12-19T14:40:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74171a20 con 0x7f0f7415d320 2023-12-19T14:40:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f7ba2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f740c3390 con 0x7f0f740aeba0 2023-12-19T14:40:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.407+0000 7f0f78fc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39756/0 (socket says 172.21.15.50:39756) 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f797ca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f740aeba0 0x7f0f741714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 -- 172.21.15.50:0/1638206070 learned_addr learned my addr 172.21.15.50:0/1638206070 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f79fcb700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f740670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 -- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f740aeba0 msgr2=0x7f0f741714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f740aeba0 0x7f0f741714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 -- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 msgr2=0x7f0f740670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f740670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 -- 172.21.15.50:0/1638206070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009a20 con 0x7f0f740c15b0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f78fc9700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400cb90 tx=0x7f0f64001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6400cea0 con 0x7f0f740c15b0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64007e90 con 0x7f0f740c15b0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f740c3670 con 0x7f0f740c15b0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640075d0 con 0x7f0f740c15b0 2023-12-19T14:40:21.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f740c3c30 con 0x7f0f740c15b0 2023-12-19T14:40:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.411+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f74071120 con 0x7f0f740c15b0 2023-12-19T14:40:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.415+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0f64004a40 con 0x7f0f740c15b0 2023-12-19T14:40:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.415+0000 7f0f697fa700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 0x7f0f58062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.415+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f0f6407bd00 con 0x7f0f740c15b0 2023-12-19T14:40:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.419+0000 7f0f797ca700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 0x7f0f58062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.419+0000 7f0f797ca700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 0x7f0f58062a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0f700099f0 tx=0x7f0f70017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.423+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f6404c190 con 0x7f0f740c15b0 2023-12-19T14:40:21.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.655+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0f740642a0 con 0x7f0f740c15b0 2023-12-19T14:40:21.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.659+0000 7f0f697fa700 1 -- 172.21.15.50:0/1638206070 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1035 (secure 0 0 0) 0x7f0f64018420 con 0x7f0f740c15b0 2023-12-19T14:40:21.663 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:21.663 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":98,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":38,"num_osds":8,"num_up_osds":8,"osd_up_since":1702996735,"num_in_osds":8,"osd_in_since":1702996725,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":2456576000,"bytes_avail":770637537280,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5091}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-12-19T14:40:04.149382+0000","services":{}},"progress_events":{}} 2023-12-19T14:40:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.659+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 msgr2=0x7f0f58062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.659+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 0x7f0f58062a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0f700099f0 tx=0x7f0f70017000 comp rx=0 tx=0).stop 2023-12-19T14:40:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.659+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 msgr2=0x7f0f741540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.659+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400cb90 tx=0x7f0f64001b50 comp rx=0 tx=0).stop 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 shutdown_connections 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f7415d320 0x7f0f740670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f58060550 0x7f0f58062a00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f740c15b0 0x7f0f741540c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 --2- 172.21.15.50:0/1638206070 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f740aeba0 0x7f0f741714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 >> 172.21.15.50:0/1638206070 conn(0x7f0f740c8850 msgr2=0x7f0f7406c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 shutdown_connections 2023-12-19T14:40:21.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.663+0000 7f0f7ba2e700 1 -- 172.21.15.50:0/1638206070 wait complete. 2023-12-19T14:40:21.686 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-12-19T14:40:21.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:40:21.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.843+0000 7f5fc6625700 1 Processor -- start 2023-12-19T14:40:21.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.843+0000 7f5fc6625700 1 -- start start 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc6625700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5fc00c15b0 0x7f5fc015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc6625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc00680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc6625700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc00c7190 con 0x7f5fc015d320 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc6625700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00aefd0 con 0x7f5fc00c15b0 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc6625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00c1b50 con 0x7f5fc00aeba0 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fbffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5fc015d320 0x7f5fc006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40006/0 (socket says 172.21.15.50:40006) 2023-12-19T14:40:21.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fbffff700 1 -- 172.21.15.50:0/1812359352 learned_addr learned my addr 172.21.15.50:0/1812359352 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fb77fe700 1 --2- 172.21.15.50:0/1812359352 >> v2:172.21.15.50:3300/0 conn(0x7f5fc00c15b0 0x7f5fc015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1812359352 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc00680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fbf7fe700 1 -- 172.21.15.50:0/1812359352 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144782693 0 0) 0x7f5fc00c7190 con 0x7f5fc015d320 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fbf7fe700 1 -- 172.21.15.50:0/1812359352 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac003570 con 0x7f5fc015d320 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1812359352 >> v2:172.21.15.50:3300/0 conn(0x7f5fc00c15b0 msgr2=0x7f5fc015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1812359352 >> v2:172.21.15.50:3300/0 conn(0x7f5fc00c15b0 0x7f5fc015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1812359352 >> v1:172.21.15.50:6789/0 conn(0x7f5fc015d320 legacy=0x7f5fc006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.847+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1812359352 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00c1e30 con 0x7f5fc00aeba0 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1812359352 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc00680f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80098e0 tx=0x7f5fb80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e5f2a94ba5ec18 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fbf7fe700 1 -- 172.21.15.50:0/1812359352 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8011630 con 0x7f5fc00aeba0 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fbf7fe700 1 -- 172.21.15.50:0/1812359352 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8011790 con 0x7f5fc00aeba0 2023-12-19T14:40:21.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fbf7fe700 1 -- 172.21.15.50:0/1812359352 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8011b30 con 0x7f5fc00aeba0 2023-12-19T14:40:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 -- 172.21.15.50:0/1812359352 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 msgr2=0x7f5fc00680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1812359352 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc00680f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80098e0 tx=0x7f5fb80091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 -- 172.21.15.50:0/1812359352 shutdown_connections 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1812359352 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc00680f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1812359352 >> v2:172.21.15.50:3300/0 conn(0x7f5fc00c15b0 0x7f5fc015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 -- 172.21.15.50:0/1812359352 >> 172.21.15.50:0/1812359352 conn(0x7f5fc00c8850 msgr2=0x7f5fc00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 -- 172.21.15.50:0/1812359352 shutdown_connections 2023-12-19T14:40:21.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 -- 172.21.15.50:0/1812359352 wait complete. 2023-12-19T14:40:21.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.851+0000 7f5fc6625700 1 Processor -- start 2023-12-19T14:40:21.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- start start 2023-12-19T14:40:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fc00c15b0 0x7f5fc006a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc0158380 con 0x7f5fc015d320 2023-12-19T14:40:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc016deb0 con 0x7f5fc00c15b0 2023-12-19T14:40:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc015dd60 con 0x7f5fc00aeba0 2023-12-19T14:40:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39760/0 (socket says 172.21.15.50:39760) 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1791700748 learned_addr learned my addr 172.21.15.50:0/1791700748 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fbffff700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fb77fe700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fc00c15b0 0x7f5fc006a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 msgr2=0x7f5fc015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fc00c15b0 msgr2=0x7f5fc006a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fc00c15b0 0x7f5fc006a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 -- 172.21.15.50:0/1791700748 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb8009420 con 0x7f5fc015d320 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc4bc2700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5fc006a2b0 tx=0x7f5fb8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb80233f0 con 0x7f5fc015d320 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8023550 con 0x7f5fc015d320 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc016d580 con 0x7f5fc015d320 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8023910 con 0x7f5fc015d320 2023-12-19T14:40:21.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.855+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc016db40 con 0x7f5fc015d320 2023-12-19T14:40:21.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.859+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc016e040 con 0x7f5fc015d320 2023-12-19T14:40:21.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.859+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5fb802c3b0 con 0x7f5fc015d320 2023-12-19T14:40:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.863+0000 7f5fbcff9700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 0x7f5fa0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.863+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5fb807e240 con 0x7f5fc015d320 2023-12-19T14:40:21.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.863+0000 7f5fbffff700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 0x7f5fa0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:21.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.863+0000 7f5fbffff700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 0x7f5fa0062a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fc015a8f0 tx=0x7f5fb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:21.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:21.867+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5fb804e6d0 con 0x7f5fc015d320 2023-12-19T14:40:22.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.039+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fc0071120 con 0x7f5fa0060580 2023-12-19T14:40:22.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.059+0000 7f5fbcff9700 1 -- 172.21.15.50:0/1791700748 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283484 (secure 0 0 0) 0x7f5fc0071120 con 0x7f5fa0060580 2023-12-19T14:40:22.068 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 msgr2=0x7f5fa0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 0x7f5fa0062a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fc015a8f0 tx=0x7f5fb0009040 comp rx=0 tx=0).stop 2023-12-19T14:40:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 msgr2=0x7f5fc006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5fc006a2b0 tx=0x7f5fb8017000 comp rx=0 tx=0).stop 2023-12-19T14:40:22.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 shutdown_connections 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fc00c15b0 0x7f5fc006a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fa0060580 0x7f5fa0062a30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fc015d320 0x7f5fc006adc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 --2- 172.21.15.50:0/1791700748 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fc00aeba0 0x7f5fc015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 >> 172.21.15.50:0/1791700748 conn(0x7f5fc00c8850 msgr2=0x7f5fc00c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 shutdown_connections 2023-12-19T14:40:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:22.067+0000 7f5fc6625700 1 -- 172.21.15.50:0/1791700748 wait complete. 2023-12-19T14:40:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:40:22.094 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-12-19T14:40:20.153579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399000,"kb_used_data":4312,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575720,"statfs":{"total":773094113280,"available":770637537280,"internally_reserved":0,"allocated":4415488,"data_stored":2533784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001747"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011324+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.015299+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010786+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032050+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032514+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994572+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984445+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012508+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996425+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996909+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013771+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991932+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989734+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.994060+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991210+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995599+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981853+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:07.226632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.032685+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.980787+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:12.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.991379+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.985539+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989176+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031534+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.982597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:03.243160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031182+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981065+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:50.763905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992991+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981044+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:20.673959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.981068+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:16.111460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988543+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030864+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.998033+0000","last_change":"2023-12-19T14:40:14.998033+0000","last_active":"2023-12-19T14:40:14.998033+0000","last_peered":"2023-12-19T14:40:14.998033+0000","last_clean":"2023-12-19T14:40:14.998033+0000","last_became_active":"2023-12-19T14:40:14.997769+0000","last_became_peered":"2023-12-19T14:40:14.997769+0000","last_unstale":"2023-12-19T14:40:14.998033+0000","last_undegraded":"2023-12-19T14:40:14.998033+0000","last_fullsized":"2023-12-19T14:40:14.998033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:05.375208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031880+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022907+0000","last_change":"2023-12-19T14:40:15.022907+0000","last_active":"2023-12-19T14:40:15.022907+0000","last_peered":"2023-12-19T14:40:15.022907+0000","last_clean":"2023-12-19T14:40:15.022907+0000","last_became_active":"2023-12-19T14:40:15.022603+0000","last_became_peered":"2023-12-19T14:40:15.022603+0000","last_unstale":"2023-12-19T14:40:15.022907+0000","last_undegraded":"2023-12-19T14:40:15.022907+0000","last_fullsized":"2023-12-19T14:40:15.022907+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987338+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021930+0000","last_change":"2023-12-19T14:40:15.021930+0000","last_active":"2023-12-19T14:40:15.021930+0000","last_peered":"2023-12-19T14:40:15.021930+0000","last_clean":"2023-12-19T14:40:15.021930+0000","last_became_active":"2023-12-19T14:40:15.021586+0000","last_became_peered":"2023-12-19T14:40:15.021586+0000","last_unstale":"2023-12-19T14:40:15.021930+0000","last_undegraded":"2023-12-19T14:40:15.021930+0000","last_fullsized":"2023-12-19T14:40:15.021930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011878+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026312+0000","last_change":"2023-12-19T14:40:15.026312+0000","last_active":"2023-12-19T14:40:15.026312+0000","last_peered":"2023-12-19T14:40:15.026312+0000","last_clean":"2023-12-19T14:40:15.026312+0000","last_became_active":"2023-12-19T14:40:15.025957+0000","last_became_peered":"2023-12-19T14:40:15.025957+0000","last_unstale":"2023-12-19T14:40:15.026312+0000","last_undegraded":"2023-12-19T14:40:15.026312+0000","last_fullsized":"2023-12-19T14:40:15.026312+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014928+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025292+0000","last_change":"2023-12-19T14:40:15.025292+0000","last_active":"2023-12-19T14:40:15.025292+0000","last_peered":"2023-12-19T14:40:15.025292+0000","last_clean":"2023-12-19T14:40:15.025292+0000","last_became_active":"2023-12-19T14:40:15.024103+0000","last_became_peered":"2023-12-19T14:40:15.024103+0000","last_unstale":"2023-12-19T14:40:15.025292+0000","last_undegraded":"2023-12-19T14:40:15.025292+0000","last_fullsized":"2023-12-19T14:40:15.025292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:59.010045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.992953+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.004455+0000","last_change":"2023-12-19T14:40:15.004455+0000","last_active":"2023-12-19T14:40:15.004455+0000","last_peered":"2023-12-19T14:40:15.004455+0000","last_clean":"2023-12-19T14:40:15.004455+0000","last_became_active":"2023-12-19T14:40:15.004022+0000","last_became_peered":"2023-12-19T14:40:15.004022+0000","last_unstale":"2023-12-19T14:40:15.004455+0000","last_undegraded":"2023-12-19T14:40:15.004455+0000","last_fullsized":"2023-12-19T14:40:15.004455+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.986757+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.015560+0000","last_change":"2023-12-19T14:40:15.015560+0000","last_active":"2023-12-19T14:40:15.015560+0000","last_peered":"2023-12-19T14:40:15.015560+0000","last_clean":"2023-12-19T14:40:15.015560+0000","last_became_active":"2023-12-19T14:40:15.015177+0000","last_became_peered":"2023-12-19T14:40:15.015177+0000","last_unstale":"2023-12-19T14:40:15.015560+0000","last_undegraded":"2023-12-19T14:40:15.015560+0000","last_fullsized":"2023-12-19T14:40:15.015560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014352+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:14.994802+0000","last_change":"2023-12-19T14:40:14.994802+0000","last_active":"2023-12-19T14:40:14.994802+0000","last_peered":"2023-12-19T14:40:14.994802+0000","last_clean":"2023-12-19T14:40:14.994802+0000","last_became_active":"2023-12-19T14:40:14.994524+0000","last_became_peered":"2023-12-19T14:40:14.994524+0000","last_unstale":"2023-12-19T14:40:14.994802+0000","last_undegraded":"2023-12-19T14:40:14.994802+0000","last_fullsized":"2023-12-19T14:40:14.994802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031430+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024050+0000","last_change":"2023-12-19T14:40:15.024050+0000","last_active":"2023-12-19T14:40:15.024050+0000","last_peered":"2023-12-19T14:40:15.024050+0000","last_clean":"2023-12-19T14:40:15.024050+0000","last_became_active":"2023-12-19T14:40:15.023710+0000","last_became_peered":"2023-12-19T14:40:15.023710+0000","last_unstale":"2023-12-19T14:40:15.024050+0000","last_undegraded":"2023-12-19T14:40:15.024050+0000","last_fullsized":"2023-12-19T14:40:15.024050+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990860+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.025954+0000","last_change":"2023-12-19T14:40:15.025954+0000","last_active":"2023-12-19T14:40:15.025954+0000","last_peered":"2023-12-19T14:40:15.025954+0000","last_clean":"2023-12-19T14:40:15.025954+0000","last_became_active":"2023-12-19T14:40:15.025658+0000","last_became_peered":"2023-12-19T14:40:15.025658+0000","last_unstale":"2023-12-19T14:40:15.025954+0000","last_undegraded":"2023-12-19T14:40:15.025954+0000","last_fullsized":"2023-12-19T14:40:15.025954+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.010376+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:26.545879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023326+0000","last_change":"2023-12-19T14:40:15.023326+0000","last_active":"2023-12-19T14:40:15.023326+0000","last_peered":"2023-12-19T14:40:15.023326+0000","last_clean":"2023-12-19T14:40:15.023326+0000","last_became_active":"2023-12-19T14:40:15.022920+0000","last_became_peered":"2023-12-19T14:40:15.022920+0000","last_unstale":"2023-12-19T14:40:15.023326+0000","last_undegraded":"2023-12-19T14:40:15.023326+0000","last_fullsized":"2023-12-19T14:40:15.023326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:44.100285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.987923+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023075+0000","last_change":"2023-12-19T14:40:15.023075+0000","last_active":"2023-12-19T14:40:15.023075+0000","last_peered":"2023-12-19T14:40:15.023075+0000","last_clean":"2023-12-19T14:40:15.023075+0000","last_became_active":"2023-12-19T14:40:15.022756+0000","last_became_peered":"2023-12-19T14:40:15.022756+0000","last_unstale":"2023-12-19T14:40:15.023075+0000","last_undegraded":"2023-12-19T14:40:15.023075+0000","last_fullsized":"2023-12-19T14:40:15.023075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.014351+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026805+0000","last_change":"2023-12-19T14:40:15.026805+0000","last_active":"2023-12-19T14:40:15.026805+0000","last_peered":"2023-12-19T14:40:15.026805+0000","last_clean":"2023-12-19T14:40:15.026805+0000","last_became_active":"2023-12-19T14:40:15.025038+0000","last_became_peered":"2023-12-19T14:40:15.025038+0000","last_unstale":"2023-12-19T14:40:15.026805+0000","last_undegraded":"2023-12-19T14:40:15.026805+0000","last_fullsized":"2023-12-19T14:40:15.026805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984566+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.017622+0000","last_change":"2023-12-19T14:40:15.017622+0000","last_active":"2023-12-19T14:40:15.017622+0000","last_peered":"2023-12-19T14:40:15.017622+0000","last_clean":"2023-12-19T14:40:15.017622+0000","last_became_active":"2023-12-19T14:40:15.017316+0000","last_became_peered":"2023-12-19T14:40:15.017316+0000","last_unstale":"2023-12-19T14:40:15.017622+0000","last_undegraded":"2023-12-19T14:40:15.017622+0000","last_fullsized":"2023-12-19T14:40:15.017622+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989643+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001820+0000","last_change":"2023-12-19T14:40:15.001820+0000","last_active":"2023-12-19T14:40:15.001820+0000","last_peered":"2023-12-19T14:40:15.001820+0000","last_clean":"2023-12-19T14:40:15.001820+0000","last_became_active":"2023-12-19T14:40:15.001305+0000","last_became_peered":"2023-12-19T14:40:15.001305+0000","last_unstale":"2023-12-19T14:40:15.001820+0000","last_undegraded":"2023-12-19T14:40:15.001820+0000","last_fullsized":"2023-12-19T14:40:15.001820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:52.732501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013809+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003285+0000","last_change":"2023-12-19T14:40:15.003285+0000","last_active":"2023-12-19T14:40:15.003285+0000","last_peered":"2023-12-19T14:40:15.003285+0000","last_clean":"2023-12-19T14:40:15.003285+0000","last_became_active":"2023-12-19T14:40:15.002941+0000","last_became_peered":"2023-12-19T14:40:15.002941+0000","last_unstale":"2023-12-19T14:40:15.003285+0000","last_undegraded":"2023-12-19T14:40:15.003285+0000","last_fullsized":"2023-12-19T14:40:15.003285+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.012459+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.003942+0000","last_change":"2023-12-19T14:40:15.003942+0000","last_active":"2023-12-19T14:40:15.003942+0000","last_peered":"2023-12-19T14:40:15.003942+0000","last_clean":"2023-12-19T14:40:15.003942+0000","last_became_active":"2023-12-19T14:40:15.003631+0000","last_became_peered":"2023-12-19T14:40:15.003631+0000","last_unstale":"2023-12-19T14:40:15.003942+0000","last_undegraded":"2023-12-19T14:40:15.003942+0000","last_fullsized":"2023-12-19T14:40:15.003942+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:22.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.995831+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.002317+0000","last_change":"2023-12-19T14:40:15.002317+0000","last_active":"2023-12-19T14:40:15.002317+0000","last_peered":"2023-12-19T14:40:15.002317+0000","last_clean":"2023-12-19T14:40:15.002317+0000","last_became_active":"2023-12-19T14:40:15.002056+0000","last_became_peered":"2023-12-19T14:40:15.002056+0000","last_unstale":"2023-12-19T14:40:15.002317+0000","last_undegraded":"2023-12-19T14:40:15.002317+0000","last_fullsized":"2023-12-19T14:40:15.002317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:38.560751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988174+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:24.075492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.010764+0000","last_change":"2023-12-19T14:40:15.010764+0000","last_active":"2023-12-19T14:40:15.010764+0000","last_peered":"2023-12-19T14:40:15.010764+0000","last_clean":"2023-12-19T14:40:15.010764+0000","last_became_active":"2023-12-19T14:40:15.010427+0000","last_became_peered":"2023-12-19T14:40:15.010427+0000","last_unstale":"2023-12-19T14:40:15.010764+0000","last_undegraded":"2023-12-19T14:40:15.010764+0000","last_fullsized":"2023-12-19T14:40:15.010764+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:11.000395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.031490+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021360+0000","last_change":"2023-12-19T14:40:15.021360+0000","last_active":"2023-12-19T14:40:15.021360+0000","last_peered":"2023-12-19T14:40:15.021360+0000","last_clean":"2023-12-19T14:40:15.021360+0000","last_became_active":"2023-12-19T14:40:15.020201+0000","last_became_peered":"2023-12-19T14:40:15.020201+0000","last_unstale":"2023-12-19T14:40:15.021360+0000","last_undegraded":"2023-12-19T14:40:15.021360+0000","last_fullsized":"2023-12-19T14:40:15.021360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989347+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026807+0000","last_change":"2023-12-19T14:40:15.026807+0000","last_active":"2023-12-19T14:40:15.026807+0000","last_peered":"2023-12-19T14:40:15.026807+0000","last_clean":"2023-12-19T14:40:15.026807+0000","last_became_active":"2023-12-19T14:40:15.026425+0000","last_became_peered":"2023-12-19T14:40:15.026425+0000","last_unstale":"2023-12-19T14:40:15.026807+0000","last_undegraded":"2023-12-19T14:40:15.026807+0000","last_fullsized":"2023-12-19T14:40:15.026807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.011013+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.013522+0000","last_change":"2023-12-19T14:40:15.013522+0000","last_active":"2023-12-19T14:40:15.013522+0000","last_peered":"2023-12-19T14:40:15.013522+0000","last_clean":"2023-12-19T14:40:15.013522+0000","last_became_active":"2023-12-19T14:40:15.013241+0000","last_became_peered":"2023-12-19T14:40:15.013241+0000","last_unstale":"2023-12-19T14:40:15.013522+0000","last_undegraded":"2023-12-19T14:40:15.013522+0000","last_fullsized":"2023-12-19T14:40:15.013522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:49.758106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996537+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:48.250195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.001340+0000","last_change":"2023-12-19T14:40:15.001340+0000","last_active":"2023-12-19T14:40:15.001340+0000","last_peered":"2023-12-19T14:40:15.001340+0000","last_clean":"2023-12-19T14:40:15.001340+0000","last_became_active":"2023-12-19T14:40:15.001076+0000","last_became_peered":"2023-12-19T14:40:15.001076+0000","last_unstale":"2023-12-19T14:40:15.001340+0000","last_undegraded":"2023-12-19T14:40:15.001340+0000","last_fullsized":"2023-12-19T14:40:15.001340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.984661+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024120+0000","last_change":"2023-12-19T14:40:15.024120+0000","last_active":"2023-12-19T14:40:15.024120+0000","last_peered":"2023-12-19T14:40:15.024120+0000","last_clean":"2023-12-19T14:40:15.024120+0000","last_became_active":"2023-12-19T14:40:15.023840+0000","last_became_peered":"2023-12-19T14:40:15.023840+0000","last_unstale":"2023-12-19T14:40:15.024120+0000","last_undegraded":"2023-12-19T14:40:15.024120+0000","last_fullsized":"2023-12-19T14:40:15.024120+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030693+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.022435+0000","last_change":"2023-12-19T14:40:15.022435+0000","last_active":"2023-12-19T14:40:15.022435+0000","last_peered":"2023-12-19T14:40:15.022435+0000","last_clean":"2023-12-19T14:40:15.022435+0000","last_became_active":"2023-12-19T14:40:15.022169+0000","last_became_peered":"2023-12-19T14:40:15.022169+0000","last_unstale":"2023-12-19T14:40:15.022435+0000","last_undegraded":"2023-12-19T14:40:15.022435+0000","last_fullsized":"2023-12-19T14:40:15.022435+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:43.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.990365+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.012638+0000","last_change":"2023-12-19T14:40:15.012638+0000","last_active":"2023-12-19T14:40:15.012638+0000","last_peered":"2023-12-19T14:40:15.012638+0000","last_clean":"2023-12-19T14:40:15.012638+0000","last_became_active":"2023-12-19T14:40:15.012351+0000","last_became_peered":"2023-12-19T14:40:15.012351+0000","last_unstale":"2023-12-19T14:40:15.012638+0000","last_undegraded":"2023-12-19T14:40:15.012638+0000","last_fullsized":"2023-12-19T14:40:15.012638+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:15.105662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013206+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.029026+0000","last_change":"2023-12-19T14:40:15.029026+0000","last_active":"2023-12-19T14:40:15.029026+0000","last_peered":"2023-12-19T14:40:15.029026+0000","last_clean":"2023-12-19T14:40:15.029026+0000","last_became_active":"2023-12-19T14:40:15.028694+0000","last_became_peered":"2023-12-19T14:40:15.028694+0000","last_unstale":"2023-12-19T14:40:15.029026+0000","last_undegraded":"2023-12-19T14:40:15.029026+0000","last_fullsized":"2023-12-19T14:40:15.029026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.013783+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.023303+0000","last_change":"2023-12-19T14:40:15.023303+0000","last_active":"2023-12-19T14:40:15.023303+0000","last_peered":"2023-12-19T14:40:15.023303+0000","last_clean":"2023-12-19T14:40:15.023303+0000","last_became_active":"2023-12-19T14:40:15.022812+0000","last_became_peered":"2023-12-19T14:40:15.022812+0000","last_unstale":"2023-12-19T14:40:15.023303+0000","last_undegraded":"2023-12-19T14:40:15.023303+0000","last_fullsized":"2023-12-19T14:40:15.023303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:15.030633+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024761+0000","last_change":"2023-12-19T14:40:15.024761+0000","last_active":"2023-12-19T14:40:15.024761+0000","last_peered":"2023-12-19T14:40:15.024761+0000","last_clean":"2023-12-19T14:40:15.024761+0000","last_became_active":"2023-12-19T14:40:15.024406+0000","last_became_peered":"2023-12-19T14:40:15.024406+0000","last_unstale":"2023-12-19T14:40:15.024761+0000","last_undegraded":"2023-12-19T14:40:15.024761+0000","last_fullsized":"2023-12-19T14:40:15.024761+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:36.341963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.988938+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:10.537896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.026100+0000","last_change":"2023-12-19T14:40:15.026100+0000","last_active":"2023-12-19T14:40:15.026100+0000","last_peered":"2023-12-19T14:40:15.026100+0000","last_clean":"2023-12-19T14:40:15.026100+0000","last_became_active":"2023-12-19T14:40:15.025647+0000","last_became_peered":"2023-12-19T14:40:15.025647+0000","last_unstale":"2023-12-19T14:40:15.026100+0000","last_undegraded":"2023-12-19T14:40:15.026100+0000","last_fullsized":"2023-12-19T14:40:15.026100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.989597+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.024302+0000","last_change":"2023-12-19T14:40:15.024302+0000","last_active":"2023-12-19T14:40:15.024302+0000","last_peered":"2023-12-19T14:40:15.024302+0000","last_clean":"2023-12-19T14:40:15.024302+0000","last_became_active":"2023-12-19T14:40:15.024017+0000","last_became_peered":"2023-12-19T14:40:15.024017+0000","last_unstale":"2023-12-19T14:40:15.024302+0000","last_undegraded":"2023-12-19T14:40:15.024302+0000","last_fullsized":"2023-12-19T14:40:15.024302+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-12-19T14:40:14.964555+0000","last_change":"2023-12-19T14:40:14.996301+0000","last_active":"2023-12-19T14:40:14.964555+0000","last_peered":"2023-12-19T14:40:14.964555+0000","last_clean":"2023-12-19T14:40:14.964555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-19T14:40:14.964555+0000","last_undegraded":"2023-12-19T14:40:14.964555+0000","last_fullsized":"2023-12-19T14:40:14.964555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":8,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.021965+0000","last_change":"2023-12-19T14:40:15.021965+0000","last_active":"2023-12-19T14:40:15.021965+0000","last_peered":"2023-12-19T14:40:15.021965+0000","last_clean":"2023-12-19T14:40:15.021965+0000","last_became_active":"2023-12-19T14:40:15.021467+0000","last_became_peered":"2023-12-19T14:40:15.021467+0000","last_unstale":"2023-12-19T14:40:15.021965+0000","last_undegraded":"2023-12-19T14:40:15.021965+0000","last_fullsized":"2023-12-19T14:40:15.021965+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-19T14:40:13.980266+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:40:13.980266+0000","last_peered":"2023-12-19T14:40:13.980266+0000","last_clean":"2023-12-19T14:40:13.980266+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:40:13.980266+0000","last_undegraded":"2023-12-19T14:40:13.980266+0000","last_fullsized":"2023-12-19T14:40:13.980266+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298260,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073580,"statfs":{"total":96636764160,"available":96331345920,"internally_reserved":0,"allocated":995328,"data_stored":758106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.90500000000000003}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.494}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.161}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.55},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.522}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.91700000000000004}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.266},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":3.0990000000000002},"last":1.125}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.222},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.1579999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":1.254},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":3.194,"5min":3.194,"15min":3.194},"last":1.321}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0150000000000001,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3600000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.367,"5min":3.367,"15min":3.367},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.51}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.3},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.143}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.1850000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.419},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.2709999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.173},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.2430000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.478},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.121}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297680,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074160,"statfs":{"total":96636764160,"available":96331939840,"internally_reserved":0,"allocated":401408,"data_stored":167738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.514},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.702}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.966},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.8740000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.8300000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":2.0840000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.742,"5min":2.742,"15min":2.742},"last":1.917}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":5.2060000000000004,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":2.02},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":2.5289999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.3939999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":4.476,"5min":4.476,"15min":4.476},"last":0.94499999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.212}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.4419999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":1.347},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.621}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.867,"5min":1.867,"15min":1.867},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.5980000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301772,"kb_used_data":388,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070068,"statfs":{"total":96636764160,"available":96327749632,"internally_reserved":0,"allocated":397312,"data_stored":166727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.871},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.2050000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.151},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.629}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.351}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":1.306},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":1.548}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.403},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.1140000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":413696,"data_stored":174321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.65200000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":3.0619999999999998,"5min":3.0619999999999998,"15min":3.0619999999999998},"last":1.123}]},{"osd":3,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2789999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":1.236}]},{"osd":5,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.591},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.266}]},{"osd":6,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.49},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":1.329},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.7669999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069488,"statfs":{"total":96636764160,"available":96327155712,"internally_reserved":0,"allocated":991232,"data_stored":757095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.83799999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.413}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.482},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":1.1819999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.143}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.077,"5min":1.077,"15min":1.077},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:40:25.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:40:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.275+0000 7fe7dc585700 1 Processor -- start 2023-12-19T14:40:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.275+0000 7fe7dc585700 1 -- start start 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7d40c75e0 0x7fe7d4068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d406e150 con 0x7fe7d4140ae0 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d406d540 con 0x7fe7d40c75e0 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dc585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d406d820 con 0x7fe7d4140710 2023-12-19T14:40:25.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7d9b20700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7d9b20700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39762/0 (socket says 172.21.15.50:39762) 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7d40c75e0 0x7fe7d4068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7d9b20700 1 -- 172.21.15.50:0/1013909208 learned_addr learned my addr 172.21.15.50:0/1013909208 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7da321700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 -- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 msgr2=0x7fe7d40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d40c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 -- 172.21.15.50:0/1013909208 >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 msgr2=0x7fe7d40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 --2- 172.21.15.50:0/1013909208 >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 -- 172.21.15.50:0/1013909208 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d406db00 con 0x7fe7d40c75e0 2023-12-19T14:40:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7d9b20700 1 --2- 172.21.15.50:0/1013909208 >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:25.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7dab22700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d40c75e0 0x7fe7d4068630 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40ae930 tx=0x7fe7cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9ba7143f809a22 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.279+0000 7fe7d931f700 1 -- 172.21.15.50:0/1013909208 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011630 con 0x7fe7d40c75e0 2023-12-19T14:40:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7d931f700 1 -- 172.21.15.50:0/1013909208 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc0117b0 con 0x7fe7d40c75e0 2023-12-19T14:40:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7d931f700 1 -- 172.21.15.50:0/1013909208 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011b70 con 0x7fe7d40c75e0 2023-12-19T14:40:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 -- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d40c75e0 msgr2=0x7fe7d4068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d40c75e0 0x7fe7d4068630 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40ae930 tx=0x7fe7cc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 -- 172.21.15.50:0/1013909208 shutdown_connections 2023-12-19T14:40:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 --2- 172.21.15.50:0/1013909208 >> v2:172.21.15.50:3300/0 conn(0x7fe7d4140ae0 0x7fe7d40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d40c75e0 0x7fe7d4068630 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 --2- 172.21.15.50:0/1013909208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 -- 172.21.15.50:0/1013909208 >> 172.21.15.50:0/1013909208 conn(0x7fe7d40c8850 msgr2=0x7fe7d40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 -- 172.21.15.50:0/1013909208 shutdown_connections 2023-12-19T14:40:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 -- 172.21.15.50:0/1013909208 wait complete. 2023-12-19T14:40:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.283+0000 7fe7dc585700 1 Processor -- start 2023-12-19T14:40:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 -- start start 2023-12-19T14:40:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d4157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d4140ae0 0x7fe7d4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d41581e0 con 0x7fe7d40c75e0 2023-12-19T14:40:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d415d030 con 0x7fe7d4140ae0 2023-12-19T14:40:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dc585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d415d4c0 con 0x7fe7d4140710 2023-12-19T14:40:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39774/0 (socket says 172.21.15.50:39774) 2023-12-19T14:40:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 -- 172.21.15.50:0/2740490476 learned_addr learned my addr 172.21.15.50:0/2740490476 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7dab22700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d4140ae0 0x7fe7d4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7d9b20700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d4157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 -- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 msgr2=0x7fe7d4157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d4157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 -- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d4140ae0 msgr2=0x7fe7d4157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d4140ae0 0x7fe7d4157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 -- 172.21.15.50:0/2740490476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40094f0 con 0x7fe7d40c75e0 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7da321700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009f10 tx=0x7fe7d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.287+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000cb10 con 0x7fe7d40c75e0 2023-12-19T14:40:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d000f040 con 0x7fe7d40c75e0 2023-12-19T14:40:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000cee0 con 0x7fe7d40c75e0 2023-12-19T14:40:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7cc009420 con 0x7fe7d40c75e0 2023-12-19T14:40:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d415c700 con 0x7fe7d40c75e0 2023-12-19T14:40:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d4071120 con 0x7fe7d40c75e0 2023-12-19T14:40:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.291+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe7d0014690 con 0x7fe7d40c75e0 2023-12-19T14:40:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.295+0000 7fe7ca7fc700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 0x7fe7b8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.295+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe7d007d280 con 0x7fe7d40c75e0 2023-12-19T14:40:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.295+0000 7fe7d9b20700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 0x7fe7b8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.295+0000 7fe7d9b20700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 0x7fe7b8062a30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40c5f10 tx=0x7fe7c4009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.303+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe7d004d690 con 0x7fe7d40c75e0 2023-12-19T14:40:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.475+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7d40642a0 con 0x7fe7b8060580 2023-12-19T14:40:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.491+0000 7fe7ca7fc700 1 -- 172.21.15.50:0/2740490476 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286301 (secure 0 0 0) 0x7fe7d40642a0 con 0x7fe7b8060580 2023-12-19T14:40:25.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 msgr2=0x7fe7b8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 0x7fe7b8062a30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40c5f10 tx=0x7fe7c4009250 comp rx=0 tx=0).stop 2023-12-19T14:40:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 msgr2=0x7fe7d415bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009f10 tx=0x7fe7d000b040 comp rx=0 tx=0).stop 2023-12-19T14:40:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 shutdown_connections 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7d4140ae0 0x7fe7d4157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe7b8060580 0x7fe7b8062a30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe7d4140710 0x7fe7d4157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 --2- 172.21.15.50:0/2740490476 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7d40c75e0 0x7fe7d415bbd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 >> 172.21.15.50:0/2740490476 conn(0x7fe7d40c8850 msgr2=0x7fe7d4068430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 shutdown_connections 2023-12-19T14:40:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.499+0000 7fe7dc585700 1 -- 172.21.15.50:0/2740490476 wait complete. 2023-12-19T14:40:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:40:25.525 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2023-12-19T14:40:24.155277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399328,"kb_used_data":4640,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575392,"statfs":{"total":773094113280,"available":770637201408,"internally_reserved":0,"allocated":4751360,"data_stored":2687422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.171562"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030677+0000","last_change":"2023-12-19T14:40:16.023506+0000","last_active":"2023-12-19T14:40:16.030677+0000","last_peered":"2023-12-19T14:40:16.030677+0000","last_clean":"2023-12-19T14:40:16.030677+0000","last_became_active":"2023-12-19T14:40:16.023229+0000","last_became_peered":"2023-12-19T14:40:16.023229+0000","last_unstale":"2023-12-19T14:40:16.030677+0000","last_undegraded":"2023-12-19T14:40:16.030677+0000","last_fullsized":"2023-12-19T14:40:16.030677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:10:18.741347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.017816+0000","last_change":"2023-12-19T14:40:16.008124+0000","last_active":"2023-12-19T14:40:16.017816+0000","last_peered":"2023-12-19T14:40:16.017816+0000","last_clean":"2023-12-19T14:40:16.017816+0000","last_became_active":"2023-12-19T14:40:16.007842+0000","last_became_peered":"2023-12-19T14:40:16.007842+0000","last_unstale":"2023-12-19T14:40:16.017816+0000","last_undegraded":"2023-12-19T14:40:16.017816+0000","last_fullsized":"2023-12-19T14:40:16.017816+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:01:14.693039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032074+0000","last_change":"2023-12-19T14:40:16.024976+0000","last_active":"2023-12-19T14:40:16.032074+0000","last_peered":"2023-12-19T14:40:16.032074+0000","last_clean":"2023-12-19T14:40:16.032074+0000","last_became_active":"2023-12-19T14:40:16.024660+0000","last_became_peered":"2023-12-19T14:40:16.024660+0000","last_unstale":"2023-12-19T14:40:16.032074+0000","last_undegraded":"2023-12-19T14:40:16.032074+0000","last_fullsized":"2023-12-19T14:40:16.032074+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028525+0000","last_change":"2023-12-19T14:40:16.025202+0000","last_active":"2023-12-19T14:40:16.028525+0000","last_peered":"2023-12-19T14:40:16.028525+0000","last_clean":"2023-12-19T14:40:16.028525+0000","last_became_active":"2023-12-19T14:40:16.024787+0000","last_became_peered":"2023-12-19T14:40:16.024787+0000","last_unstale":"2023-12-19T14:40:16.028525+0000","last_undegraded":"2023-12-19T14:40:16.028525+0000","last_fullsized":"2023-12-19T14:40:16.028525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.020648+0000","last_change":"2023-12-19T14:40:16.015187+0000","last_active":"2023-12-19T14:40:16.020648+0000","last_peered":"2023-12-19T14:40:16.020648+0000","last_clean":"2023-12-19T14:40:16.020648+0000","last_became_active":"2023-12-19T14:40:16.014854+0000","last_became_peered":"2023-12-19T14:40:16.014854+0000","last_unstale":"2023-12-19T14:40:16.020648+0000","last_undegraded":"2023-12-19T14:40:16.020648+0000","last_fullsized":"2023-12-19T14:40:16.020648+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026773+0000","last_change":"2023-12-19T14:40:16.012257+0000","last_active":"2023-12-19T14:40:16.026773+0000","last_peered":"2023-12-19T14:40:16.026773+0000","last_clean":"2023-12-19T14:40:16.026773+0000","last_became_active":"2023-12-19T14:40:16.011980+0000","last_became_peered":"2023-12-19T14:40:16.011980+0000","last_unstale":"2023-12-19T14:40:16.026773+0000","last_undegraded":"2023-12-19T14:40:16.026773+0000","last_fullsized":"2023-12-19T14:40:16.026773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024786+0000","last_change":"2023-12-19T14:40:16.016910+0000","last_active":"2023-12-19T14:40:16.024786+0000","last_peered":"2023-12-19T14:40:16.024786+0000","last_clean":"2023-12-19T14:40:16.024786+0000","last_became_active":"2023-12-19T14:40:16.016624+0000","last_became_peered":"2023-12-19T14:40:16.016624+0000","last_unstale":"2023-12-19T14:40:16.024786+0000","last_undegraded":"2023-12-19T14:40:16.024786+0000","last_fullsized":"2023-12-19T14:40:16.024786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:01:14.693039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.025627+0000","last_change":"2023-12-19T14:40:16.019456+0000","last_active":"2023-12-19T14:40:16.025627+0000","last_peered":"2023-12-19T14:40:16.025627+0000","last_clean":"2023-12-19T14:40:16.025627+0000","last_became_active":"2023-12-19T14:40:16.014077+0000","last_became_peered":"2023-12-19T14:40:16.014077+0000","last_unstale":"2023-12-19T14:40:16.025627+0000","last_undegraded":"2023-12-19T14:40:16.025627+0000","last_fullsized":"2023-12-19T14:40:16.025627+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:41.134391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.025308+0000","last_change":"2023-12-19T14:40:16.017319+0000","last_active":"2023-12-19T14:40:16.025308+0000","last_peered":"2023-12-19T14:40:16.025308+0000","last_clean":"2023-12-19T14:40:16.025308+0000","last_became_active":"2023-12-19T14:40:16.017023+0000","last_became_peered":"2023-12-19T14:40:16.017023+0000","last_unstale":"2023-12-19T14:40:16.025308+0000","last_undegraded":"2023-12-19T14:40:16.025308+0000","last_fullsized":"2023-12-19T14:40:16.025308+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024209+0000","last_change":"2023-12-19T14:40:16.017314+0000","last_active":"2023-12-19T14:40:16.024209+0000","last_peered":"2023-12-19T14:40:16.024209+0000","last_clean":"2023-12-19T14:40:16.024209+0000","last_became_active":"2023-12-19T14:40:16.013711+0000","last_became_peered":"2023-12-19T14:40:16.013711+0000","last_unstale":"2023-12-19T14:40:16.024209+0000","last_undegraded":"2023-12-19T14:40:16.024209+0000","last_fullsized":"2023-12-19T14:40:16.024209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.035493+0000","last_change":"2023-12-19T14:40:16.031097+0000","last_active":"2023-12-19T14:40:16.035493+0000","last_peered":"2023-12-19T14:40:16.035493+0000","last_clean":"2023-12-19T14:40:16.035493+0000","last_became_active":"2023-12-19T14:40:16.030819+0000","last_became_peered":"2023-12-19T14:40:16.030819+0000","last_unstale":"2023-12-19T14:40:16.035493+0000","last_undegraded":"2023-12-19T14:40:16.035493+0000","last_fullsized":"2023-12-19T14:40:16.035493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026601+0000","last_change":"2023-12-19T14:40:16.020425+0000","last_active":"2023-12-19T14:40:16.026601+0000","last_peered":"2023-12-19T14:40:16.026601+0000","last_clean":"2023-12-19T14:40:16.026601+0000","last_became_active":"2023-12-19T14:40:16.020116+0000","last_became_peered":"2023-12-19T14:40:16.020116+0000","last_unstale":"2023-12-19T14:40:16.026601+0000","last_undegraded":"2023-12-19T14:40:16.026601+0000","last_fullsized":"2023-12-19T14:40:16.026601+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:26:36.506699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.021849+0000","last_change":"2023-12-19T14:40:16.015042+0000","last_active":"2023-12-19T14:40:16.021849+0000","last_peered":"2023-12-19T14:40:16.021849+0000","last_clean":"2023-12-19T14:40:16.021849+0000","last_became_active":"2023-12-19T14:40:16.014180+0000","last_became_peered":"2023-12-19T14:40:16.014180+0000","last_unstale":"2023-12-19T14:40:16.021849+0000","last_undegraded":"2023-12-19T14:40:16.021849+0000","last_fullsized":"2023-12-19T14:40:16.021849+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:26.826515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.038173+0000","last_change":"2023-12-19T14:40:16.034467+0000","last_active":"2023-12-19T14:40:16.038173+0000","last_peered":"2023-12-19T14:40:16.038173+0000","last_clean":"2023-12-19T14:40:16.038173+0000","last_became_active":"2023-12-19T14:40:16.033875+0000","last_became_peered":"2023-12-19T14:40:16.033875+0000","last_unstale":"2023-12-19T14:40:16.038173+0000","last_undegraded":"2023-12-19T14:40:16.038173+0000","last_fullsized":"2023-12-19T14:40:16.038173+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:36:38.599323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024774+0000","last_change":"2023-12-19T14:40:16.016330+0000","last_active":"2023-12-19T14:40:16.024774+0000","last_peered":"2023-12-19T14:40:16.024774+0000","last_clean":"2023-12-19T14:40:16.024774+0000","last_became_active":"2023-12-19T14:40:16.004205+0000","last_became_peered":"2023-12-19T14:40:16.004205+0000","last_unstale":"2023-12-19T14:40:16.024774+0000","last_undegraded":"2023-12-19T14:40:16.024774+0000","last_fullsized":"2023-12-19T14:40:16.024774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.027497+0000","last_change":"2023-12-19T14:40:16.019253+0000","last_active":"2023-12-19T14:40:16.027497+0000","last_peered":"2023-12-19T14:40:16.027497+0000","last_clean":"2023-12-19T14:40:16.027497+0000","last_became_active":"2023-12-19T14:40:16.018919+0000","last_became_peered":"2023-12-19T14:40:16.018919+0000","last_unstale":"2023-12-19T14:40:16.027497+0000","last_undegraded":"2023-12-19T14:40:16.027497+0000","last_fullsized":"2023-12-19T14:40:16.027497+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028286+0000","last_change":"2023-12-19T14:40:16.023680+0000","last_active":"2023-12-19T14:40:16.028286+0000","last_peered":"2023-12-19T14:40:16.028286+0000","last_clean":"2023-12-19T14:40:16.028286+0000","last_became_active":"2023-12-19T14:40:16.018941+0000","last_became_peered":"2023-12-19T14:40:16.018941+0000","last_unstale":"2023-12-19T14:40:16.028286+0000","last_undegraded":"2023-12-19T14:40:16.028286+0000","last_fullsized":"2023-12-19T14:40:16.028286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:53:48.174057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030803+0000","last_change":"2023-12-19T14:40:16.016339+0000","last_active":"2023-12-19T14:40:16.030803+0000","last_peered":"2023-12-19T14:40:16.030803+0000","last_clean":"2023-12-19T14:40:16.030803+0000","last_became_active":"2023-12-19T14:40:16.016016+0000","last_became_peered":"2023-12-19T14:40:16.016016+0000","last_unstale":"2023-12-19T14:40:16.030803+0000","last_undegraded":"2023-12-19T14:40:16.030803+0000","last_fullsized":"2023-12-19T14:40:16.030803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:19:40.993825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.019078+0000","last_change":"2023-12-19T14:40:16.015650+0000","last_active":"2023-12-19T14:40:16.019078+0000","last_peered":"2023-12-19T14:40:16.019078+0000","last_clean":"2023-12-19T14:40:16.019078+0000","last_became_active":"2023-12-19T14:40:16.015289+0000","last_became_peered":"2023-12-19T14:40:16.015289+0000","last_unstale":"2023-12-19T14:40:16.019078+0000","last_undegraded":"2023-12-19T14:40:16.019078+0000","last_fullsized":"2023-12-19T14:40:16.019078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032171+0000","last_change":"2023-12-19T14:40:16.016717+0000","last_active":"2023-12-19T14:40:16.032171+0000","last_peered":"2023-12-19T14:40:16.032171+0000","last_clean":"2023-12-19T14:40:16.032171+0000","last_became_active":"2023-12-19T14:40:16.016405+0000","last_became_peered":"2023-12-19T14:40:16.016405+0000","last_unstale":"2023-12-19T14:40:16.032171+0000","last_undegraded":"2023-12-19T14:40:16.032171+0000","last_fullsized":"2023-12-19T14:40:16.032171+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:19.012863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029071+0000","last_change":"2023-12-19T14:40:16.021412+0000","last_active":"2023-12-19T14:40:16.029071+0000","last_peered":"2023-12-19T14:40:16.029071+0000","last_clean":"2023-12-19T14:40:16.029071+0000","last_became_active":"2023-12-19T14:40:16.021024+0000","last_became_peered":"2023-12-19T14:40:16.021024+0000","last_unstale":"2023-12-19T14:40:16.029071+0000","last_undegraded":"2023-12-19T14:40:16.029071+0000","last_fullsized":"2023-12-19T14:40:16.029071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:48.154689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.027047+0000","last_change":"2023-12-19T14:40:16.014617+0000","last_active":"2023-12-19T14:40:16.027047+0000","last_peered":"2023-12-19T14:40:16.027047+0000","last_clean":"2023-12-19T14:40:16.027047+0000","last_became_active":"2023-12-19T14:40:16.014369+0000","last_became_peered":"2023-12-19T14:40:16.014369+0000","last_unstale":"2023-12-19T14:40:16.027047+0000","last_undegraded":"2023-12-19T14:40:16.027047+0000","last_fullsized":"2023-12-19T14:40:16.027047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.017104+0000","last_change":"2023-12-19T14:40:16.008130+0000","last_active":"2023-12-19T14:40:16.017104+0000","last_peered":"2023-12-19T14:40:16.017104+0000","last_clean":"2023-12-19T14:40:16.017104+0000","last_became_active":"2023-12-19T14:40:16.007595+0000","last_became_peered":"2023-12-19T14:40:16.007595+0000","last_unstale":"2023-12-19T14:40:16.017104+0000","last_undegraded":"2023-12-19T14:40:16.017104+0000","last_fullsized":"2023-12-19T14:40:16.017104+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.006387+0000","last_change":"2023-12-19T14:40:16.003113+0000","last_active":"2023-12-19T14:40:16.006387+0000","last_peered":"2023-12-19T14:40:16.006387+0000","last_clean":"2023-12-19T14:40:16.006387+0000","last_became_active":"2023-12-19T14:40:16.002815+0000","last_became_peered":"2023-12-19T14:40:16.002815+0000","last_unstale":"2023-12-19T14:40:16.006387+0000","last_undegraded":"2023-12-19T14:40:16.006387+0000","last_fullsized":"2023-12-19T14:40:16.006387+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.033227+0000","last_change":"2023-12-19T14:40:16.018032+0000","last_active":"2023-12-19T14:40:16.033227+0000","last_peered":"2023-12-19T14:40:16.033227+0000","last_clean":"2023-12-19T14:40:16.033227+0000","last_became_active":"2023-12-19T14:40:16.017680+0000","last_became_peered":"2023-12-19T14:40:16.017680+0000","last_unstale":"2023-12-19T14:40:16.033227+0000","last_undegraded":"2023-12-19T14:40:16.033227+0000","last_fullsized":"2023-12-19T14:40:16.033227+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:07:45.173829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.027051+0000","last_change":"2023-12-19T14:40:16.018545+0000","last_active":"2023-12-19T14:40:16.027051+0000","last_peered":"2023-12-19T14:40:16.027051+0000","last_clean":"2023-12-19T14:40:16.027051+0000","last_became_active":"2023-12-19T14:40:16.014365+0000","last_became_peered":"2023-12-19T14:40:16.014365+0000","last_unstale":"2023-12-19T14:40:16.027051+0000","last_undegraded":"2023-12-19T14:40:16.027051+0000","last_fullsized":"2023-12-19T14:40:16.027051+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030629+0000","last_change":"2023-12-19T14:40:16.015178+0000","last_active":"2023-12-19T14:40:16.030629+0000","last_peered":"2023-12-19T14:40:16.030629+0000","last_clean":"2023-12-19T14:40:16.030629+0000","last_became_active":"2023-12-19T14:40:16.014833+0000","last_became_peered":"2023-12-19T14:40:16.014833+0000","last_unstale":"2023-12-19T14:40:16.030629+0000","last_undegraded":"2023-12-19T14:40:16.030629+0000","last_fullsized":"2023-12-19T14:40:16.030629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:11.371631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029615+0000","last_change":"2023-12-19T14:40:16.022595+0000","last_active":"2023-12-19T14:40:16.029615+0000","last_peered":"2023-12-19T14:40:16.029615+0000","last_clean":"2023-12-19T14:40:16.029615+0000","last_became_active":"2023-12-19T14:40:16.021915+0000","last_became_peered":"2023-12-19T14:40:16.021915+0000","last_unstale":"2023-12-19T14:40:16.029615+0000","last_undegraded":"2023-12-19T14:40:16.029615+0000","last_fullsized":"2023-12-19T14:40:16.029615+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:02:51.141453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030737+0000","last_change":"2023-12-19T14:40:16.016048+0000","last_active":"2023-12-19T14:40:16.030737+0000","last_peered":"2023-12-19T14:40:16.030737+0000","last_clean":"2023-12-19T14:40:16.030737+0000","last_became_active":"2023-12-19T14:40:16.013767+0000","last_became_peered":"2023-12-19T14:40:16.013767+0000","last_unstale":"2023-12-19T14:40:16.030737+0000","last_undegraded":"2023-12-19T14:40:16.030737+0000","last_fullsized":"2023-12-19T14:40:16.030737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:51.058291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032735+0000","last_change":"2023-12-19T14:40:16.016797+0000","last_active":"2023-12-19T14:40:16.032735+0000","last_peered":"2023-12-19T14:40:16.032735+0000","last_clean":"2023-12-19T14:40:16.032735+0000","last_became_active":"2023-12-19T14:40:16.013095+0000","last_became_peered":"2023-12-19T14:40:16.013095+0000","last_unstale":"2023-12-19T14:40:16.032735+0000","last_undegraded":"2023-12-19T14:40:16.032735+0000","last_fullsized":"2023-12-19T14:40:16.032735+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:20.301851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.017085+0000","last_change":"2023-12-19T14:40:16.007894+0000","last_active":"2023-12-19T14:40:16.017085+0000","last_peered":"2023-12-19T14:40:16.017085+0000","last_clean":"2023-12-19T14:40:16.017085+0000","last_became_active":"2023-12-19T14:40:16.007202+0000","last_became_peered":"2023-12-19T14:40:16.007202+0000","last_unstale":"2023-12-19T14:40:16.017085+0000","last_undegraded":"2023-12-19T14:40:16.017085+0000","last_fullsized":"2023-12-19T14:40:16.017085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026828+0000","last_change":"2023-12-19T14:40:16.013263+0000","last_active":"2023-12-19T14:40:16.026828+0000","last_peered":"2023-12-19T14:40:16.026828+0000","last_clean":"2023-12-19T14:40:16.026828+0000","last_became_active":"2023-12-19T14:40:16.012879+0000","last_became_peered":"2023-12-19T14:40:16.012879+0000","last_unstale":"2023-12-19T14:40:16.026828+0000","last_undegraded":"2023-12-19T14:40:16.026828+0000","last_fullsized":"2023-12-19T14:40:16.026828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:51.519092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028055+0000","last_change":"2023-12-19T14:40:16.021053+0000","last_active":"2023-12-19T14:40:16.028055+0000","last_peered":"2023-12-19T14:40:16.028055+0000","last_clean":"2023-12-19T14:40:16.028055+0000","last_became_active":"2023-12-19T14:40:16.020159+0000","last_became_peered":"2023-12-19T14:40:16.020159+0000","last_unstale":"2023-12-19T14:40:16.028055+0000","last_undegraded":"2023-12-19T14:40:16.028055+0000","last_fullsized":"2023-12-19T14:40:16.028055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.035615+0000","last_change":"2023-12-19T14:40:14.998033+0000","last_active":"2023-12-19T14:40:16.035615+0000","last_peered":"2023-12-19T14:40:16.035615+0000","last_clean":"2023-12-19T14:40:16.035615+0000","last_became_active":"2023-12-19T14:40:14.997769+0000","last_became_peered":"2023-12-19T14:40:14.997769+0000","last_unstale":"2023-12-19T14:40:16.035615+0000","last_undegraded":"2023-12-19T14:40:16.035615+0000","last_fullsized":"2023-12-19T14:40:16.035615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:05.375208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028276+0000","last_change":"2023-12-19T14:40:16.016634+0000","last_active":"2023-12-19T14:40:16.028276+0000","last_peered":"2023-12-19T14:40:16.028276+0000","last_clean":"2023-12-19T14:40:16.028276+0000","last_became_active":"2023-12-19T14:40:16.016333+0000","last_became_peered":"2023-12-19T14:40:16.016333+0000","last_unstale":"2023-12-19T14:40:16.028276+0000","last_undegraded":"2023-12-19T14:40:16.028276+0000","last_fullsized":"2023-12-19T14:40:16.028276+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029628+0000","last_change":"2023-12-19T14:40:15.022907+0000","last_active":"2023-12-19T14:40:16.029628+0000","last_peered":"2023-12-19T14:40:16.029628+0000","last_clean":"2023-12-19T14:40:16.029628+0000","last_became_active":"2023-12-19T14:40:15.022603+0000","last_became_peered":"2023-12-19T14:40:15.022603+0000","last_unstale":"2023-12-19T14:40:16.029628+0000","last_undegraded":"2023-12-19T14:40:16.029628+0000","last_fullsized":"2023-12-19T14:40:16.029628+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.035415+0000","last_change":"2023-12-19T14:40:16.030283+0000","last_active":"2023-12-19T14:40:16.035415+0000","last_peered":"2023-12-19T14:40:16.035415+0000","last_clean":"2023-12-19T14:40:16.035415+0000","last_became_active":"2023-12-19T14:40:16.029979+0000","last_became_peered":"2023-12-19T14:40:16.029979+0000","last_unstale":"2023-12-19T14:40:16.035415+0000","last_undegraded":"2023-12-19T14:40:16.035415+0000","last_fullsized":"2023-12-19T14:40:16.035415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:26.826515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030823+0000","last_change":"2023-12-19T14:40:15.021930+0000","last_active":"2023-12-19T14:40:16.030823+0000","last_peered":"2023-12-19T14:40:16.030823+0000","last_clean":"2023-12-19T14:40:16.030823+0000","last_became_active":"2023-12-19T14:40:15.021586+0000","last_became_peered":"2023-12-19T14:40:15.021586+0000","last_unstale":"2023-12-19T14:40:16.030823+0000","last_undegraded":"2023-12-19T14:40:16.030823+0000","last_fullsized":"2023-12-19T14:40:16.030823+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.020780+0000","last_change":"2023-12-19T14:40:16.006120+0000","last_active":"2023-12-19T14:40:16.020780+0000","last_peered":"2023-12-19T14:40:16.020780+0000","last_clean":"2023-12-19T14:40:16.020780+0000","last_became_active":"2023-12-19T14:40:16.005848+0000","last_became_peered":"2023-12-19T14:40:16.005848+0000","last_unstale":"2023-12-19T14:40:16.020780+0000","last_undegraded":"2023-12-19T14:40:16.020780+0000","last_fullsized":"2023-12-19T14:40:16.020780+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:21.117131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026244+0000","last_change":"2023-12-19T14:40:15.026312+0000","last_active":"2023-12-19T14:40:16.026244+0000","last_peered":"2023-12-19T14:40:16.026244+0000","last_clean":"2023-12-19T14:40:16.026244+0000","last_became_active":"2023-12-19T14:40:15.025957+0000","last_became_peered":"2023-12-19T14:40:15.025957+0000","last_unstale":"2023-12-19T14:40:16.026244+0000","last_undegraded":"2023-12-19T14:40:16.026244+0000","last_fullsized":"2023-12-19T14:40:16.026244+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032496+0000","last_change":"2023-12-19T14:40:16.020481+0000","last_active":"2023-12-19T14:40:16.032496+0000","last_peered":"2023-12-19T14:40:16.032496+0000","last_clean":"2023-12-19T14:40:16.032496+0000","last_became_active":"2023-12-19T14:40:16.019982+0000","last_became_peered":"2023-12-19T14:40:16.019982+0000","last_unstale":"2023-12-19T14:40:16.032496+0000","last_undegraded":"2023-12-19T14:40:16.032496+0000","last_fullsized":"2023-12-19T14:40:16.032496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:45.301747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.023223+0000","last_change":"2023-12-19T14:40:15.025292+0000","last_active":"2023-12-19T14:40:16.023223+0000","last_peered":"2023-12-19T14:40:16.023223+0000","last_clean":"2023-12-19T14:40:16.023223+0000","last_became_active":"2023-12-19T14:40:15.024103+0000","last_became_peered":"2023-12-19T14:40:15.024103+0000","last_unstale":"2023-12-19T14:40:16.023223+0000","last_undegraded":"2023-12-19T14:40:16.023223+0000","last_fullsized":"2023-12-19T14:40:16.023223+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:59.010045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024795+0000","last_change":"2023-12-19T14:40:16.017055+0000","last_active":"2023-12-19T14:40:16.024795+0000","last_peered":"2023-12-19T14:40:16.024795+0000","last_clean":"2023-12-19T14:40:16.024795+0000","last_became_active":"2023-12-19T14:40:16.016737+0000","last_became_peered":"2023-12-19T14:40:16.016737+0000","last_unstale":"2023-12-19T14:40:16.024795+0000","last_undegraded":"2023-12-19T14:40:16.024795+0000","last_fullsized":"2023-12-19T14:40:16.024795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:45.804177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.034938+0000","last_change":"2023-12-19T14:40:15.004455+0000","last_active":"2023-12-19T14:40:16.034938+0000","last_peered":"2023-12-19T14:40:16.034938+0000","last_clean":"2023-12-19T14:40:16.034938+0000","last_became_active":"2023-12-19T14:40:15.004022+0000","last_became_peered":"2023-12-19T14:40:15.004022+0000","last_unstale":"2023-12-19T14:40:16.034938+0000","last_undegraded":"2023-12-19T14:40:16.034938+0000","last_fullsized":"2023-12-19T14:40:16.034938+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.033628+0000","last_change":"2023-12-19T14:40:16.026942+0000","last_active":"2023-12-19T14:40:16.033628+0000","last_peered":"2023-12-19T14:40:16.033628+0000","last_clean":"2023-12-19T14:40:16.033628+0000","last_became_active":"2023-12-19T14:40:16.026621+0000","last_became_peered":"2023-12-19T14:40:16.026621+0000","last_unstale":"2023-12-19T14:40:16.033628+0000","last_undegraded":"2023-12-19T14:40:16.033628+0000","last_fullsized":"2023-12-19T14:40:16.033628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:45.804177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.000024+0000","last_change":"2023-12-19T14:40:15.015560+0000","last_active":"2023-12-19T14:40:16.000024+0000","last_peered":"2023-12-19T14:40:16.000024+0000","last_clean":"2023-12-19T14:40:16.000024+0000","last_became_active":"2023-12-19T14:40:15.015177+0000","last_became_peered":"2023-12-19T14:40:15.015177+0000","last_unstale":"2023-12-19T14:40:16.000024+0000","last_undegraded":"2023-12-19T14:40:16.000024+0000","last_fullsized":"2023-12-19T14:40:16.000024+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.017376+0000","last_change":"2023-12-19T14:40:16.010663+0000","last_active":"2023-12-19T14:40:16.017376+0000","last_peered":"2023-12-19T14:40:16.017376+0000","last_clean":"2023-12-19T14:40:16.017376+0000","last_became_active":"2023-12-19T14:40:16.010218+0000","last_became_peered":"2023-12-19T14:40:16.010218+0000","last_unstale":"2023-12-19T14:40:16.017376+0000","last_undegraded":"2023-12-19T14:40:16.017376+0000","last_fullsized":"2023-12-19T14:40:16.017376+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:45.804177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.013438+0000","last_change":"2023-12-19T14:40:14.994802+0000","last_active":"2023-12-19T14:40:16.013438+0000","last_peered":"2023-12-19T14:40:16.013438+0000","last_clean":"2023-12-19T14:40:16.013438+0000","last_became_active":"2023-12-19T14:40:14.994524+0000","last_became_peered":"2023-12-19T14:40:14.994524+0000","last_unstale":"2023-12-19T14:40:16.013438+0000","last_undegraded":"2023-12-19T14:40:16.013438+0000","last_fullsized":"2023-12-19T14:40:16.013438+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.020713+0000","last_change":"2023-12-19T14:40:16.019060+0000","last_active":"2023-12-19T14:40:16.020713+0000","last_peered":"2023-12-19T14:40:16.020713+0000","last_clean":"2023-12-19T14:40:16.020713+0000","last_became_active":"2023-12-19T14:40:16.014869+0000","last_became_peered":"2023-12-19T14:40:16.014869+0000","last_unstale":"2023-12-19T14:40:16.020713+0000","last_undegraded":"2023-12-19T14:40:16.020713+0000","last_fullsized":"2023-12-19T14:40:16.020713+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:45.106084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.995401+0000","last_change":"2023-12-19T14:40:15.024050+0000","last_active":"2023-12-19T14:40:15.995401+0000","last_peered":"2023-12-19T14:40:15.995401+0000","last_clean":"2023-12-19T14:40:15.995401+0000","last_became_active":"2023-12-19T14:40:15.023710+0000","last_became_peered":"2023-12-19T14:40:15.023710+0000","last_unstale":"2023-12-19T14:40:15.995401+0000","last_undegraded":"2023-12-19T14:40:15.995401+0000","last_fullsized":"2023-12-19T14:40:15.995401+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:50.513294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028453+0000","last_change":"2023-12-19T14:40:16.024016+0000","last_active":"2023-12-19T14:40:16.028453+0000","last_peered":"2023-12-19T14:40:16.028453+0000","last_clean":"2023-12-19T14:40:16.028453+0000","last_became_active":"2023-12-19T14:40:16.018742+0000","last_became_peered":"2023-12-19T14:40:16.018742+0000","last_unstale":"2023-12-19T14:40:16.028453+0000","last_undegraded":"2023-12-19T14:40:16.028453+0000","last_fullsized":"2023-12-19T14:40:16.028453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.999629+0000","last_change":"2023-12-19T14:40:15.025954+0000","last_active":"2023-12-19T14:40:15.999629+0000","last_peered":"2023-12-19T14:40:15.999629+0000","last_clean":"2023-12-19T14:40:15.999629+0000","last_became_active":"2023-12-19T14:40:15.025658+0000","last_became_peered":"2023-12-19T14:40:15.025658+0000","last_unstale":"2023-12-19T14:40:15.999629+0000","last_undegraded":"2023-12-19T14:40:15.999629+0000","last_fullsized":"2023-12-19T14:40:15.999629+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032293+0000","last_change":"2023-12-19T14:40:16.031007+0000","last_active":"2023-12-19T14:40:16.032293+0000","last_peered":"2023-12-19T14:40:16.032293+0000","last_clean":"2023-12-19T14:40:16.032293+0000","last_became_active":"2023-12-19T14:40:16.029753+0000","last_became_peered":"2023-12-19T14:40:16.029753+0000","last_unstale":"2023-12-19T14:40:16.032293+0000","last_undegraded":"2023-12-19T14:40:16.032293+0000","last_fullsized":"2023-12-19T14:40:16.032293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:39.566549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029610+0000","last_change":"2023-12-19T14:40:15.023326+0000","last_active":"2023-12-19T14:40:16.029610+0000","last_peered":"2023-12-19T14:40:16.029610+0000","last_clean":"2023-12-19T14:40:16.029610+0000","last_became_active":"2023-12-19T14:40:15.022920+0000","last_became_peered":"2023-12-19T14:40:15.022920+0000","last_unstale":"2023-12-19T14:40:16.029610+0000","last_undegraded":"2023-12-19T14:40:16.029610+0000","last_fullsized":"2023-12-19T14:40:16.029610+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:44.100285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.038105+0000","last_change":"2023-12-19T14:40:16.032373+0000","last_active":"2023-12-19T14:40:16.038105+0000","last_peered":"2023-12-19T14:40:16.038105+0000","last_clean":"2023-12-19T14:40:16.038105+0000","last_became_active":"2023-12-19T14:40:16.031815+0000","last_became_peered":"2023-12-19T14:40:16.031815+0000","last_unstale":"2023-12-19T14:40:16.038105+0000","last_undegraded":"2023-12-19T14:40:16.038105+0000","last_fullsized":"2023-12-19T14:40:16.038105+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.036769+0000","last_change":"2023-12-19T14:40:15.023075+0000","last_active":"2023-12-19T14:40:16.036769+0000","last_peered":"2023-12-19T14:40:16.036769+0000","last_clean":"2023-12-19T14:40:16.036769+0000","last_became_active":"2023-12-19T14:40:15.022756+0000","last_became_peered":"2023-12-19T14:40:15.022756+0000","last_unstale":"2023-12-19T14:40:16.036769+0000","last_undegraded":"2023-12-19T14:40:16.036769+0000","last_fullsized":"2023-12-19T14:40:16.036769+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:53.364137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.032514+0000","last_change":"2023-12-19T14:40:16.026259+0000","last_active":"2023-12-19T14:40:16.032514+0000","last_peered":"2023-12-19T14:40:16.032514+0000","last_clean":"2023-12-19T14:40:16.032514+0000","last_became_active":"2023-12-19T14:40:16.025991+0000","last_became_peered":"2023-12-19T14:40:16.025991+0000","last_unstale":"2023-12-19T14:40:16.032514+0000","last_undegraded":"2023-12-19T14:40:16.032514+0000","last_fullsized":"2023-12-19T14:40:16.032514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.033226+0000","last_change":"2023-12-19T14:40:15.026805+0000","last_active":"2023-12-19T14:40:16.033226+0000","last_peered":"2023-12-19T14:40:16.033226+0000","last_clean":"2023-12-19T14:40:16.033226+0000","last_became_active":"2023-12-19T14:40:15.025038+0000","last_became_peered":"2023-12-19T14:40:15.025038+0000","last_unstale":"2023-12-19T14:40:16.033226+0000","last_undegraded":"2023-12-19T14:40:16.033226+0000","last_fullsized":"2023-12-19T14:40:16.033226+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:55.662568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.025650+0000","last_change":"2023-12-19T14:40:16.023331+0000","last_active":"2023-12-19T14:40:16.025650+0000","last_peered":"2023-12-19T14:40:16.025650+0000","last_clean":"2023-12-19T14:40:16.025650+0000","last_became_active":"2023-12-19T14:40:16.023036+0000","last_became_peered":"2023-12-19T14:40:16.023036+0000","last_unstale":"2023-12-19T14:40:16.025650+0000","last_undegraded":"2023-12-19T14:40:16.025650+0000","last_fullsized":"2023-12-19T14:40:16.025650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:22:52.715735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.000209+0000","last_change":"2023-12-19T14:40:15.017622+0000","last_active":"2023-12-19T14:40:16.000209+0000","last_peered":"2023-12-19T14:40:16.000209+0000","last_clean":"2023-12-19T14:40:16.000209+0000","last_became_active":"2023-12-19T14:40:15.017316+0000","last_became_peered":"2023-12-19T14:40:15.017316+0000","last_unstale":"2023-12-19T14:40:16.000209+0000","last_undegraded":"2023-12-19T14:40:16.000209+0000","last_fullsized":"2023-12-19T14:40:16.000209+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.008790+0000","last_change":"2023-12-19T14:40:16.005814+0000","last_active":"2023-12-19T14:40:16.008790+0000","last_peered":"2023-12-19T14:40:16.008790+0000","last_clean":"2023-12-19T14:40:16.008790+0000","last_became_active":"2023-12-19T14:40:16.005492+0000","last_became_peered":"2023-12-19T14:40:16.005492+0000","last_unstale":"2023-12-19T14:40:16.008790+0000","last_undegraded":"2023-12-19T14:40:16.008790+0000","last_fullsized":"2023-12-19T14:40:16.008790+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:00.015844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.995258+0000","last_change":"2023-12-19T14:40:15.001820+0000","last_active":"2023-12-19T14:40:15.995258+0000","last_peered":"2023-12-19T14:40:15.995258+0000","last_clean":"2023-12-19T14:40:15.995258+0000","last_became_active":"2023-12-19T14:40:15.001305+0000","last_became_peered":"2023-12-19T14:40:15.001305+0000","last_unstale":"2023-12-19T14:40:15.995258+0000","last_undegraded":"2023-12-19T14:40:15.995258+0000","last_fullsized":"2023-12-19T14:40:15.995258+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:52.732501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030734+0000","last_change":"2023-12-19T14:40:16.023719+0000","last_active":"2023-12-19T14:40:16.030734+0000","last_peered":"2023-12-19T14:40:16.030734+0000","last_clean":"2023-12-19T14:40:16.030734+0000","last_became_active":"2023-12-19T14:40:16.022900+0000","last_became_peered":"2023-12-19T14:40:16.022900+0000","last_unstale":"2023-12-19T14:40:16.030734+0000","last_undegraded":"2023-12-19T14:40:16.030734+0000","last_fullsized":"2023-12-19T14:40:16.030734+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:21.350727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.035900+0000","last_change":"2023-12-19T14:40:15.003285+0000","last_active":"2023-12-19T14:40:16.035900+0000","last_peered":"2023-12-19T14:40:16.035900+0000","last_clean":"2023-12-19T14:40:16.035900+0000","last_became_active":"2023-12-19T14:40:15.002941+0000","last_became_peered":"2023-12-19T14:40:15.002941+0000","last_unstale":"2023-12-19T14:40:16.035900+0000","last_undegraded":"2023-12-19T14:40:16.035900+0000","last_fullsized":"2023-12-19T14:40:16.035900+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.019618+0000","last_change":"2023-12-19T14:40:16.007946+0000","last_active":"2023-12-19T14:40:16.019618+0000","last_peered":"2023-12-19T14:40:16.019618+0000","last_clean":"2023-12-19T14:40:16.019618+0000","last_became_active":"2023-12-19T14:40:16.007574+0000","last_became_peered":"2023-12-19T14:40:16.007574+0000","last_unstale":"2023-12-19T14:40:16.019618+0000","last_undegraded":"2023-12-19T14:40:16.019618+0000","last_fullsized":"2023-12-19T14:40:16.019618+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:59:15.105141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.030879+0000","last_change":"2023-12-19T14:40:15.003942+0000","last_active":"2023-12-19T14:40:16.030879+0000","last_peered":"2023-12-19T14:40:16.030879+0000","last_clean":"2023-12-19T14:40:16.030879+0000","last_became_active":"2023-12-19T14:40:15.003631+0000","last_became_peered":"2023-12-19T14:40:15.003631+0000","last_unstale":"2023-12-19T14:40:16.030879+0000","last_undegraded":"2023-12-19T14:40:16.030879+0000","last_fullsized":"2023-12-19T14:40:16.030879+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:22.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024100+0000","last_change":"2023-12-19T14:40:16.016021+0000","last_active":"2023-12-19T14:40:16.024100+0000","last_peered":"2023-12-19T14:40:16.024100+0000","last_clean":"2023-12-19T14:40:16.024100+0000","last_became_active":"2023-12-19T14:40:16.015757+0000","last_became_peered":"2023-12-19T14:40:16.015757+0000","last_unstale":"2023-12-19T14:40:16.024100+0000","last_undegraded":"2023-12-19T14:40:16.024100+0000","last_fullsized":"2023-12-19T14:40:16.024100+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:59:15.105141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029897+0000","last_change":"2023-12-19T14:40:15.002317+0000","last_active":"2023-12-19T14:40:16.029897+0000","last_peered":"2023-12-19T14:40:16.029897+0000","last_clean":"2023-12-19T14:40:16.029897+0000","last_became_active":"2023-12-19T14:40:15.002056+0000","last_became_peered":"2023-12-19T14:40:15.002056+0000","last_unstale":"2023-12-19T14:40:16.029897+0000","last_undegraded":"2023-12-19T14:40:16.029897+0000","last_fullsized":"2023-12-19T14:40:16.029897+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:38.560751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.038146+0000","last_change":"2023-12-19T14:40:16.032392+0000","last_active":"2023-12-19T14:40:16.038146+0000","last_peered":"2023-12-19T14:40:16.038146+0000","last_clean":"2023-12-19T14:40:16.038146+0000","last_became_active":"2023-12-19T14:40:16.032014+0000","last_became_peered":"2023-12-19T14:40:16.032014+0000","last_unstale":"2023-12-19T14:40:16.038146+0000","last_undegraded":"2023-12-19T14:40:16.038146+0000","last_fullsized":"2023-12-19T14:40:16.038146+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:58:49.397081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028993+0000","last_change":"2023-12-19T14:40:15.010764+0000","last_active":"2023-12-19T14:40:16.028993+0000","last_peered":"2023-12-19T14:40:16.028993+0000","last_clean":"2023-12-19T14:40:16.028993+0000","last_became_active":"2023-12-19T14:40:15.010427+0000","last_became_peered":"2023-12-19T14:40:15.010427+0000","last_unstale":"2023-12-19T14:40:16.028993+0000","last_undegraded":"2023-12-19T14:40:16.028993+0000","last_fullsized":"2023-12-19T14:40:16.028993+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:11.000395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029434+0000","last_change":"2023-12-19T14:40:16.026341+0000","last_active":"2023-12-19T14:40:16.029434+0000","last_peered":"2023-12-19T14:40:16.029434+0000","last_clean":"2023-12-19T14:40:16.029434+0000","last_became_active":"2023-12-19T14:40:16.015424+0000","last_became_peered":"2023-12-19T14:40:16.015424+0000","last_unstale":"2023-12-19T14:40:16.029434+0000","last_undegraded":"2023-12-19T14:40:16.029434+0000","last_fullsized":"2023-12-19T14:40:16.029434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:44.589442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.000094+0000","last_change":"2023-12-19T14:40:15.021360+0000","last_active":"2023-12-19T14:40:16.000094+0000","last_peered":"2023-12-19T14:40:16.000094+0000","last_clean":"2023-12-19T14:40:16.000094+0000","last_became_active":"2023-12-19T14:40:15.020201+0000","last_became_peered":"2023-12-19T14:40:15.020201+0000","last_unstale":"2023-12-19T14:40:16.000094+0000","last_undegraded":"2023-12-19T14:40:16.000094+0000","last_fullsized":"2023-12-19T14:40:16.000094+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:19.296052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.007367+0000","last_change":"2023-12-19T14:40:16.004066+0000","last_active":"2023-12-19T14:40:16.007367+0000","last_peered":"2023-12-19T14:40:16.007367+0000","last_clean":"2023-12-19T14:40:16.007367+0000","last_became_active":"2023-12-19T14:40:16.003715+0000","last_became_peered":"2023-12-19T14:40:16.003715+0000","last_unstale":"2023-12-19T14:40:16.007367+0000","last_undegraded":"2023-12-19T14:40:16.007367+0000","last_fullsized":"2023-12-19T14:40:16.007367+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:54.369935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.020840+0000","last_change":"2023-12-19T14:40:15.026807+0000","last_active":"2023-12-19T14:40:16.020840+0000","last_peered":"2023-12-19T14:40:16.020840+0000","last_clean":"2023-12-19T14:40:16.020840+0000","last_became_active":"2023-12-19T14:40:15.026425+0000","last_became_peered":"2023-12-19T14:40:15.026425+0000","last_unstale":"2023-12-19T14:40:16.020840+0000","last_undegraded":"2023-12-19T14:40:16.020840+0000","last_fullsized":"2023-12-19T14:40:16.020840+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:19.668161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026568+0000","last_change":"2023-12-19T14:40:16.020932+0000","last_active":"2023-12-19T14:40:16.026568+0000","last_peered":"2023-12-19T14:40:16.026568+0000","last_clean":"2023-12-19T14:40:16.026568+0000","last_became_active":"2023-12-19T14:40:16.020667+0000","last_became_peered":"2023-12-19T14:40:16.020667+0000","last_unstale":"2023-12-19T14:40:16.026568+0000","last_undegraded":"2023-12-19T14:40:16.026568+0000","last_fullsized":"2023-12-19T14:40:16.026568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.994772+0000","last_change":"2023-12-19T14:40:15.013522+0000","last_active":"2023-12-19T14:40:15.994772+0000","last_peered":"2023-12-19T14:40:15.994772+0000","last_clean":"2023-12-19T14:40:15.994772+0000","last_became_active":"2023-12-19T14:40:15.013241+0000","last_became_peered":"2023-12-19T14:40:15.013241+0000","last_unstale":"2023-12-19T14:40:15.994772+0000","last_undegraded":"2023-12-19T14:40:15.994772+0000","last_fullsized":"2023-12-19T14:40:15.994772+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:49.758106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.024819+0000","last_change":"2023-12-19T14:40:16.017375+0000","last_active":"2023-12-19T14:40:16.024819+0000","last_peered":"2023-12-19T14:40:16.024819+0000","last_clean":"2023-12-19T14:40:16.024819+0000","last_became_active":"2023-12-19T14:40:16.004944+0000","last_became_peered":"2023-12-19T14:40:16.004944+0000","last_unstale":"2023-12-19T14:40:16.024819+0000","last_undegraded":"2023-12-19T14:40:16.024819+0000","last_fullsized":"2023-12-19T14:40:16.024819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:46.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.995800+0000","last_change":"2023-12-19T14:40:15.001340+0000","last_active":"2023-12-19T14:40:15.995800+0000","last_peered":"2023-12-19T14:40:15.995800+0000","last_clean":"2023-12-19T14:40:15.995800+0000","last_became_active":"2023-12-19T14:40:15.001076+0000","last_became_peered":"2023-12-19T14:40:15.001076+0000","last_unstale":"2023-12-19T14:40:15.995800+0000","last_undegraded":"2023-12-19T14:40:15.995800+0000","last_fullsized":"2023-12-19T14:40:15.995800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:10.365832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.017356+0000","last_change":"2023-12-19T14:40:16.010481+0000","last_active":"2023-12-19T14:40:16.017356+0000","last_peered":"2023-12-19T14:40:16.017356+0000","last_clean":"2023-12-19T14:40:16.017356+0000","last_became_active":"2023-12-19T14:40:16.009836+0000","last_became_peered":"2023-12-19T14:40:16.009836+0000","last_unstale":"2023-12-19T14:40:16.017356+0000","last_undegraded":"2023-12-19T14:40:16.017356+0000","last_fullsized":"2023-12-19T14:40:16.017356+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:26.826515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.995950+0000","last_change":"2023-12-19T14:40:15.024120+0000","last_active":"2023-12-19T14:40:15.995950+0000","last_peered":"2023-12-19T14:40:15.995950+0000","last_clean":"2023-12-19T14:40:15.995950+0000","last_became_active":"2023-12-19T14:40:15.023840+0000","last_became_peered":"2023-12-19T14:40:15.023840+0000","last_unstale":"2023-12-19T14:40:15.995950+0000","last_undegraded":"2023-12-19T14:40:15.995950+0000","last_fullsized":"2023-12-19T14:40:15.995950+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:06.220833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029323+0000","last_change":"2023-12-19T14:40:16.018554+0000","last_active":"2023-12-19T14:40:16.029323+0000","last_peered":"2023-12-19T14:40:16.029323+0000","last_clean":"2023-12-19T14:40:16.029323+0000","last_became_active":"2023-12-19T14:40:16.018250+0000","last_became_peered":"2023-12-19T14:40:16.018250+0000","last_unstale":"2023-12-19T14:40:16.029323+0000","last_undegraded":"2023-12-19T14:40:16.029323+0000","last_fullsized":"2023-12-19T14:40:16.029323+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:56.668367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.021395+0000","last_change":"2023-12-19T14:40:15.022435+0000","last_active":"2023-12-19T14:40:16.021395+0000","last_peered":"2023-12-19T14:40:16.021395+0000","last_clean":"2023-12-19T14:40:16.021395+0000","last_became_active":"2023-12-19T14:40:15.022169+0000","last_became_peered":"2023-12-19T14:40:15.022169+0000","last_unstale":"2023-12-19T14:40:16.021395+0000","last_undegraded":"2023-12-19T14:40:16.021395+0000","last_fullsized":"2023-12-19T14:40:16.021395+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:43.583643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.028473+0000","last_change":"2023-12-19T14:40:16.024180+0000","last_active":"2023-12-19T14:40:16.028473+0000","last_peered":"2023-12-19T14:40:16.028473+0000","last_clean":"2023-12-19T14:40:16.028473+0000","last_became_active":"2023-12-19T14:40:16.019599+0000","last_became_peered":"2023-12-19T14:40:16.019599+0000","last_unstale":"2023-12-19T14:40:16.028473+0000","last_undegraded":"2023-12-19T14:40:16.028473+0000","last_fullsized":"2023-12-19T14:40:16.028473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:43.747967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T14:40:17.010952+0000","last_change":"2023-12-19T14:40:15.012638+0000","last_active":"2023-12-19T14:40:17.010952+0000","last_peered":"2023-12-19T14:40:17.010952+0000","last_clean":"2023-12-19T14:40:17.010952+0000","last_became_active":"2023-12-19T14:40:15.012351+0000","last_became_peered":"2023-12-19T14:40:15.012351+0000","last_unstale":"2023-12-19T14:40:17.010952+0000","last_undegraded":"2023-12-19T14:40:17.010952+0000","last_fullsized":"2023-12-19T14:40:17.010952+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:15.105662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.025516+0000","last_change":"2023-12-19T14:40:16.017939+0000","last_active":"2023-12-19T14:40:16.025516+0000","last_peered":"2023-12-19T14:40:16.025516+0000","last_clean":"2023-12-19T14:40:16.025516+0000","last_became_active":"2023-12-19T14:40:16.017607+0000","last_became_peered":"2023-12-19T14:40:16.017607+0000","last_unstale":"2023-12-19T14:40:16.025516+0000","last_undegraded":"2023-12-19T14:40:16.025516+0000","last_fullsized":"2023-12-19T14:40:16.025516+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:02:51.141453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.007354+0000","last_change":"2023-12-19T14:40:15.029026+0000","last_active":"2023-12-19T14:40:16.007354+0000","last_peered":"2023-12-19T14:40:16.007354+0000","last_clean":"2023-12-19T14:40:16.007354+0000","last_became_active":"2023-12-19T14:40:15.028694+0000","last_became_peered":"2023-12-19T14:40:15.028694+0000","last_unstale":"2023-12-19T14:40:16.007354+0000","last_undegraded":"2023-12-19T14:40:16.007354+0000","last_fullsized":"2023-12-19T14:40:16.007354+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:04:50.052492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.031617+0000","last_change":"2023-12-19T14:40:16.022991+0000","last_active":"2023-12-19T14:40:16.031617+0000","last_peered":"2023-12-19T14:40:16.031617+0000","last_clean":"2023-12-19T14:40:16.031617+0000","last_became_active":"2023-12-19T14:40:16.022681+0000","last_became_peered":"2023-12-19T14:40:16.022681+0000","last_unstale":"2023-12-19T14:40:16.031617+0000","last_undegraded":"2023-12-19T14:40:16.031617+0000","last_fullsized":"2023-12-19T14:40:16.031617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:09:56.757605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.007727+0000","last_change":"2023-12-19T14:40:15.023303+0000","last_active":"2023-12-19T14:40:16.007727+0000","last_peered":"2023-12-19T14:40:16.007727+0000","last_clean":"2023-12-19T14:40:16.007727+0000","last_became_active":"2023-12-19T14:40:15.022812+0000","last_became_peered":"2023-12-19T14:40:15.022812+0000","last_unstale":"2023-12-19T14:40:16.007727+0000","last_undegraded":"2023-12-19T14:40:16.007727+0000","last_fullsized":"2023-12-19T14:40:16.007727+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026659+0000","last_change":"2023-12-19T14:40:16.017589+0000","last_active":"2023-12-19T14:40:16.026659+0000","last_peered":"2023-12-19T14:40:16.026659+0000","last_clean":"2023-12-19T14:40:16.026659+0000","last_became_active":"2023-12-19T14:40:16.017286+0000","last_became_peered":"2023-12-19T14:40:16.017286+0000","last_unstale":"2023-12-19T14:40:16.026659+0000","last_undegraded":"2023-12-19T14:40:16.026659+0000","last_fullsized":"2023-12-19T14:40:16.026659+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:23.492397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.994944+0000","last_change":"2023-12-19T14:40:15.024761+0000","last_active":"2023-12-19T14:40:15.994944+0000","last_peered":"2023-12-19T14:40:15.994944+0000","last_clean":"2023-12-19T14:40:15.994944+0000","last_became_active":"2023-12-19T14:40:15.024406+0000","last_became_peered":"2023-12-19T14:40:15.024406+0000","last_unstale":"2023-12-19T14:40:15.994944+0000","last_undegraded":"2023-12-19T14:40:15.994944+0000","last_fullsized":"2023-12-19T14:40:15.994944+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:36.341963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.026828+0000","last_change":"2023-12-19T14:40:16.014087+0000","last_active":"2023-12-19T14:40:16.026828+0000","last_peered":"2023-12-19T14:40:16.026828+0000","last_clean":"2023-12-19T14:40:16.026828+0000","last_became_active":"2023-12-19T14:40:16.013810+0000","last_became_peered":"2023-12-19T14:40:16.013810+0000","last_unstale":"2023-12-19T14:40:16.026828+0000","last_undegraded":"2023-12-19T14:40:16.026828+0000","last_fullsized":"2023-12-19T14:40:16.026828+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:59:15.105141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.013642+0000","last_change":"2023-12-19T14:40:15.026100+0000","last_active":"2023-12-19T14:40:16.013642+0000","last_peered":"2023-12-19T14:40:16.013642+0000","last_clean":"2023-12-19T14:40:16.013642+0000","last_became_active":"2023-12-19T14:40:15.025647+0000","last_became_peered":"2023-12-19T14:40:15.025647+0000","last_unstale":"2023-12-19T14:40:16.013642+0000","last_undegraded":"2023-12-19T14:40:16.013642+0000","last_fullsized":"2023-12-19T14:40:16.013642+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.008944+0000","last_change":"2023-12-19T14:40:16.006503+0000","last_active":"2023-12-19T14:40:16.008944+0000","last_peered":"2023-12-19T14:40:16.008944+0000","last_clean":"2023-12-19T14:40:16.008944+0000","last_became_active":"2023-12-19T14:40:16.005576+0000","last_became_peered":"2023-12-19T14:40:16.005576+0000","last_unstale":"2023-12-19T14:40:16.008944+0000","last_undegraded":"2023-12-19T14:40:16.008944+0000","last_fullsized":"2023-12-19T14:40:16.008944+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:56.570495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.994201+0000","last_change":"2023-12-19T14:40:15.024302+0000","last_active":"2023-12-19T14:40:15.994201+0000","last_peered":"2023-12-19T14:40:15.994201+0000","last_clean":"2023-12-19T14:40:15.994201+0000","last_became_active":"2023-12-19T14:40:15.024017+0000","last_became_peered":"2023-12-19T14:40:15.024017+0000","last_unstale":"2023-12-19T14:40:15.994201+0000","last_undegraded":"2023-12-19T14:40:15.994201+0000","last_fullsized":"2023-12-19T14:40:15.994201+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:54:55.564697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:16.029112+0000","last_change":"2023-12-19T14:40:16.023236+0000","last_active":"2023-12-19T14:40:16.029112+0000","last_peered":"2023-12-19T14:40:16.029112+0000","last_clean":"2023-12-19T14:40:16.029112+0000","last_became_active":"2023-12-19T14:40:16.021108+0000","last_became_peered":"2023-12-19T14:40:16.021108+0000","last_unstale":"2023-12-19T14:40:16.029112+0000","last_undegraded":"2023-12-19T14:40:16.029112+0000","last_fullsized":"2023-12-19T14:40:16.029112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:14.964555+0000","last_clean_scrub_stamp":"2023-12-19T14:40:14.964555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:41.134391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.998339+0000","last_change":"2023-12-19T14:40:15.021965+0000","last_active":"2023-12-19T14:40:15.998339+0000","last_peered":"2023-12-19T14:40:15.998339+0000","last_clean":"2023-12-19T14:40:15.998339+0000","last_became_active":"2023-12-19T14:40:15.021467+0000","last_became_peered":"2023-12-19T14:40:15.021467+0000","last_unstale":"2023-12-19T14:40:15.998339+0000","last_undegraded":"2023-12-19T14:40:15.998339+0000","last_fullsized":"2023-12-19T14:40:15.998339+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:40:13.958756+0000","last_clean_scrub_stamp":"2023-12-19T14:40:13.958756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:35:02.237361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-19T14:40:15.992789+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:40:15.992789+0000","last_peered":"2023-12-19T14:40:15.992789+0000","last_clean":"2023-12-19T14:40:15.992789+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:40:15.992789+0000","last_undegraded":"2023-12-19T14:40:15.992789+0000","last_fullsized":"2023-12-19T14:40:15.992789+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298312,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073528,"statfs":{"total":96636764160,"available":96331292672,"internally_reserved":0,"allocated":1048576,"data_stored":779986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.155,"15min":1.155},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.381},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.151}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":1.2889999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":1.054,"5min":1.054,"15min":1.054},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.343},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":1.4039999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.2629999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.222},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":1.206,"5min":1.206,"15min":1.206},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.359}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.4590000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":188566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.288}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.155},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":3.0990000000000002},"last":1.0900000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.431},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.3280000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":1.395},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":3.194,"5min":3.194,"15min":3.194},"last":1.357}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":3.0150000000000001,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":1.024},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":1.4710000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.228}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":3.367,"5min":3.367,"15min":3.367},"last":1.194},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"min":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"max":{"1min":3.2839999999999998,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.2529999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":188124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.385}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.1559999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.456},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.488}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.052}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.1970000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.4790000000000001,"5min":2.4790000000000001,"15min":2.4790000000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.24}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.254}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":188588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.113}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.375},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.2709999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.325}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.232},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.0629999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.94099999999999995}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.742,"5min":2.742,"15min":2.742},"last":1.415}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":5.2060000000000004,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":1.52}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301828,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070012,"statfs":{"total":96636764160,"available":96327692288,"internally_reserved":0,"allocated":454656,"data_stored":189582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":4.476,"5min":4.476,"15min":4.476},"last":0.79600000000000004}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.6989999999999998,"5min":2.6989999999999998,"15min":2.6989999999999998},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.47}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.46,"5min":3.46,"15min":3.46},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.1579999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.087},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.2909999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"min":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"max":{"1min":4.5999999999999996,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.216}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.512},{"interface":"front","average":{"1min":1.867,"5min":1.867,"15min":1.867},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.556}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301812,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070028,"statfs":{"total":96636764160,"available":96327708672,"internally_reserved":0,"allocated":438272,"data_stored":187071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":1.095},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.3260000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.2689999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.1910000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.546,"15min":1.546},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":3.4950000000000001,"5min":3.4950000000000001,"15min":3.4950000000000001},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.5129999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.181},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":2.077}]},{"osd":6,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":3.7309999999999999,"5min":3.7309999999999999,"15min":3.7309999999999999},"last":1.859}]},{"osd":7,"last update":"Tue Dec 19 14:39:55 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":2.2679999999999998},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":1.905}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301816,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070024,"statfs":{"total":96636764160,"available":96327704576,"internally_reserved":0,"allocated":442368,"data_stored":188102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.802}]},{"osd":2,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":3.0619999999999998,"5min":3.0619999999999998,"15min":3.0619999999999998},"last":1.276}]},{"osd":3,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.681},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.1779999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.387},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":1.474}]},{"osd":5,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.7689999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"max":{"1min":2.4620000000000002,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.627,"15min":1.627},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":1.643}]},{"osd":7,"last update":"Tue Dec 19 14:39:59 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":1.897},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.8520000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302392,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069448,"statfs":{"total":96636764160,"available":96327114752,"internally_reserved":0,"allocated":1032192,"data_stored":777403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.548}]},{"osd":3,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.224},{"interface":"front","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.581}]},{"osd":4,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.105},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":1.0409999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.423},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.397}]},{"osd":6,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.504,"15min":1.504},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.359},{"interface":"front","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.077,"5min":1.077,"15min":1.077},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.514}]},{"osd":7,"last update":"Tue Dec 19 14:39:57 2023","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.482},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.25,"5min":1.25,"15min":1.25},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.4490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:40:25.530 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:40:25.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:40:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.735+0000 7fa919625700 1 Processor -- start 2023-12-19T14:40:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.735+0000 7fa919625700 1 -- start start 2023-12-19T14:40:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa914159790 0x7fa9140684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9140b0cb0 0x7fa9140b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9140b1d00 0x7fa914159b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9140b2160 con 0x7fa9140b0cb0 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa914067790 con 0x7fa914159790 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa919625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa914068030 con 0x7fa9140b1d00 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9140b0cb0 0x7fa9140b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9140b0cb0 0x7fa9140b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39782/0 (socket says 172.21.15.50:39782) 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 -- 172.21.15.50:0/657443845 learned_addr learned my addr 172.21.15.50:0/657443845 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa912ffd700 1 --2- 172.21.15.50:0/657443845 >> v2:172.21.15.125:3300/0 conn(0x7fa914159790 0x7fa9140684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9137fe700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9140b1d00 0x7fa914159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 -- 172.21.15.50:0/657443845 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9140b1d00 msgr2=0x7fa914159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:25.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9140b1d00 0x7fa914159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 -- 172.21.15.50:0/657443845 >> v2:172.21.15.125:3300/0 conn(0x7fa914159790 msgr2=0x7fa9140684c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 --2- 172.21.15.50:0/657443845 >> v2:172.21.15.125:3300/0 conn(0x7fa914159790 0x7fa9140684c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.739+0000 7fa9127fc700 1 -- 172.21.15.50:0/657443845 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140b1200 con 0x7fa9140b0cb0 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa9127fc700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b0cb0 0x7fa9140b17c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b09f0 tx=0x7fa908009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a855dd1299118fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa911ffb700 1 -- 172.21.15.50:0/657443845 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908009e70 con 0x7fa9140b0cb0 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa911ffb700 1 -- 172.21.15.50:0/657443845 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90800b860 con 0x7fa9140b0cb0 2023-12-19T14:40:25.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa911ffb700 1 -- 172.21.15.50:0/657443845 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90800bc00 con 0x7fa9140b0cb0 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 -- 172.21.15.50:0/657443845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b0cb0 msgr2=0x7fa9140b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b0cb0 0x7fa9140b17c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b09f0 tx=0x7fa908009250 comp rx=0 tx=0).stop 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 -- 172.21.15.50:0/657443845 shutdown_connections 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b0cb0 0x7fa9140b17c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 --2- 172.21.15.50:0/657443845 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9140b1d00 0x7fa914159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 --2- 172.21.15.50:0/657443845 >> v2:172.21.15.125:3300/0 conn(0x7fa914159790 0x7fa9140684c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 -- 172.21.15.50:0/657443845 >> 172.21.15.50:0/657443845 conn(0x7fa9140cdef0 msgr2=0x7fa9140ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 -- 172.21.15.50:0/657443845 shutdown_connections 2023-12-19T14:40:25.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.743+0000 7fa919625700 1 -- 172.21.15.50:0/657443845 wait complete. 2023-12-19T14:40:25.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 Processor -- start 2023-12-19T14:40:25.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 -- start start 2023-12-19T14:40:25.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9140b0cb0 0x7fa914140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa914159790 0x7fa9140c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140ca930 con 0x7fa9140b1d00 2023-12-19T14:40:25.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c9a40 con 0x7fa9140b0cb0 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa919625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c9d20 con 0x7fa914159790 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39792/0 (socket says 172.21.15.50:39792) 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 -- 172.21.15.50:0/557012694 learned_addr learned my addr 172.21.15.50:0/557012694 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa912ffd700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9140b0cb0 0x7fa914140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9137fe700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa914159790 0x7fa9140c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 -- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa914159790 msgr2=0x7fa9140c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa914159790 0x7fa9140c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 -- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9140b0cb0 msgr2=0x7fa914140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9140b0cb0 0x7fa914140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.747+0000 7fa9127fc700 1 -- 172.21.15.50:0/557012694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080094f0 con 0x7fa9140b1d00 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa9127fc700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa91406bce0 tx=0x7fa908017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90800bbe0 con 0x7fa9140b1d00 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa908000c90 con 0x7fa9140b1d00 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9140ca000 con 0x7fa9140b1d00 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908011650 con 0x7fa9140b1d00 2023-12-19T14:40:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140ca5c0 con 0x7fa9140b1d00 2023-12-19T14:40:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.751+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9140caae0 con 0x7fa9140b1d00 2023-12-19T14:40:25.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.755+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa908011a80 con 0x7fa9140b1d00 2023-12-19T14:40:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.755+0000 7fa8faffd700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 0x7fa900062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.755+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fa90807de10 con 0x7fa9140b1d00 2023-12-19T14:40:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.755+0000 7fa912ffd700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 0x7fa900062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:25.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.759+0000 7fa912ffd700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 0x7fa900062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa904009d10 tx=0x7fa9040091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.763+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa9140caae0 con 0x7fa9140b1d00 2023-12-19T14:40:25.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:25.995+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa914071120 con 0x7fa9140b1d00 2023-12-19T14:40:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.003+0000 7fa8faffd700 1 -- 172.21.15.50:0/557012694 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa908081030 con 0x7fa9140b1d00 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 msgr2=0x7fa900062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 0x7fa900062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa904009d10 tx=0x7fa9040091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 msgr2=0x7fa9140c8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa91406bce0 tx=0x7fa908017000 comp rx=0 tx=0).stop 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 shutdown_connections 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa900060550 0x7fa900062a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa914159790 0x7fa9140c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9140b1d00 0x7fa9140c8890 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 --2- 172.21.15.50:0/557012694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9140b0cb0 0x7fa914140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 >> 172.21.15.50:0/557012694 conn(0x7fa9140cdef0 msgr2=0x7fa914155040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:26.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 shutdown_connections 2023-12-19T14:40:26.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.007+0000 7fa919625700 1 -- 172.21.15.50:0/557012694 wait complete. 2023-12-19T14:40:26.013 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:40:26.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:40:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.199+0000 7efec32ae700 1 Processor -- start 2023-12-19T14:40:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.199+0000 7efec32ae700 1 -- start start 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 0x7efebc0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0c2980 0x7efebc0c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efebc0c1a80 0x7efebc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efebc06c9f0 con 0x7efebc0c1a80 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efebc0c7f20 con 0x7efebc13f890 2023-12-19T14:40:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec32ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c8190 con 0x7efebc0c2980 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efebc0c1a80 0x7efebc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efebc0c1a80 0x7efebc0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39798/0 (socket says 172.21.15.50:39798) 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 -- 172.21.15.50:0/1553701603 learned_addr learned my addr 172.21.15.50:0/1553701603 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec104a700 1 --2- 172.21.15.50:0/1553701603 >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 0x7efebc0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec0849700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0c2980 0x7efebc0c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 -- 172.21.15.50:0/1553701603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0c2980 msgr2=0x7efebc0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0c2980 0x7efebc0c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 -- 172.21.15.50:0/1553701603 >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 msgr2=0x7efebc0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 --2- 172.21.15.50:0/1553701603 >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 0x7efebc0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 -- 172.21.15.50:0/1553701603 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc0c8470 con 0x7efebc0c1a80 2023-12-19T14:40:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec104a700 1 --2- 172.21.15.50:0/1553701603 >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 0x7efebc0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efec184b700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc0c1a80 0x7efebc0680f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efebc068c90 tx=0x7efeb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cfa117af770005 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efeb3fff700 1 -- 172.21.15.50:0/1553701603 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb400e070 con 0x7efebc0c1a80 2023-12-19T14:40:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efeb3fff700 1 -- 172.21.15.50:0/1553701603 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb400ba50 con 0x7efebc0c1a80 2023-12-19T14:40:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.203+0000 7efeb3fff700 1 -- 172.21.15.50:0/1553701603 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb40092e0 con 0x7efebc0c1a80 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- 172.21.15.50:0/1553701603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc0c1a80 msgr2=0x7efebc0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc0c1a80 0x7efebc0680f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efebc068c90 tx=0x7efeb4009840 comp rx=0 tx=0).stop 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- 172.21.15.50:0/1553701603 shutdown_connections 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc0c1a80 0x7efebc0680f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 --2- 172.21.15.50:0/1553701603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0c2980 0x7efebc0c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 --2- 172.21.15.50:0/1553701603 >> v2:172.21.15.125:3300/0 conn(0x7efebc13f890 0x7efebc0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- 172.21.15.50:0/1553701603 >> 172.21.15.50:0/1553701603 conn(0x7efebc0c8850 msgr2=0x7efebc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- 172.21.15.50:0/1553701603 shutdown_connections 2023-12-19T14:40:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- 172.21.15.50:0/1553701603 wait complete. 2023-12-19T14:40:26.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 Processor -- start 2023-12-19T14:40:26.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.207+0000 7efec32ae700 1 -- start start 2023-12-19T14:40:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efebc0c2980 0x7efebc15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0a4cd0 0x7efebc067d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc15f5c0 con 0x7efebc13f890 2023-12-19T14:40:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c4710 con 0x7efebc0c2980 2023-12-19T14:40:26.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c49f0 con 0x7efebc0a4cd0 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39804/0 (socket says 172.21.15.50:39804) 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 -- 172.21.15.50:0/1136677505 learned_addr learned my addr 172.21.15.50:0/1136677505 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec104a700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efebc0c2980 0x7efebc15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec184b700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0a4cd0 0x7efebc067d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 -- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0a4cd0 msgr2=0x7efebc067d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0a4cd0 0x7efebc067d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 -- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efebc0c2980 msgr2=0x7efebc15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efebc0c2980 0x7efebc15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 -- 172.21.15.50:0/1136677505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4009ae0 con 0x7efebc13f890 2023-12-19T14:40:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec0849700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efeac00d860 tx=0x7efeac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac0125f0 con 0x7efebc13f890 2023-12-19T14:40:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac012750 con 0x7efebc13f890 2023-12-19T14:40:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc0c4cd0 con 0x7efebc13f890 2023-12-19T14:40:26.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac012af0 con 0x7efebc13f890 2023-12-19T14:40:26.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.211+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc0c5290 con 0x7efebc13f890 2023-12-19T14:40:26.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.215+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc071120 con 0x7efebc13f890 2023-12-19T14:40:26.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.215+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efeac013480 con 0x7efebc13f890 2023-12-19T14:40:26.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.219+0000 7efeb17fa700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 0x7efea0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:26.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.219+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7efeac07cd30 con 0x7efebc13f890 2023-12-19T14:40:26.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.219+0000 7efec104a700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 0x7efea0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:26.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.219+0000 7efec104a700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 0x7efea0062a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006f00 tx=0x7efeb8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:26.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.223+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efeac04d140 con 0x7efebc13f890 2023-12-19T14:40:26.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.427+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efebc0642a0 con 0x7efebc13f890 2023-12-19T14:40:26.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.431+0000 7efeb17fa700 1 -- 172.21.15.50:0/1136677505 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7efeac050760 con 0x7efebc13f890 2023-12-19T14:40:26.435 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:40:26.435 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:40:26.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.431+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 msgr2=0x7efea0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.431+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 0x7efea0062a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006f00 tx=0x7efeb8008040 comp rx=0 tx=0).stop 2023-12-19T14:40:26.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.431+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 msgr2=0x7efebc0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:26.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.431+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efeac00d860 tx=0x7efeac008040 comp rx=0 tx=0).stop 2023-12-19T14:40:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 shutdown_connections 2023-12-19T14:40:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efea0060550 0x7efea0062a00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efebc0a4cd0 0x7efebc067d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efebc13f890 0x7efebc0a4790 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 --2- 172.21.15.50:0/1136677505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efebc0c2980 0x7efebc15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 >> 172.21.15.50:0/1136677505 conn(0x7efebc0c8850 msgr2=0x7efebc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 shutdown_connections 2023-12-19T14:40:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:26.435+0000 7efec32ae700 1 -- 172.21.15.50:0/1136677505 wait complete. 2023-12-19T14:40:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:40:26.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:40:15.420564+0000', 'modified': '2023-12-19T14:40:19.019982+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5091': {'gid': 5091, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.50:6835/906596857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 906596857}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 906596857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:40:26.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:40:26.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T14:40:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.023+0000 7f3d0e216700 1 Processor -- start 2023-12-19T14:40:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.023+0000 7f3d0e216700 1 -- start start 2023-12-19T14:40:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.027+0000 7f3d0e216700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d0806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.027+0000 7f3d0e216700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d080ce810 0x7f3d080ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.027+0000 7f3d0e216700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0806b730 con 0x7f3d081591f0 2023-12-19T14:40:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.027+0000 7f3d0e216700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0806ab20 con 0x7f3d080ce810 2023-12-19T14:40:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.027+0000 7f3d0e216700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0806ae00 con 0x7f3d08158e20 2023-12-19T14:40:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0ca13700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3d081591f0 0x7f3d080ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33390/0 (socket says 172.21.15.50:33390) 2023-12-19T14:40:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0ca13700 1 -- 172.21.15.50:0/3891839790 learned_addr learned my addr 172.21.15.50:0/3891839790 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3891839790 >> v2:172.21.15.50:3300/0 conn(0x7f3d080ce810 0x7f3d080ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0d214700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d0806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3cfffff700 1 -- 172.21.15.50:0/3891839790 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852896918 0 0) 0x7f3d0806b730 con 0x7f3d081591f0 2023-12-19T14:40:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 -- 172.21.15.50:0/3891839790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 msgr2=0x7f3d0806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d0806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 -- 172.21.15.50:0/3891839790 >> v1:172.21.15.125:6789/0 conn(0x7f3d081591f0 legacy=0x7f3d080ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 -- 172.21.15.50:0/3891839790 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0806b0e0 con 0x7f3d080ce810 2023-12-19T14:40:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d080ca950 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009e60 tx=0x7f3d00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba3fe89e8aba3ad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3cfffff700 1 -- 172.21.15.50:0/3891839790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000e070 con 0x7f3d080ce810 2023-12-19T14:40:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3cfffff700 1 -- 172.21.15.50:0/3891839790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0000bc30 con 0x7f3d080ce810 2023-12-19T14:40:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3cfffff700 1 -- 172.21.15.50:0/3891839790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00012430 con 0x7f3d080ce810 2023-12-19T14:40:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 -- 172.21.15.50:0/3891839790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 msgr2=0x7f3d080ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d080ca950 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009e60 tx=0x7f3d00009780 comp rx=0 tx=0).stop 2023-12-19T14:40:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 -- 172.21.15.50:0/3891839790 shutdown_connections 2023-12-19T14:40:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d080ca950 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 --2- 172.21.15.50:0/3891839790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d0806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 -- 172.21.15.50:0/3891839790 >> 172.21.15.50:0/3891839790 conn(0x7f3d08067bb0 msgr2=0x7f3d080cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 -- 172.21.15.50:0/3891839790 shutdown_connections 2023-12-19T14:40:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.031+0000 7f3d0e216700 1 -- 172.21.15.50:0/3891839790 wait complete. 2023-12-19T14:40:27.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 Processor -- start 2023-12-19T14:40:27.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 -- start start 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d08150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d081591f0 0x7f3d0815db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c9040 con 0x7f3d080ce810 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c7660 con 0x7f3d081591f0 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0e216700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c8d60 con 0x7f3d08158e20 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.035+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54138/0 (socket says 172.21.15.50:54138) 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 -- 172.21.15.50:0/2454019571 learned_addr learned my addr 172.21.15.50:0/2454019571 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0da15700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d081591f0 0x7f3d0815db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d08150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 -- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 msgr2=0x7f3d08150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d08150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 -- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d081591f0 msgr2=0x7f3d0815db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d081591f0 0x7f3d0815db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 -- 172.21.15.50:0/2454019571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00009a20 con 0x7f3d080ce810 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0d214700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d890 tx=0x7f3d04008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3cfd7fa700 1 -- 172.21.15.50:0/2454019571 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d040125f0 con 0x7f3d080ce810 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3cfd7fa700 1 -- 172.21.15.50:0/2454019571 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d04012750 con 0x7f3d080ce810 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3cfd7fa700 1 -- 172.21.15.50:0/2454019571 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04012af0 con 0x7f3d080ce810 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d080c7010 con 0x7f3d080ce810 2023-12-19T14:40:27.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.039+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d080c9470 con 0x7f3d080ce810 2023-12-19T14:40:27.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.043+0000 7f3cfd7fa700 1 -- 172.21.15.50:0/2454019571 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3d0401e460 con 0x7f3d080ce810 2023-12-19T14:40:27.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.043+0000 7f3cfd7fa700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 0x7f3cec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.043+0000 7f3cfd7fa700 1 -- 172.21.15.50:0/2454019571 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f3d0407cf40 con 0x7f3d080ce810 2023-12-19T14:40:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 Processor -- start 2023-12-19T14:40:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 0x7f3cec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 -- start start 2023-12-19T14:40:27.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 0x7f3cec062a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b2a30 tx=0x7f3cf8007f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16d7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 0x16d7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1604f30 con 0x1459ae0 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1600ed0 con 0x1459710 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x15c70b0 con 0x14a4790 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54152/0 (socket says 172.21.15.50:54152) 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 -- 172.21.15.50:0/1840625838 learned_addr learned my addr 172.21.15.50:0/1840625838 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0da15700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 0x16d7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 -- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 msgr2=0x16d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 0x16d7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 -- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 msgr2=0x16d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16d7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 -- 172.21.15.50:0/1840625838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x14a59e0 con 0x1459ae0 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0da15700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 0x16d7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.047+0000 7f3d0d214700 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3d0407b280 tx=0x7f3d0407b1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164761a6f79b07c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3cf27fc700 1 -- 172.21.15.50:0/1840625838 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04058070 con 0x1459ae0 2023-12-19T14:40:27.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3cf27fc700 1 -- 172.21.15.50:0/1840625838 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d04051010 con 0x1459ae0 2023-12-19T14:40:27.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3cf27fc700 1 -- 172.21.15.50:0/1840625838 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0404f150 con 0x1459ae0 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- 172.21.15.50:0/1840625838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 msgr2=0x16d6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3d0407b280 tx=0x7f3d0407b1e0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- 172.21.15.50:0/1840625838 shutdown_connections 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x14a4790 0x16d7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 --2- 172.21.15.50:0/1840625838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459ae0 0x16d6f00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- 172.21.15.50:0/1840625838 >> 172.21.15.50:0/1840625838 conn(0x1459eb0 msgr2=0x1693930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- 172.21.15.50:0/1840625838 shutdown_connections 2023-12-19T14:40:27.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- 172.21.15.50:0/1840625838 wait complete. 2023-12-19T14:40:27.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 -- start start 2023-12-19T14:40:27.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.051+0000 7f3d11776740 1 Processor -- start 2023-12-19T14:40:27.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16dba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x14a4790 0x16bccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x15c6d50 con 0x1459710 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x159f970 con 0x14a4790 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x16006e0 con 0x1459ae0 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 10 client.0 _ll_register_callbacks cb 0x164a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54166/0 (socket says 172.21.15.50:54166) 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 -- 172.21.15.50:0/61016885 learned_addr learned my addr 172.21.15.50:0/61016885 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0da15700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x14a4790 0x16bccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 -- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 msgr2=0x16dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 -- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x14a4790 msgr2=0x16bccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x14a4790 0x16bccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 -- 172.21.15.50:0/61016885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d080c9470 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0d214700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d860 tx=0x7f3d0406a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51babca32e834a8c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04012e60 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f3d0406cc50 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x159fcc0 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d040512b0 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 10 client.5307 fetch_fsmap learned FSMap version 17 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x15cf370 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x159fcc0 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f3d04050500 con 0x1459710 2023-12-19T14:40:27.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 10 client.5307 fetch_fsmap finished waiting for FSMap version 17 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 10 client.5307 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 20 client.5307 populate_metadata read hostname 'smithi050' 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x16be3d0 0x16e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 4 client.5307 mds_command: new command op to 5091 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 --> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x15a33e0 con 0x16be3d0 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.055+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x16be3d0 0x16e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x16be3d0 0x16e66b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0ecd4fb3575ba68 server_cookie=491750380ecb29c2 in_seq=0 out_seq=0 2023-12-19T14:40:27.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3ce77fe700 10 client.5307 ms_handle_connect on v2:172.21.15.50:6834/906596857 2023-12-19T14:40:27.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3ce77fe700 1 -- 172.21.15.50:0/61016885 <== mds.0 v2:172.21.15.50:6834/906596857 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x15a33e0 con 0x16be3d0 2023-12-19T14:40:27.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3ce77fe700 10 client.5307 handle_command_reply: tid=0 2023-12-19T14:40:27.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 client.5307 shutdown 2023-12-19T14:40:27.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 msgr2=0x16db510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d860 tx=0x7f3d0406a000 comp rx=0 tx=0).stop 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 shutdown_connections 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x16be3d0 0x16e66b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x14a4790 0x16bccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 --2- 172.21.15.50:0/61016885 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1459710 0x16db510 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3ce77fe700 0 client.5307 ms_handle_reset on v2:172.21.15.50:6834/906596857 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 >> 172.21.15.50:0/61016885 conn(0x1459eb0 msgr2=0x168bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 shutdown_connections 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 1 -- 172.21.15.50:0/61016885 wait complete. 2023-12-19T14:40:27.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.059+0000 7f3d11776740 20 client.5307 trim_cache size 0 max 16384 2023-12-19T14:40:27.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.079+0000 7f3d11776740 1 Processor -- start 2023-12-19T14:40:27.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.079+0000 7f3d11776740 1 -- start start 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x1691b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x14d27c0 con 0x14a4790 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x15fe890 con 0x1459710 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x16049d0 con 0x1459ae0 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54170/0 (socket says 172.21.15.50:54170) 2023-12-19T14:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 -- 172.21.15.50:0/3304014506 learned_addr learned my addr 172.21.15.50:0/3304014506 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0d214700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x1691b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 -- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 msgr2=0x1691b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x1691b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 -- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 msgr2=0x16915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 -- 172.21.15.50:0/3304014506 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x15b5d80 con 0x14a4790 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0d214700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x1691b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:27.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3d0da15700 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3d00013000 tx=0x7f3d00019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba76c71d2e3c737c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.083+0000 7f3cf2ffd700 1 -- 172.21.15.50:0/3304014506 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00024040 con 0x14a4790 2023-12-19T14:40:27.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3cf2ffd700 1 -- 172.21.15.50:0/3304014506 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00012ab0 con 0x14a4790 2023-12-19T14:40:27.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3cf2ffd700 1 -- 172.21.15.50:0/3304014506 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00012e30 con 0x14a4790 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- 172.21.15.50:0/3304014506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 msgr2=0x1692050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3d00013000 tx=0x7f3d00019000 comp rx=0 tx=0).stop 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- 172.21.15.50:0/3304014506 shutdown_connections 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x1692050 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x1691b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 --2- 172.21.15.50:0/3304014506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- 172.21.15.50:0/3304014506 >> 172.21.15.50:0/3304014506 conn(0x1459eb0 msgr2=0x168bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- 172.21.15.50:0/3304014506 shutdown_connections 2023-12-19T14:40:27.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- 172.21.15.50:0/3304014506 wait complete. 2023-12-19T14:40:27.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 -- start start 2023-12-19T14:40:27.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.087+0000 7f3d11776740 1 Processor -- start 2023-12-19T14:40:27.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16b7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1650480 con 0x14a4790 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x14a4ea0 con 0x1459710 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x15c8be0 con 0x1459ae0 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 10 client.0 _ll_register_callbacks cb 0x14259d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54172/0 (socket says 172.21.15.50:54172) 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 -- 172.21.15.50:0/115870256 learned_addr learned my addr 172.21.15.50:0/115870256 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0ca13700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16b7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0d214700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 -- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 msgr2=0x16b7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16b7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 -- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 msgr2=0x16b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 -- 172.21.15.50:0/115870256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0806b0e0 con 0x14a4790 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d0da15700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009e60 tx=0x7f3d00021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf66b590829cd280 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3ce77fe700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00024040 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f3d00029860 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.091+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d04051010 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d00029c70 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 10 client.5313 fetch_fsmap learned FSMap version 17 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x15aa130 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d04051010 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f3d0001e620 con 0x14a4790 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 10 client.5313 fetch_fsmap finished waiting for FSMap version 17 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 10 client.5313 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:40:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 20 client.5313 populate_metadata read hostname 'smithi050' 2023-12-19T14:40:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x1736d30 0x1739110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 4 client.5313 mds_command: new command op to 5091 tid=0[{"prefix": "status"}] 2023-12-19T14:40:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 --> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] -- command(tid 0: {"prefix": "status"}) v1 -- 0x15d5410 con 0x1736d30 2023-12-19T14:40:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d0d214700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x1736d30 0x1739110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d0d214700 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x1736d30 0x1739110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=684b1c9b955b8aac server_cookie=6f5a575a6ad24a20 in_seq=0 out_seq=0 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 10 client.5313 ms_handle_connect on v2:172.21.15.50:6834/906596857 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 1 -- 172.21.15.50:0/115870256 <== mds.0 v2:172.21.15.50:6834/906596857 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x15d5410 con 0x1736d30 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 10 client.5313 handle_command_reply: tid=0 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 client.5313 shutdown 2023-12-19T14:40:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 msgr2=0x16b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009e60 tx=0x7f3d00021070 comp rx=0 tx=0).stop 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 shutdown_connections 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:6834/906596857,v1:172.21.15.50:6835/906596857] conn(0x1736d30 0x1739110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x14a4790 0x16b81b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1459ae0 0x16b7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3ce77fe700 0 client.5313 ms_handle_reset on v2:172.21.15.50:6834/906596857 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 --2- 172.21.15.50:0/115870256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1459710 0x16b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 >> 172.21.15.50:0/115870256 conn(0x1459eb0 msgr2=0x168c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 shutdown_connections 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 1 -- 172.21.15.50:0/115870256 wait complete. 2023-12-19T14:40:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.095+0000 7f3d11776740 20 client.5313 trim_cache size 0 max 16384 2023-12-19T14:40:27.103 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 5091, 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:40:27.104 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 11.117321992000001, 2023-12-19T14:40:27.105 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 17, 2023-12-19T14:40:27.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 38, 2023-12-19T14:40:27.105 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 38, 2023-12-19T14:40:27.105 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 19.157108507 2023-12-19T14:40:27.105 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 msgr2=0x7f3cec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 0x7f3cec062a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b2a30 tx=0x7f3cf8007f50 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 msgr2=0x7f3d08157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d890 tx=0x7f3d04008040 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 shutdown_connections 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d081591f0 0x7f3d0815db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3cec060550 0x7f3cec062a00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d08158e20 0x7f3d08150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 --2- 172.21.15.50:0/2454019571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d080ce810 0x7f3d08157b30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 >> 172.21.15.50:0/2454019571 conn(0x7f3d08067bb0 msgr2=0x7f3d080c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 shutdown_connections 2023-12-19T14:40:27.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.099+0000 7f3d0e216700 1 -- 172.21.15.50:0/2454019571 wait complete. 2023-12-19T14:40:27.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:40:27.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.283+0000 7f73e0078700 1 Processor -- start 2023-12-19T14:40:27.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.283+0000 7f73e0078700 1 -- start start 2023-12-19T14:40:27.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73e0078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73e0078700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73d8159340 0x7f73d80cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73e0078700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8068c70 con 0x7f73d8158f70 2023-12-19T14:40:27.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73e0078700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d8068060 con 0x7f73d8159340 2023-12-19T14:40:27.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73e0078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d8068340 con 0x7f73d8069100 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53752/0 (socket says 172.21.15.50:53752) 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 -- 172.21.15.50:0/2779712646 learned_addr learned my addr 172.21.15.50:0/2779712646 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73de615700 1 --2- 172.21.15.50:0/2779712646 >> v2:172.21.15.125:3300/0 conn(0x7f73d8159340 0x7f73d80cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 -- 172.21.15.50:0/2779712646 >> v2:172.21.15.125:3300/0 conn(0x7f73d8159340 msgr2=0x7f73d80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 --2- 172.21.15.50:0/2779712646 >> v2:172.21.15.125:3300/0 conn(0x7f73d8159340 0x7f73d80cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 -- 172.21.15.50:0/2779712646 >> v1:172.21.15.125:6789/0 conn(0x7f73d8158f70 legacy=0x7f73d80cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.287+0000 7f73dd613700 1 -- 172.21.15.50:0/2779712646 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8068620 con 0x7f73d8069100 2023-12-19T14:40:27.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73dd613700 1 --2- 172.21.15.50:0/2779712646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f73c80099b0 tx=0x7f73c8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753d520a44b6d70a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73dce12700 1 -- 172.21.15.50:0/2779712646 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8009ea0 con 0x7f73d8069100 2023-12-19T14:40:27.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73dce12700 1 -- 172.21.15.50:0/2779712646 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800b860 con 0x7f73d8069100 2023-12-19T14:40:27.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73dce12700 1 -- 172.21.15.50:0/2779712646 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800bc00 con 0x7f73d8069100 2023-12-19T14:40:27.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 -- 172.21.15.50:0/2779712646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 msgr2=0x7f73d8067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 --2- 172.21.15.50:0/2779712646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f73c80099b0 tx=0x7f73c8009250 comp rx=0 tx=0).stop 2023-12-19T14:40:27.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 -- 172.21.15.50:0/2779712646 shutdown_connections 2023-12-19T14:40:27.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 --2- 172.21.15.50:0/2779712646 >> v2:172.21.15.125:3300/0 conn(0x7f73d8159340 0x7f73d80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 --2- 172.21.15.50:0/2779712646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d8067980 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 -- 172.21.15.50:0/2779712646 >> 172.21.15.50:0/2779712646 conn(0x7f73d80ccc40 msgr2=0x7f73d80cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 -- 172.21.15.50:0/2779712646 shutdown_connections 2023-12-19T14:40:27.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.291+0000 7f73e0078700 1 -- 172.21.15.50:0/2779712646 wait complete. 2023-12-19T14:40:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 Processor -- start 2023-12-19T14:40:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 -- start start 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d80b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 0x7f73d814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d816dc40 con 0x7f73d8159340 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d816d030 con 0x7f73d8158f70 2023-12-19T14:40:27.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73e0078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d816d310 con 0x7f73d8069100 2023-12-19T14:40:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.295+0000 7f73de615700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54178/0 (socket says 172.21.15.50:54178) 2023-12-19T14:40:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 -- 172.21.15.50:0/480225403 learned_addr learned my addr 172.21.15.50:0/480225403 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73dd613700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 0x7f73d814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73dde14700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d80b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 -- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 msgr2=0x7f73d80b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d80b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 -- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 msgr2=0x7f73d814dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 0x7f73d814dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 -- 172.21.15.50:0/480225403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80094f0 con 0x7f73d8159340 2023-12-19T14:40:27.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73dd613700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 0x7f73d814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73dde14700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d80b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73de615700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009df0 tx=0x7f73d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000caa0 con 0x7f73d8159340 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d000f040 con 0x7f73d8159340 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000ce70 con 0x7f73d8159340 2023-12-19T14:40:27.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d816d5f0 con 0x7f73d8159340 2023-12-19T14:40:27.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.299+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d80b1740 con 0x7f73d8159340 2023-12-19T14:40:27.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.303+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d816ddb0 con 0x7f73d8159340 2023-12-19T14:40:27.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.303+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f73d0014690 con 0x7f73d8159340 2023-12-19T14:40:27.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.307+0000 7f73cdffb700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 0x7f73bc062ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:27.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.307+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f73d007d140 con 0x7f73d8159340 2023-12-19T14:40:27.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.307+0000 7f73dde14700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 0x7f73bc062ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:27.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.307+0000 7f73dde14700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 0x7f73bc062ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f73d4001d30 tx=0x7f73d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:27.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.311+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f73d0050fe0 con 0x7f73d8159340 2023-12-19T14:40:27.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.511+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f73d8071120 con 0x7f73d8159340 2023-12-19T14:40:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.511+0000 7f73cdffb700 1 -- 172.21.15.50:0/480225403 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f73d0050b70 con 0x7f73d8159340 2023-12-19T14:40:27.517 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:40:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 msgr2=0x7f73bc062ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 0x7f73bc062ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f73d4001d30 tx=0x7f73d4009040 comp rx=0 tx=0).stop 2023-12-19T14:40:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 msgr2=0x7f73d80b1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:27.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009df0 tx=0x7f73d000b040 comp rx=0 tx=0).stop 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 shutdown_connections 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d8158f70 0x7f73d814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73bc060600 0x7f73bc062ab0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d8159340 0x7f73d80b1f20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 --2- 172.21.15.50:0/480225403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d8069100 0x7f73d80b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 >> 172.21.15.50:0/480225403 conn(0x7f73d80ccc40 msgr2=0x7f73d81540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 shutdown_connections 2023-12-19T14:40:27.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:27.519+0000 7f73e0078700 1 -- 172.21.15.50:0/480225403 wait complete. 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:40:27.544 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:40:27.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.545 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:40:27.564 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 42322sec preferred_lft 42322sec 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:40:27.565 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.566 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:40:27.567 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:40:27.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.568 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:40:27.648 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:40:27.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.651 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:40:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:40:27.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:40:27.720 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:40:27.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.853 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:40:27.871 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:40:27.871 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:40:27.872 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:40:27.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:40:27.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:40:27.954 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:40:27.955 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:40:27.955 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:28.015 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:40:28.016 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:40:28.016 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:28.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:40:28.157 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:40:28.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:40:28.231 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:40:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:40:28.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:28.234 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:40:28.296 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:40:28.296 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:40:28.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:40:28.381 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:40:28.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:40:28.463 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:40:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:40:28.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:28.465 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:40:28.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:40:28.479+0000 7f175a7c9380 -1 init, newargv = 0x55f3d2a23f30 newargc=15 2023-12-19T14:40:28.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[42574]: starting ceph client 2023-12-19T14:40:28.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[42574]: starting fuse 2023-12-19T14:40:28.500 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:40:28.500 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:40:28.501 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:40:28.502 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:40:28.672 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.42574.asok 2023-12-19T14:40:28.855 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42574.asok 2023-12-19T14:40:28.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:28.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42574.asok status 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "42574", 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:29.043 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 5215, 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 5215 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2085298308 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:40:29.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2085298308 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.5215 172.21.15.50:0/2085298308", 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "172.21.15.50:0/2085298308", 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 17, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 38, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:40:29.045 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:40:29.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:29.057 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:40:29.057 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:29.079 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:40:29.079 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:29.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:29.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:40:29.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:40:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 Processor -- start 2023-12-19T14:40:29.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 -- start start 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4a8159220 0x7fb4a806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb4a80a98d0 0x7fb4a806ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a806e160 con 0x7fb4a80a98d0 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a806d0c0 con 0x7fb4a8159220 2023-12-19T14:40:29.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4aecae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a806d3a0 con 0x7fb4a8158e50 2023-12-19T14:40:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.363+0000 7fb4ad24b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb4a80a98d0 0x7fb4a806ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb4a80a98d0 0x7fb4a806ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54192/0 (socket says 172.21.15.50:54192) 2023-12-19T14:40:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 -- 172.21.15.50:0/376004972 learned_addr learned my addr 172.21.15.50:0/376004972 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4a7fff700 1 --2- 172.21.15.50:0/376004972 >> v2:172.21.15.125:3300/0 conn(0x7fb4a8159220 0x7fb4a806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aca4a700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 -- 172.21.15.50:0/376004972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 msgr2=0x7fb4a806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 -- 172.21.15.50:0/376004972 >> v2:172.21.15.125:3300/0 conn(0x7fb4a8159220 msgr2=0x7fb4a806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 --2- 172.21.15.50:0/376004972 >> v2:172.21.15.125:3300/0 conn(0x7fb4a8159220 0x7fb4a806a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 -- 172.21.15.50:0/376004972 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a806d680 con 0x7fb4a80a98d0 2023-12-19T14:40:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4ad24b700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a806ac80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4a806a370 tx=0x7fb4a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb19115469200d08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4a77fe700 1 -- 172.21.15.50:0/376004972 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000e070 con 0x7fb4a80a98d0 2023-12-19T14:40:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4a77fe700 1 -- 172.21.15.50:0/376004972 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a000ba00 con 0x7fb4a80a98d0 2023-12-19T14:40:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4a77fe700 1 -- 172.21.15.50:0/376004972 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00092e0 con 0x7fb4a80a98d0 2023-12-19T14:40:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 -- 172.21.15.50:0/376004972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 msgr2=0x7fb4a806ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a806ac80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4a806a370 tx=0x7fb4a0009840 comp rx=0 tx=0).stop 2023-12-19T14:40:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 -- 172.21.15.50:0/376004972 shutdown_connections 2023-12-19T14:40:29.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a806ac80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 --2- 172.21.15.50:0/376004972 >> v2:172.21.15.125:3300/0 conn(0x7fb4a8159220 0x7fb4a806a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 --2- 172.21.15.50:0/376004972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 -- 172.21.15.50:0/376004972 >> 172.21.15.50:0/376004972 conn(0x7fb4a8067b60 msgr2=0x7fb4a80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 -- 172.21.15.50:0/376004972 shutdown_connections 2023-12-19T14:40:29.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.367+0000 7fb4aecae700 1 -- 172.21.15.50:0/376004972 wait complete. 2023-12-19T14:40:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.371+0000 7fb4aecae700 1 Processor -- start 2023-12-19T14:40:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.371+0000 7fb4aecae700 1 -- start start 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a816da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 0x7fb4a815b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a815c4a0 con 0x7fb4a80a98d0 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8152a20 con 0x7fb4a8159220 2023-12-19T14:40:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a816d020 con 0x7fb4a8158e50 2023-12-19T14:40:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54200/0 (socket says 172.21.15.50:54200) 2023-12-19T14:40:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 -- 172.21.15.50:0/937419430 learned_addr learned my addr 172.21.15.50:0/937419430 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4a7fff700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a816da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4ad24b700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 0x7fb4a815b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 -- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 msgr2=0x7fb4a816da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a816da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 -- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 msgr2=0x7fb4a815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 0x7fb4a815b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 -- 172.21.15.50:0/937419430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0009ae0 con 0x7fb4a80a98d0 2023-12-19T14:40:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4ad24b700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 0x7fb4a815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aca4a700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ecb0 tx=0x7fb49c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00c920 con 0x7fb4a80a98d0 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c00caa0 con 0x7fb4a80a98d0 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.375+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a815bee0 con 0x7fb4a80a98d0 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.379+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00ce60 con 0x7fb4a80a98d0 2023-12-19T14:40:29.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.379+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a816d290 con 0x7fb4a80a98d0 2023-12-19T14:40:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.379+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a8071120 con 0x7fb4a80a98d0 2023-12-19T14:40:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.379+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb49c013680 con 0x7fb4a80a98d0 2023-12-19T14:40:29.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.383+0000 7fb4a4ff9700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 0x7fb48c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.383+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fb49c07cc40 con 0x7fb4a80a98d0 2023-12-19T14:40:29.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.383+0000 7fb4a7fff700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 0x7fb48c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.383+0000 7fb4a7fff700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 0x7fb48c062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4a815c600 tx=0x7fb4940091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.391+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb49c04d050 con 0x7fb4a80a98d0 2023-12-19T14:40:29.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.583+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4a806d960 con 0x7fb48c060550 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.587+0000 7fb4a4ff9700 1 -- 172.21.15.50:0/937419430 <== mgr.4102 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb4a806d960 con 0x7fb48c060550 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:40:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:40:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 msgr2=0x7fb48c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 0x7fb48c062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4a815c600 tx=0x7fb4940091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 msgr2=0x7fb4a80ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ecb0 tx=0x7fb49c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 shutdown_connections 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4a8159220 0x7fb4a815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb48c060550 0x7fb48c062a00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4a8158e50 0x7fb4a816da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 --2- 172.21.15.50:0/937419430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4a80a98d0 0x7fb4a80ca490 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 >> 172.21.15.50:0/937419430 conn(0x7fb4a8067b60 msgr2=0x7fb4a8154230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 shutdown_connections 2023-12-19T14:40:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.595+0000 7fb4aecae700 1 -- 172.21.15.50:0/937419430 wait complete. 2023-12-19T14:40:29.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:40:29.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 Processor -- start 2023-12-19T14:40:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 -- start start 2023-12-19T14:40:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f805c0cb290 0x7f805c158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c06b6c0 con 0x7f805c158b00 2023-12-19T14:40:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c06aab0 con 0x7f805c06bb50 2023-12-19T14:40:29.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f806226b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c06ad90 con 0x7f805c0cb290 2023-12-19T14:40:29.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f8061269700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f805c158b00 0x7f805c0cab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45988/0 (socket says 172.21.15.50:45988) 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.783+0000 7f8061269700 1 -- 172.21.15.50:0/1581627716 learned_addr learned my addr 172.21.15.50:0/1581627716 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8060a68700 1 --2- 172.21.15.50:0/1581627716 >> v2:172.21.15.50:3300/0 conn(0x7f805c0cb290 0x7f805c158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167687187 0 0) 0x7f805c06b6c0 con 0x7f805c158b00 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048003570 con 0x7f805c158b00 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8060a68700 1 -- 172.21.15.50:0/1581627716 >> v1:172.21.15.125:6789/0 conn(0x7f805c06bb50 legacy=0x7f805c0afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8060a68700 1 -- 172.21.15.50:0/1581627716 >> v1:172.21.15.50:6789/0 conn(0x7f805c158b00 legacy=0x7f805c0cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1457566153 0 0) 0x7f8048003570 con 0x7f805c158b00 2023-12-19T14:40:29.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8060a68700 1 -- 172.21.15.50:0/1581627716 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c06b070 con 0x7f805c0cb290 2023-12-19T14:40:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8060a68700 1 --2- 172.21.15.50:0/1581627716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c0cb290 0x7f805c158ed0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f804c0098e0 tx=0x7f804c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72535aaa337ef0a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058001470 con 0x7f805c0cb290 2023-12-19T14:40:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c011420 con 0x7f805c0cb290 2023-12-19T14:40:29.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f8053fff700 1 -- 172.21.15.50:0/1581627716 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0117c0 con 0x7f805c0cb290 2023-12-19T14:40:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 -- 172.21.15.50:0/1581627716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c0cb290 msgr2=0x7f805c158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 --2- 172.21.15.50:0/1581627716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c0cb290 0x7f805c158ed0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f804c0098e0 tx=0x7f804c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 -- 172.21.15.50:0/1581627716 shutdown_connections 2023-12-19T14:40:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 --2- 172.21.15.50:0/1581627716 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c0cb290 0x7f805c158ed0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 -- 172.21.15.50:0/1581627716 >> 172.21.15.50:0/1581627716 conn(0x7f805c067bb0 msgr2=0x7f805c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 -- 172.21.15.50:0/1581627716 shutdown_connections 2023-12-19T14:40:29.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.787+0000 7f806226b700 1 -- 172.21.15.50:0/1581627716 wait complete. 2023-12-19T14:40:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 Processor -- start 2023-12-19T14:40:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 -- start start 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c0cb290 0x7f805c150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 0x7f805c150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b3dc0 con 0x7f805c06bb50 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c1514e0 con 0x7f805c0cb290 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f806226b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b34d0 con 0x7f805c158b00 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f8061269700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f8061269700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54232/0 (socket says 172.21.15.50:54232) 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.791+0000 7f8061269700 1 -- 172.21.15.50:0/1302866899 learned_addr learned my addr 172.21.15.50:0/1302866899 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061a6a700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 0x7f805c150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8060a68700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c0cb290 0x7f805c150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 -- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 msgr2=0x7f805c150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 0x7f805c150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 -- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c0cb290 msgr2=0x7f805c150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c0cb290 0x7f805c150990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 -- 172.21.15.50:0/1302866899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c009420 con 0x7f805c06bb50 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061a6a700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 0x7f805c150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f8061269700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f805800cb90 tx=0x7f8058001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058007150 con 0x7f805c06bb50 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80580072d0 con 0x7f805c06bb50 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058007690 con 0x7f805c06bb50 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c0b3770 con 0x7f805c06bb50 2023-12-19T14:40:29.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c14ed70 con 0x7f805c06bb50 2023-12-19T14:40:29.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.795+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c04be40 con 0x7f805c06bb50 2023-12-19T14:40:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.799+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8058004a40 con 0x7f805c06bb50 2023-12-19T14:40:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.799+0000 7f80517fa700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 0x7f8040062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.799+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f805807be50 con 0x7f805c06bb50 2023-12-19T14:40:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.799+0000 7f8060a68700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 0x7f8040062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:29.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.803+0000 7f8060a68700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 0x7f8040062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f804c0091e0 tx=0x7f804c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:29.807+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f805804c2e0 con 0x7f805c06bb50 2023-12-19T14:40:30.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.019+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f805c06b350 con 0x7f805c06bb50 2023-12-19T14:40:30.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.563+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f805804c6b0 con 0x7f805c06bb50 2023-12-19T14:40:30.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.567+0000 7f80517fa700 1 -- 172.21.15.50:0/1302866899 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f805804c2e0 con 0x7f805c06bb50 2023-12-19T14:40:30.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 msgr2=0x7f8040062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 0x7f8040062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f804c0091e0 tx=0x7f804c016000 comp rx=0 tx=0).stop 2023-12-19T14:40:30.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 msgr2=0x7f805c14d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f805800cb90 tx=0x7f8058001a30 comp rx=0 tx=0).stop 2023-12-19T14:40:30.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 shutdown_connections 2023-12-19T14:40:30.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f805c0cb290 0x7f805c150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8040060550 0x7f8040062a00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f805c158b00 0x7f805c150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 --2- 172.21.15.50:0/1302866899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f805c06bb50 0x7f805c14d6f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 >> 172.21.15.50:0/1302866899 conn(0x7f805c067bb0 msgr2=0x7f805c15ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:30.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 shutdown_connections 2023-12-19T14:40:30.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.571+0000 7f806226b700 1 -- 172.21.15.50:0/1302866899 wait complete. 2023-12-19T14:40:30.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:40:30.754 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:30.747+0000 7faf0e5ab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:40:30.770 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:30.767+0000 7f5f4c962000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:40:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 Processor -- start 2023-12-19T14:40:30.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 -- start start 2023-12-19T14:40:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa83806aeb0 0x7fa838159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa8381595a0 0x7fa8381599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83806c850 con 0x7fa838158c90 2023-12-19T14:40:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380cb650 con 0x7fa8381595a0 2023-12-19T14:40:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.787+0000 7fa83cfcc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83806b470 con 0x7fa83806aeb0 2023-12-19T14:40:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa8381595a0 0x7fa8381599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa8381595a0 0x7fa8381599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54254/0 (socket says 172.21.15.50:54254) 2023-12-19T14:40:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa8377fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa838158c90 0x7fa8380caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46004/0 (socket says 172.21.15.50:46004) 2023-12-19T14:40:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 -- 172.21.15.50:0/1477821158 learned_addr learned my addr 172.21.15.50:0/1477821158 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa836ffd700 1 --2- 172.21.15.50:0/1477821158 >> v2:172.21.15.125:3300/0 conn(0x7fa83806aeb0 0x7fa838159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 -- 172.21.15.50:0/1477821158 >> v2:172.21.15.125:3300/0 conn(0x7fa83806aeb0 msgr2=0x7fa838159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 --2- 172.21.15.50:0/1477821158 >> v2:172.21.15.125:3300/0 conn(0x7fa83806aeb0 0x7fa838159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 -- 172.21.15.50:0/1477821158 >> v1:172.21.15.50:6789/0 conn(0x7fa838158c90 legacy=0x7fa8380caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 -- 172.21.15.50:0/1477821158 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83806b750 con 0x7fa8381595a0 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa8367fc700 1 -- 172.21.15.50:0/1477821158 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664025059 0 0) 0x7fa83806c850 con 0x7fa838158c90 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa837fff700 1 --2- 172.21.15.50:0/1477821158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8381595a0 0x7fa8381599c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa838068000 tx=0x7fa830009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3f814a33874e59 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa8367fc700 1 -- 172.21.15.50:0/1477821158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83000e070 con 0x7fa8381595a0 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa8367fc700 1 -- 172.21.15.50:0/1477821158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83000ba20 con 0x7fa8381595a0 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa8367fc700 1 -- 172.21.15.50:0/1477821158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8300092e0 con 0x7fa8381595a0 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 -- 172.21.15.50:0/1477821158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8381595a0 msgr2=0x7fa8381599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/1477821158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8381595a0 0x7fa8381599c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa838068000 tx=0x7fa830009840 comp rx=0 tx=0).stop 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 -- 172.21.15.50:0/1477821158 shutdown_connections 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/1477821158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8381595a0 0x7fa8381599c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/1477821158 >> v2:172.21.15.125:3300/0 conn(0x7fa83806aeb0 0x7fa838159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 -- 172.21.15.50:0/1477821158 >> 172.21.15.50:0/1477821158 conn(0x7fa838067bb0 msgr2=0x7fa8380cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 -- 172.21.15.50:0/1477821158 shutdown_connections 2023-12-19T14:40:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.791+0000 7fa83cfcc700 1 -- 172.21.15.50:0/1477821158 wait complete. 2023-12-19T14:40:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.795+0000 7fa83cfcc700 1 Processor -- start 2023-12-19T14:40:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.795+0000 7fa83cfcc700 1 -- start start 2023-12-19T14:40:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa838158c90 0x7fa83815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 0x7fa83815bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83815e7d0 con 0x7fa83806aeb0 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa838149e20 con 0x7fa838158c90 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa838169a90 con 0x7fa8381595a0 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54256/0 (socket says 172.21.15.50:54256) 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 -- 172.21.15.50:0/3748227159 learned_addr learned my addr 172.21.15.50:0/3748227159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa836ffd700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa838158c90 0x7fa83815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa837fff700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 0x7fa83815bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 -- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 msgr2=0x7fa83815bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 0x7fa83815bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 -- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa838158c90 msgr2=0x7fa83815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa838158c90 0x7fa83815b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 -- 172.21.15.50:0/3748227159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830009ae0 con 0x7fa83806aeb0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa837fff700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 0x7fa83815bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa8377fe700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa8280011e0 tx=0x7fa828007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828005a70 con 0x7fa83806aeb0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828005bd0 con 0x7fa83806aeb0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83816c390 con 0x7fa83806aeb0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8280070d0 con 0x7fa83806aeb0 2023-12-19T14:40:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.799+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83815e4f0 con 0x7fa83806aeb0 2023-12-19T14:40:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83816c160 con 0x7fa83806aeb0 2023-12-19T14:40:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7fa828007480 con 0x7fa83806aeb0 2023-12-19T14:40:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa81f7fe700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 0x7fa824062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fa82807bf20 con 0x7fa83806aeb0 2023-12-19T14:40:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa836ffd700 1 -- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 msgr2=0x7fa824062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.803+0000 7fa836ffd700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 0x7fa824062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:40:30.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.811+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa83816c160 con 0x7fa83806aeb0 2023-12-19T14:40:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.987+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fa83816c160 con 0x7fa83806aeb0 2023-12-19T14:40:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.995+0000 7fa81f7fe700 1 -- 172.21.15.50:0/3748227159 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa83816c160 con 0x7fa83806aeb0 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.999+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 msgr2=0x7fa824062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.999+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 0x7fa824062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.999+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 msgr2=0x7fa8381498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:30.999+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa8280011e0 tx=0x7fa828007e10 comp rx=0 tx=0).stop 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 shutdown_connections 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa838158c90 0x7fa83815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240605a0 0x7fa824062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa8381595a0 0x7fa83815bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 --2- 172.21.15.50:0/3748227159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa83806aeb0 0x7fa8381498e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 >> 172.21.15.50:0/3748227159 conn(0x7fa838067bb0 msgr2=0x7fa838157e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 shutdown_connections 2023-12-19T14:40:31.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.003+0000 7fa83cfcc700 1 -- 172.21.15.50:0/3748227159 wait complete. 2023-12-19T14:40:31.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:40:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.183+0000 7f5a1ed45700 1 Processor -- start 2023-12-19T14:40:31.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.183+0000 7f5a1ed45700 1 -- start start 2023-12-19T14:40:31.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1ed45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a18158e20 0x7f5a180ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1ed45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1ed45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180cbdf0 con 0x7f5a180cb970 2023-12-19T14:40:31.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1ed45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180cc0d0 con 0x7f5a18158e20 2023-12-19T14:40:31.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1ed45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180cc3b0 con 0x7f5a180ca2c0 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a18158e20 0x7f5a180ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a18158e20 0x7f5a180ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34568/0 (socket says 172.21.15.50:34568) 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a17fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5a180cb970 0x7f5a1806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33430/0 (socket says 172.21.15.50:33430) 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 -- 172.21.15.50:0/2614674472 learned_addr learned my addr 172.21.15.50:0/2614674472 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1d2e2700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 -- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 msgr2=0x7f5a180ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 -- 172.21.15.50:0/2614674472 >> v1:172.21.15.125:6789/0 conn(0x7f5a180cb970 legacy=0x7f5a1806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 -- 172.21.15.50:0/2614674472 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180cc690 con 0x7f5a18158e20 2023-12-19T14:40:31.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a1cae1700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180ce5c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0099e0 tx=0x7f5a0c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65a5dc9fc1d9151d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:31.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a177fe700 1 -- 172.21.15.50:0/2614674472 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00b820 con 0x7f5a18158e20 2023-12-19T14:40:31.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.187+0000 7f5a177fe700 1 -- 172.21.15.50:0/2614674472 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00b980 con 0x7f5a18158e20 2023-12-19T14:40:31.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a177fe700 1 -- 172.21.15.50:0/2614674472 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00bd20 con 0x7f5a18158e20 2023-12-19T14:40:31.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 -- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 msgr2=0x7f5a180ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180ce5c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0099e0 tx=0x7f5a0c009250 comp rx=0 tx=0).stop 2023-12-19T14:40:31.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 -- 172.21.15.50:0/2614674472 shutdown_connections 2023-12-19T14:40:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/2614674472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180ce5c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 -- 172.21.15.50:0/2614674472 >> 172.21.15.50:0/2614674472 conn(0x7f5a18067bb0 msgr2=0x7f5a180d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 -- 172.21.15.50:0/2614674472 shutdown_connections 2023-12-19T14:40:31.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 -- 172.21.15.50:0/2614674472 wait complete. 2023-12-19T14:40:31.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.191+0000 7f5a1ed45700 1 Processor -- start 2023-12-19T14:40:31.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 -- start start 2023-12-19T14:40:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b4300 con 0x7f5a180b5680 2023-12-19T14:40:31.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b5dc0 con 0x7f5a18158e20 2023-12-19T14:40:31.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1ed45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b5af0 con 0x7f5a180ca2c0 2023-12-19T14:40:31.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a17fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54260/0 (socket says 172.21.15.50:54260) 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 -- 172.21.15.50:0/3401340132 learned_addr learned my addr 172.21.15.50:0/3401340132 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1cae1700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 -- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 msgr2=0x7f5a180c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 -- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 msgr2=0x7f5a180b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a1d2e2700 1 -- 172.21.15.50:0/3401340132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0094f0 con 0x7f5a180b5680 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.195+0000 7f5a17fff700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a1d2e2700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009e90 tx=0x7f5a1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000cb70 con 0x7f5a180b5680 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000f040 con 0x7f5a180b5680 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180b1100 con 0x7f5a180b5680 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10014480 con 0x7f5a180b5680 2023-12-19T14:40:31.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a180b16c0 con 0x7f5a180b5680 2023-12-19T14:40:31.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.199+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18071120 con 0x7f5a180b5680 2023-12-19T14:40:31.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.203+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f5a10014830 con 0x7f5a180b5680 2023-12-19T14:40:31.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.203+0000 7f5a14ff9700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 0x7f59fc062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.203+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5a1007d6a0 con 0x7f5a180b5680 2023-12-19T14:40:31.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.203+0000 7f5a1cae1700 1 -- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 msgr2=0x7f59fc062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:31.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.203+0000 7f5a1cae1700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 0x7f59fc062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:40:31.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.207+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5a1004dab0 con 0x7f5a180b5680 2023-12-19T14:40:31.280 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:31.275+0000 7faf0e5ab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:40:31.349 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:31.343+0000 7f5f4c962000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:40:31.349 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:31.343+0000 7faf0e5ab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:40:31.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.379+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5a18064270 con 0x7f5a180b5680 2023-12-19T14:40:31.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.387+0000 7f5a14ff9700 1 -- 172.21.15.50:0/3401340132 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5a100510d0 con 0x7f5a180b5680 2023-12-19T14:40:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 msgr2=0x7f59fc062a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:40:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 0x7f59fc062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 msgr2=0x7f5a180b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009e90 tx=0x7f5a1000b040 comp rx=0 tx=0).stop 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 shutdown_connections 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a18158e20 0x7f5a180b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59fc0605d0 0x7f59fc062a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a180b5680 0x7f5a180b1ec0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 --2- 172.21.15.50:0/3401340132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a180ca2c0 0x7f5a180c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 >> 172.21.15.50:0/3401340132 conn(0x7f5a18067bb0 msgr2=0x7f5a180ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 shutdown_connections 2023-12-19T14:40:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.391+0000 7f5a1ed45700 1 -- 172.21.15.50:0/3401340132 wait complete. 2023-12-19T14:40:31.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:31.417 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T14:40:31.427 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:31.423+0000 7f5f4c962000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:40:31.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:31.453 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-12-19T14:40:31.482 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:31.479+0000 7faf0e5ab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:40:31.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T14:40:31.584 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:31.579+0000 7f5f4c962000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:40:31.730 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:31.723+0000 7faf0e5ab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:40:31.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.731+0000 7f0f8359e700 1 Processor -- start 2023-12-19T14:40:31.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 -- start start 2023-12-19T14:40:31.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f8406af50 0x7f0f840cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 0x7f0f8406db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840cc140 con 0x7f0f8406af50 2023-12-19T14:40:31.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840cb3f0 con 0x7f0f8406d6b0 2023-12-19T14:40:31.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.735+0000 7f0f8359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8406d220 con 0x7f0f840aedc0 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f8406af50 0x7f0f840cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f82d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 0x7f0f8406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f8406af50 0x7f0f840cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54276/0 (socket says 172.21.15.50:54276) 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 -- 172.21.15.50:0/917590155 learned_addr learned my addr 172.21.15.50:0/917590155 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 -- 172.21.15.50:0/917590155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 msgr2=0x7f0f840af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:31.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/917590155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 -- 172.21.15.50:0/917590155 >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 msgr2=0x7f0f8406db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/917590155 >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 0x7f0f8406db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 -- 172.21.15.50:0/917590155 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840cb190 con 0x7f0f8406af50 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f82d9d700 1 --2- 172.21.15.50:0/917590155 >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 0x7f0f8406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/917590155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406af50 0x7f0f840cac50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0f8406f890 tx=0x7f0f78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66831494b5a482a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8159a700 1 -- 172.21.15.50:0/917590155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800b640 con 0x7f0f8406af50 2023-12-19T14:40:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8159a700 1 -- 172.21.15.50:0/917590155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7800b7a0 con 0x7f0f8406af50 2023-12-19T14:40:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8159a700 1 -- 172.21.15.50:0/917590155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800bb60 con 0x7f0f8406af50 2023-12-19T14:40:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 -- 172.21.15.50:0/917590155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406af50 msgr2=0x7f0f840cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 --2- 172.21.15.50:0/917590155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406af50 0x7f0f840cac50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0f8406f890 tx=0x7f0f78009250 comp rx=0 tx=0).stop 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 -- 172.21.15.50:0/917590155 shutdown_connections 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 --2- 172.21.15.50:0/917590155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406af50 0x7f0f840cac50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 --2- 172.21.15.50:0/917590155 >> v2:172.21.15.125:3300/0 conn(0x7f0f8406d6b0 0x7f0f8406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 --2- 172.21.15.50:0/917590155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 -- 172.21.15.50:0/917590155 >> 172.21.15.50:0/917590155 conn(0x7f0f840cc760 msgr2=0x7f0f840ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 -- 172.21.15.50:0/917590155 shutdown_connections 2023-12-19T14:40:31.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.739+0000 7f0f8359e700 1 -- 172.21.15.50:0/917590155 wait complete. 2023-12-19T14:40:31.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 Processor -- start 2023-12-19T14:40:31.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 -- start start 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8406af50 0x7f0f840b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b4f00 con 0x7f0f8406d6b0 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b3570 con 0x7f0f8406af50 2023-12-19T14:40:31.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b2c40 con 0x7f0f840aedc0 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f81d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f81d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54288/0 (socket says 172.21.15.50:54288) 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f81d9b700 1 -- 172.21.15.50:0/286240180 learned_addr learned my addr 172.21.15.50:0/286240180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8406af50 0x7f0f840b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.743+0000 7f0f82d9d700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 msgr2=0x7f0f840c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8406af50 msgr2=0x7f0f840b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8406af50 0x7f0f840b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c009a20 con 0x7f0f8406d6b0 2023-12-19T14:40:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f81d9b700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0f780094c0 tx=0x7f0f78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78000d10 con 0x7f0f8406d6b0 2023-12-19T14:40:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7800b910 con 0x7f0f8406d6b0 2023-12-19T14:40:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780094f0 con 0x7f0f8406d6b0 2023-12-19T14:40:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78026430 con 0x7f0f8406d6b0 2023-12-19T14:40:31.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f840b3200 con 0x7f0f8406d6b0 2023-12-19T14:40:31.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.747+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f840b3720 con 0x7f0f8406d6b0 2023-12-19T14:40:31.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.751+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f0f78023030 con 0x7f0f8406d6b0 2023-12-19T14:40:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.751+0000 7f0f727fc700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.751+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f0f7807e100 con 0x7f0f8406d6b0 2023-12-19T14:40:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.751+0000 7f0f8259c700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.751+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:40:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.755+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f840b3720 con 0x7f0f8406d6b0 2023-12-19T14:40:31.805 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:31.799+0000 7faf0e5ab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:40:31.885 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:31.879+0000 7f5f4c962000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:40:31.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.947+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f840b36e0 con 0x7f0f640605a0 2023-12-19T14:40:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.955+0000 7f0f8259c700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:31.955+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:40:31.961 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:31.955+0000 7f5f4c962000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:40:32.024 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:32.019+0000 7faf0e5ab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:40:32.113 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:32.107+0000 7faf0e5ab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:40:32.189 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:32.183+0000 7f5f4c962000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:40:32.214 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:32.207+0000 7faf0e5ab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:40:32.280 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:32.275+0000 7f5f4c962000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:40:32.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:32.359+0000 7f0f8259c700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:32.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:32.359+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:40:32.371 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:32.367+0000 7f5f4c962000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:40:32.620 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:32.615+0000 7faf0e5ab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:40:32.788 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:32.783+0000 7f5f4c962000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:40:32.795 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:32.791+0000 7faf0e5ab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:40:32.967 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:32.963+0000 7f5f4c962000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:40:33.166 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.159+0000 7faf0e5ab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:40:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:33.163+0000 7f0f8259c700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:33.163+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:40:33.357 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:33.351+0000 7f5f4c962000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:40:33.390 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.383+0000 7faf0e5ab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:40:33.485 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.479+0000 7faf0e5ab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:40:33.567 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.563+0000 7faf0e5ab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:40:33.593 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:33.587+0000 7f5f4c962000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:40:33.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.643+0000 7faf0e5ab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:40:33.688 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:33.683+0000 7f5f4c962000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:40:33.769 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:33.763+0000 7f5f4c962000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:40:33.859 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:33.855+0000 7f5f4c962000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:40:33.964 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:33.959+0000 7faf0e5ab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:40:34.049 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:34.043+0000 7faf0e5ab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:40:34.121 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:34.115+0000 7f5f4c962000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:40:34.142 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:34.135+0000 7faf0e5ab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:40:34.207 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:34.203+0000 7f5f4c962000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:40:34.300 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:34.295+0000 7f5f4c962000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:40:34.346 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:34.339+0000 7faf0e5ab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:40:34.441 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:34.435+0000 7faf0e5ab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:40:34.506 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:34.503+0000 7f5f4c962000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:40:34.605 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:34.599+0000 7f5f4c962000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:40:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:34.767+0000 7f0f8259c700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:40:34.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:34.767+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:40:34.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:40:34.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:40:35.180 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.175+0000 7faf0e5ab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:40:35.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:35.199+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 7) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f0f7804eac0 con 0x7f0f8406d6b0 2023-12-19T14:40:35.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:35.199+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:40:35.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:35.199+0000 7f0f727fc700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:35.388 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faed147f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.388 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faed147f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.388 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faed147f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.388 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faed147f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.388 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faed147f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faecec7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faecec7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faecec7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faecec7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:40:35.383+0000 7faecec7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:40:35.421 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:40:35.415+0000 7f5f4c962000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:40:36.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.203+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 8) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f0f780211e0 con 0x7f0f8406d6b0 2023-12-19T14:40:36.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.203+0000 7f0f727fc700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.203+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f840b36e0 con 0x7f0f640605a0 2023-12-19T14:40:36.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.207+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.207+0000 7f0f8259c700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0f8415e7a0 tx=0x7f0f740091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.339+0000 7f0f727fc700 1 -- 172.21.15.50:0/286240180 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0f840b36e0 con 0x7f0f640605a0 2023-12-19T14:40:36.344 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir0 2023-12-19T14:40:36.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 msgr2=0x7f0f64062a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0f8415e7a0 tx=0x7f0f740091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 msgr2=0x7f0f84158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0f780094c0 tx=0x7f0f78017000 comp rx=0 tx=0).stop 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 shutdown_connections 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f640605a0 0x7f0f64062a50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f840aedc0 0x7f0f840c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f8406d6b0 0x7f0f84158870 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 --2- 172.21.15.50:0/286240180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f8406af50 0x7f0f840b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 >> 172.21.15.50:0/286240180 conn(0x7f0f840cc760 msgr2=0x7f0f840ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 shutdown_connections 2023-12-19T14:40:36.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.343+0000 7f0f8359e700 1 -- 172.21.15.50:0/286240180 wait complete. 2023-12-19T14:40:36.372 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-12-19T14:40:36.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:36.372 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-12-19T14:40:36.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T14:40:36.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.627+0000 7f9170e2e700 1 Processor -- start 2023-12-19T14:40:36.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.631+0000 7f9170e2e700 1 -- start start 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c156b90 0x7f916c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f916c156f60 0x7f916c0bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f916c0c0300 0x7f916c0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0c7e70 con 0x7f916c0c0300 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0c72a0 con 0x7f916c156f60 2023-12-19T14:40:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9170e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0c7540 con 0x7f916c156b90 2023-12-19T14:40:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f916ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f916c0c0300 0x7f916c0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f916c156f60 0x7f916c0bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f916c156f60 0x7f916c0bfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:46040/0 (socket says 172.21.15.50:46040) 2023-12-19T14:40:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 -- 172.21.15.50:0/3775131003 learned_addr learned my addr 172.21.15.50:0/3775131003 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:36.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f916a59c700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c156b90 0x7f916c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 -- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c156b90 msgr2=0x7f916c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c156b90 0x7f916c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 -- 172.21.15.50:0/3775131003 >> v2:172.21.15.50:3300/0 conn(0x7f916c0c0300 msgr2=0x7f916c0c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3775131003 >> v2:172.21.15.50:3300/0 conn(0x7f916c0c0300 0x7f916c0c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 -- 172.21.15.50:0/3775131003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0c7820 con 0x7f916c156f60 2023-12-19T14:40:36.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0bfdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f916c06abc0 tx=0x7f9160009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcfbf2318df389a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f916959a700 1 -- 172.21.15.50:0/3775131003 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000b640 con 0x7f916c156f60 2023-12-19T14:40:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f916959a700 1 -- 172.21.15.50:0/3775131003 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000b7a0 con 0x7f916c156f60 2023-12-19T14:40:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.635+0000 7f916959a700 1 -- 172.21.15.50:0/3775131003 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000bb60 con 0x7f916c156f60 2023-12-19T14:40:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 -- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 msgr2=0x7f916c0bfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0bfdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f916c06abc0 tx=0x7f9160009250 comp rx=0 tx=0).stop 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 -- 172.21.15.50:0/3775131003 shutdown_connections 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3775131003 >> v2:172.21.15.50:3300/0 conn(0x7f916c0c0300 0x7f916c0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0bfdc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3775131003 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c156b90 0x7f916c0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 -- 172.21.15.50:0/3775131003 >> 172.21.15.50:0/3775131003 conn(0x7f916c0c8db0 msgr2=0x7f916c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 -- 172.21.15.50:0/3775131003 shutdown_connections 2023-12-19T14:40:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 -- 172.21.15.50:0/3775131003 wait complete. 2023-12-19T14:40:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.639+0000 7f9170e2e700 1 Processor -- start 2023-12-19T14:40:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 -- start start 2023-12-19T14:40:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c0c0300 0x7f916c153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c06dfb0 con 0x7f916c156b90 2023-12-19T14:40:36.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0a9540 con 0x7f916c156f60 2023-12-19T14:40:36.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9170e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0a9820 con 0x7f916c0c0300 2023-12-19T14:40:36.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35896/0 (socket says 172.21.15.50:35896) 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 -- 172.21.15.50:0/3352300778 learned_addr learned my addr 172.21.15.50:0/3352300778 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f916a59c700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c0c0300 0x7f916c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f916ad9d700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 -- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c0c0300 msgr2=0x7f916c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c0c0300 0x7f916c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 -- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 msgr2=0x7f916c0c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 -- 172.21.15.50:0/3352300778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91600094f0 con 0x7f916c156b90 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f9169d9b700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f916c0c73e0 tx=0x7f9160017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.643+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160000d10 con 0x7f916c156b90 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000bb20 con 0x7f916c156b90 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916001a6c0 con 0x7f916c156b90 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c06dc40 con 0x7f916c156b90 2023-12-19T14:40:36.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c0a9160 con 0x7f916c156b90 2023-12-19T14:40:36.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c06be20 con 0x7f916c156b90 2023-12-19T14:40:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.647+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f916001aab0 con 0x7f916c156b90 2023-12-19T14:40:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.651+0000 7f915a7fc700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 0x7f914c0628d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.651+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f916007ded0 con 0x7f916c156b90 2023-12-19T14:40:36.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.651+0000 7f916a59c700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 0x7f914c0628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.651+0000 7f916a59c700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 0x7f914c0628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f916c0b2210 tx=0x7f915c008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.659+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f916c06be20 con 0x7f916c156b90 2023-12-19T14:40:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.851+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f916c06e160 con 0x7f914c060420 2023-12-19T14:40:36.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.975+0000 7f915a7fc700 1 -- 172.21.15.50:0/3352300778 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f916c06e160 con 0x7f914c060420 2023-12-19T14:40:36.978 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir1 2023-12-19T14:40:36.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.975+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 msgr2=0x7f914c0628d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.975+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 0x7f914c0628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f916c0b2210 tx=0x7f915c008040 comp rx=0 tx=0).stop 2023-12-19T14:40:36.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.975+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 msgr2=0x7f916c0aae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:36.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.975+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f916c0c73e0 tx=0x7f9160017000 comp rx=0 tx=0).stop 2023-12-19T14:40:36.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 shutdown_connections 2023-12-19T14:40:36.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f916c156f60 0x7f916c0c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f914c060420 0x7f914c0628d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f916c156b90 0x7f916c0aae70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 --2- 172.21.15.50:0/3352300778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f916c0c0300 0x7f916c153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 >> 172.21.15.50:0/3352300778 conn(0x7f916c0c8db0 msgr2=0x7f916c0cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 shutdown_connections 2023-12-19T14:40:36.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:36.979+0000 7f9170e2e700 1 -- 172.21.15.50:0/3352300778 wait complete. 2023-12-19T14:40:37.006 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-12-19T14:40:37.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:37.006 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-12-19T14:40:37.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T14:40:37.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.251+0000 7fbb67fff700 1 Processor -- start 2023-12-19T14:40:37.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.251+0000 7fbb67fff700 1 -- start start 2023-12-19T14:40:37.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 0x7fbb680af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb67fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb680cac50 0x7fbb680af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb67fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb680cb3f0 con 0x7fbb6806af50 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb67fff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb680cc140 con 0x7fbb680cac50 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb67fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6806d220 con 0x7fbb680aedc0 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb667fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbb6806af50 0x7fbb6806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53782/0 (socket says 172.21.15.50:53782) 2023-12-19T14:40:37.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb667fc700 1 -- 172.21.15.50:0/205893360 learned_addr learned my addr 172.21.15.50:0/205893360 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:37.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb66ffd700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 0x7fbb680af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.255+0000 7fbb677fe700 1 --2- 172.21.15.50:0/205893360 >> v2:172.21.15.125:3300/0 conn(0x7fbb680cac50 0x7fbb680af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603323245 0 0) 0x7fbb680cb3f0 con 0x7fbb6806af50 2023-12-19T14:40:37.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003570 con 0x7fbb6806af50 2023-12-19T14:40:37.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501395494 0 0) 0x7fbb50003570 con 0x7fbb6806af50 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 msgr2=0x7fbb680af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 0x7fbb680af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 >> v2:172.21.15.125:3300/0 conn(0x7fbb680cac50 msgr2=0x7fbb680af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 --2- 172.21.15.50:0/205893360 >> v2:172.21.15.125:3300/0 conn(0x7fbb680cac50 0x7fbb680af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6806b3a0 con 0x7fbb6806af50 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fbb5c001470 con 0x7fbb6806af50 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 >> v1:172.21.15.50:6789/0 conn(0x7fbb6806af50 legacy=0x7fbb6806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 0x7fbb50003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb50003fe0 0x7fbb50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50004d50 0x7fbb500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680cc140 con 0x7fbb50003fe0 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6806d220 con 0x7fbb50004d50 2023-12-19T14:40:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000ba90 con 0x7fbb680aedc0 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb50003fe0 0x7fbb50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 -- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 msgr2=0x7fbb50003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb680aedc0 0x7fbb50003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 -- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50004d50 msgr2=0x7fbb500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50004d50 0x7fbb500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 -- 172.21.15.50:0/205893360 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50003570 con 0x7fbb50003fe0 2023-12-19T14:40:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb667fc700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb50003fe0 0x7fbb50004810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c3c0 tx=0x7fbb5c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1600661a90923f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00cc30 con 0x7fbb50003fe0 2023-12-19T14:40:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c00cdd0 con 0x7fbb50003fe0 2023-12-19T14:40:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb65ffb700 1 -- 172.21.15.50:0/205893360 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c005b70 con 0x7fbb50003fe0 2023-12-19T14:40:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb67fff700 1 -- 172.21.15.50:0/205893360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb50003fe0 msgr2=0x7fbb50004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb67fff700 1 --2- 172.21.15.50:0/205893360 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb50003fe0 0x7fbb50004810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c3c0 tx=0x7fbb5c001e90 comp rx=0 tx=0).stop 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.259+0000 7fbb677fe700 1 -- 172.21.15.50:0/205893360 reap_dead start 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- 172.21.15.50:0/205893360 shutdown_connections 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- 172.21.15.50:0/205893360 >> 172.21.15.50:0/205893360 conn(0x7fbb680cc760 msgr2=0x7fbb680ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- 172.21.15.50:0/205893360 shutdown_connections 2023-12-19T14:40:37.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- 172.21.15.50:0/205893360 wait complete. 2023-12-19T14:40:37.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 Processor -- start 2023-12-19T14:40:37.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- start start 2023-12-19T14:40:37.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 0x7fbb680c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb68159eb0 0x7fbb68158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb681576a0 con 0x7fbb6815a320 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680b2e20 con 0x7fbb50003fe0 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.263+0000 7fbb67fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680b3c80 con 0x7fbb68159eb0 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb66ffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 0x7fbb680c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb667fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb68159eb0 0x7fbb68158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35900/0 (socket says 172.21.15.50:35900) 2023-12-19T14:40:37.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 -- 172.21.15.50:0/138651739 learned_addr learned my addr 172.21.15.50:0/138651739 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 -- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb68159eb0 msgr2=0x7fbb68158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb68159eb0 0x7fbb68158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 -- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 msgr2=0x7fbb680c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 0x7fbb680c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 -- 172.21.15.50:0/138651739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68159a40 con 0x7fbb6815a320 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb66ffd700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 0x7fbb680c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb677fe700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbb680cb020 tx=0x7fbb6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000c790 con 0x7fbb6815a320 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6000f040 con 0x7fbb6815a320 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb680b3070 con 0x7fbb6815a320 2023-12-19T14:40:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000cb40 con 0x7fbb6815a320 2023-12-19T14:40:37.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb680b3630 con 0x7fbb6815a320 2023-12-19T14:40:37.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.267+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb680b3df0 con 0x7fbb6815a320 2023-12-19T14:40:37.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.271+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fbb60014480 con 0x7fbb6815a320 2023-12-19T14:40:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.271+0000 7fbb4effd700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 0x7fbb54062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.271+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fbb6007d2d0 con 0x7fbb6815a320 2023-12-19T14:40:37.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.271+0000 7fbb66ffd700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 0x7fbb54062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.271+0000 7fbb66ffd700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 0x7fbb54062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011390 tx=0x7fbb580091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.279+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbb6004d250 con 0x7fbb6815a320 2023-12-19T14:40:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.471+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb6804be40 con 0x7fbb54060780 2023-12-19T14:40:37.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb4effd700 1 -- 172.21.15.50:0/138651739 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbb6804be40 con 0x7fbb54060780 2023-12-19T14:40:37.599 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir2 2023-12-19T14:40:37.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 msgr2=0x7fbb54062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 0x7fbb54062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011390 tx=0x7fbb580091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 msgr2=0x7fbb68159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbb680cb020 tx=0x7fbb6000b040 comp rx=0 tx=0).stop 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 shutdown_connections 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb54060780 0x7fbb54062c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6815a320 0x7fbb68159500 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb68159eb0 0x7fbb68158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 --2- 172.21.15.50:0/138651739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb50003fe0 0x7fbb680c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.595+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 >> 172.21.15.50:0/138651739 conn(0x7fbb680cc760 msgr2=0x7fbb680b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:37.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.599+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 shutdown_connections 2023-12-19T14:40:37.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.599+0000 7fbb67fff700 1 -- 172.21.15.50:0/138651739 wait complete. 2023-12-19T14:40:37.625 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-12-19T14:40:37.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:37.625 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-12-19T14:40:37.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T14:40:37.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.871+0000 7f96a222e700 1 Processor -- start 2023-12-19T14:40:37.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f96a222e700 1 -- start start 2023-12-19T14:40:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f96a222e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f969c06e4c0 0x7f969c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f96a222e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0c7130 con 0x7f969c0ae270 2023-12-19T14:40:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f96a222e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0c7410 con 0x7f969c06e0f0 2023-12-19T14:40:37.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f96a222e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c156140 con 0x7f969c06e4c0 2023-12-19T14:40:37.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.875+0000 7f969affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f969c06e4c0 0x7f969c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f969c06e4c0 0x7f969c0c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35902/0 (socket says 172.21.15.50:35902) 2023-12-19T14:40:37.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 -- 172.21.15.50:0/3239327174 learned_addr learned my addr 172.21.15.50:0/3239327174 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:37.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969a7fc700 1 -- 172.21.15.50:0/3239327174 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699020537 0 0) 0x7f969c0c7410 con 0x7f969c06e0f0 2023-12-19T14:40:37.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 -- 172.21.15.50:0/3239327174 >> v1:172.21.15.125:6789/0 conn(0x7f969c06e0f0 legacy=0x7f969c0c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 -- 172.21.15.50:0/3239327174 >> v1:172.21.15.50:6789/0 conn(0x7f969c0ae270 legacy=0x7f969c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 -- 172.21.15.50:0/3239327174 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0c76f0 con 0x7f969c06e4c0 2023-12-19T14:40:37.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969affd700 1 --2- 172.21.15.50:0/3239327174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e4c0 0x7f969c0c81e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f96900099b0 tx=0x7f9690009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921acbfcf223cf8b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969a7fc700 1 -- 172.21.15.50:0/3239327174 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969000b760 con 0x7f969c06e4c0 2023-12-19T14:40:37.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969a7fc700 1 -- 172.21.15.50:0/3239327174 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969000b8c0 con 0x7f969c06e4c0 2023-12-19T14:40:37.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f969a7fc700 1 -- 172.21.15.50:0/3239327174 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969000bc60 con 0x7f969c06e4c0 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f96a222e700 1 -- 172.21.15.50:0/3239327174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e4c0 msgr2=0x7f969c0c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.879+0000 7f96a222e700 1 --2- 172.21.15.50:0/3239327174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e4c0 0x7f969c0c81e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f96900099b0 tx=0x7f9690009250 comp rx=0 tx=0).stop 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 -- 172.21.15.50:0/3239327174 shutdown_connections 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 --2- 172.21.15.50:0/3239327174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e4c0 0x7f969c0c81e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 -- 172.21.15.50:0/3239327174 >> 172.21.15.50:0/3239327174 conn(0x7f969c0c8db0 msgr2=0x7f969c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 -- 172.21.15.50:0/3239327174 shutdown_connections 2023-12-19T14:40:37.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 -- 172.21.15.50:0/3239327174 wait complete. 2023-12-19T14:40:37.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 Processor -- start 2023-12-19T14:40:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.883+0000 7f96a222e700 1 -- start start 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 0x7f969c0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 0x7f969c0b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0af5b0 con 0x7f969c06e0f0 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0af890 con 0x7f969c0ae270 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0afb70 con 0x7f969c06e4c0 2023-12-19T14:40:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35916/0 (socket says 172.21.15.50:35916) 2023-12-19T14:40:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 -- 172.21.15.50:0/3733573188 learned_addr learned my addr 172.21.15.50:0/3733573188 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969affd700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 0x7f969c0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969bfff700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 0x7f969c0b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 -- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 msgr2=0x7f969c0c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 0x7f969c0c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 -- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 msgr2=0x7f969c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 0x7f969c0b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 -- 172.21.15.50:0/3733573188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900094f0 con 0x7f969c06e0f0 2023-12-19T14:40:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969affd700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 0x7f969c0c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969bfff700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 0x7f969c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f969b7fe700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c530 tx=0x7f968c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c012630 con 0x7f969c06e0f0 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0127b0 con 0x7f969c06e0f0 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0afe50 con 0x7f969c06e0f0 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c012b70 con 0x7f969c06e0f0 2023-12-19T14:40:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.887+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c0c6120 con 0x7f969c06e0f0 2023-12-19T14:40:37.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.891+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c0c6590 con 0x7f969c06e0f0 2023-12-19T14:40:37.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.891+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f968c01b400 con 0x7f969c06e0f0 2023-12-19T14:40:37.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.891+0000 7f96837fe700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 0x7f9688062c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:37.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.895+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f968c07ca20 con 0x7f969c06e0f0 2023-12-19T14:40:37.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.895+0000 7f969affd700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 0x7f9688062c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.895+0000 7f969affd700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 0x7f9688062c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f96900099b0 tx=0x7f9690016000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:37.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:37.899+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f969c0c6590 con 0x7f969c06e0f0 2023-12-19T14:40:38.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.095+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f969c0c6590 con 0x7f96880607d0 2023-12-19T14:40:38.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.219+0000 7f96837fe700 1 -- 172.21.15.50:0/3733573188 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f969c0c6590 con 0x7f96880607d0 2023-12-19T14:40:38.225 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir3 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 msgr2=0x7f9688062c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 0x7f9688062c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f96900099b0 tx=0x7f9690016000 comp rx=0 tx=0).stop 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 msgr2=0x7f969c0c5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f968c00c530 tx=0x7f968c00c8f0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 shutdown_connections 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f969c0ae270 0x7f969c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96880607d0 0x7f9688062c80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f969c06e4c0 0x7f969c0c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 --2- 172.21.15.50:0/3733573188 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c06e0f0 0x7f969c0c5be0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 >> 172.21.15.50:0/3733573188 conn(0x7f969c0c8db0 msgr2=0x7f969c0ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:38.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 shutdown_connections 2023-12-19T14:40:38.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.223+0000 7f96a222e700 1 -- 172.21.15.50:0/3733573188 wait complete. 2023-12-19T14:40:38.252 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-12-19T14:40:38.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:38.252 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-12-19T14:40:38.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T14:40:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 Processor -- start 2023-12-19T14:40:38.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 -- start start 2023-12-19T14:40:38.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f72ec156e70 0x7f72ec06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 0x7f72ec0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72ec0ac480 0x7f72ec0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec06e180 con 0x7f72ec156e70 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0ac8e0 con 0x7f72ec0ac480 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72f12f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec06d7b0 con 0x7f72ec06b060 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f72ec156e70 0x7f72ec06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f72ec156e70 0x7f72ec06dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35922/0 (socket says 172.21.15.50:35922) 2023-12-19T14:40:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 -- 172.21.15.50:0/480863800 learned_addr learned my addr 172.21.15.50:0/480863800 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eb7fe700 1 --2- 172.21.15.50:0/480863800 >> v2:172.21.15.125:3300/0 conn(0x7f72ec0ac480 0x7f72ec0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 0x7f72ec0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 -- 172.21.15.50:0/480863800 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 msgr2=0x7f72ec0abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 0x7f72ec0abf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 -- 172.21.15.50:0/480863800 >> v2:172.21.15.125:3300/0 conn(0x7f72ec0ac480 msgr2=0x7f72ec0aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 --2- 172.21.15.50:0/480863800 >> v2:172.21.15.125:3300/0 conn(0x7f72ec0ac480 0x7f72ec0aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72eaffd700 1 -- 172.21.15.50:0/480863800 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0ae570 con 0x7f72ec156e70 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.495+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 0x7f72ec0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72eaffd700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec156e70 0x7f72ec06dc40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72ec06a630 tx=0x7f72dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdf7df4d6107fbe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72e9ffb700 1 -- 172.21.15.50:0/480863800 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00b860 con 0x7f72ec156e70 2023-12-19T14:40:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72e9ffb700 1 -- 172.21.15.50:0/480863800 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00b9c0 con 0x7f72ec156e70 2023-12-19T14:40:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72e9ffb700 1 -- 172.21.15.50:0/480863800 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00bd60 con 0x7f72ec156e70 2023-12-19T14:40:38.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 -- 172.21.15.50:0/480863800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec156e70 msgr2=0x7f72ec06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec156e70 0x7f72ec06dc40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72ec06a630 tx=0x7f72dc009250 comp rx=0 tx=0).stop 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 -- 172.21.15.50:0/480863800 shutdown_connections 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 --2- 172.21.15.50:0/480863800 >> v2:172.21.15.125:3300/0 conn(0x7f72ec0ac480 0x7f72ec0aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec06b060 0x7f72ec0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 --2- 172.21.15.50:0/480863800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec156e70 0x7f72ec06dc40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 -- 172.21.15.50:0/480863800 >> 172.21.15.50:0/480863800 conn(0x7f72ec067bb0 msgr2=0x7f72ec0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 -- 172.21.15.50:0/480863800 shutdown_connections 2023-12-19T14:40:38.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.499+0000 7f72f12f0700 1 -- 172.21.15.50:0/480863800 wait complete. 2023-12-19T14:40:38.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 Processor -- start 2023-12-19T14:40:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 -- start start 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 0x7f72ec15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec156e70 0x7f72ec0c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec16a1e0 con 0x7f72ec06b060 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec16ad60 con 0x7f72ec0ac480 2023-12-19T14:40:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72f12f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0c9390 con 0x7f72ec156e70 2023-12-19T14:40:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35926/0 (socket says 172.21.15.50:35926) 2023-12-19T14:40:38.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 -- 172.21.15.50:0/3716177761 learned_addr learned my addr 172.21.15.50:0/3716177761 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 0x7f72ec15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eb7fe700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec156e70 0x7f72ec0c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 -- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec156e70 msgr2=0x7f72ec0c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec156e70 0x7f72ec0c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 -- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 msgr2=0x7f72ec15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 0x7f72ec15aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72eaffd700 1 -- 172.21.15.50:0/3716177761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc0094f0 con 0x7f72ec06b060 2023-12-19T14:40:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.503+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 0x7f72ec15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72eaffd700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00b3c0 tx=0x7f72dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc024440 con 0x7f72ec06b060 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc0245a0 con 0x7f72ec06b060 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec16a430 con 0x7f72ec06b060 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc024960 con 0x7f72ec06b060 2023-12-19T14:40:38.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec16a9f0 con 0x7f72ec06b060 2023-12-19T14:40:38.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.507+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec071120 con 0x7f72ec06b060 2023-12-19T14:40:38.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.511+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f72dc01a420 con 0x7f72ec06b060 2023-12-19T14:40:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.511+0000 7f72d2ffd700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 0x7f72d8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.511+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f72dc07ec50 con 0x7f72ec06b060 2023-12-19T14:40:38.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.511+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 0x7f72d8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:38.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.515+0000 7f72ea7fc700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 0x7f72d8062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f72ec156120 tx=0x7f72e0006f90 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.519+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72dc04ece0 con 0x7f72ec06b060 2023-12-19T14:40:38.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.711+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f72ec06cba0 con 0x7f72d8060750 2023-12-19T14:40:38.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.835+0000 7f72d2ffd700 1 -- 172.21.15.50:0/3716177761 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f72ec06cba0 con 0x7f72d8060750 2023-12-19T14:40:38.838 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir4 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 msgr2=0x7f72d8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 0x7f72d8062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f72ec156120 tx=0x7f72e0006f90 comp rx=0 tx=0).stop 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 msgr2=0x7f72ec169ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00b3c0 tx=0x7f72dc017000 comp rx=0 tx=0).stop 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 shutdown_connections 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72ec0ac480 0x7f72ec15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f72d8060750 0x7f72d8062c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f72ec156e70 0x7f72ec0c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 --2- 172.21.15.50:0/3716177761 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f72ec06b060 0x7f72ec169ab0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 >> 172.21.15.50:0/3716177761 conn(0x7f72ec067bb0 msgr2=0x7f72ec157240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 shutdown_connections 2023-12-19T14:40:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:38.839+0000 7f72f12f0700 1 -- 172.21.15.50:0/3716177761 wait complete. 2023-12-19T14:40:38.865 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-12-19T14:40:38.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:38.865 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-12-19T14:40:38.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T14:40:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.123+0000 7f61b946e700 1 Processor -- start 2023-12-19T14:40:39.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.123+0000 7f61b946e700 1 -- start start 2023-12-19T14:40:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b946e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 0x7f61b415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b946e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40b0ac0 con 0x7f61b40cb130 2023-12-19T14:40:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b946e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b406b210 con 0x7f61b415c760 2023-12-19T14:40:39.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b946e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b415cbe0 con 0x7f61b40cb500 2023-12-19T14:40:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b2ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f61b40cb130 0x7f61b406a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53802/0 (socket says 172.21.15.50:53802) 2023-12-19T14:40:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b2ffd700 1 -- 172.21.15.50:0/2145299649 learned_addr learned my addr 172.21.15.50:0/2145299649 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b27fc700 1 --2- 172.21.15.50:0/2145299649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 0x7f61b415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320496503 0 0) 0x7f61b40b0ac0 con 0x7f61b40cb130 2023-12-19T14:40:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c003570 con 0x7f61b40cb130 2023-12-19T14:40:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121748775 0 0) 0x7f61b406b210 con 0x7f61b415c760 2023-12-19T14:40:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b40b0ac0 con 0x7f61b415c760 2023-12-19T14:40:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b27fc700 1 -- 172.21.15.50:0/2145299649 >> v1:172.21.15.125:6789/0 conn(0x7f61b415c760 legacy=0x7f61b40b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b27fc700 1 -- 172.21.15.50:0/2145299649 >> v1:172.21.15.50:6789/0 conn(0x7f61b40cb130 legacy=0x7f61b406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.127+0000 7f61b27fc700 1 -- 172.21.15.50:0/2145299649 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40ca810 con 0x7f61b40cb500 2023-12-19T14:40:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b27fc700 1 --2- 172.21.15.50:0/2145299649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 0x7f61b415c220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009910 tx=0x7f61a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5758c6d7adcee3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8011630 con 0x7f61b40cb500 2023-12-19T14:40:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8011790 con 0x7f61b40cb500 2023-12-19T14:40:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b1ffb700 1 -- 172.21.15.50:0/2145299649 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8011b10 con 0x7f61b40cb500 2023-12-19T14:40:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 -- 172.21.15.50:0/2145299649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 msgr2=0x7f61b415c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 --2- 172.21.15.50:0/2145299649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 0x7f61b415c220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009910 tx=0x7f61a80091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 -- 172.21.15.50:0/2145299649 shutdown_connections 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 --2- 172.21.15.50:0/2145299649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b40cb500 0x7f61b415c220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 -- 172.21.15.50:0/2145299649 >> 172.21.15.50:0/2145299649 conn(0x7f61b4067bb0 msgr2=0x7f61b40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 -- 172.21.15.50:0/2145299649 shutdown_connections 2023-12-19T14:40:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 -- 172.21.15.50:0/2145299649 wait complete. 2023-12-19T14:40:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.131+0000 7f61b946e700 1 Processor -- start 2023-12-19T14:40:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- start start 2023-12-19T14:40:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b40cb130 0x7f61b415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b415c760 0x7f61b4154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b44d0 con 0x7f61b40cb500 2023-12-19T14:40:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b38c0 con 0x7f61b40cb130 2023-12-19T14:40:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b3ba0 con 0x7f61b415c760 2023-12-19T14:40:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35936/0 (socket says 172.21.15.50:35936) 2023-12-19T14:40:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 -- 172.21.15.50:0/2525179100 learned_addr learned my addr 172.21.15.50:0/2525179100 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b37fe700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b415c760 0x7f61b4154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b2ffd700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b40cb130 0x7f61b415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 -- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b415c760 msgr2=0x7f61b4154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b415c760 0x7f61b4154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 -- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b40cb130 msgr2=0x7f61b415a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b40cb130 0x7f61b415a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 -- 172.21.15.50:0/2525179100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8009420 con 0x7f61b40cb500 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b27fc700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f61b406f300 tx=0x7f61a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8011900 con 0x7f61b40cb500 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8011a60 con 0x7f61b40cb500 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b40b3e80 con 0x7f61b40cb500 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8025670 con 0x7f61b40cb500 2023-12-19T14:40:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.135+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b4155c20 con 0x7f61b40cb500 2023-12-19T14:40:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.139+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40b4640 con 0x7f61b40cb500 2023-12-19T14:40:39.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.139+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f61a8023070 con 0x7f61b40cb500 2023-12-19T14:40:39.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.143+0000 7f619affd700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 0x7f61a0062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.143+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f61a807e400 con 0x7f61b40cb500 2023-12-19T14:40:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.143+0000 7f61b2ffd700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 0x7f61a0062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.143+0000 7f61b2ffd700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 0x7f61a0062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b415f770 tx=0x7f61a4009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.147+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f61a804e490 con 0x7f61b40cb500 2023-12-19T14:40:39.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.339+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b4071120 con 0x7f61a0060750 2023-12-19T14:40:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.467+0000 7f619affd700 1 -- 172.21.15.50:0/2525179100 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f61b4071120 con 0x7f61a0060750 2023-12-19T14:40:39.474 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir5 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.471+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 msgr2=0x7f61a0062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.471+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 0x7f61a0062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b415f770 tx=0x7f61a4009040 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 msgr2=0x7f61b40b2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f61b406f300 tx=0x7f61a8017000 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 shutdown_connections 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61a0060750 0x7f61a0062c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f61b415c760 0x7f61b4154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f61b40cb500 0x7f61b40b2e40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 --2- 172.21.15.50:0/2525179100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61b40cb130 0x7f61b415a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 >> 172.21.15.50:0/2525179100 conn(0x7f61b4067bb0 msgr2=0x7f61b40cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 shutdown_connections 2023-12-19T14:40:39.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.475+0000 7f61b946e700 1 -- 172.21.15.50:0/2525179100 wait complete. 2023-12-19T14:40:39.502 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-12-19T14:40:39.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:39.502 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-12-19T14:40:39.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T14:40:39.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.743+0000 7fa9e50a5700 1 Processor -- start 2023-12-19T14:40:39.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.743+0000 7fa9e50a5700 1 -- start start 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9e50a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9e006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9e50a5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9e015cce0 0x7fa9e006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9e50a5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e00add70 con 0x7fa9e015c490 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9e50a5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e0159dc0 con 0x7fa9e015cce0 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9e50a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00af210 con 0x7fa9e015c860 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ded9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9e015c490 0x7fa9e01596b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53806/0 (socket says 172.21.15.50:53806) 2023-12-19T14:40:39.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ded9d700 1 -- 172.21.15.50:0/2116408176 learned_addr learned my addr 172.21.15.50:0/2116408176 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9de59c700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9e006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9df59e700 1 --2- 172.21.15.50:0/2116408176 >> v2:172.21.15.125:3300/0 conn(0x7fa9e015cce0 0x7fa9e006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298402512 0 0) 0x7fa9e00add70 con 0x7fa9e015c490 2023-12-19T14:40:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c8003570 con 0x7fa9e015c490 2023-12-19T14:40:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486170642 0 0) 0x7fa9c8003570 con 0x7fa9e015c490 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 msgr2=0x7fa9e006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9e006d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 >> v2:172.21.15.125:3300/0 conn(0x7fa9e015cce0 msgr2=0x7fa9e006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 --2- 172.21.15.50:0/2116408176 >> v2:172.21.15.125:3300/0 conn(0x7fa9e015cce0 0x7fa9e006d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0159fa0 con 0x7fa9e015c490 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fa9d0001470 con 0x7fa9e015c490 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 >> v1:172.21.15.50:6789/0 conn(0x7fa9e015c490 legacy=0x7fa9e01596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c8004810 0x7fa9c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0159dc0 con 0x7fa9c8003a60 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00af210 con 0x7fa9c8004810 2023-12-19T14:40:39.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c800bae0 con 0x7fa9e015c860 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.747+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9df59e700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9de59c700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c8004810 0x7fa9c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 -- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 msgr2=0x7fa9c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e015c860 0x7fa9c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 -- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c8004810 msgr2=0x7fa9c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:39.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9c8004810 0x7fa9c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 -- 172.21.15.50:0/2116408176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8003570 con 0x7fa9c8003a60 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9c8003e70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c3c0 tx=0x7fa9d0001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970a7640038b54f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000cc90 con 0x7fa9c8003a60 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d000ce30 con 0x7fa9c8003a60 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9ddd9b700 1 -- 172.21.15.50:0/2116408176 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0005b70 con 0x7fa9c8003a60 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 -- 172.21.15.50:0/2116408176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 msgr2=0x7fa9c8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/2116408176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9c8003e70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c3c0 tx=0x7fa9d0001e90 comp rx=0 tx=0).stop 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9df59e700 1 -- 172.21.15.50:0/2116408176 reap_dead start 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 -- 172.21.15.50:0/2116408176 shutdown_connections 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 -- 172.21.15.50:0/2116408176 >> 172.21.15.50:0/2116408176 conn(0x7fa9e00ccc20 msgr2=0x7fa9e00cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 -- 172.21.15.50:0/2116408176 shutdown_connections 2023-12-19T14:40:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.751+0000 7fa9e50a5700 1 -- 172.21.15.50:0/2116408176 wait complete. 2023-12-19T14:40:39.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 Processor -- start 2023-12-19T14:40:39.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 -- start start 2023-12-19T14:40:39.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e01567e0 0x7fa9e0155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e0157b50 0x7fa9e0157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0158500 con 0x7fa9c8003a60 2023-12-19T14:40:39.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e01570e0 con 0x7fa9e0157b50 2023-12-19T14:40:39.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.755+0000 7fa9e50a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00ab9f0 con 0x7fa9e01567e0 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35956/0 (socket says 172.21.15.50:35956) 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 -- 172.21.15.50:0/930816767 learned_addr learned my addr 172.21.15.50:0/930816767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9df59e700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e0157b50 0x7fa9e0157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9de59c700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e01567e0 0x7fa9e0155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 -- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e01567e0 msgr2=0x7fa9e0155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e01567e0 0x7fa9e0155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 -- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e0157b50 msgr2=0x7fa9e0157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e0157b50 0x7fa9e0157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 -- 172.21.15.50:0/930816767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00abcd0 con 0x7fa9c8003a60 2023-12-19T14:40:39.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9ded9d700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0011000 tx=0x7fa9d0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000cc90 con 0x7fa9c8003a60 2023-12-19T14:40:39.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d00072e0 con 0x7fa9c8003a60 2023-12-19T14:40:39.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e00abfb0 con 0x7fa9c8003a60 2023-12-19T14:40:39.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0007660 con 0x7fa9c8003a60 2023-12-19T14:40:39.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e00ab6e0 con 0x7fa9c8003a60 2023-12-19T14:40:39.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.759+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0071120 con 0x7fa9c8003a60 2023-12-19T14:40:39.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.763+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fa9d0019070 con 0x7fa9c8003a60 2023-12-19T14:40:39.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.763+0000 7fa9c6ffd700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 0x7fa9cc062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:39.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.763+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fa9d007dcb0 con 0x7fa9c8003a60 2023-12-19T14:40:39.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.763+0000 7fa9de59c700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 0x7fa9cc062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:39.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.767+0000 7fa9de59c700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 0x7fa9cc062c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0158930 tx=0x7fa9d4008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:39.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.771+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa9d004dd40 con 0x7fa9c8003a60 2023-12-19T14:40:39.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:39.971+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9e00af9c0 con 0x7fa9cc060780 2023-12-19T14:40:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.079+0000 7fa9c6ffd700 1 -- 172.21.15.50:0/930816767 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa9e00af9c0 con 0x7fa9cc060780 2023-12-19T14:40:40.085 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir6 2023-12-19T14:40:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 msgr2=0x7fa9cc062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 0x7fa9cc062c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0158930 tx=0x7fa9d4008040 comp rx=0 tx=0).stop 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 msgr2=0x7fa9e00ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0011000 tx=0x7fa9d0015000 comp rx=0 tx=0).stop 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 shutdown_connections 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e0157b50 0x7fa9e0157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9cc060780 0x7fa9cc062c30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9e01567e0 0x7fa9e0155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 --2- 172.21.15.50:0/930816767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9c8003a60 0x7fa9e00ab1a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 >> 172.21.15.50:0/930816767 conn(0x7fa9e00ccc20 msgr2=0x7fa9e006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 shutdown_connections 2023-12-19T14:40:40.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.083+0000 7fa9e50a5700 1 -- 172.21.15.50:0/930816767 wait complete. 2023-12-19T14:40:40.113 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-12-19T14:40:40.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:40.113 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-12-19T14:40:40.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T14:40:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.355+0000 7fa4ad46e700 1 Processor -- start 2023-12-19T14:40:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.355+0000 7fa4ad46e700 1 -- start start 2023-12-19T14:40:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 0x7fa4a80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a8068120 0x7fa4a80c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80c8de0 0x7fa4a80c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a80c83c0 con 0x7fa4a8068120 2023-12-19T14:40:40.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a80685e0 con 0x7fa4a8158db0 2023-12-19T14:40:40.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4ad46e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80688c0 con 0x7fa4a80c8de0 2023-12-19T14:40:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a8068120 0x7fa4a80c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4a8068120 0x7fa4a80c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35958/0 (socket says 172.21.15.50:35958) 2023-12-19T14:40:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 -- 172.21.15.50:0/1351003767 learned_addr learned my addr 172.21.15.50:0/1351003767 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:40.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a77fe700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80c8de0 0x7fa4a80c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a6ffd700 1 --2- 172.21.15.50:0/1351003767 >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 0x7fa4a80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 -- 172.21.15.50:0/1351003767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80c8de0 msgr2=0x7fa4a80c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80c8de0 0x7fa4a80c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 -- 172.21.15.50:0/1351003767 >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 msgr2=0x7fa4a80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/1351003767 >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 0x7fa4a80cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 -- 172.21.15.50:0/1351003767 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a8068ba0 con 0x7fa4a8068120 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a6ffd700 1 --2- 172.21.15.50:0/1351003767 >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 0x7fa4a80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c88a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c320 tx=0x7fa49c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3043938108045b18 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a5ffb700 1 -- 172.21.15.50:0/1351003767 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00b860 con 0x7fa4a8068120 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a5ffb700 1 -- 172.21.15.50:0/1351003767 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c00b9c0 con 0x7fa4a8068120 2023-12-19T14:40:40.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.359+0000 7fa4a5ffb700 1 -- 172.21.15.50:0/1351003767 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00bd60 con 0x7fa4a8068120 2023-12-19T14:40:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 -- 172.21.15.50:0/1351003767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 msgr2=0x7fa4a80c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c88a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c320 tx=0x7fa49c009250 comp rx=0 tx=0).stop 2023-12-19T14:40:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 -- 172.21.15.50:0/1351003767 shutdown_connections 2023-12-19T14:40:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c88a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/1351003767 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80c8de0 0x7fa4a80c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/1351003767 >> v2:172.21.15.125:3300/0 conn(0x7fa4a8158db0 0x7fa4a80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 -- 172.21.15.50:0/1351003767 >> 172.21.15.50:0/1351003767 conn(0x7fa4a80cdba0 msgr2=0x7fa4a80cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 -- 172.21.15.50:0/1351003767 shutdown_connections 2023-12-19T14:40:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.363+0000 7fa4ad46e700 1 -- 172.21.15.50:0/1351003767 wait complete. 2023-12-19T14:40:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 Processor -- start 2023-12-19T14:40:40.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 -- start start 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 0x7fa4a8155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8158db0 0x7fa4a81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80c7a20 con 0x7fa4a8068120 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8159bd0 con 0x7fa4a80c8de0 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4ad46e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80c5330 con 0x7fa4a8158db0 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35968/0 (socket says 172.21.15.50:35968) 2023-12-19T14:40:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 -- 172.21.15.50:0/824467575 learned_addr learned my addr 172.21.15.50:0/824467575 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 0x7fa4a8155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a77fe700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8158db0 0x7fa4a81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 -- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8158db0 msgr2=0x7fa4a81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8158db0 0x7fa4a81573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 -- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 msgr2=0x7fa4a8155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 0x7fa4a8155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 -- 172.21.15.50:0/824467575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0094f0 con 0x7fa4a8068120 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 0x7fa4a8155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa4a6ffd700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa49800c850 tx=0x7fa498008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.367+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498011600 con 0x7fa4a8068120 2023-12-19T14:40:40.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498011780 con 0x7fa4a8068120 2023-12-19T14:40:40.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498011b40 con 0x7fa4a8068120 2023-12-19T14:40:40.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a80c5610 con 0x7fa4a8068120 2023-12-19T14:40:40.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a8155c60 con 0x7fa4a8068120 2023-12-19T14:40:40.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a804be40 con 0x7fa4a8068120 2023-12-19T14:40:40.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.371+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fa4980073b0 con 0x7fa4a8068120 2023-12-19T14:40:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.375+0000 7fa48effd700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 0x7fa494062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.375+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fa49807cdd0 con 0x7fa4a8068120 2023-12-19T14:40:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.375+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 0x7fa494062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.375+0000 7fa4a67fc700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 0x7fa494062c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c320 tx=0x7fa49c017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.379+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa49804cdf0 con 0x7fa4a8068120 2023-12-19T14:40:40.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.571+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4a8159d10 con 0x7fa494060750 2023-12-19T14:40:40.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.699+0000 7fa48effd700 1 -- 172.21.15.50:0/824467575 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa49c00bb30 con 0x7fa494060750 2023-12-19T14:40:40.702 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir7 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.699+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 msgr2=0x7fa494062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.699+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 0x7fa494062c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c320 tx=0x7fa49c017000 comp rx=0 tx=0).stop 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 msgr2=0x7fa4a80c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa49800c850 tx=0x7fa498008040 comp rx=0 tx=0).stop 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 shutdown_connections 2023-12-19T14:40:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a80c8de0 0x7fa4a8155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa494060750 0x7fa494062c00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8158db0 0x7fa4a81573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 --2- 172.21.15.50:0/824467575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a8068120 0x7fa4a80c4c50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 >> 172.21.15.50:0/824467575 conn(0x7fa4a80cdba0 msgr2=0x7fa4a80c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 shutdown_connections 2023-12-19T14:40:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.703+0000 7fa4ad46e700 1 -- 172.21.15.50:0/824467575 wait complete. 2023-12-19T14:40:40.729 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-12-19T14:40:40.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:40.729 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-12-19T14:40:40.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T14:40:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.971+0000 7fbeb2e16700 1 Processor -- start 2023-12-19T14:40:40.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.971+0000 7fbeb2e16700 1 -- start start 2023-12-19T14:40:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2e16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbeac0aedc0 0x7fbeac0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2e16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 0x7fbeac0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2e16700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac0cb3f0 con 0x7fbeac06d6b0 2023-12-19T14:40:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2e16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeac0cc140 con 0x7fbeac0aedc0 2023-12-19T14:40:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2e16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac06d220 con 0x7fbeac06af50 2023-12-19T14:40:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbeac0aedc0 0x7fbeac0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2615700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbeac06d6b0 0x7fbeac0af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53814/0 (socket says 172.21.15.50:53814) 2023-12-19T14:40:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1613700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 0x7fbeac0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb2615700 1 -- 172.21.15.50:0/3574749912 learned_addr learned my addr 172.21.15.50:0/3574749912 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 msgr2=0x7fbeac0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 0x7fbeac0cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3574749912 >> v1:172.21.15.50:6789/0 conn(0x7fbeac06d6b0 legacy=0x7fbeac0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3574749912 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac0cb190 con 0x7fbeac0aedc0 2023-12-19T14:40:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1613700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 0x7fbeac0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb1e14700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac0af190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbea80098e0 tx=0x7fbea80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=989ce0e8647264cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb0e12700 1 -- 172.21.15.50:0/3574749912 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea800b640 con 0x7fbeac0aedc0 2023-12-19T14:40:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb0e12700 1 -- 172.21.15.50:0/3574749912 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea800b7a0 con 0x7fbeac0aedc0 2023-12-19T14:40:40.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.975+0000 7fbeb0e12700 1 -- 172.21.15.50:0/3574749912 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea800bb40 con 0x7fbeac0aedc0 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 msgr2=0x7fbeac0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac0af190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbea80098e0 tx=0x7fbea80091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3574749912 shutdown_connections 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06af50 0x7fbeac0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3574749912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac0af190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3574749912 >> 172.21.15.50:0/3574749912 conn(0x7fbeac0cc760 msgr2=0x7fbeac0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3574749912 shutdown_connections 2023-12-19T14:40:40.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3574749912 wait complete. 2023-12-19T14:40:40.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 Processor -- start 2023-12-19T14:40:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.979+0000 7fbeb2e16700 1 -- start start 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06d6b0 0x7fbeac15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac168ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0c6d00 con 0x7fbeac06af50 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0c7c90 con 0x7fbeac0aedc0 2023-12-19T14:40:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac1697a0 con 0x7fbeac06d6b0 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35970/0 (socket says 172.21.15.50:35970) 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3724143432 learned_addr learned my addr 172.21.15.50:0/3724143432 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2615700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac168ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1613700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06d6b0 0x7fbeac15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06d6b0 msgr2=0x7fbeac15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06d6b0 0x7fbeac15d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 msgr2=0x7fbeac168ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac168ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 -- 172.21.15.50:0/3724143432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c009420 con 0x7fbeac06af50 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2615700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac168ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:40.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb1e14700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbea800a010 tx=0x7fbea8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea801f070 con 0x7fbeac06af50 2023-12-19T14:40:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea801ab60 con 0x7fbeac06af50 2023-12-19T14:40:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea8011420 con 0x7fbeac06af50 2023-12-19T14:40:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8009420 con 0x7fbeac06af50 2023-12-19T14:40:40.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.983+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac0c6f50 con 0x7fbeac06af50 2023-12-19T14:40:40.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.987+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeac04be40 con 0x7fbeac06af50 2023-12-19T14:40:40.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.987+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fbea8011810 con 0x7fbeac06af50 2023-12-19T14:40:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.991+0000 7fbea1ffb700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 0x7fbe90062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.991+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fbea807dc60 con 0x7fbeac06af50 2023-12-19T14:40:40.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.991+0000 7fbeb1613700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 0x7fbe90062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:40.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.991+0000 7fbeb1613700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 0x7fbe90062c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbeac15ed20 tx=0x7fbe9c0091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:41.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:40.995+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbea804dd90 con 0x7fbeac06af50 2023-12-19T14:40:41.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.199+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeac06f3d0 con 0x7fbe90060750 2023-12-19T14:40:41.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.327+0000 7fbea1ffb700 1 -- 172.21.15.50:0/3724143432 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fbeac06f3d0 con 0x7fbe90060750 2023-12-19T14:40:41.331 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir8 2023-12-19T14:40:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 msgr2=0x7fbe90062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 0x7fbe90062c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbeac15ed20 tx=0x7fbe9c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 msgr2=0x7fbeac0c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbea800a010 tx=0x7fbea8017000 comp rx=0 tx=0).stop 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 shutdown_connections 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbeac0aedc0 0x7fbeac168ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbe90060750 0x7fbe90062c00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbeac06d6b0 0x7fbeac15d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 --2- 172.21.15.50:0/3724143432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbeac06af50 0x7fbeac0c7750 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 >> 172.21.15.50:0/3724143432 conn(0x7fbeac0cc760 msgr2=0x7fbeac15b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 shutdown_connections 2023-12-19T14:40:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.331+0000 7fbeb2e16700 1 -- 172.21.15.50:0/3724143432 wait complete. 2023-12-19T14:40:41.359 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-12-19T14:40:41.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:40:41.360 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-12-19T14:40:41.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T14:40:41.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.591+0000 7fac7306e700 1 Processor -- start 2023-12-19T14:40:41.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.591+0000 7fac7306e700 1 -- start start 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac7306e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac6c156e70 0x7fac6c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac7306e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac6c06b060 0x7fac6c0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac7306e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c06d7b0 con 0x7fac6c0ac480 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac7306e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c06e180 con 0x7fac6c06b060 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac7306e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c0ae570 con 0x7fac6c156e70 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac6bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac6c06b060 0x7fac6c0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac6bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac6c06b060 0x7fac6c0abf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35972/0 (socket says 172.21.15.50:35972) 2023-12-19T14:40:41.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac6bfff700 1 -- 172.21.15.50:0/909410071 learned_addr learned my addr 172.21.15.50:0/909410071 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:41.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.595+0000 7fac70e0a700 1 --2- 172.21.15.50:0/909410071 >> v2:172.21.15.125:3300/0 conn(0x7fac6c156e70 0x7fac6c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6bfff700 1 -- 172.21.15.50:0/909410071 >> v2:172.21.15.125:3300/0 conn(0x7fac6c156e70 msgr2=0x7fac6c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:41.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6bfff700 1 --2- 172.21.15.50:0/909410071 >> v2:172.21.15.125:3300/0 conn(0x7fac6c156e70 0x7fac6c06dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6bfff700 1 -- 172.21.15.50:0/909410071 >> v1:172.21.15.50:6789/0 conn(0x7fac6c0ac480 legacy=0x7fac6c0aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6bfff700 1 -- 172.21.15.50:0/909410071 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0ac8a0 con 0x7fac6c06b060 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6bfff700 1 --2- 172.21.15.50:0/909410071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c06b060 0x7fac6c0abf40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fac6c064820 tx=0x7fac58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e034bef9ce58d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6b7fe700 1 -- 172.21.15.50:0/909410071 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5800b860 con 0x7fac6c06b060 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6b7fe700 1 -- 172.21.15.50:0/909410071 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5800b9c0 con 0x7fac6c06b060 2023-12-19T14:40:41.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac6b7fe700 1 -- 172.21.15.50:0/909410071 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5800bd60 con 0x7fac6c06b060 2023-12-19T14:40:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 -- 172.21.15.50:0/909410071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c06b060 msgr2=0x7fac6c0abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 --2- 172.21.15.50:0/909410071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c06b060 0x7fac6c0abf40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fac6c064820 tx=0x7fac58009250 comp rx=0 tx=0).stop 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 -- 172.21.15.50:0/909410071 shutdown_connections 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 --2- 172.21.15.50:0/909410071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c06b060 0x7fac6c0abf40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 --2- 172.21.15.50:0/909410071 >> v2:172.21.15.125:3300/0 conn(0x7fac6c156e70 0x7fac6c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 -- 172.21.15.50:0/909410071 >> 172.21.15.50:0/909410071 conn(0x7fac6c067bb0 msgr2=0x7fac6c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 -- 172.21.15.50:0/909410071 shutdown_connections 2023-12-19T14:40:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.599+0000 7fac7306e700 1 -- 172.21.15.50:0/909410071 wait complete. 2023-12-19T14:40:41.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.603+0000 7fac7306e700 1 Processor -- start 2023-12-19T14:40:41.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.603+0000 7fac7306e700 1 -- start start 2023-12-19T14:40:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 0x7fac6c14b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 0x7fac6c0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0c7160 con 0x7fac6c156e70 2023-12-19T14:40:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0c7440 con 0x7fac6c06b060 2023-12-19T14:40:41.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0c7720 con 0x7fac6c0ac480 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35976/0 (socket says 172.21.15.50:35976) 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 -- 172.21.15.50:0/368789764 learned_addr learned my addr 172.21.15.50:0/368789764 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac70e0a700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 0x7fac6c14b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac6bfff700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 0x7fac6c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 -- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 msgr2=0x7fac6c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 0x7fac6c0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 -- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 msgr2=0x7fac6c14b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 0x7fac6c14b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 -- 172.21.15.50:0/368789764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580094f0 con 0x7fac6c156e70 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac6bfff700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 0x7fac6c0a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac70e0a700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 0x7fac6c14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7160b700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fac6400c8e0 tx=0x7fac64001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6400ce80 con 0x7fac6c156e70 2023-12-19T14:40:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0c7a00 con 0x7fac6c156e70 2023-12-19T14:40:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6400a7e0 con 0x7fac6c156e70 2023-12-19T14:40:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6400ab80 con 0x7fac6c156e70 2023-12-19T14:40:41.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.607+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c14bba0 con 0x7fac6c156e70 2023-12-19T14:40:41.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.611+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0c7f00 con 0x7fac6c156e70 2023-12-19T14:40:41.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.611+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fac64012420 con 0x7fac6c156e70 2023-12-19T14:40:41.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.611+0000 7fac68ff9700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 0x7fac50062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.615+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fac6407d1a0 con 0x7fac6c156e70 2023-12-19T14:40:41.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.615+0000 7fac70e0a700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 0x7fac50062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:41.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.615+0000 7fac70e0a700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 0x7fac50062c00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fac60006fd0 tx=0x7fac60008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:41.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.619+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac6404d230 con 0x7fac6c156e70 2023-12-19T14:40:41.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.823+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fac6c071120 con 0x7fac50060750 2023-12-19T14:40:41.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.947+0000 7fac68ff9700 1 -- 172.21.15.50:0/368789764 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fac6c071120 con 0x7fac50060750 2023-12-19T14:40:41.954 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/dir9 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 msgr2=0x7fac50062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 0x7fac50062c00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fac60006fd0 tx=0x7fac60008040 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 msgr2=0x7fac6c0b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fac6400c8e0 tx=0x7fac64001710 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 shutdown_connections 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac50060750 0x7fac50062c00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac6c156e70 0x7fac6c0b0840 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6c0ac480 0x7fac6c0a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 --2- 172.21.15.50:0/368789764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6c06b060 0x7fac6c14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 >> 172.21.15.50:0/368789764 conn(0x7fac6c067bb0 msgr2=0x7fac6c0ab980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 shutdown_connections 2023-12-19T14:40:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:41.955+0000 7fac7306e700 1 -- 172.21.15.50:0/368789764 wait complete. 2023-12-19T14:40:41.984 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-12-19T14:40:41.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T14:40:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.159+0000 7f149b4cd700 1 Processor -- start 2023-12-19T14:40:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 -- start start 2023-12-19T14:40:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1494161da0 0x7f149406a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f149415d720 0x7f149406abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14940c6560 con 0x7f149415d350 2023-12-19T14:40:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14940c70e0 con 0x7f1494161da0 2023-12-19T14:40:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f149b4cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14940cc900 con 0x7f149415d720 2023-12-19T14:40:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f1499269700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f149415d350 0x7f14940cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53838/0 (socket says 172.21.15.50:53838) 2023-12-19T14:40:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f1499269700 1 -- 172.21.15.50:0/949202414 learned_addr learned my addr 172.21.15.50:0/949202414 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.163+0000 7f1498a68700 1 --2- 172.21.15.50:0/949202414 >> v2:172.21.15.50:3300/0 conn(0x7f1494161da0 0x7f149406a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1499a6a700 1 --2- 172.21.15.50:0/949202414 >> v2:172.21.15.125:3300/0 conn(0x7f149415d720 0x7f149406abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f148bfff700 1 -- 172.21.15.50:0/949202414 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372786529 0 0) 0x7f14940c6560 con 0x7f149415d350 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f148bfff700 1 -- 172.21.15.50:0/949202414 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1480003570 con 0x7f149415d350 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1498a68700 1 -- 172.21.15.50:0/949202414 >> v2:172.21.15.125:3300/0 conn(0x7f149415d720 msgr2=0x7f149406abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1498a68700 1 --2- 172.21.15.50:0/949202414 >> v2:172.21.15.125:3300/0 conn(0x7f149415d720 0x7f149406abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1498a68700 1 -- 172.21.15.50:0/949202414 >> v1:172.21.15.50:6789/0 conn(0x7f149415d350 legacy=0x7f14940cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1498a68700 1 -- 172.21.15.50:0/949202414 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14940c67b0 con 0x7f1494161da0 2023-12-19T14:40:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f1498a68700 1 --2- 172.21.15.50:0/949202414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494161da0 0x7f149406a6a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f14940c9640 tx=0x7f14840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d927d8b82c097e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f148bfff700 1 -- 172.21.15.50:0/949202414 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484011630 con 0x7f1494161da0 2023-12-19T14:40:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f148bfff700 1 -- 172.21.15.50:0/949202414 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14840117b0 con 0x7f1494161da0 2023-12-19T14:40:42.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f148bfff700 1 -- 172.21.15.50:0/949202414 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484011b70 con 0x7f1494161da0 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 -- 172.21.15.50:0/949202414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494161da0 msgr2=0x7f149406a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 --2- 172.21.15.50:0/949202414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494161da0 0x7f149406a6a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f14940c9640 tx=0x7f14840091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 -- 172.21.15.50:0/949202414 shutdown_connections 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 --2- 172.21.15.50:0/949202414 >> v2:172.21.15.125:3300/0 conn(0x7f149415d720 0x7f149406abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 --2- 172.21.15.50:0/949202414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494161da0 0x7f149406a6a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 -- 172.21.15.50:0/949202414 >> 172.21.15.50:0/949202414 conn(0x7f1494067bb0 msgr2=0x7f14940cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:42.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 -- 172.21.15.50:0/949202414 shutdown_connections 2023-12-19T14:40:42.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.167+0000 7f149b4cd700 1 -- 172.21.15.50:0/949202414 wait complete. 2023-12-19T14:40:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 Processor -- start 2023-12-19T14:40:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 -- start start 2023-12-19T14:40:42.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149415d350 0x7f1494150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1494161da0 0x7f149406cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149415bb90 con 0x7f149415d720 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149415a5c0 con 0x7f149415d350 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f149b4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149415b260 con 0x7f1494161da0 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.171+0000 7f1498a68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35998/0 (socket says 172.21.15.50:35998) 2023-12-19T14:40:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 -- 172.21.15.50:0/2551008604 learned_addr learned my addr 172.21.15.50:0/2551008604 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1499269700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149415d350 0x7f1494150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1499a6a700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1494161da0 0x7f149406cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 -- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1494161da0 msgr2=0x7f149406cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1494161da0 0x7f149406cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 -- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149415d350 msgr2=0x7f1494150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149415d350 0x7f1494150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 -- 172.21.15.50:0/2551008604 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484009420 con 0x7f149415d720 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f1498a68700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1484013000 tx=0x7f1484017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148401f070 con 0x7f149415d720 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484000bc0 con 0x7f149415d720 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149415b540 con 0x7f149415d720 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484011b70 con 0x7f149415d720 2023-12-19T14:40:42.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149415c560 con 0x7f149415d720 2023-12-19T14:40:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.175+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1494071120 con 0x7f149415d720 2023-12-19T14:40:42.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.179+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f1484023070 con 0x7f149415d720 2023-12-19T14:40:42.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.179+0000 7f14897fa700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 0x7f1478062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.179+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f148407e0e0 con 0x7f149415d720 2023-12-19T14:40:42.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.179+0000 7f1499269700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 0x7f1478062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.179+0000 7f1499269700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 0x7f1478062c00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14900058d0 tx=0x7f1490009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.183+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f148404e170 con 0x7f149415d720 2023-12-19T14:40:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.379+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f14940c6a90 con 0x7f1478060750 2023-12-19T14:40:42.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.487+0000 7f14897fa700 1 -- 172.21.15.50:0/2551008604 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f14940c6a90 con 0x7f1478060750 2023-12-19T14:40:42.491 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.487+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 msgr2=0x7f1478062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.487+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 0x7f1478062c00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14900058d0 tx=0x7f1490009040 comp rx=0 tx=0).stop 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.487+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 msgr2=0x7f149415c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.487+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1484013000 tx=0x7f1484017000 comp rx=0 tx=0).stop 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 shutdown_connections 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1478060750 0x7f1478062c00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1494161da0 0x7f149406cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f149415d720 0x7f149415c020 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 --2- 172.21.15.50:0/2551008604 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f149415d350 0x7f1494150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 >> 172.21.15.50:0/2551008604 conn(0x7f1494067bb0 msgr2=0x7f14940abce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 shutdown_connections 2023-12-19T14:40:42.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.491+0000 7f149b4cd700 1 -- 172.21.15.50:0/2551008604 wait complete. 2023-12-19T14:40:42.515 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-12-19T14:40:42.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T14:40:42.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.691+0000 7fe415945700 1 Processor -- start 2023-12-19T14:40:42.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.691+0000 7fe415945700 1 -- start start 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe415945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 0x7fe4100cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe415945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41006b2c0 con 0x7fe4100c59d0 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe415945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41006dfa0 con 0x7fe4100c5600 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe415945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41006a6b0 con 0x7fe41006b750 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe4100c59d0 0x7fe4100caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53846/0 (socket says 172.21.15.50:53846) 2023-12-19T14:40:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40e7fc700 1 -- 172.21.15.50:0/193935444 learned_addr learned my addr 172.21.15.50:0/193935444 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:42.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40f7fe700 1 --2- 172.21.15.50:0/193935444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 0x7fe4100cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429027034 0 0) 0x7fe41006b2c0 con 0x7fe4100c59d0 2023-12-19T14:40:42.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8003570 con 0x7fe4100c59d0 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252889469 0 0) 0x7fe41006dfa0 con 0x7fe4100c5600 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41006b2c0 con 0x7fe4100c5600 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412999078 0 0) 0x7fe3f8003570 con 0x7fe4100c59d0 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40f7fe700 1 -- 172.21.15.50:0/193935444 >> v1:172.21.15.125:6789/0 conn(0x7fe4100c5600 legacy=0x7fe41006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40f7fe700 1 -- 172.21.15.50:0/193935444 >> v1:172.21.15.50:6789/0 conn(0x7fe4100c59d0 legacy=0x7fe4100caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40f7fe700 1 -- 172.21.15.50:0/193935444 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41006a990 con 0x7fe41006b750 2023-12-19T14:40:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40f7fe700 1 --2- 172.21.15.50:0/193935444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 0x7fe4100cb110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe408009e90 tx=0x7fe408009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8e72e13232857 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40800e070 con 0x7fe41006b750 2023-12-19T14:40:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40800b870 con 0x7fe41006b750 2023-12-19T14:40:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.695+0000 7fe40dffb700 1 -- 172.21.15.50:0/193935444 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404001470 con 0x7fe41006b750 2023-12-19T14:40:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- 172.21.15.50:0/193935444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 msgr2=0x7fe4100cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 --2- 172.21.15.50:0/193935444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 0x7fe4100cb110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe408009e90 tx=0x7fe408009780 comp rx=0 tx=0).stop 2023-12-19T14:40:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- 172.21.15.50:0/193935444 shutdown_connections 2023-12-19T14:40:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 --2- 172.21.15.50:0/193935444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe41006b750 0x7fe4100cb110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- 172.21.15.50:0/193935444 >> 172.21.15.50:0/193935444 conn(0x7fe410067bb0 msgr2=0x7fe4100cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- 172.21.15.50:0/193935444 shutdown_connections 2023-12-19T14:40:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- 172.21.15.50:0/193935444 wait complete. 2023-12-19T14:40:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 Processor -- start 2023-12-19T14:40:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.699+0000 7fe415945700 1 -- start start 2023-12-19T14:40:42.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe41006b750 0x7fe41015a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 0x7fe41014f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41015a890 con 0x7fe4100c5600 2023-12-19T14:40:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41015ab70 con 0x7fe41006b750 2023-12-19T14:40:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41015ae50 con 0x7fe4100c59d0 2023-12-19T14:40:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36000/0 (socket says 172.21.15.50:36000) 2023-12-19T14:40:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 -- 172.21.15.50:0/247014076 learned_addr learned my addr 172.21.15.50:0/247014076 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40effd700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe41006b750 0x7fe41015a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40f7fe700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 0x7fe41014f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 -- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 msgr2=0x7fe41014f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 0x7fe41014f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 -- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe41006b750 msgr2=0x7fe41015a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe41006b750 0x7fe41015a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 -- 172.21.15.50:0/247014076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe408009a20 con 0x7fe4100c5600 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40f7fe700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 0x7fe41014f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe40e7fc700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe40400cb90 tx=0x7fe404001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404007e90 con 0x7fe4100c5600 2023-12-19T14:40:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404007360 con 0x7fe4100c5600 2023-12-19T14:40:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41015b130 con 0x7fe4100c5600 2023-12-19T14:40:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404004830 con 0x7fe4100c5600 2023-12-19T14:40:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.703+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41014fcb0 con 0x7fe4100c5600 2023-12-19T14:40:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.707+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41004be40 con 0x7fe4100c5600 2023-12-19T14:40:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.707+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fe404004be0 con 0x7fe4100c5600 2023-12-19T14:40:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.711+0000 7fe3f6ffd700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 0x7fe3fc062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.711+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fe40407c4f0 con 0x7fe4100c5600 2023-12-19T14:40:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.711+0000 7fe40effd700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 0x7fe3fc062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.711+0000 7fe40effd700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 0x7fe3fc062c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe400007fd0 tx=0x7fe400007d50 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.719+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe40404c600 con 0x7fe4100c5600 2023-12-19T14:40:42.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:42.911+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4100c5f10 con 0x7fe3fc060750 2023-12-19T14:40:43.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.007+0000 7fe3f6ffd700 1 -- 172.21.15.50:0/247014076 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe4100c5f10 con 0x7fe3fc060750 2023-12-19T14:40:43.014 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-12-19T14:40:43.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 msgr2=0x7fe3fc062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 0x7fe3fc062c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe400007fd0 tx=0x7fe400007d50 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 msgr2=0x7fe410157370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe40400cb90 tx=0x7fe404001b50 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 shutdown_connections 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3fc060750 0x7fe3fc062c00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4100c59d0 0x7fe41014f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4100c5600 0x7fe410157370 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 --2- 172.21.15.50:0/247014076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe41006b750 0x7fe41015a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 >> 172.21.15.50:0/247014076 conn(0x7fe410067bb0 msgr2=0x7fe410062d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 shutdown_connections 2023-12-19T14:40:43.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.015+0000 7fe415945700 1 -- 172.21.15.50:0/247014076 wait complete. 2023-12-19T14:40:43.042 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-12-19T14:40:43.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T14:40:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.211+0000 7f7793fff700 1 Processor -- start 2023-12-19T14:40:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.211+0000 7f7793fff700 1 -- start start 2023-12-19T14:40:43.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7793fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794157210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7793fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7794157750 0x7f779406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7793fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779406c7c0 con 0x7f7794156640 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7793fff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779406ca10 con 0x7f7794157750 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7793fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779406ccf0 con 0x7f77940c8ae0 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7792ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7794156640 0x7f7794156a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53848/0 (socket says 172.21.15.50:53848) 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f7792ffd700 1 -- 172.21.15.50:0/29691353 learned_addr learned my addr 172.21.15.50:0/29691353 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 --2- 172.21.15.50:0/29691353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f77937fe700 1 --2- 172.21.15.50:0/29691353 >> v2:172.21.15.125:3300/0 conn(0x7f7794157750 0x7f779406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f77927fc700 1 -- 172.21.15.50:0/29691353 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618636792 0 0) 0x7f779406c7c0 con 0x7f7794156640 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f77927fc700 1 -- 172.21.15.50:0/29691353 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7780003570 con 0x7f7794156640 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 -- 172.21.15.50:0/29691353 >> v2:172.21.15.125:3300/0 conn(0x7f7794157750 msgr2=0x7f779406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 --2- 172.21.15.50:0/29691353 >> v2:172.21.15.125:3300/0 conn(0x7f7794157750 0x7f779406d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 -- 172.21.15.50:0/29691353 >> v1:172.21.15.50:6789/0 conn(0x7f7794156640 legacy=0x7f7794156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 -- 172.21.15.50:0/29691353 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779406cfd0 con 0x7f77940c8ae0 2023-12-19T14:40:43.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.215+0000 7f778a7fc700 1 --2- 172.21.15.50:0/29691353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794157210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f777c0098e0 tx=0x7f777c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f13c43cb934146 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f77927fc700 1 -- 172.21.15.50:0/29691353 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c011630 con 0x7f77940c8ae0 2023-12-19T14:40:43.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f77927fc700 1 -- 172.21.15.50:0/29691353 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c011790 con 0x7f77940c8ae0 2023-12-19T14:40:43.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f77927fc700 1 -- 172.21.15.50:0/29691353 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c011b30 con 0x7f77940c8ae0 2023-12-19T14:40:43.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 -- 172.21.15.50:0/29691353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 msgr2=0x7f7794157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 --2- 172.21.15.50:0/29691353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794157210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f777c0098e0 tx=0x7f777c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 -- 172.21.15.50:0/29691353 shutdown_connections 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 --2- 172.21.15.50:0/29691353 >> v2:172.21.15.125:3300/0 conn(0x7f7794157750 0x7f779406d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 --2- 172.21.15.50:0/29691353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794157210 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 -- 172.21.15.50:0/29691353 >> 172.21.15.50:0/29691353 conn(0x7f7794067bb0 msgr2=0x7f77940c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 -- 172.21.15.50:0/29691353 shutdown_connections 2023-12-19T14:40:43.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.219+0000 7f7793fff700 1 -- 172.21.15.50:0/29691353 wait complete. 2023-12-19T14:40:43.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 Processor -- start 2023-12-19T14:40:43.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 -- start start 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794167c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7794156640 0x7f7794151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794153230 con 0x7f7794157750 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794153510 con 0x7f7794156640 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77941537f0 con 0x7f77940c8ae0 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36002/0 (socket says 172.21.15.50:36002) 2023-12-19T14:40:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 -- 172.21.15.50:0/1801225155 learned_addr learned my addr 172.21.15.50:0/1801225155 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:43.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7792ffd700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794167c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f778a7fc700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7794156640 0x7f7794151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 -- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 msgr2=0x7f7794167c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794167c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 -- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7794156640 msgr2=0x7f7794151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7794156640 0x7f7794151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 -- 172.21.15.50:0/1801225155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c009420 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f77937fe700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f778c00e900 tx=0x7f778c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c00ee40 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c00c850 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794153ad0 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.223+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c00cc10 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.227+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77941542b0 con 0x7f7794157750 2023-12-19T14:40:43.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.227+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794153f50 con 0x7f7794157750 2023-12-19T14:40:43.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.227+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f778c013470 con 0x7f7794157750 2023-12-19T14:40:43.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.231+0000 7f778b7fe700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 0x7f7774062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.231+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f778c07cfb0 con 0x7f7794157750 2023-12-19T14:40:43.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.231+0000 7f7792ffd700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 0x7f7774062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.231+0000 7f7792ffd700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 0x7f7774062c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7794158c80 tx=0x7f7784009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.235+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f778c04d040 con 0x7f7794157750 2023-12-19T14:40:43.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.431+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f779404be40 con 0x7f7774060750 2023-12-19T14:40:43.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.539+0000 7f778b7fe700 1 -- 172.21.15.50:0/1801225155 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f779404be40 con 0x7f7774060750 2023-12-19T14:40:43.545 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 msgr2=0x7f7774062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 0x7f7774062c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7794158c80 tx=0x7f7784009040 comp rx=0 tx=0).stop 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 msgr2=0x7f7794154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f778c00e900 tx=0x7f778c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 shutdown_connections 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7794156640 0x7f7794151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7774060750 0x7f7774062c00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7794157750 0x7f7794154bc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 --2- 172.21.15.50:0/1801225155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77940c8ae0 0x7f7794167c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 >> 172.21.15.50:0/1801225155 conn(0x7f7794067bb0 msgr2=0x7f779406b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 shutdown_connections 2023-12-19T14:40:43.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.543+0000 7f7793fff700 1 -- 172.21.15.50:0/1801225155 wait complete. 2023-12-19T14:40:43.571 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-12-19T14:40:43.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T14:40:43.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.747+0000 7fd81b59e700 1 Processor -- start 2023-12-19T14:40:43.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.747+0000 7fd81b59e700 1 -- start start 2023-12-19T14:40:43.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81b59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd81c158b30 0x7fd81c0ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 0x7fd81c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c15d870 con 0x7fd81c0c7a00 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81b59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c0cacc0 con 0x7fd81c158b30 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0c7f70 con 0x7fd81c0c9880 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd819d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd81c0c7a00 0x7fd81c0c9170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53852/0 (socket says 172.21.15.50:53852) 2023-12-19T14:40:43.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd819d9b700 1 -- 172.21.15.50:0/4078629713 learned_addr learned my addr 172.21.15.50:0/4078629713 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:43.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81ad9d700 1 --2- 172.21.15.50:0/4078629713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 0x7fd81c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81a59c700 1 --2- 172.21.15.50:0/4078629713 >> v2:172.21.15.125:3300/0 conn(0x7fd81c158b30 0x7fd81c0ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272297769 0 0) 0x7fd810001110 con 0x7fd81c0c7a00 2023-12-19T14:40:43.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd804003570 con 0x7fd81c0c7a00 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81ad9d700 1 -- 172.21.15.50:0/4078629713 >> v2:172.21.15.125:3300/0 conn(0x7fd81c158b30 msgr2=0x7fd81c0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81ad9d700 1 --2- 172.21.15.50:0/4078629713 >> v2:172.21.15.125:3300/0 conn(0x7fd81c158b30 0x7fd81c0ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030159793 0 0) 0x7fd804003570 con 0x7fd81c0c7a00 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81ad9d700 1 -- 172.21.15.50:0/4078629713 >> v1:172.21.15.50:6789/0 conn(0x7fd81c0c7a00 legacy=0x7fd81c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.751+0000 7fd81ad9d700 1 -- 172.21.15.50:0/4078629713 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0caf10 con 0x7fd81c0c9880 2023-12-19T14:40:43.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81ad9d700 1 --2- 172.21.15.50:0/4078629713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 0x7fd81c15d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd814009e60 tx=0x7fd814009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f0cb637db20ba9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400e070 con 0x7fd81c0c9880 2023-12-19T14:40:43.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81400b7a0 con 0x7fd81c0c9880 2023-12-19T14:40:43.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81959a700 1 -- 172.21.15.50:0/4078629713 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810001600 con 0x7fd81c0c9880 2023-12-19T14:40:43.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 -- 172.21.15.50:0/4078629713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 msgr2=0x7fd81c15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 --2- 172.21.15.50:0/4078629713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 0x7fd81c15d330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd814009e60 tx=0x7fd814009780 comp rx=0 tx=0).stop 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 -- 172.21.15.50:0/4078629713 shutdown_connections 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 --2- 172.21.15.50:0/4078629713 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c0c9880 0x7fd81c15d330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 --2- 172.21.15.50:0/4078629713 >> v2:172.21.15.125:3300/0 conn(0x7fd81c158b30 0x7fd81c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 -- 172.21.15.50:0/4078629713 >> 172.21.15.50:0/4078629713 conn(0x7fd81c06b5b0 msgr2=0x7fd81c06b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 -- 172.21.15.50:0/4078629713 shutdown_connections 2023-12-19T14:40:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.755+0000 7fd81b59e700 1 -- 172.21.15.50:0/4078629713 wait complete. 2023-12-19T14:40:43.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 Processor -- start 2023-12-19T14:40:43.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 -- start start 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 0x7fd81c0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c158b30 0x7fd81c0b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0c52f0 con 0x7fd81c0c7a00 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0c46e0 con 0x7fd81c0c9880 2023-12-19T14:40:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0c49c0 con 0x7fd81c158b30 2023-12-19T14:40:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36010/0 (socket says 172.21.15.50:36010) 2023-12-19T14:40:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 -- 172.21.15.50:0/2806874040 learned_addr learned my addr 172.21.15.50:0/2806874040 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:43.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81ad9d700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c158b30 0x7fd81c0b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd819d9b700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 0x7fd81c0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 -- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c158b30 msgr2=0x7fd81c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c158b30 0x7fd81c0b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 -- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 msgr2=0x7fd81c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 0x7fd81c0b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd81a59c700 1 -- 172.21.15.50:0/2806874040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd814009a20 con 0x7fd81c0c7a00 2023-12-19T14:40:43.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.759+0000 7fd819d9b700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 0x7fd81c0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd81a59c700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ec80 tx=0x7fd80c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c00c850 con 0x7fd81c0c7a00 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c00c9b0 con 0x7fd81c0c7a00 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c0c4ca0 con 0x7fd81c0c7a00 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c00cd70 con 0x7fd81c0c7a00 2023-12-19T14:40:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c0c6830 con 0x7fd81c0c7a00 2023-12-19T14:40:43.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.763+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c0c8a10 con 0x7fd81c0c7a00 2023-12-19T14:40:43.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.767+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fd80c013680 con 0x7fd81c0c7a00 2023-12-19T14:40:43.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.767+0000 7fd80a7fc700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 0x7fd7fc062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.767+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fd80c07d040 con 0x7fd81c0c7a00 2023-12-19T14:40:43.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.767+0000 7fd819d9b700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 0x7fd7fc062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:43.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.771+0000 7fd819d9b700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 0x7fd7fc062c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8100017b0 tx=0x7fd810009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.775+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd80c0acbe0 con 0x7fd81c0c7a00 2023-12-19T14:40:43.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:43.971+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81c0c5430 con 0x7fd7fc060780 2023-12-19T14:40:44.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.083+0000 7fd80a7fc700 1 -- 172.21.15.50:0/2806874040 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd810007920 con 0x7fd7fc060780 2023-12-19T14:40:44.086 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-12-19T14:40:44.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 msgr2=0x7fd7fc062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 0x7fd7fc062c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8100017b0 tx=0x7fd810009040 comp rx=0 tx=0).stop 2023-12-19T14:40:44.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 msgr2=0x7fd81c0c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00ec80 tx=0x7fd80c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 shutdown_connections 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd81c0c9880 0x7fd81c0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd7fc060780 0x7fd7fc062c30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd81c158b30 0x7fd81c0b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 --2- 172.21.15.50:0/2806874040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd81c0c7a00 0x7fd81c0c62f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 >> 172.21.15.50:0/2806874040 conn(0x7fd81c06b5b0 msgr2=0x7fd81c157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 shutdown_connections 2023-12-19T14:40:44.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.087+0000 7fd81b59e700 1 -- 172.21.15.50:0/2806874040 wait complete. 2023-12-19T14:40:44.114 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-12-19T14:40:44.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T14:40:44.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.295+0000 7f86512e3700 1 Processor -- start 2023-12-19T14:40:44.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.295+0000 7f86512e3700 1 -- start start 2023-12-19T14:40:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f86512e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 0x7f864c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f86512e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f864c0cac50 0x7f864c0af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f86512e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0cb3f0 con 0x7f864c06af50 2023-12-19T14:40:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f86512e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c0cc140 con 0x7f864c0cac50 2023-12-19T14:40:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f86512e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c06d220 con 0x7f864c0aedc0 2023-12-19T14:40:44.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f864c0cac50 0x7f864c0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 0x7f864c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864b7fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f864c06af50 0x7f864c06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:45848/0 (socket says 172.21.15.50:45848) 2023-12-19T14:40:44.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864b7fe700 1 -- 172.21.15.50:0/1429718349 learned_addr learned my addr 172.21.15.50:0/1429718349 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:44.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f864c0cac50 0x7f864c0af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:46250/0 (socket says 172.21.15.50:46250) 2023-12-19T14:40:44.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864affd700 1 -- 172.21.15.50:0/1429718349 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662199586 0 0) 0x7f864c0cb3f0 con 0x7f864c06af50 2023-12-19T14:40:44.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864affd700 1 -- 172.21.15.50:0/1429718349 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8634003570 con 0x7f864c06af50 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 -- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 msgr2=0x7f864c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 0x7f864c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 -- 172.21.15.50:0/1429718349 >> v1:172.21.15.125:6789/0 conn(0x7f864c06af50 legacy=0x7f864c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f8650ae2700 1 -- 172.21.15.50:0/1429718349 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c06b3a0 con 0x7f864c0cac50 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.299+0000 7f864bfff700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 0x7f864c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f8650ae2700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0cac50 0x7f864c0af930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8644009e60 tx=0x7f8644009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8257f388af0f9d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f864affd700 1 -- 172.21.15.50:0/1429718349 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864400e070 con 0x7f864c0cac50 2023-12-19T14:40:44.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f864affd700 1 -- 172.21.15.50:0/1429718349 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864400bab0 con 0x7f864c0cac50 2023-12-19T14:40:44.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f864affd700 1 -- 172.21.15.50:0/1429718349 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864400be70 con 0x7f864c0cac50 2023-12-19T14:40:44.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 -- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0cac50 msgr2=0x7f864c0af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0cac50 0x7f864c0af930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8644009e60 tx=0x7f8644009780 comp rx=0 tx=0).stop 2023-12-19T14:40:44.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 -- 172.21.15.50:0/1429718349 shutdown_connections 2023-12-19T14:40:44.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0cac50 0x7f864c0af930 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 --2- 172.21.15.50:0/1429718349 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0aedc0 0x7f864c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 -- 172.21.15.50:0/1429718349 >> 172.21.15.50:0/1429718349 conn(0x7f864c0cc760 msgr2=0x7f864c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:44.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 -- 172.21.15.50:0/1429718349 shutdown_connections 2023-12-19T14:40:44.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.303+0000 7f86512e3700 1 -- 172.21.15.50:0/1429718349 wait complete. 2023-12-19T14:40:44.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 Processor -- start 2023-12-19T14:40:44.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 -- start start 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 0x7f864c14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0cac50 0x7f864c0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c15ab10 con 0x7f864c06af50 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0c93e0 con 0x7f864c0aedc0 2023-12-19T14:40:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f86512e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0c4940 con 0x7f864c0cac50 2023-12-19T14:40:44.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36020/0 (socket says 172.21.15.50:36020) 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 -- 172.21.15.50:0/356098859 learned_addr learned my addr 172.21.15.50:0/356098859 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f8650ae2700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0cac50 0x7f864c0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864b7fe700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 0x7f864c14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 -- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0cac50 msgr2=0x7f864c0c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0cac50 0x7f864c0c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 -- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 msgr2=0x7f864c14d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 0x7f864c14d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 -- 172.21.15.50:0/356098859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c009420 con 0x7f864c06af50 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864b7fe700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 0x7f864c14d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:44.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.307+0000 7f864bfff700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f863c00ecb0 tx=0x7f863c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00c950 con 0x7f864c06af50 2023-12-19T14:40:44.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00cab0 con 0x7f864c06af50 2023-12-19T14:40:44.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8644009a20 con 0x7f864c06af50 2023-12-19T14:40:44.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00ce50 con 0x7f864c06af50 2023-12-19T14:40:44.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c0c4010 con 0x7f864c06af50 2023-12-19T14:40:44.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c0ca5c0 con 0x7f864c06af50 2023-12-19T14:40:44.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.311+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f863c013680 con 0x7f864c06af50 2023-12-19T14:40:44.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.315+0000 7f8633fff700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 0x7f8638062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.315+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f863c07d110 con 0x7f864c06af50 2023-12-19T14:40:44.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.315+0000 7f864b7fe700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 0x7f8638062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.315+0000 7f864b7fe700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 0x7f8638062c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c31e0 tx=0x7f8640009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.319+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f864c0ca5c0 con 0x7f864c06af50 2023-12-19T14:40:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.519+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f864c0c4a80 con 0x7f8638060750 2023-12-19T14:40:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.619+0000 7f8633fff700 1 -- 172.21.15.50:0/356098859 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f8640001d10 con 0x7f8638060750 2023-12-19T14:40:44.625 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-12-19T14:40:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 msgr2=0x7f8638062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 0x7f8638062c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c31e0 tx=0x7f8640009040 comp rx=0 tx=0).stop 2023-12-19T14:40:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 msgr2=0x7f864c0c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f863c00ecb0 tx=0x7f863c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 shutdown_connections 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f864c0aedc0 0x7f864c14d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8638060750 0x7f8638062c00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f864c0cac50 0x7f864c0c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 --2- 172.21.15.50:0/356098859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f864c06af50 0x7f864c0c3750 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 >> 172.21.15.50:0/356098859 conn(0x7f864c0cc760 msgr2=0x7f864c06e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:44.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 shutdown_connections 2023-12-19T14:40:44.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.623+0000 7f86512e3700 1 -- 172.21.15.50:0/356098859 wait complete. 2023-12-19T14:40:44.654 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-12-19T14:40:44.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T14:40:44.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 Processor -- start 2023-12-19T14:40:44.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 -- start start 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd13c0cb130 0x7fd13c0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0b0ac0 con 0x7fd13c15c220 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13c06b030 con 0x7fd13c0cb130 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.819+0000 7fd1430cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c06b210 con 0x7fd13c06a6a0 2023-12-19T14:40:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd14166a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd13c15c220 0x7fd13c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:45860/0 (socket says 172.21.15.50:45860) 2023-12-19T14:40:44.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd14166a700 1 -- 172.21.15.50:0/1408797116 learned_addr learned my addr 172.21.15.50:0/1408797116 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:40:44.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd13c0cb130 0x7fd13c0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13b7fe700 1 -- 172.21.15.50:0/1408797116 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347070425 0 0) 0x7fd13c0b0ac0 con 0x7fd13c15c220 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13b7fe700 1 -- 172.21.15.50:0/1408797116 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c003570 con 0x7fd13c15c220 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 -- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 msgr2=0x7fd13c06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 -- 172.21.15.50:0/1408797116 >> v1:172.21.15.125:6789/0 conn(0x7fd13c15c220 legacy=0x7fd13c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 -- 172.21.15.50:0/1408797116 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0ca810 con 0x7fd13c0cb130 2023-12-19T14:40:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13bfff700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:44.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd140e69700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0cb500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd1300098e0 tx=0x7fd1300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6883d68cace79d90 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13b7fe700 1 -- 172.21.15.50:0/1408797116 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130011630 con 0x7fd13c0cb130 2023-12-19T14:40:44.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13b7fe700 1 -- 172.21.15.50:0/1408797116 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130011790 con 0x7fd13c0cb130 2023-12-19T14:40:44.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.823+0000 7fd13b7fe700 1 -- 172.21.15.50:0/1408797116 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130011b30 con 0x7fd13c0cb130 2023-12-19T14:40:44.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 msgr2=0x7fd13c0cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0cb500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd1300098e0 tx=0x7fd1300091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- 172.21.15.50:0/1408797116 shutdown_connections 2023-12-19T14:40:44.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 --2- 172.21.15.50:0/1408797116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0cb500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- 172.21.15.50:0/1408797116 >> 172.21.15.50:0/1408797116 conn(0x7fd13c067bb0 msgr2=0x7fd13c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:44.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- 172.21.15.50:0/1408797116 shutdown_connections 2023-12-19T14:40:44.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- 172.21.15.50:0/1408797116 wait complete. 2023-12-19T14:40:44.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 Processor -- start 2023-12-19T14:40:44.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.827+0000 7fd1430cd700 1 -- start start 2023-12-19T14:40:44.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0b4f90 con 0x7fd13c15c220 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c1537e0 con 0x7fd13c0cb130 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0b4380 con 0x7fd13c06a6a0 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd13bfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd140e69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36030/0 (socket says 172.21.15.50:36030) 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 -- 172.21.15.50:0/4267067799 learned_addr learned my addr 172.21.15.50:0/4267067799 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 -- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 msgr2=0x7fd13c1543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c1543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 -- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 msgr2=0x7fd13c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:44.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 -- 172.21.15.50:0/4267067799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd128009420 con 0x7fd13c15c220 2023-12-19T14:40:44.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd14166a700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd134001460 tx=0x7fd134001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400a890 con 0x7fd13c15c220 2023-12-19T14:40:44.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13400a9f0 con 0x7fd13c15c220 2023-12-19T14:40:44.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400ad70 con 0x7fd13c15c220 2023-12-19T14:40:44.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd130009420 con 0x7fd13c15c220 2023-12-19T14:40:44.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.831+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c0b4940 con 0x7fd13c15c220 2023-12-19T14:40:44.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.835+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c0b5100 con 0x7fd13c15c220 2023-12-19T14:40:44.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.835+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fd134007300 con 0x7fd13c15c220 2023-12-19T14:40:44.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.835+0000 7fd138ff9700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 0x7fd120062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:44.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.839+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fd13407d150 con 0x7fd13c15c220 2023-12-19T14:40:44.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.839+0000 7fd140e69700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 0x7fd120062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.839+0000 7fd140e69700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 0x7fd120062c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd1300093f0 tx=0x7fd130017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:44.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:44.843+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd13404d260 con 0x7fd13c15c220 2023-12-19T14:40:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.043+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd13c071120 con 0x7fd120060780 2023-12-19T14:40:45.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.159+0000 7fd138ff9700 1 -- 172.21.15.50:0/4267067799 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd13c071120 con 0x7fd120060780 2023-12-19T14:40:45.165 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-12-19T14:40:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 msgr2=0x7fd120062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 0x7fd120062c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd1300093f0 tx=0x7fd130017000 comp rx=0 tx=0).stop 2023-12-19T14:40:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 msgr2=0x7fd13c0b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd134001460 tx=0x7fd134001ae0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 shutdown_connections 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd13c0cb130 0x7fd13c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd120060780 0x7fd120062c30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd13c15c220 0x7fd13c0b5960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 --2- 172.21.15.50:0/4267067799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd13c06a6a0 0x7fd13c1543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.163+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 >> 172.21.15.50:0/4267067799 conn(0x7fd13c067bb0 msgr2=0x7fd13c0ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.167+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 shutdown_connections 2023-12-19T14:40:45.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.167+0000 7fd1430cd700 1 -- 172.21.15.50:0/4267067799 wait complete. 2023-12-19T14:40:45.194 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-12-19T14:40:45.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T14:40:45.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.379+0000 7f05cc385700 1 Processor -- start 2023-12-19T14:40:45.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.379+0000 7f05cc385700 1 -- start start 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f05c40cb130 0x7f05c40cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 0x7f05c406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c415c220 0x7f05c415c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c406b030 con 0x7f05c40cb130 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c40b0ac0 con 0x7f05c406a6a0 2023-12-19T14:40:45.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05cc385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c415cbb0 con 0x7f05c415c220 2023-12-19T14:40:45.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f05c40cb130 0x7f05c40cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f05c40cb130 0x7f05c40cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36036/0 (socket says 172.21.15.50:36036) 2023-12-19T14:40:45.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 -- 172.21.15.50:0/334315126 learned_addr learned my addr 172.21.15.50:0/334315126 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:45.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca922700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c415c220 0x7f05c415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05c9920700 1 --2- 172.21.15.50:0/334315126 >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 0x7f05c406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 -- 172.21.15.50:0/334315126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c415c220 msgr2=0x7f05c415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:45.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c415c220 0x7f05c415c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 -- 172.21.15.50:0/334315126 >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 msgr2=0x7f05c406aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:45.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 --2- 172.21.15.50:0/334315126 >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 0x7f05c406aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 -- 172.21.15.50:0/334315126 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c406b210 con 0x7f05c40cb130 2023-12-19T14:40:45.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05c9920700 1 --2- 172.21.15.50:0/334315126 >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 0x7f05c406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:45.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05ca121700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c40cb130 0x7f05c40cb500 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cc930 tx=0x7f05c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4642e7970bda8551 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05c911f700 1 -- 172.21.15.50:0/334315126 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0009e70 con 0x7f05c40cb130 2023-12-19T14:40:45.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05c911f700 1 -- 172.21.15.50:0/334315126 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c000b860 con 0x7f05c40cb130 2023-12-19T14:40:45.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.383+0000 7f05c911f700 1 -- 172.21.15.50:0/334315126 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c000bc00 con 0x7f05c40cb130 2023-12-19T14:40:45.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- 172.21.15.50:0/334315126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c40cb130 msgr2=0x7f05c40cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c40cb130 0x7f05c40cb500 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cc930 tx=0x7f05c0009250 comp rx=0 tx=0).stop 2023-12-19T14:40:45.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- 172.21.15.50:0/334315126 shutdown_connections 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c415c220 0x7f05c415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 --2- 172.21.15.50:0/334315126 >> v2:172.21.15.125:3300/0 conn(0x7f05c406a6a0 0x7f05c406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 --2- 172.21.15.50:0/334315126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c40cb130 0x7f05c40cb500 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- 172.21.15.50:0/334315126 >> 172.21.15.50:0/334315126 conn(0x7f05c4067bb0 msgr2=0x7f05c40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- 172.21.15.50:0/334315126 shutdown_connections 2023-12-19T14:40:45.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- 172.21.15.50:0/334315126 wait complete. 2023-12-19T14:40:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 Processor -- start 2023-12-19T14:40:45.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.387+0000 7f05cc385700 1 -- start start 2023-12-19T14:40:45.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05c406a6a0 0x7f05c40b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c40cb130 0x7f05c4151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40b2e40 con 0x7f05c415c220 2023-12-19T14:40:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40b4350 con 0x7f05c406a6a0 2023-12-19T14:40:45.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c4169ec0 con 0x7f05c40cb130 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36048/0 (socket says 172.21.15.50:36048) 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 -- 172.21.15.50:0/3689111739 learned_addr learned my addr 172.21.15.50:0/3689111739 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca121700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05c406a6a0 0x7f05c40b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05c9920700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c40cb130 0x7f05c4151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 -- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c40cb130 msgr2=0x7f05c4151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c40cb130 0x7f05c4151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 -- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05c406a6a0 msgr2=0x7f05c40b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05c406a6a0 0x7f05c40b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 -- 172.21.15.50:0/3689111739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c00094f0 con 0x7f05c415c220 2023-12-19T14:40:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ca922700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009df0 tx=0x7f05bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00caa0 con 0x7f05c415c220 2023-12-19T14:40:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc00f040 con 0x7f05c415c220 2023-12-19T14:40:45.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00ce70 con 0x7f05c415c220 2023-12-19T14:40:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c40b3050 con 0x7f05c415c220 2023-12-19T14:40:45.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.391+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c40b3690 con 0x7f05c415c220 2023-12-19T14:40:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.395+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c40cd7f0 con 0x7f05c415c220 2023-12-19T14:40:45.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.395+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f05bc014690 con 0x7f05c415c220 2023-12-19T14:40:45.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.399+0000 7f05ba7fc700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 0x7f05a8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.399+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f05bc07d700 con 0x7f05c415c220 2023-12-19T14:40:45.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.399+0000 7f05ca121700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 0x7f05a8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.399+0000 7f05ca121700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 0x7f05a8062c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cc930 tx=0x7f05c0017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.403+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f05c40cd7f0 con 0x7f05c415c220 2023-12-19T14:40:45.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.599+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f05c40cd7f0 con 0x7f05a8060750 2023-12-19T14:40:45.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.707+0000 7f05ba7fc700 1 -- 172.21.15.50:0/3689111739 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f05c40cd7f0 con 0x7f05a8060750 2023-12-19T14:40:45.711 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-12-19T14:40:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 msgr2=0x7f05a8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 0x7f05a8062c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05c40cc930 tx=0x7f05c0017000 comp rx=0 tx=0).stop 2023-12-19T14:40:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 msgr2=0x7f05c415a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009df0 tx=0x7f05bc00b040 comp rx=0 tx=0).stop 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 shutdown_connections 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f05a8060750 0x7f05a8062c00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05c415c220 0x7f05c415a7f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05c40cb130 0x7f05c4151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 --2- 172.21.15.50:0/3689111739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05c406a6a0 0x7f05c40b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 >> 172.21.15.50:0/3689111739 conn(0x7f05c4067bb0 msgr2=0x7f05c40cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 shutdown_connections 2023-12-19T14:40:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.711+0000 7f05cc385700 1 -- 172.21.15.50:0/3689111739 wait complete. 2023-12-19T14:40:45.741 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-12-19T14:40:45.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T14:40:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 Processor -- start 2023-12-19T14:40:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 -- start start 2023-12-19T14:40:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad100aedc0 0x7fad100af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad100af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1006b3d0 con 0x7fad1006af50 2023-12-19T14:40:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad100cc140 con 0x7fad100aedc0 2023-12-19T14:40:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1660c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad100cb3f0 con 0x7fad100cac50 2023-12-19T14:40:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.931+0000 7fad1560a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad100aedc0 0x7fad100af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fad100aedc0 0x7fad100af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36054/0 (socket says 172.21.15.50:36054) 2023-12-19T14:40:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad14e09700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fad1006af50 0x7fad1006d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:45862/0 (socket says 172.21.15.50:45862) 2023-12-19T14:40:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 -- 172.21.15.50:0/1398309740 learned_addr learned my addr 172.21.15.50:0/1398309740 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad15e0b700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad100af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad07fff700 1 -- 172.21.15.50:0/1398309740 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483143057 0 0) 0x7fad00001110 con 0x7fad1006af50 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad07fff700 1 -- 172.21.15.50:0/1398309740 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc003570 con 0x7fad1006af50 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 -- 172.21.15.50:0/1398309740 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 msgr2=0x7fad100af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad100af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 -- 172.21.15.50:0/1398309740 >> v1:172.21.15.125:6789/0 conn(0x7fad1006af50 legacy=0x7fad1006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 -- 172.21.15.50:0/1398309740 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1006d220 con 0x7fad100aedc0 2023-12-19T14:40:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1560a700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad100aedc0 0x7fad100af190 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0099b0 tx=0x7fad0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd19dc68255e7c5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad07fff700 1 -- 172.21.15.50:0/1398309740 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c00b680 con 0x7fad100aedc0 2023-12-19T14:40:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad07fff700 1 -- 172.21.15.50:0/1398309740 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c00b7e0 con 0x7fad100aedc0 2023-12-19T14:40:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad07fff700 1 -- 172.21.15.50:0/1398309740 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c00bb80 con 0x7fad100aedc0 2023-12-19T14:40:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 -- 172.21.15.50:0/1398309740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad100aedc0 msgr2=0x7fad100af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad100aedc0 0x7fad100af190 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0099b0 tx=0x7fad0c009250 comp rx=0 tx=0).stop 2023-12-19T14:40:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 -- 172.21.15.50:0/1398309740 shutdown_connections 2023-12-19T14:40:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad100af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 --2- 172.21.15.50:0/1398309740 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad100aedc0 0x7fad100af190 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.935+0000 7fad1660c700 1 -- 172.21.15.50:0/1398309740 >> 172.21.15.50:0/1398309740 conn(0x7fad100cc760 msgr2=0x7fad100ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.939+0000 7fad1660c700 1 -- 172.21.15.50:0/1398309740 shutdown_connections 2023-12-19T14:40:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.939+0000 7fad1660c700 1 -- 172.21.15.50:0/1398309740 wait complete. 2023-12-19T14:40:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.939+0000 7fad1660c700 1 Processor -- start 2023-12-19T14:40:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.939+0000 7fad1660c700 1 -- start start 2023-12-19T14:40:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad100aedc0 0x7fad100c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad1016c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad100c5fe0 con 0x7fad1006af50 2023-12-19T14:40:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1016d1c0 con 0x7fad100aedc0 2023-12-19T14:40:45.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1016d4a0 con 0x7fad100cac50 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36058/0 (socket says 172.21.15.50:36058) 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 -- 172.21.15.50:0/345302798 learned_addr learned my addr 172.21.15.50:0/345302798 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad15e0b700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad1016c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad14e09700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad100aedc0 0x7fad100c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 -- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 msgr2=0x7fad1016c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad1016c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 -- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad100aedc0 msgr2=0x7fad100c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad100aedc0 0x7fad100c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 -- 172.21.15.50:0/345302798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0094f0 con 0x7fad1006af50 2023-12-19T14:40:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1560a700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009290 tx=0x7fad0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c00b950 con 0x7fad1006af50 2023-12-19T14:40:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c00bab0 con 0x7fad1006af50 2023-12-19T14:40:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad100c5c70 con 0x7fad1006af50 2023-12-19T14:40:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c01a680 con 0x7fad1006af50 2023-12-19T14:40:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.943+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad100c58d0 con 0x7fad1006af50 2023-12-19T14:40:45.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.947+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1004be40 con 0x7fad1006af50 2023-12-19T14:40:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.947+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fad0c01aa70 con 0x7fad1006af50 2023-12-19T14:40:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.947+0000 7fad057fa700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 0x7facf4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.947+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fad0c07e9e0 con 0x7fad1006af50 2023-12-19T14:40:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.951+0000 7fad14e09700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 0x7facf4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:45.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.951+0000 7fad14e09700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 0x7facf4062c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad1016c660 tx=0x7fad00007a00 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:45.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:45.955+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad0c04eab0 con 0x7fad1006af50 2023-12-19T14:40:46.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.147+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fad100c6120 con 0x7facf4060750 2023-12-19T14:40:46.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.255+0000 7fad057fa700 1 -- 172.21.15.50:0/345302798 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fad00005960 con 0x7facf4060750 2023-12-19T14:40:46.258 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-12-19T14:40:46.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.255+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 msgr2=0x7facf4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.255+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 0x7facf4062c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad1016c660 tx=0x7fad00007a00 comp rx=0 tx=0).stop 2023-12-19T14:40:46.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.255+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 msgr2=0x7fad100c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.255+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fad0c009290 tx=0x7fad0c017000 comp rx=0 tx=0).stop 2023-12-19T14:40:46.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 shutdown_connections 2023-12-19T14:40:46.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad100aedc0 0x7fad100c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7facf4060750 0x7facf4062c00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad100cac50 0x7fad1016c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 --2- 172.21.15.50:0/345302798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad1006af50 0x7fad100c6470 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 >> 172.21.15.50:0/345302798 conn(0x7fad100cc760 msgr2=0x7fad100b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:46.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 shutdown_connections 2023-12-19T14:40:46.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.259+0000 7fad1660c700 1 -- 172.21.15.50:0/345302798 wait complete. 2023-12-19T14:40:46.292 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-12-19T14:40:46.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T14:40:46.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.459+0000 7f7078b85700 1 Processor -- start 2023-12-19T14:40:46.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.459+0000 7f7078b85700 1 -- start start 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7078b85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70740cb130 0x7f70740cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7078b85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f707406a6a0 0x7f707406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7078b85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70740b0ac0 con 0x7f707415c220 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7078b85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707406b030 con 0x7f707406a6a0 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7078b85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707406b210 con 0x7f70740cb130 2023-12-19T14:40:46.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7071d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f707406a6a0 0x7f707406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7071d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f707406a6a0 0x7f707406aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44112/0 (socket says 172.21.15.50:44112) 2023-12-19T14:40:46.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f7071d9b700 1 -- 172.21.15.50:0/1055526681 learned_addr learned my addr 172.21.15.50:0/1055526681 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.463+0000 7f707259c700 1 --2- 172.21.15.50:0/1055526681 >> v2:172.21.15.125:3300/0 conn(0x7f70740cb130 0x7f70740cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7071d9b700 1 -- 172.21.15.50:0/1055526681 >> v2:172.21.15.125:3300/0 conn(0x7f70740cb130 msgr2=0x7f70740cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7071d9b700 1 --2- 172.21.15.50:0/1055526681 >> v2:172.21.15.125:3300/0 conn(0x7f70740cb130 0x7f70740cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7071d9b700 1 -- 172.21.15.50:0/1055526681 >> v1:172.21.15.125:6789/0 conn(0x7f707415c220 legacy=0x7f707406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7071d9b700 1 -- 172.21.15.50:0/1055526681 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70740ca810 con 0x7f707406a6a0 2023-12-19T14:40:46.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7071d9b700 1 --2- 172.21.15.50:0/1055526681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707406a6a0 0x7f707406aaf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f70740694e0 tx=0x7f7068009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e1813dc05533d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f707159a700 1 -- 172.21.15.50:0/1055526681 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800b860 con 0x7f707406a6a0 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f707159a700 1 -- 172.21.15.50:0/1055526681 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706800b9c0 con 0x7f707406a6a0 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f707159a700 1 -- 172.21.15.50:0/1055526681 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800bd60 con 0x7f707406a6a0 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 -- 172.21.15.50:0/1055526681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707406a6a0 msgr2=0x7f707406aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 --2- 172.21.15.50:0/1055526681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707406a6a0 0x7f707406aaf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f70740694e0 tx=0x7f7068009250 comp rx=0 tx=0).stop 2023-12-19T14:40:46.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 -- 172.21.15.50:0/1055526681 shutdown_connections 2023-12-19T14:40:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 --2- 172.21.15.50:0/1055526681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707406a6a0 0x7f707406aaf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 --2- 172.21.15.50:0/1055526681 >> v2:172.21.15.125:3300/0 conn(0x7f70740cb130 0x7f70740cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 -- 172.21.15.50:0/1055526681 >> 172.21.15.50:0/1055526681 conn(0x7f7074067bb0 msgr2=0x7f70740cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 -- 172.21.15.50:0/1055526681 shutdown_connections 2023-12-19T14:40:46.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.467+0000 7f7078b85700 1 -- 172.21.15.50:0/1055526681 wait complete. 2023-12-19T14:40:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 Processor -- start 2023-12-19T14:40:46.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 -- start start 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 0x7f70740c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 0x7f7074147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707416a310 con 0x7f70740cb130 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707416a890 con 0x7f707406a6a0 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7078b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707416ab70 con 0x7f707415c220 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44118/0 (socket says 172.21.15.50:44118) 2023-12-19T14:40:46.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 -- 172.21.15.50:0/749100558 learned_addr learned my addr 172.21.15.50:0/749100558 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f707259c700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 0x7f70740c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 -- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 msgr2=0x7f7074147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7072d9d700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 0x7f7074147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 0x7f7074147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 -- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 msgr2=0x7f70740c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 0x7f70740c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 -- 172.21.15.50:0/749100558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680094f0 con 0x7f70740cb130 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7072d9d700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 0x7f7074147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f707259c700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 0x7f70740c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.471+0000 7f7071d9b700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f707406f300 tx=0x7f7068017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068011e60 con 0x7f70740cb130 2023-12-19T14:40:46.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706801a630 con 0x7f70740cb130 2023-12-19T14:40:46.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706801a9f0 con 0x7f70740cb130 2023-12-19T14:40:46.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707416ae50 con 0x7f70740cb130 2023-12-19T14:40:46.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707416b410 con 0x7f70740cb130 2023-12-19T14:40:46.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707416b630 con 0x7f70740cb130 2023-12-19T14:40:46.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.475+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f706802d440 con 0x7f70740cb130 2023-12-19T14:40:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.479+0000 7f70627fc700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 0x7f7054062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:46.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.479+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f706807e8b0 con 0x7f70740cb130 2023-12-19T14:40:46.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.479+0000 7f707259c700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 0x7f7054062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.483+0000 7f707259c700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 0x7f7054062c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7064006fd0 tx=0x7f7064008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:46.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.487+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f706804e9e0 con 0x7f70740cb130 2023-12-19T14:40:46.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.683+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7074071120 con 0x7f7054060750 2023-12-19T14:40:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.783+0000 7f70627fc700 1 -- 172.21.15.50:0/749100558 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7074071120 con 0x7f7054060750 2023-12-19T14:40:46.790 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-12-19T14:40:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 msgr2=0x7f7054062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 0x7f7054062c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7064006fd0 tx=0x7f7064008040 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 msgr2=0x7f7074169dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f707406f300 tx=0x7f7068017000 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 shutdown_connections 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7054060750 0x7f7054062c00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f707415c220 0x7f7074147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70740cb130 0x7f7074169dd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 --2- 172.21.15.50:0/749100558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f707406a6a0 0x7f70740c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 >> 172.21.15.50:0/749100558 conn(0x7f7074067bb0 msgr2=0x7f70740d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 shutdown_connections 2023-12-19T14:40:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.787+0000 7f7078b85700 1 -- 172.21.15.50:0/749100558 wait complete. 2023-12-19T14:40:46.817 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-12-19T14:40:46.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T14:40:47.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.995+0000 7fdf3b6cd700 1 Processor -- start 2023-12-19T14:40:47.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.999+0000 7fdf3b6cd700 1 -- start start 2023-12-19T14:40:47.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.999+0000 7fdf3b6cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf34158e50 0x7fdf340c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:46.999+0000 7fdf3b6cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf34159220 0x7fdf34068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf3b6cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3406d0d0 con 0x7fdf34069310 2023-12-19T14:40:47.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf3b6cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3406c7c0 con 0x7fdf34158e50 2023-12-19T14:40:47.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf3b6cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf340c7b40 con 0x7fdf34159220 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf34158e50 0x7fdf340c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdf34158e50 0x7fdf340c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44122/0 (socket says 172.21.15.50:44122) 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39c6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdf34069310 0x7fdf3415cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38840/0 (socket says 172.21.15.50:38840) 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 -- 172.21.15.50:0/1488525131 learned_addr learned my addr 172.21.15.50:0/1488525131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf38c68700 1 --2- 172.21.15.50:0/1488525131 >> v2:172.21.15.125:3300/0 conn(0x7fdf34159220 0x7fdf34068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 -- 172.21.15.50:0/1488525131 >> v2:172.21.15.125:3300/0 conn(0x7fdf34159220 msgr2=0x7fdf34068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:40:47.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 --2- 172.21.15.50:0/1488525131 >> v2:172.21.15.125:3300/0 conn(0x7fdf34159220 0x7fdf34068380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 -- 172.21.15.50:0/1488525131 >> v1:172.21.15.50:6789/0 conn(0x7fdf34069310 legacy=0x7fdf3415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 -- 172.21.15.50:0/1488525131 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3406ca80 con 0x7fdf34158e50 2023-12-19T14:40:47.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf39469700 1 --2- 172.21.15.50:0/1488525131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf340c72c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdf3406f030 tx=0x7fdf30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a822e42af4daaf16 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:47.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf2bfff700 1 -- 172.21.15.50:0/1488525131 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30009e70 con 0x7fdf34158e50 2023-12-19T14:40:47.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf2bfff700 1 -- 172.21.15.50:0/1488525131 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3000b860 con 0x7fdf34158e50 2023-12-19T14:40:47.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.003+0000 7fdf2bfff700 1 -- 172.21.15.50:0/1488525131 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3000bc20 con 0x7fdf34158e50 2023-12-19T14:40:47.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1488525131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 msgr2=0x7fdf340c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1488525131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf340c72c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdf3406f030 tx=0x7fdf30009250 comp rx=0 tx=0).stop 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1488525131 shutdown_connections 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1488525131 >> v2:172.21.15.125:3300/0 conn(0x7fdf34159220 0x7fdf34068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1488525131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf340c72c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1488525131 >> 172.21.15.50:0/1488525131 conn(0x7fdf340c8db0 msgr2=0x7fdf340c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1488525131 shutdown_connections 2023-12-19T14:40:47.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1488525131 wait complete. 2023-12-19T14:40:47.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 Processor -- start 2023-12-19T14:40:47.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.007+0000 7fdf3b6cd700 1 -- start start 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf34069310 0x7fdf340c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 0x7fdf3416aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340b0c10 con 0x7fdf34158e50 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340c4190 con 0x7fdf34159220 2023-12-19T14:40:47.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340c38f0 con 0x7fdf34069310 2023-12-19T14:40:47.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf39c6a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 0x7fdf3416aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44134/0 (socket says 172.21.15.50:44134) 2023-12-19T14:40:47.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf39469700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf34069310 0x7fdf340c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 -- 172.21.15.50:0/1427762815 learned_addr learned my addr 172.21.15.50:0/1427762815 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 -- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf34069310 msgr2=0x7fdf340c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf34069310 0x7fdf340c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 -- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 msgr2=0x7fdf3416aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 0x7fdf3416aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 -- 172.21.15.50:0/1427762815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf300094f0 con 0x7fdf34158e50 2023-12-19T14:40:47.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf39c6a700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 0x7fdf3416aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf38c68700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400e930 tx=0x7fdf240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24013680 con 0x7fdf34158e50 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf240137e0 con 0x7fdf34158e50 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf340c3bd0 con 0x7fdf34158e50 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.011+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24013b80 con 0x7fdf34158e50 2023-12-19T14:40:47.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.015+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf34169a70 con 0x7fdf34158e50 2023-12-19T14:40:47.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.015+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf34071120 con 0x7fdf34158e50 2023-12-19T14:40:47.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.015+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fdf2400c3e0 con 0x7fdf34158e50 2023-12-19T14:40:47.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.019+0000 7fdf297fa700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 0x7fdf18062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:40:47.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.019+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fdf2407cd90 con 0x7fdf34158e50 2023-12-19T14:40:47.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.019+0000 7fdf39469700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 0x7fdf18062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:40:47.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.019+0000 7fdf39469700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 0x7fdf18062c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdf3406f030 tx=0x7fdf30017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:40:47.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.027+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdf2404cec0 con 0x7fdf34158e50 2023-12-19T14:40:47.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.219+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf3406cd60 con 0x7fdf18060780 2023-12-19T14:40:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.315+0000 7fdf297fa700 1 -- 172.21.15.50:0/1427762815 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdf3406cd60 con 0x7fdf18060780 2023-12-19T14:40:47.319 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-12-19T14:40:47.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 msgr2=0x7fdf18062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 0x7fdf18062c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdf3406f030 tx=0x7fdf30017000 comp rx=0 tx=0).stop 2023-12-19T14:40:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 msgr2=0x7fdf341690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:40:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400e930 tx=0x7fdf240091a0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 shutdown_connections 2023-12-19T14:40:47.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf34159220 0x7fdf3416aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf18060780 0x7fdf18062c30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf34158e50 0x7fdf341690d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 --2- 172.21.15.50:0/1427762815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf34069310 0x7fdf340c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 >> 172.21.15.50:0/1427762815 conn(0x7fdf340c8db0 msgr2=0x7fdf340ab080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 shutdown_connections 2023-12-19T14:40:47.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:40:47.319+0000 7fdf3b6cd700 1 -- 172.21.15.50:0/1427762815 wait complete. 2023-12-19T14:40:47.348 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-12-19T14:41:00.036 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:41:00.030+0000 7faec145f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:41:00.036 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:41:00.030+0000 7faec145f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:41:00.036 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:41:00.030+0000 7faec145f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:41:00.037 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:41:00.030+0000 7faec145f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:41:00.037 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:41:00.030+0000 7faec145f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:41:05.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:41:05.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:41:35.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:41:35.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:42:05.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:42:05.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:42:35.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:42:35.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:43:05.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:43:05.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:43:35.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:43:35.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:44:00.219 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:44:00.214+0000 7faec8c6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:44:00.219 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:44:00.214+0000 7faec8c6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:44:00.220 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:44:00.214+0000 7faec8c6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:44:00.220 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:44:00.214+0000 7faec8c6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:44:00.220 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:44:00.214+0000 7faec8c6e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:44:05.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:44:05.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:44:35.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:44:35.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:45:05.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:45:05.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:45:35.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:45:35.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:46:05.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:46:05.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:46:36.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:46:36.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:47:06.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:47:06.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:47:36.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:47:36.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:48:06.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:48:06.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:48:36.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:48:36.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:49:06.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:49:06.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:49:36.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:49:36.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:50:06.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:50:06.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:50:36.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:50:36.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:51:06.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:51:06.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:51:21.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T14:51:21.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.519+0000 7f226ce2e700 1 Processor -- start 2023-12-19T14:51:21.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.519+0000 7f226ce2e700 1 -- start start 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22680cc4e0 0x7f2268156350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2268156890 0x7f22680c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22681572d0 con 0x7f2268156890 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2268156d10 con 0x7f22680cc4e0 2023-12-19T14:51:21.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226ce2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2268156ff0 con 0x7f22681538c0 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226659c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f2265d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22680cc4e0 0x7f2268156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f2266d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2268156890 0x7f22680c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226659c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39212/0 (socket says 172.21.15.50:39212) 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f2265d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22680cc4e0 0x7f2268156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:44444/0 (socket says 172.21.15.50:44444) 2023-12-19T14:51:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.523+0000 7f226659c700 1 -- 172.21.15.50:0/3086505104 learned_addr learned my addr 172.21.15.50:0/3086505104 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 -- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 msgr2=0x7f226806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:21.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 -- 172.21.15.50:0/3086505104 >> v2:172.21.15.50:3300/0 conn(0x7f2268156890 msgr2=0x7f22680c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:21.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 --2- 172.21.15.50:0/3086505104 >> v2:172.21.15.50:3300/0 conn(0x7f2268156890 0x7f22680c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 -- 172.21.15.50:0/3086505104 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680ace10 con 0x7f22680cc4e0 2023-12-19T14:51:21.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f226659c700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:21.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.527+0000 7f2265d9b700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f2268156350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f22680680c0 tx=0x7f225c00d630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccb76c99899b5359 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226559a700 1 -- 172.21.15.50:0/3086505104 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c012070 con 0x7f22680cc4e0 2023-12-19T14:51:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226559a700 1 -- 172.21.15.50:0/3086505104 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00d2e0 con 0x7f22680cc4e0 2023-12-19T14:51:21.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226559a700 1 -- 172.21.15.50:0/3086505104 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00f900 con 0x7f22680cc4e0 2023-12-19T14:51:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 -- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 msgr2=0x7f2268156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:21.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f2268156350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f22680680c0 tx=0x7f225c00d630 comp rx=0 tx=0).stop 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 -- 172.21.15.50:0/3086505104 shutdown_connections 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 --2- 172.21.15.50:0/3086505104 >> v2:172.21.15.50:3300/0 conn(0x7f2268156890 0x7f22680c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f2268156350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 --2- 172.21.15.50:0/3086505104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f226806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 -- 172.21.15.50:0/3086505104 >> 172.21.15.50:0/3086505104 conn(0x7f2268067bb0 msgr2=0x7f22680cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 -- 172.21.15.50:0/3086505104 shutdown_connections 2023-12-19T14:51:21.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.531+0000 7f226ce2e700 1 -- 172.21.15.50:0/3086505104 wait complete. 2023-12-19T14:51:21.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 Processor -- start 2023-12-19T14:51:21.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 -- start start 2023-12-19T14:51:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f22681546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f22681510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268151a80 con 0x7f22681515e0 2023-12-19T14:51:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22680aea30 con 0x7f22680cc4e0 2023-12-19T14:51:21.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226ce2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22680af850 con 0x7f22681538c0 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2266d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226659c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f22681546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f22681510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2266d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48736/0 (socket says 172.21.15.50:48736) 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2266d9d700 1 -- 172.21.15.50:0/66362851 learned_addr learned my addr 172.21.15.50:0/66362851 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f226659c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f22681546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:44460/0 (socket says 172.21.15.50:44460) 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 -- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 msgr2=0x7f22681546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f22681546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 -- 172.21.15.50:0/66362851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 msgr2=0x7f22680afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2265d9b700 1 -- 172.21.15.50:0/66362851 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580094f0 con 0x7f22681538c0 2023-12-19T14:51:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.535+0000 7f2266d9d700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f2265d9b700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f22681510a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f225c019000 tx=0x7f225c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00c840 con 0x7f22681538c0 2023-12-19T14:51:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00c9c0 con 0x7f22681538c0 2023-12-19T14:51:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00cd80 con 0x7f22681538c0 2023-12-19T14:51:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c00d8d0 con 0x7f22681538c0 2023-12-19T14:51:21.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22680aef20 con 0x7f22681538c0 2023-12-19T14:51:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.539+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22680af9c0 con 0x7f22681538c0 2023-12-19T14:51:21.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.543+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f225c020620 con 0x7f22681538c0 2023-12-19T14:51:21.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.543+0000 7f22567fc700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 0x7f2248062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:21.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.543+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f225c082630 con 0x7f22681538c0 2023-12-19T14:51:21.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.543+0000 7f226659c700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 0x7f2248062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:21.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.547+0000 7f226659c700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 0x7f2248062cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2258006d20 tx=0x7f2258009250 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:21.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.551+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f225c052610 con 0x7f22681538c0 2023-12-19T14:51:21.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.751+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f22681596c0 con 0x7f2248060820 2023-12-19T14:51:21.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.879+0000 7f22567fc700 1 -- 172.21.15.50:0/66362851 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f22681596c0 con 0x7f2248060820 2023-12-19T14:51:21.883 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-12-19T14:51:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 msgr2=0x7f2248062cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 0x7f2248062cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2258006d20 tx=0x7f2258009250 comp rx=0 tx=0).stop 2023-12-19T14:51:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 msgr2=0x7f22681510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:21.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f22681510a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f225c019000 tx=0x7f225c01c000 comp rx=0 tx=0).stop 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 shutdown_connections 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2248060820 0x7f2248062cd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22681515e0 0x7f22680afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22681538c0 0x7f22681510a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.883+0000 7f226ce2e700 1 --2- 172.21.15.50:0/66362851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22680cc4e0 0x7f22681546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.887+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 >> 172.21.15.50:0/66362851 conn(0x7f2268067bb0 msgr2=0x7f22680cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.887+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 shutdown_connections 2023-12-19T14:51:21.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:21.887+0000 7f226ce2e700 1 -- 172.21.15.50:0/66362851 wait complete. 2023-12-19T14:51:21.910 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-12-19T14:51:21.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T14:51:22.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.087+0000 7f56c88a3700 1 Processor -- start 2023-12-19T14:51:22.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 -- start start 2023-12-19T14:51:22.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 0x7f56c40addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 0x7f56c415b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c40c9230 con 0x7f56c406ade0 2023-12-19T14:51:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c415b680 con 0x7f56c40ad9f0 2023-12-19T14:51:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c88a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c406da90 con 0x7f56c40c8db0 2023-12-19T14:51:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f56c406ade0 0x7f56c40b0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54276/0 (socket says 172.21.15.50:54276) 2023-12-19T14:51:22.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c3fff700 1 -- 172.21.15.50:0/465511834 learned_addr learned my addr 172.21.15.50:0/465511834 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:22.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c37fe700 1 --2- 172.21.15.50:0/465511834 >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 0x7f56c40addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56bbfff700 1 --2- 172.21.15.50:0/465511834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 0x7f56c415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c2ffd700 1 -- 172.21.15.50:0/465511834 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246517734 0 0) 0x7f56c40c9230 con 0x7f56c406ade0 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.091+0000 7f56c2ffd700 1 -- 172.21.15.50:0/465511834 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b0003570 con 0x7f56c406ade0 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56bbfff700 1 -- 172.21.15.50:0/465511834 >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 msgr2=0x7f56c40addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56bbfff700 1 --2- 172.21.15.50:0/465511834 >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 0x7f56c40addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56bbfff700 1 -- 172.21.15.50:0/465511834 >> v1:172.21.15.50:6789/0 conn(0x7f56c406ade0 legacy=0x7f56c40b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56bbfff700 1 -- 172.21.15.50:0/465511834 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c406a620 con 0x7f56c40c8db0 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c37fe700 1 --2- 172.21.15.50:0/465511834 >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 0x7f56c40addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56bbfff700 1 --2- 172.21.15.50:0/465511834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 0x7f56c415b140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0098e0 tx=0x7f56ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f591045e312b6cd8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c2ffd700 1 -- 172.21.15.50:0/465511834 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac011630 con 0x7f56c40c8db0 2023-12-19T14:51:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c2ffd700 1 -- 172.21.15.50:0/465511834 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac011790 con 0x7f56c40c8db0 2023-12-19T14:51:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c2ffd700 1 -- 172.21.15.50:0/465511834 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac011b30 con 0x7f56c40c8db0 2023-12-19T14:51:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 -- 172.21.15.50:0/465511834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 msgr2=0x7f56c415b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 --2- 172.21.15.50:0/465511834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 0x7f56c415b140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0098e0 tx=0x7f56ac0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 -- 172.21.15.50:0/465511834 shutdown_connections 2023-12-19T14:51:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 --2- 172.21.15.50:0/465511834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c40c8db0 0x7f56c415b140 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 --2- 172.21.15.50:0/465511834 >> v2:172.21.15.125:3300/0 conn(0x7f56c40ad9f0 0x7f56c40addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 -- 172.21.15.50:0/465511834 >> 172.21.15.50:0/465511834 conn(0x7f56c4067bb0 msgr2=0x7f56c40c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:22.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 -- 172.21.15.50:0/465511834 shutdown_connections 2023-12-19T14:51:22.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.095+0000 7f56c88a3700 1 -- 172.21.15.50:0/465511834 wait complete. 2023-12-19T14:51:22.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 Processor -- start 2023-12-19T14:51:22.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 -- start start 2023-12-19T14:51:22.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c40ad9f0 0x7f56c40a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40aa5b0 con 0x7f56c40c8db0 2023-12-19T14:51:22.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40ab3d0 con 0x7f56c40ad9f0 2023-12-19T14:51:22.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c88a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40c0b60 con 0x7f56c406ade0 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39250/0 (socket says 172.21.15.50:39250) 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c37fe700 1 -- 172.21.15.50:0/1201657766 learned_addr learned my addr 172.21.15.50:0/1201657766 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.099+0000 7f56c3fff700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48738/0 (socket says 172.21.15.50:48738) 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56bbfff700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c40ad9f0 0x7f56c40a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 -- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 msgr2=0x7f56c40c1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 -- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c40ad9f0 msgr2=0x7f56c40a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c40ad9f0 0x7f56c40a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 -- 172.21.15.50:0/1201657766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b4009420 con 0x7f56c40c8db0 2023-12-19T14:51:22.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c37fe700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:22.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c3fff700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00cbc0 tx=0x7f56bc001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00a7e0 con 0x7f56c40c8db0 2023-12-19T14:51:22.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc00a940 con 0x7f56c40c8db0 2023-12-19T14:51:22.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00acc0 con 0x7f56c40c8db0 2023-12-19T14:51:22.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac009420 con 0x7f56c40c8db0 2023-12-19T14:51:22.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.103+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c40aaaa0 con 0x7f56c40c8db0 2023-12-19T14:51:22.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.107+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c40ab540 con 0x7f56c40c8db0 2023-12-19T14:51:22.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.107+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f56bc007300 con 0x7f56c40c8db0 2023-12-19T14:51:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.107+0000 7f56bb7fe700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 0x7f56a4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.107+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f56bc07d170 con 0x7f56c40c8db0 2023-12-19T14:51:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.107+0000 7f56c37fe700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 0x7f56a4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.111+0000 7f56c37fe700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 0x7f56a4062c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56b400f680 tx=0x7f56b40091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.111+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f56bc050c60 con 0x7f56c40c8db0 2023-12-19T14:51:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.307+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f56c404bee0 con 0x7f56a4060750 2023-12-19T14:51:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.439+0000 7f56bb7fe700 1 -- 172.21.15.50:0/1201657766 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f56c404bee0 con 0x7f56a4060750 2023-12-19T14:51:22.443 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 msgr2=0x7f56a4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 0x7f56a4062c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56b400f680 tx=0x7f56b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 msgr2=0x7f56c40a8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00cbc0 tx=0x7f56bc001a20 comp rx=0 tx=0).stop 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 shutdown_connections 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c40ad9f0 0x7f56c40a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f56a4060750 0x7f56a4062c00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56c40c8db0 0x7f56c40a8d90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 --2- 172.21.15.50:0/1201657766 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f56c406ade0 0x7f56c40c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 >> 172.21.15.50:0/1201657766 conn(0x7f56c4067bb0 msgr2=0x7f56c40acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 shutdown_connections 2023-12-19T14:51:22.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.443+0000 7f56c88a3700 1 -- 172.21.15.50:0/1201657766 wait complete. 2023-12-19T14:51:22.474 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-12-19T14:51:22.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T14:51:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.647+0000 7f8da3f45700 1 Processor -- start 2023-12-19T14:51:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 -- start start 2023-12-19T14:51:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 0x7f8d9c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d9c068a90 0x7f8d9c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c06d0d0 con 0x7f8d9c159220 2023-12-19T14:51:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9c06c7c0 con 0x7f8d9c068a90 2023-12-19T14:51:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.651+0000 7f8da3f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0c7b40 con 0x7f8d9c158e50 2023-12-19T14:51:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 0x7f8d9c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da14e0700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8d9c159220 0x7f8d9c068380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:53136/0 (socket says 172.21.15.50:53136) 2023-12-19T14:51:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da14e0700 1 -- 172.21.15.50:0/3216673262 learned_addr learned my addr 172.21.15.50:0/3216673262 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da24e2700 1 --2- 172.21.15.50:0/3216673262 >> v2:172.21.15.125:3300/0 conn(0x7f8d9c068a90 0x7f8d9c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 -- 172.21.15.50:0/3216673262 >> v2:172.21.15.125:3300/0 conn(0x7f8d9c068a90 msgr2=0x7f8d9c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 --2- 172.21.15.50:0/3216673262 >> v2:172.21.15.125:3300/0 conn(0x7f8d9c068a90 0x7f8d9c06d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da0cdf700 1 -- 172.21.15.50:0/3216673262 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264631634 0 0) 0x7f8d9c06d0d0 con 0x7f8d9c159220 2023-12-19T14:51:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 -- 172.21.15.50:0/3216673262 >> v1:172.21.15.125:6789/0 conn(0x7f8d9c159220 legacy=0x7f8d9c068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 -- 172.21.15.50:0/3216673262 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c06ca80 con 0x7f8d9c158e50 2023-12-19T14:51:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da1ce1700 1 --2- 172.21.15.50:0/3216673262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 0x7f8d9c0c72c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098e0 tx=0x7f8d980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba02f383478504b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da0cdf700 1 -- 172.21.15.50:0/3216673262 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800b640 con 0x7f8d9c158e50 2023-12-19T14:51:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da0cdf700 1 -- 172.21.15.50:0/3216673262 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9800b7a0 con 0x7f8d9c158e50 2023-12-19T14:51:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.655+0000 7f8da0cdf700 1 -- 172.21.15.50:0/3216673262 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800bb40 con 0x7f8d9c158e50 2023-12-19T14:51:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- 172.21.15.50:0/3216673262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 msgr2=0x7f8d9c0c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 --2- 172.21.15.50:0/3216673262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 0x7f8d9c0c72c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098e0 tx=0x7f8d980091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- 172.21.15.50:0/3216673262 shutdown_connections 2023-12-19T14:51:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 --2- 172.21.15.50:0/3216673262 >> v2:172.21.15.125:3300/0 conn(0x7f8d9c068a90 0x7f8d9c06d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 --2- 172.21.15.50:0/3216673262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c158e50 0x7f8d9c0c72c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- 172.21.15.50:0/3216673262 >> 172.21.15.50:0/3216673262 conn(0x7f8d9c0c8db0 msgr2=0x7f8d9c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- 172.21.15.50:0/3216673262 shutdown_connections 2023-12-19T14:51:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- 172.21.15.50:0/3216673262 wait complete. 2023-12-19T14:51:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 Processor -- start 2023-12-19T14:51:22.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.659+0000 7f8da3f45700 1 -- start start 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 0x7f8d9c169100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c159220 0x7f8d9c149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c149d60 con 0x7f8d9c068a90 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0c13f0 con 0x7f8d9c158e50 2023-12-19T14:51:22.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c1570e0 con 0x7f8d9c159220 2023-12-19T14:51:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48744/0 (socket says 172.21.15.50:48744) 2023-12-19T14:51:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 -- 172.21.15.50:0/944835643 learned_addr learned my addr 172.21.15.50:0/944835643 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da14e0700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 0x7f8d9c169100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da24e2700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c159220 0x7f8d9c149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 -- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c159220 msgr2=0x7f8d9c149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c159220 0x7f8d9c149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 -- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 msgr2=0x7f8d9c169100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 0x7f8d9c169100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 -- 172.21.15.50:0/944835643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98009420 con 0x7f8d9c068a90 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da14e0700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 0x7f8d9c169100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da1ce1700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8d980093f0 tx=0x7f8d98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98000bc0 con 0x7f8d9c068a90 2023-12-19T14:51:22.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9800b910 con 0x7f8d9c068a90 2023-12-19T14:51:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c0c07e0 con 0x7f8d9c068a90 2023-12-19T14:51:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.663+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d98025420 con 0x7f8d9c068a90 2023-12-19T14:51:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.667+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c0c0da0 con 0x7f8d9c068a90 2023-12-19T14:51:22.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.667+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c071120 con 0x7f8d9c068a90 2023-12-19T14:51:22.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.667+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f8d98023070 con 0x7f8d9c068a90 2023-12-19T14:51:22.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.671+0000 7f8d91ffb700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 0x7f8d80062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.671+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f8d9807e0d0 con 0x7f8d9c068a90 2023-12-19T14:51:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.671+0000 7f8da14e0700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 0x7f8d80062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:22.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.671+0000 7f8da14e0700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 0x7f8d80062c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0b1600 tx=0x7f8d8c007f50 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:22.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.675+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d9804e1f0 con 0x7f8d9c068a90 2023-12-19T14:51:22.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:22.867+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d9c06cd60 con 0x7f8d80060750 2023-12-19T14:51:23.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.003+0000 7f8d91ffb700 1 -- 172.21.15.50:0/944835643 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8d9c06cd60 con 0x7f8d80060750 2023-12-19T14:51:23.005 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 msgr2=0x7f8d80062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 0x7f8d80062c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0b1600 tx=0x7f8d8c007f50 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 msgr2=0x7f8d9c156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8d980093f0 tx=0x7f8d98017000 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 shutdown_connections 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d9c158e50 0x7f8d9c169100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8d80060750 0x7f8d80062c00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d9c159220 0x7f8d9c149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 --2- 172.21.15.50:0/944835643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d9c068a90 0x7f8d9c156840 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 >> 172.21.15.50:0/944835643 conn(0x7f8d9c0c8db0 msgr2=0x7f8d9c157710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 shutdown_connections 2023-12-19T14:51:23.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.007+0000 7f8da3f45700 1 -- 172.21.15.50:0/944835643 wait complete. 2023-12-19T14:51:23.033 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-12-19T14:51:23.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T14:51:23.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 Processor -- start 2023-12-19T14:51:23.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 -- start start 2023-12-19T14:51:23.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5af00aedc0 0x7f5af00af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af006b3d0 con 0x7f5af00cac50 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af00cb3f0 con 0x7f5af006af50 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af6b23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af00cc140 con 0x7f5af00aedc0 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6322700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5af00cac50 0x7f5af00af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54290/0 (socket says 172.21.15.50:54290) 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6322700 1 -- 172.21.15.50:0/1034088549 learned_addr learned my addr 172.21.15.50:0/1034088549 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:23.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.223+0000 7f5af5b21700 1 --2- 172.21.15.50:0/1034088549 >> v2:172.21.15.50:3300/0 conn(0x7f5af00aedc0 0x7f5af00af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321378349 0 0) 0x7f5ae8001490 con 0x7f5af00cac50 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5adc003570 con 0x7f5af00cac50 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982345993 0 0) 0x7f5af00cb3f0 con 0x7f5af006af50 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8001490 con 0x7f5af006af50 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af5b21700 1 -- 172.21.15.50:0/1034088549 >> v1:172.21.15.125:6789/0 conn(0x7f5af006af50 legacy=0x7f5af006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af5b21700 1 -- 172.21.15.50:0/1034088549 >> v1:172.21.15.50:6789/0 conn(0x7f5af00cac50 legacy=0x7f5af00af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af5b21700 1 -- 172.21.15.50:0/1034088549 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af006d220 con 0x7f5af00aedc0 2023-12-19T14:51:23.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af5b21700 1 --2- 172.21.15.50:0/1034088549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00af190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098e0 tx=0x7f5aec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def8627b0e351f67 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec011630 con 0x7f5af00aedc0 2023-12-19T14:51:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec011790 con 0x7f5af00aedc0 2023-12-19T14:51:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af4b1f700 1 -- 172.21.15.50:0/1034088549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec011b30 con 0x7f5af00aedc0 2023-12-19T14:51:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 -- 172.21.15.50:0/1034088549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 msgr2=0x7f5af00af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1034088549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00af190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098e0 tx=0x7f5aec0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 -- 172.21.15.50:0/1034088549 shutdown_connections 2023-12-19T14:51:23.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1034088549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00af190 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 -- 172.21.15.50:0/1034088549 >> 172.21.15.50:0/1034088549 conn(0x7f5af00cc760 msgr2=0x7f5af00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:23.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 -- 172.21.15.50:0/1034088549 shutdown_connections 2023-12-19T14:51:23.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.227+0000 7f5af6b23700 1 -- 172.21.15.50:0/1034088549 wait complete. 2023-12-19T14:51:23.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 Processor -- start 2023-12-19T14:51:23.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 -- start start 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5af006af50 0x7f5af014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5af00cac50 0x7f5af006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00c3ac0 con 0x7f5af00aedc0 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00c5230 con 0x7f5af00cac50 2023-12-19T14:51:23.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.231+0000 7f5af6b23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00b3bd0 con 0x7f5af006af50 2023-12-19T14:51:23.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48764/0 (socket says 172.21.15.50:48764) 2023-12-19T14:51:23.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 -- 172.21.15.50:0/1772555776 learned_addr learned my addr 172.21.15.50:0/1772555776 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af6322700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5af00cac50 0x7f5af006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5b21700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5af006af50 0x7f5af014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 -- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5af006af50 msgr2=0x7f5af014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5af006af50 0x7f5af014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 -- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5af00cac50 msgr2=0x7f5af006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5af00cac50 0x7f5af006d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 -- 172.21.15.50:0/1772555776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec009420 con 0x7f5af00aedc0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af5320700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c5b0 tx=0x7f5ae0007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0010510 con 0x7f5af00aedc0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae0010670 con 0x7f5af00aedc0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0010a30 con 0x7f5af00aedc0 2023-12-19T14:51:23.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af00b3eb0 con 0x7f5af00aedc0 2023-12-19T14:51:23.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.235+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af00b4470 con 0x7f5af00aedc0 2023-12-19T14:51:23.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.239+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af00c53c0 con 0x7f5af00aedc0 2023-12-19T14:51:23.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.239+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f5ae001b400 con 0x7f5af00aedc0 2023-12-19T14:51:23.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.239+0000 7f5ae5ffb700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 0x7f5ad4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.239+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f5ae007cae0 con 0x7f5af00aedc0 2023-12-19T14:51:23.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.239+0000 7f5af5b21700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 0x7f5ad4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.243+0000 7f5af5b21700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 0x7f5ad4062c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0093f0 tx=0x7f5aec017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.247+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ae004cb50 con 0x7f5af00aedc0 2023-12-19T14:51:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.439+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5af004be40 con 0x7f5ad4060750 2023-12-19T14:51:23.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.575+0000 7f5ae5ffb700 1 -- 172.21.15.50:0/1772555776 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5af004be40 con 0x7f5ad4060750 2023-12-19T14:51:23.578 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 msgr2=0x7f5ad4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 0x7f5ad4062c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0093f0 tx=0x7f5aec017000 comp rx=0 tx=0).stop 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 msgr2=0x7f5af00b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c5b0 tx=0x7f5ae0007e10 comp rx=0 tx=0).stop 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 shutdown_connections 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5af00cac50 0x7f5af006d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad4060750 0x7f5ad4062c00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5af00aedc0 0x7f5af00b5220 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 --2- 172.21.15.50:0/1772555776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5af006af50 0x7f5af014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 >> 172.21.15.50:0/1772555776 conn(0x7f5af00cc760 msgr2=0x7f5af00afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 shutdown_connections 2023-12-19T14:51:23.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.579+0000 7f5af6b23700 1 -- 172.21.15.50:0/1772555776 wait complete. 2023-12-19T14:51:23.604 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-12-19T14:51:23.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T14:51:23.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.787+0000 7f4201c4c700 1 Processor -- start 2023-12-19T14:51:23.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.787+0000 7f4201c4c700 1 -- start start 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f4201c4c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41fc0af190 0x7f41fc06af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f4201c4c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0cb3f0 con 0x7f41fc0cac50 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f4201c4c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06d220 con 0x7f41fc0aedc0 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f4201c4c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41fc0cc140 con 0x7f41fc0af190 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f420144b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f41fc0cac50 0x7f41fc0af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54300/0 (socket says 172.21.15.50:54300) 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f420144b700 1 -- 172.21.15.50:0/2497717041 learned_addr learned my addr 172.21.15.50:0/2497717041 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:23.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fbfff700 1 --2- 172.21.15.50:0/2497717041 >> v2:172.21.15.125:3300/0 conn(0x7f41fc0af190 0x7f41fc06af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915942986 0 0) 0x7f41fc06d220 con 0x7f41fc0aedc0 2023-12-19T14:51:23.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec003570 con 0x7f41fc0aedc0 2023-12-19T14:51:23.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691312392 0 0) 0x7f41fc0cb3f0 con 0x7f41fc0cac50 2023-12-19T14:51:23.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc06d220 con 0x7f41fc0cac50 2023-12-19T14:51:23.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fbfff700 1 -- 172.21.15.50:0/2497717041 >> v1:172.21.15.125:6789/0 conn(0x7f41fc0aedc0 legacy=0x7f41fc06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fbfff700 1 -- 172.21.15.50:0/2497717041 >> v1:172.21.15.50:6789/0 conn(0x7f41fc0cac50 legacy=0x7f41fc0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fbfff700 1 -- 172.21.15.50:0/2497717041 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc06c8f0 con 0x7f41fc0af190 2023-12-19T14:51:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.791+0000 7f41fbfff700 1 --2- 172.21.15.50:0/2497717041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0af190 0x7f41fc06af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098e0 tx=0x7f41e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515612740e540b8f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8011630 con 0x7f41fc0af190 2023-12-19T14:51:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8011790 con 0x7f41fc0af190 2023-12-19T14:51:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f41fb7fe700 1 -- 172.21.15.50:0/2497717041 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8011b30 con 0x7f41fc0af190 2023-12-19T14:51:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 -- 172.21.15.50:0/2497717041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0af190 msgr2=0x7f41fc06af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 --2- 172.21.15.50:0/2497717041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0af190 0x7f41fc06af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098e0 tx=0x7f41e80091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 -- 172.21.15.50:0/2497717041 shutdown_connections 2023-12-19T14:51:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 --2- 172.21.15.50:0/2497717041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0af190 0x7f41fc06af50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 -- 172.21.15.50:0/2497717041 >> 172.21.15.50:0/2497717041 conn(0x7f41fc0cc760 msgr2=0x7f41fc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 -- 172.21.15.50:0/2497717041 shutdown_connections 2023-12-19T14:51:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 -- 172.21.15.50:0/2497717041 wait complete. 2023-12-19T14:51:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.795+0000 7f4201c4c700 1 Processor -- start 2023-12-19T14:51:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 -- start start 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41fc0aedc0 0x7f41fc0c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0cac50 0x7f41fc0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0af610 con 0x7f41fc0af190 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0b2e40 con 0x7f41fc0cac50 2023-12-19T14:51:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0b2b40 con 0x7f41fc0aedc0 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48774/0 (socket says 172.21.15.50:48774) 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 -- 172.21.15.50:0/1061734666 learned_addr learned my addr 172.21.15.50:0/1061734666 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f420144b700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0cac50 0x7f41fc0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4200c4a700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41fc0aedc0 0x7f41fc0c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 -- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41fc0aedc0 msgr2=0x7f41fc0c2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41fc0aedc0 0x7f41fc0c2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 -- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0cac50 msgr2=0x7f41fc0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0cac50 0x7f41fc0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 -- 172.21.15.50:0/1061734666 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8009420 con 0x7f41fc0af190 2023-12-19T14:51:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41fbfff700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009ed0 tx=0x7f41e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e801f070 con 0x7f41fc0af190 2023-12-19T14:51:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8011b10 con 0x7f41fc0af190 2023-12-19T14:51:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc0c5d50 con 0x7f41fc0af190 2023-12-19T14:51:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.799+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8023660 con 0x7f41fc0af190 2023-12-19T14:51:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.803+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc0c3450 con 0x7f41fc0af190 2023-12-19T14:51:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.803+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc06ae40 con 0x7f41fc0af190 2023-12-19T14:51:23.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.803+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f41e802c3b0 con 0x7f41fc0af190 2023-12-19T14:51:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.807+0000 7f41f8ff9700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 0x7f41e0062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:23.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.807+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f41e807e110 con 0x7f41fc0af190 2023-12-19T14:51:23.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.807+0000 7f4200c4a700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 0x7f41e0062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:23.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.807+0000 7f4200c4a700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 0x7f41e0062c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f41f00037e0 tx=0x7f41f0009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:23.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:23.811+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f41e804e180 con 0x7f41fc0af190 2023-12-19T14:51:24.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.003+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f41fc04be40 con 0x7f41e0060780 2023-12-19T14:51:24.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.139+0000 7f41f8ff9700 1 -- 172.21.15.50:0/1061734666 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f41fc04be40 con 0x7f41e0060780 2023-12-19T14:51:24.142 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 msgr2=0x7f41e0062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 0x7f41e0062c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f41f00037e0 tx=0x7f41f0009040 comp rx=0 tx=0).stop 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 msgr2=0x7f41fc0c53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009ed0 tx=0x7f41e8017000 comp rx=0 tx=0).stop 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 shutdown_connections 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41fc0cac50 0x7f41fc0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41e0060780 0x7f41e0062c30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41fc0af190 0x7f41fc0c53e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 --2- 172.21.15.50:0/1061734666 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41fc0aedc0 0x7f41fc0c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 >> 172.21.15.50:0/1061734666 conn(0x7f41fc0cc760 msgr2=0x7f41fc0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 shutdown_connections 2023-12-19T14:51:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.143+0000 7f4201c4c700 1 -- 172.21.15.50:0/1061734666 wait complete. 2023-12-19T14:51:24.172 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-12-19T14:51:24.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T14:51:24.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.355+0000 7f7afd2ae700 1 Processor -- start 2023-12-19T14:51:24.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.355+0000 7f7afd2ae700 1 -- start start 2023-12-19T14:51:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7afd2ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7af80cb500 0x7f7af815c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7afd2ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 0x7f7af80b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7afd2ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af80b0ac0 con 0x7f7af80cb130 2023-12-19T14:51:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7afd2ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af815cbe0 con 0x7f7af80cb500 2023-12-19T14:51:24.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7afd2ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af806b210 con 0x7f7af815c760 2023-12-19T14:51:24.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7af67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7af80cb500 0x7f7af815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7af67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7af80cb500 0x7f7af815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48790/0 (socket says 172.21.15.50:48790) 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.359+0000 7f7af67fc700 1 -- 172.21.15.50:0/3567323104 learned_addr learned my addr 172.21.15.50:0/3567323104 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af77fe700 1 --2- 172.21.15.50:0/3567323104 >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 0x7f7af80b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af67fc700 1 -- 172.21.15.50:0/3567323104 >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 msgr2=0x7f7af80b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af67fc700 1 --2- 172.21.15.50:0/3567323104 >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 0x7f7af80b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af67fc700 1 -- 172.21.15.50:0/3567323104 >> v1:172.21.15.125:6789/0 conn(0x7f7af80cb130 legacy=0x7f7af806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af67fc700 1 -- 172.21.15.50:0/3567323104 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80ca810 con 0x7f7af80cb500 2023-12-19T14:51:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af77fe700 1 --2- 172.21.15.50:0/3567323104 >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 0x7f7af80b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af67fc700 1 --2- 172.21.15.50:0/3567323104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af815c220 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7af80cfc60 tx=0x7f7aec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b742b6da9cdd73d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af5ffb700 1 -- 172.21.15.50:0/3567323104 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00b820 con 0x7f7af80cb500 2023-12-19T14:51:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af5ffb700 1 -- 172.21.15.50:0/3567323104 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec00b980 con 0x7f7af80cb500 2023-12-19T14:51:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7af5ffb700 1 -- 172.21.15.50:0/3567323104 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00bd20 con 0x7f7af80cb500 2023-12-19T14:51:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 -- 172.21.15.50:0/3567323104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 msgr2=0x7f7af815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/3567323104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af815c220 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7af80cfc60 tx=0x7f7aec009250 comp rx=0 tx=0).stop 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 -- 172.21.15.50:0/3567323104 shutdown_connections 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/3567323104 >> v2:172.21.15.125:3300/0 conn(0x7f7af815c760 0x7f7af80b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/3567323104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af815c220 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 -- 172.21.15.50:0/3567323104 >> 172.21.15.50:0/3567323104 conn(0x7f7af8067bb0 msgr2=0x7f7af80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 -- 172.21.15.50:0/3567323104 shutdown_connections 2023-12-19T14:51:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.363+0000 7f7afd2ae700 1 -- 172.21.15.50:0/3567323104 wait complete. 2023-12-19T14:51:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 Processor -- start 2023-12-19T14:51:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 -- start start 2023-12-19T14:51:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 0x7f7af80b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af815c760 0x7f7af814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af815bc50 con 0x7f7af80cb500 2023-12-19T14:51:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af816dde0 con 0x7f7af815c760 2023-12-19T14:51:24.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7afd2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af815b690 con 0x7f7af80cb130 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48792/0 (socket says 172.21.15.50:48792) 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 -- 172.21.15.50:0/1885664609 learned_addr learned my addr 172.21.15.50:0/1885664609 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af6ffd700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 0x7f7af80b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af77fe700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af815c760 0x7f7af814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 -- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 msgr2=0x7f7af80b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 0x7f7af80b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 -- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af815c760 msgr2=0x7f7af814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af815c760 0x7f7af814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af67fc700 1 -- 172.21.15.50:0/1885664609 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0094f0 con 0x7f7af80cb500 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.367+0000 7f7af6ffd700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 0x7f7af80b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7af67fc700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0030c0 tx=0x7f7aec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec011d90 con 0x7f7af80cb500 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec020070 con 0x7f7af80cb500 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec01b6a0 con 0x7f7af80cb500 2023-12-19T14:51:24.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af815b970 con 0x7f7af80cb500 2023-12-19T14:51:24.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af8156340 con 0x7f7af80cb500 2023-12-19T14:51:24.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af8155ec0 con 0x7f7af80cb500 2023-12-19T14:51:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.371+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f7aec022030 con 0x7f7af80cb500 2023-12-19T14:51:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.375+0000 7f7adeffd700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 0x7f7ae4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.375+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f7aec07ed80 con 0x7f7af80cb500 2023-12-19T14:51:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.375+0000 7f7af6ffd700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 0x7f7ae4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.375+0000 7f7af6ffd700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 0x7f7ae4062c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8007fd0 tx=0x7f7ae8009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.379+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7af8155ec0 con 0x7f7af80cb500 2023-12-19T14:51:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.571+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f7af806de90 con 0x7f7ae4060750 2023-12-19T14:51:24.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.715+0000 7f7adeffd700 1 -- 172.21.15.50:0/1885664609 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7af806de90 con 0x7f7ae4060750 2023-12-19T14:51:24.717 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 msgr2=0x7f7ae4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 0x7f7ae4062c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8007fd0 tx=0x7f7ae8009040 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 msgr2=0x7f7af816d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0030c0 tx=0x7f7aec016000 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 shutdown_connections 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af815c760 0x7f7af814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ae4060750 0x7f7ae4062c00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7af80cb500 0x7f7af816d8a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 --2- 172.21.15.50:0/1885664609 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7af80cb130 0x7f7af80b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 >> 172.21.15.50:0/1885664609 conn(0x7f7af8067bb0 msgr2=0x7f7af80cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:24.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 shutdown_connections 2023-12-19T14:51:24.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.719+0000 7f7afd2ae700 1 -- 172.21.15.50:0/1885664609 wait complete. 2023-12-19T14:51:24.744 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-12-19T14:51:24.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T14:51:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.923+0000 7fab1142e700 1 Processor -- start 2023-12-19T14:51:24.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.923+0000 7fab1142e700 1 -- start start 2023-12-19T14:51:24.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab1142e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab1142e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c06b2c0 con 0x7fab0c0caa00 2023-12-19T14:51:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab1142e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c06dfa0 con 0x7fab0c06b750 2023-12-19T14:51:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab1142e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c06a6b0 con 0x7fab0c0c5600 2023-12-19T14:51:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fab0c0caa00 0x7fab0c0cae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54316/0 (socket says 172.21.15.50:54316) 2023-12-19T14:51:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0b7fe700 1 -- 172.21.15.50:0/1520570194 learned_addr learned my addr 172.21.15.50:0/1520570194 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0affd700 1 --2- 172.21.15.50:0/1520570194 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826753441 0 0) 0x7fab0c06b2c0 con 0x7fab0c0caa00 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf4003570 con 0x7fab0c0caa00 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917869585 0 0) 0x7fab0c06dfa0 con 0x7fab0c06b750 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c06b2c0 con 0x7fab0c06b750 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879316756 0 0) 0x7faaf4003570 con 0x7fab0c0caa00 2023-12-19T14:51:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0affd700 1 -- 172.21.15.50:0/1520570194 >> v1:172.21.15.125:6789/0 conn(0x7fab0c06b750 legacy=0x7fab0c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0affd700 1 -- 172.21.15.50:0/1520570194 >> v1:172.21.15.50:6789/0 conn(0x7fab0c0caa00 legacy=0x7fab0c0cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.927+0000 7fab0affd700 1 -- 172.21.15.50:0/1520570194 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c06a990 con 0x7fab0c0c5600 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab0affd700 1 --2- 172.21.15.50:0/1520570194 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c59d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faafc009910 tx=0x7faafc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a122ffb77cfb8405 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab040017f0 con 0x7fab0c0c5600 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc011420 con 0x7fab0c0c5600 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab09ffb700 1 -- 172.21.15.50:0/1520570194 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0117c0 con 0x7fab0c0c5600 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 -- 172.21.15.50:0/1520570194 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 msgr2=0x7fab0c0c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 --2- 172.21.15.50:0/1520570194 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c59d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7faafc009910 tx=0x7faafc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 -- 172.21.15.50:0/1520570194 shutdown_connections 2023-12-19T14:51:24.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 --2- 172.21.15.50:0/1520570194 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c59d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 -- 172.21.15.50:0/1520570194 >> 172.21.15.50:0/1520570194 conn(0x7fab0c067bb0 msgr2=0x7fab0c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:24.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 -- 172.21.15.50:0/1520570194 shutdown_connections 2023-12-19T14:51:24.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.931+0000 7fab1142e700 1 -- 172.21.15.50:0/1520570194 wait complete. 2023-12-19T14:51:24.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 Processor -- start 2023-12-19T14:51:24.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 -- start start 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab0c0caa00 0x7fab0c0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c14adf0 con 0x7fab0c06b750 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c0c7720 con 0x7fab0c0caa00 2023-12-19T14:51:24.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab1142e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c0c7a00 con 0x7fab0c0c5600 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48802/0 (socket says 172.21.15.50:48802) 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 -- 172.21.15.50:0/3820501558 learned_addr learned my addr 172.21.15.50:0/3820501558 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0b7fe700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab0c0caa00 0x7fab0c0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0a7fc700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 -- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 msgr2=0x7fab0c0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 -- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab0c0caa00 msgr2=0x7fab0c0b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab0c0caa00 0x7fab0c0b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 -- 172.21.15.50:0/3820501558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc009420 con 0x7fab0c06b750 2023-12-19T14:51:24.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.935+0000 7fab0affd700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faafc0091e0 tx=0x7faafc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc01cbe0 con 0x7fab0c06b750 2023-12-19T14:51:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc01cd40 con 0x7fab0c06b750 2023-12-19T14:51:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c0c7ce0 con 0x7fab0c06b750 2023-12-19T14:51:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc01a6a0 con 0x7fab0c06b750 2023-12-19T14:51:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c0c82a0 con 0x7fab0c06b750 2023-12-19T14:51:24.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c0c8490 con 0x7fab0c06b750 2023-12-19T14:51:24.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.939+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7faafc01aa90 con 0x7fab0c06b750 2023-12-19T14:51:24.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.943+0000 7faaf2ffd700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 0x7faaf8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:24.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.943+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7faafc07ebd0 con 0x7fab0c06b750 2023-12-19T14:51:24.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.943+0000 7fab0a7fc700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 0x7faaf8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:24.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.943+0000 7fab0a7fc700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 0x7faaf8062c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab0c154750 tx=0x7fab00009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:24.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:24.947+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faafc04ed90 con 0x7fab0c06b750 2023-12-19T14:51:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.143+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fab0c04be40 con 0x7faaf8060750 2023-12-19T14:51:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.279+0000 7faaf2ffd700 1 -- 172.21.15.50:0/3820501558 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fab0c04be40 con 0x7faaf8060750 2023-12-19T14:51:25.282 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-12-19T14:51:25.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 msgr2=0x7faaf8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 0x7faaf8062c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab0c154750 tx=0x7fab00009040 comp rx=0 tx=0).stop 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 msgr2=0x7fab0c14a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7faafc0091e0 tx=0x7faafc016000 comp rx=0 tx=0).stop 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 shutdown_connections 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab0c0caa00 0x7fab0c0b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faaf8060750 0x7faaf8062c00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab0c0c5600 0x7fab0c0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 --2- 172.21.15.50:0/3820501558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab0c06b750 0x7fab0c14a8b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 >> 172.21.15.50:0/3820501558 conn(0x7fab0c067bb0 msgr2=0x7fab0c0cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 shutdown_connections 2023-12-19T14:51:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.283+0000 7fab1142e700 1 -- 172.21.15.50:0/3820501558 wait complete. 2023-12-19T14:51:25.309 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-12-19T14:51:25.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T14:51:25.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 Processor -- start 2023-12-19T14:51:25.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 -- start start 2023-12-19T14:51:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce4415cf90 0x7fce4406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4406dde0 0x7fce440af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4406b550 con 0x7fce4415d360 2023-12-19T14:51:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4406af40 con 0x7fce4415cf90 2023-12-19T14:51:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce48984700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce440af5a0 con 0x7fce4406dde0 2023-12-19T14:51:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce42d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce4415cf90 0x7fce4406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce42d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce4415cf90 0x7fce4406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48806/0 (socket says 172.21.15.50:48806) 2023-12-19T14:51:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.483+0000 7fce42d9d700 1 -- 172.21.15.50:0/551415058 learned_addr learned my addr 172.21.15.50:0/551415058 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce4359e700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4406dde0 0x7fce440af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce42d9d700 1 -- 172.21.15.50:0/551415058 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4406dde0 msgr2=0x7fce440af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce42d9d700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4406dde0 0x7fce440af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce42d9d700 1 -- 172.21.15.50:0/551415058 >> v1:172.21.15.50:6789/0 conn(0x7fce4415d360 legacy=0x7fce440ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce42d9d700 1 -- 172.21.15.50:0/551415058 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce440ae0c0 con 0x7fce4415cf90 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce42d9d700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4415cf90 0x7fce4406b9e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce340099b0 tx=0x7fce34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f160ccafcebd049b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce4259c700 1 -- 172.21.15.50:0/551415058 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3400b860 con 0x7fce4415cf90 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce4259c700 1 -- 172.21.15.50:0/551415058 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3400b9c0 con 0x7fce4415cf90 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce4259c700 1 -- 172.21.15.50:0/551415058 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3400bd60 con 0x7fce4415cf90 2023-12-19T14:51:25.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 -- 172.21.15.50:0/551415058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4415cf90 msgr2=0x7fce4406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4415cf90 0x7fce4406b9e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce340099b0 tx=0x7fce34009250 comp rx=0 tx=0).stop 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 -- 172.21.15.50:0/551415058 shutdown_connections 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4406dde0 0x7fce440af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 --2- 172.21.15.50:0/551415058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4415cf90 0x7fce4406b9e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 -- 172.21.15.50:0/551415058 >> 172.21.15.50:0/551415058 conn(0x7fce44067bb0 msgr2=0x7fce440c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 -- 172.21.15.50:0/551415058 shutdown_connections 2023-12-19T14:51:25.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.487+0000 7fce48984700 1 -- 172.21.15.50:0/551415058 wait complete. 2023-12-19T14:51:25.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 Processor -- start 2023-12-19T14:51:25.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 -- start start 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce4415cf90 0x7fce4415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4415d360 0x7fce440caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4415ed50 con 0x7fce4406dde0 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4415e140 con 0x7fce4415cf90 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce48984700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4415e420 con 0x7fce4415d360 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48808/0 (socket says 172.21.15.50:48808) 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 -- 172.21.15.50:0/3833808776 learned_addr learned my addr 172.21.15.50:0/3833808776 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce4359e700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4415d360 0x7fce440caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 -- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4415d360 msgr2=0x7fce440caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4415d360 0x7fce440caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 -- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce4415cf90 msgr2=0x7fce4415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce4415cf90 0x7fce4415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.491+0000 7fce42d9d700 1 -- 172.21.15.50:0/3833808776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340094f0 con 0x7fce4406dde0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce42d9d700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fce4406a030 tx=0x7fce34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34024440 con 0x7fce4406dde0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce340245c0 con 0x7fce4406dde0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4415e700 con 0x7fce4406dde0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34024980 con 0x7fce4406dde0 2023-12-19T14:51:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4415dd80 con 0x7fce4406dde0 2023-12-19T14:51:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4404be40 con 0x7fce4406dde0 2023-12-19T14:51:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.495+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fce3401a420 con 0x7fce4406dde0 2023-12-19T14:51:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.499+0000 7fce3affd700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 0x7fce24062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.499+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fce3407ec60 con 0x7fce4406dde0 2023-12-19T14:51:25.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.499+0000 7fce3bfff700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 0x7fce24062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:25.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.499+0000 7fce3bfff700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 0x7fce24062c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fce44151760 tx=0x7fce2c007b40 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:25.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.503+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce3404ed70 con 0x7fce4406dde0 2023-12-19T14:51:25.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.699+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fce4415ee90 con 0x7fce24060750 2023-12-19T14:51:25.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.839+0000 7fce3affd700 1 -- 172.21.15.50:0/3833808776 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fce2c014410 con 0x7fce24060750 2023-12-19T14:51:25.841 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-12-19T14:51:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 msgr2=0x7fce24062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 0x7fce24062c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fce44151760 tx=0x7fce2c007b40 comp rx=0 tx=0).stop 2023-12-19T14:51:25.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 msgr2=0x7fce4416d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fce4406a030 tx=0x7fce34017000 comp rx=0 tx=0).stop 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 shutdown_connections 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce4415cf90 0x7fce4415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fce24060750 0x7fce24062c00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce4415d360 0x7fce440caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 --2- 172.21.15.50:0/3833808776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce4406dde0 0x7fce4416d310 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 >> 172.21.15.50:0/3833808776 conn(0x7fce44067bb0 msgr2=0x7fce440b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 shutdown_connections 2023-12-19T14:51:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:25.843+0000 7fce48984700 1 -- 172.21.15.50:0/3833808776 wait complete. 2023-12-19T14:51:25.870 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-12-19T14:51:25.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T14:51:26.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.047+0000 7f09532ae700 1 Processor -- start 2023-12-19T14:51:26.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.047+0000 7f09532ae700 1 -- start start 2023-12-19T14:51:26.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f09532ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c06adb0 0x7f094c168470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f09532ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f094c0c6560 con 0x7f094c15d350 2023-12-19T14:51:26.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f09532ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f094c0c70e0 con 0x7f094c161da0 2023-12-19T14:51:26.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f09532ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094c0cc900 con 0x7f094c06adb0 2023-12-19T14:51:26.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f095104a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f094c15d350 0x7f094c0cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54326/0 (socket says 172.21.15.50:54326) 2023-12-19T14:51:26.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f095104a700 1 -- 172.21.15.50:0/6493921 learned_addr learned my addr 172.21.15.50:0/6493921 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f095184b700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c06adb0 0x7f094c168470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372432483 0 0) 0x7f094c0c6560 con 0x7f094c15d350 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0938003570 con 0x7f094c15d350 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144987498 0 0) 0x7f094c0c70e0 con 0x7f094c161da0 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f094c0c6560 con 0x7f094c161da0 2023-12-19T14:51:26.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206428568 0 0) 0x7f0938003570 con 0x7f094c15d350 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c06adb0 msgr2=0x7f094c168470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c06adb0 0x7f094c168470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 >> v1:172.21.15.125:6789/0 conn(0x7f094c161da0 legacy=0x7f094c06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0c67b0 con 0x7f094c15d350 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f09480017f0 con 0x7f094c15d350 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 >> v1:172.21.15.50:6789/0 conn(0x7f094c15d350 legacy=0x7f094c0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0938003c20 0x7f0938004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0938004810 0x7f0938004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09380051c0 0x7f093800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c0cc900 con 0x7f09380051c0 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c0b0 con 0x7f0938003c20 2023-12-19T14:51:26.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.051+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c270 con 0x7f0938004810 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09380051c0 0x7f093800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095104a700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0938003c20 0x7f0938004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f0950849700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0938004810 0x7f0938004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 -- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0938004810 msgr2=0x7f0938004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0938004810 0x7f0938004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 -- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0938003c20 msgr2=0x7f0938004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0938003c20 0x7f0938004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 -- 172.21.15.50:0/6493921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938003570 con 0x7f09380051c0 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095104a700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0938003c20 0x7f0938004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09380051c0 0x7f093800ba60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f094c06b180 tx=0x7f09440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ad1d243bf5d7ed server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094400e610 con 0x7f09380051c0 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094400e770 con 0x7f09380051c0 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f0943fff700 1 -- 172.21.15.50:0/6493921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094400eb30 con 0x7f09380051c0 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 -- 172.21.15.50:0/6493921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09380051c0 msgr2=0x7f093800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 --2- 172.21.15.50:0/6493921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09380051c0 0x7f093800ba60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f094c06b180 tx=0x7f09440091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f095184b700 1 -- 172.21.15.50:0/6493921 reap_dead start 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 -- 172.21.15.50:0/6493921 shutdown_connections 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 -- 172.21.15.50:0/6493921 >> 172.21.15.50:0/6493921 conn(0x7f094c067bb0 msgr2=0x7f094c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 -- 172.21.15.50:0/6493921 shutdown_connections 2023-12-19T14:51:26.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.055+0000 7f09532ae700 1 -- 172.21.15.50:0/6493921 wait complete. 2023-12-19T14:51:26.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 Processor -- start 2023-12-19T14:51:26.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 -- start start 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 0x7f094c0b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 0x7f094c1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1567b0 con 0x7f094c171010 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1719c0 con 0x7f094c1563e0 2023-12-19T14:51:26.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f09532ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094c0b10a0 con 0x7f094c15f2d0 2023-12-19T14:51:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48830/0 (socket says 172.21.15.50:48830) 2023-12-19T14:51:26.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 -- 172.21.15.50:0/3202944979 learned_addr learned my addr 172.21.15.50:0/3202944979 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095104a700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 0x7f094c0b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f0950849700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 0x7f094c1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 -- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 msgr2=0x7f094c1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 0x7f094c1601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 -- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 msgr2=0x7f094c0b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 0x7f094c0b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095184b700 1 -- 172.21.15.50:0/3202944979 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0b0210 con 0x7f094c171010 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f095104a700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 0x7f094c0b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.059+0000 7f0950849700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 0x7f094c1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f095184b700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f094c0c9660 tx=0x7f094401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944022070 con 0x7f094c171010 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094400eea0 con 0x7f094c171010 2023-12-19T14:51:26.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c0b0490 con 0x7f094c171010 2023-12-19T14:51:26.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944009ee0 con 0x7f094c171010 2023-12-19T14:51:26.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c0b0a50 con 0x7f094c171010 2023-12-19T14:51:26.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.063+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c0b1230 con 0x7f094c171010 2023-12-19T14:51:26.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.067+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f0944027940 con 0x7f094c171010 2023-12-19T14:51:26.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.067+0000 7f09417fa700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 0x7f0930062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.067+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f094407e1b0 con 0x7f094c171010 2023-12-19T14:51:26.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.067+0000 7f095104a700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 0x7f0930062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.067+0000 7f095104a700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 0x7f0930062c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0948001190 tx=0x7f09480017f0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.071+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f094404e220 con 0x7f094c171010 2023-12-19T14:51:26.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.271+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f094c1567b0 con 0x7f0930060780 2023-12-19T14:51:26.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.403+0000 7f09417fa700 1 -- 172.21.15.50:0/3202944979 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f094c1567b0 con 0x7f0930060780 2023-12-19T14:51:26.404 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.403+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 msgr2=0x7f0930062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.403+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 0x7f0930062c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0948001190 tx=0x7f09480017f0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.403+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 msgr2=0x7f094c171480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.403+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f094c0c9660 tx=0x7f094401a000 comp rx=0 tx=0).stop 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 shutdown_connections 2023-12-19T14:51:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0930060780 0x7f0930062c30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f094c171010 0x7f094c171480 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f094c15f2d0 0x7f094c1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 --2- 172.21.15.50:0/3202944979 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f094c1563e0 0x7f094c0b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 >> 172.21.15.50:0/3202944979 conn(0x7f094c067bb0 msgr2=0x7f094c157220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 shutdown_connections 2023-12-19T14:51:26.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.407+0000 7f09532ae700 1 -- 172.21.15.50:0/3202944979 wait complete. 2023-12-19T14:51:26.431 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-12-19T14:51:26.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T14:51:26.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.607+0000 7f376b878700 1 Processor -- start 2023-12-19T14:51:26.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.611+0000 7f376b878700 1 -- start start 2023-12-19T14:51:26.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.611+0000 7f376b878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f3764155590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f376b878700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764159f60 con 0x7f37640ac9d0 2023-12-19T14:51:26.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f376b878700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764069900 con 0x7f3764159b30 2023-12-19T14:51:26.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f376b878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c7340 con 0x7f37641551c0 2023-12-19T14:51:26.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3768e13700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3764159b30 0x7f37640690d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:42698/0 (socket says 172.21.15.50:42698) 2023-12-19T14:51:26.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3768e13700 1 -- 172.21.15.50:0/887492719 learned_addr learned my addr 172.21.15.50:0/887492719 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:26.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3769614700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f3764155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367504165 0 0) 0x7f3764159f60 con 0x7f37640ac9d0 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3750003570 con 0x7f37640ac9d0 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719210605 0 0) 0x7f3764069900 con 0x7f3764159b30 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3769614700 1 -- 172.21.15.50:0/887492719 >> v1:172.21.15.125:6789/0 conn(0x7f3764159b30 legacy=0x7f37640690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3769614700 1 -- 172.21.15.50:0/887492719 >> v1:172.21.15.50:6789/0 conn(0x7f37640ac9d0 legacy=0x7f3764068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3769614700 1 -- 172.21.15.50:0/887492719 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640c6a10 con 0x7f37641551c0 2023-12-19T14:51:26.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f3769614700 1 --2- 172.21.15.50:0/887492719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f3764155590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3760009910 tx=0x7f37600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c7bd9e21c4d4870 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376000b680 con 0x7f37641551c0 2023-12-19T14:51:26.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376000b7e0 con 0x7f37641551c0 2023-12-19T14:51:26.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.615+0000 7f375bfff700 1 -- 172.21.15.50:0/887492719 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376000bba0 con 0x7f37641551c0 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- 172.21.15.50:0/887492719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 msgr2=0x7f3764155590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 --2- 172.21.15.50:0/887492719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f3764155590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3760009910 tx=0x7f37600091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- 172.21.15.50:0/887492719 shutdown_connections 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 --2- 172.21.15.50:0/887492719 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f3764155590 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- 172.21.15.50:0/887492719 >> 172.21.15.50:0/887492719 conn(0x7f37640c8db0 msgr2=0x7f37640c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- 172.21.15.50:0/887492719 shutdown_connections 2023-12-19T14:51:26.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- 172.21.15.50:0/887492719 wait complete. 2023-12-19T14:51:26.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 Processor -- start 2023-12-19T14:51:26.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.619+0000 7f376b878700 1 -- start start 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 0x7f37640aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f37640c1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764156780 con 0x7f3764159b30 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764156450 con 0x7f37640ac9d0 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640aab40 con 0x7f37641551c0 2023-12-19T14:51:26.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47484/0 (socket says 172.21.15.50:47484) 2023-12-19T14:51:26.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 -- 172.21.15.50:0/2807333606 learned_addr learned my addr 172.21.15.50:0/2807333606 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:26.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3768e13700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f37640c1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769614700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 0x7f37640aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 -- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 msgr2=0x7f37640c1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f37640c1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 -- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 msgr2=0x7f37640aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 0x7f37640aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 -- 172.21.15.50:0/2807333606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3760009420 con 0x7f3764159b30 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769614700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 0x7f37640aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f3769e15700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c910 tx=0x7f375c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c012630 con 0x7f3764159b30 2023-12-19T14:51:26.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c0127b0 con 0x7f3764159b30 2023-12-19T14:51:26.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c012b70 con 0x7f3764159b30 2023-12-19T14:51:26.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37640aae20 con 0x7f3764159b30 2023-12-19T14:51:26.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.623+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37640aa750 con 0x7f3764159b30 2023-12-19T14:51:26.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.627+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3764156930 con 0x7f3764159b30 2023-12-19T14:51:26.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.627+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f375c01b400 con 0x7f3764159b30 2023-12-19T14:51:26.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.627+0000 7f37597fa700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 0x7f3748062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:26.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.627+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f375c07d180 con 0x7f3764159b30 2023-12-19T14:51:26.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.631+0000 7f3769614700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 0x7f3748062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:26.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.631+0000 7f3769614700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 0x7f3748062c30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37600093f0 tx=0x7f3760017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:26.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.635+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3764156930 con 0x7f3764159b30 2023-12-19T14:51:26.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.831+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3764071120 con 0x7f3748060780 2023-12-19T14:51:26.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.963+0000 7f37597fa700 1 -- 172.21.15.50:0/2807333606 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3764071120 con 0x7f3748060780 2023-12-19T14:51:26.967 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-12-19T14:51:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.967+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 msgr2=0x7f3748062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.967+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 0x7f3748062c30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37600093f0 tx=0x7f3760017000 comp rx=0 tx=0).stop 2023-12-19T14:51:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.967+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 msgr2=0x7f37640c2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.967+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c910 tx=0x7f375c0027a0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 shutdown_connections 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3748060780 0x7f3748062c30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3764159b30 0x7f37640c2110 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37641551c0 0x7f37640c1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 --2- 172.21.15.50:0/2807333606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37640ac9d0 0x7f37640aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 >> 172.21.15.50:0/2807333606 conn(0x7f37640c8db0 msgr2=0x7f376406abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 shutdown_connections 2023-12-19T14:51:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:26.971+0000 7f376b878700 1 -- 172.21.15.50:0/2807333606 wait complete. 2023-12-19T14:51:26.998 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-12-19T14:51:26.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:26.998 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:27.034 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:27.034 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:27.035 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_01_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_01_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:27.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir0 2023-12-19T14:51:27.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.251+0000 7ff50ef23700 1 Processor -- start 2023-12-19T14:51:27.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50ef23700 1 -- start start 2023-12-19T14:51:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50ef23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff5080cbae0 0x7ff508158e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50ef23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50806c7c0 con 0x7ff508068870 2023-12-19T14:51:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50ef23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50806d340 con 0x7ff508158ac0 2023-12-19T14:51:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50ef23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50806ca10 con 0x7ff5080cbae0 2023-12-19T14:51:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50e722700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff508068870 0x7ff50806d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58908/0 (socket says 172.21.15.50:58908) 2023-12-19T14:51:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50df21700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff508158ac0 0x7ff5080c8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:42706/0 (socket says 172.21.15.50:42706) 2023-12-19T14:51:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50e722700 1 -- 172.21.15.50:0/210549142 learned_addr learned my addr 172.21.15.50:0/210549142 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:27.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.255+0000 7ff50d720700 1 --2- 172.21.15.50:0/210549142 >> v2:172.21.15.50:3300/0 conn(0x7ff5080cbae0 0x7ff508158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50cf1f700 1 -- 172.21.15.50:0/210549142 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768294369 0 0) 0x7ff50806d340 con 0x7ff508158ac0 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50cf1f700 1 -- 172.21.15.50:0/210549142 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f4003570 con 0x7ff508158ac0 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50d720700 1 -- 172.21.15.50:0/210549142 >> v1:172.21.15.125:6789/0 conn(0x7ff508158ac0 legacy=0x7ff5080c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50d720700 1 -- 172.21.15.50:0/210549142 >> v1:172.21.15.50:6789/0 conn(0x7ff508068870 legacy=0x7ff50806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50d720700 1 -- 172.21.15.50:0/210549142 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50806ccf0 con 0x7ff5080cbae0 2023-12-19T14:51:27.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50d720700 1 --2- 172.21.15.50:0/210549142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5080cbae0 0x7ff508158e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80098e0 tx=0x7ff4f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac16266814110fc5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50cf1f700 1 -- 172.21.15.50:0/210549142 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f800b640 con 0x7ff5080cbae0 2023-12-19T14:51:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50cf1f700 1 -- 172.21.15.50:0/210549142 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f800b7a0 con 0x7ff5080cbae0 2023-12-19T14:51:27.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50cf1f700 1 -- 172.21.15.50:0/210549142 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f800bb20 con 0x7ff5080cbae0 2023-12-19T14:51:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 -- 172.21.15.50:0/210549142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5080cbae0 msgr2=0x7ff508158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 --2- 172.21.15.50:0/210549142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5080cbae0 0x7ff508158e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80098e0 tx=0x7ff4f80091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 -- 172.21.15.50:0/210549142 shutdown_connections 2023-12-19T14:51:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 --2- 172.21.15.50:0/210549142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5080cbae0 0x7ff508158e90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 -- 172.21.15.50:0/210549142 >> 172.21.15.50:0/210549142 conn(0x7ff5080cdb70 msgr2=0x7ff5080cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:27.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 -- 172.21.15.50:0/210549142 shutdown_connections 2023-12-19T14:51:27.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.259+0000 7ff50ef23700 1 -- 172.21.15.50:0/210549142 wait complete. 2023-12-19T14:51:27.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 Processor -- start 2023-12-19T14:51:27.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 -- start start 2023-12-19T14:51:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5080cbae0 0x7ff50816cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff508158ac0 0x7ff5080c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50816d630 con 0x7ff508068870 2023-12-19T14:51:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50816d910 con 0x7ff5080cbae0 2023-12-19T14:51:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.263+0000 7ff50ef23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50816dbf0 con 0x7ff508158ac0 2023-12-19T14:51:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47494/0 (socket says 172.21.15.50:47494) 2023-12-19T14:51:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 -- 172.21.15.50:0/979099253 learned_addr learned my addr 172.21.15.50:0/979099253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50e722700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff508158ac0 0x7ff5080c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50d720700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5080cbae0 0x7ff50816cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 -- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff508158ac0 msgr2=0x7ff5080c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff508158ac0 0x7ff5080c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 -- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5080cbae0 msgr2=0x7ff50816cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5080cbae0 0x7ff50816cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 -- 172.21.15.50:0/979099253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8009420 con 0x7ff508068870 2023-12-19T14:51:27.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50df21700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c590 tx=0x7ff504007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504005a70 con 0x7ff508068870 2023-12-19T14:51:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5080ab490 con 0x7ff508068870 2023-12-19T14:51:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504005bf0 con 0x7ff508068870 2023-12-19T14:51:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5040070d0 con 0x7ff508068870 2023-12-19T14:51:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.267+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080b1f70 con 0x7ff508068870 2023-12-19T14:51:27.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.271+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5080ab2f0 con 0x7ff508068870 2023-12-19T14:51:27.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.271+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7ff5040074c0 con 0x7ff508068870 2023-12-19T14:51:27.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.271+0000 7ff4fdffb700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 0x7ff4ec062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.271+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7ff50407c3f0 con 0x7ff508068870 2023-12-19T14:51:27.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.275+0000 7ff50d720700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 0x7ff4ec062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.275+0000 7ff50d720700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 0x7ff4ec062c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80093f0 tx=0x7ff4f8017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.279+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff5080ab2f0 con 0x7ff508068870 2023-12-19T14:51:27.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.471+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5080ab2f0 con 0x7ff4ec060750 2023-12-19T14:51:27.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.487+0000 7ff4fdffb700 1 -- 172.21.15.50:0/979099253 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7ff5080ab2f0 con 0x7ff4ec060750 2023-12-19T14:51:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:27.490 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:36", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.491+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 msgr2=0x7ff4ec062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.491+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 0x7ff4ec062c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80093f0 tx=0x7ff4f8017000 comp rx=0 tx=0).stop 2023-12-19T14:51:27.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.491+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 msgr2=0x7ff5080b1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.491+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c590 tx=0x7ff504007e00 comp rx=0 tx=0).stop 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 shutdown_connections 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5080cbae0 0x7ff50816cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4ec060750 0x7ff4ec062c00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff508158ac0 0x7ff5080c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 --2- 172.21.15.50:0/979099253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff508068870 0x7ff5080b1a30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 >> 172.21.15.50:0/979099253 conn(0x7ff5080cdb70 msgr2=0x7ff508153860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 shutdown_connections 2023-12-19T14:51:27.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.495+0000 7ff50ef23700 1 -- 172.21.15.50:0/979099253 wait complete. 2023-12-19T14:51:27.517 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:36", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:27.518 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:36', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:27.518 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:27.518 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:27.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:27.518 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:27.549 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:27.551 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_01_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_01_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:27.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir1 2023-12-19T14:51:27.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.763+0000 7fc369a2e700 1 Processor -- start 2023-12-19T14:51:27.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.763+0000 7fc369a2e700 1 -- start start 2023-12-19T14:51:27.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc369a2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3640c5600 0x7fc3640c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc369a2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc36406b750 0x7fc36406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc369a2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36406b2c0 con 0x7fc36406d710 2023-12-19T14:51:27.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc369a2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36406dfa0 con 0x7fc36406b750 2023-12-19T14:51:27.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc369a2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36406a6b0 con 0x7fc3640c5600 2023-12-19T14:51:27.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc3637fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc36406d710 0x7fc3640caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58912/0 (socket says 172.21.15.50:58912) 2023-12-19T14:51:27.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc3637fe700 1 -- 172.21.15.50:0/1514653863 learned_addr learned my addr 172.21.15.50:0/1514653863 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:27.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc3627fc700 1 --2- 172.21.15.50:0/1514653863 >> v2:172.21.15.50:3300/0 conn(0x7fc36406b750 0x7fc36406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.767+0000 7fc362ffd700 1 --2- 172.21.15.50:0/1514653863 >> v2:172.21.15.125:3300/0 conn(0x7fc3640c5600 0x7fc3640c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc361ffb700 1 -- 172.21.15.50:0/1514653863 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111578925 0 0) 0x7fc36406b2c0 con 0x7fc36406d710 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc3627fc700 1 -- 172.21.15.50:0/1514653863 >> v2:172.21.15.125:3300/0 conn(0x7fc3640c5600 msgr2=0x7fc3640c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc3627fc700 1 --2- 172.21.15.50:0/1514653863 >> v2:172.21.15.125:3300/0 conn(0x7fc3640c5600 0x7fc3640c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc3627fc700 1 -- 172.21.15.50:0/1514653863 >> v1:172.21.15.50:6789/0 conn(0x7fc36406d710 legacy=0x7fc3640caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc3627fc700 1 -- 172.21.15.50:0/1514653863 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36406a990 con 0x7fc36406b750 2023-12-19T14:51:27.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc3627fc700 1 --2- 172.21.15.50:0/1514653863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc36406d1d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc364070a90 tx=0x7fc3580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b92e301429fd148 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc361ffb700 1 -- 172.21.15.50:0/1514653863 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358011630 con 0x7fc36406b750 2023-12-19T14:51:27.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc361ffb700 1 -- 172.21.15.50:0/1514653863 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358011790 con 0x7fc36406b750 2023-12-19T14:51:27.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc361ffb700 1 -- 172.21.15.50:0/1514653863 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358011b30 con 0x7fc36406b750 2023-12-19T14:51:27.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 -- 172.21.15.50:0/1514653863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 msgr2=0x7fc36406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 --2- 172.21.15.50:0/1514653863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc36406d1d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc364070a90 tx=0x7fc3580091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 -- 172.21.15.50:0/1514653863 shutdown_connections 2023-12-19T14:51:27.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 --2- 172.21.15.50:0/1514653863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc36406d1d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 --2- 172.21.15.50:0/1514653863 >> v2:172.21.15.125:3300/0 conn(0x7fc3640c5600 0x7fc3640c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 -- 172.21.15.50:0/1514653863 >> 172.21.15.50:0/1514653863 conn(0x7fc364067bb0 msgr2=0x7fc3640cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:27.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 -- 172.21.15.50:0/1514653863 shutdown_connections 2023-12-19T14:51:27.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.771+0000 7fc369a2e700 1 -- 172.21.15.50:0/1514653863 wait complete. 2023-12-19T14:51:27.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.775+0000 7fc369a2e700 1 Processor -- start 2023-12-19T14:51:27.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.775+0000 7fc369a2e700 1 -- start start 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc36406d710 0x7fc3640ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 0x7fc364143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640c6e40 con 0x7fc36406b750 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640b58d0 con 0x7fc36406d710 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640c7090 con 0x7fc3640c5600 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47520/0 (socket says 172.21.15.50:47520) 2023-12-19T14:51:27.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 -- 172.21.15.50:0/3457477663 learned_addr learned my addr 172.21.15.50:0/3457477663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc3627fc700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc36406d710 0x7fc3640ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc3637fe700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 0x7fc364143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 -- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 msgr2=0x7fc364143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 0x7fc364143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 -- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc36406d710 msgr2=0x7fc3640ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc36406d710 0x7fc3640ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 -- 172.21.15.50:0/3457477663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009420 con 0x7fc36406b750 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc3637fe700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 0x7fc364143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc362ffd700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc35400ec80 tx=0x7fc3540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400c9d0 con 0x7fc36406b750 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35400cb30 con 0x7fc36406b750 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354013470 con 0x7fc36406b750 2023-12-19T14:51:27.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3640c7370 con 0x7fc36406b750 2023-12-19T14:51:27.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.779+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3640ced50 con 0x7fc36406b750 2023-12-19T14:51:27.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.783+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3640c77f0 con 0x7fc36406b750 2023-12-19T14:51:27.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.783+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fc354013820 con 0x7fc36406b750 2023-12-19T14:51:27.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.783+0000 7fc34affd700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 0x7fc350062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:27.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.783+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fc35407d290 con 0x7fc36406b750 2023-12-19T14:51:27.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.783+0000 7fc3627fc700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 0x7fc350062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:27.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.787+0000 7fc3627fc700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 0x7fc350062c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3580093f0 tx=0x7fc358017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.791+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc35404d300 con 0x7fc36406b750 2023-12-19T14:51:27.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.983+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc36404be40 con 0x7fc350060750 2023-12-19T14:51:28.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:27.999+0000 7fc34affd700 1 -- 172.21.15.50:0/3457477663 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fc36404be40 con 0x7fc350060750 2023-12-19T14:51:28.002 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:28.003 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:36", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 msgr2=0x7fc350062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 0x7fc350062c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3580093f0 tx=0x7fc358017000 comp rx=0 tx=0).stop 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 msgr2=0x7fc3640c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc35400ec80 tx=0x7fc3540091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 shutdown_connections 2023-12-19T14:51:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc36406d710 0x7fc3640ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc350060750 0x7fc350062c00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3640c5600 0x7fc364143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 --2- 172.21.15.50:0/3457477663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc36406b750 0x7fc3640c7b70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 >> 172.21.15.50:0/3457477663 conn(0x7fc364067bb0 msgr2=0x7fc3640cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 shutdown_connections 2023-12-19T14:51:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.003+0000 7fc369a2e700 1 -- 172.21.15.50:0/3457477663 wait complete. 2023-12-19T14:51:28.030 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:36", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:28.030 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:36', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:28.030 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:28.030 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:28.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:28.030 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-12-19T14:51:28.057 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_01_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:28.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:28.059 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_01_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_01_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:28.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir2 2023-12-19T14:51:28.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.287+0000 7f8582785700 1 Processor -- start 2023-12-19T14:51:28.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.287+0000 7f8582785700 1 -- start start 2023-12-19T14:51:28.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8582785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8582785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f857c15c930 0x7f857c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8582785700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857c0b0ac0 con 0x7f857c06a6a0 2023-12-19T14:51:28.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8582785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857c06b030 con 0x7f857c15c930 2023-12-19T14:51:28.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8582785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c06b210 con 0x7f857c0cb130 2023-12-19T14:51:28.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8580d22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f857c15c930 0x7f857c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f857b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f857c06a6a0 0x7f857c15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58926/0 (socket says 172.21.15.50:58926) 2023-12-19T14:51:28.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8580d22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f857c15c930 0x7f857c06aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47526/0 (socket says 172.21.15.50:47526) 2023-12-19T14:51:28.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.291+0000 7f8580d22700 1 -- 172.21.15.50:0/143658093 learned_addr learned my addr 172.21.15.50:0/143658093 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:28.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f857bfff700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8580d22700 1 -- 172.21.15.50:0/143658093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 msgr2=0x7f857c0cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:28.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8580d22700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c0cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8580d22700 1 -- 172.21.15.50:0/143658093 >> v1:172.21.15.50:6789/0 conn(0x7f857c06a6a0 legacy=0x7f857c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8580d22700 1 -- 172.21.15.50:0/143658093 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c0ca810 con 0x7f857c15c930 2023-12-19T14:51:28.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8580d22700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c15c930 0x7f857c06aaf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8574009f20 tx=0x7f8574009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed78ee3f86ebdaa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f857affd700 1 -- 172.21.15.50:0/143658093 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857400e070 con 0x7f857c15c930 2023-12-19T14:51:28.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f857affd700 1 -- 172.21.15.50:0/143658093 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857400b9d0 con 0x7f857c15c930 2023-12-19T14:51:28.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f857affd700 1 -- 172.21.15.50:0/143658093 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85740092e0 con 0x7f857c15c930 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 -- 172.21.15.50:0/143658093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c15c930 msgr2=0x7f857c06aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c15c930 0x7f857c06aaf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8574009f20 tx=0x7f8574009840 comp rx=0 tx=0).stop 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 -- 172.21.15.50:0/143658093 shutdown_connections 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c15c930 0x7f857c06aaf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 --2- 172.21.15.50:0/143658093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c0cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 -- 172.21.15.50:0/143658093 >> 172.21.15.50:0/143658093 conn(0x7f857c067bb0 msgr2=0x7f857c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:28.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 -- 172.21.15.50:0/143658093 shutdown_connections 2023-12-19T14:51:28.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.295+0000 7f8582785700 1 -- 172.21.15.50:0/143658093 wait complete. 2023-12-19T14:51:28.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 Processor -- start 2023-12-19T14:51:28.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 -- start start 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f857c15c930 0x7f857c089d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c08a840 con 0x7f857c06a6a0 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c15e170 con 0x7f857c15c930 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f8582785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0b4140 con 0x7f857c0cb130 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f857bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f857bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47536/0 (socket says 172.21.15.50:47536) 2023-12-19T14:51:28.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.299+0000 7f857bfff700 1 -- 172.21.15.50:0/2343196348 learned_addr learned my addr 172.21.15.50:0/2343196348 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:28.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857b7fe700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8580d22700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f857c15c930 0x7f857c089d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 -- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 msgr2=0x7f857c157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 -- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f857c15c930 msgr2=0x7f857c089d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f857c15c930 0x7f857c089d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 -- 172.21.15.50:0/2343196348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574009ae0 con 0x7f857c06a6a0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f857bfff700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f856c00ecb0 tx=0x7f856c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c00c850 con 0x7f857c06a6a0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856c00c9b0 con 0x7f857c06a6a0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c157da0 con 0x7f857c06a6a0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c00cd50 con 0x7f857c06a6a0 2023-12-19T14:51:28.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c15db20 con 0x7f857c06a6a0 2023-12-19T14:51:28.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.303+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c071120 con 0x7f857c06a6a0 2023-12-19T14:51:28.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.307+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f856c013680 con 0x7f857c06a6a0 2023-12-19T14:51:28.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.307+0000 7f8563fff700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 0x7f8568062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.307+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f856c07cfe0 con 0x7f857c06a6a0 2023-12-19T14:51:28.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.307+0000 7f857b7fe700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 0x7f8568062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.311+0000 7f857b7fe700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 0x7f8568062c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f857c0b5b00 tx=0x7f8570007eb0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.315+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f856c04d050 con 0x7f857c06a6a0 2023-12-19T14:51:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.515+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f857c0cbe90 con 0x7f8568060750 2023-12-19T14:51:28.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.531+0000 7f8563fff700 1 -- 172.21.15.50:0/2343196348 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f857c0cbe90 con 0x7f8568060750 2023-12-19T14:51:28.537 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:28.537 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:37", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 msgr2=0x7f8568062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 0x7f8568062c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f857c0b5b00 tx=0x7f8570007eb0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 msgr2=0x7f857c0b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f856c00ecb0 tx=0x7f856c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 shutdown_connections 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f857c15c930 0x7f857c089d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8568060750 0x7f8568062c00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f857c0cb130 0x7f857c157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 --2- 172.21.15.50:0/2343196348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f857c06a6a0 0x7f857c0b3c00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 >> 172.21.15.50:0/2343196348 conn(0x7f857c067bb0 msgr2=0x7f857c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 shutdown_connections 2023-12-19T14:51:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.535+0000 7f8582785700 1 -- 172.21.15.50:0/2343196348 wait complete. 2023-12-19T14:51:28.561 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:37", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:28.561 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:37', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:28.562 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:28.562 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:28.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:28.562 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-12-19T14:51:28.589 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:28.589 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:28.590 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:28.591 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:28.591 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:28.592 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_01_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_01_UTC'] 2023-12-19T14:51:28.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir3 2023-12-19T14:51:28.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.803+0000 7fd02de78700 1 Processor -- start 2023-12-19T14:51:28.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.803+0000 7fd02de78700 1 -- start start 2023-12-19T14:51:28.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd02de78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd02815c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd02de78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0280b0ac0 con 0x7fd0280cb130 2023-12-19T14:51:28.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd02de78700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02806b210 con 0x7fd02815c760 2023-12-19T14:51:28.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd02de78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02815cbe0 con 0x7fd0280cb500 2023-12-19T14:51:28.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0277fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd0280cb130 0x7fd02806a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58928/0 (socket says 172.21.15.50:58928) 2023-12-19T14:51:28.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0277fe700 1 -- 172.21.15.50:0/3814162752 learned_addr learned my addr 172.21.15.50:0/3814162752 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:28.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd026ffd700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd02815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392742805 0 0) 0x7fd0280b0ac0 con 0x7fd0280cb130 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd010003570 con 0x7fd0280cb130 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261933316 0 0) 0x7fd02806b210 con 0x7fd02815c760 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0280b0ac0 con 0x7fd02815c760 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909619507 0 0) 0x7fd010003570 con 0x7fd0280cb130 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 msgr2=0x7fd02815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd02815c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 >> v1:172.21.15.125:6789/0 conn(0x7fd02815c760 legacy=0x7fd0280b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0280ca810 con 0x7fd0280cb130 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.807+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fd018001470 con 0x7fd0280cb130 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 >> v1:172.21.15.50:6789/0 conn(0x7fd0280cb130 legacy=0x7fd02806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd010004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd010004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010004d50 0x7fd0100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02815cbe0 con 0x7fd010003c20 2023-12-19T14:51:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01000bb40 con 0x7fd010004d50 2023-12-19T14:51:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01000bd00 con 0x7fd0280cb500 2023-12-19T14:51:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd010004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd027fff700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010004d50 0x7fd0100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd026ffd700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd010004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 -- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 msgr2=0x7fd010004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280cb500 0x7fd010004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 -- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010004d50 msgr2=0x7fd0100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010004d50 0x7fd0100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 -- 172.21.15.50:0/3814162752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd010003570 con 0x7fd010003c20 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0277fe700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd010004010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c2d0 tx=0x7fd018001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f128bbd78debecd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01800cba0 con 0x7fd010003c20 2023-12-19T14:51:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01800cd00 con 0x7fd010003c20 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd0267fc700 1 -- 172.21.15.50:0/3814162752 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018005b70 con 0x7fd010003c20 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 -- 172.21.15.50:0/3814162752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 msgr2=0x7fd010004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 --2- 172.21.15.50:0/3814162752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd010004010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c2d0 tx=0x7fd018001e10 comp rx=0 tx=0).stop 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd027fff700 1 -- 172.21.15.50:0/3814162752 reap_dead start 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 -- 172.21.15.50:0/3814162752 shutdown_connections 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 -- 172.21.15.50:0/3814162752 >> 172.21.15.50:0/3814162752 conn(0x7fd028067bb0 msgr2=0x7fd0280cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 -- 172.21.15.50:0/3814162752 shutdown_connections 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.811+0000 7fd02de78700 1 -- 172.21.15.50:0/3814162752 wait complete. 2023-12-19T14:51:28.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 Processor -- start 2023-12-19T14:51:28.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 -- start start 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 0x7fd02815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 0x7fd02815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0280b3dd0 con 0x7fd010003c20 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0280b31c0 con 0x7fd0280c5460 2023-12-19T14:51:28.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd02de78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0280b34a0 con 0x7fd0280c4ff0 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47548/0 (socket says 172.21.15.50:47548) 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 -- 172.21.15.50:0/1803923557 learned_addr learned my addr 172.21.15.50:0/1803923557 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd027fff700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 0x7fd02815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd026ffd700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 0x7fd02815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 -- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 msgr2=0x7fd02815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 0x7fd02815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 -- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 msgr2=0x7fd02815e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 0x7fd02815e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd0277fe700 1 -- 172.21.15.50:0/1803923557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0280b3780 con 0x7fd010003c20 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd027fff700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 0x7fd02815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.815+0000 7fd026ffd700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 0x7fd02815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:28.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd0277fe700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd010004790 tx=0x7fd018015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01800ce70 con 0x7fd010003c20 2023-12-19T14:51:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0180070d0 con 0x7fd010003c20 2023-12-19T14:51:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0280b3a60 con 0x7fd010003c20 2023-12-19T14:51:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018007490 con 0x7fd010003c20 2023-12-19T14:51:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd028147f90 con 0x7fd010003c20 2023-12-19T14:51:28.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.819+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd028071120 con 0x7fd010003c20 2023-12-19T14:51:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.823+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fd018019030 con 0x7fd010003c20 2023-12-19T14:51:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.823+0000 7fd00f7fe700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 0x7fd014062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.823+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fd01807df70 con 0x7fd010003c20 2023-12-19T14:51:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.823+0000 7fd026ffd700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 0x7fd014062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:28.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.823+0000 7fd026ffd700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 0x7fd014062c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd02815e940 tx=0x7fd01c008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:28.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:28.831+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd01804dfe0 con 0x7fd010003c20 2023-12-19T14:51:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.027+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0280b3dd0 con 0x7fd014060750 2023-12-19T14:51:29.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd00f7fe700 1 -- 172.21.15.50:0/1803923557 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fd0280b3dd0 con 0x7fd014060750 2023-12-19T14:51:29.050 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:29.050 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:38", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 msgr2=0x7fd014062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 0x7fd014062c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd02815e940 tx=0x7fd01c008040 comp rx=0 tx=0).stop 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 msgr2=0x7fd028147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd010004790 tx=0x7fd018015000 comp rx=0 tx=0).stop 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 shutdown_connections 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0280c5460 0x7fd02815e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd014060750 0x7fd014062c00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0280c4ff0 0x7fd02815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.047+0000 7fd02de78700 1 --2- 172.21.15.50:0/1803923557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd010003c20 0x7fd028147a50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.051+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 >> 172.21.15.50:0/1803923557 conn(0x7fd028067bb0 msgr2=0x7fd02815a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:29.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.051+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 shutdown_connections 2023-12-19T14:51:29.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.051+0000 7fd02de78700 1 -- 172.21.15.50:0/1803923557 wait complete. 2023-12-19T14:51:29.073 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:38", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:29.073 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:38', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:29.073 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:29.073 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:29.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:29.074 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-12-19T14:51:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:29.109 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:29.110 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_01_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_01_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:29.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir4 2023-12-19T14:51:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.331+0000 7fac8f545700 1 Processor -- start 2023-12-19T14:51:29.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.331+0000 7fac8f545700 1 -- start start 2023-12-19T14:51:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8f545700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac8815c930 0x7fac88163e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8f545700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac880b0ac0 con 0x7fac880cb130 2023-12-19T14:51:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8f545700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8806b210 con 0x7fac880cb500 2023-12-19T14:51:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8f545700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac880ca810 con 0x7fac8815c930 2023-12-19T14:51:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8d2e1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fac880cb130 0x7fac8806a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58936/0 (socket says 172.21.15.50:58936) 2023-12-19T14:51:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8d2e1700 1 -- 172.21.15.50:0/3680864001 learned_addr learned my addr 172.21.15.50:0/3680864001 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac8dae2700 1 --2- 172.21.15.50:0/3680864001 >> v2:172.21.15.125:3300/0 conn(0x7fac8815c930 0x7fac88163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233367406 0 0) 0x7fac880b0ac0 con 0x7fac880cb130 2023-12-19T14:51:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.335+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac74003570 con 0x7fac880cb130 2023-12-19T14:51:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908869840 0 0) 0x7fac74003570 con 0x7fac880cb130 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 >> v2:172.21.15.125:3300/0 conn(0x7fac8815c930 msgr2=0x7fac88163e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 --2- 172.21.15.50:0/3680864001 >> v2:172.21.15.125:3300/0 conn(0x7fac8815c930 0x7fac88163e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 >> v1:172.21.15.125:6789/0 conn(0x7fac880cb500 legacy=0x7fac8815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880b0190 con 0x7fac880cb130 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fac84001470 con 0x7fac880cb130 2023-12-19T14:51:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 >> v1:172.21.15.50:6789/0 conn(0x7fac880cb130 legacy=0x7fac8806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac74003a60 0x7fac74003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 0x7fac74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac740051c0 0x7fac7400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac880ca810 con 0x7fac74004810 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7400c0d0 con 0x7fac740051c0 2023-12-19T14:51:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7400c290 con 0x7fac74003a60 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 0x7fac74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8d2e1700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac74003a60 0x7fac74003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8dae2700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac740051c0 0x7fac7400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 -- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac74003a60 msgr2=0x7fac74003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac74003a60 0x7fac74003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 -- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac740051c0 msgr2=0x7fac7400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac740051c0 0x7fac7400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 -- 172.21.15.50:0/3680864001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74003570 con 0x7fac74004810 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8dae2700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac740051c0 0x7fac7400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8d2e1700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac74003a60 0x7fac74003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8dae2700 1 -- 172.21.15.50:0/3680864001 reap_dead start 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8cae0700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 0x7fac74004c80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac780018b0 tx=0x7fac78007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8452d468a2a532fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78012420 con 0x7fac74004810 2023-12-19T14:51:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac780125a0 con 0x7fac74004810 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac7ffff700 1 -- 172.21.15.50:0/3680864001 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78012960 con 0x7fac74004810 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 -- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 msgr2=0x7fac74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 0x7fac74004c80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fac780018b0 tx=0x7fac78007d30 comp rx=0 tx=0).stop 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 -- 172.21.15.50:0/3680864001 shutdown_connections 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 --2- 172.21.15.50:0/3680864001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac74004810 0x7fac74004c80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 -- 172.21.15.50:0/3680864001 >> 172.21.15.50:0/3680864001 conn(0x7fac88067bb0 msgr2=0x7fac880cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:29.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 -- 172.21.15.50:0/3680864001 shutdown_connections 2023-12-19T14:51:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.339+0000 7fac8f545700 1 -- 172.21.15.50:0/3680864001 wait complete. 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 Processor -- start 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 -- start start 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac74004810 0x7fac88159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac880c4dc0 0x7fac881693c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8816a2b0 con 0x7fac88169900 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac881596c0 con 0x7fac74004810 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8f545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac880b4bf0 con 0x7fac880c4dc0 2023-12-19T14:51:29.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8dae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8dae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47566/0 (socket says 172.21.15.50:47566) 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.343+0000 7fac8dae2700 1 -- 172.21.15.50:0/1068581869 learned_addr learned my addr 172.21.15.50:0/1068581869 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8d2e1700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac74004810 0x7fac88159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8cae0700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac880c4dc0 0x7fac881693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 -- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac880c4dc0 msgr2=0x7fac881693c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac880c4dc0 0x7fac881693c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 -- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac74004810 msgr2=0x7fac88159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac74004810 0x7fac88159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 -- 172.21.15.50:0/1068581869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880c5230 con 0x7fac88169900 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8dae2700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fac80007e70 tx=0x7fac8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8000fb30 con 0x7fac88169900 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8000fc90 con 0x7fac88169900 2023-12-19T14:51:29.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac80016420 con 0x7fac88169900 2023-12-19T14:51:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac880b4910 con 0x7fac88169900 2023-12-19T14:51:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac88159350 con 0x7fac88169900 2023-12-19T14:51:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.347+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac880b46e0 con 0x7fac88169900 2023-12-19T14:51:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.351+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fac80016850 con 0x7fac88169900 2023-12-19T14:51:29.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.351+0000 7fac7d7fa700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 0x7fac6c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.351+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fac8007db00 con 0x7fac88169900 2023-12-19T14:51:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.351+0000 7fac8d2e1700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 0x7fac6c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.355+0000 7fac8d2e1700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 0x7fac6c062c30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac840015a0 tx=0x7fac84009040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.359+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac8007fe60 con 0x7fac88169900 2023-12-19T14:51:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.547+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac88159850 con 0x7fac6c060780 2023-12-19T14:51:29.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.567+0000 7fac7d7fa700 1 -- 172.21.15.50:0/1068581869 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fac88159850 con 0x7fac6c060780 2023-12-19T14:51:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:29.570 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:38", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 msgr2=0x7fac6c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 0x7fac6c062c30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac840015a0 tx=0x7fac84009040 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 msgr2=0x7fac88169d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fac80007e70 tx=0x7fac8000b040 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 shutdown_connections 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac6c060780 0x7fac6c062c30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac88169900 0x7fac88169d70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac880c4dc0 0x7fac881693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 --2- 172.21.15.50:0/1068581869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac74004810 0x7fac88159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 >> 172.21.15.50:0/1068581869 conn(0x7fac88067bb0 msgr2=0x7fac88157ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 shutdown_connections 2023-12-19T14:51:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.571+0000 7fac8f545700 1 -- 172.21.15.50:0/1068581869 wait complete. 2023-12-19T14:51:29.594 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:38", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:29.594 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:38', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:29.594 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:29.594 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:29.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:29.594 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-12-19T14:51:29.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:29.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:29.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:29.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:29.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:29.631 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_01_UTC'] 2023-12-19T14:51:29.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir5 2023-12-19T14:51:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 Processor -- start 2023-12-19T14:51:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 -- start start 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efc5006af50 0x7efc500cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500cb3f0 con 0x7efc5006d6b0 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc500cc140 con 0x7efc5006af50 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.847+0000 7efc55ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5006d220 con 0x7efc500aedc0 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc4ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efc5006af50 0x7efc500cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35410/0 (socket says 172.21.15.50:35410) 2023-12-19T14:51:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 -- 172.21.15.50:0/3054480012 learned_addr learned my addr 172.21.15.50:0/3054480012 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 -- 172.21.15.50:0/3054480012 >> v2:172.21.15.50:3300/0 conn(0x7efc5006af50 msgr2=0x7efc500cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 --2- 172.21.15.50:0/3054480012 >> v2:172.21.15.50:3300/0 conn(0x7efc5006af50 0x7efc500cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 -- 172.21.15.50:0/3054480012 >> v1:172.21.15.50:6789/0 conn(0x7efc5006d6b0 legacy=0x7efc500af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 -- 172.21.15.50:0/3054480012 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500cb190 con 0x7efc500aedc0 2023-12-19T14:51:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc54ee1700 1 --2- 172.21.15.50:0/3054480012 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc440099b0 tx=0x7efc44009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c218a51cb7e6aa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc4f7fe700 1 -- 172.21.15.50:0/3054480012 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44009ea0 con 0x7efc500aedc0 2023-12-19T14:51:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc4f7fe700 1 -- 172.21.15.50:0/3054480012 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4400b860 con 0x7efc500aedc0 2023-12-19T14:51:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc4f7fe700 1 -- 172.21.15.50:0/3054480012 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4400bc00 con 0x7efc500aedc0 2023-12-19T14:51:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc55ee3700 1 -- 172.21.15.50:0/3054480012 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 msgr2=0x7efc500af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.851+0000 7efc55ee3700 1 --2- 172.21.15.50:0/3054480012 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc440099b0 tx=0x7efc44009250 comp rx=0 tx=0).stop 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- 172.21.15.50:0/3054480012 shutdown_connections 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 --2- 172.21.15.50:0/3054480012 >> v2:172.21.15.50:3300/0 conn(0x7efc5006af50 0x7efc500cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 --2- 172.21.15.50:0/3054480012 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc500aedc0 0x7efc500af190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- 172.21.15.50:0/3054480012 >> 172.21.15.50:0/3054480012 conn(0x7efc500cc760 msgr2=0x7efc500ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- 172.21.15.50:0/3054480012 shutdown_connections 2023-12-19T14:51:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- 172.21.15.50:0/3054480012 wait complete. 2023-12-19T14:51:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 Processor -- start 2023-12-19T14:51:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- start start 2023-12-19T14:51:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc5006af50 0x7efc500c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc500aedc0 0x7efc500c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b4c60 con 0x7efc5006d6b0 2023-12-19T14:51:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b4f40 con 0x7efc500aedc0 2023-12-19T14:51:29.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.855+0000 7efc55ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b5220 con 0x7efc5006af50 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47592/0 (socket says 172.21.15.50:47592) 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 -- 172.21.15.50:0/2253774934 learned_addr learned my addr 172.21.15.50:0/2253774934 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc556e2700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc500aedc0 0x7efc500c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc54ee1700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc5006af50 0x7efc500c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 -- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc5006af50 msgr2=0x7efc500c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc5006af50 0x7efc500c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 -- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc500aedc0 msgr2=0x7efc500c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc500aedc0 0x7efc500c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 -- 172.21.15.50:0/2253774934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440094f0 con 0x7efc5006d6b0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4ffff700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00e930 tx=0x7efc3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c013680 con 0x7efc5006d6b0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c0137e0 con 0x7efc5006d6b0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc500c6e80 con 0x7efc5006d6b0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c013ba0 con 0x7efc5006d6b0 2023-12-19T14:51:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.859+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc501570b0 con 0x7efc5006d6b0 2023-12-19T14:51:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.863+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5004be40 con 0x7efc5006d6b0 2023-12-19T14:51:29.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.863+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7efc3c00c3e0 con 0x7efc5006d6b0 2023-12-19T14:51:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.863+0000 7efc4cff9700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 0x7efc34062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.863+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7efc3c07ce10 con 0x7efc5006d6b0 2023-12-19T14:51:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.863+0000 7efc54ee1700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 0x7efc34062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:29.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.867+0000 7efc54ee1700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 0x7efc34062c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc440099b0 tx=0x7efc44017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:29.871+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efc3c080030 con 0x7efc5006d6b0 2023-12-19T14:51:30.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.063+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc500c6d00 con 0x7efc34060780 2023-12-19T14:51:30.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.083+0000 7efc4cff9700 1 -- 172.21.15.50:0/2253774934 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7efc44009ea0 con 0x7efc34060780 2023-12-19T14:51:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:30.086 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:39", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:30.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 msgr2=0x7efc34062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 0x7efc34062c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc440099b0 tx=0x7efc44017000 comp rx=0 tx=0).stop 2023-12-19T14:51:30.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 msgr2=0x7efc500b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00e930 tx=0x7efc3c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 shutdown_connections 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc500aedc0 0x7efc500c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efc34060780 0x7efc34062c30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc5006d6b0 0x7efc500b4430 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 --2- 172.21.15.50:0/2253774934 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc5006af50 0x7efc500c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 >> 172.21.15.50:0/2253774934 conn(0x7efc500cc760 msgr2=0x7efc500b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 shutdown_connections 2023-12-19T14:51:30.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.087+0000 7efc55ee3700 1 -- 172.21.15.50:0/2253774934 wait complete. 2023-12-19T14:51:30.114 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:39", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:30.114 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:39', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:30.114 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:30.114 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:30.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:30.114 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-12-19T14:51:30.150 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:30.150 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:30.150 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:30.150 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:30.151 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:30.152 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_01_UTC', 'scheduled-2023-12-19-14_48_00_UTC', 'scheduled-2023-12-19-14_49_01_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:30.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir6 2023-12-19T14:51:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.367+0000 7f1ddb40c700 1 Processor -- start 2023-12-19T14:51:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.367+0000 7f1ddb40c700 1 -- start start 2023-12-19T14:51:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddb40c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dd41511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddb40c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd406dca0 con 0x7f1dd4150df0 2023-12-19T14:51:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddb40c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd406d370 con 0x7f1dd40c81b0 2023-12-19T14:51:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddb40c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40c8820 con 0x7f1dd415ae50 2023-12-19T14:51:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dda40a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1dd4150df0 0x7f1dd40c8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58954/0 (socket says 172.21.15.50:58954) 2023-12-19T14:51:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dda40a700 1 -- 172.21.15.50:0/3429673872 learned_addr learned my addr 172.21.15.50:0/3429673872 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dd41511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116637241 0 0) 0x7f1dd406dca0 con 0x7f1dd4150df0 2023-12-19T14:51:30.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc0003570 con 0x7f1dd4150df0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059167805 0 0) 0x7f1dd406d370 con 0x7f1dd40c81b0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd406dca0 con 0x7f1dd40c81b0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753767868 0 0) 0x7f1dc0003570 con 0x7f1dd4150df0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 msgr2=0x7f1dd41511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dd41511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 >> v1:172.21.15.125:6789/0 conn(0x7f1dd40c81b0 legacy=0x7f1dd415a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd406d650 con 0x7f1dd4150df0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f1dd0001470 con 0x7f1dd4150df0 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dd41511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.371+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 >> v1:172.21.15.50:6789/0 conn(0x7f1dd4150df0 legacy=0x7f1dd40c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc0003c20 0x7f1dc0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc0004810 0x7f1dc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40c8820 con 0x7f1dc0003c20 2023-12-19T14:51:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc000bb10 con 0x7f1dc0004810 2023-12-19T14:51:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc000bcd0 con 0x7f1dd415ae50 2023-12-19T14:51:30.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc0003c20 0x7f1dc0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9c09700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc0004810 0x7f1dc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 -- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 msgr2=0x7f1dc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:30.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 -- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc0004810 msgr2=0x7f1dc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc0004810 0x7f1dc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 -- 172.21.15.50:0/3429673872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc009a20 con 0x7f1dc0003c20 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9c09700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc0004810 0x7f1dc0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd415ae50 0x7f1dc00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dda40a700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc0003c20 0x7f1dc0004010 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c2d0 tx=0x7f1dd0001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ea941fa99c9101 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000cba0 con 0x7f1dc0003c20 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd000cd00 con 0x7f1dc0003c20 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1dd9408700 1 -- 172.21.15.50:0/3429673872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0005b70 con 0x7f1dc0003c20 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddb40c700 1 -- 172.21.15.50:0/3429673872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc0003c20 msgr2=0x7f1dc0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/3429673872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc0003c20 0x7f1dc0004010 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c2d0 tx=0x7f1dd0001e10 comp rx=0 tx=0).stop 2023-12-19T14:51:30.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddac0b700 1 -- 172.21.15.50:0/3429673872 reap_dead start 2023-12-19T14:51:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.375+0000 7f1ddb40c700 1 -- 172.21.15.50:0/3429673872 shutdown_connections 2023-12-19T14:51:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.379+0000 7f1ddb40c700 1 -- 172.21.15.50:0/3429673872 >> 172.21.15.50:0/3429673872 conn(0x7f1dd40cdb70 msgr2=0x7f1dd40cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.379+0000 7f1ddb40c700 1 -- 172.21.15.50:0/3429673872 shutdown_connections 2023-12-19T14:51:30.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.379+0000 7f1ddb40c700 1 -- 172.21.15.50:0/3429673872 wait complete. 2023-12-19T14:51:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.379+0000 7f1ddb40c700 1 Processor -- start 2023-12-19T14:51:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.379+0000 7f1ddb40c700 1 -- start start 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc0003c20 0x7f1dd40b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd40b3090 0x7f1dd4155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4169ce0 con 0x7f1dd4155b40 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40b18c0 con 0x7f1dd40b3090 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4068d00 con 0x7f1dc0003c20 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47606/0 (socket says 172.21.15.50:47606) 2023-12-19T14:51:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 -- 172.21.15.50:0/2187489366 learned_addr learned my addr 172.21.15.50:0/2187489366 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1dda40a700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc0003c20 0x7f1dd40b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1dd9c09700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd40b3090 0x7f1dd4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 -- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc0003c20 msgr2=0x7f1dd40b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc0003c20 0x7f1dd40b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 -- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd40b3090 msgr2=0x7f1dd4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd40b3090 0x7f1dd4155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 -- 172.21.15.50:0/2187489366 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4068fe0 con 0x7f1dd4155b40 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddac0b700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4070b50 tx=0x7f1dcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc009d00 con 0x7f1dd4155b40 2023-12-19T14:51:30.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc009e60 con 0x7f1dd4155b40 2023-12-19T14:51:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd40b1270 con 0x7f1dd4155b40 2023-12-19T14:51:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc016650 con 0x7f1dd4155b40 2023-12-19T14:51:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.383+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd40b2290 con 0x7f1dd4155b40 2023-12-19T14:51:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.387+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd40b1a50 con 0x7f1dd4155b40 2023-12-19T14:51:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.387+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f1dcc016a40 con 0x7f1dd4155b40 2023-12-19T14:51:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.391+0000 7f1dca7fc700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 0x7f1db8062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.391+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f1dcc07dc60 con 0x7f1dd4155b40 2023-12-19T14:51:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.391+0000 7f1dda40a700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 0x7f1db8062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.391+0000 7f1dda40a700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 0x7f1db8062c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0001e70 tx=0x7f1dd0015000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.395+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1dcc0516f0 con 0x7f1dd4155b40 2023-12-19T14:51:30.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.591+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1dd404be40 con 0x7f1db8060780 2023-12-19T14:51:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.611+0000 7f1dca7fc700 1 -- 172.21.15.50:0/2187489366 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f1dd404be40 con 0x7f1db8060780 2023-12-19T14:51:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:30.613 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:39", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:30.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 msgr2=0x7f1db8062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 0x7f1db8062c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0001e70 tx=0x7f1dd0015000 comp rx=0 tx=0).stop 2023-12-19T14:51:30.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 msgr2=0x7f1dd40b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4070b50 tx=0x7f1dcc00b040 comp rx=0 tx=0).stop 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 shutdown_connections 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd40b3090 0x7f1dd4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db8060780 0x7f1db8062c30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd4155b40 0x7f1dd40b1d50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 --2- 172.21.15.50:0/2187489366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc0003c20 0x7f1dd40b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 >> 172.21.15.50:0/2187489366 conn(0x7f1dd40cdb70 msgr2=0x7f1dd415a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 shutdown_connections 2023-12-19T14:51:30.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.615+0000 7f1ddb40c700 1 -- 172.21.15.50:0/2187489366 wait complete. 2023-12-19T14:51:30.641 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:39", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:30.642 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:39', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:30.642 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:30.642 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:30.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:30.642 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_01_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:30.674 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:30.675 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:30.675 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:30.675 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_01_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_00_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_01_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_01_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:30.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir7 2023-12-19T14:51:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.903+0000 7f36398cd700 1 Processor -- start 2023-12-19T14:51:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.903+0000 7f36398cd700 1 -- start start 2023-12-19T14:51:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36398cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36398cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36340b0ac0 con 0x7f363415c930 2023-12-19T14:51:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36398cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363406b210 con 0x7f363406a6a0 2023-12-19T14:51:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36398cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36340ca810 con 0x7f36340cb130 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35460/0 (socket says 172.21.15.50:35460) 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36337fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f363415c930 0x7f36340b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58964/0 (socket says 172.21.15.50:58964) 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f36327fc700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f363406a6a0 0x7f363415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:42746/0 (socket says 172.21.15.50:42746) 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 -- 172.21.15.50:0/3809492541 learned_addr learned my addr 172.21.15.50:0/3809492541 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 -- 172.21.15.50:0/3809492541 >> v1:172.21.15.125:6789/0 conn(0x7f363406a6a0 legacy=0x7f363415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3631ffb700 1 -- 172.21.15.50:0/3809492541 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958681874 0 0) 0x7f36340b0ac0 con 0x7f363415c930 2023-12-19T14:51:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 -- 172.21.15.50:0/3809492541 >> v1:172.21.15.50:6789/0 conn(0x7f363415c930 legacy=0x7f36340b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 -- 172.21.15.50:0/3809492541 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340b0190 con 0x7f36340cb130 2023-12-19T14:51:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.907+0000 7f3632ffd700 1 --2- 172.21.15.50:0/3809492541 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36240099e0 tx=0x7f3624009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7807027c48fe1c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f3631ffb700 1 -- 172.21.15.50:0/3809492541 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400b860 con 0x7f36340cb130 2023-12-19T14:51:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f3631ffb700 1 -- 172.21.15.50:0/3809492541 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362400b9c0 con 0x7f36340cb130 2023-12-19T14:51:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f3631ffb700 1 -- 172.21.15.50:0/3809492541 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362400bd60 con 0x7f36340cb130 2023-12-19T14:51:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 -- 172.21.15.50:0/3809492541 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 msgr2=0x7f36340cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 --2- 172.21.15.50:0/3809492541 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36240099e0 tx=0x7f3624009250 comp rx=0 tx=0).stop 2023-12-19T14:51:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 -- 172.21.15.50:0/3809492541 shutdown_connections 2023-12-19T14:51:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 --2- 172.21.15.50:0/3809492541 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f36340cb130 0x7f36340cb500 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 -- 172.21.15.50:0/3809492541 >> 172.21.15.50:0/3809492541 conn(0x7f3634067bb0 msgr2=0x7f36340cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 -- 172.21.15.50:0/3809492541 shutdown_connections 2023-12-19T14:51:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.911+0000 7f36398cd700 1 -- 172.21.15.50:0/3809492541 wait complete. 2023-12-19T14:51:30.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 Processor -- start 2023-12-19T14:51:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 -- start start 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36340cb130 0x7f363415a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f363415c930 0x7f363415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36340b3070 con 0x7f36340cb130 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36341597d0 con 0x7f363415c930 2023-12-19T14:51:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36398cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363415b4c0 con 0x7f363406a6a0 2023-12-19T14:51:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36327fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36340cb130 0x7f363415a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f3632ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f36337fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f363415c930 0x7f363415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f3632ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35470/0 (socket says 172.21.15.50:35470) 2023-12-19T14:51:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.915+0000 7f3632ffd700 1 -- 172.21.15.50:0/3907259606 learned_addr learned my addr 172.21.15.50:0/3907259606 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 -- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f363415c930 msgr2=0x7f363415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f363415c930 0x7f363415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 -- 172.21.15.50:0/3907259606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36340cb130 msgr2=0x7f363415a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36340cb130 0x7f363415a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 -- 172.21.15.50:0/3907259606 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240094f0 con 0x7f363406a6a0 2023-12-19T14:51:30.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f3632ffd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362400c010 tx=0x7f3624017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362401f070 con 0x7f363406a6a0 2023-12-19T14:51:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624011e60 con 0x7f363406a6a0 2023-12-19T14:51:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363415b7a0 con 0x7f363406a6a0 2023-12-19T14:51:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624025420 con 0x7f363406a6a0 2023-12-19T14:51:30.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3634156f40 con 0x7f363406a6a0 2023-12-19T14:51:30.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.919+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634071120 con 0x7f363406a6a0 2023-12-19T14:51:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.923+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f3624023070 con 0x7f363406a6a0 2023-12-19T14:51:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.923+0000 7f361affd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 0x7f3620062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:30.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.927+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f362407e6e0 con 0x7f363406a6a0 2023-12-19T14:51:30.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.927+0000 7f36327fc700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 0x7f3620062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:30.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.927+0000 7f36327fc700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 0x7f3620062cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36340cccc0 tx=0x7f3628007eb0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:30.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:30.931+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f362404e890 con 0x7f363406a6a0 2023-12-19T14:51:31.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.127+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36340b31b0 con 0x7f3620060820 2023-12-19T14:51:31.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.147+0000 7f361affd700 1 -- 172.21.15.50:0/3907259606 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f3628004c60 con 0x7f3620060820 2023-12-19T14:51:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:31.148 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:40", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 msgr2=0x7f3620062cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 0x7f3620062cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36340cccc0 tx=0x7f3628007eb0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 msgr2=0x7f363414eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362400c010 tx=0x7f3624017000 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 shutdown_connections 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f363415c930 0x7f363415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3620060820 0x7f3620062cd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f36340cb130 0x7f363415a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 --2- 172.21.15.50:0/3907259606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f363406a6a0 0x7f363414eeb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 >> 172.21.15.50:0/3907259606 conn(0x7f3634067bb0 msgr2=0x7f36340cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 shutdown_connections 2023-12-19T14:51:31.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.151+0000 7f36398cd700 1 -- 172.21.15.50:0/3907259606 wait complete. 2023-12-19T14:51:31.172 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:40", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:31.172 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:40', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:31.172 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:31.172 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:31.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:31.172 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-12-19T14:51:31.207 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_01_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:31.208 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:31.209 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_01_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_01_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_01_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_01_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_00_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:31.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir8 2023-12-19T14:51:31.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.431+0000 7fc48dd63700 1 Processor -- start 2023-12-19T14:51:31.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 -- start start 2023-12-19T14:51:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc488158f10 0x7fc4880ca390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4880ca8d0 0x7fc4880cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880cade0 con 0x7fc488158b40 2023-12-19T14:51:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880cb0c0 con 0x7fc488158f10 2023-12-19T14:51:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.435+0000 7fc48dd63700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880cb3a0 con 0x7fc4880ca8d0 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc488158f10 0x7fc4880ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc488158f10 0x7fc4880ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47624/0 (socket says 172.21.15.50:47624) 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 -- 172.21.15.50:0/3203733183 learned_addr learned my addr 172.21.15.50:0/3203733183 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48cd61700 1 --1- 172.21.15.50:0/3203733183 >> v1:172.21.15.125:6789/0 conn(0x7fc488158b40 0x7fc4880afb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:42748/0 (socket says 172.21.15.50:42748) 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48d562700 1 --2- 172.21.15.50:0/3203733183 >> v2:172.21.15.125:3300/0 conn(0x7fc4880ca8d0 0x7fc4880cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 -- 172.21.15.50:0/3203733183 >> v2:172.21.15.125:3300/0 conn(0x7fc4880ca8d0 msgr2=0x7fc4880cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 --2- 172.21.15.50:0/3203733183 >> v2:172.21.15.125:3300/0 conn(0x7fc4880ca8d0 0x7fc4880cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 -- 172.21.15.50:0/3203733183 >> v1:172.21.15.125:6789/0 conn(0x7fc488158b40 legacy=0x7fc4880afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 -- 172.21.15.50:0/3203733183 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48806cf70 con 0x7fc488158f10 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc487fff700 1 --2- 172.21.15.50:0/3203733183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158f10 0x7fc4880ca390 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc4880c6d90 tx=0x7fc474009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524d9d4b822c42a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc4877fe700 1 -- 172.21.15.50:0/3203733183 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47400b860 con 0x7fc488158f10 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc4877fe700 1 -- 172.21.15.50:0/3203733183 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47400b9c0 con 0x7fc488158f10 2023-12-19T14:51:31.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc4877fe700 1 -- 172.21.15.50:0/3203733183 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47400bd60 con 0x7fc488158f10 2023-12-19T14:51:31.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 -- 172.21.15.50:0/3203733183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158f10 msgr2=0x7fc4880ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 --2- 172.21.15.50:0/3203733183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158f10 0x7fc4880ca390 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc4880c6d90 tx=0x7fc474009250 comp rx=0 tx=0).stop 2023-12-19T14:51:31.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 -- 172.21.15.50:0/3203733183 shutdown_connections 2023-12-19T14:51:31.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 --2- 172.21.15.50:0/3203733183 >> v2:172.21.15.125:3300/0 conn(0x7fc4880ca8d0 0x7fc4880cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 --2- 172.21.15.50:0/3203733183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158f10 0x7fc4880ca390 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 -- 172.21.15.50:0/3203733183 >> 172.21.15.50:0/3203733183 conn(0x7fc4880670e0 msgr2=0x7fc4880674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:31.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 -- 172.21.15.50:0/3203733183 shutdown_connections 2023-12-19T14:51:31.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.439+0000 7fc48dd63700 1 -- 172.21.15.50:0/3203733183 wait complete. 2023-12-19T14:51:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 Processor -- start 2023-12-19T14:51:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 -- start start 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4880ca8d0 0x7fc4880c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488158f10 0x7fc4880c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880c5c90 con 0x7fc488158b40 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48814a6b0 con 0x7fc488158f10 2023-12-19T14:51:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48dd63700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4880c5080 con 0x7fc4880ca8d0 2023-12-19T14:51:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc487fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc487fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47636/0 (socket says 172.21.15.50:47636) 2023-12-19T14:51:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc487fff700 1 -- 172.21.15.50:0/1981680247 learned_addr learned my addr 172.21.15.50:0/1981680247 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48cd61700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4880ca8d0 0x7fc4880c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.443+0000 7fc48d562700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488158f10 0x7fc4880c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 -- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4880ca8d0 msgr2=0x7fc4880c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4880ca8d0 0x7fc4880c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 -- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488158f10 msgr2=0x7fc4880c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488158f10 0x7fc4880c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 -- 172.21.15.50:0/1981680247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740094f0 con 0x7fc488158b40 2023-12-19T14:51:31.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc487fff700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc4880cd770 tx=0x7fc474017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc474011e40 con 0x7fc488158b40 2023-12-19T14:51:31.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47401a450 con 0x7fc488158b40 2023-12-19T14:51:31.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4880c5360 con 0x7fc488158b40 2023-12-19T14:51:31.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47401a810 con 0x7fc488158b40 2023-12-19T14:51:31.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4880c5920 con 0x7fc488158b40 2023-12-19T14:51:31.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.447+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4880c5e00 con 0x7fc488158b40 2023-12-19T14:51:31.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.451+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fc47401ac00 con 0x7fc488158b40 2023-12-19T14:51:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.451+0000 7fc484ff9700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 0x7fc46c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.451+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fc47407e810 con 0x7fc488158b40 2023-12-19T14:51:31.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.451+0000 7fc48cd61700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 0x7fc46c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.455+0000 7fc48cd61700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 0x7fc46c062c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc47c004fd0 tx=0x7fc47c007eb0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.459+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc47404e880 con 0x7fc488158b40 2023-12-19T14:51:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.655+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc48804be40 con 0x7fc46c060780 2023-12-19T14:51:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.675+0000 7fc484ff9700 1 -- 172.21.15.50:0/1981680247 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fc48804be40 con 0x7fc46c060780 2023-12-19T14:51:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:31.676 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:41", "first": "2023-12-19T14:41:01", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.675+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 msgr2=0x7fc46c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.675+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 0x7fc46c062c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc47c004fd0 tx=0x7fc47c007eb0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.675+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 msgr2=0x7fc48814a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.675+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc4880cd770 tx=0x7fc474017000 comp rx=0 tx=0).stop 2023-12-19T14:51:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 shutdown_connections 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc488158f10 0x7fc4880c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc46c060780 0x7fc46c062c30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc488158b40 0x7fc48814a170 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 --2- 172.21.15.50:0/1981680247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4880ca8d0 0x7fc4880c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 >> 172.21.15.50:0/1981680247 conn(0x7fc4880670e0 msgr2=0x7fc488158190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 shutdown_connections 2023-12-19T14:51:31.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.679+0000 7fc48dd63700 1 -- 172.21.15.50:0/1981680247 wait complete. 2023-12-19T14:51:31.701 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:41", "first": "2023-12-19T14:41:01", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:31.701 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:41', 'first': '2023-12-19T14:41:01', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:31.701 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:31.701 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:31.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:31.701 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-12-19T14:51:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:31.739 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-14_41_00_UTC', 'scheduled-2023-12-19-14_42_00_UTC', 'scheduled-2023-12-19-14_43_00_UTC', 'scheduled-2023-12-19-14_44_00_UTC', 'scheduled-2023-12-19-14_45_00_UTC', 'scheduled-2023-12-19-14_46_01_UTC', 'scheduled-2023-12-19-14_47_00_UTC', 'scheduled-2023-12-19-14_48_01_UTC', 'scheduled-2023-12-19-14_49_00_UTC', 'scheduled-2023-12-19-14_50_01_UTC', 'scheduled-2023-12-19-14_51_00_UTC'] 2023-12-19T14:51:31.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir9 2023-12-19T14:51:31.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.967+0000 7f43672f0700 1 Processor -- start 2023-12-19T14:51:31.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.967+0000 7f43672f0700 1 -- start start 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f43672f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f43600c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f43672f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436006d0d0 con 0x7f4360068a90 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f43672f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436006c7c0 con 0x7f4360159220 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f43672f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43600c7b40 con 0x7f4360158e50 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436588d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4360068a90 0x7f436006d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58968/0 (socket says 172.21.15.50:58968) 2023-12-19T14:51:31.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436588d700 1 -- 172.21.15.50:0/3330305802 learned_addr learned my addr 172.21.15.50:0/3330305802 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310338418 0 0) 0x7f436006d0d0 con 0x7f4360068a90 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436508c700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f43600c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f434c003570 con 0x7f4360068a90 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883840369 0 0) 0x7f434c003570 con 0x7f4360068a90 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 msgr2=0x7f43600c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f43600c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 >> v1:172.21.15.125:6789/0 conn(0x7f4360159220 legacy=0x7f4360068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436006ca80 con 0x7f4360068a90 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f43580017f0 con 0x7f4360068a90 2023-12-19T14:51:31.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 >> v1:172.21.15.50:6789/0 conn(0x7f4360068a90 legacy=0x7f436006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f434c003a90 0x7f434c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f434c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f434c004d50 0x7f434c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600c7b40 con 0x7f434c004d50 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c00bc20 con 0x7f434c003a90 2023-12-19T14:51:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c00bde0 con 0x7f4360158e50 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436508c700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f434c003a90 0x7f434c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436588d700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f434c004d50 0x7f434c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.971+0000 7f436488b700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f434c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 -- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 msgr2=0x7f434c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360158e50 0x7f434c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 -- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f434c003a90 msgr2=0x7f434c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:31.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f434c003a90 0x7f434c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 -- 172.21.15.50:0/3330305802 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c003570 con 0x7f434c004d50 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436508c700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f434c003a90 0x7f434c003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f434c004d50 0x7f434c0051c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f436006ae80 tx=0x7f43580021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d90e808f2dbee8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43580070d0 con 0x7f434c004d50 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358007250 con 0x7f434c004d50 2023-12-19T14:51:31.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f4357fff700 1 -- 172.21.15.50:0/3330305802 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358007610 con 0x7f434c004d50 2023-12-19T14:51:31.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 -- 172.21.15.50:0/3330305802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f434c004d50 msgr2=0x7f434c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 --2- 172.21.15.50:0/3330305802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f434c004d50 0x7f434c0051c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f436006ae80 tx=0x7f43580021d0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f436588d700 1 -- 172.21.15.50:0/3330305802 reap_dead start 2023-12-19T14:51:31.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 -- 172.21.15.50:0/3330305802 shutdown_connections 2023-12-19T14:51:31.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 -- 172.21.15.50:0/3330305802 >> 172.21.15.50:0/3330305802 conn(0x7f43600c8db0 msgr2=0x7f43600c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:31.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 -- 172.21.15.50:0/3330305802 shutdown_connections 2023-12-19T14:51:31.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.975+0000 7f43672f0700 1 -- 172.21.15.50:0/3330305802 wait complete. 2023-12-19T14:51:31.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 Processor -- start 2023-12-19T14:51:31.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 -- start start 2023-12-19T14:51:31.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360149ad0 0x7f43600cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43600c2e60 0x7f43600cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600b0070 con 0x7f4360149700 2023-12-19T14:51:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600af460 con 0x7f43600c2e60 2023-12-19T14:51:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f43672f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43600af740 con 0x7f4360149ad0 2023-12-19T14:51:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f436508c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f436508c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47648/0 (socket says 172.21.15.50:47648) 2023-12-19T14:51:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f436508c700 1 -- 172.21.15.50:0/4187000468 learned_addr learned my addr 172.21.15.50:0/4187000468 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f436588d700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43600c2e60 0x7f43600cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.979+0000 7f436488b700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360149ad0 0x7f43600cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 -- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360149ad0 msgr2=0x7f43600cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360149ad0 0x7f43600cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 -- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43600c2e60 msgr2=0x7f43600cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43600c2e60 0x7f43600cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 -- 172.21.15.50:0/4187000468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600afa20 con 0x7f4360149700 2023-12-19T14:51:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f436508c700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f435c011d70 tx=0x7f435c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c011e70 con 0x7f4360149700 2023-12-19T14:51:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00fc20 con 0x7f4360149700 2023-12-19T14:51:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43600afd00 con 0x7f4360149700 2023-12-19T14:51:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c014440 con 0x7f4360149700 2023-12-19T14:51:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4360168cc0 con 0x7f4360149700 2023-12-19T14:51:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.983+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4360071120 con 0x7f4360149700 2023-12-19T14:51:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.987+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f435c014870 con 0x7f4360149700 2023-12-19T14:51:31.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.987+0000 7f43557fa700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 0x7f4344062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:31.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.987+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f435c018070 con 0x7f4360149700 2023-12-19T14:51:31.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.987+0000 7f436488b700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 0x7f4344062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:31.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.987+0000 7f436488b700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 0x7f4344062c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43600b0220 tx=0x7f4350007d30 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:31.995+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f435c04da30 con 0x7f4360149700 2023-12-19T14:51:32.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.191+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f436006cd60 con 0x7f4344060750 2023-12-19T14:51:32.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.207+0000 7f43557fa700 1 -- 172.21.15.50:0/4187000468 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f436006cd60 con 0x7f4344060750 2023-12-19T14:51:32.210 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:32.210 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:41", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 msgr2=0x7f4344062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 0x7f4344062c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43600b0220 tx=0x7f4350007d30 comp rx=0 tx=0).stop 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 msgr2=0x7f43600c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f435c011d70 tx=0x7f435c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 shutdown_connections 2023-12-19T14:51:32.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43600c2e60 0x7f43600cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4344060750 0x7f4344062c00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4360149ad0 0x7f43600cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 --2- 172.21.15.50:0/4187000468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360149700 0x7f43600c2920 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 >> 172.21.15.50:0/4187000468 conn(0x7f43600c8db0 msgr2=0x7f43600ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 shutdown_connections 2023-12-19T14:51:32.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.211+0000 7f43672f0700 1 -- 172.21.15.50:0/4187000468 wait complete. 2023-12-19T14:51:32.235 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:40:41", "first": "2023-12-19T14:41:00", "last": "2023-12-19T14:51:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T14:51:32.235 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:40:41', 'first': '2023-12-19T14:41:00', 'last': '2023-12-19T14:51:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T14:51:32.235 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T14:51:32.235 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T14:51:32.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T14:51:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.399+0000 7ff66942e700 1 Processor -- start 2023-12-19T14:51:32.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 -- start start 2023-12-19T14:51:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff664156e70 0x7ff66406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66406b060 0x7ff6640abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6640ac8e0 con 0x7ff6640ac480 2023-12-19T14:51:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66406e180 con 0x7ff66406b060 2023-12-19T14:51:32.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff66942e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66406d7b0 con 0x7ff664156e70 2023-12-19T14:51:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff6627fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66406b060 0x7ff6640abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff6627fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff66406b060 0x7ff6640abf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47664/0 (socket says 172.21.15.50:47664) 2023-12-19T14:51:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff6627fc700 1 -- 172.21.15.50:0/1836371176 learned_addr learned my addr 172.21.15.50:0/1836371176 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff6637fe700 1 --1- 172.21.15.50:0/1836371176 >> v1:172.21.15.50:6789/0 conn(0x7ff6640ac480 0x7ff6640aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58974/0 (socket says 172.21.15.50:58974) 2023-12-19T14:51:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.403+0000 7ff662ffd700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff664156e70 0x7ff66406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff6627fc700 1 -- 172.21.15.50:0/1836371176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff664156e70 msgr2=0x7ff66406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff6627fc700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff664156e70 0x7ff66406dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff6627fc700 1 -- 172.21.15.50:0/1836371176 >> v1:172.21.15.50:6789/0 conn(0x7ff6640ac480 legacy=0x7ff6640aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff6627fc700 1 -- 172.21.15.50:0/1836371176 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640ae570 con 0x7ff66406b060 2023-12-19T14:51:32.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff6627fc700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66406b060 0x7ff6640abf40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff6580099b0 tx=0x7ff658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a21c481d553bd39 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff661ffb700 1 -- 172.21.15.50:0/1836371176 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65800b860 con 0x7ff66406b060 2023-12-19T14:51:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff661ffb700 1 -- 172.21.15.50:0/1836371176 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65800b9c0 con 0x7ff66406b060 2023-12-19T14:51:32.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff661ffb700 1 -- 172.21.15.50:0/1836371176 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65800bd60 con 0x7ff66406b060 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 -- 172.21.15.50:0/1836371176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66406b060 msgr2=0x7ff6640abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66406b060 0x7ff6640abf40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff6580099b0 tx=0x7ff658009250 comp rx=0 tx=0).stop 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 -- 172.21.15.50:0/1836371176 shutdown_connections 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff66406b060 0x7ff6640abf40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 --2- 172.21.15.50:0/1836371176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff664156e70 0x7ff66406dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 -- 172.21.15.50:0/1836371176 >> 172.21.15.50:0/1836371176 conn(0x7ff664067bb0 msgr2=0x7ff6640cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 -- 172.21.15.50:0/1836371176 shutdown_connections 2023-12-19T14:51:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.407+0000 7ff66942e700 1 -- 172.21.15.50:0/1836371176 wait complete. 2023-12-19T14:51:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 Processor -- start 2023-12-19T14:51:32.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 -- start start 2023-12-19T14:51:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff66406b060 0x7ff66415ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6640ac480 0x7ff6640b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664168e20 con 0x7ff664156e70 2023-12-19T14:51:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66415bf70 con 0x7ff66406b060 2023-12-19T14:51:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff66942e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640c4f60 con 0x7ff6640ac480 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff6637fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff6637fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47672/0 (socket says 172.21.15.50:47672) 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.411+0000 7ff6637fe700 1 -- 172.21.15.50:0/1066430322 learned_addr learned my addr 172.21.15.50:0/1066430322 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6627fc700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6640ac480 0x7ff6640b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff662ffd700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff66406b060 0x7ff66415ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 -- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6640ac480 msgr2=0x7ff6640b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6640ac480 0x7ff6640b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 -- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff66406b060 msgr2=0x7ff66415ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff66406b060 0x7ff66415ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 -- 172.21.15.50:0/1066430322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6580094f0 con 0x7ff664156e70 2023-12-19T14:51:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff6637fe700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00cd30 tx=0x7ff65c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00ce90 con 0x7ff664156e70 2023-12-19T14:51:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c005df0 con 0x7ff664156e70 2023-12-19T14:51:32.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00aac0 con 0x7ff664156e70 2023-12-19T14:51:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6640c5240 con 0x7ff664156e70 2023-12-19T14:51:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6640c5800 con 0x7ff664156e70 2023-12-19T14:51:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.415+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664168f60 con 0x7ff664156e70 2023-12-19T14:51:32.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.419+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7ff65c0070f0 con 0x7ff664156e70 2023-12-19T14:51:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.419+0000 7ff64affd700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 0x7ff650062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.419+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7ff65c07cc70 con 0x7ff664156e70 2023-12-19T14:51:32.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.419+0000 7ff662ffd700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 0x7ff650062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:32.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.419+0000 7ff662ffd700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 0x7ff650062c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff654006fd0 tx=0x7ff654008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:32.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.427+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff664168f60 con 0x7ff664156e70 2023-12-19T14:51:32.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.623+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6640ac8e0 con 0x7ff650060750 2023-12-19T14:51:32.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.779+0000 7ff64affd700 1 -- 172.21.15.50:0/1066430322 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff6640ac8e0 con 0x7ff650060750 2023-12-19T14:51:32.782 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-12-19T14:51:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 msgr2=0x7ff650062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 0x7ff650062c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff654006fd0 tx=0x7ff654008040 comp rx=0 tx=0).stop 2023-12-19T14:51:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 msgr2=0x7ff66415c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:32.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00cd30 tx=0x7ff65c001b10 comp rx=0 tx=0).stop 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 shutdown_connections 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff650060750 0x7ff650062c00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff664156e70 0x7ff66415c370 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6640ac480 0x7ff6640b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 --2- 172.21.15.50:0/1066430322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff66406b060 0x7ff66415ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 >> 172.21.15.50:0/1066430322 conn(0x7ff664067bb0 msgr2=0x7ff6640aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 shutdown_connections 2023-12-19T14:51:32.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:32.783+0000 7ff66942e700 1 -- 172.21.15.50:0/1066430322 wait complete. 2023-12-19T14:51:32.812 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-12-19T14:51:32.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:32.812 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:32.846 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:32.847 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:32.847 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:32.847 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:32.848 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:32.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:32.849 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:32.935 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:32.935 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:32.935 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:33.026 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:33.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.026 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:33.107 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:33.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.107 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:33.193 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:33.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.194 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_45_01_UTC) 2023-12-19T14:51:33.280 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:33.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.280 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:33.368 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:33.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.369 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_47_01_UTC) 2023-12-19T14:51:33.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:33.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.453 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:33.534 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:33.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.535 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:33.614 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:33.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.614 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:33.696 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:33.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.696 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:33.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:33.778 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-12-19T14:51:33.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T14:51:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.075+0000 7fc943185700 1 Processor -- start 2023-12-19T14:51:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.075+0000 7fc943185700 1 -- start start 2023-12-19T14:51:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc943185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc943185700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93c15c930 0x7fc93c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc943185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c0b0ac0 con 0x7fc93c06a6a0 2023-12-19T14:51:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc943185700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c06b030 con 0x7fc93c15c930 2023-12-19T14:51:34.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc943185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c06b210 con 0x7fc93c0cb130 2023-12-19T14:51:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc93bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc93c06a6a0 0x7fc93c15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58986/0 (socket says 172.21.15.50:58986) 2023-12-19T14:51:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc93bfff700 1 -- 172.21.15.50:0/2012542763 learned_addr learned my addr 172.21.15.50:0/2012542763 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 --2- 172.21.15.50:0/2012542763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc941722700 1 --2- 172.21.15.50:0/2012542763 >> v2:172.21.15.125:3300/0 conn(0x7fc93c15c930 0x7fc93c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc93b7fe700 1 -- 172.21.15.50:0/2012542763 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017522483 0 0) 0x7fc93c0b0ac0 con 0x7fc93c06a6a0 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc93b7fe700 1 -- 172.21.15.50:0/2012542763 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c003570 con 0x7fc93c06a6a0 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 -- 172.21.15.50:0/2012542763 >> v2:172.21.15.125:3300/0 conn(0x7fc93c15c930 msgr2=0x7fc93c06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 --2- 172.21.15.50:0/2012542763 >> v2:172.21.15.125:3300/0 conn(0x7fc93c15c930 0x7fc93c06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 -- 172.21.15.50:0/2012542763 >> v1:172.21.15.50:6789/0 conn(0x7fc93c06a6a0 legacy=0x7fc93c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 -- 172.21.15.50:0/2012542763 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c0ca810 con 0x7fc93c0cb130 2023-12-19T14:51:34.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc940f21700 1 --2- 172.21.15.50:0/2012542763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c0cb500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9300098e0 tx=0x7fc9300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=554ef5419054c789 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.079+0000 7fc93b7fe700 1 -- 172.21.15.50:0/2012542763 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930011630 con 0x7fc93c0cb130 2023-12-19T14:51:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc93b7fe700 1 -- 172.21.15.50:0/2012542763 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc930011790 con 0x7fc93c0cb130 2023-12-19T14:51:34.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc93b7fe700 1 -- 172.21.15.50:0/2012542763 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930011b30 con 0x7fc93c0cb130 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 -- 172.21.15.50:0/2012542763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 msgr2=0x7fc93c0cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 --2- 172.21.15.50:0/2012542763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c0cb500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9300098e0 tx=0x7fc9300091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 -- 172.21.15.50:0/2012542763 shutdown_connections 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 --2- 172.21.15.50:0/2012542763 >> v2:172.21.15.125:3300/0 conn(0x7fc93c15c930 0x7fc93c06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 --2- 172.21.15.50:0/2012542763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c0cb500 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 -- 172.21.15.50:0/2012542763 >> 172.21.15.50:0/2012542763 conn(0x7fc93c067bb0 msgr2=0x7fc93c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 -- 172.21.15.50:0/2012542763 shutdown_connections 2023-12-19T14:51:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 -- 172.21.15.50:0/2012542763 wait complete. 2023-12-19T14:51:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.083+0000 7fc943185700 1 Processor -- start 2023-12-19T14:51:34.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 -- start start 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 0x7fc93c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c155540 con 0x7fc93c15c930 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c158f20 con 0x7fc93c06a6a0 2023-12-19T14:51:34.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c1594e0 con 0x7fc93c0cb130 2023-12-19T14:51:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47680/0 (socket says 172.21.15.50:47680) 2023-12-19T14:51:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 -- 172.21.15.50:0/1194513847 learned_addr learned my addr 172.21.15.50:0/1194513847 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc940f21700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 0x7fc93c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc93bfff700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 -- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 msgr2=0x7fc93c15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c15abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 -- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 msgr2=0x7fc93c159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 0x7fc93c159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 -- 172.21.15.50:0/1194513847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930009420 con 0x7fc93c15c930 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc940f21700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 0x7fc93c159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc941722700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc934009da0 tx=0x7fc93400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400cb50 con 0x7fc93c15c930 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400f040 con 0x7fc93c15c930 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c1585b0 con 0x7fc93c15c930 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.087+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934014480 con 0x7fc93c15c930 2023-12-19T14:51:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.091+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c158b30 con 0x7fc93c15c930 2023-12-19T14:51:34.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.091+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c158d50 con 0x7fc93c15c930 2023-12-19T14:51:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.091+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fc9340147f0 con 0x7fc93c15c930 2023-12-19T14:51:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.095+0000 7fc938ff9700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 0x7fc920062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.095+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fc93407db40 con 0x7fc93c15c930 2023-12-19T14:51:34.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.095+0000 7fc940f21700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 0x7fc920062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:34.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.095+0000 7fc940f21700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 0x7fc920062c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc9300091e0 tx=0x7fc930017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:34.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.099+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc93404daa0 con 0x7fc93c15c930 2023-12-19T14:51:34.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.291+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc93c071120 con 0x7fc920060780 2023-12-19T14:51:34.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.447+0000 7fc938ff9700 1 -- 172.21.15.50:0/1194513847 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc93c071120 con 0x7fc920060780 2023-12-19T14:51:34.452 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 msgr2=0x7fc920062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 0x7fc920062c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc9300091e0 tx=0x7fc930017000 comp rx=0 tx=0).stop 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 msgr2=0x7fc93c15b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc934009da0 tx=0x7fc93400b040 comp rx=0 tx=0).stop 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 shutdown_connections 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc920060780 0x7fc920062c30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc93c15c930 0x7fc93c15b0f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc93c0cb130 0x7fc93c15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 --2- 172.21.15.50:0/1194513847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93c06a6a0 0x7fc93c159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.451+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 >> 172.21.15.50:0/1194513847 conn(0x7fc93c067bb0 msgr2=0x7fc93c06dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.455+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 shutdown_connections 2023-12-19T14:51:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:34.455+0000 7fc943185700 1 -- 172.21.15.50:0/1194513847 wait complete. 2023-12-19T14:51:34.479 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-12-19T14:51:34.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.479 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:34.519 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:34.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:34.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.521 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:34.609 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:34.610 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.610 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:34.693 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:34.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.693 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:34.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:34.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.776 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_44_01_UTC) 2023-12-19T14:51:34.862 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:34.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.862 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:34.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:34.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:34.945 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:35.027 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:35.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.027 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:35.114 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:35.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.115 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:35.200 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:35.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.201 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_49_01_UTC) 2023-12-19T14:51:35.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:35.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.287 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:35.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:35.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.371 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:35.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:35.457 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-12-19T14:51:35.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T14:51:35.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.743+0000 7f5c98e78700 1 Processor -- start 2023-12-19T14:51:35.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.743+0000 7f5c98e78700 1 -- start start 2023-12-19T14:51:35.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.743+0000 7f5c98e78700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c94156cd0 0x7f5c941570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c940aeaa0 0x7f5c9406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9406d3d0 con 0x7f5c940ae200 2023-12-19T14:51:35.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9406b6e0 con 0x7f5c94156cd0 2023-12-19T14:51:35.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9406c7c0 con 0x7f5c940aeaa0 2023-12-19T14:51:35.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c92d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c940aeaa0 0x7f5c9406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c94156cd0 0x7f5c941570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c94156cd0 0x7f5c941570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48110/0 (socket says 172.21.15.50:48110) 2023-12-19T14:51:35.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 -- 172.21.15.50:0/2413726215 learned_addr learned my addr 172.21.15.50:0/2413726215 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:35.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 -- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c940aeaa0 msgr2=0x7f5c9406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:35.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 --2- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c940aeaa0 0x7f5c9406d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:35.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 -- 172.21.15.50:0/2413726215 >> v1:172.21.15.125:6789/0 conn(0x7f5c940ae200 legacy=0x7f5c9406afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:35.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 -- 172.21.15.50:0/2413726215 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9406caa0 con 0x7f5c94156cd0 2023-12-19T14:51:35.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9259c700 1 --2- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c94156cd0 0x7f5c941570a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c840099b0 tx=0x7f5c84009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae771da1e29b36fa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:35.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9159a700 1 -- 172.21.15.50:0/2413726215 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84009ea0 con 0x7f5c94156cd0 2023-12-19T14:51:35.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9159a700 1 -- 172.21.15.50:0/2413726215 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8400b860 con 0x7f5c94156cd0 2023-12-19T14:51:35.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c9159a700 1 -- 172.21.15.50:0/2413726215 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400bc00 con 0x7f5c94156cd0 2023-12-19T14:51:35.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 -- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c94156cd0 msgr2=0x7f5c941570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:35.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.747+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c94156cd0 0x7f5c941570a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c840099b0 tx=0x7f5c84009250 comp rx=0 tx=0).stop 2023-12-19T14:51:35.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 -- 172.21.15.50:0/2413726215 shutdown_connections 2023-12-19T14:51:35.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c940aeaa0 0x7f5c9406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:35.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2413726215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c94156cd0 0x7f5c941570a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:35.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 -- 172.21.15.50:0/2413726215 >> 172.21.15.50:0/2413726215 conn(0x7f5c94067bb0 msgr2=0x7f5c940cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:35.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 -- 172.21.15.50:0/2413726215 shutdown_connections 2023-12-19T14:51:35.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 -- 172.21.15.50:0/2413726215 wait complete. 2023-12-19T14:51:35.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.751+0000 7f5c98e78700 1 Processor -- start 2023-12-19T14:51:35.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 -- start start 2023-12-19T14:51:35.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c940aeaa0 0x7f5c941589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 0x7f5c940c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940c4550 con 0x7f5c940ae200 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94157a50 con 0x7f5c940aeaa0 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c98e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94157d30 con 0x7f5c94156cd0 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47692/0 (socket says 172.21.15.50:47692) 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 -- 172.21.15.50:0/2333103103 learned_addr learned my addr 172.21.15.50:0/2333103103 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:35.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c92d9d700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 0x7f5c940c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c91d9b700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c940aeaa0 0x7f5c941589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 -- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 msgr2=0x7f5c940c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 0x7f5c940c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 -- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c940aeaa0 msgr2=0x7f5c941589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c940aeaa0 0x7f5c941589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c9259c700 1 -- 172.21.15.50:0/2333103103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840094f0 con 0x7f5c940ae200 2023-12-19T14:51:35.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.755+0000 7f5c92d9d700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 0x7f5c940c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:35.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c9259c700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5c940697b0 tx=0x7f5c84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:35.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400b9d0 con 0x7f5c940ae200 2023-12-19T14:51:35.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8400bb30 con 0x7f5c940ae200 2023-12-19T14:51:35.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84011670 con 0x7f5c940ae200 2023-12-19T14:51:35.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c94158010 con 0x7f5c940ae200 2023-12-19T14:51:35.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c94158f00 con 0x7f5c940ae200 2023-12-19T14:51:35.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.759+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c94071120 con 0x7f5c940ae200 2023-12-19T14:51:35.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.763+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f5c84011a60 con 0x7f5c940ae200 2023-12-19T14:51:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.763+0000 7f5c827fc700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 0x7f5c74062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.763+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f5c8407e680 con 0x7f5c940ae200 2023-12-19T14:51:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.763+0000 7f5c91d9b700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 0x7f5c74062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:35.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.763+0000 7f5c91d9b700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 0x7f5c74062c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c940ca8a0 tx=0x7f5c88007ea0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:35.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.771+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c8404e6f0 con 0x7f5c940ae200 2023-12-19T14:51:35.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:35.967+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c940c5ed0 con 0x7f5c74060780 2023-12-19T14:51:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.111+0000 7f5c827fc700 1 -- 172.21.15.50:0/2333103103 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5c940c5ed0 con 0x7f5c74060780 2023-12-19T14:51:36.112 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 msgr2=0x7f5c74062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 0x7f5c74062c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c940ca8a0 tx=0x7f5c88007ea0 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 msgr2=0x7f5c94144910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5c940697b0 tx=0x7f5c84017000 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 shutdown_connections 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c940aeaa0 0x7f5c941589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c74060780 0x7f5c74062c30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5c94156cd0 0x7f5c940c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 --2- 172.21.15.50:0/2333103103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c940ae200 0x7f5c94144910 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 >> 172.21.15.50:0/2333103103 conn(0x7f5c94067bb0 msgr2=0x7f5c940cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:36.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 shutdown_connections 2023-12-19T14:51:36.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:36.115+0000 7f5c98e78700 1 -- 172.21.15.50:0/2333103103 wait complete. 2023-12-19T14:51:36.139 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-12-19T14:51:36.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.139 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_01_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:36.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:36.179 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:36.179 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:36.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.180 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:36.260 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_42_01_UTC 2023-12-19T14:51:36.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.261 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_42_01_UTC) 2023-12-19T14:51:36.347 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:36.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.348 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:36.436 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:36.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.437 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:36.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:36.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.520 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:36.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:36.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.603 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:36.687 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:36.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.687 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:36.772 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:36.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.773 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_48_01_UTC) 2023-12-19T14:51:36.859 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:36.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.860 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:36.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:51:36.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:51:36.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:36.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:36.948 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:37.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:37.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:37.078 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:37.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:37.158 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-12-19T14:51:37.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T14:51:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.455+0000 7f6376345700 1 Processor -- start 2023-12-19T14:51:37.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.455+0000 7f6376345700 1 -- start start 2023-12-19T14:51:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f6376345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63700cb130 0x7f63700cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f6376345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f637006a6a0 0x7f637006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f6376345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63700b0ac0 con 0x7f637015c220 2023-12-19T14:51:37.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f6376345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637006b030 con 0x7f637006a6a0 2023-12-19T14:51:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f6376345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637006b210 con 0x7f63700cb130 2023-12-19T14:51:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f637006a6a0 0x7f637006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f636ffff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63700cb130 0x7f63700cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f637006a6a0 0x7f637006aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51978/0 (socket says 172.21.15.50:51978) 2023-12-19T14:51:37.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 -- 172.21.15.50:0/1537602986 learned_addr learned my addr 172.21.15.50:0/1537602986 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 -- 172.21.15.50:0/1537602986 >> v2:172.21.15.125:3300/0 conn(0x7f63700cb130 msgr2=0x7f63700cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 --2- 172.21.15.50:0/1537602986 >> v2:172.21.15.125:3300/0 conn(0x7f63700cb130 0x7f63700cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 -- 172.21.15.50:0/1537602986 >> v1:172.21.15.125:6789/0 conn(0x7f637015c220 legacy=0x7f637006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.459+0000 7f63677fe700 1 -- 172.21.15.50:0/1537602986 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63700ca810 con 0x7f637006a6a0 2023-12-19T14:51:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f63677fe700 1 --2- 172.21.15.50:0/1537602986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f637006a6a0 0x7f637006aaf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f63700694e0 tx=0x7f6358009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a802f67a1520407 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:37.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f636f7fe700 1 -- 172.21.15.50:0/1537602986 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358009e70 con 0x7f637006a6a0 2023-12-19T14:51:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f636f7fe700 1 -- 172.21.15.50:0/1537602986 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800b860 con 0x7f637006a6a0 2023-12-19T14:51:37.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f636f7fe700 1 -- 172.21.15.50:0/1537602986 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800bc00 con 0x7f637006a6a0 2023-12-19T14:51:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 -- 172.21.15.50:0/1537602986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f637006a6a0 msgr2=0x7f637006aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 --2- 172.21.15.50:0/1537602986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f637006a6a0 0x7f637006aaf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f63700694e0 tx=0x7f6358009250 comp rx=0 tx=0).stop 2023-12-19T14:51:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 -- 172.21.15.50:0/1537602986 shutdown_connections 2023-12-19T14:51:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 --2- 172.21.15.50:0/1537602986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f637006a6a0 0x7f637006aaf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 --2- 172.21.15.50:0/1537602986 >> v2:172.21.15.125:3300/0 conn(0x7f63700cb130 0x7f63700cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 -- 172.21.15.50:0/1537602986 >> 172.21.15.50:0/1537602986 conn(0x7f6370067bb0 msgr2=0x7f63700cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 -- 172.21.15.50:0/1537602986 shutdown_connections 2023-12-19T14:51:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.463+0000 7f6376345700 1 -- 172.21.15.50:0/1537602986 wait complete. 2023-12-19T14:51:37.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 Processor -- start 2023-12-19T14:51:37.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 -- start start 2023-12-19T14:51:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f637006a6a0 0x7f63700c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f637015c220 0x7f6370152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370089350 con 0x7f63700cb130 2023-12-19T14:51:37.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637008a600 con 0x7f637006a6a0 2023-12-19T14:51:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6370089560 con 0x7f637015c220 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51992/0 (socket says 172.21.15.50:51992) 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 -- 172.21.15.50:0/2433881779 learned_addr learned my addr 172.21.15.50:0/2433881779 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f636ffff700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f637006a6a0 0x7f63700c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63748e2700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f637015c220 0x7f6370152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 -- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f637015c220 msgr2=0x7f6370152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f637015c220 0x7f6370152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 -- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f637006a6a0 msgr2=0x7f63700c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f637006a6a0 0x7f63700c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 -- 172.21.15.50:0/2433881779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63580094f0 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f63677fe700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f637006f300 tx=0x7f6358017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800b9d0 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800bb30 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.467+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6370089840 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.471+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358025670 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.471+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6370089e00 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.471+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370071120 con 0x7f63700cb130 2023-12-19T14:51:37.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.471+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f6358023070 con 0x7f63700cb130 2023-12-19T14:51:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.475+0000 7f636cff9700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 0x7f6350062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.475+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f635807e390 con 0x7f63700cb130 2023-12-19T14:51:37.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.475+0000 7f636ffff700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 0x7f6350062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:37.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.475+0000 7f636ffff700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 0x7f6350062c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f636000f570 tx=0x7f636000b040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:37.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.479+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f635804e400 con 0x7f63700cb130 2023-12-19T14:51:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.675+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f63700cbe90 con 0x7f6350060750 2023-12-19T14:51:37.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.823+0000 7f636cff9700 1 -- 172.21.15.50:0/2433881779 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f63700cbe90 con 0x7f6350060750 2023-12-19T14:51:37.826 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 msgr2=0x7f6350062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 0x7f6350062c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f636000f570 tx=0x7f636000b040 comp rx=0 tx=0).stop 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 msgr2=0x7f6370088e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f637006f300 tx=0x7f6358017000 comp rx=0 tx=0).stop 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 shutdown_connections 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6350060750 0x7f6350062c00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f637015c220 0x7f6370152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63700cb130 0x7f6370088e10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 --2- 172.21.15.50:0/2433881779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f637006a6a0 0x7f63700c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:37.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 >> 172.21.15.50:0/2433881779 conn(0x7f6370067bb0 msgr2=0x7f637015b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:37.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 shutdown_connections 2023-12-19T14:51:37.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:37.827+0000 7f6376345700 1 -- 172.21.15.50:0/2433881779 wait complete. 2023-12-19T14:51:37.850 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-12-19T14:51:37.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:37.851 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:37.893 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:37.894 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:37.894 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:37.894 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:37.894 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:37.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:37.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:37.895 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:37.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:37.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:37.982 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:38.063 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:38.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.063 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:38.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:38.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.148 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:38.235 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:38.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.235 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:38.321 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:38.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.322 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:38.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:38.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.403 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_47_01_UTC) 2023-12-19T14:51:38.493 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:38.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.493 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:38.577 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:38.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.577 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:38.665 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:38.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.665 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:38.749 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:38.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.749 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-14_51_01_UTC) 2023-12-19T14:51:38.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:38.835 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-12-19T14:51:38.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T14:51:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.155+0000 7fa10b416700 1 Processor -- start 2023-12-19T14:51:39.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.155+0000 7fa10b416700 1 -- start start 2023-12-19T14:51:39.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa10b416700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa104154bc0 0x7fa10406b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa10b416700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa104154f90 0x7fa10406a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa10b416700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040af100 con 0x7fa10406aac0 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa10b416700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10406af10 con 0x7fa104154f90 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa10b416700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1040aead0 con 0x7fa104154bc0 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa109c13700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa104154f90 0x7fa10406a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa109c13700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa104154f90 0x7fa10406a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52004/0 (socket says 172.21.15.50:52004) 2023-12-19T14:51:39.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.159+0000 7fa109c13700 1 -- 172.21.15.50:0/2657295618 learned_addr learned my addr 172.21.15.50:0/2657295618 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa10a414700 1 --2- 172.21.15.50:0/2657295618 >> v2:172.21.15.125:3300/0 conn(0x7fa104154bc0 0x7fa10406b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109c13700 1 -- 172.21.15.50:0/2657295618 >> v2:172.21.15.125:3300/0 conn(0x7fa104154bc0 msgr2=0x7fa10406b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109c13700 1 --2- 172.21.15.50:0/2657295618 >> v2:172.21.15.125:3300/0 conn(0x7fa104154bc0 0x7fa10406b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109c13700 1 -- 172.21.15.50:0/2657295618 >> v1:172.21.15.50:6789/0 conn(0x7fa10406aac0 legacy=0x7fa10406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:39.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109c13700 1 -- 172.21.15.50:0/2657295618 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10406bc40 con 0x7fa104154f90 2023-12-19T14:51:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109c13700 1 --2- 172.21.15.50:0/2657295618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa104154f90 0x7fa10406a580 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa104068020 tx=0x7fa0f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca47e90e450ee806 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:39.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109412700 1 -- 172.21.15.50:0/2657295618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4009e70 con 0x7fa104154f90 2023-12-19T14:51:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109412700 1 -- 172.21.15.50:0/2657295618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f400b820 con 0x7fa104154f90 2023-12-19T14:51:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa109412700 1 -- 172.21.15.50:0/2657295618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400bbc0 con 0x7fa104154f90 2023-12-19T14:51:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa10b416700 1 -- 172.21.15.50:0/2657295618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa104154f90 msgr2=0x7fa10406a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.163+0000 7fa10b416700 1 --2- 172.21.15.50:0/2657295618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa104154f90 0x7fa10406a580 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa104068020 tx=0x7fa0f4009250 comp rx=0 tx=0).stop 2023-12-19T14:51:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 -- 172.21.15.50:0/2657295618 shutdown_connections 2023-12-19T14:51:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 --2- 172.21.15.50:0/2657295618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa104154f90 0x7fa10406a580 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 --2- 172.21.15.50:0/2657295618 >> v2:172.21.15.125:3300/0 conn(0x7fa104154bc0 0x7fa10406b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 -- 172.21.15.50:0/2657295618 >> 172.21.15.50:0/2657295618 conn(0x7fa104067bb0 msgr2=0x7fa1040c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:39.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 -- 172.21.15.50:0/2657295618 shutdown_connections 2023-12-19T14:51:39.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.167+0000 7fa10b416700 1 -- 172.21.15.50:0/2657295618 wait complete. 2023-12-19T14:51:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 Processor -- start 2023-12-19T14:51:39.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 -- start start 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 0x7fa104157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa104154bc0 0x7fa104143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 0x7fa1040c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041444b0 con 0x7fa10406aac0 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104156de0 con 0x7fa104154f90 2023-12-19T14:51:39.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10b416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041570c0 con 0x7fa104154bc0 2023-12-19T14:51:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10a414700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 0x7fa104157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10a414700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 0x7fa104157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52018/0 (socket says 172.21.15.50:52018) 2023-12-19T14:51:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa10a414700 1 -- 172.21.15.50:0/3256991342 learned_addr learned my addr 172.21.15.50:0/3256991342 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.171+0000 7fa109c13700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa104154bc0 0x7fa104143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 0x7fa1040c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 -- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa104154bc0 msgr2=0x7fa104143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa104154bc0 0x7fa104143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 -- 172.21.15.50:0/3256991342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 msgr2=0x7fa104157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 0x7fa104157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 -- 172.21.15.50:0/3256991342 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40094f0 con 0x7fa104154f90 2023-12-19T14:51:39.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.175+0000 7fa10ac15700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 0x7fa1040c9500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00bb00 tx=0x7fa0fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.179+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00c7c0 con 0x7fa104154f90 2023-12-19T14:51:39.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.179+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc00c920 con 0x7fa104154f90 2023-12-19T14:51:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.179+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00cce0 con 0x7fa104154f90 2023-12-19T14:51:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.179+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1041573a0 con 0x7fa104154f90 2023-12-19T14:51:39.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.179+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa104157960 con 0x7fa104154f90 2023-12-19T14:51:39.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.183+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fa0fc014620 con 0x7fa104154f90 2023-12-19T14:51:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.183+0000 7fa0fa7fc700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 0x7fa0e8062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:39.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.183+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fa0fc07cfd0 con 0x7fa104154f90 2023-12-19T14:51:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.183+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10404be40 con 0x7fa104154f90 2023-12-19T14:51:39.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.183+0000 7fa10a414700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 0x7fa0e8062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:39.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.191+0000 7fa10a414700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 0x7fa0e8062cd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e70 tx=0x7fa1000091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.195+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa0fc04abe0 con 0x7fa104154f90 2023-12-19T14:51:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.471+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa104157b50 con 0x7fa0e8060820 2023-12-19T14:51:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.623+0000 7fa0fa7fc700 1 -- 172.21.15.50:0/3256991342 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa104157b50 con 0x7fa0e8060820 2023-12-19T14:51:39.624 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 msgr2=0x7fa0e8062cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 0x7fa0e8062cd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e70 tx=0x7fa1000091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 msgr2=0x7fa1040c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 0x7fa1040c9500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00bb00 tx=0x7fa0fc009780 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 shutdown_connections 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa104154f90 0x7fa1040c9500 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa0e8060820 0x7fa0e8062cd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa104154bc0 0x7fa104143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 --2- 172.21.15.50:0/3256991342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa10406aac0 0x7fa104157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 >> 172.21.15.50:0/3256991342 conn(0x7fa104067bb0 msgr2=0x7fa1040aadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 shutdown_connections 2023-12-19T14:51:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:39.627+0000 7fa10b416700 1 -- 172.21.15.50:0/3256991342 wait complete. 2023-12-19T14:51:39.655 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-12-19T14:51:39.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:39.655 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-12-19T14:51:39.694 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:39.695 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:39.696 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:39.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:39.697 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:39.787 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:39.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:39.787 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:39.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:39.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:39.904 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:39.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:39.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:39.951 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:40.031 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_45_01_UTC 2023-12-19T14:51:40.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.031 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_45_01_UTC) 2023-12-19T14:51:40.118 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:40.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.118 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:40.201 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:40.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.202 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_47_01_UTC) 2023-12-19T14:51:40.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:40.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.283 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:40.359 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:40.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.360 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:40.445 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:40.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.445 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:40.530 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:40.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.530 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:40.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:40.619 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-12-19T14:51:40.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T14:51:40.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.907+0000 7f25e1416700 1 Processor -- start 2023-12-19T14:51:40.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 -- start start 2023-12-19T14:51:40.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25dc0af190 0x7f25dc06af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0cb3f0 con 0x7f25dc0aedc0 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc0cc140 con 0x7f25dc0af190 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.911+0000 7f25e1416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0af610 con 0x7f25dc0cac50 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25dbfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f25dc0aedc0 0x7f25dc06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:35786/0 (socket says 172.21.15.50:35786) 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25dbfff700 1 -- 172.21.15.50:0/3745134247 learned_addr learned my addr 172.21.15.50:0/3745134247 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25d3fff700 1 --2- 172.21.15.50:0/3745134247 >> v2:172.21.15.125:3300/0 conn(0x7f25dc0af190 0x7f25dc06af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25db7fe700 1 -- 172.21.15.50:0/3745134247 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410457290 0 0) 0x7f25dc0cb3f0 con 0x7f25dc0aedc0 2023-12-19T14:51:40.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 -- 172.21.15.50:0/3745134247 >> v2:172.21.15.125:3300/0 conn(0x7f25dc0af190 msgr2=0x7f25dc06af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:40.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 --2- 172.21.15.50:0/3745134247 >> v2:172.21.15.125:3300/0 conn(0x7f25dc0af190 0x7f25dc06af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:40.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 -- 172.21.15.50:0/3745134247 >> v1:172.21.15.125:6789/0 conn(0x7f25dc0aedc0 legacy=0x7f25dc06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:40.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 -- 172.21.15.50:0/3745134247 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc06d220 con 0x7f25dc0cac50 2023-12-19T14:51:40.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e0c15700 1 --2- 172.21.15.50:0/3745134247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0af930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009e60 tx=0x7f25d4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d289163081fea3f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:40.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25db7fe700 1 -- 172.21.15.50:0/3745134247 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d400e070 con 0x7f25dc0cac50 2023-12-19T14:51:40.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25db7fe700 1 -- 172.21.15.50:0/3745134247 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d400bb60 con 0x7f25dc0cac50 2023-12-19T14:51:40.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25db7fe700 1 -- 172.21.15.50:0/3745134247 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4012430 con 0x7f25dc0cac50 2023-12-19T14:51:40.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 -- 172.21.15.50:0/3745134247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 msgr2=0x7f25dc0af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:40.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 --2- 172.21.15.50:0/3745134247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0af930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009e60 tx=0x7f25d4009780 comp rx=0 tx=0).stop 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 -- 172.21.15.50:0/3745134247 shutdown_connections 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 --2- 172.21.15.50:0/3745134247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0af930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 --2- 172.21.15.50:0/3745134247 >> v2:172.21.15.125:3300/0 conn(0x7f25dc0af190 0x7f25dc06af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 -- 172.21.15.50:0/3745134247 >> 172.21.15.50:0/3745134247 conn(0x7f25dc0cc760 msgr2=0x7f25dc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.915+0000 7f25e1416700 1 -- 172.21.15.50:0/3745134247 shutdown_connections 2023-12-19T14:51:40.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 -- 172.21.15.50:0/3745134247 wait complete. 2023-12-19T14:51:40.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 Processor -- start 2023-12-19T14:51:40.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 -- start start 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 0x7f25dc0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0c7390 con 0x7f25dc0af190 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc15d600 con 0x7f25dc0aedc0 2023-12-19T14:51:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.919+0000 7f25e1416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc15d8e0 con 0x7f25dc0cac50 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 0x7f25dc0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:53104/0 (socket says 172.21.15.50:53104) 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 -- 172.21.15.50:0/694164597 learned_addr learned my addr 172.21.15.50:0/694164597 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25d3fff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 0x7f25dc0c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52022/0 (socket says 172.21.15.50:52022) 2023-12-19T14:51:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25e0c15700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 -- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 msgr2=0x7f25dc0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 -- 172.21.15.50:0/694164597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 msgr2=0x7f25dc0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 0x7f25dc0c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 -- 172.21.15.50:0/694164597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4009a20 con 0x7f25dc0aedc0 2023-12-19T14:51:40.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25dbfff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c5b0 tx=0x7f25cc001070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc005b30 con 0x7f25dc0aedc0 2023-12-19T14:51:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc005c90 con 0x7f25dc0aedc0 2023-12-19T14:51:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc007300 con 0x7f25dc0aedc0 2023-12-19T14:51:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc15dbc0 con 0x7f25dc0aedc0 2023-12-19T14:51:40.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.923+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc0b2fe0 con 0x7f25dc0aedc0 2023-12-19T14:51:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.927+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc04be40 con 0x7f25dc0aedc0 2023-12-19T14:51:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.927+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f25cc01b3b0 con 0x7f25dc0aedc0 2023-12-19T14:51:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.931+0000 7f25d8ff9700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 0x7f25bc062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.931+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6832+0+0 (secure 0 0 0) 0x7f25cc07c450 con 0x7f25dc0aedc0 2023-12-19T14:51:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.931+0000 7f25d3fff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 0x7f25bc062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:40.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.931+0000 7f25d3fff700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 0x7f25bc062c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0c9d10 tx=0x7f25c40091a0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:40.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:40.935+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f25cc04c2e0 con 0x7f25dc0aedc0 2023-12-19T14:51:41.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.135+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f25dc15d740 con 0x7f25bc060750 2023-12-19T14:51:41.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.287+0000 7f25d8ff9700 1 -- 172.21.15.50:0/694164597 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f25dc15d740 con 0x7f25bc060750 2023-12-19T14:51:41.289 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 msgr2=0x7f25bc062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 0x7f25bc062c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0c9d10 tx=0x7f25c40091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 msgr2=0x7f25dc0c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00c5b0 tx=0x7f25cc001070 comp rx=0 tx=0).stop 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 shutdown_connections 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25bc060750 0x7f25bc062c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25dc0cac50 0x7f25dc0b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25dc0af190 0x7f25dc0c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 --2- 172.21.15.50:0/694164597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25dc0aedc0 0x7f25dc0c6d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:41.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 >> 172.21.15.50:0/694164597 conn(0x7f25dc0cc760 msgr2=0x7f25dc0b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:41.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 shutdown_connections 2023-12-19T14:51:41.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:41.291+0000 7f25e1416700 1 -- 172.21.15.50:0/694164597 wait complete. 2023-12-19T14:51:41.315 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-12-19T14:51:41.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.315 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-12-19T14:51:41.351 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:41.351 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:41.351 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:41.352 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:41.353 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:41.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.353 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:41.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:41.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.439 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:41.521 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:41.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.521 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:41.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:41.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.602 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:41.685 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:41.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.686 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:41.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:41.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.768 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:41.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:41.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.852 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:41.964 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:41.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:41.965 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:42.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:42.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.024 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:42.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:42.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.062 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:42.142 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_51_01_UTC 2023-12-19T14:51:42.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.143 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-14_51_01_UTC) 2023-12-19T14:51:42.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.226 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-12-19T14:51:42.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T14:51:42.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.515+0000 7f3e0cf85700 1 Processor -- start 2023-12-19T14:51:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.515+0000 7f3e0cf85700 1 -- start start 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0cf85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e080c4dc0 0x7f3e080c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0cf85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e0806c7c0 0x7f3e0806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0cf85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080c6270 con 0x7f3e08142910 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0cf85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0806a6a0 con 0x7f3e0806c7c0 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0cf85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080aea30 con 0x7f3e080c4dc0 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e06d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e0806c7c0 0x7f3e0806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e06d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3e0806c7c0 0x7f3e0806cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52032/0 (socket says 172.21.15.50:52032) 2023-12-19T14:51:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4067675042 learned_addr learned my addr 172.21.15.50:0/4067675042 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e05d9b700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e080c4dc0 0x7f3e080c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.519+0000 7f3e0659c700 1 --1- 172.21.15.50:0/4067675042 >> v1:172.21.15.125:6789/0 conn(0x7f3e08142910 0x7f3e0806d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:35792/0 (socket says 172.21.15.50:35792) 2023-12-19T14:51:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4067675042 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e080c4dc0 msgr2=0x7f3e080c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e06d9d700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e080c4dc0 0x7f3e080c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4067675042 >> v1:172.21.15.125:6789/0 conn(0x7f3e08142910 legacy=0x7f3e0806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4067675042 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080c5750 con 0x7f3e0806c7c0 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0559a700 1 -- 172.21.15.50:0/4067675042 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439031913 0 0) 0x7f3e080c6270 con 0x7f3e08142910 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e06d9d700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0806c7c0 0x7f3e0806cc40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009f20 tx=0x7f3e00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63e3dfb27b8b764 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0559a700 1 -- 172.21.15.50:0/4067675042 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000e070 con 0x7f3e0806c7c0 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0559a700 1 -- 172.21.15.50:0/4067675042 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0000b9d0 con 0x7f3e0806c7c0 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0559a700 1 -- 172.21.15.50:0/4067675042 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e000092e0 con 0x7f3e0806c7c0 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4067675042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0806c7c0 msgr2=0x7f3e0806cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0806c7c0 0x7f3e0806cc40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009f20 tx=0x7f3e00009840 comp rx=0 tx=0).stop 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4067675042 shutdown_connections 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e0806c7c0 0x7f3e0806cc40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4067675042 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e080c4dc0 0x7f3e080c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4067675042 >> 172.21.15.50:0/4067675042 conn(0x7f3e08067bb0 msgr2=0x7f3e080cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4067675042 shutdown_connections 2023-12-19T14:51:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.523+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4067675042 wait complete. 2023-12-19T14:51:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 Processor -- start 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 -- start start 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e0806c7c0 0x7f3e080b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e080c4dc0 0x7f3e08138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08158fe0 con 0x7f3e08142910 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08137ba0 con 0x7f3e080c4dc0 2023-12-19T14:51:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0cf85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08137890 con 0x7f3e0806c7c0 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52042/0 (socket says 172.21.15.50:52042) 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4094867832 learned_addr learned my addr 172.21.15.50:0/4094867832 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e0659c700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e0806c7c0 0x7f3e080b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e05d9b700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e080c4dc0 0x7f3e08138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e0806c7c0 msgr2=0x7f3e080b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e0806c7c0 0x7f3e080b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e080c4dc0 msgr2=0x7f3e08138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e080c4dc0 0x7f3e08138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.527+0000 7f3e06d9d700 1 -- 172.21.15.50:0/4094867832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00009ae0 con 0x7f3e08142910 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3e06d9d700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000b610 tx=0x7f3e00018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00008ad0 con 0x7f3e08142910 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e00008c70 con 0x7f3e08142910 2023-12-19T14:51:42.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00026460 con 0x7f3e08142910 2023-12-19T14:51:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e080b12b0 con 0x7f3e08142910 2023-12-19T14:51:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080b1850 con 0x7f3e08142910 2023-12-19T14:51:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.531+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e08137d50 con 0x7f3e08142910 2023-12-19T14:51:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.535+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f3e00003040 con 0x7f3e08142910 2023-12-19T14:51:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.535+0000 7f3df67fc700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 0x7f3de8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:42.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.535+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6992+0+0 (secure 0 0 0) 0x7f3e0007ea20 con 0x7f3e08142910 2023-12-19T14:51:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.535+0000 7f3e0659c700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 0x7f3de8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.535+0000 7f3e0659c700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 0x7f3de8062c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df8007180 tx=0x7f3df8007ef0 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.543+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3e08137d50 con 0x7f3e08142910 2023-12-19T14:51:42.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.735+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e08137d50 con 0x7f3de8060750 2023-12-19T14:51:42.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.919+0000 7f3df67fc700 1 -- 172.21.15.50:0/4094867832 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3e08137d50 con 0x7f3de8060750 2023-12-19T14:51:42.920 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-12-19T14:51:42.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.919+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 msgr2=0x7f3de8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.919+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 0x7f3de8062c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df8007180 tx=0x7f3df8007ef0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.919+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 msgr2=0x7f3e08138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.919+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000b610 tx=0x7f3e00018000 comp rx=0 tx=0).stop 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 shutdown_connections 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e080c4dc0 0x7f3e08138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3de8060750 0x7f3de8062c00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3e08142910 0x7f3e08138570 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 --2- 172.21.15.50:0/4094867832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3e0806c7c0 0x7f3e080b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 >> 172.21.15.50:0/4094867832 conn(0x7f3e08067bb0 msgr2=0x7f3e0806cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 shutdown_connections 2023-12-19T14:51:42.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:42.923+0000 7f3e0cf85700 1 -- 172.21.15.50:0/4094867832 wait complete. 2023-12-19T14:51:42.945 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-12-19T14:51:42.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.945 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-12-19T14:51:42.990 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:42.990 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:42.990 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:42.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:42.992 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:42.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:42.993 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:43.072 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:43.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.072 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:43.157 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:43.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.158 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:43.248 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:43.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.248 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:43.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:43.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.330 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:43.410 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:43.410 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.410 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:43.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_47_01_UTC 2023-12-19T14:51:43.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.490 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_47_01_UTC) 2023-12-19T14:51:43.571 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_48_00_UTC 2023-12-19T14:51:43.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.572 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_48_00_UTC) 2023-12-19T14:51:43.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_49_01_UTC 2023-12-19T14:51:43.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.655 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_49_01_UTC) 2023-12-19T14:51:43.736 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:43.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.736 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:43.820 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:43.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.820 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:43.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:43.914 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-12-19T14:51:43.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T14:51:44.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.203+0000 7f39f8985700 1 Processor -- start 2023-12-19T14:51:44.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.203+0000 7f39f8985700 1 -- start start 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f8985700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f39f40cb130 0x7f39f40cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f8985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 0x7f39f406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f8985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f40b0ac0 con 0x7f39f415c220 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f8985700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f406b030 con 0x7f39f40cb130 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f8985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f406b210 con 0x7f39f406a6a0 2023-12-19T14:51:44.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f39f40cb130 0x7f39f40cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f39f40cb130 0x7f39f40cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52046/0 (socket says 172.21.15.50:52046) 2023-12-19T14:51:44.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 -- 172.21.15.50:0/1515032704 learned_addr learned my addr 172.21.15.50:0/1515032704 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:44.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f1d9b700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 0x7f39f406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 -- 172.21.15.50:0/1515032704 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 msgr2=0x7f39f406aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:44.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 0x7f39f406aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 -- 172.21.15.50:0/1515032704 >> v1:172.21.15.125:6789/0 conn(0x7f39f415c220 legacy=0x7f39f406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:44.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 -- 172.21.15.50:0/1515032704 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f40ca810 con 0x7f39f40cb130 2023-12-19T14:51:44.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f1d9b700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 0x7f39f406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:44.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f259c700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f40cb130 0x7f39f40cb500 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f39e40099b0 tx=0x7f39e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2927c7d140c062ed server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:44.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f159a700 1 -- 172.21.15.50:0/1515032704 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400b860 con 0x7f39f40cb130 2023-12-19T14:51:44.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f159a700 1 -- 172.21.15.50:0/1515032704 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e400b9c0 con 0x7f39f40cb130 2023-12-19T14:51:44.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.207+0000 7f39f159a700 1 -- 172.21.15.50:0/1515032704 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e400bd60 con 0x7f39f40cb130 2023-12-19T14:51:44.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- 172.21.15.50:0/1515032704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f40cb130 msgr2=0x7f39f40cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:44.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f40cb130 0x7f39f40cb500 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f39e40099b0 tx=0x7f39e4009250 comp rx=0 tx=0).stop 2023-12-19T14:51:44.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- 172.21.15.50:0/1515032704 shutdown_connections 2023-12-19T14:51:44.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f406a6a0 0x7f39f406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 --2- 172.21.15.50:0/1515032704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f40cb130 0x7f39f40cb500 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- 172.21.15.50:0/1515032704 >> 172.21.15.50:0/1515032704 conn(0x7f39f4067bb0 msgr2=0x7f39f40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:44.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- 172.21.15.50:0/1515032704 shutdown_connections 2023-12-19T14:51:44.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- 172.21.15.50:0/1515032704 wait complete. 2023-12-19T14:51:44.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 Processor -- start 2023-12-19T14:51:44.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.211+0000 7f39f8985700 1 -- start start 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f40cb130 0x7f39f4147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39f415c220 0x7f39f40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40c71d0 con 0x7f39f406a6a0 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40c65c0 con 0x7f39f415c220 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40c68a0 con 0x7f39f40cb130 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52050/0 (socket says 172.21.15.50:52050) 2023-12-19T14:51:44.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 -- 172.21.15.50:0/279761002 learned_addr learned my addr 172.21.15.50:0/279761002 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:44.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f1d9b700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f40cb130 0x7f39f4147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f2d9d700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39f415c220 0x7f39f40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 -- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f40cb130 msgr2=0x7f39f4147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f40cb130 0x7f39f4147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 -- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39f415c220 msgr2=0x7f39f40b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39f415c220 0x7f39f40b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 -- 172.21.15.50:0/279761002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e40094f0 con 0x7f39f406a6a0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f259c700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009290 tx=0x7f39e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4026480 con 0x7f39f406a6a0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e401f070 con 0x7f39f406a6a0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4026830 con 0x7f39f406a6a0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f40c6b80 con 0x7f39f406a6a0 2023-12-19T14:51:44.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f416d8d0 con 0x7f39f406a6a0 2023-12-19T14:51:44.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.215+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f40c7000 con 0x7f39f406a6a0 2023-12-19T14:51:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.219+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f39e4026ba0 con 0x7f39f406a6a0 2023-12-19T14:51:44.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.219+0000 7f39e27fc700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 0x7f39d4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:44.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.219+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f39e4021070 con 0x7f39f406a6a0 2023-12-19T14:51:44.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.219+0000 7f39f1d9b700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 0x7f39d4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:44.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.223+0000 7f39f1d9b700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 0x7f39d4062c00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39e8006f00 tx=0x7f39e8008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:44.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.227+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39e404ed90 con 0x7f39f406a6a0 2023-12-19T14:51:44.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.423+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f39f4071120 con 0x7f39d4060750 2023-12-19T14:51:44.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.583+0000 7f39e27fc700 1 -- 172.21.15.50:0/279761002 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f39f4071120 con 0x7f39d4060750 2023-12-19T14:51:44.591 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-12-19T14:51:44.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 msgr2=0x7f39d4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:44.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 0x7f39d4062c00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39e8006f00 tx=0x7f39e8008040 comp rx=0 tx=0).stop 2023-12-19T14:51:44.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 msgr2=0x7f39f416d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:44.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009290 tx=0x7f39e4017000 comp rx=0 tx=0).stop 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 shutdown_connections 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39f415c220 0x7f39f40b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39d4060750 0x7f39d4062c00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39f40cb130 0x7f39f4147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 --2- 172.21.15.50:0/279761002 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39f406a6a0 0x7f39f416d390 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:44.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.591+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 >> 172.21.15.50:0/279761002 conn(0x7f39f4067bb0 msgr2=0x7f39f40b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:44.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.595+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 shutdown_connections 2023-12-19T14:51:44.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:44.595+0000 7f39f8985700 1 -- 172.21.15.50:0/279761002 wait complete. 2023-12-19T14:51:44.622 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-12-19T14:51:44.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:44.622 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_01_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:44.653 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:44.654 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:44.655 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:44.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:44.655 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:44.738 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:44.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:44.739 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:44.823 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_43_01_UTC 2023-12-19T14:51:44.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:44.824 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_43_01_UTC) 2023-12-19T14:51:44.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:44.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:44.950 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:45.000 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:45.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.000 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:45.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_46_00_UTC 2023-12-19T14:51:45.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.031 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_46_00_UTC) 2023-12-19T14:51:45.110 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:45.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.110 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:45.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:45.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.192 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_48_01_UTC) 2023-12-19T14:51:45.277 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:45.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.277 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:45.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:45.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.364 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_50_01_UTC) 2023-12-19T14:51:45.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:45.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.448 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:45.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:45.530 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-12-19T14:51:45.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T14:51:45.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.819+0000 7f5ed6c4c700 1 Processor -- start 2023-12-19T14:51:45.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.819+0000 7f5ed6c4c700 1 -- start start 2023-12-19T14:51:45.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed6c4c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ed00aedc0 0x7f5ed00af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:45.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed6c4c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed006b3d0 con 0x7f5ed00cac50 2023-12-19T14:51:45.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed6c4c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00cb3f0 con 0x7f5ed006af50 2023-12-19T14:51:45.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed6c4c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed00cc140 con 0x7f5ed00aedc0 2023-12-19T14:51:45.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5c4a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ed00aedc0 0x7f5ed00af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed644b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5ed00cac50 0x7f5ed00af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43694/0 (socket says 172.21.15.50:43694) 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed644b700 1 -- 172.21.15.50:0/575573999 learned_addr learned my addr 172.21.15.50:0/575573999 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5449700 1 --1- 172.21.15.50:0/575573999 >> v1:172.21.15.125:6789/0 conn(0x7f5ed006af50 0x7f5ed006d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:35808/0 (socket says 172.21.15.50:35808) 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/575573999 >> v1:172.21.15.125:6789/0 conn(0x7f5ed006af50 legacy=0x7f5ed006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/575573999 >> v1:172.21.15.50:6789/0 conn(0x7f5ed00cac50 legacy=0x7f5ed00af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed4c48700 1 -- 172.21.15.50:0/575573999 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135678980 0 0) 0x7f5ec8001490 con 0x7f5ed00cac50 2023-12-19T14:51:45.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/575573999 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed006d220 con 0x7f5ed00aedc0 2023-12-19T14:51:45.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed5c4a700 1 --2- 172.21.15.50:0/575573999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed00aedc0 0x7f5ed00af190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0098e0 tx=0x7f5ecc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8eb9da405f563d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:45.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed4c48700 1 -- 172.21.15.50:0/575573999 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc009e80 con 0x7f5ed00aedc0 2023-12-19T14:51:45.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed4c48700 1 -- 172.21.15.50:0/575573999 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc00b860 con 0x7f5ed00aedc0 2023-12-19T14:51:45.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.823+0000 7f5ed4c48700 1 -- 172.21.15.50:0/575573999 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00bbe0 con 0x7f5ed00aedc0 2023-12-19T14:51:45.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/575573999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed00aedc0 msgr2=0x7f5ed00af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:45.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/575573999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed00aedc0 0x7f5ed00af190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0098e0 tx=0x7f5ecc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:45.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/575573999 shutdown_connections 2023-12-19T14:51:45.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/575573999 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed00aedc0 0x7f5ed00af190 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:45.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/575573999 >> 172.21.15.50:0/575573999 conn(0x7f5ed00cc760 msgr2=0x7f5ed00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:45.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/575573999 shutdown_connections 2023-12-19T14:51:45.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/575573999 wait complete. 2023-12-19T14:51:45.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 Processor -- start 2023-12-19T14:51:45.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.827+0000 7f5ed6c4c700 1 -- start start 2023-12-19T14:51:45.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:45.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ed00aedc0 0x7f5ed015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:45.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ed00cac50 0x7f5ed00c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed01698f0 con 0x7f5ed006af50 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b47d0 con 0x7f5ed00aedc0 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00c4ec0 con 0x7f5ed00cac50 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52078/0 (socket says 172.21.15.50:52078) 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/870644656 learned_addr learned my addr 172.21.15.50:0/870644656 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5449700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ed00aedc0 0x7f5ed015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ed00cac50 msgr2=0x7f5ed00c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ed00cac50 0x7f5ed00c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ed00aedc0 msgr2=0x7f5ed015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ed00aedc0 0x7f5ed015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 -- 172.21.15.50:0/870644656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc009420 con 0x7f5ed006af50 2023-12-19T14:51:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed5c4a700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0068ed0 tx=0x7f5ecc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc01f070 con 0x7f5ed006af50 2023-12-19T14:51:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc00b9d0 con 0x7f5ed006af50 2023-12-19T14:51:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc025650 con 0x7f5ed006af50 2023-12-19T14:51:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed00b3ea0 con 0x7f5ed006af50 2023-12-19T14:51:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.831+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed00b44e0 con 0x7f5ed006af50 2023-12-19T14:51:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.835+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed004be40 con 0x7f5ed006af50 2023-12-19T14:51:45.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.835+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f5ecc023070 con 0x7f5ed006af50 2023-12-19T14:51:45.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.835+0000 7f5ec5ffb700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 0x7f5eb4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:45.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.835+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6912+0+0 (secure 0 0 0) 0x7f5ecc07e650 con 0x7f5ed006af50 2023-12-19T14:51:45.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.839+0000 7f5ed5449700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 0x7f5eb4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:45.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.839+0000 7f5ed5449700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 0x7f5eb4062c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0007fd0 tx=0x7f5ec0007d70 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:45.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:45.839+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ecc04e510 con 0x7f5ed006af50 2023-12-19T14:51:46.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.051+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ed00b4910 con 0x7f5eb4060750 2023-12-19T14:51:46.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.215+0000 7f5ec5ffb700 1 -- 172.21.15.50:0/870644656 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5ec0013420 con 0x7f5eb4060750 2023-12-19T14:51:46.216 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-12-19T14:51:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 msgr2=0x7f5eb4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 0x7f5eb4062c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0007fd0 tx=0x7f5ec0007d70 comp rx=0 tx=0).stop 2023-12-19T14:51:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 msgr2=0x7f5ed01693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:46.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0068ed0 tx=0x7f5ecc017000 comp rx=0 tx=0).stop 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 shutdown_connections 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ed00aedc0 0x7f5ed015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5eb4060750 0x7f5eb4062c00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ed00cac50 0x7f5ed00c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 --2- 172.21.15.50:0/870644656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ed006af50 0x7f5ed01693b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 >> 172.21.15.50:0/870644656 conn(0x7f5ed00cc760 msgr2=0x7f5ed00ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 shutdown_connections 2023-12-19T14:51:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:46.219+0000 7f5ed6c4c700 1 -- 172.21.15.50:0/870644656 wait complete. 2023-12-19T14:51:46.243 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-12-19T14:51:46.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.243 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-12-19T14:51:46.281 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_01_UTC 2023-12-19T14:51:46.281 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:46.281 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:46.281 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:46.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:46.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_41_01_UTC 2023-12-19T14:51:46.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.283 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_41_01_UTC) 2023-12-19T14:51:46.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:46.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.372 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:46.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:46.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.454 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:46.537 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_44_01_UTC 2023-12-19T14:51:46.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.538 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_44_01_UTC) 2023-12-19T14:51:46.617 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:46.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.618 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:46.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:46.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.700 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_46_01_UTC) 2023-12-19T14:51:46.778 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:46.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.778 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:46.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:46.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.897 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_48_01_UTC) 2023-12-19T14:51:46.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:46.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.966 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:46.989 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_50_00_UTC 2023-12-19T14:51:46.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:46.990 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_50_00_UTC) 2023-12-19T14:51:47.069 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:47.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:47.069 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:47.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:47.151 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-12-19T14:51:47.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T14:51:47.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.459+0000 7f751759e700 1 Processor -- start 2023-12-19T14:51:47.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.459+0000 7f751759e700 1 -- start start 2023-12-19T14:51:47.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f7518156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 0x7f75180ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751806beb0 con 0x7f75181569e0 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751806cb80 con 0x7f75180adf40 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751806ce60 con 0x7f75180aef60 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751659c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75181569e0 0x7f75180b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58760/0 (socket says 172.21.15.50:58760) 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751659c700 1 -- 172.21.15.50:0/475316199 learned_addr learned my addr 172.21.15.50:0/475316199 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:47.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 --2- 172.21.15.50:0/475316199 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f7518156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7516d9d700 1 --2- 172.21.15.50:0/475316199 >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 0x7f75180ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751559a700 1 -- 172.21.15.50:0/475316199 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340528564 0 0) 0x7f751806beb0 con 0x7f75181569e0 2023-12-19T14:51:47.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751559a700 1 -- 172.21.15.50:0/475316199 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500003570 con 0x7f75181569e0 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 -- 172.21.15.50:0/475316199 >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 msgr2=0x7f75180ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 --2- 172.21.15.50:0/475316199 >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 0x7f75180ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 -- 172.21.15.50:0/475316199 >> v1:172.21.15.50:6789/0 conn(0x7f75181569e0 legacy=0x7f75180b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 -- 172.21.15.50:0/475316199 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751806d140 con 0x7f75180aef60 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7516d9d700 1 --2- 172.21.15.50:0/475316199 >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 0x7f75180ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:47.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f7515d9b700 1 --2- 172.21.15.50:0/475316199 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f7518156db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f750c0098e0 tx=0x7f750c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cae7f7675abd102 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751559a700 1 -- 172.21.15.50:0/475316199 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c011630 con 0x7f75180aef60 2023-12-19T14:51:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751559a700 1 -- 172.21.15.50:0/475316199 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c011790 con 0x7f75180aef60 2023-12-19T14:51:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751559a700 1 -- 172.21.15.50:0/475316199 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c011b30 con 0x7f75180aef60 2023-12-19T14:51:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.463+0000 7f751759e700 1 -- 172.21.15.50:0/475316199 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 msgr2=0x7f7518156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 --2- 172.21.15.50:0/475316199 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f7518156db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f750c0098e0 tx=0x7f750c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 -- 172.21.15.50:0/475316199 shutdown_connections 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 --2- 172.21.15.50:0/475316199 >> v2:172.21.15.50:3300/0 conn(0x7f75180adf40 0x7f75180ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 --2- 172.21.15.50:0/475316199 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f7518156db0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 -- 172.21.15.50:0/475316199 >> 172.21.15.50:0/475316199 conn(0x7f75180c8c00 msgr2=0x7f75180c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 -- 172.21.15.50:0/475316199 shutdown_connections 2023-12-19T14:51:47.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 -- 172.21.15.50:0/475316199 wait complete. 2023-12-19T14:51:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 Processor -- start 2023-12-19T14:51:47.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.467+0000 7f751759e700 1 -- start start 2023-12-19T14:51:47.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f75180c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 0x7f7518152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75180c5220 con 0x7f75180adf40 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75180c4180 con 0x7f75181569e0 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180c4460 con 0x7f75180aef60 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56358/0 (socket says 172.21.15.50:56358) 2023-12-19T14:51:47.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 -- 172.21.15.50:0/1515512792 learned_addr learned my addr 172.21.15.50:0/1515512792 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:47.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f7515d9b700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f75180c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f7516d9d700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 0x7f7518152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 -- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 msgr2=0x7f75180c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f75180c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 -- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 msgr2=0x7f7518152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 0x7f7518152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 -- 172.21.15.50:0/1515512792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510009a20 con 0x7f75180adf40 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f7515d9b700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f75180c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f7516d9d700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 0x7f7518152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751659c700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f750800c7c0 tx=0x7f7508007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508005b30 con 0x7f75180adf40 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508005c90 con 0x7f75180adf40 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c009420 con 0x7f75180adf40 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508007340 con 0x7f75180adf40 2023-12-19T14:51:47.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.471+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75180c47f0 con 0x7f75180adf40 2023-12-19T14:51:47.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.475+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751804be40 con 0x7f75180adf40 2023-12-19T14:51:47.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.475+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f750801b400 con 0x7f75180adf40 2023-12-19T14:51:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.475+0000 7f75067fc700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 0x7f74f8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:47.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.479+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6928+0+0 (secure 0 0 0) 0x7f750807cb60 con 0x7f75180adf40 2023-12-19T14:51:47.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.479+0000 7f7515d9b700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 0x7f74f8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:47.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.479+0000 7f7515d9b700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 0x7f74f8062c00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f750c0091e0 tx=0x7f750c017000 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.483+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f750804ca10 con 0x7f75180adf40 2023-12-19T14:51:47.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.679+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f75180c4620 con 0x7f74f8060750 2023-12-19T14:51:47.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.835+0000 7f75067fc700 1 -- 172.21.15.50:0/1515512792 <== mgr.5236 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f750c011900 con 0x7f74f8060750 2023-12-19T14:51:47.838 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-12-19T14:51:47.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 msgr2=0x7f74f8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 0x7f74f8062c00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f750c0091e0 tx=0x7f750c017000 comp rx=0 tx=0).stop 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 msgr2=0x7f75180bf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f750800c7c0 tx=0x7f7508007e00 comp rx=0 tx=0).stop 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 shutdown_connections 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75181569e0 0x7f7518152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f74f8060750 0x7f74f8062c00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75180aef60 0x7f75180c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 --2- 172.21.15.50:0/1515512792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75180adf40 0x7f75180bf2b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:47.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 >> 172.21.15.50:0/1515512792 conn(0x7f75180c8c00 msgr2=0x7f75180670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 shutdown_connections 2023-12-19T14:51:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:47.839+0000 7f751759e700 1 -- 172.21.15.50:0/1515512792 wait complete. 2023-12-19T14:51:47.870 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-12-19T14:51:47.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:47.870 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:47.935 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:47.936 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:47.936 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:47.936 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:47.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_41_00_UTC 2023-12-19T14:51:47.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:47.937 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_41_00_UTC) 2023-12-19T14:51:47.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_42_00_UTC 2023-12-19T14:51:47.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:47.966 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_42_00_UTC) 2023-12-19T14:51:48.041 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_43_00_UTC 2023-12-19T14:51:48.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.042 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_43_00_UTC) 2023-12-19T14:51:48.124 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_44_00_UTC 2023-12-19T14:51:48.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.124 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_44_00_UTC) 2023-12-19T14:51:48.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_45_00_UTC 2023-12-19T14:51:48.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.239 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_45_00_UTC) 2023-12-19T14:51:48.270 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_46_01_UTC 2023-12-19T14:51:48.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.271 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_46_01_UTC) 2023-12-19T14:51:48.355 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_47_00_UTC 2023-12-19T14:51:48.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.355 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_47_00_UTC) 2023-12-19T14:51:48.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_48_01_UTC 2023-12-19T14:51:48.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.447 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_48_01_UTC) 2023-12-19T14:51:48.536 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_49_00_UTC 2023-12-19T14:51:48.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.536 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_49_00_UTC) 2023-12-19T14:51:48.622 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_50_01_UTC 2023-12-19T14:51:48.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.622 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_50_01_UTC) 2023-12-19T14:51:48.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_51_00_UTC 2023-12-19T14:51:48.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.708 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-14_51_00_UTC) 2023-12-19T14:51:48.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:51:48.794 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-12-19T14:51:48.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:51:49.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.087+0000 7f67e3fff700 1 Processor -- start 2023-12-19T14:51:49.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.087+0000 7f67e3fff700 1 -- start start 2023-12-19T14:51:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f67e40caa10 0x7f67e415d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e3fff700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e40cbbb0 con 0x7f67e40cc040 2023-12-19T14:51:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e3fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e415d960 con 0x7f67e40caa10 2023-12-19T14:51:49.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e3fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e406dc90 con 0x7f67e40ce2d0 2023-12-19T14:51:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f67e40caa10 0x7f67e415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f67e40caa10 0x7f67e415d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56374/0 (socket says 172.21.15.50:56374) 2023-12-19T14:51:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 -- 172.21.15.50:0/2789582214 learned_addr learned my addr 172.21.15.50:0/2789582214 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e37fe700 1 --1- 172.21.15.50:0/2789582214 >> v1:172.21.15.125:6789/0 conn(0x7f67e40cc040 0x7f67e4158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50750/0 (socket says 172.21.15.50:50750) 2023-12-19T14:51:49.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e2ffd700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 -- 172.21.15.50:0/2789582214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 msgr2=0x7f67e40ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e40ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e1ffb700 1 -- 172.21.15.50:0/2789582214 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985730999 0 0) 0x7f67e40cbbb0 con 0x7f67e40cc040 2023-12-19T14:51:49.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 -- 172.21.15.50:0/2789582214 >> v1:172.21.15.125:6789/0 conn(0x7f67e40cc040 legacy=0x7f67e4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 -- 172.21.15.50:0/2789582214 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e40cafa0 con 0x7f67e40caa10 2023-12-19T14:51:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e27fc700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40caa10 0x7f67e415d420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f67d80099b0 tx=0x7f67d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfc1ac12c932e08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e1ffb700 1 -- 172.21.15.50:0/2789582214 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8009ea0 con 0x7f67e40caa10 2023-12-19T14:51:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e1ffb700 1 -- 172.21.15.50:0/2789582214 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d800b860 con 0x7f67e40caa10 2023-12-19T14:51:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.091+0000 7f67e1ffb700 1 -- 172.21.15.50:0/2789582214 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800bc00 con 0x7f67e40caa10 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 -- 172.21.15.50:0/2789582214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40caa10 msgr2=0x7f67e415d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40caa10 0x7f67e415d420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f67d80099b0 tx=0x7f67d8009250 comp rx=0 tx=0).stop 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 -- 172.21.15.50:0/2789582214 shutdown_connections 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40caa10 0x7f67e415d420 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2789582214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e40ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 -- 172.21.15.50:0/2789582214 >> 172.21.15.50:0/2789582214 conn(0x7f67e4067bb0 msgr2=0x7f67e40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 -- 172.21.15.50:0/2789582214 shutdown_connections 2023-12-19T14:51:49.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.095+0000 7f67e3fff700 1 -- 172.21.15.50:0/2789582214 wait complete. 2023-12-19T14:51:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 Processor -- start 2023-12-19T14:51:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 -- start start 2023-12-19T14:51:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67e40caa10 0x7f67e414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e40c6e10 con 0x7f67e40cc040 2023-12-19T14:51:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e414f7b0 con 0x7f67e40caa10 2023-12-19T14:51:49.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e3fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e41539d0 con 0x7f67e40ce2d0 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56378/0 (socket says 172.21.15.50:56378) 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 -- 172.21.15.50:0/2812702952 learned_addr learned my addr 172.21.15.50:0/2812702952 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e2ffd700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67e40caa10 0x7f67e414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e37fe700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 -- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 msgr2=0x7f67e415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e415c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 -- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67e40caa10 msgr2=0x7f67e414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67e40caa10 0x7f67e414f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.099+0000 7f67e27fc700 1 -- 172.21.15.50:0/2812702952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80094f0 con 0x7f67e40cc040 2023-12-19T14:51:49.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67e27fc700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f67e40634d0 tx=0x7f67d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800bbe0 con 0x7f67e40cc040 2023-12-19T14:51:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8011420 con 0x7f67e40cc040 2023-12-19T14:51:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e41592f0 con 0x7f67e40cc040 2023-12-19T14:51:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d80117e0 con 0x7f67e40cc040 2023-12-19T14:51:49.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e4153c20 con 0x7f67e40cc040 2023-12-19T14:51:49.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.103+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e404be40 con 0x7f67e40cc040 2023-12-19T14:51:49.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.107+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f67d8011c10 con 0x7f67e40cc040 2023-12-19T14:51:49.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.107+0000 7f67caffd700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 0x7f67d0062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:49.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.107+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6944+0+0 (secure 0 0 0) 0x7f67d807e930 con 0x7f67e40cc040 2023-12-19T14:51:49.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.107+0000 7f67e2ffd700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 0x7f67d0062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:49.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.111+0000 7f67e2ffd700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 0x7f67d0062c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009ca0 tx=0x7f67d4008040 comp rx=0 tx=0).ready entity=mgr.5236 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:49.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.115+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f67d804e750 con 0x7f67e40cc040 2023-12-19T14:51:49.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.327+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f67e40cb280 con 0x7f67e40cc040 2023-12-19T14:51:49.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.879+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f67d804eb20 con 0x7f67e40cc040 2023-12-19T14:51:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.879+0000 7f67caffd700 1 -- 172.21.15.50:0/2812702952 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v13) v1 ==== 93+0+0 (secure 0 0 0) 0x7f67d80521e0 con 0x7f67e40cc040 2023-12-19T14:51:49.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.883+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 msgr2=0x7f67d0062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.883+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 0x7f67d0062c30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009ca0 tx=0x7f67d4008040 comp rx=0 tx=0).stop 2023-12-19T14:51:49.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.883+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 msgr2=0x7f67e41588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:49.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.883+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f67e40634d0 tx=0x7f67d8017000 comp rx=0 tx=0).stop 2023-12-19T14:51:49.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 shutdown_connections 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67d0060780 0x7f67d0062c30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f67e40ce2d0 0x7f67e415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67e40cc040 0x7f67e41588e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 --2- 172.21.15.50:0/2812702952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67e40caa10 0x7f67e414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 >> 172.21.15.50:0/2812702952 conn(0x7f67e4067bb0 msgr2=0x7f67e40c4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:49.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 shutdown_connections 2023-12-19T14:51:49.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:49.887+0000 7f67e3fff700 1 -- 172.21.15.50:0/2812702952 wait complete. 2023-12-19T14:51:49.918 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:51:49.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:51:49.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:51:49.984 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:51:49.985 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:51:49.985 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:51:49.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:51:50.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[42574]: fuse finished with error 0 and tester_r 0 2023-12-19T14:51:50.083 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:51:50.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:51:50.083 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:51:50.152 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:50.145+0000 7fe1e7b0c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:51:50.187 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:50.183+0000 7f68bbbc3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:51:50.227 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:51:50.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:51:50.227 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:51:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:51:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:51:50.249 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:51:50.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:51:50.249 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:51:50.296 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:51:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:51:50.297 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:51:50.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:51:50.298 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:51:50.317 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:51:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:51:50.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:51:50.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 Processor -- start 2023-12-19T14:51:50.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 -- start start 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 0x7fb87c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c06b730 con 0x7fb87c158e20 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c06ab20 con 0x7fb87c1591f0 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.503+0000 7fb87b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c06ae00 con 0x7fb87c0ce810 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58240/0 (socket says 172.21.15.50:58240) 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 -- 172.21.15.50:0/3128699517 learned_addr learned my addr 172.21.15.50:0/3128699517 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb879d9b700 1 --2- 172.21.15.50:0/3128699517 >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 0x7fb87c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87a59c700 1 --1- 172.21.15.50:0/3128699517 >> v1:172.21.15.50:6789/0 conn(0x7fb87c158e20 0x7fb87c0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58782/0 (socket says 172.21.15.50:58782) 2023-12-19T14:51:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 -- 172.21.15.50:0/3128699517 >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 msgr2=0x7fb87c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 --2- 172.21.15.50:0/3128699517 >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 0x7fb87c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 -- 172.21.15.50:0/3128699517 >> v1:172.21.15.50:6789/0 conn(0x7fb87c158e20 legacy=0x7fb87c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 -- 172.21.15.50:0/3128699517 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c06b0e0 con 0x7fb87c0ce810 2023-12-19T14:51:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb879d9b700 1 --2- 172.21.15.50:0/3128699517 >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 0x7fb87c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87ad9d700 1 --2- 172.21.15.50:0/3128699517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb874009f20 tx=0x7fb874009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d05a5b6976e8cf34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87959a700 1 -- 172.21.15.50:0/3128699517 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87400e070 con 0x7fb87c0ce810 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87959a700 1 -- 172.21.15.50:0/3128699517 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87400b9d0 con 0x7fb87c0ce810 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87959a700 1 -- 172.21.15.50:0/3128699517 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740092e0 con 0x7fb87c0ce810 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 -- 172.21.15.50:0/3128699517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 msgr2=0x7fb87c0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 --2- 172.21.15.50:0/3128699517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb874009f20 tx=0x7fb874009840 comp rx=0 tx=0).stop 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 -- 172.21.15.50:0/3128699517 shutdown_connections 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 --2- 172.21.15.50:0/3128699517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c0ce810 0x7fb87c0ca950 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 --2- 172.21.15.50:0/3128699517 >> v2:172.21.15.125:3300/0 conn(0x7fb87c1591f0 0x7fb87c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 -- 172.21.15.50:0/3128699517 >> 172.21.15.50:0/3128699517 conn(0x7fb87c067bb0 msgr2=0x7fb87c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 -- 172.21.15.50:0/3128699517 shutdown_connections 2023-12-19T14:51:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.507+0000 7fb87b59e700 1 -- 172.21.15.50:0/3128699517 wait complete. 2023-12-19T14:51:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 Processor -- start 2023-12-19T14:51:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 -- start start 2023-12-19T14:51:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 0x7fb87c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 0x7fb87c0c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb87c1591f0 0x7fb87c0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c71a0 con 0x7fb87c0ce810 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c6e70 con 0x7fb87c1591f0 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c15a9f0 con 0x7fb87c158e20 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 0x7fb87c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 0x7fb87c155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56402/0 (socket says 172.21.15.50:56402) 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87a59c700 1 -- 172.21.15.50:0/558667883 learned_addr learned my addr 172.21.15.50:0/558667883 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb879d9b700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 0x7fb87c0c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.511+0000 7fb87ad9d700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb87c1591f0 0x7fb87c0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb87c1591f0 msgr2=0x7fb87c0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb87c1591f0 0x7fb87c0c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 msgr2=0x7fb87c155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 0x7fb87c155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 -- 172.21.15.50:0/558667883 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb870009420 con 0x7fb87c158e20 2023-12-19T14:51:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb879d9b700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 0x7fb87c0c6200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb87000ea00 tx=0x7fb8700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87000c850 con 0x7fb87c158e20 2023-12-19T14:51:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87000c9b0 con 0x7fb87c158e20 2023-12-19T14:51:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87000cd50 con 0x7fb87c158e20 2023-12-19T14:51:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb874009ae0 con 0x7fb87c158e20 2023-12-19T14:51:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0c9950 con 0x7fb87c158e20 2023-12-19T14:51:50.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.515+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c0c7310 con 0x7fb87c158e20 2023-12-19T14:51:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.519+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fb870013680 con 0x7fb87c158e20 2023-12-19T14:51:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.519+0000 7fb86a7fc700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 0x7fb85c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.519+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fb87007d240 con 0x7fb87c158e20 2023-12-19T14:51:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.519+0000 7fb87a59c700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 msgr2=0x7fb85c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.519+0000 7fb87a59c700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 0x7fb85c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:50.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.527+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb87004d080 con 0x7fb87c158e20 2023-12-19T14:51:50.697 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:50.689+0000 7fe1e7b0c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:51:50.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.723+0000 7fb87a59c700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 msgr2=0x7fb85c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:50.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.723+0000 7fb87a59c700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 0x7fb85c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:51:50.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.739+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb87c04be40 con 0x7fb87c158e20 2023-12-19T14:51:50.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.739+0000 7fb86a7fc700 1 -- 172.21.15.50:0/558667883 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fb8700506a0 con 0x7fb87c158e20 2023-12-19T14:51:50.743 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:50.743 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.743+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 msgr2=0x7fb85c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.743+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 0x7fb85c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.743+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 msgr2=0x7fb87c0c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.743+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 0x7fb87c0c6200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb87000ea00 tx=0x7fb8700091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 shutdown_connections 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb87c1591f0 0x7fb87c0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb85c060700 0x7fb85c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb87c158e20 0x7fb87c0c6200 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 --2- 172.21.15.50:0/558667883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb87c0ce810 0x7fb87c155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 >> 172.21.15.50:0/558667883 conn(0x7fb87c067bb0 msgr2=0x7fb87c152400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 shutdown_connections 2023-12-19T14:51:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.747+0000 7fb87b59e700 1 -- 172.21.15.50:0/558667883 wait complete. 2023-12-19T14:51:50.750 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:51:50.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:51:50.774 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:50.765+0000 7fe1e7b0c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:51:50.785 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:50.783+0000 7f68bbbc3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:51:50.876 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:50.875+0000 7f68bbbc3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:51:50.919 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:50.913+0000 7fe1e7b0c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:51:50.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 Processor -- start 2023-12-19T14:51:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 -- start start 2023-12-19T14:51:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efeb00ae510 0x7efeb00a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb006dc20 con 0x7efeb00c1d60 2023-12-19T14:51:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb006d940 con 0x7efeb00ae140 2023-12-19T14:51:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeb542e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb00c2200 con 0x7efeb00ae510 2023-12-19T14:51:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeaf7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efeb00c1d60 0x7efeb00680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58792/0 (socket says 172.21.15.50:58792) 2023-12-19T14:51:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeaf7fe700 1 -- 172.21.15.50:0/393224576 learned_addr learned my addr 172.21.15.50:0/393224576 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efea7fff700 1 --2- 172.21.15.50:0/393224576 >> v2:172.21.15.125:3300/0 conn(0x7efeb00ae510 0x7efeb00a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.931+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613709773 0 0) 0x7efeb006dc20 con 0x7efeb00c1d60 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c003570 con 0x7efeb00c1d60 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853792033 0 0) 0x7efeb006d940 con 0x7efeb00ae140 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb006dc20 con 0x7efeb00ae140 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efea7fff700 1 -- 172.21.15.50:0/393224576 >> v1:172.21.15.125:6789/0 conn(0x7efeb00ae140 legacy=0x7efeb006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efea7fff700 1 -- 172.21.15.50:0/393224576 >> v1:172.21.15.50:6789/0 conn(0x7efeb00c1d60 legacy=0x7efeb00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efea7fff700 1 -- 172.21.15.50:0/393224576 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00c6e20 con 0x7efeb00ae510 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efea7fff700 1 --2- 172.21.15.50:0/393224576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb00a8850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe980098e0 tx=0x7efe980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec3a0c91fd71c26 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98011630 con 0x7efeb00ae510 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe98011790 con 0x7efeb00ae510 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeae7fc700 1 -- 172.21.15.50:0/393224576 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98011b30 con 0x7efeb00ae510 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 -- 172.21.15.50:0/393224576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 msgr2=0x7efeb00a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 --2- 172.21.15.50:0/393224576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb00a8850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efe980098e0 tx=0x7efe980091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 -- 172.21.15.50:0/393224576 shutdown_connections 2023-12-19T14:51:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 --2- 172.21.15.50:0/393224576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb00a8850 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 -- 172.21.15.50:0/393224576 >> 172.21.15.50:0/393224576 conn(0x7efeb00c8850 msgr2=0x7efeb00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 -- 172.21.15.50:0/393224576 shutdown_connections 2023-12-19T14:51:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.935+0000 7efeb542e700 1 -- 172.21.15.50:0/393224576 wait complete. 2023-12-19T14:51:50.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 Processor -- start 2023-12-19T14:51:50.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 -- start start 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 0x7efeb015b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb0159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 0x7efeb0156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00aa630 con 0x7efeb00c1d60 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb015a8a0 con 0x7efeb00ae510 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00aa8d0 con 0x7efeb00ae140 2023-12-19T14:51:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 0x7efeb0156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 0x7efeb015b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 0x7efeb0156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56412/0 (socket says 172.21.15.50:56412) 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaf7fe700 1 -- 172.21.15.50:0/1244145206 learned_addr learned my addr 172.21.15.50:0/1244145206 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efea7fff700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb0159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 -- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 msgr2=0x7efeb0159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb0159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 -- 172.21.15.50:0/1244145206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 msgr2=0x7efeb0156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 0x7efeb0156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 -- 172.21.15.50:0/1244145206 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98009420 con 0x7efeb00ae140 2023-12-19T14:51:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeaeffd700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 0x7efeb015b530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efea000c560 tx=0x7efea0007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0011590 con 0x7efeb00ae140 2023-12-19T14:51:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea00116f0 con 0x7efeb00ae140 2023-12-19T14:51:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0011ab0 con 0x7efeb00ae140 2023-12-19T14:51:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.939+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb0159d60 con 0x7efeb00ae140 2023-12-19T14:51:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.943+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb015a2e0 con 0x7efeb00ae140 2023-12-19T14:51:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.943+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0071120 con 0x7efeb00ae140 2023-12-19T14:51:50.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.943+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7efea001c460 con 0x7efeb00ae140 2023-12-19T14:51:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.943+0000 7efea6ffd700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 0x7efe90062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.947+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7efea007cc10 con 0x7efeb00ae140 2023-12-19T14:51:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.947+0000 7efea7fff700 1 -- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 msgr2=0x7efe90062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.947+0000 7efea7fff700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 0x7efe90062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:50.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:50.955+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efea004ca10 con 0x7efeb00ae140 2023-12-19T14:51:51.026 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.023+0000 7f68bbbc3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:51:51.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.131+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeb00642a0 con 0x7efeb00ae140 2023-12-19T14:51:51.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.131+0000 7efea6ffd700 1 -- 172.21.15.50:0/1244145206 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17005 (secure 0 0 0) 0x7efea0050030 con 0x7efeb00ae140 2023-12-19T14:51:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:51.134 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":53,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:51:49.861436+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3449382498":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/751498889":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/4107501911":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6833/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6832/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/1210582553":"2023-12-20T14:40:35.182753+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":4,"length":3},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":17,"length":3},{"begin":22,"length":2},{"begin":26,"length":1},{"begin":28,"length":1},{"begin":30,"length":2},{"begin":33,"length":4},{"begin":39,"length":1},{"begin":41,"length":1},{"begin":43,"length":1},{"begin":46,"length":5},{"begin":52,"length":2},{"begin":55,"length":2},{"begin":60,"length":3},{"begin":64,"length":3},{"begin":68,"length":2},{"begin":72,"length":3},{"begin":78,"length":3},{"begin":84,"length":4},{"begin":89,"length":2},{"begin":92,"length":1},{"begin":96,"length":2},{"begin":99,"length":4},{"begin":104,"length":2},{"begin":108,"length":1},{"begin":110,"length":2},{"begin":155,"length":66}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":6,"length":1},{"begin":19,"length":1},{"begin":22,"length":1},{"begin":33,"length":1},{"begin":46,"length":1},{"begin":60,"length":1},{"begin":64,"length":1},{"begin":78,"length":1},{"begin":85,"length":1},{"begin":100,"length":1},{"begin":211,"length":10}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:51:51.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 msgr2=0x7efe90062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 0x7efe90062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 msgr2=0x7efeb015b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 0x7efeb015b530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efea000c560 tx=0x7efea0007d50 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 shutdown_connections 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efeb00ae510 0x7efeb0159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efe90060700 0x7efe90062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efeb00c1d60 0x7efeb0156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 --2- 172.21.15.50:0/1244145206 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efeb00ae140 0x7efeb015b530 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 >> 172.21.15.50:0/1244145206 conn(0x7efeb00c8850 msgr2=0x7efeb0158800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 shutdown_connections 2023-12-19T14:51:51.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.135+0000 7efeb542e700 1 -- 172.21.15.50:0/1244145206 wait complete. 2023-12-19T14:51:51.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:40:13.626675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:40:14.423960+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:51:51.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:51:51.180 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.173+0000 7fe1e7b0c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:51:51.252 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.245+0000 7fe1e7b0c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:51:51.316 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.315+0000 7f68bbbc3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:51:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 Processor -- start 2023-12-19T14:51:51.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 -- start start 2023-12-19T14:51:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac4415c520 0x7fac440a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac440ca240 0x7fac440ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac440cabc0 con 0x7fac4415c8f0 2023-12-19T14:51:51.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4406b520 con 0x7fac440ca240 2023-12-19T14:51:51.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac496ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac440cb910 con 0x7fac4415c520 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac3a7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fac4415c8f0 0x7fac4406a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58804/0 (socket says 172.21.15.50:58804) 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac3a7fc700 1 -- 172.21.15.50:0/4161652150 learned_addr learned my addr 172.21.15.50:0/4161652150 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac437fe700 1 --2- 172.21.15.50:0/4161652150 >> v2:172.21.15.125:3300/0 conn(0x7fac440ca240 0x7fac440ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac42ffd700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac4415c520 0x7fac440a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac437fe700 1 -- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac4415c520 msgr2=0x7fac440a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac437fe700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac4415c520 0x7fac440a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac437fe700 1 -- 172.21.15.50:0/4161652150 >> v1:172.21.15.50:6789/0 conn(0x7fac4415c8f0 legacy=0x7fac4406a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac437fe700 1 -- 172.21.15.50:0/4161652150 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac440c6ab0 con 0x7fac440ca240 2023-12-19T14:51:51.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.371+0000 7fac427fc700 1 -- 172.21.15.50:0/4161652150 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732937545 0 0) 0x7fac440cabc0 con 0x7fac4415c8f0 2023-12-19T14:51:51.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac437fe700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac440ca240 0x7fac440ca680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0098e0 tx=0x7fac3c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825be98cb716b95f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:51.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac427fc700 1 -- 172.21.15.50:0/4161652150 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00b6d0 con 0x7fac440ca240 2023-12-19T14:51:51.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac427fc700 1 -- 172.21.15.50:0/4161652150 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c00b830 con 0x7fac440ca240 2023-12-19T14:51:51.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac427fc700 1 -- 172.21.15.50:0/4161652150 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00bbd0 con 0x7fac440ca240 2023-12-19T14:51:51.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac496ae700 1 -- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac440ca240 msgr2=0x7fac440ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.375+0000 7fac496ae700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac440ca240 0x7fac440ca680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0098e0 tx=0x7fac3c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 -- 172.21.15.50:0/4161652150 shutdown_connections 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac440ca240 0x7fac440ca680 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 --2- 172.21.15.50:0/4161652150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac4415c520 0x7fac440a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 -- 172.21.15.50:0/4161652150 >> 172.21.15.50:0/4161652150 conn(0x7fac44067bb0 msgr2=0x7fac440cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 -- 172.21.15.50:0/4161652150 shutdown_connections 2023-12-19T14:51:51.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 -- 172.21.15.50:0/4161652150 wait complete. 2023-12-19T14:51:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 Processor -- start 2023-12-19T14:51:51.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.379+0000 7fac496ae700 1 -- start start 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 0x7fac440c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 0x7fac4414c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 0x7fac4415edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac44169400 con 0x7fac4415c8f0 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac440afd40 con 0x7fac4415c520 2023-12-19T14:51:51.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac440b0300 con 0x7fac440ca240 2023-12-19T14:51:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac42ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 0x7fac440c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 0x7fac4414c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac437fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 0x7fac4415edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac42ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 0x7fac440c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58276/0 (socket says 172.21.15.50:58276) 2023-12-19T14:51:51.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac42ffd700 1 -- 172.21.15.50:0/709363074 learned_addr learned my addr 172.21.15.50:0/709363074 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 -- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 msgr2=0x7fac440c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 0x7fac440c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 -- 172.21.15.50:0/709363074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 msgr2=0x7fac4415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 0x7fac4415edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 -- 172.21.15.50:0/709363074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c009420 con 0x7fac4415c520 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac437fe700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 0x7fac4415edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3a7fc700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 0x7fac4414c750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c4e0 tx=0x7fac2c007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00ce10 con 0x7fac4415c520 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c005b30 con 0x7fac4415c520 2023-12-19T14:51:51.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac440c3f00 con 0x7fac4415c520 2023-12-19T14:51:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0070d0 con 0x7fac4415c520 2023-12-19T14:51:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.383+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac440aff90 con 0x7fac4415c520 2023-12-19T14:51:51.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac44071120 con 0x7fac4415c520 2023-12-19T14:51:51.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fac2c007500 con 0x7fac4415c520 2023-12-19T14:51:51.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac3b7fe700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 0x7fac24062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7fac2c07c320 con 0x7fac4415c520 2023-12-19T14:51:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac42ffd700 1 -- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 msgr2=0x7fac24062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:51.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.387+0000 7fac42ffd700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 0x7fac24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:51.397 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.395+0000 7f68bbbc3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:51:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.395+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac2c04c120 con 0x7fac4415c520 2023-12-19T14:51:51.456 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.449+0000 7fe1e7b0c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:51:51.536 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.529+0000 7fe1e7b0c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:51:51.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.563+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac4406b240 con 0x7fac4415c520 2023-12-19T14:51:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.567+0000 7fac3b7fe700 1 -- 172.21.15.50:0/709363074 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac2c04f740 con 0x7fac4415c520 2023-12-19T14:51:51.568 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 msgr2=0x7fac24062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 0x7fac24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 msgr2=0x7fac4414c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 0x7fac4414c750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c4e0 tx=0x7fac2c007f30 comp rx=0 tx=0).stop 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 shutdown_connections 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac4415c520 0x7fac4414c750 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac24060700 0x7fac24062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac4415c8f0 0x7fac4415edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 --2- 172.21.15.50:0/709363074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac440ca240 0x7fac440c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 >> 172.21.15.50:0/709363074 conn(0x7fac44067bb0 msgr2=0x7fac440c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:51.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 shutdown_connections 2023-12-19T14:51:51.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.571+0000 7fac496ae700 1 -- 172.21.15.50:0/709363074 wait complete. 2023-12-19T14:51:51.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:51:51.615 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.609+0000 7fe1e7b0c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:51:51.660 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.659+0000 7f68bbbc3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:51:51.753 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.751+0000 7f68bbbc3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:51:51.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 Processor -- start 2023-12-19T14:51:51.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 -- start start 2023-12-19T14:51:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f97bc0cc040 0x7f97bc159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0cbbb0 con 0x7f97bc0ce2d0 2023-12-19T14:51:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc06dc90 con 0x7f97bc0caa10 2023-12-19T14:51:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c2123700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0cafa0 con 0x7f97bc0cc040 2023-12-19T14:51:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c1121700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f97bc0ce2d0 0x7f97bc158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58814/0 (socket says 172.21.15.50:58814) 2023-12-19T14:51:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c1121700 1 -- 172.21.15.50:0/3223767478 learned_addr learned my addr 172.21.15.50:0/3223767478 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:51.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.767+0000 7f97c1922700 1 --2- 172.21.15.50:0/3223767478 >> v2:172.21.15.125:3300/0 conn(0x7f97bc0cc040 0x7f97bc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97b3fff700 1 -- 172.21.15.50:0/3223767478 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821477535 0 0) 0x7f97bc0cbbb0 con 0x7f97bc0ce2d0 2023-12-19T14:51:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97b3fff700 1 -- 172.21.15.50:0/3223767478 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a8003570 con 0x7f97bc0ce2d0 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c1922700 1 -- 172.21.15.50:0/3223767478 >> v1:172.21.15.125:6789/0 conn(0x7f97bc0caa10 legacy=0x7f97bc06a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c1922700 1 -- 172.21.15.50:0/3223767478 >> v1:172.21.15.50:6789/0 conn(0x7f97bc0ce2d0 legacy=0x7f97bc158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c1922700 1 -- 172.21.15.50:0/3223767478 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0cb280 con 0x7f97bc0cc040 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c1922700 1 --2- 172.21.15.50:0/3223767478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc159010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009e60 tx=0x7f97b4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=935aa61ffb38c7f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97b3fff700 1 -- 172.21.15.50:0/3223767478 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b400e070 con 0x7f97bc0cc040 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97b3fff700 1 -- 172.21.15.50:0/3223767478 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b400bb60 con 0x7f97bc0cc040 2023-12-19T14:51:51.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97b3fff700 1 -- 172.21.15.50:0/3223767478 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4012430 con 0x7f97bc0cc040 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 -- 172.21.15.50:0/3223767478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 msgr2=0x7f97bc159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 --2- 172.21.15.50:0/3223767478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc159010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009e60 tx=0x7f97b4009780 comp rx=0 tx=0).stop 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 -- 172.21.15.50:0/3223767478 shutdown_connections 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 --2- 172.21.15.50:0/3223767478 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc159010 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 -- 172.21.15.50:0/3223767478 >> 172.21.15.50:0/3223767478 conn(0x7f97bc067bb0 msgr2=0x7f97bc0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 -- 172.21.15.50:0/3223767478 shutdown_connections 2023-12-19T14:51:51.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.771+0000 7f97c2123700 1 -- 172.21.15.50:0/3223767478 wait complete. 2023-12-19T14:51:51.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 Processor -- start 2023-12-19T14:51:51.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 -- start start 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97bc0ce2d0 0x7f97bc156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0b4dd0 con 0x7f97bc0caa10 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc151d20 con 0x7f97bc0cc040 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c2123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc16cd70 con 0x7f97bc0ce2d0 2023-12-19T14:51:51.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56420/0 (socket says 172.21.15.50:56420) 2023-12-19T14:51:51.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 -- 172.21.15.50:0/2877641994 learned_addr learned my addr 172.21.15.50:0/2877641994 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:51.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c0920700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1922700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97bc0ce2d0 0x7f97bc156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:51.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 -- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97bc0ce2d0 msgr2=0x7f97bc156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97bc0ce2d0 0x7f97bc156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 -- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 msgr2=0x7f97bc14eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc14eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.775+0000 7f97c1121700 1 -- 172.21.15.50:0/2877641994 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4009a20 con 0x7f97bc0caa10 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97c1121700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c5c0 tx=0x7f97b8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8005b30 con 0x7f97bc0caa10 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8005cb0 con 0x7f97bc0caa10 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8007320 con 0x7f97bc0caa10 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc16d050 con 0x7f97bc0caa10 2023-12-19T14:51:51.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc151960 con 0x7f97bc0caa10 2023-12-19T14:51:51.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.779+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc04be40 con 0x7f97bc0caa10 2023-12-19T14:51:51.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.783+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f97b801b3b0 con 0x7f97bc0caa10 2023-12-19T14:51:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.783+0000 7f97b17fa700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 0x7f97a0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.783+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f97b807c440 con 0x7f97bc0caa10 2023-12-19T14:51:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.783+0000 7f97c0920700 1 -- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 msgr2=0x7f97a0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:51.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.783+0000 7f97c0920700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 0x7f97a0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:51.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.791+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f97b804c240 con 0x7f97bc0caa10 2023-12-19T14:51:51.848 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:51.847+0000 7f68bbbc3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:51:51.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.967+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f97bc0cb560 con 0x7f97bc0caa10 2023-12-19T14:51:51.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97b17fa700 1 -- 172.21.15.50:0/2877641994 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f97b804f860 con 0x7f97bc0caa10 2023-12-19T14:51:51.972 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:51:51.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 msgr2=0x7f97a0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:51.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 0x7f97a0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 msgr2=0x7f97bc16c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:51.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c5c0 tx=0x7f97b8007e00 comp rx=0 tx=0).stop 2023-12-19T14:51:51.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 shutdown_connections 2023-12-19T14:51:51.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0cc040 0x7f97bc14eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f97a0060700 0x7f97a0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97bc0ce2d0 0x7f97bc156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 --2- 172.21.15.50:0/2877641994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97bc0caa10 0x7f97bc16c670 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.971+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 >> 172.21.15.50:0/2877641994 conn(0x7f97bc067bb0 msgr2=0x7f97bc0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.975+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 shutdown_connections 2023-12-19T14:51:51.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:51.975+0000 7f97c2123700 1 -- 172.21.15.50:0/2877641994 wait complete. 2023-12-19T14:51:51.983 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:51.977+0000 7fe1e7b0c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:51:51.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:51:52.141 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.133+0000 7fe1e7b0c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:51:52.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.174+0000 7f6c80acd700 1 Processor -- start 2023-12-19T14:51:52.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.174+0000 7f6c80acd700 1 -- start start 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c80acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0cb970 0x7f6c7c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c80acd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0cbdf0 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c80acd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0cc0d0 con 0x7f6c7c158e20 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c80acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cc3b0 con 0x7f6c7c0cb970 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6c7c0cecd0 0x7f6c7c06b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58828/0 (socket says 172.21.15.50:58828) 2023-12-19T14:51:52.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/169212046 learned_addr learned my addr 172.21.15.50:0/169212046 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c79d9b700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0cb970 0x7f6c7c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379762631 0 0) 0x7f6c7c0cbdf0 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c64003570 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666312576 0 0) 0x7f6c7c0cc0d0 con 0x7f6c7c158e20 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c7c0cbdf0 con 0x7f6c7c158e20 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870416066 0 0) 0x7f6c64003570 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0cb970 msgr2=0x7f6c7c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0cb970 0x7f6c7c06acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 >> v1:172.21.15.125:6789/0 conn(0x7f6c7c158e20 legacy=0x7f6c7c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c0cc690 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f6c740017f0 con 0x7f6c7c0cecd0 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 >> v1:172.21.15.50:6789/0 conn(0x7f6c7c0cecd0 legacy=0x7f6c7c06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c64003c20 0x7f6c64004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c64004810 0x7f6c64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c6400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0cc3b0 con 0x7f6c64003c20 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6400bfa0 con 0x7f6c64005180 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6400c0e0 con 0x7f6c64004810 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7a59c700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c64003c20 0x7f6c64004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c6400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c79d9b700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c64004810 0x7f6c64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c64004810 msgr2=0x7f6c64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c64004810 0x7f6c64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/169212046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c64003c20 msgr2=0x7f6c64004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c64003c20 0x7f6c64004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/169212046 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64003570 con 0x7f6c64005180 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.178+0000 7f6c7a59c700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c64003c20 0x7f6c64004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c7ad9d700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c6400ba60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0cfd80 tx=0x7f6c74002190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fa23e8e2ae962cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740070d0 con 0x7f6c64005180 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c74007230 con 0x7f6c64005180 2023-12-19T14:51:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c7959a700 1 -- 172.21.15.50:0/169212046 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740075f0 con 0x7f6c64005180 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 -- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 msgr2=0x7f6c6400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 --2- 172.21.15.50:0/169212046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c6400ba60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0cfd80 tx=0x7f6c74002190 comp rx=0 tx=0).stop 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/169212046 reap_dead start 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 -- 172.21.15.50:0/169212046 shutdown_connections 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 -- 172.21.15.50:0/169212046 >> 172.21.15.50:0/169212046 conn(0x7f6c7c067bb0 msgr2=0x7f6c7c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 -- 172.21.15.50:0/169212046 shutdown_connections 2023-12-19T14:51:52.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.182+0000 7f6c80acd700 1 -- 172.21.15.50:0/169212046 wait complete. 2023-12-19T14:51:52.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 Processor -- start 2023-12-19T14:51:52.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 -- start start 2023-12-19T14:51:52.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c7c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0c60b0 0x7f6c7c0c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0b5db0 con 0x7f6c7c159400 2023-12-19T14:51:52.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0c6a60 con 0x7f6c64005180 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c80acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c159de0 con 0x7f6c7c0c60b0 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c7c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56434/0 (socket says 172.21.15.50:56434) 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7ad9d700 1 -- 172.21.15.50:0/469754378 learned_addr learned my addr 172.21.15.50:0/469754378 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:52.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c79d9b700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0c60b0 0x7f6c7c0c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 -- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0c60b0 msgr2=0x7f6c7c0c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0c60b0 0x7f6c7c0c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 -- 172.21.15.50:0/469754378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 msgr2=0x7f6c7c1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 -- 172.21.15.50:0/469754378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c16d5f0 con 0x7f6c64005180 2023-12-19T14:51:52.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7ad9d700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.186+0000 7f6c7a59c700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c7c15bc10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00f910 tx=0x7f6c6c007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c006820 con 0x7f6c64005180 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c006980 con 0x7f6c64005180 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c15a010 con 0x7f6c64005180 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c006d20 con 0x7f6c64005180 2023-12-19T14:51:52.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c16d280 con 0x7f6c64005180 2023-12-19T14:51:52.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c071120 con 0x7f6c64005180 2023-12-19T14:51:52.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.190+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f6c6c007230 con 0x7f6c64005180 2023-12-19T14:51:52.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.194+0000 7f6c6a7fc700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 0x7f6c5c062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.194+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f6c6c07cae0 con 0x7f6c64005180 2023-12-19T14:51:52.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.194+0000 7f6c79d9b700 1 -- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 msgr2=0x7f6c5c062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:52.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.194+0000 7f6c79d9b700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 0x7f6c5c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.202+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c6c04c960 con 0x7f6c64005180 2023-12-19T14:51:52.293 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:52.290+0000 7f68bbbc3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:51:52.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.374+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c7c08b160 con 0x7f6c64005180 2023-12-19T14:51:52.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.374+0000 7f6c6a7fc700 1 -- 172.21.15.50:0/469754378 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c6c04ff80 con 0x7f6c64005180 2023-12-19T14:51:52.379 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 msgr2=0x7f6c5c062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 0x7f6c5c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 msgr2=0x7f6c7c15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c7c15bc10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00f910 tx=0x7f6c6c007d30 comp rx=0 tx=0).stop 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 shutdown_connections 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060730 0x7f6c5c062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7c159400 0x7f6c7c1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c7c0c60b0 0x7f6c7c0c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 --2- 172.21.15.50:0/469754378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c64005180 0x7f6c7c15bc10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 >> 172.21.15.50:0/469754378 conn(0x7f6c7c067bb0 msgr2=0x7f6c7c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:52.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 shutdown_connections 2023-12-19T14:51:52.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.378+0000 7f6c80acd700 1 -- 172.21.15.50:0/469754378 wait complete. 2023-12-19T14:51:52.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:51:52.467 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:52.462+0000 7f68bbbc3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:51:52.509 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.501+0000 7fe1e7b0c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:51:52.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.558+0000 7f83bbacd700 1 Processor -- start 2023-12-19T14:51:52.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.558+0000 7f83bbacd700 1 -- start start 2023-12-19T14:51:52.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83bbacd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83b4140710 0x7f83b40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83bbacd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83b4068800 0x7f83b40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83bbacd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b406e150 con 0x7f83b4140ae0 2023-12-19T14:51:52.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83bbacd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b406d540 con 0x7f83b4068800 2023-12-19T14:51:52.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83bbacd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b406d820 con 0x7f83b4140710 2023-12-19T14:51:52.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83b4068800 0x7f83b40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f83b4068800 0x7f83b40ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56448/0 (socket says 172.21.15.50:56448) 2023-12-19T14:51:52.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 -- 172.21.15.50:0/2656001391 learned_addr learned my addr 172.21.15.50:0/2656001391 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83b9869700 1 --2- 172.21.15.50:0/2656001391 >> v2:172.21.15.125:3300/0 conn(0x7f83b4140710 0x7f83b40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 -- 172.21.15.50:0/2656001391 >> v2:172.21.15.125:3300/0 conn(0x7f83b4140710 msgr2=0x7f83b40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 --2- 172.21.15.50:0/2656001391 >> v2:172.21.15.125:3300/0 conn(0x7f83b4140710 0x7f83b40c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 -- 172.21.15.50:0/2656001391 >> v1:172.21.15.125:6789/0 conn(0x7f83b4140ae0 legacy=0x7f83b40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 -- 172.21.15.50:0/2656001391 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b406db00 con 0x7f83b4068800 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83ba06a700 1 --2- 172.21.15.50:0/2656001391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4068800 0x7f83b40ac250 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f83b4068c30 tx=0x7f83ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d39f1a8599f6ee server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.562+0000 7f83b8867700 1 -- 172.21.15.50:0/2656001391 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00e070 con 0x7f83b4068800 2023-12-19T14:51:52.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83b8867700 1 -- 172.21.15.50:0/2656001391 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac00ba20 con 0x7f83b4068800 2023-12-19T14:51:52.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83b8867700 1 -- 172.21.15.50:0/2656001391 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0092e0 con 0x7f83b4068800 2023-12-19T14:51:52.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- 172.21.15.50:0/2656001391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4068800 msgr2=0x7f83b40ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- 172.21.15.50:0/2656001391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4068800 0x7f83b40ac250 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f83b4068c30 tx=0x7f83ac009840 comp rx=0 tx=0).stop 2023-12-19T14:51:52.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- 172.21.15.50:0/2656001391 shutdown_connections 2023-12-19T14:51:52.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- 172.21.15.50:0/2656001391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4068800 0x7f83b40ac250 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- 172.21.15.50:0/2656001391 >> v2:172.21.15.125:3300/0 conn(0x7f83b4140710 0x7f83b40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- 172.21.15.50:0/2656001391 >> 172.21.15.50:0/2656001391 conn(0x7f83b40c8850 msgr2=0x7f83b40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:52.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- 172.21.15.50:0/2656001391 shutdown_connections 2023-12-19T14:51:52.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- 172.21.15.50:0/2656001391 wait complete. 2023-12-19T14:51:52.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 Processor -- start 2023-12-19T14:51:52.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- start start 2023-12-19T14:51:52.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4068800 0x7f83b4169b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f83b4140710 0x7f83b40aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41696c0 con 0x7f83b4140ae0 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40ab0a0 con 0x7f83b4068800 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.566+0000 7f83bbacd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40ab380 con 0x7f83b4140710 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56462/0 (socket says 172.21.15.50:56462) 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 -- 172.21.15.50:0/4115776624 learned_addr learned my addr 172.21.15.50:0/4115776624 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83b9068700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f83b4140710 0x7f83b40aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83b9869700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4068800 0x7f83b4169b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f83b4140710 msgr2=0x7f83b40aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f83b4140710 0x7f83b40aaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4068800 msgr2=0x7f83b4169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4068800 0x7f83b4169b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 -- 172.21.15.50:0/4115776624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac009ae0 con 0x7f83b4140ae0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83ba06a700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009880 tx=0x7f83ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac008be0 con 0x7f83b4140ae0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac008d60 con 0x7f83b4140ae0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac01c660 con 0x7f83b4140ae0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b4169070 con 0x7f83b4140ae0 2023-12-19T14:51:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.570+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4168d70 con 0x7f83b4140ae0 2023-12-19T14:51:52.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4169870 con 0x7f83b4140ae0 2023-12-19T14:51:52.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f83ac003040 con 0x7f83b4140ae0 2023-12-19T14:51:52.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83a9ffb700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 0x7f8398062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6976+0+0 (secure 0 0 0) 0x7f83ac07e980 con 0x7f83b4140ae0 2023-12-19T14:51:52.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83b9869700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 msgr2=0x7f8398062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:52.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.574+0000 7f83b9869700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 0x7f8398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:52.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.582+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f83b4169870 con 0x7f83b4140ae0 2023-12-19T14:51:52.723 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.717+0000 7fe1e7b0c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:51:52.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.778+0000 7f83b9869700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 msgr2=0x7f8398062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:52.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.778+0000 7f83b9869700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 0x7f8398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:51:52.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.790+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b4071120 con 0x7f83b4140ae0 2023-12-19T14:51:52.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.790+0000 7f83a9ffb700 1 -- 172.21.15.50:0/4115776624 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f83ac04e6f0 con 0x7f83b4140ae0 2023-12-19T14:51:52.793 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:52.793 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 msgr2=0x7f8398062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 0x7f8398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 msgr2=0x7f83b4168830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009880 tx=0x7f83ac018000 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 shutdown_connections 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8398060700 0x7f8398062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83b4140ae0 0x7f83b4168830 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f83b4140710 0x7f83b40aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 --2- 172.21.15.50:0/4115776624 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83b4068800 0x7f83b4169b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 >> 172.21.15.50:0/4115776624 conn(0x7f83b40c8850 msgr2=0x7f83b413e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:52.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 shutdown_connections 2023-12-19T14:51:52.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.794+0000 7f83bbacd700 1 -- 172.21.15.50:0/4115776624 wait complete. 2023-12-19T14:51:52.800 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:51:52.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:51:52.819 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.809+0000 7fe1e7b0c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:51:52.869 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:52.866+0000 7f68bbbc3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:51:52.895 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.889+0000 7fe1e7b0c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:51:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.974+0000 7f0396a78700 1 Processor -- start 2023-12-19T14:51:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.974+0000 7f0396a78700 1 -- start start 2023-12-19T14:51:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.974+0000 7f0396a78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03900c2980 0x7f03900aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.974+0000 7f0396a78700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 0x7f0390068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.974+0000 7f0396a78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039006c9f0 con 0x7f039013f890 2023-12-19T14:51:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f0396a78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03900c7f20 con 0x7f03900c2980 2023-12-19T14:51:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f0396a78700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0390068a50 con 0x7f03900680f0 2023-12-19T14:51:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f0394814700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f039013f890 0x7f03900c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58838/0 (socket says 172.21.15.50:58838) 2023-12-19T14:51:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f0394814700 1 -- 172.21.15.50:0/733410353 learned_addr learned my addr 172.21.15.50:0/733410353 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:52.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f038ffff700 1 --2- 172.21.15.50:0/733410353 >> v2:172.21.15.50:3300/0 conn(0x7f03900c2980 0x7f03900aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f038ffff700 1 -- 172.21.15.50:0/733410353 >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 msgr2=0x7f0390068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.978+0000 7f0395015700 1 --2- 172.21.15.50:0/733410353 >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 0x7f0390068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038ffff700 1 --2- 172.21.15.50:0/733410353 >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 0x7f0390068510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038ffff700 1 -- 172.21.15.50:0/733410353 >> v1:172.21.15.50:6789/0 conn(0x7f039013f890 legacy=0x7f03900c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038ffff700 1 -- 172.21.15.50:0/733410353 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03900c1c50 con 0x7f03900c2980 2023-12-19T14:51:52.984 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:52.977+0000 7fe1e7b0c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:51:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0395015700 1 --2- 172.21.15.50:0/733410353 >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 0x7f0390068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038ffff700 1 --2- 172.21.15.50:0/733410353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900aeba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f039006a6a0 tx=0x7f037c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410be06b6146e93f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038f7fe700 1 -- 172.21.15.50:0/733410353 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00b6b0 con 0x7f03900c2980 2023-12-19T14:51:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038f7fe700 1 -- 172.21.15.50:0/733410353 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c00b810 con 0x7f03900c2980 2023-12-19T14:51:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f038f7fe700 1 -- 172.21.15.50:0/733410353 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00bbb0 con 0x7f03900c2980 2023-12-19T14:51:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 -- 172.21.15.50:0/733410353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 msgr2=0x7f03900aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 --2- 172.21.15.50:0/733410353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900aeba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f039006a6a0 tx=0x7f037c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 -- 172.21.15.50:0/733410353 shutdown_connections 2023-12-19T14:51:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 --2- 172.21.15.50:0/733410353 >> v2:172.21.15.125:3300/0 conn(0x7f03900680f0 0x7f0390068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 --2- 172.21.15.50:0/733410353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900aeba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 -- 172.21.15.50:0/733410353 >> 172.21.15.50:0/733410353 conn(0x7f03900c8850 msgr2=0x7f03900c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 -- 172.21.15.50:0/733410353 shutdown_connections 2023-12-19T14:51:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.982+0000 7f0396a78700 1 -- 172.21.15.50:0/733410353 wait complete. 2023-12-19T14:51:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 Processor -- start 2023-12-19T14:51:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 -- start start 2023-12-19T14:51:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03900680f0 0x7f0390149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f039013f890 0x7f039006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03900c4960 con 0x7f03900c2980 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03900c44d0 con 0x7f039013f890 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0396a78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03900c38c0 con 0x7f03900680f0 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56470/0 (socket says 172.21.15.50:56470) 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 -- 172.21.15.50:0/551621691 learned_addr learned my addr 172.21.15.50:0/551621691 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0394814700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03900680f0 0x7f0390149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f0395015700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f039013f890 0x7f039006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 -- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03900680f0 msgr2=0x7f0390149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03900680f0 0x7f0390149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 -- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f039013f890 msgr2=0x7f039006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f039013f890 0x7f039006a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 -- 172.21.15.50:0/551621691 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c009420 con 0x7f03900c2980 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.986+0000 7f038ffff700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f037c013000 tx=0x7f037c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00bb90 con 0x7f03900c2980 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c011420 con 0x7f03900c2980 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c0117a0 con 0x7f03900c2980 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03900c3ba0 con 0x7f03900c2980 2023-12-19T14:51:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03900c4160 con 0x7f03900c2980 2023-12-19T14:51:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.990+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0390071120 con 0x7f03900c2980 2023-12-19T14:51:52.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.994+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f037c011b90 con 0x7f03900c2980 2023-12-19T14:51:52.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.994+0000 7f038cff9700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 0x7f0374062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.994+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f037c07f0e0 con 0x7f03900c2980 2023-12-19T14:51:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.994+0000 7f0394814700 1 -- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 msgr2=0x7f0374062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.994+0000 7f0394814700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 0x7f0374062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:53.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:52.998+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f037c04ef60 con 0x7f03900c2980 2023-12-19T14:51:53.127 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.122+0000 7f68bbbc3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:51:53.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.174+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03900c4610 con 0x7f03900c2980 2023-12-19T14:51:53.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.174+0000 7f038cff9700 1 -- 172.21.15.50:0/551621691 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17199 (secure 0 0 0) 0x7f037c052580 con 0x7f03900c2980 2023-12-19T14:51:53.178 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:53.178 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":54,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:51:52.837454+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3449382498":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/751498889":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/4107501911":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6833/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6832/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/1210582553":"2023-12-20T14:40:35.182753+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":15,"length":1},{"begin":18,"length":2},{"begin":22,"length":1},{"begin":28,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":41,"length":1},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":60,"length":2},{"begin":64,"length":3},{"begin":78,"length":3},{"begin":85,"length":3},{"begin":92,"length":1},{"begin":100,"length":2},{"begin":104,"length":1},{"begin":108,"length":1},{"begin":190,"length":31}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":2},{"begin":9,"length":2},{"begin":12,"length":2},{"begin":17,"length":1},{"begin":23,"length":1},{"begin":26,"length":1},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":36,"length":1},{"begin":39,"length":1},{"begin":43,"length":1},{"begin":48,"length":1},{"begin":50,"length":1},{"begin":52,"length":1},{"begin":55,"length":2},{"begin":62,"length":1},{"begin":68,"length":2},{"begin":72,"length":3},{"begin":84,"length":1},{"begin":89,"length":2},{"begin":96,"length":2},{"begin":99,"length":1},{"begin":102,"length":1},{"begin":105,"length":1},{"begin":110,"length":2},{"begin":155,"length":35}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 msgr2=0x7f0374062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 0x7f0374062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 msgr2=0x7f03900c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f037c013000 tx=0x7f037c017000 comp rx=0 tx=0).stop 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 shutdown_connections 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f039013f890 0x7f039006a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0374060700 0x7f0374062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03900c2980 0x7f03900c2e90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 --2- 172.21.15.50:0/551621691 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03900680f0 0x7f0390149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 >> 172.21.15.50:0/551621691 conn(0x7f03900c8850 msgr2=0x7f03901553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 shutdown_connections 2023-12-19T14:51:53.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.178+0000 7f0396a78700 1 -- 172.21.15.50:0/551621691 wait complete. 2023-12-19T14:51:53.197 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:53.189+0000 7fe1e7b0c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:51:53.202 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:51:53.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:51:53.216 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.214+0000 7f68bbbc3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:51:53.274 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:53.265+0000 7fe1e7b0c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:51:53.307 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.306+0000 7f68bbbc3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:51:53.357 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:53.349+0000 7fe1e7b0c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:51:53.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 Processor -- start 2023-12-19T14:51:53.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 -- start start 2023-12-19T14:51:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc06b730 con 0x7fa3fc158e20 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc06ab20 con 0x7fa3fc1591f0 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa401ce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc06ae00 con 0x7fa3fc0ce810 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa3f3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa4014e2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa3f3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:58252/0 (socket says 172.21.15.50:58252) 2023-12-19T14:51:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa3f3fff700 1 -- 172.21.15.50:0/3886947637 learned_addr learned my addr 172.21.15.50:0/3886947637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa4014e2700 1 -- 172.21.15.50:0/3886947637 >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 msgr2=0x7fa3fc06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa4014e2700 1 --2- 172.21.15.50:0/3886947637 >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa4014e2700 1 -- 172.21.15.50:0/3886947637 >> v1:172.21.15.125:6789/0 conn(0x7fa3fc158e20 legacy=0x7fa3fc0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa4014e2700 1 -- 172.21.15.50:0/3886947637 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc06b0e0 con 0x7fa3fc0ce810 2023-12-19T14:51:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.366+0000 7fa3f3fff700 1 --2- 172.21.15.50:0/3886947637 >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:53.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa4014e2700 1 --2- 172.21.15.50:0/3886947637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc0ca950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40098e0 tx=0x7fa3f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b9e28ec8ad6edb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa3f37fe700 1 -- 172.21.15.50:0/3886947637 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400b640 con 0x7fa3fc0ce810 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa3f37fe700 1 -- 172.21.15.50:0/3886947637 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f400b7a0 con 0x7fa3fc0ce810 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa3f37fe700 1 -- 172.21.15.50:0/3886947637 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400bb40 con 0x7fa3fc0ce810 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 -- 172.21.15.50:0/3886947637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 msgr2=0x7fa3fc0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 --2- 172.21.15.50:0/3886947637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc0ca950 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40098e0 tx=0x7fa3f40091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 -- 172.21.15.50:0/3886947637 shutdown_connections 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 --2- 172.21.15.50:0/3886947637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc0ca950 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 --2- 172.21.15.50:0/3886947637 >> v2:172.21.15.125:3300/0 conn(0x7fa3fc1591f0 0x7fa3fc06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 -- 172.21.15.50:0/3886947637 >> 172.21.15.50:0/3886947637 conn(0x7fa3fc067bb0 msgr2=0x7fa3fc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:53.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 -- 172.21.15.50:0/3886947637 shutdown_connections 2023-12-19T14:51:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 -- 172.21.15.50:0/3886947637 wait complete. 2023-12-19T14:51:53.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.370+0000 7fa401ce3700 1 Processor -- start 2023-12-19T14:51:53.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 -- start start 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc16d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3fc158e20 0x7fa3fc16c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c81f0 con 0x7fa3fc1591f0 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc16dbc0 con 0x7fa3fc158e20 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c9ad0 con 0x7fa3fc0ce810 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56474/0 (socket says 172.21.15.50:56474) 2023-12-19T14:51:53.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 -- 172.21.15.50:0/117155618 learned_addr learned my addr 172.21.15.50:0/117155618 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:53.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa3f3fff700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3fc158e20 0x7fa3fc16c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa400ce1700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc16d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 msgr2=0x7fa3fc16d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc16d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3fc158e20 msgr2=0x7fa3fc16c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3fc158e20 0x7fa3fc16c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 -- 172.21.15.50:0/117155618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec0094f0 con 0x7fa3fc1591f0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa4014e2700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06af40 tx=0x7fa3f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4011420 con 0x7fa3fc1591f0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4011580 con 0x7fa3fc1591f0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4011940 con 0x7fa3fc1591f0 2023-12-19T14:51:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.374+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f4009420 con 0x7fa3fc1591f0 2023-12-19T14:51:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.378+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc0c78c0 con 0x7fa3fc1591f0 2023-12-19T14:51:53.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.378+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc16a800 con 0x7fa3fc1591f0 2023-12-19T14:51:53.381 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.378+0000 7f68bbbc3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:51:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.378+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fa3f4011cb0 con 0x7fa3fc1591f0 2023-12-19T14:51:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.382+0000 7fa3f0ff9700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 0x7fa3e8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.382+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7104+0+0 (secure 0 0 0) 0x7fa3f407ec30 con 0x7fa3fc1591f0 2023-12-19T14:51:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.382+0000 7fa400ce1700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 msgr2=0x7fa3e8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:53.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.382+0000 7fa400ce1700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 0x7fa3e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:53.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.386+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa3fc16a800 con 0x7fa3fc1591f0 2023-12-19T14:51:53.549 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:53.541+0000 7fe1e7b0c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:51:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.586+0000 7fa400ce1700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 msgr2=0x7fa3e8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.586+0000 7fa400ce1700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 0x7fa3e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:51:53.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.598+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3fc16a800 con 0x7fa3fc1591f0 2023-12-19T14:51:53.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.598+0000 7fa3f0ff9700 1 -- 172.21.15.50:0/117155618 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fa3fc16a800 con 0x7fa3fc1591f0 2023-12-19T14:51:53.603 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:53.603 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:51:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 msgr2=0x7fa3e8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 0x7fa3e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 msgr2=0x7fa3fc16c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06af40 tx=0x7fa3f4017000 comp rx=0 tx=0).stop 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 shutdown_connections 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3fc158e20 0x7fa3fc16c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa3e8060700 0x7fa3e8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3fc1591f0 0x7fa3fc16c720 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 --2- 172.21.15.50:0/117155618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3fc0ce810 0x7fa3fc16d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 >> 172.21.15.50:0/117155618 conn(0x7fa3fc067bb0 msgr2=0x7fa3fc152840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 shutdown_connections 2023-12-19T14:51:53.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.602+0000 7fa401ce3700 1 -- 172.21.15.50:0/117155618 wait complete. 2023-12-19T14:51:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:51:53.609 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.606+0000 7f68bbbc3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:51:53.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:51:53.633 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:53.625+0000 7fe1e7b0c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:51:53.692 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.690+0000 7f68bbbc3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:51:53.791 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:53.790+0000 7f68bbbc3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:51:53.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.826+0000 7f58a84f0700 1 Processor -- start 2023-12-19T14:51:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.826+0000 7f58a84f0700 1 -- start start 2023-12-19T14:51:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 0x7f58a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a00c7190 con 0x7f58a00c15b0 2023-12-19T14:51:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a00c1b50 con 0x7f58a00aeba0 2023-12-19T14:51:53.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c1e30 con 0x7f58a015d320 2023-12-19T14:51:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a5a8b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f58a00c15b0 0x7f58a006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58846/0 (socket says 172.21.15.50:58846) 2023-12-19T14:51:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a5a8b700 1 -- 172.21.15.50:0/2381990663 learned_addr learned my addr 172.21.15.50:0/2381990663 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:53.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a628c700 1 --2- 172.21.15.50:0/2381990663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 0x7f58a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804381253 0 0) 0x7f58a00c7190 con 0x7f58a00c15b0 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f588c003570 con 0x7f58a00c15b0 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a628c700 1 -- 172.21.15.50:0/2381990663 >> v1:172.21.15.125:6789/0 conn(0x7f58a00aeba0 legacy=0x7f58a00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224027642 0 0) 0x7f58a00c1b50 con 0x7f58a00aeba0 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a628c700 1 -- 172.21.15.50:0/2381990663 >> v1:172.21.15.50:6789/0 conn(0x7f58a00c15b0 legacy=0x7f58a006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a628c700 1 -- 172.21.15.50:0/2381990663 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a00c6e20 con 0x7f58a015d320 2023-12-19T14:51:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a628c700 1 --2- 172.21.15.50:0/2381990663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 0x7f58a015d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e90 tx=0x7f589c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3e64e4adaa64bc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00e070 con 0x7f58a015d320 2023-12-19T14:51:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00bc30 con 0x7f58a015d320 2023-12-19T14:51:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a528a700 1 -- 172.21.15.50:0/2381990663 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c012430 con 0x7f58a015d320 2023-12-19T14:51:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 -- 172.21.15.50:0/2381990663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 msgr2=0x7f58a015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.830+0000 7f58a84f0700 1 --2- 172.21.15.50:0/2381990663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 0x7f58a015d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e90 tx=0x7f589c009780 comp rx=0 tx=0).stop 2023-12-19T14:51:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 -- 172.21.15.50:0/2381990663 shutdown_connections 2023-12-19T14:51:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 --2- 172.21.15.50:0/2381990663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a015d320 0x7f58a015d6f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 -- 172.21.15.50:0/2381990663 >> 172.21.15.50:0/2381990663 conn(0x7f58a00c8850 msgr2=0x7f58a00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:53.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 -- 172.21.15.50:0/2381990663 shutdown_connections 2023-12-19T14:51:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 -- 172.21.15.50:0/2381990663 wait complete. 2023-12-19T14:51:53.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 Processor -- start 2023-12-19T14:51:53.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 -- start start 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.834+0000 7f58a84f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a00c15b0 0x7f58a0170ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a015d320 0x7f58a0171400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0154520 con 0x7f58a015d320 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0171d60 con 0x7f58a00aeba0 2023-12-19T14:51:53.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00afe70 con 0x7f58a00c15b0 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a6a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a015d320 0x7f58a0171400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:58270/0 (socket says 172.21.15.50:58270) 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 -- 172.21.15.50:0/5490883 learned_addr learned my addr 172.21.15.50:0/5490883 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a5a8b700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a00c15b0 0x7f58a0170ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a00c15b0 msgr2=0x7f58a0170ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a00c15b0 0x7f58a0170ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a015d320 msgr2=0x7f58a0171400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a015d320 0x7f58a0171400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 -- 172.21.15.50:0/5490883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c009a20 con 0x7f58a00aeba0 2023-12-19T14:51:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a628c700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f589c0099f0 tx=0x7f589c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c012e60 con 0x7f58a00aeba0 2023-12-19T14:51:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c026480 con 0x7f58a00aeba0 2023-12-19T14:51:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a0171a80 con 0x7f58a00aeba0 2023-12-19T14:51:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c026840 con 0x7f58a00aeba0 2023-12-19T14:51:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a00afb00 con 0x7f58a00aeba0 2023-12-19T14:51:53.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.838+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a0071120 con 0x7f58a00aeba0 2023-12-19T14:51:53.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.842+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f589c01c400 con 0x7f58a00aeba0 2023-12-19T14:51:53.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.842+0000 7f58967fc700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 0x7f5884062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:53.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.842+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f589c07efc0 con 0x7f58a00aeba0 2023-12-19T14:51:53.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.842+0000 7f58a5a8b700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 msgr2=0x7f5884062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:53.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.842+0000 7f58a5a8b700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 0x7f5884062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:53.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:53.854+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f589c04ec30 con 0x7f58a00aeba0 2023-12-19T14:51:54.011 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:54.010+0000 7f68bbbc3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:51:54.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.046+0000 7f58a5a8b700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 msgr2=0x7f5884062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:54.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.046+0000 7f58a5a8b700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 0x7f5884062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:51:54.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.066+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58a00642a0 con 0x7f58a00aeba0 2023-12-19T14:51:54.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.066+0000 7f58967fc700 1 -- 172.21.15.50:0/5490883 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f589c052250 con 0x7f58a00aeba0 2023-12-19T14:51:54.069 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:54.069 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:40:15.420564+0000","modified":"2023-12-19T14:40:19.019982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.50:6835/906596857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":906596857},{"type":"v1","addr":"172.21.15.50:6835","nonce":906596857}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 msgr2=0x7f5884062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 0x7f5884062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 msgr2=0x7f58a01574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f589c0099f0 tx=0x7f589c019000 comp rx=0 tx=0).stop 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 shutdown_connections 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5884060700 0x7f5884062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58a015d320 0x7f58a0171400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58a00c15b0 0x7f58a0170ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 --2- 172.21.15.50:0/5490883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58a00aeba0 0x7f58a01574d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 >> 172.21.15.50:0/5490883 conn(0x7f58a00c8850 msgr2=0x7f58a006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:54.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 shutdown_connections 2023-12-19T14:51:54.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.070+0000 7f58a84f0700 1 -- 172.21.15.50:0/5490883 wait complete. 2023-12-19T14:51:54.074 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 17 2023-12-19T14:51:54.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:51:54.098 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:54.094+0000 7f68bbbc3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:51:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 Processor -- start 2023-12-19T14:51:54.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 -- start start 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2941591f0 0x7fe29406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29406b730 con 0x7fe2940ce810 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29406ab20 con 0x7fe294158e20 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe29968e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29406ae00 con 0x7fe2941591f0 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe298e8d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe2940ce810 0x7fe2940ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58858/0 (socket says 172.21.15.50:58858) 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe298e8d700 1 -- 172.21.15.50:0/1697775051 learned_addr learned my addr 172.21.15.50:0/1697775051 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe2937fe700 1 --2- 172.21.15.50:0/1697775051 >> v2:172.21.15.50:3300/0 conn(0x7fe2941591f0 0x7fe29406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe292ffd700 1 -- 172.21.15.50:0/1697775051 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146758722 0 0) 0x7fe29406b730 con 0x7fe2940ce810 2023-12-19T14:51:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe292ffd700 1 -- 172.21.15.50:0/1697775051 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe27c003570 con 0x7fe2940ce810 2023-12-19T14:51:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe2937fe700 1 -- 172.21.15.50:0/1697775051 >> v1:172.21.15.125:6789/0 conn(0x7fe294158e20 legacy=0x7fe2940ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe2937fe700 1 -- 172.21.15.50:0/1697775051 >> v1:172.21.15.50:6789/0 conn(0x7fe2940ce810 legacy=0x7fe2940ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.258+0000 7fe2937fe700 1 -- 172.21.15.50:0/1697775051 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29406b0e0 con 0x7fe2941591f0 2023-12-19T14:51:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe2937fe700 1 --2- 172.21.15.50:0/1697775051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2941591f0 0x7fe29406a540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe2880098e0 tx=0x7fe2880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254ec2fc968c3c0e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe292ffd700 1 -- 172.21.15.50:0/1697775051 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28800b640 con 0x7fe2941591f0 2023-12-19T14:51:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe292ffd700 1 -- 172.21.15.50:0/1697775051 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28800b7a0 con 0x7fe2941591f0 2023-12-19T14:51:54.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe292ffd700 1 -- 172.21.15.50:0/1697775051 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28800bb40 con 0x7fe2941591f0 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 -- 172.21.15.50:0/1697775051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2941591f0 msgr2=0x7fe29406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 --2- 172.21.15.50:0/1697775051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2941591f0 0x7fe29406a540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe2880098e0 tx=0x7fe2880091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 -- 172.21.15.50:0/1697775051 shutdown_connections 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 --2- 172.21.15.50:0/1697775051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2941591f0 0x7fe29406a540 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 -- 172.21.15.50:0/1697775051 >> 172.21.15.50:0/1697775051 conn(0x7fe294067bb0 msgr2=0x7fe2940cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 -- 172.21.15.50:0/1697775051 shutdown_connections 2023-12-19T14:51:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 -- 172.21.15.50:0/1697775051 wait complete. 2023-12-19T14:51:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.262+0000 7fe29968e700 1 Processor -- start 2023-12-19T14:51:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- start start 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 0x7fe294152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2941591f0 0x7fe294155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29416a450 con 0x7fe294158e20 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe294153470 con 0x7fe2941591f0 2023-12-19T14:51:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe294169580 con 0x7fe2940ce810 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56502/0 (socket says 172.21.15.50:56502) 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 -- 172.21.15.50:0/1963515869 learned_addr learned my addr 172.21.15.50:0/1963515869 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe293fff700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 0x7fe294152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe298e8d700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2941591f0 0x7fe294155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 -- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 msgr2=0x7fe294152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 0x7fe294152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 -- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2941591f0 msgr2=0x7fe294155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2941591f0 0x7fe294155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 -- 172.21.15.50:0/1963515869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288009420 con 0x7fe294158e20 2023-12-19T14:51:54.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe293fff700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 0x7fe294152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe2937fe700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe288009f80 tx=0x7fe288017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28800bb20 con 0x7fe294158e20 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe288011420 con 0x7fe294158e20 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe294169840 con 0x7fe294158e20 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2880117a0 con 0x7fe294158e20 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.266+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe294169e00 con 0x7fe294158e20 2023-12-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.270+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29404be40 con 0x7fe294158e20 2023-12-19T14:51:54.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.270+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fe288011b10 con 0x7fe294158e20 2023-12-19T14:51:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.274+0000 7fe27bfff700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 0x7fe280062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.274+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fe28807e650 con 0x7fe294158e20 2023-12-19T14:51:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.274+0000 7fe293fff700 1 -- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 msgr2=0x7fe280062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:51:54.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.274+0000 7fe293fff700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 0x7fe280062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:51:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.278+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe28804e540 con 0x7fe294158e20 2023-12-19T14:51:54.362 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.353+0000 7fe1e7b0c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:51:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.378+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 14) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fe28804e910 con 0x7fe294158e20 2023-12-19T14:51:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.378+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 msgr2=0x7fe280062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:51:54.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.378+0000 7fe27bfff700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 0x7fe280062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.458+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe29406b3c0 con 0x7fe294158e20 2023-12-19T14:51:54.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.458+0000 7fe27bfff700 1 -- 172.21.15.50:0/1963515869 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17017 (secure 0 0 0) 0x7fe288080400 con 0x7fe294158e20 2023-12-19T14:51:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:51:54.462 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":56,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:51:54.368828+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:40:13.626675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T14:40:14.423960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/2960058572":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/902571518":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3904162499":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3771030607":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3779418184":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3338034462":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/1210582553":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6832/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6833/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/4107501911":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/1437130881":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/751498889":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/3449382498":"2023-12-20T14:40:35.182753+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":15,"length":1},{"begin":18,"length":2},{"begin":22,"length":1},{"begin":28,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":41,"length":1},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":60,"length":2},{"begin":64,"length":3},{"begin":78,"length":3},{"begin":85,"length":3},{"begin":92,"length":1},{"begin":100,"length":2},{"begin":104,"length":1},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":190,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 msgr2=0x7fe2941556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe288009f80 tx=0x7fe288017000 comp rx=0 tx=0).stop 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 shutdown_connections 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2941591f0 0x7fe294155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe280060700 0x7fe280062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe294158e20 0x7fe2941556f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 --2- 172.21.15.50:0/1963515869 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2940ce810 0x7fe294152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 >> 172.21.15.50:0/1963515869 conn(0x7fe294067bb0 msgr2=0x7fe2940b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 shutdown_connections 2023-12-19T14:51:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.462+0000 7fe29968e700 1 -- 172.21.15.50:0/1963515869 wait complete. 2023-12-19T14:51:54.476 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b11f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.476 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b11f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.476 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b11f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.476 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b11f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.476 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b11f4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b41fa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b41fa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b41fa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b41fa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:51:54.469+0000 7fe1b41fa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:51:54.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:51:54.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 Processor -- start 2023-12-19T14:51:54.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 -- start start 2023-12-19T14:51:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 0x7fe3bc0cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe3bc0cc360 0x7fe3bc0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc155ea0 con 0x7fe3bc1556c0 2023-12-19T14:51:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cc7c0 con 0x7fe3bc0cc360 2023-12-19T14:51:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.638+0000 7fe3bb59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cb640 con 0x7fe3bc155a90 2023-12-19T14:51:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3ba59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe3bc1556c0 0x7fe3bc0c2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58868/0 (socket says 172.21.15.50:58868) 2023-12-19T14:51:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3ba59c700 1 -- 172.21.15.50:0/2861353162 learned_addr learned my addr 172.21.15.50:0/2861353162 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 --2- 172.21.15.50:0/2861353162 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 0x7fe3bc0cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bad9d700 1 --2- 172.21.15.50:0/2861353162 >> v2:172.21.15.125:3300/0 conn(0x7fe3bc0cc360 0x7fe3bc0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b959a700 1 -- 172.21.15.50:0/2861353162 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356610235 0 0) 0x7fe3bc155ea0 con 0x7fe3bc1556c0 2023-12-19T14:51:54.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b959a700 1 -- 172.21.15.50:0/2861353162 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4003570 con 0x7fe3bc1556c0 2023-12-19T14:51:54.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 -- 172.21.15.50:0/2861353162 >> v2:172.21.15.125:3300/0 conn(0x7fe3bc0cc360 msgr2=0x7fe3bc0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:54.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 --2- 172.21.15.50:0/2861353162 >> v2:172.21.15.125:3300/0 conn(0x7fe3bc0cc360 0x7fe3bc0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 -- 172.21.15.50:0/2861353162 >> v1:172.21.15.50:6789/0 conn(0x7fe3bc1556c0 legacy=0x7fe3bc0c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 -- 172.21.15.50:0/2861353162 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0c2930 con 0x7fe3bc155a90 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b9d9b700 1 --2- 172.21.15.50:0/2861353162 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 0x7fe3bc0cbe20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00098e0 tx=0x7fe3b00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd8b7df0ebc82e67 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b959a700 1 -- 172.21.15.50:0/2861353162 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0011630 con 0x7fe3bc155a90 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b959a700 1 -- 172.21.15.50:0/2861353162 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0011790 con 0x7fe3bc155a90 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3b959a700 1 -- 172.21.15.50:0/2861353162 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0011b30 con 0x7fe3bc155a90 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 -- 172.21.15.50:0/2861353162 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 msgr2=0x7fe3bc0cbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/2861353162 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 0x7fe3bc0cbe20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00098e0 tx=0x7fe3b00091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 -- 172.21.15.50:0/2861353162 shutdown_connections 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/2861353162 >> v2:172.21.15.125:3300/0 conn(0x7fe3bc0cc360 0x7fe3bc0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/2861353162 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc155a90 0x7fe3bc0cbe20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.642+0000 7fe3bb59e700 1 -- 172.21.15.50:0/2861353162 >> 172.21.15.50:0/2861353162 conn(0x7fe3bc06b670 msgr2=0x7fe3bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.646+0000 7fe3bb59e700 1 -- 172.21.15.50:0/2861353162 shutdown_connections 2023-12-19T14:51:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.646+0000 7fe3bb59e700 1 -- 172.21.15.50:0/2861353162 wait complete. 2023-12-19T14:51:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.646+0000 7fe3bb59e700 1 Processor -- start 2023-12-19T14:51:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.646+0000 7fe3bb59e700 1 -- start start 2023-12-19T14:51:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 0x7fe3bc150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3bc1556c0 0x7fe3bc0c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c5200 con 0x7fe3bc155a90 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c45f0 con 0x7fe3bc1556c0 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c48d0 con 0x7fe3bc0cc360 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56516/0 (socket says 172.21.15.50:56516) 2023-12-19T14:51:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 -- 172.21.15.50:0/710245611 learned_addr learned my addr 172.21.15.50:0/710245611 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:54.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3b9d9b700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3bc1556c0 0x7fe3bc0c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3ba59c700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 0x7fe3bc150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 -- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 msgr2=0x7fe3bc150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 0x7fe3bc150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 -- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3bc1556c0 msgr2=0x7fe3bc0c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3bc1556c0 0x7fe3bc0c3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 -- 172.21.15.50:0/710245611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b0009420 con 0x7fe3bc155a90 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3ba59c700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 0x7fe3bc150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bad9d700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009df0 tx=0x7fe3b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400cb50 con 0x7fe3bc155a90 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b400f040 con 0x7fe3bc155a90 2023-12-19T14:51:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc0c4bb0 con 0x7fe3bc155a90 2023-12-19T14:51:54.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4014480 con 0x7fe3bc155a90 2023-12-19T14:51:54.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.650+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0c5cf0 con 0x7fe3bc155a90 2023-12-19T14:51:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.654+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc04be40 con 0x7fe3bc155a90 2023-12-19T14:51:54.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.654+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 14) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fe3b4014830 con 0x7fe3bc155a90 2023-12-19T14:51:54.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.658+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fe3b407d0b0 con 0x7fe3bc155a90 2023-12-19T14:51:54.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.662+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe3b404d550 con 0x7fe3bc155a90 2023-12-19T14:51:54.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.866+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe3bc0c2b40 con 0x7fe3bc155a90 2023-12-19T14:51:54.871 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:51:54.866+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:51:54.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.878+0000 7fe3aa7fc700 1 -- 172.21.15.50:0/710245611 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe3b4018090 con 0x7fe3bc155a90 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 msgr2=0x7fe3bc0670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009df0 tx=0x7fe3b400b040 comp rx=0 tx=0).stop 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 shutdown_connections 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3bc1556c0 0x7fe3bc0c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3bc155a90 0x7fe3bc0670e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 --2- 172.21.15.50:0/710245611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3bc0cc360 0x7fe3bc150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 >> 172.21.15.50:0/710245611 conn(0x7fe3bc06b670 msgr2=0x7fe3bc0c1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 shutdown_connections 2023-12-19T14:51:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:54.882+0000 7fe3bb59e700 1 -- 172.21.15.50:0/710245611 wait complete. 2023-12-19T14:51:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:51:54.898 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:51:54.894+0000 7f68bbbc3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:51:54.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:51:54.947 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:51:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.062+0000 7fafc8d63700 1 Processor -- start 2023-12-19T14:51:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.062+0000 7fafc8d63700 1 -- start start 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc8d63700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafc4141e70 0x7fafc40c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc8d63700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc4159b80 con 0x7fafc40c7a80 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc8d63700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc40c8220 con 0x7fafc40c53f0 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc8d63700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc40c7f40 con 0x7fafc4141e70 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafbbfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fafc40c7a80 0x7fafc40c4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58872/0 (socket says 172.21.15.50:58872) 2023-12-19T14:51:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafbbfff700 1 -- 172.21.15.50:0/4043706445 learned_addr learned my addr 172.21.15.50:0/4043706445 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:55.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc37fe700 1 --2- 172.21.15.50:0/4043706445 >> v2:172.21.15.125:3300/0 conn(0x7fafc4141e70 0x7fafc40c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043595669 0 0) 0x7fafc4159b80 con 0x7fafc40c7a80 2023-12-19T14:51:55.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb0003570 con 0x7fafc40c7a80 2023-12-19T14:51:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045724476 0 0) 0x7fafc40c8220 con 0x7fafc40c53f0 2023-12-19T14:51:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4159b80 con 0x7fafc40c53f0 2023-12-19T14:51:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343198129 0 0) 0x7fafb0003570 con 0x7fafc40c7a80 2023-12-19T14:51:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 >> v2:172.21.15.125:3300/0 conn(0x7fafc4141e70 msgr2=0x7fafc40c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 --2- 172.21.15.50:0/4043706445 >> v2:172.21.15.125:3300/0 conn(0x7fafc4141e70 0x7fafc40c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 >> v1:172.21.15.125:6789/0 conn(0x7fafc40c53f0 legacy=0x7fafc415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4159530 con 0x7fafc40c7a80 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fafac001470 con 0x7fafc40c7a80 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 >> v1:172.21.15.50:6789/0 conn(0x7fafc40c7a80 legacy=0x7fafc40c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 0x7fafb0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0004810 0x7fafb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafb00051c0 0x7fafb000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c7f40 con 0x7fafb0003bf0 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb000c160 con 0x7fafb0004810 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb000c320 con 0x7fafb00051c0 2023-12-19T14:51:55.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc37fe700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 0x7fafb0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafbbfff700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0004810 0x7fafb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.066+0000 7fafc3fff700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafb00051c0 0x7fafb000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 -- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafb00051c0 msgr2=0x7fafb000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafb00051c0 0x7fafb000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 -- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0004810 msgr2=0x7fafb0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0004810 0x7fafb0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 -- 172.21.15.50:0/4043706445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0003570 con 0x7fafb0003bf0 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafbbfff700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0004810 0x7fafb0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc3fff700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafb00051c0 0x7fafb000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc3fff700 1 -- 172.21.15.50:0/4043706445 reap_dead start 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc37fe700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 0x7fafb0003fe0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006f00 tx=0x7fafb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170ca4d8a202c960 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400e710 con 0x7fafb0003bf0 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb400e870 con 0x7fafb0003bf0 2023-12-19T14:51:55.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc2ffd700 1 -- 172.21.15.50:0/4043706445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400ec30 con 0x7fafb0003bf0 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 -- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 msgr2=0x7fafb0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 0x7fafb0003fe0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006f00 tx=0x7fafb40091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 -- 172.21.15.50:0/4043706445 shutdown_connections 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 --2- 172.21.15.50:0/4043706445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafb0003bf0 0x7fafb0003fe0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 -- 172.21.15.50:0/4043706445 >> 172.21.15.50:0/4043706445 conn(0x7fafc4067b50 msgr2=0x7fafc40cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 -- 172.21.15.50:0/4043706445 shutdown_connections 2023-12-19T14:51:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.070+0000 7fafc8d63700 1 -- 172.21.15.50:0/4043706445 wait complete. 2023-12-19T14:51:55.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 Processor -- start 2023-12-19T14:51:55.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 -- start start 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0003bf0 0x7fafc4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafc40af310 0x7fafc40af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc415b490 con 0x7fafc415aae0 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4169140 con 0x7fafb0003bf0 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.074+0000 7fafc8d63700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40afe20 con 0x7fafc40af310 2023-12-19T14:51:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56556/0 (socket says 172.21.15.50:56556) 2023-12-19T14:51:55.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 -- 172.21.15.50:0/743664786 learned_addr learned my addr 172.21.15.50:0/743664786 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafbbfff700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafc40af310 0x7fafc40af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc37fe700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0003bf0 0x7fafc4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 -- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafc40af310 msgr2=0x7fafc40af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafc40af310 0x7fafc40af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 -- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0003bf0 msgr2=0x7fafc4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0003bf0 0x7fafc4158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 -- 172.21.15.50:0/743664786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40b0100 con 0x7fafc415aae0 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc3fff700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fafc40cca30 tx=0x7fafbc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:55.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc003300 con 0x7fafc415aae0 2023-12-19T14:51:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc003460 con 0x7fafc415aae0 2023-12-19T14:51:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc006df0 con 0x7fafc415aae0 2023-12-19T14:51:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40b03e0 con 0x7fafc415aae0 2023-12-19T14:51:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc4159110 con 0x7fafc415aae0 2023-12-19T14:51:55.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.078+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc41692d0 con 0x7fafc415aae0 2023-12-19T14:51:55.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.082+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 14) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fafbc0018f0 con 0x7fafc415aae0 2023-12-19T14:51:55.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.082+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7079+0+0 (secure 0 0 0) 0x7fafbc07bde0 con 0x7fafc415aae0 2023-12-19T14:51:55.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.090+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fafbc04c1c0 con 0x7fafc415aae0 2023-12-19T14:51:55.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.298+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fafc404be40 con 0x7fafc415aae0 2023-12-19T14:51:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.386+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 15) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fafbc0550d0 con 0x7fafc415aae0 2023-12-19T14:51:55.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.386+0000 7fafbb7fe700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 0x7fafa4068c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:55.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.386+0000 7fafc37fe700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 0x7fafa4068c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.390+0000 7fafc37fe700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 0x7fafa4068c10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafb40091e0 tx=0x7fafb401a000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:55.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.894+0000 7fafbb7fe700 1 -- 172.21.15.50:0/743664786 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fafbc077d30 con 0x7fafc415aae0 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 msgr2=0x7fafa4068c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 0x7fafa4068c10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafb40091e0 tx=0x7fafb401a000 comp rx=0 tx=0).stop 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 msgr2=0x7fafc415af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fafc40cca30 tx=0x7fafbc009040 comp rx=0 tx=0).stop 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 shutdown_connections 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fafa4066830 0x7fafa4068c10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafc415aae0 0x7fafc415af50 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafc40af310 0x7fafc40af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 --2- 172.21.15.50:0/743664786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafb0003bf0 0x7fafc4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 >> 172.21.15.50:0/743664786 conn(0x7fafc4067b50 msgr2=0x7fafc413b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 shutdown_connections 2023-12-19T14:51:55.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:55.898+0000 7fafc8d63700 1 -- 172.21.15.50:0/743664786 wait complete. 2023-12-19T14:51:55.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:51:56.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.138+0000 7fe27b22e700 1 Processor -- start 2023-12-19T14:51:56.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.138+0000 7fe27b22e700 1 -- start start 2023-12-19T14:51:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe27b22e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2740cb130 0x7fe2740cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe27b22e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe27b22e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740b0ac0 con 0x7fe27406a6a0 2023-12-19T14:51:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe27b22e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe27406b030 con 0x7fe2740cb130 2023-12-19T14:51:56.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe27b22e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27406b210 con 0x7fe27415c930 2023-12-19T14:51:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe278fca700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2740cb130 0x7fe2740cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58452/0 (socket says 172.21.15.50:58452) 2023-12-19T14:51:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 -- 172.21.15.50:0/1158241180 learned_addr learned my addr 172.21.15.50:0/1158241180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:56.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe273fff700 1 --1- 172.21.15.50:0/1158241180 >> v1:172.21.15.125:6789/0 conn(0x7fe27406a6a0 0x7fe27415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50842/0 (socket says 172.21.15.50:50842) 2023-12-19T14:51:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 -- 172.21.15.50:0/1158241180 >> v2:172.21.15.125:3300/0 conn(0x7fe2740cb130 msgr2=0x7fe2740cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 --2- 172.21.15.50:0/1158241180 >> v2:172.21.15.125:3300/0 conn(0x7fe2740cb130 0x7fe2740cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 -- 172.21.15.50:0/1158241180 >> v1:172.21.15.125:6789/0 conn(0x7fe27406a6a0 legacy=0x7fe27415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2797cb700 1 -- 172.21.15.50:0/1158241180 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2740ca810 con 0x7fe27415c930 2023-12-19T14:51:56.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.142+0000 7fe2737fe700 1 -- 172.21.15.50:0/1158241180 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797871316 0 0) 0x7fe2740b0ac0 con 0x7fe27406a6a0 2023-12-19T14:51:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe2797cb700 1 --2- 172.21.15.50:0/1158241180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009f20 tx=0x7fe26c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d782585bfee80c24 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe2737fe700 1 -- 172.21.15.50:0/1158241180 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c00e070 con 0x7fe27415c930 2023-12-19T14:51:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe2737fe700 1 -- 172.21.15.50:0/1158241180 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26c00b9d0 con 0x7fe27415c930 2023-12-19T14:51:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe2737fe700 1 -- 172.21.15.50:0/1158241180 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c0092e0 con 0x7fe27415c930 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 -- 172.21.15.50:0/1158241180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 msgr2=0x7fe27406aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 --2- 172.21.15.50:0/1158241180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009f20 tx=0x7fe26c009840 comp rx=0 tx=0).stop 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 -- 172.21.15.50:0/1158241180 shutdown_connections 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 --2- 172.21.15.50:0/1158241180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27415c930 0x7fe27406aaf0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 --2- 172.21.15.50:0/1158241180 >> v2:172.21.15.125:3300/0 conn(0x7fe2740cb130 0x7fe2740cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 -- 172.21.15.50:0/1158241180 >> 172.21.15.50:0/1158241180 conn(0x7fe274067bb0 msgr2=0x7fe2740cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 -- 172.21.15.50:0/1158241180 shutdown_connections 2023-12-19T14:51:56.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.146+0000 7fe27b22e700 1 -- 172.21.15.50:0/1158241180 wait complete. 2023-12-19T14:51:56.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 Processor -- start 2023-12-19T14:51:56.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 -- start start 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2740cb130 0x7fe274147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe27415c930 0x7fe2741531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740b4cb0 con 0x7fe2740cb130 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740b40a0 con 0x7fe27415c930 2023-12-19T14:51:56.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.150+0000 7fe27b22e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740b4380 con 0x7fe27406a6a0 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe273fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2740cb130 0x7fe274147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58462/0 (socket says 172.21.15.50:58462) 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 -- 172.21.15.50:0/4005425115 learned_addr learned my addr 172.21.15.50:0/4005425115 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe2797cb700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe27415c930 0x7fe2741531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 -- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe27415c930 msgr2=0x7fe2741531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe27415c930 0x7fe2741531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 -- 172.21.15.50:0/4005425115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2740cb130 msgr2=0x7fe274147d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2740cb130 0x7fe274147d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 -- 172.21.15.50:0/4005425115 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c009ae0 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe278fca700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe26800ec80 tx=0x7fe2680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26800c850 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26800c9b0 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2740b4660 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26800cd50 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe274153720 con 0x7fe27406a6a0 2023-12-19T14:51:56.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.154+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe274071120 con 0x7fe27406a6a0 2023-12-19T14:51:56.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.158+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 15) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fe268013680 con 0x7fe27406a6a0 2023-12-19T14:51:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.158+0000 7fe270ff9700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 0x7fe258062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:56.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.158+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7079+0+0 (secure 0 0 0) 0x7fe26807cc80 con 0x7fe27406a6a0 2023-12-19T14:51:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.162+0000 7fe273fff700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 0x7fe258062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:56.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.162+0000 7fe273fff700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 0x7fe258062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe27415bcf0 tx=0x7fe260007d50 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:56.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.166+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe26804cef0 con 0x7fe27406a6a0 2023-12-19T14:51:56.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.338+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe2740cbe90 con 0x7fe27406a6a0 2023-12-19T14:51:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.926+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe26804d2c0 con 0x7fe27406a6a0 2023-12-19T14:51:56.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.926+0000 7fe270ff9700 1 -- 172.21.15.50:0/4005425115 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe268049b60 con 0x7fe27406a6a0 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 msgr2=0x7fe258062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 0x7fe258062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe27415bcf0 tx=0x7fe260007d50 comp rx=0 tx=0).stop 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 msgr2=0x7fe2740b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe26800ec80 tx=0x7fe2680091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 shutdown_connections 2023-12-19T14:51:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe27415c930 0x7fe2741531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2580603d0 0x7fe258062880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2740cb130 0x7fe274147d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 --2- 172.21.15.50:0/4005425115 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe27406a6a0 0x7fe2740b5140 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 >> 172.21.15.50:0/4005425115 conn(0x7fe274067bb0 msgr2=0x7fe2740b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 shutdown_connections 2023-12-19T14:51:56.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:56.930+0000 7fe27b22e700 1 -- 172.21.15.50:0/4005425115 wait complete. 2023-12-19T14:51:56.934 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:51:56.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:51:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.130+0000 7f7aaf48e700 1 Processor -- start 2023-12-19T14:51:57.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.130+0000 7f7aaf48e700 1 -- start start 2023-12-19T14:51:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.130+0000 7f7aaf48e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7aa80af190 0x7f7aa806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaf48e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80cb3f0 con 0x7f7aa80cac50 2023-12-19T14:51:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaf48e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa806d220 con 0x7f7aa80aedc0 2023-12-19T14:51:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaf48e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa80cc140 con 0x7f7aa80af190 2023-12-19T14:51:57.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7aa80cac50 0x7f7aa80af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38344/0 (socket says 172.21.15.50:38344) 2023-12-19T14:51:57.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3293218234 learned_addr learned my addr 172.21.15.50:0/3293218234 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aadc8b700 1 --2- 172.21.15.50:0/3293218234 >> v2:172.21.15.125:3300/0 conn(0x7f7aa80af190 0x7f7aa806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279081593 0 0) 0x7f7aa80cb3f0 con 0x7f7aa80cac50 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a94003570 con 0x7f7aa80cac50 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310850513 0 0) 0x7f7a94003570 con 0x7f7aa80cac50 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 >> v2:172.21.15.125:3300/0 conn(0x7f7aa80af190 msgr2=0x7f7aa806af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 --2- 172.21.15.50:0/3293218234 >> v2:172.21.15.125:3300/0 conn(0x7f7aa80af190 0x7f7aa806af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 >> v1:172.21.15.125:6789/0 conn(0x7f7aa80aedc0 legacy=0x7f7aa806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa806c8f0 con 0x7f7aa80cac50 2023-12-19T14:51:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f7aa00017f0 con 0x7f7aa80cac50 2023-12-19T14:51:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 >> v1:172.21.15.50:6789/0 conn(0x7f7aa80cac50 legacy=0x7f7aa80af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a94003a60 0x7f7a94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a94004810 0x7f7a94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 0x7f7a9400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80cc140 con 0x7f7a940051c0 2023-12-19T14:51:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9400c160 con 0x7f7a94004810 2023-12-19T14:51:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9400c320 con 0x7f7a94003a60 2023-12-19T14:51:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 0x7f7a9400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aadc8b700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a94004810 0x7f7a94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aae48c700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a94003a60 0x7f7a94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a94003a60 msgr2=0x7f7a94003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a94003a60 0x7f7a94003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a94004810 msgr2=0x7f7a94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a94004810 0x7f7a94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3293218234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94003570 con 0x7f7a940051c0 2023-12-19T14:51:57.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aadc8b700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a94004810 0x7f7a94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3293218234 reap_dead start 2023-12-19T14:51:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.134+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 0x7f7a9400ba60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000a930 tx=0x7f7aa00021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2743aa2ef9dd4e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00070d0 con 0x7f7a940051c0 2023-12-19T14:51:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0007230 con 0x7f7a940051c0 2023-12-19T14:51:57.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aad48a700 1 -- 172.21.15.50:0/3293218234 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00075d0 con 0x7f7a940051c0 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 msgr2=0x7f7a9400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 0x7f7a9400ba60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000a930 tx=0x7f7aa00021d0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3293218234 shutdown_connections 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3293218234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a940051c0 0x7f7a9400ba60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3293218234 >> 172.21.15.50:0/3293218234 conn(0x7f7aa80cc760 msgr2=0x7f7aa80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3293218234 shutdown_connections 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3293218234 wait complete. 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 Processor -- start 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.138+0000 7f7aaf48e700 1 -- start start 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 0x7f7aa80b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aa814c350 0x7f7aa80c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80af610 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80b19a0 con 0x7f7aa814c350 2023-12-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80b4010 con 0x7f7a940051c0 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46500/0 (socket says 172.21.15.50:46500) 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3635398752 learned_addr learned my addr 172.21.15.50:0/3635398752 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aadc8b700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aa814c350 0x7f7aa80c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aae48c700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 0x7f7aa80b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 msgr2=0x7f7aa80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 0x7f7aa80b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aa814c350 msgr2=0x7f7aa80c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aa814c350 0x7f7aa80c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 -- 172.21.15.50:0/3635398752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80c4030 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aae48c700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 0x7f7aa80b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaec8d700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7aa806f890 tx=0x7f7aa0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00070d0 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0007940 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00073a0 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa80b3160 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.142+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa80b3720 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.146+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa804be40 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.146+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7aa001e640 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.146+0000 7f7a9e7fc700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 0x7f7a8c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:57.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.146+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6641+0+0 (secure 0 0 0) 0x7f7aa007f410 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.146+0000 7f7aae48c700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 0x7f7a8c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:57.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.150+0000 7f7aae48c700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 0x7f7a8c062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80b1cd0 tx=0x7f7aa4009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:57.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.154+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7aa004f540 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.326+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7aa80b4150 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.862+0000 7f7a9e7fc700 1 -- 172.21.15.50:0/3635398752 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7aa0052b60 con 0x7f7aa80b0fc0 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 msgr2=0x7f7a8c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 0x7f7a8c062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80b1cd0 tx=0x7f7aa4009040 comp rx=0 tx=0).stop 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 msgr2=0x7f7aa80b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7aa806f890 tx=0x7f7aa0016000 comp rx=0 tx=0).stop 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 shutdown_connections 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7aa814c350 0x7f7aa80c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7a8c0606a0 0x7f7a8c062b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7aa80b0fc0 0x7f7aa80b1460 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 --2- 172.21.15.50:0/3635398752 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7a940051c0 0x7f7aa80b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 >> 172.21.15.50:0/3635398752 conn(0x7f7aa80cc760 msgr2=0x7f7aa815c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 shutdown_connections 2023-12-19T14:51:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:57.866+0000 7f7aaf48e700 1 -- 172.21.15.50:0/3635398752 wait complete. 2023-12-19T14:51:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:51:57.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:51:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 Processor -- start 2023-12-19T14:51:58.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 -- start start 2023-12-19T14:51:58.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff24406aeb0 0x7ff244159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2441595a0 0x7ff244159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24406c850 con 0x7ff244158c90 2023-12-19T14:51:58.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2440cb650 con 0x7ff24406aeb0 2023-12-19T14:51:58.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.058+0000 7ff249416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24406b470 con 0x7ff2441595a0 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff244158c90 0x7ff2440caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38352/0 (socket says 172.21.15.50:38352) 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 -- 172.21.15.50:0/2503623678 learned_addr learned my addr 172.21.15.50:0/2503623678 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff23bfff700 1 --2- 172.21.15.50:0/2503623678 >> v2:172.21.15.125:3300/0 conn(0x7ff24406aeb0 0x7ff244159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff248c15700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2441595a0 0x7ff244159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678665408 0 0) 0x7ff24406c850 con 0x7ff244158c90 2023-12-19T14:51:58.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff230003570 con 0x7ff244158c90 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253465125 0 0) 0x7ff230003570 con 0x7ff244158c90 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2441595a0 msgr2=0x7ff244159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2441595a0 0x7ff244159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 >> v2:172.21.15.125:3300/0 conn(0x7ff24406aeb0 msgr2=0x7ff244159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 --2- 172.21.15.50:0/2503623678 >> v2:172.21.15.125:3300/0 conn(0x7ff24406aeb0 0x7ff244159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24406b750 con 0x7ff244158c90 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7ff234001470 con 0x7ff244158c90 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 >> v1:172.21.15.50:6789/0 conn(0x7ff244158c90 legacy=0x7ff2440caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff230003a60 0x7ff230003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff230004810 0x7ff230004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2300051c0 0x7ff23000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2440cb650 con 0x7ff230004810 2023-12-19T14:51:58.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24406b470 con 0x7ff230003a60 2023-12-19T14:51:58.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23000bfa0 con 0x7ff2300051c0 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff230003a60 0x7ff230003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff23bfff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff230004810 0x7ff230004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff248c15700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2300051c0 0x7ff23000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 -- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2300051c0 msgr2=0x7ff23000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff2300051c0 0x7ff23000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 -- 172.21.15.50:0/2503623678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff230004810 msgr2=0x7ff230004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff230004810 0x7ff230004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff243fff700 1 -- 172.21.15.50:0/2503623678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff230003570 con 0x7ff230003a60 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.062+0000 7ff23bfff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff230004810 0x7ff230004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:51:58.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff243fff700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff230003a60 0x7ff230003e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c3c0 tx=0x7ff234001e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420a6cb6eca61a97 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23400cc90 con 0x7ff230003a60 2023-12-19T14:51:58.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23400ce30 con 0x7ff230003a60 2023-12-19T14:51:58.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff2437fe700 1 -- 172.21.15.50:0/2503623678 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234005b70 con 0x7ff230003a60 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 -- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff230003a60 msgr2=0x7ff230003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 --2- 172.21.15.50:0/2503623678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff230003a60 0x7ff230003e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c3c0 tx=0x7ff234001e90 comp rx=0 tx=0).stop 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff248c15700 1 -- 172.21.15.50:0/2503623678 reap_dead start 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 -- 172.21.15.50:0/2503623678 shutdown_connections 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 -- 172.21.15.50:0/2503623678 >> 172.21.15.50:0/2503623678 conn(0x7ff244067bb0 msgr2=0x7ff2440cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 -- 172.21.15.50:0/2503623678 shutdown_connections 2023-12-19T14:51:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 -- 172.21.15.50:0/2503623678 wait complete. 2023-12-19T14:51:58.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.066+0000 7ff249416700 1 Processor -- start 2023-12-19T14:51:58.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- start start 2023-12-19T14:51:58.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff230003a60 0x7ff2440c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 0x7ff2440b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2441511d0 con 0x7ff2441576f0 2023-12-19T14:51:58.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2440b54b0 con 0x7ff2440b3d30 2023-12-19T14:51:58.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff244158060 con 0x7ff230003a60 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46518/0 (socket says 172.21.15.50:46518) 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 -- 172.21.15.50:0/2123889208 learned_addr learned my addr 172.21.15.50:0/2123889208 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff243fff700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff230003a60 0x7ff2440c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff23bfff700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 0x7ff2440b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 -- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff230003a60 msgr2=0x7ff2440c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff230003a60 0x7ff2440c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 -- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 msgr2=0x7ff2440b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 0x7ff2440b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 -- 172.21.15.50:0/2123889208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244169780 con 0x7ff2441576f0 2023-12-19T14:51:58.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff23bfff700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 0x7ff2440b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:58.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff248c15700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff244069d00 tx=0x7ff23c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c00e040 con 0x7ff2441576f0 2023-12-19T14:51:58.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c00f040 con 0x7ff2441576f0 2023-12-19T14:51:58.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2440b48a0 con 0x7ff2441576f0 2023-12-19T14:51:58.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c00fcb0 con 0x7ff2441576f0 2023-12-19T14:51:58.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.070+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2440b4e60 con 0x7ff2441576f0 2023-12-19T14:51:58.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.074+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2440b5660 con 0x7ff2441576f0 2023-12-19T14:51:58.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.074+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff23c015640 con 0x7ff2441576f0 2023-12-19T14:51:58.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.074+0000 7ff240ff9700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 0x7ff224062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.074+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ff23c07d400 con 0x7ff2441576f0 2023-12-19T14:51:58.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.078+0000 7ff243fff700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 0x7ff224062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.078+0000 7ff243fff700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 0x7ff224062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c3c0 tx=0x7ff234015000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.086+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff2440b5660 con 0x7ff2441576f0 2023-12-19T14:51:58.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.262+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff2440b5660 con 0x7ff2441576f0 2023-12-19T14:51:58.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.270+0000 7ff240ff9700 1 -- 172.21.15.50:0/2123889208 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff2440b5660 con 0x7ff2441576f0 2023-12-19T14:51:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 msgr2=0x7ff224062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 0x7ff224062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff23400c3c0 tx=0x7ff234015000 comp rx=0 tx=0).stop 2023-12-19T14:51:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 msgr2=0x7ff244157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff244069d00 tx=0x7ff23c00b040 comp rx=0 tx=0).stop 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 shutdown_connections 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2440b3d30 0x7ff2440b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff224060670 0x7ff224062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2441576f0 0x7ff244157b20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 --2- 172.21.15.50:0/2123889208 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff230003a60 0x7ff2440c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 >> 172.21.15.50:0/2123889208 conn(0x7ff244067bb0 msgr2=0x7ff2440cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 shutdown_connections 2023-12-19T14:51:58.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.274+0000 7ff249416700 1 -- 172.21.15.50:0/2123889208 wait complete. 2023-12-19T14:51:58.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:51:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 Processor -- start 2023-12-19T14:51:58.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 -- start start 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbc04160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc040b0e40 con 0x7fbc040ab870 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc040b1120 con 0x7fbc040b08d0 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc0a42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040b1400 con 0x7fbc0415a180 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc03fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbc040ab870 0x7fbc040afdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38366/0 (socket says 172.21.15.50:38366) 2023-12-19T14:51:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc03fff700 1 -- 172.21.15.50:0/3831208410 learned_addr learned my addr 172.21.15.50:0/3831208410 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:51:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666955921 0 0) 0x7fbc040b0e40 con 0x7fbc040ab870 2023-12-19T14:51:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0003570 con 0x7fbc040ab870 2023-12-19T14:51:58.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.466+0000 7fbc089cb700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbc04160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927673982 0 0) 0x7fbc040b1120 con 0x7fbc040b08d0 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc040b0e40 con 0x7fbc040b08d0 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626623737 0 0) 0x7fbbf0003570 con 0x7fbc040ab870 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 msgr2=0x7fbc04160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbc04160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 >> v1:172.21.15.125:6789/0 conn(0x7fbc040b08d0 legacy=0x7fbc04159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040b16e0 con 0x7fbc040ab870 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fbbf40013f0 con 0x7fbc040ab870 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 >> v1:172.21.15.50:6789/0 conn(0x7fbc040ab870 legacy=0x7fbc040afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbbf0003c20 0x7fbbf0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbbf0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbbf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040b1400 con 0x7fbbf0004d50 2023-12-19T14:51:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf000bbf0 con 0x7fbbf0003c20 2023-12-19T14:51:58.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf000bd60 con 0x7fbc0415a180 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbbf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc03fff700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbbf0003c20 0x7fbbf0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbbfbfff700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbbf0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 -- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 msgr2=0x7fbbf0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0415a180 0x7fbbf0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 -- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbbf0003c20 msgr2=0x7fbbf0004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbbf0003c20 0x7fbbf0004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 -- 172.21.15.50:0/3831208410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0003570 con 0x7fbbf0004d50 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc03fff700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbbf0003c20 0x7fbbf0004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc089cb700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbbf00051c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0094d0 tx=0x7fbbfc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876cd086542b2f30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00ea60 con 0x7fbbf0004d50 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc00f040 con 0x7fbbf0004d50 2023-12-19T14:51:58.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.470+0000 7fbc037fe700 1 -- 172.21.15.50:0/3831208410 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00edf0 con 0x7fbbf0004d50 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- 172.21.15.50:0/3831208410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 msgr2=0x7fbbf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/3831208410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbbf00051c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0094d0 tx=0x7fbbfc00b040 comp rx=0 tx=0).stop 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc089cb700 1 -- 172.21.15.50:0/3831208410 reap_dead start 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- 172.21.15.50:0/3831208410 shutdown_connections 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- 172.21.15.50:0/3831208410 >> 172.21.15.50:0/3831208410 conn(0x7fbc040cc580 msgr2=0x7fbc040cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- 172.21.15.50:0/3831208410 shutdown_connections 2023-12-19T14:51:58.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- 172.21.15.50:0/3831208410 wait complete. 2023-12-19T14:51:58.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 Processor -- start 2023-12-19T14:51:58.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.474+0000 7fbc0a42e700 1 -- start start 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc04165580 0x7fbc041659f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 0x7fbc0414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0414bc60 con 0x7fbbf0004d50 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040ac8f0 con 0x7fbc04165580 2023-12-19T14:51:58.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04149240 con 0x7fbc0414b2b0 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46530/0 (socket says 172.21.15.50:46530) 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 -- 172.21.15.50:0/46586570 learned_addr learned my addr 172.21.15.50:0/46586570 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbbfbfff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc04165580 0x7fbc041659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc089cb700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 0x7fbc0414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 -- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 msgr2=0x7fbc0414b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 0x7fbc0414b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 -- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc04165580 msgr2=0x7fbc041659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc04165580 0x7fbc041659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 -- 172.21.15.50:0/46586570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040abce0 con 0x7fbbf0004d50 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc089cb700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 0x7fbc0414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc03fff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400e780 tx=0x7fbbf4000c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4010490 con 0x7fbbf0004d50 2023-12-19T14:51:58.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf4010040 con 0x7fbbf0004d50 2023-12-19T14:51:58.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4010820 con 0x7fbbf0004d50 2023-12-19T14:51:58.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc040abfc0 con 0x7fbbf0004d50 2023-12-19T14:51:58.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.478+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc040ac580 con 0x7fbbf0004d50 2023-12-19T14:51:58.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.482+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc04071120 con 0x7fbbf0004d50 2023-12-19T14:51:58.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.482+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbbf4010b90 con 0x7fbbf0004d50 2023-12-19T14:51:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.482+0000 7fbc00ff9700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 0x7fbbe4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.482+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fbbf407be80 con 0x7fbbf0004d50 2023-12-19T14:51:58.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.482+0000 7fbbfbfff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 0x7fbbe4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.486+0000 7fbbfbfff700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 0x7fbbe4062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc04153a90 tx=0x7fbbec00c040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.490+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbbf404c070 con 0x7fbbf0004d50 2023-12-19T14:51:58.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.662+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbc040cd070 con 0x7fbbf0004d50 2023-12-19T14:51:58.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.674+0000 7fbc00ff9700 1 -- 172.21.15.50:0/46586570 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v14) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbbf404f690 con 0x7fbbf0004d50 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 msgr2=0x7fbbe4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 0x7fbbe4062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc04153a90 tx=0x7fbbec00c040 comp rx=0 tx=0).stop 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 msgr2=0x7fbc04148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400e780 tx=0x7fbbf4000c50 comp rx=0 tx=0).stop 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 shutdown_connections 2023-12-19T14:51:58.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc04165580 0x7fbc041659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbbe4060670 0x7fbbe4062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbc0414b2b0 0x7fbc0414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 --2- 172.21.15.50:0/46586570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbbf0004d50 0x7fbc04148740 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 >> 172.21.15.50:0/46586570 conn(0x7fbc040cc580 msgr2=0x7fbc04154120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 shutdown_connections 2023-12-19T14:51:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.678+0000 7fbc0a42e700 1 -- 172.21.15.50:0/46586570 wait complete. 2023-12-19T14:51:58.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:51:58.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 Processor -- start 2023-12-19T14:51:58.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 -- start start 2023-12-19T14:51:58.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74158c90 0x7f2e740caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e74159770 0x7f2e7406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7406c850 con 0x7f2e7406aeb0 2023-12-19T14:51:58.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7406b470 con 0x7f2e74159770 2023-12-19T14:51:58.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a8a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7406b750 con 0x7f2e74158c90 2023-12-19T14:51:58.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a0a2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e74159770 0x7f2e7406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e798a1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74158c90 0x7f2e740caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a0a2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e74159770 0x7f2e7406cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46538/0 (socket says 172.21.15.50:46538) 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e7a0a2700 1 -- 172.21.15.50:0/577374652 learned_addr learned my addr 172.21.15.50:0/577374652 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.870+0000 7f2e790a0700 1 --1- 172.21.15.50:0/577374652 >> v1:172.21.15.125:6789/0 conn(0x7f2e7406aeb0 0x7f2e74159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:48582/0 (socket says 172.21.15.50:48582) 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a0a2700 1 -- 172.21.15.50:0/577374652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74158c90 msgr2=0x7f2e740caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a0a2700 1 --2- 172.21.15.50:0/577374652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74158c90 0x7f2e740caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7889f700 1 -- 172.21.15.50:0/577374652 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001968143 0 0) 0x7f2e7406c850 con 0x7f2e7406aeb0 2023-12-19T14:51:58.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a0a2700 1 -- 172.21.15.50:0/577374652 >> v1:172.21.15.125:6789/0 conn(0x7f2e7406aeb0 legacy=0x7f2e74159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a0a2700 1 -- 172.21.15.50:0/577374652 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7406ba30 con 0x7f2e74159770 2023-12-19T14:51:58.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a0a2700 1 --2- 172.21.15.50:0/577374652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e74159770 0x7f2e7406cce0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009f20 tx=0x7f2e6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2803c84780952783 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7889f700 1 -- 172.21.15.50:0/577374652 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c00e070 con 0x7f2e74159770 2023-12-19T14:51:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7889f700 1 -- 172.21.15.50:0/577374652 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c00b9d0 con 0x7f2e74159770 2023-12-19T14:51:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7889f700 1 -- 172.21.15.50:0/577374652 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c0092e0 con 0x7f2e74159770 2023-12-19T14:51:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/577374652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e74159770 msgr2=0x7f2e7406cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/577374652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e74159770 0x7f2e7406cce0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009f20 tx=0x7f2e6c009840 comp rx=0 tx=0).stop 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/577374652 shutdown_connections 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/577374652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e74159770 0x7f2e7406cce0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/577374652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74158c90 0x7f2e740caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/577374652 >> 172.21.15.50:0/577374652 conn(0x7f2e74067bb0 msgr2=0x7f2e740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/577374652 shutdown_connections 2023-12-19T14:51:58.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.874+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/577374652 wait complete. 2023-12-19T14:51:58.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 Processor -- start 2023-12-19T14:51:58.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 -- start start 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e74158c90 0x7f2e740b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74159770 0x7f2e740b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e740c4ce0 con 0x7f2e7406aeb0 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e740c52a0 con 0x7f2e74158c90 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.878+0000 7f2e7a8a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e740c9130 con 0x7f2e74159770 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46544/0 (socket says 172.21.15.50:46544) 2023-12-19T14:51:58.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 -- 172.21.15.50:0/2568604907 learned_addr learned my addr 172.21.15.50:0/2568604907 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:58.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e7a0a2700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74159770 0x7f2e740b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e790a0700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e74158c90 0x7f2e740b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 -- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74159770 msgr2=0x7f2e740b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74159770 0x7f2e740b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 -- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e74158c90 msgr2=0x7f2e740b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e74158c90 0x7f2e740b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 -- 172.21.15.50:0/2568604907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c009ae0 con 0x7f2e7406aeb0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e798a1700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000ecb0 tx=0x7f2e700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000c9d0 con 0x7f2e7406aeb0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e740c9410 con 0x7f2e7406aeb0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7000cb30 con 0x7f2e7406aeb0 2023-12-19T14:51:58.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000ced0 con 0x7f2e7406aeb0 2023-12-19T14:51:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e740c8400 con 0x7f2e7406aeb0 2023-12-19T14:51:58.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.882+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7404be40 con 0x7f2e7406aeb0 2023-12-19T14:51:58.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.886+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2e70013680 con 0x7f2e7406aeb0 2023-12-19T14:51:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.886+0000 7f2e69ffb700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 0x7f2e58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.886+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f2e7007cfe0 con 0x7f2e7406aeb0 2023-12-19T14:51:58.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.886+0000 7f2e790a0700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 0x7f2e58062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:58.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.890+0000 7f2e790a0700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 0x7f2e58062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e74156a80 tx=0x7f2e64007bf0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:58.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:58.894+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e7004d230 con 0x7f2e7406aeb0 2023-12-19T14:51:59.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.066+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2e7406bd10 con 0x7f2e7406aeb0 2023-12-19T14:51:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.078+0000 7f2e69ffb700 1 -- 172.21.15.50:0/2568604907 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2e70050850 con 0x7f2e7406aeb0 2023-12-19T14:51:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 msgr2=0x7f2e58062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 0x7f2e58062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e74156a80 tx=0x7f2e64007bf0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 msgr2=0x7f2e74153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000ecb0 tx=0x7f2e700091a0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 shutdown_connections 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e74158c90 0x7f2e740b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e58060670 0x7f2e58062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e74159770 0x7f2e740b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 --2- 172.21.15.50:0/2568604907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e7406aeb0 0x7f2e74153780 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.082+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 >> 172.21.15.50:0/2568604907 conn(0x7f2e74067bb0 msgr2=0x7f2e740cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.086+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 shutdown_connections 2023-12-19T14:51:59.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.086+0000 7f2e7a8a3700 1 -- 172.21.15.50:0/2568604907 wait complete. 2023-12-19T14:51:59.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-12-19T14:51:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 Processor -- start 2023-12-19T14:51:59.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 -- start start 2023-12-19T14:51:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f3c159160 0x7f1f3c0adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0c63a0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c06a7b0 con 0x7f1f3c159160 2023-12-19T14:51:59.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f3b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0ad550 con 0x7f1f3c0c5fd0 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f31d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f3c159160 0x7f1f3c0adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.258+0000 7f1f31d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f3c159160 0x7f1f3c0adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46548/0 (socket says 172.21.15.50:46548) 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 -- 172.21.15.50:0/2487395987 learned_addr learned my addr 172.21.15.50:0/2487395987 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3a59c700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 -- 172.21.15.50:0/2487395987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c0c5fd0 msgr2=0x7f1f3c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 -- 172.21.15.50:0/2487395987 >> v1:172.21.15.50:6789/0 conn(0x7f1f3c0ae210 legacy=0x7f1f3c158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:51:59.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 -- 172.21.15.50:0/2487395987 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0accb0 con 0x7f1f3c159160 2023-12-19T14:51:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c159160 0x7f1f3c0adcd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1f240099b0 tx=0x7f1f24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b76280926a52f8a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f39d9b700 1 -- 172.21.15.50:0/2487395987 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400b820 con 0x7f1f3c159160 2023-12-19T14:51:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f39d9b700 1 -- 172.21.15.50:0/2487395987 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2400b980 con 0x7f1f3c159160 2023-12-19T14:51:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f39d9b700 1 -- 172.21.15.50:0/2487395987 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400bd20 con 0x7f1f3c159160 2023-12-19T14:51:59.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 -- 172.21.15.50:0/2487395987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c159160 msgr2=0x7f1f3c0adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c159160 0x7f1f3c0adcd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1f240099b0 tx=0x7f1f24009250 comp rx=0 tx=0).stop 2023-12-19T14:51:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 -- 172.21.15.50:0/2487395987 shutdown_connections 2023-12-19T14:51:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c159160 0x7f1f3c0adcd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/2487395987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 -- 172.21.15.50:0/2487395987 >> 172.21.15.50:0/2487395987 conn(0x7f1f3c067bb0 msgr2=0x7f1f3c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 -- 172.21.15.50:0/2487395987 shutdown_connections 2023-12-19T14:51:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.262+0000 7f1f3b59e700 1 -- 172.21.15.50:0/2487395987 wait complete. 2023-12-19T14:51:59.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.266+0000 7f1f3b59e700 1 Processor -- start 2023-12-19T14:51:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.266+0000 7f1f3b59e700 1 -- start start 2023-12-19T14:51:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c159160 0x7f1f3c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0c9600 con 0x7f1f3c0ae210 2023-12-19T14:51:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0b1790 con 0x7f1f3c0c5fd0 2023-12-19T14:51:59.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0c98c0 con 0x7f1f3c159160 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46562/0 (socket says 172.21.15.50:46562) 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 -- 172.21.15.50:0/3467617707 learned_addr learned my addr 172.21.15.50:0/3467617707 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3ad9d700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c159160 0x7f1f3c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 -- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c159160 msgr2=0x7f1f3c0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c159160 0x7f1f3c0cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 -- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 msgr2=0x7f1f3c0cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 -- 172.21.15.50:0/3467617707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240094f0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:51:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3a59c700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00d8c0 tx=0x7f1f2c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:59.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00de60 con 0x7f1f3c0ae210 2023-12-19T14:51:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c00b7e0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c0b0ef0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00bba0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.270+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c0b14b0 con 0x7f1f3c0ae210 2023-12-19T14:51:59.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.274+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c04be40 con 0x7f1f3c0ae210 2023-12-19T14:51:59.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.274+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1f2c01e460 con 0x7f1f3c0ae210 2023-12-19T14:51:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.278+0000 7f1f32ffd700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 0x7f1f1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:51:59.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.278+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f1f2c07d300 con 0x7f1f3c0ae210 2023-12-19T14:51:59.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.278+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 0x7f1f1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:51:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.278+0000 7f1f31d9b700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 0x7f1f1c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f240099b0 tx=0x7f1f24016000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:51:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.282+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1f2c04d550 con 0x7f1f3c0ae210 2023-12-19T14:51:59.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.514+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f1f3c0acf90 con 0x7f1f3c0ae210 2023-12-19T14:51:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.926+0000 7f1f32ffd700 1 -- 172.21.15.50:0/3467617707 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v471) v1 ==== 158+0+0 (secure 0 0 0) 0x7f1f2c050b70 con 0x7f1f3c0ae210 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 msgr2=0x7f1f1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 0x7f1f1c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f240099b0 tx=0x7f1f24016000 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 msgr2=0x7f1f3c0b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00d8c0 tx=0x7f1f2c008040 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 shutdown_connections 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f3c0c5fd0 0x7f1f3c0cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f1c060670 0x7f1f1c062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f3c159160 0x7f1f3c0cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 --2- 172.21.15.50:0/3467617707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f3c0ae210 0x7f1f3c0b06a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 >> 172.21.15.50:0/3467617707 conn(0x7f1f3c067bb0 msgr2=0x7f1f3c1576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 shutdown_connections 2023-12-19T14:51:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:51:59.930+0000 7f1f3b59e700 1 -- 172.21.15.50:0/3467617707 wait complete. 2023-12-19T14:51:59.957 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-12-19T14:51:59.958 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-12-19T14:51:59.958 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:51:59.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:52:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.122+0000 7f79bf06e700 1 Processor -- start 2023-12-19T14:52:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.122+0000 7f79bf06e700 1 -- start start 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bf06e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b8161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bf06e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b806e150 con 0x7f79b8140ae0 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bf06e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b806d540 con 0x7f79b8140710 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bf06e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806d820 con 0x7f79b8068800 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b7fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f79b8140ae0 0x7f79b80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38384/0 (socket says 172.21.15.50:38384) 2023-12-19T14:52:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b7fff700 1 -- 172.21.15.50:0/3829694244 learned_addr learned my addr 172.21.15.50:0/3829694244 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bd60b700 1 --2- 172.21.15.50:0/3829694244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b8161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538404773 0 0) 0x7f79b806e150 con 0x7f79b8140ae0 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a0003570 con 0x7f79b8140ae0 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813872558 0 0) 0x7f79b806d540 con 0x7f79b8140710 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b806e150 con 0x7f79b8140710 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bd60b700 1 -- 172.21.15.50:0/3829694244 >> v1:172.21.15.125:6789/0 conn(0x7f79b8140710 legacy=0x7f79b80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bd60b700 1 -- 172.21.15.50:0/3829694244 >> v1:172.21.15.50:6789/0 conn(0x7f79b8140ae0 legacy=0x7f79b80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.126+0000 7f79bd60b700 1 -- 172.21.15.50:0/3829694244 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b806db00 con 0x7f79b8068800 2023-12-19T14:52:00.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bd60b700 1 --2- 172.21.15.50:0/3829694244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b8161cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e90 tx=0x7f79b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62ee39a1805bfae8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000e070 con 0x7f79b8068800 2023-12-19T14:52:00.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b000bb80 con 0x7f79b8068800 2023-12-19T14:52:00.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79b77fe700 1 -- 172.21.15.50:0/3829694244 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b0012430 con 0x7f79b8068800 2023-12-19T14:52:00.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 -- 172.21.15.50:0/3829694244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 msgr2=0x7f79b8161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 --2- 172.21.15.50:0/3829694244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b8161cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e90 tx=0x7f79b0009780 comp rx=0 tx=0).stop 2023-12-19T14:52:00.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 -- 172.21.15.50:0/3829694244 shutdown_connections 2023-12-19T14:52:00.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 --2- 172.21.15.50:0/3829694244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b8161cd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 -- 172.21.15.50:0/3829694244 >> 172.21.15.50:0/3829694244 conn(0x7f79b80c8850 msgr2=0x7f79b80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:00.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 -- 172.21.15.50:0/3829694244 shutdown_connections 2023-12-19T14:52:00.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.130+0000 7f79bf06e700 1 -- 172.21.15.50:0/3829694244 wait complete. 2023-12-19T14:52:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 Processor -- start 2023-12-19T14:52:00.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 -- start start 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b80b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79b8140710 0x7f79b8159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b813f220 con 0x7f79b8140ae0 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8169200 con 0x7f79b8140710 2023-12-19T14:52:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bf06e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b816a020 con 0x7f79b8068800 2023-12-19T14:52:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46572/0 (socket says 172.21.15.50:46572) 2023-12-19T14:52:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 -- 172.21.15.50:0/2364898376 learned_addr learned my addr 172.21.15.50:0/2364898376 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bce0a700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b80b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79b7fff700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79b8140710 0x7f79b8159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 -- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 msgr2=0x7f79b80b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b80b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 -- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79b8140710 msgr2=0x7f79b8159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79b8140710 0x7f79b8159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.134+0000 7f79bd60b700 1 -- 172.21.15.50:0/2364898376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009a20 con 0x7f79b8140ae0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79bd60b700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f79b0013000 tx=0x7f79b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b001b420 con 0x7f79b8140ae0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b001b5a0 con 0x7f79b8140ae0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b001b960 con 0x7f79b8140ae0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b815a4b0 con 0x7f79b8140ae0 2023-12-19T14:52:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b81696f0 con 0x7f79b8140ae0 2023-12-19T14:52:00.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.138+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8071120 con 0x7f79b8140ae0 2023-12-19T14:52:00.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.142+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f79b001bd50 con 0x7f79b8140ae0 2023-12-19T14:52:00.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.142+0000 7f79b4ff9700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 0x7f79a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.142+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f79b007e560 con 0x7f79b8140ae0 2023-12-19T14:52:00.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.142+0000 7f79bce0a700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 0x7f79a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.146+0000 7f79bce0a700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 0x7f79a4062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79ac007fd0 tx=0x7f79ac007d50 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.150+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f79b004e790 con 0x7f79b8140ae0 2023-12-19T14:52:00.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.318+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79b80642a0 con 0x7f79b8140ae0 2023-12-19T14:52:00.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.318+0000 7f79b4ff9700 1 -- 172.21.15.50:0/2364898376 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13856 (secure 0 0 0) 0x7f79b0051d70 con 0x7f79b8140ae0 2023-12-19T14:52:00.322 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:00.322 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":59,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:51:57.849802+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/906596857":"2023-12-20T14:51:54.872395+0000","172.21.15.50:6834/906596857":"2023-12-20T14:51:54.872395+0000","172.21.15.125:0/2960058572":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/902571518":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3904162499":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3771030607":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3779418184":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/3338034462":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/1210582553":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6832/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:6833/33827":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/4107501911":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/1437130881":"2023-12-20T14:51:54.368748+0000","172.21.15.125:0/751498889":"2023-12-20T14:40:35.182753+0000","172.21.15.125:0/3449382498":"2023-12-20T14:40:35.182753+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":6,"length":1},{"begin":11,"length":1},{"begin":15,"length":1},{"begin":18,"length":2},{"begin":22,"length":1},{"begin":28,"length":1},{"begin":31,"length":1},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":41,"length":1},{"begin":46,"length":2},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":60,"length":2},{"begin":64,"length":3},{"begin":78,"length":3},{"begin":85,"length":3},{"begin":92,"length":1},{"begin":100,"length":2},{"begin":104,"length":1},{"begin":106,"length":1},{"begin":108,"length":1},{"begin":190,"length":32}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 msgr2=0x7f79a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 0x7f79a4062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79ac007fd0 tx=0x7f79ac007d50 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 msgr2=0x7f79b80a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f79b0013000 tx=0x7f79b0019000 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 shutdown_connections 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79b8140710 0x7f79b8159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79a4060670 0x7f79a4062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8140ae0 0x7f79b80a9820 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 --2- 172.21.15.50:0/2364898376 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79b8068800 0x7f79b80b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 >> 172.21.15.50:0/2364898376 conn(0x7f79b80c8850 msgr2=0x7f79b80adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:00.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 shutdown_connections 2023-12-19T14:52:00.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.322+0000 7f79bf06e700 1 -- 172.21.15.50:0/2364898376 wait complete. 2023-12-19T14:52:00.345 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:52:00.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:52:00.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 Processor -- start 2023-12-19T14:52:00.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 -- start start 2023-12-19T14:52:00.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac600c6d20 0x7fac600c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac6006ac30 0x7fac6006b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6006a7a0 con 0x7fac6006c7c0 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac600c7630 con 0x7fac600c6d20 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac66985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6006d4b0 con 0x7fac6006ac30 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac600c6d20 0x7fac600c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac600c6d20 0x7fac600c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46574/0 (socket says 172.21.15.50:46574) 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 -- 172.21.15.50:0/3937737150 learned_addr learned my addr 172.21.15.50:0/3937737150 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:00.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac64f22700 1 --2- 172.21.15.50:0/3937737150 >> v2:172.21.15.125:3300/0 conn(0x7fac6006ac30 0x7fac6006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 -- 172.21.15.50:0/3937737150 >> v2:172.21.15.125:3300/0 conn(0x7fac6006ac30 msgr2=0x7fac6006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:00.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 --2- 172.21.15.50:0/3937737150 >> v2:172.21.15.125:3300/0 conn(0x7fac6006ac30 0x7fac6006b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 -- 172.21.15.50:0/3937737150 >> v1:172.21.15.125:6789/0 conn(0x7fac6006c7c0 legacy=0x7fac6006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.518+0000 7fac5ffff700 1 -- 172.21.15.50:0/3937737150 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6006d790 con 0x7fac600c6d20 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac5ffff700 1 --2- 172.21.15.50:0/3937737150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac600c70f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac600695c0 tx=0x7fac50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a816299df10c0a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac5effd700 1 -- 172.21.15.50:0/3937737150 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5000b860 con 0x7fac600c6d20 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac5effd700 1 -- 172.21.15.50:0/3937737150 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5000b9c0 con 0x7fac600c6d20 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac5effd700 1 -- 172.21.15.50:0/3937737150 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5000bd60 con 0x7fac600c6d20 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 -- 172.21.15.50:0/3937737150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 msgr2=0x7fac600c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 --2- 172.21.15.50:0/3937737150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac600c70f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac600695c0 tx=0x7fac50009250 comp rx=0 tx=0).stop 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 -- 172.21.15.50:0/3937737150 shutdown_connections 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 --2- 172.21.15.50:0/3937737150 >> v2:172.21.15.125:3300/0 conn(0x7fac6006ac30 0x7fac6006b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 --2- 172.21.15.50:0/3937737150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac600c70f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 -- 172.21.15.50:0/3937737150 >> 172.21.15.50:0/3937737150 conn(0x7fac60067bb0 msgr2=0x7fac600cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 -- 172.21.15.50:0/3937737150 shutdown_connections 2023-12-19T14:52:00.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.522+0000 7fac66985700 1 -- 172.21.15.50:0/3937737150 wait complete. 2023-12-19T14:52:00.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 Processor -- start 2023-12-19T14:52:00.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 -- start start 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6006ac30 0x7fac60148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6006c7c0 0x7fac600c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac600aff30 con 0x7fac600c6d20 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac600af0d0 con 0x7fac6006ac30 2023-12-19T14:52:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac66985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac600af3b0 con 0x7fac6006c7c0 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46582/0 (socket says 172.21.15.50:46582) 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 -- 172.21.15.50:0/1836993451 learned_addr learned my addr 172.21.15.50:0/1836993451 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac5ffff700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6006ac30 0x7fac60148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac5f7fe700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6006c7c0 0x7fac600c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 -- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6006c7c0 msgr2=0x7fac600c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6006c7c0 0x7fac600c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 -- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6006ac30 msgr2=0x7fac60148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6006ac30 0x7fac60148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 -- 172.21.15.50:0/1836993451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500094f0 con 0x7fac600c6d20 2023-12-19T14:52:00.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.526+0000 7fac64f22700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fac58009fa0 tx=0x7fac58009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5800ebc0 con 0x7fac600c6d20 2023-12-19T14:52:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5800ed20 con 0x7fac600c6d20 2023-12-19T14:52:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac600af690 con 0x7fac600c6d20 2023-12-19T14:52:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58014670 con 0x7fac600c6d20 2023-12-19T14:52:00.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac60148c50 con 0x7fac600c6d20 2023-12-19T14:52:00.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac600b00a0 con 0x7fac600c6d20 2023-12-19T14:52:00.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.530+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fac580149e0 con 0x7fac600c6d20 2023-12-19T14:52:00.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.534+0000 7fac47fff700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 0x7fac4c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.534+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fac5807d210 con 0x7fac600c6d20 2023-12-19T14:52:00.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.534+0000 7fac5ffff700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 0x7fac4c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.534+0000 7fac5ffff700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 0x7fac4c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac600695c0 tx=0x7fac50017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.542+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac600b00a0 con 0x7fac600c6d20 2023-12-19T14:52:00.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.718+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac600b00a0 con 0x7fac600c6d20 2023-12-19T14:52:00.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.718+0000 7fac47fff700 1 -- 172.21.15.50:0/1836993451 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac600b00a0 con 0x7fac600c6d20 2023-12-19T14:52:00.721 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 msgr2=0x7fac4c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 0x7fac4c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac600695c0 tx=0x7fac50017000 comp rx=0 tx=0).stop 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 msgr2=0x7fac60155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fac58009fa0 tx=0x7fac58009900 comp rx=0 tx=0).stop 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 shutdown_connections 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac4c060670 0x7fac4c062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac600c6d20 0x7fac60155f30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac6006c7c0 0x7fac600c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 --2- 172.21.15.50:0/1836993451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac6006ac30 0x7fac60148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 >> 172.21.15.50:0/1836993451 conn(0x7fac60067bb0 msgr2=0x7fac600c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:00.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 shutdown_connections 2023-12-19T14:52:00.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.722+0000 7fac66985700 1 -- 172.21.15.50:0/1836993451 wait complete. 2023-12-19T14:52:00.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-12-19T14:52:00.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.910+0000 7f13c13c5700 1 Processor -- start 2023-12-19T14:52:00.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.910+0000 7f13c13c5700 1 -- start start 2023-12-19T14:52:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13bc0cea90 0x7f13bc0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 0x7f13bc068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0cbea0 con 0x7f13bc0cc7b0 2023-12-19T14:52:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0cb030 con 0x7f13bc0cea90 2023-12-19T14:52:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0cb2d0 con 0x7f13bc0cc330 2023-12-19T14:52:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13bc0cea90 0x7f13bc0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f13bc0cea90 0x7f13bc0cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46584/0 (socket says 172.21.15.50:46584) 2023-12-19T14:52:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 -- 172.21.15.50:0/1918923930 learned_addr learned my addr 172.21.15.50:0/1918923930 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13ba7fc700 1 --2- 172.21.15.50:0/1918923930 >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 0x7f13bc068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 -- 172.21.15.50:0/1918923930 >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 msgr2=0x7f13bc068a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 --2- 172.21.15.50:0/1918923930 >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 0x7f13bc068a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 -- 172.21.15.50:0/1918923930 >> v1:172.21.15.50:6789/0 conn(0x7f13bc0cc7b0 legacy=0x7f13bc142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 -- 172.21.15.50:0/1918923930 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc067e00 con 0x7f13bc0cea90 2023-12-19T14:52:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13ba7fc700 1 --2- 172.21.15.50:0/1918923930 >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 0x7f13bc068a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13baffd700 1 --2- 172.21.15.50:0/1918923930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc0cee60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f13bc06a330 tx=0x7f13ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebe7c931cd14764 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13b9ffb700 1 -- 172.21.15.50:0/1918923930 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00b860 con 0x7f13bc0cea90 2023-12-19T14:52:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13b9ffb700 1 -- 172.21.15.50:0/1918923930 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac00b9c0 con 0x7f13bc0cea90 2023-12-19T14:52:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13b9ffb700 1 -- 172.21.15.50:0/1918923930 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00bd60 con 0x7f13bc0cea90 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 -- 172.21.15.50:0/1918923930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 msgr2=0x7f13bc0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.914+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1918923930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc0cee60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f13bc06a330 tx=0x7f13ac009250 comp rx=0 tx=0).stop 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 -- 172.21.15.50:0/1918923930 shutdown_connections 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1918923930 >> v2:172.21.15.125:3300/0 conn(0x7f13bc0cc330 0x7f13bc068a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1918923930 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc0cee60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 -- 172.21.15.50:0/1918923930 >> 172.21.15.50:0/1918923930 conn(0x7f13bc0cd650 msgr2=0x7f13bc06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:00.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 -- 172.21.15.50:0/1918923930 shutdown_connections 2023-12-19T14:52:00.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 -- 172.21.15.50:0/1918923930 wait complete. 2023-12-19T14:52:00.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 Processor -- start 2023-12-19T14:52:00.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.918+0000 7f13c13c5700 1 -- start start 2023-12-19T14:52:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13bc0cc330 0x7f13bc13ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13bc0cc7b0 0x7f13bc15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc13f1a0 con 0x7f13bc0cea90 2023-12-19T14:52:00.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc15bec0 con 0x7f13bc0cc7b0 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc15b2b0 con 0x7f13bc0cc330 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46586/0 (socket says 172.21.15.50:46586) 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 -- 172.21.15.50:0/1898802347 learned_addr learned my addr 172.21.15.50:0/1898802347 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13baffd700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13bc0cc330 0x7f13bc13ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13ba7fc700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13bc0cc7b0 0x7f13bc15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 -- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13bc0cc330 msgr2=0x7f13bc13ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13bc0cc330 0x7f13bc13ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 -- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13bc0cc7b0 msgr2=0x7f13bc15a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13bc0cc7b0 0x7f13bc15a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 -- 172.21.15.50:0/1898802347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac0094f0 con 0x7f13bc0cea90 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13bb7fe700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f13b400c8e0 tx=0x7f13b4001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4012630 con 0x7f13bc0cea90 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4012790 con 0x7f13bc0cea90 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc15b590 con 0x7f13bc0cea90 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4012b50 con 0x7f13bc0cea90 2023-12-19T14:52:00.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.922+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc15bb50 con 0x7f13bc0cea90 2023-12-19T14:52:00.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.926+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc15c050 con 0x7f13bc0cea90 2023-12-19T14:52:00.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.926+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f13b401a3a0 con 0x7f13bc0cea90 2023-12-19T14:52:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.926+0000 7f13a2ffd700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 0x7f13a8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.926+0000 7f13baffd700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 0x7f13a8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.930+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f13b4051430 con 0x7f13bc0cea90 2023-12-19T14:52:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.930+0000 7f13baffd700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 0x7f13a8062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13bc06a330 tx=0x7f13ac017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:00.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:00.934+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f13b404d030 con 0x7f13bc0cea90 2023-12-19T14:52:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.166+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f13bc071120 con 0x7f13bc0cea90 2023-12-19T14:52:01.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.946+0000 7f13a2ffd700 1 -- 172.21.15.50:0/1898802347 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v473) v1 ==== 157+0+0 (secure 0 0 0) 0x7f13b4050650 con 0x7f13bc0cea90 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 msgr2=0x7f13a8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 0x7f13a8062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f13bc06a330 tx=0x7f13ac017000 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 msgr2=0x7f13bc15acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f13b400c8e0 tx=0x7f13b4001710 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 shutdown_connections 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13bc0cc7b0 0x7f13bc15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f13a8060670 0x7f13a8062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f13bc0cea90 0x7f13bc15acc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 --2- 172.21.15.50:0/1898802347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f13bc0cc330 0x7f13bc13ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 >> 172.21.15.50:0/1898802347 conn(0x7f13bc0cd650 msgr2=0x7f13bc1419c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:01.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 shutdown_connections 2023-12-19T14:52:01.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:01.950+0000 7f13c13c5700 1 -- 172.21.15.50:0/1898802347 wait complete. 2023-12-19T14:52:01.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:52:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.154+0000 7fb3624cd700 1 Processor -- start 2023-12-19T14:52:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.154+0000 7fb3624cd700 1 -- start start 2023-12-19T14:52:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb3624cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb35c158e20 0x7fb35c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb3624cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 0x7fb35c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb3624cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0cbdf0 con 0x7fb35c0ca2c0 2023-12-19T14:52:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb3624cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0cc0d0 con 0x7fb35c158e20 2023-12-19T14:52:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb3624cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0cc3b0 con 0x7fb35c0cb970 2023-12-19T14:52:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb35bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb35c158e20 0x7fb35c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb360a6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb35c0ca2c0 0x7fb35c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38404/0 (socket says 172.21.15.50:38404) 2023-12-19T14:52:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.158+0000 7fb360a6a700 1 -- 172.21.15.50:0/3478595557 learned_addr learned my addr 172.21.15.50:0/3478595557 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35b7fe700 1 --2- 172.21.15.50:0/3478595557 >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 0x7fb35c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35affd700 1 -- 172.21.15.50:0/3478595557 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740094217 0 0) 0x7fb35c0cbdf0 con 0x7fb35c0ca2c0 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35affd700 1 -- 172.21.15.50:0/3478595557 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344003570 con 0x7fb35c0ca2c0 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35bfff700 1 -- 172.21.15.50:0/3478595557 >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 msgr2=0x7fb35c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35bfff700 1 --2- 172.21.15.50:0/3478595557 >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 0x7fb35c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35bfff700 1 -- 172.21.15.50:0/3478595557 >> v1:172.21.15.50:6789/0 conn(0x7fb35c0ca2c0 legacy=0x7fb35c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35bfff700 1 -- 172.21.15.50:0/3478595557 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0cc690 con 0x7fb35c158e20 2023-12-19T14:52:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35b7fe700 1 --2- 172.21.15.50:0/3478595557 >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 0x7fb35c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35bfff700 1 --2- 172.21.15.50:0/3478595557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c158e20 0x7fb35c0ce5c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0098e0 tx=0x7fb34c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a59aab7e052511 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35affd700 1 -- 172.21.15.50:0/3478595557 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c011630 con 0x7fb35c158e20 2023-12-19T14:52:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35affd700 1 -- 172.21.15.50:0/3478595557 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c011790 con 0x7fb35c158e20 2023-12-19T14:52:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb35affd700 1 -- 172.21.15.50:0/3478595557 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c011b30 con 0x7fb35c158e20 2023-12-19T14:52:02.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 -- 172.21.15.50:0/3478595557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c158e20 msgr2=0x7fb35c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 --2- 172.21.15.50:0/3478595557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c158e20 0x7fb35c0ce5c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0098e0 tx=0x7fb34c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 -- 172.21.15.50:0/3478595557 shutdown_connections 2023-12-19T14:52:02.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 --2- 172.21.15.50:0/3478595557 >> v2:172.21.15.125:3300/0 conn(0x7fb35c0cb970 0x7fb35c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 --2- 172.21.15.50:0/3478595557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c158e20 0x7fb35c0ce5c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 -- 172.21.15.50:0/3478595557 >> 172.21.15.50:0/3478595557 conn(0x7fb35c067bb0 msgr2=0x7fb35c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:02.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 -- 172.21.15.50:0/3478595557 shutdown_connections 2023-12-19T14:52:02.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.162+0000 7fb3624cd700 1 -- 172.21.15.50:0/3478595557 wait complete. 2023-12-19T14:52:02.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.166+0000 7fb3624cd700 1 Processor -- start 2023-12-19T14:52:02.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.166+0000 7fb3624cd700 1 -- start start 2023-12-19T14:52:02.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 0x7fb35c14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb35c158e20 0x7fb35c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c14d350 con 0x7fb35c14fef0 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c14c740 con 0x7fb35c0ca2c0 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c14ca20 con 0x7fb35c158e20 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46598/0 (socket says 172.21.15.50:46598) 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 -- 172.21.15.50:0/2660329277 learned_addr learned my addr 172.21.15.50:0/2660329277 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb35bfff700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 0x7fb35c14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb35b7fe700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb35c158e20 0x7fb35c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 -- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb35c158e20 msgr2=0x7fb35c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb35c158e20 0x7fb35c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 -- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 msgr2=0x7fb35c14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 0x7fb35c14f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 -- 172.21.15.50:0/2660329277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c009420 con 0x7fb35c14fef0 2023-12-19T14:52:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb35bfff700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 0x7fb35c14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb360a6a700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb354001460 tx=0x7fb354001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35400ce80 con 0x7fb35c14fef0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35400ab70 con 0x7fb35c14fef0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3540070f0 con 0x7fb35c14fef0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c14cd00 con 0x7fb35c14fef0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.170+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c14d7e0 con 0x7fb35c14fef0 2023-12-19T14:52:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.174+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c14d500 con 0x7fb35c14fef0 2023-12-19T14:52:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.174+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb354007520 con 0x7fb35c14fef0 2023-12-19T14:52:02.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.174+0000 7fb343fff700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 0x7fb348062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.178+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fb35407d0b0 con 0x7fb35c14fef0 2023-12-19T14:52:02.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.178+0000 7fb35bfff700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 0x7fb348062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.178+0000 7fb35bfff700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 0x7fb348062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0091e0 tx=0x7fb34c017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.182+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb35c14d500 con 0x7fb35c14fef0 2023-12-19T14:52:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.366+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb35c14d490 con 0x7fb35c14fef0 2023-12-19T14:52:02.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.374+0000 7fb343fff700 1 -- 172.21.15.50:0/2660329277 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb35404d2f0 con 0x7fb35c14fef0 2023-12-19T14:52:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 msgr2=0x7fb348062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 0x7fb348062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0091e0 tx=0x7fb34c017000 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 msgr2=0x7fb35c0c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb354001460 tx=0x7fb354001ae0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 shutdown_connections 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb348060670 0x7fb348062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb35c14fef0 0x7fb35c0c9c80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb35c158e20 0x7fb35c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 --2- 172.21.15.50:0/2660329277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb35c0ca2c0 0x7fb35c14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 >> 172.21.15.50:0/2660329277 conn(0x7fb35c067bb0 msgr2=0x7fb35c0cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 shutdown_connections 2023-12-19T14:52:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.378+0000 7fb3624cd700 1 -- 172.21.15.50:0/2660329277 wait complete. 2023-12-19T14:52:02.403 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:52:02.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:52:02.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:52:02.489 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:52:02.489 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:52:02.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:52:02.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 Processor -- start 2023-12-19T14:52:02.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 -- start start 2023-12-19T14:52:02.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0f815d320 0x7fe0f815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f80c15b0 0x7fe0f80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0f806d690 0x7fe0f80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f80c7190 con 0x7fe0f806d690 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f80c1b50 con 0x7fe0f815d320 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.670+0000 7fe0ffb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80c1e30 con 0x7fe0f80c15b0 2023-12-19T14:52:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0f806d690 0x7fe0f80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0f806d690 0x7fe0f80680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46604/0 (socket says 172.21.15.50:46604) 2023-12-19T14:52:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3824926551 learned_addr learned my addr 172.21.15.50:0/3824926551 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fd8e1700 1 --2- 172.21.15.50:0/3824926551 >> v2:172.21.15.125:3300/0 conn(0x7fe0f815d320 0x7fe0f815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fd0e0700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f80c15b0 0x7fe0f80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3824926551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f80c15b0 msgr2=0x7fe0f80aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f80c15b0 0x7fe0f80aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3824926551 >> v2:172.21.15.125:3300/0 conn(0x7fe0f815d320 msgr2=0x7fe0f815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3824926551 >> v2:172.21.15.125:3300/0 conn(0x7fe0f815d320 0x7fe0f815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3824926551 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f80c6e20 con 0x7fe0f806d690 2023-12-19T14:52:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f806d690 0x7fe0f80680f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80cbd40 tx=0x7fe0f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf789aab7041fff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fc8df700 1 -- 172.21.15.50:0/3824926551 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f000e070 con 0x7fe0f806d690 2023-12-19T14:52:02.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fc8df700 1 -- 172.21.15.50:0/3824926551 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f000b970 con 0x7fe0f806d690 2023-12-19T14:52:02.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0fc8df700 1 -- 172.21.15.50:0/3824926551 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00092e0 con 0x7fe0f806d690 2023-12-19T14:52:02.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3824926551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f806d690 msgr2=0x7fe0f80680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.674+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f806d690 0x7fe0f80680f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80cbd40 tx=0x7fe0f0009840 comp rx=0 tx=0).stop 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3824926551 shutdown_connections 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f806d690 0x7fe0f80680f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3824926551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f80c15b0 0x7fe0f80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3824926551 >> v2:172.21.15.125:3300/0 conn(0x7fe0f815d320 0x7fe0f815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3824926551 >> 172.21.15.50:0/3824926551 conn(0x7fe0f80c8850 msgr2=0x7fe0f80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3824926551 shutdown_connections 2023-12-19T14:52:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3824926551 wait complete. 2023-12-19T14:52:02.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 Processor -- start 2023-12-19T14:52:02.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- start start 2023-12-19T14:52:02.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f806d690 0x7fe0f80c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f80c15b0 0x7fe0f80b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81546f0 con 0x7fe0f815d320 2023-12-19T14:52:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81608d0 con 0x7fe0f80c15b0 2023-12-19T14:52:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.678+0000 7fe0ffb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f815ffe0 con 0x7fe0f806d690 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46620/0 (socket says 172.21.15.50:46620) 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3069788248 learned_addr learned my addr 172.21.15.50:0/3069788248 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fd8e1700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f806d690 0x7fe0f80c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fd0e0700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f80c15b0 0x7fe0f80b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f806d690 msgr2=0x7fe0f80c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f806d690 0x7fe0f80c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f80c15b0 msgr2=0x7fe0f80b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f80c15b0 0x7fe0f80b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 -- 172.21.15.50:0/3069788248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0009ae0 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0fe0e2700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000b610 tx=0x7fe0f0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0008b00 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f0008c80 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f80c3ae0 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0026460 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.682+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8160560 con 0x7fe0f815d320 2023-12-19T14:52:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.686+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f8160a60 con 0x7fe0f815d320 2023-12-19T14:52:02.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.686+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe0f0003040 con 0x7fe0f815d320 2023-12-19T14:52:02.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.686+0000 7fe0edffb700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 0x7fe0dc062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:02.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.686+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fe0f007e9b0 con 0x7fe0f815d320 2023-12-19T14:52:02.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.686+0000 7fe0fd8e1700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 0x7fe0dc062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:02.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.690+0000 7fe0fd8e1700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 0x7fe0dc062bd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009d10 tx=0x7fe0f40091a0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:02.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.694+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe0f004eba0 con 0x7fe0f815d320 2023-12-19T14:52:02.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.902+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0f8071120 con 0x7fe0f815d320 2023-12-19T14:52:02.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.902+0000 7fe0edffb700 1 -- 172.21.15.50:0/3069788248 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fe0f0021de0 con 0x7fe0f815d320 2023-12-19T14:52:02.907 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:02.907 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4862,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/542876175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":542876175},{"type":"v1","addr":"172.21.15.125:6837","nonce":542876175}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2649941206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2649941206},{"type":"v1","addr":"172.21.15.125:6835","nonce":2649941206}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5088,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2162741746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2162741746},{"type":"v1","addr":"172.21.15.50:6833","nonce":2162741746}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5871,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2562006430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2562006430},{"type":"v1","addr":"172.21.15.50:6835","nonce":2562006430}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 msgr2=0x7fe0dc062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 0x7fe0dc062bd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009d10 tx=0x7fe0f40091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 msgr2=0x7fe0f80b1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000b610 tx=0x7fe0f0018000 comp rx=0 tx=0).stop 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 shutdown_connections 2023-12-19T14:52:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0f80c15b0 0x7fe0f80b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe0dc060720 0x7fe0dc062bd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0f815d320 0x7fe0f80b1640 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 --2- 172.21.15.50:0/3069788248 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0f806d690 0x7fe0f80c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 >> 172.21.15.50:0/3069788248 conn(0x7fe0f80c8850 msgr2=0x7fe0f8067c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 shutdown_connections 2023-12-19T14:52:02.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:02.906+0000 7fe0ffb45700 1 -- 172.21.15.50:0/3069788248 wait complete. 2023-12-19T14:52:02.912 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 19 2023-12-19T14:52:02.930 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:52:02.930 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:52:02.930 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:52:02.930 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:52:02.931 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:52:02.931 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:52:02.931 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:52:02.931 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:52:02.931 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:52:02.931 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:52:02.931 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:52:02.932 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:52:02.932 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:52:02.932 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:52:02.932 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:52:02.932 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:52:02.949 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:52:02.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:52:02.951 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:52:02.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:52:02.959 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:52:02.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:52:02.960 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:52:02.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:52:02.999 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:52:03.000 DEBUG:teuthology.parallel:result is None 2023-12-19T14:52:03.004 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:52:03.004 DEBUG:teuthology.parallel:result is None 2023-12-19T14:52:03.009 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:52:03.009 DEBUG:teuthology.parallel:result is None 2023-12-19T14:52:03.015 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:52:03.015 DEBUG:teuthology.parallel:result is None 2023-12-19T14:52:03.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:52:03.108 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:52:03.110 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:52:03.122 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:52:03.125 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:52:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 Processor -- start 2023-12-19T14:52:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 -- start start 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2860158cc0 0x7f286006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f28600b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286006b0d0 con 0x7f286006dc00 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600b4790 con 0x7f2860158cc0 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f28654f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286006d710 con 0x7f28600b3870 2023-12-19T14:52:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f285e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f286006dc00 0x7f286006b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38416/0 (socket says 172.21.15.50:38416) 2023-12-19T14:52:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f285e7fc700 1 -- 172.21.15.50:0/3265860599 learned_addr learned my addr 172.21.15.50:0/3265860599 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f285effd700 1 --2- 172.21.15.50:0/3265860599 >> v2:172.21.15.125:3300/0 conn(0x7f2860158cc0 0x7f286006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.182+0000 7f285f7fe700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f28600b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121637635 0 0) 0x7f286006b0d0 con 0x7f286006dc00 2023-12-19T14:52:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848003570 con 0x7f286006dc00 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285effd700 1 -- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 msgr2=0x7f28600b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987736816 0 0) 0x7f2848003570 con 0x7f286006dc00 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285effd700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f28600b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285effd700 1 -- 172.21.15.50:0/3265860599 >> v1:172.21.15.50:6789/0 conn(0x7f286006dc00 legacy=0x7f286006b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285effd700 1 -- 172.21.15.50:0/3265860599 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286006a7e0 con 0x7f2860158cc0 2023-12-19T14:52:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285f7fe700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f28600b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285effd700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2860158cc0 0x7f286006d1d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f28500098e0 tx=0x7f28500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43bd4b11724c3d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854001470 con 0x7f2860158cc0 2023-12-19T14:52:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850011420 con 0x7f2860158cc0 2023-12-19T14:52:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f285dffb700 1 -- 172.21.15.50:0/3265860599 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28500117c0 con 0x7f2860158cc0 2023-12-19T14:52:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 -- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2860158cc0 msgr2=0x7f286006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2860158cc0 0x7f286006d1d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f28500098e0 tx=0x7f28500091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 -- 172.21.15.50:0/3265860599 shutdown_connections 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f28600b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 --2- 172.21.15.50:0/3265860599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2860158cc0 0x7f286006d1d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 -- 172.21.15.50:0/3265860599 >> 172.21.15.50:0/3265860599 conn(0x7f2860067bb0 msgr2=0x7f28600cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 -- 172.21.15.50:0/3265860599 shutdown_connections 2023-12-19T14:52:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.186+0000 7f28654f0700 1 -- 172.21.15.50:0/3265860599 wait complete. 2023-12-19T14:52:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.190+0000 7f28654f0700 1 Processor -- start 2023-12-19T14:52:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.190+0000 7f28654f0700 1 -- start start 2023-12-19T14:52:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f286006dc00 0x7f2860156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f2860088c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600cec20 con 0x7f2860158cc0 2023-12-19T14:52:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860088800 con 0x7f286006dc00 2023-12-19T14:52:03.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2860087bf0 con 0x7f28600b3870 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46658/0 (socket says 172.21.15.50:46658) 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 -- 172.21.15.50:0/801250895 learned_addr learned my addr 172.21.15.50:0/801250895 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285e7fc700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f2860088c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285effd700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f286006dc00 0x7f2860156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 -- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 msgr2=0x7f2860088c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f2860088c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 -- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f286006dc00 msgr2=0x7f2860156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:03.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f286006dc00 0x7f2860156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 -- 172.21.15.50:0/801250895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009a20 con 0x7f2860158cc0 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f285f7fe700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2858009fc0 tx=0x7f285800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800cbd0 con 0x7f2860158cc0 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800f040 con 0x7f2860158cc0 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2850009420 con 0x7f2860158cc0 2023-12-19T14:52:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858014480 con 0x7f2860158cc0 2023-12-19T14:52:03.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.194+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28600881b0 con 0x7f2860158cc0 2023-12-19T14:52:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.198+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286015c690 con 0x7f2860158cc0 2023-12-19T14:52:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.198+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2858003020 con 0x7f2860158cc0 2023-12-19T14:52:03.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.198+0000 7f2846ffd700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 0x7f284c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.198+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f285807d520 con 0x7f2860158cc0 2023-12-19T14:52:03.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.198+0000 7f285effd700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 0x7f284c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:03.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.202+0000 7f285effd700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 0x7f284c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f28500030c0 tx=0x7f2850016000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.206+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f286015c690 con 0x7f2860158cc0 2023-12-19T14:52:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:03.406+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f286015c690 con 0x7f2860158cc0 2023-12-19T14:52:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.410+0000 7f2846ffd700 1 -- 172.21.15.50:0/801250895 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f286015c690 con 0x7f2860158cc0 2023-12-19T14:52:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 msgr2=0x7f284c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 0x7f284c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f28500030c0 tx=0x7f2850016000 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 msgr2=0x7f286016d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2858009fc0 tx=0x7f285800b040 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 shutdown_connections 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f284c060670 0x7f284c062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2860158cc0 0x7f286016d5b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28600b3870 0x7f2860088c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 --2- 172.21.15.50:0/801250895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f286006dc00 0x7f2860156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 >> 172.21.15.50:0/801250895 conn(0x7f2860067bb0 msgr2=0x7f28600a88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 shutdown_connections 2023-12-19T14:52:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.414+0000 7f28654f0700 1 -- 172.21.15.50:0/801250895 wait complete. 2023-12-19T14:52:04.419 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:52:04.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:52:04.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 Processor -- start 2023-12-19T14:52:04.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 -- start start 2023-12-19T14:52:04.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff20158e70 0x7eff200afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff200cc470 0x7eff200cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200ca5a0 con 0x7eff2006a540 2023-12-19T14:52:04.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2006dac0 con 0x7eff20158e70 2023-12-19T14:52:04.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff25c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff200b0260 con 0x7eff200cc470 2023-12-19T14:52:04.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff24c8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff20158e70 0x7eff200afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff1ffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7eff2006a540 0x7eff2006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38418/0 (socket says 172.21.15.50:38418) 2023-12-19T14:52:04.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff1ffff700 1 -- 172.21.15.50:0/2970855080 learned_addr learned my addr 172.21.15.50:0/2970855080 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:04.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff2548d700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff200cc470 0x7eff200cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff24c8c700 1 -- 172.21.15.50:0/2970855080 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff200cc470 msgr2=0x7eff200cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:04.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff24c8c700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff200cc470 0x7eff200cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff24c8c700 1 -- 172.21.15.50:0/2970855080 >> v1:172.21.15.50:6789/0 conn(0x7eff2006a540 legacy=0x7eff2006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff1f7fe700 1 -- 172.21.15.50:0/2970855080 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701193991 0 0) 0x7eff200ca5a0 con 0x7eff2006a540 2023-12-19T14:52:04.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.610+0000 7eff24c8c700 1 -- 172.21.15.50:0/2970855080 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2006d220 con 0x7eff20158e70 2023-12-19T14:52:04.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff24c8c700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff20158e70 0x7eff200afd20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7eff140098e0 tx=0x7eff140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d574ce70f96aeb1b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:04.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff1f7fe700 1 -- 172.21.15.50:0/2970855080 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14009e80 con 0x7eff20158e70 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff1f7fe700 1 -- 172.21.15.50:0/2970855080 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1400b860 con 0x7eff20158e70 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff1f7fe700 1 -- 172.21.15.50:0/2970855080 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1400bbe0 con 0x7eff20158e70 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 -- 172.21.15.50:0/2970855080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff20158e70 msgr2=0x7eff200afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff20158e70 0x7eff200afd20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7eff140098e0 tx=0x7eff140091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 -- 172.21.15.50:0/2970855080 shutdown_connections 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff200cc470 0x7eff200cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 --2- 172.21.15.50:0/2970855080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff20158e70 0x7eff200afd20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 -- 172.21.15.50:0/2970855080 >> 172.21.15.50:0/2970855080 conn(0x7eff200670e0 msgr2=0x7eff200674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 -- 172.21.15.50:0/2970855080 shutdown_connections 2023-12-19T14:52:04.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.614+0000 7eff25c8e700 1 -- 172.21.15.50:0/2970855080 wait complete. 2023-12-19T14:52:04.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 Processor -- start 2023-12-19T14:52:04.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- start start 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff200cc470 0x7eff2015d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff20158e70 0x7eff2015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200b2c70 con 0x7eff2006a540 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200b2f50 con 0x7eff200cc470 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2015b930 con 0x7eff20158e70 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46684/0 (socket says 172.21.15.50:46684) 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 -- 172.21.15.50:0/1302433096 learned_addr learned my addr 172.21.15.50:0/1302433096 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff2548d700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff20158e70 0x7eff2015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff1ffff700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff200cc470 0x7eff2015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 -- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff20158e70 msgr2=0x7eff2015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff20158e70 0x7eff2015dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 -- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff200cc470 msgr2=0x7eff2015d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff200cc470 0x7eff2015d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 -- 172.21.15.50:0/1302433096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14009420 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff24c8c700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7eff1400b3c0 tx=0x7eff14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1400bbe0 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14011420 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2015bc10 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140117a0 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.618+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff201555b0 con 0x7eff2006a540 2023-12-19T14:52:04.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.622+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2006cee0 con 0x7eff2006a540 2023-12-19T14:52:04.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.622+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7eff14011b90 con 0x7eff2006a540 2023-12-19T14:52:04.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.626+0000 7eff1cff9700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 0x7eff04062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:04.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.626+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5173+0+0 (secure 0 0 0) 0x7eff1407dcb0 con 0x7eff2006a540 2023-12-19T14:52:04.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.626+0000 7eff1ffff700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 0x7eff04062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:04.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.626+0000 7eff1ffff700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 0x7eff04062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff200afef0 tx=0x7eff0c009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:04.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.634+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7eff2006cee0 con 0x7eff2006a540 2023-12-19T14:52:04.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.862+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7eff2006cee0 con 0x7eff2006a540 2023-12-19T14:52:04.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.862+0000 7eff1cff9700 1 -- 172.21.15.50:0/1302433096 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7eff2006cee0 con 0x7eff2006a540 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.867 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:52:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.870 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.872 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:52:04.875 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:52:04.876 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.878 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:52:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:52:04.881 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 msgr2=0x7eff04062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 0x7eff04062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff200afef0 tx=0x7eff0c009040 comp rx=0 tx=0).stop 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 msgr2=0x7eff20155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7eff1400b3c0 tx=0x7eff14017000 comp rx=0 tx=0).stop 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 shutdown_connections 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff200cc470 0x7eff2015d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff040606a0 0x7eff04062b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff20158e70 0x7eff2015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 --2- 172.21.15.50:0/1302433096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff2006a540 0x7eff20155070 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:04.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 >> 172.21.15.50:0/1302433096 conn(0x7eff200670e0 msgr2=0x7eff200cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:04.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 shutdown_connections 2023-12-19T14:52:04.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:04.866+0000 7eff25c8e700 1 -- 172.21.15.50:0/1302433096 wait complete. 2023-12-19T14:52:04.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:52:05.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 Processor -- start 2023-12-19T14:52:05.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 -- start start 2023-12-19T14:52:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f06c81502b0 0x7f06c8068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f06c80c5900 0x7f06c80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c80c6630 con 0x7f06c8067c90 2023-12-19T14:52:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80c6910 con 0x7f06c81502b0 2023-12-19T14:52:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.062+0000 7f06cf6ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80c6bf0 con 0x7f06c80c5900 2023-12-19T14:52:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cd44a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f06c8067c90 0x7f06c814fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38420/0 (socket says 172.21.15.50:38420) 2023-12-19T14:52:05.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cd44a700 1 -- 172.21.15.50:0/2497118033 learned_addr learned my addr 172.21.15.50:0/2497118033 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cdc4b700 1 --2- 172.21.15.50:0/2497118033 >> v2:172.21.15.125:3300/0 conn(0x7f06c80c5900 0x7f06c80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2497118033 >> v2:172.21.15.50:3300/0 conn(0x7f06c81502b0 0x7f06c8068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06bffff700 1 -- 172.21.15.50:0/2497118033 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981233320 0 0) 0x7f06c80c6630 con 0x7f06c8067c90 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06bffff700 1 -- 172.21.15.50:0/2497118033 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b4003570 con 0x7f06c8067c90 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 -- 172.21.15.50:0/2497118033 >> v2:172.21.15.125:3300/0 conn(0x7f06c80c5900 msgr2=0x7f06c80c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2497118033 >> v2:172.21.15.125:3300/0 conn(0x7f06c80c5900 0x7f06c80c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 -- 172.21.15.50:0/2497118033 >> v1:172.21.15.50:6789/0 conn(0x7f06c8067c90 legacy=0x7f06c814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 -- 172.21.15.50:0/2497118033 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c80c6ed0 con 0x7f06c81502b0 2023-12-19T14:52:05.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2497118033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c81502b0 0x7f06c8068060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f06c8068a60 tx=0x7f06b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b44549af94000e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06bffff700 1 -- 172.21.15.50:0/2497118033 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8011630 con 0x7f06c81502b0 2023-12-19T14:52:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06bffff700 1 -- 172.21.15.50:0/2497118033 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b80117b0 con 0x7f06c81502b0 2023-12-19T14:52:05.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06bffff700 1 -- 172.21.15.50:0/2497118033 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8011b70 con 0x7f06c81502b0 2023-12-19T14:52:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2497118033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c81502b0 msgr2=0x7f06c8068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2497118033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c81502b0 0x7f06c8068060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f06c8068a60 tx=0x7f06b80091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2497118033 shutdown_connections 2023-12-19T14:52:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2497118033 >> v2:172.21.15.125:3300/0 conn(0x7f06c80c5900 0x7f06c80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2497118033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c81502b0 0x7f06c8068060 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.066+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2497118033 >> 172.21.15.50:0/2497118033 conn(0x7f06c80cd6d0 msgr2=0x7f06c806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2497118033 shutdown_connections 2023-12-19T14:52:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2497118033 wait complete. 2023-12-19T14:52:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 Processor -- start 2023-12-19T14:52:05.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- start start 2023-12-19T14:52:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06c80c5900 0x7f06c8169200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06c81502b0 0x7f06c81498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80b3da0 con 0x7f06c8067c90 2023-12-19T14:52:05.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80b4080 con 0x7f06c81502b0 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.070+0000 7f06cf6ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80b4360 con 0x7f06c80c5900 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46706/0 (socket says 172.21.15.50:46706) 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 -- 172.21.15.50:0/2948026838 learned_addr learned my addr 172.21.15.50:0/2948026838 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cdc4b700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06c81502b0 0x7f06c81498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06c80c5900 0x7f06c8169200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 -- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06c80c5900 msgr2=0x7f06c8169200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06c80c5900 0x7f06c8169200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 -- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06c81502b0 msgr2=0x7f06c81498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06c81502b0 0x7f06c81498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 -- 172.21.15.50:0/2948026838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b8009420 con 0x7f06c8067c90 2023-12-19T14:52:05.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cd44a700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7c0 tx=0x7f06c4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4005cd0 con 0x7f06c8067c90 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4005e30 con 0x7f06c8067c90 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4007320 con 0x7f06c8067c90 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c80b4640 con 0x7f06c8067c90 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c81513e0 con 0x7f06c8067c90 2023-12-19T14:52:05.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.074+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c8071120 con 0x7f06c8067c90 2023-12-19T14:52:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.078+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f06c401b3b0 con 0x7f06c8067c90 2023-12-19T14:52:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.078+0000 7f06bd7fa700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 0x7f06ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.078+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f06c407be60 con 0x7f06c8067c90 2023-12-19T14:52:05.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.078+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 0x7f06ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.082+0000 7f06ccc49700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 0x7f06ac062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06c8068a60 tx=0x7f06b8017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.082+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f06c404c4b0 con 0x7f06c8067c90 2023-12-19T14:52:05.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.258+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06c80642a0 con 0x7f06c8067c90 2023-12-19T14:52:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06bd7fa700 1 -- 172.21.15.50:0/2948026838 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12339 (secure 0 0 0) 0x7f06c4017020 con 0x7f06c8067c90 2023-12-19T14:52:05.263 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:05.263 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":64,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:52:04.403541+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:52:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 msgr2=0x7f06ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 0x7f06ac062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f06c8068a60 tx=0x7f06b8017000 comp rx=0 tx=0).stop 2023-12-19T14:52:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 msgr2=0x7f06c80b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7c0 tx=0x7f06c4007e00 comp rx=0 tx=0).stop 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 shutdown_connections 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06c81502b0 0x7f06c81498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f06ac060670 0x7f06ac062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06c80c5900 0x7f06c8169200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 --2- 172.21.15.50:0/2948026838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06c8067c90 0x7f06c80b37c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 >> 172.21.15.50:0/2948026838 conn(0x7f06c80cd6d0 msgr2=0x7f06c806d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.262+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 shutdown_connections 2023-12-19T14:52:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.266+0000 7f06cf6ae700 1 -- 172.21.15.50:0/2948026838 wait complete. 2023-12-19T14:52:05.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:52:05.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:52:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 Processor -- start 2023-12-19T14:52:05.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 -- start start 2023-12-19T14:52:05.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fca0c06db40 0x7fca0c0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca0c06df10 0x7fca0c159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0c9db0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0ca950 con 0x7fca0c06df10 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.458+0000 7fca11545700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0cb2e0 con 0x7fca0c06db40 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca0c06df10 0x7fca0c159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0affd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fca0c06db40 0x7fca0c0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fca0c06df10 0x7fca0c159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46708/0 (socket says 172.21.15.50:46708) 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 -- 172.21.15.50:0/300537551 learned_addr learned my addr 172.21.15.50:0/300537551 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca09ffb700 1 -- 172.21.15.50:0/300537551 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788700641 0 0) 0x7fca0c0c9db0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 -- 172.21.15.50:0/300537551 >> v2:172.21.15.125:3300/0 conn(0x7fca0c06db40 msgr2=0x7fca0c0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 --2- 172.21.15.50:0/300537551 >> v2:172.21.15.125:3300/0 conn(0x7fca0c06db40 0x7fca0c0cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 -- 172.21.15.50:0/300537551 >> v1:172.21.15.125:6789/0 conn(0x7fca0c0c63f0 legacy=0x7fca0c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 -- 172.21.15.50:0/300537551 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0cb5c0 con 0x7fca0c06df10 2023-12-19T14:52:05.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca0b7fe700 1 --2- 172.21.15.50:0/300537551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c06df10 0x7fca0c159900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fca0c067fe0 tx=0x7fca04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b1c9047543ccde server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca09ffb700 1 -- 172.21.15.50:0/300537551 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400e070 con 0x7fca0c06df10 2023-12-19T14:52:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca09ffb700 1 -- 172.21.15.50:0/300537551 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400ba20 con 0x7fca0c06df10 2023-12-19T14:52:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca09ffb700 1 -- 172.21.15.50:0/300537551 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040092e0 con 0x7fca0c06df10 2023-12-19T14:52:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 -- 172.21.15.50:0/300537551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c06df10 msgr2=0x7fca0c159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 --2- 172.21.15.50:0/300537551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c06df10 0x7fca0c159900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fca0c067fe0 tx=0x7fca04009840 comp rx=0 tx=0).stop 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 -- 172.21.15.50:0/300537551 shutdown_connections 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 --2- 172.21.15.50:0/300537551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c06df10 0x7fca0c159900 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 --2- 172.21.15.50:0/300537551 >> v2:172.21.15.125:3300/0 conn(0x7fca0c06db40 0x7fca0c0cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 -- 172.21.15.50:0/300537551 >> 172.21.15.50:0/300537551 conn(0x7fca0c067bb0 msgr2=0x7fca0c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.462+0000 7fca11545700 1 -- 172.21.15.50:0/300537551 shutdown_connections 2023-12-19T14:52:05.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.466+0000 7fca11545700 1 -- 172.21.15.50:0/300537551 wait complete. 2023-12-19T14:52:05.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.466+0000 7fca11545700 1 Processor -- start 2023-12-19T14:52:05.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.466+0000 7fca11545700 1 -- start start 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca0c06db40 0x7fca0c15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 0x7fca0c15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0c2d80 con 0x7fca0c0c63f0 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0c2fd0 con 0x7fca0c06df10 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0c32b0 con 0x7fca0c06db40 2023-12-19T14:52:05.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46720/0 (socket says 172.21.15.50:46720) 2023-12-19T14:52:05.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 -- 172.21.15.50:0/3695779415 learned_addr learned my addr 172.21.15.50:0/3695779415 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:05.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0affd700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca0c06db40 0x7fca0c15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0a7fc700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 0x7fca0c15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 -- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca0c06db40 msgr2=0x7fca0c15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca0c06db40 0x7fca0c15f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 -- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 msgr2=0x7fca0c15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 0x7fca0c15a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 -- 172.21.15.50:0/3695779415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca04009ae0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0a7fc700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 0x7fca0c15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca0b7fe700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fca04009880 tx=0x7fca04018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04008b70 con 0x7fca0c0c63f0 2023-12-19T14:52:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04008cf0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040266b0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c0c3590 con 0x7fca0c0c63f0 2023-12-19T14:52:05.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.470+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c0c3b50 con 0x7fca0c0c63f0 2023-12-19T14:52:05.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.474+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c0c3ce0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.474+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fca04003040 con 0x7fca0c0c63f0 2023-12-19T14:52:05.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.474+0000 7fc9f2ffd700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 0x7fc9f8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.474+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fca0407e1a0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.474+0000 7fca0affd700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 0x7fc9f8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.478+0000 7fca0affd700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 0x7fc9f8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009d10 tx=0x7fc9fc0091a0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.482+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca0404e6e0 con 0x7fca0c0c63f0 2023-12-19T14:52:05.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.650+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca0c071120 con 0x7fca0c0c63f0 2023-12-19T14:52:05.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.650+0000 7fc9f2ffd700 1 -- 172.21.15.50:0/3695779415 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca04051d00 con 0x7fca0c0c63f0 2023-12-19T14:52:05.653 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 msgr2=0x7fc9f8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 0x7fc9f8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009d10 tx=0x7fc9fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 msgr2=0x7fca0c0c4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fca04009880 tx=0x7fca04018000 comp rx=0 tx=0).stop 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 shutdown_connections 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fca0c06df10 0x7fca0c15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc9f8060670 0x7fc9f8062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca0c0c63f0 0x7fca0c0c4050 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 --2- 172.21.15.50:0/3695779415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fca0c06db40 0x7fca0c15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 >> 172.21.15.50:0/3695779415 conn(0x7fca0c067bb0 msgr2=0x7fca0c0c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 shutdown_connections 2023-12-19T14:52:05.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.654+0000 7fca11545700 1 -- 172.21.15.50:0/3695779415 wait complete. 2023-12-19T14:52:05.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:52:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.842+0000 7f6812f85700 1 Processor -- start 2023-12-19T14:52:05.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.842+0000 7f6812f85700 1 -- start start 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f680c140ae0 0x7f680c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c068630 0x7f680c0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c06e150 con 0x7f680c140710 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c068ab0 con 0x7f680c140ae0 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c06d540 con 0x7f680c068630 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f680c140ae0 0x7f680c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f680c140ae0 0x7f680c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46730/0 (socket says 172.21.15.50:46730) 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 -- 172.21.15.50:0/1762284760 learned_addr learned my addr 172.21.15.50:0/1762284760 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6811522700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c068630 0x7f680c0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6810d21700 1 --1- 172.21.15.50:0/1762284760 >> v1:172.21.15.50:6789/0 conn(0x7f680c140710 0x7f680c0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38432/0 (socket says 172.21.15.50:38432) 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 -- 172.21.15.50:0/1762284760 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c068630 msgr2=0x7f680c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c068630 0x7f680c0ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 -- 172.21.15.50:0/1762284760 >> v1:172.21.15.50:6789/0 conn(0x7f680c140710 legacy=0x7f680c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 -- 172.21.15.50:0/1762284760 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c06d820 con 0x7f680c140ae0 2023-12-19T14:52:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680bfff700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c140ae0 0x7f680c0680f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f67f80099b0 tx=0x7f67f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa48d7319f309d52 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680b7fe700 1 -- 172.21.15.50:0/1762284760 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8009ea0 con 0x7f680c140ae0 2023-12-19T14:52:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680b7fe700 1 -- 172.21.15.50:0/1762284760 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f800b860 con 0x7f680c140ae0 2023-12-19T14:52:05.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f680b7fe700 1 -- 172.21.15.50:0/1762284760 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f800bc00 con 0x7f680c140ae0 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- 172.21.15.50:0/1762284760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c140ae0 msgr2=0x7f680c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c140ae0 0x7f680c0680f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f67f80099b0 tx=0x7f67f8009250 comp rx=0 tx=0).stop 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- 172.21.15.50:0/1762284760 shutdown_connections 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c068630 0x7f680c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 --2- 172.21.15.50:0/1762284760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c140ae0 0x7f680c0680f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.846+0000 7f6812f85700 1 -- 172.21.15.50:0/1762284760 >> 172.21.15.50:0/1762284760 conn(0x7f680c0c8850 msgr2=0x7f680c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 -- 172.21.15.50:0/1762284760 shutdown_connections 2023-12-19T14:52:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 -- 172.21.15.50:0/1762284760 wait complete. 2023-12-19T14:52:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 Processor -- start 2023-12-19T14:52:05.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 -- start start 2023-12-19T14:52:05.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.850+0000 7f6812f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c140710 0x7f680c0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 0x7f680c13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0aeba0 con 0x7f680c068630 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0af720 con 0x7f680c140ae0 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0aedf0 con 0x7f680c140710 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46740/0 (socket says 172.21.15.50:46740) 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 -- 172.21.15.50:0/693945567 learned_addr learned my addr 172.21.15.50:0/693945567 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:05.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f680bfff700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c140710 0x7f680c0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6811522700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 0x7f680c13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 -- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c140710 msgr2=0x7f680c0cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c140710 0x7f680c0cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 -- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 msgr2=0x7f680c13dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 0x7f680c13dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 -- 172.21.15.50:0/693945567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80094f0 con 0x7f680c068630 2023-12-19T14:52:05.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6811522700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 0x7f680c13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6810d21700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f680000c590 tx=0x7f6800007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800005a70 con 0x7f680c068630 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6800005bd0 con 0x7f680c068630 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68000070d0 con 0x7f680c068630 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c0af0d0 con 0x7f680c068630 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.854+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c156c90 con 0x7f680c068630 2023-12-19T14:52:05.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.858+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c071120 con 0x7f680c068630 2023-12-19T14:52:05.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.858+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6800007480 con 0x7f680c068630 2023-12-19T14:52:05.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.858+0000 7f6808ff9700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 0x7f67f0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:05.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.862+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f680007bbc0 con 0x7f680c068630 2023-12-19T14:52:05.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.862+0000 7f680bfff700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 0x7f67f0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:05.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.862+0000 7f680bfff700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 0x7f67f0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67f80099b0 tx=0x7f67f8017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:05.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:05.866+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f680004c210 con 0x7f680c068630 2023-12-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.042+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f680c0642a0 con 0x7f680c068630 2023-12-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.042+0000 7f6808ff9700 1 -- 172.21.15.50:0/693945567 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6800017090 con 0x7f680c068630 2023-12-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:52:06.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:06.047 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:52:06.047 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:52:06.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:06.047 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 msgr2=0x7f67f0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 0x7f67f0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67f80099b0 tx=0x7f67f8017000 comp rx=0 tx=0).stop 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 msgr2=0x7f680c0cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f680000c590 tx=0x7f6800007f70 comp rx=0 tx=0).stop 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 shutdown_connections 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f680c140ae0 0x7f680c13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f67f0060670 0x7f67f0062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f680c140710 0x7f680c0cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 --2- 172.21.15.50:0/693945567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f680c068630 0x7f680c0cae20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 >> 172.21.15.50:0/693945567 conn(0x7f680c0c8850 msgr2=0x7f680c06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 shutdown_connections 2023-12-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.046+0000 7f6812f85700 1 -- 172.21.15.50:0/693945567 wait complete. 2023-12-19T14:52:06.070 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:52:06.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:52:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.238+0000 7fdb1e9c5700 1 Processor -- start 2023-12-19T14:52:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.238+0000 7fdb1e9c5700 1 -- start start 2023-12-19T14:52:06.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb1e9c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb180c4dc0 0x7fdb180c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:06.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb1e9c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180c6270 con 0x7fdb1806c7c0 2023-12-19T14:52:06.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb1e9c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180c5750 con 0x7fdb18142910 2023-12-19T14:52:06.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb1e9c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1806a6a0 con 0x7fdb180c4dc0 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb180c4dc0 0x7fdb180c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb180c4dc0 0x7fdb180c5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46752/0 (socket says 172.21.15.50:46752) 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 -- 172.21.15.50:0/2962932041 learned_addr learned my addr 172.21.15.50:0/2962932041 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 -- 172.21.15.50:0/2962932041 >> v1:172.21.15.125:6789/0 conn(0x7fdb18142910 legacy=0x7fdb1806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 -- 172.21.15.50:0/2962932041 >> v1:172.21.15.50:6789/0 conn(0x7fdb1806c7c0 legacy=0x7fdb1806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 -- 172.21.15.50:0/2962932041 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1806a880 con 0x7fdb180c4dc0 2023-12-19T14:52:06.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb177fe700 1 --2- 172.21.15.50:0/2962932041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb180c4dc0 0x7fdb180c5210 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0099b0 tx=0x7fdb0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6717f01f0cd04d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:06.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.242+0000 7fdb16ffd700 1 -- 172.21.15.50:0/2962932041 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00b860 con 0x7fdb180c4dc0 2023-12-19T14:52:06.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb16ffd700 1 -- 172.21.15.50:0/2962932041 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c00b9c0 con 0x7fdb180c4dc0 2023-12-19T14:52:06.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb16ffd700 1 -- 172.21.15.50:0/2962932041 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00bd60 con 0x7fdb180c4dc0 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/2962932041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb180c4dc0 msgr2=0x7fdb180c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/2962932041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb180c4dc0 0x7fdb180c5210 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0099b0 tx=0x7fdb0c009250 comp rx=0 tx=0).stop 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/2962932041 shutdown_connections 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/2962932041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb180c4dc0 0x7fdb180c5210 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/2962932041 >> 172.21.15.50:0/2962932041 conn(0x7fdb18067bb0 msgr2=0x7fdb180cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/2962932041 shutdown_connections 2023-12-19T14:52:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/2962932041 wait complete. 2023-12-19T14:52:06.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 Processor -- start 2023-12-19T14:52:06.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.246+0000 7fdb1e9c5700 1 -- start start 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb180c4dc0 0x7fdb18157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb18142910 0x7fdb1816a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18157dd0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1815b620 con 0x7fdb180c4dc0 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1815ce40 con 0x7fdb18142910 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46756/0 (socket says 172.21.15.50:46756) 2023-12-19T14:52:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 -- 172.21.15.50:0/842043905 learned_addr learned my addr 172.21.15.50:0/842043905 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1cf62700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb18142910 0x7fdb1816a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb177fe700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb180c4dc0 0x7fdb18157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 -- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb18142910 msgr2=0x7fdb1816a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb18142910 0x7fdb1816a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 -- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb180c4dc0 msgr2=0x7fdb18157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb180c4dc0 0x7fdb18157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 -- 172.21.15.50:0/842043905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0094f0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb17fff700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800c510 tx=0x7fdb0800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08011590 con 0x7fdb1806c7c0 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb08011710 con 0x7fdb1806c7c0 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1815b870 con 0x7fdb1806c7c0 2023-12-19T14:52:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.250+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08011ad0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.254+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb18169e10 con 0x7fdb1806c7c0 2023-12-19T14:52:06.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.254+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb18071120 con 0x7fdb1806c7c0 2023-12-19T14:52:06.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.254+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdb0801b3b0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.258+0000 7fdafffff700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 0x7fdb04062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:06.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.258+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fdb0807bb50 con 0x7fdb1806c7c0 2023-12-19T14:52:06.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.258+0000 7fdb177fe700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 0x7fdb04062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:06.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.258+0000 7fdb177fe700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 0x7fdb04062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0099b0 tx=0x7fdb0c017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:06.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.262+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdb0804c1a0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.434+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb1806b7f0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.986+0000 7fdafffff700 1 -- 172.21.15.50:0/842043905 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb0804f7c0 con 0x7fdb1806c7c0 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 msgr2=0x7fdb04062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 0x7fdb04062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0099b0 tx=0x7fdb0c017000 comp rx=0 tx=0).stop 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 msgr2=0x7fdb18139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800c510 tx=0x7fdb0800c820 comp rx=0 tx=0).stop 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 shutdown_connections 2023-12-19T14:52:06.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb180c4dc0 0x7fdb18157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb04060670 0x7fdb04062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb18142910 0x7fdb1816a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 --2- 172.21.15.50:0/842043905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb1806c7c0 0x7fdb18139410 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 >> 172.21.15.50:0/842043905 conn(0x7fdb18067bb0 msgr2=0x7fdb180caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 shutdown_connections 2023-12-19T14:52:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:06.994+0000 7fdb1e9c5700 1 -- 172.21.15.50:0/842043905 wait complete. 2023-12-19T14:52:06.999 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:52:07.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:52:07.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:52:07.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:52:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 Processor -- start 2023-12-19T14:52:07.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 -- start start 2023-12-19T14:52:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2160158e20 0x7f21600ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216006d5e0 con 0x7f21600cb970 2023-12-19T14:52:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21600cbdf0 con 0x7f21600ca2c0 2023-12-19T14:52:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f2165345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21600cc0d0 con 0x7f2160158e20 2023-12-19T14:52:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f215e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f21600cb970 0x7f216006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57788/0 (socket says 172.21.15.50:57788) 2023-12-19T14:52:07.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f215e7fc700 1 -- 172.21.15.50:0/1604959060 learned_addr learned my addr 172.21.15.50:0/1604959060 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.286+0000 7f215effd700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2160158e20 0x7f21600ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221615623 0 0) 0x7f216006d5e0 con 0x7f21600cb970 2023-12-19T14:52:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2148003570 con 0x7f21600cb970 2023-12-19T14:52:07.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847220487 0 0) 0x7f21600cbdf0 con 0x7f21600ca2c0 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f216006d5e0 con 0x7f21600ca2c0 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227479370 0 0) 0x7f2148003570 con 0x7f21600cb970 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2160158e20 msgr2=0x7f21600ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2160158e20 0x7f21600ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 >> v1:172.21.15.125:6789/0 conn(0x7f21600ca2c0 legacy=0x7f216006ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600cc3b0 con 0x7f21600cb970 2023-12-19T14:52:07.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f2154001470 con 0x7f21600cb970 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 >> v1:172.21.15.50:6789/0 conn(0x7f21600cb970 legacy=0x7f216006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2148003c20 0x7f2148004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2148004810 0x7f2148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21480051c0 0x7f214800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21600cc0d0 con 0x7f2148003c20 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214800c0b0 con 0x7f2148004810 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214800c270 con 0x7f21480051c0 2023-12-19T14:52:07.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215effd700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2160158e20 0x7f21600ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215e7fc700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2148004810 0x7f2148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21480051c0 0x7f214800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215effd700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2148003c20 0x7f2148004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 -- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2148004810 msgr2=0x7f2148004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2148004810 0x7f2148004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 -- 172.21.15.50:0/1604959060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2148003c20 msgr2=0x7f2148004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2148003c20 0x7f2148004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 -- 172.21.15.50:0/1604959060 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148003570 con 0x7f21480051c0 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215effd700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2148003c20 0x7f2148004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215e7fc700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2148004810 0x7f2148004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215f7fe700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21480051c0 0x7f214800ba60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2158002110 tx=0x7f21580017c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d781df2dc0c27ad8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.290+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158001ad0 con 0x7f21480051c0 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215800c980 con 0x7f21480051c0 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f215dffb700 1 -- 172.21.15.50:0/1604959060 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215800cd40 con 0x7f21480051c0 2023-12-19T14:52:07.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 -- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21480051c0 msgr2=0x7f214800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 --2- 172.21.15.50:0/1604959060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21480051c0 0x7f214800ba60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2158002110 tx=0x7f21580017c0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f215f7fe700 1 -- 172.21.15.50:0/1604959060 reap_dead start 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 -- 172.21.15.50:0/1604959060 shutdown_connections 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 -- 172.21.15.50:0/1604959060 >> 172.21.15.50:0/1604959060 conn(0x7f2160067bb0 msgr2=0x7f21600d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 -- 172.21.15.50:0/1604959060 shutdown_connections 2023-12-19T14:52:07.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.294+0000 7f2165345700 1 -- 172.21.15.50:0/1604959060 wait complete. 2023-12-19T14:52:07.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 Processor -- start 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 -- start start 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 0x7f2160169570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21600c94a0 0x7f216014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21600c9910 con 0x7f216016a720 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21600a8980 con 0x7f21480051c0 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2165345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21600c9fd0 con 0x7f21600c94a0 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33480/0 (socket says 172.21.15.50:33480) 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 -- 172.21.15.50:0/2533909446 learned_addr learned my addr 172.21.15.50:0/2533909446 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215e7fc700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21600c94a0 0x7f216014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215effd700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 0x7f2160169570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 -- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21600c94a0 msgr2=0x7f216014dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21600c94a0 0x7f216014dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 -- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 msgr2=0x7f2160169570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 0x7f2160169570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 -- 172.21.15.50:0/2533909446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216014e0e0 con 0x7f216016a720 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215effd700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 0x7f2160169570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f215f7fe700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f215800c6a0 tx=0x7f2158016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215800c6d0 con 0x7f216016a720 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.298+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2158012e80 con 0x7f216016a720 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.302+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2160169ba0 con 0x7f216016a720 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.302+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215801f440 con 0x7f216016a720 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.302+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216016a160 con 0x7f216016a720 2023-12-19T14:52:07.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.302+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216006e990 con 0x7f216016a720 2023-12-19T14:52:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.302+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2158025020 con 0x7f216016a720 2023-12-19T14:52:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.306+0000 7f2146ffd700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 0x7f214c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.306+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5543+0+0 (secure 0 0 0) 0x7f215807e6f0 con 0x7f216016a720 2023-12-19T14:52:07.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.306+0000 7f215effd700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 0x7f214c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:07.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.306+0000 7f215effd700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 0x7f214c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f21600cdc10 tx=0x7f2150008040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:07.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.314+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f216006e990 con 0x7f216016a720 2023-12-19T14:52:07.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.486+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f216006e990 con 0x7f216016a720 2023-12-19T14:52:08.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:07.998+0000 7f2146ffd700 1 -- 172.21.15.50:0/2533909446 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7f215804ec90 con 0x7f216016a720 2023-12-19T14:52:08.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.010+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 msgr2=0x7f214c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.010+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 0x7f214c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f21600cdc10 tx=0x7f2150008040 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.010+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 msgr2=0x7f21600c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.010+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f215800c6a0 tx=0x7f2158016000 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 shutdown_connections 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f214c0606a0 0x7f214c062b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f216016a720 0x7f21600c6390 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21600c94a0 0x7f216014dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 --2- 172.21.15.50:0/2533909446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f21480051c0 0x7f2160169570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 >> 172.21.15.50:0/2533909446 conn(0x7f2160067bb0 msgr2=0x7f216006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 shutdown_connections 2023-12-19T14:52:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.014+0000 7f2165345700 1 -- 172.21.15.50:0/2533909446 wait complete. 2023-12-19T14:52:08.028 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:52:08.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:52:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.218+0000 7fd9e6123700 1 Processor -- start 2023-12-19T14:52:08.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.218+0000 7fd9e6123700 1 -- start start 2023-12-19T14:52:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e6123700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd9e006baa0 0x7fd9e0157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e6123700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e015ae30 con 0x7fd9e00adab0 2023-12-19T14:52:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e6123700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e00abf70 con 0x7fd9e006baa0 2023-12-19T14:52:08.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0154b50 con 0x7fd9e0154780 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e5922700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd9e00adab0 0x7fd9e00adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57796/0 (socket says 172.21.15.50:57796) 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e5922700 1 -- 172.21.15.50:0/2432413100 learned_addr learned my addr 172.21.15.50:0/2432413100 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e4920700 1 --2- 172.21.15.50:0/2432413100 >> v2:172.21.15.125:3300/0 conn(0x7fd9e006baa0 0x7fd9e0157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e5121700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9d7fff700 1 -- 172.21.15.50:0/2432413100 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233758317 0 0) 0x7fd9e015ae30 con 0x7fd9e00adab0 2023-12-19T14:52:08.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e4920700 1 -- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 msgr2=0x7fd9e015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e4920700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e4920700 1 -- 172.21.15.50:0/2432413100 >> v1:172.21.15.50:6789/0 conn(0x7fd9e00adab0 legacy=0x7fd9e00adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.222+0000 7fd9e4920700 1 -- 172.21.15.50:0/2432413100 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e01580a0 con 0x7fd9e006baa0 2023-12-19T14:52:08.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e4920700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e0157b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098e0 tx=0x7fd9d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5468230e300beadf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9d7fff700 1 -- 172.21.15.50:0/2432413100 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000b640 con 0x7fd9e006baa0 2023-12-19T14:52:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9d7fff700 1 -- 172.21.15.50:0/2432413100 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d000b7a0 con 0x7fd9e006baa0 2023-12-19T14:52:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9d7fff700 1 -- 172.21.15.50:0/2432413100 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000bb40 con 0x7fd9e006baa0 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 -- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 msgr2=0x7fd9e0157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e0157b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098e0 tx=0x7fd9d00091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 -- 172.21.15.50:0/2432413100 shutdown_connections 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e0157b60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 --2- 172.21.15.50:0/2432413100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 -- 172.21.15.50:0/2432413100 >> 172.21.15.50:0/2432413100 conn(0x7fd9e0067b50 msgr2=0x7fd9e00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 -- 172.21.15.50:0/2432413100 shutdown_connections 2023-12-19T14:52:08.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.226+0000 7fd9e6123700 1 -- 172.21.15.50:0/2432413100 wait complete. 2023-12-19T14:52:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 Processor -- start 2023-12-19T14:52:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 -- start start 2023-12-19T14:52:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 0x7fd9e00af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e01442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00b0750 con 0x7fd9e00adab0 2023-12-19T14:52:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00af8a0 con 0x7fd9e006baa0 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00afb40 con 0x7fd9e0154780 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e4920700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 0x7fd9e00af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e4920700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 0x7fd9e00af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33488/0 (socket says 172.21.15.50:33488) 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e4920700 1 -- 172.21.15.50:0/3491183351 learned_addr learned my addr 172.21.15.50:0/3491183351 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:08.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5922700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e01442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 -- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 msgr2=0x7fd9e01442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e01442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 -- 172.21.15.50:0/3491183351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 msgr2=0x7fd9e00af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 0x7fd9e00af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.230+0000 7fd9e5121700 1 -- 172.21.15.50:0/3491183351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0009420 con 0x7fd9e006baa0 2023-12-19T14:52:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9e5121700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e015bb90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00e900 tx=0x7fd9dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc013680 con 0x7fd9e006baa0 2023-12-19T14:52:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc0137e0 con 0x7fd9e006baa0 2023-12-19T14:52:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e00afe20 con 0x7fd9e006baa0 2023-12-19T14:52:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc013ba0 con 0x7fd9e006baa0 2023-12-19T14:52:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e00b03e0 con 0x7fd9e006baa0 2023-12-19T14:52:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e00cc880 con 0x7fd9e006baa0 2023-12-19T14:52:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.234+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd9dc0123d0 con 0x7fd9e006baa0 2023-12-19T14:52:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.238+0000 7fd9d57fa700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 0x7fd9c4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.238+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5925+0+0 (secure 0 0 0) 0x7fd9dc07cb30 con 0x7fd9e006baa0 2023-12-19T14:52:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.238+0000 7fd9e4920700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 0x7fd9c4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:08.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.238+0000 7fd9e4920700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 0x7fd9c4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0152df0 tx=0x7fd9d0017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.250+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd9e00cc880 con 0x7fd9e006baa0 2023-12-19T14:52:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:08.462+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd9e00cc880 con 0x7fd9e006baa0 2023-12-19T14:52:09.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.038+0000 7fd9d57fa700 1 -- 172.21.15.50:0/3491183351 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd9e00cc880 con 0x7fd9e006baa0 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 msgr2=0x7fd9c4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 0x7fd9c4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0152df0 tx=0x7fd9d0017000 comp rx=0 tx=0).stop 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 msgr2=0x7fd9e015bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e015bb90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00e900 tx=0x7fd9dc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 shutdown_connections 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd9c4060670 0x7fd9c4062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9e0154780 0x7fd9e01442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9e00adab0 0x7fd9e00af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 --2- 172.21.15.50:0/3491183351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9e006baa0 0x7fd9e015bb90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 >> 172.21.15.50:0/3491183351 conn(0x7fd9e0067b50 msgr2=0x7fd9e00c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 shutdown_connections 2023-12-19T14:52:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.062+0000 7fd9e6123700 1 -- 172.21.15.50:0/3491183351 wait complete. 2023-12-19T14:52:09.067 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 7 and data pool 8 2023-12-19T14:52:09.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:52:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 Processor -- start 2023-12-19T14:52:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 -- start start 2023-12-19T14:52:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7c0cb750 con 0x7f8a7c15d230 2023-12-19T14:52:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7c0cc5a0 con 0x7f8a7c0ca310 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a818f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7c0c8070 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a73fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a73fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33504/0 (socket says 172.21.15.50:33504) 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.250+0000 7f8a73fff700 1 -- 172.21.15.50:0/817945642 learned_addr learned my addr 172.21.15.50:0/817945642 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a73fff700 1 -- 172.21.15.50:0/817945642 >> v1:172.21.15.125:6789/0 conn(0x7f8a7c0ca310 legacy=0x7f8a7c0c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a73fff700 1 -- 172.21.15.50:0/817945642 >> v1:172.21.15.50:6789/0 conn(0x7f8a7c15d230 legacy=0x7f8a7c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a73fff700 1 -- 172.21.15.50:0/817945642 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c0ca710 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a73fff700 1 --2- 172.21.15.50:0/817945642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8a640099b0 tx=0x7f8a64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f325cf2a21f42baf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a7a7fc700 1 -- 172.21.15.50:0/817945642 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400b820 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a7a7fc700 1 -- 172.21.15.50:0/817945642 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6400b980 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a7a7fc700 1 -- 172.21.15.50:0/817945642 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400bd20 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 -- 172.21.15.50:0/817945642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 msgr2=0x7f8a7c0c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 --2- 172.21.15.50:0/817945642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8a640099b0 tx=0x7f8a64009250 comp rx=0 tx=0).stop 2023-12-19T14:52:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 -- 172.21.15.50:0/817945642 shutdown_connections 2023-12-19T14:52:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 --2- 172.21.15.50:0/817945642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c0c9dd0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 -- 172.21.15.50:0/817945642 >> 172.21.15.50:0/817945642 conn(0x7f8a7c068500 msgr2=0x7f8a7c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 -- 172.21.15.50:0/817945642 shutdown_connections 2023-12-19T14:52:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.254+0000 7f8a818f0700 1 -- 172.21.15.50:0/817945642 wait complete. 2023-12-19T14:52:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.258+0000 7f8a818f0700 1 Processor -- start 2023-12-19T14:52:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.258+0000 7f8a818f0700 1 -- start start 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7c0ca310 0x7f8a7c158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 0x7f8a7c14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c15c350 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c0c5480 con 0x7f8a7c15d230 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c0c46c0 con 0x7f8a7c0ca310 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33520/0 (socket says 172.21.15.50:33520) 2023-12-19T14:52:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 -- 172.21.15.50:0/3925173562 learned_addr learned my addr 172.21.15.50:0/3925173562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7bfff700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 0x7f8a7c14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a73fff700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7c0ca310 0x7f8a7c158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 -- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7c0ca310 msgr2=0x7f8a7c158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7c0ca310 0x7f8a7c158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 -- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 msgr2=0x7f8a7c14f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 0x7f8a7c14f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 -- 172.21.15.50:0/3925173562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a640094f0 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7bfff700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 0x7f8a7c14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a7affd700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c510 tx=0x7f8a6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011590 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c011710 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011ad0 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7c0c49a0 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.262+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7c0c4f60 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.266+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7c071120 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.266+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8a6c01b3b0 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.266+0000 7f8a72ffd700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 0x7f8a5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.266+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f8a6c07be80 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.270+0000 7f8a73fff700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 0x7f8a5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.270+0000 7f8a73fff700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 0x7f8a5c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a640099b0 tx=0x7f8a64016000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.274+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a6c04c180 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.450+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a7c0cb750 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.454+0000 7f8a72ffd700 1 -- 172.21.15.50:0/3925173562 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+15568 (secure 0 0 0) 0x7f8a7c0cb750 con 0x7f8a7c06d8e0 2023-12-19T14:52:09.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:09.457 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":67,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:52:08.468884+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:52:09.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 msgr2=0x7f8a5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 0x7f8a5c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a640099b0 tx=0x7f8a64016000 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 msgr2=0x7f8a7c15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c510 tx=0x7f8a6c00c8d0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 shutdown_connections 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a7c15d230 0x7f8a7c14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a5c060670 0x7f8a5c062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7c0ca310 0x7f8a7c158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 --2- 172.21.15.50:0/3925173562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7c06d8e0 0x7f8a7c15bb20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 >> 172.21.15.50:0/3925173562 conn(0x7f8a7c068500 msgr2=0x7f8a7c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 shutdown_connections 2023-12-19T14:52:09.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.458+0000 7f8a818f0700 1 -- 172.21.15.50:0/3925173562 wait complete. 2023-12-19T14:52:09.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:52:09.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.638+0000 7f354d2ae700 1 Processor -- start 2023-12-19T14:52:09.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.638+0000 7f354d2ae700 1 -- start start 2023-12-19T14:52:09.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f354d2ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35480c15b0 0x7f354815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f354d2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f35480680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f354d2ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480c7190 con 0x7f354815d320 2023-12-19T14:52:09.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f354d2ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35480aefd0 con 0x7f35480c15b0 2023-12-19T14:52:09.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f354d2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35480c1b50 con 0x7f35480aeba0 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35480c15b0 0x7f354815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35480c15b0 0x7f354815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33526/0 (socket says 172.21.15.50:33526) 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 -- 172.21.15.50:0/3968511 learned_addr learned my addr 172.21.15.50:0/3968511 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f35477fe700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f35480680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 -- 172.21.15.50:0/3968511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 msgr2=0x7f35480680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f35480680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 -- 172.21.15.50:0/3968511 >> v1:172.21.15.125:6789/0 conn(0x7f354815d320 legacy=0x7f354806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 -- 172.21.15.50:0/3968511 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480c1e30 con 0x7f35480c15b0 2023-12-19T14:52:09.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.642+0000 7f353e7fc700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35480c15b0 0x7f354815d6f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35300099b0 tx=0x7f3530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e939a199e85e45e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f35467fc700 1 -- 172.21.15.50:0/3968511 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353000b860 con 0x7f35480c15b0 2023-12-19T14:52:09.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f35467fc700 1 -- 172.21.15.50:0/3968511 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353000b9c0 con 0x7f35480c15b0 2023-12-19T14:52:09.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f35467fc700 1 -- 172.21.15.50:0/3968511 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353000bd60 con 0x7f35480c15b0 2023-12-19T14:52:09.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 -- 172.21.15.50:0/3968511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35480c15b0 msgr2=0x7f354815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35480c15b0 0x7f354815d6f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35300099b0 tx=0x7f3530009250 comp rx=0 tx=0).stop 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 -- 172.21.15.50:0/3968511 shutdown_connections 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f35480680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 --2- 172.21.15.50:0/3968511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35480c15b0 0x7f354815d6f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 -- 172.21.15.50:0/3968511 >> 172.21.15.50:0/3968511 conn(0x7f35480c8850 msgr2=0x7f35480c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 -- 172.21.15.50:0/3968511 shutdown_connections 2023-12-19T14:52:09.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.646+0000 7f354d2ae700 1 -- 172.21.15.50:0/3968511 wait complete. 2023-12-19T14:52:09.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 Processor -- start 2023-12-19T14:52:09.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 -- start start 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f3548067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35480c15b0 0x7f35480c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35480c4110 con 0x7f354815d320 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3548155720 con 0x7f35480c15b0 2023-12-19T14:52:09.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.650+0000 7f354d2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35481559c0 con 0x7f35480aeba0 2023-12-19T14:52:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f3546ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f3548067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33540/0 (socket says 172.21.15.50:33540) 2023-12-19T14:52:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 -- 172.21.15.50:0/595248562 learned_addr learned my addr 172.21.15.50:0/595248562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f353e7fc700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35480c15b0 0x7f35480c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 -- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 msgr2=0x7f3548067ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f3548067ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 -- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35480c15b0 msgr2=0x7f35480c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35480c15b0 0x7f35480c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 -- 172.21.15.50:0/595248562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35300094f0 con 0x7f354815d320 2023-12-19T14:52:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f35477fe700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3540007f20 tx=0x7f3540009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354000eb90 con 0x7f354815d320 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354000ed10 con 0x7f354815d320 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35480c3870 con 0x7f354815d320 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540014670 con 0x7f354815d320 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35480c3eb0 con 0x7f354815d320 2023-12-19T14:52:09.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.654+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3548071120 con 0x7f354815d320 2023-12-19T14:52:09.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.658+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f35400149e0 con 0x7f354815d320 2023-12-19T14:52:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.658+0000 7f353f7fe700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 0x7f3528062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.658+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f354007d050 con 0x7f354815d320 2023-12-19T14:52:09.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.658+0000 7f3546ffd700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 0x7f3528062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:09.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.662+0000 7f3546ffd700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 0x7f3528062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3538003a80 tx=0x7f3538007ec0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:09.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.666+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f35400acbe0 con 0x7f354815d320 2023-12-19T14:52:09.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.838+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35480642a0 con 0x7f354815d320 2023-12-19T14:52:09.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.842+0000 7f353f7fe700 1 -- 172.21.15.50:0/595248562 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+15568 (secure 0 0 0) 0x7f354004d350 con 0x7f354815d320 2023-12-19T14:52:09.845 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:09.845 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":67,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:52:08.468884+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 msgr2=0x7f3528062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 0x7f3528062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3538003a80 tx=0x7f3538007ec0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 msgr2=0x7f35481541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3540007f20 tx=0x7f3540009860 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 shutdown_connections 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35480c15b0 0x7f35480c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f35280606a0 0x7f3528062b50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f354815d320 0x7f35481541d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 --2- 172.21.15.50:0/595248562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35480aeba0 0x7f3548067ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 >> 172.21.15.50:0/595248562 conn(0x7f35480c8850 msgr2=0x7f35480ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 shutdown_connections 2023-12-19T14:52:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:09.846+0000 7f354d2ae700 1 -- 172.21.15.50:0/595248562 wait complete. 2023-12-19T14:52:09.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:52:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.038+0000 7fc2edc4c700 1 Processor -- start 2023-12-19T14:52:10.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.038+0000 7fc2edc4c700 1 -- start start 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.038+0000 7fc2edc4c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2e80c7170 0x7fc2e80ac720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.038+0000 7fc2edc4c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80c7610 con 0x7fc2e80acc60 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.038+0000 7fc2edc4c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80ad080 con 0x7fc2e8150200 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80af530 con 0x7fc2e80c7170 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2e80c7170 0x7fc2e80ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:10.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc2e80c7170 0x7fc2e80ac720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33554/0 (socket says 172.21.15.50:33554) 2023-12-19T14:52:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 -- 172.21.15.50:0/2143335120 learned_addr learned my addr 172.21.15.50:0/2143335120 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 -- 172.21.15.50:0/2143335120 >> v1:172.21.15.125:6789/0 conn(0x7fc2e8150200 legacy=0x7fc2e80ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 -- 172.21.15.50:0/2143335120 >> v1:172.21.15.50:6789/0 conn(0x7fc2e80acc60 legacy=0x7fc2e80cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:10.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 -- 172.21.15.50:0/2143335120 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80af250 con 0x7fc2e80c7170 2023-12-19T14:52:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e77fe700 1 -- 172.21.15.50:0/2143335120 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218712604 0 0) 0x7fc2e80c7610 con 0x7fc2e80acc60 2023-12-19T14:52:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e7fff700 1 --2- 172.21.15.50:0/2143335120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e80ac720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40099b0 tx=0x7fc2d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3431d842984dfd1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e77fe700 1 -- 172.21.15.50:0/2143335120 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400b860 con 0x7fc2e80c7170 2023-12-19T14:52:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e77fe700 1 -- 172.21.15.50:0/2143335120 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d400b9c0 con 0x7fc2e80c7170 2023-12-19T14:52:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2e77fe700 1 -- 172.21.15.50:0/2143335120 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400bd60 con 0x7fc2e80c7170 2023-12-19T14:52:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- 172.21.15.50:0/2143335120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 msgr2=0x7fc2e80ac720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/2143335120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e80ac720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40099b0 tx=0x7fc2d4009250 comp rx=0 tx=0).stop 2023-12-19T14:52:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- 172.21.15.50:0/2143335120 shutdown_connections 2023-12-19T14:52:10.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/2143335120 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e80ac720 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- 172.21.15.50:0/2143335120 >> 172.21.15.50:0/2143335120 conn(0x7fc2e80ccb30 msgr2=0x7fc2e80ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- 172.21.15.50:0/2143335120 shutdown_connections 2023-12-19T14:52:10.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.042+0000 7fc2edc4c700 1 -- 172.21.15.50:0/2143335120 wait complete. 2023-12-19T14:52:10.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.046+0000 7fc2edc4c700 1 Processor -- start 2023-12-19T14:52:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.046+0000 7fc2edc4c700 1 -- start start 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2e80acc60 0x7fc2e80aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2e8150200 0x7fc2e814a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8168ff0 con 0x7fc2e80c7170 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e813fd20 con 0x7fc2e8150200 2023-12-19T14:52:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e816c980 con 0x7fc2e80acc60 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33566/0 (socket says 172.21.15.50:33566) 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 -- 172.21.15.50:0/3013365879 learned_addr learned my addr 172.21.15.50:0/3013365879 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2ecc4a700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2e80acc60 0x7fc2e80aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2ed44b700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2e8150200 0x7fc2e814a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 -- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2e80acc60 msgr2=0x7fc2e80aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2e80acc60 0x7fc2e80aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:10.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 -- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2e8150200 msgr2=0x7fc2e814a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2e8150200 0x7fc2e814a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 -- 172.21.15.50:0/3013365879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40094f0 con 0x7fc2e80c7170 2023-12-19T14:52:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e7fff700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc2e806b3a0 tx=0x7fc2d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d401f070 con 0x7fc2e80c7170 2023-12-19T14:52:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4011c60 con 0x7fc2e80c7170 2023-12-19T14:52:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4025420 con 0x7fc2e80c7170 2023-12-19T14:52:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e816cc60 con 0x7fc2e80c7170 2023-12-19T14:52:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.054+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8148ac0 con 0x7fc2e80c7170 2023-12-19T14:52:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.054+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e804be40 con 0x7fc2e80c7170 2023-12-19T14:52:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.054+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc2d4023070 con 0x7fc2e80c7170 2023-12-19T14:52:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.054+0000 7fc2e4ff9700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 0x7fc2cc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.054+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc2d407dc40 con 0x7fc2e80c7170 2023-12-19T14:52:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.058+0000 7fc2ecc4a700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 0x7fc2cc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:10.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.058+0000 7fc2ecc4a700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 0x7fc2cc062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0039e0 tx=0x7fc2dc007ec0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.062+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2d404dff0 con 0x7fc2e80c7170 2023-12-19T14:52:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:10.274+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc2e806cbf0 con 0x7fc2e80c7170 2023-12-19T14:52:11.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2e4ff9700 1 -- 172.21.15.50:0/3013365879 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc2d40150b0 con 0x7fc2e80c7170 2023-12-19T14:52:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 msgr2=0x7fc2cc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 0x7fc2cc062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0039e0 tx=0x7fc2dc007ec0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 msgr2=0x7fc2e813f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc2e806b3a0 tx=0x7fc2d4017000 comp rx=0 tx=0).stop 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 shutdown_connections 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2e8150200 0x7fc2e814a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc2cc060670 0x7fc2cc062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc2e80c7170 0x7fc2e813f7e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 --2- 172.21.15.50:0/3013365879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc2e80acc60 0x7fc2e80aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 >> 172.21.15.50:0/3013365879 conn(0x7fc2e80ccb30 msgr2=0x7fc2e80ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 shutdown_connections 2023-12-19T14:52:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.050+0000 7fc2edc4c700 1 -- 172.21.15.50:0/3013365879 wait complete. 2023-12-19T14:52:11.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:52:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.242+0000 7f31c6d85700 1 Processor -- start 2023-12-19T14:52:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.242+0000 7f31c6d85700 1 -- start start 2023-12-19T14:52:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c6d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 0x7f31c00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c6d85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c006d5e0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c6d85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00cbdf0 con 0x7f31c00cb970 2023-12-19T14:52:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c6d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00cc0d0 con 0x7f31c0158e20 2023-12-19T14:52:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c5322700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f31c00ca2c0 0x7f31c006ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57808/0 (socket says 172.21.15.50:57808) 2023-12-19T14:52:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c5322700 1 -- 172.21.15.50:0/3042361330 learned_addr learned my addr 172.21.15.50:0/3042361330 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:11.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c4b21700 1 --2- 172.21.15.50:0/3042361330 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 0x7f31c00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:11.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461328385 0 0) 0x7f31c006d5e0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b0003570 con 0x7f31c00ca2c0 2023-12-19T14:52:11.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261378718 0 0) 0x7f31c00cbdf0 con 0x7f31c00cb970 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c006d5e0 con 0x7f31c00cb970 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c4b21700 1 -- 172.21.15.50:0/3042361330 >> v1:172.21.15.125:6789/0 conn(0x7f31c00cb970 legacy=0x7f31c006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c4b21700 1 -- 172.21.15.50:0/3042361330 >> v1:172.21.15.50:6789/0 conn(0x7f31c00ca2c0 legacy=0x7f31c006ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3560238212 0 0) 0x7f31b0003570 con 0x7f31c00ca2c0 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c4b21700 1 -- 172.21.15.50:0/3042361330 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00cc3b0 con 0x7f31c0158e20 2023-12-19T14:52:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31c4b21700 1 --2- 172.21.15.50:0/3042361330 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 0x7f31c00ce5c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009910 tx=0x7f31b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13302b13b1f7c378 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.246+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80017f0 con 0x7f31c0158e20 2023-12-19T14:52:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b4011420 con 0x7f31c0158e20 2023-12-19T14:52:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31bf7fe700 1 -- 172.21.15.50:0/3042361330 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b40117c0 con 0x7f31c0158e20 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- 172.21.15.50:0/3042361330 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 msgr2=0x7f31c00ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3042361330 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 0x7f31c00ce5c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009910 tx=0x7f31b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- 172.21.15.50:0/3042361330 shutdown_connections 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3042361330 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c0158e20 0x7f31c00ce5c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- 172.21.15.50:0/3042361330 >> 172.21.15.50:0/3042361330 conn(0x7f31c0067bb0 msgr2=0x7f31c00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- 172.21.15.50:0/3042361330 shutdown_connections 2023-12-19T14:52:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- 172.21.15.50:0/3042361330 wait complete. 2023-12-19T14:52:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 Processor -- start 2023-12-19T14:52:11.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- start start 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c0158e20 0x7f31c016cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c00c8600 0x7f31c01526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c016d500 con 0x7f31c00ca2c0 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0169650 con 0x7f31c0158e20 2023-12-19T14:52:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.250+0000 7f31c6d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0169930 con 0x7f31c00c8600 2023-12-19T14:52:11.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:11.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33576/0 (socket says 172.21.15.50:33576) 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 -- 172.21.15.50:0/3600688561 learned_addr learned my addr 172.21.15.50:0/3600688561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31bffff700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c0158e20 0x7f31c016cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c5322700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c00c8600 0x7f31c01526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 -- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c00c8600 msgr2=0x7f31c01526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c00c8600 0x7f31c01526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 -- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c0158e20 msgr2=0x7f31c016cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c0158e20 0x7f31c016cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 -- 172.21.15.50:0/3600688561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4009420 con 0x7f31c00ca2c0 2023-12-19T14:52:11.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c4b21700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f31b40091e0 tx=0x7f31b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b401cbe0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b401cd40 con 0x7f31c00ca2c0 2023-12-19T14:52:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b401a6a0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c0169c10 con 0x7f31c00ca2c0 2023-12-19T14:52:11.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c01509d0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.254+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c0150bc0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.258+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f31b401aad0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.258+0000 7f31bcff9700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 0x7f31a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:11.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.258+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f31b407e860 con 0x7f31c00ca2c0 2023-12-19T14:52:11.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.258+0000 7f31bffff700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 0x7f31a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:11.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.258+0000 7f31bffff700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 0x7f31a4062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f31c00cf6e0 tx=0x7f31ac007d50 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:11.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.266+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f31b404ebe0 con 0x7f31c00ca2c0 2023-12-19T14:52:11.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:11.474+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f31c0071120 con 0x7f31c00ca2c0 2023-12-19T14:52:12.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.054+0000 7f31bcff9700 1 -- 172.21.15.50:0/3600688561 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f31b4052200 con 0x7f31c00ca2c0 2023-12-19T14:52:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 msgr2=0x7f31a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 0x7f31a4062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f31c00cf6e0 tx=0x7f31ac007d50 comp rx=0 tx=0).stop 2023-12-19T14:52:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 msgr2=0x7f31c0169000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f31b40091e0 tx=0x7f31b4016000 comp rx=0 tx=0).stop 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 shutdown_connections 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31c0158e20 0x7f31c016cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31a4060670 0x7f31a4062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31c00c8600 0x7f31c01526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 --2- 172.21.15.50:0/3600688561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31c00ca2c0 0x7f31c0169000 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 >> 172.21.15.50:0/3600688561 conn(0x7f31c0067bb0 msgr2=0x7f31c0156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 shutdown_connections 2023-12-19T14:52:12.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.058+0000 7f31c6d85700 1 -- 172.21.15.50:0/3600688561 wait complete. 2023-12-19T14:52:12.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:52:12.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 Processor -- start 2023-12-19T14:52:12.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 -- start start 2023-12-19T14:52:12.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96401591f0 0x7f964006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964006b730 con 0x7f9640158e20 2023-12-19T14:52:12.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964006ab20 con 0x7f96400ce810 2023-12-19T14:52:12.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963f59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964006ae00 con 0x7f96401591f0 2023-12-19T14:52:12.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963e59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9640158e20 0x7f96400ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57820/0 (socket says 172.21.15.50:57820) 2023-12-19T14:52:12.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.242+0000 7f963e59c700 1 -- 172.21.15.50:0/3308384430 learned_addr learned my addr 172.21.15.50:0/3308384430 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:12.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963dd9b700 1 --2- 172.21.15.50:0/3308384430 >> v2:172.21.15.125:3300/0 conn(0x7f96401591f0 0x7f964006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963d59a700 1 -- 172.21.15.50:0/3308384430 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25389608 0 0) 0x7f964006b730 con 0x7f9640158e20 2023-12-19T14:52:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963d59a700 1 -- 172.21.15.50:0/3308384430 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9628003570 con 0x7f9640158e20 2023-12-19T14:52:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963dd9b700 1 -- 172.21.15.50:0/3308384430 >> v1:172.21.15.125:6789/0 conn(0x7f96400ce810 legacy=0x7f96400ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963dd9b700 1 -- 172.21.15.50:0/3308384430 >> v1:172.21.15.50:6789/0 conn(0x7f9640158e20 legacy=0x7f96400ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963dd9b700 1 -- 172.21.15.50:0/3308384430 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964006b0e0 con 0x7f96401591f0 2023-12-19T14:52:12.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963dd9b700 1 --2- 172.21.15.50:0/3308384430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96401591f0 0x7f964006a540 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96340098e0 tx=0x7f96340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eeb22d582cc160d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963d59a700 1 -- 172.21.15.50:0/3308384430 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963400b640 con 0x7f96401591f0 2023-12-19T14:52:12.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963d59a700 1 -- 172.21.15.50:0/3308384430 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963400b7a0 con 0x7f96401591f0 2023-12-19T14:52:12.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963d59a700 1 -- 172.21.15.50:0/3308384430 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963400bb40 con 0x7f96401591f0 2023-12-19T14:52:12.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 -- 172.21.15.50:0/3308384430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96401591f0 msgr2=0x7f964006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 --2- 172.21.15.50:0/3308384430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96401591f0 0x7f964006a540 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96340098e0 tx=0x7f96340091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 -- 172.21.15.50:0/3308384430 shutdown_connections 2023-12-19T14:52:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 --2- 172.21.15.50:0/3308384430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96401591f0 0x7f964006a540 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 -- 172.21.15.50:0/3308384430 >> 172.21.15.50:0/3308384430 conn(0x7f9640067bb0 msgr2=0x7f96400cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 -- 172.21.15.50:0/3308384430 shutdown_connections 2023-12-19T14:52:12.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.246+0000 7f963f59e700 1 -- 172.21.15.50:0/3308384430 wait complete. 2023-12-19T14:52:12.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 Processor -- start 2023-12-19T14:52:12.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 -- start start 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 0x7f964015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9640153080 con 0x7f96401591f0 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964015b8b0 con 0x7f9640158e20 2023-12-19T14:52:12.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963f59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96401428d0 con 0x7f96400ce810 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963e59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 0x7f964015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963dd9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963dd9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:41864/0 (socket says 172.21.15.50:41864) 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963dd9b700 1 -- 172.21.15.50:0/3525200536 learned_addr learned my addr 172.21.15.50:0/3525200536 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33590/0 (socket says 172.21.15.50:33590) 2023-12-19T14:52:12.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.250+0000 7f963e59c700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 0x7f964015bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35974/0 (socket says 172.21.15.50:35974) 2023-12-19T14:52:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 -- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 msgr2=0x7f964015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 0x7f964015bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 -- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 msgr2=0x7f9640142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 -- 172.21.15.50:0/3525200536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9634009420 con 0x7f96401591f0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963dd9b700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963ed9d700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f963800c860 tx=0x7f9638001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638012630 con 0x7f96401591f0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638012790 con 0x7f96401591f0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964015aca0 con 0x7f96401591f0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638012b50 con 0x7f96401591f0 2023-12-19T14:52:12.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964015b260 con 0x7f96401591f0 2023-12-19T14:52:12.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.254+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964015ba60 con 0x7f96401591f0 2023-12-19T14:52:12.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.258+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f963801a3a0 con 0x7f96401591f0 2023-12-19T14:52:12.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.258+0000 7f962e7fc700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 0x7f9620062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.258+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f963807cbe0 con 0x7f96401591f0 2023-12-19T14:52:12.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.258+0000 7f963e59c700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 0x7f9620062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.266+0000 7f963e59c700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 0x7f9620062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9630007fd0 tx=0x7f9630009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.266+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f964015ba60 con 0x7f96401591f0 2023-12-19T14:52:12.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.474+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f964015ba60 con 0x7f96401591f0 2023-12-19T14:52:12.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f962e7fc700 1 -- 172.21.15.50:0/3525200536 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f964015ba60 con 0x7f96401591f0 2023-12-19T14:52:12.480 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:12.480 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 msgr2=0x7f9620062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 0x7f9620062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9630007fd0 tx=0x7f9630009040 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 msgr2=0x7f9640157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f963800c860 tx=0x7f9638001690 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 shutdown_connections 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9640158e20 0x7f9640142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9620060670 0x7f9620062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96401591f0 0x7f9640157ba0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 --2- 172.21.15.50:0/3525200536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96400ce810 0x7f964015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.478+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 >> 172.21.15.50:0/3525200536 conn(0x7f9640067bb0 msgr2=0x7f96400cad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.482+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 shutdown_connections 2023-12-19T14:52:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.482+0000 7f963f59e700 1 -- 172.21.15.50:0/3525200536 wait complete. 2023-12-19T14:52:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:52:12.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:52:12.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.666+0000 7f53afd45700 1 Processor -- start 2023-12-19T14:52:12.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.666+0000 7f53afd45700 1 -- start start 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53afd45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f53a8158f00 0x7f53a80c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53afd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f53a806b0f0 0x7f53a806b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53afd45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a806ba90 con 0x7f53a81592d0 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53afd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a806e470 con 0x7f53a806b0f0 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53afd45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a806e900 con 0x7f53a8158f00 2023-12-19T14:52:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f53a806b0f0 0x7f53a806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53adae1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f53a8158f00 0x7f53a80c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f53a806b0f0 0x7f53a806b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33604/0 (socket says 172.21.15.50:33604) 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 -- 172.21.15.50:0/832478688 learned_addr learned my addr 172.21.15.50:0/832478688 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 -- 172.21.15.50:0/832478688 >> v2:172.21.15.125:3300/0 conn(0x7f53a8158f00 msgr2=0x7f53a80c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 --2- 172.21.15.50:0/832478688 >> v2:172.21.15.125:3300/0 conn(0x7f53a8158f00 0x7f53a80c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 -- 172.21.15.50:0/832478688 >> v1:172.21.15.125:6789/0 conn(0x7f53a81592d0 legacy=0x7f53a806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 -- 172.21.15.50:0/832478688 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a806d860 con 0x7f53a806b0f0 2023-12-19T14:52:12.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53ae2e2700 1 --2- 172.21.15.50:0/832478688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a806b0f0 0x7f53a806b550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009f20 tx=0x7f53a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed8e0efe5cc36ab server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53acadf700 1 -- 172.21.15.50:0/832478688 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a000e070 con 0x7f53a806b0f0 2023-12-19T14:52:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53acadf700 1 -- 172.21.15.50:0/832478688 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a000b990 con 0x7f53a806b0f0 2023-12-19T14:52:12.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.670+0000 7f53acadf700 1 -- 172.21.15.50:0/832478688 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00092e0 con 0x7f53a806b0f0 2023-12-19T14:52:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 -- 172.21.15.50:0/832478688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a806b0f0 msgr2=0x7f53a806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 --2- 172.21.15.50:0/832478688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a806b0f0 0x7f53a806b550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009f20 tx=0x7f53a0009840 comp rx=0 tx=0).stop 2023-12-19T14:52:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 -- 172.21.15.50:0/832478688 shutdown_connections 2023-12-19T14:52:12.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 --2- 172.21.15.50:0/832478688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a806b0f0 0x7f53a806b550 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 --2- 172.21.15.50:0/832478688 >> v2:172.21.15.125:3300/0 conn(0x7f53a8158f00 0x7f53a80c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 -- 172.21.15.50:0/832478688 >> 172.21.15.50:0/832478688 conn(0x7f53a80c90f0 msgr2=0x7f53a80c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 -- 172.21.15.50:0/832478688 shutdown_connections 2023-12-19T14:52:12.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.674+0000 7f53afd45700 1 -- 172.21.15.50:0/832478688 wait complete. 2023-12-19T14:52:12.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 Processor -- start 2023-12-19T14:52:12.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 -- start start 2023-12-19T14:52:12.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53a806b0f0 0x7f53a816b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f53a8158f00 0x7f53a8169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a816be10 con 0x7f53a81592d0 2023-12-19T14:52:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8169970 con 0x7f53a806b0f0 2023-12-19T14:52:12.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53afd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8159790 con 0x7f53a8158f00 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.678+0000 7f53ae2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33614/0 (socket says 172.21.15.50:33614) 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 -- 172.21.15.50:0/1097650244 learned_addr learned my addr 172.21.15.50:0/1097650244 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53adae1700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53a806b0f0 0x7f53a816b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ad2e0700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f53a8158f00 0x7f53a8169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 -- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f53a8158f00 msgr2=0x7f53a8169be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f53a8158f00 0x7f53a8169be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 -- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53a806b0f0 msgr2=0x7f53a816b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53a806b0f0 0x7f53a816b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 -- 172.21.15.50:0/1097650244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0009ae0 con 0x7f53a81592d0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53ae2e2700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009880 tx=0x7f53a0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0008ba0 con 0x7f53a81592d0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0008d00 con 0x7f53a81592d0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00266b0 con 0x7f53a81592d0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a8159a70 con 0x7f53a81592d0 2023-12-19T14:52:12.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.682+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a815a030 con 0x7f53a81592d0 2023-12-19T14:52:12.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.686+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a815a270 con 0x7f53a81592d0 2023-12-19T14:52:12.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.686+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f53a0003040 con 0x7f53a81592d0 2023-12-19T14:52:12.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.690+0000 7f539dffb700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 0x7f538c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:12.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.690+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f53a007e470 con 0x7f53a81592d0 2023-12-19T14:52:12.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.690+0000 7f53adae1700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 0x7f538c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:12.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.690+0000 7f53adae1700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 0x7f538c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009db0 tx=0x7f53a40091a0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:12.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.694+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f53a004e660 con 0x7f53a81592d0 2023-12-19T14:52:12.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.870+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a8071120 con 0x7f53a81592d0 2023-12-19T14:52:12.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f539dffb700 1 -- 172.21.15.50:0/1097650244 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+15555 (secure 0 0 0) 0x7f53a0051c80 con 0x7f53a81592d0 2023-12-19T14:52:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:12.876 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":68,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:52:10.005685+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 msgr2=0x7f538c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 0x7f538c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009db0 tx=0x7f53a40091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 msgr2=0x7f53a815b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009880 tx=0x7f53a0018000 comp rx=0 tx=0).stop 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 shutdown_connections 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f538c060670 0x7f538c062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53a81592d0 0x7f53a815b9c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f53a8158f00 0x7f53a8169be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 --2- 172.21.15.50:0/1097650244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53a806b0f0 0x7f53a816b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:12.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 >> 172.21.15.50:0/1097650244 conn(0x7f53a80c90f0 msgr2=0x7f53a80caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:12.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.874+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 shutdown_connections 2023-12-19T14:52:12.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:12.878+0000 7f53afd45700 1 -- 172.21.15.50:0/1097650244 wait complete. 2023-12-19T14:52:12.900 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:52:12.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:52:13.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.066+0000 7f817f59e700 1 Processor -- start 2023-12-19T14:52:13.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.066+0000 7f817f59e700 1 -- start start 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f81800b0810 0x7f81800b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f81800af9c0 0x7f8180154b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81800af530 con 0x7f81800b0810 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8180159660 con 0x7f81800af9c0 2023-12-19T14:52:13.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817f59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8180159940 con 0x7f818006be10 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f81800b0810 0x7f81800b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f81800b0810 0x7f81800b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33620/0 (socket says 172.21.15.50:33620) 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 -- 172.21.15.50:0/4187594863 learned_addr learned my addr 172.21.15.50:0/4187594863 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817dd9b700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817ed9d700 1 --2- 172.21.15.50:0/4187594863 >> v2:172.21.15.125:3300/0 conn(0x7f81800af9c0 0x7f8180154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 -- 172.21.15.50:0/4187594863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 msgr2=0x7f8180158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 -- 172.21.15.50:0/4187594863 >> v2:172.21.15.125:3300/0 conn(0x7f81800af9c0 msgr2=0x7f8180154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 --2- 172.21.15.50:0/4187594863 >> v2:172.21.15.125:3300/0 conn(0x7f81800af9c0 0x7f8180154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 -- 172.21.15.50:0/4187594863 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180159c20 con 0x7f81800b0810 2023-12-19T14:52:13.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.070+0000 7f817e59c700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f81800b0be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcfcf26811658a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817d59a700 1 -- 172.21.15.50:0/4187594863 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817000b690 con 0x7f81800b0810 2023-12-19T14:52:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817d59a700 1 -- 172.21.15.50:0/4187594863 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817000b7f0 con 0x7f81800b0810 2023-12-19T14:52:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817d59a700 1 -- 172.21.15.50:0/4187594863 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817000bb90 con 0x7f81800b0810 2023-12-19T14:52:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 -- 172.21.15.50:0/4187594863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 msgr2=0x7f81800b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f81800b0be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170009250 comp rx=0 tx=0).stop 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 -- 172.21.15.50:0/4187594863 shutdown_connections 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 --2- 172.21.15.50:0/4187594863 >> v2:172.21.15.125:3300/0 conn(0x7f81800af9c0 0x7f8180154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 --2- 172.21.15.50:0/4187594863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f81800b0be0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 -- 172.21.15.50:0/4187594863 >> 172.21.15.50:0/4187594863 conn(0x7f8180067bb0 msgr2=0x7f81800add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 -- 172.21.15.50:0/4187594863 shutdown_connections 2023-12-19T14:52:13.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 -- 172.21.15.50:0/4187594863 wait complete. 2023-12-19T14:52:13.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.074+0000 7f817f59e700 1 Processor -- start 2023-12-19T14:52:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- start start 2023-12-19T14:52:13.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 0x7f8180149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81801533f0 con 0x7f81800b0810 2023-12-19T14:52:13.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81801527e0 con 0x7f81800af9c0 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8180152ac0 con 0x7f818006be10 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 0x7f8180149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33630/0 (socket says 172.21.15.50:33630) 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817e59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817ed9d700 1 -- 172.21.15.50:0/3413845901 learned_addr learned my addr 172.21.15.50:0/3413845901 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:13.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 -- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 msgr2=0x7f8180153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 -- 172.21.15.50:0/3413845901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 msgr2=0x7f818014dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 -- 172.21.15.50:0/3413845901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700094f0 con 0x7f81800af9c0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817ed9d700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817dd9b700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 0x7f8180149d30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f817400ea50 tx=0x7f81740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400ee90 con 0x7f81800af9c0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817400c7f0 con 0x7f81800af9c0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8180152da0 con 0x7f81800af9c0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400cb90 con 0x7f81800af9c0 2023-12-19T14:52:13.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.078+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8180153dc0 con 0x7f81800af9c0 2023-12-19T14:52:13.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.082+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81801535a0 con 0x7f81800af9c0 2023-12-19T14:52:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.082+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8174013470 con 0x7f81800af9c0 2023-12-19T14:52:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.086+0000 7f816e7fc700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 0x7f8160062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.086+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f817407cc70 con 0x7f81800af9c0 2023-12-19T14:52:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.086+0000 7f817e59c700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 0x7f8160062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.086+0000 7f817e59c700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 0x7f8160062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170017000 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.094+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f81801535a0 con 0x7f81800af9c0 2023-12-19T14:52:13.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.266+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8180153560 con 0x7f8160060670 2023-12-19T14:52:13.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.282+0000 7f816e7fc700 1 -- 172.21.15.50:0/3413845901 <== mgr.5626 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286473 (secure 0 0 0) 0x7f8180153560 con 0x7f8160060670 2023-12-19T14:52:13.291 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 msgr2=0x7f8160062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 0x7f8160062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81700099e0 tx=0x7f8170017000 comp rx=0 tx=0).stop 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 msgr2=0x7f8180149d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 0x7f8180149d30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f817400ea50 tx=0x7f81740091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 shutdown_connections 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81800af9c0 0x7f8180149d30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8160060670 0x7f8160062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81800b0810 0x7f818014dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 --2- 172.21.15.50:0/3413845901 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f818006be10 0x7f8180153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 >> 172.21.15.50:0/3413845901 conn(0x7f8180067bb0 msgr2=0x7f818014d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 shutdown_connections 2023-12-19T14:52:13.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.294+0000 7f817f59e700 1 -- 172.21.15.50:0/3413845901 wait complete. 2023-12-19T14:52:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:52:13.316 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:52:12.399482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439088,"kb_used_data":7536,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752535632,"statfs":{"total":773094113280,"available":770596487168,"internally_reserved":0,"allocated":7716864,"data_stored":4369398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413102"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040626+0000","last_change":"2023-12-19T14:52:09.036116+0000","last_active":"2023-12-19T14:52:09.040626+0000","last_peered":"2023-12-19T14:52:09.040626+0000","last_clean":"2023-12-19T14:52:09.040626+0000","last_became_active":"2023-12-19T14:52:09.035803+0000","last_became_peered":"2023-12-19T14:52:09.035803+0000","last_unstale":"2023-12-19T14:52:09.040626+0000","last_undegraded":"2023-12-19T14:52:09.040626+0000","last_fullsized":"2023-12-19T14:52:09.040626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:48.180290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043337+0000","last_change":"2023-12-19T14:52:09.037446+0000","last_active":"2023-12-19T14:52:09.043337+0000","last_peered":"2023-12-19T14:52:09.043337+0000","last_clean":"2023-12-19T14:52:09.043337+0000","last_became_active":"2023-12-19T14:52:09.036847+0000","last_became_peered":"2023-12-19T14:52:09.036847+0000","last_unstale":"2023-12-19T14:52:09.043337+0000","last_undegraded":"2023-12-19T14:52:09.043337+0000","last_fullsized":"2023-12-19T14:52:09.043337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:48.180290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052057+0000","last_change":"2023-12-19T14:52:09.043187+0000","last_active":"2023-12-19T14:52:09.052057+0000","last_peered":"2023-12-19T14:52:09.052057+0000","last_clean":"2023-12-19T14:52:09.052057+0000","last_became_active":"2023-12-19T14:52:09.042783+0000","last_became_peered":"2023-12-19T14:52:09.042783+0000","last_unstale":"2023-12-19T14:52:09.052057+0000","last_undegraded":"2023-12-19T14:52:09.052057+0000","last_fullsized":"2023-12-19T14:52:09.052057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:17.623185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043398+0000","last_change":"2023-12-19T14:52:09.037992+0000","last_active":"2023-12-19T14:52:09.043398+0000","last_peered":"2023-12-19T14:52:09.043398+0000","last_clean":"2023-12-19T14:52:09.043398+0000","last_became_active":"2023-12-19T14:52:09.037716+0000","last_became_peered":"2023-12-19T14:52:09.037716+0000","last_unstale":"2023-12-19T14:52:09.043398+0000","last_undegraded":"2023-12-19T14:52:09.043398+0000","last_fullsized":"2023-12-19T14:52:09.043398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:06.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045993+0000","last_change":"2023-12-19T14:52:09.040387+0000","last_active":"2023-12-19T14:52:09.045993+0000","last_peered":"2023-12-19T14:52:09.045993+0000","last_clean":"2023-12-19T14:52:09.045993+0000","last_became_active":"2023-12-19T14:52:09.040083+0000","last_became_peered":"2023-12-19T14:52:09.040083+0000","last_unstale":"2023-12-19T14:52:09.045993+0000","last_undegraded":"2023-12-19T14:52:09.045993+0000","last_fullsized":"2023-12-19T14:52:09.045993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:56.125818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042231+0000","last_change":"2023-12-19T14:52:09.038334+0000","last_active":"2023-12-19T14:52:09.042231+0000","last_peered":"2023-12-19T14:52:09.042231+0000","last_clean":"2023-12-19T14:52:09.042231+0000","last_became_active":"2023-12-19T14:52:09.038070+0000","last_became_peered":"2023-12-19T14:52:09.038070+0000","last_unstale":"2023-12-19T14:52:09.042231+0000","last_undegraded":"2023-12-19T14:52:09.042231+0000","last_fullsized":"2023-12-19T14:52:09.042231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:06.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052109+0000","last_change":"2023-12-19T14:52:09.044230+0000","last_active":"2023-12-19T14:52:09.052109+0000","last_peered":"2023-12-19T14:52:09.052109+0000","last_clean":"2023-12-19T14:52:09.052109+0000","last_became_active":"2023-12-19T14:52:09.039210+0000","last_became_peered":"2023-12-19T14:52:09.039210+0000","last_unstale":"2023-12-19T14:52:09.052109+0000","last_undegraded":"2023-12-19T14:52:09.052109+0000","last_fullsized":"2023-12-19T14:52:09.052109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054605+0000","last_change":"2023-12-19T14:52:09.046828+0000","last_active":"2023-12-19T14:52:09.054605+0000","last_peered":"2023-12-19T14:52:09.054605+0000","last_clean":"2023-12-19T14:52:09.054605+0000","last_became_active":"2023-12-19T14:52:09.046500+0000","last_became_peered":"2023-12-19T14:52:09.046500+0000","last_unstale":"2023-12-19T14:52:09.054605+0000","last_undegraded":"2023-12-19T14:52:09.054605+0000","last_fullsized":"2023-12-19T14:52:09.054605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:03:00.136172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.048595+0000","last_change":"2023-12-19T14:52:09.045957+0000","last_active":"2023-12-19T14:52:09.048595+0000","last_peered":"2023-12-19T14:52:09.048595+0000","last_clean":"2023-12-19T14:52:09.048595+0000","last_became_active":"2023-12-19T14:52:09.045476+0000","last_became_peered":"2023-12-19T14:52:09.045476+0000","last_unstale":"2023-12-19T14:52:09.048595+0000","last_undegraded":"2023-12-19T14:52:09.048595+0000","last_fullsized":"2023-12-19T14:52:09.048595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:02:51.357979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036047+0000","last_change":"2023-12-19T14:52:09.029448+0000","last_active":"2023-12-19T14:52:09.036047+0000","last_peered":"2023-12-19T14:52:09.036047+0000","last_clean":"2023-12-19T14:52:09.036047+0000","last_became_active":"2023-12-19T14:52:09.029155+0000","last_became_peered":"2023-12-19T14:52:09.029155+0000","last_unstale":"2023-12-19T14:52:09.036047+0000","last_undegraded":"2023-12-19T14:52:09.036047+0000","last_fullsized":"2023-12-19T14:52:09.036047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:42.953083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054772+0000","last_change":"2023-12-19T14:52:09.051508+0000","last_active":"2023-12-19T14:52:09.054772+0000","last_peered":"2023-12-19T14:52:09.054772+0000","last_clean":"2023-12-19T14:52:09.054772+0000","last_became_active":"2023-12-19T14:52:09.051217+0000","last_became_peered":"2023-12-19T14:52:09.051217+0000","last_unstale":"2023-12-19T14:52:09.054772+0000","last_undegraded":"2023-12-19T14:52:09.054772+0000","last_fullsized":"2023-12-19T14:52:09.054772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:12.498326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039891+0000","last_change":"2023-12-19T14:52:09.033517+0000","last_active":"2023-12-19T14:52:09.039891+0000","last_peered":"2023-12-19T14:52:09.039891+0000","last_clean":"2023-12-19T14:52:09.039891+0000","last_became_active":"2023-12-19T14:52:09.032500+0000","last_became_peered":"2023-12-19T14:52:09.032500+0000","last_unstale":"2023-12-19T14:52:09.039891+0000","last_undegraded":"2023-12-19T14:52:09.039891+0000","last_fullsized":"2023-12-19T14:52:09.039891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:17.623185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036052+0000","last_change":"2023-12-19T14:52:09.028928+0000","last_active":"2023-12-19T14:52:09.036052+0000","last_peered":"2023-12-19T14:52:09.036052+0000","last_clean":"2023-12-19T14:52:09.036052+0000","last_became_active":"2023-12-19T14:52:09.028616+0000","last_became_peered":"2023-12-19T14:52:09.028616+0000","last_unstale":"2023-12-19T14:52:09.036052+0000","last_undegraded":"2023-12-19T14:52:09.036052+0000","last_fullsized":"2023-12-19T14:52:09.036052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:30.584084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054504+0000","last_change":"2023-12-19T14:52:09.044159+0000","last_active":"2023-12-19T14:52:09.054504+0000","last_peered":"2023-12-19T14:52:09.054504+0000","last_clean":"2023-12-19T14:52:09.054504+0000","last_became_active":"2023-12-19T14:52:09.043861+0000","last_became_peered":"2023-12-19T14:52:09.043861+0000","last_unstale":"2023-12-19T14:52:09.054504+0000","last_undegraded":"2023-12-19T14:52:09.054504+0000","last_fullsized":"2023-12-19T14:52:09.054504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043300+0000","last_change":"2023-12-19T14:52:09.035737+0000","last_active":"2023-12-19T14:52:09.043300+0000","last_peered":"2023-12-19T14:52:09.043300+0000","last_clean":"2023-12-19T14:52:09.043300+0000","last_became_active":"2023-12-19T14:52:09.035448+0000","last_became_peered":"2023-12-19T14:52:09.035448+0000","last_unstale":"2023-12-19T14:52:09.043300+0000","last_undegraded":"2023-12-19T14:52:09.043300+0000","last_fullsized":"2023-12-19T14:52:09.043300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:05.290781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036306+0000","last_change":"2023-12-19T14:52:09.027883+0000","last_active":"2023-12-19T14:52:09.036306+0000","last_peered":"2023-12-19T14:52:09.036306+0000","last_clean":"2023-12-19T14:52:09.036306+0000","last_became_active":"2023-12-19T14:52:09.027588+0000","last_became_peered":"2023-12-19T14:52:09.027588+0000","last_unstale":"2023-12-19T14:52:09.036306+0000","last_undegraded":"2023-12-19T14:52:09.036306+0000","last_fullsized":"2023-12-19T14:52:09.036306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:00.157458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052014+0000","last_change":"2023-12-19T14:52:09.037729+0000","last_active":"2023-12-19T14:52:09.052014+0000","last_peered":"2023-12-19T14:52:09.052014+0000","last_clean":"2023-12-19T14:52:09.052014+0000","last_became_active":"2023-12-19T14:52:09.037403+0000","last_became_peered":"2023-12-19T14:52:09.037403+0000","last_unstale":"2023-12-19T14:52:09.052014+0000","last_undegraded":"2023-12-19T14:52:09.052014+0000","last_fullsized":"2023-12-19T14:52:09.052014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:17:25.394194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043354+0000","last_change":"2023-12-19T14:52:09.037504+0000","last_active":"2023-12-19T14:52:09.043354+0000","last_peered":"2023-12-19T14:52:09.043354+0000","last_clean":"2023-12-19T14:52:09.043354+0000","last_became_active":"2023-12-19T14:52:09.037207+0000","last_became_peered":"2023-12-19T14:52:09.037207+0000","last_unstale":"2023-12-19T14:52:09.043354+0000","last_undegraded":"2023-12-19T14:52:09.043354+0000","last_fullsized":"2023-12-19T14:52:09.043354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051885+0000","last_change":"2023-12-19T14:52:09.035999+0000","last_active":"2023-12-19T14:52:09.051885+0000","last_peered":"2023-12-19T14:52:09.051885+0000","last_clean":"2023-12-19T14:52:09.051885+0000","last_became_active":"2023-12-19T14:52:09.035702+0000","last_became_peered":"2023-12-19T14:52:09.035702+0000","last_unstale":"2023-12-19T14:52:09.051885+0000","last_undegraded":"2023-12-19T14:52:09.051885+0000","last_fullsized":"2023-12-19T14:52:09.051885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:06.248039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.035992+0000","last_change":"2023-12-19T14:52:09.030172+0000","last_active":"2023-12-19T14:52:09.035992+0000","last_peered":"2023-12-19T14:52:09.035992+0000","last_clean":"2023-12-19T14:52:09.035992+0000","last_became_active":"2023-12-19T14:52:09.029673+0000","last_became_peered":"2023-12-19T14:52:09.029673+0000","last_unstale":"2023-12-19T14:52:09.035992+0000","last_undegraded":"2023-12-19T14:52:09.035992+0000","last_fullsized":"2023-12-19T14:52:09.035992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:06.973983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036221+0000","last_change":"2023-12-19T14:52:09.032774+0000","last_active":"2023-12-19T14:52:09.036221+0000","last_peered":"2023-12-19T14:52:09.036221+0000","last_clean":"2023-12-19T14:52:09.036221+0000","last_became_active":"2023-12-19T14:52:09.032314+0000","last_became_peered":"2023-12-19T14:52:09.032314+0000","last_unstale":"2023-12-19T14:52:09.036221+0000","last_undegraded":"2023-12-19T14:52:09.036221+0000","last_fullsized":"2023-12-19T14:52:09.036221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:32.311495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.032674+0000","last_change":"2023-12-19T14:52:09.029439+0000","last_active":"2023-12-19T14:52:09.032674+0000","last_peered":"2023-12-19T14:52:09.032674+0000","last_clean":"2023-12-19T14:52:09.032674+0000","last_became_active":"2023-12-19T14:52:09.027954+0000","last_became_peered":"2023-12-19T14:52:09.027954+0000","last_unstale":"2023-12-19T14:52:09.032674+0000","last_undegraded":"2023-12-19T14:52:09.032674+0000","last_fullsized":"2023-12-19T14:52:09.032674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:02:06.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.055647+0000","last_change":"2023-12-19T14:52:09.054039+0000","last_active":"2023-12-19T14:52:09.055647+0000","last_peered":"2023-12-19T14:52:09.055647+0000","last_clean":"2023-12-19T14:52:09.055647+0000","last_became_active":"2023-12-19T14:52:09.053518+0000","last_became_peered":"2023-12-19T14:52:09.053518+0000","last_unstale":"2023-12-19T14:52:09.055647+0000","last_undegraded":"2023-12-19T14:52:09.055647+0000","last_fullsized":"2023-12-19T14:52:09.055647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:06.973983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047840+0000","last_change":"2023-12-19T14:52:09.034409+0000","last_active":"2023-12-19T14:52:09.047840+0000","last_peered":"2023-12-19T14:52:09.047840+0000","last_clean":"2023-12-19T14:52:09.047840+0000","last_became_active":"2023-12-19T14:52:09.033991+0000","last_became_peered":"2023-12-19T14:52:09.033991+0000","last_unstale":"2023-12-19T14:52:09.047840+0000","last_undegraded":"2023-12-19T14:52:09.047840+0000","last_fullsized":"2023-12-19T14:52:09.047840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:05.290781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052064+0000","last_change":"2023-12-19T14:52:09.038449+0000","last_active":"2023-12-19T14:52:09.052064+0000","last_peered":"2023-12-19T14:52:09.052064+0000","last_clean":"2023-12-19T14:52:09.052064+0000","last_became_active":"2023-12-19T14:52:09.038091+0000","last_became_peered":"2023-12-19T14:52:09.038091+0000","last_unstale":"2023-12-19T14:52:09.052064+0000","last_undegraded":"2023-12-19T14:52:09.052064+0000","last_fullsized":"2023-12-19T14:52:09.052064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:19.264675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046383+0000","last_change":"2023-12-19T14:52:09.039434+0000","last_active":"2023-12-19T14:52:09.046383+0000","last_peered":"2023-12-19T14:52:09.046383+0000","last_clean":"2023-12-19T14:52:09.046383+0000","last_became_active":"2023-12-19T14:52:09.039107+0000","last_became_peered":"2023-12-19T14:52:09.039107+0000","last_unstale":"2023-12-19T14:52:09.046383+0000","last_undegraded":"2023-12-19T14:52:09.046383+0000","last_fullsized":"2023-12-19T14:52:09.046383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:06.096141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052821+0000","last_change":"2023-12-19T14:52:09.046591+0000","last_active":"2023-12-19T14:52:09.052821+0000","last_peered":"2023-12-19T14:52:09.052821+0000","last_clean":"2023-12-19T14:52:09.052821+0000","last_became_active":"2023-12-19T14:52:09.044905+0000","last_became_peered":"2023-12-19T14:52:09.044905+0000","last_unstale":"2023-12-19T14:52:09.052821+0000","last_undegraded":"2023-12-19T14:52:09.052821+0000","last_fullsized":"2023-12-19T14:52:09.052821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:56.125818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.041930+0000","last_change":"2023-12-19T14:52:09.036092+0000","last_active":"2023-12-19T14:52:09.041930+0000","last_peered":"2023-12-19T14:52:09.041930+0000","last_clean":"2023-12-19T14:52:09.041930+0000","last_became_active":"2023-12-19T14:52:09.035820+0000","last_became_peered":"2023-12-19T14:52:09.035820+0000","last_unstale":"2023-12-19T14:52:09.041930+0000","last_undegraded":"2023-12-19T14:52:09.041930+0000","last_fullsized":"2023-12-19T14:52:09.041930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:12.498326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.053264+0000","last_change":"2023-12-19T14:52:09.039107+0000","last_active":"2023-12-19T14:52:09.053264+0000","last_peered":"2023-12-19T14:52:09.053264+0000","last_clean":"2023-12-19T14:52:09.053264+0000","last_became_active":"2023-12-19T14:52:09.038746+0000","last_became_peered":"2023-12-19T14:52:09.038746+0000","last_unstale":"2023-12-19T14:52:09.053264+0000","last_undegraded":"2023-12-19T14:52:09.053264+0000","last_fullsized":"2023-12-19T14:52:09.053264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:09:32.544274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042682+0000","last_change":"2023-12-19T14:52:09.035192+0000","last_active":"2023-12-19T14:52:09.042682+0000","last_peered":"2023-12-19T14:52:09.042682+0000","last_clean":"2023-12-19T14:52:09.042682+0000","last_became_active":"2023-12-19T14:52:09.034892+0000","last_became_peered":"2023-12-19T14:52:09.034892+0000","last_unstale":"2023-12-19T14:52:09.042682+0000","last_undegraded":"2023-12-19T14:52:09.042682+0000","last_fullsized":"2023-12-19T14:52:09.042682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:53.099678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051944+0000","last_change":"2023-12-19T14:52:09.042731+0000","last_active":"2023-12-19T14:52:09.051944+0000","last_peered":"2023-12-19T14:52:09.051944+0000","last_clean":"2023-12-19T14:52:09.051944+0000","last_became_active":"2023-12-19T14:52:09.038819+0000","last_became_peered":"2023-12-19T14:52:09.038819+0000","last_unstale":"2023-12-19T14:52:09.051944+0000","last_undegraded":"2023-12-19T14:52:09.051944+0000","last_fullsized":"2023-12-19T14:52:09.051944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:44.147509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.031110+0000","last_change":"2023-12-19T14:52:09.025661+0000","last_active":"2023-12-19T14:52:09.031110+0000","last_peered":"2023-12-19T14:52:09.031110+0000","last_clean":"2023-12-19T14:52:09.031110+0000","last_became_active":"2023-12-19T14:52:09.025379+0000","last_became_peered":"2023-12-19T14:52:09.025379+0000","last_unstale":"2023-12-19T14:52:09.031110+0000","last_undegraded":"2023-12-19T14:52:09.031110+0000","last_fullsized":"2023-12-19T14:52:09.031110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:59.545272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043194+0000","last_change":"2023-12-19T14:52:09.035334+0000","last_active":"2023-12-19T14:52:09.043194+0000","last_peered":"2023-12-19T14:52:09.043194+0000","last_clean":"2023-12-19T14:52:09.043194+0000","last_became_active":"2023-12-19T14:52:09.034998+0000","last_became_peered":"2023-12-19T14:52:09.034998+0000","last_unstale":"2023-12-19T14:52:09.043194+0000","last_undegraded":"2023-12-19T14:52:09.043194+0000","last_fullsized":"2023-12-19T14:52:09.043194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:44.147509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.061765+0000","last_change":"2023-12-19T14:52:08.022078+0000","last_active":"2023-12-19T14:52:09.061765+0000","last_peered":"2023-12-19T14:52:09.061765+0000","last_clean":"2023-12-19T14:52:09.061765+0000","last_became_active":"2023-12-19T14:52:08.020328+0000","last_became_peered":"2023-12-19T14:52:08.020328+0000","last_unstale":"2023-12-19T14:52:09.061765+0000","last_undegraded":"2023-12-19T14:52:09.061765+0000","last_fullsized":"2023-12-19T14:52:09.061765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:27.268706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045077+0000","last_change":"2023-12-19T14:52:09.040850+0000","last_active":"2023-12-19T14:52:09.045077+0000","last_peered":"2023-12-19T14:52:09.045077+0000","last_clean":"2023-12-19T14:52:09.045077+0000","last_became_active":"2023-12-19T14:52:09.040540+0000","last_became_peered":"2023-12-19T14:52:09.040540+0000","last_unstale":"2023-12-19T14:52:09.045077+0000","last_undegraded":"2023-12-19T14:52:09.045077+0000","last_fullsized":"2023-12-19T14:52:09.045077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:43:01.592858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068289+0000","last_change":"2023-12-19T14:52:08.040833+0000","last_active":"2023-12-19T14:52:09.068289+0000","last_peered":"2023-12-19T14:52:09.068289+0000","last_clean":"2023-12-19T14:52:09.068289+0000","last_became_active":"2023-12-19T14:52:08.040279+0000","last_became_peered":"2023-12-19T14:52:08.040279+0000","last_unstale":"2023-12-19T14:52:09.068289+0000","last_undegraded":"2023-12-19T14:52:09.068289+0000","last_fullsized":"2023-12-19T14:52:09.068289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:14.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045567+0000","last_change":"2023-12-19T14:52:09.043710+0000","last_active":"2023-12-19T14:52:09.045567+0000","last_peered":"2023-12-19T14:52:09.045567+0000","last_clean":"2023-12-19T14:52:09.045567+0000","last_became_active":"2023-12-19T14:52:09.043236+0000","last_became_peered":"2023-12-19T14:52:09.043236+0000","last_unstale":"2023-12-19T14:52:09.045567+0000","last_undegraded":"2023-12-19T14:52:09.045567+0000","last_fullsized":"2023-12-19T14:52:09.045567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:05.260747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.071154+0000","last_change":"2023-12-19T14:52:08.027276+0000","last_active":"2023-12-19T14:52:09.071154+0000","last_peered":"2023-12-19T14:52:09.071154+0000","last_clean":"2023-12-19T14:52:09.071154+0000","last_became_active":"2023-12-19T14:52:08.026691+0000","last_became_peered":"2023-12-19T14:52:08.026691+0000","last_unstale":"2023-12-19T14:52:09.071154+0000","last_undegraded":"2023-12-19T14:52:09.071154+0000","last_fullsized":"2023-12-19T14:52:09.071154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:03:42.400730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047313+0000","last_change":"2023-12-19T14:52:09.041963+0000","last_active":"2023-12-19T14:52:09.047313+0000","last_peered":"2023-12-19T14:52:09.047313+0000","last_clean":"2023-12-19T14:52:09.047313+0000","last_became_active":"2023-12-19T14:52:09.041308+0000","last_became_peered":"2023-12-19T14:52:09.041308+0000","last_unstale":"2023-12-19T14:52:09.047313+0000","last_undegraded":"2023-12-19T14:52:09.047313+0000","last_fullsized":"2023-12-19T14:52:09.047313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:57.525750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.056734+0000","last_change":"2023-12-19T14:52:08.034807+0000","last_active":"2023-12-19T14:52:09.056734+0000","last_peered":"2023-12-19T14:52:09.056734+0000","last_clean":"2023-12-19T14:52:09.056734+0000","last_became_active":"2023-12-19T14:52:08.034424+0000","last_became_peered":"2023-12-19T14:52:08.034424+0000","last_unstale":"2023-12-19T14:52:09.056734+0000","last_undegraded":"2023-12-19T14:52:09.056734+0000","last_fullsized":"2023-12-19T14:52:09.056734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:56.512280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046003+0000","last_change":"2023-12-19T14:52:09.044433+0000","last_active":"2023-12-19T14:52:09.046003+0000","last_peered":"2023-12-19T14:52:09.046003+0000","last_clean":"2023-12-19T14:52:09.046003+0000","last_became_active":"2023-12-19T14:52:09.044083+0000","last_became_peered":"2023-12-19T14:52:09.044083+0000","last_unstale":"2023-12-19T14:52:09.046003+0000","last_undegraded":"2023-12-19T14:52:09.046003+0000","last_fullsized":"2023-12-19T14:52:09.046003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:56.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.066062+0000","last_change":"2023-12-19T14:52:08.034042+0000","last_active":"2023-12-19T14:52:09.066062+0000","last_peered":"2023-12-19T14:52:09.066062+0000","last_clean":"2023-12-19T14:52:09.066062+0000","last_became_active":"2023-12-19T14:52:08.033238+0000","last_became_peered":"2023-12-19T14:52:08.033238+0000","last_unstale":"2023-12-19T14:52:09.066062+0000","last_undegraded":"2023-12-19T14:52:09.066062+0000","last_fullsized":"2023-12-19T14:52:09.066062+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:04.365847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045297+0000","last_change":"2023-12-19T14:52:09.040510+0000","last_active":"2023-12-19T14:52:09.045297+0000","last_peered":"2023-12-19T14:52:09.045297+0000","last_clean":"2023-12-19T14:52:09.045297+0000","last_became_active":"2023-12-19T14:52:09.039004+0000","last_became_peered":"2023-12-19T14:52:09.039004+0000","last_unstale":"2023-12-19T14:52:09.045297+0000","last_undegraded":"2023-12-19T14:52:09.045297+0000","last_fullsized":"2023-12-19T14:52:09.045297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:13.392136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.062095+0000","last_change":"2023-12-19T14:52:08.020047+0000","last_active":"2023-12-19T14:52:09.062095+0000","last_peered":"2023-12-19T14:52:09.062095+0000","last_clean":"2023-12-19T14:52:09.062095+0000","last_became_active":"2023-12-19T14:52:08.019117+0000","last_became_peered":"2023-12-19T14:52:08.019117+0000","last_unstale":"2023-12-19T14:52:09.062095+0000","last_undegraded":"2023-12-19T14:52:09.062095+0000","last_fullsized":"2023-12-19T14:52:09.062095+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:12.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046363+0000","last_change":"2023-12-19T14:52:09.039377+0000","last_active":"2023-12-19T14:52:09.046363+0000","last_peered":"2023-12-19T14:52:09.046363+0000","last_clean":"2023-12-19T14:52:09.046363+0000","last_became_active":"2023-12-19T14:52:09.039011+0000","last_became_peered":"2023-12-19T14:52:09.039011+0000","last_unstale":"2023-12-19T14:52:09.046363+0000","last_undegraded":"2023-12-19T14:52:09.046363+0000","last_fullsized":"2023-12-19T14:52:09.046363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:01:38.929717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016383+0000","last_change":"2023-12-19T14:52:08.019134+0000","last_active":"2023-12-19T14:52:09.016383+0000","last_peered":"2023-12-19T14:52:09.016383+0000","last_clean":"2023-12-19T14:52:09.016383+0000","last_became_active":"2023-12-19T14:52:08.018648+0000","last_became_peered":"2023-12-19T14:52:08.018648+0000","last_unstale":"2023-12-19T14:52:09.016383+0000","last_undegraded":"2023-12-19T14:52:09.016383+0000","last_fullsized":"2023-12-19T14:52:09.016383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:56.166925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046428+0000","last_change":"2023-12-19T14:52:09.044533+0000","last_active":"2023-12-19T14:52:09.046428+0000","last_peered":"2023-12-19T14:52:09.046428+0000","last_clean":"2023-12-19T14:52:09.046428+0000","last_became_active":"2023-12-19T14:52:09.044221+0000","last_became_peered":"2023-12-19T14:52:09.044221+0000","last_unstale":"2023-12-19T14:52:09.046428+0000","last_undegraded":"2023-12-19T14:52:09.046428+0000","last_fullsized":"2023-12-19T14:52:09.046428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:42.953083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022399+0000","last_change":"2023-12-19T14:52:08.025323+0000","last_active":"2023-12-19T14:52:09.022399+0000","last_peered":"2023-12-19T14:52:09.022399+0000","last_clean":"2023-12-19T14:52:09.022399+0000","last_became_active":"2023-12-19T14:52:08.024950+0000","last_became_peered":"2023-12-19T14:52:08.024950+0000","last_unstale":"2023-12-19T14:52:09.022399+0000","last_undegraded":"2023-12-19T14:52:09.022399+0000","last_fullsized":"2023-12-19T14:52:09.022399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:33:11.975978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.035663+0000","last_change":"2023-12-19T14:52:09.032340+0000","last_active":"2023-12-19T14:52:09.035663+0000","last_peered":"2023-12-19T14:52:09.035663+0000","last_clean":"2023-12-19T14:52:09.035663+0000","last_became_active":"2023-12-19T14:52:09.029110+0000","last_became_peered":"2023-12-19T14:52:09.029110+0000","last_unstale":"2023-12-19T14:52:09.035663+0000","last_undegraded":"2023-12-19T14:52:09.035663+0000","last_fullsized":"2023-12-19T14:52:09.035663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:27.440702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022098+0000","last_change":"2023-12-19T14:52:08.024083+0000","last_active":"2023-12-19T14:52:09.022098+0000","last_peered":"2023-12-19T14:52:09.022098+0000","last_clean":"2023-12-19T14:52:09.022098+0000","last_became_active":"2023-12-19T14:52:08.023712+0000","last_became_peered":"2023-12-19T14:52:08.023712+0000","last_unstale":"2023-12-19T14:52:09.022098+0000","last_undegraded":"2023-12-19T14:52:09.022098+0000","last_fullsized":"2023-12-19T14:52:09.022098+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:04.247276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042011+0000","last_change":"2023-12-19T14:52:09.036542+0000","last_active":"2023-12-19T14:52:09.042011+0000","last_peered":"2023-12-19T14:52:09.042011+0000","last_clean":"2023-12-19T14:52:09.042011+0000","last_became_active":"2023-12-19T14:52:09.035966+0000","last_became_peered":"2023-12-19T14:52:09.035966+0000","last_unstale":"2023-12-19T14:52:09.042011+0000","last_undegraded":"2023-12-19T14:52:09.042011+0000","last_fullsized":"2023-12-19T14:52:09.042011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:36.950907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019733+0000","last_change":"2023-12-19T14:52:08.036732+0000","last_active":"2023-12-19T14:52:09.019733+0000","last_peered":"2023-12-19T14:52:09.019733+0000","last_clean":"2023-12-19T14:52:09.019733+0000","last_became_active":"2023-12-19T14:52:08.036331+0000","last_became_peered":"2023-12-19T14:52:08.036331+0000","last_unstale":"2023-12-19T14:52:09.019733+0000","last_undegraded":"2023-12-19T14:52:09.019733+0000","last_fullsized":"2023-12-19T14:52:09.019733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.134038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054485+0000","last_change":"2023-12-19T14:52:09.035652+0000","last_active":"2023-12-19T14:52:09.054485+0000","last_peered":"2023-12-19T14:52:09.054485+0000","last_clean":"2023-12-19T14:52:09.054485+0000","last_became_active":"2023-12-19T14:52:09.035321+0000","last_became_peered":"2023-12-19T14:52:09.035321+0000","last_unstale":"2023-12-19T14:52:09.054485+0000","last_undegraded":"2023-12-19T14:52:09.054485+0000","last_fullsized":"2023-12-19T14:52:09.054485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:02:06.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.063930+0000","last_change":"2023-12-19T14:52:08.032194+0000","last_active":"2023-12-19T14:52:09.063930+0000","last_peered":"2023-12-19T14:52:09.063930+0000","last_clean":"2023-12-19T14:52:09.063930+0000","last_became_active":"2023-12-19T14:52:08.031812+0000","last_became_peered":"2023-12-19T14:52:08.031812+0000","last_unstale":"2023-12-19T14:52:09.063930+0000","last_undegraded":"2023-12-19T14:52:09.063930+0000","last_fullsized":"2023-12-19T14:52:09.063930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:03:42.400730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045396+0000","last_change":"2023-12-19T14:52:09.038267+0000","last_active":"2023-12-19T14:52:09.045396+0000","last_peered":"2023-12-19T14:52:09.045396+0000","last_clean":"2023-12-19T14:52:09.045396+0000","last_became_active":"2023-12-19T14:52:09.037992+0000","last_became_peered":"2023-12-19T14:52:09.037992+0000","last_unstale":"2023-12-19T14:52:09.045396+0000","last_undegraded":"2023-12-19T14:52:09.045396+0000","last_fullsized":"2023-12-19T14:52:09.045396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:00.157458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068188+0000","last_change":"2023-12-19T14:52:08.040447+0000","last_active":"2023-12-19T14:52:09.068188+0000","last_peered":"2023-12-19T14:52:09.068188+0000","last_clean":"2023-12-19T14:52:09.068188+0000","last_became_active":"2023-12-19T14:52:08.036832+0000","last_became_peered":"2023-12-19T14:52:08.036832+0000","last_unstale":"2023-12-19T14:52:09.068188+0000","last_undegraded":"2023-12-19T14:52:09.068188+0000","last_fullsized":"2023-12-19T14:52:09.068188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:05.712073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045038+0000","last_change":"2023-12-19T14:52:09.040709+0000","last_active":"2023-12-19T14:52:09.045038+0000","last_peered":"2023-12-19T14:52:09.045038+0000","last_clean":"2023-12-19T14:52:09.045038+0000","last_became_active":"2023-12-19T14:52:09.040041+0000","last_became_peered":"2023-12-19T14:52:09.040041+0000","last_unstale":"2023-12-19T14:52:09.045038+0000","last_undegraded":"2023-12-19T14:52:09.045038+0000","last_fullsized":"2023-12-19T14:52:09.045038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:16:56.760376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068331+0000","last_change":"2023-12-19T14:52:08.037963+0000","last_active":"2023-12-19T14:52:09.068331+0000","last_peered":"2023-12-19T14:52:09.068331+0000","last_clean":"2023-12-19T14:52:09.068331+0000","last_became_active":"2023-12-19T14:52:08.037688+0000","last_became_peered":"2023-12-19T14:52:08.037688+0000","last_unstale":"2023-12-19T14:52:09.068331+0000","last_undegraded":"2023-12-19T14:52:09.068331+0000","last_fullsized":"2023-12-19T14:52:09.068331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:14.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051932+0000","last_change":"2023-12-19T14:52:09.038137+0000","last_active":"2023-12-19T14:52:09.051932+0000","last_peered":"2023-12-19T14:52:09.051932+0000","last_clean":"2023-12-19T14:52:09.051932+0000","last_became_active":"2023-12-19T14:52:09.037768+0000","last_became_peered":"2023-12-19T14:52:09.037768+0000","last_unstale":"2023-12-19T14:52:09.051932+0000","last_undegraded":"2023-12-19T14:52:09.051932+0000","last_fullsized":"2023-12-19T14:52:09.051932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:01.072438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016715+0000","last_change":"2023-12-19T14:52:08.031686+0000","last_active":"2023-12-19T14:52:09.016715+0000","last_peered":"2023-12-19T14:52:09.016715+0000","last_clean":"2023-12-19T14:52:09.016715+0000","last_became_active":"2023-12-19T14:52:08.031387+0000","last_became_peered":"2023-12-19T14:52:08.031387+0000","last_unstale":"2023-12-19T14:52:09.016715+0000","last_undegraded":"2023-12-19T14:52:09.016715+0000","last_fullsized":"2023-12-19T14:52:09.016715+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:56.628667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039832+0000","last_change":"2023-12-19T14:52:09.036312+0000","last_active":"2023-12-19T14:52:09.039832+0000","last_peered":"2023-12-19T14:52:09.039832+0000","last_clean":"2023-12-19T14:52:09.039832+0000","last_became_active":"2023-12-19T14:52:09.036071+0000","last_became_peered":"2023-12-19T14:52:09.036071+0000","last_unstale":"2023-12-19T14:52:09.039832+0000","last_undegraded":"2023-12-19T14:52:09.039832+0000","last_fullsized":"2023-12-19T14:52:09.039832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:19.264675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019445+0000","last_change":"2023-12-19T14:52:08.020553+0000","last_active":"2023-12-19T14:52:09.019445+0000","last_peered":"2023-12-19T14:52:09.019445+0000","last_clean":"2023-12-19T14:52:09.019445+0000","last_became_active":"2023-12-19T14:52:08.020292+0000","last_became_peered":"2023-12-19T14:52:08.020292+0000","last_unstale":"2023-12-19T14:52:09.019445+0000","last_undegraded":"2023-12-19T14:52:09.019445+0000","last_fullsized":"2023-12-19T14:52:09.019445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:24.637246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052038+0000","last_change":"2023-12-19T14:52:09.038339+0000","last_active":"2023-12-19T14:52:09.052038+0000","last_peered":"2023-12-19T14:52:09.052038+0000","last_clean":"2023-12-19T14:52:09.052038+0000","last_became_active":"2023-12-19T14:52:09.037978+0000","last_became_peered":"2023-12-19T14:52:09.037978+0000","last_unstale":"2023-12-19T14:52:09.052038+0000","last_undegraded":"2023-12-19T14:52:09.052038+0000","last_fullsized":"2023-12-19T14:52:09.052038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:43:11.519796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.065994+0000","last_change":"2023-12-19T14:52:08.038201+0000","last_active":"2023-12-19T14:52:09.065994+0000","last_peered":"2023-12-19T14:52:09.065994+0000","last_clean":"2023-12-19T14:52:09.065994+0000","last_became_active":"2023-12-19T14:52:08.037044+0000","last_became_peered":"2023-12-19T14:52:08.037044+0000","last_unstale":"2023-12-19T14:52:09.065994+0000","last_undegraded":"2023-12-19T14:52:09.065994+0000","last_fullsized":"2023-12-19T14:52:09.065994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.134038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039769+0000","last_change":"2023-12-19T14:52:09.033290+0000","last_active":"2023-12-19T14:52:09.039769+0000","last_peered":"2023-12-19T14:52:09.039769+0000","last_clean":"2023-12-19T14:52:09.039769+0000","last_became_active":"2023-12-19T14:52:09.033047+0000","last_became_peered":"2023-12-19T14:52:09.033047+0000","last_unstale":"2023-12-19T14:52:09.039769+0000","last_undegraded":"2023-12-19T14:52:09.039769+0000","last_fullsized":"2023-12-19T14:52:09.039769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:43:11.519796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.065061+0000","last_change":"2023-12-19T14:52:08.040052+0000","last_active":"2023-12-19T14:52:09.065061+0000","last_peered":"2023-12-19T14:52:09.065061+0000","last_clean":"2023-12-19T14:52:09.065061+0000","last_became_active":"2023-12-19T14:52:08.039686+0000","last_became_peered":"2023-12-19T14:52:08.039686+0000","last_unstale":"2023-12-19T14:52:09.065061+0000","last_undegraded":"2023-12-19T14:52:09.065061+0000","last_fullsized":"2023-12-19T14:52:09.065061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:20:52.055179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046480+0000","last_change":"2023-12-19T14:52:09.044018+0000","last_active":"2023-12-19T14:52:09.046480+0000","last_peered":"2023-12-19T14:52:09.046480+0000","last_clean":"2023-12-19T14:52:09.046480+0000","last_became_active":"2023-12-19T14:52:09.043679+0000","last_became_peered":"2023-12-19T14:52:09.043679+0000","last_unstale":"2023-12-19T14:52:09.046480+0000","last_undegraded":"2023-12-19T14:52:09.046480+0000","last_fullsized":"2023-12-19T14:52:09.046480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.063373+0000","last_change":"2023-12-19T14:52:08.020383+0000","last_active":"2023-12-19T14:52:09.063373+0000","last_peered":"2023-12-19T14:52:09.063373+0000","last_clean":"2023-12-19T14:52:09.063373+0000","last_became_active":"2023-12-19T14:52:08.020063+0000","last_became_peered":"2023-12-19T14:52:08.020063+0000","last_unstale":"2023-12-19T14:52:09.063373+0000","last_undegraded":"2023-12-19T14:52:09.063373+0000","last_fullsized":"2023-12-19T14:52:09.063373+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:14:24.180643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045407+0000","last_change":"2023-12-19T14:52:09.040393+0000","last_active":"2023-12-19T14:52:09.045407+0000","last_peered":"2023-12-19T14:52:09.045407+0000","last_clean":"2023-12-19T14:52:09.045407+0000","last_became_active":"2023-12-19T14:52:09.040117+0000","last_became_peered":"2023-12-19T14:52:09.040117+0000","last_unstale":"2023-12-19T14:52:09.045407+0000","last_undegraded":"2023-12-19T14:52:09.045407+0000","last_fullsized":"2023-12-19T14:52:09.045407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.056347+0000","last_change":"2023-12-19T14:52:08.018391+0000","last_active":"2023-12-19T14:52:09.056347+0000","last_peered":"2023-12-19T14:52:09.056347+0000","last_clean":"2023-12-19T14:52:09.056347+0000","last_became_active":"2023-12-19T14:52:08.018089+0000","last_became_peered":"2023-12-19T14:52:08.018089+0000","last_unstale":"2023-12-19T14:52:09.056347+0000","last_undegraded":"2023-12-19T14:52:09.056347+0000","last_fullsized":"2023-12-19T14:52:09.056347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:15:41.764443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040260+0000","last_change":"2023-12-19T14:52:09.032437+0000","last_active":"2023-12-19T14:52:09.040260+0000","last_peered":"2023-12-19T14:52:09.040260+0000","last_clean":"2023-12-19T14:52:09.040260+0000","last_became_active":"2023-12-19T14:52:09.032104+0000","last_became_peered":"2023-12-19T14:52:09.032104+0000","last_unstale":"2023-12-19T14:52:09.040260+0000","last_undegraded":"2023-12-19T14:52:09.040260+0000","last_fullsized":"2023-12-19T14:52:09.040260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:56.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.018667+0000","last_change":"2023-12-19T14:52:08.036151+0000","last_active":"2023-12-19T14:52:09.018667+0000","last_peered":"2023-12-19T14:52:09.018667+0000","last_clean":"2023-12-19T14:52:09.018667+0000","last_became_active":"2023-12-19T14:52:08.035753+0000","last_became_peered":"2023-12-19T14:52:08.035753+0000","last_unstale":"2023-12-19T14:52:09.018667+0000","last_undegraded":"2023-12-19T14:52:09.018667+0000","last_fullsized":"2023-12-19T14:52:09.018667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.074810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.041338+0000","last_change":"2023-12-19T14:52:09.034497+0000","last_active":"2023-12-19T14:52:09.041338+0000","last_peered":"2023-12-19T14:52:09.041338+0000","last_clean":"2023-12-19T14:52:09.041338+0000","last_became_active":"2023-12-19T14:52:09.034189+0000","last_became_peered":"2023-12-19T14:52:09.034189+0000","last_unstale":"2023-12-19T14:52:09.041338+0000","last_undegraded":"2023-12-19T14:52:09.041338+0000","last_fullsized":"2023-12-19T14:52:09.041338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:00.328643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068461+0000","last_change":"2023-12-19T14:52:08.028420+0000","last_active":"2023-12-19T14:52:09.068461+0000","last_peered":"2023-12-19T14:52:09.068461+0000","last_clean":"2023-12-19T14:52:09.068461+0000","last_became_active":"2023-12-19T14:52:08.028149+0000","last_became_peered":"2023-12-19T14:52:08.028149+0000","last_unstale":"2023-12-19T14:52:09.068461+0000","last_undegraded":"2023-12-19T14:52:09.068461+0000","last_fullsized":"2023-12-19T14:52:09.068461+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:24.637246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040586+0000","last_change":"2023-12-19T14:52:09.034234+0000","last_active":"2023-12-19T14:52:09.040586+0000","last_peered":"2023-12-19T14:52:09.040586+0000","last_clean":"2023-12-19T14:52:09.040586+0000","last_became_active":"2023-12-19T14:52:09.033814+0000","last_became_peered":"2023-12-19T14:52:09.033814+0000","last_unstale":"2023-12-19T14:52:09.040586+0000","last_undegraded":"2023-12-19T14:52:09.040586+0000","last_fullsized":"2023-12-19T14:52:09.040586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:46:39.304602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015366+0000","last_change":"2023-12-19T14:52:08.039472+0000","last_active":"2023-12-19T14:52:09.015366+0000","last_peered":"2023-12-19T14:52:09.015366+0000","last_clean":"2023-12-19T14:52:09.015366+0000","last_became_active":"2023-12-19T14:52:08.038342+0000","last_became_peered":"2023-12-19T14:52:08.038342+0000","last_unstale":"2023-12-19T14:52:09.015366+0000","last_undegraded":"2023-12-19T14:52:09.015366+0000","last_fullsized":"2023-12-19T14:52:09.015366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:53.467987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051900+0000","last_change":"2023-12-19T14:52:09.042382+0000","last_active":"2023-12-19T14:52:09.051900+0000","last_peered":"2023-12-19T14:52:09.051900+0000","last_clean":"2023-12-19T14:52:09.051900+0000","last_became_active":"2023-12-19T14:52:09.042051+0000","last_became_peered":"2023-12-19T14:52:09.042051+0000","last_unstale":"2023-12-19T14:52:09.051900+0000","last_undegraded":"2023-12-19T14:52:09.051900+0000","last_fullsized":"2023-12-19T14:52:09.051900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:01:38.929717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022719+0000","last_change":"2023-12-19T14:52:08.029511+0000","last_active":"2023-12-19T14:52:09.022719+0000","last_peered":"2023-12-19T14:52:09.022719+0000","last_clean":"2023-12-19T14:52:09.022719+0000","last_became_active":"2023-12-19T14:52:08.028039+0000","last_became_peered":"2023-12-19T14:52:08.028039+0000","last_unstale":"2023-12-19T14:52:09.022719+0000","last_undegraded":"2023-12-19T14:52:09.022719+0000","last_fullsized":"2023-12-19T14:52:09.022719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:55.601122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.025305+0000","last_change":"2023-12-19T14:52:09.023628+0000","last_active":"2023-12-19T14:52:09.025305+0000","last_peered":"2023-12-19T14:52:09.025305+0000","last_clean":"2023-12-19T14:52:09.025305+0000","last_became_active":"2023-12-19T14:52:09.022637+0000","last_became_peered":"2023-12-19T14:52:09.022637+0000","last_unstale":"2023-12-19T14:52:09.025305+0000","last_undegraded":"2023-12-19T14:52:09.025305+0000","last_fullsized":"2023-12-19T14:52:09.025305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:41.755184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.032923+0000","last_change":"2023-12-19T14:52:08.017821+0000","last_active":"2023-12-19T14:52:09.032923+0000","last_peered":"2023-12-19T14:52:09.032923+0000","last_clean":"2023-12-19T14:52:09.032923+0000","last_became_active":"2023-12-19T14:52:08.017436+0000","last_became_peered":"2023-12-19T14:52:08.017436+0000","last_unstale":"2023-12-19T14:52:09.032923+0000","last_undegraded":"2023-12-19T14:52:09.032923+0000","last_fullsized":"2023-12-19T14:52:09.032923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:32.518002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.031145+0000","last_change":"2023-12-19T14:52:09.027220+0000","last_active":"2023-12-19T14:52:09.031145+0000","last_peered":"2023-12-19T14:52:09.031145+0000","last_clean":"2023-12-19T14:52:09.031145+0000","last_became_active":"2023-12-19T14:52:09.025613+0000","last_became_peered":"2023-12-19T14:52:09.025613+0000","last_unstale":"2023-12-19T14:52:09.031145+0000","last_undegraded":"2023-12-19T14:52:09.031145+0000","last_fullsized":"2023-12-19T14:52:09.031145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:24.852226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.058083+0000","last_change":"2023-12-19T14:52:08.031733+0000","last_active":"2023-12-19T14:52:09.058083+0000","last_peered":"2023-12-19T14:52:09.058083+0000","last_clean":"2023-12-19T14:52:09.058083+0000","last_became_active":"2023-12-19T14:52:08.031422+0000","last_became_peered":"2023-12-19T14:52:08.031422+0000","last_unstale":"2023-12-19T14:52:09.058083+0000","last_undegraded":"2023-12-19T14:52:09.058083+0000","last_fullsized":"2023-12-19T14:52:09.058083+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:39:29.554303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042212+0000","last_change":"2023-12-19T14:52:09.037799+0000","last_active":"2023-12-19T14:52:09.042212+0000","last_peered":"2023-12-19T14:52:09.042212+0000","last_clean":"2023-12-19T14:52:09.042212+0000","last_became_active":"2023-12-19T14:52:09.037498+0000","last_became_peered":"2023-12-19T14:52:09.037498+0000","last_unstale":"2023-12-19T14:52:09.042212+0000","last_undegraded":"2023-12-19T14:52:09.042212+0000","last_fullsized":"2023-12-19T14:52:09.042212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:53.099678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022252+0000","last_change":"2023-12-19T14:52:08.022071+0000","last_active":"2023-12-19T14:52:09.022252+0000","last_peered":"2023-12-19T14:52:09.022252+0000","last_clean":"2023-12-19T14:52:09.022252+0000","last_became_active":"2023-12-19T14:52:08.021756+0000","last_became_peered":"2023-12-19T14:52:08.021756+0000","last_unstale":"2023-12-19T14:52:09.022252+0000","last_undegraded":"2023-12-19T14:52:09.022252+0000","last_fullsized":"2023-12-19T14:52:09.022252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:31.298025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.030579+0000","last_change":"2023-12-19T14:52:09.027470+0000","last_active":"2023-12-19T14:52:09.030579+0000","last_peered":"2023-12-19T14:52:09.030579+0000","last_clean":"2023-12-19T14:52:09.030579+0000","last_became_active":"2023-12-19T14:52:09.027194+0000","last_became_peered":"2023-12-19T14:52:09.027194+0000","last_unstale":"2023-12-19T14:52:09.030579+0000","last_undegraded":"2023-12-19T14:52:09.030579+0000","last_fullsized":"2023-12-19T14:52:09.030579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:50.893600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022909+0000","last_change":"2023-12-19T14:52:08.036747+0000","last_active":"2023-12-19T14:52:09.022909+0000","last_peered":"2023-12-19T14:52:09.022909+0000","last_clean":"2023-12-19T14:52:09.022909+0000","last_became_active":"2023-12-19T14:52:08.036283+0000","last_became_peered":"2023-12-19T14:52:08.036283+0000","last_unstale":"2023-12-19T14:52:09.022909+0000","last_undegraded":"2023-12-19T14:52:09.022909+0000","last_fullsized":"2023-12-19T14:52:09.022909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:14.721168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047751+0000","last_change":"2023-12-19T14:52:09.033922+0000","last_active":"2023-12-19T14:52:09.047751+0000","last_peered":"2023-12-19T14:52:09.047751+0000","last_clean":"2023-12-19T14:52:09.047751+0000","last_became_active":"2023-12-19T14:52:09.033637+0000","last_became_peered":"2023-12-19T14:52:09.033637+0000","last_unstale":"2023-12-19T14:52:09.047751+0000","last_undegraded":"2023-12-19T14:52:09.047751+0000","last_fullsized":"2023-12-19T14:52:09.047751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:03:00.136172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016725+0000","last_change":"2023-12-19T14:52:08.031238+0000","last_active":"2023-12-19T14:52:09.016725+0000","last_peered":"2023-12-19T14:52:09.016725+0000","last_clean":"2023-12-19T14:52:09.016725+0000","last_became_active":"2023-12-19T14:52:08.030960+0000","last_became_peered":"2023-12-19T14:52:08.030960+0000","last_unstale":"2023-12-19T14:52:09.016725+0000","last_undegraded":"2023-12-19T14:52:09.016725+0000","last_fullsized":"2023-12-19T14:52:09.016725+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:47.166820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045780+0000","last_change":"2023-12-19T14:52:09.043548+0000","last_active":"2023-12-19T14:52:09.045780+0000","last_peered":"2023-12-19T14:52:09.045780+0000","last_clean":"2023-12-19T14:52:09.045780+0000","last_became_active":"2023-12-19T14:52:09.043249+0000","last_became_peered":"2023-12-19T14:52:09.043249+0000","last_unstale":"2023-12-19T14:52:09.045780+0000","last_undegraded":"2023-12-19T14:52:09.045780+0000","last_fullsized":"2023-12-19T14:52:09.045780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:42:49.888902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015493+0000","last_change":"2023-12-19T14:52:08.036803+0000","last_active":"2023-12-19T14:52:09.015493+0000","last_peered":"2023-12-19T14:52:09.015493+0000","last_clean":"2023-12-19T14:52:09.015493+0000","last_became_active":"2023-12-19T14:52:08.036477+0000","last_became_peered":"2023-12-19T14:52:08.036477+0000","last_unstale":"2023-12-19T14:52:09.015493+0000","last_undegraded":"2023-12-19T14:52:09.015493+0000","last_fullsized":"2023-12-19T14:52:09.015493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:41.939612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052239+0000","last_change":"2023-12-19T14:52:09.044777+0000","last_active":"2023-12-19T14:52:09.052239+0000","last_peered":"2023-12-19T14:52:09.052239+0000","last_clean":"2023-12-19T14:52:09.052239+0000","last_became_active":"2023-12-19T14:52:09.044457+0000","last_became_peered":"2023-12-19T14:52:09.044457+0000","last_unstale":"2023-12-19T14:52:09.052239+0000","last_undegraded":"2023-12-19T14:52:09.052239+0000","last_fullsized":"2023-12-19T14:52:09.052239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:50.893600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015643+0000","last_change":"2023-12-19T14:52:08.033941+0000","last_active":"2023-12-19T14:52:09.015643+0000","last_peered":"2023-12-19T14:52:09.015643+0000","last_clean":"2023-12-19T14:52:09.015643+0000","last_became_active":"2023-12-19T14:52:08.033664+0000","last_became_peered":"2023-12-19T14:52:08.033664+0000","last_unstale":"2023-12-19T14:52:09.015643+0000","last_undegraded":"2023-12-19T14:52:09.015643+0000","last_fullsized":"2023-12-19T14:52:09.015643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:04.277310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047632+0000","last_change":"2023-12-19T14:52:09.039210+0000","last_active":"2023-12-19T14:52:09.047632+0000","last_peered":"2023-12-19T14:52:09.047632+0000","last_clean":"2023-12-19T14:52:09.047632+0000","last_became_active":"2023-12-19T14:52:09.038893+0000","last_became_peered":"2023-12-19T14:52:09.038893+0000","last_unstale":"2023-12-19T14:52:09.047632+0000","last_undegraded":"2023-12-19T14:52:09.047632+0000","last_fullsized":"2023-12-19T14:52:09.047632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:06.248039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016998+0000","last_change":"2023-12-19T14:52:08.027246+0000","last_active":"2023-12-19T14:52:09.016998+0000","last_peered":"2023-12-19T14:52:09.016998+0000","last_clean":"2023-12-19T14:52:09.016998+0000","last_became_active":"2023-12-19T14:52:08.026970+0000","last_became_peered":"2023-12-19T14:52:08.026970+0000","last_unstale":"2023-12-19T14:52:09.016998+0000","last_undegraded":"2023-12-19T14:52:09.016998+0000","last_fullsized":"2023-12-19T14:52:09.016998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:16.609715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040718+0000","last_change":"2023-12-19T14:52:09.034829+0000","last_active":"2023-12-19T14:52:09.040718+0000","last_peered":"2023-12-19T14:52:09.040718+0000","last_clean":"2023-12-19T14:52:09.040718+0000","last_became_active":"2023-12-19T14:52:09.034074+0000","last_became_peered":"2023-12-19T14:52:09.034074+0000","last_unstale":"2023-12-19T14:52:09.040718+0000","last_undegraded":"2023-12-19T14:52:09.040718+0000","last_fullsized":"2023-12-19T14:52:09.040718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:25.650717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.024049+0000","last_change":"2023-12-19T14:52:08.033730+0000","last_active":"2023-12-19T14:52:09.024049+0000","last_peered":"2023-12-19T14:52:09.024049+0000","last_clean":"2023-12-19T14:52:09.024049+0000","last_became_active":"2023-12-19T14:52:08.033421+0000","last_became_peered":"2023-12-19T14:52:08.033421+0000","last_unstale":"2023-12-19T14:52:09.024049+0000","last_undegraded":"2023-12-19T14:52:09.024049+0000","last_fullsized":"2023-12-19T14:52:09.024049+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:48:00.344139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019508+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:52:09.019508+0000","last_peered":"2023-12-19T14:52:09.019508+0000","last_clean":"2023-12-19T14:52:09.019508+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:52:09.019508+0000","last_undegraded":"2023-12-19T14:52:09.019508+0000","last_fullsized":"2023-12-19T14:52:09.019508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310948,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060892,"statfs":{"total":96636764160,"available":96318353408,"internally_reserved":0,"allocated":1404928,"data_stored":985447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.109,"15min":1.099},"min":{"1min":0.59699999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.48,"5min":1.579,"15min":1.579},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.123,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.488,"5min":1.5669999999999999,"15min":2.2799999999999998},"last":1.246}]},{"osd":1,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1120000000000001,"15min":1.0940000000000001},"min":{"1min":0.76400000000000001,"5min":0.67600000000000005,"15min":0.59299999999999997},"max":{"1min":1.48,"5min":1.73,"15min":1.73},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.181,"15min":1.1799999999999999},"min":{"1min":0.89400000000000002,"5min":0.77700000000000002,"15min":0.63},"max":{"1min":1.488,"5min":1.6060000000000001,"15min":1.984},"last":0.81999999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.236,"15min":1.2310000000000001},"min":{"1min":0.95399999999999996,"5min":0.67900000000000005,"15min":0.623},"max":{"1min":1.534,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.2929999999999999,"15min":1.296},"min":{"1min":1.0640000000000001,"5min":0.85199999999999998,"15min":0.70999999999999996},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.96799999999999997,"5min":0.879,"15min":0.76200000000000001},"max":{"1min":1.5680000000000001,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3180000000000001,"15min":1.3200000000000001},"min":{"1min":1.0189999999999999,"5min":0.999,"15min":0.79700000000000004},"max":{"1min":1.6379999999999999,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.159,"15min":1.206},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.58999999999999997},"max":{"1min":1.4890000000000001,"5min":1.75,"15min":2.1930000000000001},"last":1.113},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2050000000000001,"15min":1.2569999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.67500000000000004},"max":{"1min":1.6180000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.5389999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2050000000000001,"15min":1.256},"min":{"1min":0.84399999999999997,"5min":0.69599999999999995,"15min":0.69099999999999995},"max":{"1min":1.675,"5min":1.8109999999999999,"15min":1.976},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.276,"15min":1.3620000000000001},"min":{"1min":0.84099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.581,"5min":1.6950000000000001,"15min":2.3620000000000001},"last":1.3660000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.387,"15min":1.4470000000000001},"min":{"1min":0.94099999999999995,"5min":0.88100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5669999999999999,"5min":1.8029999999999999,"15min":3.1589999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.393},"min":{"1min":0.96599999999999997,"5min":0.85499999999999998,"15min":0.70299999999999996},"max":{"1min":1.6850000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.431}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":823296,"data_stored":402187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1439999999999999,"15min":1.1919999999999999},"min":{"1min":0.70299999999999996,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.877,"15min":1.8939999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2090000000000001,"15min":1.2330000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.64000000000000001},"max":{"1min":1.512,"5min":1.6830000000000001,"15min":2.335},"last":1.7949999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.196,"15min":1.218},"min":{"1min":0.96599999999999997,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":1.325,"5min":1.7649999999999999,"15min":2.0339999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.274,"15min":1.3149999999999999},"min":{"1min":0.85599999999999998,"5min":0.84499999999999997,"15min":0.747},"max":{"1min":1.51,"5min":1.714,"15min":3.0990000000000002},"last":1.24}]},{"osd":2,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.23,"15min":1.276},"min":{"1min":0.99099999999999999,"5min":0.84999999999999998,"15min":0.78800000000000003},"max":{"1min":1.472,"5min":1.7390000000000001,"15min":2.4900000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.248,"15min":1.272},"min":{"1min":0.95399999999999996,"5min":0.70799999999999996,"15min":0.67100000000000004},"max":{"1min":1.5149999999999999,"5min":1.849,"15min":2.4550000000000001},"last":1.6279999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.319,"15min":1.3779999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.65800000000000003},"max":{"1min":1.7010000000000001,"5min":1.778,"15min":3.2330000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.329,"15min":1.3879999999999999},"min":{"1min":0.996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6759999999999999,"5min":1.879,"15min":3.194},"last":1.4530000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2110000000000001,"15min":1.292},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":1.776,"5min":2.073,"15min":3.0150000000000001},"last":1.825},{"interface":"front","average":{"1min":1.107,"5min":1.2789999999999999,"15min":1.383},"min":{"1min":0.79800000000000004,"5min":0.68500000000000005,"15min":0.57599999999999996},"max":{"1min":1.611,"5min":2.3450000000000002,"15min":3.4399999999999999},"last":2.0350000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2989999999999999,"15min":1.3959999999999999},"min":{"1min":1.0089999999999999,"5min":0.74199999999999999,"15min":0.63100000000000001},"max":{"1min":1.8580000000000001,"5min":2.0219999999999998,"15min":2.8620000000000001},"last":1.734},{"interface":"front","average":{"1min":1.335,"5min":1.331,"15min":1.4119999999999999},"min":{"1min":1.05,"5min":0.76300000000000001,"15min":0.51000000000000001},"max":{"1min":1.6299999999999999,"5min":1.7609999999999999,"15min":2.9420000000000002},"last":2.1110000000000002}]},{"osd":7,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4119999999999999,"15min":1.482},"min":{"1min":0.97299999999999998,"5min":0.93400000000000005,"15min":0.53000000000000003},"max":{"1min":1.766,"5min":2.0249999999999999,"15min":3.367},"last":2.214},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.462,"15min":1.5469999999999999},"min":{"1min":0.93100000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.97,"5min":2.234,"15min":3.2839999999999998},"last":1.974}]}]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":394955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0640000000000001,"15min":1.0580000000000001},"min":{"1min":0.71899999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.325,"5min":1.5800000000000001,"15min":1.619},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.141,"15min":1.133},"min":{"1min":0.78000000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6399999999999999,"5min":1.6579999999999999,"15min":1.718},"last":1.153}]},{"osd":1,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.286,"15min":1.2909999999999999},"min":{"1min":0.95999999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.554,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.202,"5min":1.232,"15min":1.2509999999999999},"min":{"1min":0.95099999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.482,"5min":1.571,"15min":1.7869999999999999},"last":0.95299999999999996}]},{"osd":2,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.268,"15min":1.228},"min":{"1min":0.80200000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5760000000000001,"5min":1.7450000000000001,"15min":1.847},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.2,"15min":1.2330000000000001},"min":{"1min":0.82899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.589,"5min":1.8600000000000001,"15min":2.0739999999999998},"last":0.92100000000000004}]},{"osd":3,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2769999999999999,"15min":1.2869999999999999},"min":{"1min":1.024,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6679999999999999,"5min":1.71,"15min":1.9770000000000001},"last":1.411},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.256,"15min":1.3049999999999999},"min":{"1min":0.82299999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.524,"5min":1.78,"15min":2.2149999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0880000000000001,"15min":1.145},"min":{"1min":0.77500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5149999999999999,"5min":1.5369999999999999,"15min":2.1099999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.234,"15min":1.246},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.49299999999999999},"max":{"1min":1.649,"5min":1.6990000000000001,"15min":2.073},"last":1.268}]},{"osd":6,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.3200000000000001,"15min":1.3660000000000001},"min":{"1min":0.91900000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6599999999999999,"5min":1.984,"15min":2.4790000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.349,"5min":1.3280000000000001,"15min":1.3540000000000001},"min":{"1min":0.95999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.641,"5min":1.7270000000000001,"15min":2.0030000000000001},"last":1.0940000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3240000000000001,"15min":1.375},"min":{"1min":0.89700000000000002,"5min":0.82399999999999995,"15min":0.58899999999999997},"max":{"1min":1.5980000000000001,"5min":1.764,"15min":2.4470000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.349,"15min":1.429},"min":{"1min":0.89800000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.6080000000000001,"5min":2.0379999999999998,"15min":2.411},"last":1.105}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302184,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069656,"statfs":{"total":96636764160,"available":96327327744,"internally_reserved":0,"allocated":819200,"data_stored":402117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0880000000000001,"15min":1.079},"min":{"1min":0.75700000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.579,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.391},{"interface":"front","average":{"1min":1.129,"5min":1.095,"15min":1.1020000000000001},"min":{"1min":0.67900000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.377,"5min":1.5089999999999999,"15min":1.75},"last":0.90800000000000003}]},{"osd":1,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2350000000000001,"15min":1.2430000000000001},"min":{"1min":0.79800000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.6259999999999999,"5min":1.6819999999999999,"15min":2.1259999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.88200000000000001,"5min":0.71599999999999997,"15min":0.65100000000000002},"max":{"1min":1.55,"5min":1.9790000000000001,"15min":2.2109999999999999},"last":1.47}]},{"osd":2,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.86599999999999999,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.498,"5min":1.6040000000000001,"15min":1.891},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.226,"15min":1.2230000000000001},"min":{"1min":1.0209999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5720000000000001,"5min":1.893,"15min":1.893},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1859999999999999,"15min":1.198},"min":{"1min":0.95599999999999996,"5min":0.85899999999999999,"15min":0.59599999999999997},"max":{"1min":1.399,"5min":1.6379999999999999,"15min":1.8740000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":0.95799999999999996,"5min":0.75600000000000001,"15min":0.60599999999999998},"max":{"1min":1.712,"5min":1.712,"15min":2.0190000000000001},"last":1.226}]},{"osd":5,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.127,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.425,"5min":2.0449999999999999,"15min":2.3500000000000001},"last":1.296},{"interface":"front","average":{"1min":1.099,"5min":1.119,"15min":1.194},"min":{"1min":0.79900000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.581,"5min":1.7210000000000001,"15min":2.161},"last":1.278}]},{"osd":6,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2809999999999999,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.6639999999999999,"5min":1.845,"15min":3.27},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.381,"15min":1.4630000000000001},"min":{"1min":0.94599999999999995,"5min":0.68899999999999995,"15min":0.67600000000000005},"max":{"1min":1.665,"5min":2.0960000000000001,"15min":2.742},"last":1.5580000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3069999999999999,"15min":1.4390000000000001},"min":{"1min":1.0249999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7649999999999999,"5min":1.9079999999999999,"15min":5.2060000000000004},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3999999999999999,"15min":1.518},"min":{"1min":0.97399999999999998,"5min":0.72599999999999998,"15min":0.70599999999999996},"max":{"1min":1.825,"5min":1.825,"15min":3.6739999999999999},"last":1.5900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306280,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065560,"statfs":{"total":96636764160,"available":96323133440,"internally_reserved":0,"allocated":819200,"data_stored":396513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.873,"15min":0.96799999999999997},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.53500000000000003},"max":{"1min":1.3640000000000001,"5min":1.419,"15min":4.1159999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.026,"15min":1.069},"min":{"1min":0.79200000000000004,"5min":0.59099999999999997,"15min":0.50900000000000001},"max":{"1min":1.4059999999999999,"5min":1.7270000000000001,"15min":3.3929999999999998},"last":0.91200000000000003}]},{"osd":1,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.94099999999999995,"15min":1.02},"min":{"1min":0.66700000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.224,"5min":1.4299999999999999,"15min":2.4390000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1080000000000001,"15min":1.1759999999999999},"min":{"1min":0.72999999999999998,"5min":0.66900000000000004,"15min":0.60199999999999998},"max":{"1min":1.532,"5min":1.758,"15min":4.476},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1379999999999999,"15min":1.1890000000000001},"min":{"1min":0.73299999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6879999999999999,"5min":1.76,"15min":2.6989999999999998},"last":1.159},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2170000000000001,"15min":1.2789999999999999},"min":{"1min":0.85799999999999998,"5min":0.68000000000000005,"15min":0.61699999999999999},"max":{"1min":1.647,"5min":1.7190000000000001,"15min":2.5110000000000001},"last":1.0549999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3180000000000001,"15min":1.4079999999999999},"min":{"1min":0.97999999999999998,"5min":0.72999999999999998,"15min":0.69899999999999995},"max":{"1min":1.506,"5min":1.768,"15min":3.46},"last":1.101},{"interface":"front","average":{"1min":1.319,"5min":1.3,"15min":1.3560000000000001},"min":{"1min":0.96099999999999997,"5min":0.85899999999999999,"15min":0.84799999999999998},"max":{"1min":1.6659999999999999,"5min":1.7989999999999999,"15min":2.9409999999999998},"last":1.448}]},{"osd":5,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3080000000000001,"15min":1.3959999999999999},"min":{"1min":1.1499999999999999,"5min":0.85999999999999999,"15min":0.77100000000000002},"max":{"1min":1.5309999999999999,"5min":1.6599999999999999,"15min":3.1019999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3799999999999999,"15min":1.4890000000000001},"min":{"1min":1.1379999999999999,"5min":0.97499999999999998,"15min":0.94499999999999995},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":4.3650000000000002},"last":1.244}]},{"osd":6,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4510000000000001,"15min":1.55},"min":{"1min":1.1850000000000001,"5min":1.1060000000000001,"15min":0.97499999999999998},"max":{"1min":1.883,"5min":1.9219999999999999,"15min":4.5999999999999996},"last":1.653},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.391,"15min":1.524},"min":{"1min":1.147,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.74,"5min":2.0489999999999999,"15min":3.4329999999999998},"last":1.3280000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.498,"15min":1.629},"min":{"1min":1.177,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.796,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4419999999999999,"15min":1.5740000000000001},"min":{"1min":1.224,"5min":0.97599999999999998,"15min":0.89700000000000002},"max":{"1min":1.738,"5min":1.99,"15min":3.7240000000000002},"last":1.546}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306284,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065556,"statfs":{"total":96636764160,"available":96323129344,"internally_reserved":0,"allocated":823296,"data_stored":402088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.89600000000000002,"15min":0.93200000000000005},"min":{"1min":0.629,"5min":0.52800000000000002,"15min":0.36599999999999999},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.946},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.109,"15min":1.091},"min":{"1min":0.92100000000000004,"5min":0.69999999999999996,"15min":0.54100000000000004},"max":{"1min":1.4410000000000001,"5min":1.546,"15min":1.8560000000000001},"last":1.083}]},{"osd":1,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1519999999999999,"15min":1.143},"min":{"1min":0.81399999999999995,"5min":0.61799999999999999,"15min":0.46500000000000002},"max":{"1min":1.554,"5min":1.6479999999999999,"15min":1.8},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.091,"15min":1.111},"min":{"1min":0.72099999999999997,"5min":0.63600000000000001,"15min":0.62},"max":{"1min":1.429,"5min":1.655,"15min":1.716},"last":0.754}]},{"osd":3,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.997,"15min":1.026},"min":{"1min":0.79900000000000004,"5min":0.64200000000000002,"15min":0.55600000000000005},"max":{"1min":1.1639999999999999,"5min":1.623,"15min":1.9370000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.034,"5min":1.097,"15min":1.1160000000000001},"min":{"1min":0.80100000000000005,"5min":0.70999999999999996,"15min":0.55000000000000004},"max":{"1min":1.3120000000000001,"5min":1.518,"15min":1.6930000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.401,"15min":1.425},"min":{"1min":1.1200000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.724,"5min":2.044,"15min":3.4950000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4099999999999999,"15min":1.4159999999999999},"min":{"1min":1.244,"5min":1.018,"15min":0.82799999999999996},"max":{"1min":1.619,"5min":1.9339999999999999,"15min":2.0409999999999999},"last":1.272}]},{"osd":5,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.339,"15min":1.403},"min":{"1min":1.085,"5min":0.89300000000000002,"15min":0.89000000000000001},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":3.8599999999999999},"last":1.484},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.347,"15min":1.3859999999999999},"min":{"1min":0.94399999999999995,"5min":0.93400000000000005,"15min":0.755},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.2070000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.369,"15min":1.399},"min":{"1min":1.079,"5min":0.83299999999999996,"15min":0.80600000000000005},"max":{"1min":1.6739999999999999,"5min":1.9850000000000001,"15min":2.8759999999999999},"last":1.119},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4330000000000001,"15min":1.49},"min":{"1min":1.1799999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.714,"5min":2.137,"15min":3.7309999999999999},"last":1.3240000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4750000000000001,"15min":1.53},"min":{"1min":1.1899999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.903,"5min":2.2080000000000002,"15min":2.5550000000000002},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.464,"15min":1.514},"min":{"1min":1.0740000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.788,"5min":2.181,"15min":2.4049999999999998},"last":1.3799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306272,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065568,"statfs":{"total":96636764160,"available":96323141632,"internally_reserved":0,"allocated":811008,"data_stored":395115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.85499999999999998,"15min":0.88100000000000001},"min":{"1min":0.69499999999999995,"5min":0.51000000000000001,"15min":0.443},"max":{"1min":1.2769999999999999,"5min":1.871,"15min":1.871},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0049999999999999,"15min":1.028},"min":{"1min":0.76200000000000001,"5min":0.53100000000000003,"15min":0.50700000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.468},"last":0.95799999999999996}]},{"osd":2,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0209999999999999,"15min":1.0469999999999999},"min":{"1min":0.82199999999999995,"5min":0.59699999999999998,"15min":0.46500000000000002},"max":{"1min":1.714,"5min":1.714,"15min":1.7829999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.079,"5min":1.131,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68899999999999995,"15min":0.67800000000000005},"max":{"1min":1.429,"5min":1.607,"15min":3.0619999999999998},"last":1.373}]},{"osd":3,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.256,"15min":1.2350000000000001},"min":{"1min":0.94999999999999996,"5min":0.69699999999999995,"15min":0.59199999999999997},"max":{"1min":1.6499999999999999,"5min":1.6919999999999999,"15min":1.9490000000000001},"last":1.538},{"interface":"front","average":{"1min":1.163,"5min":1.0780000000000001,"15min":1.103},"min":{"1min":0.86699999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.26}]},{"osd":4,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3240000000000001,"15min":1.3640000000000001},"min":{"1min":0.997,"5min":0.89200000000000002,"15min":0.72299999999999998},"max":{"1min":1.6439999999999999,"5min":1.8480000000000001,"15min":2.6440000000000001},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.413,"15min":1.4239999999999999},"min":{"1min":1.109,"5min":0.96999999999999997,"15min":0.69499999999999995},"max":{"1min":1.7569999999999999,"5min":1.7749999999999999,"15min":2.7389999999999999},"last":1.302}]},{"osd":5,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.296,"15min":1.329},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.92200000000000004},"max":{"1min":1.5880000000000001,"5min":1.6120000000000001,"15min":2.1320000000000001},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3560000000000001,"15min":1.4259999999999999},"min":{"1min":1.0509999999999999,"5min":1.006,"15min":0.81000000000000005},"max":{"1min":1.6479999999999999,"5min":1.9530000000000001,"15min":2.79},"last":1.4099999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3520000000000001,"15min":1.389},"min":{"1min":1.173,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.629,"5min":1.9179999999999999,"15min":2.4620000000000002},"last":1.581},{"interface":"front","average":{"1min":1.47,"5min":1.4490000000000001,"15min":1.498},"min":{"1min":1.046,"5min":1.02,"15min":0.996},"max":{"1min":1.8129999999999999,"5min":2.0089999999999999,"15min":2.8999999999999999},"last":1.6240000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.498,"15min":1.536},"min":{"1min":1.3140000000000001,"5min":1.0660000000000001,"15min":0.73599999999999999},"max":{"1min":2.0470000000000002,"5min":2.0590000000000002,"15min":3.782},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.4099999999999999,"15min":1.478},"min":{"1min":1.1479999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":2.5390000000000001},"last":1.6870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306860,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064980,"statfs":{"total":96636764160,"available":96322539520,"internally_reserved":0,"allocated":1413120,"data_stored":990976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.65600000000000003,"5min":0.54600000000000004,"15min":0.505},"max":{"1min":1.266,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.96599999999999997,"15min":0.99199999999999999},"min":{"1min":0.67000000000000004,"5min":0.54500000000000004,"15min":0.53600000000000003},"max":{"1min":1.423,"5min":1.5840000000000001,"15min":1.7450000000000001},"last":1.222}]},{"osd":2,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.88,"5min":0.66100000000000003,"15min":0.55200000000000005},"max":{"1min":1.6339999999999999,"5min":1.653,"15min":1.8979999999999999},"last":1.014},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2529999999999999,"15min":1.2370000000000001},"min":{"1min":0.91400000000000003,"5min":0.68799999999999994,"15min":0.54300000000000004},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.97399999999999998}]},{"osd":3,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.054,"15min":1.0349999999999999},"min":{"1min":0.77100000000000002,"5min":0.70199999999999996,"15min":0.67800000000000005},"max":{"1min":1.708,"5min":1.708,"15min":2.0299999999999998},"last":1.085},{"interface":"front","average":{"1min":1.22,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.89400000000000002,"5min":0.81100000000000005,"15min":0.71199999999999997},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.8859999999999999},"last":1.3859999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3140000000000001,"15min":1.3340000000000001},"min":{"1min":1.0309999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":2.298},"last":1.542},{"interface":"front","average":{"1min":1.294,"5min":1.292,"15min":1.3160000000000001},"min":{"1min":1.0369999999999999,"5min":0.89400000000000002,"15min":0.80200000000000005},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.5980000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3720000000000001,"15min":1.3959999999999999},"min":{"1min":1.1950000000000001,"5min":0.86799999999999999,"15min":0.76200000000000001},"max":{"1min":1.7549999999999999,"5min":1.79,"15min":2.4630000000000001},"last":1.802},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.405,"15min":1.4339999999999999},"min":{"1min":1.2030000000000001,"5min":0.74099999999999999,"15min":0.58299999999999996},"max":{"1min":1.7290000000000001,"5min":1.881,"15min":2.6160000000000001},"last":1.851}]},{"osd":6,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.353,"15min":1.3819999999999999},"min":{"1min":1.101,"5min":0.91500000000000004,"15min":0.88300000000000001},"max":{"1min":1.6319999999999999,"5min":1.827,"15min":2.339},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4430000000000001,"15min":1.4870000000000001},"min":{"1min":1.1850000000000001,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.716,"5min":2.1240000000000001,"15min":2.5680000000000001},"last":1.9710000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.502,"15min":1.508},"min":{"1min":1.218,"5min":0.94799999999999995,"15min":0.91500000000000004},"max":{"1min":1.927,"5min":1.9299999999999999,"15min":2.5129999999999999},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.429,"15min":1.4930000000000001},"min":{"1min":1.153,"5min":1.0009999999999999,"15min":0.88200000000000001},"max":{"1min":1.835,"5min":1.907,"15min":2.4500000000000002},"last":1.931}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:52:13.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:52:13.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 Processor -- start 2023-12-19T14:52:13.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 -- start start 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f88fc140ae0 0x7f88fc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc06e150 con 0x7f88fc068630 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc068a70 con 0x7f88fc140710 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.530+0000 7f8900aa5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc06d540 con 0x7f88fc140ae0 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88fa59c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f88fc140710 0x7f88fc0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:55178/0 (socket says 172.21.15.50:55178) 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f88fc140ae0 0x7f88fc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88fa59c700 1 -- 172.21.15.50:0/3171025861 learned_addr learned my addr 172.21.15.50:0/3171025861 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:13.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f9d9b700 1 -- 172.21.15.50:0/3171025861 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661160590 0 0) 0x7f88fc068a70 con 0x7f88fc140710 2023-12-19T14:52:13.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f9d9b700 1 -- 172.21.15.50:0/3171025861 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e8003570 con 0x7f88fc140710 2023-12-19T14:52:13.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f3fff700 1 -- 172.21.15.50:0/3171025861 >> v1:172.21.15.125:6789/0 conn(0x7f88fc140710 legacy=0x7f88fc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f3fff700 1 -- 172.21.15.50:0/3171025861 >> v1:172.21.15.50:6789/0 conn(0x7f88fc068630 legacy=0x7f88fc0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f3fff700 1 -- 172.21.15.50:0/3171025861 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc06d820 con 0x7f88fc140ae0 2023-12-19T14:52:13.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f3fff700 1 --2- 172.21.15.50:0/3171025861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140ae0 0x7f88fc0680f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098e0 tx=0x7f88e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fafd11740e6af7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f9d9b700 1 -- 172.21.15.50:0/3171025861 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400b640 con 0x7f88fc140ae0 2023-12-19T14:52:13.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f9d9b700 1 -- 172.21.15.50:0/3171025861 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e400b7a0 con 0x7f88fc140ae0 2023-12-19T14:52:13.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f88f9d9b700 1 -- 172.21.15.50:0/3171025861 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400bb40 con 0x7f88fc140ae0 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f8900aa5700 1 -- 172.21.15.50:0/3171025861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140ae0 msgr2=0x7f88fc0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.534+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3171025861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140ae0 0x7f88fc0680f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098e0 tx=0x7f88e40091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 -- 172.21.15.50:0/3171025861 shutdown_connections 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3171025861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140ae0 0x7f88fc0680f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 -- 172.21.15.50:0/3171025861 >> 172.21.15.50:0/3171025861 conn(0x7f88fc0c8850 msgr2=0x7f88fc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:13.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 -- 172.21.15.50:0/3171025861 shutdown_connections 2023-12-19T14:52:13.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 -- 172.21.15.50:0/3171025861 wait complete. 2023-12-19T14:52:13.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 Processor -- start 2023-12-19T14:52:13.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.538+0000 7f8900aa5700 1 -- start start 2023-12-19T14:52:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 0x7f88fc1327c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f88fc140ae0 0x7f88fc0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc132d00 con 0x7f88fc140710 2023-12-19T14:52:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc13fd40 con 0x7f88fc068630 2023-12-19T14:52:13.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc159ae0 con 0x7f88fc140ae0 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33646/0 (socket says 172.21.15.50:33646) 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 -- 172.21.15.50:0/3146874048 learned_addr learned my addr 172.21.15.50:0/3146874048 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88fa59c700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 0x7f88fc1327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88fad9d700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f88fc140ae0 0x7f88fc0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 -- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f88fc140ae0 msgr2=0x7f88fc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f88fc140ae0 0x7f88fc0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 -- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 msgr2=0x7f88fc1327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 0x7f88fc1327c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 -- 172.21.15.50:0/3146874048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009420 con 0x7f88fc140710 2023-12-19T14:52:13.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88fa59c700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 0x7f88fc1327c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f3fff700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009f80 tx=0x7f88e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400bb20 con 0x7f88fc140710 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4011420 con 0x7f88fc140710 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc159dc0 con 0x7f88fc140710 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40117a0 con 0x7f88fc140710 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.542+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc15a380 con 0x7f88fc140710 2023-12-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.546+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc071120 con 0x7f88fc140710 2023-12-19T14:52:13.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.546+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f88e4011b10 con 0x7f88fc140710 2023-12-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.550+0000 7f88f27fc700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 0x7f88dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.550+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f88e407dfc0 con 0x7f88fc140710 2023-12-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.550+0000 7f88fa59c700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 0x7f88dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:13.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.550+0000 7f88fa59c700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 0x7f88dc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0058d0 tx=0x7f88ec009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:13.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.554+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f88e404e360 con 0x7f88fc140710 2023-12-19T14:52:13.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.722+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88fc0642a0 con 0x7f88dc060670 2023-12-19T14:52:13.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.742+0000 7f88f27fc700 1 -- 172.21.15.50:0/3146874048 <== mgr.5626 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286473 (secure 0 0 0) 0x7f88fc0642a0 con 0x7f88dc060670 2023-12-19T14:52:13.750 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 msgr2=0x7f88dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 0x7f88dc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0058d0 tx=0x7f88ec009040 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 msgr2=0x7f88fc0b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009f80 tx=0x7f88e4017000 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 shutdown_connections 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88dc060670 0x7f88dc062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f88fc140ae0 0x7f88fc0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88fc140710 0x7f88fc0b1120 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 --2- 172.21.15.50:0/3146874048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88fc068630 0x7f88fc1327c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:13.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 >> 172.21.15.50:0/3146874048 conn(0x7f88fc0c8850 msgr2=0x7f88fc0c2140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:13.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 shutdown_connections 2023-12-19T14:52:13.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.750+0000 7f8900aa5700 1 -- 172.21.15.50:0/3146874048 wait complete. 2023-12-19T14:52:13.757 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:52:13.777 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:52:12.399482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2439088,"kb_used_data":7536,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752535632,"statfs":{"total":773094113280,"available":770596487168,"internally_reserved":0,"allocated":7716864,"data_stored":4369398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413102"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040626+0000","last_change":"2023-12-19T14:52:09.036116+0000","last_active":"2023-12-19T14:52:09.040626+0000","last_peered":"2023-12-19T14:52:09.040626+0000","last_clean":"2023-12-19T14:52:09.040626+0000","last_became_active":"2023-12-19T14:52:09.035803+0000","last_became_peered":"2023-12-19T14:52:09.035803+0000","last_unstale":"2023-12-19T14:52:09.040626+0000","last_undegraded":"2023-12-19T14:52:09.040626+0000","last_fullsized":"2023-12-19T14:52:09.040626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:48.180290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043337+0000","last_change":"2023-12-19T14:52:09.037446+0000","last_active":"2023-12-19T14:52:09.043337+0000","last_peered":"2023-12-19T14:52:09.043337+0000","last_clean":"2023-12-19T14:52:09.043337+0000","last_became_active":"2023-12-19T14:52:09.036847+0000","last_became_peered":"2023-12-19T14:52:09.036847+0000","last_unstale":"2023-12-19T14:52:09.043337+0000","last_undegraded":"2023-12-19T14:52:09.043337+0000","last_fullsized":"2023-12-19T14:52:09.043337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:48.180290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052057+0000","last_change":"2023-12-19T14:52:09.043187+0000","last_active":"2023-12-19T14:52:09.052057+0000","last_peered":"2023-12-19T14:52:09.052057+0000","last_clean":"2023-12-19T14:52:09.052057+0000","last_became_active":"2023-12-19T14:52:09.042783+0000","last_became_peered":"2023-12-19T14:52:09.042783+0000","last_unstale":"2023-12-19T14:52:09.052057+0000","last_undegraded":"2023-12-19T14:52:09.052057+0000","last_fullsized":"2023-12-19T14:52:09.052057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:17.623185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043398+0000","last_change":"2023-12-19T14:52:09.037992+0000","last_active":"2023-12-19T14:52:09.043398+0000","last_peered":"2023-12-19T14:52:09.043398+0000","last_clean":"2023-12-19T14:52:09.043398+0000","last_became_active":"2023-12-19T14:52:09.037716+0000","last_became_peered":"2023-12-19T14:52:09.037716+0000","last_unstale":"2023-12-19T14:52:09.043398+0000","last_undegraded":"2023-12-19T14:52:09.043398+0000","last_fullsized":"2023-12-19T14:52:09.043398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:06.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045993+0000","last_change":"2023-12-19T14:52:09.040387+0000","last_active":"2023-12-19T14:52:09.045993+0000","last_peered":"2023-12-19T14:52:09.045993+0000","last_clean":"2023-12-19T14:52:09.045993+0000","last_became_active":"2023-12-19T14:52:09.040083+0000","last_became_peered":"2023-12-19T14:52:09.040083+0000","last_unstale":"2023-12-19T14:52:09.045993+0000","last_undegraded":"2023-12-19T14:52:09.045993+0000","last_fullsized":"2023-12-19T14:52:09.045993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:56.125818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042231+0000","last_change":"2023-12-19T14:52:09.038334+0000","last_active":"2023-12-19T14:52:09.042231+0000","last_peered":"2023-12-19T14:52:09.042231+0000","last_clean":"2023-12-19T14:52:09.042231+0000","last_became_active":"2023-12-19T14:52:09.038070+0000","last_became_peered":"2023-12-19T14:52:09.038070+0000","last_unstale":"2023-12-19T14:52:09.042231+0000","last_undegraded":"2023-12-19T14:52:09.042231+0000","last_fullsized":"2023-12-19T14:52:09.042231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:06.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052109+0000","last_change":"2023-12-19T14:52:09.044230+0000","last_active":"2023-12-19T14:52:09.052109+0000","last_peered":"2023-12-19T14:52:09.052109+0000","last_clean":"2023-12-19T14:52:09.052109+0000","last_became_active":"2023-12-19T14:52:09.039210+0000","last_became_peered":"2023-12-19T14:52:09.039210+0000","last_unstale":"2023-12-19T14:52:09.052109+0000","last_undegraded":"2023-12-19T14:52:09.052109+0000","last_fullsized":"2023-12-19T14:52:09.052109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054605+0000","last_change":"2023-12-19T14:52:09.046828+0000","last_active":"2023-12-19T14:52:09.054605+0000","last_peered":"2023-12-19T14:52:09.054605+0000","last_clean":"2023-12-19T14:52:09.054605+0000","last_became_active":"2023-12-19T14:52:09.046500+0000","last_became_peered":"2023-12-19T14:52:09.046500+0000","last_unstale":"2023-12-19T14:52:09.054605+0000","last_undegraded":"2023-12-19T14:52:09.054605+0000","last_fullsized":"2023-12-19T14:52:09.054605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:03:00.136172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.048595+0000","last_change":"2023-12-19T14:52:09.045957+0000","last_active":"2023-12-19T14:52:09.048595+0000","last_peered":"2023-12-19T14:52:09.048595+0000","last_clean":"2023-12-19T14:52:09.048595+0000","last_became_active":"2023-12-19T14:52:09.045476+0000","last_became_peered":"2023-12-19T14:52:09.045476+0000","last_unstale":"2023-12-19T14:52:09.048595+0000","last_undegraded":"2023-12-19T14:52:09.048595+0000","last_fullsized":"2023-12-19T14:52:09.048595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:02:51.357979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036047+0000","last_change":"2023-12-19T14:52:09.029448+0000","last_active":"2023-12-19T14:52:09.036047+0000","last_peered":"2023-12-19T14:52:09.036047+0000","last_clean":"2023-12-19T14:52:09.036047+0000","last_became_active":"2023-12-19T14:52:09.029155+0000","last_became_peered":"2023-12-19T14:52:09.029155+0000","last_unstale":"2023-12-19T14:52:09.036047+0000","last_undegraded":"2023-12-19T14:52:09.036047+0000","last_fullsized":"2023-12-19T14:52:09.036047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:42.953083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054772+0000","last_change":"2023-12-19T14:52:09.051508+0000","last_active":"2023-12-19T14:52:09.054772+0000","last_peered":"2023-12-19T14:52:09.054772+0000","last_clean":"2023-12-19T14:52:09.054772+0000","last_became_active":"2023-12-19T14:52:09.051217+0000","last_became_peered":"2023-12-19T14:52:09.051217+0000","last_unstale":"2023-12-19T14:52:09.054772+0000","last_undegraded":"2023-12-19T14:52:09.054772+0000","last_fullsized":"2023-12-19T14:52:09.054772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:12.498326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039891+0000","last_change":"2023-12-19T14:52:09.033517+0000","last_active":"2023-12-19T14:52:09.039891+0000","last_peered":"2023-12-19T14:52:09.039891+0000","last_clean":"2023-12-19T14:52:09.039891+0000","last_became_active":"2023-12-19T14:52:09.032500+0000","last_became_peered":"2023-12-19T14:52:09.032500+0000","last_unstale":"2023-12-19T14:52:09.039891+0000","last_undegraded":"2023-12-19T14:52:09.039891+0000","last_fullsized":"2023-12-19T14:52:09.039891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:17.623185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036052+0000","last_change":"2023-12-19T14:52:09.028928+0000","last_active":"2023-12-19T14:52:09.036052+0000","last_peered":"2023-12-19T14:52:09.036052+0000","last_clean":"2023-12-19T14:52:09.036052+0000","last_became_active":"2023-12-19T14:52:09.028616+0000","last_became_peered":"2023-12-19T14:52:09.028616+0000","last_unstale":"2023-12-19T14:52:09.036052+0000","last_undegraded":"2023-12-19T14:52:09.036052+0000","last_fullsized":"2023-12-19T14:52:09.036052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:30.584084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054504+0000","last_change":"2023-12-19T14:52:09.044159+0000","last_active":"2023-12-19T14:52:09.054504+0000","last_peered":"2023-12-19T14:52:09.054504+0000","last_clean":"2023-12-19T14:52:09.054504+0000","last_became_active":"2023-12-19T14:52:09.043861+0000","last_became_peered":"2023-12-19T14:52:09.043861+0000","last_unstale":"2023-12-19T14:52:09.054504+0000","last_undegraded":"2023-12-19T14:52:09.054504+0000","last_fullsized":"2023-12-19T14:52:09.054504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043300+0000","last_change":"2023-12-19T14:52:09.035737+0000","last_active":"2023-12-19T14:52:09.043300+0000","last_peered":"2023-12-19T14:52:09.043300+0000","last_clean":"2023-12-19T14:52:09.043300+0000","last_became_active":"2023-12-19T14:52:09.035448+0000","last_became_peered":"2023-12-19T14:52:09.035448+0000","last_unstale":"2023-12-19T14:52:09.043300+0000","last_undegraded":"2023-12-19T14:52:09.043300+0000","last_fullsized":"2023-12-19T14:52:09.043300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:05.290781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036306+0000","last_change":"2023-12-19T14:52:09.027883+0000","last_active":"2023-12-19T14:52:09.036306+0000","last_peered":"2023-12-19T14:52:09.036306+0000","last_clean":"2023-12-19T14:52:09.036306+0000","last_became_active":"2023-12-19T14:52:09.027588+0000","last_became_peered":"2023-12-19T14:52:09.027588+0000","last_unstale":"2023-12-19T14:52:09.036306+0000","last_undegraded":"2023-12-19T14:52:09.036306+0000","last_fullsized":"2023-12-19T14:52:09.036306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:00.157458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052014+0000","last_change":"2023-12-19T14:52:09.037729+0000","last_active":"2023-12-19T14:52:09.052014+0000","last_peered":"2023-12-19T14:52:09.052014+0000","last_clean":"2023-12-19T14:52:09.052014+0000","last_became_active":"2023-12-19T14:52:09.037403+0000","last_became_peered":"2023-12-19T14:52:09.037403+0000","last_unstale":"2023-12-19T14:52:09.052014+0000","last_undegraded":"2023-12-19T14:52:09.052014+0000","last_fullsized":"2023-12-19T14:52:09.052014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:17:25.394194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043354+0000","last_change":"2023-12-19T14:52:09.037504+0000","last_active":"2023-12-19T14:52:09.043354+0000","last_peered":"2023-12-19T14:52:09.043354+0000","last_clean":"2023-12-19T14:52:09.043354+0000","last_became_active":"2023-12-19T14:52:09.037207+0000","last_became_peered":"2023-12-19T14:52:09.037207+0000","last_unstale":"2023-12-19T14:52:09.043354+0000","last_undegraded":"2023-12-19T14:52:09.043354+0000","last_fullsized":"2023-12-19T14:52:09.043354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051885+0000","last_change":"2023-12-19T14:52:09.035999+0000","last_active":"2023-12-19T14:52:09.051885+0000","last_peered":"2023-12-19T14:52:09.051885+0000","last_clean":"2023-12-19T14:52:09.051885+0000","last_became_active":"2023-12-19T14:52:09.035702+0000","last_became_peered":"2023-12-19T14:52:09.035702+0000","last_unstale":"2023-12-19T14:52:09.051885+0000","last_undegraded":"2023-12-19T14:52:09.051885+0000","last_fullsized":"2023-12-19T14:52:09.051885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:06.248039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.035992+0000","last_change":"2023-12-19T14:52:09.030172+0000","last_active":"2023-12-19T14:52:09.035992+0000","last_peered":"2023-12-19T14:52:09.035992+0000","last_clean":"2023-12-19T14:52:09.035992+0000","last_became_active":"2023-12-19T14:52:09.029673+0000","last_became_peered":"2023-12-19T14:52:09.029673+0000","last_unstale":"2023-12-19T14:52:09.035992+0000","last_undegraded":"2023-12-19T14:52:09.035992+0000","last_fullsized":"2023-12-19T14:52:09.035992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:06.973983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.036221+0000","last_change":"2023-12-19T14:52:09.032774+0000","last_active":"2023-12-19T14:52:09.036221+0000","last_peered":"2023-12-19T14:52:09.036221+0000","last_clean":"2023-12-19T14:52:09.036221+0000","last_became_active":"2023-12-19T14:52:09.032314+0000","last_became_peered":"2023-12-19T14:52:09.032314+0000","last_unstale":"2023-12-19T14:52:09.036221+0000","last_undegraded":"2023-12-19T14:52:09.036221+0000","last_fullsized":"2023-12-19T14:52:09.036221+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:32.311495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.032674+0000","last_change":"2023-12-19T14:52:09.029439+0000","last_active":"2023-12-19T14:52:09.032674+0000","last_peered":"2023-12-19T14:52:09.032674+0000","last_clean":"2023-12-19T14:52:09.032674+0000","last_became_active":"2023-12-19T14:52:09.027954+0000","last_became_peered":"2023-12-19T14:52:09.027954+0000","last_unstale":"2023-12-19T14:52:09.032674+0000","last_undegraded":"2023-12-19T14:52:09.032674+0000","last_fullsized":"2023-12-19T14:52:09.032674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:02:06.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.055647+0000","last_change":"2023-12-19T14:52:09.054039+0000","last_active":"2023-12-19T14:52:09.055647+0000","last_peered":"2023-12-19T14:52:09.055647+0000","last_clean":"2023-12-19T14:52:09.055647+0000","last_became_active":"2023-12-19T14:52:09.053518+0000","last_became_peered":"2023-12-19T14:52:09.053518+0000","last_unstale":"2023-12-19T14:52:09.055647+0000","last_undegraded":"2023-12-19T14:52:09.055647+0000","last_fullsized":"2023-12-19T14:52:09.055647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:06.973983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047840+0000","last_change":"2023-12-19T14:52:09.034409+0000","last_active":"2023-12-19T14:52:09.047840+0000","last_peered":"2023-12-19T14:52:09.047840+0000","last_clean":"2023-12-19T14:52:09.047840+0000","last_became_active":"2023-12-19T14:52:09.033991+0000","last_became_peered":"2023-12-19T14:52:09.033991+0000","last_unstale":"2023-12-19T14:52:09.047840+0000","last_undegraded":"2023-12-19T14:52:09.047840+0000","last_fullsized":"2023-12-19T14:52:09.047840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:05.290781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052064+0000","last_change":"2023-12-19T14:52:09.038449+0000","last_active":"2023-12-19T14:52:09.052064+0000","last_peered":"2023-12-19T14:52:09.052064+0000","last_clean":"2023-12-19T14:52:09.052064+0000","last_became_active":"2023-12-19T14:52:09.038091+0000","last_became_peered":"2023-12-19T14:52:09.038091+0000","last_unstale":"2023-12-19T14:52:09.052064+0000","last_undegraded":"2023-12-19T14:52:09.052064+0000","last_fullsized":"2023-12-19T14:52:09.052064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:19.264675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046383+0000","last_change":"2023-12-19T14:52:09.039434+0000","last_active":"2023-12-19T14:52:09.046383+0000","last_peered":"2023-12-19T14:52:09.046383+0000","last_clean":"2023-12-19T14:52:09.046383+0000","last_became_active":"2023-12-19T14:52:09.039107+0000","last_became_peered":"2023-12-19T14:52:09.039107+0000","last_unstale":"2023-12-19T14:52:09.046383+0000","last_undegraded":"2023-12-19T14:52:09.046383+0000","last_fullsized":"2023-12-19T14:52:09.046383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:06.096141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052821+0000","last_change":"2023-12-19T14:52:09.046591+0000","last_active":"2023-12-19T14:52:09.052821+0000","last_peered":"2023-12-19T14:52:09.052821+0000","last_clean":"2023-12-19T14:52:09.052821+0000","last_became_active":"2023-12-19T14:52:09.044905+0000","last_became_peered":"2023-12-19T14:52:09.044905+0000","last_unstale":"2023-12-19T14:52:09.052821+0000","last_undegraded":"2023-12-19T14:52:09.052821+0000","last_fullsized":"2023-12-19T14:52:09.052821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:56.125818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.041930+0000","last_change":"2023-12-19T14:52:09.036092+0000","last_active":"2023-12-19T14:52:09.041930+0000","last_peered":"2023-12-19T14:52:09.041930+0000","last_clean":"2023-12-19T14:52:09.041930+0000","last_became_active":"2023-12-19T14:52:09.035820+0000","last_became_peered":"2023-12-19T14:52:09.035820+0000","last_unstale":"2023-12-19T14:52:09.041930+0000","last_undegraded":"2023-12-19T14:52:09.041930+0000","last_fullsized":"2023-12-19T14:52:09.041930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:12.498326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.053264+0000","last_change":"2023-12-19T14:52:09.039107+0000","last_active":"2023-12-19T14:52:09.053264+0000","last_peered":"2023-12-19T14:52:09.053264+0000","last_clean":"2023-12-19T14:52:09.053264+0000","last_became_active":"2023-12-19T14:52:09.038746+0000","last_became_peered":"2023-12-19T14:52:09.038746+0000","last_unstale":"2023-12-19T14:52:09.053264+0000","last_undegraded":"2023-12-19T14:52:09.053264+0000","last_fullsized":"2023-12-19T14:52:09.053264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:09:32.544274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042682+0000","last_change":"2023-12-19T14:52:09.035192+0000","last_active":"2023-12-19T14:52:09.042682+0000","last_peered":"2023-12-19T14:52:09.042682+0000","last_clean":"2023-12-19T14:52:09.042682+0000","last_became_active":"2023-12-19T14:52:09.034892+0000","last_became_peered":"2023-12-19T14:52:09.034892+0000","last_unstale":"2023-12-19T14:52:09.042682+0000","last_undegraded":"2023-12-19T14:52:09.042682+0000","last_fullsized":"2023-12-19T14:52:09.042682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:53.099678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051944+0000","last_change":"2023-12-19T14:52:09.042731+0000","last_active":"2023-12-19T14:52:09.051944+0000","last_peered":"2023-12-19T14:52:09.051944+0000","last_clean":"2023-12-19T14:52:09.051944+0000","last_became_active":"2023-12-19T14:52:09.038819+0000","last_became_peered":"2023-12-19T14:52:09.038819+0000","last_unstale":"2023-12-19T14:52:09.051944+0000","last_undegraded":"2023-12-19T14:52:09.051944+0000","last_fullsized":"2023-12-19T14:52:09.051944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:44.147509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.031110+0000","last_change":"2023-12-19T14:52:09.025661+0000","last_active":"2023-12-19T14:52:09.031110+0000","last_peered":"2023-12-19T14:52:09.031110+0000","last_clean":"2023-12-19T14:52:09.031110+0000","last_became_active":"2023-12-19T14:52:09.025379+0000","last_became_peered":"2023-12-19T14:52:09.025379+0000","last_unstale":"2023-12-19T14:52:09.031110+0000","last_undegraded":"2023-12-19T14:52:09.031110+0000","last_fullsized":"2023-12-19T14:52:09.031110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:59.545272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.043194+0000","last_change":"2023-12-19T14:52:09.035334+0000","last_active":"2023-12-19T14:52:09.043194+0000","last_peered":"2023-12-19T14:52:09.043194+0000","last_clean":"2023-12-19T14:52:09.043194+0000","last_became_active":"2023-12-19T14:52:09.034998+0000","last_became_peered":"2023-12-19T14:52:09.034998+0000","last_unstale":"2023-12-19T14:52:09.043194+0000","last_undegraded":"2023-12-19T14:52:09.043194+0000","last_fullsized":"2023-12-19T14:52:09.043194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:44.147509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.061765+0000","last_change":"2023-12-19T14:52:08.022078+0000","last_active":"2023-12-19T14:52:09.061765+0000","last_peered":"2023-12-19T14:52:09.061765+0000","last_clean":"2023-12-19T14:52:09.061765+0000","last_became_active":"2023-12-19T14:52:08.020328+0000","last_became_peered":"2023-12-19T14:52:08.020328+0000","last_unstale":"2023-12-19T14:52:09.061765+0000","last_undegraded":"2023-12-19T14:52:09.061765+0000","last_fullsized":"2023-12-19T14:52:09.061765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:27.268706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045077+0000","last_change":"2023-12-19T14:52:09.040850+0000","last_active":"2023-12-19T14:52:09.045077+0000","last_peered":"2023-12-19T14:52:09.045077+0000","last_clean":"2023-12-19T14:52:09.045077+0000","last_became_active":"2023-12-19T14:52:09.040540+0000","last_became_peered":"2023-12-19T14:52:09.040540+0000","last_unstale":"2023-12-19T14:52:09.045077+0000","last_undegraded":"2023-12-19T14:52:09.045077+0000","last_fullsized":"2023-12-19T14:52:09.045077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:43:01.592858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068289+0000","last_change":"2023-12-19T14:52:08.040833+0000","last_active":"2023-12-19T14:52:09.068289+0000","last_peered":"2023-12-19T14:52:09.068289+0000","last_clean":"2023-12-19T14:52:09.068289+0000","last_became_active":"2023-12-19T14:52:08.040279+0000","last_became_peered":"2023-12-19T14:52:08.040279+0000","last_unstale":"2023-12-19T14:52:09.068289+0000","last_undegraded":"2023-12-19T14:52:09.068289+0000","last_fullsized":"2023-12-19T14:52:09.068289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:14.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045567+0000","last_change":"2023-12-19T14:52:09.043710+0000","last_active":"2023-12-19T14:52:09.045567+0000","last_peered":"2023-12-19T14:52:09.045567+0000","last_clean":"2023-12-19T14:52:09.045567+0000","last_became_active":"2023-12-19T14:52:09.043236+0000","last_became_peered":"2023-12-19T14:52:09.043236+0000","last_unstale":"2023-12-19T14:52:09.045567+0000","last_undegraded":"2023-12-19T14:52:09.045567+0000","last_fullsized":"2023-12-19T14:52:09.045567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:05.260747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.071154+0000","last_change":"2023-12-19T14:52:08.027276+0000","last_active":"2023-12-19T14:52:09.071154+0000","last_peered":"2023-12-19T14:52:09.071154+0000","last_clean":"2023-12-19T14:52:09.071154+0000","last_became_active":"2023-12-19T14:52:08.026691+0000","last_became_peered":"2023-12-19T14:52:08.026691+0000","last_unstale":"2023-12-19T14:52:09.071154+0000","last_undegraded":"2023-12-19T14:52:09.071154+0000","last_fullsized":"2023-12-19T14:52:09.071154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:03:42.400730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047313+0000","last_change":"2023-12-19T14:52:09.041963+0000","last_active":"2023-12-19T14:52:09.047313+0000","last_peered":"2023-12-19T14:52:09.047313+0000","last_clean":"2023-12-19T14:52:09.047313+0000","last_became_active":"2023-12-19T14:52:09.041308+0000","last_became_peered":"2023-12-19T14:52:09.041308+0000","last_unstale":"2023-12-19T14:52:09.047313+0000","last_undegraded":"2023-12-19T14:52:09.047313+0000","last_fullsized":"2023-12-19T14:52:09.047313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:57.525750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.056734+0000","last_change":"2023-12-19T14:52:08.034807+0000","last_active":"2023-12-19T14:52:09.056734+0000","last_peered":"2023-12-19T14:52:09.056734+0000","last_clean":"2023-12-19T14:52:09.056734+0000","last_became_active":"2023-12-19T14:52:08.034424+0000","last_became_peered":"2023-12-19T14:52:08.034424+0000","last_unstale":"2023-12-19T14:52:09.056734+0000","last_undegraded":"2023-12-19T14:52:09.056734+0000","last_fullsized":"2023-12-19T14:52:09.056734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:56.512280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046003+0000","last_change":"2023-12-19T14:52:09.044433+0000","last_active":"2023-12-19T14:52:09.046003+0000","last_peered":"2023-12-19T14:52:09.046003+0000","last_clean":"2023-12-19T14:52:09.046003+0000","last_became_active":"2023-12-19T14:52:09.044083+0000","last_became_peered":"2023-12-19T14:52:09.044083+0000","last_unstale":"2023-12-19T14:52:09.046003+0000","last_undegraded":"2023-12-19T14:52:09.046003+0000","last_fullsized":"2023-12-19T14:52:09.046003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:56.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.066062+0000","last_change":"2023-12-19T14:52:08.034042+0000","last_active":"2023-12-19T14:52:09.066062+0000","last_peered":"2023-12-19T14:52:09.066062+0000","last_clean":"2023-12-19T14:52:09.066062+0000","last_became_active":"2023-12-19T14:52:08.033238+0000","last_became_peered":"2023-12-19T14:52:08.033238+0000","last_unstale":"2023-12-19T14:52:09.066062+0000","last_undegraded":"2023-12-19T14:52:09.066062+0000","last_fullsized":"2023-12-19T14:52:09.066062+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:04.365847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045297+0000","last_change":"2023-12-19T14:52:09.040510+0000","last_active":"2023-12-19T14:52:09.045297+0000","last_peered":"2023-12-19T14:52:09.045297+0000","last_clean":"2023-12-19T14:52:09.045297+0000","last_became_active":"2023-12-19T14:52:09.039004+0000","last_became_peered":"2023-12-19T14:52:09.039004+0000","last_unstale":"2023-12-19T14:52:09.045297+0000","last_undegraded":"2023-12-19T14:52:09.045297+0000","last_fullsized":"2023-12-19T14:52:09.045297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:13.392136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.062095+0000","last_change":"2023-12-19T14:52:08.020047+0000","last_active":"2023-12-19T14:52:09.062095+0000","last_peered":"2023-12-19T14:52:09.062095+0000","last_clean":"2023-12-19T14:52:09.062095+0000","last_became_active":"2023-12-19T14:52:08.019117+0000","last_became_peered":"2023-12-19T14:52:08.019117+0000","last_unstale":"2023-12-19T14:52:09.062095+0000","last_undegraded":"2023-12-19T14:52:09.062095+0000","last_fullsized":"2023-12-19T14:52:09.062095+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:12.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046363+0000","last_change":"2023-12-19T14:52:09.039377+0000","last_active":"2023-12-19T14:52:09.046363+0000","last_peered":"2023-12-19T14:52:09.046363+0000","last_clean":"2023-12-19T14:52:09.046363+0000","last_became_active":"2023-12-19T14:52:09.039011+0000","last_became_peered":"2023-12-19T14:52:09.039011+0000","last_unstale":"2023-12-19T14:52:09.046363+0000","last_undegraded":"2023-12-19T14:52:09.046363+0000","last_fullsized":"2023-12-19T14:52:09.046363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:01:38.929717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016383+0000","last_change":"2023-12-19T14:52:08.019134+0000","last_active":"2023-12-19T14:52:09.016383+0000","last_peered":"2023-12-19T14:52:09.016383+0000","last_clean":"2023-12-19T14:52:09.016383+0000","last_became_active":"2023-12-19T14:52:08.018648+0000","last_became_peered":"2023-12-19T14:52:08.018648+0000","last_unstale":"2023-12-19T14:52:09.016383+0000","last_undegraded":"2023-12-19T14:52:09.016383+0000","last_fullsized":"2023-12-19T14:52:09.016383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:56.166925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046428+0000","last_change":"2023-12-19T14:52:09.044533+0000","last_active":"2023-12-19T14:52:09.046428+0000","last_peered":"2023-12-19T14:52:09.046428+0000","last_clean":"2023-12-19T14:52:09.046428+0000","last_became_active":"2023-12-19T14:52:09.044221+0000","last_became_peered":"2023-12-19T14:52:09.044221+0000","last_unstale":"2023-12-19T14:52:09.046428+0000","last_undegraded":"2023-12-19T14:52:09.046428+0000","last_fullsized":"2023-12-19T14:52:09.046428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:42.953083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022399+0000","last_change":"2023-12-19T14:52:08.025323+0000","last_active":"2023-12-19T14:52:09.022399+0000","last_peered":"2023-12-19T14:52:09.022399+0000","last_clean":"2023-12-19T14:52:09.022399+0000","last_became_active":"2023-12-19T14:52:08.024950+0000","last_became_peered":"2023-12-19T14:52:08.024950+0000","last_unstale":"2023-12-19T14:52:09.022399+0000","last_undegraded":"2023-12-19T14:52:09.022399+0000","last_fullsized":"2023-12-19T14:52:09.022399+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:33:11.975978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.035663+0000","last_change":"2023-12-19T14:52:09.032340+0000","last_active":"2023-12-19T14:52:09.035663+0000","last_peered":"2023-12-19T14:52:09.035663+0000","last_clean":"2023-12-19T14:52:09.035663+0000","last_became_active":"2023-12-19T14:52:09.029110+0000","last_became_peered":"2023-12-19T14:52:09.029110+0000","last_unstale":"2023-12-19T14:52:09.035663+0000","last_undegraded":"2023-12-19T14:52:09.035663+0000","last_fullsized":"2023-12-19T14:52:09.035663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:27.440702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022098+0000","last_change":"2023-12-19T14:52:08.024083+0000","last_active":"2023-12-19T14:52:09.022098+0000","last_peered":"2023-12-19T14:52:09.022098+0000","last_clean":"2023-12-19T14:52:09.022098+0000","last_became_active":"2023-12-19T14:52:08.023712+0000","last_became_peered":"2023-12-19T14:52:08.023712+0000","last_unstale":"2023-12-19T14:52:09.022098+0000","last_undegraded":"2023-12-19T14:52:09.022098+0000","last_fullsized":"2023-12-19T14:52:09.022098+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:04.247276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042011+0000","last_change":"2023-12-19T14:52:09.036542+0000","last_active":"2023-12-19T14:52:09.042011+0000","last_peered":"2023-12-19T14:52:09.042011+0000","last_clean":"2023-12-19T14:52:09.042011+0000","last_became_active":"2023-12-19T14:52:09.035966+0000","last_became_peered":"2023-12-19T14:52:09.035966+0000","last_unstale":"2023-12-19T14:52:09.042011+0000","last_undegraded":"2023-12-19T14:52:09.042011+0000","last_fullsized":"2023-12-19T14:52:09.042011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:36.950907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019733+0000","last_change":"2023-12-19T14:52:08.036732+0000","last_active":"2023-12-19T14:52:09.019733+0000","last_peered":"2023-12-19T14:52:09.019733+0000","last_clean":"2023-12-19T14:52:09.019733+0000","last_became_active":"2023-12-19T14:52:08.036331+0000","last_became_peered":"2023-12-19T14:52:08.036331+0000","last_unstale":"2023-12-19T14:52:09.019733+0000","last_undegraded":"2023-12-19T14:52:09.019733+0000","last_fullsized":"2023-12-19T14:52:09.019733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.134038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.054485+0000","last_change":"2023-12-19T14:52:09.035652+0000","last_active":"2023-12-19T14:52:09.054485+0000","last_peered":"2023-12-19T14:52:09.054485+0000","last_clean":"2023-12-19T14:52:09.054485+0000","last_became_active":"2023-12-19T14:52:09.035321+0000","last_became_peered":"2023-12-19T14:52:09.035321+0000","last_unstale":"2023-12-19T14:52:09.054485+0000","last_undegraded":"2023-12-19T14:52:09.054485+0000","last_fullsized":"2023-12-19T14:52:09.054485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:02:06.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.063930+0000","last_change":"2023-12-19T14:52:08.032194+0000","last_active":"2023-12-19T14:52:09.063930+0000","last_peered":"2023-12-19T14:52:09.063930+0000","last_clean":"2023-12-19T14:52:09.063930+0000","last_became_active":"2023-12-19T14:52:08.031812+0000","last_became_peered":"2023-12-19T14:52:08.031812+0000","last_unstale":"2023-12-19T14:52:09.063930+0000","last_undegraded":"2023-12-19T14:52:09.063930+0000","last_fullsized":"2023-12-19T14:52:09.063930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:03:42.400730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045396+0000","last_change":"2023-12-19T14:52:09.038267+0000","last_active":"2023-12-19T14:52:09.045396+0000","last_peered":"2023-12-19T14:52:09.045396+0000","last_clean":"2023-12-19T14:52:09.045396+0000","last_became_active":"2023-12-19T14:52:09.037992+0000","last_became_peered":"2023-12-19T14:52:09.037992+0000","last_unstale":"2023-12-19T14:52:09.045396+0000","last_undegraded":"2023-12-19T14:52:09.045396+0000","last_fullsized":"2023-12-19T14:52:09.045396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:00.157458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068188+0000","last_change":"2023-12-19T14:52:08.040447+0000","last_active":"2023-12-19T14:52:09.068188+0000","last_peered":"2023-12-19T14:52:09.068188+0000","last_clean":"2023-12-19T14:52:09.068188+0000","last_became_active":"2023-12-19T14:52:08.036832+0000","last_became_peered":"2023-12-19T14:52:08.036832+0000","last_unstale":"2023-12-19T14:52:09.068188+0000","last_undegraded":"2023-12-19T14:52:09.068188+0000","last_fullsized":"2023-12-19T14:52:09.068188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:33:05.712073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045038+0000","last_change":"2023-12-19T14:52:09.040709+0000","last_active":"2023-12-19T14:52:09.045038+0000","last_peered":"2023-12-19T14:52:09.045038+0000","last_clean":"2023-12-19T14:52:09.045038+0000","last_became_active":"2023-12-19T14:52:09.040041+0000","last_became_peered":"2023-12-19T14:52:09.040041+0000","last_unstale":"2023-12-19T14:52:09.045038+0000","last_undegraded":"2023-12-19T14:52:09.045038+0000","last_fullsized":"2023-12-19T14:52:09.045038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:16:56.760376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068331+0000","last_change":"2023-12-19T14:52:08.037963+0000","last_active":"2023-12-19T14:52:09.068331+0000","last_peered":"2023-12-19T14:52:09.068331+0000","last_clean":"2023-12-19T14:52:09.068331+0000","last_became_active":"2023-12-19T14:52:08.037688+0000","last_became_peered":"2023-12-19T14:52:08.037688+0000","last_unstale":"2023-12-19T14:52:09.068331+0000","last_undegraded":"2023-12-19T14:52:09.068331+0000","last_fullsized":"2023-12-19T14:52:09.068331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:14.126369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051932+0000","last_change":"2023-12-19T14:52:09.038137+0000","last_active":"2023-12-19T14:52:09.051932+0000","last_peered":"2023-12-19T14:52:09.051932+0000","last_clean":"2023-12-19T14:52:09.051932+0000","last_became_active":"2023-12-19T14:52:09.037768+0000","last_became_peered":"2023-12-19T14:52:09.037768+0000","last_unstale":"2023-12-19T14:52:09.051932+0000","last_undegraded":"2023-12-19T14:52:09.051932+0000","last_fullsized":"2023-12-19T14:52:09.051932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:01.072438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016715+0000","last_change":"2023-12-19T14:52:08.031686+0000","last_active":"2023-12-19T14:52:09.016715+0000","last_peered":"2023-12-19T14:52:09.016715+0000","last_clean":"2023-12-19T14:52:09.016715+0000","last_became_active":"2023-12-19T14:52:08.031387+0000","last_became_peered":"2023-12-19T14:52:08.031387+0000","last_unstale":"2023-12-19T14:52:09.016715+0000","last_undegraded":"2023-12-19T14:52:09.016715+0000","last_fullsized":"2023-12-19T14:52:09.016715+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:56.628667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039832+0000","last_change":"2023-12-19T14:52:09.036312+0000","last_active":"2023-12-19T14:52:09.039832+0000","last_peered":"2023-12-19T14:52:09.039832+0000","last_clean":"2023-12-19T14:52:09.039832+0000","last_became_active":"2023-12-19T14:52:09.036071+0000","last_became_peered":"2023-12-19T14:52:09.036071+0000","last_unstale":"2023-12-19T14:52:09.039832+0000","last_undegraded":"2023-12-19T14:52:09.039832+0000","last_fullsized":"2023-12-19T14:52:09.039832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:19.264675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019445+0000","last_change":"2023-12-19T14:52:08.020553+0000","last_active":"2023-12-19T14:52:09.019445+0000","last_peered":"2023-12-19T14:52:09.019445+0000","last_clean":"2023-12-19T14:52:09.019445+0000","last_became_active":"2023-12-19T14:52:08.020292+0000","last_became_peered":"2023-12-19T14:52:08.020292+0000","last_unstale":"2023-12-19T14:52:09.019445+0000","last_undegraded":"2023-12-19T14:52:09.019445+0000","last_fullsized":"2023-12-19T14:52:09.019445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:24.637246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052038+0000","last_change":"2023-12-19T14:52:09.038339+0000","last_active":"2023-12-19T14:52:09.052038+0000","last_peered":"2023-12-19T14:52:09.052038+0000","last_clean":"2023-12-19T14:52:09.052038+0000","last_became_active":"2023-12-19T14:52:09.037978+0000","last_became_peered":"2023-12-19T14:52:09.037978+0000","last_unstale":"2023-12-19T14:52:09.052038+0000","last_undegraded":"2023-12-19T14:52:09.052038+0000","last_fullsized":"2023-12-19T14:52:09.052038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:43:11.519796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.065994+0000","last_change":"2023-12-19T14:52:08.038201+0000","last_active":"2023-12-19T14:52:09.065994+0000","last_peered":"2023-12-19T14:52:09.065994+0000","last_clean":"2023-12-19T14:52:09.065994+0000","last_became_active":"2023-12-19T14:52:08.037044+0000","last_became_peered":"2023-12-19T14:52:08.037044+0000","last_unstale":"2023-12-19T14:52:09.065994+0000","last_undegraded":"2023-12-19T14:52:09.065994+0000","last_fullsized":"2023-12-19T14:52:09.065994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.134038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.039769+0000","last_change":"2023-12-19T14:52:09.033290+0000","last_active":"2023-12-19T14:52:09.039769+0000","last_peered":"2023-12-19T14:52:09.039769+0000","last_clean":"2023-12-19T14:52:09.039769+0000","last_became_active":"2023-12-19T14:52:09.033047+0000","last_became_peered":"2023-12-19T14:52:09.033047+0000","last_unstale":"2023-12-19T14:52:09.039769+0000","last_undegraded":"2023-12-19T14:52:09.039769+0000","last_fullsized":"2023-12-19T14:52:09.039769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:43:11.519796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.065061+0000","last_change":"2023-12-19T14:52:08.040052+0000","last_active":"2023-12-19T14:52:09.065061+0000","last_peered":"2023-12-19T14:52:09.065061+0000","last_clean":"2023-12-19T14:52:09.065061+0000","last_became_active":"2023-12-19T14:52:08.039686+0000","last_became_peered":"2023-12-19T14:52:08.039686+0000","last_unstale":"2023-12-19T14:52:09.065061+0000","last_undegraded":"2023-12-19T14:52:09.065061+0000","last_fullsized":"2023-12-19T14:52:09.065061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:20:52.055179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.046480+0000","last_change":"2023-12-19T14:52:09.044018+0000","last_active":"2023-12-19T14:52:09.046480+0000","last_peered":"2023-12-19T14:52:09.046480+0000","last_clean":"2023-12-19T14:52:09.046480+0000","last_became_active":"2023-12-19T14:52:09.043679+0000","last_became_peered":"2023-12-19T14:52:09.043679+0000","last_unstale":"2023-12-19T14:52:09.046480+0000","last_undegraded":"2023-12-19T14:52:09.046480+0000","last_fullsized":"2023-12-19T14:52:09.046480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:54.481458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.063373+0000","last_change":"2023-12-19T14:52:08.020383+0000","last_active":"2023-12-19T14:52:09.063373+0000","last_peered":"2023-12-19T14:52:09.063373+0000","last_clean":"2023-12-19T14:52:09.063373+0000","last_became_active":"2023-12-19T14:52:08.020063+0000","last_became_peered":"2023-12-19T14:52:08.020063+0000","last_unstale":"2023-12-19T14:52:09.063373+0000","last_undegraded":"2023-12-19T14:52:09.063373+0000","last_fullsized":"2023-12-19T14:52:09.063373+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:14:24.180643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045407+0000","last_change":"2023-12-19T14:52:09.040393+0000","last_active":"2023-12-19T14:52:09.045407+0000","last_peered":"2023-12-19T14:52:09.045407+0000","last_clean":"2023-12-19T14:52:09.045407+0000","last_became_active":"2023-12-19T14:52:09.040117+0000","last_became_peered":"2023-12-19T14:52:09.040117+0000","last_unstale":"2023-12-19T14:52:09.045407+0000","last_undegraded":"2023-12-19T14:52:09.045407+0000","last_fullsized":"2023-12-19T14:52:09.045407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.874904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.056347+0000","last_change":"2023-12-19T14:52:08.018391+0000","last_active":"2023-12-19T14:52:09.056347+0000","last_peered":"2023-12-19T14:52:09.056347+0000","last_clean":"2023-12-19T14:52:09.056347+0000","last_became_active":"2023-12-19T14:52:08.018089+0000","last_became_peered":"2023-12-19T14:52:08.018089+0000","last_unstale":"2023-12-19T14:52:09.056347+0000","last_undegraded":"2023-12-19T14:52:09.056347+0000","last_fullsized":"2023-12-19T14:52:09.056347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:15:41.764443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040260+0000","last_change":"2023-12-19T14:52:09.032437+0000","last_active":"2023-12-19T14:52:09.040260+0000","last_peered":"2023-12-19T14:52:09.040260+0000","last_clean":"2023-12-19T14:52:09.040260+0000","last_became_active":"2023-12-19T14:52:09.032104+0000","last_became_peered":"2023-12-19T14:52:09.032104+0000","last_unstale":"2023-12-19T14:52:09.040260+0000","last_undegraded":"2023-12-19T14:52:09.040260+0000","last_fullsized":"2023-12-19T14:52:09.040260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:56.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.018667+0000","last_change":"2023-12-19T14:52:08.036151+0000","last_active":"2023-12-19T14:52:09.018667+0000","last_peered":"2023-12-19T14:52:09.018667+0000","last_clean":"2023-12-19T14:52:09.018667+0000","last_became_active":"2023-12-19T14:52:08.035753+0000","last_became_peered":"2023-12-19T14:52:08.035753+0000","last_unstale":"2023-12-19T14:52:09.018667+0000","last_undegraded":"2023-12-19T14:52:09.018667+0000","last_fullsized":"2023-12-19T14:52:09.018667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:49.074810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.041338+0000","last_change":"2023-12-19T14:52:09.034497+0000","last_active":"2023-12-19T14:52:09.041338+0000","last_peered":"2023-12-19T14:52:09.041338+0000","last_clean":"2023-12-19T14:52:09.041338+0000","last_became_active":"2023-12-19T14:52:09.034189+0000","last_became_peered":"2023-12-19T14:52:09.034189+0000","last_unstale":"2023-12-19T14:52:09.041338+0000","last_undegraded":"2023-12-19T14:52:09.041338+0000","last_fullsized":"2023-12-19T14:52:09.041338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:24:00.328643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.068461+0000","last_change":"2023-12-19T14:52:08.028420+0000","last_active":"2023-12-19T14:52:09.068461+0000","last_peered":"2023-12-19T14:52:09.068461+0000","last_clean":"2023-12-19T14:52:09.068461+0000","last_became_active":"2023-12-19T14:52:08.028149+0000","last_became_peered":"2023-12-19T14:52:08.028149+0000","last_unstale":"2023-12-19T14:52:09.068461+0000","last_undegraded":"2023-12-19T14:52:09.068461+0000","last_fullsized":"2023-12-19T14:52:09.068461+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:24.637246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040586+0000","last_change":"2023-12-19T14:52:09.034234+0000","last_active":"2023-12-19T14:52:09.040586+0000","last_peered":"2023-12-19T14:52:09.040586+0000","last_clean":"2023-12-19T14:52:09.040586+0000","last_became_active":"2023-12-19T14:52:09.033814+0000","last_became_peered":"2023-12-19T14:52:09.033814+0000","last_unstale":"2023-12-19T14:52:09.040586+0000","last_undegraded":"2023-12-19T14:52:09.040586+0000","last_fullsized":"2023-12-19T14:52:09.040586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:46:39.304602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015366+0000","last_change":"2023-12-19T14:52:08.039472+0000","last_active":"2023-12-19T14:52:09.015366+0000","last_peered":"2023-12-19T14:52:09.015366+0000","last_clean":"2023-12-19T14:52:09.015366+0000","last_became_active":"2023-12-19T14:52:08.038342+0000","last_became_peered":"2023-12-19T14:52:08.038342+0000","last_unstale":"2023-12-19T14:52:09.015366+0000","last_undegraded":"2023-12-19T14:52:09.015366+0000","last_fullsized":"2023-12-19T14:52:09.015366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:53.467987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.051900+0000","last_change":"2023-12-19T14:52:09.042382+0000","last_active":"2023-12-19T14:52:09.051900+0000","last_peered":"2023-12-19T14:52:09.051900+0000","last_clean":"2023-12-19T14:52:09.051900+0000","last_became_active":"2023-12-19T14:52:09.042051+0000","last_became_peered":"2023-12-19T14:52:09.042051+0000","last_unstale":"2023-12-19T14:52:09.051900+0000","last_undegraded":"2023-12-19T14:52:09.051900+0000","last_fullsized":"2023-12-19T14:52:09.051900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:01:38.929717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022719+0000","last_change":"2023-12-19T14:52:08.029511+0000","last_active":"2023-12-19T14:52:09.022719+0000","last_peered":"2023-12-19T14:52:09.022719+0000","last_clean":"2023-12-19T14:52:09.022719+0000","last_became_active":"2023-12-19T14:52:08.028039+0000","last_became_peered":"2023-12-19T14:52:08.028039+0000","last_unstale":"2023-12-19T14:52:09.022719+0000","last_undegraded":"2023-12-19T14:52:09.022719+0000","last_fullsized":"2023-12-19T14:52:09.022719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:55.601122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.025305+0000","last_change":"2023-12-19T14:52:09.023628+0000","last_active":"2023-12-19T14:52:09.025305+0000","last_peered":"2023-12-19T14:52:09.025305+0000","last_clean":"2023-12-19T14:52:09.025305+0000","last_became_active":"2023-12-19T14:52:09.022637+0000","last_became_peered":"2023-12-19T14:52:09.022637+0000","last_unstale":"2023-12-19T14:52:09.025305+0000","last_undegraded":"2023-12-19T14:52:09.025305+0000","last_fullsized":"2023-12-19T14:52:09.025305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:41.755184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.032923+0000","last_change":"2023-12-19T14:52:08.017821+0000","last_active":"2023-12-19T14:52:09.032923+0000","last_peered":"2023-12-19T14:52:09.032923+0000","last_clean":"2023-12-19T14:52:09.032923+0000","last_became_active":"2023-12-19T14:52:08.017436+0000","last_became_peered":"2023-12-19T14:52:08.017436+0000","last_unstale":"2023-12-19T14:52:09.032923+0000","last_undegraded":"2023-12-19T14:52:09.032923+0000","last_fullsized":"2023-12-19T14:52:09.032923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:32.518002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.031145+0000","last_change":"2023-12-19T14:52:09.027220+0000","last_active":"2023-12-19T14:52:09.031145+0000","last_peered":"2023-12-19T14:52:09.031145+0000","last_clean":"2023-12-19T14:52:09.031145+0000","last_became_active":"2023-12-19T14:52:09.025613+0000","last_became_peered":"2023-12-19T14:52:09.025613+0000","last_unstale":"2023-12-19T14:52:09.031145+0000","last_undegraded":"2023-12-19T14:52:09.031145+0000","last_fullsized":"2023-12-19T14:52:09.031145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:24.852226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.058083+0000","last_change":"2023-12-19T14:52:08.031733+0000","last_active":"2023-12-19T14:52:09.058083+0000","last_peered":"2023-12-19T14:52:09.058083+0000","last_clean":"2023-12-19T14:52:09.058083+0000","last_became_active":"2023-12-19T14:52:08.031422+0000","last_became_peered":"2023-12-19T14:52:08.031422+0000","last_unstale":"2023-12-19T14:52:09.058083+0000","last_undegraded":"2023-12-19T14:52:09.058083+0000","last_fullsized":"2023-12-19T14:52:09.058083+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:39:29.554303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.042212+0000","last_change":"2023-12-19T14:52:09.037799+0000","last_active":"2023-12-19T14:52:09.042212+0000","last_peered":"2023-12-19T14:52:09.042212+0000","last_clean":"2023-12-19T14:52:09.042212+0000","last_became_active":"2023-12-19T14:52:09.037498+0000","last_became_peered":"2023-12-19T14:52:09.037498+0000","last_unstale":"2023-12-19T14:52:09.042212+0000","last_undegraded":"2023-12-19T14:52:09.042212+0000","last_fullsized":"2023-12-19T14:52:09.042212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:53.099678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022252+0000","last_change":"2023-12-19T14:52:08.022071+0000","last_active":"2023-12-19T14:52:09.022252+0000","last_peered":"2023-12-19T14:52:09.022252+0000","last_clean":"2023-12-19T14:52:09.022252+0000","last_became_active":"2023-12-19T14:52:08.021756+0000","last_became_peered":"2023-12-19T14:52:08.021756+0000","last_unstale":"2023-12-19T14:52:09.022252+0000","last_undegraded":"2023-12-19T14:52:09.022252+0000","last_fullsized":"2023-12-19T14:52:09.022252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:31.298025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.030579+0000","last_change":"2023-12-19T14:52:09.027470+0000","last_active":"2023-12-19T14:52:09.030579+0000","last_peered":"2023-12-19T14:52:09.030579+0000","last_clean":"2023-12-19T14:52:09.030579+0000","last_became_active":"2023-12-19T14:52:09.027194+0000","last_became_peered":"2023-12-19T14:52:09.027194+0000","last_unstale":"2023-12-19T14:52:09.030579+0000","last_undegraded":"2023-12-19T14:52:09.030579+0000","last_fullsized":"2023-12-19T14:52:09.030579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:50.893600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.022909+0000","last_change":"2023-12-19T14:52:08.036747+0000","last_active":"2023-12-19T14:52:09.022909+0000","last_peered":"2023-12-19T14:52:09.022909+0000","last_clean":"2023-12-19T14:52:09.022909+0000","last_became_active":"2023-12-19T14:52:08.036283+0000","last_became_peered":"2023-12-19T14:52:08.036283+0000","last_unstale":"2023-12-19T14:52:09.022909+0000","last_undegraded":"2023-12-19T14:52:09.022909+0000","last_fullsized":"2023-12-19T14:52:09.022909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:40:14.721168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047751+0000","last_change":"2023-12-19T14:52:09.033922+0000","last_active":"2023-12-19T14:52:09.047751+0000","last_peered":"2023-12-19T14:52:09.047751+0000","last_clean":"2023-12-19T14:52:09.047751+0000","last_became_active":"2023-12-19T14:52:09.033637+0000","last_became_peered":"2023-12-19T14:52:09.033637+0000","last_unstale":"2023-12-19T14:52:09.047751+0000","last_undegraded":"2023-12-19T14:52:09.047751+0000","last_fullsized":"2023-12-19T14:52:09.047751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:03:00.136172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016725+0000","last_change":"2023-12-19T14:52:08.031238+0000","last_active":"2023-12-19T14:52:09.016725+0000","last_peered":"2023-12-19T14:52:09.016725+0000","last_clean":"2023-12-19T14:52:09.016725+0000","last_became_active":"2023-12-19T14:52:08.030960+0000","last_became_peered":"2023-12-19T14:52:08.030960+0000","last_unstale":"2023-12-19T14:52:09.016725+0000","last_undegraded":"2023-12-19T14:52:09.016725+0000","last_fullsized":"2023-12-19T14:52:09.016725+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:47.166820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.045780+0000","last_change":"2023-12-19T14:52:09.043548+0000","last_active":"2023-12-19T14:52:09.045780+0000","last_peered":"2023-12-19T14:52:09.045780+0000","last_clean":"2023-12-19T14:52:09.045780+0000","last_became_active":"2023-12-19T14:52:09.043249+0000","last_became_peered":"2023-12-19T14:52:09.043249+0000","last_unstale":"2023-12-19T14:52:09.045780+0000","last_undegraded":"2023-12-19T14:52:09.045780+0000","last_fullsized":"2023-12-19T14:52:09.045780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:42:49.888902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015493+0000","last_change":"2023-12-19T14:52:08.036803+0000","last_active":"2023-12-19T14:52:09.015493+0000","last_peered":"2023-12-19T14:52:09.015493+0000","last_clean":"2023-12-19T14:52:09.015493+0000","last_became_active":"2023-12-19T14:52:08.036477+0000","last_became_peered":"2023-12-19T14:52:08.036477+0000","last_unstale":"2023-12-19T14:52:09.015493+0000","last_undegraded":"2023-12-19T14:52:09.015493+0000","last_fullsized":"2023-12-19T14:52:09.015493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:41.939612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.052239+0000","last_change":"2023-12-19T14:52:09.044777+0000","last_active":"2023-12-19T14:52:09.052239+0000","last_peered":"2023-12-19T14:52:09.052239+0000","last_clean":"2023-12-19T14:52:09.052239+0000","last_became_active":"2023-12-19T14:52:09.044457+0000","last_became_peered":"2023-12-19T14:52:09.044457+0000","last_unstale":"2023-12-19T14:52:09.052239+0000","last_undegraded":"2023-12-19T14:52:09.052239+0000","last_fullsized":"2023-12-19T14:52:09.052239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:50.893600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.015643+0000","last_change":"2023-12-19T14:52:08.033941+0000","last_active":"2023-12-19T14:52:09.015643+0000","last_peered":"2023-12-19T14:52:09.015643+0000","last_clean":"2023-12-19T14:52:09.015643+0000","last_became_active":"2023-12-19T14:52:08.033664+0000","last_became_peered":"2023-12-19T14:52:08.033664+0000","last_unstale":"2023-12-19T14:52:09.015643+0000","last_undegraded":"2023-12-19T14:52:09.015643+0000","last_fullsized":"2023-12-19T14:52:09.015643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:04.277310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.047632+0000","last_change":"2023-12-19T14:52:09.039210+0000","last_active":"2023-12-19T14:52:09.047632+0000","last_peered":"2023-12-19T14:52:09.047632+0000","last_clean":"2023-12-19T14:52:09.047632+0000","last_became_active":"2023-12-19T14:52:09.038893+0000","last_became_peered":"2023-12-19T14:52:09.038893+0000","last_unstale":"2023-12-19T14:52:09.047632+0000","last_undegraded":"2023-12-19T14:52:09.047632+0000","last_fullsized":"2023-12-19T14:52:09.047632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:06.248039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.016998+0000","last_change":"2023-12-19T14:52:08.027246+0000","last_active":"2023-12-19T14:52:09.016998+0000","last_peered":"2023-12-19T14:52:09.016998+0000","last_clean":"2023-12-19T14:52:09.016998+0000","last_became_active":"2023-12-19T14:52:08.026970+0000","last_became_peered":"2023-12-19T14:52:08.026970+0000","last_unstale":"2023-12-19T14:52:09.016998+0000","last_undegraded":"2023-12-19T14:52:09.016998+0000","last_fullsized":"2023-12-19T14:52:09.016998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:16.609715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.040718+0000","last_change":"2023-12-19T14:52:09.034829+0000","last_active":"2023-12-19T14:52:09.040718+0000","last_peered":"2023-12-19T14:52:09.040718+0000","last_clean":"2023-12-19T14:52:09.040718+0000","last_became_active":"2023-12-19T14:52:09.034074+0000","last_became_peered":"2023-12-19T14:52:09.034074+0000","last_unstale":"2023-12-19T14:52:09.040718+0000","last_undegraded":"2023-12-19T14:52:09.040718+0000","last_fullsized":"2023-12-19T14:52:09.040718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:07.989739+0000","last_clean_scrub_stamp":"2023-12-19T14:52:07.989739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:25.650717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.024049+0000","last_change":"2023-12-19T14:52:08.033730+0000","last_active":"2023-12-19T14:52:09.024049+0000","last_peered":"2023-12-19T14:52:09.024049+0000","last_clean":"2023-12-19T14:52:09.024049+0000","last_became_active":"2023-12-19T14:52:08.033421+0000","last_became_peered":"2023-12-19T14:52:08.033421+0000","last_unstale":"2023-12-19T14:52:09.024049+0000","last_undegraded":"2023-12-19T14:52:09.024049+0000","last_fullsized":"2023-12-19T14:52:09.024049+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:52:06.976268+0000","last_clean_scrub_stamp":"2023-12-19T14:52:06.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:48:00.344139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":281,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T14:52:09.019508+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:52:09.019508+0000","last_peered":"2023-12-19T14:52:09.019508+0000","last_clean":"2023-12-19T14:52:09.019508+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:52:09.019508+0000","last_undegraded":"2023-12-19T14:52:09.019508+0000","last_fullsized":"2023-12-19T14:52:09.019508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310948,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060892,"statfs":{"total":96636764160,"available":96318353408,"internally_reserved":0,"allocated":1404928,"data_stored":985447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.109,"15min":1.099},"min":{"1min":0.59699999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.48,"5min":1.579,"15min":1.579},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.123,"15min":1.1579999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.488,"5min":1.5669999999999999,"15min":2.2799999999999998},"last":1.246}]},{"osd":1,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1120000000000001,"15min":1.0940000000000001},"min":{"1min":0.76400000000000001,"5min":0.67600000000000005,"15min":0.59299999999999997},"max":{"1min":1.48,"5min":1.73,"15min":1.73},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.181,"15min":1.1799999999999999},"min":{"1min":0.89400000000000002,"5min":0.77700000000000002,"15min":0.63},"max":{"1min":1.488,"5min":1.6060000000000001,"15min":1.984},"last":0.81999999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.236,"15min":1.2310000000000001},"min":{"1min":0.95399999999999996,"5min":0.67900000000000005,"15min":0.623},"max":{"1min":1.534,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.2929999999999999,"15min":1.296},"min":{"1min":1.0640000000000001,"5min":0.85199999999999998,"15min":0.70999999999999996},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.296,"15min":1.3089999999999999},"min":{"1min":0.96799999999999997,"5min":0.879,"15min":0.76200000000000001},"max":{"1min":1.5680000000000001,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3180000000000001,"15min":1.3200000000000001},"min":{"1min":1.0189999999999999,"5min":0.999,"15min":0.79700000000000004},"max":{"1min":1.6379999999999999,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.0489999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.159,"15min":1.206},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.58999999999999997},"max":{"1min":1.4890000000000001,"5min":1.75,"15min":2.1930000000000001},"last":1.113},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2050000000000001,"15min":1.2569999999999999},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.67500000000000004},"max":{"1min":1.6180000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.5389999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2050000000000001,"15min":1.256},"min":{"1min":0.84399999999999997,"5min":0.69599999999999995,"15min":0.69099999999999995},"max":{"1min":1.675,"5min":1.8109999999999999,"15min":1.976},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.276,"15min":1.3620000000000001},"min":{"1min":0.84099999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.581,"5min":1.6950000000000001,"15min":2.3620000000000001},"last":1.3660000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.387,"15min":1.4470000000000001},"min":{"1min":0.94099999999999995,"5min":0.88100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5669999999999999,"5min":1.8029999999999999,"15min":3.1589999999999998},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.393},"min":{"1min":0.96599999999999997,"5min":0.85499999999999998,"15min":0.70299999999999996},"max":{"1min":1.6850000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.431}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":823296,"data_stored":402187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1439999999999999,"15min":1.1919999999999999},"min":{"1min":0.70299999999999996,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.6659999999999999,"5min":1.877,"15min":1.8939999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2090000000000001,"15min":1.2330000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.64000000000000001},"max":{"1min":1.512,"5min":1.6830000000000001,"15min":2.335},"last":1.7949999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.196,"15min":1.218},"min":{"1min":0.96599999999999997,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":1.325,"5min":1.7649999999999999,"15min":2.0339999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.274,"15min":1.3149999999999999},"min":{"1min":0.85599999999999998,"5min":0.84499999999999997,"15min":0.747},"max":{"1min":1.51,"5min":1.714,"15min":3.0990000000000002},"last":1.24}]},{"osd":2,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.23,"15min":1.276},"min":{"1min":0.99099999999999999,"5min":0.84999999999999998,"15min":0.78800000000000003},"max":{"1min":1.472,"5min":1.7390000000000001,"15min":2.4900000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.248,"15min":1.272},"min":{"1min":0.95399999999999996,"5min":0.70799999999999996,"15min":0.67100000000000004},"max":{"1min":1.5149999999999999,"5min":1.849,"15min":2.4550000000000001},"last":1.6279999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.319,"15min":1.3779999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.65800000000000003},"max":{"1min":1.7010000000000001,"5min":1.778,"15min":3.2330000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.329,"15min":1.3879999999999999},"min":{"1min":0.996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6759999999999999,"5min":1.879,"15min":3.194},"last":1.4530000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2110000000000001,"15min":1.292},"min":{"1min":0.83399999999999996,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":1.776,"5min":2.073,"15min":3.0150000000000001},"last":1.825},{"interface":"front","average":{"1min":1.107,"5min":1.2789999999999999,"15min":1.383},"min":{"1min":0.79800000000000004,"5min":0.68500000000000005,"15min":0.57599999999999996},"max":{"1min":1.611,"5min":2.3450000000000002,"15min":3.4399999999999999},"last":2.0350000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2989999999999999,"15min":1.3959999999999999},"min":{"1min":1.0089999999999999,"5min":0.74199999999999999,"15min":0.63100000000000001},"max":{"1min":1.8580000000000001,"5min":2.0219999999999998,"15min":2.8620000000000001},"last":1.734},{"interface":"front","average":{"1min":1.335,"5min":1.331,"15min":1.4119999999999999},"min":{"1min":1.05,"5min":0.76300000000000001,"15min":0.51000000000000001},"max":{"1min":1.6299999999999999,"5min":1.7609999999999999,"15min":2.9420000000000002},"last":2.1110000000000002}]},{"osd":7,"last update":"Tue Dec 19 14:51:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4119999999999999,"15min":1.482},"min":{"1min":0.97299999999999998,"5min":0.93400000000000005,"15min":0.53000000000000003},"max":{"1min":1.766,"5min":2.0249999999999999,"15min":3.367},"last":2.214},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.462,"15min":1.5469999999999999},"min":{"1min":0.93100000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.97,"5min":2.234,"15min":3.2839999999999998},"last":1.974}]}]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":394955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0640000000000001,"15min":1.0580000000000001},"min":{"1min":0.71899999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.325,"5min":1.5800000000000001,"15min":1.619},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.141,"15min":1.133},"min":{"1min":0.78000000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6399999999999999,"5min":1.6579999999999999,"15min":1.718},"last":1.153}]},{"osd":1,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.286,"15min":1.2909999999999999},"min":{"1min":0.95999999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.554,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.202,"5min":1.232,"15min":1.2509999999999999},"min":{"1min":0.95099999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.482,"5min":1.571,"15min":1.7869999999999999},"last":0.95299999999999996}]},{"osd":2,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.268,"15min":1.228},"min":{"1min":0.80200000000000005,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5760000000000001,"5min":1.7450000000000001,"15min":1.847},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.2,"15min":1.2330000000000001},"min":{"1min":0.82899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.589,"5min":1.8600000000000001,"15min":2.0739999999999998},"last":0.92100000000000004}]},{"osd":3,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.2769999999999999,"15min":1.2869999999999999},"min":{"1min":1.024,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6679999999999999,"5min":1.71,"15min":1.9770000000000001},"last":1.411},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.256,"15min":1.3049999999999999},"min":{"1min":0.82299999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.524,"5min":1.78,"15min":2.2149999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0880000000000001,"15min":1.145},"min":{"1min":0.77500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5149999999999999,"5min":1.5369999999999999,"15min":2.1099999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.234,"15min":1.246},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.49299999999999999},"max":{"1min":1.649,"5min":1.6990000000000001,"15min":2.073},"last":1.268}]},{"osd":6,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.3200000000000001,"15min":1.3660000000000001},"min":{"1min":0.91900000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6599999999999999,"5min":1.984,"15min":2.4790000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.349,"5min":1.3280000000000001,"15min":1.3540000000000001},"min":{"1min":0.95999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.641,"5min":1.7270000000000001,"15min":2.0030000000000001},"last":1.0940000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3240000000000001,"15min":1.375},"min":{"1min":0.89700000000000002,"5min":0.82399999999999995,"15min":0.58899999999999997},"max":{"1min":1.5980000000000001,"5min":1.764,"15min":2.4470000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.349,"15min":1.429},"min":{"1min":0.89800000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.6080000000000001,"5min":2.0379999999999998,"15min":2.411},"last":1.105}]}]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302184,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069656,"statfs":{"total":96636764160,"available":96327327744,"internally_reserved":0,"allocated":819200,"data_stored":402117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0880000000000001,"15min":1.079},"min":{"1min":0.75700000000000001,"5min":0.58599999999999997,"15min":0.58199999999999996},"max":{"1min":1.579,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.391},{"interface":"front","average":{"1min":1.129,"5min":1.095,"15min":1.1020000000000001},"min":{"1min":0.67900000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.377,"5min":1.5089999999999999,"15min":1.75},"last":0.90800000000000003}]},{"osd":1,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2350000000000001,"15min":1.2430000000000001},"min":{"1min":0.79800000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.6259999999999999,"5min":1.6819999999999999,"15min":2.1259999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2729999999999999,"15min":1.2869999999999999},"min":{"1min":0.88200000000000001,"5min":0.71599999999999997,"15min":0.65100000000000002},"max":{"1min":1.55,"5min":1.9790000000000001,"15min":2.2109999999999999},"last":1.47}]},{"osd":2,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.86599999999999999,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.498,"5min":1.6040000000000001,"15min":1.891},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.226,"15min":1.2230000000000001},"min":{"1min":1.0209999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.5720000000000001,"5min":1.893,"15min":1.893},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1859999999999999,"15min":1.198},"min":{"1min":0.95599999999999996,"5min":0.85899999999999999,"15min":0.59599999999999997},"max":{"1min":1.399,"5min":1.6379999999999999,"15min":1.8740000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":0.95799999999999996,"5min":0.75600000000000001,"15min":0.60599999999999998},"max":{"1min":1.712,"5min":1.712,"15min":2.0190000000000001},"last":1.226}]},{"osd":5,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.127,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.425,"5min":2.0449999999999999,"15min":2.3500000000000001},"last":1.296},{"interface":"front","average":{"1min":1.099,"5min":1.119,"15min":1.194},"min":{"1min":0.79900000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.581,"5min":1.7210000000000001,"15min":2.161},"last":1.278}]},{"osd":6,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2809999999999999,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.6639999999999999,"5min":1.845,"15min":3.27},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.381,"15min":1.4630000000000001},"min":{"1min":0.94599999999999995,"5min":0.68899999999999995,"15min":0.67600000000000005},"max":{"1min":1.665,"5min":2.0960000000000001,"15min":2.742},"last":1.5580000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.3069999999999999,"15min":1.4390000000000001},"min":{"1min":1.0249999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7649999999999999,"5min":1.9079999999999999,"15min":5.2060000000000004},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3999999999999999,"15min":1.518},"min":{"1min":0.97399999999999998,"5min":0.72599999999999998,"15min":0.70599999999999996},"max":{"1min":1.825,"5min":1.825,"15min":3.6739999999999999},"last":1.5900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306280,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065560,"statfs":{"total":96636764160,"available":96323133440,"internally_reserved":0,"allocated":819200,"data_stored":396513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.873,"15min":0.96799999999999997},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.53500000000000003},"max":{"1min":1.3640000000000001,"5min":1.419,"15min":4.1159999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.026,"15min":1.069},"min":{"1min":0.79200000000000004,"5min":0.59099999999999997,"15min":0.50900000000000001},"max":{"1min":1.4059999999999999,"5min":1.7270000000000001,"15min":3.3929999999999998},"last":0.91200000000000003}]},{"osd":1,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.94099999999999995,"15min":1.02},"min":{"1min":0.66700000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.224,"5min":1.4299999999999999,"15min":2.4390000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1080000000000001,"15min":1.1759999999999999},"min":{"1min":0.72999999999999998,"5min":0.66900000000000004,"15min":0.60199999999999998},"max":{"1min":1.532,"5min":1.758,"15min":4.476},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1379999999999999,"15min":1.1890000000000001},"min":{"1min":0.73299999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6879999999999999,"5min":1.76,"15min":2.6989999999999998},"last":1.159},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2170000000000001,"15min":1.2789999999999999},"min":{"1min":0.85799999999999998,"5min":0.68000000000000005,"15min":0.61699999999999999},"max":{"1min":1.647,"5min":1.7190000000000001,"15min":2.5110000000000001},"last":1.0549999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3180000000000001,"15min":1.4079999999999999},"min":{"1min":0.97999999999999998,"5min":0.72999999999999998,"15min":0.69899999999999995},"max":{"1min":1.506,"5min":1.768,"15min":3.46},"last":1.101},{"interface":"front","average":{"1min":1.319,"5min":1.3,"15min":1.3560000000000001},"min":{"1min":0.96099999999999997,"5min":0.85899999999999999,"15min":0.84799999999999998},"max":{"1min":1.6659999999999999,"5min":1.7989999999999999,"15min":2.9409999999999998},"last":1.448}]},{"osd":5,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3080000000000001,"15min":1.3959999999999999},"min":{"1min":1.1499999999999999,"5min":0.85999999999999999,"15min":0.77100000000000002},"max":{"1min":1.5309999999999999,"5min":1.6599999999999999,"15min":3.1019999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3799999999999999,"15min":1.4890000000000001},"min":{"1min":1.1379999999999999,"5min":0.97499999999999998,"15min":0.94499999999999995},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":4.3650000000000002},"last":1.244}]},{"osd":6,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4510000000000001,"15min":1.55},"min":{"1min":1.1850000000000001,"5min":1.1060000000000001,"15min":0.97499999999999998},"max":{"1min":1.883,"5min":1.9219999999999999,"15min":4.5999999999999996},"last":1.653},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.391,"15min":1.524},"min":{"1min":1.147,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.74,"5min":2.0489999999999999,"15min":3.4329999999999998},"last":1.3280000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:20 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.498,"15min":1.629},"min":{"1min":1.177,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.796,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4419999999999999,"15min":1.5740000000000001},"min":{"1min":1.224,"5min":0.97599999999999998,"15min":0.89700000000000002},"max":{"1min":1.738,"5min":1.99,"15min":3.7240000000000002},"last":1.546}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306284,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065556,"statfs":{"total":96636764160,"available":96323129344,"internally_reserved":0,"allocated":823296,"data_stored":402088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.89600000000000002,"15min":0.93200000000000005},"min":{"1min":0.629,"5min":0.52800000000000002,"15min":0.36599999999999999},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.946},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.109,"15min":1.091},"min":{"1min":0.92100000000000004,"5min":0.69999999999999996,"15min":0.54100000000000004},"max":{"1min":1.4410000000000001,"5min":1.546,"15min":1.8560000000000001},"last":1.083}]},{"osd":1,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1519999999999999,"15min":1.143},"min":{"1min":0.81399999999999995,"5min":0.61799999999999999,"15min":0.46500000000000002},"max":{"1min":1.554,"5min":1.6479999999999999,"15min":1.8},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.091,"15min":1.111},"min":{"1min":0.72099999999999997,"5min":0.63600000000000001,"15min":0.62},"max":{"1min":1.429,"5min":1.655,"15min":1.716},"last":0.754}]},{"osd":3,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.997,"15min":1.026},"min":{"1min":0.79900000000000004,"5min":0.64200000000000002,"15min":0.55600000000000005},"max":{"1min":1.1639999999999999,"5min":1.623,"15min":1.9370000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.034,"5min":1.097,"15min":1.1160000000000001},"min":{"1min":0.80100000000000005,"5min":0.70999999999999996,"15min":0.55000000000000004},"max":{"1min":1.3120000000000001,"5min":1.518,"15min":1.6930000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.401,"15min":1.425},"min":{"1min":1.1200000000000001,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.724,"5min":2.044,"15min":3.4950000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4099999999999999,"15min":1.4159999999999999},"min":{"1min":1.244,"5min":1.018,"15min":0.82799999999999996},"max":{"1min":1.619,"5min":1.9339999999999999,"15min":2.0409999999999999},"last":1.272}]},{"osd":5,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.339,"15min":1.403},"min":{"1min":1.085,"5min":0.89300000000000002,"15min":0.89000000000000001},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":3.8599999999999999},"last":1.484},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.347,"15min":1.3859999999999999},"min":{"1min":0.94399999999999995,"5min":0.93400000000000005,"15min":0.755},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.2070000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.369,"15min":1.399},"min":{"1min":1.079,"5min":0.83299999999999996,"15min":0.80600000000000005},"max":{"1min":1.6739999999999999,"5min":1.9850000000000001,"15min":2.8759999999999999},"last":1.119},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4330000000000001,"15min":1.49},"min":{"1min":1.1799999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.714,"5min":2.137,"15min":3.7309999999999999},"last":1.3240000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:13 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4750000000000001,"15min":1.53},"min":{"1min":1.1899999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.903,"5min":2.2080000000000002,"15min":2.5550000000000002},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.464,"15min":1.514},"min":{"1min":1.0740000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.788,"5min":2.181,"15min":2.4049999999999998},"last":1.3799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306272,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065568,"statfs":{"total":96636764160,"available":96323141632,"internally_reserved":0,"allocated":811008,"data_stored":395115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.85499999999999998,"15min":0.88100000000000001},"min":{"1min":0.69499999999999995,"5min":0.51000000000000001,"15min":0.443},"max":{"1min":1.2769999999999999,"5min":1.871,"15min":1.871},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0049999999999999,"15min":1.028},"min":{"1min":0.76200000000000001,"5min":0.53100000000000003,"15min":0.50700000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.468},"last":0.95799999999999996}]},{"osd":2,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0209999999999999,"15min":1.0469999999999999},"min":{"1min":0.82199999999999995,"5min":0.59699999999999998,"15min":0.46500000000000002},"max":{"1min":1.714,"5min":1.714,"15min":1.7829999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.079,"5min":1.131,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68899999999999995,"15min":0.67800000000000005},"max":{"1min":1.429,"5min":1.607,"15min":3.0619999999999998},"last":1.373}]},{"osd":3,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.256,"15min":1.2350000000000001},"min":{"1min":0.94999999999999996,"5min":0.69699999999999995,"15min":0.59199999999999997},"max":{"1min":1.6499999999999999,"5min":1.6919999999999999,"15min":1.9490000000000001},"last":1.538},{"interface":"front","average":{"1min":1.163,"5min":1.0780000000000001,"15min":1.103},"min":{"1min":0.86699999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.26}]},{"osd":4,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3240000000000001,"15min":1.3640000000000001},"min":{"1min":0.997,"5min":0.89200000000000002,"15min":0.72299999999999998},"max":{"1min":1.6439999999999999,"5min":1.8480000000000001,"15min":2.6440000000000001},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.413,"15min":1.4239999999999999},"min":{"1min":1.109,"5min":0.96999999999999997,"15min":0.69499999999999995},"max":{"1min":1.7569999999999999,"5min":1.7749999999999999,"15min":2.7389999999999999},"last":1.302}]},{"osd":5,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.296,"15min":1.329},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.92200000000000004},"max":{"1min":1.5880000000000001,"5min":1.6120000000000001,"15min":2.1320000000000001},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3560000000000001,"15min":1.4259999999999999},"min":{"1min":1.0509999999999999,"5min":1.006,"15min":0.81000000000000005},"max":{"1min":1.6479999999999999,"5min":1.9530000000000001,"15min":2.79},"last":1.4099999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3520000000000001,"15min":1.389},"min":{"1min":1.173,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.629,"5min":1.9179999999999999,"15min":2.4620000000000002},"last":1.581},{"interface":"front","average":{"1min":1.47,"5min":1.4490000000000001,"15min":1.498},"min":{"1min":1.046,"5min":1.02,"15min":0.996},"max":{"1min":1.8129999999999999,"5min":2.0089999999999999,"15min":2.8999999999999999},"last":1.6240000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:19 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.498,"15min":1.536},"min":{"1min":1.3140000000000001,"5min":1.0660000000000001,"15min":0.73599999999999999},"max":{"1min":2.0470000000000002,"5min":2.0590000000000002,"15min":3.782},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.4099999999999999,"15min":1.478},"min":{"1min":1.1479999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":2.5390000000000001},"last":1.6870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306860,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064980,"statfs":{"total":96636764160,"available":96322539520,"internally_reserved":0,"allocated":1413120,"data_stored":990976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.65600000000000003,"5min":0.54600000000000004,"15min":0.505},"max":{"1min":1.266,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.96599999999999997,"15min":0.99199999999999999},"min":{"1min":0.67000000000000004,"5min":0.54500000000000004,"15min":0.53600000000000003},"max":{"1min":1.423,"5min":1.5840000000000001,"15min":1.7450000000000001},"last":1.222}]},{"osd":2,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1339999999999999,"15min":1.1180000000000001},"min":{"1min":0.88,"5min":0.66100000000000003,"15min":0.55200000000000005},"max":{"1min":1.6339999999999999,"5min":1.653,"15min":1.8979999999999999},"last":1.014},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2529999999999999,"15min":1.2370000000000001},"min":{"1min":0.91400000000000003,"5min":0.68799999999999994,"15min":0.54300000000000004},"max":{"1min":1.893,"5min":1.893,"15min":1.893},"last":0.97399999999999998}]},{"osd":3,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.054,"15min":1.0349999999999999},"min":{"1min":0.77100000000000002,"5min":0.70199999999999996,"15min":0.67800000000000005},"max":{"1min":1.708,"5min":1.708,"15min":2.0299999999999998},"last":1.085},{"interface":"front","average":{"1min":1.22,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.89400000000000002,"5min":0.81100000000000005,"15min":0.71199999999999997},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.8859999999999999},"last":1.3859999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3140000000000001,"15min":1.3340000000000001},"min":{"1min":1.0309999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":2.298},"last":1.542},{"interface":"front","average":{"1min":1.294,"5min":1.292,"15min":1.3160000000000001},"min":{"1min":1.0369999999999999,"5min":0.89400000000000002,"15min":0.80200000000000005},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.5980000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3720000000000001,"15min":1.3959999999999999},"min":{"1min":1.1950000000000001,"5min":0.86799999999999999,"15min":0.76200000000000001},"max":{"1min":1.7549999999999999,"5min":1.79,"15min":2.4630000000000001},"last":1.802},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.405,"15min":1.4339999999999999},"min":{"1min":1.2030000000000001,"5min":0.74099999999999999,"15min":0.58299999999999996},"max":{"1min":1.7290000000000001,"5min":1.881,"15min":2.6160000000000001},"last":1.851}]},{"osd":6,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.353,"15min":1.3819999999999999},"min":{"1min":1.101,"5min":0.91500000000000004,"15min":0.88300000000000001},"max":{"1min":1.6319999999999999,"5min":1.827,"15min":2.339},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4430000000000001,"15min":1.4870000000000001},"min":{"1min":1.1850000000000001,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":1.716,"5min":2.1240000000000001,"15min":2.5680000000000001},"last":1.9710000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:51:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.502,"15min":1.508},"min":{"1min":1.218,"5min":0.94799999999999995,"15min":0.91500000000000004},"max":{"1min":1.927,"5min":1.9299999999999999,"15min":2.5129999999999999},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.429,"15min":1.4930000000000001},"min":{"1min":1.153,"5min":1.0009999999999999,"15min":0.88200000000000001},"max":{"1min":1.835,"5min":1.907,"15min":2.4500000000000002},"last":1.931}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:52:13.782 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:52:13.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:52:13.973 INFO:tasks.ceph.mds.a.smithi050.stderr:2023-12-19T14:52:13.970+0000 7f79fb916700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:52:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 Processor -- start 2023-12-19T14:52:14.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 -- start start 2023-12-19T14:52:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 0x7f79ac159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 0x7f79ac0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac0cbbb0 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79ac06dc90 con 0x7f79ac0caa10 2023-12-19T14:52:14.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b1a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0cafa0 con 0x7f79ac0cc040 2023-12-19T14:52:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79abfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 0x7f79ac159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79abfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 0x7f79ac159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:41912/0 (socket says 172.21.15.50:41912) 2023-12-19T14:52:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79abfff700 1 -- 172.21.15.50:0/3291578189 learned_addr learned my addr 172.21.15.50:0/3291578189 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b0a0a700 1 --1- 172.21.15.50:0/3291578189 >> v1:172.21.15.125:6789/0 conn(0x7f79ac0ce2d0 0x7f79ac158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:55188/0 (socket says 172.21.15.50:55188) 2023-12-19T14:52:14.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:13.998+0000 7f79b120b700 1 --2- 172.21.15.50:0/3291578189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 0x7f79ac0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79ab7fe700 1 -- 172.21.15.50:0/3291578189 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386148463 0 0) 0x7f79ac0cbbb0 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79ab7fe700 1 -- 172.21.15.50:0/3291578189 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c003570 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b120b700 1 -- 172.21.15.50:0/3291578189 >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 msgr2=0x7f79ac159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b120b700 1 --2- 172.21.15.50:0/3291578189 >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 0x7f79ac159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b120b700 1 -- 172.21.15.50:0/3291578189 >> v1:172.21.15.125:6789/0 conn(0x7f79ac0ce2d0 legacy=0x7f79ac158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b120b700 1 -- 172.21.15.50:0/3291578189 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac0cb280 con 0x7f79ac0cc040 2023-12-19T14:52:14.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b120b700 1 --2- 172.21.15.50:0/3291578189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 0x7f79ac0ce6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009e60 tx=0x7f79a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d3fe92808ff8a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79ab7fe700 1 -- 172.21.15.50:0/3291578189 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400e070 con 0x7f79ac0cc040 2023-12-19T14:52:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79ab7fe700 1 -- 172.21.15.50:0/3291578189 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a400bab0 con 0x7f79ac0cc040 2023-12-19T14:52:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79ab7fe700 1 -- 172.21.15.50:0/3291578189 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400be70 con 0x7f79ac0cc040 2023-12-19T14:52:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 -- 172.21.15.50:0/3291578189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 msgr2=0x7f79ac0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/3291578189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 0x7f79ac0ce6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009e60 tx=0x7f79a4009780 comp rx=0 tx=0).stop 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 -- 172.21.15.50:0/3291578189 shutdown_connections 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/3291578189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0cc040 0x7f79ac0ce6a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/3291578189 >> v2:172.21.15.125:3300/0 conn(0x7f79ac0caa10 0x7f79ac159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.002+0000 7f79b1a0c700 1 -- 172.21.15.50:0/3291578189 >> 172.21.15.50:0/3291578189 conn(0x7f79ac067bb0 msgr2=0x7f79ac0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.006+0000 7f79b1a0c700 1 -- 172.21.15.50:0/3291578189 shutdown_connections 2023-12-19T14:52:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.006+0000 7f79b1a0c700 1 -- 172.21.15.50:0/3291578189 wait complete. 2023-12-19T14:52:14.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.006+0000 7f79b1a0c700 1 Processor -- start 2023-12-19T14:52:14.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.006+0000 7f79b1a0c700 1 -- start start 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0caa10 0x7f79ac151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79ac0cc040 0x7f79ac14f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0c82c0 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0b2d90 con 0x7f79ac0cc040 2023-12-19T14:52:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac0b3070 con 0x7f79ac0caa10 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33650/0 (socket says 172.21.15.50:33650) 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b0a0a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0caa10 0x7f79ac151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 -- 172.21.15.50:0/889372921 learned_addr learned my addr 172.21.15.50:0/889372921 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79abfff700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79ac0cc040 0x7f79ac14f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 -- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0caa10 msgr2=0x7f79ac151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0caa10 0x7f79ac151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 -- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79ac0cc040 msgr2=0x7f79ac14f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79ac0cc040 0x7f79ac14f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 -- 172.21.15.50:0/889372921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a4009a20 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b120b700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0634d0 tx=0x7f79a4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4012e50 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4027470 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac0b3350 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4027830 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.010+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac0b5540 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.014+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac04be40 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.014+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f79a4024070 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.018+0000 7f79a8ff9700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 0x7f7990062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.018+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f79a407f0f0 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.018+0000 7f79b0a0a700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 0x7f7990062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.018+0000 7f79b0a0a700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 0x7f7990062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f79a0004f70 tx=0x7f79a0007d40 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.022+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f79a404f430 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.254+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f79ac0cb560 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.266+0000 7f79a8ff9700 1 -- 172.21.15.50:0/889372921 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f79a4052a10 con 0x7f79ac0ce2d0 2023-12-19T14:52:14.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 msgr2=0x7f7990062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 0x7f7990062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f79a0004f70 tx=0x7f79a0007d40 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 msgr2=0x7f79ac0c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f79ac0634d0 tx=0x7f79a4019000 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 shutdown_connections 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79ac0cc040 0x7f79ac14f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f79900606a0 0x7f7990062b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79ac0ce2d0 0x7f79ac0c77d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 --2- 172.21.15.50:0/889372921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79ac0caa10 0x7f79ac151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 >> 172.21.15.50:0/889372921 conn(0x7f79ac067bb0 msgr2=0x7f79ac0c6290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 shutdown_connections 2023-12-19T14:52:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.270+0000 7f79b1a0c700 1 -- 172.21.15.50:0/889372921 wait complete. 2023-12-19T14:52:14.276 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:52:14.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:52:14.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.462+0000 7f44159cc700 1 Processor -- start 2023-12-19T14:52:14.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.462+0000 7f44159cc700 1 -- start start 2023-12-19T14:52:14.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44159cc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f441006aa00 0x7f441006d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44159cc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441006c7c0 con 0x7f44100b1410 2023-12-19T14:52:14.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44159cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441006cd80 con 0x7f441006a580 2023-12-19T14:52:14.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44159cc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441006b8f0 con 0x7f441006aa00 2023-12-19T14:52:14.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44149ca700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f44100b1410 0x7f44100ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57838/0 (socket says 172.21.15.50:57838) 2023-12-19T14:52:14.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44149ca700 1 -- 172.21.15.50:0/656593972 learned_addr learned my addr 172.21.15.50:0/656593972 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:14.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44151cb700 1 --2- 172.21.15.50:0/656593972 >> v2:172.21.15.125:3300/0 conn(0x7f441006aa00 0x7f441006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44077fe700 1 -- 172.21.15.50:0/656593972 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261635514 0 0) 0x7f441006c7c0 con 0x7f44100b1410 2023-12-19T14:52:14.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44077fe700 1 -- 172.21.15.50:0/656593972 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f8003570 con 0x7f44100b1410 2023-12-19T14:52:14.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44151cb700 1 -- 172.21.15.50:0/656593972 >> v1:172.21.15.125:6789/0 conn(0x7f441006a580 legacy=0x7f441006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44151cb700 1 -- 172.21.15.50:0/656593972 >> v1:172.21.15.50:6789/0 conn(0x7f44100b1410 legacy=0x7f44100ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.466+0000 7f44151cb700 1 -- 172.21.15.50:0/656593972 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100ac1e0 con 0x7f441006aa00 2023-12-19T14:52:14.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44151cb700 1 --2- 172.21.15.50:0/656593972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f441006aa00 0x7f441006d920 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4408009e60 tx=0x7f4408009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=929c433c5795c6eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44077fe700 1 -- 172.21.15.50:0/656593972 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800e070 con 0x7f441006aa00 2023-12-19T14:52:14.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44077fe700 1 -- 172.21.15.50:0/656593972 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440800bb60 con 0x7f441006aa00 2023-12-19T14:52:14.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44077fe700 1 -- 172.21.15.50:0/656593972 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408012430 con 0x7f441006aa00 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 -- 172.21.15.50:0/656593972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f441006aa00 msgr2=0x7f441006d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 --2- 172.21.15.50:0/656593972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f441006aa00 0x7f441006d920 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4408009e60 tx=0x7f4408009780 comp rx=0 tx=0).stop 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 -- 172.21.15.50:0/656593972 shutdown_connections 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 --2- 172.21.15.50:0/656593972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f441006aa00 0x7f441006d920 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 -- 172.21.15.50:0/656593972 >> 172.21.15.50:0/656593972 conn(0x7f4410067bb0 msgr2=0x7f44100cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 -- 172.21.15.50:0/656593972 shutdown_connections 2023-12-19T14:52:14.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.470+0000 7f44159cc700 1 -- 172.21.15.50:0/656593972 wait complete. 2023-12-19T14:52:14.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 Processor -- start 2023-12-19T14:52:14.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 -- start start 2023-12-19T14:52:14.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f441006aa00 0x7f44101704f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44100b1410 0x7f441016ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441016f450 con 0x7f441006a580 2023-12-19T14:52:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441016f730 con 0x7f44100b1410 2023-12-19T14:52:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44159cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441016fa10 con 0x7f441006aa00 2023-12-19T14:52:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33660/0 (socket says 172.21.15.50:33660) 2023-12-19T14:52:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 -- 172.21.15.50:0/2118170540 learned_addr learned my addr 172.21.15.50:0/2118170540 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f4407fff700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f441006aa00 0x7f44101704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44151cb700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44100b1410 0x7f441016ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 -- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f441006aa00 msgr2=0x7f44101704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f441006aa00 0x7f44101704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 -- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44100b1410 msgr2=0x7f441016ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44100b1410 0x7f441016ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 -- 172.21.15.50:0/2118170540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408009a20 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f44149ca700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f440c00c590 tx=0x7f440c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.474+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c005b30 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c005cb0 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441016fcf0 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c007320 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4410154b80 con 0x7f441006a580 2023-12-19T14:52:14.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441004be40 con 0x7f441006a580 2023-12-19T14:52:14.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.478+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f440c01b3b0 con 0x7f441006a580 2023-12-19T14:52:14.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.482+0000 7f4404ff9700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 0x7f43fc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:14.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.482+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f440c07c0c0 con 0x7f441006a580 2023-12-19T14:52:14.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.482+0000 7f4407fff700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 0x7f43fc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:14.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.482+0000 7f4407fff700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 0x7f43fc062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f44100c78c0 tx=0x7f4400009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:14.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.486+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f440c04c3c0 con 0x7f441006a580 2023-12-19T14:52:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.690+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44100cc340 con 0x7f441006a580 2023-12-19T14:52:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.690+0000 7f4404ff9700 1 -- 172.21.15.50:0/2118170540 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f440c017090 con 0x7f441006a580 2023-12-19T14:52:14.694 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:14.694 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:52:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 msgr2=0x7f43fc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 0x7f43fc062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f44100c78c0 tx=0x7f4400009040 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 msgr2=0x7f4410154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f440c00c590 tx=0x7f440c007e00 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 shutdown_connections 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44100b1410 0x7f441016ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43fc060670 0x7f43fc062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f441006aa00 0x7f44101704f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 --2- 172.21.15.50:0/2118170540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f441006a580 0x7f4410154640 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 >> 172.21.15.50:0/2118170540 conn(0x7f4410067bb0 msgr2=0x7f4410156480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 shutdown_connections 2023-12-19T14:52:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:14.694+0000 7f44159cc700 1 -- 172.21.15.50:0/2118170540 wait complete. 2023-12-19T14:52:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:52:15.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:52:08.469774+0000', 'modified': '2023-12-19T14:52:12.052742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5964}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5964': {'gid': 5964, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/1875488305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1875488305}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1875488305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:52:15.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:52:15.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T14:52:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.294+0000 7f478c86e700 1 Processor -- start 2023-12-19T14:52:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.294+0000 7f478c86e700 1 -- start start 2023-12-19T14:52:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478c86e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f478806dc00 0x7f478806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478c86e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478806b0d0 con 0x7f4788158cc0 2023-12-19T14:52:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478c86e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880b4790 con 0x7f478806baa0 2023-12-19T14:52:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478c86e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478806bec0 con 0x7f478806dc00 2023-12-19T14:52:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f477ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f478806dc00 0x7f478806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478659c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4788158cc0 0x7f478806d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57842/0 (socket says 172.21.15.50:57842) 2023-12-19T14:52:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.298+0000 7f478659c700 1 -- 172.21.15.50:0/1563681327 learned_addr learned my addr 172.21.15.50:0/1563681327 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f4785d9b700 1 -- 172.21.15.50:0/1563681327 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933099482 0 0) 0x7f478806b0d0 con 0x7f4788158cc0 2023-12-19T14:52:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f4785d9b700 1 -- 172.21.15.50:0/1563681327 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003570 con 0x7f4788158cc0 2023-12-19T14:52:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f477ffff700 1 -- 172.21.15.50:0/1563681327 >> v1:172.21.15.125:6789/0 conn(0x7f478806baa0 legacy=0x7f47880b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f477ffff700 1 -- 172.21.15.50:0/1563681327 >> v1:172.21.15.50:6789/0 conn(0x7f4788158cc0 legacy=0x7f478806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f477ffff700 1 -- 172.21.15.50:0/1563681327 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478806a7e0 con 0x7f478806dc00 2023-12-19T14:52:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f477ffff700 1 --2- 172.21.15.50:0/1563681327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f478806dc00 0x7f478806b560 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f47700098e0 tx=0x7f47700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bdbf07419b1b28 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f4785d9b700 1 -- 172.21.15.50:0/1563681327 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000b640 con 0x7f478806dc00 2023-12-19T14:52:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f4785d9b700 1 -- 172.21.15.50:0/1563681327 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000b7a0 con 0x7f478806dc00 2023-12-19T14:52:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f4785d9b700 1 -- 172.21.15.50:0/1563681327 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000bb20 con 0x7f478806dc00 2023-12-19T14:52:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 -- 172.21.15.50:0/1563681327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f478806dc00 msgr2=0x7f478806b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 --2- 172.21.15.50:0/1563681327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f478806dc00 0x7f478806b560 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f47700098e0 tx=0x7f47700091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 -- 172.21.15.50:0/1563681327 shutdown_connections 2023-12-19T14:52:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 --2- 172.21.15.50:0/1563681327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f478806dc00 0x7f478806b560 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 -- 172.21.15.50:0/1563681327 >> 172.21.15.50:0/1563681327 conn(0x7f4788067bb0 msgr2=0x7f47880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 -- 172.21.15.50:0/1563681327 shutdown_connections 2023-12-19T14:52:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.302+0000 7f478c86e700 1 -- 172.21.15.50:0/1563681327 wait complete. 2023-12-19T14:52:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.306+0000 7f478c86e700 1 Processor -- start 2023-12-19T14:52:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.306+0000 7f478c86e700 1 -- start start 2023-12-19T14:52:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f478806baa0 0x7f47881566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f478806dc00 0x7f47880b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880aff90 con 0x7f4788158cc0 2023-12-19T14:52:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880ae440 con 0x7f478806dc00 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880af030 con 0x7f478806baa0 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33674/0 (socket says 172.21.15.50:33674) 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 -- 172.21.15.50:0/209904102 learned_addr learned my addr 172.21.15.50:0/209904102 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f477ffff700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f478806dc00 0x7f47880b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478659c700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f478806baa0 0x7f47881566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 -- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f478806baa0 msgr2=0x7f47881566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f478806baa0 0x7f47881566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 -- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f478806dc00 msgr2=0x7f47880b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f478806dc00 0x7f47880b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 -- 172.21.15.50:0/209904102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009420 con 0x7f4788158cc0 2023-12-19T14:52:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f4786d9d700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f478000c870 tx=0x7f4780001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f477e7fc700 1 -- 172.21.15.50:0/209904102 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780012630 con 0x7f4788158cc0 2023-12-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f477e7fc700 1 -- 172.21.15.50:0/209904102 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47800127b0 con 0x7f4788158cc0 2023-12-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f477e7fc700 1 -- 172.21.15.50:0/209904102 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780012b70 con 0x7f4788158cc0 2023-12-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.310+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47880af310 con 0x7f4788158cc0 2023-12-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.314+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47880af8d0 con 0x7f4788158cc0 2023-12-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.314+0000 7f477e7fc700 1 -- 172.21.15.50:0/209904102 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4780003030 con 0x7f4788158cc0 2023-12-19T14:52:15.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.318+0000 7f477e7fc700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 0x7f4768062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.318+0000 7f477e7fc700 1 -- 172.21.15.50:0/209904102 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f478007c960 con 0x7f4788158cc0 2023-12-19T14:52:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.318+0000 7f478eb20740 1 Processor -- start 2023-12-19T14:52:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.318+0000 7f478eb20740 1 -- start start 2023-12-19T14:52:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.318+0000 7f478659c700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 0x7f4768062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 0x7f4768062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4778007fd0 tx=0x7f4778009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1563440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1563980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1490f30 con 0x12e5ae0 2023-12-19T14:52:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x148ced0 con 0x12e5710 2023-12-19T14:52:15.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x14530b0 con 0x1330790 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f477ffff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1563440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33690/0 (socket says 172.21.15.50:33690) 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 -- 172.21.15.50:0/2630601180 learned_addr learned my addr 172.21.15.50:0/2630601180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f4786d9d700 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1563980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 -- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 msgr2=0x1563980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1563980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 -- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 msgr2=0x1563440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1563440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 -- 172.21.15.50:0/2630601180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x13319e0 con 0x12e5ae0 2023-12-19T14:52:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478659c700 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f477800f980 tx=0x7f4778015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70876b9b5337691f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f47677fe700 1 -- 172.21.15.50:0/2630601180 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800fe40 con 0x12e5ae0 2023-12-19T14:52:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f47677fe700 1 -- 172.21.15.50:0/2630601180 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778004a40 con 0x12e5ae0 2023-12-19T14:52:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f47677fe700 1 -- 172.21.15.50:0/2630601180 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778004dc0 con 0x12e5ae0 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- 172.21.15.50:0/2630601180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 msgr2=0x1562f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f477800f980 tx=0x7f4778015000 comp rx=0 tx=0).stop 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- 172.21.15.50:0/2630601180 shutdown_connections 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1563440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1563980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 --2- 172.21.15.50:0/2630601180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1562f00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.322+0000 7f478eb20740 1 -- 172.21.15.50:0/2630601180 >> 172.21.15.50:0/2630601180 conn(0x12e5eb0 msgr2=0x151f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- 172.21.15.50:0/2630601180 shutdown_connections 2023-12-19T14:52:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- 172.21.15.50:0/2630601180 wait complete. 2023-12-19T14:52:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- start start 2023-12-19T14:52:15.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 Processor -- start 2023-12-19T14:52:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1567510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1548cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1452d50 con 0x12e5ae0 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x142b970 con 0x12e5710 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1493400 con 0x1330790 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.326+0000 7f478eb20740 10 client.0 _ll_register_callbacks cb 0x14d66a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33706/0 (socket says 172.21.15.50:33706) 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 -- 172.21.15.50:0/1962195832 learned_addr learned my addr 172.21.15.50:0/1962195832 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478659c700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1567510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4786d9d700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1548cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 -- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 msgr2=0x1548cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1548cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 -- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 msgr2=0x1567510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1567510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 -- 172.21.15.50:0/1962195832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4774003570 con 0x12e5ae0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4786d9d700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1548cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478659c700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1567510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f477ffff700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f478806eb60 tx=0x7f4770017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622496ce17c0ae76 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770021050 con 0x12e5ae0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f477001a630 con 0x12e5ae0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x142bcc0 con 0x12e5ae0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f477000b1a0 con 0x12e5ae0 2023-12-19T14:52:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 10 client.6114 fetch_fsmap learned FSMap version 25 2023-12-19T14:52:15.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x145b370 con 0x12e5ae0 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x142bcc0 con 0x12e5ae0 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f477001aa00 con 0x12e5ae0 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 10 client.6114 fetch_fsmap finished waiting for FSMap version 25 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 10 client.6114 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 20 client.6114 populate_metadata read hostname 'smithi050' 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x154a310 0x15726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 4 client.6114 mds_command: new command op to 5964 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.330+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 --> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x142f3e0 con 0x154a310 2023-12-19T14:52:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478659c700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x154a310 0x15726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478659c700 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x154a310 0x15726b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22116b34c2e580eb server_cookie=9386b05b112fee2c in_seq=0 out_seq=0 2023-12-19T14:52:15.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f4764ff9700 10 client.6114 ms_handle_connect on v2:172.21.15.50:6834/1875488305 2023-12-19T14:52:15.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f4764ff9700 1 -- 172.21.15.50:0/1962195832 <== mds.0 v2:172.21.15.50:6834/1875488305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x142f3e0 con 0x154a310 2023-12-19T14:52:15.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f4764ff9700 10 client.6114 handle_command_reply: tid=0 2023-12-19T14:52:15.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 client.6114 shutdown 2023-12-19T14:52:15.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 msgr2=0x1567a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f478806eb60 tx=0x7f4770017000 comp rx=0 tx=0).stop 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 shutdown_connections 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x154a310 0x15726b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1330790 0x1548cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1567a50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f4764ff9700 0 client.6114 ms_handle_reset on v2:172.21.15.50:6834/1875488305 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 --2- 172.21.15.50:0/1962195832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5710 0x1567510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 >> 172.21.15.50:0/1962195832 conn(0x12e5eb0 msgr2=0x1517f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 shutdown_connections 2023-12-19T14:52:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.334+0000 7f478eb20740 1 -- 172.21.15.50:0/1962195832 wait complete. 2023-12-19T14:52:15.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.338+0000 7f478eb20740 20 client.6114 trim_cache size 0 max 16384 2023-12-19T14:52:15.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 Processor -- start 2023-12-19T14:52:15.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 -- start start 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x151d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5ae0 0x151da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x135e7c0 con 0x1330790 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1460040 con 0x12e5ae0 2023-12-19T14:52:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1488b50 con 0x12e5710 2023-12-19T14:52:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33720/0 (socket says 172.21.15.50:33720) 2023-12-19T14:52:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 -- 172.21.15.50:0/579040586 learned_addr learned my addr 172.21.15.50:0/579040586 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f477ffff700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5ae0 0x151da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478659c700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x151d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 -- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 msgr2=0x151d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x151d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 -- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5ae0 msgr2=0x151da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5ae0 0x151da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 -- 172.21.15.50:0/579040586 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1441d80 con 0x1330790 2023-12-19T14:52:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f478659c700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x151d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.358+0000 7f4786d9d700 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4780050620 tx=0x7f4780053040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e48b12113ac256 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f4767fff700 1 -- 172.21.15.50:0/579040586 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780050280 con 0x1330790 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f4767fff700 1 -- 172.21.15.50:0/579040586 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478004ea60 con 0x1330790 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f4767fff700 1 -- 172.21.15.50:0/579040586 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478004d030 con 0x1330790 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- 172.21.15.50:0/579040586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 msgr2=0x151df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4780050620 tx=0x7f4780053040 comp rx=0 tx=0).stop 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- 172.21.15.50:0/579040586 shutdown_connections 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e5ae0 0x151da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1330790 0x151df80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- 172.21.15.50:0/579040586 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x151d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- 172.21.15.50:0/579040586 >> 172.21.15.50:0/579040586 conn(0x12e5eb0 msgr2=0x1517f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- 172.21.15.50:0/579040586 shutdown_connections 2023-12-19T14:52:15.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- 172.21.15.50:0/579040586 wait complete. 2023-12-19T14:52:15.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- start start 2023-12-19T14:52:15.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 Processor -- start 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x15c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1330790 0x1543c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x14909d0 con 0x12e5ae0 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1491ef0 con 0x1330790 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x12e3430 con 0x12e5710 2023-12-19T14:52:15.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.362+0000 7f478eb20740 10 client.0 _ll_register_callbacks cb 0x12b19d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:52:15.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33730/0 (socket says 172.21.15.50:33730) 2023-12-19T14:52:15.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478659c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x15c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 -- 172.21.15.50:0/1558346553 learned_addr learned my addr 172.21.15.50:0/1558346553 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4786d9d700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1330790 0x1543c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 -- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 msgr2=0x15c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x15c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 -- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1330790 msgr2=0x1543c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1330790 0x1543c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 -- 172.21.15.50:0/1558346553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47880af8d0 con 0x12e5ae0 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f477ffff700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f477000bf60 tx=0x7f477001e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c163a25262e0388f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:52:15.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770021050 con 0x12e5ae0 2023-12-19T14:52:15.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f477001a630 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4778004a40 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f477000b640 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 10 client.6120 fetch_fsmap learned FSMap version 25 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x1436130 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4778004a40 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 25) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f47700274b0 con 0x12e5ae0 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 10 client.6120 fetch_fsmap finished waiting for FSMap version 25 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 10 client.6120 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 20 client.6120 populate_metadata read hostname 'smithi050' 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x1544600 0x15449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 4 client.6120 mds_command: new command op to 5964 tid=0[{"prefix": "status"}] 2023-12-19T14:52:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 --> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1461410 con 0x1544600 2023-12-19T14:52:15.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478659c700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x1544600 0x15449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f478659c700 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x1544600 0x15449d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21096a6ede2ff4a0 server_cookie=7eb13425ccea4f2f in_seq=0 out_seq=0 2023-12-19T14:52:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.366+0000 7f4764ff9700 10 client.6120 ms_handle_connect on v2:172.21.15.50:6834/1875488305 2023-12-19T14:52:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f4764ff9700 1 -- 172.21.15.50:0/1558346553 <== mds.0 v2:172.21.15.50:6834/1875488305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x1461410 con 0x1544600 2023-12-19T14:52:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f4764ff9700 10 client.6120 handle_command_reply: tid=0 2023-12-19T14:52:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 client.6120 shutdown 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 msgr2=0x1543730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f477000bf60 tx=0x7f477001e070 comp rx=0 tx=0).stop 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 shutdown_connections 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:6834/1875488305,v1:172.21.15.50:6835/1875488305] conn(0x1544600 0x15449d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1330790 0x1543c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e5ae0 0x1543730 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f4764ff9700 0 client.6120 ms_handle_reset on v2:172.21.15.50:6834/1875488305 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 --2- 172.21.15.50:0/1558346553 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e5710 0x15c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 >> 172.21.15.50:0/1558346553 conn(0x12e5eb0 msgr2=0x151a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 shutdown_connections 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 1 -- 172.21.15.50:0/1558346553 wait complete. 2023-12-19T14:52:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478eb20740 20 client.6120 trim_cache size 0 max 16384 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 5964, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3426612760000003, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 25, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 68, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 68, 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.254279943 2023-12-19T14:52:15.375 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:52:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 msgr2=0x7f4768062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 0x7f4768062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4778007fd0 tx=0x7f4778009040 comp rx=0 tx=0).stop 2023-12-19T14:52:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 msgr2=0x7f47880ad640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f478000c870 tx=0x7f4780001670 comp rx=0 tx=0).stop 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 shutdown_connections 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f478806dc00 0x7f47880b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47680606a0 0x7f4768062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4788158cc0 0x7f47880ad640 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 --2- 172.21.15.50:0/209904102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f478806baa0 0x7f47881566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 >> 172.21.15.50:0/209904102 conn(0x7f4788067bb0 msgr2=0x7f47880b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 shutdown_connections 2023-12-19T14:52:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.370+0000 7f478c86e700 1 -- 172.21.15.50:0/209904102 wait complete. 2023-12-19T14:52:15.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:52:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.566+0000 7f9dd5345700 1 Processor -- start 2023-12-19T14:52:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.566+0000 7f9dd5345700 1 -- start start 2023-12-19T14:52:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dd5345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9dd006de60 0x7f9dd0158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dd5345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd006ac30 con 0x7f9dd0158c10 2023-12-19T14:52:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dd5345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd006d180 con 0x7f9dd006b7d0 2023-12-19T14:52:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dd5345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd006a620 con 0x7f9dd006de60 2023-12-19T14:52:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dceffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9dd0158c10 0x7f9dd006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57852/0 (socket says 172.21.15.50:57852) 2023-12-19T14:52:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dceffd700 1 -- 172.21.15.50:0/709463732 learned_addr learned my addr 172.21.15.50:0/709463732 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:15.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dc67fc700 1 --2- 172.21.15.50:0/709463732 >> v2:172.21.15.125:3300/0 conn(0x7f9dd006de60 0x7f9dd0158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dce7fc700 1 -- 172.21.15.50:0/709463732 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185338128 0 0) 0x7f9dd006ac30 con 0x7f9dd0158c10 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dce7fc700 1 -- 172.21.15.50:0/709463732 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dbc003570 con 0x7f9dd0158c10 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dc67fc700 1 -- 172.21.15.50:0/709463732 >> v1:172.21.15.125:6789/0 conn(0x7f9dd006b7d0 legacy=0x7f9dd00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dc67fc700 1 -- 172.21.15.50:0/709463732 >> v1:172.21.15.50:6789/0 conn(0x7f9dd0158c10 legacy=0x7f9dd006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dc67fc700 1 -- 172.21.15.50:0/709463732 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd006d420 con 0x7f9dd006de60 2023-12-19T14:52:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.570+0000 7f9dc67fc700 1 --2- 172.21.15.50:0/709463732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd006de60 0x7f9dd0158fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9db80098e0 tx=0x7f9db80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbb641e7466bf547 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dce7fc700 1 -- 172.21.15.50:0/709463732 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800b640 con 0x7f9dd006de60 2023-12-19T14:52:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dce7fc700 1 -- 172.21.15.50:0/709463732 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db800b7a0 con 0x7f9dd006de60 2023-12-19T14:52:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dce7fc700 1 -- 172.21.15.50:0/709463732 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800bb40 con 0x7f9dd006de60 2023-12-19T14:52:15.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 -- 172.21.15.50:0/709463732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd006de60 msgr2=0x7f9dd0158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 --2- 172.21.15.50:0/709463732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd006de60 0x7f9dd0158fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9db80098e0 tx=0x7f9db80091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 -- 172.21.15.50:0/709463732 shutdown_connections 2023-12-19T14:52:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 --2- 172.21.15.50:0/709463732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd006de60 0x7f9dd0158fe0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 -- 172.21.15.50:0/709463732 >> 172.21.15.50:0/709463732 conn(0x7f9dd00d0920 msgr2=0x7f9dd00b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 -- 172.21.15.50:0/709463732 shutdown_connections 2023-12-19T14:52:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.574+0000 7f9dd5345700 1 -- 172.21.15.50:0/709463732 wait complete. 2023-12-19T14:52:15.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 Processor -- start 2023-12-19T14:52:15.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 -- start start 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9dd006b7d0 0x7f9dd00c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 0x7f9dd00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0154e90 con 0x7f9dd006de60 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0155cf0 con 0x7f9dd0158c10 2023-12-19T14:52:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dd5345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0153020 con 0x7f9dd006b7d0 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33742/0 (socket says 172.21.15.50:33742) 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 -- 172.21.15.50:0/6663234 learned_addr learned my addr 172.21.15.50:0/6663234 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dceffd700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9dd006b7d0 0x7f9dd00c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dcf7fe700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 0x7f9dd00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 -- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9dd006b7d0 msgr2=0x7f9dd00c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9dd006b7d0 0x7f9dd00c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 -- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 msgr2=0x7f9dd00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 0x7f9dd00b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 -- 172.21.15.50:0/6663234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db8009420 con 0x7f9dd006de60 2023-12-19T14:52:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dcf7fe700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 0x7f9dd00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.578+0000 7f9dc67fc700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009f80 tx=0x7f9db8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800bb20 con 0x7f9dd006de60 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db8011420 con 0x7f9dd006de60 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80117a0 con 0x7f9dd006de60 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0146e00 con 0x7f9dd006de60 2023-12-19T14:52:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd01553c0 con 0x7f9dd006de60 2023-12-19T14:52:15.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd0155e80 con 0x7f9dd006de60 2023-12-19T14:52:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.582+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9db8011b10 con 0x7f9dd006de60 2023-12-19T14:52:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.586+0000 7f9dc77fe700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 0x7f9db0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.586+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f9db807e210 con 0x7f9dd006de60 2023-12-19T14:52:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.586+0000 7f9dceffd700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 0x7f9db0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:15.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.586+0000 7f9dceffd700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 0x7f9db0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0007fd0 tx=0x7f9dc0009040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.594+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9db804e5b0 con 0x7f9dd006de60 2023-12-19T14:52:15.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.798+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9dd004be40 con 0x7f9dd006de60 2023-12-19T14:52:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.798+0000 7f9dc77fe700 1 -- 172.21.15.50:0/6663234 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9db8051bd0 con 0x7f9dd006de60 2023-12-19T14:52:15.803 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:52:15.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 msgr2=0x7f9db0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 0x7f9db0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0007fd0 tx=0x7f9dc0009040 comp rx=0 tx=0).stop 2023-12-19T14:52:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 msgr2=0x7f9dd01468c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009f80 tx=0x7f9db8017000 comp rx=0 tx=0).stop 2023-12-19T14:52:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 shutdown_connections 2023-12-19T14:52:15.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9dd0158c10 0x7f9dd00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9db0060670 0x7f9db0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dd006de60 0x7f9dd01468c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 --2- 172.21.15.50:0/6663234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9dd006b7d0 0x7f9dd00c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 >> 172.21.15.50:0/6663234 conn(0x7f9dd00d0920 msgr2=0x7f9dd006c850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 shutdown_connections 2023-12-19T14:52:15.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:15.802+0000 7f9dd5345700 1 -- 172.21.15.50:0/6663234 wait complete. 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:15.826 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:52:15.827 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:52:15.827 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:52:15.827 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:52:15.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:15.827 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:52:15.850 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:52:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:52:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41614sec preferred_lft 41614sec 2023-12-19T14:52:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:52:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.853 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:52:15.853 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:52:15.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:15.854 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:52:15.938 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:52:15.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:15.941 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:52:16.008 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:52:16.008 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:52:16.008 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:52:16.008 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:52:16.009 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:52:16.009 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:52:16.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:52:16.009 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:52:16.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:16.145 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:52:16.163 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:52:16.163 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:52:16.164 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:52:16.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:52:16.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:52:16.245 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:52:16.246 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:52:16.246 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:16.305 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:52:16.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:52:16.306 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:16.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:52:16.447 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:52:16.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:52:16.527 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:52:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:52:16.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:16.531 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:52:16.598 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:52:16.598 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:52:16.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:52:16.680 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:52:16.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:52:16.763 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:52:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:52:16.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:16.766 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:52:16.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:52:16.778+0000 7ff13c509380 -1 init, newargv = 0x562021d99d70 newargc=15 2023-12-19T14:52:16.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[45886]: starting ceph client 2023-12-19T14:52:16.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[45886]: starting fuse 2023-12-19T14:52:16.796 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:52:16.796 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:52:16.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:52:16.796 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:52:16.796 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:52:16.796 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:52:16.797 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:52:16.798 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:52:16.962 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.45886.asok 2023-12-19T14:52:17.149 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45886.asok 2023-12-19T14:52:17.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:17.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45886.asok status 2023-12-19T14:52:17.340 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:52:17.340 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:52:17.340 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "45886", 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:52:17.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 6132, 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:52:17.342 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6132 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2165341627 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:17.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2165341627 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.6132 192.168.0.1:0/2165341627", 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/2165341627", 2023-12-19T14:52:17.344 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 25, 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 68, 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:52:17.345 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:52:17.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:17.354 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:52:17.355 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:17.370 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:52:17.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:17.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:17.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:52:17.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:52:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.658+0000 7f62cc80c700 1 Processor -- start 2023-12-19T14:52:17.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.658+0000 7f62cc80c700 1 -- start start 2023-12-19T14:52:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62cc80c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62c80c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62cc80c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c806cf00 con 0x7f62c80b1580 2023-12-19T14:52:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62cc80c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80b01d0 con 0x7f62c80b0670 2023-12-19T14:52:17.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62cc80c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80ae7e0 con 0x7f62c8155990 2023-12-19T14:52:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c7fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f62c80b1580 0x7f62c806d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39926/0 (socket says 172.21.15.50:39926) 2023-12-19T14:52:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c7fff700 1 -- 172.21.15.50:0/1054193743 learned_addr learned my addr 172.21.15.50:0/1054193743 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:17.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c77fe700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62c80c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073182641 0 0) 0x7f62c806cf00 con 0x7f62c80b1580 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0003570 con 0x7f62c80b1580 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017504147 0 0) 0x7f62c80b01d0 con 0x7f62c80b0670 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c806cf00 con 0x7f62c80b0670 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381331286 0 0) 0x7f62b0003570 con 0x7f62c80b1580 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 msgr2=0x7f62c80c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62c80c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 >> v1:172.21.15.125:6789/0 conn(0x7f62c80b0670 legacy=0x7f62c80b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c806c8f0 con 0x7f62c80b1580 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f62c00017f0 con 0x7f62c80b1580 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 >> v1:172.21.15.50:6789/0 conn(0x7f62c80b1580 legacy=0x7f62c806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b0004c80 0x7f62b0005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62b00037b0 0x7f62b0003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62b00041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b000de00 con 0x7f62b00037b0 2023-12-19T14:52:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b000df40 con 0x7f62b0004c80 2023-12-19T14:52:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b000e100 con 0x7f62c8155990 2023-12-19T14:52:17.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.662+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62b00037b0 0x7f62b0003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c7fff700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62b00041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c77fe700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b0004c80 0x7f62b0005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 msgr2=0x7f62b00041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8155990 0x7f62b00041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b0004c80 msgr2=0x7f62b0005070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b0004c80 0x7f62b0005070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1054193743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0003570 con 0x7f62b00037b0 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62b00037b0 0x7f62b0003c20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f62bc001940 tx=0x7f62bc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b256b9976e0cf06a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:17.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc001d80 con 0x7f62b00037b0 2023-12-19T14:52:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc00c5d0 con 0x7f62b00037b0 2023-12-19T14:52:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c67fc700 1 -- 172.21.15.50:0/1054193743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00c970 con 0x7f62b00037b0 2023-12-19T14:52:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 -- 172.21.15.50:0/1054193743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62b00037b0 msgr2=0x7f62b0003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1054193743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62b00037b0 0x7f62b0003c20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f62bc001940 tx=0x7f62bc007d30 comp rx=0 tx=0).stop 2023-12-19T14:52:17.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62c7fff700 1 -- 172.21.15.50:0/1054193743 reap_dead start 2023-12-19T14:52:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 -- 172.21.15.50:0/1054193743 shutdown_connections 2023-12-19T14:52:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 -- 172.21.15.50:0/1054193743 >> 172.21.15.50:0/1054193743 conn(0x7f62c8067bb0 msgr2=0x7f62c80add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 -- 172.21.15.50:0/1054193743 shutdown_connections 2023-12-19T14:52:17.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.666+0000 7f62cc80c700 1 -- 172.21.15.50:0/1054193743 wait complete. 2023-12-19T14:52:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 Processor -- start 2023-12-19T14:52:17.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 -- start start 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b00037b0 0x7f62c8168e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8157eb0 0x7f62c8158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8158860 con 0x7f62c815d250 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c815a760 con 0x7f62b00037b0 2023-12-19T14:52:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62cc80c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c815d7a0 con 0x7f62c8157eb0 2023-12-19T14:52:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56018/0 (socket says 172.21.15.50:56018) 2023-12-19T14:52:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1221246221 learned_addr learned my addr 172.21.15.50:0/1221246221 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c77fe700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b00037b0 0x7f62c8168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c7fff700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8157eb0 0x7f62c8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8157eb0 msgr2=0x7f62c8158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8157eb0 0x7f62c8158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b00037b0 msgr2=0x7f62c8168e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b00037b0 0x7f62c8168e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.670+0000 7f62c6ffd700 1 -- 172.21.15.50:0/1221246221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c815da80 con 0x7f62c815d250 2023-12-19T14:52:17.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62c6ffd700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f62b0004790 tx=0x7f62bc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc023070 con 0x7f62c815d250 2023-12-19T14:52:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc001290 con 0x7f62c815d250 2023-12-19T14:52:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc000ed0 con 0x7f62c815d250 2023-12-19T14:52:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c815dd60 con 0x7f62c815d250 2023-12-19T14:52:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c8169370 con 0x7f62c815d250 2023-12-19T14:52:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c815a8d0 con 0x7f62c815d250 2023-12-19T14:52:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.674+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f62bc019070 con 0x7f62c815d250 2023-12-19T14:52:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.678+0000 7f62af7fe700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 0x7f62b4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.678+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f62bc07d640 con 0x7f62c815d250 2023-12-19T14:52:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.678+0000 7f62c77fe700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 0x7f62b4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.678+0000 7f62c77fe700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 0x7f62b4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62b8006f00 tx=0x7f62b80091a0 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.682+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f62bc04d9e0 con 0x7f62c815d250 2023-12-19T14:52:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.878+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f62c804be40 con 0x7f62b4060670 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.882+0000 7f62af7fe700 1 -- 172.21.15.50:0/1221246221 <== mgr.5626 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f62c804be40 con 0x7f62b4060670 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:52:17.886 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 msgr2=0x7f62b4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 0x7f62b4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62b8006f00 tx=0x7f62b80091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 msgr2=0x7f62c80adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f62b0004790 tx=0x7f62bc015000 comp rx=0 tx=0).stop 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 shutdown_connections 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f62b4060670 0x7f62b4062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62c8157eb0 0x7f62c8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62c815d250 0x7f62c80adb30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 --2- 172.21.15.50:0/1221246221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62b00037b0 0x7f62c8168e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 >> 172.21.15.50:0/1221246221 conn(0x7f62c8067bb0 msgr2=0x7f62c80c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 shutdown_connections 2023-12-19T14:52:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:17.886+0000 7f62cc80c700 1 -- 172.21.15.50:0/1221246221 wait complete. 2023-12-19T14:52:17.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:52:18.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.074+0000 7f65e822e700 1 Processor -- start 2023-12-19T14:52:18.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.074+0000 7f65e822e700 1 -- start start 2023-12-19T14:52:18.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e822e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65e015d290 0x7f65e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e822e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e015d660 0x7f65e006d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e822e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e0161eb0 con 0x7f65e006ace0 2023-12-19T14:52:18.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e822e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e00cc5c0 con 0x7f65e015d660 2023-12-19T14:52:18.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e822e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e006d290 con 0x7f65e015d290 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e015d660 0x7f65e006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e015d660 0x7f65e006d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56034/0 (socket says 172.21.15.50:56034) 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 -- 172.21.15.50:0/1530632699 learned_addr learned my addr 172.21.15.50:0/1530632699 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e5fca700 1 --2- 172.21.15.50:0/1530632699 >> v2:172.21.15.125:3300/0 conn(0x7f65e015d290 0x7f65e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 -- 172.21.15.50:0/1530632699 >> v2:172.21.15.125:3300/0 conn(0x7f65e015d290 msgr2=0x7f65e006a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 --2- 172.21.15.50:0/1530632699 >> v2:172.21.15.125:3300/0 conn(0x7f65e015d290 0x7f65e006a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 -- 172.21.15.50:0/1530632699 >> v1:172.21.15.50:6789/0 conn(0x7f65e006ace0 legacy=0x7f65e0160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 -- 172.21.15.50:0/1530632699 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e006c960 con 0x7f65e015d660 2023-12-19T14:52:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e57c9700 1 --2- 172.21.15.50:0/1530632699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e006d720 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f65e00d0980 tx=0x7f65d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333c8f75cceb91f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e4fc8700 1 -- 172.21.15.50:0/1530632699 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000b860 con 0x7f65e015d660 2023-12-19T14:52:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e4fc8700 1 -- 172.21.15.50:0/1530632699 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000b9c0 con 0x7f65e015d660 2023-12-19T14:52:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.078+0000 7f65e4fc8700 1 -- 172.21.15.50:0/1530632699 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000bd60 con 0x7f65e015d660 2023-12-19T14:52:18.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- 172.21.15.50:0/1530632699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 msgr2=0x7f65e006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 --2- 172.21.15.50:0/1530632699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e006d720 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f65e00d0980 tx=0x7f65d0009250 comp rx=0 tx=0).stop 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- 172.21.15.50:0/1530632699 shutdown_connections 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 --2- 172.21.15.50:0/1530632699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e006d720 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 --2- 172.21.15.50:0/1530632699 >> v2:172.21.15.125:3300/0 conn(0x7f65e015d290 0x7f65e006a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- 172.21.15.50:0/1530632699 >> 172.21.15.50:0/1530632699 conn(0x7f65e0067bb0 msgr2=0x7f65e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- 172.21.15.50:0/1530632699 shutdown_connections 2023-12-19T14:52:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- 172.21.15.50:0/1530632699 wait complete. 2023-12-19T14:52:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 Processor -- start 2023-12-19T14:52:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.082+0000 7f65e822e700 1 -- start start 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e006ace0 0x7f65e016df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e015d290 0x7f65e0154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e015ba70 con 0x7f65e015d660 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e015bd50 con 0x7f65e015d290 2023-12-19T14:52:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e015c030 con 0x7f65e006ace0 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56046/0 (socket says 172.21.15.50:56046) 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 -- 172.21.15.50:0/3056582941 learned_addr learned my addr 172.21.15.50:0/3056582941 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e5fca700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e006ace0 0x7f65e016df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e57c9700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e015d290 0x7f65e0154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 -- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e006ace0 msgr2=0x7f65e016df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e006ace0 0x7f65e016df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 -- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e015d290 msgr2=0x7f65e0154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e015d290 0x7f65e0154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 -- 172.21.15.50:0/3056582941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00094f0 con 0x7f65e015d660 2023-12-19T14:52:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e67cb700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c8e0 tx=0x7f65d8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8012630 con 0x7f65e015d660 2023-12-19T14:52:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8012790 con 0x7f65e015d660 2023-12-19T14:52:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e015c310 con 0x7f65e015d660 2023-12-19T14:52:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.086+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8012b50 con 0x7f65e015d660 2023-12-19T14:52:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.090+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e016d6a0 con 0x7f65e015d660 2023-12-19T14:52:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.090+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e015ce10 con 0x7f65e015d660 2023-12-19T14:52:18.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.090+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f65d801a3a0 con 0x7f65e015d660 2023-12-19T14:52:18.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.094+0000 7f65d5ffb700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 0x7f65c4062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:18.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.094+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f65d807cba0 con 0x7f65e015d660 2023-12-19T14:52:18.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.094+0000 7f65e5fca700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 0x7f65c4062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:18.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.094+0000 7f65e5fca700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 0x7f65c4062bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65dc006fd0 tx=0x7f65dc008040 comp rx=0 tx=0).ready entity=mgr.5626 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:18.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.102+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f65d8050a50 con 0x7f65e015d660 2023-12-19T14:52:18.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.314+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f65e016dc10 con 0x7f65e015d660 2023-12-19T14:52:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.886+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f65d804d390 con 0x7f65e015d660 2023-12-19T14:52:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.886+0000 7f65d5ffb700 1 -- 172.21.15.50:0/3056582941 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7f65d80507c0 con 0x7f65e015d660 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 msgr2=0x7f65c4062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 0x7f65c4062bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65dc006fd0 tx=0x7f65dc008040 comp rx=0 tx=0).stop 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 msgr2=0x7f65e00c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f65d800c8e0 tx=0x7f65d8001710 comp rx=0 tx=0).stop 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 shutdown_connections 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e015d290 0x7f65e0154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c4060740 0x7f65c4062bf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e015d660 0x7f65e00c7200 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 --2- 172.21.15.50:0/3056582941 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e006ace0 0x7f65e016df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:18.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 >> 172.21.15.50:0/3056582941 conn(0x7f65e0067bb0 msgr2=0x7f65e0156670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:18.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 shutdown_connections 2023-12-19T14:52:18.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:18.890+0000 7f65e822e700 1 -- 172.21.15.50:0/3056582941 wait complete. 2023-12-19T14:52:18.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:52:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.078+0000 7eff4f48e700 1 Processor -- start 2023-12-19T14:52:19.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.078+0000 7eff4f48e700 1 -- start start 2023-12-19T14:52:19.080 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:19.077+0000 7f8479eef000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:52:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff4813f2d0 0x7eff480ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff480adc60 con 0x7eff4813f6a0 2023-12-19T14:52:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff480ad050 con 0x7eff480c7780 2023-12-19T14:52:19.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff480ad330 con 0x7eff4813f2d0 2023-12-19T14:52:19.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff4813f2d0 0x7eff480ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff4813f2d0 0x7eff480ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56068/0 (socket says 172.21.15.50:56068) 2023-12-19T14:52:19.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 -- 172.21.15.50:0/4244595632 learned_addr learned my addr 172.21.15.50:0/4244595632 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:19.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 -- 172.21.15.50:0/4244595632 >> v1:172.21.15.125:6789/0 conn(0x7eff480c7780 legacy=0x7eff4806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 -- 172.21.15.50:0/4244595632 >> v1:172.21.15.50:6789/0 conn(0x7eff4813f6a0 legacy=0x7eff4815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 -- 172.21.15.50:0/4244595632 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff480ad610 con 0x7eff4813f2d0 2023-12-19T14:52:19.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4e48c700 1 --2- 172.21.15.50:0/4244595632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f2d0 0x7eff480ac9b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7eff440099b0 tx=0x7eff44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78565052f448210 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:19.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4d48a700 1 -- 172.21.15.50:0/4244595632 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400b860 con 0x7eff4813f2d0 2023-12-19T14:52:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4d48a700 1 -- 172.21.15.50:0/4244595632 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4400b9c0 con 0x7eff4813f2d0 2023-12-19T14:52:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4d48a700 1 -- 172.21.15.50:0/4244595632 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400bd60 con 0x7eff4813f2d0 2023-12-19T14:52:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 -- 172.21.15.50:0/4244595632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f2d0 msgr2=0x7eff480ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.082+0000 7eff4f48e700 1 --2- 172.21.15.50:0/4244595632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f2d0 0x7eff480ac9b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7eff440099b0 tx=0x7eff44009250 comp rx=0 tx=0).stop 2023-12-19T14:52:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- 172.21.15.50:0/4244595632 shutdown_connections 2023-12-19T14:52:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 --2- 172.21.15.50:0/4244595632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f2d0 0x7eff480ac9b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- 172.21.15.50:0/4244595632 >> 172.21.15.50:0/4244595632 conn(0x7eff48067b50 msgr2=0x7eff480cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- 172.21.15.50:0/4244595632 shutdown_connections 2023-12-19T14:52:19.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- 172.21.15.50:0/4244595632 wait complete. 2023-12-19T14:52:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 Processor -- start 2023-12-19T14:52:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- start start 2023-12-19T14:52:19.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff480c7780 0x7eff4813b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff4813f2d0 0x7eff4815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.090 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:19.086+0000 7fb6f03d0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4815d040 con 0x7eff4813f6a0 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4815c430 con 0x7eff4813f2d0 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4f48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4815c710 con 0x7eff480c7780 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.086+0000 7eff4ec8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56078/0 (socket says 172.21.15.50:56078) 2023-12-19T14:52:19.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 -- 172.21.15.50:0/3899300811 learned_addr learned my addr 172.21.15.50:0/3899300811 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:19.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4e48c700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff480c7780 0x7eff4813b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4dc8b700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff4813f2d0 0x7eff4815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff480c7780 msgr2=0x7eff4813b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff480c7780 0x7eff4813b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff4813f2d0 msgr2=0x7eff4815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff4813f2d0 0x7eff4815ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 -- 172.21.15.50:0/3899300811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff440094f0 con 0x7eff4813f6a0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4ec8d700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c840 tx=0x7eff40001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40012630 con 0x7eff4813f6a0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff400127b0 con 0x7eff4813f6a0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4815c9f0 con 0x7eff4813f6a0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40012b70 con 0x7eff4813f6a0 2023-12-19T14:52:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4813b7d0 con 0x7eff4813f6a0 2023-12-19T14:52:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.090+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4815d1b0 con 0x7eff4813f6a0 2023-12-19T14:52:19.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.094+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7eff4001a3a0 con 0x7eff4813f6a0 2023-12-19T14:52:19.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.094+0000 7eff3e7fc700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 0x7eff2c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.094+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7eff4007cc20 con 0x7eff4813f6a0 2023-12-19T14:52:19.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.094+0000 7eff4e48c700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 msgr2=0x7eff2c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:19.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.094+0000 7eff4e48c700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 0x7eff2c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:52:19.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.102+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7eff4004cf20 con 0x7eff4813f6a0 2023-12-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.298+0000 7eff4e48c700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 msgr2=0x7eff2c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.298+0000 7eff4e48c700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 0x7eff2c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.322+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7eff4804be40 con 0x7eff4813f6a0 2023-12-19T14:52:19.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.330+0000 7eff3e7fc700 1 -- 172.21.15.50:0/3899300811 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff40050540 con 0x7eff4813f6a0 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 msgr2=0x7eff2c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 0x7eff2c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 msgr2=0x7eff480ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c840 tx=0x7eff40001670 comp rx=0 tx=0).stop 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 shutdown_connections 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff4813f2d0 0x7eff4815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff2c0606c0 0x7eff2c062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff4813f6a0 0x7eff480ae950 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 --2- 172.21.15.50:0/3899300811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff480c7780 0x7eff4813b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 >> 172.21.15.50:0/3899300811 conn(0x7eff48067b50 msgr2=0x7eff4806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:19.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 shutdown_connections 2023-12-19T14:52:19.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.334+0000 7eff4f48e700 1 -- 172.21.15.50:0/3899300811 wait complete. 2023-12-19T14:52:19.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:52:19.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 Processor -- start 2023-12-19T14:52:19.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 -- start start 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa406b2c0 0x7ffaa406b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaa406d210 0x7ffaa406d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffaa40afbd0 0x7ffaa40b0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa406c7c0 con 0x7ffaa406d210 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa406cd80 con 0x7ffaa40afbd0 2023-12-19T14:52:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaaaef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40b0940 con 0x7ffaa406b2c0 2023-12-19T14:52:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaa406d210 0x7ffaa406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaa406d210 0x7ffaa406d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56084/0 (socket says 172.21.15.50:56084) 2023-12-19T14:52:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 -- 172.21.15.50:0/769354434 learned_addr learned my addr 172.21.15.50:0/769354434 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa948d700 1 --2- 172.21.15.50:0/769354434 >> v2:172.21.15.125:3300/0 conn(0x7ffaa40afbd0 0x7ffaa40b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa8c8c700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa406b2c0 0x7ffaa406b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 -- 172.21.15.50:0/769354434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa406b2c0 msgr2=0x7ffaa406b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa406b2c0 0x7ffaa406b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 -- 172.21.15.50:0/769354434 >> v2:172.21.15.125:3300/0 conn(0x7ffaa40afbd0 msgr2=0x7ffaa40b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 --2- 172.21.15.50:0/769354434 >> v2:172.21.15.125:3300/0 conn(0x7ffaa40afbd0 0x7ffaa40b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.526+0000 7ffaa3fff700 1 -- 172.21.15.50:0/769354434 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa40b0c20 con 0x7ffaa406d210 2023-12-19T14:52:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaa3fff700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406d210 0x7ffaa406d680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40cc480 tx=0x7ffa90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b51aaf3f0180d45 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:19.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaa37fe700 1 -- 172.21.15.50:0/769354434 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90009e70 con 0x7ffaa406d210 2023-12-19T14:52:19.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaa37fe700 1 -- 172.21.15.50:0/769354434 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9000b860 con 0x7ffaa406d210 2023-12-19T14:52:19.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaa37fe700 1 -- 172.21.15.50:0/769354434 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9000bc00 con 0x7ffaa406d210 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 -- 172.21.15.50:0/769354434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406d210 msgr2=0x7ffaa406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406d210 0x7ffaa406d680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40cc480 tx=0x7ffa90009250 comp rx=0 tx=0).stop 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 -- 172.21.15.50:0/769354434 shutdown_connections 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406d210 0x7ffaa406d680 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/769354434 >> v2:172.21.15.125:3300/0 conn(0x7ffaa40afbd0 0x7ffaa40b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/769354434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa406b2c0 0x7ffaa406b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 -- 172.21.15.50:0/769354434 >> 172.21.15.50:0/769354434 conn(0x7ffaa40cc580 msgr2=0x7ffaa40cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 -- 172.21.15.50:0/769354434 shutdown_connections 2023-12-19T14:52:19.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.530+0000 7ffaaaef0700 1 -- 172.21.15.50:0/769354434 wait complete. 2023-12-19T14:52:19.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 Processor -- start 2023-12-19T14:52:19.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 -- start start 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaa406d210 0x7ffaa415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa40afbd0 0x7ffaa415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4159dc0 con 0x7ffaa406b2c0 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4159490 con 0x7ffaa406d210 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaaaef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4159770 con 0x7ffaa40afbd0 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56088/0 (socket says 172.21.15.50:56088) 2023-12-19T14:52:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 -- 172.21.15.50:0/968763157 learned_addr learned my addr 172.21.15.50:0/968763157 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa3fff700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaa406d210 0x7ffaa415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 -- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa40afbd0 msgr2=0x7ffaa415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa40afbd0 0x7ffaa415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 -- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaa406d210 msgr2=0x7ffaa415a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaa406d210 0x7ffaa415a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 -- 172.21.15.50:0/968763157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa900094f0 con 0x7ffaa406b2c0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.534+0000 7ffaa8c8c700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800d890 tx=0x7ffa98008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.538+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800b600 con 0x7ffaa406b2c0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.538+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9800b760 con 0x7ffaa406b2c0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.538+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4159a50 con 0x7ffaa406b2c0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.538+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800bae0 con 0x7ffaa406b2c0 2023-12-19T14:52:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.538+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa4154960 con 0x7ffaa406b2c0 2023-12-19T14:52:19.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa4154b80 con 0x7ffaa406b2c0 2023-12-19T14:52:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7ffa9801e460 con 0x7ffaa406b2c0 2023-12-19T14:52:19.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaa0ff9700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 0x7ffa88062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:19.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ffa9807d220 con 0x7ffaa406b2c0 2023-12-19T14:52:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaa3fff700 1 -- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 msgr2=0x7ffa88062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:19.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.542+0000 7ffaa3fff700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 0x7ffa88062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:52:19.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.550+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffa9804d520 con 0x7ffaa406b2c0 2023-12-19T14:52:19.622 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:19.617+0000 7f8479eef000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:52:19.701 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:19.697+0000 7f8479eef000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:52:19.714 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:19.710+0000 7fb6f03d0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:52:19.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.730+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ffaa4071120 con 0x7ffaa406b2c0 2023-12-19T14:52:19.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.738+0000 7ffaa0ff9700 1 -- 172.21.15.50:0/968763157 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7ffa98050b40 con 0x7ffaa406b2c0 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 msgr2=0x7ffa88062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 0x7ffa88062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 msgr2=0x7ffaa415a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800d890 tx=0x7ffa98008040 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 shutdown_connections 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaa406d210 0x7ffaa415a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffa880606c0 0x7ffa88062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaa40afbd0 0x7ffaa415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 --2- 172.21.15.50:0/968763157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaa406b2c0 0x7ffaa415a250 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:19.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 >> 172.21.15.50:0/968763157 conn(0x7ffaa40cc580 msgr2=0x7ffaa40a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:19.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 shutdown_connections 2023-12-19T14:52:19.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:19.742+0000 7ffaaaef0700 1 -- 172.21.15.50:0/968763157 wait complete. 2023-12-19T14:52:19.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:19.767 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T14:52:19.788 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:19.786+0000 7fb6f03d0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:52:19.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T14:52:19.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:19.837+0000 7f8479eef000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:52:19.940 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:19.938+0000 7fb6f03d0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:52:20.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 Processor -- start 2023-12-19T14:52:20.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 -- start start 2023-12-19T14:52:20.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:20.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bdc06da90 con 0x7f1bdc0addc0 2023-12-19T14:52:20.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bdc06a620 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be128e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc06dd30 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be0a8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be0a8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56102/0 (socket says 172.21.15.50:56102) 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.006+0000 7f1be0a8d700 1 -- 172.21.15.50:0/4044084159 learned_addr learned my addr 172.21.15.50:0/4044084159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be0a8d700 1 -- 172.21.15.50:0/4044084159 >> v1:172.21.15.125:6789/0 conn(0x7f1bdc0ad9f0 legacy=0x7f1bdc06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be0a8d700 1 -- 172.21.15.50:0/4044084159 >> v1:172.21.15.50:6789/0 conn(0x7f1bdc0addc0 legacy=0x7f1bdc0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be0a8d700 1 -- 172.21.15.50:0/4044084159 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc06a8c0 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be0a8d700 1 --2- 172.21.15.50:0/4044084159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009f20 tx=0x7f1bd4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41fa3b521eda6dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:20.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1bdb7fe700 1 -- 172.21.15.50:0/4044084159 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd400e070 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1bdb7fe700 1 -- 172.21.15.50:0/4044084159 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd400ba80 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1bdb7fe700 1 -- 172.21.15.50:0/4044084159 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40092e0 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 -- 172.21.15.50:0/4044084159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0b0a50 msgr2=0x7f1bdc0c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:20.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 --2- 172.21.15.50:0/4044084159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009f20 tx=0x7f1bd4009840 comp rx=0 tx=0).stop 2023-12-19T14:52:20.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 -- 172.21.15.50:0/4044084159 shutdown_connections 2023-12-19T14:52:20.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 --2- 172.21.15.50:0/4044084159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc0c8db0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:20.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 -- 172.21.15.50:0/4044084159 >> 172.21.15.50:0/4044084159 conn(0x7f1bdc067bb0 msgr2=0x7f1bdc0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:20.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 -- 172.21.15.50:0/4044084159 shutdown_connections 2023-12-19T14:52:20.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.010+0000 7f1be128e700 1 -- 172.21.15.50:0/4044084159 wait complete. 2023-12-19T14:52:20.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.014+0000 7f1be128e700 1 Processor -- start 2023-12-19T14:52:20.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.014+0000 7f1be128e700 1 -- start start 2023-12-19T14:52:20.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 0x7f1bdc13b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc1579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc13bd50 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0a85b0 con 0x7f1bdc0b0a50 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc1581e0 con 0x7f1bdc0addc0 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56106/0 (socket says 172.21.15.50:56106) 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 -- 172.21.15.50:0/1360144072 learned_addr learned my addr 172.21.15.50:0/1360144072 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 0x7f1bdc13b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:20.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be0a8d700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc1579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:20.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 msgr2=0x7f1bdc13b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:20.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 0x7f1bdc13b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bdc0b0a50 msgr2=0x7f1bdc1579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc1579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4009ae0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 0x7f1bdc13b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bdbfff700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00c590 tx=0x7f1bcc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc011590 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc0116f0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc1584c0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc011ab0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc157ef0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.018+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc04be40 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.022+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f1bcc01b3b0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.022+0000 7f1bd8ff9700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:20.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.022+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1bcc07c050 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.022+0000 7f1bd37fe700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:20.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.022+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:52:20.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.026+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1bcc04c3f0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:20.096 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.093+0000 7f8479eef000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:52:20.167 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.165+0000 7f8479eef000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:52:20.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.218+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bdc0a86f0 con 0x7f1bbc0606c0 2023-12-19T14:52:20.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.226+0000 7f1bd37fe700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:20.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.226+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:52:20.228 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:20.226+0000 7fb6f03d0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:52:20.303 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:20.302+0000 7fb6f03d0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:52:20.381 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.377+0000 7f8479eef000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:52:20.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.457+0000 7f8479eef000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:52:20.540 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:20.538+0000 7fb6f03d0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:52:20.544 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.541+0000 7f8479eef000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:52:20.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.630+0000 7f1bd37fe700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:20.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:20.630+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:52:20.632 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:20.630+0000 7fb6f03d0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:52:20.723 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:20.718+0000 7fb6f03d0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:52:20.916 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:20.913+0000 7f8479eef000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:52:21.075 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.073+0000 7f8479eef000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:52:21.145 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:21.142+0000 7fb6f03d0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:52:21.324 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:21.322+0000 7fb6f03d0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:52:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:21.434+0000 7f1bd37fe700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:21.434+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:52:21.453 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.449+0000 7f8479eef000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:52:21.682 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.677+0000 7f8479eef000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:52:21.696 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:21.694+0000 7fb6f03d0000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:52:21.780 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.777+0000 7f8479eef000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:52:21.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.857+0000 7f8479eef000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:52:21.929 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:21.926+0000 7fb6f03d0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:52:21.944 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:21.941+0000 7f8479eef000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:52:22.017 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.014+0000 7fb6f03d0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:52:22.092 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.090+0000 7fb6f03d0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:52:22.175 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.170+0000 7fb6f03d0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:52:22.182 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:22.177+0000 7f8479eef000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:52:22.267 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:22.261+0000 7f8479eef000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:52:22.357 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:22.353+0000 7f8479eef000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:52:22.389 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.386+0000 7fb6f03d0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:52:22.466 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.462+0000 7fb6f03d0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:52:22.556 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.554+0000 7fb6f03d0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:52:22.558 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:22.553+0000 7f8479eef000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:52:22.645 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:22.641+0000 7f8479eef000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:52:22.754 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.750+0000 7fb6f03d0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:52:22.846 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:22.842+0000 7fb6f03d0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:52:23.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:23.038+0000 7f1bd37fe700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:52:23.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:23.038+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:52:23.374 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.369+0000 7f8479eef000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:52:23.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:23.390+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 18) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f1bcc04c7c0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:23.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:23.390+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:52:23.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:23.390+0000 7f1bd8ff9700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:23.577 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843d54a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.577 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843d54a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.577 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843d54a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.577 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843d54a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.577 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843d54a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.578 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843a544700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.578 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843a544700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.578 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843a544700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.578 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843a544700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.578 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:52:23.573+0000 7f843a544700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:52:23.639 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:52:23.634+0000 7fb6f03d0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:52:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.398+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f1bcc07a3f0 con 0x7f1bdc0ad9f0 2023-12-19T14:52:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.402+0000 7f1bd8ff9700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.402+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bbc068af0 con 0x7f1bbc0606c0 2023-12-19T14:52:24.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.406+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.410+0000 7f1bd37fe700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0c8360 tx=0x7f1bc4001210 comp rx=0 tx=0).ready entity=mgr.6016 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.514+0000 7f1bd8ff9700 1 -- 172.21.15.50:0/1360144072 <== mgr.6016 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1bbc068af0 con 0x7f1bbc0606c0 2023-12-19T14:52:24.518 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T14:52:24.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 msgr2=0x7f1bbc062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0c8360 tx=0x7f1bc4001210 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 msgr2=0x7f1bdc159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00c590 tx=0x7f1bcc00c950 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 shutdown_connections 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bdc0b0a50 0x7f1bdc1579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1bbc0606c0 0x7f1bbc062b70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1bdc0addc0 0x7f1bdc13b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 --2- 172.21.15.50:0/1360144072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1bdc0ad9f0 0x7f1bdc159fe0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 >> 172.21.15.50:0/1360144072 conn(0x7f1bdc067bb0 msgr2=0x7f1bdc0cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 shutdown_connections 2023-12-19T14:52:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.518+0000 7f1be128e700 1 -- 172.21.15.50:0/1360144072 wait complete. 2023-12-19T14:52:24.544 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T14:52:24.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 2M 2023-12-19T14:52:24.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.718+0000 7f49d642e700 1 Processor -- start 2023-12-19T14:52:24.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.718+0000 7f49d642e700 1 -- start start 2023-12-19T14:52:24.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d642e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f49d006dc40 0x7f49d00abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d642e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d00aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d642e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00ac8e0 con 0x7f49d0156e70 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d642e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d006d7b0 con 0x7f49d006dc40 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d642e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00ae570 con 0x7f49d00ac480 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f49d006dc40 0x7f49d00abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49cffff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f49d0156e70 0x7f49d006b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:59860/0 (socket says 172.21.15.50:59860) 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f49d006dc40 0x7f49d00abf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56116/0 (socket says 172.21.15.50:56116) 2023-12-19T14:52:24.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49cffff700 1 -- 172.21.15.50:0/132481036 learned_addr learned my addr 172.21.15.50:0/132481036 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:24.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49d49cb700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d00aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 -- 172.21.15.50:0/132481036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 msgr2=0x7f49d00aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:52:24.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d00aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 -- 172.21.15.50:0/132481036 >> v1:172.21.15.125:6789/0 conn(0x7f49d0156e70 legacy=0x7f49d006b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 -- 172.21.15.50:0/132481036 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d006e0e0 con 0x7f49d006dc40 2023-12-19T14:52:24.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49c7fff700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d006dc40 0x7f49d00abf40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f49b80098e0 tx=0x7f49b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9720f9e70f6bd856 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:24.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49cf7fe700 1 -- 172.21.15.50:0/132481036 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8009e80 con 0x7f49d006dc40 2023-12-19T14:52:24.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49cf7fe700 1 -- 172.21.15.50:0/132481036 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b800b860 con 0x7f49d006dc40 2023-12-19T14:52:24.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.722+0000 7f49cf7fe700 1 -- 172.21.15.50:0/132481036 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b800bbe0 con 0x7f49d006dc40 2023-12-19T14:52:24.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 -- 172.21.15.50:0/132481036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d006dc40 msgr2=0x7f49d00abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d006dc40 0x7f49d00abf40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f49b80098e0 tx=0x7f49b80091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 -- 172.21.15.50:0/132481036 shutdown_connections 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d00aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 --2- 172.21.15.50:0/132481036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d006dc40 0x7f49d00abf40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 -- 172.21.15.50:0/132481036 >> 172.21.15.50:0/132481036 conn(0x7f49d0067bb0 msgr2=0x7f49d00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 -- 172.21.15.50:0/132481036 shutdown_connections 2023-12-19T14:52:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.726+0000 7f49d642e700 1 -- 172.21.15.50:0/132481036 wait complete. 2023-12-19T14:52:24.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 Processor -- start 2023-12-19T14:52:24.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 -- start start 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49d006dc40 0x7f49d0149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d0157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0158100 con 0x7f49d0156e70 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00c9cc0 con 0x7f49d006dc40 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d642e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00ca240 con 0x7f49d00ac480 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56126/0 (socket says 172.21.15.50:56126) 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 -- 172.21.15.50:0/2964174159 learned_addr learned my addr 172.21.15.50:0/2964174159 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49cffff700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49d006dc40 0x7f49d0149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49c7fff700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d0157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 -- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 msgr2=0x7f49d0157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d0157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 -- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49d006dc40 msgr2=0x7f49d0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:24.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49d006dc40 0x7f49d0149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:24.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.730+0000 7f49d49cb700 1 -- 172.21.15.50:0/2964174159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b8009420 con 0x7f49d0156e70 2023-12-19T14:52:24.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49d49cb700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009fa0 tx=0x7f49c8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c800ebc0 con 0x7f49d0156e70 2023-12-19T14:52:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c800ed20 con 0x7f49d0156e70 2023-12-19T14:52:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0157ba0 con 0x7f49d0156e70 2023-12-19T14:52:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8014670 con 0x7f49d0156e70 2023-12-19T14:52:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d014a040 con 0x7f49d0156e70 2023-12-19T14:52:24.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.734+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d00ca3b0 con 0x7f49d0156e70 2023-12-19T14:52:24.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.738+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f49c80149e0 con 0x7f49d0156e70 2023-12-19T14:52:24.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.738+0000 7f49ccff9700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 0x7f49b00628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:24.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.738+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6369+0+0 (secure 0 0 0) 0x7f49c807cea0 con 0x7f49d0156e70 2023-12-19T14:52:24.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.742+0000 7f49cffff700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 0x7f49b00628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:24.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.742+0000 7f49cffff700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 0x7f49b00628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49c0001950 tx=0x7f49c0009040 comp rx=0 tx=0).ready entity=mgr.6016 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:24.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.746+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f49c804d3d0 con 0x7f49d0156e70 2023-12-19T14:52:24.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:24.938+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "2M", "target": ["mon-mgr", ""]}) v1 -- 0x7f49d0071120 con 0x7f49b00603f0 2023-12-19T14:52:25.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.038+0000 7f49ccff9700 1 -- 172.21.15.50:0/2964174159 <== mgr.6016 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f49d0071120 con 0x7f49b00603f0 2023-12-19T14:52:25.042 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 msgr2=0x7f49b00628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 0x7f49b00628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49c0001950 tx=0x7f49c0009040 comp rx=0 tx=0).stop 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 msgr2=0x7f49d00ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009fa0 tx=0x7f49c8009900 comp rx=0 tx=0).stop 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 shutdown_connections 2023-12-19T14:52:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f49b00603f0 0x7f49b00628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49d0156e70 0x7f49d00ca6d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49d00ac480 0x7f49d0157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 --2- 172.21.15.50:0/2964174159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49d006dc40 0x7f49d0149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 >> 172.21.15.50:0/2964174159 conn(0x7f49d0067bb0 msgr2=0x7f49d00aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 shutdown_connections 2023-12-19T14:52:25.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.042+0000 7f49d642e700 1 -- 172.21.15.50:0/2964174159 wait complete. 2023-12-19T14:52:25.066 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T14:52:25.066 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-14_53_00 in ~35s... 2023-12-19T14:52:25.066 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-14_54_00 in ~95s... 2023-12-19T14:52:25.067 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M', '2M'], expected_retention: [] 2023-12-19T14:52:25.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T14:52:25.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.246+0000 7f938e5c3700 1 Processor -- start 2023-12-19T14:52:25.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.246+0000 7f938e5c3700 1 -- start start 2023-12-19T14:52:25.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938e5c3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 0x7f93880af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:25.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938e5c3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880cb3f0 con 0x7f93880cac50 2023-12-19T14:52:25.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938e5c3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938806d220 con 0x7f938806af50 2023-12-19T14:52:25.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938e5c3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880cc140 con 0x7f93880aedc0 2023-12-19T14:52:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938ddc2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f93880cac50 0x7f93880af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39978/0 (socket says 172.21.15.50:39978) 2023-12-19T14:52:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938ddc2700 1 -- 172.21.15.50:0/110396297 learned_addr learned my addr 172.21.15.50:0/110396297 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:52:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f938d5c1700 1 --2- 172.21.15.50:0/110396297 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 0x7f93880af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150565488 0 0) 0x7f93880cb3f0 con 0x7f93880cac50 2023-12-19T14:52:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9374003570 con 0x7f93880cac50 2023-12-19T14:52:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041409468 0 0) 0x7f938806d220 con 0x7f938806af50 2023-12-19T14:52:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.250+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93880cb3f0 con 0x7f938806af50 2023-12-19T14:52:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938d5c1700 1 -- 172.21.15.50:0/110396297 >> v1:172.21.15.125:6789/0 conn(0x7f938806af50 legacy=0x7f938806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938d5c1700 1 -- 172.21.15.50:0/110396297 >> v1:172.21.15.50:6789/0 conn(0x7f93880cac50 legacy=0x7f93880af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938d5c1700 1 -- 172.21.15.50:0/110396297 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938806b3a0 con 0x7f93880aedc0 2023-12-19T14:52:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938d5c1700 1 --2- 172.21.15.50:0/110396297 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 0x7f93880af190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9384009910 tx=0x7f93840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dda7ddacf8f90edc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384011630 con 0x7f93880aedc0 2023-12-19T14:52:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384011790 con 0x7f93880aedc0 2023-12-19T14:52:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f937ffff700 1 -- 172.21.15.50:0/110396297 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384011b30 con 0x7f93880aedc0 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 -- 172.21.15.50:0/110396297 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 msgr2=0x7f93880af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 --2- 172.21.15.50:0/110396297 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 0x7f93880af190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9384009910 tx=0x7f93840091a0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 -- 172.21.15.50:0/110396297 shutdown_connections 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 --2- 172.21.15.50:0/110396297 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880aedc0 0x7f93880af190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 -- 172.21.15.50:0/110396297 >> 172.21.15.50:0/110396297 conn(0x7f93880cc760 msgr2=0x7f93880ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 -- 172.21.15.50:0/110396297 shutdown_connections 2023-12-19T14:52:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.254+0000 7f938e5c3700 1 -- 172.21.15.50:0/110396297 wait complete. 2023-12-19T14:52:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 Processor -- start 2023-12-19T14:52:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 -- start start 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93880aedc0 0x7f938816bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 0x7f938806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c3be0 con 0x7f938806af50 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c3ec0 con 0x7f93880aedc0 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938e5c3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c41a0 con 0x7f93880cac50 2023-12-19T14:52:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56142/0 (socket says 172.21.15.50:56142) 2023-12-19T14:52:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 -- 172.21.15.50:0/1639336803 learned_addr learned my addr 172.21.15.50:0/1639336803 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:52:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938ddc2700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 0x7f938806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938cdc0700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93880aedc0 0x7f938816bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 -- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 msgr2=0x7f938806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 0x7f938806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 -- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93880aedc0 msgr2=0x7f938816bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93880aedc0 0x7f938816bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938d5c1700 1 -- 172.21.15.50:0/1639336803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9384009420 con 0x7f938806af50 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.258+0000 7f938ddc2700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 0x7f938806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f938d5c1700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9384009f00 tx=0x7f9384017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938401f070 con 0x7f938806af50 2023-12-19T14:52:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9384000bc0 con 0x7f938806af50 2023-12-19T14:52:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9384025420 con 0x7f938806af50 2023-12-19T14:52:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93880c4480 con 0x7f938806af50 2023-12-19T14:52:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93880b5770 con 0x7f938806af50 2023-12-19T14:52:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93880c4980 con 0x7f938806af50 2023-12-19T14:52:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.262+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f9384023070 con 0x7f938806af50 2023-12-19T14:52:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.266+0000 7f937d7fa700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 0x7f936c062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:52:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.266+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6369+0+0 (secure 0 0 0) 0x7f938407db20 con 0x7f938806af50 2023-12-19T14:52:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.266+0000 7f938cdc0700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 0x7f936c062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:52:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.266+0000 7f938cdc0700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 0x7f936c062970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93880c6cb0 tx=0x7f9378007d50 comp rx=0 tx=0).ready entity=mgr.6016 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:52:25.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.274+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f938404e110 con 0x7f938806af50 2023-12-19T14:52:25.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.410+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 20) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f938404e4e0 con 0x7f938806af50 2023-12-19T14:52:25.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.474+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f938804be40 con 0x7f936c0604c0 2023-12-19T14:52:25.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.490+0000 7f937d7fa700 1 -- 172.21.15.50:0/1639336803 <== mgr.6016 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f938804be40 con 0x7f936c0604c0 2023-12-19T14:52:25.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:52:25.493 INFO:teuthology.orchestra.run.smithi050.stdout:{"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-12-19T14:52:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 msgr2=0x7f936c062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 0x7f936c062970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93880c6cb0 tx=0x7f9378007d50 comp rx=0 tx=0).stop 2023-12-19T14:52:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 msgr2=0x7f93880b5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:52:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9384009f00 tx=0x7f9384017000 comp rx=0 tx=0).stop 2023-12-19T14:52:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 shutdown_connections 2023-12-19T14:52:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93880aedc0 0x7f938816bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f936c0604c0 0x7f936c062970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93880cac50 0x7f938806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 --2- 172.21.15.50:0/1639336803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f938806af50 0x7f93880b5230 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:52:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 >> 172.21.15.50:0/1639336803 conn(0x7f93880cc760 msgr2=0x7f938815e120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:52:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 shutdown_connections 2023-12-19T14:52:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:52:25.494+0000 7f938e5c3700 1 -- 172.21.15.50:0/1639336803 wait complete. 2023-12-19T14:52:25.520 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-12-19T14:52:25.521 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M', '2M'], 'retention': [{}, {}]} 2023-12-19T14:52:25.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:25.521 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:25.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:25.552 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:25.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:26.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:26.751 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:26.787 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:26.787 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:26.787 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:27.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:27.789 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:27.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:27.825 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:27.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:28.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:28.826 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:28.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:28.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:28.863 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:29.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:29.864 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:29.898 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:29.898 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:29.898 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:30.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:30.900 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:30.933 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:30.934 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:30.934 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:31.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:31.935 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:31.970 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:31.970 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:31.970 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:32.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:32.971 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:33.006 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:33.007 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:33.007 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:34.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:34.007 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:34.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:34.043 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:34.043 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:35.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:35.044 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:35.078 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:35.079 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:35.079 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:36.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:36.079 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:36.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:36.117 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:36.117 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:37.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:52:37.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:52:37.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:37.118 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:37.169 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:37.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:37.170 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:38.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:38.171 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:38.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:38.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:38.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:39.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:39.209 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:39.242 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:39.242 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:39.242 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:40.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:40.243 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:40.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:40.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:40.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:41.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:41.277 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:41.314 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:41.314 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:41.314 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:42.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:42.315 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:42.351 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:42.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:42.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:43.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:43.352 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:43.386 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:43.386 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:43.386 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:44.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:44.387 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:44.421 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:44.421 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:44.421 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:45.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:45.423 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:45.457 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:45.457 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:45.457 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:46.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:46.458 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:46.494 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:46.494 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:46.494 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:47.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:47.495 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:47.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:47.529 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:47.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:48.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:48.530 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:48.566 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:48.566 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:48.566 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:49.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:49.567 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:49.602 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:49.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:49.603 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:50.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:50.603 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:50.640 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:50.641 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:50.641 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:51.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:51.641 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:51.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:51.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:51.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:52.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:52.679 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:52.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:52.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:52.714 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:53.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:53.715 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:53.748 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:53.748 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:53.748 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:54.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:54.749 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:54.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:54.786 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:54.786 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:55.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:55.787 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:55.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:55.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:55.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:56.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:56.821 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:56.855 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:56.855 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:56.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:57.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:57.856 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:57.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:57.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:57.892 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:58.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:58.893 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:58.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:58.928 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:58.928 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:52:59.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:52:59.928 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:52:59.963 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:52:59.963 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:52:59.963 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:00.060 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:53:00.056+0000 7f842c528700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:53:00.061 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:53:00.056+0000 7f842c528700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:53:00.061 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:53:00.056+0000 7f842c528700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:53:00.061 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:53:00.056+0000 7f842c528700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:53:00.061 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:53:00.056+0000 7f842c528700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:53:00.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:00.965 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:00.998 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:00.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:00.999 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:00.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:00.999 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-14_53_00_UTC'] 2023-12-19T14:53:00.999 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702997545.0667605, now = 1702997580.9997263, timo = 35 2023-12-19T14:53:02.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:02.000 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:02.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:02.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:02.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:03.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:03.036 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:03.068 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:03.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:03.069 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:03.069 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:04.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:04.070 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:04.100 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:04.102 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:04.102 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:04.102 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:05.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:05.103 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:05.136 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:05.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:05.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:05.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:06.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:06.140 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:06.174 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:06.175 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:06.175 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:06.175 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:07.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:07.176 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:07.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:53:07.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:53:07.205 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:07.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:07.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:07.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:08.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:08.209 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:08.240 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:08.241 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:08.241 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:08.241 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:09.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:09.243 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:09.277 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:09.278 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:09.278 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:09.278 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:10.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:10.279 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:10.311 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:10.313 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:10.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:10.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:11.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:11.314 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:11.350 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:11.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:11.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:11.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:12.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:12.351 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:12.386 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:12.387 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:12.387 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:12.387 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:13.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:13.388 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:13.420 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:13.421 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:13.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:13.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:14.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:14.422 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:14.455 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:14.457 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:14.457 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:14.457 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:15.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:15.458 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:15.491 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:15.493 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:15.493 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:15.493 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:16.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:16.494 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:16.526 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:16.528 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:16.528 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:16.528 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:17.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:17.530 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:17.563 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:17.565 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:17.565 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:17.565 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:18.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:18.566 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:18.599 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:18.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:18.601 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:18.601 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:19.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:19.601 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:19.633 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:19.635 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:19.635 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:19.635 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:20.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:20.636 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:20.664 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:20.665 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:20.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:20.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:21.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:21.669 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:21.703 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:21.704 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:21.704 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:21.704 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:22.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:22.705 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:22.739 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:22.740 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:22.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:22.741 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:23.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:23.742 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:23.776 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:23.777 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:23.777 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:23.778 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:24.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:24.778 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:24.813 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:24.815 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:24.815 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:24.815 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:25.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:25.816 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:25.851 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:25.853 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:25.853 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:25.853 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:26.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:26.854 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:26.889 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:26.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:26.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:26.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:27.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:27.891 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:27.922 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:27.924 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:27.924 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:27.924 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:28.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:28.925 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:28.957 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:28.959 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:28.959 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:28.959 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:29.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:29.960 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:29.991 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:29.993 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:29.993 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:29.993 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:30.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:30.994 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:31.022 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:31.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:31.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:31.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:32.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:32.025 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:32.058 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:32.060 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:32.060 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:32.060 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:33.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:33.061 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:33.093 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:33.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:33.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:33.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:34.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:34.096 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:34.129 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:34.130 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:34.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:34.131 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:35.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:35.132 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:35.165 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:35.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:35.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:35.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:36.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:36.168 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:36.201 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:36.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:36.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:36.203 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:37.204 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:37.204 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:37.241 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:37.241 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:37.242 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:37.242 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:37.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:53:37.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:53:38.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:38.243 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:38.274 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:38.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:38.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:38.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:39.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:39.277 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:39.311 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:39.313 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:39.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:39.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:40.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:40.313 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:40.347 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:40.348 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:40.348 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:40.349 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:41.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:41.350 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:41.384 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:41.386 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:41.386 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:41.386 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:42.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:42.387 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:42.421 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:42.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:42.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:42.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:43.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:43.425 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:43.458 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:43.459 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:43.459 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:44.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:44.460 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:44.492 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:44.494 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:44.494 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:44.494 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:45.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:45.495 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:45.525 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:45.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:45.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:45.527 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:46.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:46.528 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:46.561 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:46.563 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:46.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:46.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:47.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:47.565 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:47.598 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:47.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:47.600 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:47.600 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:48.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:48.602 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:48.635 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:48.636 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:48.636 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:48.636 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:49.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:49.638 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:49.671 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:49.673 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:49.673 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:49.673 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:50.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:50.674 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:50.710 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:50.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:50.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:50.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:51.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:51.713 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:51.746 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:51.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:51.748 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:51.748 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:52.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:52.749 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:52.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:52.786 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:52.786 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:53.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:53.787 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:53.818 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:53.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:53.820 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:53.820 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:54.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:54.822 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:54.855 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:54.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:54.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:54.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:55.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:55.858 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:55.891 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:55.893 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:55.893 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:55.893 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:56.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:56.895 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:56.928 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:56.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:56.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:56.930 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:57.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:57.931 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:57.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:57.964 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:57.964 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:57.964 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:53:58.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:53:58.966 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:53:59.000 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:53:59.001 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:53:59.001 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:53:59.002 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:54:00.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:00.003 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:54:00.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:54:00.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:54:00.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:54:00.034 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:54:01.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:01.035 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:54:01.060 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:54:01.060 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_54_00_UTC 2023-12-19T14:54:01.061 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_54_00_UTC', 'scheduled-2023-12-19-14_53_00_UTC'} 2023-12-19T14:54:01.061 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-14_54_00_UTC'} 2023-12-19T14:54:01.062 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:54:01.062 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-14_54_00_UTC'] 2023-12-19T14:54:01.062 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702997545.0667605, now = 1702997641.062238, timo = 95 2023-12-19T14:54:02.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T14:54:02.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.247+0000 7f18c88ae700 1 Processor -- start 2023-12-19T14:54:02.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.247+0000 7f18c88ae700 1 -- start start 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c88ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18c415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c88ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18c415c760 0x7f18c40b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c88ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40b0ac0 con 0x7f18c40cb130 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c88ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c415cbe0 con 0x7f18c415c760 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c88ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c406b210 con 0x7f18c40cb500 2023-12-19T14:54:02.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f18c40cb130 0x7f18c406a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33428/0 (socket says 172.21.15.50:33428) 2023-12-19T14:54:02.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.251+0000 7f18c259c700 1 -- 172.21.15.50:0/1312435625 learned_addr learned my addr 172.21.15.50:0/1312435625 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:02.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18c415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c2d9d700 1 --2- 172.21.15.50:0/1312435625 >> v2:172.21.15.125:3300/0 conn(0x7f18c415c760 0x7f18c40b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620621463 0 0) 0x7f18c40b0ac0 con 0x7f18c40cb130 2023-12-19T14:54:02.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac003570 con 0x7f18c40cb130 2023-12-19T14:54:02.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782935201 0 0) 0x7f18ac003570 con 0x7f18c40cb130 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 msgr2=0x7f18c415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18c415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 >> v2:172.21.15.125:3300/0 conn(0x7f18c415c760 msgr2=0x7f18c40b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 --2- 172.21.15.50:0/1312435625 >> v2:172.21.15.125:3300/0 conn(0x7f18c415c760 0x7f18c40b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40ca810 con 0x7f18c40cb130 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f18b4001470 con 0x7f18c40cb130 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 >> v1:172.21.15.50:6789/0 conn(0x7f18c40cb130 legacy=0x7f18c406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac003a60 0x7f18ac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18ac004810 0x7f18ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415cbe0 con 0x7f18ac003a60 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c406b210 con 0x7f18ac004810 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac00ba60 con 0x7f18c40cb500 2023-12-19T14:54:02.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c259c700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac003a60 0x7f18ac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18ac004810 0x7f18ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c2d9d700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 -- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 msgr2=0x7f18ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c40cb500 0x7f18ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 -- 172.21.15.50:0/1312435625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac003a60 msgr2=0x7f18ac003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac003a60 0x7f18ac003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 -- 172.21.15.50:0/1312435625 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac003570 con 0x7f18ac004810 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18ac004810 0x7f18ac004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009c30 tx=0x7f18b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ad4b0d817fa4542 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b800e710 con 0x7f18ac004810 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b800e870 con 0x7f18ac004810 2023-12-19T14:54:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.255+0000 7f18c159a700 1 -- 172.21.15.50:0/1312435625 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b800ec30 con 0x7f18ac004810 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18ac004810 msgr2=0x7f18ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1312435625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18ac004810 0x7f18ac004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009c30 tx=0x7f18b80091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c2d9d700 1 -- 172.21.15.50:0/1312435625 reap_dead start 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- 172.21.15.50:0/1312435625 shutdown_connections 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- 172.21.15.50:0/1312435625 >> 172.21.15.50:0/1312435625 conn(0x7f18c4067bb0 msgr2=0x7f18c40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:02.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- 172.21.15.50:0/1312435625 shutdown_connections 2023-12-19T14:54:02.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- 172.21.15.50:0/1312435625 wait complete. 2023-12-19T14:54:02.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 Processor -- start 2023-12-19T14:54:02.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.259+0000 7f18c88ae700 1 -- start start 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18c415b980 0x7f18c4144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c415dee0 0x7f18c415e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415e880 con 0x7f18ac004810 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415b4f0 con 0x7f18c415b980 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40c6a20 con 0x7f18c415dee0 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53236/0 (socket says 172.21.15.50:53236) 2023-12-19T14:54:02.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 -- 172.21.15.50:0/1618821214 learned_addr learned my addr 172.21.15.50:0/1618821214 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18c415b980 0x7f18c4144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c2d9d700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c415dee0 0x7f18c415e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 -- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c415dee0 msgr2=0x7f18c415e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c415dee0 0x7f18c415e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 -- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18c415b980 msgr2=0x7f18c4144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18c415b980 0x7f18c4144590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 -- 172.21.15.50:0/1618821214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c415a8e0 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c259c700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f18b40073c0 tx=0x7f18b4001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4018030 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b400f920 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c415abc0 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b400fcc0 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.263+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c415b180 con 0x7f18ac004810 2023-12-19T14:54:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.267+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4071120 con 0x7f18ac004810 2023-12-19T14:54:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.267+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f18b401b070 con 0x7f18ac004810 2023-12-19T14:54:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.271+0000 7f18b27fc700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 0x7f18a4062c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.271+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f18b4019070 con 0x7f18ac004810 2023-12-19T14:54:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.271+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 0x7f18a4062c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.271+0000 7f18c1d9b700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 0x7f18a4062c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18b80091e0 tx=0x7f18b801a000 comp rx=0 tx=0).ready entity=mgr.6016 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:02.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.275+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18b404ce70 con 0x7f18ac004810 2023-12-19T14:54:02.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.467+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f18c415b630 con 0x7f18a4060770 2023-12-19T14:54:02.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.599+0000 7f18b27fc700 1 -- 172.21.15.50:0/1618821214 <== mgr.6016 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f18b8023040 con 0x7f18a4060770 2023-12-19T14:54:02.601 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path snap_test_dir1 2023-12-19T14:54:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 msgr2=0x7f18a4062c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 0x7f18a4062c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18b80091e0 tx=0x7f18b801a000 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 msgr2=0x7f18c40c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f18b40073c0 tx=0x7f18b4001e90 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 shutdown_connections 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18c415b980 0x7f18c4144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18a4060770 0x7f18a4062c20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18c415dee0 0x7f18c415e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 --2- 172.21.15.50:0/1618821214 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18ac004810 0x7f18c40c64e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:02.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 >> 172.21.15.50:0/1618821214 conn(0x7f18c4067bb0 msgr2=0x7f18c406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 shutdown_connections 2023-12-19T14:54:02.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:02.603+0000 7f18c88ae700 1 -- 172.21.15.50:0/1618821214 wait complete. 2023-12-19T14:54:02.628 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T14:54:02.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:02.628 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:54:02.661 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:54:02.662 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_54_00_UTC 2023-12-19T14:54:02.663 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-14_53_00_UTC 2023-12-19T14:54:02.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:02.663 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-14_53_00_UTC) 2023-12-19T14:54:02.745 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-14_54_00_UTC 2023-12-19T14:54:02.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:02.745 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-14_54_00_UTC) 2023-12-19T14:54:02.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:02.824 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T14:54:02.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:54:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 Processor -- start 2023-12-19T14:54:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 -- start start 2023-12-19T14:54:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbd806c7c0 0x7ffbd806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd806a7a0 con 0x7ffbd80c6d20 2023-12-19T14:54:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd806d4b0 con 0x7ffbd806ac30 2023-12-19T14:54:03.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbdd478700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd806d790 con 0x7ffbd806c7c0 2023-12-19T14:54:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbd6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffbd80c6d20 0x7ffbd80c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33444/0 (socket says 172.21.15.50:33444) 2023-12-19T14:54:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbd6ffd700 1 -- 172.21.15.50:0/2017299667 learned_addr learned my addr 172.21.15.50:0/2017299667 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbd67fc700 1 --2- 172.21.15.50:0/2017299667 >> v2:172.21.15.50:3300/0 conn(0x7ffbd806c7c0 0x7ffbd806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839892208 0 0) 0x7ffbd806a7a0 con 0x7ffbd80c6d20 2023-12-19T14:54:03.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.107+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc0003570 con 0x7ffbd80c6d20 2023-12-19T14:54:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd67fc700 1 -- 172.21.15.50:0/2017299667 >> v1:172.21.15.125:6789/0 conn(0x7ffbd806ac30 legacy=0x7ffbd806b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 415781309 0 0) 0x7ffbc0003570 con 0x7ffbd80c6d20 2023-12-19T14:54:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd67fc700 1 -- 172.21.15.50:0/2017299667 >> v1:172.21.15.50:6789/0 conn(0x7ffbd80c6d20 legacy=0x7ffbd80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd67fc700 1 -- 172.21.15.50:0/2017299667 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd806da70 con 0x7ffbd806c7c0 2023-12-19T14:54:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd67fc700 1 --2- 172.21.15.50:0/2017299667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806c7c0 0x7ffbd806cc60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0098e0 tx=0x7ffbcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71585185f540df3e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8001470 con 0x7ffbd806c7c0 2023-12-19T14:54:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbcc011420 con 0x7ffbd806c7c0 2023-12-19T14:54:03.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbd5ffb700 1 -- 172.21.15.50:0/2017299667 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc0117c0 con 0x7ffbd806c7c0 2023-12-19T14:54:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 -- 172.21.15.50:0/2017299667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806c7c0 msgr2=0x7ffbd806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 --2- 172.21.15.50:0/2017299667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806c7c0 0x7ffbd806cc60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0098e0 tx=0x7ffbcc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 -- 172.21.15.50:0/2017299667 shutdown_connections 2023-12-19T14:54:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 --2- 172.21.15.50:0/2017299667 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806c7c0 0x7ffbd806cc60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 -- 172.21.15.50:0/2017299667 >> 172.21.15.50:0/2017299667 conn(0x7ffbd8067bb0 msgr2=0x7ffbd80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 -- 172.21.15.50:0/2017299667 shutdown_connections 2023-12-19T14:54:03.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.111+0000 7ffbdd478700 1 -- 172.21.15.50:0/2017299667 wait complete. 2023-12-19T14:54:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 Processor -- start 2023-12-19T14:54:03.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 -- start start 2023-12-19T14:54:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:03.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbd806c7c0 0x7ffbd8152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 0x7ffbd80a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd8169600 con 0x7ffbd806ac30 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80c6580 con 0x7ffbd806c7c0 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbdd478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80b1b40 con 0x7ffbd80c6d20 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbd6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbd6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53258/0 (socket says 172.21.15.50:53258) 2023-12-19T14:54:03.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.115+0000 7ffbd6ffd700 1 -- 172.21.15.50:0/577659641 learned_addr learned my addr 172.21.15.50:0/577659641 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd67fc700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbd806c7c0 0x7ffbd8152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd77fe700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 0x7ffbd80a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 -- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 msgr2=0x7ffbd80a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 0x7ffbd80a9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 -- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbd806c7c0 msgr2=0x7ffbd8152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbd806c7c0 0x7ffbd8152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 -- 172.21.15.50:0/577659641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc009420 con 0x7ffbd806ac30 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd77fe700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 0x7ffbd80a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbd6ffd700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800cbc0 tx=0x7ffbc8001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8007360 con 0x7ffbd806ac30 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc80074e0 con 0x7ffbd806ac30 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd80a9700 con 0x7ffbd806ac30 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8004830 con 0x7ffbd806ac30 2023-12-19T14:54:03.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd80b1860 con 0x7ffbd806ac30 2023-12-19T14:54:03.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.119+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd8071120 con 0x7ffbd806ac30 2023-12-19T14:54:03.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.123+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) v1 ==== 80092+0+0 (secure 0 0 0) 0x7ffbc8004be0 con 0x7ffbd806ac30 2023-12-19T14:54:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.123+0000 7ffbbeffd700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 0x7ffbc4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:03.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.123+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6427+0+0 (secure 0 0 0) 0x7ffbc807c3b0 con 0x7ffbd806ac30 2023-12-19T14:54:03.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.123+0000 7ffbd67fc700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 0x7ffbc4062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:03.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.127+0000 7ffbd67fc700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 0x7ffbc4062b70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0091e0 tx=0x7ffbcc016000 comp rx=0 tx=0).ready entity=mgr.6016 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.131+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffbc804c5a0 con 0x7ffbd806ac30 2023-12-19T14:54:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.343+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ffbd80ae6f0 con 0x7ffbd806ac30 2023-12-19T14:54:03.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.935+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7ffbc804c970 con 0x7ffbd806ac30 2023-12-19T14:54:03.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.939+0000 7ffbbeffd700 1 -- 172.21.15.50:0/577659641 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v23) v1 ==== 93+0+0 (secure 0 0 0) 0x7ffbc8050030 con 0x7ffbd806ac30 2023-12-19T14:54:03.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 msgr2=0x7ffbc4062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 0x7ffbc4062b70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0091e0 tx=0x7ffbcc016000 comp rx=0 tx=0).stop 2023-12-19T14:54:03.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 msgr2=0x7ffbd80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:03.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800cbc0 tx=0x7ffbc8001a30 comp rx=0 tx=0).stop 2023-12-19T14:54:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 shutdown_connections 2023-12-19T14:54:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbd806c7c0 0x7ffbd8152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffbc40606c0 0x7ffbc4062b70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbd80c6d20 0x7ffbd80a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 --2- 172.21.15.50:0/577659641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbd806ac30 0x7ffbd80aeba0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:03.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 >> 172.21.15.50:0/577659641 conn(0x7ffbd8067bb0 msgr2=0x7ffbd80cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:03.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 shutdown_connections 2023-12-19T14:54:03.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:03.943+0000 7ffbdd478700 1 -- 172.21.15.50:0/577659641 wait complete. 2023-12-19T14:54:03.966 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:54:03.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:04.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:04.046 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:54:04.046 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:54:04.046 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:54:04.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:04.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[45886]: fuse finished with error 0 and tester_r 0 2023-12-19T14:54:04.092 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:54:04.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:04.092 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:54:04.140 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:04.135+0000 7f138f968000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:04.149 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:04.147+0000 7f75cb7ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:04.150 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:54:04.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:04.150 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:54:04.171 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:54:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:04.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:54:04.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:04.173 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:54:04.208 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:54:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:04.209 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:54:04.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:04.210 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:54:04.234 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:54:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:04.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:04.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.411+0000 7feee6e6e700 1 Processor -- start 2023-12-19T14:54:04.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.411+0000 7feee6e6e700 1 -- start start 2023-12-19T14:54:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee6e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee6e6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feee00cc4c0 0x7feee00c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee6e6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00cc030 con 0x7feee00c6900 2023-12-19T14:54:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee6e6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee00cb420 con 0x7feee00cc4c0 2023-12-19T14:54:04.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee6e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee00cb700 con 0x7feee015d220 2023-12-19T14:54:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedffff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feee00cc4c0 0x7feee00c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:57272/0 (socket says 172.21.15.50:57272) 2023-12-19T14:54:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 -- 172.21.15.50:0/35847249 learned_addr learned my addr 172.21.15.50:0/35847249 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:04.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee540b700 1 --1- 172.21.15.50:0/35847249 >> v1:172.21.15.125:6789/0 conn(0x7feee00c6900 0x7feee0161170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39220/0 (socket says 172.21.15.50:39220) 2023-12-19T14:54:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedf7fe700 1 -- 172.21.15.50:0/35847249 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010074151 0 0) 0x7feee00cc030 con 0x7feee00c6900 2023-12-19T14:54:04.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedf7fe700 1 -- 172.21.15.50:0/35847249 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed0003570 con 0x7feee00c6900 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 -- 172.21.15.50:0/35847249 >> v2:172.21.15.125:3300/0 conn(0x7feee00cc4c0 msgr2=0x7feee00c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 --2- 172.21.15.50:0/35847249 >> v2:172.21.15.125:3300/0 conn(0x7feee00cc4c0 0x7feee00c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 -- 172.21.15.50:0/35847249 >> v1:172.21.15.125:6789/0 conn(0x7feee00c6900 legacy=0x7feee0161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 -- 172.21.15.50:0/35847249 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee00cb9e0 con 0x7feee015d220 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feee4c0a700 1 --2- 172.21.15.50:0/35847249 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feed40099b0 tx=0x7feed4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92816d9a948326a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedf7fe700 1 -- 172.21.15.50:0/35847249 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400b680 con 0x7feee015d220 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedf7fe700 1 -- 172.21.15.50:0/35847249 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed400b7e0 con 0x7feee015d220 2023-12-19T14:54:04.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.415+0000 7feedf7fe700 1 -- 172.21.15.50:0/35847249 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400bb80 con 0x7feee015d220 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- 172.21.15.50:0/35847249 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 msgr2=0x7feee015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 --2- 172.21.15.50:0/35847249 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feed40099b0 tx=0x7feed4009250 comp rx=0 tx=0).stop 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- 172.21.15.50:0/35847249 shutdown_connections 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 --2- 172.21.15.50:0/35847249 >> v2:172.21.15.125:3300/0 conn(0x7feee00cc4c0 0x7feee00c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 --2- 172.21.15.50:0/35847249 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee015d5f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- 172.21.15.50:0/35847249 >> 172.21.15.50:0/35847249 conn(0x7feee00cd6d0 msgr2=0x7feee006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- 172.21.15.50:0/35847249 shutdown_connections 2023-12-19T14:54:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- 172.21.15.50:0/35847249 wait complete. 2023-12-19T14:54:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 Processor -- start 2023-12-19T14:54:04.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.419+0000 7feee6e6e700 1 -- start start 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee00c6900 0x7feee014b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee00b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee00c7bb0 con 0x7feee00cc4c0 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0158d90 con 0x7feee00c6900 2023-12-19T14:54:04.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee014b970 con 0x7feee015d220 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53284/0 (socket says 172.21.15.50:53284) 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 -- 172.21.15.50:0/3380992602 learned_addr learned my addr 172.21.15.50:0/3380992602 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee540b700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee00b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee4c0a700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee00c6900 0x7feee014b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 msgr2=0x7feee00b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee00b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee00c6900 msgr2=0x7feee014b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee00c6900 0x7feee014b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 -- 172.21.15.50:0/3380992602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40094f0 con 0x7feee00cc4c0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedffff700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7feecc00ec80 tx=0x7feecc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc00c9d0 con 0x7feee00cc4c0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc00cb30 con 0x7feee00cc4c0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc013470 con 0x7feee00cc4c0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee00c72a0 con 0x7feee00cc4c0 2023-12-19T14:54:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.423+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee00c7560 con 0x7feee00cc4c0 2023-12-19T14:54:04.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee00c7d20 con 0x7feee00cc4c0 2023-12-19T14:54:04.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7feecc013820 con 0x7feee00cc4c0 2023-12-19T14:54:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feedcff9700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 0x7feec4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6515+0+0 (secure 0 0 0) 0x7feecc07d060 con 0x7feee00cc4c0 2023-12-19T14:54:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feee4c0a700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 msgr2=0x7feec4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:04.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.427+0000 7feee4c0a700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 0x7feec4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:04.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.435+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feecc0acbe0 con 0x7feee00cc4c0 2023-12-19T14:54:04.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.631+0000 7feee4c0a700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 msgr2=0x7feec4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:04.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.631+0000 7feee4c0a700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 0x7feec4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:04.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.647+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee0071120 con 0x7feee00cc4c0 2023-12-19T14:54:04.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.647+0000 7feedcff9700 1 -- 172.21.15.50:0/3380992602 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7feecc04d170 con 0x7feee00cc4c0 2023-12-19T14:54:04.650 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:04.650 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:54:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 msgr2=0x7feec4062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 0x7feec4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 msgr2=0x7feee00b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7feecc00ec80 tx=0x7feecc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 shutdown_connections 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feec4060670 0x7feec4062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feee015d220 0x7feee00b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feee00cc4c0 0x7feee00b1c00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 --2- 172.21.15.50:0/3380992602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee00c6900 0x7feee014b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 >> 172.21.15.50:0/3380992602 conn(0x7feee00cd6d0 msgr2=0x7feee00cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 shutdown_connections 2023-12-19T14:54:04.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.651+0000 7feee6e6e700 1 -- 172.21.15.50:0/3380992602 wait complete. 2023-12-19T14:54:04.656 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:54:04.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:04.696 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:04.691+0000 7f138f968000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:04.725 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:04.723+0000 7f75cb7ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:04.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:04.767+0000 7f138f968000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:04.807 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:04.807+0000 7f75cb7ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 Processor -- start 2023-12-19T14:54:04.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 -- start start 2023-12-19T14:54:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4198158e20 0x7f419806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41980ce810 0x7f41980ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419806b730 con 0x7f41981591f0 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419806ab20 con 0x7f4198158e20 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419e0e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419806ae00 con 0x7f41980ce810 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419d0e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4198158e20 0x7f419806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419d0e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4198158e20 0x7f419806a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53298/0 (socket says 172.21.15.50:53298) 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.827+0000 7f419d0e1700 1 -- 172.21.15.50:0/1235271846 learned_addr learned my addr 172.21.15.50:0/1235271846 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d8e2700 1 --2- 172.21.15.50:0/1235271846 >> v2:172.21.15.125:3300/0 conn(0x7f41980ce810 0x7f41980ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d0e1700 1 -- 172.21.15.50:0/1235271846 >> v2:172.21.15.125:3300/0 conn(0x7f41980ce810 msgr2=0x7f41980ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d0e1700 1 --2- 172.21.15.50:0/1235271846 >> v2:172.21.15.125:3300/0 conn(0x7f41980ce810 0x7f41980ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d0e1700 1 -- 172.21.15.50:0/1235271846 >> v1:172.21.15.50:6789/0 conn(0x7f41981591f0 legacy=0x7f41980ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d0e1700 1 -- 172.21.15.50:0/1235271846 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419806b0e0 con 0x7f4198158e20 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419d0e1700 1 --2- 172.21.15.50:0/1235271846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4198158e20 0x7f419806a540 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f419806dc00 tx=0x7f4194009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7db908bc0a393a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f418ffff700 1 -- 172.21.15.50:0/1235271846 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194009e70 con 0x7f4198158e20 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f418ffff700 1 -- 172.21.15.50:0/1235271846 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419400b860 con 0x7f4198158e20 2023-12-19T14:54:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f418ffff700 1 -- 172.21.15.50:0/1235271846 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419400bc00 con 0x7f4198158e20 2023-12-19T14:54:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 -- 172.21.15.50:0/1235271846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4198158e20 msgr2=0x7f419806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 --2- 172.21.15.50:0/1235271846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4198158e20 0x7f419806a540 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f419806dc00 tx=0x7f4194009250 comp rx=0 tx=0).stop 2023-12-19T14:54:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 -- 172.21.15.50:0/1235271846 shutdown_connections 2023-12-19T14:54:04.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 --2- 172.21.15.50:0/1235271846 >> v2:172.21.15.125:3300/0 conn(0x7f41980ce810 0x7f41980ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 --2- 172.21.15.50:0/1235271846 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4198158e20 0x7f419806a540 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 -- 172.21.15.50:0/1235271846 >> 172.21.15.50:0/1235271846 conn(0x7f4198067bb0 msgr2=0x7f41980cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 -- 172.21.15.50:0/1235271846 shutdown_connections 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 -- 172.21.15.50:0/1235271846 wait complete. 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.831+0000 7f419e0e3700 1 Processor -- start 2023-12-19T14:54:04.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- start start 2023-12-19T14:54:04.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41980ce810 0x7f419815bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 0x7f41980c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980c73b0 con 0x7f41981591f0 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980c81b0 con 0x7f4198158e20 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41980c75a0 con 0x7f41980ce810 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419d8e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419d8e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53314/0 (socket says 172.21.15.50:53314) 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419d0e1700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41980ce810 0x7f419815bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419d8e2700 1 -- 172.21.15.50:0/832599195 learned_addr learned my addr 172.21.15.50:0/832599195 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:04.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 0x7f41980c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 -- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41980ce810 msgr2=0x7f419815bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41980ce810 0x7f419815bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 -- 172.21.15.50:0/832599195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 msgr2=0x7f41981436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 -- 172.21.15.50:0/832599195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940094f0 con 0x7f4198158e20 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419d8e2700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419c8e0700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 0x7f41980c6e70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f418800c490 tx=0x7f4188007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418800ce40 con 0x7f4198158e20 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4188011590 con 0x7f4198158e20 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188011950 con 0x7f4198158e20 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41980c7880 con 0x7f4198158e20 2023-12-19T14:54:04.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.835+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41980c7e40 con 0x7f4198158e20 2023-12-19T14:54:04.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.839+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419804be40 con 0x7f4198158e20 2023-12-19T14:54:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.839+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f4188011cc0 con 0x7f4198158e20 2023-12-19T14:54:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.843+0000 7f418d7fa700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 0x7f417c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.843+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6515+0+0 (secure 0 0 0) 0x7f418807bfe0 con 0x7f4198158e20 2023-12-19T14:54:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.843+0000 7f419d0e1700 1 -- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 msgr2=0x7f417c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:04.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.843+0000 7f419d0e1700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 0x7f417c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:04.851+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f418804c110 con 0x7f4198158e20 2023-12-19T14:54:04.930 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:04.923+0000 7f138f968000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:04.968 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:04.967+0000 7f75cb7ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:05.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.031+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41981595f0 con 0x7f4198158e20 2023-12-19T14:54:05.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.031+0000 7f418d7fa700 1 -- 172.21.15.50:0/832599195 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16278 (secure 0 0 0) 0x7f418804f730 con 0x7f4198158e20 2023-12-19T14:54:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":73,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:04.930441+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.2a","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3803250607":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/1814796945":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2588274817":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6833/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6832/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/3551203859":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/62382090":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2430576648":"2023-12-20T14:52:23.379995+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 msgr2=0x7f417c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 0x7f417c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 msgr2=0x7f41980c6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 0x7f41980c6e70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f418800c490 tx=0x7f4188007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 shutdown_connections 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4198158e20 0x7f41980c6e70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f417c060670 0x7f417c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41981591f0 0x7f41981436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 --2- 172.21.15.50:0/832599195 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41980ce810 0x7f419815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 >> 172.21.15.50:0/832599195 conn(0x7f4198067bb0 msgr2=0x7f4198152b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 shutdown_connections 2023-12-19T14:54:05.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.035+0000 7f419e0e3700 1 -- 172.21.15.50:0/832599195 wait complete. 2023-12-19T14:54:05.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:52:06.501664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:52:07.553305+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 73, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:54:05.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:05.205 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:05.199+0000 7f138f968000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:05.241 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:05.239+0000 7f75cb7ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 Processor -- start 2023-12-19T14:54:05.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 -- start start 2023-12-19T14:54:05.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f382015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3820156d30 0x7f38200b1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382015b930 con 0x7f3820156d30 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f382006d270 con 0x7f382015b4e0 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.271+0000 7f3827bc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382006d550 con 0x7f382015abd0 2023-12-19T14:54:05.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3825160700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3820156d30 0x7f38200b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3825160700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:44264/0 (socket says 172.21.15.50:44264) 2023-12-19T14:54:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3825160700 1 -- 172.21.15.50:0/220284283 learned_addr learned my addr 172.21.15.50:0/220284283 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:05.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3825961700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f382015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 -- 172.21.15.50:0/220284283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 msgr2=0x7f382015afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f382015afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 -- 172.21.15.50:0/220284283 >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 msgr2=0x7f38201567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 --2- 172.21.15.50:0/220284283 >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 -- 172.21.15.50:0/220284283 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382006d830 con 0x7f3820156d30 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3825160700 1 --2- 172.21.15.50:0/220284283 >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:05.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3826162700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3820156d30 0x7f38200b1460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f382006b4b0 tx=0x7f381800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf81d9a60e274ce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f382495f700 1 -- 172.21.15.50:0/220284283 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818012070 con 0x7f3820156d30 2023-12-19T14:54:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f382495f700 1 -- 172.21.15.50:0/220284283 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381800fe50 con 0x7f3820156d30 2023-12-19T14:54:05.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f382495f700 1 -- 172.21.15.50:0/220284283 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800d320 con 0x7f3820156d30 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3827bc5700 1 -- 172.21.15.50:0/220284283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3820156d30 msgr2=0x7f38200b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.275+0000 7f3827bc5700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3820156d30 0x7f38200b1460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f382006b4b0 tx=0x7f381800db90 comp rx=0 tx=0).stop 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 -- 172.21.15.50:0/220284283 shutdown_connections 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3820156d30 0x7f38200b1460 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 --2- 172.21.15.50:0/220284283 >> v2:172.21.15.125:3300/0 conn(0x7f382015b4e0 0x7f38201567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 --2- 172.21.15.50:0/220284283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f382015afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 -- 172.21.15.50:0/220284283 >> 172.21.15.50:0/220284283 conn(0x7f38200cdef0 msgr2=0x7f38200ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 -- 172.21.15.50:0/220284283 shutdown_connections 2023-12-19T14:54:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 -- 172.21.15.50:0/220284283 wait complete. 2023-12-19T14:54:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 Processor -- start 2023-12-19T14:54:05.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.279+0000 7f3827bc5700 1 -- start start 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3820156d30 0x7f38200ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f3820164fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820158600 con 0x7f382015b4e0 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38200ae000 con 0x7f3820156d30 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38200ae2e0 con 0x7f382015abd0 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53324/0 (socket says 172.21.15.50:53324) 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 -- 172.21.15.50:0/1508880106 learned_addr learned my addr 172.21.15.50:0/1508880106 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3825961700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3820156d30 0x7f38200ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3825160700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f3820164fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 -- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 msgr2=0x7f3820164fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f3820164fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 -- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3820156d30 msgr2=0x7f38200ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3820156d30 0x7f38200ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 -- 172.21.15.50:0/1508880106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38100094f0 con 0x7f382015b4e0 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3826162700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f381800dbd0 tx=0x7f381801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800ce20 con 0x7f382015b4e0 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381802a460 con 0x7f382015b4e0 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381802a820 con 0x7f382015b4e0 2023-12-19T14:54:05.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381800de30 con 0x7f382015b4e0 2023-12-19T14:54:05.287 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:05.279+0000 7f138f968000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.283+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38200add10 con 0x7f382015b4e0 2023-12-19T14:54:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.287+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3820158770 con 0x7f382015b4e0 2023-12-19T14:54:05.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.287+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f3818003040 con 0x7f382015b4e0 2023-12-19T14:54:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.291+0000 7f3815ffb700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 0x7f3804062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.291+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f3818082950 con 0x7f382015b4e0 2023-12-19T14:54:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.291+0000 7f3825961700 1 -- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 msgr2=0x7f3804062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.291+0000 7f3825961700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 0x7f3804062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:05.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.295+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3818052960 con 0x7f382015b4e0 2023-12-19T14:54:05.320 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:05.319+0000 7f75cb7ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:05.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.475+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f382006db10 con 0x7f382015b4e0 2023-12-19T14:54:05.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.475+0000 7f3815ffb700 1 -- 172.21.15.50:0/1508880106 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3818055f80 con 0x7f382015b4e0 2023-12-19T14:54:05.478 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:05.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 msgr2=0x7f3804062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:05.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 0x7f3804062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 msgr2=0x7f3820148a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f381800dbd0 tx=0x7f381801c000 comp rx=0 tx=0).stop 2023-12-19T14:54:05.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 shutdown_connections 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3804060670 0x7f3804062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f382015b4e0 0x7f3820148a10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f382015abd0 0x7f3820164fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 --2- 172.21.15.50:0/1508880106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3820156d30 0x7f38200ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 >> 172.21.15.50:0/1508880106 conn(0x7f38200cdef0 msgr2=0x7f382014b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 shutdown_connections 2023-12-19T14:54:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.479+0000 7f3827bc5700 1 -- 172.21.15.50:0/1508880106 wait complete. 2023-12-19T14:54:05.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:54:05.521 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:05.515+0000 7f138f968000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:05.564 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:05.563+0000 7f75cb7ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:05.612 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:05.607+0000 7f138f968000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:05.657 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:05.655+0000 7f75cb7ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:05.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.671+0000 7fa4afef0700 1 Processor -- start 2023-12-19T14:54:05.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.671+0000 7fa4afef0700 1 -- start start 2023-12-19T14:54:05.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4afef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 0x7fa4a806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4afef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a80cbdf0 con 0x7fa4a8158e20 2023-12-19T14:54:05.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4afef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a80cc0d0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4afef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80cc3b0 con 0x7fa4a80cb970 2023-12-19T14:54:05.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4adc8c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa4a8158e20 0x7fa4a80ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33474/0 (socket says 172.21.15.50:33474) 2023-12-19T14:54:05.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4adc8c700 1 -- 172.21.15.50:0/694967033 learned_addr learned my addr 172.21.15.50:0/694967033 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:05.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4ad48b700 1 --2- 172.21.15.50:0/694967033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 0x7fa4a806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299828048 0 0) 0x7fa4a80cbdf0 con 0x7fa4a8158e20 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa494003570 con 0x7fa4a8158e20 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014014349 0 0) 0x7fa4a80cc0d0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a80cbdf0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4ad48b700 1 -- 172.21.15.50:0/694967033 >> v1:172.21.15.125:6789/0 conn(0x7fa4a80cecd0 legacy=0x7fa4a806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4ad48b700 1 -- 172.21.15.50:0/694967033 >> v1:172.21.15.50:6789/0 conn(0x7fa4a8158e20 legacy=0x7fa4a80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.675+0000 7fa4ad48b700 1 -- 172.21.15.50:0/694967033 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80cc690 con 0x7fa4a80cb970 2023-12-19T14:54:05.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4ad48b700 1 --2- 172.21.15.50:0/694967033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 0x7fa4a806acc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa498009910 tx=0x7fa4980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8814324a2b94be84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:05.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498011630 con 0x7fa4a80cb970 2023-12-19T14:54:05.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498011790 con 0x7fa4a80cb970 2023-12-19T14:54:05.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4acc8a700 1 -- 172.21.15.50:0/694967033 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498011b30 con 0x7fa4a80cb970 2023-12-19T14:54:05.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 -- 172.21.15.50:0/694967033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 msgr2=0x7fa4a806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 --2- 172.21.15.50:0/694967033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 0x7fa4a806acc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa498009910 tx=0x7fa4980091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 -- 172.21.15.50:0/694967033 shutdown_connections 2023-12-19T14:54:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 --2- 172.21.15.50:0/694967033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a80cb970 0x7fa4a806acc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 -- 172.21.15.50:0/694967033 >> 172.21.15.50:0/694967033 conn(0x7fa4a8067bb0 msgr2=0x7fa4a80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 -- 172.21.15.50:0/694967033 shutdown_connections 2023-12-19T14:54:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.679+0000 7fa4afef0700 1 -- 172.21.15.50:0/694967033 wait complete. 2023-12-19T14:54:05.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 Processor -- start 2023-12-19T14:54:05.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 -- start start 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a8158e20 0x7fa4a815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 0x7fa4a80c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80b2fb0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80b3e10 con 0x7fa4a8158e20 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4afef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a814d160 con 0x7fa4a8150b00 2023-12-19T14:54:05.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4adc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4adc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53340/0 (socket says 172.21.15.50:53340) 2023-12-19T14:54:05.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4adc8c700 1 -- 172.21.15.50:0/2901807384 learned_addr learned my addr 172.21.15.50:0/2901807384 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:05.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4ad48b700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a8158e20 0x7fa4a815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.683+0000 7fa4ae48d700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 0x7fa4a80c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 -- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 msgr2=0x7fa4a80c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 0x7fa4a80c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 -- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a8158e20 msgr2=0x7fa4a815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a8158e20 0x7fa4a815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 -- 172.21.15.50:0/2901807384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498009420 con 0x7fa4a80cecd0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4ae48d700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 0x7fa4a80c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4adc8c700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c7c0 tx=0x7fa4a4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4005c00 con 0x7fa4a80cecd0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a4005d60 con 0x7fa4a80cecd0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4007300 con 0x7fa4a80cecd0 2023-12-19T14:54:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a80b3200 con 0x7fa4a80cecd0 2023-12-19T14:54:05.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a80b37c0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.687+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a8071120 con 0x7fa4a80cecd0 2023-12-19T14:54:05.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.691+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fa4a401b3b0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.691+0000 7fa49dffb700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 0x7fa48c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:05.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.691+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fa4a407c280 con 0x7fa4a80cecd0 2023-12-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.691+0000 7fa4ad48b700 1 -- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 msgr2=0x7fa48c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.691+0000 7fa4ad48b700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 0x7fa48c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:05.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.699+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa4a404c3a0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.703 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:05.699+0000 7f138f968000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:05.755 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:05.751+0000 7f75cb7ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:05.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.871+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa4a806a7a0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.871+0000 7fa49dffb700 1 -- 172.21.15.50:0/2901807384 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa4a404f9c0 con 0x7fa4a80cecd0 2023-12-19T14:54:05.874 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 msgr2=0x7fa48c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 0x7fa48c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 msgr2=0x7fa4a814cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c7c0 tx=0x7fa4a4007ea0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 shutdown_connections 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4a8158e20 0x7fa4a815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa48c060670 0x7fa48c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4a8150b00 0x7fa4a80c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 --2- 172.21.15.50:0/2901807384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4a80cecd0 0x7fa4a814cc20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 >> 172.21.15.50:0/2901807384 conn(0x7fa4a8067bb0 msgr2=0x7fa4a80c6190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 shutdown_connections 2023-12-19T14:54:05.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:05.879+0000 7fa4afef0700 1 -- 172.21.15.50:0/2901807384 wait complete. 2023-12-19T14:54:05.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:54:06.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.075+0000 7f37a9f23700 1 Processor -- start 2023-12-19T14:54:06.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.075+0000 7f37a9f23700 1 -- start start 2023-12-19T14:54:06.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9f23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f37a4158c90 0x7f37a40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9f23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a406aeb0 0x7f37a4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9f23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a406c850 con 0x7f37a41595a0 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9f23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a406b470 con 0x7f37a4158c90 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9f23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a406b750 con 0x7f37a406aeb0 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9722700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f37a41595a0 0x7f37a415c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33480/0 (socket says 172.21.15.50:33480) 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a9722700 1 -- 172.21.15.50:0/999452684 learned_addr learned my addr 172.21.15.50:0/999452684 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 --2- 172.21.15.50:0/999452684 >> v2:172.21.15.50:3300/0 conn(0x7f37a4158c90 0x7f37a40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a3fff700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a406aeb0 0x7f37a4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a37fe700 1 -- 172.21.15.50:0/999452684 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857017618 0 0) 0x7f37a406c850 con 0x7f37a41595a0 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a37fe700 1 -- 172.21.15.50:0/999452684 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003570 con 0x7f37a41595a0 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 -- 172.21.15.50:0/999452684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a406aeb0 msgr2=0x7f37a4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a406aeb0 0x7f37a4159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 -- 172.21.15.50:0/999452684 >> v1:172.21.15.50:6789/0 conn(0x7f37a41595a0 legacy=0x7f37a415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 -- 172.21.15.50:0/999452684 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a406ba30 con 0x7f37a4158c90 2023-12-19T14:54:06.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a8f21700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a4158c90 0x7f37a40caf40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f37980098e0 tx=0x7f37980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae799254a50d297 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.079+0000 7f37a37fe700 1 -- 172.21.15.50:0/999452684 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798011630 con 0x7f37a4158c90 2023-12-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a37fe700 1 -- 172.21.15.50:0/999452684 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798011790 con 0x7f37a4158c90 2023-12-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a37fe700 1 -- 172.21.15.50:0/999452684 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798011b30 con 0x7f37a4158c90 2023-12-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 -- 172.21.15.50:0/999452684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a4158c90 msgr2=0x7f37a40caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a4158c90 0x7f37a40caf40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f37980098e0 tx=0x7f37980091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 -- 172.21.15.50:0/999452684 shutdown_connections 2023-12-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a406aeb0 0x7f37a4159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 --2- 172.21.15.50:0/999452684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a4158c90 0x7f37a40caf40 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 -- 172.21.15.50:0/999452684 >> 172.21.15.50:0/999452684 conn(0x7f37a4067bb0 msgr2=0x7f37a40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:06.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 -- 172.21.15.50:0/999452684 shutdown_connections 2023-12-19T14:54:06.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 -- 172.21.15.50:0/999452684 wait complete. 2023-12-19T14:54:06.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.083+0000 7f37a9f23700 1 Processor -- start 2023-12-19T14:54:06.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 -- start start 2023-12-19T14:54:06.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37a4158c90 0x7f37a40c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 0x7f37a4152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4154990 con 0x7f37a406aeb0 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4154c70 con 0x7f37a4158c90 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9f23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40c9060 con 0x7f37a41595a0 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53364/0 (socket says 172.21.15.50:53364) 2023-12-19T14:54:06.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 -- 172.21.15.50:0/3247130218 learned_addr learned my addr 172.21.15.50:0/3247130218 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a3fff700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37a4158c90 0x7f37a40c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9722700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 0x7f37a4152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 -- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 msgr2=0x7f37a4152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 0x7f37a4152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 -- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37a4158c90 msgr2=0x7f37a40c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37a4158c90 0x7f37a40c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a8f21700 1 -- 172.21.15.50:0/3247130218 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798009420 con 0x7f37a406aeb0 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.087+0000 7f37a9722700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 0x7f37a4152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a8f21700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3798009ed0 tx=0x7f3798017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798011900 con 0x7f37a406aeb0 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798011a60 con 0x7f37a406aeb0 2023-12-19T14:54:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a40c92c0 con 0x7f37a406aeb0 2023-12-19T14:54:06.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798025620 con 0x7f37a406aeb0 2023-12-19T14:54:06.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a40c9880 con 0x7f37a406aeb0 2023-12-19T14:54:06.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a404be40 con 0x7f37a406aeb0 2023-12-19T14:54:06.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.091+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f3798025a10 con 0x7f37a406aeb0 2023-12-19T14:54:06.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.095+0000 7f37a0ff9700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 0x7f3788062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.095+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f379807e480 con 0x7f37a406aeb0 2023-12-19T14:54:06.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.095+0000 7f37a3fff700 1 -- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 msgr2=0x7f3788062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:06.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.095+0000 7f37a3fff700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 0x7f3788062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:06.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.103+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f379804e5a0 con 0x7f37a406aeb0 2023-12-19T14:54:06.105 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:06.099+0000 7f138f968000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:06.184 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:06.183+0000 7f75cb7ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:06.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.271+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37a406bd10 con 0x7f37a406aeb0 2023-12-19T14:54:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a0ff9700 1 -- 172.21.15.50:0/3247130218 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3798015020 con 0x7f37a406aeb0 2023-12-19T14:54:06.275 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:54:06.275 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:06.271+0000 7f138f968000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:06.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 msgr2=0x7f3788062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 0x7f3788062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 msgr2=0x7f37a41535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3798009ed0 tx=0x7f3798017000 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 shutdown_connections 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37a4158c90 0x7f37a40c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3788060670 0x7f3788062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37a41595a0 0x7f37a4152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 --2- 172.21.15.50:0/3247130218 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37a406aeb0 0x7f37a41535a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 >> 172.21.15.50:0/3247130218 conn(0x7f37a4067bb0 msgr2=0x7f37a40cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 shutdown_connections 2023-12-19T14:54:06.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.275+0000 7f37a9f23700 1 -- 172.21.15.50:0/3247130218 wait complete. 2023-12-19T14:54:06.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:06.364 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:06.363+0000 7f75cb7ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:06.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.467+0000 7fb0d188e700 1 Processor -- start 2023-12-19T14:54:06.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.467+0000 7fb0d188e700 1 -- start start 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d188e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb0cc0c5380 0x7fb0cc163ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d188e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc06c7c0 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d188e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc06caa0 con 0x7fb0cc158d60 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d188e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc15d130 con 0x7fb0cc0c5380 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb0cc06b7a0 0x7fb0cc0c4c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48052/0 (socket says 172.21.15.50:48052) 2023-12-19T14:54:06.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 -- 172.21.15.50:0/3802279404 learned_addr learned my addr 172.21.15.50:0/3802279404 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d108d700 1 --2- 172.21.15.50:0/3802279404 >> v2:172.21.15.125:3300/0 conn(0x7fb0cc0c5380 0x7fb0cc163ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365071964 0 0) 0x7fb0cc06c7c0 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b4003570 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780468178 0 0) 0x7fb0b4003570 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 >> v2:172.21.15.125:3300/0 conn(0x7fb0cc0c5380 msgr2=0x7fb0cc163ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 --2- 172.21.15.50:0/3802279404 >> v2:172.21.15.125:3300/0 conn(0x7fb0cc0c5380 0x7fb0cc163ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 >> v1:172.21.15.125:6789/0 conn(0x7fb0cc158d60 legacy=0x7fb0cc15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc15d410 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fb0bc001470 con 0x7fb0cc06b7a0 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 >> v1:172.21.15.50:6789/0 conn(0x7fb0cc06b7a0 legacy=0x7fb0cc0c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0b4003a60 0x7fb0b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 0x7fb0b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b40052e0 0x7fb0b400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc15d130 con 0x7fb0b4004810 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b400c230 con 0x7fb0b40052e0 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b400c3f0 con 0x7fb0b4003a60 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 0x7fb0b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d108d700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b40052e0 0x7fb0b400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d088c700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0b4003a60 0x7fb0b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 -- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0b4003a60 msgr2=0x7fb0b4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0b4003a60 0x7fb0b4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 -- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b40052e0 msgr2=0x7fb0b400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b40052e0 0x7fb0b400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 -- 172.21.15.50:0/3802279404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4003570 con 0x7fb0b4004810 2023-12-19T14:54:06.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d108d700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b40052e0 0x7fb0b400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0d108d700 1 -- 172.21.15.50:0/3802279404 reap_dead start 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 0x7fb0b4004c80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c3c0 tx=0x7fb0bc001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c164eee70d709aa3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00cc50 con 0x7fb0b4004810 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc00cdb0 con 0x7fb0b4004810 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.471+0000 7fb0c37fe700 1 -- 172.21.15.50:0/3802279404 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc005b50 con 0x7fb0b4004810 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 msgr2=0x7fb0b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 0x7fb0b4004c80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c3c0 tx=0x7fb0bc001e50 comp rx=0 tx=0).stop 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- 172.21.15.50:0/3802279404 shutdown_connections 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 --2- 172.21.15.50:0/3802279404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0b4004810 0x7fb0b4004c80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- 172.21.15.50:0/3802279404 >> 172.21.15.50:0/3802279404 conn(0x7fb0cc067bb0 msgr2=0x7fb0cc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- 172.21.15.50:0/3802279404 shutdown_connections 2023-12-19T14:54:06.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- 172.21.15.50:0/3802279404 wait complete. 2023-12-19T14:54:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 Processor -- start 2023-12-19T14:54:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.475+0000 7fb0d188e700 1 -- start start 2023-12-19T14:54:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0cc0c9b00 0x7fb0cc0c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0cc06cd80 0x7fb0cc06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc157060 con 0x7fb0cc142490 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc06d730 con 0x7fb0cc0c9b00 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc0ca550 con 0x7fb0cc06cd80 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50472/0 (socket says 172.21.15.50:50472) 2023-12-19T14:54:06.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 -- 172.21.15.50:0/4011806426 learned_addr learned my addr 172.21.15.50:0/4011806426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0cc0c9b00 0x7fb0cc0c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d108d700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0cc06cd80 0x7fb0cc06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0cc06cd80 msgr2=0x7fb0cc06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0cc06cd80 0x7fb0cc06d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0cc0c9b00 msgr2=0x7fb0cc0c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0cc0c9b00 0x7fb0cc0c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 -- 172.21.15.50:0/4011806426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc157bc0 con 0x7fb0cc142490 2023-12-19T14:54:06.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d088c700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e590 tx=0x7fb0c8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8012590 con 0x7fb0cc142490 2023-12-19T14:54:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c80126f0 con 0x7fb0cc142490 2023-12-19T14:54:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0cc158590 con 0x7fb0cc142490 2023-12-19T14:54:06.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8012ab0 con 0x7fb0cc142490 2023-12-19T14:54:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.479+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc157570 con 0x7fb0cc142490 2023-12-19T14:54:06.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.483+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc04be40 con 0x7fb0cc142490 2023-12-19T14:54:06.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.483+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fb0c8013420 con 0x7fb0cc142490 2023-12-19T14:54:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.483+0000 7fb0c0ff9700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 0x7fb0b8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.483+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fb0c807cb30 con 0x7fb0cc142490 2023-12-19T14:54:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.487+0000 7fb0c3fff700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 msgr2=0x7fb0b8062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:06.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.487+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 0x7fb0b8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.491+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb0c804cc50 con 0x7fb0cc142490 2023-12-19T14:54:06.651 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:06.647+0000 7f138f968000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:06.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.691+0000 7fb0c3fff700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 msgr2=0x7fb0b8062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:06.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.691+0000 7fb0c3fff700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 0x7fb0b8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:06.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.699+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0cc15d6f0 con 0x7fb0cc142490 2023-12-19T14:54:06.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.699+0000 7fb0c0ff9700 1 -- 172.21.15.50:0/4011806426 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fb0c8050270 con 0x7fb0cc142490 2023-12-19T14:54:06.702 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:06.702 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 msgr2=0x7fb0b8062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 0x7fb0b8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 msgr2=0x7fb0cc158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e590 tx=0x7fb0c8009040 comp rx=0 tx=0).stop 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 shutdown_connections 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0cc0c9b00 0x7fb0cc0c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb0b80606a0 0x7fb0b8062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb0cc06cd80 0x7fb0cc06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 --2- 172.21.15.50:0/4011806426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0cc142490 0x7fb0cc158050 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 >> 172.21.15.50:0/4011806426 conn(0x7fb0cc067bb0 msgr2=0x7fb0cc0cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 shutdown_connections 2023-12-19T14:54:06.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.703+0000 7fb0d188e700 1 -- 172.21.15.50:0/4011806426 wait complete. 2023-12-19T14:54:06.708 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:54:06.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:06.755 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:06.755+0000 7f75cb7ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:06.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:06.855+0000 7f138f968000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:06.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.899+0000 7fd852c2e700 1 Processor -- start 2023-12-19T14:54:06.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.899+0000 7fd852c2e700 1 -- start start 2023-12-19T14:54:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd84c140710 0x7fd84c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c068800 0x7fd84c0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c06e150 con 0x7fd84c140ae0 2023-12-19T14:54:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c06d540 con 0x7fd84c140710 2023-12-19T14:54:06.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c06d820 con 0x7fd84c068800 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd84c140710 0x7fd84c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd84c140710 0x7fd84c0c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50482/0 (socket says 172.21.15.50:50482) 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 -- 172.21.15.50:0/161087793 learned_addr learned my addr 172.21.15.50:0/161087793 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8511cb700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c068800 0x7fd84c0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 -- 172.21.15.50:0/161087793 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c068800 msgr2=0x7fd84c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c068800 0x7fd84c0ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 -- 172.21.15.50:0/161087793 >> v1:172.21.15.50:6789/0 conn(0x7fd84c140ae0 legacy=0x7fd84c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 -- 172.21.15.50:0/161087793 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c06db00 con 0x7fd84c140710 2023-12-19T14:54:06.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd8509ca700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c140710 0x7fd84c0c70a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd8400099b0 tx=0x7fd840009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f8c59ca508e210 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd84b7fe700 1 -- 172.21.15.50:0/161087793 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84000b860 con 0x7fd84c140710 2023-12-19T14:54:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd84b7fe700 1 -- 172.21.15.50:0/161087793 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84000b9c0 con 0x7fd84c140710 2023-12-19T14:54:06.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd84b7fe700 1 -- 172.21.15.50:0/161087793 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84000bd60 con 0x7fd84c140710 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- 172.21.15.50:0/161087793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c140710 msgr2=0x7fd84c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c140710 0x7fd84c0c70a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd8400099b0 tx=0x7fd840009250 comp rx=0 tx=0).stop 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- 172.21.15.50:0/161087793 shutdown_connections 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c068800 0x7fd84c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 --2- 172.21.15.50:0/161087793 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c140710 0x7fd84c0c70a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- 172.21.15.50:0/161087793 >> 172.21.15.50:0/161087793 conn(0x7fd84c0c8850 msgr2=0x7fd84c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- 172.21.15.50:0/161087793 shutdown_connections 2023-12-19T14:54:06.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.903+0000 7fd852c2e700 1 -- 172.21.15.50:0/161087793 wait complete. 2023-12-19T14:54:06.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 Processor -- start 2023-12-19T14:54:06.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 -- start start 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c140710 0x7fd84c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 0x7fd84c0c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0c1a00 con 0x7fd84c068800 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0c1ce0 con 0x7fd84c140ae0 2023-12-19T14:54:06.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd852c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0c1fc0 con 0x7fd84c140710 2023-12-19T14:54:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.907+0000 7fd8509ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50494/0 (socket says 172.21.15.50:50494) 2023-12-19T14:54:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8511cb700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 0x7fd84c0c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 -- 172.21.15.50:0/376665627 learned_addr learned my addr 172.21.15.50:0/376665627 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:06.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd84bfff700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c140710 0x7fd84c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 -- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c140710 msgr2=0x7fd84c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c140710 0x7fd84c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 -- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 msgr2=0x7fd84c0c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 0x7fd84c0c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 -- 172.21.15.50:0/376665627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8400094f0 con 0x7fd84c068800 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8511cb700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 0x7fd84c0c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:06.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd8509ca700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0ae960 tx=0x7fd840017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840011e40 con 0x7fd84c068800 2023-12-19T14:54:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840024440 con 0x7fd84c068800 2023-12-19T14:54:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c0c22a0 con 0x7fd84c068800 2023-12-19T14:54:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840024800 con 0x7fd84c068800 2023-12-19T14:54:06.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c0c1430 con 0x7fd84c068800 2023-12-19T14:54:06.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.911+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c071120 con 0x7fd84c068800 2023-12-19T14:54:06.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.915+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fd84001a420 con 0x7fd84c068800 2023-12-19T14:54:06.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.915+0000 7fd848ff9700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 0x7fd830062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:06.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.915+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fd84007e970 con 0x7fd84c068800 2023-12-19T14:54:06.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.915+0000 7fd84bfff700 1 -- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 msgr2=0x7fd830062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:06.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.915+0000 7fd84bfff700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 0x7fd830062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:06.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:06.919+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd84004eb30 con 0x7fd84c068800 2023-12-19T14:54:06.945 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:06.939+0000 7f138f968000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:06.998 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:06.995+0000 7f75cb7ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:07.021 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.015+0000 7f138f968000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:07.099 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.099+0000 7f75cb7ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:07.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.099+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd84c0642a0 con 0x7fd84c068800 2023-12-19T14:54:07.102 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.095+0000 7f138f968000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:07.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.099+0000 7fd848ff9700 1 -- 172.21.15.50:0/376665627 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16278 (secure 0 0 0) 0x7fd840052150 con 0x7fd84c068800 2023-12-19T14:54:07.103 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:07.103 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":73,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:04.930441+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.2a","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3803250607":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/1814796945":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2588274817":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6833/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6832/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/3551203859":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/62382090":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2430576648":"2023-12-20T14:52:23.379995+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 msgr2=0x7fd830062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 0x7fd830062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 msgr2=0x7fd84c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0ae960 tx=0x7fd840017000 comp rx=0 tx=0).stop 2023-12-19T14:54:07.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 shutdown_connections 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd84c140ae0 0x7fd84c0c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8300606a0 0x7fd830062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd84c140710 0x7fd84c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 --2- 172.21.15.50:0/376665627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd84c068800 0x7fd84c15ac00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 >> 172.21.15.50:0/376665627 conn(0x7fd84c0c8850 msgr2=0x7fd84c067b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 shutdown_connections 2023-12-19T14:54:07.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.103+0000 7fd852c2e700 1 -- 172.21.15.50:0/376665627 wait complete. 2023-12-19T14:54:07.126 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:54:07.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:07.183 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.183+0000 7f75cb7ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:07.272 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.271+0000 7f75cb7ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.319+0000 7f838320c700 1 Processor -- start 2023-12-19T14:54:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.319+0000 7f838320c700 1 -- start start 2023-12-19T14:54:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f837c155a30 0x7f837c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0cd1f0 0x7f837c0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837c0c2b00 con 0x7f837c0c8ba0 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c0c1ef0 con 0x7f837c155a30 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c0c21d0 con 0x7f837c0cd1f0 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f837c155a30 0x7f837c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f837c155a30 0x7f837c0c1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50498/0 (socket says 172.21.15.50:50498) 2023-12-19T14:54:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 -- 172.21.15.50:0/2041730118 learned_addr learned my addr 172.21.15.50:0/2041730118 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f8381a09700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0cd1f0 0x7f837c0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 -- 172.21.15.50:0/2041730118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0cd1f0 msgr2=0x7f837c0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0cd1f0 0x7f837c0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 -- 172.21.15.50:0/2041730118 >> v1:172.21.15.125:6789/0 conn(0x7f837c0c8ba0 legacy=0x7f837c155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 -- 172.21.15.50:0/2041730118 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c0c24b0 con 0x7f837c155a30 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838220a700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c0c1880 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f83780099b0 tx=0x7f8378009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a998478df2ae4f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f8381208700 1 -- 172.21.15.50:0/2041730118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837800b820 con 0x7f837c155a30 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f8381208700 1 -- 172.21.15.50:0/2041730118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837800b980 con 0x7f837c155a30 2023-12-19T14:54:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f8381208700 1 -- 172.21.15.50:0/2041730118 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837800bd40 con 0x7f837c155a30 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- 172.21.15.50:0/2041730118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 msgr2=0x7f837c0c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c0c1880 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f83780099b0 tx=0x7f8378009250 comp rx=0 tx=0).stop 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- 172.21.15.50:0/2041730118 shutdown_connections 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0cd1f0 0x7f837c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 --2- 172.21.15.50:0/2041730118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c0c1880 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- 172.21.15.50:0/2041730118 >> 172.21.15.50:0/2041730118 conn(0x7f837c06c0a0 msgr2=0x7f837c0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- 172.21.15.50:0/2041730118 shutdown_connections 2023-12-19T14:54:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.323+0000 7f838320c700 1 -- 172.21.15.50:0/2041730118 wait complete. 2023-12-19T14:54:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 Processor -- start 2023-12-19T14:54:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 -- start start 2023-12-19T14:54:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f837c0cd1f0 0x7f837c0679f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c16a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c143180 con 0x7f837c155a30 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c143460 con 0x7f837c0cd1f0 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838320c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c143740 con 0x7f837c0c8ba0 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838220a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838220a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:56580/0 (socket says 172.21.15.50:56580) 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f838220a700 1 -- 172.21.15.50:0/815780177 learned_addr learned my addr 172.21.15.50:0/815780177 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f8381a09700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f837c0cd1f0 0x7f837c0679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.327+0000 7f8382a0b700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c16a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f837c0cd1f0 msgr2=0x7f837c0679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f837c0cd1f0 0x7f837c0679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 msgr2=0x7f837c16a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c16a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 -- 172.21.15.50:0/815780177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83780094f0 con 0x7f837c0c8ba0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838220a700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8378006fd0 tx=0x7f8378016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378006a90 con 0x7f837c0c8ba0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378020070 con 0x7f837c0c8ba0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378011d70 con 0x7f837c0c8ba0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c0b01d0 con 0x7f837c0c8ba0 2023-12-19T14:54:07.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c068620 con 0x7f837c0c8ba0 2023-12-19T14:54:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f8378022030 con 0x7f837c0c8ba0 2023-12-19T14:54:07.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.331+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c04be40 con 0x7f837c0c8ba0 2023-12-19T14:54:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.335+0000 7f83727fc700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 0x7f8360062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.335+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f837807e840 con 0x7f837c0c8ba0 2023-12-19T14:54:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.335+0000 7f8381a09700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 msgr2=0x7f8360062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:07.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.335+0000 7f8381a09700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 0x7f8360062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:07.343 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.339+0000 7f138f968000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:07.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.343+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f837804e960 con 0x7f837c0c8ba0 2023-12-19T14:54:07.421 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.415+0000 7f138f968000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:07.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:54:07.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:54:07.511 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.507+0000 7f138f968000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:07.516 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.515+0000 7f75cb7ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.539+0000 7f8381a09700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 msgr2=0x7f8360062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.539+0000 7f8381a09700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 0x7f8360062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.559+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f837c04be40 con 0x7f837c0c8ba0 2023-12-19T14:54:07.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.559+0000 7f83727fc700 1 -- 172.21.15.50:0/815780177 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f8378051f80 con 0x7f837c0c8ba0 2023-12-19T14:54:07.562 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:07.562 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:54:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 msgr2=0x7f8360062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 0x7f8360062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 msgr2=0x7f837c146850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8378006fd0 tx=0x7f8378016000 comp rx=0 tx=0).stop 2023-12-19T14:54:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 shutdown_connections 2023-12-19T14:54:07.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f837c0cd1f0 0x7f837c0679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8360060670 0x7f8360062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f837c155a30 0x7f837c16a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 --2- 172.21.15.50:0/815780177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f837c0c8ba0 0x7f837c146850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 >> 172.21.15.50:0/815780177 conn(0x7f837c06c0a0 msgr2=0x7f837c0c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 shutdown_connections 2023-12-19T14:54:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.563+0000 7f838320c700 1 -- 172.21.15.50:0/815780177 wait complete. 2023-12-19T14:54:07.567 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:54:07.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:07.600 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.599+0000 7f75cb7ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:07.699 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.699+0000 7f75cb7ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:07.709 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.703+0000 7f138f968000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.759+0000 7fa658ee3700 1 Processor -- start 2023-12-19T14:54:07.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.759+0000 7fa658ee3700 1 -- start start 2023-12-19T14:54:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa658ee3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa6540b1410 0x7fa6540ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa658ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa65406a580 0x7fa65406a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa658ee3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65406c7c0 con 0x7fa65415a860 2023-12-19T14:54:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa658ee3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65406cd80 con 0x7fa6540b1410 2023-12-19T14:54:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa658ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65406af10 con 0x7fa65406a580 2023-12-19T14:54:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa6540b1410 0x7fa6540ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa6540b1410 0x7fa6540ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50502/0 (socket says 172.21.15.50:50502) 2023-12-19T14:54:07.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 -- 172.21.15.50:0/3464143263 learned_addr learned my addr 172.21.15.50:0/3464143263 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa64bfff700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa65406a580 0x7fa65406a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 -- 172.21.15.50:0/3464143263 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa65406a580 msgr2=0x7fa65406a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa65406a580 0x7fa65406a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 -- 172.21.15.50:0/3464143263 >> v1:172.21.15.125:6789/0 conn(0x7fa65415a860 legacy=0x7fa65415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 -- 172.21.15.50:0/3464143263 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65406b8f0 con 0x7fa6540b1410 2023-12-19T14:54:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa6537fe700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6540b1410 0x7fa6540ab440 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa6440099b0 tx=0x7fa644009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e5a33342e4dd73 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa652ffd700 1 -- 172.21.15.50:0/3464143263 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644009ea0 con 0x7fa6540b1410 2023-12-19T14:54:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa652ffd700 1 -- 172.21.15.50:0/3464143263 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400b860 con 0x7fa6540b1410 2023-12-19T14:54:07.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.763+0000 7fa652ffd700 1 -- 172.21.15.50:0/3464143263 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400bc00 con 0x7fa6540b1410 2023-12-19T14:54:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- 172.21.15.50:0/3464143263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6540b1410 msgr2=0x7fa6540ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6540b1410 0x7fa6540ab440 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa6440099b0 tx=0x7fa644009250 comp rx=0 tx=0).stop 2023-12-19T14:54:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- 172.21.15.50:0/3464143263 shutdown_connections 2023-12-19T14:54:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa65406a580 0x7fa65406a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- 172.21.15.50:0/3464143263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa6540b1410 0x7fa6540ab440 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- 172.21.15.50:0/3464143263 >> 172.21.15.50:0/3464143263 conn(0x7fa654067bb0 msgr2=0x7fa6540cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:07.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- 172.21.15.50:0/3464143263 shutdown_connections 2023-12-19T14:54:07.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- 172.21.15.50:0/3464143263 wait complete. 2023-12-19T14:54:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 Processor -- start 2023-12-19T14:54:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- start start 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa65406a580 0x7fa6541494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6541581f0 0x7fa654169540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65416a540 con 0x7fa65415a860 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654149a20 con 0x7fa65406a580 2023-12-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.767+0000 7fa658ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa654169f80 con 0x7fa6541581f0 2023-12-19T14:54:07.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50510/0 (socket says 172.21.15.50:50510) 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 -- 172.21.15.50:0/1841806715 learned_addr learned my addr 172.21.15.50:0/1841806715 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa653fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6541581f0 0x7fa654169540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa6537fe700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa65406a580 0x7fa6541494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6541581f0 msgr2=0x7fa654169540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6541581f0 0x7fa654169540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa65406a580 msgr2=0x7fa6541494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa65406a580 0x7fa6541494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:07.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 -- 172.21.15.50:0/1841806715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6440094f0 con 0x7fa65415a860 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64bfff700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00ec80 tx=0x7fa63c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00c850 con 0x7fa65415a860 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c00c9b0 con 0x7fa65415a860 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00cd50 con 0x7fa65415a860 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65416a260 con 0x7fa65415a860 2023-12-19T14:54:07.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa654157790 con 0x7fa65415a860 2023-12-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.771+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa654157310 con 0x7fa65415a860 2023-12-19T14:54:07.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.775+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fa63c013680 con 0x7fa65415a860 2023-12-19T14:54:07.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.775+0000 7fa64b7fe700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 0x7fa634062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:07.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.775+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7fa63c07cf50 con 0x7fa65415a860 2023-12-19T14:54:07.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.775+0000 7fa6537fe700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 msgr2=0x7fa634062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:07.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.775+0000 7fa6537fe700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 0x7fa634062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:07.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.779+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa654157310 con 0x7fa65415a860 2023-12-19T14:54:07.790 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:07.783+0000 7f138f968000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.979+0000 7fa6537fe700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 msgr2=0x7fa634062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:07.979+0000 7fa6537fe700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 0x7fa634062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:07.988 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:07.987+0000 7f75cb7ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:08.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.067+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6541572e0 con 0x7fa65415a860 2023-12-19T14:54:08.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.071+0000 7fa64b7fe700 1 -- 172.21.15.50:0/1841806715 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fa63c04d070 con 0x7fa65415a860 2023-12-19T14:54:08.071 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:08.071 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:52:08.469774+0000","modified":"2023-12-19T14:52:12.052742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5964},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5964":{"gid":5964,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1875488305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1875488305},{"type":"v1","addr":"172.21.15.50:6835","nonce":1875488305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T14:54:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.071+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 msgr2=0x7fa634062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 0x7fa634062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 msgr2=0x7fa654157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00ec80 tx=0x7fa63c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 shutdown_connections 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa634060670 0x7fa634062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa6541581f0 0x7fa654169540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa65415a860 0x7fa654157cb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 --2- 172.21.15.50:0/1841806715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa65406a580 0x7fa6541494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 >> 172.21.15.50:0/1841806715 conn(0x7fa654067bb0 msgr2=0x7fa6541562f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 shutdown_connections 2023-12-19T14:54:08.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.075+0000 7fa658ee3700 1 -- 172.21.15.50:0/1841806715 wait complete. 2023-12-19T14:54:08.078 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 25 2023-12-19T14:54:08.085 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:08.083+0000 7f75cb7ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:08.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:08.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.255+0000 7f5f02b45700 1 Processor -- start 2023-12-19T14:54:08.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.255+0000 7f5f02b45700 1 -- start start 2023-12-19T14:54:08.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f02b45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5efc0cc610 0x7f5efc0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f02b45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0c6630 con 0x7f5efc1502b0 2023-12-19T14:54:08.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f02b45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0c6910 con 0x7f5efc067c90 2023-12-19T14:54:08.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f02b45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5efc0c6bf0 con 0x7f5efc0cc610 2023-12-19T14:54:08.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5ef3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5efc1502b0 0x7f5efc0c5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48074/0 (socket says 172.21.15.50:48074) 2023-12-19T14:54:08.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5ef3fff700 1 -- 172.21.15.50:0/4131856434 learned_addr learned my addr 172.21.15.50:0/4131856434 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:08.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f010e2700 1 --2- 172.21.15.50:0/4131856434 >> v2:172.21.15.50:3300/0 conn(0x7f5efc0cc610 0x7f5efc0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429920235 0 0) 0x7f5efc0c6630 con 0x7f5efc1502b0 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee4003570 con 0x7f5efc1502b0 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f010e2700 1 -- 172.21.15.50:0/4131856434 >> v1:172.21.15.125:6789/0 conn(0x7f5efc067c90 legacy=0x7f5efc14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613608290 0 0) 0x7f5efc0c6910 con 0x7f5efc067c90 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f010e2700 1 -- 172.21.15.50:0/4131856434 >> v1:172.21.15.50:6789/0 conn(0x7f5efc1502b0 legacy=0x7f5efc0c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.259+0000 7f5f010e2700 1 -- 172.21.15.50:0/4131856434 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0c6ed0 con 0x7f5efc0cc610 2023-12-19T14:54:08.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f010e2700 1 --2- 172.21.15.50:0/4131856434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c6010 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009e60 tx=0x7f5ef4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4b1dff0a129285 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400e070 con 0x7f5efc0cc610 2023-12-19T14:54:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef400bb60 con 0x7f5efc0cc610 2023-12-19T14:54:08.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5ef37fe700 1 -- 172.21.15.50:0/4131856434 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4012430 con 0x7f5efc0cc610 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 -- 172.21.15.50:0/4131856434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc0cc610 msgr2=0x7f5efc0c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4131856434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c6010 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009e60 tx=0x7f5ef4009780 comp rx=0 tx=0).stop 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 -- 172.21.15.50:0/4131856434 shutdown_connections 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4131856434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c6010 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 -- 172.21.15.50:0/4131856434 >> 172.21.15.50:0/4131856434 conn(0x7f5efc0cd6d0 msgr2=0x7f5efc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 -- 172.21.15.50:0/4131856434 shutdown_connections 2023-12-19T14:54:08.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.263+0000 7f5f02b45700 1 -- 172.21.15.50:0/4131856434 wait complete. 2023-12-19T14:54:08.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 Processor -- start 2023-12-19T14:54:08.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 -- start start 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5efc1502b0 0x7f5efc16e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc0c9c20 con 0x7f5efc067c90 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc0b3390 con 0x7f5efc0cc610 2023-12-19T14:54:08.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f02b45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc0b3670 con 0x7f5efc1502b0 2023-12-19T14:54:08.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f008e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f008e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50524/0 (socket says 172.21.15.50:50524) 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.267+0000 7f5f008e1700 1 -- 172.21.15.50:0/4265988916 learned_addr learned my addr 172.21.15.50:0/4265988916 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5ef3fff700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f010e2700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5efc1502b0 0x7f5efc16e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 -- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5efc1502b0 msgr2=0x7f5efc16e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5efc1502b0 0x7f5efc16e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 -- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 msgr2=0x7f5efc0c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 -- 172.21.15.50:0/4265988916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4009a20 con 0x7f5efc067c90 2023-12-19T14:54:08.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5ef3fff700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f008e1700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7e0 tx=0x7f5ef8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80105c0 con 0x7f5efc067c90 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8010740 con 0x7f5efc067c90 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc0b3950 con 0x7f5efc067c90 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8010b00 con 0x7f5efc067c90 2023-12-19T14:54:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc0b3f10 con 0x7f5efc067c90 2023-12-19T14:54:08.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.271+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc0b4160 con 0x7f5efc067c90 2023-12-19T14:54:08.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.275+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f5ef801b3b0 con 0x7f5efc067c90 2023-12-19T14:54:08.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.275+0000 7f5ef0ff9700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 0x7f5ee8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.275+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f5ef807c2a0 con 0x7f5efc067c90 2023-12-19T14:54:08.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.275+0000 7f5ef3fff700 1 -- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 msgr2=0x7f5ee8062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:08.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.275+0000 7f5ef3fff700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 0x7f5ee8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:08.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.283+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ef804c3c0 con 0x7f5efc067c90 2023-12-19T14:54:08.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.455+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5efc071120 con 0x7f5efc067c90 2023-12-19T14:54:08.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.455+0000 7f5ef0ff9700 1 -- 172.21.15.50:0/4265988916 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16278 (secure 0 0 0) 0x7f5ef8017090 con 0x7f5efc067c90 2023-12-19T14:54:08.459 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:08.459 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":73,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:04.930441+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:52:06.501664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T14:52:07.553305+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.2a","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3803250607":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/1814796945":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2588274817":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6833/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6832/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/3551203859":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/62382090":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2430576648":"2023-12-20T14:52:23.379995+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 msgr2=0x7f5ee8062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 0x7f5ee8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 msgr2=0x7f5efc0b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800c7e0 tx=0x7f5ef8007df0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 shutdown_connections 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5efc0cc610 0x7f5efc0c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ee8060670 0x7f5ee8062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5efc1502b0 0x7f5efc16e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 --2- 172.21.15.50:0/4265988916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5efc067c90 0x7f5efc0b4430 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 >> 172.21.15.50:0/4265988916 conn(0x7f5efc0cd6d0 msgr2=0x7f5efc06d600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 shutdown_connections 2023-12-19T14:54:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.463+0000 7f5f02b45700 1 -- 172.21.15.50:0/4265988916 wait complete. 2023-12-19T14:54:08.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:54:08.541 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.535+0000 7f138f968000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:08.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f1356da3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f1356da3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f1356da3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f1356da3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.648 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f1356da3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f135a5aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f135a5aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f135a5aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f135a5aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:08.643+0000 7f135a5aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.655+0000 7f8f11545700 1 Processor -- start 2023-12-19T14:54:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.655+0000 7f8f11545700 1 -- start start 2023-12-19T14:54:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 0x7f8f0c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f0c06dc00 0x7f8f0c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c06b0d0 con 0x7f8f0c06baa0 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c06bec0 con 0x7f8f0c06dc00 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c0b4790 con 0x7f8f0c158cc0 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f0c06dc00 0x7f8f0c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0affd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 0x7f8f0c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f0c06baa0 0x7f8f0c0b3870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48080/0 (socket says 172.21.15.50:48080) 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0b7fe700 1 -- 172.21.15.50:0/3811160064 learned_addr learned my addr 172.21.15.50:0/3811160064 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f0c06dc00 0x7f8f0c06b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50530/0 (socket says 172.21.15.50:50530) 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3811160064 >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 msgr2=0x7f8f0c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 --2- 172.21.15.50:0/3811160064 >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 0x7f8f0c06d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3811160064 >> v1:172.21.15.50:6789/0 conn(0x7f8f0c06baa0 legacy=0x7f8f0c0b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3811160064 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c06d710 con 0x7f8f0c06dc00 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0affd700 1 --2- 172.21.15.50:0/3811160064 >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 0x7f8f0c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f0a7fc700 1 --2- 172.21.15.50:0/3811160064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c06dc00 0x7f8f0c06b560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0d0830 tx=0x7f8f000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9232821c1b305199 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f09ffb700 1 -- 172.21.15.50:0/3811160064 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00011630 con 0x7f8f0c06dc00 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f09ffb700 1 -- 172.21.15.50:0/3811160064 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00011790 con 0x7f8f0c06dc00 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f09ffb700 1 -- 172.21.15.50:0/3811160064 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00011b30 con 0x7f8f0c06dc00 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- 172.21.15.50:0/3811160064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c06dc00 msgr2=0x7f8f0c06b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 --2- 172.21.15.50:0/3811160064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c06dc00 0x7f8f0c06b560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0d0830 tx=0x7f8f000091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- 172.21.15.50:0/3811160064 shutdown_connections 2023-12-19T14:54:08.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 --2- 172.21.15.50:0/3811160064 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c06dc00 0x7f8f0c06b560 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 --2- 172.21.15.50:0/3811160064 >> v2:172.21.15.125:3300/0 conn(0x7f8f0c158cc0 0x7f8f0c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- 172.21.15.50:0/3811160064 >> 172.21.15.50:0/3811160064 conn(0x7f8f0c067bb0 msgr2=0x7f8f0c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:08.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- 172.21.15.50:0/3811160064 shutdown_connections 2023-12-19T14:54:08.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.659+0000 7f8f11545700 1 -- 172.21.15.50:0/3811160064 wait complete. 2023-12-19T14:54:08.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 Processor -- start 2023-12-19T14:54:08.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 -- start start 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 0x7f8f0c14f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 0x7f8f0c169e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1693e0 con 0x7f8f0c158cc0 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c151560 con 0x7f8f0c06baa0 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.663+0000 7f8f11545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c151840 con 0x7f8f0c06dc00 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0affd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 0x7f8f0c14f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 0x7f8f0c169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:08.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50546/0 (socket says 172.21.15.50:50546) 2023-12-19T14:54:08.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0affd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 0x7f8f0c14f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34600/0 (socket says 172.21.15.50:34600) 2023-12-19T14:54:08.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0b7fe700 1 -- 172.21.15.50:0/3524788357 learned_addr learned my addr 172.21.15.50:0/3524788357 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 msgr2=0x7f8f0c14f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 0x7f8f0c14f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3524788357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 msgr2=0x7f8f0c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 -- 172.21.15.50:0/3524788357 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc009420 con 0x7f8f0c06dc00 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0b7fe700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:08.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f0a7fc700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 0x7f8f0c169e70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8f00016000 tx=0x7f8f00017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00000d10 con 0x7f8f0c06dc00 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00011b10 con 0x7f8f0c06dc00 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f00009420 con 0x7f8f0c06dc00 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00025640 con 0x7f8f0c06dc00 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c169950 con 0x7f8f0c06dc00 2023-12-19T14:54:08.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.667+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c169b40 con 0x7f8f0c06dc00 2023-12-19T14:54:08.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.671+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 24) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f8f00025a30 con 0x7f8f0c06dc00 2023-12-19T14:54:08.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.671+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6713+0+0 (secure 0 0 0) 0x7f8f0007dff0 con 0x7f8f0c06dc00 2023-12-19T14:54:08.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.679+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f000adbe0 con 0x7f8f0c06dc00 2023-12-19T14:54:08.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.899+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8f0c1519d0 con 0x7f8f0c06dc00 2023-12-19T14:54:08.901 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:54:08.899+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:54:08.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.911+0000 7f8ef2ffd700 1 -- 172.21.15.50:0/3524788357 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8f0004e4c0 con 0x7f8f0c06dc00 2023-12-19T14:54:08.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 msgr2=0x7f8f0c169e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:08.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 0x7f8f0c169e70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8f00016000 tx=0x7f8f00017000 comp rx=0 tx=0).stop 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 shutdown_connections 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f0c158cc0 0x7f8f0c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f0c06dc00 0x7f8f0c169e70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 --2- 172.21.15.50:0/3524788357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f0c06baa0 0x7f8f0c14f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 >> 172.21.15.50:0/3524788357 conn(0x7f8f0c067bb0 msgr2=0x7f8f0c0aba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 shutdown_connections 2023-12-19T14:54:08.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:08.915+0000 7f8f11545700 1 -- 172.21.15.50:0/3524788357 wait complete. 2023-12-19T14:54:08.968 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:54:08.968 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:08.919+0000 7f75cb7ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:08.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:54:08.979 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:54:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.139+0000 7f5810c8e700 1 Processor -- start 2023-12-19T14:54:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.139+0000 7f5810c8e700 1 -- start start 2023-12-19T14:54:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f5810c8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f580c0c69a0 0x7f580c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f5810c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f5810c8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c06d970 con 0x7f580c06c7c0 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f5810c8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c06cd60 con 0x7f580c0c69a0 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f5810c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c06d040 con 0x7f580c06de00 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f580affd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f580b7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f580c0c69a0 0x7f580c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f580affd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:56626/0 (socket says 172.21.15.50:56626) 2023-12-19T14:54:09.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.143+0000 7f580affd700 1 -- 172.21.15.50:0/3428852246 learned_addr learned my addr 172.21.15.50:0/3428852246 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580affd700 1 -- 172.21.15.50:0/3428852246 >> v2:172.21.15.125:3300/0 conn(0x7f580c0c69a0 msgr2=0x7f580c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580affd700 1 --2- 172.21.15.50:0/3428852246 >> v2:172.21.15.125:3300/0 conn(0x7f580c0c69a0 0x7f580c0c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580affd700 1 -- 172.21.15.50:0/3428852246 >> v1:172.21.15.125:6789/0 conn(0x7f580c06c7c0 legacy=0x7f580c15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580affd700 1 -- 172.21.15.50:0/3428852246 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c06d320 con 0x7f580c06de00 2023-12-19T14:54:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580affd700 1 --2- 172.21.15.50:0/3428852246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58000099e0 tx=0x7f5800009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdeab8cc8297f59 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580a7fc700 1 -- 172.21.15.50:0/3428852246 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580000b820 con 0x7f580c06de00 2023-12-19T14:54:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580a7fc700 1 -- 172.21.15.50:0/3428852246 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580000b980 con 0x7f580c06de00 2023-12-19T14:54:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f580a7fc700 1 -- 172.21.15.50:0/3428852246 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580000bd20 con 0x7f580c06de00 2023-12-19T14:54:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 -- 172.21.15.50:0/3428852246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 msgr2=0x7f580c06b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 --2- 172.21.15.50:0/3428852246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58000099e0 tx=0x7f5800009250 comp rx=0 tx=0).stop 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 -- 172.21.15.50:0/3428852246 shutdown_connections 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 --2- 172.21.15.50:0/3428852246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c06de00 0x7f580c06b880 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 --2- 172.21.15.50:0/3428852246 >> v2:172.21.15.125:3300/0 conn(0x7f580c0c69a0 0x7f580c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 -- 172.21.15.50:0/3428852246 >> 172.21.15.50:0/3428852246 conn(0x7f580c067b50 msgr2=0x7f580c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 -- 172.21.15.50:0/3428852246 shutdown_connections 2023-12-19T14:54:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.147+0000 7f5810c8e700 1 -- 172.21.15.50:0/3428852246 wait complete. 2023-12-19T14:54:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 Processor -- start 2023-12-19T14:54:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 -- start start 2023-12-19T14:54:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f580c06c7c0 0x7f580c15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c0c69a0 0x7f580c0c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c0c00 con 0x7f580c06de00 2023-12-19T14:54:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c1780 con 0x7f580c06c7c0 2023-12-19T14:54:09.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f5810c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c0e50 con 0x7f580c0c69a0 2023-12-19T14:54:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f580affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f580bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c0c69a0 0x7f580c0c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f580affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50578/0 (socket says 172.21.15.50:50578) 2023-12-19T14:54:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f580affd700 1 -- 172.21.15.50:0/1260106226 learned_addr learned my addr 172.21.15.50:0/1260106226 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.151+0000 7f580b7fe700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f580c06c7c0 0x7f580c15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 -- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c0c69a0 msgr2=0x7f580c0c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c0c69a0 0x7f580c0c1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 -- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f580c06c7c0 msgr2=0x7f580c15a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f580c06c7c0 0x7f580c15a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 -- 172.21.15.50:0/1260106226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58000094f0 con 0x7f580c06de00 2023-12-19T14:54:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f580affd700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800011d30 con 0x7f580c06de00 2023-12-19T14:54:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800024440 con 0x7f580c06de00 2023-12-19T14:54:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800024800 con 0x7f580c06de00 2023-12-19T14:54:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c0c1130 con 0x7f580c06de00 2023-12-19T14:54:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c15a6a0 con 0x7f580c06de00 2023-12-19T14:54:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.155+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c04be40 con 0x7f580c06de00 2023-12-19T14:54:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.159+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 24) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f580002d440 con 0x7f580c06de00 2023-12-19T14:54:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.159+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f580007e160 con 0x7f580c06de00 2023-12-19T14:54:09.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.167+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f580004e5e0 con 0x7f580c06de00 2023-12-19T14:54:09.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.379+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f580c0c18c0 con 0x7f580c06de00 2023-12-19T14:54:09.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.567+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 25) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f580004e090 con 0x7f580c06de00 2023-12-19T14:54:09.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.567+0000 7f57f37fe700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 0x7f57f8068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.567+0000 7f580b7fe700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 0x7f57f8068860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:09.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.571+0000 7f580b7fe700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 0x7f57f8068860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d10 tx=0x7f57fc0091a0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:09.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.959+0000 7f57f37fe700 1 -- 172.21.15.50:0/1260106226 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5800053840 con 0x7f580c06de00 2023-12-19T14:54:09.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 msgr2=0x7f57f8068860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 0x7f57f8068860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d10 tx=0x7f57fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 msgr2=0x7f580c15fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800017000 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 shutdown_connections 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57f8066480 0x7f57f8068860 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f580c0c69a0 0x7f580c0c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f580c06de00 0x7f580c15fea0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 --2- 172.21.15.50:0/1260106226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f580c06c7c0 0x7f580c15a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 >> 172.21.15.50:0/1260106226 conn(0x7f580c067b50 msgr2=0x7f580c0ca070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 shutdown_connections 2023-12-19T14:54:09.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:09.967+0000 7f5810c8e700 1 -- 172.21.15.50:0/1260106226 wait complete. 2023-12-19T14:54:09.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:54:10.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 Processor -- start 2023-12-19T14:54:10.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 -- start start 2023-12-19T14:54:10.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71c4158da0 0x7f71c406e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c415d720 con 0x7f71c406df10 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c40ca290 con 0x7f71c4158da0 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.215+0000 7f71ca68e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40ca4a0 con 0x7f71c40caca0 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71c4158da0 0x7f71c406e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71c4158da0 0x7f71c406e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34638/0 (socket says 172.21.15.50:34638) 2023-12-19T14:54:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c9e8d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 -- 172.21.15.50:0/2585224657 learned_addr learned my addr 172.21.15.50:0/2585224657 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 -- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 msgr2=0x7f71c406ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:10.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 --2- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c406ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 -- 172.21.15.50:0/2585224657 >> v1:172.21.15.125:6789/0 conn(0x7f71c406df10 legacy=0x7f71c406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 -- 172.21.15.50:0/2585224657 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40ca780 con 0x7f71c4158da0 2023-12-19T14:54:10.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71c968c700 1 --2- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c406e670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f71c00099b0 tx=0x7f71c0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bf5776b71114860 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:10.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71bbfff700 1 -- 172.21.15.50:0/2585224657 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0009ea0 con 0x7f71c4158da0 2023-12-19T14:54:10.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71bbfff700 1 -- 172.21.15.50:0/2585224657 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c000b860 con 0x7f71c4158da0 2023-12-19T14:54:10.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.219+0000 7f71bbfff700 1 -- 172.21.15.50:0/2585224657 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c000bc00 con 0x7f71c4158da0 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 -- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 msgr2=0x7f71c406e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c406e670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f71c00099b0 tx=0x7f71c0009250 comp rx=0 tx=0).stop 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 -- 172.21.15.50:0/2585224657 shutdown_connections 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c406ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2585224657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c406e670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 -- 172.21.15.50:0/2585224657 >> 172.21.15.50:0/2585224657 conn(0x7f71c40cc760 msgr2=0x7f71c40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 -- 172.21.15.50:0/2585224657 shutdown_connections 2023-12-19T14:54:10.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 -- 172.21.15.50:0/2585224657 wait complete. 2023-12-19T14:54:10.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.223+0000 7f71ca68e700 1 Processor -- start 2023-12-19T14:54:10.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 -- start start 2023-12-19T14:54:10.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c40c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c40c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c41554e0 con 0x7f71c406df10 2023-12-19T14:54:10.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c7e40 con 0x7f71c4158da0 2023-12-19T14:54:10.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71ca68e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c40c7880 con 0x7f71c40caca0 2023-12-19T14:54:10.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50592/0 (socket says 172.21.15.50:50592) 2023-12-19T14:54:10.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 -- 172.21.15.50:0/2980317798 learned_addr learned my addr 172.21.15.50:0/2980317798 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c9e8d700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c40c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c8e8b700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c40c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 -- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 msgr2=0x7f71c40c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c40c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 -- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 msgr2=0x7f71c40c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c40c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.227+0000 7f71c968c700 1 -- 172.21.15.50:0/2980317798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c00094f0 con 0x7f71c406df10 2023-12-19T14:54:10.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71c968c700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f71c4068230 tx=0x7f71c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0000d10 con 0x7f71c406df10 2023-12-19T14:54:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c000bbe0 con 0x7f71c406df10 2023-12-19T14:54:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0011670 con 0x7f71c406df10 2023-12-19T14:54:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c40c7b60 con 0x7f71c406df10 2023-12-19T14:54:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c40c3b80 con 0x7f71c406df10 2023-12-19T14:54:10.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.231+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c40c3da0 con 0x7f71c406df10 2023-12-19T14:54:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.235+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 25) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f71c0011aa0 con 0x7f71c406df10 2023-12-19T14:54:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.235+0000 7f71b97fa700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 0x7f71a8062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.235+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f71c007e070 con 0x7f71c406df10 2023-12-19T14:54:10.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.235+0000 7f71c8e8b700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 0x7f71a8062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.235+0000 7f71c8e8b700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 0x7f71a8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71c4157b60 tx=0x7f71b4007ec0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:10.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.243+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f71c00adbe0 con 0x7f71c406df10 2023-12-19T14:54:10.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.411+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71c404be40 con 0x7f71c406df10 2023-12-19T14:54:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.987+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f71c0011d50 con 0x7f71c406df10 2023-12-19T14:54:10.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.987+0000 7f71b97fa700 1 -- 172.21.15.50:0/2980317798 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) v1 ==== 181+0+0 (secure 0 0 0) 0x7f71c00602b0 con 0x7f71c406df10 2023-12-19T14:54:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 msgr2=0x7f71a8062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 0x7f71a8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71c4157b60 tx=0x7f71b4007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 msgr2=0x7f71c4154900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f71c4068230 tx=0x7f71c0017000 comp rx=0 tx=0).stop 2023-12-19T14:54:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 shutdown_connections 2023-12-19T14:54:10.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71c4158da0 0x7f71c40c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71a80603a0 0x7f71a8062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71c40caca0 0x7f71c40c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 --2- 172.21.15.50:0/2980317798 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71c406df10 0x7f71c4154900 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:10.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.991+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 >> 172.21.15.50:0/2980317798 conn(0x7f71c40cc760 msgr2=0x7f71c40af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.995+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 shutdown_connections 2023-12-19T14:54:10.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:10.995+0000 7f71ca68e700 1 -- 172.21.15.50:0/2980317798 wait complete. 2023-12-19T14:54:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:54:11.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:54:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.207+0000 7f249a2f0700 1 Processor -- start 2023-12-19T14:54:11.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.207+0000 7f249a2f0700 1 -- start start 2023-12-19T14:54:11.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249a2f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f24940cb130 0x7f24940cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249a2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 0x7f249406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249a2f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940b0ac0 con 0x7f249406a6a0 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249a2f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249406b030 con 0x7f24940cb130 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249a2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249406b210 con 0x7f249415c930 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f24940cb130 0x7f24940cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f24937fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f249406a6a0 0x7f249415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:32924/0 (socket says 172.21.15.50:32924) 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f24937fe700 1 -- 172.21.15.50:0/1838357913 learned_addr learned my addr 172.21.15.50:0/1838357913 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:11.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249888d700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 0x7f249406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2492ffd700 1 -- 172.21.15.50:0/1838357913 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713105613 0 0) 0x7f24940b0ac0 con 0x7f249406a6a0 2023-12-19T14:54:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2492ffd700 1 -- 172.21.15.50:0/1838357913 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c003570 con 0x7f249406a6a0 2023-12-19T14:54:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 -- 172.21.15.50:0/1838357913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 msgr2=0x7f249406aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 0x7f249406aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 -- 172.21.15.50:0/1838357913 >> v1:172.21.15.125:6789/0 conn(0x7f249406a6a0 legacy=0x7f249415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 -- 172.21.15.50:0/1838357913 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940ca810 con 0x7f24940cb130 2023-12-19T14:54:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f249888d700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 0x7f249406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2493fff700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24940cb130 0x7f24940cb500 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f24840098e0 tx=0x7f24840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38303498e8dd500 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2492ffd700 1 -- 172.21.15.50:0/1838357913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484011630 con 0x7f24940cb130 2023-12-19T14:54:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2492ffd700 1 -- 172.21.15.50:0/1838357913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2484011790 con 0x7f24940cb130 2023-12-19T14:54:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.211+0000 7f2492ffd700 1 -- 172.21.15.50:0/1838357913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484011b10 con 0x7f24940cb130 2023-12-19T14:54:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 -- 172.21.15.50:0/1838357913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24940cb130 msgr2=0x7f24940cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:11.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24940cb130 0x7f24940cb500 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f24840098e0 tx=0x7f24840091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 -- 172.21.15.50:0/1838357913 shutdown_connections 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249415c930 0x7f249406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1838357913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24940cb130 0x7f24940cb500 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 -- 172.21.15.50:0/1838357913 >> 172.21.15.50:0/1838357913 conn(0x7f2494067bb0 msgr2=0x7f24940cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 -- 172.21.15.50:0/1838357913 shutdown_connections 2023-12-19T14:54:11.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 -- 172.21.15.50:0/1838357913 wait complete. 2023-12-19T14:54:11.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.215+0000 7f249a2f0700 1 Processor -- start 2023-12-19T14:54:11.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 -- start start 2023-12-19T14:54:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24940cb130 0x7f249414f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 0x7f24940cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494151ce0 con 0x7f249415c930 2023-12-19T14:54:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249414fa20 con 0x7f24940cb130 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249a2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249414fc10 con 0x7f249406a6a0 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f249888d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 0x7f24940cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f2493fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f2493fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:56684/0 (socket says 172.21.15.50:56684) 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f2493fff700 1 -- 172.21.15.50:0/1253441874 learned_addr learned my addr 172.21.15.50:0/1253441874 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.219+0000 7f24937fe700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24940cb130 0x7f249414f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 -- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24940cb130 msgr2=0x7f249414f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:11.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24940cb130 0x7f249414f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 -- 172.21.15.50:0/1253441874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 msgr2=0x7f24940cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:11.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 0x7f24940cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:11.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 -- 172.21.15.50:0/1253441874 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009a20 con 0x7f249406a6a0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f249888d700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 0x7f24940cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f2493fff700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f249406f300 tx=0x7f2484017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484011900 con 0x7f249406a6a0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2484011a80 con 0x7f249406a6a0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248401a680 con 0x7f249406a6a0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2484009420 con 0x7f249406a6a0 2023-12-19T14:54:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2494151620 con 0x7f249406a6a0 2023-12-19T14:54:11.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.223+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2494151e20 con 0x7f249406a6a0 2023-12-19T14:54:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.227+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f248401aa70 con 0x7f249406a6a0 2023-12-19T14:54:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.227+0000 7f247bfff700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 0x7f2480062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.227+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f248407e780 con 0x7f249406a6a0 2023-12-19T14:54:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.227+0000 7f24937fe700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 0x7f2480062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:11.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.231+0000 7f24937fe700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 0x7f2480062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f24940b2430 tx=0x7f2488009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:11.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.235+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f248404ea70 con 0x7f249406a6a0 2023-12-19T14:54:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.411+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2494151e20 con 0x7f249406a6a0 2023-12-19T14:54:11.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.995+0000 7f247bfff700 1 -- 172.21.15.50:0/1253441874 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2484052090 con 0x7f249406a6a0 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 msgr2=0x7f2480062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 0x7f2480062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f24940b2430 tx=0x7f2488009040 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 msgr2=0x7f2494156b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f249406f300 tx=0x7f2484017000 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 shutdown_connections 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24940cb130 0x7f249414f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2480060670 0x7f2480062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f249415c930 0x7f24940cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 --2- 172.21.15.50:0/1253441874 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f249406a6a0 0x7f2494156b60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 >> 172.21.15.50:0/1253441874 conn(0x7f2494067bb0 msgr2=0x7f249414ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:12.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 shutdown_connections 2023-12-19T14:54:12.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:11.999+0000 7f249a2f0700 1 -- 172.21.15.50:0/1253441874 wait complete. 2023-12-19T14:54:12.006 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:54:12.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:54:12.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 Processor -- start 2023-12-19T14:54:12.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 -- start start 2023-12-19T14:54:12.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1210158e20 0x7f12100ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 0x7f12100ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100cbdf0 con 0x7f12100ca2c0 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12100cc0d0 con 0x7f1210158e20 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f12162ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12100cc3b0 con 0x7f12100cb970 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f120ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1210158e20 0x7f12100ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f120ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1210158e20 0x7f12100ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50622/0 (socket says 172.21.15.50:50622) 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f120ffff700 1 -- 172.21.15.50:0/1037005132 learned_addr learned my addr 172.21.15.50:0/1037005132 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:12.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.203+0000 7f120f7fe700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 0x7f12100ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120ffff700 1 -- 172.21.15.50:0/1037005132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 msgr2=0x7f12100ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:12.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120ffff700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 0x7f12100ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120ffff700 1 -- 172.21.15.50:0/1037005132 >> v1:172.21.15.125:6789/0 conn(0x7f12100ca2c0 legacy=0x7f121006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120ffff700 1 -- 172.21.15.50:0/1037005132 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12100cc690 con 0x7f1210158e20 2023-12-19T14:54:12.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120f7fe700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 0x7f12100ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:12.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120ffff700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100ce5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1200009f20 tx=0x7f1200009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64151f4bbd2163e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120effd700 1 -- 172.21.15.50:0/1037005132 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120000e070 con 0x7f1210158e20 2023-12-19T14:54:12.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120effd700 1 -- 172.21.15.50:0/1037005132 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120000ba80 con 0x7f1210158e20 2023-12-19T14:54:12.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f120effd700 1 -- 172.21.15.50:0/1037005132 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12000092e0 con 0x7f1210158e20 2023-12-19T14:54:12.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 -- 172.21.15.50:0/1037005132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 msgr2=0x7f12100ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100ce5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1200009f20 tx=0x7f1200009840 comp rx=0 tx=0).stop 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 -- 172.21.15.50:0/1037005132 shutdown_connections 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100cb970 0x7f12100ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 --2- 172.21.15.50:0/1037005132 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100ce5c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 -- 172.21.15.50:0/1037005132 >> 172.21.15.50:0/1037005132 conn(0x7f1210067bb0 msgr2=0x7f12100d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 -- 172.21.15.50:0/1037005132 shutdown_connections 2023-12-19T14:54:12.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.207+0000 7f12162ae700 1 -- 172.21.15.50:0/1037005132 wait complete. 2023-12-19T14:54:12.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 Processor -- start 2023-12-19T14:54:12.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 -- start start 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12100ca2c0 0x7f121014f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100c8120 0x7f121015a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121014fe80 con 0x7f1210158e20 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100b40e0 con 0x7f12100ca2c0 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.211+0000 7f12162ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1210169930 con 0x7f12100c8120 2023-12-19T14:54:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120ffff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12100ca2c0 0x7f121014f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50638/0 (socket says 172.21.15.50:50638) 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 -- 172.21.15.50:0/273068957 learned_addr learned my addr 172.21.15.50:0/273068957 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f121484b700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100c8120 0x7f121015a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 -- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100c8120 msgr2=0x7f121015a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100c8120 0x7f121015a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 -- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12100ca2c0 msgr2=0x7f121014f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12100ca2c0 0x7f121014f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 -- 172.21.15.50:0/273068957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1200009ae0 con 0x7f1210158e20 2023-12-19T14:54:12.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f120f7fe700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f120400e8d0 tx=0x7f1204008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204013410 con 0x7f1210158e20 2023-12-19T14:54:12.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1204013570 con 0x7f1210158e20 2023-12-19T14:54:12.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121016a6b0 con 0x7f1210158e20 2023-12-19T14:54:12.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204013910 con 0x7f1210158e20 2023-12-19T14:54:12.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.215+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1210169eb0 con 0x7f1210158e20 2023-12-19T14:54:12.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.219+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1210071120 con 0x7f1210158e20 2023-12-19T14:54:12.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.223+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1204013d00 con 0x7f1210158e20 2023-12-19T14:54:12.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.223+0000 7f11f7fff700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 0x7f11fc062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.223+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f120407c840 con 0x7f1210158e20 2023-12-19T14:54:12.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.223+0000 7f120ffff700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 0x7f11fc062c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.227+0000 7f120ffff700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 0x7f11fc062c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1200009880 tx=0x7f1200018000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.231+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f120404cb30 con 0x7f1210158e20 2023-12-19T14:54:12.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.403+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1210150570 con 0x7f1210158e20 2023-12-19T14:54:12.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.411+0000 7f11f7fff700 1 -- 172.21.15.50:0/273068957 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1204050150 con 0x7f1210158e20 2023-12-19T14:54:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 msgr2=0x7f11fc062c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 0x7f11fc062c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1200009880 tx=0x7f1200018000 comp rx=0 tx=0).stop 2023-12-19T14:54:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 msgr2=0x7f12100c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f120400e8d0 tx=0x7f1204008040 comp rx=0 tx=0).stop 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 shutdown_connections 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11fc0607c0 0x7f11fc062c70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12100c8120 0x7f121015a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1210158e20 0x7f12100c7be0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 --2- 172.21.15.50:0/273068957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12100ca2c0 0x7f121014f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 >> 172.21.15.50:0/273068957 conn(0x7f1210067bb0 msgr2=0x7f12100c65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 shutdown_connections 2023-12-19T14:54:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.415+0000 7f12162ae700 1 -- 172.21.15.50:0/273068957 wait complete. 2023-12-19T14:54:12.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:54:12.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.599+0000 7f18f884c700 1 Processor -- start 2023-12-19T14:54:12.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 -- start start 2023-12-19T14:54:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 0x7f18f40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18f40caa10 0x7f18f415d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40cbbb0 con 0x7f18f40cc040 2023-12-19T14:54:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f415d960 con 0x7f18f40caa10 2023-12-19T14:54:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f884c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f406dc90 con 0x7f18f40ce2d0 2023-12-19T14:54:12.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 0x7f18f40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f2ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18f40caa10 0x7f18f415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f3fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f18f40cc040 0x7f18f4158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:32936/0 (socket says 172.21.15.50:32936) 2023-12-19T14:54:12.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f3fff700 1 -- 172.21.15.50:0/732002753 learned_addr learned my addr 172.21.15.50:0/732002753 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:12.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.603+0000 7f18f27fc700 1 -- 172.21.15.50:0/732002753 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500267905 0 0) 0x7f18f40cbbb0 con 0x7f18f40cc040 2023-12-19T14:54:12.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f37fe700 1 -- 172.21.15.50:0/732002753 >> v2:172.21.15.125:3300/0 conn(0x7f18f40caa10 msgr2=0x7f18f415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:12.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f37fe700 1 --2- 172.21.15.50:0/732002753 >> v2:172.21.15.125:3300/0 conn(0x7f18f40caa10 0x7f18f415d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f37fe700 1 -- 172.21.15.50:0/732002753 >> v1:172.21.15.125:6789/0 conn(0x7f18f40cc040 legacy=0x7f18f4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f37fe700 1 -- 172.21.15.50:0/732002753 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f40cafa0 con 0x7f18f40ce2d0 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f37fe700 1 --2- 172.21.15.50:0/732002753 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 0x7f18f40ce6a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18e40098e0 tx=0x7f18e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b5970beec8e1afb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f27fc700 1 -- 172.21.15.50:0/732002753 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400b640 con 0x7f18f40ce2d0 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f27fc700 1 -- 172.21.15.50:0/732002753 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e400b7a0 con 0x7f18f40ce2d0 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f27fc700 1 -- 172.21.15.50:0/732002753 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400bb40 con 0x7f18f40ce2d0 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 -- 172.21.15.50:0/732002753 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 msgr2=0x7f18f40ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 --2- 172.21.15.50:0/732002753 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 0x7f18f40ce6a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18e40098e0 tx=0x7f18e40091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 -- 172.21.15.50:0/732002753 shutdown_connections 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 --2- 172.21.15.50:0/732002753 >> v2:172.21.15.125:3300/0 conn(0x7f18f40caa10 0x7f18f415d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 --2- 172.21.15.50:0/732002753 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40ce2d0 0x7f18f40ce6a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 -- 172.21.15.50:0/732002753 >> 172.21.15.50:0/732002753 conn(0x7f18f4067bb0 msgr2=0x7f18f40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 -- 172.21.15.50:0/732002753 shutdown_connections 2023-12-19T14:54:12.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.607+0000 7f18f884c700 1 -- 172.21.15.50:0/732002753 wait complete. 2023-12-19T14:54:12.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 Processor -- start 2023-12-19T14:54:12.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 -- start start 2023-12-19T14:54:12.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40cc040 0x7f18f415a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18f40ce2d0 0x7f18f40c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f416a0f0 con 0x7f18f40caa10 2023-12-19T14:54:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f415c170 con 0x7f18f40ce2d0 2023-12-19T14:54:12.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f884c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f415ae30 con 0x7f18f40cc040 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50642/0 (socket says 172.21.15.50:50642) 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 -- 172.21.15.50:0/3901575292 learned_addr learned my addr 172.21.15.50:0/3901575292 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f3fff700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18f40ce2d0 0x7f18f40c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f2ffd700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40cc040 0x7f18f415a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 -- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40cc040 msgr2=0x7f18f415a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40cc040 0x7f18f415a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 -- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18f40ce2d0 msgr2=0x7f18f40c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18f40ce2d0 0x7f18f40c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 -- 172.21.15.50:0/3901575292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e4009420 con 0x7f18f40caa10 2023-12-19T14:54:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.611+0000 7f18f37fe700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f18e40093f0 tx=0x7f18e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400bb20 con 0x7f18f40caa10 2023-12-19T14:54:12.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4000c90 con 0x7f18f40caa10 2023-12-19T14:54:12.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f415b0d0 con 0x7f18f40caa10 2023-12-19T14:54:12.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4011650 con 0x7f18f40caa10 2023-12-19T14:54:12.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f415b970 con 0x7f18f40caa10 2023-12-19T14:54:12.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f404be40 con 0x7f18f40caa10 2023-12-19T14:54:12.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.615+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f18e4011a40 con 0x7f18f40caa10 2023-12-19T14:54:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.619+0000 7f18db7fe700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 0x7f18e0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.619+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f18e407e260 con 0x7f18f40caa10 2023-12-19T14:54:12.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.619+0000 7f18f2ffd700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 0x7f18e0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:12.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.623+0000 7f18f2ffd700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 0x7f18e0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18f40c6cf0 tx=0x7f18e80091a0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:12.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.627+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18e404e5e0 con 0x7f18f40caa10 2023-12-19T14:54:12.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.795+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f18f40cb280 con 0x7f18f40caa10 2023-12-19T14:54:12.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.807+0000 7f18db7fe700 1 -- 172.21.15.50:0/3901575292 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7f18e4051c00 con 0x7f18f40caa10 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.807+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 msgr2=0x7f18e0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.807+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 0x7f18e0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18f40c6cf0 tx=0x7f18e80091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.807+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 msgr2=0x7f18f4169bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.807+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f18e40093f0 tx=0x7f18e4017000 comp rx=0 tx=0).stop 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 shutdown_connections 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18f40ce2d0 0x7f18f40c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f18e0060670 0x7f18e0062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18f40cc040 0x7f18f415a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 --2- 172.21.15.50:0/3901575292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18f40caa10 0x7f18f4169bb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 >> 172.21.15.50:0/3901575292 conn(0x7f18f4067bb0 msgr2=0x7f18f40c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 shutdown_connections 2023-12-19T14:54:12.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:12.811+0000 7f18f884c700 1 -- 172.21.15.50:0/3901575292 wait complete. 2023-12-19T14:54:12.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:54:13.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.015+0000 7f87f24f0700 1 Processor -- start 2023-12-19T14:54:13.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.015+0000 7f87f24f0700 1 -- start start 2023-12-19T14:54:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f24f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87ec0cecd0 0x7f87ec06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f24f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0cbdf0 con 0x7f87ec158e20 2023-12-19T14:54:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f24f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0cc0d0 con 0x7f87ec0cb970 2023-12-19T14:54:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f24f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec0cc3b0 con 0x7f87ec0cecd0 2023-12-19T14:54:13.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87ebfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f87ec158e20 0x7f87ec0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48092/0 (socket says 172.21.15.50:48092) 2023-12-19T14:54:13.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87ebfff700 1 -- 172.21.15.50:0/350920363 learned_addr learned my addr 172.21.15.50:0/350920363 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:13.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f0a8d700 1 --2- 172.21.15.50:0/350920363 >> v2:172.21.15.125:3300/0 conn(0x7f87ec0cecd0 0x7f87ec06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569116371 0 0) 0x7f87ec0cc0d0 con 0x7f87ec0cb970 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d8003570 con 0x7f87ec0cb970 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268088923 0 0) 0x7f87ec0cbdf0 con 0x7f87ec158e20 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ec0cc0d0 con 0x7f87ec158e20 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f0a8d700 1 -- 172.21.15.50:0/350920363 >> v1:172.21.15.125:6789/0 conn(0x7f87ec0cb970 legacy=0x7f87ec06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f0a8d700 1 -- 172.21.15.50:0/350920363 >> v1:172.21.15.50:6789/0 conn(0x7f87ec158e20 legacy=0x7f87ec0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f0a8d700 1 -- 172.21.15.50:0/350920363 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec0cc690 con 0x7f87ec0cecd0 2023-12-19T14:54:13.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87f0a8d700 1 --2- 172.21.15.50:0/350920363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec0cecd0 0x7f87ec06b3d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009910 tx=0x7f87e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6515a13dc9c4c435 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4011630 con 0x7f87ec0cecd0 2023-12-19T14:54:13.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4011790 con 0x7f87ec0cecd0 2023-12-19T14:54:13.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.019+0000 7f87eb7fe700 1 -- 172.21.15.50:0/350920363 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4011b30 con 0x7f87ec0cecd0 2023-12-19T14:54:13.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 -- 172.21.15.50:0/350920363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec0cecd0 msgr2=0x7f87ec06b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 --2- 172.21.15.50:0/350920363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec0cecd0 0x7f87ec06b3d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009910 tx=0x7f87e40091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 -- 172.21.15.50:0/350920363 shutdown_connections 2023-12-19T14:54:13.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 --2- 172.21.15.50:0/350920363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec0cecd0 0x7f87ec06b3d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 -- 172.21.15.50:0/350920363 >> 172.21.15.50:0/350920363 conn(0x7f87ec067bb0 msgr2=0x7f87ec0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:13.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 -- 172.21.15.50:0/350920363 shutdown_connections 2023-12-19T14:54:13.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 -- 172.21.15.50:0/350920363 wait complete. 2023-12-19T14:54:13.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.023+0000 7f87f24f0700 1 Processor -- start 2023-12-19T14:54:13.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 -- start start 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec158e20 0x7f87ec0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f87ec0b3d10 0x7f87ec0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec148710 con 0x7f87ec0cecd0 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c9b90 con 0x7f87ec158e20 2023-12-19T14:54:13.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f24f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c7230 con 0x7f87ec0b3d10 2023-12-19T14:54:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50652/0 (socket says 172.21.15.50:50652) 2023-12-19T14:54:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 -- 172.21.15.50:0/1369567312 learned_addr learned my addr 172.21.15.50:0/1369567312 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87f0a8d700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f87ec0b3d10 0x7f87ec0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87e37fe700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec158e20 0x7f87ec0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 -- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f87ec0b3d10 msgr2=0x7f87ec0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f87ec0b3d10 0x7f87ec0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 -- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec158e20 msgr2=0x7f87ec0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec158e20 0x7f87ec0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 -- 172.21.15.50:0/1369567312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4009420 con 0x7f87ec0cecd0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.027+0000 7f87ebfff700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c7c0 tx=0x7f87dc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc005b30 con 0x7f87ec0cecd0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc005c90 con 0x7f87ec0cecd0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec14e790 con 0x7f87ec0cecd0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc007320 con 0x7f87ec0cecd0 2023-12-19T14:54:13.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec0c74d0 con 0x7f87ec0cecd0 2023-12-19T14:54:13.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec0c76f0 con 0x7f87ec0cecd0 2023-12-19T14:54:13.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.031+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f87dc01b3b0 con 0x7f87ec0cecd0 2023-12-19T14:54:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.035+0000 7f87e8ff9700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 0x7f87cc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.035+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f87dc07bf70 con 0x7f87ec0cecd0 2023-12-19T14:54:13.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.035+0000 7f87e37fe700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 0x7f87cc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.035+0000 7f87e37fe700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 0x7f87cc062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87ec148c60 tx=0x7f87d4002660 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.043+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f87dc04c2f0 con 0x7f87ec0cecd0 2023-12-19T14:54:13.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.219+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f87ec071120 con 0x7f87ec0cecd0 2023-12-19T14:54:13.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87e8ff9700 1 -- 172.21.15.50:0/1369567312 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7f87dc04f910 con 0x7f87ec0cecd0 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 msgr2=0x7f87cc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 0x7f87cc062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87ec148c60 tx=0x7f87d4002660 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 msgr2=0x7f87ec0c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c7c0 tx=0x7f87dc007e00 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.231+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 shutdown_connections 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87ec158e20 0x7f87ec0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f87cc0606a0 0x7f87cc062b50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f87ec0b3d10 0x7f87ec0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 --2- 172.21.15.50:0/1369567312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87ec0cecd0 0x7f87ec0c9650 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 >> 172.21.15.50:0/1369567312 conn(0x7f87ec067bb0 msgr2=0x7f87ec148f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:13.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 shutdown_connections 2023-12-19T14:54:13.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.235+0000 7f87f24f0700 1 -- 172.21.15.50:0/1369567312 wait complete. 2023-12-19T14:54:13.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-12-19T14:54:13.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 Processor -- start 2023-12-19T14:54:13.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 -- start start 2023-12-19T14:54:13.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe10140700 0x7efe10140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100c8570 0x7efe100ace70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100cbb70 con 0x7efe100caac0 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe100c89d0 con 0x7efe10140700 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.423+0000 7efe16878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe100caf60 con 0x7efe100c8570 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe10140700 0x7efe10140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe10140700 0x7efe10140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34706/0 (socket says 172.21.15.50:34706) 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 -- 172.21.15.50:0/2981671706 learned_addr learned my addr 172.21.15.50:0/2981671706 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:13.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe14e15700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100c8570 0x7efe100ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 -- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100c8570 msgr2=0x7efe100ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:13.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100c8570 0x7efe100ace70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 -- 172.21.15.50:0/2981671706 >> v1:172.21.15.50:6789/0 conn(0x7efe100caac0 legacy=0x7efe100cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 -- 172.21.15.50:0/2981671706 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe100cb240 con 0x7efe10140700 2023-12-19T14:54:13.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0ffff700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe10140ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efe00009f20 tx=0x7efe00009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d2e81a69bcc09fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0f7fe700 1 -- 172.21.15.50:0/2981671706 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0000e070 con 0x7efe10140700 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0f7fe700 1 -- 172.21.15.50:0/2981671706 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0000b9d0 con 0x7efe10140700 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe0f7fe700 1 -- 172.21.15.50:0/2981671706 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000092e0 con 0x7efe10140700 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 -- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 msgr2=0x7efe10140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe10140ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efe00009f20 tx=0x7efe00009840 comp rx=0 tx=0).stop 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 -- 172.21.15.50:0/2981671706 shutdown_connections 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100c8570 0x7efe100ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 --2- 172.21.15.50:0/2981671706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe10140ad0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 -- 172.21.15.50:0/2981671706 >> 172.21.15.50:0/2981671706 conn(0x7efe100cd9c0 msgr2=0x7efe100c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 -- 172.21.15.50:0/2981671706 shutdown_connections 2023-12-19T14:54:13.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.427+0000 7efe16878700 1 -- 172.21.15.50:0/2981671706 wait complete. 2023-12-19T14:54:13.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.431+0000 7efe16878700 1 Processor -- start 2023-12-19T14:54:13.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.431+0000 7efe16878700 1 -- start start 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100caac0 0x7efe1015b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe100b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1015b610 con 0x7efe100c8570 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe10167410 con 0x7efe10140700 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe101676f0 con 0x7efe100caac0 2023-12-19T14:54:13.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50668/0 (socket says 172.21.15.50:50668) 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 -- 172.21.15.50:0/3394599870 learned_addr learned my addr 172.21.15.50:0/3394599870 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe077fe700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100caac0 0x7efe1015b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe14e15700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe100b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 -- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100caac0 msgr2=0x7efe1015b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100caac0 0x7efe1015b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 -- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 msgr2=0x7efe100b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe100b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 -- 172.21.15.50:0/3394599870 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00009ae0 con 0x7efe100c8570 2023-12-19T14:54:13.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0ffff700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efe0000b610 tx=0x7efe00018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000088f0 con 0x7efe100c8570 2023-12-19T14:54:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe00008a50 con 0x7efe100c8570 2023-12-19T14:54:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe101679d0 con 0x7efe100c8570 2023-12-19T14:54:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00008e10 con 0x7efe100c8570 2023-12-19T14:54:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.435+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe10166fa0 con 0x7efe100c8570 2023-12-19T14:54:13.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.439+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe10071120 con 0x7efe100c8570 2023-12-19T14:54:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.439+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7efe00003040 con 0x7efe100c8570 2023-12-19T14:54:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.443+0000 7efe0cff9700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 0x7efdf0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.443+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7efe0007e3b0 con 0x7efe100c8570 2023-12-19T14:54:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.443+0000 7efe077fe700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 0x7efdf0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.443+0000 7efe077fe700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 0x7efdf0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efe100c6c50 tx=0x7efdf8007ec0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.451+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efe0004e770 con 0x7efe100c8570 2023-12-19T14:54:13.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:13.687+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7efe100cb520 con 0x7efe100c8570 2023-12-19T14:54:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.007+0000 7efe0cff9700 1 -- 172.21.15.50:0/3394599870 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v555) v1 ==== 154+0+0 (secure 0 0 0) 0x7efe00051d90 con 0x7efe100c8570 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 msgr2=0x7efdf0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 0x7efdf0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efe100c6c50 tx=0x7efdf8007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 msgr2=0x7efe1013d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efe0000b610 tx=0x7efe00018000 comp rx=0 tx=0).stop 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 shutdown_connections 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe10140700 0x7efe100b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efdf00606a0 0x7efdf0062b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efe100caac0 0x7efe1015b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 --2- 172.21.15.50:0/3394599870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efe100c8570 0x7efe1013d670 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 >> 172.21.15.50:0/3394599870 conn(0x7efe100cd9c0 msgr2=0x7efe100c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 shutdown_connections 2023-12-19T14:54:14.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.011+0000 7efe16878700 1 -- 172.21.15.50:0/3394599870 wait complete. 2023-12-19T14:54:14.037 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-12-19T14:54:14.037 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-12-19T14:54:14.037 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:54:14.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.239+0000 7f493d345700 1 Processor -- start 2023-12-19T14:54:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.239+0000 7f493d345700 1 -- start start 2023-12-19T14:54:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f493d345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4938156c50 0x7f49380ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f493d345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f49380adc20 0x7f493806a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f493d345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493806dd10 con 0x7f4938156880 2023-12-19T14:54:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f493d345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49380c7300 con 0x7f49380adc20 2023-12-19T14:54:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f493d345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493806d1d0 con 0x7f4938156c50 2023-12-19T14:54:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49367fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4938156c50 0x7f49380ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f4936ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4938156880 0x7f49380c6880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:32968/0 (socket says 172.21.15.50:32968) 2023-12-19T14:54:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f4936ffd700 1 -- 172.21.15.50:0/3634667479 learned_addr learned my addr 172.21.15.50:0/3634667479 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:14.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49377fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f49380adc20 0x7f493806a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f4935ffb700 1 -- 172.21.15.50:0/3634667479 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110309421 0 0) 0x7f493806dd10 con 0x7f4938156880 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f4935ffb700 1 -- 172.21.15.50:0/3634667479 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4920003570 con 0x7f4938156880 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49377fe700 1 -- 172.21.15.50:0/3634667479 >> v2:172.21.15.125:3300/0 conn(0x7f4938156c50 msgr2=0x7f49380ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49377fe700 1 --2- 172.21.15.50:0/3634667479 >> v2:172.21.15.125:3300/0 conn(0x7f4938156c50 0x7f49380ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49377fe700 1 -- 172.21.15.50:0/3634667479 >> v1:172.21.15.125:6789/0 conn(0x7f4938156880 legacy=0x7f49380c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.243+0000 7f49377fe700 1 -- 172.21.15.50:0/3634667479 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49380ae050 con 0x7f49380adc20 2023-12-19T14:54:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f49377fe700 1 --2- 172.21.15.50:0/3634667479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49380adc20 0x7f493806a820 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f493806a380 tx=0x7f4930009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48a4c59950f3876 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f4935ffb700 1 -- 172.21.15.50:0/3634667479 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493000e070 con 0x7f49380adc20 2023-12-19T14:54:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f4935ffb700 1 -- 172.21.15.50:0/3634667479 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493000bab0 con 0x7f49380adc20 2023-12-19T14:54:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f4935ffb700 1 -- 172.21.15.50:0/3634667479 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493000be70 con 0x7f49380adc20 2023-12-19T14:54:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 -- 172.21.15.50:0/3634667479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49380adc20 msgr2=0x7f493806a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 --2- 172.21.15.50:0/3634667479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49380adc20 0x7f493806a820 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f493806a380 tx=0x7f4930009780 comp rx=0 tx=0).stop 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 -- 172.21.15.50:0/3634667479 shutdown_connections 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 --2- 172.21.15.50:0/3634667479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49380adc20 0x7f493806a820 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 --2- 172.21.15.50:0/3634667479 >> v2:172.21.15.125:3300/0 conn(0x7f4938156c50 0x7f49380ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 -- 172.21.15.50:0/3634667479 >> 172.21.15.50:0/3634667479 conn(0x7f4938067bb0 msgr2=0x7f49380cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 -- 172.21.15.50:0/3634667479 shutdown_connections 2023-12-19T14:54:14.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.247+0000 7f493d345700 1 -- 172.21.15.50:0/3634667479 wait complete. 2023-12-19T14:54:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.251+0000 7f493d345700 1 Processor -- start 2023-12-19T14:54:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.251+0000 7f493d345700 1 -- start start 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.251+0000 7f493d345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49380adc20 0x7f49381513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4938156c50 0x7f49380c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493815bed0 con 0x7f4938156880 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49380af9a0 con 0x7f4938156c50 2023-12-19T14:54:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493815b880 con 0x7f49380adc20 2023-12-19T14:54:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50684/0 (socket says 172.21.15.50:50684) 2023-12-19T14:54:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 -- 172.21.15.50:0/541953820 learned_addr learned my addr 172.21.15.50:0/541953820 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f4936ffd700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49380adc20 0x7f49381513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49377fe700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4938156c50 0x7f49380c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 -- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49380adc20 msgr2=0x7f49381513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49380adc20 0x7f49381513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 -- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4938156c50 msgr2=0x7f49380c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4938156c50 0x7f49380c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 -- 172.21.15.50:0/541953820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930009a20 con 0x7f4938156880 2023-12-19T14:54:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f49367fc700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f492c00ec80 tx=0x7f492c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c00c850 con 0x7f4938156880 2023-12-19T14:54:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c00c9b0 con 0x7f4938156880 2023-12-19T14:54:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493815bb60 con 0x7f4938156880 2023-12-19T14:54:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c00cd50 con 0x7f4938156880 2023-12-19T14:54:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.255+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49380af5e0 con 0x7f4938156880 2023-12-19T14:54:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.259+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493815b6e0 con 0x7f4938156880 2023-12-19T14:54:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.259+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f492c013680 con 0x7f4938156880 2023-12-19T14:54:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.259+0000 7f491effd700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 0x7f4924062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.263+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f492c07cd10 con 0x7f4938156880 2023-12-19T14:54:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.263+0000 7f4936ffd700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 0x7f4924062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.263+0000 7f4936ffd700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 0x7f4924062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.267+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f493815b6e0 con 0x7f4938156880 2023-12-19T14:54:14.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.447+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f493815b6e0 con 0x7f4938156880 2023-12-19T14:54:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.447+0000 7f491effd700 1 -- 172.21.15.50:0/541953820 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13386 (secure 0 0 0) 0x7f493815b6e0 con 0x7f4938156880 2023-12-19T14:54:14.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:14.450 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":77,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:11.982136+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/1875488305":"2023-12-20T14:54:08.903509+0000","172.21.15.50:6834/1875488305":"2023-12-20T14:54:08.903509+0000","172.21.15.125:0/1197233029":"2023-12-20T14:54:08.547821+0000","172.21.15.125:0/2607665024":"2023-12-20T14:54:08.547821+0000","172.21.15.125:0/3060160198":"2023-12-20T14:54:08.547821+0000","172.21.15.125:0/190789488":"2023-12-20T14:54:08.547821+0000","172.21.15.125:0/3803250607":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/1814796945":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/3303833902":"2023-12-20T14:54:08.547821+0000","172.21.15.125:0/2588274817":"2023-12-20T14:52:23.379995+0000","172.21.15.125:6833/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2296181963":"2023-12-20T14:54:08.547821+0000","172.21.15.125:6832/33827":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/3551203859":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/62382090":"2023-12-20T14:52:23.379995+0000","172.21.15.125:0/2430576648":"2023-12-20T14:52:23.379995+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:14.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 msgr2=0x7f4924062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 0x7f4924062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928009040 comp rx=0 tx=0).stop 2023-12-19T14:54:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 msgr2=0x7f49380c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f492c00ec80 tx=0x7f492c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 shutdown_connections 2023-12-19T14:54:14.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4938156c50 0x7f49380c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4924060670 0x7f4924062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4938156880 0x7f49380c1710 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 --2- 172.21.15.50:0/541953820 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49380adc20 0x7f49381513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 >> 172.21.15.50:0/541953820 conn(0x7f4938067bb0 msgr2=0x7f4938152250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 shutdown_connections 2023-12-19T14:54:14.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.451+0000 7f493d345700 1 -- 172.21.15.50:0/541953820 wait complete. 2023-12-19T14:54:14.473 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:54:14.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.643+0000 7f22e568e700 1 Processor -- start 2023-12-19T14:54:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.643+0000 7f22e568e700 1 -- start start 2023-12-19T14:54:14.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e568e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22e00c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e568e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e006d970 con 0x7f22e006de00 2023-12-19T14:54:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e568e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e006cd60 con 0x7f22e00c69a0 2023-12-19T14:54:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e568e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e006d040 con 0x7f22e006c7c0 2023-12-19T14:54:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22df7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f22e006de00 0x7f22e015cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48114/0 (socket says 172.21.15.50:48114) 2023-12-19T14:54:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22df7fe700 1 -- 172.21.15.50:0/1719814369 learned_addr learned my addr 172.21.15.50:0/1719814369 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22e00c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766061763 0 0) 0x7f22e006d970 con 0x7f22e006de00 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c8003570 con 0x7f22e006de00 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850945854 0 0) 0x7f22e006cd60 con 0x7f22e00c69a0 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e006d970 con 0x7f22e00c69a0 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485057870 0 0) 0x7f22c8003570 con 0x7f22e006de00 2023-12-19T14:54:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 msgr2=0x7f22e00c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22e00c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 >> v1:172.21.15.125:6789/0 conn(0x7f22e00c69a0 legacy=0x7f22e006b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e006d320 con 0x7f22e006de00 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f22d4001470 con 0x7f22e006de00 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 >> v1:172.21.15.50:6789/0 conn(0x7f22e006de00 legacy=0x7f22e015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22c8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22c8004180 0x7f22c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22c8004d50 0x7f22c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e006d040 con 0x7f22c8004d50 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c800bba0 con 0x7f22c8004180 2023-12-19T14:54:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c800bd60 con 0x7f22e006c7c0 2023-12-19T14:54:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22c8004d50 0x7f22c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22dffff700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22c8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.647+0000 7f22df7fe700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22c8004180 0x7f22c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 -- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 msgr2=0x7f22c8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22c8003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 -- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22c8004180 msgr2=0x7f22c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22c8004180 0x7f22c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 -- 172.21.15.50:0/1719814369 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c8003570 con 0x7f22c8004d50 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22dffff700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22e006c7c0 0x7f22c8003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22c8004d50 0x7f22c80051c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f22d800e210 tx=0x7f22d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd444f5f6a60438f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d800eac0 con 0x7f22c8004d50 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d800f040 con 0x7f22c8004d50 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22deffd700 1 -- 172.21.15.50:0/1719814369 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d800ee70 con 0x7f22c8004d50 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 -- 172.21.15.50:0/1719814369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22c8004d50 msgr2=0x7f22c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 --2- 172.21.15.50:0/1719814369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22c8004d50 0x7f22c80051c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f22d800e210 tx=0x7f22d800b040 comp rx=0 tx=0).stop 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e4e8d700 1 -- 172.21.15.50:0/1719814369 reap_dead start 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 -- 172.21.15.50:0/1719814369 shutdown_connections 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 -- 172.21.15.50:0/1719814369 >> 172.21.15.50:0/1719814369 conn(0x7f22e0067b50 msgr2=0x7f22e00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 -- 172.21.15.50:0/1719814369 shutdown_connections 2023-12-19T14:54:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.651+0000 7f22e568e700 1 -- 172.21.15.50:0/1719814369 wait complete. 2023-12-19T14:54:14.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 Processor -- start 2023-12-19T14:54:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 -- start start 2023-12-19T14:54:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 0x7f22e0170d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22e00b0290 0x7f22e0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00c1730 con 0x7f22e015fec0 2023-12-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0171550 con 0x7f22e00b0290 2023-12-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22e568e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0171830 con 0x7f22c8004d50 2023-12-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22df7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22df7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50708/0 (socket says 172.21.15.50:50708) 2023-12-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.655+0000 7f22df7fe700 1 -- 172.21.15.50:0/4064503912 learned_addr learned my addr 172.21.15.50:0/4064503912 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22dffff700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 0x7f22e0170d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22e4e8d700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22e00b0290 0x7f22e0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 -- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 msgr2=0x7f22e0170d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 0x7f22e0170d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 -- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22e00b0290 msgr2=0x7f22e0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22e00b0290 0x7f22e0159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 -- 172.21.15.50:0/4064503912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0171b10 con 0x7f22e015fec0 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22dffff700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 0x7f22e0170d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22df7fe700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f22d400e830 tx=0x7f22d4001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4012500 con 0x7f22e015fec0 2023-12-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4010040 con 0x7f22e015fec0 2023-12-19T14:54:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d40128b0 con 0x7f22e015fec0 2023-12-19T14:54:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e015e310 con 0x7f22e015fec0 2023-12-19T14:54:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e0171240 con 0x7f22e015fec0 2023-12-19T14:54:14.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.659+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e004be40 con 0x7f22e015fec0 2023-12-19T14:54:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.663+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f22d4012c20 con 0x7f22e015fec0 2023-12-19T14:54:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.663+0000 7f22c7fff700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 0x7f22cc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.663+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f22d407bfa0 con 0x7f22e015fec0 2023-12-19T14:54:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.663+0000 7f22dffff700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 0x7f22cc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.667+0000 7f22dffff700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 0x7f22cc062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000f60 tx=0x7f22d0009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.671+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f22d404fca0 con 0x7f22e015fec0 2023-12-19T14:54:14.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.847+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22e006d600 con 0x7f22e015fec0 2023-12-19T14:54:14.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.851+0000 7f22c7fff700 1 -- 172.21.15.50:0/4064503912 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22d4019020 con 0x7f22e015fec0 2023-12-19T14:54:14.852 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 msgr2=0x7f22cc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 0x7f22cc062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22d0000f60 tx=0x7f22d0009040 comp rx=0 tx=0).stop 2023-12-19T14:54:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 msgr2=0x7f22e00c2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f22d400e830 tx=0x7f22d4001010 comp rx=0 tx=0).stop 2023-12-19T14:54:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 shutdown_connections 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22e00b0290 0x7f22e0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f22cc0606a0 0x7f22cc062b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22e015fec0 0x7f22e00c2af0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 --2- 172.21.15.50:0/4064503912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22c8004d50 0x7f22e0170d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 >> 172.21.15.50:0/4064503912 conn(0x7f22e0067b50 msgr2=0x7f22e015b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 shutdown_connections 2023-12-19T14:54:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:14.855+0000 7f22e568e700 1 -- 172.21.15.50:0/4064503912 wait complete. 2023-12-19T14:54:14.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-12-19T14:54:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 Processor -- start 2023-12-19T14:54:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 -- start start 2023-12-19T14:54:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 0x7ff6e006a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006c7c0 con 0x7ff6e00b1410 2023-12-19T14:54:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006cd80 con 0x7ff6e006d210 2023-12-19T14:54:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.039+0000 7ff6df59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e006af10 con 0x7ff6e006a580 2023-12-19T14:54:15.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6de59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff6e00b1410 0x7ff6e00ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48116/0 (socket says 172.21.15.50:48116) 2023-12-19T14:54:15.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6de59c700 1 -- 172.21.15.50:0/2798039160 learned_addr learned my addr 172.21.15.50:0/2798039160 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:15.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6d7fff700 1 --2- 172.21.15.50:0/2798039160 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 0x7ff6e006a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6ddd9b700 1 -- 172.21.15.50:0/2798039160 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812737807 0 0) 0x7ff6e006c7c0 con 0x7ff6e00b1410 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6ddd9b700 1 -- 172.21.15.50:0/2798039160 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc003570 con 0x7ff6e00b1410 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6d7fff700 1 -- 172.21.15.50:0/2798039160 >> v1:172.21.15.125:6789/0 conn(0x7ff6e006d210 legacy=0x7ff6e006d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6d7fff700 1 -- 172.21.15.50:0/2798039160 >> v1:172.21.15.50:6789/0 conn(0x7ff6e00b1410 legacy=0x7ff6e00ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6d7fff700 1 -- 172.21.15.50:0/2798039160 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e006b8f0 con 0x7ff6e006a580 2023-12-19T14:54:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6d7fff700 1 --2- 172.21.15.50:0/2798039160 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 0x7ff6e006a9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009910 tx=0x7ff6c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb6eb811d66c9cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6ddd9b700 1 -- 172.21.15.50:0/2798039160 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800b680 con 0x7ff6e006a580 2023-12-19T14:54:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6ddd9b700 1 -- 172.21.15.50:0/2798039160 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c800b7e0 con 0x7ff6e006a580 2023-12-19T14:54:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6ddd9b700 1 -- 172.21.15.50:0/2798039160 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800bba0 con 0x7ff6e006a580 2023-12-19T14:54:15.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 -- 172.21.15.50:0/2798039160 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 msgr2=0x7ff6e006a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:15.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 --2- 172.21.15.50:0/2798039160 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 0x7ff6e006a9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009910 tx=0x7ff6c80091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 -- 172.21.15.50:0/2798039160 shutdown_connections 2023-12-19T14:54:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 --2- 172.21.15.50:0/2798039160 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e006a580 0x7ff6e006a9d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 -- 172.21.15.50:0/2798039160 >> 172.21.15.50:0/2798039160 conn(0x7ff6e0067bb0 msgr2=0x7ff6e00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 -- 172.21.15.50:0/2798039160 shutdown_connections 2023-12-19T14:54:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.043+0000 7ff6df59e700 1 -- 172.21.15.50:0/2798039160 wait complete. 2023-12-19T14:54:15.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 Processor -- start 2023-12-19T14:54:15.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 -- start start 2023-12-19T14:54:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6e006d210 0x7ff6e015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e00b1410 0x7ff6e015c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e015a5b0 con 0x7ff6e006a580 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e00c70b0 con 0x7ff6e006d210 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6df59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e00c7540 con 0x7ff6e00b1410 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.047+0000 7ff6de59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50712/0 (socket says 172.21.15.50:50712) 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 -- 172.21.15.50:0/918361983 learned_addr learned my addr 172.21.15.50:0/918361983 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6d7fff700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6e006d210 0x7ff6e015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6ded9d700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e00b1410 0x7ff6e015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 -- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e00b1410 msgr2=0x7ff6e015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e00b1410 0x7ff6e015c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 -- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6e006d210 msgr2=0x7ff6e015bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6e006d210 0x7ff6e015bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 -- 172.21.15.50:0/918361983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c8009420 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6de59c700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c5c0 tx=0x7ff6d0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0005b30 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d0005cb0 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e00c6780 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0007320 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e00c6d40 con 0x7ff6e006a580 2023-12-19T14:54:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.051+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e004be40 con 0x7ff6e006a580 2023-12-19T14:54:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.055+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff6d001b3b0 con 0x7ff6e006a580 2023-12-19T14:54:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.055+0000 7ff6d67fc700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 0x7ff6c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.055+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7ff6d007bf30 con 0x7ff6e006a580 2023-12-19T14:54:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.055+0000 7ff6d7fff700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 0x7ff6c0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.059+0000 7ff6d7fff700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 0x7ff6c0062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80093f0 tx=0x7ff6c8017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.063+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff6d004c2b0 con 0x7ff6e006a580 2023-12-19T14:54:15.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:15.295+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7ff6e00ab220 con 0x7ff6e006a580 2023-12-19T14:54:16.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.019+0000 7ff6d67fc700 1 -- 172.21.15.50:0/918361983 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v557) v1 ==== 159+0+0 (secure 0 0 0) 0x7ff6d00170e0 con 0x7ff6e006a580 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 msgr2=0x7ff6c0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 0x7ff6c0062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80093f0 tx=0x7ff6c8017000 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 msgr2=0x7ff6e0157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c5c0 tx=0x7ff6d0007e00 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 shutdown_connections 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6e006d210 0x7ff6e015bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c0060670 0x7ff6c0062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6e00b1410 0x7ff6e015c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 --2- 172.21.15.50:0/918361983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e006a580 0x7ff6e0157400 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.023+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 >> 172.21.15.50:0/918361983 conn(0x7ff6e0067bb0 msgr2=0x7ff6e0156290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.027+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 shutdown_connections 2023-12-19T14:54:16.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.027+0000 7ff6df59e700 1 -- 172.21.15.50:0/918361983 wait complete. 2023-12-19T14:54:16.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:54:16.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.223+0000 7fc67c5c5700 1 Processor -- start 2023-12-19T14:54:16.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 -- start start 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc67406db40 0x7fc6740cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc674159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6740c9db0 con 0x7fc6740c63f0 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6740ca950 con 0x7fc67406db40 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67c5c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740cb2e0 con 0x7fc67406df10 2023-12-19T14:54:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.227+0000 7fc67a361700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc67406db40 0x7fc6740cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc679b60700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc6740c63f0 0x7fc6740ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33002/0 (socket says 172.21.15.50:33002) 2023-12-19T14:54:16.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc679b60700 1 -- 172.21.15.50:0/1517680395 learned_addr learned my addr 172.21.15.50:0/1517680395 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:16.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67ab62700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc674159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67935f700 1 -- 172.21.15.50:0/1517680395 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097798308 0 0) 0x7fc6740c9db0 con 0x7fc6740c63f0 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67935f700 1 -- 172.21.15.50:0/1517680395 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc660003570 con 0x7fc6740c63f0 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67a361700 1 -- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 msgr2=0x7fc674159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67a361700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc674159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67a361700 1 -- 172.21.15.50:0/1517680395 >> v1:172.21.15.125:6789/0 conn(0x7fc6740c63f0 legacy=0x7fc6740ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67a361700 1 -- 172.21.15.50:0/1517680395 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740cb5c0 con 0x7fc67406db40 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67ab62700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc674159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67a361700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740cc3e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc6700098e0 tx=0x7fc6700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4c0125d88eef4b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67935f700 1 -- 172.21.15.50:0/1517680395 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670011630 con 0x7fc67406db40 2023-12-19T14:54:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67935f700 1 -- 172.21.15.50:0/1517680395 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670011790 con 0x7fc67406db40 2023-12-19T14:54:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67935f700 1 -- 172.21.15.50:0/1517680395 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670011b30 con 0x7fc67406db40 2023-12-19T14:54:16.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67c5c5700 1 -- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 msgr2=0x7fc6740cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.231+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740cc3e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc6700098e0 tx=0x7fc6700091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- 172.21.15.50:0/1517680395 shutdown_connections 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc674159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/1517680395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740cc3e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- 172.21.15.50:0/1517680395 >> 172.21.15.50:0/1517680395 conn(0x7fc674067bb0 msgr2=0x7fc6740cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- 172.21.15.50:0/1517680395 shutdown_connections 2023-12-19T14:54:16.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- 172.21.15.50:0/1517680395 wait complete. 2023-12-19T14:54:16.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 Processor -- start 2023-12-19T14:54:16.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- start start 2023-12-19T14:54:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc6740b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740aeba0 con 0x7fc6740c63f0 2023-12-19T14:54:16.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740aed80 con 0x7fc67406db40 2023-12-19T14:54:16.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.235+0000 7fc67c5c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740af060 con 0x7fc67406df10 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67a361700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50716/0 (socket says 172.21.15.50:50716) 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 -- 172.21.15.50:0/734116536 learned_addr learned my addr 172.21.15.50:0/734116536 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc679b60700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc6740b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 -- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 msgr2=0x7fc6740b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc6740b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 -- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 msgr2=0x7fc6740c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 -- 172.21.15.50:0/734116536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c009a20 con 0x7fc6740c63f0 2023-12-19T14:54:16.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc679b60700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc6740b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:16.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67ab62700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00ef90 tx=0x7fc66c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c00cc80 con 0x7fc6740c63f0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c00f040 con 0x7fc6740c63f0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c014480 con 0x7fc6740c63f0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc670009420 con 0x7fc6740c63f0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6740af620 con 0x7fc6740c63f0 2023-12-19T14:54:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.239+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67406b780 con 0x7fc6740c63f0 2023-12-19T14:54:16.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.243+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc66c0148b0 con 0x7fc6740c63f0 2023-12-19T14:54:16.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.243+0000 7fc66a7fc700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 0x7fc658062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.243+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7fc66c07d8e0 con 0x7fc6740c63f0 2023-12-19T14:54:16.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.243+0000 7fc67a361700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 0x7fc658062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.247+0000 7fc67a361700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 0x7fc658062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc6700091e0 tx=0x7fc670017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.251+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc67406b780 con 0x7fc6740c63f0 2023-12-19T14:54:16.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.419+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc6740af840 con 0x7fc6740c63f0 2023-12-19T14:54:16.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.427+0000 7fc66a7fc700 1 -- 172.21.15.50:0/734116536 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc66c018020 con 0x7fc6740c63f0 2023-12-19T14:54:16.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 msgr2=0x7fc658062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 0x7fc658062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc6700091e0 tx=0x7fc670017000 comp rx=0 tx=0).stop 2023-12-19T14:54:16.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 msgr2=0x7fc67413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00ef90 tx=0x7fc66c00b040 comp rx=0 tx=0).stop 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 shutdown_connections 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc658060670 0x7fc658062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6740c63f0 0x7fc67413f920 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc67406df10 0x7fc6740b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 --2- 172.21.15.50:0/734116536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc67406db40 0x7fc6740c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 >> 172.21.15.50:0/734116536 conn(0x7fc674067bb0 msgr2=0x7fc674146030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 shutdown_connections 2023-12-19T14:54:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.431+0000 7fc67c5c5700 1 -- 172.21.15.50:0/734116536 wait complete. 2023-12-19T14:54:16.457 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:54:16.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:16.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:16.548 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:54:16.548 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:54:16.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:16.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.715+0000 7f77af0ae700 1 Processor -- start 2023-12-19T14:54:16.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.715+0000 7f77af0ae700 1 -- start start 2023-12-19T14:54:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77af0ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77a8140ae0 0x7f77a80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77af0ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8068630 0x7f77a80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77af0ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a806e150 con 0x7f77a8140710 2023-12-19T14:54:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77af0ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a8068ab0 con 0x7f77a8140ae0 2023-12-19T14:54:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77af0ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a806d540 con 0x7f77a8068630 2023-12-19T14:54:16.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77a7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77a8140ae0 0x7f77a80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77a7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77a8140ae0 0x7f77a80680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41166/0 (socket says 172.21.15.50:41166) 2023-12-19T14:54:16.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77a7fff700 1 -- 172.21.15.50:0/2435753137 learned_addr learned my addr 172.21.15.50:0/2435753137 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:16.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.719+0000 7f77ad64b700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8068630 0x7f77a80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a7fff700 1 -- 172.21.15.50:0/2435753137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8068630 msgr2=0x7f77a80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a7fff700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8068630 0x7f77a80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a7fff700 1 -- 172.21.15.50:0/2435753137 >> v1:172.21.15.125:6789/0 conn(0x7f77a8140710 legacy=0x7f77a80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a7fff700 1 -- 172.21.15.50:0/2435753137 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a806d820 con 0x7f77a8140ae0 2023-12-19T14:54:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a7fff700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80680f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f77940099b0 tx=0x7f7794009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6110b2c31a4f212d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a77fe700 1 -- 172.21.15.50:0/2435753137 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779400b860 con 0x7f77a8140ae0 2023-12-19T14:54:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a77fe700 1 -- 172.21.15.50:0/2435753137 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779400b9c0 con 0x7f77a8140ae0 2023-12-19T14:54:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77a77fe700 1 -- 172.21.15.50:0/2435753137 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779400bd60 con 0x7f77a8140ae0 2023-12-19T14:54:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 -- 172.21.15.50:0/2435753137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 msgr2=0x7f77a80680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80680f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f77940099b0 tx=0x7f7794009250 comp rx=0 tx=0).stop 2023-12-19T14:54:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 -- 172.21.15.50:0/2435753137 shutdown_connections 2023-12-19T14:54:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8068630 0x7f77a80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 --2- 172.21.15.50:0/2435753137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80680f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 -- 172.21.15.50:0/2435753137 >> 172.21.15.50:0/2435753137 conn(0x7f77a80c8850 msgr2=0x7f77a80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 -- 172.21.15.50:0/2435753137 shutdown_connections 2023-12-19T14:54:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.723+0000 7f77af0ae700 1 -- 172.21.15.50:0/2435753137 wait complete. 2023-12-19T14:54:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 Processor -- start 2023-12-19T14:54:16.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 -- start start 2023-12-19T14:54:16.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a8068630 0x7f77a80c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8140710 0x7f77a80aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80af0b0 con 0x7f77a8140ae0 2023-12-19T14:54:16.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80afed0 con 0x7f77a8068630 2023-12-19T14:54:16.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77af0ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80af2c0 con 0x7f77a8140710 2023-12-19T14:54:16.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41180/0 (socket says 172.21.15.50:41180) 2023-12-19T14:54:16.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 -- 172.21.15.50:0/3347474674 learned_addr learned my addr 172.21.15.50:0/3347474674 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:16.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77a7fff700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8140710 0x7f77a80aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ace4a700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a8068630 0x7f77a80c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 -- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8140710 msgr2=0x7f77a80aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8140710 0x7f77a80aa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 -- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a8068630 msgr2=0x7f77a80c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a8068630 0x7f77a80c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.727+0000 7f77ad64b700 1 -- 172.21.15.50:0/3347474674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77940094f0 con 0x7f77a8140ae0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77ad64b700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f77a0007f20 tx=0x7f77a0009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a000eb90 con 0x7f77a8140ae0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a000ed10 con 0x7f77a8140ae0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0014670 con 0x7f77a8140ae0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80af5a0 con 0x7f77a8140ae0 2023-12-19T14:54:16.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a80afb60 con 0x7f77a8140ae0 2023-12-19T14:54:16.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.731+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a80b0060 con 0x7f77a8140ae0 2023-12-19T14:54:16.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.735+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f77a00149e0 con 0x7f77a8140ae0 2023-12-19T14:54:16.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.735+0000 7f77a4ff9700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 0x7f778c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:16.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.735+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f77a007cff0 con 0x7f77a8140ae0 2023-12-19T14:54:16.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.735+0000 7f77ace4a700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 0x7f778c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:16.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.739+0000 7f77ace4a700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 0x7f778c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f779c0039e0 tx=0x7f779c007ec0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:16.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.743+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f77a004d3d0 con 0x7f77a8140ae0 2023-12-19T14:54:16.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.951+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77a8071120 con 0x7f77a8140ae0 2023-12-19T14:54:16.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.951+0000 7f77a4ff9700 1 -- 172.21.15.50:0/3347474674 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f77a00509f0 con 0x7f77a8140ae0 2023-12-19T14:54:16.953 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:16.953 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/1101930703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1101930703},{"type":"v1","addr":"172.21.15.125:6835","nonce":1101930703}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5851,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3901818181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":3901818181},{"type":"v1","addr":"172.21.15.125:6837","nonce":3901818181}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/2362382021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2362382021},{"type":"v1","addr":"172.21.15.50:6833","nonce":2362382021}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6390,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2204024131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2204024131},{"type":"v1","addr":"172.21.15.50:6835","nonce":2204024131}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2023-12-19T14:54:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 msgr2=0x7f778c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 0x7f778c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f779c0039e0 tx=0x7f779c007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 msgr2=0x7f77a80aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f77a0007f20 tx=0x7f77a0009860 comp rx=0 tx=0).stop 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 shutdown_connections 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f778c060670 0x7f778c062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77a8140ae0 0x7f77a80aada0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77a8140710 0x7f77a80aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 --2- 172.21.15.50:0/3347474674 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77a8068630 0x7f77a80c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 >> 172.21.15.50:0/3347474674 conn(0x7f77a80c8850 msgr2=0x7f77a80ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 shutdown_connections 2023-12-19T14:54:16.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:16.955+0000 7f77af0ae700 1 -- 172.21.15.50:0/3347474674 wait complete. 2023-12-19T14:54:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 27 2023-12-19T14:54:16.980 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:54:16.980 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:54:16.980 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:54:16.980 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:16.980 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:54:16.980 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:54:16.981 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:54:16.981 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:16.981 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:54:16.981 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:54:16.981 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:54:16.981 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:16.981 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:54:16.981 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:54:16.982 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:54:16.982 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:17.000 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:54:17.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:54:17.004 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:54:17.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:54:17.013 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:54:17.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:54:17.014 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:54:17.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:54:17.052 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:54:17.052 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:17.056 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:54:17.056 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:17.064 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:54:17.064 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:17.067 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:54:17.067 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:17.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:54:17.160 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:54:17.164 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:54:17.174 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:54:17.175 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:54:17.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-12-19T14:54:17.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-12-19T14:54:17.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-12-19T14:54:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.239+0000 7f82638cd700 1 Processor -- start 2023-12-19T14:54:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.239+0000 7f82638cd700 1 -- start start 2023-12-19T14:54:17.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f82638cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c139d50 0x7f825c0cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f82638cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f825c13a120 0x7f825c0b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f82638cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0cc430 con 0x7f825c0b25f0 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f82638cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c0b2a70 con 0x7f825c13a120 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f82638cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c0cce00 con 0x7f825c139d50 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f8260e68700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f825c13a120 0x7f825c0b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f8260e68700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f825c13a120 0x7f825c0b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41222/0 (socket says 172.21.15.50:41222) 2023-12-19T14:54:17.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f8260e68700 1 -- 172.21.15.50:0/3197216772 learned_addr learned my addr 172.21.15.50:0/3197216772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.243+0000 7f8261669700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c139d50 0x7f825c0cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8260e68700 1 -- 172.21.15.50:0/3197216772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c139d50 msgr2=0x7f825c0cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8260e68700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c139d50 0x7f825c0cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8260e68700 1 -- 172.21.15.50:0/3197216772 >> v1:172.21.15.125:6789/0 conn(0x7f825c0b25f0 legacy=0x7f825c15ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8253fff700 1 -- 172.21.15.50:0/3197216772 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72490907 0 0) 0x7f825c0cc430 con 0x7f825c0b25f0 2023-12-19T14:54:17.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8260e68700 1 -- 172.21.15.50:0/3197216772 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c15bad0 con 0x7f825c13a120 2023-12-19T14:54:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8260e68700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c13a120 0x7f825c0b20b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f824c0099b0 tx=0x7f824c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eccd2f557294ec5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8253fff700 1 -- 172.21.15.50:0/3197216772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c009ea0 con 0x7f825c13a120 2023-12-19T14:54:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8253fff700 1 -- 172.21.15.50:0/3197216772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c00b860 con 0x7f825c13a120 2023-12-19T14:54:17.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f8253fff700 1 -- 172.21.15.50:0/3197216772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c00bc00 con 0x7f825c13a120 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 -- 172.21.15.50:0/3197216772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c13a120 msgr2=0x7f825c0b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c13a120 0x7f825c0b20b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f824c0099b0 tx=0x7f824c009250 comp rx=0 tx=0).stop 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 -- 172.21.15.50:0/3197216772 shutdown_connections 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c13a120 0x7f825c0b20b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 --2- 172.21.15.50:0/3197216772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c139d50 0x7f825c0cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 -- 172.21.15.50:0/3197216772 >> 172.21.15.50:0/3197216772 conn(0x7f825c069310 msgr2=0x7f825c0696e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 -- 172.21.15.50:0/3197216772 shutdown_connections 2023-12-19T14:54:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.247+0000 7f82638cd700 1 -- 172.21.15.50:0/3197216772 wait complete. 2023-12-19T14:54:17.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.251+0000 7f82638cd700 1 Processor -- start 2023-12-19T14:54:17.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.251+0000 7f82638cd700 1 -- start start 2023-12-19T14:54:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f825c139d50 0x7f825c0682f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 0x7f825c068830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c15a220 con 0x7f825c0b25f0 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c15b040 con 0x7f825c139d50 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c137280 con 0x7f825c13a120 2023-12-19T14:54:17.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41234/0 (socket says 172.21.15.50:41234) 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 -- 172.21.15.50:0/192374841 learned_addr learned my addr 172.21.15.50:0/192374841 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8260e68700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f825c139d50 0x7f825c0682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261e6a700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 0x7f825c068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 -- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 msgr2=0x7f825c068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 0x7f825c068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 -- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f825c139d50 msgr2=0x7f825c0682f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f825c139d50 0x7f825c0682f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 -- 172.21.15.50:0/192374841 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c0094f0 con 0x7f825c0b25f0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261e6a700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 0x7f825c068830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f8261669700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f825800ecb0 tx=0x7f82580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825800c9d0 con 0x7f825c0b25f0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825800cb30 con 0x7f825c0b25f0 2023-12-19T14:54:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c15a430 con 0x7f825c0b25f0 2023-12-19T14:54:17.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825800ced0 con 0x7f825c0b25f0 2023-12-19T14:54:17.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.255+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c15a9f0 con 0x7f825c0b25f0 2023-12-19T14:54:17.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.259+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c071120 con 0x7f825c0b25f0 2023-12-19T14:54:17.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.259+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8258013680 con 0x7f825c0b25f0 2023-12-19T14:54:17.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.263+0000 7f82517fa700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 0x7f8240062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:17.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.263+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5889+0+0 (secure 0 0 0) 0x7f825807cd90 con 0x7f825c0b25f0 2023-12-19T14:54:17.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.263+0000 7f8260e68700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 0x7f8240062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:17.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.263+0000 7f8260e68700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 0x7f8240062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f824c0099b0 tx=0x7f824c017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:17.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.267+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f825804d1b0 con 0x7f825c0b25f0 2023-12-19T14:54:17.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:17.451+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f825c0cbb00 con 0x7f825c0b25f0 2023-12-19T14:54:18.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.459+0000 7f82517fa700 1 -- 172.21.15.50:0/192374841 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) v1 ==== 97+0+0 (secure 0 0 0) 0x7f82580507d0 con 0x7f825c0b25f0 2023-12-19T14:54:18.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.459+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 msgr2=0x7f8240062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.459+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 0x7f8240062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f824c0099b0 tx=0x7f824c017000 comp rx=0 tx=0).stop 2023-12-19T14:54:18.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.459+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 msgr2=0x7f825c136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.459+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f825800ecb0 tx=0x7f82580091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 shutdown_connections 2023-12-19T14:54:18.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f825c139d50 0x7f825c0682f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8240060670 0x7f8240062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f825c13a120 0x7f825c068830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 --2- 172.21.15.50:0/192374841 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c0b25f0 0x7f825c136d40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 >> 172.21.15.50:0/192374841 conn(0x7f825c069310 msgr2=0x7f825c14ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:18.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 shutdown_connections 2023-12-19T14:54:18.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.463+0000 7f82638cd700 1 -- 172.21.15.50:0/192374841 wait complete. 2023-12-19T14:54:18.466 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:54:18.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:54:18.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 Processor -- start 2023-12-19T14:54:18.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 -- start start 2023-12-19T14:54:18.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 0x7f4c84067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:18.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c84068c70 con 0x7f4c84159340 2023-12-19T14:54:18.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c84068060 con 0x7f4c84158f70 2023-12-19T14:54:18.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8c078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84068340 con 0x7f4c84069100 2023-12-19T14:54:18.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8a615700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4c84159340 0x7f4c840cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43806/0 (socket says 172.21.15.50:43806) 2023-12-19T14:54:18.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.659+0000 7f4c8a615700 1 -- 172.21.15.50:0/2333850262 learned_addr learned my addr 172.21.15.50:0/2333850262 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:18.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c89613700 1 --2- 172.21.15.50:0/2333850262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 0x7f4c84067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402214102 0 0) 0x7f4c84068c70 con 0x7f4c84159340 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c70003570 con 0x7f4c84159340 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932531852 0 0) 0x7f4c84068060 con 0x7f4c84158f70 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c84068c70 con 0x7f4c84158f70 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c89613700 1 -- 172.21.15.50:0/2333850262 >> v1:172.21.15.125:6789/0 conn(0x7f4c84158f70 legacy=0x7f4c840cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c89613700 1 -- 172.21.15.50:0/2333850262 >> v1:172.21.15.50:6789/0 conn(0x7f4c84159340 legacy=0x7f4c840cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c89613700 1 -- 172.21.15.50:0/2333850262 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c84068620 con 0x7f4c84069100 2023-12-19T14:54:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c89613700 1 --2- 172.21.15.50:0/2333850262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 0x7f4c84067980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009910 tx=0x7f4c740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3bda6d2e133ac16 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74011630 con 0x7f4c84069100 2023-12-19T14:54:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74011790 con 0x7f4c84069100 2023-12-19T14:54:18.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c88e12700 1 -- 172.21.15.50:0/2333850262 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74011b30 con 0x7f4c84069100 2023-12-19T14:54:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c8c078700 1 -- 172.21.15.50:0/2333850262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 msgr2=0x7f4c84067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c8c078700 1 --2- 172.21.15.50:0/2333850262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 0x7f4c84067980 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009910 tx=0x7f4c740091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c8c078700 1 -- 172.21.15.50:0/2333850262 shutdown_connections 2023-12-19T14:54:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c8c078700 1 --2- 172.21.15.50:0/2333850262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84069100 0x7f4c84067980 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.663+0000 7f4c8c078700 1 -- 172.21.15.50:0/2333850262 >> 172.21.15.50:0/2333850262 conn(0x7f4c840ccc40 msgr2=0x7f4c840cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:18.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.667+0000 7f4c8c078700 1 -- 172.21.15.50:0/2333850262 shutdown_connections 2023-12-19T14:54:18.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.667+0000 7f4c8c078700 1 -- 172.21.15.50:0/2333850262 wait complete. 2023-12-19T14:54:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.667+0000 7f4c8c078700 1 Processor -- start 2023-12-19T14:54:18.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.667+0000 7f4c8c078700 1 -- start start 2023-12-19T14:54:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 0x7f4c8414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84159340 0x7f4c8414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c840b1770 con 0x7f4c84069100 2023-12-19T14:54:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8416a6c0 con 0x7f4c84158f70 2023-12-19T14:54:18.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84169880 con 0x7f4c84159340 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41240/0 (socket says 172.21.15.50:41240) 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89613700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 0x7f4c8414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 -- 172.21.15.50:0/1374047625 learned_addr learned my addr 172.21.15.50:0/1374047625 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8a615700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84159340 0x7f4c8414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 -- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84159340 msgr2=0x7f4c8414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84159340 0x7f4c8414cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 -- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 msgr2=0x7f4c8414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 0x7f4c8414c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 -- 172.21.15.50:0/1374047625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74009420 con 0x7f4c84069100 2023-12-19T14:54:18.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89613700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 0x7f4c8414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c89e14700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000c790 tx=0x7f4c80007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80010510 con 0x7f4c84069100 2023-12-19T14:54:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c80010670 con 0x7f4c84069100 2023-12-19T14:54:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c80010a10 con 0x7f4c84069100 2023-12-19T14:54:18.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.671+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c84169ab0 con 0x7f4c84069100 2023-12-19T14:54:18.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.675+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8416a070 con 0x7f4c84069100 2023-12-19T14:54:18.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.675+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c84071120 con 0x7f4c84069100 2023-12-19T14:54:18.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.675+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4c8001b3b0 con 0x7f4c84069100 2023-12-19T14:54:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.679+0000 7f4c79ffb700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 0x7f4c68062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:18.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.679+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f4c8007bff0 con 0x7f4c84069100 2023-12-19T14:54:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.679+0000 7f4c89613700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 0x7f4c68062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.679+0000 7f4c89613700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 0x7f4c68062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009860 tx=0x7f4c74017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:18.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.683+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4c8004c6c0 con 0x7f4c84069100 2023-12-19T14:54:18.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.923+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4c84068900 con 0x7f4c84069100 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c79ffb700 1 -- 172.21.15.50:0/1374047625 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4c80017020 con 0x7f4c84069100 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.927 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:18.928 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:54:18.930 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.933 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:54:18.937 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:18.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:54:18.939 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:18.940 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:54:18.941 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:54:18.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 msgr2=0x7f4c68062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 0x7f4c68062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009860 tx=0x7f4c74017000 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 msgr2=0x7f4c84169340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000c790 tx=0x7f4c80007e10 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 shutdown_connections 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c84158f70 0x7f4c8414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4c68060670 0x7f4c68062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c84159340 0x7f4c8414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 --2- 172.21.15.50:0/1374047625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c84069100 0x7f4c84169340 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 >> 172.21.15.50:0/1374047625 conn(0x7f4c840ccc40 msgr2=0x7f4c8414dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 shutdown_connections 2023-12-19T14:54:18.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:18.927+0000 7f4c8c078700 1 -- 172.21.15.50:0/1374047625 wait complete. 2023-12-19T14:54:18.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:19.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.107+0000 7f34650e3700 1 Processor -- start 2023-12-19T14:54:19.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.107+0000 7f34650e3700 1 -- start start 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3460159160 0x7f3460159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f34600c5fd0 0x7f34600adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 0x7f3460158200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006a7b0 con 0x7f34600c5fd0 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34600ad550 con 0x7f34600ae210 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34650e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c6400 con 0x7f3460159160 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.111+0000 7f34577fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f34600c5fd0 0x7f34600adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f34600c5fd0 0x7f34600adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41256/0 (socket says 172.21.15.50:41256) 2023-12-19T14:54:19.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 -- 172.21.15.50:0/2240950283 learned_addr learned my addr 172.21.15.50:0/2240950283 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:19.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34648e2700 1 --2- 172.21.15.50:0/2240950283 >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 0x7f3460158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f345ffff700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3460159160 0x7f3460159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 -- 172.21.15.50:0/2240950283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3460159160 msgr2=0x7f3460159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:19.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3460159160 0x7f3460159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 -- 172.21.15.50:0/2240950283 >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 msgr2=0x7f3460158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 --2- 172.21.15.50:0/2240950283 >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 0x7f3460158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 -- 172.21.15.50:0/2240950283 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600accb0 con 0x7f34600c5fd0 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34648e2700 1 --2- 172.21.15.50:0/2240950283 >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 0x7f3460158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34577fe700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600c5fd0 0x7f34600adcd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f34600ab0f0 tx=0x7f3448009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6ff276e1705067 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f345f7fe700 1 -- 172.21.15.50:0/2240950283 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800b640 con 0x7f34600c5fd0 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f345f7fe700 1 -- 172.21.15.50:0/2240950283 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344800b7a0 con 0x7f34600c5fd0 2023-12-19T14:54:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f345f7fe700 1 -- 172.21.15.50:0/2240950283 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800bb60 con 0x7f34600c5fd0 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 -- 172.21.15.50:0/2240950283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600c5fd0 msgr2=0x7f34600adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600c5fd0 0x7f34600adcd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f34600ab0f0 tx=0x7f3448009250 comp rx=0 tx=0).stop 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 -- 172.21.15.50:0/2240950283 shutdown_connections 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600c5fd0 0x7f34600adcd0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 --2- 172.21.15.50:0/2240950283 >> v2:172.21.15.125:3300/0 conn(0x7f34600ae210 0x7f3460158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 --2- 172.21.15.50:0/2240950283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3460159160 0x7f3460159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 -- 172.21.15.50:0/2240950283 >> 172.21.15.50:0/2240950283 conn(0x7f3460067bb0 msgr2=0x7f34600cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:19.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 -- 172.21.15.50:0/2240950283 shutdown_connections 2023-12-19T14:54:19.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.115+0000 7f34650e3700 1 -- 172.21.15.50:0/2240950283 wait complete. 2023-12-19T14:54:19.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.119+0000 7f34650e3700 1 Processor -- start 2023-12-19T14:54:19.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.119+0000 7f34650e3700 1 -- start start 2023-12-19T14:54:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 0x7f34600c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 0x7f34600a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600caf60 con 0x7f34600ae210 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600aa3a0 con 0x7f3460159160 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600ca350 con 0x7f34600c5fd0 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41258/0 (socket says 172.21.15.50:41258) 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 -- 172.21.15.50:0/2834739921 learned_addr learned my addr 172.21.15.50:0/2834739921 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34648e2700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 0x7f34600a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 -- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 msgr2=0x7f34600c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34577fe700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 0x7f34600c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 0x7f34600c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 -- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 msgr2=0x7f34600a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 0x7f34600a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 -- 172.21.15.50:0/2834739921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009a20 con 0x7f34600ae210 2023-12-19T14:54:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34648e2700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 0x7f34600a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34577fe700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 0x7f34600c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345ffff700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f345000e980 tx=0x7f34500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000c850 con 0x7f34600ae210 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345000c9b0 con 0x7f34600ae210 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000cdb0 con 0x7f34600ae210 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34480094f0 con 0x7f34600ae210 2023-12-19T14:54:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.123+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34600ca910 con 0x7f34600ae210 2023-12-19T14:54:19.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.127+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346004be40 con 0x7f34600ae210 2023-12-19T14:54:19.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.127+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3450013680 con 0x7f34600ae210 2023-12-19T14:54:19.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.127+0000 7f345cff9700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 0x7f3440062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.127+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f345007c940 con 0x7f34600ae210 2023-12-19T14:54:19.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.127+0000 7f34577fe700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 0x7f3440062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.131+0000 7f34577fe700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 0x7f3440062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34600ab0f0 tx=0x7f3448017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.135+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f345004d010 con 0x7f34600ae210 2023-12-19T14:54:19.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.311+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34600acf90 con 0x7f34600ae210 2023-12-19T14:54:19.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.311+0000 7f345cff9700 1 -- 172.21.15.50:0/2834739921 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f3450050630 con 0x7f34600ae210 2023-12-19T14:54:19.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:19.315 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:18.451390+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:19.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 msgr2=0x7f3440062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 0x7f3440062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34600ab0f0 tx=0x7f3448017000 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 msgr2=0x7f34600a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f345000e980 tx=0x7f34500091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 shutdown_connections 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3460159160 0x7f34600a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3440060670 0x7f3440062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34600c5fd0 0x7f34600c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 --2- 172.21.15.50:0/2834739921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34600ae210 0x7f34600a9e60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 >> 172.21.15.50:0/2834739921 conn(0x7f3460067bb0 msgr2=0x7f34600c8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 shutdown_connections 2023-12-19T14:54:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.315+0000 7f34650e3700 1 -- 172.21.15.50:0/2834739921 wait complete. 2023-12-19T14:54:19.339 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:54:19.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.507+0000 7fdea0ce3700 1 Processor -- start 2023-12-19T14:54:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 -- start start 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde9c0c8bf0 0x7fde9c0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c06d390 0x7fde9c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9c0ae110 con 0x7fde9c0ace10 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9c157110 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fdea0ce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c06e0c0 con 0x7fde9c06d390 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fde9c0ace10 0x7fde9c0c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43814/0 (socket says 172.21.15.50:43814) 2023-12-19T14:54:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9b7fe700 1 -- 172.21.15.50:0/4024466401 learned_addr learned my addr 172.21.15.50:0/4024466401 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde93fff700 1 --2- 172.21.15.50:0/4024466401 >> v2:172.21.15.50:3300/0 conn(0x7fde9c0c8bf0 0x7fde9c0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9bfff700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c06d390 0x7fde9c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219339674 0 0) 0x7fde9c0ae110 con 0x7fde9c0ace10 2023-12-19T14:54:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde88003570 con 0x7fde9c0ace10 2023-12-19T14:54:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2817976055 0 0) 0x7fde88003570 con 0x7fde9c0ace10 2023-12-19T14:54:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde93fff700 1 -- 172.21.15.50:0/4024466401 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c06d390 msgr2=0x7fde9c06d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde93fff700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c06d390 0x7fde9c06d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde93fff700 1 -- 172.21.15.50:0/4024466401 >> v1:172.21.15.50:6789/0 conn(0x7fde9c0ace10 legacy=0x7fde9c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.511+0000 7fde93fff700 1 -- 172.21.15.50:0/4024466401 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c0ad7e0 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fde93fff700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c0ad1e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fde840098e0 tx=0x7fde840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881617bfa48c32fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c001470 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde84011420 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fde9affd700 1 -- 172.21.15.50:0/4024466401 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840117c0 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 -- 172.21.15.50:0/4024466401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c0c8bf0 msgr2=0x7fde9c0ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c0ad1e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fde840098e0 tx=0x7fde840091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 -- 172.21.15.50:0/4024466401 shutdown_connections 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c06d390 0x7fde9c06d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/4024466401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c0ad1e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 -- 172.21.15.50:0/4024466401 >> 172.21.15.50:0/4024466401 conn(0x7fde9c0cdc00 msgr2=0x7fde9c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 -- 172.21.15.50:0/4024466401 shutdown_connections 2023-12-19T14:54:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.515+0000 7fdea0ce3700 1 -- 172.21.15.50:0/4024466401 wait complete. 2023-12-19T14:54:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 Processor -- start 2023-12-19T14:54:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 -- start start 2023-12-19T14:54:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c0ace10 0x7fde9c0b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c15b580 con 0x7fde9c06d390 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c15a970 con 0x7fde9c0c8bf0 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fdea0ce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c15ac50 con 0x7fde9c0ace10 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41276/0 (socket says 172.21.15.50:41276) 2023-12-19T14:54:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 -- 172.21.15.50:0/3271534418 learned_addr learned my addr 172.21.15.50:0/3271534418 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9bfff700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde93fff700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c0ace10 0x7fde9c0b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 -- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c0ace10 msgr2=0x7fde9c0b1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c0ace10 0x7fde9c0b1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 -- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde9c0c8bf0 msgr2=0x7fde9c14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.519+0000 7fde9b7fe700 1 -- 172.21.15.50:0/3271534418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde84009420 con 0x7fde9c06d390 2023-12-19T14:54:19.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fde9b7fe700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00cb90 tx=0x7fde8c001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c007360 con 0x7fde9c06d390 2023-12-19T14:54:19.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c0074e0 con 0x7fde9c06d390 2023-12-19T14:54:19.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9c15af30 con 0x7fde9c06d390 2023-12-19T14:54:19.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c004830 con 0x7fde9c06d390 2023-12-19T14:54:19.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9c14cc30 con 0x7fde9c06d390 2023-12-19T14:54:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.523+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9c04be40 con 0x7fde9c06d390 2023-12-19T14:54:19.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.527+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fde8c004ba0 con 0x7fde9c06d390 2023-12-19T14:54:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.527+0000 7fde937fe700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 0x7fde7c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.527+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fde8c07c150 con 0x7fde9c06d390 2023-12-19T14:54:19.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.527+0000 7fde93fff700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 0x7fde7c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.531+0000 7fde93fff700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 0x7fde7c062bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0c8ff0 tx=0x7fde84016000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.535+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde8c04c880 con 0x7fde9c06d390 2023-12-19T14:54:19.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.707+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde9c0adac0 con 0x7fde9c06d390 2023-12-19T14:54:19.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.707+0000 7fde937fe700 1 -- 172.21.15.50:0/3271534418 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde8c04fea0 con 0x7fde9c06d390 2023-12-19T14:54:19.708 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 msgr2=0x7fde7c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 0x7fde7c062bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0c8ff0 tx=0x7fde84016000 comp rx=0 tx=0).stop 2023-12-19T14:54:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 msgr2=0x7fde9c14c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00cb90 tx=0x7fde8c001b50 comp rx=0 tx=0).stop 2023-12-19T14:54:19.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 shutdown_connections 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde9c0c8bf0 0x7fde9c14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde7c060740 0x7fde7c062bf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde9c0ace10 0x7fde9c0b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 --2- 172.21.15.50:0/3271534418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde9c06d390 0x7fde9c14c6f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 >> 172.21.15.50:0/3271534418 conn(0x7fde9c0cdc00 msgr2=0x7fde9c0c4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 shutdown_connections 2023-12-19T14:54:19.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.711+0000 7fdea0ce3700 1 -- 172.21.15.50:0/3271534418 wait complete. 2023-12-19T14:54:19.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:54:19.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 Processor -- start 2023-12-19T14:54:19.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 -- start start 2023-12-19T14:54:19.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44600c6900 0x7f44600c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 0x7f446006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446006b470 con 0x7f446015cc20 2023-12-19T14:54:19.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446015d7b0 con 0x7f44600c6900 2023-12-19T14:54:19.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.899+0000 7f44656e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4460161d10 con 0x7f446006b900 2023-12-19T14:54:19.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44600c6900 0x7f44600c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f4464ee2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f446015cc20 0x7f446015d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43826/0 (socket says 172.21.15.50:43826) 2023-12-19T14:54:19.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f4464ee2700 1 -- 172.21.15.50:0/1636622011 learned_addr learned my addr 172.21.15.50:0/1636622011 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:19.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f4457fff700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 0x7f446006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445f7fe700 1 -- 172.21.15.50:0/1636622011 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306505753 0 0) 0x7f446006b470 con 0x7f446015cc20 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445f7fe700 1 -- 172.21.15.50:0/1636622011 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f444c003570 con 0x7f446015cc20 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 -- 172.21.15.50:0/1636622011 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 msgr2=0x7f446006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 0x7f446006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 -- 172.21.15.50:0/1636622011 >> v1:172.21.15.50:6789/0 conn(0x7f446015cc20 legacy=0x7f446015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 -- 172.21.15.50:0/1636622011 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446006bd70 con 0x7f44600c6900 2023-12-19T14:54:19.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f4457fff700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 0x7f446006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:19.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445ffff700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44600c6900 0x7f44600c6cd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4450009910 tx=0x7f44500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1f4556ff890581 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445f7fe700 1 -- 172.21.15.50:0/1636622011 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011630 con 0x7f44600c6900 2023-12-19T14:54:19.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445f7fe700 1 -- 172.21.15.50:0/1636622011 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450011790 con 0x7f44600c6900 2023-12-19T14:54:19.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f445f7fe700 1 -- 172.21.15.50:0/1636622011 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011b30 con 0x7f44600c6900 2023-12-19T14:54:19.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 -- 172.21.15.50:0/1636622011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44600c6900 msgr2=0x7f44600c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44600c6900 0x7f44600c6cd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4450009910 tx=0x7f44500091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 -- 172.21.15.50:0/1636622011 shutdown_connections 2023-12-19T14:54:19.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f446006b900 0x7f446006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 --2- 172.21.15.50:0/1636622011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44600c6900 0x7f44600c6cd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 -- 172.21.15.50:0/1636622011 >> 172.21.15.50:0/1636622011 conn(0x7f4460067bb0 msgr2=0x7f44600cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:19.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.903+0000 7f44656e3700 1 -- 172.21.15.50:0/1636622011 shutdown_connections 2023-12-19T14:54:19.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 -- 172.21.15.50:0/1636622011 wait complete. 2023-12-19T14:54:19.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 Processor -- start 2023-12-19T14:54:19.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 -- start start 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f446006b900 0x7f4460171270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44600c6900 0x7f44600b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600afbc0 con 0x7f446015cc20 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44601504b0 con 0x7f446006b900 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.907+0000 7f44656e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600afdd0 con 0x7f44600c6900 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41306/0 (socket says 172.21.15.50:41306) 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 -- 172.21.15.50:0/3777527174 learned_addr learned my addr 172.21.15.50:0/3777527174 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f445ffff700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f446006b900 0x7f4460171270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4457fff700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44600c6900 0x7f44600b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 -- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44600c6900 msgr2=0x7f44600b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44600c6900 0x7f44600b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 -- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f446006b900 msgr2=0x7f4460171270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f446006b900 0x7f4460171270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:19.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 -- 172.21.15.50:0/3777527174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448009420 con 0x7f446015cc20 2023-12-19T14:54:19.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f4464ee2700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4458001460 tx=0x7f4458001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800a870 con 0x7f446015cc20 2023-12-19T14:54:19.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445800a9d0 con 0x7f446015cc20 2023-12-19T14:54:19.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800ad50 con 0x7f446015cc20 2023-12-19T14:54:19.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450009420 con 0x7f446015cc20 2023-12-19T14:54:19.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.911+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44600b0390 con 0x7f446015cc20 2023-12-19T14:54:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.915+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446004be40 con 0x7f446015cc20 2023-12-19T14:54:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.915+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4458007300 con 0x7f446015cc20 2023-12-19T14:54:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.915+0000 7f445cff9700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 0x7f4440062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.915+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f445807caf0 con 0x7f446015cc20 2023-12-19T14:54:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.915+0000 7f445ffff700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 0x7f4440062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.919+0000 7f445ffff700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 0x7f4440062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44500091e0 tx=0x7f4450017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:19.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:19.923+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f445804d110 con 0x7f446015cc20 2023-12-19T14:54:20.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.095+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f446006a580 con 0x7f446015cc20 2023-12-19T14:54:20.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.095+0000 7f445cff9700 1 -- 172.21.15.50:0/3777527174 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4458050730 con 0x7f446015cc20 2023-12-19T14:54:20.097 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 msgr2=0x7f4440062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 0x7f4440062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44500091e0 tx=0x7f4450017000 comp rx=0 tx=0).stop 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 msgr2=0x7f446014f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4458001460 tx=0x7f4458001a40 comp rx=0 tx=0).stop 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 shutdown_connections 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4440060670 0x7f4440062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f446015cc20 0x7f446014f5d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44600c6900 0x7f44600b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 --2- 172.21.15.50:0/3777527174 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f446006b900 0x7f4460171270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 >> 172.21.15.50:0/3777527174 conn(0x7f4460067bb0 msgr2=0x7f4460161ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 shutdown_connections 2023-12-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.099+0000 7f44656e3700 1 -- 172.21.15.50:0/3777527174 wait complete. 2023-12-19T14:54:20.122 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:54:20.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:54:20.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.295+0000 7f5e8b48e700 1 Processor -- start 2023-12-19T14:54:20.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.295+0000 7f5e8b48e700 1 -- start start 2023-12-19T14:54:20.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8b48e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e840aedc0 0x7f5e840af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8b48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 0x7f5e840af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8b48e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8406b3d0 con 0x7f5e8406af50 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8b48e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e840cc140 con 0x7f5e840aedc0 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8b48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840cb3f0 con 0x7f5e840cac50 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8a48c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e840aedc0 0x7f5e840af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e89c8b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5e8406af50 0x7f5e8406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50772/0 (socket says 172.21.15.50:50772) 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 0x7f5e840af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e89c8b700 1 -- 172.21.15.50:0/4246684665 learned_addr learned my addr 172.21.15.50:0/4246684665 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8948a700 1 -- 172.21.15.50:0/4246684665 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174493085 0 0) 0x7f5e74001110 con 0x7f5e8406af50 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8948a700 1 -- 172.21.15.50:0/4246684665 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e70003570 con 0x7f5e8406af50 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/4246684665 >> v2:172.21.15.50:3300/0 conn(0x7f5e840aedc0 msgr2=0x7f5e840af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 --2- 172.21.15.50:0/4246684665 >> v2:172.21.15.50:3300/0 conn(0x7f5e840aedc0 0x7f5e840af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/4246684665 >> v1:172.21.15.125:6789/0 conn(0x7f5e8406af50 legacy=0x7f5e8406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/4246684665 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8406d220 con 0x7f5e840cac50 2023-12-19T14:54:20.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.299+0000 7f5e8ac8d700 1 --2- 172.21.15.50:0/4246684665 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 0x7f5e840af930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009e60 tx=0x7f5e7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b91d079f5d70bf9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:20.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8948a700 1 -- 172.21.15.50:0/4246684665 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00e070 con 0x7f5e840cac50 2023-12-19T14:54:20.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8948a700 1 -- 172.21.15.50:0/4246684665 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c00bab0 con 0x7f5e840cac50 2023-12-19T14:54:20.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8948a700 1 -- 172.21.15.50:0/4246684665 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00be70 con 0x7f5e840cac50 2023-12-19T14:54:20.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- 172.21.15.50:0/4246684665 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 msgr2=0x7f5e840af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/4246684665 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 0x7f5e840af930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009e60 tx=0x7f5e7c009780 comp rx=0 tx=0).stop 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- 172.21.15.50:0/4246684665 shutdown_connections 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/4246684665 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e840cac50 0x7f5e840af930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/4246684665 >> v2:172.21.15.50:3300/0 conn(0x7f5e840aedc0 0x7f5e840af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- 172.21.15.50:0/4246684665 >> 172.21.15.50:0/4246684665 conn(0x7f5e840cc760 msgr2=0x7f5e840ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- 172.21.15.50:0/4246684665 shutdown_connections 2023-12-19T14:54:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- 172.21.15.50:0/4246684665 wait complete. 2023-12-19T14:54:20.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 Processor -- start 2023-12-19T14:54:20.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.303+0000 7f5e8b48e700 1 -- start start 2023-12-19T14:54:20.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e8406af50 0x7f5e8415bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e840aedc0 0x7f5e84154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8415c330 con 0x7f5e840cac50 2023-12-19T14:54:20.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840b4720 con 0x7f5e840aedc0 2023-12-19T14:54:20.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840b4a00 con 0x7f5e8406af50 2023-12-19T14:54:20.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41326/0 (socket says 172.21.15.50:41326) 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/1053916008 learned_addr learned my addr 172.21.15.50:0/1053916008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8a48c700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e8406af50 0x7f5e8415bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e89c8b700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e840aedc0 0x7f5e84154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e8406af50 msgr2=0x7f5e8415bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e8406af50 0x7f5e8415bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e840aedc0 msgr2=0x7f5e84154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e840aedc0 0x7f5e84154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 -- 172.21.15.50:0/1053916008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c009a20 con 0x7f5e840cac50 2023-12-19T14:54:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8ac8d700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5e840cb530 tx=0x7f5e7c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c012e10 con 0x7f5e840cac50 2023-12-19T14:54:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c027470 con 0x7f5e840cac50 2023-12-19T14:54:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e840b4ce0 con 0x7f5e840cac50 2023-12-19T14:54:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c027830 con 0x7f5e840cac50 2023-12-19T14:54:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.307+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e840b52a0 con 0x7f5e840cac50 2023-12-19T14:54:20.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.311+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8404be40 con 0x7f5e840cac50 2023-12-19T14:54:20.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.311+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5e7c024070 con 0x7f5e840cac50 2023-12-19T14:54:20.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.315+0000 7f5e7a7fc700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 0x7f5e68062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:20.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.315+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f5e7c07f0f0 con 0x7f5e840cac50 2023-12-19T14:54:20.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.315+0000 7f5e8a48c700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 0x7f5e68062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:20.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.315+0000 7f5e8a48c700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 0x7f5e68062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e840c6c70 tx=0x7f5e800091a0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:20.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.319+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5e7c04f720 con 0x7f5e840cac50 2023-12-19T14:54:20.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:20.487+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5e8406c8f0 con 0x7f5e840cac50 2023-12-19T14:54:21.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.055+0000 7f5e7a7fc700 1 -- 172.21.15.50:0/1053916008 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5e7c052d40 con 0x7f5e840cac50 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 msgr2=0x7f5e68062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 0x7f5e68062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e840c6c70 tx=0x7f5e800091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 msgr2=0x7f5e840c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5e840cb530 tx=0x7f5e7c019000 comp rx=0 tx=0).stop 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 shutdown_connections 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e840aedc0 0x7f5e84154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e68060670 0x7f5e68062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e840cac50 0x7f5e840c8720 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 --2- 172.21.15.50:0/1053916008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5e8406af50 0x7f5e8415bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 >> 172.21.15.50:0/1053916008 conn(0x7f5e840cc760 msgr2=0x7f5e840af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 shutdown_connections 2023-12-19T14:54:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.059+0000 7f5e8b48e700 1 -- 172.21.15.50:0/1053916008 wait complete. 2023-12-19T14:54:21.064 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:54:21.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:54:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.251+0000 7ff62a2ae700 1 Processor -- start 2023-12-19T14:54:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.255+0000 7ff62a2ae700 1 -- start start 2023-12-19T14:54:21.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.255+0000 7ff62a2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 0x7ff62406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.255+0000 7ff62a2ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240cbdf0 con 0x7ff624158e20 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff62a2ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240cc0d0 con 0x7ff6240cecd0 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff62a2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cc3b0 con 0x7ff6240cb970 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff623fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff624158e20 0x7ff6240ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43840/0 (socket says 172.21.15.50:43840) 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff623fff700 1 -- 172.21.15.50:0/3697179698 learned_addr learned my addr 172.21.15.50:0/3697179698 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff6237fe700 1 --2- 172.21.15.50:0/3697179698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 0x7ff62406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486820582 0 0) 0x7ff6240cbdf0 con 0x7ff624158e20 2023-12-19T14:54:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60c003570 con 0x7ff624158e20 2023-12-19T14:54:21.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177028164 0 0) 0x7ff6240cc0d0 con 0x7ff6240cecd0 2023-12-19T14:54:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6240cbdf0 con 0x7ff6240cecd0 2023-12-19T14:54:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff6237fe700 1 -- 172.21.15.50:0/3697179698 >> v1:172.21.15.125:6789/0 conn(0x7ff6240cecd0 legacy=0x7ff62406b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff6237fe700 1 -- 172.21.15.50:0/3697179698 >> v1:172.21.15.50:6789/0 conn(0x7ff624158e20 legacy=0x7ff6240ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff6237fe700 1 -- 172.21.15.50:0/3697179698 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240cc690 con 0x7ff6240cb970 2023-12-19T14:54:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff6237fe700 1 --2- 172.21.15.50:0/3697179698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 0x7ff62406acc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff618009910 tx=0x7ff6180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca583b5823e28f7b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011630 con 0x7ff6240cb970 2023-12-19T14:54:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff618011790 con 0x7ff6240cb970 2023-12-19T14:54:21.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.259+0000 7ff622ffd700 1 -- 172.21.15.50:0/3697179698 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011b30 con 0x7ff6240cb970 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- 172.21.15.50:0/3697179698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 msgr2=0x7ff62406acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/3697179698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 0x7ff62406acc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff618009910 tx=0x7ff6180091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- 172.21.15.50:0/3697179698 shutdown_connections 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/3697179698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240cb970 0x7ff62406acc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- 172.21.15.50:0/3697179698 >> 172.21.15.50:0/3697179698 conn(0x7ff624067bb0 msgr2=0x7ff6240d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- 172.21.15.50:0/3697179698 shutdown_connections 2023-12-19T14:54:21.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- 172.21.15.50:0/3697179698 wait complete. 2023-12-19T14:54:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 Processor -- start 2023-12-19T14:54:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- start start 2023-12-19T14:54:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:21.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff624158e20 0x7ff624152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 0x7ff6240b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240b5ac0 con 0x7ff6240cecd0 2023-12-19T14:54:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62415a0c0 con 0x7ff624158e20 2023-12-19T14:54:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.263+0000 7ff62a2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6241591f0 con 0x7ff6240b5100 2023-12-19T14:54:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff623fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff623fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41328/0 (socket says 172.21.15.50:41328) 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff623fff700 1 -- 172.21.15.50:0/782054326 learned_addr learned my addr 172.21.15.50:0/782054326 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 0x7ff6240b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff6237fe700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff624158e20 0x7ff624152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 -- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff624158e20 msgr2=0x7ff624152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff624158e20 0x7ff624152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 -- 172.21.15.50:0/782054326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 msgr2=0x7ff62416d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 -- 172.21.15.50:0/782054326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618009420 con 0x7ff6240b5100 2023-12-19T14:54:21.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff623fff700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62884b700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 0x7ff6240b5580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c810 tx=0x7ff61c002660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c012630 con 0x7ff6240b5100 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c012790 con 0x7ff6240b5100 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6241594b0 con 0x7ff6240b5100 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c012b50 con 0x7ff6240b5100 2023-12-19T14:54:21.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.267+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff624159a70 con 0x7ff6240b5100 2023-12-19T14:54:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.271+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62415a250 con 0x7ff6240b5100 2023-12-19T14:54:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.271+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff61c01b400 con 0x7ff6240b5100 2023-12-19T14:54:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.271+0000 7ff60bfff700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 0x7ff610062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.275+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5571+0+0 (secure 0 0 0) 0x7ff61c07c880 con 0x7ff6240b5100 2023-12-19T14:54:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.275+0000 7ff623fff700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 0x7ff610062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:21.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.275+0000 7ff623fff700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 0x7ff610062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6140058d0 tx=0x7ff614009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.283+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff61c04cd30 con 0x7ff6240b5100 2023-12-19T14:54:21.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:21.459+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff624071120 con 0x7ff6240b5100 2023-12-19T14:54:22.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.067+0000 7ff60bfff700 1 -- 172.21.15.50:0/782054326 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff61c0160d0 con 0x7ff6240b5100 2023-12-19T14:54:22.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 msgr2=0x7ff610062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 0x7ff610062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6140058d0 tx=0x7ff614009040 comp rx=0 tx=0).stop 2023-12-19T14:54:22.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 msgr2=0x7ff6240b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 0x7ff6240b5580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c810 tx=0x7ff61c002660 comp rx=0 tx=0).stop 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 shutdown_connections 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff624158e20 0x7ff624152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff610060740 0x7ff610062bf0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6140058d0 tx=0x7ff614009040 comp rx=0 tx=0).stop 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6240b5100 0x7ff6240b5580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 --2- 172.21.15.50:0/782054326 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6240cecd0 0x7ff62416d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.075+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 >> 172.21.15.50:0/782054326 conn(0x7ff624067bb0 msgr2=0x7ff6240d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:22.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.079+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 shutdown_connections 2023-12-19T14:54:22.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.079+0000 7ff62a2ae700 1 -- 172.21.15.50:0/782054326 wait complete. 2023-12-19T14:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:54:22.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:54:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.287+0000 7f394b4e3700 1 Processor -- start 2023-12-19T14:54:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.287+0000 7f394b4e3700 1 -- start start 2023-12-19T14:54:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394b4e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f39440caa10 0x7f3944159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394b4e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39440cbbb0 con 0x7f39440ce2d0 2023-12-19T14:54:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394b4e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394406dc90 con 0x7f39440cc040 2023-12-19T14:54:22.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394b4e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39440cafa0 con 0x7f39440caa10 2023-12-19T14:54:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394a4e1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f39440ce2d0 0x7f3944158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43850/0 (socket says 172.21.15.50:43850) 2023-12-19T14:54:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f394a4e1700 1 -- 172.21.15.50:0/1699349319 learned_addr learned my addr 172.21.15.50:0/1699349319 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f3949ce0700 1 --2- 172.21.15.50:0/1699349319 >> v2:172.21.15.50:3300/0 conn(0x7f39440caa10 0x7f3944159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047554964 0 0) 0x7f394406dc90 con 0x7f39440cc040 2023-12-19T14:54:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003570 con 0x7f39440cc040 2023-12-19T14:54:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764814398 0 0) 0x7f39440cbbb0 con 0x7f39440ce2d0 2023-12-19T14:54:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394406dc90 con 0x7f39440ce2d0 2023-12-19T14:54:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f3949ce0700 1 -- 172.21.15.50:0/1699349319 >> v1:172.21.15.125:6789/0 conn(0x7f39440cc040 legacy=0x7f394406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502347082 0 0) 0x7f394406dc90 con 0x7f39440ce2d0 2023-12-19T14:54:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f3949ce0700 1 -- 172.21.15.50:0/1699349319 >> v1:172.21.15.50:6789/0 conn(0x7f39440ce2d0 legacy=0x7f3944158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f3949ce0700 1 -- 172.21.15.50:0/1699349319 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39440cb280 con 0x7f39440caa10 2023-12-19T14:54:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.291+0000 7f3949ce0700 1 --2- 172.21.15.50:0/1699349319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440caa10 0x7f3944159010 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f39340098e0 tx=0x7f39340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e42ef201fe2055 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940001470 con 0x7f39440caa10 2023-12-19T14:54:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3934011420 con 0x7f39440caa10 2023-12-19T14:54:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f39494df700 1 -- 172.21.15.50:0/1699349319 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39340117c0 con 0x7f39440caa10 2023-12-19T14:54:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 -- 172.21.15.50:0/1699349319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440caa10 msgr2=0x7f3944159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 --2- 172.21.15.50:0/1699349319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440caa10 0x7f3944159010 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f39340098e0 tx=0x7f39340091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 -- 172.21.15.50:0/1699349319 shutdown_connections 2023-12-19T14:54:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 --2- 172.21.15.50:0/1699349319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440caa10 0x7f3944159010 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 -- 172.21.15.50:0/1699349319 >> 172.21.15.50:0/1699349319 conn(0x7f3944067bb0 msgr2=0x7f39440cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 -- 172.21.15.50:0/1699349319 shutdown_connections 2023-12-19T14:54:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.295+0000 7f394b4e3700 1 -- 172.21.15.50:0/1699349319 wait complete. 2023-12-19T14:54:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 Processor -- start 2023-12-19T14:54:22.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 -- start start 2023-12-19T14:54:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39440caa10 0x7f39441691f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39440cc040 0x7f3944159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394415a730 con 0x7f39440ce2d0 2023-12-19T14:54:22.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39440c7b90 con 0x7f39440caa10 2023-12-19T14:54:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394b4e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39440c7e70 con 0x7f39440cc040 2023-12-19T14:54:22.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41344/0 (socket says 172.21.15.50:41344) 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 -- 172.21.15.50:0/940789681 learned_addr learned my addr 172.21.15.50:0/940789681 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f3949ce0700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39440cc040 0x7f3944159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394a4e1700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39440caa10 0x7f39441691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 -- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39440cc040 msgr2=0x7f3944159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39440cc040 0x7f3944159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 -- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39440caa10 msgr2=0x7f39441691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.299+0000 7f394ace2700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39440caa10 0x7f39441691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:22.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f394ace2700 1 -- 172.21.15.50:0/940789681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009420 con 0x7f39440ce2d0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f394ace2700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c910 tx=0x7f393c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c00a7e0 con 0x7f39440ce2d0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c00a960 con 0x7f39440ce2d0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39440c8150 con 0x7f39440ce2d0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c00ad20 con 0x7f39440ce2d0 2023-12-19T14:54:22.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394415a010 con 0x7f39440ce2d0 2023-12-19T14:54:22.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.303+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394415a8a0 con 0x7f39440ce2d0 2023-12-19T14:54:22.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.307+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f393c012630 con 0x7f39440ce2d0 2023-12-19T14:54:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.307+0000 7f393a7fc700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 0x7f3928062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:22.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.307+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5953+0+0 (secure 0 0 0) 0x7f393c07d4d0 con 0x7f39440ce2d0 2023-12-19T14:54:22.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.307+0000 7f394a4e1700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 0x7f3928062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:22.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.311+0000 7f394a4e1700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 0x7f3928062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3940001ed0 tx=0x7f3940001520 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.315+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f393c04d890 con 0x7f39440ce2d0 2023-12-19T14:54:22.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:22.519+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f394404be40 con 0x7f39440ce2d0 2023-12-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.087+0000 7f393a7fc700 1 -- 172.21.15.50:0/940789681 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) v1 ==== 173+0+0 (secure 0 0 0) 0x7f393c050eb0 con 0x7f39440ce2d0 2023-12-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 msgr2=0x7f3928062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 0x7f3928062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3940001ed0 tx=0x7f3940001520 comp rx=0 tx=0).stop 2023-12-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 msgr2=0x7f3944149700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c910 tx=0x7f393c0027a0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 shutdown_connections 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3928060670 0x7f3928062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39440ce2d0 0x7f3944149700 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39440cc040 0x7f3944159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 --2- 172.21.15.50:0/940789681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39440caa10 0x7f39441691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 >> 172.21.15.50:0/940789681 conn(0x7f3944067bb0 msgr2=0x7f39440c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 shutdown_connections 2023-12-19T14:54:23.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.091+0000 7f394b4e3700 1 -- 172.21.15.50:0/940789681 wait complete. 2023-12-19T14:54:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 9 and data pool 10 2023-12-19T14:54:23.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:23.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.283+0000 7f753f278700 1 Processor -- start 2023-12-19T14:54:23.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.283+0000 7f753f278700 1 -- start start 2023-12-19T14:54:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753f278700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f753815d320 0x7f753815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753f278700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75380c7190 con 0x7f75380c15b0 2023-12-19T14:54:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753f278700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75380c1b50 con 0x7f75380aeba0 2023-12-19T14:54:23.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753f278700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380aefd0 con 0x7f753815d320 2023-12-19T14:54:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753c813700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75380c15b0 0x7f753806d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43862/0 (socket says 172.21.15.50:43862) 2023-12-19T14:54:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753c813700 1 -- 172.21.15.50:0/2791544564 learned_addr learned my addr 172.21.15.50:0/2791544564 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:23.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753d014700 1 --2- 172.21.15.50:0/2791544564 >> v2:172.21.15.50:3300/0 conn(0x7f753815d320 0x7f753815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f752ffff700 1 -- 172.21.15.50:0/2791544564 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89660827 0 0) 0x7f75380c7190 con 0x7f75380c15b0 2023-12-19T14:54:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f752ffff700 1 -- 172.21.15.50:0/2791544564 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7524003570 con 0x7f75380c15b0 2023-12-19T14:54:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753d014700 1 -- 172.21.15.50:0/2791544564 >> v1:172.21.15.125:6789/0 conn(0x7f75380aeba0 legacy=0x7f75380680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753d014700 1 -- 172.21.15.50:0/2791544564 >> v1:172.21.15.50:6789/0 conn(0x7f75380c15b0 legacy=0x7f753806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753d014700 1 -- 172.21.15.50:0/2791544564 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380c1e30 con 0x7f753815d320 2023-12-19T14:54:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f753d014700 1 --2- 172.21.15.50:0/2791544564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f753815d6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f75340098e0 tx=0x7f75340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5c25bd57f39528 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f752ffff700 1 -- 172.21.15.50:0/2791544564 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753400b640 con 0x7f753815d320 2023-12-19T14:54:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f752ffff700 1 -- 172.21.15.50:0/2791544564 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753400b7a0 con 0x7f753815d320 2023-12-19T14:54:23.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.287+0000 7f752ffff700 1 -- 172.21.15.50:0/2791544564 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753400bb40 con 0x7f753815d320 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- 172.21.15.50:0/2791544564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 msgr2=0x7f753815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 --2- 172.21.15.50:0/2791544564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f753815d6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f75340098e0 tx=0x7f75340091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- 172.21.15.50:0/2791544564 shutdown_connections 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 --2- 172.21.15.50:0/2791544564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f753815d6f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- 172.21.15.50:0/2791544564 >> 172.21.15.50:0/2791544564 conn(0x7f75380c8850 msgr2=0x7f75380c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- 172.21.15.50:0/2791544564 shutdown_connections 2023-12-19T14:54:23.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- 172.21.15.50:0/2791544564 wait complete. 2023-12-19T14:54:23.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 Processor -- start 2023-12-19T14:54:23.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.291+0000 7f753f278700 1 -- start start 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75380aeba0 0x7f75380c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f75381542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753815d6f0 con 0x7f753815d320 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75380b1030 con 0x7f75380c15b0 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75380b1310 con 0x7f75380aeba0 2023-12-19T14:54:23.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753d815700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f75381542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:39004/0 (socket says 172.21.15.50:39004) 2023-12-19T14:54:23.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 -- 172.21.15.50:0/3560895062 learned_addr learned my addr 172.21.15.50:0/3560895062 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:23.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753d014700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75380aeba0 0x7f75380c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 -- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75380aeba0 msgr2=0x7f75380c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75380aeba0 0x7f75380c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 -- 172.21.15.50:0/3560895062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 msgr2=0x7f75381542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f75381542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 -- 172.21.15.50:0/3560895062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7534009420 con 0x7f75380c15b0 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753d815700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f75381542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:23.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753c813700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f752800c7c0 tx=0x7f7528007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528005cd0 con 0x7f75380c15b0 2023-12-19T14:54:23.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7528005e30 con 0x7f75380c15b0 2023-12-19T14:54:23.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753815d940 con 0x7f75380c15b0 2023-12-19T14:54:23.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.295+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528007320 con 0x7f75380c15b0 2023-12-19T14:54:23.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.299+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75380c46d0 con 0x7f75380c15b0 2023-12-19T14:54:23.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.299+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7538071120 con 0x7f75380c15b0 2023-12-19T14:54:23.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.299+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f752801b3b0 con 0x7f75380c15b0 2023-12-19T14:54:23.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.303+0000 7f752d7fa700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 0x7f751c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.303+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f752807c180 con 0x7f75380c15b0 2023-12-19T14:54:23.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.303+0000 7f753d014700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 0x7f751c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.303+0000 7f753d014700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 0x7f751c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75340093f0 tx=0x7f7534017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.311+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f752804c460 con 0x7f75380c15b0 2023-12-19T14:54:23.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.483+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75380642a0 con 0x7f75380c15b0 2023-12-19T14:54:23.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.487+0000 7f752d7fa700 1 -- 172.21.15.50:0/3560895062 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+15613 (secure 0 0 0) 0x7f752804fa80 con 0x7f75380c15b0 2023-12-19T14:54:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:23.490 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":85,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:22.525101+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 msgr2=0x7f751c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 0x7f751c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75340093f0 tx=0x7f7534017000 comp rx=0 tx=0).stop 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 msgr2=0x7f75380b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f752800c7c0 tx=0x7f7528007e00 comp rx=0 tx=0).stop 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 shutdown_connections 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75380c15b0 0x7f75380b0520 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f751c060670 0x7f751c062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f753815d320 0x7f75381542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 --2- 172.21.15.50:0/3560895062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75380aeba0 0x7f75380c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 >> 172.21.15.50:0/3560895062 conn(0x7f75380c8850 msgr2=0x7f753815c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 shutdown_connections 2023-12-19T14:54:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.491+0000 7f753f278700 1 -- 172.21.15.50:0/3560895062 wait complete. 2023-12-19T14:54:23.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.683+0000 7f3cbd363700 1 Processor -- start 2023-12-19T14:54:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.683+0000 7f3cbd363700 1 -- start start 2023-12-19T14:54:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbd363700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cb81591f0 0x7f3cb806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbd363700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb806b730 con 0x7f3cb80ce810 2023-12-19T14:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbd363700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb806ab20 con 0x7f3cb8158e20 2023-12-19T14:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbd363700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb806ae00 con 0x7f3cb81591f0 2023-12-19T14:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbcb62700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3cb80ce810 0x7f3cb80ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43866/0 (socket says 172.21.15.50:43866) 2023-12-19T14:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cbcb62700 1 -- 172.21.15.50:0/3208356657 learned_addr learned my addr 172.21.15.50:0/3208356657 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3208356657 >> v2:172.21.15.125:3300/0 conn(0x7f3cb81591f0 0x7f3cb806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb6ffd700 1 -- 172.21.15.50:0/3208356657 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916859316 0 0) 0x7f3cb806b730 con 0x7f3cb80ce810 2023-12-19T14:54:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb6ffd700 1 -- 172.21.15.50:0/3208356657 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca0003570 con 0x7f3cb80ce810 2023-12-19T14:54:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb77fe700 1 -- 172.21.15.50:0/3208356657 >> v1:172.21.15.125:6789/0 conn(0x7f3cb8158e20 legacy=0x7f3cb80ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb77fe700 1 -- 172.21.15.50:0/3208356657 >> v1:172.21.15.50:6789/0 conn(0x7f3cb80ce810 legacy=0x7f3cb80ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb77fe700 1 -- 172.21.15.50:0/3208356657 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb806b0e0 con 0x7f3cb81591f0 2023-12-19T14:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3208356657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb81591f0 0x7f3cb806a540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0098e0 tx=0x7f3cac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f56363caccfa36 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb6ffd700 1 -- 172.21.15.50:0/3208356657 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac00b640 con 0x7f3cb81591f0 2023-12-19T14:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb6ffd700 1 -- 172.21.15.50:0/3208356657 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac00b7a0 con 0x7f3cb81591f0 2023-12-19T14:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.687+0000 7f3cb6ffd700 1 -- 172.21.15.50:0/3208356657 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac00bb40 con 0x7f3cb81591f0 2023-12-19T14:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- 172.21.15.50:0/3208356657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb81591f0 msgr2=0x7f3cb806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3208356657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb81591f0 0x7f3cb806a540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0098e0 tx=0x7f3cac0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- 172.21.15.50:0/3208356657 shutdown_connections 2023-12-19T14:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3208356657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb81591f0 0x7f3cb806a540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- 172.21.15.50:0/3208356657 >> 172.21.15.50:0/3208356657 conn(0x7f3cb8067bb0 msgr2=0x7f3cb80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- 172.21.15.50:0/3208356657 shutdown_connections 2023-12-19T14:54:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- 172.21.15.50:0/3208356657 wait complete. 2023-12-19T14:54:23.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 Processor -- start 2023-12-19T14:54:23.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- start start 2023-12-19T14:54:23.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 0x7f3cb816d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 0x7f3cb8149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb816db60 con 0x7f3cb80ce810 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80b4f00 con 0x7f3cb8158e20 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.691+0000 7f3cbd363700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8169770 con 0x7f3cb81591f0 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41368/0 (socket says 172.21.15.50:41368) 2023-12-19T14:54:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 -- 172.21.15.50:0/3091044059 learned_addr learned my addr 172.21.15.50:0/3091044059 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cbcb62700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 0x7f3cb8149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 0x7f3cb816d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 -- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 msgr2=0x7f3cb8149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 0x7f3cb8149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 -- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 msgr2=0x7f3cb816d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 0x7f3cb816d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 -- 172.21.15.50:0/3091044059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac009420 con 0x7f3cb80ce810 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cbcb62700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 0x7f3cb8149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 0x7f3cb816d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cb7fff700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800c5c0 tx=0x7f3ca800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8011590 con 0x7f3cb80ce810 2023-12-19T14:54:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8011710 con 0x7f3cb80ce810 2023-12-19T14:54:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80b5150 con 0x7f3cb80ce810 2023-12-19T14:54:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8011ad0 con 0x7f3cb80ce810 2023-12-19T14:54:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.695+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb8169cc0 con 0x7f3cb80ce810 2023-12-19T14:54:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.699+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb8169ee0 con 0x7f3cb80ce810 2023-12-19T14:54:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.699+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3ca801b3b0 con 0x7f3cb80ce810 2023-12-19T14:54:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.699+0000 7f3c9ffff700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 0x7f3ca4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.699+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3ca807bf60 con 0x7f3cb80ce810 2023-12-19T14:54:23.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.703+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 0x7f3ca4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:23.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.703+0000 7f3cb77fe700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 0x7f3ca4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0093f0 tx=0x7f3cac017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:23.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.707+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3ca804c240 con 0x7f3cb80ce810 2023-12-19T14:54:23.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.879+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cb804be40 con 0x7f3cb80ce810 2023-12-19T14:54:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.879+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3091044059 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+15613 (secure 0 0 0) 0x7f3ca8017020 con 0x7f3cb80ce810 2023-12-19T14:54:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:23.883 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":85,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:22.525101+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 msgr2=0x7f3ca4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 0x7f3ca4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0093f0 tx=0x7f3cac017000 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 msgr2=0x7f3cb816a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800c5c0 tx=0x7f3ca800c8d0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 shutdown_connections 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cb8158e20 0x7f3cb816d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3ca4060670 0x7f3ca4062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cb81591f0 0x7f3cb8149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 --2- 172.21.15.50:0/3091044059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cb80ce810 0x7f3cb816a1e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:23.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 >> 172.21.15.50:0/3091044059 conn(0x7f3cb8067bb0 msgr2=0x7f3cb80c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 shutdown_connections 2023-12-19T14:54:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:23.883+0000 7f3cbd363700 1 -- 172.21.15.50:0/3091044059 wait complete. 2023-12-19T14:54:23.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:54:23.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2023-12-19T14:54:23.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-12-19T14:54:23.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-12-19T14:54:24.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 Processor -- start 2023-12-19T14:54:24.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 -- start start 2023-12-19T14:54:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 0x7fee3c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c06c850 con 0x7fee3c1595a0 2023-12-19T14:54:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c0cb650 con 0x7fee3c158c90 2023-12-19T14:54:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee41523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c06b470 con 0x7fee3c06aeb0 2023-12-19T14:54:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee40d22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fee3c1595a0 0x7fee3c06cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43874/0 (socket says 172.21.15.50:43874) 2023-12-19T14:54:24.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.063+0000 7fee40d22700 1 -- 172.21.15.50:0/709509620 learned_addr learned my addr 172.21.15.50:0/709509620 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:24.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3b7fe700 1 --2- 172.21.15.50:0/709509620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 0x7fee3c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:24.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3affd700 1 -- 172.21.15.50:0/709509620 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901244906 0 0) 0x7fee3c06c850 con 0x7fee3c1595a0 2023-12-19T14:54:24.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3affd700 1 -- 172.21.15.50:0/709509620 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee24003570 con 0x7fee3c1595a0 2023-12-19T14:54:24.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3b7fe700 1 -- 172.21.15.50:0/709509620 >> v1:172.21.15.125:6789/0 conn(0x7fee3c158c90 legacy=0x7fee3c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:24.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3b7fe700 1 -- 172.21.15.50:0/709509620 >> v1:172.21.15.50:6789/0 conn(0x7fee3c1595a0 legacy=0x7fee3c06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:24.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3b7fe700 1 -- 172.21.15.50:0/709509620 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c06b750 con 0x7fee3c06aeb0 2023-12-19T14:54:24.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3b7fe700 1 --2- 172.21.15.50:0/709509620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 0x7fee3c159060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee30009910 tx=0x7fee300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa3fd257df189143 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:24.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3affd700 1 -- 172.21.15.50:0/709509620 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3000b680 con 0x7fee3c06aeb0 2023-12-19T14:54:24.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3affd700 1 -- 172.21.15.50:0/709509620 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3000b7e0 con 0x7fee3c06aeb0 2023-12-19T14:54:24.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee3affd700 1 -- 172.21.15.50:0/709509620 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3000bba0 con 0x7fee3c06aeb0 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 -- 172.21.15.50:0/709509620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 msgr2=0x7fee3c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 --2- 172.21.15.50:0/709509620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 0x7fee3c159060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee30009910 tx=0x7fee300091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 -- 172.21.15.50:0/709509620 shutdown_connections 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 --2- 172.21.15.50:0/709509620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c06aeb0 0x7fee3c159060 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 -- 172.21.15.50:0/709509620 >> 172.21.15.50:0/709509620 conn(0x7fee3c067bb0 msgr2=0x7fee3c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 -- 172.21.15.50:0/709509620 shutdown_connections 2023-12-19T14:54:24.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.067+0000 7fee41523700 1 -- 172.21.15.50:0/709509620 wait complete. 2023-12-19T14:54:24.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 Processor -- start 2023-12-19T14:54:24.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 -- start start 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 0x7fee3c153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0c8fe0 con 0x7fee3c06aeb0 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0c92c0 con 0x7fee3c1595a0 2023-12-19T14:54:24.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.071+0000 7fee41523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0c95a0 con 0x7fee3c158c90 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3b7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 0x7fee3c153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3b7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:41916/0 (socket says 172.21.15.50:41916) 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3b7fe700 1 -- 172.21.15.50:0/3504673415 learned_addr learned my addr 172.21.15.50:0/3504673415 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41380/0 (socket says 172.21.15.50:41380) 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 -- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 msgr2=0x7fee3c0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 -- 172.21.15.50:0/3504673415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 msgr2=0x7fee3c16a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 -- 172.21.15.50:0/3504673415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30009420 con 0x7fee3c1595a0 2023-12-19T14:54:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3bfff700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:24.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee3b7fe700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:24.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee40d22700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 0x7fee3c153280 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fee3400cb90 tx=0x7fee34001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:24.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400a7e0 con 0x7fee3c1595a0 2023-12-19T14:54:24.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3400a940 con 0x7fee3c1595a0 2023-12-19T14:54:24.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c0c9880 con 0x7fee3c1595a0 2023-12-19T14:54:24.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400acc0 con 0x7fee3c1595a0 2023-12-19T14:54:24.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.075+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c16a620 con 0x7fee3c1595a0 2023-12-19T14:54:24.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.079+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c0c65e0 con 0x7fee3c1595a0 2023-12-19T14:54:24.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.079+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fee34007300 con 0x7fee3c1595a0 2023-12-19T14:54:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.083+0000 7fee23fff700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 0x7fee28062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:24.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.083+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fee3407c5f0 con 0x7fee3c1595a0 2023-12-19T14:54:24.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.083+0000 7fee3bfff700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 0x7fee28062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:24.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.083+0000 7fee3bfff700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 0x7fee28062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee2c001dc0 tx=0x7fee2c009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.091+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fee34049be0 con 0x7fee3c1595a0 2023-12-19T14:54:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:24.299+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fee3c04be40 con 0x7fee3c1595a0 2023-12-19T14:54:25.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.115+0000 7fee23fff700 1 -- 172.21.15.50:0/3504673415 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fee3c0c65e0 con 0x7fee3c1595a0 2023-12-19T14:54:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 msgr2=0x7fee28062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 0x7fee28062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee2c001dc0 tx=0x7fee2c009040 comp rx=0 tx=0).stop 2023-12-19T14:54:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 msgr2=0x7fee3c153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 0x7fee3c153280 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fee3400cb90 tx=0x7fee34001980 comp rx=0 tx=0).stop 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 shutdown_connections 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee3c1595a0 0x7fee3c153280 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee28060670 0x7fee28062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee3c158c90 0x7fee3c0c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 --2- 172.21.15.50:0/3504673415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee3c06aeb0 0x7fee3c16a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 >> 172.21.15.50:0/3504673415 conn(0x7fee3c067bb0 msgr2=0x7fee3c0afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:25.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 shutdown_connections 2023-12-19T14:54:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.123+0000 7fee41523700 1 -- 172.21.15.50:0/3504673415 wait complete. 2023-12-19T14:54:25.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:54:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.315+0000 7f590d6f0700 1 Processor -- start 2023-12-19T14:54:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.319+0000 7f590d6f0700 1 -- start start 2023-12-19T14:54:25.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.319+0000 7f590d6f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59080cb970 0x7f590806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:25.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f590d6f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080cbdf0 con 0x7f5908158e20 2023-12-19T14:54:25.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f590d6f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080cc0d0 con 0x7f59080cecd0 2023-12-19T14:54:25.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f590d6f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080cc3b0 con 0x7f59080cb970 2023-12-19T14:54:25.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59077fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f59080cecd0 0x7f590806b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50836/0 (socket says 172.21.15.50:50836) 2023-12-19T14:54:25.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59077fe700 1 -- 172.21.15.50:0/1720661572 learned_addr learned my addr 172.21.15.50:0/1720661572 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:25.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5906ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5908158e20 0x7f59080ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43880/0 (socket says 172.21.15.50:43880) 2023-12-19T14:54:25.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59067fc700 1 --2- 172.21.15.50:0/1720661572 >> v2:172.21.15.125:3300/0 conn(0x7f59080cb970 0x7f590806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:25.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580857589 0 0) 0x7f59080cc0d0 con 0x7f59080cecd0 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f0003570 con 0x7f59080cecd0 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800271203 0 0) 0x7f59080cbdf0 con 0x7f5908158e20 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59080cc0d0 con 0x7f5908158e20 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59067fc700 1 -- 172.21.15.50:0/1720661572 >> v1:172.21.15.125:6789/0 conn(0x7f59080cecd0 legacy=0x7f590806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59067fc700 1 -- 172.21.15.50:0/1720661572 >> v1:172.21.15.50:6789/0 conn(0x7f5908158e20 legacy=0x7f59080ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59067fc700 1 -- 172.21.15.50:0/1720661572 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080cc690 con 0x7f59080cb970 2023-12-19T14:54:25.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f59067fc700 1 --2- 172.21.15.50:0/1720661572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cb970 0x7f590806acc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009910 tx=0x7f58fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=257f7e631489e8a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:25.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc011630 con 0x7f59080cb970 2023-12-19T14:54:25.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc011790 con 0x7f59080cb970 2023-12-19T14:54:25.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.323+0000 7f5905ffb700 1 -- 172.21.15.50:0/1720661572 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc011b30 con 0x7f59080cb970 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 -- 172.21.15.50:0/1720661572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cb970 msgr2=0x7f590806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 --2- 172.21.15.50:0/1720661572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cb970 0x7f590806acc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009910 tx=0x7f58fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 -- 172.21.15.50:0/1720661572 shutdown_connections 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 --2- 172.21.15.50:0/1720661572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cb970 0x7f590806acc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 -- 172.21.15.50:0/1720661572 >> 172.21.15.50:0/1720661572 conn(0x7f5908067bb0 msgr2=0x7f59080d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 -- 172.21.15.50:0/1720661572 shutdown_connections 2023-12-19T14:54:25.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 -- 172.21.15.50:0/1720661572 wait complete. 2023-12-19T14:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.327+0000 7f590d6f0700 1 Processor -- start 2023-12-19T14:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 -- start start 2023-12-19T14:54:25.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cecd0 0x7f59080b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:25.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 0x7f5908154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c82c0 con 0x7f5908158e20 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c8b60 con 0x7f59080cecd0 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f590d6f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590814d3a0 con 0x7f59080b3c70 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41386/0 (socket says 172.21.15.50:41386) 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 -- 172.21.15.50:0/3816033307 learned_addr learned my addr 172.21.15.50:0/3816033307 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f5906ffd700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cecd0 0x7f59080b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59077fe700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 0x7f5908154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 -- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 msgr2=0x7f5908154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 0x7f5908154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 -- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cecd0 msgr2=0x7f59080b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cecd0 0x7f59080b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59067fc700 1 -- 172.21.15.50:0/3816033307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc009420 con 0x7f5908158e20 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.331+0000 7f59077fe700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 0x7f5908154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:25.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f59067fc700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f58fc013000 tx=0x7f58fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc01f070 con 0x7f5908158e20 2023-12-19T14:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc006af0 con 0x7f5908158e20 2023-12-19T14:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc011900 con 0x7f5908158e20 2023-12-19T14:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59080b5de0 con 0x7f5908158e20 2023-12-19T14:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59080c87f0 con 0x7f5908158e20 2023-12-19T14:54:25.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.335+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5908071120 con 0x7f5908158e20 2023-12-19T14:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.339+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f58fc023070 con 0x7f5908158e20 2023-12-19T14:54:25.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.339+0000 7f58eeffd700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 0x7f58f4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:25.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.339+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f58fc07d8b0 con 0x7f5908158e20 2023-12-19T14:54:25.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.339+0000 7f5906ffd700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 0x7f58f4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:25.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.339+0000 7f5906ffd700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 0x7f58f4062b50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58f8004f70 tx=0x7f58f8007d40 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:25.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.347+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f58fc04db90 con 0x7f5908158e20 2023-12-19T14:54:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:25.559+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f590806a7a0 con 0x7f5908158e20 2023-12-19T14:54:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.127+0000 7f58eeffd700 1 -- 172.21.15.50:0/3816033307 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f58fc0150c0 con 0x7f5908158e20 2023-12-19T14:54:26.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 msgr2=0x7f58f4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 0x7f58f4062b50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f58f8004f70 tx=0x7f58f8007d40 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 msgr2=0x7f590814bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f58fc013000 tx=0x7f58fc017000 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 shutdown_connections 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58f40606a0 0x7f58f4062b50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f59080b3c70 0x7f5908154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5908158e20 0x7f590814bd40 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 --2- 172.21.15.50:0/3816033307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080cecd0 0x7f59080b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 >> 172.21.15.50:0/3816033307 conn(0x7f5908067bb0 msgr2=0x7f59080cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 shutdown_connections 2023-12-19T14:54:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.131+0000 7f590d6f0700 1 -- 172.21.15.50:0/3816033307 wait complete. 2023-12-19T14:54:26.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.327+0000 7f808eaae700 1 Processor -- start 2023-12-19T14:54:26.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.327+0000 7f808eaae700 1 -- start start 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808eaae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8088140710 0x7f80880c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808eaae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 0x7f80880ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808eaae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808806e150 con 0x7f8088140ae0 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808eaae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808806d540 con 0x7f8088140710 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808eaae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808806d820 con 0x7f8088068800 2023-12-19T14:54:26.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8088140710 0x7f80880c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8088140710 0x7f80880c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41402/0 (socket says 172.21.15.50:41402) 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 -- 172.21.15.50:0/948442393 learned_addr learned my addr 172.21.15.50:0/948442393 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f807ffff700 1 --1- 172.21.15.50:0/948442393 >> v1:172.21.15.50:6789/0 conn(0x7f8088140ae0 0x7f80880680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43888/0 (socket says 172.21.15.50:43888) 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808d04b700 1 --2- 172.21.15.50:0/948442393 >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 0x7f80880ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 -- 172.21.15.50:0/948442393 >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 msgr2=0x7f80880ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:26.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 --2- 172.21.15.50:0/948442393 >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 0x7f80880ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f807f7fe700 1 -- 172.21.15.50:0/948442393 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330647831 0 0) 0x7f808806e150 con 0x7f8088140ae0 2023-12-19T14:54:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 -- 172.21.15.50:0/948442393 >> v1:172.21.15.50:6789/0 conn(0x7f8088140ae0 legacy=0x7f80880680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 -- 172.21.15.50:0/948442393 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808806db00 con 0x7f8088140710 2023-12-19T14:54:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808d04b700 1 --2- 172.21.15.50:0/948442393 >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 0x7f80880ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:26.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.331+0000 7f808c84a700 1 --2- 172.21.15.50:0/948442393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c70a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f808806bfc0 tx=0x7f8084009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb57a16ddefc2dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f807f7fe700 1 -- 172.21.15.50:0/948442393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808400b860 con 0x7f8088140710 2023-12-19T14:54:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f807f7fe700 1 -- 172.21.15.50:0/948442393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808400b9c0 con 0x7f8088140710 2023-12-19T14:54:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f807f7fe700 1 -- 172.21.15.50:0/948442393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808400bd60 con 0x7f8088140710 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 -- 172.21.15.50:0/948442393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 msgr2=0x7f80880c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 --2- 172.21.15.50:0/948442393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c70a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f808806bfc0 tx=0x7f8084009250 comp rx=0 tx=0).stop 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 -- 172.21.15.50:0/948442393 shutdown_connections 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 --2- 172.21.15.50:0/948442393 >> v2:172.21.15.125:3300/0 conn(0x7f8088068800 0x7f80880ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 --2- 172.21.15.50:0/948442393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c70a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 -- 172.21.15.50:0/948442393 >> 172.21.15.50:0/948442393 conn(0x7f80880c8850 msgr2=0x7f80880c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 -- 172.21.15.50:0/948442393 shutdown_connections 2023-12-19T14:54:26.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.335+0000 7f808eaae700 1 -- 172.21.15.50:0/948442393 wait complete. 2023-12-19T14:54:26.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 Processor -- start 2023-12-19T14:54:26.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 -- start start 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8088068800 0x7f808813f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8088140ae0 0x7f80880c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c4cb0 con 0x7f8088140710 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c5480 con 0x7f8088068800 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808eaae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80880c4010 con 0x7f8088140ae0 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f807ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f807ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41404/0 (socket says 172.21.15.50:41404) 2023-12-19T14:54:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f807ffff700 1 -- 172.21.15.50:0/1068721939 learned_addr learned my addr 172.21.15.50:0/1068721939 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808d04b700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8088140ae0 0x7f80880c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.339+0000 7f808c84a700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8088068800 0x7f808813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 -- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8088140ae0 msgr2=0x7f80880c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8088140ae0 0x7f80880c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 -- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8088068800 msgr2=0x7f808813f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8088068800 0x7f808813f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 -- 172.21.15.50:0/1068721939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840094f0 con 0x7f8088140710 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807ffff700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f807800c7e0 tx=0x7f8078007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078005a70 con 0x7f8088140710 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8078005bd0 con 0x7f8088140710 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80880c3400 con 0x7f8088140710 2023-12-19T14:54:26.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780070d0 con 0x7f8088140710 2023-12-19T14:54:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80880c39c0 con 0x7f8088140710 2023-12-19T14:54:26.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.343+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80880c7c00 con 0x7f8088140710 2023-12-19T14:54:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.347+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8078007440 con 0x7f8088140710 2023-12-19T14:54:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.347+0000 7f807cff9700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 0x7f8074062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.347+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f807807bf90 con 0x7f8088140710 2023-12-19T14:54:26.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.347+0000 7f808c84a700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 0x7f8074062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.351+0000 7f808c84a700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 0x7f8074062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f808806bfc0 tx=0x7f8084017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.355+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f80780abbe0 con 0x7f8088140710 2023-12-19T14:54:26.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.571+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8088071120 con 0x7f8088140710 2023-12-19T14:54:26.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f807cff9700 1 -- 172.21.15.50:0/1068721939 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f8078017090 con 0x7f8088140710 2023-12-19T14:54:26.575 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:26.575 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:26.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 msgr2=0x7f8074062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 0x7f8074062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f808806bfc0 tx=0x7f8084017000 comp rx=0 tx=0).stop 2023-12-19T14:54:26.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 msgr2=0x7f80880c4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f807800c7e0 tx=0x7f8078007e10 comp rx=0 tx=0).stop 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 shutdown_connections 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8074060670 0x7f8074062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8088140ae0 0x7f80880c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8088140710 0x7f80880c4f40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 --2- 172.21.15.50:0/1068721939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8088068800 0x7f808813f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 >> 172.21.15.50:0/1068721939 conn(0x7f80880c8850 msgr2=0x7f80880aab10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 shutdown_connections 2023-12-19T14:54:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.575+0000 7f808eaae700 1 -- 172.21.15.50:0/1068721939 wait complete. 2023-12-19T14:54:26.580 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:26.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.767+0000 7fab82065700 1 Processor -- start 2023-12-19T14:54:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.767+0000 7fab82065700 1 -- start start 2023-12-19T14:54:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.771+0000 7fab82065700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab7c1502b0 0x7fab7c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.771+0000 7fab82065700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c0c5900 0x7fab7c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.771+0000 7fab82065700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c0c6630 con 0x7fab7c067c90 2023-12-19T14:54:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.771+0000 7fab82065700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c0c6910 con 0x7fab7c1502b0 2023-12-19T14:54:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.771+0000 7fab82065700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0c6bf0 con 0x7fab7c0c5900 2023-12-19T14:54:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fab7c067c90 0x7fab7c14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53904/0 (socket says 172.21.15.50:53904) 2023-12-19T14:54:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7b7fe700 1 -- 172.21.15.50:0/78614984 learned_addr learned my addr 172.21.15.50:0/78614984 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 --2- 172.21.15.50:0/78614984 >> v2:172.21.15.125:3300/0 conn(0x7fab7c1502b0 0x7fab7c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7bfff700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c0c5900 0x7fab7c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7a7fc700 1 -- 172.21.15.50:0/78614984 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133828307 0 0) 0x7fab7c0c6630 con 0x7fab7c067c90 2023-12-19T14:54:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7a7fc700 1 -- 172.21.15.50:0/78614984 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab64003570 con 0x7fab7c067c90 2023-12-19T14:54:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 -- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c0c5900 msgr2=0x7fab7c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c0c5900 0x7fab7c0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 -- 172.21.15.50:0/78614984 >> v1:172.21.15.50:6789/0 conn(0x7fab7c067c90 legacy=0x7fab7c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 -- 172.21.15.50:0/78614984 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0c6ed0 con 0x7fab7c1502b0 2023-12-19T14:54:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7affd700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c068060 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab700098e0 tx=0x7fab700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88b32595b006b57a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7a7fc700 1 -- 172.21.15.50:0/78614984 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70011630 con 0x7fab7c1502b0 2023-12-19T14:54:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7a7fc700 1 -- 172.21.15.50:0/78614984 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab70011790 con 0x7fab7c1502b0 2023-12-19T14:54:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab7a7fc700 1 -- 172.21.15.50:0/78614984 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70011b30 con 0x7fab7c1502b0 2023-12-19T14:54:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 -- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 msgr2=0x7fab7c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c068060 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab700098e0 tx=0x7fab700091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 -- 172.21.15.50:0/78614984 shutdown_connections 2023-12-19T14:54:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c0c5900 0x7fab7c0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 --2- 172.21.15.50:0/78614984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c068060 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 -- 172.21.15.50:0/78614984 >> 172.21.15.50:0/78614984 conn(0x7fab7c0cd6d0 msgr2=0x7fab7c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 -- 172.21.15.50:0/78614984 shutdown_connections 2023-12-19T14:54:26.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.775+0000 7fab82065700 1 -- 172.21.15.50:0/78614984 wait complete. 2023-12-19T14:54:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 Processor -- start 2023-12-19T14:54:26.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 -- start start 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c067c90 0x7fab7c16d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c16dce0 con 0x7fab7c0c5900 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c169790 con 0x7fab7c1502b0 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab82065700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c169a50 con 0x7fab7c067c90 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35236/0 (socket says 172.21.15.50:35236) 2023-12-19T14:54:26.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab7affd700 1 -- 172.21.15.50:0/1200598112 learned_addr learned my addr 172.21.15.50:0/1200598112 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab7bfff700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.779+0000 7fab7b7fe700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c067c90 0x7fab7c16d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 -- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c067c90 msgr2=0x7fab7c16d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c067c90 0x7fab7c16d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 -- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 msgr2=0x7fab7c0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 -- 172.21.15.50:0/1200598112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70009420 con 0x7fab7c0c5900 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7bfff700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab7affd700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fab7c06a0f0 tx=0x7fab70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70011b10 con 0x7fab7c0c5900 2023-12-19T14:54:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab700253f0 con 0x7fab7c0c5900 2023-12-19T14:54:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70025790 con 0x7fab7c0c5900 2023-12-19T14:54:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c169d30 con 0x7fab7c0c5900 2023-12-19T14:54:26.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c16a2f0 con 0x7fab7c0c5900 2023-12-19T14:54:26.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.783+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c071120 con 0x7fab7c0c5900 2023-12-19T14:54:26.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.787+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fab70025b80 con 0x7fab7c0c5900 2023-12-19T14:54:26.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.787+0000 7fab637fe700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 0x7fab68062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:26.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.787+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fab7007e2e0 con 0x7fab7c0c5900 2023-12-19T14:54:26.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.787+0000 7fab7b7fe700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 0x7fab68062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:26.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.791+0000 7fab7b7fe700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 0x7fab68062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab7c16ccf0 tx=0x7fab6c009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:26.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.795+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fab7004e5c0 con 0x7fab7c0c5900 2023-12-19T14:54:26.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.971+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab7c0642a0 con 0x7fab7c0c5900 2023-12-19T14:54:26.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.971+0000 7fab637fe700 1 -- 172.21.15.50:0/1200598112 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+15600 (secure 0 0 0) 0x7fab70051be0 con 0x7fab7c0c5900 2023-12-19T14:54:26.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:26.973 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":86,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:23.927647+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:26.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 msgr2=0x7fab68062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 0x7fab68062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab7c16ccf0 tx=0x7fab6c009040 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 msgr2=0x7fab7c14bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fab7c06a0f0 tx=0x7fab70017000 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 shutdown_connections 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab7c1502b0 0x7fab7c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fab68060670 0x7fab68062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7c0c5900 0x7fab7c14bc40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 --2- 172.21.15.50:0/1200598112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7c067c90 0x7fab7c16d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 >> 172.21.15.50:0/1200598112 conn(0x7fab7c0cd6d0 msgr2=0x7fab7c0ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 shutdown_connections 2023-12-19T14:54:26.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:26.975+0000 7fab82065700 1 -- 172.21.15.50:0/1200598112 wait complete. 2023-12-19T14:54:26.995 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:54:26.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:54:27.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.159+0000 7f3889cf0700 1 Processor -- start 2023-12-19T14:54:27.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.159+0000 7f3889cf0700 1 -- start start 2023-12-19T14:54:27.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3889cf0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 0x7f3884068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3889cf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38840c5900 0x7f38840c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3889cf0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840c6630 con 0x7f3884067c90 2023-12-19T14:54:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3889cf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c6910 con 0x7f38840c5900 2023-12-19T14:54:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3889cf0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c6bf0 con 0x7f38841502b0 2023-12-19T14:54:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38840c5900 0x7f38840c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3882ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 0x7f3884068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3882ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 0x7f3884068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42442/0 (socket says 172.21.15.50:42442) 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3882ffd700 1 -- 172.21.15.50:0/408673011 learned_addr learned my addr 172.21.15.50:0/408673011 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38840c5900 0x7f38840c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35250/0 (socket says 172.21.15.50:35250) 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 -- 172.21.15.50:0/408673011 >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 msgr2=0x7f3884068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 --2- 172.21.15.50:0/408673011 >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 0x7f3884068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 -- 172.21.15.50:0/408673011 >> v1:172.21.15.125:6789/0 conn(0x7f3884067c90 legacy=0x7f388414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 -- 172.21.15.50:0/408673011 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840c6ed0 con 0x7f38840c5900 2023-12-19T14:54:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f3883fff700 1 --2- 172.21.15.50:0/408673011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840c5d60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f38840c56b0 tx=0x7f387c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305e10339d7bbe4c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f38827fc700 1 -- 172.21.15.50:0/408673011 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c00e070 con 0x7f38840c5900 2023-12-19T14:54:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.163+0000 7f38827fc700 1 -- 172.21.15.50:0/408673011 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c00bb60 con 0x7f38840c5900 2023-12-19T14:54:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f38827fc700 1 -- 172.21.15.50:0/408673011 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c012430 con 0x7f38840c5900 2023-12-19T14:54:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 -- 172.21.15.50:0/408673011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 msgr2=0x7f38840c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 --2- 172.21.15.50:0/408673011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840c5d60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f38840c56b0 tx=0x7f387c009780 comp rx=0 tx=0).stop 2023-12-19T14:54:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 -- 172.21.15.50:0/408673011 shutdown_connections 2023-12-19T14:54:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 --2- 172.21.15.50:0/408673011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840c5d60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 --2- 172.21.15.50:0/408673011 >> v2:172.21.15.125:3300/0 conn(0x7f38841502b0 0x7f3884068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 -- 172.21.15.50:0/408673011 >> 172.21.15.50:0/408673011 conn(0x7f38840cd6d0 msgr2=0x7f388406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 -- 172.21.15.50:0/408673011 shutdown_connections 2023-12-19T14:54:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.167+0000 7f3889cf0700 1 -- 172.21.15.50:0/408673011 wait complete. 2023-12-19T14:54:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 Processor -- start 2023-12-19T14:54:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 -- start start 2023-12-19T14:54:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3884067c90 0x7f3884169d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 0x7f38840b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c72a0 con 0x7f38840c5900 2023-12-19T14:54:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c8100 con 0x7f38841502b0 2023-12-19T14:54:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.171+0000 7f3889cf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388416a2c0 con 0x7f3884067c90 2023-12-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3883fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 0x7f38840b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35266/0 (socket says 172.21.15.50:35266) 2023-12-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 -- 172.21.15.50:0/3420976104 learned_addr learned my addr 172.21.15.50:0/3420976104 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f38837fe700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3884067c90 0x7f3884169d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 -- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3884067c90 msgr2=0x7f3884169d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3884067c90 0x7f3884169d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 -- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 msgr2=0x7f38840b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 0x7f38840b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 -- 172.21.15.50:0/3420976104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c009a20 con 0x7f38840c5900 2023-12-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3883fff700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 0x7f38840b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3882ffd700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f387800e980 tx=0x7f3878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878013680 con 0x7f38840c5900 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38780137e0 con 0x7f38840c5900 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840c74f0 con 0x7f38840c5900 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878013b60 con 0x7f38840c5900 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.175+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840c7b30 con 0x7f38840c5900 2023-12-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.179+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884071120 con 0x7f38840c5900 2023-12-19T14:54:27.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.179+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f387800c400 con 0x7f38840c5900 2023-12-19T14:54:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.183+0000 7f386b7fe700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 0x7f3870062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.183+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f387807cb30 con 0x7f38840c5900 2023-12-19T14:54:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.183+0000 7f38837fe700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 0x7f3870062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.183+0000 7f38837fe700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 0x7f3870062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3874003a80 tx=0x7f3874007ec0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.187+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3878050940 con 0x7f38840c5900 2023-12-19T14:54:27.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.359+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38840642a0 con 0x7f3870060670 2023-12-19T14:54:27.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.379+0000 7f386b7fe700 1 -- 172.21.15.50:0/3420976104 <== mgr.5720 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286787 (secure 0 0 0) 0x7f38840642a0 con 0x7f3870060670 2023-12-19T14:54:27.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:27.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 msgr2=0x7f3870062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 0x7f3870062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3874003a80 tx=0x7f3874007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 msgr2=0x7f38840b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f387800e980 tx=0x7f3878009250 comp rx=0 tx=0).stop 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 shutdown_connections 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38841502b0 0x7f38840b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3870060670 0x7f3870062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38840c5900 0x7f38840b5100 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 --2- 172.21.15.50:0/3420976104 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3884067c90 0x7f3884169d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 >> 172.21.15.50:0/3420976104 conn(0x7f38840cd6d0 msgr2=0x7f38840c5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 shutdown_connections 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.387+0000 7f3889cf0700 1 -- 172.21.15.50:0/3420976104 wait complete. 2023-12-19T14:54:27.390 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:54:27.408 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:54:26.576027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2440832,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752533888,"statfs":{"total":773094113280,"available":770594701312,"internally_reserved":0,"allocated":9502720,"data_stored":5295758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.524457"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951012+0000","last_change":"2023-12-19T14:54:22.951012+0000","last_active":"2023-12-19T14:54:22.951012+0000","last_peered":"2023-12-19T14:54:22.951012+0000","last_clean":"2023-12-19T14:54:22.951012+0000","last_became_active":"2023-12-19T14:54:22.950613+0000","last_became_peered":"2023-12-19T14:54:22.950613+0000","last_unstale":"2023-12-19T14:54:22.951012+0000","last_undegraded":"2023-12-19T14:54:22.951012+0000","last_fullsized":"2023-12-19T14:54:22.951012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.961303+0000","last_change":"2023-12-19T14:54:22.961303+0000","last_active":"2023-12-19T14:54:22.961303+0000","last_peered":"2023-12-19T14:54:22.961303+0000","last_clean":"2023-12-19T14:54:22.961303+0000","last_became_active":"2023-12-19T14:54:22.960933+0000","last_became_peered":"2023-12-19T14:54:22.960933+0000","last_unstale":"2023-12-19T14:54:22.961303+0000","last_undegraded":"2023-12-19T14:54:22.961303+0000","last_fullsized":"2023-12-19T14:54:22.961303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.956249+0000","last_change":"2023-12-19T14:54:22.956249+0000","last_active":"2023-12-19T14:54:22.956249+0000","last_peered":"2023-12-19T14:54:22.956249+0000","last_clean":"2023-12-19T14:54:22.956249+0000","last_became_active":"2023-12-19T14:54:22.955982+0000","last_became_peered":"2023-12-19T14:54:22.955982+0000","last_unstale":"2023-12-19T14:54:22.956249+0000","last_undegraded":"2023-12-19T14:54:22.956249+0000","last_fullsized":"2023-12-19T14:54:22.956249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962227+0000","last_change":"2023-12-19T14:54:22.962227+0000","last_active":"2023-12-19T14:54:22.962227+0000","last_peered":"2023-12-19T14:54:22.962227+0000","last_clean":"2023-12-19T14:54:22.962227+0000","last_became_active":"2023-12-19T14:54:22.961400+0000","last_became_peered":"2023-12-19T14:54:22.961400+0000","last_unstale":"2023-12-19T14:54:22.962227+0000","last_undegraded":"2023-12-19T14:54:22.962227+0000","last_fullsized":"2023-12-19T14:54:22.962227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:16.693921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957210+0000","last_change":"2023-12-19T14:54:22.955591+0000","last_active":"2023-12-19T14:54:22.957210+0000","last_peered":"2023-12-19T14:54:22.957210+0000","last_clean":"2023-12-19T14:54:22.957210+0000","last_became_active":"2023-12-19T14:54:22.955082+0000","last_became_peered":"2023-12-19T14:54:22.955082+0000","last_unstale":"2023-12-19T14:54:22.957210+0000","last_undegraded":"2023-12-19T14:54:22.957210+0000","last_fullsized":"2023-12-19T14:54:22.957210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:47.300540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959812+0000","last_change":"2023-12-19T14:54:22.959812+0000","last_active":"2023-12-19T14:54:22.959812+0000","last_peered":"2023-12-19T14:54:22.959812+0000","last_clean":"2023-12-19T14:54:22.959812+0000","last_became_active":"2023-12-19T14:54:22.959448+0000","last_became_peered":"2023-12-19T14:54:22.959448+0000","last_unstale":"2023-12-19T14:54:22.959812+0000","last_undegraded":"2023-12-19T14:54:22.959812+0000","last_fullsized":"2023-12-19T14:54:22.959812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:13:43.962908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960007+0000","last_change":"2023-12-19T14:54:22.956689+0000","last_active":"2023-12-19T14:54:22.960007+0000","last_peered":"2023-12-19T14:54:22.960007+0000","last_clean":"2023-12-19T14:54:22.960007+0000","last_became_active":"2023-12-19T14:54:22.956408+0000","last_became_peered":"2023-12-19T14:54:22.956408+0000","last_unstale":"2023-12-19T14:54:22.960007+0000","last_undegraded":"2023-12-19T14:54:22.960007+0000","last_fullsized":"2023-12-19T14:54:22.960007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:03.845407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947337+0000","last_change":"2023-12-19T14:54:22.947337+0000","last_active":"2023-12-19T14:54:22.947337+0000","last_peered":"2023-12-19T14:54:22.947337+0000","last_clean":"2023-12-19T14:54:22.947337+0000","last_became_active":"2023-12-19T14:54:22.947048+0000","last_became_peered":"2023-12-19T14:54:22.947048+0000","last_unstale":"2023-12-19T14:54:22.947337+0000","last_undegraded":"2023-12-19T14:54:22.947337+0000","last_fullsized":"2023-12-19T14:54:22.947337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:33:50.174690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959356+0000","last_change":"2023-12-19T14:54:22.959356+0000","last_active":"2023-12-19T14:54:22.959356+0000","last_peered":"2023-12-19T14:54:22.959356+0000","last_clean":"2023-12-19T14:54:22.959356+0000","last_became_active":"2023-12-19T14:54:22.959099+0000","last_became_peered":"2023-12-19T14:54:22.959099+0000","last_unstale":"2023-12-19T14:54:22.959356+0000","last_undegraded":"2023-12-19T14:54:22.959356+0000","last_fullsized":"2023-12-19T14:54:22.959356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:09.263637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952132+0000","last_change":"2023-12-19T14:54:22.952132+0000","last_active":"2023-12-19T14:54:22.952132+0000","last_peered":"2023-12-19T14:54:22.952132+0000","last_clean":"2023-12-19T14:54:22.952132+0000","last_became_active":"2023-12-19T14:54:22.951855+0000","last_became_peered":"2023-12-19T14:54:22.951855+0000","last_unstale":"2023-12-19T14:54:22.952132+0000","last_undegraded":"2023-12-19T14:54:22.952132+0000","last_fullsized":"2023-12-19T14:54:22.952132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:04:18.531600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958836+0000","last_change":"2023-12-19T14:54:22.958836+0000","last_active":"2023-12-19T14:54:22.958836+0000","last_peered":"2023-12-19T14:54:22.958836+0000","last_clean":"2023-12-19T14:54:22.958836+0000","last_became_active":"2023-12-19T14:54:22.958517+0000","last_became_peered":"2023-12-19T14:54:22.958517+0000","last_unstale":"2023-12-19T14:54:22.958836+0000","last_undegraded":"2023-12-19T14:54:22.958836+0000","last_fullsized":"2023-12-19T14:54:22.958836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:20.969411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952064+0000","last_change":"2023-12-19T14:54:22.952064+0000","last_active":"2023-12-19T14:54:22.952064+0000","last_peered":"2023-12-19T14:54:22.952064+0000","last_clean":"2023-12-19T14:54:22.952064+0000","last_became_active":"2023-12-19T14:54:22.949570+0000","last_became_peered":"2023-12-19T14:54:22.949570+0000","last_unstale":"2023-12-19T14:54:22.952064+0000","last_undegraded":"2023-12-19T14:54:22.952064+0000","last_fullsized":"2023-12-19T14:54:22.952064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:03:09.806691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964019+0000","last_change":"2023-12-19T14:54:22.964019+0000","last_active":"2023-12-19T14:54:22.964019+0000","last_peered":"2023-12-19T14:54:22.964019+0000","last_clean":"2023-12-19T14:54:22.964019+0000","last_became_active":"2023-12-19T14:54:22.963742+0000","last_became_peered":"2023-12-19T14:54:22.963742+0000","last_unstale":"2023-12-19T14:54:22.964019+0000","last_undegraded":"2023-12-19T14:54:22.964019+0000","last_fullsized":"2023-12-19T14:54:22.964019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:07:30.035158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957890+0000","last_change":"2023-12-19T14:54:22.957890+0000","last_active":"2023-12-19T14:54:22.957890+0000","last_peered":"2023-12-19T14:54:22.957890+0000","last_clean":"2023-12-19T14:54:22.957890+0000","last_became_active":"2023-12-19T14:54:22.957600+0000","last_became_peered":"2023-12-19T14:54:22.957600+0000","last_unstale":"2023-12-19T14:54:22.957890+0000","last_undegraded":"2023-12-19T14:54:22.957890+0000","last_fullsized":"2023-12-19T14:54:22.957890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949326+0000","last_change":"2023-12-19T14:54:22.949326+0000","last_active":"2023-12-19T14:54:22.949326+0000","last_peered":"2023-12-19T14:54:22.949326+0000","last_clean":"2023-12-19T14:54:22.949326+0000","last_became_active":"2023-12-19T14:54:22.949010+0000","last_became_peered":"2023-12-19T14:54:22.949010+0000","last_unstale":"2023-12-19T14:54:22.949326+0000","last_undegraded":"2023-12-19T14:54:22.949326+0000","last_fullsized":"2023-12-19T14:54:22.949326+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959699+0000","last_change":"2023-12-19T14:54:22.955463+0000","last_active":"2023-12-19T14:54:22.959699+0000","last_peered":"2023-12-19T14:54:22.959699+0000","last_clean":"2023-12-19T14:54:22.959699+0000","last_became_active":"2023-12-19T14:54:22.954416+0000","last_became_peered":"2023-12-19T14:54:22.954416+0000","last_unstale":"2023-12-19T14:54:22.959699+0000","last_undegraded":"2023-12-19T14:54:22.959699+0000","last_fullsized":"2023-12-19T14:54:22.959699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:52:47.375220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958901+0000","last_change":"2023-12-19T14:54:22.958901+0000","last_active":"2023-12-19T14:54:22.958901+0000","last_peered":"2023-12-19T14:54:22.958901+0000","last_clean":"2023-12-19T14:54:22.958901+0000","last_became_active":"2023-12-19T14:54:22.958471+0000","last_became_peered":"2023-12-19T14:54:22.958471+0000","last_unstale":"2023-12-19T14:54:22.958901+0000","last_undegraded":"2023-12-19T14:54:22.958901+0000","last_fullsized":"2023-12-19T14:54:22.958901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.961903+0000","last_change":"2023-12-19T14:54:22.961903+0000","last_active":"2023-12-19T14:54:22.961903+0000","last_peered":"2023-12-19T14:54:22.961903+0000","last_clean":"2023-12-19T14:54:22.961903+0000","last_became_active":"2023-12-19T14:54:22.961258+0000","last_became_peered":"2023-12-19T14:54:22.961258+0000","last_unstale":"2023-12-19T14:54:22.961903+0000","last_undegraded":"2023-12-19T14:54:22.961903+0000","last_fullsized":"2023-12-19T14:54:22.961903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:09.263637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954525+0000","last_change":"2023-12-19T14:54:22.951844+0000","last_active":"2023-12-19T14:54:22.954525+0000","last_peered":"2023-12-19T14:54:22.954525+0000","last_clean":"2023-12-19T14:54:22.954525+0000","last_became_active":"2023-12-19T14:54:22.950094+0000","last_became_peered":"2023-12-19T14:54:22.950094+0000","last_unstale":"2023-12-19T14:54:22.954525+0000","last_undegraded":"2023-12-19T14:54:22.954525+0000","last_fullsized":"2023-12-19T14:54:22.954525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:57.078664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949491+0000","last_change":"2023-12-19T14:54:22.949491+0000","last_active":"2023-12-19T14:54:22.949491+0000","last_peered":"2023-12-19T14:54:22.949491+0000","last_clean":"2023-12-19T14:54:22.949491+0000","last_became_active":"2023-12-19T14:54:22.949191+0000","last_became_peered":"2023-12-19T14:54:22.949191+0000","last_unstale":"2023-12-19T14:54:22.949491+0000","last_undegraded":"2023-12-19T14:54:22.949491+0000","last_fullsized":"2023-12-19T14:54:22.949491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:34.627955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948377+0000","last_change":"2023-12-19T14:54:22.948377+0000","last_active":"2023-12-19T14:54:22.948377+0000","last_peered":"2023-12-19T14:54:22.948377+0000","last_clean":"2023-12-19T14:54:22.948377+0000","last_became_active":"2023-12-19T14:54:22.948042+0000","last_became_peered":"2023-12-19T14:54:22.948042+0000","last_unstale":"2023-12-19T14:54:22.948377+0000","last_undegraded":"2023-12-19T14:54:22.948377+0000","last_fullsized":"2023-12-19T14:54:22.948377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:46.333985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951521+0000","last_change":"2023-12-19T14:54:22.951521+0000","last_active":"2023-12-19T14:54:22.951521+0000","last_peered":"2023-12-19T14:54:22.951521+0000","last_clean":"2023-12-19T14:54:22.951521+0000","last_became_active":"2023-12-19T14:54:22.951211+0000","last_became_peered":"2023-12-19T14:54:22.951211+0000","last_unstale":"2023-12-19T14:54:22.951521+0000","last_undegraded":"2023-12-19T14:54:22.951521+0000","last_fullsized":"2023-12-19T14:54:22.951521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:13:43.962908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950530+0000","last_change":"2023-12-19T14:54:22.950530+0000","last_active":"2023-12-19T14:54:22.950530+0000","last_peered":"2023-12-19T14:54:22.950530+0000","last_clean":"2023-12-19T14:54:22.950530+0000","last_became_active":"2023-12-19T14:54:22.950238+0000","last_became_peered":"2023-12-19T14:54:22.950238+0000","last_unstale":"2023-12-19T14:54:22.950530+0000","last_undegraded":"2023-12-19T14:54:22.950530+0000","last_fullsized":"2023-12-19T14:54:22.950530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:26.149145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958637+0000","last_change":"2023-12-19T14:54:22.958637+0000","last_active":"2023-12-19T14:54:22.958637+0000","last_peered":"2023-12-19T14:54:22.958637+0000","last_clean":"2023-12-19T14:54:22.958637+0000","last_became_active":"2023-12-19T14:54:22.957866+0000","last_became_peered":"2023-12-19T14:54:22.957866+0000","last_unstale":"2023-12-19T14:54:22.958637+0000","last_undegraded":"2023-12-19T14:54:22.958637+0000","last_fullsized":"2023-12-19T14:54:22.958637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954805+0000","last_change":"2023-12-19T14:54:22.954805+0000","last_active":"2023-12-19T14:54:22.954805+0000","last_peered":"2023-12-19T14:54:22.954805+0000","last_clean":"2023-12-19T14:54:22.954805+0000","last_became_active":"2023-12-19T14:54:22.954533+0000","last_became_peered":"2023-12-19T14:54:22.954533+0000","last_unstale":"2023-12-19T14:54:22.954805+0000","last_undegraded":"2023-12-19T14:54:22.954805+0000","last_fullsized":"2023-12-19T14:54:22.954805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:20:32.162345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951333+0000","last_change":"2023-12-19T14:54:22.951333+0000","last_active":"2023-12-19T14:54:22.951333+0000","last_peered":"2023-12-19T14:54:22.951333+0000","last_clean":"2023-12-19T14:54:22.951333+0000","last_became_active":"2023-12-19T14:54:22.951074+0000","last_became_peered":"2023-12-19T14:54:22.951074+0000","last_unstale":"2023-12-19T14:54:22.951333+0000","last_undegraded":"2023-12-19T14:54:22.951333+0000","last_fullsized":"2023-12-19T14:54:22.951333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:35:39.864176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959436+0000","last_change":"2023-12-19T14:54:22.959436+0000","last_active":"2023-12-19T14:54:22.959436+0000","last_peered":"2023-12-19T14:54:22.959436+0000","last_clean":"2023-12-19T14:54:22.959436+0000","last_became_active":"2023-12-19T14:54:22.959119+0000","last_became_peered":"2023-12-19T14:54:22.959119+0000","last_unstale":"2023-12-19T14:54:22.959436+0000","last_undegraded":"2023-12-19T14:54:22.959436+0000","last_fullsized":"2023-12-19T14:54:22.959436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949150+0000","last_change":"2023-12-19T14:54:22.949150+0000","last_active":"2023-12-19T14:54:22.949150+0000","last_peered":"2023-12-19T14:54:22.949150+0000","last_clean":"2023-12-19T14:54:22.949150+0000","last_became_active":"2023-12-19T14:54:22.948392+0000","last_became_peered":"2023-12-19T14:54:22.948392+0000","last_unstale":"2023-12-19T14:54:22.949150+0000","last_undegraded":"2023-12-19T14:54:22.949150+0000","last_fullsized":"2023-12-19T14:54:22.949150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:17.648233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964117+0000","last_change":"2023-12-19T14:54:22.964117+0000","last_active":"2023-12-19T14:54:22.964117+0000","last_peered":"2023-12-19T14:54:22.964117+0000","last_clean":"2023-12-19T14:54:22.964117+0000","last_became_active":"2023-12-19T14:54:22.963271+0000","last_became_peered":"2023-12-19T14:54:22.963271+0000","last_unstale":"2023-12-19T14:54:22.964117+0000","last_undegraded":"2023-12-19T14:54:22.964117+0000","last_fullsized":"2023-12-19T14:54:22.964117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:16.693921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947931+0000","last_change":"2023-12-19T14:54:22.947931+0000","last_active":"2023-12-19T14:54:22.947931+0000","last_peered":"2023-12-19T14:54:22.947931+0000","last_clean":"2023-12-19T14:54:22.947931+0000","last_became_active":"2023-12-19T14:54:22.947651+0000","last_became_peered":"2023-12-19T14:54:22.947651+0000","last_unstale":"2023-12-19T14:54:22.947931+0000","last_undegraded":"2023-12-19T14:54:22.947931+0000","last_fullsized":"2023-12-19T14:54:22.947931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:21:14.211842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954874+0000","last_change":"2023-12-19T14:54:22.954874+0000","last_active":"2023-12-19T14:54:22.954874+0000","last_peered":"2023-12-19T14:54:22.954874+0000","last_clean":"2023-12-19T14:54:22.954874+0000","last_became_active":"2023-12-19T14:54:22.954516+0000","last_became_peered":"2023-12-19T14:54:22.954516+0000","last_unstale":"2023-12-19T14:54:22.954874+0000","last_undegraded":"2023-12-19T14:54:22.954874+0000","last_fullsized":"2023-12-19T14:54:22.954874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960000+0000","last_change":"2023-12-19T14:54:22.960000+0000","last_active":"2023-12-19T14:54:22.960000+0000","last_peered":"2023-12-19T14:54:22.960000+0000","last_clean":"2023-12-19T14:54:22.960000+0000","last_became_active":"2023-12-19T14:54:22.959686+0000","last_became_peered":"2023-12-19T14:54:22.959686+0000","last_unstale":"2023-12-19T14:54:22.960000+0000","last_undegraded":"2023-12-19T14:54:22.960000+0000","last_fullsized":"2023-12-19T14:54:22.960000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:45.306170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948280+0000","last_change":"2023-12-19T14:54:22.944055+0000","last_active":"2023-12-19T14:54:22.948280+0000","last_peered":"2023-12-19T14:54:22.948280+0000","last_clean":"2023-12-19T14:54:22.948280+0000","last_became_active":"2023-12-19T14:54:22.943691+0000","last_became_peered":"2023-12-19T14:54:22.943691+0000","last_unstale":"2023-12-19T14:54:22.948280+0000","last_undegraded":"2023-12-19T14:54:22.948280+0000","last_fullsized":"2023-12-19T14:54:22.948280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:08.837335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"85'4","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.107141+0000","last_change":"2023-12-19T14:54:22.082231+0000","last_active":"2023-12-19T14:54:23.107141+0000","last_peered":"2023-12-19T14:54:23.107141+0000","last_clean":"2023-12-19T14:54:23.107141+0000","last_became_active":"2023-12-19T14:54:22.080837+0000","last_became_peered":"2023-12-19T14:54:22.080837+0000","last_unstale":"2023-12-19T14:54:23.107141+0000","last_undegraded":"2023-12-19T14:54:23.107141+0000","last_fullsized":"2023-12-19T14:54:23.107141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962609+0000","last_change":"2023-12-19T14:54:22.962609+0000","last_active":"2023-12-19T14:54:22.962609+0000","last_peered":"2023-12-19T14:54:22.962609+0000","last_clean":"2023-12-19T14:54:22.962609+0000","last_became_active":"2023-12-19T14:54:22.961296+0000","last_became_peered":"2023-12-19T14:54:22.961296+0000","last_unstale":"2023-12-19T14:54:22.962609+0000","last_undegraded":"2023-12-19T14:54:22.962609+0000","last_fullsized":"2023-12-19T14:54:22.962609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947940+0000","last_change":"2023-12-19T14:54:22.097609+0000","last_active":"2023-12-19T14:54:22.947940+0000","last_peered":"2023-12-19T14:54:22.947940+0000","last_clean":"2023-12-19T14:54:22.947940+0000","last_became_active":"2023-12-19T14:54:22.096727+0000","last_became_peered":"2023-12-19T14:54:22.096727+0000","last_unstale":"2023-12-19T14:54:22.947940+0000","last_undegraded":"2023-12-19T14:54:22.947940+0000","last_fullsized":"2023-12-19T14:54:22.947940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:38.774987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958987+0000","last_change":"2023-12-19T14:54:22.958987+0000","last_active":"2023-12-19T14:54:22.958987+0000","last_peered":"2023-12-19T14:54:22.958987+0000","last_clean":"2023-12-19T14:54:22.958987+0000","last_became_active":"2023-12-19T14:54:22.958645+0000","last_became_peered":"2023-12-19T14:54:22.958645+0000","last_unstale":"2023-12-19T14:54:22.958987+0000","last_undegraded":"2023-12-19T14:54:22.958987+0000","last_fullsized":"2023-12-19T14:54:22.958987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:59.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102693+0000","last_change":"2023-12-19T14:54:22.100000+0000","last_active":"2023-12-19T14:54:23.102693+0000","last_peered":"2023-12-19T14:54:23.102693+0000","last_clean":"2023-12-19T14:54:23.102693+0000","last_became_active":"2023-12-19T14:54:22.099669+0000","last_became_peered":"2023-12-19T14:54:22.099669+0000","last_unstale":"2023-12-19T14:54:23.102693+0000","last_undegraded":"2023-12-19T14:54:23.102693+0000","last_fullsized":"2023-12-19T14:54:23.102693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:15.681450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954603+0000","last_change":"2023-12-19T14:54:22.954603+0000","last_active":"2023-12-19T14:54:22.954603+0000","last_peered":"2023-12-19T14:54:22.954603+0000","last_clean":"2023-12-19T14:54:22.954603+0000","last_became_active":"2023-12-19T14:54:22.953201+0000","last_became_peered":"2023-12-19T14:54:22.953201+0000","last_unstale":"2023-12-19T14:54:22.954603+0000","last_undegraded":"2023-12-19T14:54:22.954603+0000","last_fullsized":"2023-12-19T14:54:22.954603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.103829+0000","last_change":"2023-12-19T14:54:22.088620+0000","last_active":"2023-12-19T14:54:23.103829+0000","last_peered":"2023-12-19T14:54:23.103829+0000","last_clean":"2023-12-19T14:54:23.103829+0000","last_became_active":"2023-12-19T14:54:22.086527+0000","last_became_peered":"2023-12-19T14:54:22.086527+0000","last_unstale":"2023-12-19T14:54:23.103829+0000","last_undegraded":"2023-12-19T14:54:23.103829+0000","last_fullsized":"2023-12-19T14:54:23.103829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:19.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.955572+0000","last_change":"2023-12-19T14:54:22.947054+0000","last_active":"2023-12-19T14:54:22.955572+0000","last_peered":"2023-12-19T14:54:22.955572+0000","last_clean":"2023-12-19T14:54:22.955572+0000","last_became_active":"2023-12-19T14:54:22.946734+0000","last_became_peered":"2023-12-19T14:54:22.946734+0000","last_unstale":"2023-12-19T14:54:22.955572+0000","last_undegraded":"2023-12-19T14:54:22.955572+0000","last_fullsized":"2023-12-19T14:54:22.955572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:33:34.772110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.938789+0000","last_change":"2023-12-19T14:54:22.088785+0000","last_active":"2023-12-19T14:54:22.938789+0000","last_peered":"2023-12-19T14:54:22.938789+0000","last_clean":"2023-12-19T14:54:22.938789+0000","last_became_active":"2023-12-19T14:54:22.088529+0000","last_became_peered":"2023-12-19T14:54:22.088529+0000","last_unstale":"2023-12-19T14:54:22.938789+0000","last_undegraded":"2023-12-19T14:54:22.938789+0000","last_fullsized":"2023-12-19T14:54:22.938789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:19.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.965269+0000","last_change":"2023-12-19T14:54:22.965269+0000","last_active":"2023-12-19T14:54:22.965269+0000","last_peered":"2023-12-19T14:54:22.965269+0000","last_clean":"2023-12-19T14:54:22.965269+0000","last_became_active":"2023-12-19T14:54:22.964328+0000","last_became_peered":"2023-12-19T14:54:22.964328+0000","last_unstale":"2023-12-19T14:54:22.965269+0000","last_undegraded":"2023-12-19T14:54:22.965269+0000","last_fullsized":"2023-12-19T14:54:22.965269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.939322+0000","last_change":"2023-12-19T14:54:22.095788+0000","last_active":"2023-12-19T14:54:22.939322+0000","last_peered":"2023-12-19T14:54:22.939322+0000","last_clean":"2023-12-19T14:54:22.939322+0000","last_became_active":"2023-12-19T14:54:22.095449+0000","last_became_peered":"2023-12-19T14:54:22.095449+0000","last_unstale":"2023-12-19T14:54:22.939322+0000","last_undegraded":"2023-12-19T14:54:22.939322+0000","last_fullsized":"2023-12-19T14:54:22.939322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:15.681450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962443+0000","last_change":"2023-12-19T14:54:22.957103+0000","last_active":"2023-12-19T14:54:22.962443+0000","last_peered":"2023-12-19T14:54:22.962443+0000","last_clean":"2023-12-19T14:54:22.962443+0000","last_became_active":"2023-12-19T14:54:22.956844+0000","last_became_peered":"2023-12-19T14:54:22.956844+0000","last_unstale":"2023-12-19T14:54:22.962443+0000","last_undegraded":"2023-12-19T14:54:22.962443+0000","last_fullsized":"2023-12-19T14:54:22.962443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:10:37.977882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098264+0000","last_change":"2023-12-19T14:54:22.094744+0000","last_active":"2023-12-19T14:54:22.098264+0000","last_peered":"2023-12-19T14:54:22.098264+0000","last_clean":"2023-12-19T14:54:22.098264+0000","last_became_active":"2023-12-19T14:54:22.092224+0000","last_became_peered":"2023-12-19T14:54:22.092224+0000","last_unstale":"2023-12-19T14:54:22.098264+0000","last_undegraded":"2023-12-19T14:54:22.098264+0000","last_fullsized":"2023-12-19T14:54:22.098264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:28.510850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.967881+0000","last_change":"2023-12-19T14:54:22.967881+0000","last_active":"2023-12-19T14:54:22.967881+0000","last_peered":"2023-12-19T14:54:22.967881+0000","last_clean":"2023-12-19T14:54:22.967881+0000","last_became_active":"2023-12-19T14:54:22.967554+0000","last_became_peered":"2023-12-19T14:54:22.967554+0000","last_unstale":"2023-12-19T14:54:22.967881+0000","last_undegraded":"2023-12-19T14:54:22.967881+0000","last_fullsized":"2023-12-19T14:54:22.967881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:45.306170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.938996+0000","last_change":"2023-12-19T14:54:22.087757+0000","last_active":"2023-12-19T14:54:22.938996+0000","last_peered":"2023-12-19T14:54:22.938996+0000","last_clean":"2023-12-19T14:54:22.938996+0000","last_became_active":"2023-12-19T14:54:22.086515+0000","last_became_peered":"2023-12-19T14:54:22.086515+0000","last_unstale":"2023-12-19T14:54:22.938996+0000","last_undegraded":"2023-12-19T14:54:22.938996+0000","last_fullsized":"2023-12-19T14:54:22.938996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:46:32.604895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952303+0000","last_change":"2023-12-19T14:54:22.952303+0000","last_active":"2023-12-19T14:54:22.952303+0000","last_peered":"2023-12-19T14:54:22.952303+0000","last_clean":"2023-12-19T14:54:22.952303+0000","last_became_active":"2023-12-19T14:54:22.951825+0000","last_became_peered":"2023-12-19T14:54:22.951825+0000","last_unstale":"2023-12-19T14:54:22.952303+0000","last_undegraded":"2023-12-19T14:54:22.952303+0000","last_fullsized":"2023-12-19T14:54:22.952303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.093843+0000","last_change":"2023-12-19T14:54:22.090063+0000","last_active":"2023-12-19T14:54:22.093843+0000","last_peered":"2023-12-19T14:54:22.093843+0000","last_clean":"2023-12-19T14:54:22.093843+0000","last_became_active":"2023-12-19T14:54:22.089745+0000","last_became_peered":"2023-12-19T14:54:22.089745+0000","last_unstale":"2023-12-19T14:54:22.093843+0000","last_undegraded":"2023-12-19T14:54:22.093843+0000","last_fullsized":"2023-12-19T14:54:22.093843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:43.975646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948695+0000","last_change":"2023-12-19T14:54:22.948695+0000","last_active":"2023-12-19T14:54:22.948695+0000","last_peered":"2023-12-19T14:54:22.948695+0000","last_clean":"2023-12-19T14:54:22.948695+0000","last_became_active":"2023-12-19T14:54:22.948320+0000","last_became_peered":"2023-12-19T14:54:22.948320+0000","last_unstale":"2023-12-19T14:54:22.948695+0000","last_undegraded":"2023-12-19T14:54:22.948695+0000","last_fullsized":"2023-12-19T14:54:22.948695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:59.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.096110+0000","last_change":"2023-12-19T14:54:22.093910+0000","last_active":"2023-12-19T14:54:22.096110+0000","last_peered":"2023-12-19T14:54:22.096110+0000","last_clean":"2023-12-19T14:54:22.096110+0000","last_became_active":"2023-12-19T14:54:22.093554+0000","last_became_peered":"2023-12-19T14:54:22.093554+0000","last_unstale":"2023-12-19T14:54:22.096110+0000","last_undegraded":"2023-12-19T14:54:22.096110+0000","last_fullsized":"2023-12-19T14:54:22.096110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954000+0000","last_change":"2023-12-19T14:54:22.949728+0000","last_active":"2023-12-19T14:54:22.954000+0000","last_peered":"2023-12-19T14:54:22.954000+0000","last_clean":"2023-12-19T14:54:22.954000+0000","last_became_active":"2023-12-19T14:54:22.948660+0000","last_became_peered":"2023-12-19T14:54:22.948660+0000","last_unstale":"2023-12-19T14:54:22.954000+0000","last_undegraded":"2023-12-19T14:54:22.954000+0000","last_fullsized":"2023-12-19T14:54:22.954000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:39.502278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"85'3","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.103976+0000","last_change":"2023-12-19T14:54:22.088152+0000","last_active":"2023-12-19T14:54:23.103976+0000","last_peered":"2023-12-19T14:54:23.103976+0000","last_clean":"2023-12-19T14:54:23.103976+0000","last_became_active":"2023-12-19T14:54:22.087870+0000","last_became_peered":"2023-12-19T14:54:22.087870+0000","last_unstale":"2023-12-19T14:54:23.103976+0000","last_undegraded":"2023-12-19T14:54:23.103976+0000","last_fullsized":"2023-12-19T14:54:23.103976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:04.952498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959993+0000","last_change":"2023-12-19T14:54:22.959993+0000","last_active":"2023-12-19T14:54:22.959993+0000","last_peered":"2023-12-19T14:54:22.959993+0000","last_clean":"2023-12-19T14:54:22.959993+0000","last_became_active":"2023-12-19T14:54:22.958434+0000","last_became_peered":"2023-12-19T14:54:22.958434+0000","last_unstale":"2023-12-19T14:54:22.959993+0000","last_undegraded":"2023-12-19T14:54:22.959993+0000","last_fullsized":"2023-12-19T14:54:22.959993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:17.648233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"85'6","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.107457+0000","last_change":"2023-12-19T14:54:22.093878+0000","last_active":"2023-12-19T14:54:23.107457+0000","last_peered":"2023-12-19T14:54:23.107457+0000","last_clean":"2023-12-19T14:54:23.107457+0000","last_became_active":"2023-12-19T14:54:22.093556+0000","last_became_peered":"2023-12-19T14:54:22.093556+0000","last_unstale":"2023-12-19T14:54:23.107457+0000","last_undegraded":"2023-12-19T14:54:23.107457+0000","last_fullsized":"2023-12-19T14:54:23.107457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:46.288069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.955764+0000","last_change":"2023-12-19T14:54:22.955764+0000","last_active":"2023-12-19T14:54:22.955764+0000","last_peered":"2023-12-19T14:54:22.955764+0000","last_clean":"2023-12-19T14:54:22.955764+0000","last_became_active":"2023-12-19T14:54:22.955431+0000","last_became_peered":"2023-12-19T14:54:22.955431+0000","last_unstale":"2023-12-19T14:54:22.955764+0000","last_undegraded":"2023-12-19T14:54:22.955764+0000","last_fullsized":"2023-12-19T14:54:22.955764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102059+0000","last_change":"2023-12-19T14:54:22.087341+0000","last_active":"2023-12-19T14:54:23.102059+0000","last_peered":"2023-12-19T14:54:23.102059+0000","last_clean":"2023-12-19T14:54:23.102059+0000","last_became_active":"2023-12-19T14:54:22.087061+0000","last_became_peered":"2023-12-19T14:54:22.087061+0000","last_unstale":"2023-12-19T14:54:23.102059+0000","last_undegraded":"2023-12-19T14:54:23.102059+0000","last_fullsized":"2023-12-19T14:54:23.102059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:38.489807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950613+0000","last_change":"2023-12-19T14:54:22.950613+0000","last_active":"2023-12-19T14:54:22.950613+0000","last_peered":"2023-12-19T14:54:22.950613+0000","last_clean":"2023-12-19T14:54:22.950613+0000","last_became_active":"2023-12-19T14:54:22.950339+0000","last_became_peered":"2023-12-19T14:54:22.950339+0000","last_unstale":"2023-12-19T14:54:22.950613+0000","last_undegraded":"2023-12-19T14:54:22.950613+0000","last_fullsized":"2023-12-19T14:54:22.950613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:03.971129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095972+0000","last_change":"2023-12-19T14:54:22.094574+0000","last_active":"2023-12-19T14:54:22.095972+0000","last_peered":"2023-12-19T14:54:22.095972+0000","last_clean":"2023-12-19T14:54:22.095972+0000","last_became_active":"2023-12-19T14:54:22.094262+0000","last_became_peered":"2023-12-19T14:54:22.094262+0000","last_unstale":"2023-12-19T14:54:22.095972+0000","last_undegraded":"2023-12-19T14:54:22.095972+0000","last_fullsized":"2023-12-19T14:54:22.095972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:35.881438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954322+0000","last_change":"2023-12-19T14:54:22.950865+0000","last_active":"2023-12-19T14:54:22.954322+0000","last_peered":"2023-12-19T14:54:22.954322+0000","last_clean":"2023-12-19T14:54:22.954322+0000","last_became_active":"2023-12-19T14:54:22.950159+0000","last_became_peered":"2023-12-19T14:54:22.950159+0000","last_unstale":"2023-12-19T14:54:22.954322+0000","last_undegraded":"2023-12-19T14:54:22.954322+0000","last_fullsized":"2023-12-19T14:54:22.954322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:48.373908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095211+0000","last_change":"2023-12-19T14:54:22.092245+0000","last_active":"2023-12-19T14:54:22.095211+0000","last_peered":"2023-12-19T14:54:22.095211+0000","last_clean":"2023-12-19T14:54:22.095211+0000","last_became_active":"2023-12-19T14:54:22.091985+0000","last_became_peered":"2023-12-19T14:54:22.091985+0000","last_unstale":"2023-12-19T14:54:22.095211+0000","last_undegraded":"2023-12-19T14:54:22.095211+0000","last_fullsized":"2023-12-19T14:54:22.095211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:46.288069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954381+0000","last_change":"2023-12-19T14:54:22.954381+0000","last_active":"2023-12-19T14:54:22.954381+0000","last_peered":"2023-12-19T14:54:22.954381+0000","last_clean":"2023-12-19T14:54:22.954381+0000","last_became_active":"2023-12-19T14:54:22.953840+0000","last_became_peered":"2023-12-19T14:54:22.953840+0000","last_unstale":"2023-12-19T14:54:22.954381+0000","last_undegraded":"2023-12-19T14:54:22.954381+0000","last_fullsized":"2023-12-19T14:54:22.954381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:24:08.256925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"85'8","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.105454+0000","last_change":"2023-12-19T14:54:22.087039+0000","last_active":"2023-12-19T14:54:23.105454+0000","last_peered":"2023-12-19T14:54:23.105454+0000","last_clean":"2023-12-19T14:54:23.105454+0000","last_became_active":"2023-12-19T14:54:22.086741+0000","last_became_peered":"2023-12-19T14:54:22.086741+0000","last_unstale":"2023-12-19T14:54:23.105454+0000","last_undegraded":"2023-12-19T14:54:23.105454+0000","last_fullsized":"2023-12-19T14:54:23.105454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960903+0000","last_change":"2023-12-19T14:54:22.960903+0000","last_active":"2023-12-19T14:54:22.960903+0000","last_peered":"2023-12-19T14:54:22.960903+0000","last_clean":"2023-12-19T14:54:22.960903+0000","last_became_active":"2023-12-19T14:54:22.960558+0000","last_became_peered":"2023-12-19T14:54:22.960558+0000","last_unstale":"2023-12-19T14:54:22.960903+0000","last_undegraded":"2023-12-19T14:54:22.960903+0000","last_fullsized":"2023-12-19T14:54:22.960903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.940395+0000","last_change":"2023-12-19T14:54:22.095403+0000","last_active":"2023-12-19T14:54:22.940395+0000","last_peered":"2023-12-19T14:54:22.940395+0000","last_clean":"2023-12-19T14:54:22.940395+0000","last_became_active":"2023-12-19T14:54:22.095005+0000","last_became_peered":"2023-12-19T14:54:22.095005+0000","last_unstale":"2023-12-19T14:54:22.940395+0000","last_undegraded":"2023-12-19T14:54:22.940395+0000","last_fullsized":"2023-12-19T14:54:22.940395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:16.166610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.966650+0000","last_change":"2023-12-19T14:54:22.961917+0000","last_active":"2023-12-19T14:54:22.966650+0000","last_peered":"2023-12-19T14:54:22.966650+0000","last_clean":"2023-12-19T14:54:22.966650+0000","last_became_active":"2023-12-19T14:54:22.960852+0000","last_became_peered":"2023-12-19T14:54:22.960852+0000","last_unstale":"2023-12-19T14:54:22.966650+0000","last_undegraded":"2023-12-19T14:54:22.966650+0000","last_fullsized":"2023-12-19T14:54:22.966650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:20.710143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102211+0000","last_change":"2023-12-19T14:54:22.092851+0000","last_active":"2023-12-19T14:54:23.102211+0000","last_peered":"2023-12-19T14:54:23.102211+0000","last_clean":"2023-12-19T14:54:23.102211+0000","last_became_active":"2023-12-19T14:54:22.092583+0000","last_became_peered":"2023-12-19T14:54:22.092583+0000","last_unstale":"2023-12-19T14:54:23.102211+0000","last_undegraded":"2023-12-19T14:54:23.102211+0000","last_fullsized":"2023-12-19T14:54:23.102211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959319+0000","last_change":"2023-12-19T14:54:22.959319+0000","last_active":"2023-12-19T14:54:22.959319+0000","last_peered":"2023-12-19T14:54:22.959319+0000","last_clean":"2023-12-19T14:54:22.959319+0000","last_became_active":"2023-12-19T14:54:22.959046+0000","last_became_peered":"2023-12-19T14:54:22.959046+0000","last_unstale":"2023-12-19T14:54:22.959319+0000","last_undegraded":"2023-12-19T14:54:22.959319+0000","last_fullsized":"2023-12-19T14:54:22.959319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.941908+0000","last_change":"2023-12-19T14:54:22.091281+0000","last_active":"2023-12-19T14:54:22.941908+0000","last_peered":"2023-12-19T14:54:22.941908+0000","last_clean":"2023-12-19T14:54:22.941908+0000","last_became_active":"2023-12-19T14:54:22.090956+0000","last_became_peered":"2023-12-19T14:54:22.090956+0000","last_unstale":"2023-12-19T14:54:22.941908+0000","last_undegraded":"2023-12-19T14:54:22.941908+0000","last_fullsized":"2023-12-19T14:54:22.941908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:47:42.172427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.953629+0000","last_change":"2023-12-19T14:54:22.946425+0000","last_active":"2023-12-19T14:54:22.953629+0000","last_peered":"2023-12-19T14:54:22.953629+0000","last_clean":"2023-12-19T14:54:22.953629+0000","last_became_active":"2023-12-19T14:54:22.946100+0000","last_became_peered":"2023-12-19T14:54:22.946100+0000","last_unstale":"2023-12-19T14:54:22.953629+0000","last_undegraded":"2023-12-19T14:54:22.953629+0000","last_fullsized":"2023-12-19T14:54:22.953629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:44.988117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098055+0000","last_change":"2023-12-19T14:54:22.094014+0000","last_active":"2023-12-19T14:54:22.098055+0000","last_peered":"2023-12-19T14:54:22.098055+0000","last_clean":"2023-12-19T14:54:22.098055+0000","last_became_active":"2023-12-19T14:54:22.093735+0000","last_became_peered":"2023-12-19T14:54:22.093735+0000","last_unstale":"2023-12-19T14:54:22.098055+0000","last_undegraded":"2023-12-19T14:54:22.098055+0000","last_fullsized":"2023-12-19T14:54:22.098055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:03.028588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958158+0000","last_change":"2023-12-19T14:54:22.958158+0000","last_active":"2023-12-19T14:54:22.958158+0000","last_peered":"2023-12-19T14:54:22.958158+0000","last_clean":"2023-12-19T14:54:22.958158+0000","last_became_active":"2023-12-19T14:54:22.957834+0000","last_became_peered":"2023-12-19T14:54:22.957834+0000","last_unstale":"2023-12-19T14:54:22.958158+0000","last_undegraded":"2023-12-19T14:54:22.958158+0000","last_fullsized":"2023-12-19T14:54:22.958158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:34.627955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.104761+0000","last_change":"2023-12-19T14:54:22.094508+0000","last_active":"2023-12-19T14:54:23.104761+0000","last_peered":"2023-12-19T14:54:23.104761+0000","last_clean":"2023-12-19T14:54:23.104761+0000","last_became_active":"2023-12-19T14:54:22.091569+0000","last_became_peered":"2023-12-19T14:54:22.091569+0000","last_unstale":"2023-12-19T14:54:23.104761+0000","last_undegraded":"2023-12-19T14:54:23.104761+0000","last_fullsized":"2023-12-19T14:54:23.104761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:08.251166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949665+0000","last_change":"2023-12-19T14:54:22.949665+0000","last_active":"2023-12-19T14:54:22.949665+0000","last_peered":"2023-12-19T14:54:22.949665+0000","last_clean":"2023-12-19T14:54:22.949665+0000","last_became_active":"2023-12-19T14:54:22.949377+0000","last_became_peered":"2023-12-19T14:54:22.949377+0000","last_unstale":"2023-12-19T14:54:22.949665+0000","last_undegraded":"2023-12-19T14:54:22.949665+0000","last_fullsized":"2023-12-19T14:54:22.949665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:17.716909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.091242+0000","last_change":"2023-12-19T14:54:22.089876+0000","last_active":"2023-12-19T14:54:22.091242+0000","last_peered":"2023-12-19T14:54:22.091242+0000","last_clean":"2023-12-19T14:54:22.091242+0000","last_became_active":"2023-12-19T14:54:22.089585+0000","last_became_peered":"2023-12-19T14:54:22.089585+0000","last_unstale":"2023-12-19T14:54:22.091242+0000","last_undegraded":"2023-12-19T14:54:22.091242+0000","last_fullsized":"2023-12-19T14:54:22.091242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:55.035096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958359+0000","last_change":"2023-12-19T14:54:22.956348+0000","last_active":"2023-12-19T14:54:22.958359+0000","last_peered":"2023-12-19T14:54:22.958359+0000","last_clean":"2023-12-19T14:54:22.958359+0000","last_became_active":"2023-12-19T14:54:22.956050+0000","last_became_peered":"2023-12-19T14:54:22.956050+0000","last_unstale":"2023-12-19T14:54:22.958359+0000","last_undegraded":"2023-12-19T14:54:22.958359+0000","last_fullsized":"2023-12-19T14:54:22.958359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:31.728767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098105+0000","last_change":"2023-12-19T14:54:22.094252+0000","last_active":"2023-12-19T14:54:22.098105+0000","last_peered":"2023-12-19T14:54:22.098105+0000","last_clean":"2023-12-19T14:54:22.098105+0000","last_became_active":"2023-12-19T14:54:22.093935+0000","last_became_peered":"2023-12-19T14:54:22.093935+0000","last_unstale":"2023-12-19T14:54:22.098105+0000","last_undegraded":"2023-12-19T14:54:22.098105+0000","last_fullsized":"2023-12-19T14:54:22.098105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:51.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964693+0000","last_change":"2023-12-19T14:54:22.964693+0000","last_active":"2023-12-19T14:54:22.964693+0000","last_peered":"2023-12-19T14:54:22.964693+0000","last_clean":"2023-12-19T14:54:22.964693+0000","last_became_active":"2023-12-19T14:54:22.964414+0000","last_became_peered":"2023-12-19T14:54:22.964414+0000","last_unstale":"2023-12-19T14:54:22.964693+0000","last_undegraded":"2023-12-19T14:54:22.964693+0000","last_fullsized":"2023-12-19T14:54:22.964693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:26.149145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.098570+0000","last_change":"2023-12-19T14:54:22.086565+0000","last_active":"2023-12-19T14:54:23.098570+0000","last_peered":"2023-12-19T14:54:23.098570+0000","last_clean":"2023-12-19T14:54:23.098570+0000","last_became_active":"2023-12-19T14:54:22.086285+0000","last_became_peered":"2023-12-19T14:54:22.086285+0000","last_unstale":"2023-12-19T14:54:23.098570+0000","last_undegraded":"2023-12-19T14:54:23.098570+0000","last_fullsized":"2023-12-19T14:54:23.098570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:25.356833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.967055+0000","last_change":"2023-12-19T14:54:22.963254+0000","last_active":"2023-12-19T14:54:22.967055+0000","last_peered":"2023-12-19T14:54:22.967055+0000","last_clean":"2023-12-19T14:54:22.967055+0000","last_became_active":"2023-12-19T14:54:22.962933+0000","last_became_peered":"2023-12-19T14:54:22.962933+0000","last_unstale":"2023-12-19T14:54:22.967055+0000","last_undegraded":"2023-12-19T14:54:22.967055+0000","last_fullsized":"2023-12-19T14:54:22.967055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:35:41.801705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.092981+0000","last_change":"2023-12-19T14:54:22.091910+0000","last_active":"2023-12-19T14:54:22.092981+0000","last_peered":"2023-12-19T14:54:22.092981+0000","last_clean":"2023-12-19T14:54:22.092981+0000","last_became_active":"2023-12-19T14:54:22.091514+0000","last_became_peered":"2023-12-19T14:54:22.091514+0000","last_unstale":"2023-12-19T14:54:22.092981+0000","last_undegraded":"2023-12-19T14:54:22.092981+0000","last_fullsized":"2023-12-19T14:54:22.092981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:25:09.268439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957216+0000","last_change":"2023-12-19T14:54:22.957216+0000","last_active":"2023-12-19T14:54:22.957216+0000","last_peered":"2023-12-19T14:54:22.957216+0000","last_clean":"2023-12-19T14:54:22.957216+0000","last_became_active":"2023-12-19T14:54:22.956914+0000","last_became_peered":"2023-12-19T14:54:22.956914+0000","last_unstale":"2023-12-19T14:54:22.957216+0000","last_undegraded":"2023-12-19T14:54:22.957216+0000","last_fullsized":"2023-12-19T14:54:22.957216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:18.690127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.101577+0000","last_change":"2023-12-19T14:54:22.099895+0000","last_active":"2023-12-19T14:54:22.101577+0000","last_peered":"2023-12-19T14:54:22.101577+0000","last_clean":"2023-12-19T14:54:22.101577+0000","last_became_active":"2023-12-19T14:54:22.099584+0000","last_became_peered":"2023-12-19T14:54:22.099584+0000","last_unstale":"2023-12-19T14:54:22.101577+0000","last_undegraded":"2023-12-19T14:54:22.101577+0000","last_fullsized":"2023-12-19T14:54:22.101577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:28.510850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.953646+0000","last_change":"2023-12-19T14:54:22.948439+0000","last_active":"2023-12-19T14:54:22.953646+0000","last_peered":"2023-12-19T14:54:22.953646+0000","last_clean":"2023-12-19T14:54:22.953646+0000","last_became_active":"2023-12-19T14:54:22.948101+0000","last_became_peered":"2023-12-19T14:54:22.948101+0000","last_unstale":"2023-12-19T14:54:22.953646+0000","last_undegraded":"2023-12-19T14:54:22.953646+0000","last_fullsized":"2023-12-19T14:54:22.953646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:47:43.184898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.097533+0000","last_change":"2023-12-19T14:54:22.093771+0000","last_active":"2023-12-19T14:54:22.097533+0000","last_peered":"2023-12-19T14:54:22.097533+0000","last_clean":"2023-12-19T14:54:22.097533+0000","last_became_active":"2023-12-19T14:54:22.093450+0000","last_became_peered":"2023-12-19T14:54:22.093450+0000","last_unstale":"2023-12-19T14:54:22.097533+0000","last_undegraded":"2023-12-19T14:54:22.097533+0000","last_fullsized":"2023-12-19T14:54:22.097533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954301+0000","last_change":"2023-12-19T14:54:22.950822+0000","last_active":"2023-12-19T14:54:22.954301+0000","last_peered":"2023-12-19T14:54:22.954301+0000","last_clean":"2023-12-19T14:54:22.954301+0000","last_became_active":"2023-12-19T14:54:22.950547+0000","last_became_peered":"2023-12-19T14:54:22.950547+0000","last_unstale":"2023-12-19T14:54:22.954301+0000","last_undegraded":"2023-12-19T14:54:22.954301+0000","last_fullsized":"2023-12-19T14:54:22.954301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:05.964970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095529+0000","last_change":"2023-12-19T14:54:22.093235+0000","last_active":"2023-12-19T14:54:22.095529+0000","last_peered":"2023-12-19T14:54:22.095529+0000","last_clean":"2023-12-19T14:54:22.095529+0000","last_became_active":"2023-12-19T14:54:22.090494+0000","last_became_peered":"2023-12-19T14:54:22.090494+0000","last_unstale":"2023-12-19T14:54:22.095529+0000","last_undegraded":"2023-12-19T14:54:22.095529+0000","last_fullsized":"2023-12-19T14:54:22.095529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958890+0000","last_change":"2023-12-19T14:54:22.958890+0000","last_active":"2023-12-19T14:54:22.958890+0000","last_peered":"2023-12-19T14:54:22.958890+0000","last_clean":"2023-12-19T14:54:22.958890+0000","last_became_active":"2023-12-19T14:54:22.958521+0000","last_became_peered":"2023-12-19T14:54:22.958521+0000","last_unstale":"2023-12-19T14:54:22.958890+0000","last_undegraded":"2023-12-19T14:54:22.958890+0000","last_fullsized":"2023-12-19T14:54:22.958890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095208+0000","last_change":"2023-12-19T14:54:22.087737+0000","last_active":"2023-12-19T14:54:22.095208+0000","last_peered":"2023-12-19T14:54:22.095208+0000","last_clean":"2023-12-19T14:54:22.095208+0000","last_became_active":"2023-12-19T14:54:22.087431+0000","last_became_peered":"2023-12-19T14:54:22.087431+0000","last_unstale":"2023-12-19T14:54:22.095208+0000","last_undegraded":"2023-12-19T14:54:22.095208+0000","last_fullsized":"2023-12-19T14:54:22.095208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:53.397056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.965720+0000","last_change":"2023-12-19T14:54:22.965720+0000","last_active":"2023-12-19T14:54:22.965720+0000","last_peered":"2023-12-19T14:54:22.965720+0000","last_clean":"2023-12-19T14:54:22.965720+0000","last_became_active":"2023-12-19T14:54:22.963707+0000","last_became_peered":"2023-12-19T14:54:22.963707+0000","last_unstale":"2023-12-19T14:54:22.965720+0000","last_undegraded":"2023-12-19T14:54:22.965720+0000","last_fullsized":"2023-12-19T14:54:22.965720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098282+0000","last_change":"2023-12-19T14:54:22.094912+0000","last_active":"2023-12-19T14:54:22.098282+0000","last_peered":"2023-12-19T14:54:22.098282+0000","last_clean":"2023-12-19T14:54:22.098282+0000","last_became_active":"2023-12-19T14:54:22.094485+0000","last_became_peered":"2023-12-19T14:54:22.094485+0000","last_unstale":"2023-12-19T14:54:22.098282+0000","last_undegraded":"2023-12-19T14:54:22.098282+0000","last_fullsized":"2023-12-19T14:54:22.098282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:38.774987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957827+0000","last_change":"2023-12-19T14:54:22.951004+0000","last_active":"2023-12-19T14:54:22.957827+0000","last_peered":"2023-12-19T14:54:22.957827+0000","last_clean":"2023-12-19T14:54:22.957827+0000","last_became_active":"2023-12-19T14:54:22.950715+0000","last_became_peered":"2023-12-19T14:54:22.950715+0000","last_unstale":"2023-12-19T14:54:22.957827+0000","last_undegraded":"2023-12-19T14:54:22.957827+0000","last_fullsized":"2023-12-19T14:54:22.957827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:49:15.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.081824+0000","last_change":"2023-12-19T14:54:22.079441+0000","last_active":"2023-12-19T14:54:22.081824+0000","last_peered":"2023-12-19T14:54:22.081824+0000","last_clean":"2023-12-19T14:54:22.081824+0000","last_became_active":"2023-12-19T14:54:22.079118+0000","last_became_peered":"2023-12-19T14:54:22.079118+0000","last_unstale":"2023-12-19T14:54:22.081824+0000","last_undegraded":"2023-12-19T14:54:22.081824+0000","last_fullsized":"2023-12-19T14:54:22.081824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:19.697671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950337+0000","last_change":"2023-12-19T14:54:22.945328+0000","last_active":"2023-12-19T14:54:22.950337+0000","last_peered":"2023-12-19T14:54:22.950337+0000","last_clean":"2023-12-19T14:54:22.950337+0000","last_became_active":"2023-12-19T14:54:22.945033+0000","last_became_peered":"2023-12-19T14:54:22.945033+0000","last_unstale":"2023-12-19T14:54:22.950337+0000","last_undegraded":"2023-12-19T14:54:22.950337+0000","last_fullsized":"2023-12-19T14:54:22.950337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:48:13.499494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.093232+0000","last_change":"2023-12-19T14:54:22.089091+0000","last_active":"2023-12-19T14:54:22.093232+0000","last_peered":"2023-12-19T14:54:22.093232+0000","last_clean":"2023-12-19T14:54:22.093232+0000","last_became_active":"2023-12-19T14:54:22.088777+0000","last_became_peered":"2023-12-19T14:54:22.088777+0000","last_unstale":"2023-12-19T14:54:22.093232+0000","last_undegraded":"2023-12-19T14:54:22.093232+0000","last_fullsized":"2023-12-19T14:54:22.093232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:08.251166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":373,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.075964+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:54:22.075964+0000","last_peered":"2023-12-19T14:54:22.075964+0000","last_clean":"2023-12-19T14:54:22.075964+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:54:22.075964+0000","last_undegraded":"2023-12-19T14:54:22.075964+0000","last_fullsized":"2023-12-19T14:54:22.075964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":1105170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.498,"5min":3.544,"15min":3.544},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81699999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5369999999999999,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.093},"min":{"1min":0.91700000000000004,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.4379999999999999,"5min":1.73,"15min":1.73},"last":1.262},{"interface":"front","average":{"1min":1.276,"5min":1.1950000000000001,"15min":1.1819999999999999},"min":{"1min":0.93400000000000005,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.984},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2729999999999999,"15min":1.2330000000000001},"min":{"1min":0.96699999999999997,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.571,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":1.546},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.274,"15min":1.278},"min":{"1min":1.1359999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.667,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.28,"15min":1.29},"min":{"1min":0.92500000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.5409999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.093},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.3109999999999999},"min":{"1min":1.038,"5min":0.95899999999999996,"15min":0.79700000000000004},"max":{"1min":1.6100000000000001,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.5860000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.1839999999999999,"15min":1.1699999999999999},"min":{"1min":0.68500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.752,"5min":1.752,"15min":2.1930000000000001},"last":1.673},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2410000000000001,"15min":1.222},"min":{"1min":0.90300000000000002,"5min":0.75700000000000001,"15min":0.67500000000000004},"max":{"1min":1.6000000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.1859999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2210000000000001,"15min":1.2270000000000001},"min":{"1min":1.034,"5min":0.496,"15min":0.496},"max":{"1min":1.6930000000000001,"5min":1.8109999999999999,"15min":1.976},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3029999999999999,"15min":1.3180000000000001},"min":{"1min":0.90200000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.3620000000000001},"last":1.121}]},{"osd":6,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3899999999999999,"15min":1.3999999999999999},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.65600000000000003},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":3.1589999999999998},"last":1.371},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3620000000000001,"15min":1.361},"min":{"1min":1.0620000000000001,"5min":0.88500000000000001,"15min":0.70299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.298}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302396,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069444,"statfs":{"total":96636764160,"available":96327110656,"internally_reserved":0,"allocated":1036288,"data_stored":514088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.1659999999999999},"min":{"1min":0.499,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6100000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.169,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.845,"5min":1.845,"15min":2.335},"last":1.0920000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.2010000000000001,"15min":1.2150000000000001},"min":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":2.016,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.026},{"interface":"front","average":{"1min":1.254,"5min":1.268,"15min":1.2849999999999999},"min":{"1min":0.78000000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.526,"5min":3.226,"15min":3.226},"last":0.76400000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2829999999999999,"15min":1.274},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7110000000000001,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.363},{"interface":"front","average":{"1min":1.25,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.82399999999999995,"5min":0.70799999999999996,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.337}]},{"osd":3,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3160000000000001,"15min":1.3420000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.7769999999999999,"5min":1.9239999999999999,"15min":3.2330000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3680000000000001,"15min":1.3720000000000001},"min":{"1min":0.84299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.645,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2609999999999999},"min":{"1min":0.70899999999999996,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":2.1040000000000001,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.294,"15min":1.3500000000000001},"min":{"1min":0.90200000000000002,"5min":0.79800000000000004,"15min":0.57599999999999996},"max":{"1min":1.901,"5min":3.3460000000000001,"15min":3.4399999999999999},"last":1.23}]},{"osd":5,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.321,"15min":1.351},"min":{"1min":0.93300000000000005,"5min":0.71799999999999997,"15min":0.63100000000000001},"max":{"1min":1.736,"5min":2.02,"15min":2.8620000000000001},"last":1.425},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.4019999999999999},"min":{"1min":0.82799999999999996,"5min":0.76300000000000001,"15min":0.51000000000000001},"max":{"1min":1.931,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.2709999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.456,"15min":1.456},"min":{"1min":0.97599999999999998,"5min":0.85899999999999999,"15min":0.53000000000000003},"max":{"1min":1.8140000000000001,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.496,"15min":1.512},"min":{"1min":1.0720000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.821,"5min":3.742,"15min":3.742},"last":1.3009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298300,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073540,"statfs":{"total":96636764160,"available":96331304960,"internally_reserved":0,"allocated":1036288,"data_stored":514076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.0389999999999999,"15min":1.0569999999999999},"min":{"1min":0.80400000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.601,"5min":1.601,"15min":1.619},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.198,"5min":1.141,"15min":1.139},"min":{"1min":0.875,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6539999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.93000000000000005}]},{"osd":1,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2729999999999999,"15min":1.29},"min":{"1min":0.97099999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.8029999999999999,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.016},{"interface":"front","average":{"1min":1.272,"5min":1.2210000000000001,"15min":1.244},"min":{"1min":1.0660000000000001,"5min":0.70299999999999996,"15min":0.55700000000000005},"max":{"1min":1.494,"5min":1.5649999999999999,"15min":1.7869999999999999},"last":1.3400000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.204,"15min":1.2290000000000001},"min":{"1min":0.83099999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.645,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.052},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1759999999999999,"15min":1.21},"min":{"1min":0.88900000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.448,"5min":1.8600000000000001,"15min":2.0739999999999998},"last":1.1180000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.2689999999999999,"15min":1.2769999999999999},"min":{"1min":0.88200000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.726,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.409},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2549999999999999,"15min":1.284},"min":{"1min":0.98799999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.73,"5min":1.73,"15min":2.2149999999999999},"last":1.286}]},{"osd":4,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.83299999999999996,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5089999999999999,"5min":1.603,"15min":2.1099999999999999},"last":1.488},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.76700000000000002,"5min":0.69099999999999995,"15min":0.49299999999999999},"max":{"1min":1.7709999999999999,"5min":2.27,"15min":2.27},"last":1.6319999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3009999999999999,"15min":1.3440000000000001},"min":{"1min":0.83099999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6140000000000001,"5min":1.913,"15min":2.4790000000000001},"last":1.377},{"interface":"front","average":{"1min":1.278,"5min":1.319,"15min":1.3360000000000001},"min":{"1min":0.70999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6859999999999999,"5min":1.8540000000000001,"15min":2.0030000000000001},"last":1.5900000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3160000000000001,"15min":1.363},"min":{"1min":0.89200000000000002,"5min":0.76700000000000002,"15min":0.58899999999999997},"max":{"1min":1.6699999999999999,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.556},{"interface":"front","average":{"1min":1.353,"5min":1.3360000000000001,"15min":1.3999999999999999},"min":{"1min":0.95299999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6779999999999999,"5min":2.0070000000000001,"15min":2.411},"last":1.522}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302404,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069436,"statfs":{"total":96636764160,"available":96327102464,"internally_reserved":0,"allocated":1044480,"data_stored":515590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.077,"15min":1.073},"min":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.58199999999999996},"max":{"1min":1.452,"5min":1.579,"15min":1.7869999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1140000000000001,"15min":1.0960000000000001},"min":{"1min":0.68400000000000005,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.4450000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.69999999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.2370000000000001,"15min":1.244},"min":{"1min":1.073,"5min":0.623,"15min":0.623},"max":{"1min":1.5680000000000001,"5min":1.6819999999999999,"15min":2.1259999999999999},"last":0.75},{"interface":"front","average":{"1min":1.244,"5min":1.262,"15min":1.28},"min":{"1min":0.89200000000000002,"5min":0.70299999999999996,"15min":0.65100000000000002},"max":{"1min":1.5960000000000001,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":0.81599999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.1499999999999999,"15min":1.1519999999999999},"min":{"1min":0.877,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.891},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2270000000000001,"15min":1.2170000000000001},"min":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.65500000000000003},"max":{"1min":1.514,"5min":1.629,"15min":1.893},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.84399999999999997,"5min":0.82099999999999995,"15min":0.59599999999999997},"max":{"1min":1.5620000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.93999999999999995,"5min":0.83799999999999997,"15min":0.60599999999999998},"max":{"1min":1.593,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.6259999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1599999999999999,"15min":1.173},"min":{"1min":0.91200000000000003,"5min":0.71099999999999997,"15min":0.57899999999999996},"max":{"1min":1.5629999999999999,"5min":2.1800000000000002,"15min":2.3500000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.143,"15min":1.1779999999999999},"min":{"1min":0.84699999999999998,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.526,"5min":2.077,"15min":2.161},"last":0.83699999999999997}]},{"osd":6,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.294,"15min":1.347},"min":{"1min":0.80300000000000005,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.671,"5min":2.8119999999999998,"15min":3.27},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.4159999999999999,"15min":1.4390000000000001},"min":{"1min":1.0369999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.633,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":0.86899999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.3879999999999999},"min":{"1min":0.94599999999999995,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.6200000000000001,"5min":2.9910000000000001,"15min":5.2060000000000004},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.429,"15min":1.48},"min":{"1min":0.872,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":1.669,"5min":3.008,"15min":3.6739999999999999},"last":0.92300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575033,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":514054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84999999999999998,"15min":0.89000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2010000000000001,"5min":1.3640000000000001,"15min":4.1159999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.026,"5min":1.01,"15min":1.0129999999999999},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.542,"5min":1.542,"15min":3.3929999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":0.95799999999999996},"min":{"1min":0.69599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0149999999999999,"5min":1.611,"15min":2.4390000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.121,"15min":1.1220000000000001},"min":{"1min":0.749,"5min":0.63500000000000001,"15min":0.60199999999999998},"max":{"1min":1.5189999999999999,"5min":1.661,"15min":4.476},"last":1.224}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1459999999999999,"15min":1.1419999999999999},"min":{"1min":0.89900000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.639,"5min":1.76,"15min":2.6989999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61699999999999999},"max":{"1min":1.5309999999999999,"5min":1.647,"15min":2.5110000000000001},"last":1.0129999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.345},"min":{"1min":0.84899999999999998,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.526,"5min":1.768,"15min":3.46},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2829999999999999,"15min":1.3069999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.542,"5min":1.6970000000000001,"15min":2.9409999999999998},"last":1.1870000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.286,"15min":1.3280000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.629,"5min":1.7709999999999999,"15min":3.1019999999999999},"last":1.375},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.4159999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.833,"5min":2.1619999999999999,"15min":4.3650000000000002},"last":1.3500000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.446,"15min":1.4730000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9099999999999999,"5min":2.2709999999999999,"15min":4.5999999999999996},"last":1.294},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.419,"15min":1.444},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":3.4329999999999998},"last":1.321}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.516,"15min":1.5429999999999999},"min":{"1min":0.93300000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.7629999999999999,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.45,"15min":1.488},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.728,"5min":2.0670000000000002,"15min":3.7240000000000002},"last":1.5009999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306484,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065356,"statfs":{"total":96636764160,"available":96322924544,"internally_reserved":0,"allocated":1028096,"data_stored":513043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89400000000000002,"15min":0.90400000000000003},"min":{"1min":0.60599999999999998,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.946},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.056,"15min":1.0720000000000001},"min":{"1min":0.63300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.605,"5min":1.605,"15min":1.8560000000000001},"last":1.218}]},{"osd":1,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.82199999999999995,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.085},{"interface":"front","average":{"1min":1.169,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.3169999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1.0009999999999999},"min":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.304,"5min":1.5669999999999999,"15min":1.9370000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.081,"15min":1.105},"min":{"1min":0.80100000000000005,"5min":0.66000000000000003,"15min":0.55000000000000004},"max":{"1min":1.603,"5min":1.603,"15min":1.6930000000000001},"last":1.131}]},{"osd":4,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":0.95199999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":1.893,"15min":3.4950000000000001},"last":1.165},{"interface":"front","average":{"1min":1.448,"5min":1.403,"15min":1.3999999999999999},"min":{"1min":1.2230000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.923,"5min":1.96,"15min":2.0409999999999999},"last":1.2969999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3740000000000001,"15min":1.3600000000000001},"min":{"1min":1.1040000000000001,"5min":0.876,"15min":0.876},"max":{"1min":1.7669999999999999,"5min":2.0760000000000001,"15min":3.8599999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.93899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.6759999999999999,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.347}]},{"osd":6,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.80600000000000005},"max":{"1min":1.6950000000000001,"5min":1.829,"15min":2.8759999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4370000000000001,"15min":1.45},"min":{"1min":1.2170000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8280000000000001,"5min":2.1200000000000001,"15min":3.7309999999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.482,"15min":1.492},"min":{"1min":1.171,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.044,"5min":2.0699999999999998,"15min":2.5550000000000002},"last":1.484},{"interface":"front","average":{"1min":1.599,"5min":1.4710000000000001,"15min":1.4870000000000001},"min":{"1min":1.258,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.4049999999999998},"last":1.4630000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306516,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065324,"statfs":{"total":96636764160,"available":96322891776,"internally_reserved":0,"allocated":1060864,"data_stored":516326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.877,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":1.238,"5min":1.464,"15min":1.871},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.98899999999999999,"15min":0.996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.50700000000000001},"max":{"1min":1.393,"5min":1.7070000000000001,"15min":2.468},"last":0.77400000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0169999999999999,"15min":1.0209999999999999},"min":{"1min":0.71699999999999997,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":1.2589999999999999,"5min":1.714,"15min":1.7829999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.66300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.5660000000000001,"5min":2.0230000000000001,"15min":3.0619999999999998},"last":1.2669999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.236,"15min":1.2350000000000001},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.554},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.0820000000000001},"min":{"1min":0.81200000000000006,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6439999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2869999999999999,"15min":1.3109999999999999},"min":{"1min":0.93000000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.724,"5min":1.7370000000000001,"15min":2.6440000000000001},"last":1.341},{"interface":"front","average":{"1min":1.405,"5min":1.401,"15min":1.403},"min":{"1min":1.0009999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.677,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.6699999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.262,"15min":1.296},"min":{"1min":1.028,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.538,"5min":1.6719999999999999,"15min":2.1320000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.3720000000000001,"15min":1.391},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.76,"5min":1.76,"15min":2.79},"last":1.6339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.333,"15min":1.3640000000000001},"min":{"1min":1.0129999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7070000000000001,"5min":1.78,"15min":2.4620000000000002},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4319999999999999,"15min":1.456},"min":{"1min":1.073,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.8999999999999999},"last":1.4730000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.474,"15min":1.488},"min":{"1min":1.0740000000000001,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":2.0470000000000002,"15min":3.782},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":1.046,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":2.5390000000000001},"last":1.3129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307064,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064776,"statfs":{"total":96636764160,"available":96322330624,"internally_reserved":0,"allocated":1622016,"data_stored":1103411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.97299999999999998},"min":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.53600000000000003},"max":{"1min":1.4810000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.80000000000000004,"5min":0.69699999999999995,"15min":0.55200000000000005},"max":{"1min":1.5189999999999999,"5min":1.6339999999999999,"15min":1.8979999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.2410000000000001,"15min":1.2210000000000001},"min":{"1min":0.82199999999999995,"5min":0.70799999999999996,"15min":0.54300000000000004},"max":{"1min":1.728,"5min":1.982,"15min":1.982},"last":1.327}]},{"osd":3,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6359999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.091,"5min":1.1679999999999999,"15min":1.1819999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.1140000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3169999999999999,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.70199999999999996},"max":{"1min":1.754,"5min":2.153,"15min":2.298},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.302,"15min":1.298},"min":{"1min":1.026,"5min":0.83999999999999997,"15min":0.80200000000000005},"max":{"1min":1.579,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.2829999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3600000000000001,"15min":1.375},"min":{"1min":0.89300000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9410000000000001,"5min":1.974,"15min":2.4630000000000001},"last":1.24},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":2.2189999999999999,"15min":2.6160000000000001},"last":1.2090000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.343,"15min":1.3460000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.88300000000000001},"max":{"1min":1.7130000000000001,"5min":2.0150000000000001,"15min":2.339},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4299999999999999,"15min":1.452},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.0590000000000002,"5min":2.113,"15min":2.5680000000000001},"last":1.361}]},{"osd":7,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.488,"15min":1.4710000000000001},"min":{"1min":1.083,"5min":1.0660000000000001,"15min":0.91500000000000004},"max":{"1min":2.0019999999999998,"5min":2.2629999999999999,"15min":2.5129999999999999},"last":1.083},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":1.04,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.802,"5min":1.931,"15min":2.4500000000000002},"last":1.4570000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:54:27.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:54:27.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.607+0000 7fbb5e2e3700 1 Processor -- start 2023-12-19T14:54:27.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 -- start start 2023-12-19T14:54:27.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbb5815d1a0 0x7fbb5806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb580c8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb580c7ca0 con 0x7fbb5806d590 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5806da10 con 0x7fbb5815d1a0 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5e2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815e100 con 0x7fbb5815dc80 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbb5815d1a0 0x7fbb5806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbb5815d1a0 0x7fbb5806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35276/0 (socket says 172.21.15.50:35276) 2023-12-19T14:54:27.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/3585816411 learned_addr learned my addr 172.21.15.50:0/3585816411 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/3585816411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 msgr2=0x7fbb580c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 --2- 172.21.15.50:0/3585816411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb580c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/3585816411 >> v1:172.21.15.125:6789/0 conn(0x7fbb5806d590 legacy=0x7fbb5815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/3585816411 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5815e380 con 0x7fbb5815d1a0 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb5d2e1700 1 --2- 172.21.15.50:0/3585816411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5815d1a0 0x7fbb5806d050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbb540099b0 tx=0x7fbb54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ae246722faf56d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb4ffff700 1 -- 172.21.15.50:0/3585816411 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5400b820 con 0x7fbb5815d1a0 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb4ffff700 1 -- 172.21.15.50:0/3585816411 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5400b980 con 0x7fbb5815d1a0 2023-12-19T14:54:27.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.611+0000 7fbb4ffff700 1 -- 172.21.15.50:0/3585816411 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5400bd40 con 0x7fbb5815d1a0 2023-12-19T14:54:27.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/3585816411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5815d1a0 msgr2=0x7fbb5806d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/3585816411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5815d1a0 0x7fbb5806d050 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbb540099b0 tx=0x7fbb54009250 comp rx=0 tx=0).stop 2023-12-19T14:54:27.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/3585816411 shutdown_connections 2023-12-19T14:54:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/3585816411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb580c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/3585816411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5815d1a0 0x7fbb5806d050 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/3585816411 >> 172.21.15.50:0/3585816411 conn(0x7fbb5806c080 msgr2=0x7fbb580ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/3585816411 shutdown_connections 2023-12-19T14:54:27.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.615+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/3585816411 wait complete. 2023-12-19T14:54:27.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 Processor -- start 2023-12-19T14:54:27.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 -- start start 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb5815d1a0 0x7fbb58158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb58148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815a8a0 con 0x7fbb5806d590 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815b6e0 con 0x7fbb5815d1a0 2023-12-19T14:54:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.619+0000 7fbb5e2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5815aad0 con 0x7fbb5815dc80 2023-12-19T14:54:27.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35286/0 (socket says 172.21.15.50:35286) 2023-12-19T14:54:27.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/2789746020 learned_addr learned my addr 172.21.15.50:0/2789746020 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5cae0700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb5815d1a0 0x7fbb58158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5dae2700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb58148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 msgr2=0x7fbb58148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb58148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb5815d1a0 msgr2=0x7fbb58158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb5815d1a0 0x7fbb58158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 -- 172.21.15.50:0/2789746020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb540094f0 con 0x7fbb5806d590 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5d2e1700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009fd0 tx=0x7fbb54016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54011da0 con 0x7fbb5806d590 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5401f070 con 0x7fbb5806d590 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540266d0 con 0x7fbb5806d590 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5815adb0 con 0x7fbb5806d590 2023-12-19T14:54:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5815b370 con 0x7fbb5806d590 2023-12-19T14:54:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.623+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5804be40 con 0x7fbb5806d590 2023-12-19T14:54:27.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.627+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbb54026b00 con 0x7fbb5806d590 2023-12-19T14:54:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.627+0000 7fbb4d7fa700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 0x7fbb3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.627+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbb54021070 con 0x7fbb5806d590 2023-12-19T14:54:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.627+0000 7fbb5cae0700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 0x7fbb3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.627+0000 7fbb5cae0700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 0x7fbb3c062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb580c2530 tx=0x7fbb480090f0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.635+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbb5404ee60 con 0x7fbb5806d590 2023-12-19T14:54:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.807+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb580670e0 con 0x7fbb3c060670 2023-12-19T14:54:27.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.827+0000 7fbb4d7fa700 1 -- 172.21.15.50:0/2789746020 <== mgr.5720 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286787 (secure 0 0 0) 0x7fbb580670e0 con 0x7fbb3c060670 2023-12-19T14:54:27.831 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:27.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 msgr2=0x7fbb3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 0x7fbb3c062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb580c2530 tx=0x7fbb480090f0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 msgr2=0x7fbb580c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009fd0 tx=0x7fbb54016000 comp rx=0 tx=0).stop 2023-12-19T14:54:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 shutdown_connections 2023-12-19T14:54:27.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb5815d1a0 0x7fbb58158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb3c060670 0x7fbb3c062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb5815dc80 0x7fbb58148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 --2- 172.21.15.50:0/2789746020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb5806d590 0x7fbb580c45f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 >> 172.21.15.50:0/2789746020 conn(0x7fbb5806c080 msgr2=0x7fbb58067300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 shutdown_connections 2023-12-19T14:54:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:27.839+0000 7fbb5e2e3700 1 -- 172.21.15.50:0/2789746020 wait complete. 2023-12-19T14:54:27.841 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:54:27.860 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:54:26.576027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2440832,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752533888,"statfs":{"total":773094113280,"available":770594701312,"internally_reserved":0,"allocated":9502720,"data_stored":5295758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.524457"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951012+0000","last_change":"2023-12-19T14:54:22.951012+0000","last_active":"2023-12-19T14:54:22.951012+0000","last_peered":"2023-12-19T14:54:22.951012+0000","last_clean":"2023-12-19T14:54:22.951012+0000","last_became_active":"2023-12-19T14:54:22.950613+0000","last_became_peered":"2023-12-19T14:54:22.950613+0000","last_unstale":"2023-12-19T14:54:22.951012+0000","last_undegraded":"2023-12-19T14:54:22.951012+0000","last_fullsized":"2023-12-19T14:54:22.951012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.961303+0000","last_change":"2023-12-19T14:54:22.961303+0000","last_active":"2023-12-19T14:54:22.961303+0000","last_peered":"2023-12-19T14:54:22.961303+0000","last_clean":"2023-12-19T14:54:22.961303+0000","last_became_active":"2023-12-19T14:54:22.960933+0000","last_became_peered":"2023-12-19T14:54:22.960933+0000","last_unstale":"2023-12-19T14:54:22.961303+0000","last_undegraded":"2023-12-19T14:54:22.961303+0000","last_fullsized":"2023-12-19T14:54:22.961303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.956249+0000","last_change":"2023-12-19T14:54:22.956249+0000","last_active":"2023-12-19T14:54:22.956249+0000","last_peered":"2023-12-19T14:54:22.956249+0000","last_clean":"2023-12-19T14:54:22.956249+0000","last_became_active":"2023-12-19T14:54:22.955982+0000","last_became_peered":"2023-12-19T14:54:22.955982+0000","last_unstale":"2023-12-19T14:54:22.956249+0000","last_undegraded":"2023-12-19T14:54:22.956249+0000","last_fullsized":"2023-12-19T14:54:22.956249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962227+0000","last_change":"2023-12-19T14:54:22.962227+0000","last_active":"2023-12-19T14:54:22.962227+0000","last_peered":"2023-12-19T14:54:22.962227+0000","last_clean":"2023-12-19T14:54:22.962227+0000","last_became_active":"2023-12-19T14:54:22.961400+0000","last_became_peered":"2023-12-19T14:54:22.961400+0000","last_unstale":"2023-12-19T14:54:22.962227+0000","last_undegraded":"2023-12-19T14:54:22.962227+0000","last_fullsized":"2023-12-19T14:54:22.962227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:16.693921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957210+0000","last_change":"2023-12-19T14:54:22.955591+0000","last_active":"2023-12-19T14:54:22.957210+0000","last_peered":"2023-12-19T14:54:22.957210+0000","last_clean":"2023-12-19T14:54:22.957210+0000","last_became_active":"2023-12-19T14:54:22.955082+0000","last_became_peered":"2023-12-19T14:54:22.955082+0000","last_unstale":"2023-12-19T14:54:22.957210+0000","last_undegraded":"2023-12-19T14:54:22.957210+0000","last_fullsized":"2023-12-19T14:54:22.957210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:47.300540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959812+0000","last_change":"2023-12-19T14:54:22.959812+0000","last_active":"2023-12-19T14:54:22.959812+0000","last_peered":"2023-12-19T14:54:22.959812+0000","last_clean":"2023-12-19T14:54:22.959812+0000","last_became_active":"2023-12-19T14:54:22.959448+0000","last_became_peered":"2023-12-19T14:54:22.959448+0000","last_unstale":"2023-12-19T14:54:22.959812+0000","last_undegraded":"2023-12-19T14:54:22.959812+0000","last_fullsized":"2023-12-19T14:54:22.959812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:13:43.962908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960007+0000","last_change":"2023-12-19T14:54:22.956689+0000","last_active":"2023-12-19T14:54:22.960007+0000","last_peered":"2023-12-19T14:54:22.960007+0000","last_clean":"2023-12-19T14:54:22.960007+0000","last_became_active":"2023-12-19T14:54:22.956408+0000","last_became_peered":"2023-12-19T14:54:22.956408+0000","last_unstale":"2023-12-19T14:54:22.960007+0000","last_undegraded":"2023-12-19T14:54:22.960007+0000","last_fullsized":"2023-12-19T14:54:22.960007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:03.845407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947337+0000","last_change":"2023-12-19T14:54:22.947337+0000","last_active":"2023-12-19T14:54:22.947337+0000","last_peered":"2023-12-19T14:54:22.947337+0000","last_clean":"2023-12-19T14:54:22.947337+0000","last_became_active":"2023-12-19T14:54:22.947048+0000","last_became_peered":"2023-12-19T14:54:22.947048+0000","last_unstale":"2023-12-19T14:54:22.947337+0000","last_undegraded":"2023-12-19T14:54:22.947337+0000","last_fullsized":"2023-12-19T14:54:22.947337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:33:50.174690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959356+0000","last_change":"2023-12-19T14:54:22.959356+0000","last_active":"2023-12-19T14:54:22.959356+0000","last_peered":"2023-12-19T14:54:22.959356+0000","last_clean":"2023-12-19T14:54:22.959356+0000","last_became_active":"2023-12-19T14:54:22.959099+0000","last_became_peered":"2023-12-19T14:54:22.959099+0000","last_unstale":"2023-12-19T14:54:22.959356+0000","last_undegraded":"2023-12-19T14:54:22.959356+0000","last_fullsized":"2023-12-19T14:54:22.959356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:09.263637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952132+0000","last_change":"2023-12-19T14:54:22.952132+0000","last_active":"2023-12-19T14:54:22.952132+0000","last_peered":"2023-12-19T14:54:22.952132+0000","last_clean":"2023-12-19T14:54:22.952132+0000","last_became_active":"2023-12-19T14:54:22.951855+0000","last_became_peered":"2023-12-19T14:54:22.951855+0000","last_unstale":"2023-12-19T14:54:22.952132+0000","last_undegraded":"2023-12-19T14:54:22.952132+0000","last_fullsized":"2023-12-19T14:54:22.952132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:04:18.531600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958836+0000","last_change":"2023-12-19T14:54:22.958836+0000","last_active":"2023-12-19T14:54:22.958836+0000","last_peered":"2023-12-19T14:54:22.958836+0000","last_clean":"2023-12-19T14:54:22.958836+0000","last_became_active":"2023-12-19T14:54:22.958517+0000","last_became_peered":"2023-12-19T14:54:22.958517+0000","last_unstale":"2023-12-19T14:54:22.958836+0000","last_undegraded":"2023-12-19T14:54:22.958836+0000","last_fullsized":"2023-12-19T14:54:22.958836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:20.969411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952064+0000","last_change":"2023-12-19T14:54:22.952064+0000","last_active":"2023-12-19T14:54:22.952064+0000","last_peered":"2023-12-19T14:54:22.952064+0000","last_clean":"2023-12-19T14:54:22.952064+0000","last_became_active":"2023-12-19T14:54:22.949570+0000","last_became_peered":"2023-12-19T14:54:22.949570+0000","last_unstale":"2023-12-19T14:54:22.952064+0000","last_undegraded":"2023-12-19T14:54:22.952064+0000","last_fullsized":"2023-12-19T14:54:22.952064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:03:09.806691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964019+0000","last_change":"2023-12-19T14:54:22.964019+0000","last_active":"2023-12-19T14:54:22.964019+0000","last_peered":"2023-12-19T14:54:22.964019+0000","last_clean":"2023-12-19T14:54:22.964019+0000","last_became_active":"2023-12-19T14:54:22.963742+0000","last_became_peered":"2023-12-19T14:54:22.963742+0000","last_unstale":"2023-12-19T14:54:22.964019+0000","last_undegraded":"2023-12-19T14:54:22.964019+0000","last_fullsized":"2023-12-19T14:54:22.964019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:07:30.035158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957890+0000","last_change":"2023-12-19T14:54:22.957890+0000","last_active":"2023-12-19T14:54:22.957890+0000","last_peered":"2023-12-19T14:54:22.957890+0000","last_clean":"2023-12-19T14:54:22.957890+0000","last_became_active":"2023-12-19T14:54:22.957600+0000","last_became_peered":"2023-12-19T14:54:22.957600+0000","last_unstale":"2023-12-19T14:54:22.957890+0000","last_undegraded":"2023-12-19T14:54:22.957890+0000","last_fullsized":"2023-12-19T14:54:22.957890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949326+0000","last_change":"2023-12-19T14:54:22.949326+0000","last_active":"2023-12-19T14:54:22.949326+0000","last_peered":"2023-12-19T14:54:22.949326+0000","last_clean":"2023-12-19T14:54:22.949326+0000","last_became_active":"2023-12-19T14:54:22.949010+0000","last_became_peered":"2023-12-19T14:54:22.949010+0000","last_unstale":"2023-12-19T14:54:22.949326+0000","last_undegraded":"2023-12-19T14:54:22.949326+0000","last_fullsized":"2023-12-19T14:54:22.949326+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959699+0000","last_change":"2023-12-19T14:54:22.955463+0000","last_active":"2023-12-19T14:54:22.959699+0000","last_peered":"2023-12-19T14:54:22.959699+0000","last_clean":"2023-12-19T14:54:22.959699+0000","last_became_active":"2023-12-19T14:54:22.954416+0000","last_became_peered":"2023-12-19T14:54:22.954416+0000","last_unstale":"2023-12-19T14:54:22.959699+0000","last_undegraded":"2023-12-19T14:54:22.959699+0000","last_fullsized":"2023-12-19T14:54:22.959699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:52:47.375220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958901+0000","last_change":"2023-12-19T14:54:22.958901+0000","last_active":"2023-12-19T14:54:22.958901+0000","last_peered":"2023-12-19T14:54:22.958901+0000","last_clean":"2023-12-19T14:54:22.958901+0000","last_became_active":"2023-12-19T14:54:22.958471+0000","last_became_peered":"2023-12-19T14:54:22.958471+0000","last_unstale":"2023-12-19T14:54:22.958901+0000","last_undegraded":"2023-12-19T14:54:22.958901+0000","last_fullsized":"2023-12-19T14:54:22.958901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.961903+0000","last_change":"2023-12-19T14:54:22.961903+0000","last_active":"2023-12-19T14:54:22.961903+0000","last_peered":"2023-12-19T14:54:22.961903+0000","last_clean":"2023-12-19T14:54:22.961903+0000","last_became_active":"2023-12-19T14:54:22.961258+0000","last_became_peered":"2023-12-19T14:54:22.961258+0000","last_unstale":"2023-12-19T14:54:22.961903+0000","last_undegraded":"2023-12-19T14:54:22.961903+0000","last_fullsized":"2023-12-19T14:54:22.961903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:09.263637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954525+0000","last_change":"2023-12-19T14:54:22.951844+0000","last_active":"2023-12-19T14:54:22.954525+0000","last_peered":"2023-12-19T14:54:22.954525+0000","last_clean":"2023-12-19T14:54:22.954525+0000","last_became_active":"2023-12-19T14:54:22.950094+0000","last_became_peered":"2023-12-19T14:54:22.950094+0000","last_unstale":"2023-12-19T14:54:22.954525+0000","last_undegraded":"2023-12-19T14:54:22.954525+0000","last_fullsized":"2023-12-19T14:54:22.954525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:57.078664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949491+0000","last_change":"2023-12-19T14:54:22.949491+0000","last_active":"2023-12-19T14:54:22.949491+0000","last_peered":"2023-12-19T14:54:22.949491+0000","last_clean":"2023-12-19T14:54:22.949491+0000","last_became_active":"2023-12-19T14:54:22.949191+0000","last_became_peered":"2023-12-19T14:54:22.949191+0000","last_unstale":"2023-12-19T14:54:22.949491+0000","last_undegraded":"2023-12-19T14:54:22.949491+0000","last_fullsized":"2023-12-19T14:54:22.949491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:34.627955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948377+0000","last_change":"2023-12-19T14:54:22.948377+0000","last_active":"2023-12-19T14:54:22.948377+0000","last_peered":"2023-12-19T14:54:22.948377+0000","last_clean":"2023-12-19T14:54:22.948377+0000","last_became_active":"2023-12-19T14:54:22.948042+0000","last_became_peered":"2023-12-19T14:54:22.948042+0000","last_unstale":"2023-12-19T14:54:22.948377+0000","last_undegraded":"2023-12-19T14:54:22.948377+0000","last_fullsized":"2023-12-19T14:54:22.948377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:46.333985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951521+0000","last_change":"2023-12-19T14:54:22.951521+0000","last_active":"2023-12-19T14:54:22.951521+0000","last_peered":"2023-12-19T14:54:22.951521+0000","last_clean":"2023-12-19T14:54:22.951521+0000","last_became_active":"2023-12-19T14:54:22.951211+0000","last_became_peered":"2023-12-19T14:54:22.951211+0000","last_unstale":"2023-12-19T14:54:22.951521+0000","last_undegraded":"2023-12-19T14:54:22.951521+0000","last_fullsized":"2023-12-19T14:54:22.951521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:13:43.962908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950530+0000","last_change":"2023-12-19T14:54:22.950530+0000","last_active":"2023-12-19T14:54:22.950530+0000","last_peered":"2023-12-19T14:54:22.950530+0000","last_clean":"2023-12-19T14:54:22.950530+0000","last_became_active":"2023-12-19T14:54:22.950238+0000","last_became_peered":"2023-12-19T14:54:22.950238+0000","last_unstale":"2023-12-19T14:54:22.950530+0000","last_undegraded":"2023-12-19T14:54:22.950530+0000","last_fullsized":"2023-12-19T14:54:22.950530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:26.149145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958637+0000","last_change":"2023-12-19T14:54:22.958637+0000","last_active":"2023-12-19T14:54:22.958637+0000","last_peered":"2023-12-19T14:54:22.958637+0000","last_clean":"2023-12-19T14:54:22.958637+0000","last_became_active":"2023-12-19T14:54:22.957866+0000","last_became_peered":"2023-12-19T14:54:22.957866+0000","last_unstale":"2023-12-19T14:54:22.958637+0000","last_undegraded":"2023-12-19T14:54:22.958637+0000","last_fullsized":"2023-12-19T14:54:22.958637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954805+0000","last_change":"2023-12-19T14:54:22.954805+0000","last_active":"2023-12-19T14:54:22.954805+0000","last_peered":"2023-12-19T14:54:22.954805+0000","last_clean":"2023-12-19T14:54:22.954805+0000","last_became_active":"2023-12-19T14:54:22.954533+0000","last_became_peered":"2023-12-19T14:54:22.954533+0000","last_unstale":"2023-12-19T14:54:22.954805+0000","last_undegraded":"2023-12-19T14:54:22.954805+0000","last_fullsized":"2023-12-19T14:54:22.954805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:20:32.162345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.951333+0000","last_change":"2023-12-19T14:54:22.951333+0000","last_active":"2023-12-19T14:54:22.951333+0000","last_peered":"2023-12-19T14:54:22.951333+0000","last_clean":"2023-12-19T14:54:22.951333+0000","last_became_active":"2023-12-19T14:54:22.951074+0000","last_became_peered":"2023-12-19T14:54:22.951074+0000","last_unstale":"2023-12-19T14:54:22.951333+0000","last_undegraded":"2023-12-19T14:54:22.951333+0000","last_fullsized":"2023-12-19T14:54:22.951333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:35:39.864176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959436+0000","last_change":"2023-12-19T14:54:22.959436+0000","last_active":"2023-12-19T14:54:22.959436+0000","last_peered":"2023-12-19T14:54:22.959436+0000","last_clean":"2023-12-19T14:54:22.959436+0000","last_became_active":"2023-12-19T14:54:22.959119+0000","last_became_peered":"2023-12-19T14:54:22.959119+0000","last_unstale":"2023-12-19T14:54:22.959436+0000","last_undegraded":"2023-12-19T14:54:22.959436+0000","last_fullsized":"2023-12-19T14:54:22.959436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949150+0000","last_change":"2023-12-19T14:54:22.949150+0000","last_active":"2023-12-19T14:54:22.949150+0000","last_peered":"2023-12-19T14:54:22.949150+0000","last_clean":"2023-12-19T14:54:22.949150+0000","last_became_active":"2023-12-19T14:54:22.948392+0000","last_became_peered":"2023-12-19T14:54:22.948392+0000","last_unstale":"2023-12-19T14:54:22.949150+0000","last_undegraded":"2023-12-19T14:54:22.949150+0000","last_fullsized":"2023-12-19T14:54:22.949150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:17.648233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964117+0000","last_change":"2023-12-19T14:54:22.964117+0000","last_active":"2023-12-19T14:54:22.964117+0000","last_peered":"2023-12-19T14:54:22.964117+0000","last_clean":"2023-12-19T14:54:22.964117+0000","last_became_active":"2023-12-19T14:54:22.963271+0000","last_became_peered":"2023-12-19T14:54:22.963271+0000","last_unstale":"2023-12-19T14:54:22.964117+0000","last_undegraded":"2023-12-19T14:54:22.964117+0000","last_fullsized":"2023-12-19T14:54:22.964117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:16.693921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947931+0000","last_change":"2023-12-19T14:54:22.947931+0000","last_active":"2023-12-19T14:54:22.947931+0000","last_peered":"2023-12-19T14:54:22.947931+0000","last_clean":"2023-12-19T14:54:22.947931+0000","last_became_active":"2023-12-19T14:54:22.947651+0000","last_became_peered":"2023-12-19T14:54:22.947651+0000","last_unstale":"2023-12-19T14:54:22.947931+0000","last_undegraded":"2023-12-19T14:54:22.947931+0000","last_fullsized":"2023-12-19T14:54:22.947931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:21:14.211842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954874+0000","last_change":"2023-12-19T14:54:22.954874+0000","last_active":"2023-12-19T14:54:22.954874+0000","last_peered":"2023-12-19T14:54:22.954874+0000","last_clean":"2023-12-19T14:54:22.954874+0000","last_became_active":"2023-12-19T14:54:22.954516+0000","last_became_peered":"2023-12-19T14:54:22.954516+0000","last_unstale":"2023-12-19T14:54:22.954874+0000","last_undegraded":"2023-12-19T14:54:22.954874+0000","last_fullsized":"2023-12-19T14:54:22.954874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:52.110889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960000+0000","last_change":"2023-12-19T14:54:22.960000+0000","last_active":"2023-12-19T14:54:22.960000+0000","last_peered":"2023-12-19T14:54:22.960000+0000","last_clean":"2023-12-19T14:54:22.960000+0000","last_became_active":"2023-12-19T14:54:22.959686+0000","last_became_peered":"2023-12-19T14:54:22.959686+0000","last_unstale":"2023-12-19T14:54:22.960000+0000","last_undegraded":"2023-12-19T14:54:22.960000+0000","last_fullsized":"2023-12-19T14:54:22.960000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:45.306170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948280+0000","last_change":"2023-12-19T14:54:22.944055+0000","last_active":"2023-12-19T14:54:22.948280+0000","last_peered":"2023-12-19T14:54:22.948280+0000","last_clean":"2023-12-19T14:54:22.948280+0000","last_became_active":"2023-12-19T14:54:22.943691+0000","last_became_peered":"2023-12-19T14:54:22.943691+0000","last_unstale":"2023-12-19T14:54:22.948280+0000","last_undegraded":"2023-12-19T14:54:22.948280+0000","last_fullsized":"2023-12-19T14:54:22.948280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:08:08.837335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"85'4","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.107141+0000","last_change":"2023-12-19T14:54:22.082231+0000","last_active":"2023-12-19T14:54:23.107141+0000","last_peered":"2023-12-19T14:54:23.107141+0000","last_clean":"2023-12-19T14:54:23.107141+0000","last_became_active":"2023-12-19T14:54:22.080837+0000","last_became_peered":"2023-12-19T14:54:22.080837+0000","last_unstale":"2023-12-19T14:54:23.107141+0000","last_undegraded":"2023-12-19T14:54:23.107141+0000","last_fullsized":"2023-12-19T14:54:23.107141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962609+0000","last_change":"2023-12-19T14:54:22.962609+0000","last_active":"2023-12-19T14:54:22.962609+0000","last_peered":"2023-12-19T14:54:22.962609+0000","last_clean":"2023-12-19T14:54:22.962609+0000","last_became_active":"2023-12-19T14:54:22.961296+0000","last_became_peered":"2023-12-19T14:54:22.961296+0000","last_unstale":"2023-12-19T14:54:22.962609+0000","last_undegraded":"2023-12-19T14:54:22.962609+0000","last_fullsized":"2023-12-19T14:54:22.962609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:04.041059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.947940+0000","last_change":"2023-12-19T14:54:22.097609+0000","last_active":"2023-12-19T14:54:22.947940+0000","last_peered":"2023-12-19T14:54:22.947940+0000","last_clean":"2023-12-19T14:54:22.947940+0000","last_became_active":"2023-12-19T14:54:22.096727+0000","last_became_peered":"2023-12-19T14:54:22.096727+0000","last_unstale":"2023-12-19T14:54:22.947940+0000","last_undegraded":"2023-12-19T14:54:22.947940+0000","last_fullsized":"2023-12-19T14:54:22.947940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:38.774987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958987+0000","last_change":"2023-12-19T14:54:22.958987+0000","last_active":"2023-12-19T14:54:22.958987+0000","last_peered":"2023-12-19T14:54:22.958987+0000","last_clean":"2023-12-19T14:54:22.958987+0000","last_became_active":"2023-12-19T14:54:22.958645+0000","last_became_peered":"2023-12-19T14:54:22.958645+0000","last_unstale":"2023-12-19T14:54:22.958987+0000","last_undegraded":"2023-12-19T14:54:22.958987+0000","last_fullsized":"2023-12-19T14:54:22.958987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:59.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102693+0000","last_change":"2023-12-19T14:54:22.100000+0000","last_active":"2023-12-19T14:54:23.102693+0000","last_peered":"2023-12-19T14:54:23.102693+0000","last_clean":"2023-12-19T14:54:23.102693+0000","last_became_active":"2023-12-19T14:54:22.099669+0000","last_became_peered":"2023-12-19T14:54:22.099669+0000","last_unstale":"2023-12-19T14:54:23.102693+0000","last_undegraded":"2023-12-19T14:54:23.102693+0000","last_fullsized":"2023-12-19T14:54:23.102693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:15.681450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954603+0000","last_change":"2023-12-19T14:54:22.954603+0000","last_active":"2023-12-19T14:54:22.954603+0000","last_peered":"2023-12-19T14:54:22.954603+0000","last_clean":"2023-12-19T14:54:22.954603+0000","last_became_active":"2023-12-19T14:54:22.953201+0000","last_became_peered":"2023-12-19T14:54:22.953201+0000","last_unstale":"2023-12-19T14:54:22.954603+0000","last_undegraded":"2023-12-19T14:54:22.954603+0000","last_fullsized":"2023-12-19T14:54:22.954603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.103829+0000","last_change":"2023-12-19T14:54:22.088620+0000","last_active":"2023-12-19T14:54:23.103829+0000","last_peered":"2023-12-19T14:54:23.103829+0000","last_clean":"2023-12-19T14:54:23.103829+0000","last_became_active":"2023-12-19T14:54:22.086527+0000","last_became_peered":"2023-12-19T14:54:22.086527+0000","last_unstale":"2023-12-19T14:54:23.103829+0000","last_undegraded":"2023-12-19T14:54:23.103829+0000","last_fullsized":"2023-12-19T14:54:23.103829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:19.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.955572+0000","last_change":"2023-12-19T14:54:22.947054+0000","last_active":"2023-12-19T14:54:22.955572+0000","last_peered":"2023-12-19T14:54:22.955572+0000","last_clean":"2023-12-19T14:54:22.955572+0000","last_became_active":"2023-12-19T14:54:22.946734+0000","last_became_peered":"2023-12-19T14:54:22.946734+0000","last_unstale":"2023-12-19T14:54:22.955572+0000","last_undegraded":"2023-12-19T14:54:22.955572+0000","last_fullsized":"2023-12-19T14:54:22.955572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:33:34.772110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.938789+0000","last_change":"2023-12-19T14:54:22.088785+0000","last_active":"2023-12-19T14:54:22.938789+0000","last_peered":"2023-12-19T14:54:22.938789+0000","last_clean":"2023-12-19T14:54:22.938789+0000","last_became_active":"2023-12-19T14:54:22.088529+0000","last_became_peered":"2023-12-19T14:54:22.088529+0000","last_unstale":"2023-12-19T14:54:22.938789+0000","last_undegraded":"2023-12-19T14:54:22.938789+0000","last_fullsized":"2023-12-19T14:54:22.938789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:05:19.956940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.965269+0000","last_change":"2023-12-19T14:54:22.965269+0000","last_active":"2023-12-19T14:54:22.965269+0000","last_peered":"2023-12-19T14:54:22.965269+0000","last_clean":"2023-12-19T14:54:22.965269+0000","last_became_active":"2023-12-19T14:54:22.964328+0000","last_became_peered":"2023-12-19T14:54:22.964328+0000","last_unstale":"2023-12-19T14:54:22.965269+0000","last_undegraded":"2023-12-19T14:54:22.965269+0000","last_fullsized":"2023-12-19T14:54:22.965269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.939322+0000","last_change":"2023-12-19T14:54:22.095788+0000","last_active":"2023-12-19T14:54:22.939322+0000","last_peered":"2023-12-19T14:54:22.939322+0000","last_clean":"2023-12-19T14:54:22.939322+0000","last_became_active":"2023-12-19T14:54:22.095449+0000","last_became_peered":"2023-12-19T14:54:22.095449+0000","last_unstale":"2023-12-19T14:54:22.939322+0000","last_undegraded":"2023-12-19T14:54:22.939322+0000","last_fullsized":"2023-12-19T14:54:22.939322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:38:15.681450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.962443+0000","last_change":"2023-12-19T14:54:22.957103+0000","last_active":"2023-12-19T14:54:22.962443+0000","last_peered":"2023-12-19T14:54:22.962443+0000","last_clean":"2023-12-19T14:54:22.962443+0000","last_became_active":"2023-12-19T14:54:22.956844+0000","last_became_peered":"2023-12-19T14:54:22.956844+0000","last_unstale":"2023-12-19T14:54:22.962443+0000","last_undegraded":"2023-12-19T14:54:22.962443+0000","last_fullsized":"2023-12-19T14:54:22.962443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:10:37.977882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098264+0000","last_change":"2023-12-19T14:54:22.094744+0000","last_active":"2023-12-19T14:54:22.098264+0000","last_peered":"2023-12-19T14:54:22.098264+0000","last_clean":"2023-12-19T14:54:22.098264+0000","last_became_active":"2023-12-19T14:54:22.092224+0000","last_became_peered":"2023-12-19T14:54:22.092224+0000","last_unstale":"2023-12-19T14:54:22.098264+0000","last_undegraded":"2023-12-19T14:54:22.098264+0000","last_fullsized":"2023-12-19T14:54:22.098264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:28.510850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.967881+0000","last_change":"2023-12-19T14:54:22.967881+0000","last_active":"2023-12-19T14:54:22.967881+0000","last_peered":"2023-12-19T14:54:22.967881+0000","last_clean":"2023-12-19T14:54:22.967881+0000","last_became_active":"2023-12-19T14:54:22.967554+0000","last_became_peered":"2023-12-19T14:54:22.967554+0000","last_unstale":"2023-12-19T14:54:22.967881+0000","last_undegraded":"2023-12-19T14:54:22.967881+0000","last_fullsized":"2023-12-19T14:54:22.967881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:48:45.306170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.938996+0000","last_change":"2023-12-19T14:54:22.087757+0000","last_active":"2023-12-19T14:54:22.938996+0000","last_peered":"2023-12-19T14:54:22.938996+0000","last_clean":"2023-12-19T14:54:22.938996+0000","last_became_active":"2023-12-19T14:54:22.086515+0000","last_became_peered":"2023-12-19T14:54:22.086515+0000","last_unstale":"2023-12-19T14:54:22.938996+0000","last_undegraded":"2023-12-19T14:54:22.938996+0000","last_fullsized":"2023-12-19T14:54:22.938996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:46:32.604895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.952303+0000","last_change":"2023-12-19T14:54:22.952303+0000","last_active":"2023-12-19T14:54:22.952303+0000","last_peered":"2023-12-19T14:54:22.952303+0000","last_clean":"2023-12-19T14:54:22.952303+0000","last_became_active":"2023-12-19T14:54:22.951825+0000","last_became_peered":"2023-12-19T14:54:22.951825+0000","last_unstale":"2023-12-19T14:54:22.952303+0000","last_undegraded":"2023-12-19T14:54:22.952303+0000","last_fullsized":"2023-12-19T14:54:22.952303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.093843+0000","last_change":"2023-12-19T14:54:22.090063+0000","last_active":"2023-12-19T14:54:22.093843+0000","last_peered":"2023-12-19T14:54:22.093843+0000","last_clean":"2023-12-19T14:54:22.093843+0000","last_became_active":"2023-12-19T14:54:22.089745+0000","last_became_peered":"2023-12-19T14:54:22.089745+0000","last_unstale":"2023-12-19T14:54:22.093843+0000","last_undegraded":"2023-12-19T14:54:22.093843+0000","last_fullsized":"2023-12-19T14:54:22.093843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:43.975646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.948695+0000","last_change":"2023-12-19T14:54:22.948695+0000","last_active":"2023-12-19T14:54:22.948695+0000","last_peered":"2023-12-19T14:54:22.948695+0000","last_clean":"2023-12-19T14:54:22.948695+0000","last_became_active":"2023-12-19T14:54:22.948320+0000","last_became_peered":"2023-12-19T14:54:22.948320+0000","last_unstale":"2023-12-19T14:54:22.948695+0000","last_undegraded":"2023-12-19T14:54:22.948695+0000","last_fullsized":"2023-12-19T14:54:22.948695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:59.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.096110+0000","last_change":"2023-12-19T14:54:22.093910+0000","last_active":"2023-12-19T14:54:22.096110+0000","last_peered":"2023-12-19T14:54:22.096110+0000","last_clean":"2023-12-19T14:54:22.096110+0000","last_became_active":"2023-12-19T14:54:22.093554+0000","last_became_peered":"2023-12-19T14:54:22.093554+0000","last_unstale":"2023-12-19T14:54:22.096110+0000","last_undegraded":"2023-12-19T14:54:22.096110+0000","last_fullsized":"2023-12-19T14:54:22.096110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954000+0000","last_change":"2023-12-19T14:54:22.949728+0000","last_active":"2023-12-19T14:54:22.954000+0000","last_peered":"2023-12-19T14:54:22.954000+0000","last_clean":"2023-12-19T14:54:22.954000+0000","last_became_active":"2023-12-19T14:54:22.948660+0000","last_became_peered":"2023-12-19T14:54:22.948660+0000","last_unstale":"2023-12-19T14:54:22.954000+0000","last_undegraded":"2023-12-19T14:54:22.954000+0000","last_fullsized":"2023-12-19T14:54:22.954000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:39.502278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"85'3","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.103976+0000","last_change":"2023-12-19T14:54:22.088152+0000","last_active":"2023-12-19T14:54:23.103976+0000","last_peered":"2023-12-19T14:54:23.103976+0000","last_clean":"2023-12-19T14:54:23.103976+0000","last_became_active":"2023-12-19T14:54:22.087870+0000","last_became_peered":"2023-12-19T14:54:22.087870+0000","last_unstale":"2023-12-19T14:54:23.103976+0000","last_undegraded":"2023-12-19T14:54:23.103976+0000","last_fullsized":"2023-12-19T14:54:23.103976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:04.952498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959993+0000","last_change":"2023-12-19T14:54:22.959993+0000","last_active":"2023-12-19T14:54:22.959993+0000","last_peered":"2023-12-19T14:54:22.959993+0000","last_clean":"2023-12-19T14:54:22.959993+0000","last_became_active":"2023-12-19T14:54:22.958434+0000","last_became_peered":"2023-12-19T14:54:22.958434+0000","last_unstale":"2023-12-19T14:54:22.959993+0000","last_undegraded":"2023-12-19T14:54:22.959993+0000","last_fullsized":"2023-12-19T14:54:22.959993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:17.648233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"85'6","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.107457+0000","last_change":"2023-12-19T14:54:22.093878+0000","last_active":"2023-12-19T14:54:23.107457+0000","last_peered":"2023-12-19T14:54:23.107457+0000","last_clean":"2023-12-19T14:54:23.107457+0000","last_became_active":"2023-12-19T14:54:22.093556+0000","last_became_peered":"2023-12-19T14:54:22.093556+0000","last_unstale":"2023-12-19T14:54:23.107457+0000","last_undegraded":"2023-12-19T14:54:23.107457+0000","last_fullsized":"2023-12-19T14:54:23.107457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:46.288069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.955764+0000","last_change":"2023-12-19T14:54:22.955764+0000","last_active":"2023-12-19T14:54:22.955764+0000","last_peered":"2023-12-19T14:54:22.955764+0000","last_clean":"2023-12-19T14:54:22.955764+0000","last_became_active":"2023-12-19T14:54:22.955431+0000","last_became_peered":"2023-12-19T14:54:22.955431+0000","last_unstale":"2023-12-19T14:54:22.955764+0000","last_undegraded":"2023-12-19T14:54:22.955764+0000","last_fullsized":"2023-12-19T14:54:22.955764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102059+0000","last_change":"2023-12-19T14:54:22.087341+0000","last_active":"2023-12-19T14:54:23.102059+0000","last_peered":"2023-12-19T14:54:23.102059+0000","last_clean":"2023-12-19T14:54:23.102059+0000","last_became_active":"2023-12-19T14:54:22.087061+0000","last_became_peered":"2023-12-19T14:54:22.087061+0000","last_unstale":"2023-12-19T14:54:23.102059+0000","last_undegraded":"2023-12-19T14:54:23.102059+0000","last_fullsized":"2023-12-19T14:54:23.102059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:38.489807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950613+0000","last_change":"2023-12-19T14:54:22.950613+0000","last_active":"2023-12-19T14:54:22.950613+0000","last_peered":"2023-12-19T14:54:22.950613+0000","last_clean":"2023-12-19T14:54:22.950613+0000","last_became_active":"2023-12-19T14:54:22.950339+0000","last_became_peered":"2023-12-19T14:54:22.950339+0000","last_unstale":"2023-12-19T14:54:22.950613+0000","last_undegraded":"2023-12-19T14:54:22.950613+0000","last_fullsized":"2023-12-19T14:54:22.950613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:03.971129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095972+0000","last_change":"2023-12-19T14:54:22.094574+0000","last_active":"2023-12-19T14:54:22.095972+0000","last_peered":"2023-12-19T14:54:22.095972+0000","last_clean":"2023-12-19T14:54:22.095972+0000","last_became_active":"2023-12-19T14:54:22.094262+0000","last_became_peered":"2023-12-19T14:54:22.094262+0000","last_unstale":"2023-12-19T14:54:22.095972+0000","last_undegraded":"2023-12-19T14:54:22.095972+0000","last_fullsized":"2023-12-19T14:54:22.095972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:35.881438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954322+0000","last_change":"2023-12-19T14:54:22.950865+0000","last_active":"2023-12-19T14:54:22.954322+0000","last_peered":"2023-12-19T14:54:22.954322+0000","last_clean":"2023-12-19T14:54:22.954322+0000","last_became_active":"2023-12-19T14:54:22.950159+0000","last_became_peered":"2023-12-19T14:54:22.950159+0000","last_unstale":"2023-12-19T14:54:22.954322+0000","last_undegraded":"2023-12-19T14:54:22.954322+0000","last_fullsized":"2023-12-19T14:54:22.954322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:48.373908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095211+0000","last_change":"2023-12-19T14:54:22.092245+0000","last_active":"2023-12-19T14:54:22.095211+0000","last_peered":"2023-12-19T14:54:22.095211+0000","last_clean":"2023-12-19T14:54:22.095211+0000","last_became_active":"2023-12-19T14:54:22.091985+0000","last_became_peered":"2023-12-19T14:54:22.091985+0000","last_unstale":"2023-12-19T14:54:22.095211+0000","last_undegraded":"2023-12-19T14:54:22.095211+0000","last_fullsized":"2023-12-19T14:54:22.095211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:46.288069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954381+0000","last_change":"2023-12-19T14:54:22.954381+0000","last_active":"2023-12-19T14:54:22.954381+0000","last_peered":"2023-12-19T14:54:22.954381+0000","last_clean":"2023-12-19T14:54:22.954381+0000","last_became_active":"2023-12-19T14:54:22.953840+0000","last_became_peered":"2023-12-19T14:54:22.953840+0000","last_unstale":"2023-12-19T14:54:22.954381+0000","last_undegraded":"2023-12-19T14:54:22.954381+0000","last_fullsized":"2023-12-19T14:54:22.954381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:24:08.256925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"85'8","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.105454+0000","last_change":"2023-12-19T14:54:22.087039+0000","last_active":"2023-12-19T14:54:23.105454+0000","last_peered":"2023-12-19T14:54:23.105454+0000","last_clean":"2023-12-19T14:54:23.105454+0000","last_became_active":"2023-12-19T14:54:22.086741+0000","last_became_peered":"2023-12-19T14:54:22.086741+0000","last_unstale":"2023-12-19T14:54:23.105454+0000","last_undegraded":"2023-12-19T14:54:23.105454+0000","last_fullsized":"2023-12-19T14:54:23.105454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.960903+0000","last_change":"2023-12-19T14:54:22.960903+0000","last_active":"2023-12-19T14:54:22.960903+0000","last_peered":"2023-12-19T14:54:22.960903+0000","last_clean":"2023-12-19T14:54:22.960903+0000","last_became_active":"2023-12-19T14:54:22.960558+0000","last_became_peered":"2023-12-19T14:54:22.960558+0000","last_unstale":"2023-12-19T14:54:22.960903+0000","last_undegraded":"2023-12-19T14:54:22.960903+0000","last_fullsized":"2023-12-19T14:54:22.960903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:29.523321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.940395+0000","last_change":"2023-12-19T14:54:22.095403+0000","last_active":"2023-12-19T14:54:22.940395+0000","last_peered":"2023-12-19T14:54:22.940395+0000","last_clean":"2023-12-19T14:54:22.940395+0000","last_became_active":"2023-12-19T14:54:22.095005+0000","last_became_peered":"2023-12-19T14:54:22.095005+0000","last_unstale":"2023-12-19T14:54:22.940395+0000","last_undegraded":"2023-12-19T14:54:22.940395+0000","last_fullsized":"2023-12-19T14:54:22.940395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:16.166610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.966650+0000","last_change":"2023-12-19T14:54:22.961917+0000","last_active":"2023-12-19T14:54:22.966650+0000","last_peered":"2023-12-19T14:54:22.966650+0000","last_clean":"2023-12-19T14:54:22.966650+0000","last_became_active":"2023-12-19T14:54:22.960852+0000","last_became_peered":"2023-12-19T14:54:22.960852+0000","last_unstale":"2023-12-19T14:54:22.966650+0000","last_undegraded":"2023-12-19T14:54:22.966650+0000","last_fullsized":"2023-12-19T14:54:22.966650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:20.710143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.102211+0000","last_change":"2023-12-19T14:54:22.092851+0000","last_active":"2023-12-19T14:54:23.102211+0000","last_peered":"2023-12-19T14:54:23.102211+0000","last_clean":"2023-12-19T14:54:23.102211+0000","last_became_active":"2023-12-19T14:54:22.092583+0000","last_became_peered":"2023-12-19T14:54:22.092583+0000","last_unstale":"2023-12-19T14:54:23.102211+0000","last_undegraded":"2023-12-19T14:54:23.102211+0000","last_fullsized":"2023-12-19T14:54:23.102211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:17:56.066193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.959319+0000","last_change":"2023-12-19T14:54:22.959319+0000","last_active":"2023-12-19T14:54:22.959319+0000","last_peered":"2023-12-19T14:54:22.959319+0000","last_clean":"2023-12-19T14:54:22.959319+0000","last_became_active":"2023-12-19T14:54:22.959046+0000","last_became_peered":"2023-12-19T14:54:22.959046+0000","last_unstale":"2023-12-19T14:54:22.959319+0000","last_undegraded":"2023-12-19T14:54:22.959319+0000","last_fullsized":"2023-12-19T14:54:22.959319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:38.352211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"85'1","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.941908+0000","last_change":"2023-12-19T14:54:22.091281+0000","last_active":"2023-12-19T14:54:22.941908+0000","last_peered":"2023-12-19T14:54:22.941908+0000","last_clean":"2023-12-19T14:54:22.941908+0000","last_became_active":"2023-12-19T14:54:22.090956+0000","last_became_peered":"2023-12-19T14:54:22.090956+0000","last_unstale":"2023-12-19T14:54:22.941908+0000","last_undegraded":"2023-12-19T14:54:22.941908+0000","last_fullsized":"2023-12-19T14:54:22.941908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:47:42.172427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.953629+0000","last_change":"2023-12-19T14:54:22.946425+0000","last_active":"2023-12-19T14:54:22.953629+0000","last_peered":"2023-12-19T14:54:22.953629+0000","last_clean":"2023-12-19T14:54:22.953629+0000","last_became_active":"2023-12-19T14:54:22.946100+0000","last_became_peered":"2023-12-19T14:54:22.946100+0000","last_unstale":"2023-12-19T14:54:22.953629+0000","last_undegraded":"2023-12-19T14:54:22.953629+0000","last_fullsized":"2023-12-19T14:54:22.953629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:44.988117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098055+0000","last_change":"2023-12-19T14:54:22.094014+0000","last_active":"2023-12-19T14:54:22.098055+0000","last_peered":"2023-12-19T14:54:22.098055+0000","last_clean":"2023-12-19T14:54:22.098055+0000","last_became_active":"2023-12-19T14:54:22.093735+0000","last_became_peered":"2023-12-19T14:54:22.093735+0000","last_unstale":"2023-12-19T14:54:22.098055+0000","last_undegraded":"2023-12-19T14:54:22.098055+0000","last_fullsized":"2023-12-19T14:54:22.098055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:03.028588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958158+0000","last_change":"2023-12-19T14:54:22.958158+0000","last_active":"2023-12-19T14:54:22.958158+0000","last_peered":"2023-12-19T14:54:22.958158+0000","last_clean":"2023-12-19T14:54:22.958158+0000","last_became_active":"2023-12-19T14:54:22.957834+0000","last_became_peered":"2023-12-19T14:54:22.957834+0000","last_unstale":"2023-12-19T14:54:22.958158+0000","last_undegraded":"2023-12-19T14:54:22.958158+0000","last_fullsized":"2023-12-19T14:54:22.958158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:34.627955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.104761+0000","last_change":"2023-12-19T14:54:22.094508+0000","last_active":"2023-12-19T14:54:23.104761+0000","last_peered":"2023-12-19T14:54:23.104761+0000","last_clean":"2023-12-19T14:54:23.104761+0000","last_became_active":"2023-12-19T14:54:22.091569+0000","last_became_peered":"2023-12-19T14:54:22.091569+0000","last_unstale":"2023-12-19T14:54:23.104761+0000","last_undegraded":"2023-12-19T14:54:23.104761+0000","last_fullsized":"2023-12-19T14:54:23.104761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:08.251166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.949665+0000","last_change":"2023-12-19T14:54:22.949665+0000","last_active":"2023-12-19T14:54:22.949665+0000","last_peered":"2023-12-19T14:54:22.949665+0000","last_clean":"2023-12-19T14:54:22.949665+0000","last_became_active":"2023-12-19T14:54:22.949377+0000","last_became_peered":"2023-12-19T14:54:22.949377+0000","last_unstale":"2023-12-19T14:54:22.949665+0000","last_undegraded":"2023-12-19T14:54:22.949665+0000","last_fullsized":"2023-12-19T14:54:22.949665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:17.716909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.091242+0000","last_change":"2023-12-19T14:54:22.089876+0000","last_active":"2023-12-19T14:54:22.091242+0000","last_peered":"2023-12-19T14:54:22.091242+0000","last_clean":"2023-12-19T14:54:22.091242+0000","last_became_active":"2023-12-19T14:54:22.089585+0000","last_became_peered":"2023-12-19T14:54:22.089585+0000","last_unstale":"2023-12-19T14:54:22.091242+0000","last_undegraded":"2023-12-19T14:54:22.091242+0000","last_fullsized":"2023-12-19T14:54:22.091242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:55.035096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958359+0000","last_change":"2023-12-19T14:54:22.956348+0000","last_active":"2023-12-19T14:54:22.958359+0000","last_peered":"2023-12-19T14:54:22.958359+0000","last_clean":"2023-12-19T14:54:22.958359+0000","last_became_active":"2023-12-19T14:54:22.956050+0000","last_became_peered":"2023-12-19T14:54:22.956050+0000","last_unstale":"2023-12-19T14:54:22.958359+0000","last_undegraded":"2023-12-19T14:54:22.958359+0000","last_fullsized":"2023-12-19T14:54:22.958359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:31.728767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098105+0000","last_change":"2023-12-19T14:54:22.094252+0000","last_active":"2023-12-19T14:54:22.098105+0000","last_peered":"2023-12-19T14:54:22.098105+0000","last_clean":"2023-12-19T14:54:22.098105+0000","last_became_active":"2023-12-19T14:54:22.093935+0000","last_became_peered":"2023-12-19T14:54:22.093935+0000","last_unstale":"2023-12-19T14:54:22.098105+0000","last_undegraded":"2023-12-19T14:54:22.098105+0000","last_fullsized":"2023-12-19T14:54:22.098105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:51.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.964693+0000","last_change":"2023-12-19T14:54:22.964693+0000","last_active":"2023-12-19T14:54:22.964693+0000","last_peered":"2023-12-19T14:54:22.964693+0000","last_clean":"2023-12-19T14:54:22.964693+0000","last_became_active":"2023-12-19T14:54:22.964414+0000","last_became_peered":"2023-12-19T14:54:22.964414+0000","last_unstale":"2023-12-19T14:54:22.964693+0000","last_undegraded":"2023-12-19T14:54:22.964693+0000","last_fullsized":"2023-12-19T14:54:22.964693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:26.149145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"85'2","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:23.098570+0000","last_change":"2023-12-19T14:54:22.086565+0000","last_active":"2023-12-19T14:54:23.098570+0000","last_peered":"2023-12-19T14:54:23.098570+0000","last_clean":"2023-12-19T14:54:23.098570+0000","last_became_active":"2023-12-19T14:54:22.086285+0000","last_became_peered":"2023-12-19T14:54:22.086285+0000","last_unstale":"2023-12-19T14:54:23.098570+0000","last_undegraded":"2023-12-19T14:54:23.098570+0000","last_fullsized":"2023-12-19T14:54:23.098570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:25.356833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.967055+0000","last_change":"2023-12-19T14:54:22.963254+0000","last_active":"2023-12-19T14:54:22.967055+0000","last_peered":"2023-12-19T14:54:22.967055+0000","last_clean":"2023-12-19T14:54:22.967055+0000","last_became_active":"2023-12-19T14:54:22.962933+0000","last_became_peered":"2023-12-19T14:54:22.962933+0000","last_unstale":"2023-12-19T14:54:22.967055+0000","last_undegraded":"2023-12-19T14:54:22.967055+0000","last_fullsized":"2023-12-19T14:54:22.967055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:35:41.801705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.092981+0000","last_change":"2023-12-19T14:54:22.091910+0000","last_active":"2023-12-19T14:54:22.092981+0000","last_peered":"2023-12-19T14:54:22.092981+0000","last_clean":"2023-12-19T14:54:22.092981+0000","last_became_active":"2023-12-19T14:54:22.091514+0000","last_became_peered":"2023-12-19T14:54:22.091514+0000","last_unstale":"2023-12-19T14:54:22.092981+0000","last_undegraded":"2023-12-19T14:54:22.092981+0000","last_fullsized":"2023-12-19T14:54:22.092981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:25:09.268439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957216+0000","last_change":"2023-12-19T14:54:22.957216+0000","last_active":"2023-12-19T14:54:22.957216+0000","last_peered":"2023-12-19T14:54:22.957216+0000","last_clean":"2023-12-19T14:54:22.957216+0000","last_became_active":"2023-12-19T14:54:22.956914+0000","last_became_peered":"2023-12-19T14:54:22.956914+0000","last_unstale":"2023-12-19T14:54:22.957216+0000","last_undegraded":"2023-12-19T14:54:22.957216+0000","last_fullsized":"2023-12-19T14:54:22.957216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:18.690127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.101577+0000","last_change":"2023-12-19T14:54:22.099895+0000","last_active":"2023-12-19T14:54:22.101577+0000","last_peered":"2023-12-19T14:54:22.101577+0000","last_clean":"2023-12-19T14:54:22.101577+0000","last_became_active":"2023-12-19T14:54:22.099584+0000","last_became_peered":"2023-12-19T14:54:22.099584+0000","last_unstale":"2023-12-19T14:54:22.101577+0000","last_undegraded":"2023-12-19T14:54:22.101577+0000","last_fullsized":"2023-12-19T14:54:22.101577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:28.510850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.953646+0000","last_change":"2023-12-19T14:54:22.948439+0000","last_active":"2023-12-19T14:54:22.953646+0000","last_peered":"2023-12-19T14:54:22.953646+0000","last_clean":"2023-12-19T14:54:22.953646+0000","last_became_active":"2023-12-19T14:54:22.948101+0000","last_became_peered":"2023-12-19T14:54:22.948101+0000","last_unstale":"2023-12-19T14:54:22.953646+0000","last_undegraded":"2023-12-19T14:54:22.953646+0000","last_fullsized":"2023-12-19T14:54:22.953646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:47:43.184898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.097533+0000","last_change":"2023-12-19T14:54:22.093771+0000","last_active":"2023-12-19T14:54:22.097533+0000","last_peered":"2023-12-19T14:54:22.097533+0000","last_clean":"2023-12-19T14:54:22.097533+0000","last_became_active":"2023-12-19T14:54:22.093450+0000","last_became_peered":"2023-12-19T14:54:22.093450+0000","last_unstale":"2023-12-19T14:54:22.097533+0000","last_undegraded":"2023-12-19T14:54:22.097533+0000","last_fullsized":"2023-12-19T14:54:22.097533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.954301+0000","last_change":"2023-12-19T14:54:22.950822+0000","last_active":"2023-12-19T14:54:22.954301+0000","last_peered":"2023-12-19T14:54:22.954301+0000","last_clean":"2023-12-19T14:54:22.954301+0000","last_became_active":"2023-12-19T14:54:22.950547+0000","last_became_peered":"2023-12-19T14:54:22.950547+0000","last_unstale":"2023-12-19T14:54:22.954301+0000","last_undegraded":"2023-12-19T14:54:22.954301+0000","last_fullsized":"2023-12-19T14:54:22.954301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:05.964970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095529+0000","last_change":"2023-12-19T14:54:22.093235+0000","last_active":"2023-12-19T14:54:22.095529+0000","last_peered":"2023-12-19T14:54:22.095529+0000","last_clean":"2023-12-19T14:54:22.095529+0000","last_became_active":"2023-12-19T14:54:22.090494+0000","last_became_peered":"2023-12-19T14:54:22.090494+0000","last_unstale":"2023-12-19T14:54:22.095529+0000","last_undegraded":"2023-12-19T14:54:22.095529+0000","last_fullsized":"2023-12-19T14:54:22.095529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:55:47.361437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.958890+0000","last_change":"2023-12-19T14:54:22.958890+0000","last_active":"2023-12-19T14:54:22.958890+0000","last_peered":"2023-12-19T14:54:22.958890+0000","last_clean":"2023-12-19T14:54:22.958890+0000","last_became_active":"2023-12-19T14:54:22.958521+0000","last_became_peered":"2023-12-19T14:54:22.958521+0000","last_unstale":"2023-12-19T14:54:22.958890+0000","last_undegraded":"2023-12-19T14:54:22.958890+0000","last_fullsized":"2023-12-19T14:54:22.958890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:39.787458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.095208+0000","last_change":"2023-12-19T14:54:22.087737+0000","last_active":"2023-12-19T14:54:22.095208+0000","last_peered":"2023-12-19T14:54:22.095208+0000","last_clean":"2023-12-19T14:54:22.095208+0000","last_became_active":"2023-12-19T14:54:22.087431+0000","last_became_peered":"2023-12-19T14:54:22.087431+0000","last_unstale":"2023-12-19T14:54:22.095208+0000","last_undegraded":"2023-12-19T14:54:22.095208+0000","last_fullsized":"2023-12-19T14:54:22.095208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:53.397056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":8,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.965720+0000","last_change":"2023-12-19T14:54:22.965720+0000","last_active":"2023-12-19T14:54:22.965720+0000","last_peered":"2023-12-19T14:54:22.965720+0000","last_clean":"2023-12-19T14:54:22.965720+0000","last_became_active":"2023-12-19T14:54:22.963707+0000","last_became_peered":"2023-12-19T14:54:22.963707+0000","last_unstale":"2023-12-19T14:54:22.965720+0000","last_undegraded":"2023-12-19T14:54:22.965720+0000","last_fullsized":"2023-12-19T14:54:22.965720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:47:54.409527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.098282+0000","last_change":"2023-12-19T14:54:22.094912+0000","last_active":"2023-12-19T14:54:22.098282+0000","last_peered":"2023-12-19T14:54:22.098282+0000","last_clean":"2023-12-19T14:54:22.098282+0000","last_became_active":"2023-12-19T14:54:22.094485+0000","last_became_peered":"2023-12-19T14:54:22.094485+0000","last_unstale":"2023-12-19T14:54:22.098282+0000","last_undegraded":"2023-12-19T14:54:22.098282+0000","last_fullsized":"2023-12-19T14:54:22.098282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:38.774987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.957827+0000","last_change":"2023-12-19T14:54:22.951004+0000","last_active":"2023-12-19T14:54:22.957827+0000","last_peered":"2023-12-19T14:54:22.957827+0000","last_clean":"2023-12-19T14:54:22.957827+0000","last_became_active":"2023-12-19T14:54:22.950715+0000","last_became_peered":"2023-12-19T14:54:22.950715+0000","last_unstale":"2023-12-19T14:54:22.957827+0000","last_undegraded":"2023-12-19T14:54:22.957827+0000","last_fullsized":"2023-12-19T14:54:22.957827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:49:15.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.081824+0000","last_change":"2023-12-19T14:54:22.079441+0000","last_active":"2023-12-19T14:54:22.081824+0000","last_peered":"2023-12-19T14:54:22.081824+0000","last_clean":"2023-12-19T14:54:22.081824+0000","last_became_active":"2023-12-19T14:54:22.079118+0000","last_became_peered":"2023-12-19T14:54:22.079118+0000","last_unstale":"2023-12-19T14:54:22.081824+0000","last_undegraded":"2023-12-19T14:54:22.081824+0000","last_fullsized":"2023-12-19T14:54:22.081824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:19.697671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.950337+0000","last_change":"2023-12-19T14:54:22.945328+0000","last_active":"2023-12-19T14:54:22.950337+0000","last_peered":"2023-12-19T14:54:22.950337+0000","last_clean":"2023-12-19T14:54:22.950337+0000","last_became_active":"2023-12-19T14:54:22.945033+0000","last_became_peered":"2023-12-19T14:54:22.945033+0000","last_unstale":"2023-12-19T14:54:22.950337+0000","last_undegraded":"2023-12-19T14:54:22.950337+0000","last_fullsized":"2023-12-19T14:54:22.950337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:22.053384+0000","last_clean_scrub_stamp":"2023-12-19T14:54:22.053384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:48:13.499494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.093232+0000","last_change":"2023-12-19T14:54:22.089091+0000","last_active":"2023-12-19T14:54:22.093232+0000","last_peered":"2023-12-19T14:54:22.093232+0000","last_clean":"2023-12-19T14:54:22.093232+0000","last_became_active":"2023-12-19T14:54:22.088777+0000","last_became_peered":"2023-12-19T14:54:22.088777+0000","last_unstale":"2023-12-19T14:54:22.093232+0000","last_undegraded":"2023-12-19T14:54:22.093232+0000","last_fullsized":"2023-12-19T14:54:22.093232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:21.040913+0000","last_clean_scrub_stamp":"2023-12-19T14:54:21.040913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:08.251166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'161","reported_seq":373,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T14:54:22.075964+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:54:22.075964+0000","last_peered":"2023-12-19T14:54:22.075964+0000","last_clean":"2023-12-19T14:54:22.075964+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:54:22.075964+0000","last_undegraded":"2023-12-19T14:54:22.075964+0000","last_fullsized":"2023-12-19T14:54:22.075964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575034,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":1105170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.498,"5min":3.544,"15min":3.544},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81699999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5369999999999999,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.093},"min":{"1min":0.91700000000000004,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.4379999999999999,"5min":1.73,"15min":1.73},"last":1.262},{"interface":"front","average":{"1min":1.276,"5min":1.1950000000000001,"15min":1.1819999999999999},"min":{"1min":0.93400000000000005,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.984},"last":1.5089999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2729999999999999,"15min":1.2330000000000001},"min":{"1min":0.96699999999999997,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.571,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":1.546},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.274,"15min":1.278},"min":{"1min":1.1359999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.667,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.28,"15min":1.29},"min":{"1min":0.92500000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.5409999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.093},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.3109999999999999},"min":{"1min":1.038,"5min":0.95899999999999996,"15min":0.79700000000000004},"max":{"1min":1.6100000000000001,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.5860000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.1839999999999999,"15min":1.1699999999999999},"min":{"1min":0.68500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.752,"5min":1.752,"15min":2.1930000000000001},"last":1.673},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2410000000000001,"15min":1.222},"min":{"1min":0.90300000000000002,"5min":0.75700000000000001,"15min":0.67500000000000004},"max":{"1min":1.6000000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.1859999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2210000000000001,"15min":1.2270000000000001},"min":{"1min":1.034,"5min":0.496,"15min":0.496},"max":{"1min":1.6930000000000001,"5min":1.8109999999999999,"15min":1.976},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3029999999999999,"15min":1.3180000000000001},"min":{"1min":0.90200000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.3620000000000001},"last":1.121}]},{"osd":6,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3899999999999999,"15min":1.3999999999999999},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.65600000000000003},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":3.1589999999999998},"last":1.371},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3620000000000001,"15min":1.361},"min":{"1min":1.0620000000000001,"5min":0.88500000000000001,"15min":0.70299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.298}]}]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302396,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069444,"statfs":{"total":96636764160,"available":96327110656,"internally_reserved":0,"allocated":1036288,"data_stored":514088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.1659999999999999},"min":{"1min":0.499,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6100000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.169,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.845,"5min":1.845,"15min":2.335},"last":1.0920000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.2010000000000001,"15min":1.2150000000000001},"min":{"1min":0.80600000000000005,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":2.016,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.026},{"interface":"front","average":{"1min":1.254,"5min":1.268,"15min":1.2849999999999999},"min":{"1min":0.78000000000000003,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.526,"5min":3.226,"15min":3.226},"last":0.76400000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2829999999999999,"15min":1.274},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7110000000000001,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.363},{"interface":"front","average":{"1min":1.25,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.82399999999999995,"5min":0.70799999999999996,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.337}]},{"osd":3,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3160000000000001,"15min":1.3420000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.7769999999999999,"5min":1.9239999999999999,"15min":3.2330000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3680000000000001,"15min":1.3720000000000001},"min":{"1min":0.84299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.645,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2609999999999999},"min":{"1min":0.70899999999999996,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":2.1040000000000001,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.294,"15min":1.3500000000000001},"min":{"1min":0.90200000000000002,"5min":0.79800000000000004,"15min":0.57599999999999996},"max":{"1min":1.901,"5min":3.3460000000000001,"15min":3.4399999999999999},"last":1.23}]},{"osd":5,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.321,"15min":1.351},"min":{"1min":0.93300000000000005,"5min":0.71799999999999997,"15min":0.63100000000000001},"max":{"1min":1.736,"5min":2.02,"15min":2.8620000000000001},"last":1.425},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.4019999999999999},"min":{"1min":0.82799999999999996,"5min":0.76300000000000001,"15min":0.51000000000000001},"max":{"1min":1.931,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.2709999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:53:24 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.456,"15min":1.456},"min":{"1min":0.97599999999999998,"5min":0.85899999999999999,"15min":0.53000000000000003},"max":{"1min":1.8140000000000001,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.496,"15min":1.512},"min":{"1min":1.0720000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.821,"5min":3.742,"15min":3.742},"last":1.3009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575034,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298300,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073540,"statfs":{"total":96636764160,"available":96331304960,"internally_reserved":0,"allocated":1036288,"data_stored":514076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.0389999999999999,"15min":1.0569999999999999},"min":{"1min":0.80400000000000005,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.601,"5min":1.601,"15min":1.619},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.198,"5min":1.141,"15min":1.139},"min":{"1min":0.875,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.6539999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.93000000000000005}]},{"osd":1,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2729999999999999,"15min":1.29},"min":{"1min":0.97099999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.8029999999999999,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.016},{"interface":"front","average":{"1min":1.272,"5min":1.2210000000000001,"15min":1.244},"min":{"1min":1.0660000000000001,"5min":0.70299999999999996,"15min":0.55700000000000005},"max":{"1min":1.494,"5min":1.5649999999999999,"15min":1.7869999999999999},"last":1.3400000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.204,"15min":1.2290000000000001},"min":{"1min":0.83099999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.645,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.052},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1759999999999999,"15min":1.21},"min":{"1min":0.88900000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.448,"5min":1.8600000000000001,"15min":2.0739999999999998},"last":1.1180000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.2689999999999999,"15min":1.2769999999999999},"min":{"1min":0.88200000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.726,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.409},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2549999999999999,"15min":1.284},"min":{"1min":0.98799999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.73,"5min":1.73,"15min":2.2149999999999999},"last":1.286}]},{"osd":4,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.83299999999999996,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5089999999999999,"5min":1.603,"15min":2.1099999999999999},"last":1.488},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2210000000000001,"15min":1.2390000000000001},"min":{"1min":0.76700000000000002,"5min":0.69099999999999995,"15min":0.49299999999999999},"max":{"1min":1.7709999999999999,"5min":2.27,"15min":2.27},"last":1.6319999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3009999999999999,"15min":1.3440000000000001},"min":{"1min":0.83099999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6140000000000001,"5min":1.913,"15min":2.4790000000000001},"last":1.377},{"interface":"front","average":{"1min":1.278,"5min":1.319,"15min":1.3360000000000001},"min":{"1min":0.70999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6859999999999999,"5min":1.8540000000000001,"15min":2.0030000000000001},"last":1.5900000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:53:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3160000000000001,"15min":1.363},"min":{"1min":0.89200000000000002,"5min":0.76700000000000002,"15min":0.58899999999999997},"max":{"1min":1.6699999999999999,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.556},{"interface":"front","average":{"1min":1.353,"5min":1.3360000000000001,"15min":1.3999999999999999},"min":{"1min":0.95299999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.6779999999999999,"5min":2.0070000000000001,"15min":2.411},"last":1.522}]}]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302404,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069436,"statfs":{"total":96636764160,"available":96327102464,"internally_reserved":0,"allocated":1044480,"data_stored":515590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.077,"15min":1.073},"min":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.58199999999999996},"max":{"1min":1.452,"5min":1.579,"15min":1.7869999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1140000000000001,"15min":1.0960000000000001},"min":{"1min":0.68400000000000005,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.4450000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.69999999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.2370000000000001,"15min":1.244},"min":{"1min":1.073,"5min":0.623,"15min":0.623},"max":{"1min":1.5680000000000001,"5min":1.6819999999999999,"15min":2.1259999999999999},"last":0.75},{"interface":"front","average":{"1min":1.244,"5min":1.262,"15min":1.28},"min":{"1min":0.89200000000000002,"5min":0.70299999999999996,"15min":0.65100000000000002},"max":{"1min":1.5960000000000001,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":0.81599999999999995}]},{"osd":2,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.1499999999999999,"15min":1.1519999999999999},"min":{"1min":0.877,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.891},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2270000000000001,"15min":1.2170000000000001},"min":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.65500000000000003},"max":{"1min":1.514,"5min":1.629,"15min":1.893},"last":0.64900000000000002}]},{"osd":3,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.84399999999999997,"5min":0.82099999999999995,"15min":0.59599999999999997},"max":{"1min":1.5620000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.93999999999999995,"5min":0.83799999999999997,"15min":0.60599999999999998},"max":{"1min":1.593,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.6259999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1599999999999999,"15min":1.173},"min":{"1min":0.91200000000000003,"5min":0.71099999999999997,"15min":0.57899999999999996},"max":{"1min":1.5629999999999999,"5min":2.1800000000000002,"15min":2.3500000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.143,"15min":1.1779999999999999},"min":{"1min":0.84699999999999998,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.526,"5min":2.077,"15min":2.161},"last":0.83699999999999997}]},{"osd":6,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.294,"15min":1.347},"min":{"1min":0.80300000000000005,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.671,"5min":2.8119999999999998,"15min":3.27},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.4159999999999999,"15min":1.4390000000000001},"min":{"1min":1.0369999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.633,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":0.86899999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:53:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.343,"15min":1.3879999999999999},"min":{"1min":0.94599999999999995,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.6200000000000001,"5min":2.9910000000000001,"15min":5.2060000000000004},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.429,"15min":1.48},"min":{"1min":0.872,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":1.669,"5min":3.008,"15min":3.6739999999999999},"last":0.92300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575033,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306488,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065352,"statfs":{"total":96636764160,"available":96322920448,"internally_reserved":0,"allocated":1032192,"data_stored":514054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84999999999999998,"15min":0.89000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2010000000000001,"5min":1.3640000000000001,"15min":4.1159999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.026,"5min":1.01,"15min":1.0129999999999999},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.542,"5min":1.542,"15min":3.3929999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":0.95799999999999996},"min":{"1min":0.69599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0149999999999999,"5min":1.611,"15min":2.4390000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.121,"15min":1.1220000000000001},"min":{"1min":0.749,"5min":0.63500000000000001,"15min":0.60199999999999998},"max":{"1min":1.5189999999999999,"5min":1.661,"15min":4.476},"last":1.224}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1459999999999999,"15min":1.1419999999999999},"min":{"1min":0.89900000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.639,"5min":1.76,"15min":2.6989999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61699999999999999},"max":{"1min":1.5309999999999999,"5min":1.647,"15min":2.5110000000000001},"last":1.0129999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.345},"min":{"1min":0.84899999999999998,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.526,"5min":1.768,"15min":3.46},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2829999999999999,"15min":1.3069999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.542,"5min":1.6970000000000001,"15min":2.9409999999999998},"last":1.1870000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.286,"15min":1.3280000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.629,"5min":1.7709999999999999,"15min":3.1019999999999999},"last":1.375},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.4159999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.833,"5min":2.1619999999999999,"15min":4.3650000000000002},"last":1.3500000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.446,"15min":1.4730000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9099999999999999,"5min":2.2709999999999999,"15min":4.5999999999999996},"last":1.294},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.419,"15min":1.444},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":3.4329999999999998},"last":1.321}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.516,"15min":1.5429999999999999},"min":{"1min":0.93300000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.7629999999999999,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.45,"15min":1.488},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.728,"5min":2.0670000000000002,"15min":3.7240000000000002},"last":1.5009999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306484,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065356,"statfs":{"total":96636764160,"available":96322924544,"internally_reserved":0,"allocated":1028096,"data_stored":513043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89400000000000002,"15min":0.90400000000000003},"min":{"1min":0.60599999999999998,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.946},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.056,"15min":1.0720000000000001},"min":{"1min":0.63300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.605,"5min":1.605,"15min":1.8560000000000001},"last":1.218}]},{"osd":1,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.82199999999999995,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.085},{"interface":"front","average":{"1min":1.169,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.3169999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1.0009999999999999},"min":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.304,"5min":1.5669999999999999,"15min":1.9370000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.081,"15min":1.105},"min":{"1min":0.80100000000000005,"5min":0.66000000000000003,"15min":0.55000000000000004},"max":{"1min":1.603,"5min":1.603,"15min":1.6930000000000001},"last":1.131}]},{"osd":4,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":0.95199999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":1.893,"15min":3.4950000000000001},"last":1.165},{"interface":"front","average":{"1min":1.448,"5min":1.403,"15min":1.3999999999999999},"min":{"1min":1.2230000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.923,"5min":1.96,"15min":2.0409999999999999},"last":1.2969999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3740000000000001,"15min":1.3600000000000001},"min":{"1min":1.1040000000000001,"5min":0.876,"15min":0.876},"max":{"1min":1.7669999999999999,"5min":2.0760000000000001,"15min":3.8599999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.93899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.6759999999999999,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.347}]},{"osd":6,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.80600000000000005},"max":{"1min":1.6950000000000001,"5min":1.829,"15min":2.8759999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4370000000000001,"15min":1.45},"min":{"1min":1.2170000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8280000000000001,"5min":2.1200000000000001,"15min":3.7309999999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.482,"15min":1.492},"min":{"1min":1.171,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.044,"5min":2.0699999999999998,"15min":2.5550000000000002},"last":1.484},{"interface":"front","average":{"1min":1.599,"5min":1.4710000000000001,"15min":1.4870000000000001},"min":{"1min":1.258,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.4049999999999998},"last":1.4630000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306516,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065324,"statfs":{"total":96636764160,"available":96322891776,"internally_reserved":0,"allocated":1060864,"data_stored":516326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.877,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":1.238,"5min":1.464,"15min":1.871},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.98899999999999999,"15min":0.996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.50700000000000001},"max":{"1min":1.393,"5min":1.7070000000000001,"15min":2.468},"last":0.77400000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0169999999999999,"15min":1.0209999999999999},"min":{"1min":0.71699999999999997,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":1.2589999999999999,"5min":1.714,"15min":1.7829999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.66300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.5660000000000001,"5min":2.0230000000000001,"15min":3.0619999999999998},"last":1.2669999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.236,"15min":1.2350000000000001},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.554},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.0820000000000001},"min":{"1min":0.81200000000000006,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6439999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2869999999999999,"15min":1.3109999999999999},"min":{"1min":0.93000000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.724,"5min":1.7370000000000001,"15min":2.6440000000000001},"last":1.341},{"interface":"front","average":{"1min":1.405,"5min":1.401,"15min":1.403},"min":{"1min":1.0009999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.677,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.6699999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.262,"15min":1.296},"min":{"1min":1.028,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.538,"5min":1.6719999999999999,"15min":2.1320000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.3720000000000001,"15min":1.391},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.76,"5min":1.76,"15min":2.79},"last":1.6339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.333,"15min":1.3640000000000001},"min":{"1min":1.0129999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7070000000000001,"5min":1.78,"15min":2.4620000000000002},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4319999999999999,"15min":1.456},"min":{"1min":1.073,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.8999999999999999},"last":1.4730000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.474,"15min":1.488},"min":{"1min":1.0740000000000001,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":2.0470000000000002,"15min":3.782},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":1.046,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":2.5390000000000001},"last":1.3129999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307064,"kb_used_data":1584,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064776,"statfs":{"total":96636764160,"available":96322330624,"internally_reserved":0,"allocated":1622016,"data_stored":1103411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.97299999999999998},"min":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.53600000000000003},"max":{"1min":1.4810000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.3859999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.80000000000000004,"5min":0.69699999999999995,"15min":0.55200000000000005},"max":{"1min":1.5189999999999999,"5min":1.6339999999999999,"15min":1.8979999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.2410000000000001,"15min":1.2210000000000001},"min":{"1min":0.82199999999999995,"5min":0.70799999999999996,"15min":0.54300000000000004},"max":{"1min":1.728,"5min":1.982,"15min":1.982},"last":1.327}]},{"osd":3,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6359999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.091,"5min":1.1679999999999999,"15min":1.1819999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.1140000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3169999999999999,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.70199999999999996},"max":{"1min":1.754,"5min":2.153,"15min":2.298},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.302,"15min":1.298},"min":{"1min":1.026,"5min":0.83999999999999997,"15min":0.80200000000000005},"max":{"1min":1.579,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.2829999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3600000000000001,"15min":1.375},"min":{"1min":0.89300000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9410000000000001,"5min":1.974,"15min":2.4630000000000001},"last":1.24},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":2.2189999999999999,"15min":2.6160000000000001},"last":1.2090000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.343,"15min":1.3460000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.88300000000000001},"max":{"1min":1.7130000000000001,"5min":2.0150000000000001,"15min":2.339},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4299999999999999,"15min":1.452},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.0590000000000002,"5min":2.113,"15min":2.5680000000000001},"last":1.361}]},{"osd":7,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.488,"15min":1.4710000000000001},"min":{"1min":1.083,"5min":1.0660000000000001,"15min":0.91500000000000004},"max":{"1min":2.0019999999999998,"5min":2.2629999999999999,"15min":2.5129999999999999},"last":1.083},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":1.04,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.802,"5min":1.931,"15min":2.4500000000000002},"last":1.4570000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:54:27.865 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:54:27.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:54:28.037 INFO:tasks.ceph.mds.a.smithi050.stderr:2023-12-19T14:54:28.035+0000 7f4919f5b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:54:28.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.075+0000 7f75a700c700 1 Processor -- start 2023-12-19T14:54:28.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.075+0000 7f75a700c700 1 -- start start 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a700c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75a0158c90 0x7f75a00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a700c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a700c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a006c850 con 0x7f75a01595a0 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a700c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a006b470 con 0x7f75a0158c90 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a700c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a006b750 con 0x7f75a006aeb0 2023-12-19T14:54:28.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a600a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75a0158c90 0x7f75a00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a5809700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a600a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75a0158c90 0x7f75a00caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35300/0 (socket says 172.21.15.50:35300) 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.079+0000 7f75a600a700 1 -- 172.21.15.50:0/1588242314 learned_addr learned my addr 172.21.15.50:0/1588242314 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a600a700 1 -- 172.21.15.50:0/1588242314 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 msgr2=0x7f75a0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a600a700 1 --2- 172.21.15.50:0/1588242314 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a600a700 1 -- 172.21.15.50:0/1588242314 >> v1:172.21.15.125:6789/0 conn(0x7f75a01595a0 legacy=0x7f75a015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a5008700 1 -- 172.21.15.50:0/1588242314 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940330384 0 0) 0x7f75a006c850 con 0x7f75a01595a0 2023-12-19T14:54:28.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a600a700 1 -- 172.21.15.50:0/1588242314 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a006ba30 con 0x7f75a0158c90 2023-12-19T14:54:28.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a600a700 1 --2- 172.21.15.50:0/1588242314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a00caf40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099b0 tx=0x7f759c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce678ce0a71dc5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a5008700 1 -- 172.21.15.50:0/1588242314 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c009ea0 con 0x7f75a0158c90 2023-12-19T14:54:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a5008700 1 -- 172.21.15.50:0/1588242314 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c00b860 con 0x7f75a0158c90 2023-12-19T14:54:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a5008700 1 -- 172.21.15.50:0/1588242314 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00bc00 con 0x7f75a0158c90 2023-12-19T14:54:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 -- 172.21.15.50:0/1588242314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 msgr2=0x7f75a00caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 --2- 172.21.15.50:0/1588242314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a00caf40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099b0 tx=0x7f759c009250 comp rx=0 tx=0).stop 2023-12-19T14:54:28.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 -- 172.21.15.50:0/1588242314 shutdown_connections 2023-12-19T14:54:28.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 --2- 172.21.15.50:0/1588242314 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 --2- 172.21.15.50:0/1588242314 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a00caf40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 -- 172.21.15.50:0/1588242314 >> 172.21.15.50:0/1588242314 conn(0x7f75a0067bb0 msgr2=0x7f75a00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:28.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 -- 172.21.15.50:0/1588242314 shutdown_connections 2023-12-19T14:54:28.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.083+0000 7f75a700c700 1 -- 172.21.15.50:0/1588242314 wait complete. 2023-12-19T14:54:28.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.087+0000 7f75a700c700 1 Processor -- start 2023-12-19T14:54:28.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.087+0000 7f75a700c700 1 -- start start 2023-12-19T14:54:28.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a01595a0 0x7f75a00c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00c5dd0 con 0x7f75a0158c90 2023-12-19T14:54:28.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0157640 con 0x7f75a01595a0 2023-12-19T14:54:28.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0169e70 con 0x7f75a006aeb0 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35314/0 (socket says 172.21.15.50:35314) 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 -- 172.21.15.50:0/4042733509 learned_addr learned my addr 172.21.15.50:0/4042733509 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a680b700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a01595a0 0x7f75a00c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a600a700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 -- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 msgr2=0x7f75a0169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0169930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 -- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a01595a0 msgr2=0x7f75a00c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a01595a0 0x7f75a00c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 -- 172.21.15.50:0/4042733509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0094f0 con 0x7f75a0158c90 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a5809700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f759000ec80 tx=0x7f75900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000c850 con 0x7f75a0158c90 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759000c9b0 con 0x7f75a0158c90 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a00c51c0 con 0x7f75a0158c90 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000cd50 con 0x7f75a0158c90 2023-12-19T14:54:28.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.091+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a00c5780 con 0x7f75a0158c90 2023-12-19T14:54:28.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.095+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a00c5f40 con 0x7f75a0158c90 2023-12-19T14:54:28.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.095+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7590013680 con 0x7f75a0158c90 2023-12-19T14:54:28.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.099+0000 7f75967fc700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 0x7f7584062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.099+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f759007cd60 con 0x7f75a0158c90 2023-12-19T14:54:28.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.099+0000 7f75a600a700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 0x7f7584062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.099+0000 7f75a600a700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 0x7f7584062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099b0 tx=0x7f759c017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.107+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f759004d040 con 0x7f75a0158c90 2023-12-19T14:54:28.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.331+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f75a004be40 con 0x7f75a0158c90 2023-12-19T14:54:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.343+0000 7f75967fc700 1 -- 172.21.15.50:0/4042733509 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7590050660 con 0x7f75a0158c90 2023-12-19T14:54:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 msgr2=0x7f7584062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 0x7f7584062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099b0 tx=0x7f759c017000 comp rx=0 tx=0).stop 2023-12-19T14:54:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 msgr2=0x7f75a0153420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f759000ec80 tx=0x7f75900091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 shutdown_connections 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75a01595a0 0x7f75a00c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7584060670 0x7f7584062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75a0158c90 0x7f75a0153420 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 --2- 172.21.15.50:0/4042733509 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75a006aeb0 0x7f75a0169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 >> 172.21.15.50:0/4042733509 conn(0x7f75a0067bb0 msgr2=0x7f75a00cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 shutdown_connections 2023-12-19T14:54:28.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.347+0000 7f75a700c700 1 -- 172.21.15.50:0/4042733509 wait complete. 2023-12-19T14:54:28.350 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:54:28.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.539+0000 7fc6bfa6e700 1 Processor -- start 2023-12-19T14:54:28.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.539+0000 7fc6bfa6e700 1 -- start start 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bfa6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae140 0x7fc6b80a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bfa6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6b80c1d60 0x7fc6b80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bfa6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b806dc20 con 0x7fc6b80ae510 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bfa6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b80c2200 con 0x7fc6b80c1d60 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bfa6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b806d940 con 0x7fc6b80ae140 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bd009700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc6b80ae510 0x7fc6b806d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53910/0 (socket says 172.21.15.50:53910) 2023-12-19T14:54:28.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bd009700 1 -- 172.21.15.50:0/2266284240 learned_addr learned my addr 172.21.15.50:0/2266284240 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bd80a700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae140 0x7fc6b80a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 --2- 172.21.15.50:0/2266284240 >> v2:172.21.15.125:3300/0 conn(0x7fc6b80c1d60 0x7fc6b80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bc808700 1 -- 172.21.15.50:0/2266284240 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246538387 0 0) 0x7fc6b806dc20 con 0x7fc6b80ae510 2023-12-19T14:54:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6bc808700 1 -- 172.21.15.50:0/2266284240 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a4003570 con 0x7fc6b80ae510 2023-12-19T14:54:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 -- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae140 msgr2=0x7fc6b80a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae140 0x7fc6b80a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 -- 172.21.15.50:0/2266284240 >> v1:172.21.15.50:6789/0 conn(0x7fc6b80ae510 legacy=0x7fc6b806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 -- 172.21.15.50:0/2266284240 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b80c6e20 con 0x7fc6b80c1d60 2023-12-19T14:54:28.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.543+0000 7fc6be00b700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80680f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009e60 tx=0x7fc6b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20b0a3cbd6e12027 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bc808700 1 -- 172.21.15.50:0/2266284240 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000e070 con 0x7fc6b80c1d60 2023-12-19T14:54:28.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bc808700 1 -- 172.21.15.50:0/2266284240 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b000bab0 con 0x7fc6b80c1d60 2023-12-19T14:54:28.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bc808700 1 -- 172.21.15.50:0/2266284240 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000be70 con 0x7fc6b80c1d60 2023-12-19T14:54:28.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 msgr2=0x7fc6b80680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80680f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009e60 tx=0x7fc6b0009780 comp rx=0 tx=0).stop 2023-12-19T14:54:28.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/2266284240 shutdown_connections 2023-12-19T14:54:28.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80680f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/2266284240 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae140 0x7fc6b80a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/2266284240 >> 172.21.15.50:0/2266284240 conn(0x7fc6b80c8850 msgr2=0x7fc6b80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:28.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/2266284240 shutdown_connections 2023-12-19T14:54:28.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.547+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/2266284240 wait complete. 2023-12-19T14:54:28.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 Processor -- start 2023-12-19T14:54:28.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 -- start start 2023-12-19T14:54:28.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 0x7fc6b80b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80aa800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8169490 con 0x7fc6b80ae140 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8168eb0 con 0x7fc6b80c1d60 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80b1930 con 0x7fc6b80ae510 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35330/0 (socket says 172.21.15.50:35330) 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 -- 172.21.15.50:0/3617653788 learned_addr learned my addr 172.21.15.50:0/3617653788 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6be00b700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80aa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd009700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 0x7fc6b80b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 -- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 msgr2=0x7fc6b80b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 0x7fc6b80b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 -- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 msgr2=0x7fc6b80aa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80aa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 -- 172.21.15.50:0/3617653788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b0009a20 con 0x7fc6b80ae140 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd009700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 0x7fc6b80b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bd80a700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400ec80 tx=0x7fc6b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b400c9d0 con 0x7fc6b80ae140 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b400cb30 con 0x7fc6b80ae140 2023-12-19T14:54:28.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b80ab170 con 0x7fc6b80ae140 2023-12-19T14:54:28.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.551+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4013470 con 0x7fc6b80ae140 2023-12-19T14:54:28.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.555+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b80ab450 con 0x7fc6b80ae140 2023-12-19T14:54:28.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.555+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8071120 con 0x7fc6b80ae140 2023-12-19T14:54:28.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.555+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc6b4013820 con 0x7fc6b80ae140 2023-12-19T14:54:28.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.559+0000 7fc6adffb700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 0x7fc69c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:28.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.559+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc6b407cde0 con 0x7fc6b80ae140 2023-12-19T14:54:28.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.559+0000 7fc6bd009700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 0x7fc69c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:28.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.559+0000 7fc6bd009700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 0x7fc69c062b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8007fd0 tx=0x7fc6a8009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:28.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.567+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc6b404d0c0 con 0x7fc6b80ae140 2023-12-19T14:54:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.771+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b80642a0 con 0x7fc6b80ae140 2023-12-19T14:54:28.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6adffb700 1 -- 172.21.15.50:0/3617653788 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fc6b40506e0 con 0x7fc6b80ae140 2023-12-19T14:54:28.776 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:28.776 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 msgr2=0x7fc69c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 0x7fc69c062b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8007fd0 tx=0x7fc6a8009040 comp rx=0 tx=0).stop 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 msgr2=0x7fc6b8169920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400ec80 tx=0x7fc6b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 shutdown_connections 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6b80c1d60 0x7fc6b80aa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc69c0606a0 0x7fc69c062b50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc6b80ae510 0x7fc6b80b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 --2- 172.21.15.50:0/3617653788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b80ae140 0x7fc6b8169920 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.775+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 >> 172.21.15.50:0/3617653788 conn(0x7fc6b80c8850 msgr2=0x7fc6b80a8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.779+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 shutdown_connections 2023-12-19T14:54:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:28.779+0000 7fc6bfa6e700 1 -- 172.21.15.50:0/3617653788 wait complete. 2023-12-19T14:54:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:29.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:54:22.525941+0000', 'modified': '2023-12-19T14:54:26.116986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6465}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6465': {'gid': 6465, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/3558598512', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 3558598512}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 3558598512}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:54:29.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:54:29.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T14:54:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.371+0000 7f5a2d145700 1 Processor -- start 2023-12-19T14:54:29.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.371+0000 7f5a2d145700 1 -- start start 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a28158e50 0x7f5a2806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a28159220 0x7f5a2806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2806e160 con 0x7f5a280a98d0 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2806d0c0 con 0x7f5a28158e50 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2806d3a0 con 0x7f5a28159220 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a28158e50 0x7f5a2806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2759e700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5a280a98d0 0x7f5a280a8850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53922/0 (socket says 172.21.15.50:53922) 2023-12-19T14:54:29.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2759e700 1 -- 172.21.15.50:0/3484225415 learned_addr learned my addr 172.21.15.50:0/3484225415 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2659c700 1 --2- 172.21.15.50:0/3484225415 >> v2:172.21.15.125:3300/0 conn(0x7f5a28159220 0x7f5a2806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3484225415 >> v2:172.21.15.125:3300/0 conn(0x7f5a28159220 msgr2=0x7f5a2806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/3484225415 >> v2:172.21.15.125:3300/0 conn(0x7f5a28159220 0x7f5a2806a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3484225415 >> v1:172.21.15.50:6789/0 conn(0x7f5a280a98d0 legacy=0x7f5a280a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3484225415 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2806d680 con 0x7f5a28158e50 2023-12-19T14:54:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/3484225415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a2806c7c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5a280cb810 tx=0x7f5a180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ae28a82d2966e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a25d9b700 1 -- 172.21.15.50:0/3484225415 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800b6b0 con 0x7f5a28158e50 2023-12-19T14:54:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a25d9b700 1 -- 172.21.15.50:0/3484225415 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1800b810 con 0x7f5a28158e50 2023-12-19T14:54:29.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a25d9b700 1 -- 172.21.15.50:0/3484225415 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800bbb0 con 0x7f5a28158e50 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 -- 172.21.15.50:0/3484225415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 msgr2=0x7f5a2806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.375+0000 7f5a2d145700 1 --2- 172.21.15.50:0/3484225415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a2806c7c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5a280cb810 tx=0x7f5a180091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 -- 172.21.15.50:0/3484225415 shutdown_connections 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 --2- 172.21.15.50:0/3484225415 >> v2:172.21.15.125:3300/0 conn(0x7f5a28159220 0x7f5a2806a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 --2- 172.21.15.50:0/3484225415 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a2806c7c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 -- 172.21.15.50:0/3484225415 >> 172.21.15.50:0/3484225415 conn(0x7f5a28067b60 msgr2=0x7f5a280ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 -- 172.21.15.50:0/3484225415 shutdown_connections 2023-12-19T14:54:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 -- 172.21.15.50:0/3484225415 wait complete. 2023-12-19T14:54:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 Processor -- start 2023-12-19T14:54:29.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 -- start start 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.379+0000 7f5a2d145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a280a98d0 0x7f5a2815ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a28159220 0x7f5a280abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28156c80 con 0x7f5a28158e50 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28169b60 con 0x7f5a28159220 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28169e40 con 0x7f5a280a98d0 2023-12-19T14:54:29.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35348/0 (socket says 172.21.15.50:35348) 2023-12-19T14:54:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 -- 172.21.15.50:0/2125020580 learned_addr learned my addr 172.21.15.50:0/2125020580 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a280a98d0 0x7f5a2815ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2759e700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a28159220 0x7f5a280abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 -- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a280a98d0 msgr2=0x7f5a2815ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a280a98d0 0x7f5a2815ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 -- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a28159220 msgr2=0x7f5a280abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a28159220 0x7f5a280abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 -- 172.21.15.50:0/2125020580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009420 con 0x7f5a28158e50 2023-12-19T14:54:29.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2659c700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d860 tx=0x7f5a1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a0effd700 1 -- 172.21.15.50:0/2125020580 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c0125f0 con 0x7f5a28158e50 2023-12-19T14:54:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a0effd700 1 -- 172.21.15.50:0/2125020580 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c012750 con 0x7f5a28158e50 2023-12-19T14:54:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a0effd700 1 -- 172.21.15.50:0/2125020580 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c012af0 con 0x7f5a28158e50 2023-12-19T14:54:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2816a120 con 0x7f5a28158e50 2023-12-19T14:54:29.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.383+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a281552f0 con 0x7f5a28158e50 2023-12-19T14:54:29.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.387+0000 7f5a0effd700 1 -- 172.21.15.50:0/2125020580 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5a1c01e460 con 0x7f5a28158e50 2023-12-19T14:54:29.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.387+0000 7f5a0effd700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 0x7f5a14062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.387+0000 7f5a0effd700 1 -- 172.21.15.50:0/2125020580 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5a1c07d110 con 0x7f5a28158e50 2023-12-19T14:54:29.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.387+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 0x7f5a14062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 0x7f5a14062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a180093f0 tx=0x7f5a18017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 Processor -- start 2023-12-19T14:54:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 -- start start 2023-12-19T14:54:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x156dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x156e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1492f30 con 0x12e7ae0 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x148eed0 con 0x12e7710 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.391+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x14550b0 con 0x1332790 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35352/0 (socket says 172.21.15.50:35352) 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3775794362 learned_addr learned my addr 172.21.15.50:0/3775794362 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2659c700 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x156dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2759e700 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x156e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 msgr2=0x156e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x156e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 msgr2=0x156dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x156dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 -- 172.21.15.50:0/3775794362 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x13339e0 con 0x12e7ae0 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5a280aab90 tx=0x7f5a18021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a091e11e90606036 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a0cff9700 1 -- 172.21.15.50:0/3775794362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18021070 con 0x12e7ae0 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a0cff9700 1 -- 172.21.15.50:0/3775794362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1802ac30 con 0x12e7ae0 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a0cff9700 1 -- 172.21.15.50:0/3775794362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802adc0 con 0x12e7ae0 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- 172.21.15.50:0/3775794362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 msgr2=0x156d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5a280aab90 tx=0x7f5a18021000 comp rx=0 tx=0).stop 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- 172.21.15.50:0/3775794362 shutdown_connections 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x156dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x156e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 --2- 172.21.15.50:0/3775794362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7ae0 0x156d750 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- 172.21.15.50:0/3775794362 >> 172.21.15.50:0/3775794362 conn(0x12e7eb0 msgr2=0x1521930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- 172.21.15.50:0/3775794362 shutdown_connections 2023-12-19T14:54:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- 172.21.15.50:0/3775794362 wait complete. 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.395+0000 7f5a2f424740 1 -- start start 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 Processor -- start 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x1549160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x152be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1454d50 con 0x1332790 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x142d970 con 0x12e7710 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1495290 con 0x12e7ae0 2023-12-19T14:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 10 client.0 _ll_register_callbacks cb 0x14d86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35368/0 (socket says 172.21.15.50:35368) 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 -- 172.21.15.50:0/230038652 learned_addr learned my addr 172.21.15.50:0/230038652 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2659c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x152be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x1549160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 -- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 msgr2=0x152be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x152be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 -- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 msgr2=0x1549160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x1549160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 -- 172.21.15.50:0/230038652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2806d680 con 0x1332790 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2759e700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007380 tx=0x7f5a20009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fab896a8eea3b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a057fa700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20013570 con 0x1332790 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x142dcc0 con 0x1332790 2023-12-19T14:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.399+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5a200136d0 con 0x1332790 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a20013aa0 con 0x1332790 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 10 client.6618 fetch_fsmap learned FSMap version 33 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x145d370 con 0x1332790 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x142dcc0 con 0x1332790 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5a20013c00 con 0x1332790 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 10 client.6618 fetch_fsmap finished waiting for FSMap version 33 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 10 client.6618 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 20 client.6618 populate_metadata read hostname 'smithi050' 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x152d130 0x157cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 4 client.6618 mds_command: new command op to 6465 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 --> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x14313e0 con 0x152d130 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x152d130 0x157cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x152d130 0x157cae0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba6c676d5c7b532 server_cookie=b8b0de06e4c08da9 in_seq=0 out_seq=0 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 10 client.6618 ms_handle_connect on v2:172.21.15.50:6834/3558598512 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 1 -- 172.21.15.50:0/230038652 <== mds.0 v2:172.21.15.50:6834/3558598512 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x14313e0 con 0x152d130 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a057fa700 10 client.6618 handle_command_reply: tid=0 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 client.6618 shutdown 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 msgr2=0x152c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.403+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5a20007380 tx=0x7f5a20009040 comp rx=0 tx=0).stop 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 shutdown_connections 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x152d130 0x157cae0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1332790 0x152c370 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x152be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 --2- 172.21.15.50:0/230038652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7710 0x1549160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a057fa700 0 client.6618 ms_handle_reset on v2:172.21.15.50:6834/3558598512 2023-12-19T14:54:29.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 >> 172.21.15.50:0/230038652 conn(0x12e7eb0 msgr2=0x1519f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 shutdown_connections 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 1 -- 172.21.15.50:0/230038652 wait complete. 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.407+0000 7f5a2f424740 20 client.6618 trim_cache size 0 max 16384 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 Processor -- start 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 -- start start 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7ae0 0x152b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x15206c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1464550 con 0x12e7710 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1458400 con 0x12e7ae0 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1462040 con 0x1332790 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a26d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a26d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35380/0 (socket says 172.21.15.50:35380) 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a26d9d700 1 -- 172.21.15.50:0/2854337008 learned_addr learned my addr 172.21.15.50:0/2854337008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2759e700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x15206c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.431+0000 7f5a2659c700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7ae0 0x152b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 -- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 msgr2=0x15206c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x15206c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 -- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7ae0 msgr2=0x152b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7ae0 0x152b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 -- 172.21.15.50:0/2854337008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1443d80 con 0x12e7710 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2759e700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x15206c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5a18034040 tx=0x7f5a1802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8650e787b9df98fc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a0d7fa700 1 -- 172.21.15.50:0/2854337008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18021040 con 0x12e7710 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a0d7fa700 1 -- 172.21.15.50:0/2854337008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1801a630 con 0x12e7710 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a0d7fa700 1 -- 172.21.15.50:0/2854337008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1803d0f0 con 0x12e7710 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- 172.21.15.50:0/2854337008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 msgr2=0x152ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5a18034040 tx=0x7f5a1802c040 comp rx=0 tx=0).stop 2023-12-19T14:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- 172.21.15.50:0/2854337008 shutdown_connections 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x12e7ae0 0x152b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1332790 0x15206c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 --2- 172.21.15.50:0/2854337008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x12e7710 0x152ae60 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- 172.21.15.50:0/2854337008 >> 172.21.15.50:0/2854337008 conn(0x12e7eb0 msgr2=0x152bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- 172.21.15.50:0/2854337008 shutdown_connections 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- 172.21.15.50:0/2854337008 wait complete. 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 -- start start 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.435+0000 7f5a2f424740 1 Processor -- start 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x15c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1332790 0x15c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1463610 con 0x1568da0 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x148ab50 con 0x1332790 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1440550 con 0x12e7ae0 2023-12-19T14:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 10 client.0 _ll_register_callbacks cb 0x12b39d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35382/0 (socket says 172.21.15.50:35382) 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 -- 172.21.15.50:0/867515097 learned_addr learned my addr 172.21.15.50:0/867515097 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2659c700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1332790 0x15c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x15c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 -- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 msgr2=0x15c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x15c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 -- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1332790 msgr2=0x15c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1332790 0x15c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 -- 172.21.15.50:0/867515097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x13339e0 con 0x1568da0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2759e700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001370 tx=0x7f5a2001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed41cd2a410591a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a057fa700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a200070f0 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5a20013aa0 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a1802ac30 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a20007970 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 10 client.6624 fetch_fsmap learned FSMap version 33 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.439+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1438130 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1802ac30 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 33) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f5a20014540 con 0x1568da0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 10 client.6624 fetch_fsmap finished waiting for FSMap version 33 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 10 client.6624 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 20 client.6624 populate_metadata read hostname 'smithi050' 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x15cda90 0x15cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 4 client.6624 mds_command: new command op to 6465 tid=0[{"prefix": "status"}] 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 --> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1463410 con 0x15cda90 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x15cda90 0x15cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a26d9d700 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x15cda90 0x15cfe70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530c269a394593f2 server_cookie=4886481e2db983a9 in_seq=0 out_seq=0 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 10 client.6624 ms_handle_connect on v2:172.21.15.50:6834/3558598512 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 1 -- 172.21.15.50:0/867515097 <== mds.0 v2:172.21.15.50:6834/3558598512 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x1463410 con 0x15cda90 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 10 client.6624 handle_command_reply: tid=0 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 client.6624 shutdown 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 msgr2=0x1569170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001370 tx=0x7f5a2001f000 comp rx=0 tx=0).stop 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 shutdown_connections 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1332790 0x15c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:6834/3558598512,v1:172.21.15.50:6835/3558598512] conn(0x15cda90 0x15cfe70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568da0 0x1569170 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 --2- 172.21.15.50:0/867515097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x12e7ae0 0x15c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a057fa700 0 client.6624 ms_handle_reset on v2:172.21.15.50:6834/3558598512 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 >> 172.21.15.50:0/867515097 conn(0x12e7eb0 msgr2=0x151a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 shutdown_connections 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 1 -- 172.21.15.50:0/867515097 wait complete. 2023-12-19T14:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.443+0000 7f5a2f424740 20 client.6624 trim_cache size 0 max 16384 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 6465, 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3601925719999999, 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 33, 2023-12-19T14:54:29.496 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 86, 2023-12-19T14:54:29.497 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 86, 2023-12-19T14:54:29.497 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.275173577 2023-12-19T14:54:29.497 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 msgr2=0x7f5a14062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 0x7f5a14062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a180093f0 tx=0x7f5a18017000 comp rx=0 tx=0).stop 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 msgr2=0x7f5a28149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00d860 tx=0x7f5a1c008040 comp rx=0 tx=0).stop 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 shutdown_connections 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a28159220 0x7f5a280abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a14060670 0x7f5a14062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a28158e50 0x7f5a28149920 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 --2- 172.21.15.50:0/2125020580 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a280a98d0 0x7f5a2815ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 >> 172.21.15.50:0/2125020580 conn(0x7f5a28067b60 msgr2=0x7f5a281546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 shutdown_connections 2023-12-19T14:54:29.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.447+0000 7f5a2d145700 1 -- 172.21.15.50:0/2125020580 wait complete. 2023-12-19T14:54:29.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:54:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.659+0000 7ff78acf0700 1 Processor -- start 2023-12-19T14:54:29.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.659+0000 7ff78acf0700 1 -- start start 2023-12-19T14:54:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff78acf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff784140ae0 0x7ff7840680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff78acf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78406e150 con 0x7ff784068630 2023-12-19T14:54:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff78acf0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff784068a70 con 0x7ff784140710 2023-12-19T14:54:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff78acf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78406d540 con 0x7ff784140ae0 2023-12-19T14:54:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff784140ae0 0x7ff7840680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff784140ae0 0x7ff7840680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35386/0 (socket says 172.21.15.50:35386) 2023-12-19T14:54:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff78928d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff784068630 0x7ff7840ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53930/0 (socket says 172.21.15.50:53930) 2023-12-19T14:54:29.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 -- 172.21.15.50:0/3287109429 learned_addr learned my addr 172.21.15.50:0/3287109429 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 -- 172.21.15.50:0/3287109429 >> v1:172.21.15.125:6789/0 conn(0x7ff784140710 legacy=0x7ff7840c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 -- 172.21.15.50:0/3287109429 >> v1:172.21.15.50:6789/0 conn(0x7ff784068630 legacy=0x7ff7840ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 -- 172.21.15.50:0/3287109429 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78406d820 con 0x7ff784140ae0 2023-12-19T14:54:29.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77bfff700 1 --2- 172.21.15.50:0/3287109429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff7840680f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff7740099b0 tx=0x7ff774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b95202b489b286 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3287109429 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400b860 con 0x7ff784140ae0 2023-12-19T14:54:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3287109429 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77400b9c0 con 0x7ff784140ae0 2023-12-19T14:54:29.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.663+0000 7ff77b7fe700 1 -- 172.21.15.50:0/3287109429 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400bd60 con 0x7ff784140ae0 2023-12-19T14:54:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- 172.21.15.50:0/3287109429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 msgr2=0x7ff7840680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3287109429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff7840680f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff7740099b0 tx=0x7ff774009250 comp rx=0 tx=0).stop 2023-12-19T14:54:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- 172.21.15.50:0/3287109429 shutdown_connections 2023-12-19T14:54:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3287109429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff7840680f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- 172.21.15.50:0/3287109429 >> 172.21.15.50:0/3287109429 conn(0x7ff7840c8850 msgr2=0x7ff7840c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- 172.21.15.50:0/3287109429 shutdown_connections 2023-12-19T14:54:29.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- 172.21.15.50:0/3287109429 wait complete. 2023-12-19T14:54:29.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 Processor -- start 2023-12-19T14:54:29.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.667+0000 7ff78acf0700 1 -- start start 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 0x7ff784157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff784140710 0x7ff7840c3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78415a580 con 0x7ff784140ae0 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78415a860 con 0x7ff784068630 2023-12-19T14:54:29.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7840afb70 con 0x7ff784140710 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35396/0 (socket says 172.21.15.50:35396) 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 -- 172.21.15.50:0/3367929631 learned_addr learned my addr 172.21.15.50:0/3367929631 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff77bfff700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff784140710 0x7ff7840c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff788a8c700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 0x7ff784157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 -- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff784140710 msgr2=0x7ff7840c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff784140710 0x7ff7840c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 -- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 msgr2=0x7ff784157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 0x7ff784157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 -- 172.21.15.50:0/3367929631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7740094f0 con 0x7ff784140ae0 2023-12-19T14:54:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff788a8c700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 0x7ff784157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78928d700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00c590 tx=0x7ff77c007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c005b40 con 0x7ff784140ae0 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c005cc0 con 0x7ff784140ae0 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7840afe50 con 0x7ff784140ae0 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c007320 con 0x7ff784140ae0 2023-12-19T14:54:29.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.671+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78415a130 con 0x7ff784140ae0 2023-12-19T14:54:29.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.675+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78415a320 con 0x7ff784140ae0 2023-12-19T14:54:29.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.675+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff77c01b3b0 con 0x7ff784140ae0 2023-12-19T14:54:29.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.679+0000 7ff778ff9700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 0x7ff770062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:29.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.679+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff77c07c070 con 0x7ff784140ae0 2023-12-19T14:54:29.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.679+0000 7ff788a8c700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 0x7ff770062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:29.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.679+0000 7ff788a8c700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 0x7ff770062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff780007fd0 tx=0x7ff780009040 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:29.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.683+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff77c04c350 con 0x7ff784140ae0 2023-12-19T14:54:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.883+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff784071120 con 0x7ff784140ae0 2023-12-19T14:54:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff778ff9700 1 -- 172.21.15.50:0/3367929631 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff77c017070 con 0x7ff784140ae0 2023-12-19T14:54:29.887 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 msgr2=0x7ff770062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 0x7ff770062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff780007fd0 tx=0x7ff780009040 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 msgr2=0x7ff784138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00c590 tx=0x7ff77c007ef0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 shutdown_connections 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff770060670 0x7ff770062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff784140ae0 0x7ff784138350 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff784140710 0x7ff7840c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 --2- 172.21.15.50:0/3367929631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff784068630 0x7ff784157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:29.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.887+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 >> 172.21.15.50:0/3367929631 conn(0x7ff7840c8850 msgr2=0x7ff7840ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.891+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 shutdown_connections 2023-12-19T14:54:29.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:29.891+0000 7ff78acf0700 1 -- 172.21.15.50:0/3367929631 wait complete. 2023-12-19T14:54:29.909 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:54:29.909 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:54:29.909 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:54:29.910 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:54:29.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:29.910 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:54:29.929 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41480sec preferred_lft 41480sec 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:54:29.930 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:54:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:54:29.932 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:54:29.932 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:54:29.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:29.933 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:54:30.017 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:54:30.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:30.022 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:54:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:54:30.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:54:30.091 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:54:30.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:30.233 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:54:30.248 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:54:30.248 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:54:30.249 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:54:30.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:30.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:54:30.329 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:54:30.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:54:30.329 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:30.388 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:54:30.388 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:54:30.388 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:30.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:54:30.532 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:54:30.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:54:30.615 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:54:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:54:30.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:30.618 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:54:30.676 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:54:30.676 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:54:30.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:54:30.758 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:54:30.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:54:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:54:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:54:30.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:30.836 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:54:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:54:30.867+0000 7f96ecfcb380 -1 init, newargv = 0x55efca7ab060 newargc=15 2023-12-19T14:54:30.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[47802]: starting ceph client 2023-12-19T14:54:30.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[47802]: starting fuse 2023-12-19T14:54:30.883 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:54:30.884 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:54:30.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:54:30.884 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:54:30.884 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:54:30.884 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:54:30.884 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:54:30.885 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:54:30.886 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:54:31.057 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.47802.asok 2023-12-19T14:54:31.241 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47802.asok 2023-12-19T14:54:31.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:31.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47802.asok status 2023-12-19T14:54:31.431 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:54:31.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "47802", 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 6639, 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:54:31.433 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 6639 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2215890147 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:54:31.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2215890147 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.6639 192.168.0.1:0/2215890147", 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/2215890147", 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 33, 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 86, 2023-12-19T14:54:31.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:54:31.436 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:54:31.436 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:54:31.436 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:54:31.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:31.447 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:54:31.447 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:31.467 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:54:31.467 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:31.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:31.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:31.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:54:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 Processor -- start 2023-12-19T14:54:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 -- start start 2023-12-19T14:54:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4f806ae50 0x7ff4f8163bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f806e160 con 0x7ff4f8159220 2023-12-19T14:54:31.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f806d0c0 con 0x7ff4f8158e50 2023-12-19T14:54:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.763+0000 7ff4fd2ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f806ced0 con 0x7ff4f806ae50 2023-12-19T14:54:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f67fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff4f8159220 0x7ff4f806a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53934/0 (socket says 172.21.15.50:53934) 2023-12-19T14:54:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f67fc700 1 -- 172.21.15.50:0/3401979121 learned_addr learned my addr 172.21.15.50:0/3401979121 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f77fe700 1 --2- 172.21.15.50:0/3401979121 >> v2:172.21.15.125:3300/0 conn(0x7ff4f806ae50 0x7ff4f8163bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f5ffb700 1 -- 172.21.15.50:0/3401979121 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572440819 0 0) 0x7ff4f806e160 con 0x7ff4f8159220 2023-12-19T14:54:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f5ffb700 1 -- 172.21.15.50:0/3401979121 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e0003570 con 0x7ff4f8159220 2023-12-19T14:54:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f77fe700 1 -- 172.21.15.50:0/3401979121 >> v1:172.21.15.125:6789/0 conn(0x7ff4f8158e50 legacy=0x7ff4f806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f77fe700 1 -- 172.21.15.50:0/3401979121 >> v1:172.21.15.50:6789/0 conn(0x7ff4f8159220 legacy=0x7ff4f806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f77fe700 1 -- 172.21.15.50:0/3401979121 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f806d3a0 con 0x7ff4f806ae50 2023-12-19T14:54:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f77fe700 1 --2- 172.21.15.50:0/3401979121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f8163bc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009e90 tx=0x7ff4f0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b903959a50ab47a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f5ffb700 1 -- 172.21.15.50:0/3401979121 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f000e070 con 0x7ff4f806ae50 2023-12-19T14:54:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f5ffb700 1 -- 172.21.15.50:0/3401979121 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f000bc40 con 0x7ff4f806ae50 2023-12-19T14:54:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4f5ffb700 1 -- 172.21.15.50:0/3401979121 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0012430 con 0x7ff4f806ae50 2023-12-19T14:54:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/3401979121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 msgr2=0x7ff4f8163bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/3401979121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f8163bc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009e90 tx=0x7ff4f0009780 comp rx=0 tx=0).stop 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/3401979121 shutdown_connections 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/3401979121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f8163bc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/3401979121 >> 172.21.15.50:0/3401979121 conn(0x7ff4f8067b60 msgr2=0x7ff4f80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/3401979121 shutdown_connections 2023-12-19T14:54:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.767+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/3401979121 wait complete. 2023-12-19T14:54:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 Processor -- start 2023-12-19T14:54:31.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 -- start start 2023-12-19T14:54:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:31.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8159220 0x7ff4f80ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80ad3e0 con 0x7ff4f8158e50 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80ca690 con 0x7ff4f806ae50 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4fd2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80ca910 con 0x7ff4f8159220 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4f67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4f67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35410/0 (socket says 172.21.15.50:35410) 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.771+0000 7ff4f67fc700 1 -- 172.21.15.50:0/1625037532 learned_addr learned my addr 172.21.15.50:0/1625037532 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f77fe700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8159220 0x7ff4f80ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f6ffd700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 -- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8159220 msgr2=0x7ff4f80ab9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8159220 0x7ff4f80ab9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 -- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 msgr2=0x7ff4f80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 -- 172.21.15.50:0/1625037532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0009a20 con 0x7ff4f8158e50 2023-12-19T14:54:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4f67fc700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00c590 tx=0x7ff4ec007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec005b30 con 0x7ff4f8158e50 2023-12-19T14:54:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ec005c90 con 0x7ff4f8158e50 2023-12-19T14:54:31.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec007300 con 0x7ff4f8158e50 2023-12-19T14:54:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f815bd40 con 0x7ff4f8158e50 2023-12-19T14:54:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f815c300 con 0x7ff4f8158e50 2023-12-19T14:54:31.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.775+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f80caa80 con 0x7ff4f8158e50 2023-12-19T14:54:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.779+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff4ec01b3b0 con 0x7ff4f8158e50 2023-12-19T14:54:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.779+0000 7ff4deffd700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 0x7ff4e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.779+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff4ec07bf20 con 0x7ff4f8158e50 2023-12-19T14:54:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.779+0000 7ff4f6ffd700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 0x7ff4e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.783+0000 7ff4f6ffd700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 0x7ff4e4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8003a80 tx=0x7ff4e8007ec0 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.787+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff4ec04c200 con 0x7ff4f8158e50 2023-12-19T14:54:31.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.979+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4f8071120 con 0x7ff4e4060670 2023-12-19T14:54:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.983+0000 7ff4deffd700 1 -- 172.21.15.50:0/1625037532 <== mgr.5720 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff4f8071120 con 0x7ff4e4060670 2023-12-19T14:54:31.983 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:54:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 msgr2=0x7ff4e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 0x7ff4e4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8003a80 tx=0x7ff4e8007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 msgr2=0x7ff4f80ab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00c590 tx=0x7ff4ec007dd0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 shutdown_connections 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4e4060670 0x7ff4e4062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8159220 0x7ff4f80ab9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8158e50 0x7ff4f80ab460 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 --2- 172.21.15.50:0/1625037532 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f806ae50 0x7ff4f80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 >> 172.21.15.50:0/1625037532 conn(0x7ff4f8067b60 msgr2=0x7ff4f80b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 shutdown_connections 2023-12-19T14:54:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:31.987+0000 7ff4fd2ae700 1 -- 172.21.15.50:0/1625037532 wait complete. 2023-12-19T14:54:32.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:54:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 Processor -- start 2023-12-19T14:54:32.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 -- start start 2023-12-19T14:54:32.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880acea0 0x7f688806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f68880cbbe0 0x7f68880c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 0x7f68880ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68880ccd20 con 0x7f68880cbbe0 2023-12-19T14:54:32.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68880c8dc0 con 0x7f68880c8990 2023-12-19T14:54:32.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.179+0000 7f688759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688806cec0 con 0x7f68880acea0 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f68880cbbe0 0x7f68880c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f6886d9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 0x7f68880ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688659c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880acea0 0x7f688806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f68880cbbe0 0x7f68880c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35422/0 (socket says 172.21.15.50:35422) 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 -- 172.21.15.50:0/2732788387 learned_addr learned my addr 172.21.15.50:0/2732788387 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 -- 172.21.15.50:0/2732788387 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880acea0 msgr2=0x7f688806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 --2- 172.21.15.50:0/2732788387 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880acea0 0x7f688806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 -- 172.21.15.50:0/2732788387 >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 msgr2=0x7f68880ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 --2- 172.21.15.50:0/2732788387 >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 0x7f68880ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 -- 172.21.15.50:0/2732788387 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688806d1a0 con 0x7f68880cbbe0 2023-12-19T14:54:32.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f6886d9d700 1 --2- 172.21.15.50:0/2732788387 >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 0x7f68880ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f687ffff700 1 --2- 172.21.15.50:0/2732788387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880cbbe0 0x7f68880c8450 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f688806a6f0 tx=0x7f6870009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68259e5f5aec1788 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f6885d9b700 1 -- 172.21.15.50:0/2732788387 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687000b640 con 0x7f68880cbbe0 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f6885d9b700 1 -- 172.21.15.50:0/2732788387 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687000b7a0 con 0x7f68880cbbe0 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f6885d9b700 1 -- 172.21.15.50:0/2732788387 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687000bb60 con 0x7f68880cbbe0 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 -- 172.21.15.50:0/2732788387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880cbbe0 msgr2=0x7f68880c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 --2- 172.21.15.50:0/2732788387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880cbbe0 0x7f68880c8450 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f688806a6f0 tx=0x7f6870009250 comp rx=0 tx=0).stop 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 -- 172.21.15.50:0/2732788387 shutdown_connections 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 --2- 172.21.15.50:0/2732788387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880cbbe0 0x7f68880c8450 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 --2- 172.21.15.50:0/2732788387 >> v2:172.21.15.125:3300/0 conn(0x7f68880c8990 0x7f68880ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 --2- 172.21.15.50:0/2732788387 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880acea0 0x7f688806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 -- 172.21.15.50:0/2732788387 >> 172.21.15.50:0/2732788387 conn(0x7f68880cdc00 msgr2=0x7f68880ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:32.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 -- 172.21.15.50:0/2732788387 shutdown_connections 2023-12-19T14:54:32.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.183+0000 7f688759e700 1 -- 172.21.15.50:0/2732788387 wait complete. 2023-12-19T14:54:32.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 Processor -- start 2023-12-19T14:54:32.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 -- start start 2023-12-19T14:54:32.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880c8990 0x7f68880b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 0x7f6888157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888158090 con 0x7f68880acea0 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881537c0 con 0x7f68880cbbe0 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68880b2840 con 0x7f68880c8990 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35428/0 (socket says 172.21.15.50:35428) 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 -- 172.21.15.50:0/2445488910 learned_addr learned my addr 172.21.15.50:0/2445488910 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f687ffff700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880c8990 0x7f68880b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f6886d9d700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 0x7f6888157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 -- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880c8990 msgr2=0x7f68880b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880c8990 0x7f68880b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 -- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 msgr2=0x7f6888157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 0x7f6888157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f688659c700 1 -- 172.21.15.50:0/2445488910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6880009a20 con 0x7f68880acea0 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.187+0000 7f6886d9d700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 0x7f6888157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f688659c700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f687800ecb0 tx=0x7f68780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800c8c0 con 0x7f68880acea0 2023-12-19T14:54:32.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687800ca40 con 0x7f68880acea0 2023-12-19T14:54:32.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700094f0 con 0x7f68880acea0 2023-12-19T14:54:32.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800ce00 con 0x7f68880acea0 2023-12-19T14:54:32.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6888157db0 con 0x7f68880acea0 2023-12-19T14:54:32.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688804be40 con 0x7f68880acea0 2023-12-19T14:54:32.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.191+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6878013680 con 0x7f68880acea0 2023-12-19T14:54:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.195+0000 7f687e7fc700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 0x7f6868062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.195+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f687807cdb0 con 0x7f68880acea0 2023-12-19T14:54:32.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.195+0000 7f687ffff700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 0x7f6868062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:32.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.195+0000 7f687ffff700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 0x7f6868062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f688806a6f0 tx=0x7f6870017000 comp rx=0 tx=0).ready entity=mgr.5720 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:32.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.199+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6878081070 con 0x7f68880acea0 2023-12-19T14:54:32.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.411+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f688806ea80 con 0x7f68880acea0 2023-12-19T14:54:32.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.959+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f687804d460 con 0x7f68880acea0 2023-12-19T14:54:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.959+0000 7f687e7fc700 1 -- 172.21.15.50:0/2445488910 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v27) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6878050b20 con 0x7f68880acea0 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 msgr2=0x7f6868062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 0x7f6868062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f688806a6f0 tx=0x7f6870017000 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 msgr2=0x7f6888153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f687800ecb0 tx=0x7f68780091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 shutdown_connections 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68880cbbe0 0x7f6888157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f68680606a0 0x7f6868062b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68880c8990 0x7f68880b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 --2- 172.21.15.50:0/2445488910 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68880acea0 0x7f6888153280 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:32.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 >> 172.21.15.50:0/2445488910 conn(0x7f68880cdc00 msgr2=0x7f6888155000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:32.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 shutdown_connections 2023-12-19T14:54:32.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:32.963+0000 7f688759e700 1 -- 172.21.15.50:0/2445488910 wait complete. 2023-12-19T14:54:32.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:54:33.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 Processor -- start 2023-12-19T14:54:33.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 -- start start 2023-12-19T14:54:33.152 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:33.146+0000 7f9c2435a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:33.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 0x7ff0bc06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0bc0cecd0 0x7ff0bc06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0bc0cbdf0 con 0x7ff0bc158e20 2023-12-19T14:54:33.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0bc0cc0d0 con 0x7ff0bc0cecd0 2023-12-19T14:54:33.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c266e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0cc3b0 con 0x7ff0bc0cb970 2023-12-19T14:54:33.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0bbfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff0bc158e20 0x7ff0bc0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53938/0 (socket says 172.21.15.50:53938) 2023-12-19T14:54:33.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0bbfff700 1 -- 172.21.15.50:0/596449025 learned_addr learned my addr 172.21.15.50:0/596449025 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:33.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/596449025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 0x7ff0bc06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.151+0000 7ff0c0c0b700 1 --2- 172.21.15.50:0/596449025 >> v2:172.21.15.125:3300/0 conn(0x7ff0bc0cecd0 0x7ff0bc06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.155 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:33.151+0000 7fa8327ca000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0baffd700 1 -- 172.21.15.50:0/596449025 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137293199 0 0) 0x7ff0bc0cbdf0 con 0x7ff0bc158e20 2023-12-19T14:54:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0baffd700 1 -- 172.21.15.50:0/596449025 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a4003570 con 0x7ff0bc158e20 2023-12-19T14:54:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/596449025 >> v2:172.21.15.125:3300/0 conn(0x7ff0bc0cecd0 msgr2=0x7ff0bc06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/596449025 >> v2:172.21.15.125:3300/0 conn(0x7ff0bc0cecd0 0x7ff0bc06b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/596449025 >> v1:172.21.15.50:6789/0 conn(0x7ff0bc158e20 legacy=0x7ff0bc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/596449025 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc0cc690 con 0x7ff0bc0cb970 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/596449025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 0x7ff0bc06acc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00098e0 tx=0x7ff0b00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3af1556c8879f69f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0baffd700 1 -- 172.21.15.50:0/596449025 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0011630 con 0x7ff0bc0cb970 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0baffd700 1 -- 172.21.15.50:0/596449025 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0011790 con 0x7ff0bc0cb970 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0baffd700 1 -- 172.21.15.50:0/596449025 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0011b30 con 0x7ff0bc0cb970 2023-12-19T14:54:33.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 -- 172.21.15.50:0/596449025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 msgr2=0x7ff0bc06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 --2- 172.21.15.50:0/596449025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 0x7ff0bc06acc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00098e0 tx=0x7ff0b00091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 -- 172.21.15.50:0/596449025 shutdown_connections 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 --2- 172.21.15.50:0/596449025 >> v2:172.21.15.125:3300/0 conn(0x7ff0bc0cecd0 0x7ff0bc06b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 --2- 172.21.15.50:0/596449025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0cb970 0x7ff0bc06acc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 -- 172.21.15.50:0/596449025 >> 172.21.15.50:0/596449025 conn(0x7ff0bc067bb0 msgr2=0x7ff0bc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 -- 172.21.15.50:0/596449025 shutdown_connections 2023-12-19T14:54:33.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.155+0000 7ff0c266e700 1 -- 172.21.15.50:0/596449025 wait complete. 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 Processor -- start 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 -- start start 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0bc0cecd0 0x7ff0bc16c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0c9330 0x7ff0bc0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc16c610 con 0x7ff0bc158e20 2023-12-19T14:54:33.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc14d3a0 con 0x7ff0bc0cecd0 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c266e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0c7b30 con 0x7ff0bc0c9330 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35456/0 (socket says 172.21.15.50:35456) 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/374535109 learned_addr learned my addr 172.21.15.50:0/374535109 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0c0c0b700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0c9330 0x7ff0bc0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bbfff700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0bc0cecd0 0x7ff0bc16c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0c9330 msgr2=0x7ff0bc0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0c9330 0x7ff0bc0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0bc0cecd0 msgr2=0x7ff0bc16c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0bc0cecd0 0x7ff0bc16c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 -- 172.21.15.50:0/374535109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0009420 con 0x7ff0bc158e20 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.159+0000 7ff0bb7fe700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0cc4f0 tx=0x7ff0b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0011900 con 0x7ff0bc158e20 2023-12-19T14:54:33.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0011a60 con 0x7ff0bc158e20 2023-12-19T14:54:33.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0023620 con 0x7ff0bc158e20 2023-12-19T14:54:33.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc0c7e10 con 0x7ff0bc158e20 2023-12-19T14:54:33.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc14d5f0 con 0x7ff0bc158e20 2023-12-19T14:54:33.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc14d7e0 con 0x7ff0bc158e20 2023-12-19T14:54:33.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.163+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7ff0b002c3b0 con 0x7ff0bc158e20 2023-12-19T14:54:33.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.167+0000 7ff0a3fff700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 0x7ff0a8062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.167+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff0b007ddb0 con 0x7ff0bc158e20 2023-12-19T14:54:33.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.167+0000 7ff0bbfff700 1 -- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 msgr2=0x7ff0a8062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:33.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.167+0000 7ff0bbfff700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 0x7ff0a8062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:33.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.175+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff0b004e1b0 con 0x7ff0bc158e20 2023-12-19T14:54:33.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.351+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff0bc071120 con 0x7ff0bc158e20 2023-12-19T14:54:33.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.359+0000 7ff0a3fff700 1 -- 172.21.15.50:0/374535109 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff0b00517d0 con 0x7ff0bc158e20 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 msgr2=0x7ff0a8062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 0x7ff0a8062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 msgr2=0x7ff0bc169210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0cc4f0 tx=0x7ff0b0017000 comp rx=0 tx=0).stop 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 shutdown_connections 2023-12-19T14:54:33.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0a80606f0 0x7ff0a8062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff0bc0c9330 0x7ff0bc0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0bc158e20 0x7ff0bc169210 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 --2- 172.21.15.50:0/374535109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0bc0cecd0 0x7ff0bc16c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 >> 172.21.15.50:0/374535109 conn(0x7ff0bc067bb0 msgr2=0x7ff0bc14ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 shutdown_connections 2023-12-19T14:54:33.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.363+0000 7ff0c266e700 1 -- 172.21.15.50:0/374535109 wait complete. 2023-12-19T14:54:33.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:54:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.551+0000 7f0d7d02e700 1 Processor -- start 2023-12-19T14:54:33.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.551+0000 7f0d7d02e700 1 -- start start 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7d02e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0d780cb970 0x7f0d7806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7d02e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0d780cecd0 0x7f0d7806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7d02e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780cbdf0 con 0x7f0d78158e20 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7d02e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d780cc0d0 con 0x7f0d780cb970 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7d02e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d780cc3b0 con 0x7f0d780cecd0 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0d780cb970 0x7f0d7806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0d780cb970 0x7f0d7806acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35472/0 (socket says 172.21.15.50:35472) 2023-12-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 -- 172.21.15.50:0/33560303 learned_addr learned my addr 172.21.15.50:0/33560303 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:33.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7759e700 1 --2- 172.21.15.50:0/33560303 >> v2:172.21.15.125:3300/0 conn(0x7f0d780cecd0 0x7f0d7806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 -- 172.21.15.50:0/33560303 >> v2:172.21.15.125:3300/0 conn(0x7f0d780cecd0 msgr2=0x7f0d7806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 --2- 172.21.15.50:0/33560303 >> v2:172.21.15.125:3300/0 conn(0x7f0d780cecd0 0x7f0d7806b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 -- 172.21.15.50:0/33560303 >> v1:172.21.15.125:6789/0 conn(0x7f0d78158e20 legacy=0x7f0d780ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.555+0000 7f0d7659c700 1 -- 172.21.15.50:0/33560303 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d780cc690 con 0x7f0d780cb970 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7659c700 1 --2- 172.21.15.50:0/33560303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cb970 0x7f0d7806acc0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099b0 tx=0x7f0d6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885239a87b483b95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d75d9b700 1 -- 172.21.15.50:0/33560303 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c00b820 con 0x7f0d780cb970 2023-12-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d75d9b700 1 -- 172.21.15.50:0/33560303 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00b980 con 0x7f0d780cb970 2023-12-19T14:54:33.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d75d9b700 1 -- 172.21.15.50:0/33560303 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c00bd20 con 0x7f0d780cb970 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 -- 172.21.15.50:0/33560303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cb970 msgr2=0x7f0d7806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/33560303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cb970 0x7f0d7806acc0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099b0 tx=0x7f0d6c009250 comp rx=0 tx=0).stop 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 -- 172.21.15.50:0/33560303 shutdown_connections 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/33560303 >> v2:172.21.15.125:3300/0 conn(0x7f0d780cecd0 0x7f0d7806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/33560303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cb970 0x7f0d7806acc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 -- 172.21.15.50:0/33560303 >> 172.21.15.50:0/33560303 conn(0x7f0d78067bb0 msgr2=0x7f0d780d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 -- 172.21.15.50:0/33560303 shutdown_connections 2023-12-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.559+0000 7f0d7d02e700 1 -- 172.21.15.50:0/33560303 wait complete. 2023-12-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 Processor -- start 2023-12-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 -- start start 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0d78158e20 0x7f0d780c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d780b40d0 0x7f0d780b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780b4f20 con 0x7f0d780cecd0 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780b4910 con 0x7f0d780b40d0 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d7d02e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780c8230 con 0x7f0d78158e20 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d76d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d76d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35476/0 (socket says 172.21.15.50:35476) 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.563+0000 7f0d76d9d700 1 -- 172.21.15.50:0/3877652431 learned_addr learned my addr 172.21.15.50:0/3877652431 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d7759e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d780b40d0 0x7f0d780b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d7659c700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0d78158e20 0x7f0d780c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:33.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 -- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0d78158e20 msgr2=0x7f0d780c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0d78158e20 0x7f0d780c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 -- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d780b40d0 msgr2=0x7f0d780b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d780b40d0 0x7f0d780b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 -- 172.21.15.50:0/3877652431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0094f0 con 0x7f0d780cecd0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d76d9d700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c590 tx=0x7f0d6800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68011590 con 0x7f0d780cecd0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d680116f0 con 0x7f0d780cecd0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d780b4bb0 con 0x7f0d780cecd0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68011ab0 con 0x7f0d780cecd0 2023-12-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d780c8510 con 0x7f0d780cecd0 2023-12-19T14:54:33.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.567+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d78071120 con 0x7f0d780cecd0 2023-12-19T14:54:33.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.571+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f0d6801b3b0 con 0x7f0d780cecd0 2023-12-19T14:54:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.571+0000 7f0d5effd700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 0x7f0d64062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.571+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0d6807c110 con 0x7f0d780cecd0 2023-12-19T14:54:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.571+0000 7f0d7659c700 1 -- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 msgr2=0x7f0d64062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.571+0000 7f0d7659c700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 0x7f0d64062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:33.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.579+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d6804c3f0 con 0x7f0d780cecd0 2023-12-19T14:54:33.691 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:33.686+0000 7f9c2435a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:33.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.755+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0d781585b0 con 0x7f0d780cecd0 2023-12-19T14:54:33.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.759+0000 7f0d5effd700 1 -- 172.21.15.50:0/3877652431 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0d680170c0 con 0x7f0d780cecd0 2023-12-19T14:54:33.761 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:33.754+0000 7f9c2435a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 msgr2=0x7f0d64062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 0x7f0d64062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 msgr2=0x7f0d78156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800c590 tx=0x7f0d6800c950 comp rx=0 tx=0).stop 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 shutdown_connections 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d780b40d0 0x7f0d780b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0d640606c0 0x7f0d64062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0d78158e20 0x7f0d780c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 --2- 172.21.15.50:0/3877652431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d780cecd0 0x7f0d78156110 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 >> 172.21.15.50:0/3877652431 conn(0x7f0d78067bb0 msgr2=0x7f0d7814cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 shutdown_connections 2023-12-19T14:54:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:33.763+0000 7f0d7d02e700 1 -- 172.21.15.50:0/3877652431 wait complete. 2023-12-19T14:54:33.774 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:33.771+0000 7fa8327ca000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:33.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:33.786 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T14:54:33.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T14:54:33.854 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:33.851+0000 7fa8327ca000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:33.893 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:33.886+0000 7f9c2435a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:34.025 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.023+0000 7fa8327ca000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:34.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.023+0000 7fe48deae700 1 Processor -- start 2023-12-19T14:54:34.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.023+0000 7fe48deae700 1 -- start start 2023-12-19T14:54:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe48deae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe4880cb500 0x7fe48815c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe48deae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe48815c760 0x7fe4880b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe48deae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4880b0ac0 con 0x7fe4880cb130 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe48deae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48815cbb0 con 0x7fe4880cb500 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe48deae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48806b210 con 0x7fe48815c760 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe4877fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe4880cb130 0x7fe48806a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53952/0 (socket says 172.21.15.50:53952) 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe4877fe700 1 -- 172.21.15.50:0/2626674386 learned_addr learned my addr 172.21.15.50:0/2626674386 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:34.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe486ffd700 1 --2- 172.21.15.50:0/2626674386 >> v2:172.21.15.50:3300/0 conn(0x7fe4880cb500 0x7fe48815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe487fff700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe48815c760 0x7fe4880b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe4867fc700 1 -- 172.21.15.50:0/2626674386 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037432053 0 0) 0x7fe4880b0ac0 con 0x7fe4880cb130 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe4867fc700 1 -- 172.21.15.50:0/2626674386 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe470003570 con 0x7fe4880cb130 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe486ffd700 1 -- 172.21.15.50:0/2626674386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe48815c760 msgr2=0x7fe4880b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe486ffd700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe48815c760 0x7fe4880b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.027+0000 7fe486ffd700 1 -- 172.21.15.50:0/2626674386 >> v1:172.21.15.50:6789/0 conn(0x7fe4880cb130 legacy=0x7fe48806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:34.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe486ffd700 1 -- 172.21.15.50:0/2626674386 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880ca810 con 0x7fe4880cb500 2023-12-19T14:54:34.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe486ffd700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4880cb500 0x7fe48815c220 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0098e0 tx=0x7fe47c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a306cfb59d789c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe4867fc700 1 -- 172.21.15.50:0/2626674386 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011630 con 0x7fe4880cb500 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe4867fc700 1 -- 172.21.15.50:0/2626674386 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c011790 con 0x7fe4880cb500 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe4867fc700 1 -- 172.21.15.50:0/2626674386 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011b30 con 0x7fe4880cb500 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 -- 172.21.15.50:0/2626674386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4880cb500 msgr2=0x7fe48815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4880cb500 0x7fe48815c220 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0098e0 tx=0x7fe47c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 -- 172.21.15.50:0/2626674386 shutdown_connections 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe48815c760 0x7fe4880b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 --2- 172.21.15.50:0/2626674386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe4880cb500 0x7fe48815c220 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 -- 172.21.15.50:0/2626674386 >> 172.21.15.50:0/2626674386 conn(0x7fe488067bb0 msgr2=0x7fe4880cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 -- 172.21.15.50:0/2626674386 shutdown_connections 2023-12-19T14:54:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.031+0000 7fe48deae700 1 -- 172.21.15.50:0/2626674386 wait complete. 2023-12-19T14:54:34.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 Processor -- start 2023-12-19T14:54:34.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- start start 2023-12-19T14:54:34.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4880cb130 0x7fe4880c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4880cb500 0x7fe4880c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe48815c760 0x7fe4880c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880c6b50 con 0x7fe48815c760 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488169180 con 0x7fe4880cb500 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880c7970 con 0x7fe4880cb130 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe48815c760 0x7fe4880c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe48815c760 0x7fe4880c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35498/0 (socket says 172.21.15.50:35498) 2023-12-19T14:54:34.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 -- 172.21.15.50:0/1005355452 learned_addr learned my addr 172.21.15.50:0/1005355452 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:34.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe486ffd700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4880cb500 0x7fe4880c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:34.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4880cb130 0x7fe4880c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:34.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4880cb130 msgr2=0x7fe4880c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:34.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe4880cb130 0x7fe4880c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4880cb500 msgr2=0x7fe4880c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4880cb500 0x7fe4880c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c009420 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe487fff700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe48815c760 0x7fe4880c8880 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe480009df0 tx=0x7fe48000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000cb50 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48000f040 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4880c6d60 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480014480 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.035+0000 7fe48deae700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4880c7320 con 0x7fe48815c760 2023-12-19T14:54:34.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.039+0000 7fe48deae700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4880c7b00 con 0x7fe48815c760 2023-12-19T14:54:34.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.039+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe480014830 con 0x7fe48815c760 2023-12-19T14:54:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.039+0000 7fe46f7fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.043+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe48007d5c0 con 0x7fe48815c760 2023-12-19T14:54:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.043+0000 7fe4877fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:34.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.043+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:34.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.047+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe48004d790 con 0x7fe48815c760 2023-12-19T14:54:34.155 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.150+0000 7f9c2435a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:34.223 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.218+0000 7f9c2435a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:34.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.243+0000 7fe48deae700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe488071120 con 0x7fe4740606f0 2023-12-19T14:54:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.247+0000 7fe4877fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:34.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.247+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:34.313 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.311+0000 7fa8327ca000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:34.395 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.395+0000 7fa8327ca000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:34.430 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.422+0000 7f9c2435a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:34.507 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.502+0000 7f9c2435a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:34.590 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.582+0000 7f9c2435a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:34.636 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.635+0000 7fa8327ca000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.651+0000 7fe4877fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:34.651+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:54:34.724 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.723+0000 7fa8327ca000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:34.820 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:34.819+0000 7fa8327ca000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:34.956 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:34.950+0000 7f9c2435a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:35.115 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.110+0000 7f9c2435a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:35.240 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:35.239+0000 7fa8327ca000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:35.411 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:35.411+0000 7fa8327ca000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:35.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:35.455+0000 7fe4877fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:35.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:35.455+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:54:35.456 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.450+0000 7f9c2435a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:35.670 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.662+0000 7f9c2435a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:35.757 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.750+0000 7f9c2435a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:35.805 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:35.803+0000 7fa8327ca000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:35.829 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.822+0000 7f9c2435a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:35.905 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:35.898+0000 7f9c2435a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:36.040 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.039+0000 7fa8327ca000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:36.125 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:36.118+0000 7f9c2435a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:36.130 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.127+0000 7fa8327ca000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:36.202 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:36.194+0000 7f9c2435a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:36.209 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.207+0000 7fa8327ca000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:36.287 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:36.282+0000 7f9c2435a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:36.292 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.291+0000 7fa8327ca000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:36.471 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:36.466+0000 7f9c2435a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:36.509 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.507+0000 7fa8327ca000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:36.558 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:36.550+0000 7f9c2435a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:36.589 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.587+0000 7fa8327ca000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:36.681 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.679+0000 7fa8327ca000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:36.875 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.871+0000 7fa8327ca000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:36.966 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:36.963+0000 7fa8327ca000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:37.059+0000 7fe4877fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:37.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:37.059+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:54:37.252 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.246+0000 7f9c2435a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:37.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:37.271+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 28) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fe48004db60 con 0x7fe48815c760 2023-12-19T14:54:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:37.271+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 msgr2=0x7fe474062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:37.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:37.271+0000 7fe46f7fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:37.460 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be5f8a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.460 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be5f8a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.460 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be5f8a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.460 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be5f8a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be5f8a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be2f84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be2f84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be2f84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be2f84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:37.454+0000 7f9be2f84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:37.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:54:37.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:54:37.785 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:37.783+0000 7fa8327ca000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:38.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.279+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 29) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fe48001a2c0 con 0x7fe48815c760 2023-12-19T14:54:38.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.283+0000 7fe46f7fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.283+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe488071120 con 0x7fe4740606f0 2023-12-19T14:54:38.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.287+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.287+0000 7fe4877fe700 1 --2- 172.21.15.50:0/1005355452 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe4740606f0 0x7fe474062ba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe4780058d0 tx=0x7fe478009040 comp rx=0 tx=0).ready entity=mgr.6505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:38.300 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:38.294+0000 7f9becf98700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T14:54:38.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.299+0000 7fe46f7fe700 1 -- 172.21.15.50:0/1005355452 <== mgr.6505 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fe488071120 con 0x7fe4740606f0 2023-12-19T14:54:38.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T14:54:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T14:54:38.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:54:38.316 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T14:54:38.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:54:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.559+0000 7feb3d8e3700 1 Processor -- start 2023-12-19T14:54:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.559+0000 7feb3d8e3700 1 -- start start 2023-12-19T14:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d8e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 0x7feb3815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d8e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb380adab0 0x7feb380adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d8e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3815ae30 con 0x7feb3806baa0 2023-12-19T14:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d8e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb380abf70 con 0x7feb380adab0 2023-12-19T14:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d8e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb38154b50 con 0x7feb38154780 2023-12-19T14:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb380adab0 0x7feb380adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7feb380adab0 0x7feb380adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46526/0 (socket says 172.21.15.50:46526) 2023-12-19T14:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 -- 172.21.15.50:0/2694144870 learned_addr learned my addr 172.21.15.50:0/2694144870 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3c8e1700 1 --2- 172.21.15.50:0/2694144870 >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 0x7feb3815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb377fe700 1 -- 172.21.15.50:0/2694144870 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930277631 0 0) 0x7feb3815ae30 con 0x7feb3806baa0 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb377fe700 1 -- 172.21.15.50:0/2694144870 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20003570 con 0x7feb3806baa0 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 -- 172.21.15.50:0/2694144870 >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 msgr2=0x7feb3815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 --2- 172.21.15.50:0/2694144870 >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 0x7feb3815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 -- 172.21.15.50:0/2694144870 >> v1:172.21.15.125:6789/0 conn(0x7feb3806baa0 legacy=0x7feb38157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3d0e2700 1 -- 172.21.15.50:0/2694144870 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380ac210 con 0x7feb380adab0 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.563+0000 7feb3c8e1700 1 --2- 172.21.15.50:0/2694144870 >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 0x7feb3815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d0e2700 1 --2- 172.21.15.50:0/2694144870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380adeb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feb3806a3f0 tx=0x7feb30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2d7cf45618f469 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb377fe700 1 -- 172.21.15.50:0/2694144870 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3000e070 con 0x7feb380adab0 2023-12-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb377fe700 1 -- 172.21.15.50:0/2694144870 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3000ba20 con 0x7feb380adab0 2023-12-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb377fe700 1 -- 172.21.15.50:0/2694144870 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb300092e0 con 0x7feb380adab0 2023-12-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2694144870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 msgr2=0x7feb380adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2694144870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380adeb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feb3806a3f0 tx=0x7feb30009840 comp rx=0 tx=0).stop 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2694144870 shutdown_connections 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2694144870 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380adeb0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2694144870 >> v2:172.21.15.125:3300/0 conn(0x7feb38154780 0x7feb3815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2694144870 >> 172.21.15.50:0/2694144870 conn(0x7feb38067b50 msgr2=0x7feb380cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2694144870 shutdown_connections 2023-12-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.567+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2694144870 wait complete. 2023-12-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.571+0000 7feb3d8e3700 1 Processor -- start 2023-12-19T14:54:38.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.571+0000 7feb3d8e3700 1 -- start start 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb38154780 0x7feb3815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38158b30 con 0x7feb380adab0 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38158240 con 0x7feb38154780 2023-12-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb381584e0 con 0x7feb3806baa0 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb37fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55002/0 (socket says 172.21.15.50:55002) 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 -- 172.21.15.50:0/2630210850 learned_addr learned my addr 172.21.15.50:0/2630210850 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d0e2700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb38154780 0x7feb3815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 -- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb38154780 msgr2=0x7feb3815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb38154780 0x7feb3815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 -- 172.21.15.50:0/2630210850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 msgr2=0x7feb380c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 -- 172.21.15.50:0/2630210850 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30009ae0 con 0x7feb3806baa0 2023-12-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb37fff700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3c8e1700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e8d0 tx=0x7feb2c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c013410 con 0x7feb3806baa0 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c013570 con 0x7feb3806baa0 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb381587c0 con 0x7feb3806baa0 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c013910 con 0x7feb3806baa0 2023-12-19T14:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.575+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb381537e0 con 0x7feb3806baa0 2023-12-19T14:54:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.579+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 29) v1 ==== 79816+0+0 (secure 0 0 0) 0x7feb2c013d40 con 0x7feb3806baa0 2023-12-19T14:54:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.579+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3804be40 con 0x7feb3806baa0 2023-12-19T14:54:38.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.579+0000 7feb34ff9700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 0x7feb240628d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:38.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.579+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feb2c07c4b0 con 0x7feb3806baa0 2023-12-19T14:54:38.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.583+0000 7feb37fff700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 0x7feb240628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:38.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.583+0000 7feb37fff700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 0x7feb240628d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb38155ef0 tx=0x7feb28007d50 comp rx=0 tx=0).ready entity=mgr.6505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:38.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.591+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feb2c04c9d0 con 0x7feb3806baa0 2023-12-19T14:54:38.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:38.803+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7feb380cc1a0 con 0x7feb3806baa0 2023-12-19T14:54:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.331+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7feb2c04cda0 con 0x7feb3806baa0 2023-12-19T14:54:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.335+0000 7feb34ff9700 1 -- 172.21.15.50:0/2630210850 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v30) v1 ==== 93+0+0 (secure 0 0 0) 0x7feb2c050460 con 0x7feb3806baa0 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 msgr2=0x7feb240628d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 0x7feb240628d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb38155ef0 tx=0x7feb28007d50 comp rx=0 tx=0).stop 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 msgr2=0x7feb381532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e8d0 tx=0x7feb2c008040 comp rx=0 tx=0).stop 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 shutdown_connections 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb38154780 0x7feb3815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feb24060420 0x7feb240628d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb380adab0 0x7feb380c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 --2- 172.21.15.50:0/2630210850 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feb3806baa0 0x7feb381532a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 >> 172.21.15.50:0/2630210850 conn(0x7feb38067b50 msgr2=0x7feb38154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 shutdown_connections 2023-12-19T14:54:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.339+0000 7feb3d8e3700 1 -- 172.21.15.50:0/2630210850 wait complete. 2023-12-19T14:54:39.359 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:54:39.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:39.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:39.439 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:54:39.439 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:54:39.439 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:54:39.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:54:39.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[47802]: fuse finished with error 0 and tester_r 0 2023-12-19T14:54:39.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:54:39.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:39.478 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:54:39.524 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:39.518+0000 7fbb8415e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:39.533 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:39.531+0000 7f51243c3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:54:39.661 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:54:39.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:39.662 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:54:39.683 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:54:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:39.685 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:54:39.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:39.685 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:54:39.723 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:54:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:39.724 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:54:39.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:54:39.724 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:54:39.746 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:54:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:54:39.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:39.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 Processor -- start 2023-12-19T14:54:39.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 -- start start 2023-12-19T14:54:39.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6cc80cc610 0x7f6cc80c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:39.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80c6630 con 0x7f6cc81502b0 2023-12-19T14:54:39.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80c6910 con 0x7f6cc8067c90 2023-12-19T14:54:39.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6ccdc78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc80c6bf0 con 0x7f6cc80cc610 2023-12-19T14:54:39.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cbffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6cc81502b0 0x7f6cc80c5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38026/0 (socket says 172.21.15.50:38026) 2023-12-19T14:54:39.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cbffff700 1 -- 172.21.15.50:0/989083501 learned_addr learned my addr 172.21.15.50:0/989083501 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:39.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc7fff700 1 --2- 172.21.15.50:0/989083501 >> v2:172.21.15.50:3300/0 conn(0x7f6cc80cc610 0x7f6cc80c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:39.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849131462 0 0) 0x7f6cc80c6630 con 0x7f6cc81502b0 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb4003570 con 0x7f6cc81502b0 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666259067 0 0) 0x7f6cc80c6910 con 0x7f6cc8067c90 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc7fff700 1 -- 172.21.15.50:0/989083501 >> v1:172.21.15.125:6789/0 conn(0x7f6cc8067c90 legacy=0x7f6cc814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.939+0000 7f6cc7fff700 1 -- 172.21.15.50:0/989083501 >> v1:172.21.15.50:6789/0 conn(0x7f6cc81502b0 legacy=0x7f6cc80c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6cc7fff700 1 -- 172.21.15.50:0/989083501 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80c6ed0 con 0x7f6cc80cc610 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6cc7fff700 1 --2- 172.21.15.50:0/989083501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc80cc610 0x7f6cc80c6010 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009e60 tx=0x7f6cc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfeff55205e9978b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:39.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc000e070 con 0x7f6cc80cc610 2023-12-19T14:54:39.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc000b850 con 0x7f6cc80cc610 2023-12-19T14:54:39.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6cc6ffd700 1 -- 172.21.15.50:0/989083501 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0001470 con 0x7f6cc80cc610 2023-12-19T14:54:39.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 -- 172.21.15.50:0/989083501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc80cc610 msgr2=0x7f6cc80c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/989083501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc80cc610 0x7f6cc80c6010 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009e60 tx=0x7f6cc0009780 comp rx=0 tx=0).stop 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 -- 172.21.15.50:0/989083501 shutdown_connections 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/989083501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc80cc610 0x7f6cc80c6010 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 -- 172.21.15.50:0/989083501 >> 172.21.15.50:0/989083501 conn(0x7f6cc80cd6d0 msgr2=0x7f6cc806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 -- 172.21.15.50:0/989083501 shutdown_connections 2023-12-19T14:54:39.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.943+0000 7f6ccdc78700 1 -- 172.21.15.50:0/989083501 wait complete. 2023-12-19T14:54:39.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 Processor -- start 2023-12-19T14:54:39.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- start start 2023-12-19T14:54:39.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:39.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6cc80cc610 0x7f6cc80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:39.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 0x7f6cc80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:39.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8151030 con 0x7f6cc8067c90 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8150ce0 con 0x7f6cc81502b0 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc80b5ae0 con 0x7f6cc80cc610 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46562/0 (socket says 172.21.15.50:46562) 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 -- 172.21.15.50:0/915595990 learned_addr learned my addr 172.21.15.50:0/915595990 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc7fff700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 0x7f6cc80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cbffff700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6cc80cc610 0x7f6cc80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:39.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6cc80cc610 msgr2=0x7f6cc80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6cc80cc610 0x7f6cc80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 msgr2=0x7f6cc80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 0x7f6cc80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 -- 172.21.15.50:0/915595990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc0009a20 con 0x7f6cc8067c90 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc7fff700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 0x7f6cc80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cc77fe700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800c7e0 tx=0x7f6cb8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8012700 con 0x7f6cc8067c90 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc80b5dc0 con 0x7f6cc8067c90 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb8012880 con 0x7f6cc8067c90 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8012c40 con 0x7f6cc8067c90 2023-12-19T14:54:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.947+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc80b34f0 con 0x7f6cc8067c90 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc8071120 con 0x7f6cc8067c90 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f6cb801b400 con 0x7f6cc8067c90 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6cbf7fe700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 0x7f6ca8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6cb807c970 con 0x7f6cc8067c90 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6cbffff700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 msgr2=0x7f6ca8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.951+0000 7f6cbffff700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 0x7f6ca8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:39.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:39.959+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6cb804ca30 con 0x7f6cc8067c90 2023-12-19T14:54:40.057 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.050+0000 7fbb8415e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:40.109 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.107+0000 7f51243c3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:54:40.130 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.122+0000 7fbb8415e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.155+0000 7f6cbffff700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 msgr2=0x7f6ca8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:40.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.155+0000 7f6cbffff700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 0x7f6ca8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:40.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.179+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc80b5fb0 con 0x7f6cc8067c90 2023-12-19T14:54:40.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.179+0000 7f6cbf7fe700 1 -- 172.21.15.50:0/915595990 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f6cb8050050 con 0x7f6cc8067c90 2023-12-19T14:54:40.182 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:40.182 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 msgr2=0x7f6ca8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 0x7f6ca8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 msgr2=0x7f6cc8153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800c7e0 tx=0x7f6cb8007df0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 shutdown_connections 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cc81502b0 0x7f6cc80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ca8060700 0x7f6ca8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6cc80cc610 0x7f6cc80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 --2- 172.21.15.50:0/915595990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6cc8067c90 0x7f6cc8153090 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.183+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 >> 172.21.15.50:0/915595990 conn(0x7f6cc80cd6d0 msgr2=0x7f6cc806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.187+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 shutdown_connections 2023-12-19T14:54:40.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.187+0000 7f6ccdc78700 1 -- 172.21.15.50:0/915595990 wait complete. 2023-12-19T14:54:40.189 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:40.192 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.191+0000 7f51243c3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:54:40.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:40.266 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.258+0000 7fbb8415e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:40.349 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.346+0000 7f51243c3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:40.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.374+0000 7f1d23b45700 1 Processor -- start 2023-12-19T14:54:40.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.374+0000 7f1d23b45700 1 -- start start 2023-12-19T14:54:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c068c70 con 0x7f1d1c159340 2023-12-19T14:54:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1c068060 con 0x7f1d1c158f70 2023-12-19T14:54:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c068340 con 0x7f1d1c069100 2023-12-19T14:54:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d218e1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d218e1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:55558/0 (socket says 172.21.15.50:55558) 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55046/0 (socket says 172.21.15.50:55046) 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d218e1700 1 -- 172.21.15.50:0/3049173036 learned_addr learned my addr 172.21.15.50:0/3049173036 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d220e2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1d1c159340 0x7f1d1c0cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38032/0 (socket says 172.21.15.50:38032) 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 -- 172.21.15.50:0/3049173036 >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 msgr2=0x7f1d1c067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d208df700 1 -- 172.21.15.50:0/3049173036 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591728657 0 0) 0x7f1d1c068c70 con 0x7f1d1c159340 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 --2- 172.21.15.50:0/3049173036 >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 -- 172.21.15.50:0/3049173036 >> v1:172.21.15.50:6789/0 conn(0x7f1d1c159340 legacy=0x7f1d1c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 -- 172.21.15.50:0/3049173036 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c068620 con 0x7f1d1c069100 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d218e1700 1 --2- 172.21.15.50:0/3049173036 >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d210e0700 1 --2- 172.21.15.50:0/3049173036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0098e0 tx=0x7f1d0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfdbc5d5abc727f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d208df700 1 -- 172.21.15.50:0/3049173036 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00b640 con 0x7f1d1c069100 2023-12-19T14:54:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d208df700 1 -- 172.21.15.50:0/3049173036 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c00b7a0 con 0x7f1d1c069100 2023-12-19T14:54:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d208df700 1 -- 172.21.15.50:0/3049173036 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00bb40 con 0x7f1d1c069100 2023-12-19T14:54:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- 172.21.15.50:0/3049173036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 msgr2=0x7f1d1c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 --2- 172.21.15.50:0/3049173036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0098e0 tx=0x7f1d0c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- 172.21.15.50:0/3049173036 shutdown_connections 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 --2- 172.21.15.50:0/3049173036 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c069100 0x7f1d1c0cf7e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 --2- 172.21.15.50:0/3049173036 >> v2:172.21.15.125:3300/0 conn(0x7f1d1c158f70 0x7f1d1c067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.378+0000 7f1d23b45700 1 -- 172.21.15.50:0/3049173036 >> 172.21.15.50:0/3049173036 conn(0x7f1d1c0ccc40 msgr2=0x7f1d1c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- 172.21.15.50:0/3049173036 shutdown_connections 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- 172.21.15.50:0/3049173036 wait complete. 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 Processor -- start 2023-12-19T14:54:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- start start 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d1c158f70 0x7f1d1c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c159340 0x7f1d1c15e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0b3eb0 con 0x7f1d1c069100 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0b3580 con 0x7f1d1c158f70 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d23b45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0b3860 con 0x7f1d1c159340 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d218e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d218e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46570/0 (socket says 172.21.15.50:46570) 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d218e1700 1 -- 172.21.15.50:0/1683577813 learned_addr learned my addr 172.21.15.50:0/1683577813 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d220e2700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c159340 0x7f1d1c15e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.382+0000 7f1d210e0700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d1c158f70 0x7f1d1c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 -- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c159340 msgr2=0x7f1d1c15e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c159340 0x7f1d1c15e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 -- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d1c158f70 msgr2=0x7f1d1c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d1c158f70 0x7f1d1c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 -- 172.21.15.50:0/1683577813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180094f0 con 0x7f1d1c069100 2023-12-19T14:54:40.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d218e1700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800ec50 tx=0x7f1d18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:40.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800c850 con 0x7f1d1c069100 2023-12-19T14:54:40.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1800c9b0 con 0x7f1d1c069100 2023-12-19T14:54:40.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c009420 con 0x7f1d1c069100 2023-12-19T14:54:40.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800cd50 con 0x7f1d1c069100 2023-12-19T14:54:40.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c14e620 con 0x7f1d1c069100 2023-12-19T14:54:40.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.386+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c071120 con 0x7f1d1c069100 2023-12-19T14:54:40.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.390+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f1d18013680 con 0x7f1d1c069100 2023-12-19T14:54:40.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.390+0000 7f1d11ffb700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 0x7f1d00062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.390+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1d1807d0d0 con 0x7f1d1c069100 2023-12-19T14:54:40.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.390+0000 7f1d210e0700 1 -- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 msgr2=0x7f1d00062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:40.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.390+0000 7f1d210e0700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 0x7f1d00062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:40.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.398+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1d1804d0f0 con 0x7f1d1c069100 2023-12-19T14:54:40.521 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.514+0000 7fbb8415e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:40.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.566+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d1c068900 con 0x7f1d1c069100 2023-12-19T14:54:40.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.566+0000 7f1d11ffb700 1 -- 172.21.15.50:0/1683577813 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16098 (secure 0 0 0) 0x7f1d18050710 con 0x7f1d1c069100 2023-12-19T14:54:40.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:40.569 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":87,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:37.258411+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/552631771":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3782271172":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/4106781597":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2371597704":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2154078294":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2430752556":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6833/33827":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6832/33827":"2023-12-20T14:54:37.258339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 msgr2=0x7f1d00062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 0x7f1d00062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 msgr2=0x7f1d1c14e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800ec50 tx=0x7f1d18009250 comp rx=0 tx=0).stop 2023-12-19T14:54:40.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 shutdown_connections 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d1c158f70 0x7f1d1c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1d00060700 0x7f1d00062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d1c159340 0x7f1d1c15e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 --2- 172.21.15.50:0/1683577813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d1c069100 0x7f1d1c14e0e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 >> 172.21.15.50:0/1683577813 conn(0x7f1d1c0ccc40 msgr2=0x7f1d1c0b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 shutdown_connections 2023-12-19T14:54:40.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.570+0000 7f1d23b45700 1 -- 172.21.15.50:0/1683577813 wait complete. 2023-12-19T14:54:40.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:54:20.550186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:54:21.522239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:54:40.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:40.594 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.586+0000 7fbb8415e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:40.646 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.642+0000 7f51243c3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:54:40.734 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.730+0000 7f51243c3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:40.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 Processor -- start 2023-12-19T14:54:40.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 -- start start 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb47806da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780aff50 con 0x7fb47806b7a0 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780af380 con 0x7fb47806df70 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47fcf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780c95b0 con 0x7fb478155420 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47e28d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb47806b7a0 0x7fb4780b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38046/0 (socket says 172.21.15.50:38046) 2023-12-19T14:54:40.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.758+0000 7fb47e28d700 1 -- 172.21.15.50:0/479664564 learned_addr learned my addr 172.21.15.50:0/479664564 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:40.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47da8c700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb47806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473804543 0 0) 0x7fb4780aff50 con 0x7fb47806b7a0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb464004680 con 0x7fb47806b7a0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377927500 0 0) 0x7fb4780af380 con 0x7fb47806df70 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4780aff50 con 0x7fb47806df70 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 193033878 0 0) 0x7fb464004680 con 0x7fb47806b7a0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 msgr2=0x7fb47806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb47806da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 >> v1:172.21.15.125:6789/0 conn(0x7fb47806df70 legacy=0x7fb47806b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780af620 con 0x7fb47806b7a0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fb4700017f0 con 0x7fb47806b7a0 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 >> v1:172.21.15.50:6789/0 conn(0x7fb47806b7a0 legacy=0x7fb4780b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb464005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4640048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb464004e00 0x7fb464005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780c95b0 con 0x7fb464006340 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb464003490 con 0x7fb464004e00 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb464003650 con 0x7fb478155420 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4640048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47e28d700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb464004e00 0x7fb464005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47da8c700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb464005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 -- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 msgr2=0x7fb464005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478155420 0x7fb464005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 -- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb464004e00 msgr2=0x7fb464005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb464004e00 0x7fb464005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 -- 172.21.15.50:0/479664564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464004680 con 0x7fb464006340 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47d28b700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4640048c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb468001940 tx=0x7fb468007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac8c326f9e6c1fe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468001d80 con 0x7fb464006340 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46800c5d0 con 0x7fb464006340 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.762+0000 7fb47ca8a700 1 -- 172.21.15.50:0/479664564 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800c970 con 0x7fb464006340 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- 172.21.15.50:0/479664564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 msgr2=0x7fb4640048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/479664564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4640048c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb468001940 tx=0x7fb468007d30 comp rx=0 tx=0).stop 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47e28d700 1 -- 172.21.15.50:0/479664564 reap_dead start 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- 172.21.15.50:0/479664564 shutdown_connections 2023-12-19T14:54:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- 172.21.15.50:0/479664564 >> 172.21.15.50:0/479664564 conn(0x7fb4780cc580 msgr2=0x7fb4780cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:40.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- 172.21.15.50:0/479664564 shutdown_connections 2023-12-19T14:54:40.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- 172.21.15.50:0/479664564 wait complete. 2023-12-19T14:54:40.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 Processor -- start 2023-12-19T14:54:40.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.766+0000 7fb47fcf0700 1 -- start start 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4780c1860 0x7fb4781684f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478168a30 0x7fb478168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780c1cd0 con 0x7fb464006340 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478157210 con 0x7fb4780c1860 2023-12-19T14:54:40.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb478156600 con 0x7fb478168a30 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46598/0 (socket says 172.21.15.50:46598) 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 -- 172.21.15.50:0/295328147 learned_addr learned my addr 172.21.15.50:0/295328147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47d28b700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4780c1860 0x7fb4781684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47e28d700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478168a30 0x7fb478168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 -- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478168a30 msgr2=0x7fb478168ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478168a30 0x7fb478168ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 -- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4780c1860 msgr2=0x7fb4781684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4780c1860 0x7fb4781684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 -- 172.21.15.50:0/295328147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4781568e0 con 0x7fb464006340 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47da8c700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb474011990 tx=0x7fb474008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474015670 con 0x7fb464006340 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4740157d0 con 0x7fb464006340 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb478156bc0 con 0x7fb464006340 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474015b90 con 0x7fb464006340 2023-12-19T14:54:40.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.770+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4780caca0 con 0x7fb464006340 2023-12-19T14:54:40.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.774+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb478157040 con 0x7fb464006340 2023-12-19T14:54:40.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.774+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fb474013410 con 0x7fb464006340 2023-12-19T14:54:40.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.778+0000 7fb46dffb700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 0x7fb45c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:40.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.778+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb47407d3b0 con 0x7fb464006340 2023-12-19T14:54:40.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.778+0000 7fb47d28b700 1 -- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 msgr2=0x7fb45c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:40.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.778+0000 7fb47d28b700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 0x7fb45c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.782+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb47404d470 con 0x7fb464006340 2023-12-19T14:54:40.805 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.798+0000 7fbb8415e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:40.884 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.878+0000 7fbb8415e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.954+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb478071120 con 0x7fb464006340 2023-12-19T14:54:40.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.958+0000 7fb46dffb700 1 -- 172.21.15.50:0/295328147 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb474050a90 con 0x7fb464006340 2023-12-19T14:54:40.959 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 msgr2=0x7fb45c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 0x7fb45c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 msgr2=0x7fb4780c1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb474011990 tx=0x7fb474008040 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 shutdown_connections 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4780c1860 0x7fb4781684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb45c060700 0x7fb45c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb478168a30 0x7fb478168ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 --2- 172.21.15.50:0/295328147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb464006340 0x7fb4780c1320 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 >> 172.21.15.50:0/295328147 conn(0x7fb4780cc580 msgr2=0x7fb478154310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 shutdown_connections 2023-12-19T14:54:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:40.962+0000 7fb47fcf0700 1 -- 172.21.15.50:0/295328147 wait complete. 2023-12-19T14:54:40.966 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:40.962+0000 7f51243c3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:54:40.967 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:40.962+0000 7fbb8415e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:40.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:54:41.051 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:41.050+0000 7f51243c3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:54:41.149 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:41.146+0000 7f51243c3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:54:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.154+0000 7f25c900c700 1 Processor -- start 2023-12-19T14:54:41.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.154+0000 7f25c900c700 1 -- start start 2023-12-19T14:54:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c900c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f25c406aeb0 0x7f25c4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c900c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c41595a0 0x7f25c4159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c900c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c406c850 con 0x7f25c4158c90 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c900c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c40cb650 con 0x7f25c406aeb0 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c900c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406b470 con 0x7f25c41595a0 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f25c406aeb0 0x7f25c4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f25c406aeb0 0x7f25c4159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46610/0 (socket says 172.21.15.50:46610) 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 -- 172.21.15.50:0/2954753818 learned_addr learned my addr 172.21.15.50:0/2954753818 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c880b700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c41595a0 0x7f25c4159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 -- 172.21.15.50:0/2954753818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c41595a0 msgr2=0x7f25c4159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c41595a0 0x7f25c4159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 -- 172.21.15.50:0/2954753818 >> v1:172.21.15.125:6789/0 conn(0x7f25c4158c90 legacy=0x7f25c40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 -- 172.21.15.50:0/2954753818 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c406b750 con 0x7f25c406aeb0 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c37fe700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c406aeb0 0x7f25c4159060 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f25b80099b0 tx=0x7f25b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bae10bf330fa18 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c2ffd700 1 -- 172.21.15.50:0/2954753818 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8009ea0 con 0x7f25c406aeb0 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c2ffd700 1 -- 172.21.15.50:0/2954753818 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b800b860 con 0x7f25c406aeb0 2023-12-19T14:54:41.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.158+0000 7f25c2ffd700 1 -- 172.21.15.50:0/2954753818 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b800bc00 con 0x7f25c406aeb0 2023-12-19T14:54:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- 172.21.15.50:0/2954753818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c406aeb0 msgr2=0x7f25c4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c406aeb0 0x7f25c4159060 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f25b80099b0 tx=0x7f25b8009250 comp rx=0 tx=0).stop 2023-12-19T14:54:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- 172.21.15.50:0/2954753818 shutdown_connections 2023-12-19T14:54:41.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c41595a0 0x7f25c4159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- 172.21.15.50:0/2954753818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c406aeb0 0x7f25c4159060 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- 172.21.15.50:0/2954753818 >> 172.21.15.50:0/2954753818 conn(0x7f25c4067bb0 msgr2=0x7f25c40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- 172.21.15.50:0/2954753818 shutdown_connections 2023-12-19T14:54:41.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- 172.21.15.50:0/2954753818 wait complete. 2023-12-19T14:54:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 Processor -- start 2023-12-19T14:54:41.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- start start 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c406aeb0 0x7f25c41488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 0x7f25c41575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 0x7f25c40c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c4148e10 con 0x7f25c4158c90 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40c5a50 con 0x7f25c41595a0 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.162+0000 7f25c900c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40c4e80 con 0x7f25c406aeb0 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 0x7f25c41575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 0x7f25c41575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46618/0 (socket says 172.21.15.50:46618) 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 0x7f25c40c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c37fe700 1 -- 172.21.15.50:0/538197482 learned_addr learned my addr 172.21.15.50:0/538197482 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c3fff700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c406aeb0 0x7f25c41488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 -- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c406aeb0 msgr2=0x7f25c41488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c406aeb0 0x7f25c41488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 -- 172.21.15.50:0/538197482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 msgr2=0x7f25c41575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 0x7f25c41575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 -- 172.21.15.50:0/538197482 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80094f0 con 0x7f25c41595a0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c880b700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 0x7f25c40c66b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00ecc0 tx=0x7f25bc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00c8c0 con 0x7f25c41595a0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00f040 con 0x7f25c41595a0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00cc90 con 0x7f25c41595a0 2023-12-19T14:54:41.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c40c5120 con 0x7f25c41595a0 2023-12-19T14:54:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c40c56e0 con 0x7f25c41595a0 2023-12-19T14:54:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.166+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c404be40 con 0x7f25c41595a0 2023-12-19T14:54:41.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.170+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f25bc014690 con 0x7f25c41595a0 2023-12-19T14:54:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.170+0000 7f25abfff700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 0x7f25b0062c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.170+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f25bc07d900 con 0x7f25c41595a0 2023-12-19T14:54:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.170+0000 7f25c3fff700 1 -- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 msgr2=0x7f25b0062c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:41.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.170+0000 7f25c3fff700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 0x7f25b0062c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:41.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.178+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f25bc04d890 con 0x7f25c41595a0 2023-12-19T14:54:41.340 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:41.334+0000 7fbb8415e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:41.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.354+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25c40c5b90 con 0x7f25c41595a0 2023-12-19T14:54:41.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.354+0000 7f25abfff700 1 -- 172.21.15.50:0/538197482 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f25bc018020 con 0x7f25c41595a0 2023-12-19T14:54:41.356 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 msgr2=0x7f25b0062c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 0x7f25b0062c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 msgr2=0x7f25c40c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 0x7f25c40c66b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00ecc0 tx=0x7f25bc00b040 comp rx=0 tx=0).stop 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 shutdown_connections 2023-12-19T14:54:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25c41595a0 0x7f25c40c66b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f25b00607d0 0x7f25b0062c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f25c4158c90 0x7f25c41575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 --2- 172.21.15.50:0/538197482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25c406aeb0 0x7f25c41488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 >> 172.21.15.50:0/538197482 conn(0x7f25c4067bb0 msgr2=0x7f25c4062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 shutdown_connections 2023-12-19T14:54:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.358+0000 7f25c900c700 1 -- 172.21.15.50:0/538197482 wait complete. 2023-12-19T14:54:41.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:54:41.500 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:41.494+0000 7fbb8415e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:41.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.550+0000 7f8014ee3700 1 Processor -- start 2023-12-19T14:54:41.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.550+0000 7f8014ee3700 1 -- start start 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.554+0000 7f8014ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8010154780 0x7f801015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.554+0000 7f8014ee3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f80100adab0 0x7f80100adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.554+0000 7f8014ee3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801015ae30 con 0x7f801006baa0 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.554+0000 7f8014ee3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100abf70 con 0x7f80100adab0 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.554+0000 7f8014ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8010154b50 con 0x7f8010154780 2023-12-19T14:54:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f80100adab0 0x7f80100adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f80100adab0 0x7f80100adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46620/0 (socket says 172.21.15.50:46620) 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 -- 172.21.15.50:0/2067644459 learned_addr learned my addr 172.21.15.50:0/2067644459 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800f7fe700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8010154780 0x7f801015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 -- 172.21.15.50:0/2067644459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8010154780 msgr2=0x7f801015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8010154780 0x7f801015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 -- 172.21.15.50:0/2067644459 >> v1:172.21.15.125:6789/0 conn(0x7f801006baa0 legacy=0x7f8010157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 -- 172.21.15.50:0/2067644459 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100ac210 con 0x7f80100adab0 2023-12-19T14:54:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800ffff700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100adf10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8008009f20 tx=0x7f8008009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb13b54869b3a49 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800effd700 1 -- 172.21.15.50:0/2067644459 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800800e070 con 0x7f80100adab0 2023-12-19T14:54:41.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800effd700 1 -- 172.21.15.50:0/2067644459 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800800b9d0 con 0x7f80100adab0 2023-12-19T14:54:41.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f800effd700 1 -- 172.21.15.50:0/2067644459 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80080092e0 con 0x7f80100adab0 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 -- 172.21.15.50:0/2067644459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 msgr2=0x7f80100adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100adf10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8008009f20 tx=0x7f8008009840 comp rx=0 tx=0).stop 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 -- 172.21.15.50:0/2067644459 shutdown_connections 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100adf10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 --2- 172.21.15.50:0/2067644459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8010154780 0x7f801015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 -- 172.21.15.50:0/2067644459 >> 172.21.15.50:0/2067644459 conn(0x7f8010067b50 msgr2=0x7f80100cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 -- 172.21.15.50:0/2067644459 shutdown_connections 2023-12-19T14:54:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.558+0000 7f8014ee3700 1 -- 172.21.15.50:0/2067644459 wait complete. 2023-12-19T14:54:41.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.562+0000 7f8014ee3700 1 Processor -- start 2023-12-19T14:54:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.562+0000 7f8014ee3700 1 -- start start 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8010154780 0x7f80100b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801014afd0 con 0x7f80100adab0 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101539f0 con 0x7f8010154780 2023-12-19T14:54:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801014e410 con 0x7f801006baa0 2023-12-19T14:54:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8006ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55146/0 (socket says 172.21.15.50:55146) 2023-12-19T14:54:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 -- 172.21.15.50:0/1630790706 learned_addr learned my addr 172.21.15.50:0/1630790706 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800ffff700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8010154780 0x7f80100b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 -- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8010154780 msgr2=0x7f80100b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8010154780 0x7f80100b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 -- 172.21.15.50:0/1630790706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 msgr2=0x7f80100af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 -- 172.21.15.50:0/1630790706 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008009ae0 con 0x7f801006baa0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f800f7fe700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f800000e930 tx=0x7f80000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000013680 con 0x7f801006baa0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80000137e0 con 0x7f801006baa0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000013b80 con 0x7f801006baa0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801014f1a0 con 0x7f801006baa0 2023-12-19T14:54:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.566+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801014f760 con 0x7f801006baa0 2023-12-19T14:54:41.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801004be40 con 0x7f801006baa0 2023-12-19T14:54:41.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f800000c3e0 con 0x7f801006baa0 2023-12-19T14:54:41.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8007fff700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 0x7f7ff0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f800007d280 con 0x7f801006baa0 2023-12-19T14:54:41.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8006ffd700 1 -- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 msgr2=0x7f7ff0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:41.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.570+0000 7f8006ffd700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 0x7f7ff0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:41.581 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:41.578+0000 7f51243c3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:54:41.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.578+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f800004d2a0 con 0x7f801006baa0 2023-12-19T14:54:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.750+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80100ab870 con 0x7f801006baa0 2023-12-19T14:54:41.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8007fff700 1 -- 172.21.15.50:0/1630790706 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f80000508c0 con 0x7f801006baa0 2023-12-19T14:54:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 msgr2=0x7f7ff0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 0x7f7ff0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 msgr2=0x7f80101534b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f800000e930 tx=0x7f80000091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 shutdown_connections 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8010154780 0x7f80100b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060700 0x7f7ff0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f80100adab0 0x7f80100af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 --2- 172.21.15.50:0/1630790706 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f801006baa0 0x7f80101534b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 >> 172.21.15.50:0/1630790706 conn(0x7f8010067b50 msgr2=0x7f80100cacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:41.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 shutdown_connections 2023-12-19T14:54:41.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.754+0000 7f8014ee3700 1 -- 172.21.15.50:0/1630790706 wait complete. 2023-12-19T14:54:41.762 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:41.758+0000 7f51243c3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:54:41.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:41.840 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:41.834+0000 7fbb8415e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:41.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.950+0000 7fce0f59e700 1 Processor -- start 2023-12-19T14:54:41.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.950+0000 7fce0f59e700 1 -- start start 2023-12-19T14:54:41.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce100b1410 0x7fce100ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 0x7fce1006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 0x7fce1015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006af40 con 0x7fce100b1410 2023-12-19T14:54:41.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006c7c0 con 0x7fce1006a580 2023-12-19T14:54:41.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0f59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1006cd80 con 0x7fce1015a860 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce100b1410 0x7fce100ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fce100b1410 0x7fce100ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46640/0 (socket says 172.21.15.50:46640) 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 -- 172.21.15.50:0/1023761385 learned_addr learned my addr 172.21.15.50:0/1023761385 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0dd9b700 1 --2- 172.21.15.50:0/1023761385 >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 0x7fce1006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0ed9d700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 0x7fce1015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 -- 172.21.15.50:0/1023761385 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 msgr2=0x7fce1015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 0x7fce1015ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 -- 172.21.15.50:0/1023761385 >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 msgr2=0x7fce1006aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1023761385 >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 0x7fce1006aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 -- 172.21.15.50:0/1023761385 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1006b8f0 con 0x7fce100b1410 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0dd9b700 1 --2- 172.21.15.50:0/1023761385 >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 0x7fce1006aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0ed9d700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 0x7fce1015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.954+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100b1410 0x7fce100ab440 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fce10068040 tx=0x7fce00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e93d8cafeb49e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0d59a700 1 -- 172.21.15.50:0/1023761385 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000b860 con 0x7fce100b1410 2023-12-19T14:54:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0d59a700 1 -- 172.21.15.50:0/1023761385 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0000b9c0 con 0x7fce100b1410 2023-12-19T14:54:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0d59a700 1 -- 172.21.15.50:0/1023761385 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000bd60 con 0x7fce100b1410 2023-12-19T14:54:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 -- 172.21.15.50:0/1023761385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100b1410 msgr2=0x7fce100ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100b1410 0x7fce100ab440 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fce10068040 tx=0x7fce00009250 comp rx=0 tx=0).stop 2023-12-19T14:54:41.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 -- 172.21.15.50:0/1023761385 shutdown_connections 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce1015a860 0x7fce1015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1023761385 >> v2:172.21.15.125:3300/0 conn(0x7fce1006a580 0x7fce1006aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1023761385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce100b1410 0x7fce100ab440 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 -- 172.21.15.50:0/1023761385 >> 172.21.15.50:0/1023761385 conn(0x7fce10067bb0 msgr2=0x7fce100cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 -- 172.21.15.50:0/1023761385 shutdown_connections 2023-12-19T14:54:41.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.958+0000 7fce0f59e700 1 -- 172.21.15.50:0/1023761385 wait complete. 2023-12-19T14:54:41.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 Processor -- start 2023-12-19T14:54:41.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 -- start start 2023-12-19T14:54:41.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce100b1410 0x7fce10155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce1015a860 0x7fce100c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1016d390 con 0x7fce1006a580 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1016d670 con 0x7fce1015a860 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1016d950 con 0x7fce100b1410 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46652/0 (socket says 172.21.15.50:46652) 2023-12-19T14:54:41.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 -- 172.21.15.50:0/1305037060 learned_addr learned my addr 172.21.15.50:0/1305037060 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:41.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0ed9d700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce1015a860 0x7fce100c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0dd9b700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce100b1410 0x7fce10155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:41.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce100b1410 msgr2=0x7fce10155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce100b1410 0x7fce10155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce1015a860 msgr2=0x7fce100c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce1015a860 0x7fce100c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 -- 172.21.15.50:0/1305037060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000094f0 con 0x7fce1006a580 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0e59c700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fce00009290 tx=0x7fce00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00011e40 con 0x7fce1006a580 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0001f070 con 0x7fce1006a580 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce100c5fd0 con 0x7fce1006a580 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.962+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce000266d0 con 0x7fce1006a580 2023-12-19T14:54:41.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.966+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1016d080 con 0x7fce1006a580 2023-12-19T14:54:41.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.966+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce100c64d0 con 0x7fce1006a580 2023-12-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.966+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fce00026a40 con 0x7fce1006a580 2023-12-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.970+0000 7fcdfe7fc700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 0x7fcdf0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.970+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fce00021070 con 0x7fce1006a580 2023-12-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.970+0000 7fce0dd9b700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 msgr2=0x7fcdf0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.970+0000 7fce0dd9b700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 0x7fcdf0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:41.974+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce0004ebb0 con 0x7fce1006a580 2023-12-19T14:54:42.049 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.042+0000 7fbb8415e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:42.141 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.134+0000 7fbb8415e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:42.155 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.154+0000 7f51243c3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:54:42.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.174+0000 7fce0dd9b700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 msgr2=0x7fcdf0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:42.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.174+0000 7fce0dd9b700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 0x7fcdf0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.186+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce1004be40 con 0x7fce1006a580 2023-12-19T14:54:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.186+0000 7fcdfe7fc700 1 -- 172.21.15.50:0/1305037060 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fce00015090 con 0x7fce1006a580 2023-12-19T14:54:42.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:42.188 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 msgr2=0x7fcdf0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 0x7fcdf0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 msgr2=0x7fce1016cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fce00009290 tx=0x7fce00017000 comp rx=0 tx=0).stop 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 shutdown_connections 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce1015a860 0x7fce100c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcdf0060700 0x7fcdf0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fce100b1410 0x7fce10155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 --2- 172.21.15.50:0/1305037060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fce1006a580 0x7fce1016cb40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 >> 172.21.15.50:0/1305037060 conn(0x7fce10067bb0 msgr2=0x7fce100ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 shutdown_connections 2023-12-19T14:54:42.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.190+0000 7fce0f59e700 1 -- 172.21.15.50:0/1305037060 wait complete. 2023-12-19T14:54:42.194 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:42.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:42.217 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.210+0000 7fbb8415e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:42.297 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.290+0000 7fbb8415e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.374+0000 7f3a4744c700 1 Processor -- start 2023-12-19T14:54:42.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.374+0000 7f3a4744c700 1 -- start start 2023-12-19T14:54:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a400ce810 0x7f3a400ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4006b730 con 0x7f3a401591f0 2023-12-19T14:54:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4006ab20 con 0x7f3a400ce810 2023-12-19T14:54:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4006ae00 con 0x7f3a40158e20 2023-12-19T14:54:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a400ce810 0x7f3a400ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a400ce810 0x7f3a400ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46662/0 (socket says 172.21.15.50:46662) 2023-12-19T14:54:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 -- 172.21.15.50:0/2425781981 learned_addr learned my addr 172.21.15.50:0/2425781981 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4644a700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 -- 172.21.15.50:0/2425781981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 msgr2=0x7f3a4006a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4006a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 -- 172.21.15.50:0/2425781981 >> v1:172.21.15.125:6789/0 conn(0x7f3a401591f0 legacy=0x7f3a400ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 -- 172.21.15.50:0/2425781981 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4006b0e0 con 0x7f3a400ce810 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4644a700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a46c4b700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a400ca950 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009f20 tx=0x7f3a38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30288c7c2b9cc48a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:42.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a45448700 1 -- 172.21.15.50:0/2425781981 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800e070 con 0x7f3a400ce810 2023-12-19T14:54:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a45448700 1 -- 172.21.15.50:0/2425781981 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3800ba80 con 0x7f3a400ce810 2023-12-19T14:54:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a45448700 1 -- 172.21.15.50:0/2425781981 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380092e0 con 0x7f3a400ce810 2023-12-19T14:54:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 -- 172.21.15.50:0/2425781981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 msgr2=0x7f3a400ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.378+0000 7f3a4744c700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a400ca950 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009f20 tx=0x7f3a38009840 comp rx=0 tx=0).stop 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- 172.21.15.50:0/2425781981 shutdown_connections 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a400ca950 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 --2- 172.21.15.50:0/2425781981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- 172.21.15.50:0/2425781981 >> 172.21.15.50:0/2425781981 conn(0x7f3a40067bb0 msgr2=0x7f3a400cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- 172.21.15.50:0/2425781981 shutdown_connections 2023-12-19T14:54:42.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- 172.21.15.50:0/2425781981 wait complete. 2023-12-19T14:54:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 Processor -- start 2023-12-19T14:54:42.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- start start 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a401503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4014cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4014f600 con 0x7f3a400ce810 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400c86f0 con 0x7f3a401591f0 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.382+0000 7f3a4744c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400c7e00 con 0x7f3a40158e20 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a4644a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a401503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:55616/0 (socket says 172.21.15.50:55616) 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 -- 172.21.15.50:0/1267188449 learned_addr learned my addr 172.21.15.50:0/1267188449 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a45c49700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4014cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 -- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 msgr2=0x7f3a4014cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4014cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 -- 172.21.15.50:0/1267188449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 msgr2=0x7f3a401503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a401503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 -- 172.21.15.50:0/1267188449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009ae0 con 0x7f3a401591f0 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a4644a700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a401503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:42.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a46c4b700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3a4006e810 tx=0x7f3a38018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380094e0 con 0x7f3a401591f0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a38008d20 con 0x7f3a401591f0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4014d190 con 0x7f3a401591f0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380266b0 con 0x7f3a401591f0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400c80a0 con 0x7f3a401591f0 2023-12-19T14:54:42.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.386+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a400c8880 con 0x7f3a401591f0 2023-12-19T14:54:42.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.390+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f3a38003040 con 0x7f3a401591f0 2023-12-19T14:54:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.390+0000 7f3a367fc700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 0x7f3a24062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.390+0000 7f3a4644a700 1 -- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 msgr2=0x7f3a24062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.390+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3a3807ee60 con 0x7f3a401591f0 2023-12-19T14:54:42.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.394+0000 7f3a4644a700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 0x7f3a24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:42.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.398+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3a3804ed70 con 0x7f3a401591f0 2023-12-19T14:54:42.406 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.402+0000 7f51243c3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:54:42.508 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.506+0000 7f51243c3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:54:42.516 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.510+0000 7fbb8415e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:42.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.574+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a4004be40 con 0x7f3a401591f0 2023-12-19T14:54:42.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.578+0000 7f3a367fc700 1 -- 172.21.15.50:0/1267188449 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16098 (secure 0 0 0) 0x7f3a38052390 con 0x7f3a401591f0 2023-12-19T14:54:42.579 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:42.579 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":87,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:37.258411+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/552631771":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3782271172":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/4106781597":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2371597704":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2154078294":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2430752556":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6833/33827":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6832/33827":"2023-12-20T14:54:37.258339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:42.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 msgr2=0x7f3a24062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:42.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 0x7f3a24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 msgr2=0x7f3a4014dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3a4006e810 tx=0x7f3a38018000 comp rx=0 tx=0).stop 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 shutdown_connections 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401591f0 0x7f3a4014dde0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3a24060700 0x7f3a24062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3a40158e20 0x7f3a4014cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 --2- 172.21.15.50:0/1267188449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a400ce810 0x7f3a401503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 >> 172.21.15.50:0/1267188449 conn(0x7f3a40067bb0 msgr2=0x7f3a400b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 shutdown_connections 2023-12-19T14:54:42.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.582+0000 7f3a4744c700 1 -- 172.21.15.50:0/1267188449 wait complete. 2023-12-19T14:54:42.591 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.590+0000 7f51243c3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:54:42.594 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.590+0000 7fbb8415e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:42.608 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:54:42.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:42.676 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.670+0000 7fbb8415e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:42.713 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.710+0000 7f51243c3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:54:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.762+0000 7fb49359e700 1 Processor -- start 2023-12-19T14:54:42.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49359e700 1 -- start start 2023-12-19T14:54:42.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb494158e20 0x7fb49406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49406b730 con 0x7fb4941591f0 2023-12-19T14:54:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49406ab20 con 0x7fb4940ce810 2023-12-19T14:54:42.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49406ae00 con 0x7fb494158e20 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb491d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb4941591f0 0x7fb4940ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38056/0 (socket says 172.21.15.50:38056) 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb491d9b700 1 -- 172.21.15.50:0/1689591392 learned_addr learned my addr 172.21.15.50:0/1689591392 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49259c700 1 --2- 172.21.15.50:0/1689591392 >> v2:172.21.15.125:3300/0 conn(0x7fb494158e20 0x7fb49406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910254788 0 0) 0x7fb49406b730 con 0x7fb4941591f0 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47c003570 con 0x7fb4941591f0 2023-12-19T14:54:42.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49259c700 1 -- 172.21.15.50:0/1689591392 >> v1:172.21.15.125:6789/0 conn(0x7fb4940ce810 legacy=0x7fb4940ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754803276 0 0) 0x7fb49406ab20 con 0x7fb4940ce810 2023-12-19T14:54:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49259c700 1 -- 172.21.15.50:0/1689591392 >> v1:172.21.15.50:6789/0 conn(0x7fb4941591f0 legacy=0x7fb4940ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.766+0000 7fb49259c700 1 -- 172.21.15.50:0/1689591392 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49406b0e0 con 0x7fb494158e20 2023-12-19T14:54:42.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49259c700 1 --2- 172.21.15.50:0/1689591392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb494158e20 0x7fb49406a540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb4840098e0 tx=0x7fb4840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba59cbf951586ef6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400b640 con 0x7fb494158e20 2023-12-19T14:54:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48400b7a0 con 0x7fb494158e20 2023-12-19T14:54:42.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49159a700 1 -- 172.21.15.50:0/1689591392 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400bb40 con 0x7fb494158e20 2023-12-19T14:54:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 -- 172.21.15.50:0/1689591392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb494158e20 msgr2=0x7fb49406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 --2- 172.21.15.50:0/1689591392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb494158e20 0x7fb49406a540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb4840098e0 tx=0x7fb4840091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 -- 172.21.15.50:0/1689591392 shutdown_connections 2023-12-19T14:54:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 --2- 172.21.15.50:0/1689591392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb494158e20 0x7fb49406a540 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 -- 172.21.15.50:0/1689591392 >> 172.21.15.50:0/1689591392 conn(0x7fb494067bb0 msgr2=0x7fb4940cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 -- 172.21.15.50:0/1689591392 shutdown_connections 2023-12-19T14:54:42.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.770+0000 7fb49359e700 1 -- 172.21.15.50:0/1689591392 wait complete. 2023-12-19T14:54:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.774+0000 7fb49359e700 1 Processor -- start 2023-12-19T14:54:42.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.774+0000 7fb49359e700 1 -- start start 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4940ce810 0x7fb494153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 0x7fb494140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4941407a0 con 0x7fb494158e20 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494157630 con 0x7fb4941591f0 2023-12-19T14:54:42.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49415bdd0 con 0x7fb4940ce810 2023-12-19T14:54:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb491d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb491d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46674/0 (socket says 172.21.15.50:46674) 2023-12-19T14:54:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb491d9b700 1 -- 172.21.15.50:0/2951736384 learned_addr learned my addr 172.21.15.50:0/2951736384 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 0x7fb494140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49259c700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4940ce810 0x7fb494153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4940ce810 msgr2=0x7fb494153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4940ce810 0x7fb494153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 msgr2=0x7fb49413fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 -- 172.21.15.50:0/2951736384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb484009420 con 0x7fb4941591f0 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb491d9b700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:42.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb492d9d700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 0x7fb494140260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb48c001560 tx=0x7fb48c0016c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c012630 con 0x7fb4941591f0 2023-12-19T14:54:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c012790 con 0x7fb4941591f0 2023-12-19T14:54:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c012b50 con 0x7fb4941591f0 2023-12-19T14:54:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49415c340 con 0x7fb4941591f0 2023-12-19T14:54:42.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.778+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb494154830 con 0x7fb4941591f0 2023-12-19T14:54:42.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.782+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb494154a80 con 0x7fb4941591f0 2023-12-19T14:54:42.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.786+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fb48c003030 con 0x7fb4941591f0 2023-12-19T14:54:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.786+0000 7fb4827fc700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 0x7fb474062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.786+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb48c07cd70 con 0x7fb4941591f0 2023-12-19T14:54:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.786+0000 7fb49259c700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 msgr2=0x7fb474062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:42.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.786+0000 7fb49259c700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 0x7fb474062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:42.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.794+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb48c04cd90 con 0x7fb4941591f0 2023-12-19T14:54:42.863 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.858+0000 7fbb8415e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:42.949 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:42.942+0000 7fbb8415e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:42.959 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:42.958+0000 7f51243c3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:54:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.990+0000 7fb49259c700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 msgr2=0x7fb474062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:42.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:42.990+0000 7fb49259c700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 0x7fb474062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:43.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.010+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb49406b3c0 con 0x7fb4941591f0 2023-12-19T14:54:43.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.010+0000 7fb4827fc700 1 -- 172.21.15.50:0/2951736384 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fb48c0503b0 con 0x7fb4941591f0 2023-12-19T14:54:43.013 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:43.013 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:43.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 msgr2=0x7fb474062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:43.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 0x7fb474062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 msgr2=0x7fb494140260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 0x7fb494140260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb48c001560 tx=0x7fb48c0016c0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 shutdown_connections 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4941591f0 0x7fb494140260 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb474060700 0x7fb474062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb494158e20 0x7fb49413fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 --2- 172.21.15.50:0/2951736384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb4940ce810 0x7fb494153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 >> 172.21.15.50:0/2951736384 conn(0x7fb494067bb0 msgr2=0x7fb494152630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 shutdown_connections 2023-12-19T14:54:43.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.014+0000 7fb49359e700 1 -- 172.21.15.50:0/2951736384 wait complete. 2023-12-19T14:54:43.018 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:43.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:43.048 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:43.046+0000 7f51243c3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:54:43.148 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:43.146+0000 7f51243c3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:54:43.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.198+0000 7efc1302e700 1 Processor -- start 2023-12-19T14:54:43.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.198+0000 7efc1302e700 1 -- start start 2023-12-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc1302e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 0x7efc0c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc1302e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0c068c70 con 0x7efc0c069100 2023-12-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc1302e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0c068060 con 0x7efc0c159340 2023-12-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc1302e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c068340 con 0x7efc0c158f70 2023-12-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efc0c069100 0x7efc0c0cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38062/0 (socket says 172.21.15.50:38062) 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0bfff700 1 -- 172.21.15.50:0/128089321 learned_addr learned my addr 172.21.15.50:0/128089321 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc10dca700 1 --2- 172.21.15.50:0/128089321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 0x7efc0c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791413412 0 0) 0x7efc0c068c70 con 0x7efc0c069100 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf4003570 con 0x7efc0c069100 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205806906 0 0) 0x7efc0c068060 con 0x7efc0c159340 2023-12-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.202+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc0c068c70 con 0x7efc0c159340 2023-12-19T14:54:43.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250174836 0 0) 0x7efbf4003570 con 0x7efc0c069100 2023-12-19T14:54:43.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc10dca700 1 -- 172.21.15.50:0/128089321 >> v1:172.21.15.125:6789/0 conn(0x7efc0c159340 legacy=0x7efc0c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc10dca700 1 -- 172.21.15.50:0/128089321 >> v1:172.21.15.50:6789/0 conn(0x7efc0c069100 legacy=0x7efc0c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc10dca700 1 -- 172.21.15.50:0/128089321 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c068620 con 0x7efc0c158f70 2023-12-19T14:54:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc10dca700 1 --2- 172.21.15.50:0/128089321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 0x7efc0c067980 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efc00009910 tx=0x7efc000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907a686d66d497e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc001470 con 0x7efc0c158f70 2023-12-19T14:54:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00011420 con 0x7efc0c158f70 2023-12-19T14:54:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc0b7fe700 1 -- 172.21.15.50:0/128089321 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000117c0 con 0x7efc0c158f70 2023-12-19T14:54:43.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 -- 172.21.15.50:0/128089321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 msgr2=0x7efc0c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 --2- 172.21.15.50:0/128089321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 0x7efc0c067980 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efc00009910 tx=0x7efc000091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 -- 172.21.15.50:0/128089321 shutdown_connections 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 --2- 172.21.15.50:0/128089321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c158f70 0x7efc0c067980 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 -- 172.21.15.50:0/128089321 >> 172.21.15.50:0/128089321 conn(0x7efc0c0ccc40 msgr2=0x7efc0c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 -- 172.21.15.50:0/128089321 shutdown_connections 2023-12-19T14:54:43.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.206+0000 7efc1302e700 1 -- 172.21.15.50:0/128089321 wait complete. 2023-12-19T14:54:43.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 Processor -- start 2023-12-19T14:54:43.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 -- start start 2023-12-19T14:54:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc0c158f70 0x7efc0c0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c159340 0x7efc0c0caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0b5050 con 0x7efc0c069100 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0b44b0 con 0x7efc0c158f70 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.210+0000 7efc1302e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0b4720 con 0x7efc0c159340 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46690/0 (socket says 172.21.15.50:46690) 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 -- 172.21.15.50:0/1343543253 learned_addr learned my addr 172.21.15.50:0/1343543253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:43.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc0bfff700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc0c158f70 0x7efc0c0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc115cb700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c159340 0x7efc0c0caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c159340 msgr2=0x7efc0c0caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c159340 0x7efc0c0caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc0c158f70 msgr2=0x7efc0c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc0c158f70 0x7efc0c0ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 -- 172.21.15.50:0/1343543253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00009420 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc10dca700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efc000091e0 tx=0x7efc00016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0001cc70 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0001cdd0 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c0b4a00 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0001a6a0 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c14e1f0 con 0x7efc0c069100 2023-12-19T14:54:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.214+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c071120 con 0x7efc0c069100 2023-12-19T14:54:43.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.218+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7efc0001aad0 con 0x7efc0c069100 2023-12-19T14:54:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.218+0000 7efc08ff9700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 0x7efbf8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.218+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efc0007ec30 con 0x7efc0c069100 2023-12-19T14:54:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.218+0000 7efc0bfff700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 msgr2=0x7efbf8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:43.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.218+0000 7efc0bfff700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 0x7efbf8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:43.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.226+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efc0004ec50 con 0x7efc0c069100 2023-12-19T14:54:43.361 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:43.358+0000 7f51243c3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:54:43.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.422+0000 7efc0bfff700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 msgr2=0x7efbf8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:43.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.422+0000 7efc0bfff700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 0x7efbf8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:54:43.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.434+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc0c068900 con 0x7efc0c069100 2023-12-19T14:54:43.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.434+0000 7efc08ff9700 1 -- 172.21.15.50:0/1343543253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4604 (secure 0 0 0) 0x7efc00052270 con 0x7efc0c069100 2023-12-19T14:54:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:43.438 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:22.525941+0000","modified":"2023-12-19T14:54:26.116986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6465},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6465":{"gid":6465,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3558598512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3558598512},{"type":"v1","addr":"172.21.15.50:6835","nonce":3558598512}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 msgr2=0x7efbf8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 0x7efbf8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 msgr2=0x7efc0c14dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efc000091e0 tx=0x7efc00016000 comp rx=0 tx=0).stop 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 shutdown_connections 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc0c158f70 0x7efc0c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efbf8060700 0x7efbf8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc0c159340 0x7efc0c0caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 --2- 172.21.15.50:0/1343543253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc0c069100 0x7efc0c14dcb0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 >> 172.21.15.50:0/1343543253 conn(0x7efc0c0ccc40 msgr2=0x7efc0c0cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:43.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 shutdown_connections 2023-12-19T14:54:43.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.438+0000 7efc1302e700 1 -- 172.21.15.50:0/1343543253 wait complete. 2023-12-19T14:54:43.444 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 33 2023-12-19T14:54:43.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:43.460 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:43.458+0000 7f51243c3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:54:43.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.622+0000 7f2297c2e700 1 Processor -- start 2023-12-19T14:54:43.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.622+0000 7f2297c2e700 1 -- start start 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f2297c2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f22900cc3b0 0x7f22900cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f2297c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2290157060 0x7f229015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f2297c2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900cd5a0 con 0x7f2290156c90 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f2297c2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2290157470 con 0x7f22900cc3b0 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f2297c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900cc990 con 0x7f2290157060 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f22900cc3b0 0x7f22900cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f22900cc3b0 0x7f22900cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46702/0 (socket says 172.21.15.50:46702) 2023-12-19T14:54:43.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 -- 172.21.15.50:0/1594429945 learned_addr learned my addr 172.21.15.50:0/1594429945 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:43.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22961cb700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2290157060 0x7f229015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 -- 172.21.15.50:0/1594429945 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2290157060 msgr2=0x7f229015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:43.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2290157060 0x7f229015ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 -- 172.21.15.50:0/1594429945 >> v1:172.21.15.125:6789/0 conn(0x7f2290156c90 legacy=0x7f22900c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 -- 172.21.15.50:0/1594429945 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900ccc70 con 0x7f22900cc3b0 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22951c9700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22900cc3b0 0x7f22900cda30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f22800099b0 tx=0x7f2280009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b8b8910bdaeb4b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.626+0000 7f22949c8700 1 -- 172.21.15.50:0/1594429945 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228000b820 con 0x7f22900cc3b0 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f22949c8700 1 -- 172.21.15.50:0/1594429945 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228000b980 con 0x7f22900cc3b0 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f22949c8700 1 -- 172.21.15.50:0/1594429945 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228000bd20 con 0x7f22900cc3b0 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 -- 172.21.15.50:0/1594429945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22900cc3b0 msgr2=0x7f22900cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22900cc3b0 0x7f22900cda30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f22800099b0 tx=0x7f2280009250 comp rx=0 tx=0).stop 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 -- 172.21.15.50:0/1594429945 shutdown_connections 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2290157060 0x7f229015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1594429945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22900cc3b0 0x7f22900cda30 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 -- 172.21.15.50:0/1594429945 >> 172.21.15.50:0/1594429945 conn(0x7f229006c7c0 msgr2=0x7f229006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 -- 172.21.15.50:0/1594429945 shutdown_connections 2023-12-19T14:54:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.630+0000 7f2297c2e700 1 -- 172.21.15.50:0/1594429945 wait complete. 2023-12-19T14:54:43.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 Processor -- start 2023-12-19T14:54:43.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 -- start start 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22900cc3b0 0x7f229014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290156c90 0x7f22900b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290159170 con 0x7f2290157060 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2290158ce0 con 0x7f2290156c90 2023-12-19T14:54:43.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f2297c2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22901580d0 con 0x7f22900cc3b0 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46710/0 (socket says 172.21.15.50:46710) 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 -- 172.21.15.50:0/1665378253 learned_addr learned my addr 172.21.15.50:0/1665378253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22951c9700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290156c90 0x7f22900b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22959ca700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22900cc3b0 0x7f229014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 -- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22900cc3b0 msgr2=0x7f229014de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22900cc3b0 0x7f229014de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 -- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290156c90 msgr2=0x7f22900b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290156c90 0x7f22900b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 -- 172.21.15.50:0/1665378253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800094f0 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.634+0000 7f22961cb700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f228800cdb0 tx=0x7f22880077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228800aa20 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228800ab80 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288017430 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22901583b0 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2290158970 con 0x7f2290157060 2023-12-19T14:54:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2290158e50 con 0x7f2290157060 2023-12-19T14:54:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.638+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 30) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f2288003020 con 0x7f2290157060 2023-12-19T14:54:43.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.642+0000 7f2285ffb700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 0x7f2274062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:43.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.642+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2288013070 con 0x7f2290157060 2023-12-19T14:54:43.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.642+0000 7f22959ca700 1 -- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 msgr2=0x7f2274062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:54:43.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.642+0000 7f22959ca700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 0x7f2274062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:54:43.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.650+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f228804d280 con 0x7f2290157060 2023-12-19T14:54:43.662 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.654+0000 7fbb8415e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:43.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.682+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 31) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f228800d040 con 0x7f2290157060 2023-12-19T14:54:43.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.682+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 msgr2=0x7f2274062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:54:43.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.682+0000 7f2285ffb700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 0x7f2274062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.761 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4ec5c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.761 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4ec5c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.761 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4ec5c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.761 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4ec5c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.762 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4ec5c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.762 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4b455700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.762 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4b455700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.762 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4b455700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.762 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4b455700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.763 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:54:43.754+0000 7fbb4b455700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:54:43.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.906+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2290071120 con 0x7f2290157060 2023-12-19T14:54:43.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2285ffb700 1 -- 172.21.15.50:0/1665378253 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16539 (secure 0 0 0) 0x7f22880791c0 con 0x7f2290157060 2023-12-19T14:54:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:43.911 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":88,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:43.668770+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:20.550186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:21.522239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1777832927":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1579636953":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/3439418072":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/2922501785":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1878386883":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1822502716":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/552631771":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3782271172":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/4106781597":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2371597704":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2154078294":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2430752556":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3554920661":"2023-12-20T14:54:43.668698+0000","172.21.15.125:6833/33827":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6832/33827":"2023-12-20T14:54:37.258339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:43.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 msgr2=0x7f22900b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:43.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f228800cdb0 tx=0x7f22880077d0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 shutdown_connections 2023-12-19T14:54:43.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2290156c90 0x7f22900b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2274060700 0x7f2274062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2290157060 0x7f22900b12d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 --2- 172.21.15.50:0/1665378253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f22900cc3b0 0x7f229014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:43.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 >> 172.21.15.50:0/1665378253 conn(0x7f229006c7c0 msgr2=0x7f22900acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:43.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 shutdown_connections 2023-12-19T14:54:43.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:43.910+0000 7f2297c2e700 1 -- 172.21.15.50:0/1665378253 wait complete. 2023-12-19T14:54:43.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:54:44.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 Processor -- start 2023-12-19T14:54:44.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 -- start start 2023-12-19T14:54:44.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffabc06d760 0x7ffabc068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc068fb0 con 0x7ffabc159160 2023-12-19T14:54:44.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc06de50 con 0x7ffabc068620 2023-12-19T14:54:44.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.098+0000 7ffac2985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffabc0b0180 con 0x7ffabc06d760 2023-12-19T14:54:44.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabbfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffabc159160 0x7ffabc0afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38064/0 (socket says 172.21.15.50:38064) 2023-12-19T14:54:44.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabbfff700 1 -- 172.21.15.50:0/645958905 learned_addr learned my addr 172.21.15.50:0/645958905 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:44.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac0f22700 1 --2- 172.21.15.50:0/645958905 >> v2:172.21.15.125:3300/0 conn(0x7ffabc06d760 0x7ffabc068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabaffd700 1 -- 172.21.15.50:0/645958905 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338142491 0 0) 0x7ffabc068fb0 con 0x7ffabc159160 2023-12-19T14:54:44.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabaffd700 1 -- 172.21.15.50:0/645958905 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4003570 con 0x7ffabc159160 2023-12-19T14:54:44.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac0f22700 1 -- 172.21.15.50:0/645958905 >> v1:172.21.15.125:6789/0 conn(0x7ffabc068620 legacy=0x7ffabc06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac0f22700 1 -- 172.21.15.50:0/645958905 >> v1:172.21.15.50:6789/0 conn(0x7ffabc159160 legacy=0x7ffabc0afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac0f22700 1 -- 172.21.15.50:0/645958905 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc159530 con 0x7ffabc06d760 2023-12-19T14:54:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac0f22700 1 --2- 172.21.15.50:0/645958905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc06d760 0x7ffabc068a70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009e60 tx=0x7ffab4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a80e409fe2d084 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabaffd700 1 -- 172.21.15.50:0/645958905 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400e070 con 0x7ffabc06d760 2023-12-19T14:54:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabaffd700 1 -- 172.21.15.50:0/645958905 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab400bb60 con 0x7ffabc06d760 2023-12-19T14:54:44.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffabaffd700 1 -- 172.21.15.50:0/645958905 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4012430 con 0x7ffabc06d760 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 -- 172.21.15.50:0/645958905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc06d760 msgr2=0x7ffabc068a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 --2- 172.21.15.50:0/645958905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc06d760 0x7ffabc068a70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009e60 tx=0x7ffab4009780 comp rx=0 tx=0).stop 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 -- 172.21.15.50:0/645958905 shutdown_connections 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 --2- 172.21.15.50:0/645958905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc06d760 0x7ffabc068a70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 -- 172.21.15.50:0/645958905 >> 172.21.15.50:0/645958905 conn(0x7ffabc0cd740 msgr2=0x7ffabc0c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 -- 172.21.15.50:0/645958905 shutdown_connections 2023-12-19T14:54:44.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.102+0000 7ffac2985700 1 -- 172.21.15.50:0/645958905 wait complete. 2023-12-19T14:54:44.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 Processor -- start 2023-12-19T14:54:44.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 -- start start 2023-12-19T14:54:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc068620 0x7ffabc14de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffabc159160 0x7ffabc0c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc15d100 con 0x7ffabc06d760 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc159c20 con 0x7ffabc068620 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac2985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc159f00 con 0x7ffabc159160 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffabb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffabb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46712/0 (socket says 172.21.15.50:46712) 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffabb7fe700 1 -- 172.21.15.50:0/2107159576 learned_addr learned my addr 172.21.15.50:0/2107159576 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffac0f22700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffabc159160 0x7ffabc0c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.106+0000 7ffabbfff700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc068620 0x7ffabc14de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 -- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffabc159160 msgr2=0x7ffabc0c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffabc159160 0x7ffabc0c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 -- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc068620 msgr2=0x7ffabc14de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc068620 0x7ffabc14de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 -- 172.21.15.50:0/2107159576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab4009a20 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffabb7fe700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c4e0 tx=0x7ffab000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0011590 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0011710 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc15a1e0 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0011ad0 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc16cdf0 con 0x7ffabc06d760 2023-12-19T14:54:44.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.110+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffabc15d270 con 0x7ffabc06d760 2023-12-19T14:54:44.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.114+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 31) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ffab001b3b0 con 0x7ffabc06d760 2023-12-19T14:54:44.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.114+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6698+0+0 (secure 0 0 0) 0x7ffab007bbc0 con 0x7ffabc06d760 2023-12-19T14:54:44.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.122+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffab004c190 con 0x7ffabc06d760 2023-12-19T14:54:44.329 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:54:44.326+0000 7f51243c3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:54:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.330+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffabc071120 con 0x7ffabc06d760 2023-12-19T14:54:44.332 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:54:44.330+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:54:44.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.338+0000 7ffaa3fff700 1 -- 172.21.15.50:0/2107159576 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffab0017020 con 0x7ffabc06d760 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 msgr2=0x7ffabc0c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c4e0 tx=0x7ffab000c8a0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 shutdown_connections 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffabc159160 0x7ffabc0c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffabc06d760 0x7ffabc0c6530 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 --2- 172.21.15.50:0/2107159576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffabc068620 0x7ffabc14de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 >> 172.21.15.50:0/2107159576 conn(0x7ffabc0cd740 msgr2=0x7ffabc06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 shutdown_connections 2023-12-19T14:54:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.342+0000 7ffac2985700 1 -- 172.21.15.50:0/2107159576 wait complete. 2023-12-19T14:54:44.346 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:54:44.351 INFO:tasks.ceph.mds.a.smithi050.stderr: -2524> 2023-12-19T14:54:28.035+0000 7f4919f5b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:54:44.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:54:44.393 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:54:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 Processor -- start 2023-12-19T14:54:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 -- start start 2023-12-19T14:54:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6d80d02a0 0x7ff6d806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d806cca0 con 0x7ff6d80cfed0 2023-12-19T14:54:44.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d806cf80 con 0x7ff6d806c7c0 2023-12-19T14:54:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6dd665700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d80b3510 con 0x7ff6d80d02a0 2023-12-19T14:54:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6d77fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6d80d02a0 0x7ff6d806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6d6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff6d80cfed0 0x7ff6d80b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:38078/0 (socket says 172.21.15.50:38078) 2023-12-19T14:54:44.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.518+0000 7ff6d6ffd700 1 -- 172.21.15.50:0/2112142362 learned_addr learned my addr 172.21.15.50:0/2112142362 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d5ffb700 1 -- 172.21.15.50:0/2112142362 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447203498 0 0) 0x7ff6d806cf80 con 0x7ff6d806c7c0 2023-12-19T14:54:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d5ffb700 1 -- 172.21.15.50:0/2112142362 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0003570 con 0x7ff6d806c7c0 2023-12-19T14:54:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d77fe700 1 -- 172.21.15.50:0/2112142362 >> v1:172.21.15.125:6789/0 conn(0x7ff6d806c7c0 legacy=0x7ff6d8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d77fe700 1 -- 172.21.15.50:0/2112142362 >> v1:172.21.15.50:6789/0 conn(0x7ff6d80cfed0 legacy=0x7ff6d80b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d77fe700 1 -- 172.21.15.50:0/2112142362 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d806d260 con 0x7ff6d80d02a0 2023-12-19T14:54:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d77fe700 1 --2- 172.21.15.50:0/2112142362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d806dd40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e60 tx=0x7ff6d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84037b79f1337239 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d5ffb700 1 -- 172.21.15.50:0/2112142362 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000e070 con 0x7ff6d80d02a0 2023-12-19T14:54:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d5ffb700 1 -- 172.21.15.50:0/2112142362 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000bb60 con 0x7ff6d80d02a0 2023-12-19T14:54:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6d5ffb700 1 -- 172.21.15.50:0/2112142362 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0012430 con 0x7ff6d80d02a0 2023-12-19T14:54:44.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 -- 172.21.15.50:0/2112142362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 msgr2=0x7ff6d806dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 --2- 172.21.15.50:0/2112142362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d806dd40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e60 tx=0x7ff6d0009780 comp rx=0 tx=0).stop 2023-12-19T14:54:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 -- 172.21.15.50:0/2112142362 shutdown_connections 2023-12-19T14:54:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 --2- 172.21.15.50:0/2112142362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d806dd40 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 -- 172.21.15.50:0/2112142362 >> 172.21.15.50:0/2112142362 conn(0x7ff6d80670e0 msgr2=0x7ff6d80674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 -- 172.21.15.50:0/2112142362 shutdown_connections 2023-12-19T14:54:44.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.522+0000 7ff6dd665700 1 -- 172.21.15.50:0/2112142362 wait complete. 2023-12-19T14:54:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 Processor -- start 2023-12-19T14:54:44.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 -- start start 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d806c7c0 0x7ff6d8158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6d80cfed0 0x7ff6d815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d816d140 con 0x7ff6d80d02a0 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d815d600 con 0x7ff6d806c7c0 2023-12-19T14:54:44.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.526+0000 7ff6dd665700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d815cd10 con 0x7ff6d80cfed0 2023-12-19T14:54:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46734/0 (socket says 172.21.15.50:46734) 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 -- 172.21.15.50:0/3589628985 learned_addr learned my addr 172.21.15.50:0/3589628985 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d67fc700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6d80cfed0 0x7ff6d815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d6ffd700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d806c7c0 0x7ff6d8158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 -- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6d80cfed0 msgr2=0x7ff6d815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6d80cfed0 0x7ff6d815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 -- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d806c7c0 msgr2=0x7ff6d8158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d806c7c0 0x7ff6d8158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 -- 172.21.15.50:0/3589628985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009a20 con 0x7ff6d80d02a0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6d77fe700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff6d806c010 tx=0x7ff6d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0027470 con 0x7ff6d80d02a0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d00275f0 con 0x7ff6d80d02a0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00279b0 con 0x7ff6d80d02a0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d816ce60 con 0x7ff6d80d02a0 2023-12-19T14:54:44.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d815d290 con 0x7ff6d80d02a0 2023-12-19T14:54:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.530+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d815d790 con 0x7ff6d80d02a0 2023-12-19T14:54:44.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.534+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 31) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ff6d0024070 con 0x7ff6d80d02a0 2023-12-19T14:54:44.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.534+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6784+0+0 (secure 0 0 0) 0x7ff6d007f080 con 0x7ff6d80d02a0 2023-12-19T14:54:44.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.542+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff6d00aebe0 con 0x7ff6d80d02a0 2023-12-19T14:54:44.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.686+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 32) v1 ==== 79799+0+0 (secure 0 0 0) 0x7ff6d0003040 con 0x7ff6d80d02a0 2023-12-19T14:54:44.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.686+0000 7ff6beffd700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 0x7ff6c40687d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:44.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.686+0000 7ff6d6ffd700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 0x7ff6c40687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:44.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.690+0000 7ff6d6ffd700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 0x7ff6c40687d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8007130 tx=0x7ff6c8007eb0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:44.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:44.758+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff6d806d540 con 0x7ff6d80d02a0 2023-12-19T14:54:45.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.354+0000 7ff6beffd700 1 -- 172.21.15.50:0/3589628985 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff6d00281d0 con 0x7ff6d80d02a0 2023-12-19T14:54:45.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 msgr2=0x7ff6c40687d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 0x7ff6c40687d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8007130 tx=0x7ff6c8007eb0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 msgr2=0x7ff6d815da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff6d806c010 tx=0x7ff6d0019000 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 shutdown_connections 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff6c40663f0 0x7ff6c40687d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6d80d02a0 0x7ff6d815da90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6d80cfed0 0x7ff6d815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 --2- 172.21.15.50:0/3589628985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6d806c7c0 0x7ff6d8158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 >> 172.21.15.50:0/3589628985 conn(0x7ff6d80670e0 msgr2=0x7ff6d81529e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 shutdown_connections 2023-12-19T14:54:45.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.358+0000 7ff6dd665700 1 -- 172.21.15.50:0/3589628985 wait complete. 2023-12-19T14:54:45.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:54:45.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.606+0000 7ffbbd88e700 1 Processor -- start 2023-12-19T14:54:45.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.606+0000 7ffbbd88e700 1 -- start start 2023-12-19T14:54:45.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd88e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbb80aedc0 0x7ffbb80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd88e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd88e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb806b3d0 con 0x7ffbb806af50 2023-12-19T14:54:45.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd88e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb80cc140 con 0x7ffbb80aedc0 2023-12-19T14:54:45.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd88e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80cb3f0 con 0x7ffbb80cac50 2023-12-19T14:54:45.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbc88c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbb80aedc0 0x7ffbb80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55310/0 (socket says 172.21.15.50:55310) 2023-12-19T14:54:45.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 -- 172.21.15.50:0/2114808520 learned_addr learned my addr 172.21.15.50:0/2114808520 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:45.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 -- 172.21.15.50:0/2114808520 >> v2:172.21.15.50:3300/0 conn(0x7ffbb80aedc0 msgr2=0x7ffbb80af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:45.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 --2- 172.21.15.50:0/2114808520 >> v2:172.21.15.50:3300/0 conn(0x7ffbb80aedc0 0x7ffbb80af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 -- 172.21.15.50:0/2114808520 >> v1:172.21.15.50:6789/0 conn(0x7ffbb806af50 legacy=0x7ffbb806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 -- 172.21.15.50:0/2114808520 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb806d220 con 0x7ffbb80cac50 2023-12-19T14:54:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.610+0000 7ffbbd08d700 1 --2- 172.21.15.50:0/2114808520 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009f20 tx=0x7ffbb0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60cb7f4c9cf89d25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbb77fe700 1 -- 172.21.15.50:0/2114808520 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb000e070 con 0x7ffbb80cac50 2023-12-19T14:54:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbb77fe700 1 -- 172.21.15.50:0/2114808520 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb000b9d0 con 0x7ffbb80cac50 2023-12-19T14:54:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbb77fe700 1 -- 172.21.15.50:0/2114808520 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00092e0 con 0x7ffbb80cac50 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 -- 172.21.15.50:0/2114808520 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 msgr2=0x7ffbb80af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/2114808520 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009f20 tx=0x7ffbb0009840 comp rx=0 tx=0).stop 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 -- 172.21.15.50:0/2114808520 shutdown_connections 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/2114808520 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb80cac50 0x7ffbb80af930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/2114808520 >> v2:172.21.15.50:3300/0 conn(0x7ffbb80aedc0 0x7ffbb80af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 -- 172.21.15.50:0/2114808520 >> 172.21.15.50:0/2114808520 conn(0x7ffbb80cc760 msgr2=0x7ffbb80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 -- 172.21.15.50:0/2114808520 shutdown_connections 2023-12-19T14:54:45.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.614+0000 7ffbbd88e700 1 -- 172.21.15.50:0/2114808520 wait complete. 2023-12-19T14:54:45.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 Processor -- start 2023-12-19T14:54:45.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 -- start start 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb806af50 0x7ffbb8169ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbb80cac50 0x7ffbb806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80c6710 con 0x7ffbb80aedc0 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb816a000 con 0x7ffbb80cac50 2023-12-19T14:54:45.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbd88e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb815a320 con 0x7ffbb806af50 2023-12-19T14:54:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46764/0 (socket says 172.21.15.50:46764) 2023-12-19T14:54:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbbc88c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb806af50 0x7ffbb8169ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.618+0000 7ffbb7fff700 1 -- 172.21.15.50:0/3343474978 learned_addr learned my addr 172.21.15.50:0/3343474978 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:45.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbbd08d700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbb80cac50 0x7ffbb806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 -- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb806af50 msgr2=0x7ffbb8169ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb806af50 0x7ffbb8169ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 -- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbb80cac50 msgr2=0x7ffbb806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbb80cac50 0x7ffbb806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 -- 172.21.15.50:0/3343474978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0009ae0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb7fff700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c4e0 tx=0x7ffba400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:45.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4011590 con 0x7ffbb80aedc0 2023-12-19T14:54:45.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4011710 con 0x7ffbb80aedc0 2023-12-19T14:54:45.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4011ad0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb80c5b00 con 0x7ffbb80aedc0 2023-12-19T14:54:45.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.622+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb80c60c0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.626+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb80c68c0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.626+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 32) v1 ==== 79799+0+0 (secure 0 0 0) 0x7ffba401b3b0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.626+0000 7ffbb4ff9700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 0x7ffb9c062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:45.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.626+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6784+0+0 (secure 0 0 0) 0x7ffba407bad0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.626+0000 7ffbbc88c700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 0x7ffb9c062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:45.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.630+0000 7ffbbc88c700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 0x7ffb9c062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cfbf0 tx=0x7ffbac0091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:45.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.634+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffbb80c68c0 con 0x7ffbb80aedc0 2023-12-19T14:54:45.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:45.810+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffbb806b3d0 con 0x7ffbb80aedc0 2023-12-19T14:54:46.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.378+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ffba404c310 con 0x7ffbb80aedc0 2023-12-19T14:54:46.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.698+0000 7ffbb4ff9700 1 -- 172.21.15.50:0/3343474978 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) v1 ==== 181+0+0 (secure 0 0 0) 0x7ffba40745b0 con 0x7ffbb80aedc0 2023-12-19T14:54:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 msgr2=0x7ffb9c062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 0x7ffb9c062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cfbf0 tx=0x7ffbac0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 msgr2=0x7ffbb8159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c4e0 tx=0x7ffba400c7f0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 shutdown_connections 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbb80cac50 0x7ffbb806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb9c0603a0 0x7ffb9c062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbb80aedc0 0x7ffbb8159de0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 --2- 172.21.15.50:0/3343474978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbb806af50 0x7ffbb8169ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 >> 172.21.15.50:0/3343474978 conn(0x7ffbb80cc760 msgr2=0x7ffbb815c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.706+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 shutdown_connections 2023-12-19T14:54:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.710+0000 7ffbbd88e700 1 -- 172.21.15.50:0/3343474978 wait complete. 2023-12-19T14:54:46.716 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:54:46.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:54:46.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.914+0000 7fdc3ffc5700 1 Processor -- start 2023-12-19T14:54:46.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.914+0000 7fdc3ffc5700 1 -- start start 2023-12-19T14:54:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3ffc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc380ad0f0 0x7fdc380cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3ffc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 0x7fdc3806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3ffc5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc380adf30 con 0x7fdc380acd20 2023-12-19T14:54:46.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3ffc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380cd6f0 con 0x7fdc380ad0f0 2023-12-19T14:54:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3ffc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380cd9d0 con 0x7fdc380af3d0 2023-12-19T14:54:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc380ad0f0 0x7fdc380cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc380ad0f0 0x7fdc380cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37546/0 (socket says 172.21.15.50:37546) 2023-12-19T14:54:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 -- 172.21.15.50:0/838407943 learned_addr learned my addr 172.21.15.50:0/838407943 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3e562700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 0x7fdc3806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 -- 172.21.15.50:0/838407943 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 msgr2=0x7fdc3806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 0x7fdc3806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 -- 172.21.15.50:0/838407943 >> v1:172.21.15.125:6789/0 conn(0x7fdc380acd20 legacy=0x7fdc380cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3d560700 1 -- 172.21.15.50:0/838407943 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc380adbc0 con 0x7fdc380ad0f0 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.918+0000 7fdc3e562700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 0x7fdc3806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3d560700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380ad0f0 0x7fdc380cd080 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e08ef97a1311eef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3cd5f700 1 -- 172.21.15.50:0/838407943 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800b860 con 0x7fdc380ad0f0 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3cd5f700 1 -- 172.21.15.50:0/838407943 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b9c0 con 0x7fdc380ad0f0 2023-12-19T14:54:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3cd5f700 1 -- 172.21.15.50:0/838407943 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800bd60 con 0x7fdc380ad0f0 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/838407943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380ad0f0 msgr2=0x7fdc380cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380ad0f0 0x7fdc380cd080 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).stop 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/838407943 shutdown_connections 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380af3d0 0x7fdc3806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/838407943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380ad0f0 0x7fdc380cd080 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/838407943 >> 172.21.15.50:0/838407943 conn(0x7fdc380cdce0 msgr2=0x7fdc380ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/838407943 shutdown_connections 2023-12-19T14:54:46.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.922+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/838407943 wait complete. 2023-12-19T14:54:46.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 Processor -- start 2023-12-19T14:54:46.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 -- start start 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380acd20 0x7fdc380b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 0x7fdc381436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc381533f0 con 0x7fdc380af3d0 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38167240 con 0x7fdc380ad0f0 2023-12-19T14:54:46.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3ffc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38167520 con 0x7fdc380acd20 2023-12-19T14:54:46.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.926+0000 7fdc3e562700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37552/0 (socket says 172.21.15.50:37552) 2023-12-19T14:54:46.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 -- 172.21.15.50:0/693455856 learned_addr learned my addr 172.21.15.50:0/693455856 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:46.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3dd61700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380acd20 0x7fdc380b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3d560700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 0x7fdc381436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 -- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380acd20 msgr2=0x7fdc380b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380acd20 0x7fdc380b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 -- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 msgr2=0x7fdc381436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 0x7fdc381436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 -- 172.21.15.50:0/693455856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280094f0 con 0x7fdc380af3d0 2023-12-19T14:54:46.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3d560700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 0x7fdc381436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3e562700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d180 tx=0x7fdc30009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000de20 con 0x7fdc380af3d0 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3000f040 con 0x7fdc380af3d0 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30015690 con 0x7fdc380af3d0 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38167800 con 0x7fdc380af3d0 2023-12-19T14:54:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.930+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc380b0fc0 con 0x7fdc380af3d0 2023-12-19T14:54:46.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.934+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38071120 con 0x7fdc380af3d0 2023-12-19T14:54:46.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.934+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdc3001e3b0 con 0x7fdc380af3d0 2023-12-19T14:54:46.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.934+0000 7fdc2dffb700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 0x7fdc1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:46.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.934+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6346+0+0 (secure 0 0 0) 0x7fdc3007d3c0 con 0x7fdc380af3d0 2023-12-19T14:54:46.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.934+0000 7fdc3dd61700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 0x7fdc1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:46.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.938+0000 7fdc3dd61700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 0x7fdc1c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc34003720 tx=0x7fdc34009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:46.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:46.942+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc3004d470 con 0x7fdc380af3d0 2023-12-19T14:54:47.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.114+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc380ad7e0 con 0x7fdc380af3d0 2023-12-19T14:54:47.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.706+0000 7fdc2dffb700 1 -- 172.21.15.50:0/693455856 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc30050a90 con 0x7fdc380af3d0 2023-12-19T14:54:47.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 msgr2=0x7fdc1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 0x7fdc1c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc34003720 tx=0x7fdc34009040 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 msgr2=0x7fdc38156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d180 tx=0x7fdc30009a50 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 shutdown_connections 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc380ad0f0 0x7fdc381436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdc1c060670 0x7fdc1c062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc380af3d0 0x7fdc38156220 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 --2- 172.21.15.50:0/693455856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc380acd20 0x7fdc380b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.710+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 >> 172.21.15.50:0/693455856 conn(0x7fdc380cdce0 msgr2=0x7fdc380af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.714+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 shutdown_connections 2023-12-19T14:54:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.714+0000 7fdc3ffc5700 1 -- 172.21.15.50:0/693455856 wait complete. 2023-12-19T14:54:47.717 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:54:47.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:54:47.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.906+0000 7fd8f080c700 1 Processor -- start 2023-12-19T14:54:47.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.906+0000 7fd8f080c700 1 -- start start 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8f080c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8ec157fc0 0x7fd8ec158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8f080c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd8ec06d050 0x7fd8ec06d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8f080c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0c7820 con 0x7fd8ec1547a0 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8f080c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0ade40 con 0x7fd8ec06d050 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8f080c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec06d9d0 con 0x7fd8ec157fc0 2023-12-19T14:54:47.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ebfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd8ec06d050 0x7fd8ec06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8eb7fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd8ec1547a0 0x7fd8ec0ad730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52600/0 (socket says 172.21.15.50:52600) 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8eb7fe700 1 -- 172.21.15.50:0/272124454 learned_addr learned my addr 172.21.15.50:0/272124454 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8eaffd700 1 --2- 172.21.15.50:0/272124454 >> v2:172.21.15.125:3300/0 conn(0x7fd8ec157fc0 0x7fd8ec158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ebfff700 1 -- 172.21.15.50:0/272124454 >> v2:172.21.15.125:3300/0 conn(0x7fd8ec157fc0 msgr2=0x7fd8ec158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ea7fc700 1 -- 172.21.15.50:0/272124454 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421463634 0 0) 0x7fd8ec0c7820 con 0x7fd8ec1547a0 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ebfff700 1 --2- 172.21.15.50:0/272124454 >> v2:172.21.15.125:3300/0 conn(0x7fd8ec157fc0 0x7fd8ec158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ebfff700 1 -- 172.21.15.50:0/272124454 >> v1:172.21.15.125:6789/0 conn(0x7fd8ec1547a0 legacy=0x7fd8ec0ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.910+0000 7fd8ebfff700 1 -- 172.21.15.50:0/272124454 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0c7210 con 0x7fd8ec06d050 2023-12-19T14:54:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8ebfff700 1 --2- 172.21.15.50:0/272124454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec06d490 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0cd930 tx=0x7fd8e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125d58bb1365fa7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8ea7fc700 1 -- 172.21.15.50:0/272124454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e400e070 con 0x7fd8ec06d050 2023-12-19T14:54:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8ea7fc700 1 -- 172.21.15.50:0/272124454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e400bb60 con 0x7fd8ec06d050 2023-12-19T14:54:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8ea7fc700 1 -- 172.21.15.50:0/272124454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4012430 con 0x7fd8ec06d050 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 -- 172.21.15.50:0/272124454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 msgr2=0x7fd8ec06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 --2- 172.21.15.50:0/272124454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec06d490 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0cd930 tx=0x7fd8e4009780 comp rx=0 tx=0).stop 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 -- 172.21.15.50:0/272124454 shutdown_connections 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 --2- 172.21.15.50:0/272124454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec06d490 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 --2- 172.21.15.50:0/272124454 >> v2:172.21.15.125:3300/0 conn(0x7fd8ec157fc0 0x7fd8ec158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 -- 172.21.15.50:0/272124454 >> 172.21.15.50:0/272124454 conn(0x7fd8ec0ccb30 msgr2=0x7fd8ec0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 -- 172.21.15.50:0/272124454 shutdown_connections 2023-12-19T14:54:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.914+0000 7fd8f080c700 1 -- 172.21.15.50:0/272124454 wait complete. 2023-12-19T14:54:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 Processor -- start 2023-12-19T14:54:47.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 -- start start 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 0x7fd8ec069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 0x7fd8ec0698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec069df0 con 0x7fd8ec06d050 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec16a390 con 0x7fd8ec157fc0 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8f080c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec169dd0 con 0x7fd8ec1547a0 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8eb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8eb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37574/0 (socket says 172.21.15.50:37574) 2023-12-19T14:54:47.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8eb7fe700 1 -- 172.21.15.50:0/2117874527 learned_addr learned my addr 172.21.15.50:0/2117874527 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.918+0000 7fd8eaffd700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 0x7fd8ec069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8ebfff700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 0x7fd8ec0698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 -- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 msgr2=0x7fd8ec069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 0x7fd8ec069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 -- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 msgr2=0x7fd8ec0698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:47.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 0x7fd8ec0698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 -- 172.21.15.50:0/2117874527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4009a20 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8ebfff700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 0x7fd8ec0698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eaffd700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 0x7fd8ec069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8eb7fe700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00c7e0 tx=0x7fd8dc007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc005b30 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc005c90 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec16a0b0 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc007300 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec0cab10 con 0x7fd8ec06d050 2023-12-19T14:54:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.922+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec04be40 con 0x7fd8ec06d050 2023-12-19T14:54:47.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.926+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd8dc01b400 con 0x7fd8ec06d050 2023-12-19T14:54:47.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.926+0000 7fd8d37fe700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 0x7fd8d8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:47.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.926+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7fd8dc07c550 con 0x7fd8ec06d050 2023-12-19T14:54:47.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.926+0000 7fd8eaffd700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 0x7fd8d8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:47.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.930+0000 7fd8eaffd700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 0x7fd8d8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0c4220 tx=0x7fd8e0008040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:47.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:47.934+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8dc04c930 con 0x7fd8ec06d050 2023-12-19T14:54:48.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.110+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd8ec0ae030 con 0x7fd8ec06d050 2023-12-19T14:54:48.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.122+0000 7fd8d37fe700 1 -- 172.21.15.50:0/2117874527 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd8dc04ff50 con 0x7fd8ec06d050 2023-12-19T14:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 msgr2=0x7fd8d8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 0x7fd8d8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0c4220 tx=0x7fd8e0008040 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 msgr2=0x7fd8ec1693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00c7e0 tx=0x7fd8dc007df0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 shutdown_connections 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8ec157fc0 0x7fd8ec0698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd8d8060670 0x7fd8d8062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8ec1547a0 0x7fd8ec069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 --2- 172.21.15.50:0/2117874527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8ec06d050 0x7fd8ec1693f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 >> 172.21.15.50:0/2117874527 conn(0x7fd8ec0ccb30 msgr2=0x7fd8ec0ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 shutdown_connections 2023-12-19T14:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.130+0000 7fd8f080c700 1 -- 172.21.15.50:0/2117874527 wait complete. 2023-12-19T14:54:48.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:54:48.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 Processor -- start 2023-12-19T14:54:48.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 -- start start 2023-12-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41b40cab10 0x7f41b40ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b406b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b40c7450 con 0x7f41b415d410 2023-12-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b406bea0 con 0x7f41b40cab10 2023-12-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.318+0000 7f41b882e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40c6e00 con 0x7f41b406b1b0 2023-12-19T14:54:48.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b406b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b259c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f41b415d410 0x7f41b40ca400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52610/0 (socket says 172.21.15.50:52610) 2023-12-19T14:54:48.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b259c700 1 -- 172.21.15.50:0/2399326898 learned_addr learned my addr 172.21.15.50:0/2399326898 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:48.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41abfff700 1 --2- 172.21.15.50:0/2399326898 >> v2:172.21.15.125:3300/0 conn(0x7f41b40cab10 0x7f41b40ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b1d9b700 1 -- 172.21.15.50:0/2399326898 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254766114 0 0) 0x7f41b40c7450 con 0x7f41b415d410 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b1d9b700 1 -- 172.21.15.50:0/2399326898 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0003570 con 0x7f41b415d410 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 -- 172.21.15.50:0/2399326898 >> v2:172.21.15.125:3300/0 conn(0x7f41b40cab10 msgr2=0x7f41b40ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 --2- 172.21.15.50:0/2399326898 >> v2:172.21.15.125:3300/0 conn(0x7f41b40cab10 0x7f41b40ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 -- 172.21.15.50:0/2399326898 >> v1:172.21.15.125:6789/0 conn(0x7f41b415d410 legacy=0x7f41b40ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 -- 172.21.15.50:0/2399326898 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40c70e0 con 0x7f41b406b1b0 2023-12-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b2d9d700 1 --2- 172.21.15.50:0/2399326898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b406b650 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009e60 tx=0x7f41ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecbbc1278220d0e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b1d9b700 1 -- 172.21.15.50:0/2399326898 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00e070 con 0x7f41b406b1b0 2023-12-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b1d9b700 1 -- 172.21.15.50:0/2399326898 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac00bab0 con 0x7f41b406b1b0 2023-12-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b1d9b700 1 -- 172.21.15.50:0/2399326898 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00be70 con 0x7f41b406b1b0 2023-12-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 -- 172.21.15.50:0/2399326898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 msgr2=0x7f41b406b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 --2- 172.21.15.50:0/2399326898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b406b650 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009e60 tx=0x7f41ac009780 comp rx=0 tx=0).stop 2023-12-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 -- 172.21.15.50:0/2399326898 shutdown_connections 2023-12-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 --2- 172.21.15.50:0/2399326898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b406b650 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 --2- 172.21.15.50:0/2399326898 >> v2:172.21.15.125:3300/0 conn(0x7f41b40cab10 0x7f41b40ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.322+0000 7f41b882e700 1 -- 172.21.15.50:0/2399326898 >> 172.21.15.50:0/2399326898 conn(0x7f41b4067bb0 msgr2=0x7f41b40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.326+0000 7f41b882e700 1 -- 172.21.15.50:0/2399326898 shutdown_connections 2023-12-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.326+0000 7f41b882e700 1 -- 172.21.15.50:0/2399326898 wait complete. 2023-12-19T14:54:48.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.326+0000 7f41b882e700 1 Processor -- start 2023-12-19T14:54:48.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.326+0000 7f41b882e700 1 -- start start 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b40c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41b40cab10 0x7f41b41461c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4146700 con 0x7f41b415d410 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40c4a60 con 0x7f41b40cab10 2023-12-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40c4c90 con 0x7f41b406b1b0 2023-12-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37590/0 (socket says 172.21.15.50:37590) 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 -- 172.21.15.50:0/3323625347 learned_addr learned my addr 172.21.15.50:0/3323625347 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b259c700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b40c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41abfff700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41b40cab10 0x7f41b41461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 -- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 msgr2=0x7f41b40c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b40c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 -- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41b40cab10 msgr2=0x7f41b41461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41b40cab10 0x7f41b41461c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 -- 172.21.15.50:0/3323625347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac009a20 con 0x7f41b415d410 2023-12-19T14:54:48.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b2d9d700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c6f40 tx=0x7f41ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac012e10 con 0x7f41b415d410 2023-12-19T14:54:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac027470 con 0x7f41b415d410 2023-12-19T14:54:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac027830 con 0x7f41b415d410 2023-12-19T14:54:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b40c4f70 con 0x7f41b415d410 2023-12-19T14:54:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.330+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b40c5530 con 0x7f41b415d410 2023-12-19T14:54:48.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.334+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b4071120 con 0x7f41b415d410 2023-12-19T14:54:48.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.334+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f41ac024070 con 0x7f41b415d410 2023-12-19T14:54:48.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.334+0000 7f41aa7fc700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 0x7f4194062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.334+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f41ac07f350 con 0x7f41b415d410 2023-12-19T14:54:48.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.338+0000 7f41b259c700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 0x7f4194062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.338+0000 7f41b259c700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 0x7f4194062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007fd0 tx=0x7f41a4009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.342+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f41ac04f7d0 con 0x7f41b415d410 2023-12-19T14:54:48.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.518+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f41b40ae0a0 con 0x7f41b415d410 2023-12-19T14:54:48.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.530+0000 7f41aa7fc700 1 -- 172.21.15.50:0/3323625347 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v26) v1 ==== 115+0+0 (secure 0 0 0) 0x7f41ac052df0 con 0x7f41b415d410 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 msgr2=0x7f4194062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 0x7f4194062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007fd0 tx=0x7f41a4009040 comp rx=0 tx=0).stop 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 msgr2=0x7f41b40c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c6f40 tx=0x7f41ac019000 comp rx=0 tx=0).stop 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 shutdown_connections 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41b40cab10 0x7f41b41461c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4194060740 0x7f4194062bf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41b415d410 0x7f41b40c2e30 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 --2- 172.21.15.50:0/3323625347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41b406b1b0 0x7f41b40c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 >> 172.21.15.50:0/3323625347 conn(0x7f41b4067bb0 msgr2=0x7f41b40c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:48.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 shutdown_connections 2023-12-19T14:54:48.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.534+0000 7f41b882e700 1 -- 172.21.15.50:0/3323625347 wait complete. 2023-12-19T14:54:48.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:54:48.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 Processor -- start 2023-12-19T14:54:48.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 -- start start 2023-12-19T14:54:48.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 0x7f55a00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55a0159770 0x7f55a006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a006c850 con 0x7f55a006aeb0 2023-12-19T14:54:48.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a006b470 con 0x7f55a0159770 2023-12-19T14:54:48.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.718+0000 7f55a6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a006b750 con 0x7f55a0158c90 2023-12-19T14:54:48.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a4920700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f55a006aeb0 0x7f55a0159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52894/0 (socket says 172.21.15.50:52894) 2023-12-19T14:54:48.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 0x7f55a00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a4920700 1 -- 172.21.15.50:0/1532721646 learned_addr learned my addr 172.21.15.50:0/1532721646 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:48.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5922700 1 --2- 172.21.15.50:0/1532721646 >> v2:172.21.15.125:3300/0 conn(0x7f55a0159770 0x7f55a006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 -- 172.21.15.50:0/1532721646 >> v2:172.21.15.125:3300/0 conn(0x7f55a0159770 msgr2=0x7f55a006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:48.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 --2- 172.21.15.50:0/1532721646 >> v2:172.21.15.125:3300/0 conn(0x7f55a0159770 0x7f55a006cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f5597fff700 1 -- 172.21.15.50:0/1532721646 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926257119 0 0) 0x7f55a006c850 con 0x7f55a006aeb0 2023-12-19T14:54:48.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 -- 172.21.15.50:0/1532721646 >> v1:172.21.15.50:6789/0 conn(0x7f55a006aeb0 legacy=0x7f55a0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 -- 172.21.15.50:0/1532721646 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a006ba30 con 0x7f55a0158c90 2023-12-19T14:54:48.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a5121700 1 --2- 172.21.15.50:0/1532721646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 0x7f55a00caf40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f559c009e60 tx=0x7f559c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57fce9935aa35ffd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f5597fff700 1 -- 172.21.15.50:0/1532721646 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c00e070 con 0x7f55a0158c90 2023-12-19T14:54:48.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f5597fff700 1 -- 172.21.15.50:0/1532721646 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c00bb60 con 0x7f55a0158c90 2023-12-19T14:54:48.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f5597fff700 1 -- 172.21.15.50:0/1532721646 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c012430 con 0x7f55a0158c90 2023-12-19T14:54:48.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 -- 172.21.15.50:0/1532721646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 msgr2=0x7f55a00caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 --2- 172.21.15.50:0/1532721646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 0x7f55a00caf40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f559c009e60 tx=0x7f559c009780 comp rx=0 tx=0).stop 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 -- 172.21.15.50:0/1532721646 shutdown_connections 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 --2- 172.21.15.50:0/1532721646 >> v2:172.21.15.125:3300/0 conn(0x7f55a0159770 0x7f55a006cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 --2- 172.21.15.50:0/1532721646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0158c90 0x7f55a00caf40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.722+0000 7f55a6123700 1 -- 172.21.15.50:0/1532721646 >> 172.21.15.50:0/1532721646 conn(0x7f55a0067bb0 msgr2=0x7f55a00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.726+0000 7f55a6123700 1 -- 172.21.15.50:0/1532721646 shutdown_connections 2023-12-19T14:54:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.726+0000 7f55a6123700 1 -- 172.21.15.50:0/1532721646 wait complete. 2023-12-19T14:54:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.726+0000 7f55a6123700 1 Processor -- start 2023-12-19T14:54:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.726+0000 7f55a6123700 1 -- start start 2023-12-19T14:54:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a0158c90 0x7f55a00b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 0x7f55a0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00b2d60 con 0x7f55a006aeb0 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00b57e0 con 0x7f55a0158c90 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0150e90 con 0x7f55a0159770 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37592/0 (socket says 172.21.15.50:37592) 2023-12-19T14:54:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 -- 172.21.15.50:0/4068011991 learned_addr learned my addr 172.21.15.50:0/4068011991 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5922700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 0x7f55a0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a4920700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a0158c90 0x7f55a00b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 -- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 msgr2=0x7f55a0150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 0x7f55a0150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 -- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a0158c90 msgr2=0x7f55a00b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a0158c90 0x7f55a00b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 -- 172.21.15.50:0/4068011991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c009a20 con 0x7f55a006aeb0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5922700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 0x7f55a0150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55a5121700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f55a006e2e0 tx=0x7f559c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.730+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c012de0 con 0x7f55a006aeb0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c027470 con 0x7f55a006aeb0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a00b2fb0 con 0x7f55a006aeb0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c027830 con 0x7f55a006aeb0 2023-12-19T14:54:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a00b3570 con 0x7f55a006aeb0 2023-12-19T14:54:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a00c9fb0 con 0x7f55a006aeb0 2023-12-19T14:54:48.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.734+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f559c024070 con 0x7f55a006aeb0 2023-12-19T14:54:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.738+0000 7f55957fa700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 0x7f5584062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.738+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f559c07efe0 con 0x7f55a006aeb0 2023-12-19T14:54:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.738+0000 7f55a4920700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 0x7f5584062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.742+0000 7f55a4920700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 0x7f5584062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55a0166ff0 tx=0x7f5590009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:48.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.746+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f55a00c9fb0 con 0x7f55a006aeb0 2023-12-19T14:54:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.918+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f55a00c9fb0 con 0x7f55a006aeb0 2023-12-19T14:54:48.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.926+0000 7f55957fa700 1 -- 172.21.15.50:0/4068011991 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7f55a00c9fb0 con 0x7f55a006aeb0 2023-12-19T14:54:48.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 msgr2=0x7f5584062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 0x7f5584062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55a0166ff0 tx=0x7f5590009040 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 msgr2=0x7f55a016a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f55a006e2e0 tx=0x7f559c019000 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 shutdown_connections 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a0158c90 0x7f55a00b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5584060670 0x7f5584062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a0159770 0x7f55a0150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 --2- 172.21.15.50:0/4068011991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006aeb0 0x7f55a016a060 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 >> 172.21.15.50:0/4068011991 conn(0x7f55a0067bb0 msgr2=0x7f55a00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:48.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 shutdown_connections 2023-12-19T14:54:48.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:48.930+0000 7f55a6123700 1 -- 172.21.15.50:0/4068011991 wait complete. 2023-12-19T14:54:48.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-12-19T14:54:49.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 Processor -- start 2023-12-19T14:54:49.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 -- start start 2023-12-19T14:54:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff98158f70 0x7eff98067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 0x7eff980cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98068c70 con 0x7eff98159340 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff98068060 con 0x7eff98158f70 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff98068340 con 0x7eff98069100 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9d1ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff98158f70 0x7eff98067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9d1ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff98158f70 0x7eff98067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37602/0 (socket says 172.21.15.50:37602) 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9d1ca700 1 -- 172.21.15.50:0/841965822 learned_addr learned my addr 172.21.15.50:0/841965822 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.126+0000 7eff9c9c9700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 0x7eff980cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9d1ca700 1 -- 172.21.15.50:0/841965822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 msgr2=0x7eff980cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:49.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9d1ca700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 0x7eff980cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff8ffff700 1 -- 172.21.15.50:0/841965822 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481354568 0 0) 0x7eff98068c70 con 0x7eff98159340 2023-12-19T14:54:49.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9d1ca700 1 -- 172.21.15.50:0/841965822 >> v1:172.21.15.125:6789/0 conn(0x7eff98159340 legacy=0x7eff980cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9d1ca700 1 -- 172.21.15.50:0/841965822 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98068620 con 0x7eff98158f70 2023-12-19T14:54:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9c9c9700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 0x7eff980cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9d1ca700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98158f70 0x7eff98067980 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7eff940099b0 tx=0x7eff94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b603f7ae24e66a30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff8ffff700 1 -- 172.21.15.50:0/841965822 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9400b860 con 0x7eff98158f70 2023-12-19T14:54:49.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff8ffff700 1 -- 172.21.15.50:0/841965822 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9400b9c0 con 0x7eff98158f70 2023-12-19T14:54:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff8ffff700 1 -- 172.21.15.50:0/841965822 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9400bd60 con 0x7eff98158f70 2023-12-19T14:54:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 -- 172.21.15.50:0/841965822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98158f70 msgr2=0x7eff98067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98158f70 0x7eff98067980 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7eff940099b0 tx=0x7eff94009250 comp rx=0 tx=0).stop 2023-12-19T14:54:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 -- 172.21.15.50:0/841965822 shutdown_connections 2023-12-19T14:54:49.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98069100 0x7eff980cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 --2- 172.21.15.50:0/841965822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98158f70 0x7eff98067980 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 -- 172.21.15.50:0/841965822 >> 172.21.15.50:0/841965822 conn(0x7eff980ccc40 msgr2=0x7eff980cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:49.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 -- 172.21.15.50:0/841965822 shutdown_connections 2023-12-19T14:54:49.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.130+0000 7eff9f42e700 1 -- 172.21.15.50:0/841965822 wait complete. 2023-12-19T14:54:49.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.134+0000 7eff9f42e700 1 Processor -- start 2023-12-19T14:54:49.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.134+0000 7eff9f42e700 1 -- start start 2023-12-19T14:54:49.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff98069100 0x7eff980b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98158f70 0x7eff9815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9816ce30 con 0x7eff98159340 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9816dc50 con 0x7eff98069100 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9816d040 con 0x7eff98158f70 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37618/0 (socket says 172.21.15.50:37618) 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 -- 172.21.15.50:0/3550784274 learned_addr learned my addr 172.21.15.50:0/3550784274 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d1ca700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff98069100 0x7eff980b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9c9c9700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98158f70 0x7eff9815adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 -- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98158f70 msgr2=0x7eff9815adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98158f70 0x7eff9815adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 -- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff98069100 msgr2=0x7eff980b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff98069100 0x7eff980b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 -- 172.21.15.50:0/3550784274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940094f0 con 0x7eff98159340 2023-12-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9d9cb700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c890 tx=0x7eff90001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90012630 con 0x7eff98159340 2023-12-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90012790 con 0x7eff98159340 2023-12-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9816d320 con 0x7eff98159340 2023-12-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90012b50 con 0x7eff98159340 2023-12-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.138+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9816d8e0 con 0x7eff98159340 2023-12-19T14:54:49.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.142+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff98071120 con 0x7eff98159340 2023-12-19T14:54:49.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.142+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7eff9001a3a0 con 0x7eff98159340 2023-12-19T14:54:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.146+0000 7eff8d7fa700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 0x7eff7c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.146+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7eff9007cc30 con 0x7eff98159340 2023-12-19T14:54:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.146+0000 7eff9d1ca700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 0x7eff7c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.146+0000 7eff9d1ca700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 0x7eff7c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff940099b0 tx=0x7eff94017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.150+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7eff9004cf90 con 0x7eff98159340 2023-12-19T14:54:49.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.382+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7eff9816dd90 con 0x7eff98159340 2023-12-19T14:54:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.710+0000 7eff8d7fa700 1 -- 172.21.15.50:0/3550784274 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v586) v1 ==== 156+0+0 (secure 0 0 0) 0x7eff9816dd90 con 0x7eff98159340 2023-12-19T14:54:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 msgr2=0x7eff7c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 0x7eff7c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff940099b0 tx=0x7eff94017000 comp rx=0 tx=0).stop 2023-12-19T14:54:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 msgr2=0x7eff980c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c890 tx=0x7eff90001670 comp rx=0 tx=0).stop 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 shutdown_connections 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7eff7c060670 0x7eff7c062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98159340 0x7eff980c96f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7eff98158f70 0x7eff9815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.714+0000 7eff9f42e700 1 --2- 172.21.15.50:0/3550784274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff98069100 0x7eff980b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.718+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 >> 172.21.15.50:0/3550784274 conn(0x7eff980ccc40 msgr2=0x7eff980c5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.718+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 shutdown_connections 2023-12-19T14:54:49.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.718+0000 7eff9f42e700 1 -- 172.21.15.50:0/3550784274 wait complete. 2023-12-19T14:54:49.741 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-12-19T14:54:49.741 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-12-19T14:54:49.741 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:54:49.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:49.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.910+0000 7f3caf2f0700 1 Processor -- start 2023-12-19T14:54:49.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 -- start start 2023-12-19T14:54:49.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3ca8140710 0x7f3ca80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca806e150 con 0x7f3ca8140ae0 2023-12-19T14:54:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca806d540 con 0x7f3ca8140710 2023-12-19T14:54:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3caf2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca806d820 con 0x7f3ca8068800 2023-12-19T14:54:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cac88b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3ca8140ae0 0x7f3ca80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52902/0 (socket says 172.21.15.50:52902) 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cac88b700 1 -- 172.21.15.50:0/3315121433 learned_addr learned my addr 172.21.15.50:0/3315121433 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad08c700 1 --2- 172.21.15.50:0/3315121433 >> v2:172.21.15.50:3300/0 conn(0x7f3ca8140710 0x7f3ca80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad88d700 1 --2- 172.21.15.50:0/3315121433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad88d700 1 -- 172.21.15.50:0/3315121433 >> v2:172.21.15.50:3300/0 conn(0x7f3ca8140710 msgr2=0x7f3ca80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad88d700 1 --2- 172.21.15.50:0/3315121433 >> v2:172.21.15.50:3300/0 conn(0x7f3ca8140710 0x7f3ca80c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad88d700 1 -- 172.21.15.50:0/3315121433 >> v1:172.21.15.50:6789/0 conn(0x7f3ca8140ae0 legacy=0x7f3ca80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3cad88d700 1 -- 172.21.15.50:0/3315121433 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca806db00 con 0x7f3ca8068800 2023-12-19T14:54:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.914+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3315121433 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169551922 0 0) 0x7f3ca806e150 con 0x7f3ca8140ae0 2023-12-19T14:54:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3cad88d700 1 --2- 172.21.15.50:0/3315121433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80ac250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009e60 tx=0x7f3ca0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7adaf2998da3bce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3315121433 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca000e070 con 0x7f3ca8068800 2023-12-19T14:54:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3315121433 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca000bb60 con 0x7f3ca8068800 2023-12-19T14:54:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3c9ffff700 1 -- 172.21.15.50:0/3315121433 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0012430 con 0x7f3ca8068800 2023-12-19T14:54:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 -- 172.21.15.50:0/3315121433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 msgr2=0x7f3ca80ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/3315121433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80ac250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009e60 tx=0x7f3ca0009780 comp rx=0 tx=0).stop 2023-12-19T14:54:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 -- 172.21.15.50:0/3315121433 shutdown_connections 2023-12-19T14:54:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/3315121433 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80ac250 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/3315121433 >> v2:172.21.15.50:3300/0 conn(0x7f3ca8140710 0x7f3ca80c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 -- 172.21.15.50:0/3315121433 >> 172.21.15.50:0/3315121433 conn(0x7f3ca80c8850 msgr2=0x7f3ca80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:49.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 -- 172.21.15.50:0/3315121433 shutdown_connections 2023-12-19T14:54:49.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.918+0000 7f3caf2f0700 1 -- 172.21.15.50:0/3315121433 wait complete. 2023-12-19T14:54:49.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 Processor -- start 2023-12-19T14:54:49.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 -- start start 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 0x7f3ca813d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca816a0b0 con 0x7f3ca8140ae0 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca816cad0 con 0x7f3ca8140710 2023-12-19T14:54:49.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3caf2f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca816cdb0 con 0x7f3ca8068800 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3cad88d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3cad88d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37638/0 (socket says 172.21.15.50:37638) 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3cad88d700 1 -- 172.21.15.50:0/2546844126 learned_addr learned my addr 172.21.15.50:0/2546844126 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.922+0000 7f3cad08c700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cac88b700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 0x7f3ca813d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 -- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 msgr2=0x7f3ca80caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 -- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 msgr2=0x7f3ca813d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:49.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 0x7f3ca813d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:49.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 -- 172.21.15.50:0/2546844126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0009a20 con 0x7f3ca8140ae0 2023-12-19T14:54:49.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cac88b700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 0x7f3ca813d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:49.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3cad88d700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0013000 tx=0x7f3ca0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0027470 con 0x7f3ca8140ae0 2023-12-19T14:54:49.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca0027630 con 0x7f3ca8140ae0 2023-12-19T14:54:49.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca00279f0 con 0x7f3ca8140ae0 2023-12-19T14:54:49.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca816d090 con 0x7f3ca8140ae0 2023-12-19T14:54:49.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.926+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca80cb1f0 con 0x7f3ca8140ae0 2023-12-19T14:54:49.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.930+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca816c930 con 0x7f3ca8140ae0 2023-12-19T14:54:49.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.930+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3ca0024070 con 0x7f3ca8140ae0 2023-12-19T14:54:49.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.930+0000 7f3c9d7fa700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 0x7f3c8c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:49.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.930+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f3ca007f1a0 con 0x7f3ca8140ae0 2023-12-19T14:54:49.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.934+0000 7f3cad08c700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 0x7f3c8c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:49.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.934+0000 7f3cad08c700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 0x7f3c8c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009d90 tx=0x7f3ca40091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:49.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:49.938+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3ca816c930 con 0x7f3ca8140ae0 2023-12-19T14:54:50.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.198+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ca816c930 con 0x7f3ca8140ae0 2023-12-19T14:54:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.202+0000 7f3c9d7fa700 1 -- 172.21.15.50:0/2546844126 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13452 (secure 0 0 0) 0x7f3ca816c930 con 0x7f3ca8140ae0 2023-12-19T14:54:50.204 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:50.204 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":91,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:47.700117+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6834/3558598512":"2023-12-20T14:54:44.333986+0000","172.21.15.125:0/1777832927":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1579636953":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/3439418072":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/2922501785":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1878386883":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/1822502716":"2023-12-20T14:54:43.668698+0000","172.21.15.125:0/552631771":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3782271172":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/4106781597":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2371597704":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/2154078294":"2023-12-20T14:54:37.258339+0000","172.21.15.50:6835/3558598512":"2023-12-20T14:54:44.333986+0000","172.21.15.125:0/2430752556":"2023-12-20T14:54:37.258339+0000","172.21.15.125:0/3554920661":"2023-12-20T14:54:43.668698+0000","172.21.15.125:6833/33827":"2023-12-20T14:54:37.258339+0000","172.21.15.125:6832/33827":"2023-12-20T14:54:37.258339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:50.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 msgr2=0x7f3c8c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 0x7f3c8c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009d90 tx=0x7f3ca40091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 msgr2=0x7f3ca813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0013000 tx=0x7f3ca0019000 comp rx=0 tx=0).stop 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 shutdown_connections 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3ca8140710 0x7f3ca813d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c8c060670 0x7f3c8c062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ca8140ae0 0x7f3ca813d6b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 --2- 172.21.15.50:0/2546844126 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3ca8068800 0x7f3ca80caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 >> 172.21.15.50:0/2546844126 conn(0x7f3ca80c8850 msgr2=0x7f3ca80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 shutdown_connections 2023-12-19T14:54:50.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.206+0000 7f3caf2f0700 1 -- 172.21.15.50:0/2546844126 wait complete. 2023-12-19T14:54:50.229 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:54:50.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:50.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 Processor -- start 2023-12-19T14:54:50.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 -- start start 2023-12-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55a00c6d20 0x7f55a00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55a006c7c0 0x7f55a006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a006a7a0 con 0x7f55a015aa30 2023-12-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a00c7630 con 0x7f55a006c7c0 2023-12-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.402+0000 7f55a542e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a006d4b0 con 0x7f55a00c6d20 2023-12-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55a006c7c0 0x7f55a006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55a006c7c0 0x7f55a006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37646/0 (socket says 172.21.15.50:37646) 2023-12-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 -- 172.21.15.50:0/4036943187 learned_addr learned my addr 172.21.15.50:0/4036943187 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559effd700 1 --2- 172.21.15.50:0/4036943187 >> v2:172.21.15.125:3300/0 conn(0x7f55a00c6d20 0x7f55a00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 -- 172.21.15.50:0/4036943187 >> v2:172.21.15.125:3300/0 conn(0x7f55a00c6d20 msgr2=0x7f55a00c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 --2- 172.21.15.50:0/4036943187 >> v2:172.21.15.125:3300/0 conn(0x7f55a00c6d20 0x7f55a00c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 -- 172.21.15.50:0/4036943187 >> v1:172.21.15.125:6789/0 conn(0x7f55a015aa30 legacy=0x7f55a015ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 -- 172.21.15.50:0/4036943187 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a006d790 con 0x7f55a006c7c0 2023-12-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559e7fc700 1 --2- 172.21.15.50:0/4036943187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a006cc60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f55a0068030 tx=0x7f5594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af87884718007b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559dffb700 1 -- 172.21.15.50:0/4036943187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5594009e70 con 0x7f55a006c7c0 2023-12-19T14:54:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559dffb700 1 -- 172.21.15.50:0/4036943187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559400b860 con 0x7f55a006c7c0 2023-12-19T14:54:50.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f559dffb700 1 -- 172.21.15.50:0/4036943187 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559400bc00 con 0x7f55a006c7c0 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 -- 172.21.15.50:0/4036943187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 msgr2=0x7f55a006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 --2- 172.21.15.50:0/4036943187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a006cc60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f55a0068030 tx=0x7f5594009250 comp rx=0 tx=0).stop 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 -- 172.21.15.50:0/4036943187 shutdown_connections 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 --2- 172.21.15.50:0/4036943187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a006cc60 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 --2- 172.21.15.50:0/4036943187 >> v2:172.21.15.125:3300/0 conn(0x7f55a00c6d20 0x7f55a00c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 -- 172.21.15.50:0/4036943187 >> 172.21.15.50:0/4036943187 conn(0x7f55a0067bb0 msgr2=0x7f55a00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 -- 172.21.15.50:0/4036943187 shutdown_connections 2023-12-19T14:54:50.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.406+0000 7f55a542e700 1 -- 172.21.15.50:0/4036943187 wait complete. 2023-12-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.410+0000 7f55a542e700 1 Processor -- start 2023-12-19T14:54:50.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.410+0000 7f55a542e700 1 -- start start 2023-12-19T14:54:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a00c6d20 0x7f55a00ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 0x7f55a01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00ad3f0 con 0x7f55a006c7c0 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00ad110 con 0x7f55a00c6d20 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0155800 con 0x7f55a015aa30 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37656/0 (socket says 172.21.15.50:37656) 2023-12-19T14:54:50.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 -- 172.21.15.50:0/444404601 learned_addr learned my addr 172.21.15.50:0/444404601 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559e7fc700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a00c6d20 0x7f55a00ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559f7fe700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 0x7f55a01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 -- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 msgr2=0x7f55a01562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 0x7f55a01562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 -- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a00c6d20 msgr2=0x7f55a00ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a00c6d20 0x7f55a00ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 -- 172.21.15.50:0/444404601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55940094f0 con 0x7f55a006c7c0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559f7fe700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 0x7f55a01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f559effd700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f559000ec80 tx=0x7f55900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559000c850 con 0x7f55a006c7c0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559000c9b0 con 0x7f55a006c7c0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559000cd50 con 0x7f55a006c7c0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.414+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a0155ae0 con 0x7f55a006c7c0 2023-12-19T14:54:50.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.418+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a0154fa0 con 0x7f55a006c7c0 2023-12-19T14:54:50.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.418+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a0155f60 con 0x7f55a006c7c0 2023-12-19T14:54:50.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.418+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5590013680 con 0x7f55a006c7c0 2023-12-19T14:54:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.422+0000 7f5586ffd700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 0x7f558c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.422+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f559007cd70 con 0x7f55a006c7c0 2023-12-19T14:54:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.422+0000 7f559e7fc700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 0x7f558c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.422+0000 7f559e7fc700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 0x7f558c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f55a0068030 tx=0x7f5594017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.430+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f559004d0d0 con 0x7f55a006c7c0 2023-12-19T14:54:50.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.602+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55a006da70 con 0x7f55a006c7c0 2023-12-19T14:54:50.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f5586ffd700 1 -- 172.21.15.50:0/444404601 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55900506f0 con 0x7f55a006c7c0 2023-12-19T14:54:50.607 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:50.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 msgr2=0x7f558c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 0x7f558c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f55a0068030 tx=0x7f5594017000 comp rx=0 tx=0).stop 2023-12-19T14:54:50.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 msgr2=0x7f55a015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f559000ec80 tx=0x7f55900091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 shutdown_connections 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55a00c6d20 0x7f55a00ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f558c060670 0x7f558c062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55a015aa30 0x7f55a01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 --2- 172.21.15.50:0/444404601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55a006c7c0 0x7f55a015d130 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 >> 172.21.15.50:0/444404601 conn(0x7f55a0067bb0 msgr2=0x7f55a00cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 shutdown_connections 2023-12-19T14:54:50.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.606+0000 7f55a542e700 1 -- 172.21.15.50:0/444404601 wait complete. 2023-12-19T14:54:50.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-12-19T14:54:50.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.794+0000 7f4b07385700 1 Processor -- start 2023-12-19T14:54:50.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 -- start start 2023-12-19T14:54:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b000c15b0 0x7f4b0015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000aeba0 0x7f4b000680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000c7190 con 0x7f4b0015d320 2023-12-19T14:54:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b000aefd0 con 0x7f4b000c15b0 2023-12-19T14:54:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b07385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000c1b50 con 0x7f4b000aeba0 2023-12-19T14:54:50.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b04920700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b000c15b0 0x7f4b0015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b05121700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4b0015d320 0x7f4b0006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52644/0 (socket says 172.21.15.50:52644) 2023-12-19T14:54:50.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b05121700 1 -- 172.21.15.50:0/861123950 learned_addr learned my addr 172.21.15.50:0/861123950 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:50.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.798+0000 7f4b05922700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000aeba0 0x7f4b000680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4af7fff700 1 -- 172.21.15.50:0/861123950 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212466140 0 0) 0x7f4b000c7190 con 0x7f4b0015d320 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4af7fff700 1 -- 172.21.15.50:0/861123950 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec003570 con 0x7f4b0015d320 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b04920700 1 -- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000aeba0 msgr2=0x7f4b000680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b04920700 1 --2- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000aeba0 0x7f4b000680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b04920700 1 -- 172.21.15.50:0/861123950 >> v1:172.21.15.125:6789/0 conn(0x7f4b0015d320 legacy=0x7f4b0006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b04920700 1 -- 172.21.15.50:0/861123950 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b000c1e30 con 0x7f4b000c15b0 2023-12-19T14:54:50.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b04920700 1 --2- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000c15b0 0x7f4b0015d6f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4af00098e0 tx=0x7f4af00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac652c821b08219a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4af7fff700 1 -- 172.21.15.50:0/861123950 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0011630 con 0x7f4b000c15b0 2023-12-19T14:54:50.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4af7fff700 1 -- 172.21.15.50:0/861123950 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af0011790 con 0x7f4b000c15b0 2023-12-19T14:54:50.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4af7fff700 1 -- 172.21.15.50:0/861123950 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0011b30 con 0x7f4b000c15b0 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 -- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000c15b0 msgr2=0x7f4b0015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 --2- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000c15b0 0x7f4b0015d6f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4af00098e0 tx=0x7f4af00091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 -- 172.21.15.50:0/861123950 shutdown_connections 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 --2- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000aeba0 0x7f4b000680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 --2- 172.21.15.50:0/861123950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000c15b0 0x7f4b0015d6f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 -- 172.21.15.50:0/861123950 >> 172.21.15.50:0/861123950 conn(0x7f4b000c8850 msgr2=0x7f4b000c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:50.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 -- 172.21.15.50:0/861123950 shutdown_connections 2023-12-19T14:54:50.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.802+0000 7f4b07385700 1 -- 172.21.15.50:0/861123950 wait complete. 2023-12-19T14:54:50.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 Processor -- start 2023-12-19T14:54:50.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 -- start start 2023-12-19T14:54:50.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000aeba0 0x7f4b00154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000c15b0 0x7f4b00160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00161d50 con 0x7f4b0015d320 2023-12-19T14:54:50.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00161140 con 0x7f4b000aeba0 2023-12-19T14:54:50.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.806+0000 7f4b07385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b00161420 con 0x7f4b000c15b0 2023-12-19T14:54:50.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37660/0 (socket says 172.21.15.50:37660) 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 -- 172.21.15.50:0/238660978 learned_addr learned my addr 172.21.15.50:0/238660978 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b04920700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000c15b0 0x7f4b00160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05121700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000aeba0 0x7f4b00154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 -- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000c15b0 msgr2=0x7f4b00160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000c15b0 0x7f4b00160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 -- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000aeba0 msgr2=0x7f4b00154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000aeba0 0x7f4b00154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 -- 172.21.15.50:0/238660978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0009420 con 0x7f4b0015d320 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b05922700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009df0 tx=0x7f4af800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af800caa0 con 0x7f4b0015d320 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af800f040 con 0x7f4b0015d320 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af800ce70 con 0x7f4b0015d320 2023-12-19T14:54:50.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00161700 con 0x7f4b0015d320 2023-12-19T14:54:50.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b001546c0 con 0x7f4b0015d320 2023-12-19T14:54:50.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.810+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b00161ec0 con 0x7f4b0015d320 2023-12-19T14:54:50.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.814+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4af8014690 con 0x7f4b0015d320 2023-12-19T14:54:50.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.814+0000 7f4af57fa700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 0x7f4ae4062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:50.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.814+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f4af807d3a0 con 0x7f4b0015d320 2023-12-19T14:54:50.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.814+0000 7f4b05121700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 0x7f4ae4062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.818+0000 7f4b05121700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 0x7f4ae4062ba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007fd0 tx=0x7f4afc009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:50.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:50.822+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4af8051080 con 0x7f4b0015d320 2023-12-19T14:54:51.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.058+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f4b00071120 con 0x7f4b0015d320 2023-12-19T14:54:51.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.722+0000 7f4af57fa700 1 -- 172.21.15.50:0/238660978 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v588) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4af8050c10 con 0x7f4b0015d320 2023-12-19T14:54:51.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 msgr2=0x7f4ae4062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 0x7f4ae4062ba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007fd0 tx=0x7f4afc009040 comp rx=0 tx=0).stop 2023-12-19T14:54:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 msgr2=0x7f4b0016d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009df0 tx=0x7f4af800b040 comp rx=0 tx=0).stop 2023-12-19T14:54:51.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 shutdown_connections 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ae40606f0 0x7f4ae4062ba0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b0015d320 0x7f4b0016d720 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4b000c15b0 0x7f4b00160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 --2- 172.21.15.50:0/238660978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b000aeba0 0x7f4b00154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 >> 172.21.15.50:0/238660978 conn(0x7f4b000c8850 msgr2=0x7f4b000c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 shutdown_connections 2023-12-19T14:54:51.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.726+0000 7f4b07385700 1 -- 172.21.15.50:0/238660978 wait complete. 2023-12-19T14:54:51.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:54:51.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.922+0000 7f3733e78700 1 Processor -- start 2023-12-19T14:54:51.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.922+0000 7f3733e78700 1 -- start start 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.922+0000 7f3733e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f372c06c7c0 0x7f372c06cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f372c15aa30 0x7f372c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c06a7a0 con 0x7f372c15aa30 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c0c7630 con 0x7f372c06c7c0 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c06d4b0 con 0x7f372c0c6d20 2023-12-19T14:54:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f372c15aa30 0x7f372c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f372c15aa30 0x7f372c15ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37674/0 (socket says 172.21.15.50:37674) 2023-12-19T14:54:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 -- 172.21.15.50:0/1509597985 learned_addr learned my addr 172.21.15.50:0/1509597985 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3731413700 1 --2- 172.21.15.50:0/1509597985 >> v2:172.21.15.125:3300/0 conn(0x7f372c06c7c0 0x7f372c06cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3731c14700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 -- 172.21.15.50:0/1509597985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 msgr2=0x7f372c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 -- 172.21.15.50:0/1509597985 >> v2:172.21.15.125:3300/0 conn(0x7f372c06c7c0 msgr2=0x7f372c06cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 --2- 172.21.15.50:0/1509597985 >> v2:172.21.15.125:3300/0 conn(0x7f372c06c7c0 0x7f372c06cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 -- 172.21.15.50:0/1509597985 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c06d790 con 0x7f372c15aa30 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3732415700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c15aa30 0x7f372c15ae30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f372c068060 tx=0x7f3724009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77914fb1ebcae258 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3730c12700 1 -- 172.21.15.50:0/1509597985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372400e070 con 0x7f372c15aa30 2023-12-19T14:54:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3730c12700 1 -- 172.21.15.50:0/1509597985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372400b970 con 0x7f372c15aa30 2023-12-19T14:54:51.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3730c12700 1 -- 172.21.15.50:0/1509597985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37240092e0 con 0x7f372c15aa30 2023-12-19T14:54:51.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 -- 172.21.15.50:0/1509597985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c15aa30 msgr2=0x7f372c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:51.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.926+0000 7f3733e78700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c15aa30 0x7f372c15ae30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f372c068060 tx=0x7f3724009840 comp rx=0 tx=0).stop 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 -- 172.21.15.50:0/1509597985 shutdown_connections 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c15aa30 0x7f372c15ae30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 --2- 172.21.15.50:0/1509597985 >> v2:172.21.15.125:3300/0 conn(0x7f372c06c7c0 0x7f372c06cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 --2- 172.21.15.50:0/1509597985 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 -- 172.21.15.50:0/1509597985 >> 172.21.15.50:0/1509597985 conn(0x7f372c067bb0 msgr2=0x7f372c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 -- 172.21.15.50:0/1509597985 shutdown_connections 2023-12-19T14:54:51.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 -- 172.21.15.50:0/1509597985 wait complete. 2023-12-19T14:54:51.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 Processor -- start 2023-12-19T14:54:51.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.930+0000 7f3733e78700 1 -- start start 2023-12-19T14:54:51.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c14aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f372c15aa30 0x7f372c0b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c155d60 con 0x7f372c06c7c0 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0aef50 con 0x7f372c15aa30 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0af230 con 0x7f372c0c6d20 2023-12-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37684/0 (socket says 172.21.15.50:37684) 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 -- 172.21.15.50:0/563550030 learned_addr learned my addr 172.21.15.50:0/563550030 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3732415700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f372c15aa30 0x7f372c0b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731413700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c14aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 -- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 msgr2=0x7f372c14aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c14aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 -- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f372c15aa30 msgr2=0x7f372c0b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:51.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f372c15aa30 0x7f372c0b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:51.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 -- 172.21.15.50:0/563550030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724009ae0 con 0x7f372c06c7c0 2023-12-19T14:54:51.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731413700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:51.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3731c14700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f372800ec80 tx=0x7f37280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800c850 con 0x7f372c06c7c0 2023-12-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372800c9b0 con 0x7f372c06c7c0 2023-12-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800cd70 con 0x7f372c06c7c0 2023-12-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.934+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c0af510 con 0x7f372c06c7c0 2023-12-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.938+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c0afad0 con 0x7f372c06c7c0 2023-12-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.938+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c071120 con 0x7f372c06c7c0 2023-12-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.938+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3728013680 con 0x7f372c06c7c0 2023-12-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.942+0000 7f3721ffb700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 0x7f3710062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.942+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f372807cd30 con 0x7f372c06c7c0 2023-12-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.942+0000 7f3731413700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 0x7f3710062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:51.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.942+0000 7f3731413700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 0x7f3710062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f372c154700 tx=0x7f371c008040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:51.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:51.950+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f372804d090 con 0x7f372c06c7c0 2023-12-19T14:54:52.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.122+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f372c0ae6f0 con 0x7f372c06c7c0 2023-12-19T14:54:52.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.134+0000 7f3721ffb700 1 -- 172.21.15.50:0/563550030 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37280506b0 con 0x7f372c06c7c0 2023-12-19T14:54:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 msgr2=0x7f3710062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 0x7f3710062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f372c154700 tx=0x7f371c008040 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 msgr2=0x7f372c14f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f372800ec80 tx=0x7f37280091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 shutdown_connections 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f372c15aa30 0x7f372c0b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3710060670 0x7f3710062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f372c0c6d20 0x7f372c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 --2- 172.21.15.50:0/563550030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f372c06c7c0 0x7f372c14f140 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 >> 172.21.15.50:0/563550030 conn(0x7f372c067bb0 msgr2=0x7f372c157990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 shutdown_connections 2023-12-19T14:54:52.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.138+0000 7f3733e78700 1 -- 172.21.15.50:0/563550030 wait complete. 2023-12-19T14:54:52.166 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:54:52.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:52.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:54:52.261 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:54:52.261 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:54:52.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:54:52.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.442+0000 7f6ad19cc700 1 Processor -- start 2023-12-19T14:54:52.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.442+0000 7f6ad19cc700 1 -- start start 2023-12-19T14:54:52.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad19cc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6acc158e70 0x7f6acc0afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad19cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad19cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0ca5a0 con 0x7f6acc06a540 2023-12-19T14:54:52.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad19cc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc06dac0 con 0x7f6acc158e70 2023-12-19T14:54:52.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad19cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0260 con 0x7f6acc0cc470 2023-12-19T14:54:52.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad09ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6acc158e70 0x7f6acc0afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad09ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6acc158e70 0x7f6acc0afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37692/0 (socket says 172.21.15.50:37692) 2023-12-19T14:54:52.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad09ca700 1 -- 172.21.15.50:0/346948804 learned_addr learned my addr 172.21.15.50:0/346948804 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:52.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.446+0000 7f6ad11cb700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad09ca700 1 -- 172.21.15.50:0/346948804 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 msgr2=0x7f6acc0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:52.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad09ca700 1 -- 172.21.15.50:0/346948804 >> v1:172.21.15.125:6789/0 conn(0x7f6acc06a540 legacy=0x7f6acc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad09ca700 1 -- 172.21.15.50:0/346948804 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc06d220 con 0x7f6acc158e70 2023-12-19T14:54:52.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0afd20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00099b0 tx=0x7f6ac0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58be219df22af34c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6acb7fe700 1 -- 172.21.15.50:0/346948804 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0009ea0 con 0x7f6acc158e70 2023-12-19T14:54:52.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6acb7fe700 1 -- 172.21.15.50:0/346948804 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac000b860 con 0x7f6acc158e70 2023-12-19T14:54:52.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6acb7fe700 1 -- 172.21.15.50:0/346948804 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac000bc00 con 0x7f6acc158e70 2023-12-19T14:54:52.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 -- 172.21.15.50:0/346948804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 msgr2=0x7f6acc0afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0afd20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00099b0 tx=0x7f6ac0009250 comp rx=0 tx=0).stop 2023-12-19T14:54:52.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 -- 172.21.15.50:0/346948804 shutdown_connections 2023-12-19T14:54:52.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/346948804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0afd20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 -- 172.21.15.50:0/346948804 >> 172.21.15.50:0/346948804 conn(0x7f6acc0670e0 msgr2=0x7f6acc0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:52.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 -- 172.21.15.50:0/346948804 shutdown_connections 2023-12-19T14:54:52.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.450+0000 7f6ad19cc700 1 -- 172.21.15.50:0/346948804 wait complete. 2023-12-19T14:54:52.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 Processor -- start 2023-12-19T14:54:52.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 -- start start 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 0x7f6acc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c4090 con 0x7f6acc158e70 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc15bed0 con 0x7f6acc06a540 2023-12-19T14:54:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.454+0000 7f6ad19cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0c3a40 con 0x7f6acc0cc470 2023-12-19T14:54:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad11cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad11cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37706/0 (socket says 172.21.15.50:37706) 2023-12-19T14:54:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad11cb700 1 -- 172.21.15.50:0/543585928 learned_addr learned my addr 172.21.15.50:0/543585928 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 0x7f6acc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6acbfff700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 -- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 msgr2=0x7f6acc0ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 -- 172.21.15.50:0/543585928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 msgr2=0x7f6acc0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 -- 172.21.15.50:0/543585928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00094f0 con 0x7f6acc06a540 2023-12-19T14:54:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad11cb700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad09ca700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 0x7f6acc1554a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0cfd20 tx=0x7f6ac0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0000d10 con 0x7f6acc06a540 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac000bbe0 con 0x7f6acc06a540 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc0c3d20 con 0x7f6acc06a540 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0011650 con 0x7f6acc06a540 2023-12-19T14:54:52.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.458+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc15b180 con 0x7f6acc06a540 2023-12-19T14:54:52.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.462+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc0c3860 con 0x7f6acc06a540 2023-12-19T14:54:52.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.462+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6ac0011a80 con 0x7f6acc06a540 2023-12-19T14:54:52.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.462+0000 7f6ac8ff9700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.462+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f6ac007da50 con 0x7f6acc06a540 2023-12-19T14:54:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.462+0000 7f6acbfff700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.466+0000 7f6acbfff700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 0x7f6ab0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0aedd0 tx=0x7f6ab8007ec0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.474+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ac004ded0 con 0x7f6acc06a540 2023-12-19T14:54:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.686+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6acc0c3b80 con 0x7f6acc06a540 2023-12-19T14:54:52.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.686+0000 7f6ac8ff9700 1 -- 172.21.15.50:0/543585928 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f6ac0015de0 con 0x7f6acc06a540 2023-12-19T14:54:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:52.690 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1002837656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1002837656},{"type":"v1","addr":"172.21.15.125:6836","nonce":1002837656}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1491717860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1491717860},{"type":"v1","addr":"172.21.15.125:6837","nonce":1491717860}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3602695467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3602695467},{"type":"v1","addr":"172.21.15.50:6833","nonce":3602695467}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6834,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1902728546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1902728546},{"type":"v1","addr":"172.21.15.50:6835","nonce":1902728546}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[]} 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 msgr2=0x7f6ab0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 0x7f6ab0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0aedd0 tx=0x7f6ab8007ec0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 msgr2=0x7f6acc1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 0x7f6acc1554a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0cfd20 tx=0x7f6ac0017000 comp rx=0 tx=0).stop 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 shutdown_connections 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6acc158e70 0x7f6acc0cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6acc0cc470 0x7f6acc0ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 --2- 172.21.15.50:0/543585928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc06a540 0x7f6acc1554a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 >> 172.21.15.50:0/543585928 conn(0x7f6acc0670e0 msgr2=0x7f6acc0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:52.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 shutdown_connections 2023-12-19T14:54:52.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.690+0000 7f6ad19cc700 1 -- 172.21.15.50:0/543585928 wait complete. 2023-12-19T14:54:52.695 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 35 2023-12-19T14:54:52.714 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:54:52.715 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:54:52.715 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:54:52.715 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:52.715 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:54:52.715 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:54:52.715 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:54:52.715 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:52.716 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:54:52.716 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:54:52.716 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:54:52.716 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:52.716 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:54:52.716 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:54:52.716 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:54:52.716 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:54:52.734 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:54:52.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:54:52.737 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:54:52.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:54:52.739 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:54:52.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:54:52.741 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:54:52.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:54:52.784 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:54:52.785 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:52.788 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:54:52.788 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:52.791 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:54:52.791 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:52.795 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:54:52.795 DEBUG:teuthology.parallel:result is None 2023-12-19T14:54:52.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:54:52.892 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:54:52.895 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:54:52.900 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:54:52.901 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:54:52.905 INFO:tasks.ceph.mds.d.smithi125.stderr:2023-12-19T14:54:52.898+0000 7fe9523df700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6835/0: (98) Address already in use 2023-12-19T14:54:52.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 Processor -- start 2023-12-19T14:54:52.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 -- start start 2023-12-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe8158e20 0x7f5fe806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5fe81591f0 0x7f5fe80ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe806b730 con 0x7f5fe815ca80 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe806ab20 con 0x7f5fe81591f0 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.970+0000 7f5feee4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe806ae00 con 0x7f5fe8158e20 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5fe81591f0 0x7f5fe80ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5fe81591f0 0x7f5fe80ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37752/0 (socket says 172.21.15.50:37752) 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 -- 172.21.15.50:0/1056706860 learned_addr learned my addr 172.21.15.50:0/1056706860 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fee64b700 1 --1- 172.21.15.50:0/1056706860 >> v1:172.21.15.50:6789/0 conn(0x7f5fe815ca80 0x7f5fe806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52922/0 (socket says 172.21.15.50:52922) 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fede4a700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe8158e20 0x7f5fe806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 -- 172.21.15.50:0/1056706860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe8158e20 msgr2=0x7f5fe806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe8158e20 0x7f5fe806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 -- 172.21.15.50:0/1056706860 >> v1:172.21.15.50:6789/0 conn(0x7f5fe815ca80 legacy=0x7f5fe806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 -- 172.21.15.50:0/1056706860 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe806b0e0 con 0x7f5fe81591f0 2023-12-19T14:54:52.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.974+0000 7f5fed649700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe81591f0 0x7f5fe80ce100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099b0 tx=0x7f5fd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997bbe7bb040ecd3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5fece48700 1 -- 172.21.15.50:0/1056706860 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8009ea0 con 0x7f5fe81591f0 2023-12-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5fece48700 1 -- 172.21.15.50:0/1056706860 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd800b860 con 0x7f5fe81591f0 2023-12-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5fece48700 1 -- 172.21.15.50:0/1056706860 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd800bc00 con 0x7f5fe81591f0 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 -- 172.21.15.50:0/1056706860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe81591f0 msgr2=0x7f5fe80ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe81591f0 0x7f5fe80ce100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099b0 tx=0x7f5fd8009250 comp rx=0 tx=0).stop 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 -- 172.21.15.50:0/1056706860 shutdown_connections 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe81591f0 0x7f5fe80ce100 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1056706860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe8158e20 0x7f5fe806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 -- 172.21.15.50:0/1056706860 >> 172.21.15.50:0/1056706860 conn(0x7f5fe8067bb0 msgr2=0x7f5fe80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 -- 172.21.15.50:0/1056706860 shutdown_connections 2023-12-19T14:54:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.978+0000 7f5feee4c700 1 -- 172.21.15.50:0/1056706860 wait complete. 2023-12-19T14:54:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 Processor -- start 2023-12-19T14:54:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 -- start start 2023-12-19T14:54:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 0x7f5fe80ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe815ca80 0x7f5fe80ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815c230 con 0x7f5fe8158e20 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8169800 con 0x7f5fe815ca80 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5feee4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815b900 con 0x7f5fe81591f0 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37766/0 (socket says 172.21.15.50:37766) 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 -- 172.21.15.50:0/1943024373 learned_addr learned my addr 172.21.15.50:0/1943024373 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fee64b700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe815ca80 0x7f5fe80ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fed649700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 0x7f5fe80ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 -- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 msgr2=0x7f5fe80ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 0x7f5fe80ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 -- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe815ca80 msgr2=0x7f5fe80ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:52.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe815ca80 0x7f5fe80ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fede4a700 1 -- 172.21.15.50:0/1943024373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80094f0 con 0x7f5fe8158e20 2023-12-19T14:54:52.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.982+0000 7f5fed649700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 0x7f5fe80ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5fede4a700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400ec80 tx=0x7f5fe40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400c850 con 0x7f5fe8158e20 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400c9b0 con 0x7f5fe8158e20 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe815bbe0 con 0x7f5fe8158e20 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe400cd50 con 0x7f5fe8158e20 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe815c6c0 con 0x7f5fe8158e20 2023-12-19T14:54:52.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.986+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe804be40 con 0x7f5fe8158e20 2023-12-19T14:54:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.990+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5fe4013680 con 0x7f5fe8158e20 2023-12-19T14:54:52.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.990+0000 7f5fddffb700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 0x7f5fcc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.990+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f5fe407cc80 con 0x7f5fe8158e20 2023-12-19T14:54:52.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.990+0000 7f5fed649700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 0x7f5fcc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:52.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.990+0000 7f5fed649700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 0x7f5fcc062b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099b0 tx=0x7f5fd8017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:52.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:52.994+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5fe404d080 con 0x7f5fe8158e20 2023-12-19T14:54:53.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.178+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5fe806b3c0 con 0x7f5fe8158e20 2023-12-19T14:54:53.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.986+0000 7f5fddffb700 1 -- 172.21.15.50:0/1943024373 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5fe40506a0 con 0x7f5fe8158e20 2023-12-19T14:54:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 msgr2=0x7f5fcc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 0x7f5fcc062b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099b0 tx=0x7f5fd8017000 comp rx=0 tx=0).stop 2023-12-19T14:54:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 msgr2=0x7f5fe81692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400ec80 tx=0x7f5fe40091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:53.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 shutdown_connections 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe815ca80 0x7f5fe80ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5fcc0606a0 0x7f5fcc062b50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5fe81591f0 0x7f5fe80ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 --2- 172.21.15.50:0/1943024373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5fe8158e20 0x7f5fe81692c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 >> 172.21.15.50:0/1943024373 conn(0x7f5fe8067bb0 msgr2=0x7f5fe8157e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.990+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 shutdown_connections 2023-12-19T14:54:53.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:53.994+0000 7f5feee4c700 1 -- 172.21.15.50:0/1943024373 wait complete. 2023-12-19T14:54:53.996 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:54:54.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:54:54.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.194+0000 7f208bfc5700 1 Processor -- start 2023-12-19T14:54:54.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.194+0000 7f208bfc5700 1 -- start start 2023-12-19T14:54:54.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208bfc5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f20841558b0 0x7f2084155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208bfc5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840cb8a0 con 0x7f2084159ba0 2023-12-19T14:54:54.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208bfc5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2084156250 con 0x7f20840ca420 2023-12-19T14:54:54.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208bfc5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20840cac90 con 0x7f20841558b0 2023-12-19T14:54:54.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089d61700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2084159ba0 0x7f20840670e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52928/0 (socket says 172.21.15.50:52928) 2023-12-19T14:54:54.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089d61700 1 -- 172.21.15.50:0/3047436165 learned_addr learned my addr 172.21.15.50:0/3047436165 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:54.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208a562700 1 --2- 172.21.15.50:0/3047436165 >> v2:172.21.15.125:3300/0 conn(0x7f20841558b0 0x7f2084155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647674581 0 0) 0x7f20840cb8a0 con 0x7f2084159ba0 2023-12-19T14:54:54.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2070003570 con 0x7f2084159ba0 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616808737 0 0) 0x7f2070003570 con 0x7f2084159ba0 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 >> v2:172.21.15.125:3300/0 conn(0x7f20841558b0 msgr2=0x7f2084155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 --2- 172.21.15.50:0/3047436165 >> v2:172.21.15.125:3300/0 conn(0x7f20841558b0 0x7f2084155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 >> v1:172.21.15.125:6789/0 conn(0x7f20840ca420 legacy=0x7f2084159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840caf70 con 0x7f2084159ba0 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f20800017f0 con 0x7f2084159ba0 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 >> v1:172.21.15.50:6789/0 conn(0x7f2084159ba0 legacy=0x7f20840670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070003a90 0x7f2070003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 0x7f2070004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2070005210 0x7f207000dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840cac90 con 0x7f2070004810 2023-12-19T14:54:54.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207000e3c0 con 0x7f2070005210 2023-12-19T14:54:54.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f207000e580 con 0x7f2070003a90 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 0x7f2070004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089d61700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070003a90 0x7f2070003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208a562700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2070005210 0x7f207000dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 -- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070003a90 msgr2=0x7f2070003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070003a90 0x7f2070003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 -- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2070005210 msgr2=0x7f207000dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2070005210 0x7f207000dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f2089560700 1 -- 172.21.15.50:0/3047436165 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2070003570 con 0x7f2070004810 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208a562700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2070005210 0x7f207000dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.198+0000 7f208a562700 1 -- 172.21.15.50:0/3047436165 reap_dead start 2023-12-19T14:54:54.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f2089560700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 0x7f2070004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2074001880 tx=0x7f2074007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f792725d84bd16 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074012420 con 0x7f2070004810 2023-12-19T14:54:54.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20740125a0 con 0x7f2070004810 2023-12-19T14:54:54.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f2088d5f700 1 -- 172.21.15.50:0/3047436165 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074012960 con 0x7f2070004810 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 -- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 msgr2=0x7f2070004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 0x7f2070004c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2074001880 tx=0x7f2074007d30 comp rx=0 tx=0).stop 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 -- 172.21.15.50:0/3047436165 shutdown_connections 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3047436165 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2070004810 0x7f2070004c80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 -- 172.21.15.50:0/3047436165 >> 172.21.15.50:0/3047436165 conn(0x7f20840cd990 msgr2=0x7f20840ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 -- 172.21.15.50:0/3047436165 shutdown_connections 2023-12-19T14:54:54.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.202+0000 7f208bfc5700 1 -- 172.21.15.50:0/3047436165 wait complete. 2023-12-19T14:54:54.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 Processor -- start 2023-12-19T14:54:54.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 -- start start 2023-12-19T14:54:54.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 0x7f2084158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2084166730 0x7f2084154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840a6100 con 0x7f2084156470 2023-12-19T14:54:54.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20841572b0 con 0x7f2084166730 2023-12-19T14:54:54.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.206+0000 7f208bfc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2084156a10 con 0x7f2070004810 2023-12-19T14:54:54.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37782/0 (socket says 172.21.15.50:37782) 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 -- 172.21.15.50:0/3892122561 learned_addr learned my addr 172.21.15.50:0/3892122561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089d61700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 0x7f2084158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f208a562700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2084166730 0x7f2084154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 -- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 msgr2=0x7f2084158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 0x7f2084158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 -- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2084166730 msgr2=0x7f2084154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2084166730 0x7f2084154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 -- 172.21.15.50:0/3892122561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084156cf0 con 0x7f2084156470 2023-12-19T14:54:54.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089d61700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 0x7f2084158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2089560700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f207000e390 tx=0x7f2074015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074006820 con 0x7f2084156470 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20740069a0 con 0x7f2084156470 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074006d60 con 0x7f2084156470 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2084156fd0 con 0x7f2084156470 2023-12-19T14:54:54.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.210+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20841680b0 con 0x7f2084156470 2023-12-19T14:54:54.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.214+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20841652d0 con 0x7f2084156470 2023-12-19T14:54:54.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.214+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2074019030 con 0x7f2084156470 2023-12-19T14:54:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.214+0000 7f2079ffb700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 0x7f2068062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.214+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f207407d7c0 con 0x7f2084156470 2023-12-19T14:54:54.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.214+0000 7f2089d61700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 0x7f2068062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.218+0000 7f2089d61700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 0x7f2068062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2080001190 tx=0x7f2080006040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.222+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f207404ddf0 con 0x7f2084156470 2023-12-19T14:54:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.458+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2084071120 con 0x7f2084156470 2023-12-19T14:54:54.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.458+0000 7f2079ffb700 1 -- 172.21.15.50:0/3892122561 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2074017090 con 0x7f2084156470 2023-12-19T14:54:54.460 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.462 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.464 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:54:54.465 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.466 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:54:54.467 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.469 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.470 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:54:54.472 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:54:54.473 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:54:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 msgr2=0x7f2068062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 0x7f2068062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2080001190 tx=0x7f2080006040 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 msgr2=0x7f20841661f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f207000e390 tx=0x7f2074015000 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 shutdown_connections 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2084166730 0x7f2084154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2068060670 0x7f2068062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2084156470 0x7f20841661f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 --2- 172.21.15.50:0/3892122561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2070004810 0x7f2084158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 >> 172.21.15.50:0/3892122561 conn(0x7f20840cd990 msgr2=0x7f20840a7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 shutdown_connections 2023-12-19T14:54:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.462+0000 7f208bfc5700 1 -- 172.21.15.50:0/3892122561 wait complete. 2023-12-19T14:54:54.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:54.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 Processor -- start 2023-12-19T14:54:54.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 -- start start 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b8015d320 0x7f6b8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800c7190 con 0x7f6b800aeba0 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800c1b50 con 0x7f6b800c15b0 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b8642e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c1e30 con 0x7f6b8015d320 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b849cb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6b800aeba0 0x7f6b800680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52940/0 (socket says 172.21.15.50:52940) 2023-12-19T14:54:54.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b849cb700 1 -- 172.21.15.50:0/242998493 learned_addr learned my addr 172.21.15.50:0/242998493 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.642+0000 7f6b7ffff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b8015d320 0x7f6b8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348092004 0 0) 0x7f6b800c7190 con 0x7f6b800aeba0 2023-12-19T14:54:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c003570 con 0x7f6b800aeba0 2023-12-19T14:54:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847414019 0 0) 0x7f6b800c1b50 con 0x7f6b800c15b0 2023-12-19T14:54:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b800c7190 con 0x7f6b800c15b0 2023-12-19T14:54:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2555886001 0 0) 0x7f6b6c003570 con 0x7f6b800aeba0 2023-12-19T14:54:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b8015d320 msgr2=0x7f6b8015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b8015d320 0x7f6b8015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 >> v1:172.21.15.125:6789/0 conn(0x7f6b800c15b0 legacy=0x7f6b8006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800c6e20 con 0x7f6b800aeba0 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f6b780017f0 con 0x7f6b800aeba0 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 >> v1:172.21.15.50:6789/0 conn(0x7f6b800aeba0 legacy=0x7f6b800680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b6c003c20 0x7f6b6c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b6c004810 0x7f6b6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c0051c0 0x7f6b6c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c1e30 con 0x7f6b6c004810 2023-12-19T14:54:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c00c020 con 0x7f6b6c0051c0 2023-12-19T14:54:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c00c1e0 con 0x7f6b6c003c20 2023-12-19T14:54:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b6c004810 0x7f6b6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7ffff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b6c003c20 0x7f6b6c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b849cb700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c0051c0 0x7f6b6c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 -- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b6c003c20 msgr2=0x7f6b6c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b6c003c20 0x7f6b6c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 -- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c0051c0 msgr2=0x7f6b6c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c0051c0 0x7f6b6c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 -- 172.21.15.50:0/242998493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c003570 con 0x7f6b6c004810 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b77fff700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b6c004810 0x7f6b6c004c80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6b68001a10 tx=0x7f6b68007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c022421d13febd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800c5d0 con 0x7f6b6c004810 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6800c730 con 0x7f6b6c004810 2023-12-19T14:54:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.646+0000 7f6b7f7fe700 1 -- 172.21.15.50:0/242998493 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800cad0 con 0x7f6b6c004810 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- 172.21.15.50:0/242998493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b6c004810 msgr2=0x7f6b6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 --2- 172.21.15.50:0/242998493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b6c004810 0x7f6b6c004c80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6b68001a10 tx=0x7f6b68007d30 comp rx=0 tx=0).stop 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b849cb700 1 -- 172.21.15.50:0/242998493 reap_dead start 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- 172.21.15.50:0/242998493 shutdown_connections 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- 172.21.15.50:0/242998493 >> 172.21.15.50:0/242998493 conn(0x7f6b800c8850 msgr2=0x7f6b800c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- 172.21.15.50:0/242998493 shutdown_connections 2023-12-19T14:54:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- 172.21.15.50:0/242998493 wait complete. 2023-12-19T14:54:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 Processor -- start 2023-12-19T14:54:54.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.650+0000 7f6b8642e700 1 -- start start 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c004810 0x7f6b80143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b80159040 0x7f6b801704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c4cd0 con 0x7f6b801594b0 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c40c0 con 0x7f6b6c004810 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800c43a0 con 0x7f6b80159040 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37804/0 (socket says 172.21.15.50:37804) 2023-12-19T14:54:54.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 -- 172.21.15.50:0/756320169 learned_addr learned my addr 172.21.15.50:0/756320169 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:54.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b7ffff700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c004810 0x7f6b80143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b77fff700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b80159040 0x7f6b801704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 -- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b80159040 msgr2=0x7f6b801704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b80159040 0x7f6b801704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 -- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c004810 msgr2=0x7f6b80143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c004810 0x7f6b80143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 -- 172.21.15.50:0/756320169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800c4680 con 0x7f6b801594b0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b849cb700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006a850 tx=0x7f6b78002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800ac30 con 0x7f6b801594b0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7800adb0 con 0x7f6b801594b0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b800c4960 con 0x7f6b801594b0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78013640 con 0x7f6b801594b0 2023-12-19T14:54:54.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.654+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b800c3e80 con 0x7f6b801594b0 2023-12-19T14:54:54.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.658+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b800c4e40 con 0x7f6b801594b0 2023-12-19T14:54:54.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.658+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6b78013a70 con 0x7f6b801594b0 2023-12-19T14:54:54.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.662+0000 7f6b7cff9700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 0x7f6b60062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:54.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.662+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f6b7807ce40 con 0x7f6b801594b0 2023-12-19T14:54:54.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.662+0000 7f6b7ffff700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 0x7f6b60062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:54.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.662+0000 7f6b7ffff700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 0x7f6b60062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cbd40 tx=0x7f6b700091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:54.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.666+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6b7804d470 con 0x7f6b801594b0 2023-12-19T14:54:54.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.842+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b800c3e80 con 0x7f6b801594b0 2023-12-19T14:54:54.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.846+0000 7f6b7cff9700 1 -- 172.21.15.50:0/756320169 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+12383 (secure 0 0 0) 0x7f6b800c3e80 con 0x7f6b801594b0 2023-12-19T14:54:54.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:54.848 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":96,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:53.977871+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.846+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 msgr2=0x7f6b60062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.846+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 0x7f6b60062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cbd40 tx=0x7f6b700091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.846+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 msgr2=0x7f6b801709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.846+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6b8006a850 tx=0x7f6b78002170 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 shutdown_connections 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b600606a0 0x7f6b60062b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b801594b0 0x7f6b801709e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b80159040 0x7f6b801704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 --2- 172.21.15.50:0/756320169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b6c004810 0x7f6b80143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 >> 172.21.15.50:0/756320169 conn(0x7f6b800c8850 msgr2=0x7f6b800c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 shutdown_connections 2023-12-19T14:54:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:54.850+0000 7f6b8642e700 1 -- 172.21.15.50:0/756320169 wait complete. 2023-12-19T14:54:54.874 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:54:54.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:54:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.038+0000 7fa4fa4e3700 1 Processor -- start 2023-12-19T14:54:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.038+0000 7fa4fa4e3700 1 -- start start 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.042+0000 7fa4fa4e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f4158c90 0x7fa4f40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.042+0000 7fa4fa4e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4f406aeb0 0x7fa4f4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.042+0000 7fa4fa4e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f406c850 con 0x7fa4f41595a0 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.042+0000 7fa4fa4e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f406b470 con 0x7fa4f406aeb0 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.042+0000 7fa4fa4e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f406b750 con 0x7fa4f4158c90 2023-12-19T14:54:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4f406aeb0 0x7fa4f4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f94e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f4158c90 0x7fa4f40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4f406aeb0 0x7fa4f4159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:47084/0 (socket says 172.21.15.50:47084) 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 -- 172.21.15.50:0/4110258662 learned_addr learned my addr 172.21.15.50:0/4110258662 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 -- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f4158c90 msgr2=0x7fa4f40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 --2- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f4158c90 0x7fa4f40caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 -- 172.21.15.50:0/4110258662 >> v1:172.21.15.125:6789/0 conn(0x7fa4f41595a0 legacy=0x7fa4f415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 -- 172.21.15.50:0/4110258662 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f406ba30 con 0x7fa4f406aeb0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4f8ce0700 1 --2- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f406aeb0 0x7fa4f4159060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40099b0 tx=0x7fa4e4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78c19ebb7c3398cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4ebfff700 1 -- 172.21.15.50:0/4110258662 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4009ea0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4ebfff700 1 -- 172.21.15.50:0/4110258662 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e400b860 con 0x7fa4f406aeb0 2023-12-19T14:54:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4ebfff700 1 -- 172.21.15.50:0/4110258662 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400bc00 con 0x7fa4f406aeb0 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f406aeb0 msgr2=0x7fa4f4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f406aeb0 0x7fa4f4159060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40099b0 tx=0x7fa4e4009250 comp rx=0 tx=0).stop 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/4110258662 shutdown_connections 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f406aeb0 0x7fa4f4159060 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/4110258662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f4158c90 0x7fa4f40caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/4110258662 >> 172.21.15.50:0/4110258662 conn(0x7fa4f4067bb0 msgr2=0x7fa4f40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/4110258662 shutdown_connections 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.046+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/4110258662 wait complete. 2023-12-19T14:54:55.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.050+0000 7fa4fa4e3700 1 Processor -- start 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.050+0000 7fa4fa4e3700 1 -- start start 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f4158c90 0x7fa4f416a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f41595a0 0x7fa4f40c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f416a7a0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f40c8a70 con 0x7fa4f4158c90 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f40af5c0 con 0x7fa4f41595a0 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37820/0 (socket says 172.21.15.50:37820) 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 -- 172.21.15.50:0/302883477 learned_addr learned my addr 172.21.15.50:0/302883477 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f8ce0700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f4158c90 0x7fa4f416a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f9ce2700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f41595a0 0x7fa4f40c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 -- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f41595a0 msgr2=0x7fa4f40c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f41595a0 0x7fa4f40c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 -- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f4158c90 msgr2=0x7fa4f416a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f4158c90 0x7fa4f416a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 -- 172.21.15.50:0/302883477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e40094f0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4f94e1700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000ec80 tx=0x7fa4f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000c850 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f000c9b0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000cd50 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f4157d10 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.054+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f40c8fb0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.058+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f404be40 con 0x7fa4f406aeb0 2023-12-19T14:54:55.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.058+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa4f0013680 con 0x7fa4f406aeb0 2023-12-19T14:54:55.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.062+0000 7fa4e97fa700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 0x7fa4d8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.062+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa4f007c9f0 con 0x7fa4f406aeb0 2023-12-19T14:54:55.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.062+0000 7fa4f8ce0700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 0x7fa4d8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.062+0000 7fa4f8ce0700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 0x7fa4d8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40099b0 tx=0x7fa4e4017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.066+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa4f004d020 con 0x7fa4f406aeb0 2023-12-19T14:54:55.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.242+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4f406bd10 con 0x7fa4f406aeb0 2023-12-19T14:54:55.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4e97fa700 1 -- 172.21.15.50:0/302883477 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4f0050640 con 0x7fa4f406aeb0 2023-12-19T14:54:55.248 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:54:55.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 msgr2=0x7fa4d8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 0x7fa4d8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40099b0 tx=0x7fa4e4017000 comp rx=0 tx=0).stop 2023-12-19T14:54:55.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 msgr2=0x7fa4f40c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000ec80 tx=0x7fa4f00091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 shutdown_connections 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4f4158c90 0x7fa4f416a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa4d8060670 0x7fa4d8062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4f41595a0 0x7fa4f40c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.246+0000 7fa4fa4e3700 1 --2- 172.21.15.50:0/302883477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4f406aeb0 0x7fa4f40c97b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.250+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 >> 172.21.15.50:0/302883477 conn(0x7fa4f4067bb0 msgr2=0x7fa4f40afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.250+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 shutdown_connections 2023-12-19T14:54:55.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.250+0000 7fa4fa4e3700 1 -- 172.21.15.50:0/302883477 wait complete. 2023-12-19T14:54:55.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:54:55.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.434+0000 7fe2f1ce3700 1 Processor -- start 2023-12-19T14:54:55.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.434+0000 7fe2f1ce3700 1 -- start start 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f1ce3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f1ce3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2ec06b900 0x7fe2ec06da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f1ce3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec06b470 con 0x7fe2ec15cc20 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f1ce3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2ec15d790 con 0x7fe2ec0c6900 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f1ce3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2ec161d10 con 0x7fe2ec06b900 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37832/0 (socket says 172.21.15.50:37832) 2023-12-19T14:54:55.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 -- 172.21.15.50:0/3278075644 learned_addr learned my addr 172.21.15.50:0/3278075644 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:55.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2e3fff700 1 --2- 172.21.15.50:0/3278075644 >> v2:172.21.15.125:3300/0 conn(0x7fe2ec06b900 0x7fe2ec06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 -- 172.21.15.50:0/3278075644 >> v2:172.21.15.125:3300/0 conn(0x7fe2ec06b900 msgr2=0x7fe2ec06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:55.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 --2- 172.21.15.50:0/3278075644 >> v2:172.21.15.125:3300/0 conn(0x7fe2ec06b900 0x7fe2ec06da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 -- 172.21.15.50:0/3278075644 >> v1:172.21.15.50:6789/0 conn(0x7fe2ec15cc20 legacy=0x7fe2ec15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 -- 172.21.15.50:0/3278075644 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec06bd70 con 0x7fe2ec0c6900 2023-12-19T14:54:55.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2e37fe700 1 -- 172.21.15.50:0/3278075644 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317087685 0 0) 0x7fe2ec06b470 con 0x7fe2ec15cc20 2023-12-19T14:54:55.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2f0ce1700 1 --2- 172.21.15.50:0/3278075644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06a030 tx=0x7fe2e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa9b48c2c7490b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2e37fe700 1 -- 172.21.15.50:0/3278075644 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800b860 con 0x7fe2ec0c6900 2023-12-19T14:54:55.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2e37fe700 1 -- 172.21.15.50:0/3278075644 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e800b9c0 con 0x7fe2ec0c6900 2023-12-19T14:54:55.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.438+0000 7fe2e37fe700 1 -- 172.21.15.50:0/3278075644 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800bd60 con 0x7fe2ec0c6900 2023-12-19T14:54:55.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3278075644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec0c6900 msgr2=0x7fe2ec0c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3278075644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06a030 tx=0x7fe2e8009250 comp rx=0 tx=0).stop 2023-12-19T14:54:55.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3278075644 shutdown_connections 2023-12-19T14:54:55.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3278075644 >> v2:172.21.15.125:3300/0 conn(0x7fe2ec06b900 0x7fe2ec06da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3278075644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec0c6900 0x7fe2ec0c6cd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3278075644 >> 172.21.15.50:0/3278075644 conn(0x7fe2ec067bb0 msgr2=0x7fe2ec0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:55.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3278075644 shutdown_connections 2023-12-19T14:54:55.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3278075644 wait complete. 2023-12-19T14:54:55.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 Processor -- start 2023-12-19T14:54:55.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.442+0000 7fe2f1ce3700 1 -- start start 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2ec06b900 0x7fe2ec15e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2ec0c6900 0x7fe2ec15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec16e9b0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0c0c60 con 0x7fe2ec06b900 2023-12-19T14:54:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec15f190 con 0x7fe2ec0c6900 2023-12-19T14:54:55.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37836/0 (socket says 172.21.15.50:37836) 2023-12-19T14:54:55.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 -- 172.21.15.50:0/3930391598 learned_addr learned my addr 172.21.15.50:0/3930391598 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:55.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2e3fff700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2ec0c6900 0x7fe2ec15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f0ce1700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2ec06b900 0x7fe2ec15e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 -- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2ec0c6900 msgr2=0x7fe2ec15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2ec0c6900 0x7fe2ec15eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 -- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2ec06b900 msgr2=0x7fe2ec15e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2ec06b900 0x7fe2ec15e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 -- 172.21.15.50:0/3930391598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80094f0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f14e2700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c890 tx=0x7fe2e4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4012630 con 0x7fe2ec15cc20 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4012790 con 0x7fe2ec15cc20 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec15f470 con 0x7fe2ec15cc20 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4012b50 con 0x7fe2ec15cc20 2023-12-19T14:54:55.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.446+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec16ecb0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.450+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec0cb5f0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.450+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe2e401a3a0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.450+0000 7fe2e0ff9700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 0x7fe2d8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.450+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fe2e407c980 con 0x7fe2ec15cc20 2023-12-19T14:54:55.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.450+0000 7fe2f0ce1700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 0x7fe2d8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.454+0000 7fe2f0ce1700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 0x7fe2d8062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06a030 tx=0x7fe2e8017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.458+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2ec0cb5f0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.630+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe2ec0cb5f0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.630+0000 7fe2e0ff9700 1 -- 172.21.15.50:0/3930391598 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe2ec0cb5f0 con 0x7fe2ec15cc20 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:54:55.634 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:54:55.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 msgr2=0x7fe2d8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 0x7fe2d8062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06a030 tx=0x7fe2e8017000 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 msgr2=0x7fe2ec16d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c890 tx=0x7fe2e4001670 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 shutdown_connections 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2d8060670 0x7fe2d8062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2ec15cc20 0x7fe2ec16d7e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2ec0c6900 0x7fe2ec15eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 --2- 172.21.15.50:0/3930391598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2ec06b900 0x7fe2ec15e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 >> 172.21.15.50:0/3930391598 conn(0x7fe2ec067bb0 msgr2=0x7fe2ec159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 shutdown_connections 2023-12-19T14:54:55.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.634+0000 7fe2f1ce3700 1 -- 172.21.15.50:0/3930391598 wait complete. 2023-12-19T14:54:55.657 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:54:55.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:54:55.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.830+0000 7fee988a5700 1 Processor -- start 2023-12-19T14:54:55.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.834+0000 7fee988a5700 1 -- start start 2023-12-19T14:54:55.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee988a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee940cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee988a5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee940b0ac0 con 0x7fee9406a6a0 2023-12-19T14:54:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee988a5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9406b210 con 0x7fee9415c930 2023-12-19T14:54:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee988a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee940ca810 con 0x7fee940cb130 2023-12-19T14:54:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fee9406a6a0 0x7fee9415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52952/0 (socket says 172.21.15.50:52952) 2023-12-19T14:54:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 -- 172.21.15.50:0/2316552764 learned_addr learned my addr 172.21.15.50:0/2316552764 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:55.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9259c700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee940cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227487761 0 0) 0x7fee940b0ac0 con 0x7fee9406a6a0 2023-12-19T14:54:55.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee7c003570 con 0x7fee9406a6a0 2023-12-19T14:54:55.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292517143 0 0) 0x7fee9406b210 con 0x7fee9415c930 2023-12-19T14:54:55.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee940b0ac0 con 0x7fee9415c930 2023-12-19T14:54:55.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437199917 0 0) 0x7fee7c003570 con 0x7fee9406a6a0 2023-12-19T14:54:55.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 msgr2=0x7fee940cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee940cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 >> v1:172.21.15.125:6789/0 conn(0x7fee9415c930 legacy=0x7fee940b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940b0190 con 0x7fee9406a6a0 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fee88001470 con 0x7fee9406a6a0 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 >> v1:172.21.15.50:6789/0 conn(0x7fee9406a6a0 legacy=0x7fee9415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee7c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee7c004180 0x7fee7c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee7c004d50 0x7fee7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940ca810 con 0x7fee7c004180 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c00bba0 con 0x7fee7c004d50 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c00bd60 con 0x7fee940cb130 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee7c004180 0x7fee7c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee9259c700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee7c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee92d9d700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee7c004d50 0x7fee7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 -- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 msgr2=0x7fee7c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee940cb130 0x7fee7c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 -- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee7c004d50 msgr2=0x7fee7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee7c004d50 0x7fee7c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee91d9b700 1 -- 172.21.15.50:0/2316552764 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c003570 con 0x7fee7c004180 2023-12-19T14:54:55.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.838+0000 7fee92d9d700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee7c004d50 0x7fee7c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:55.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee91d9b700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee7c004180 0x7fee7c004810 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fee8800c2d0 tx=0x7fee88001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bdb648eb1e5eed server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8800cba0 con 0x7fee7c004180 2023-12-19T14:54:55.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8800cd20 con 0x7fee7c004180 2023-12-19T14:54:55.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee9159a700 1 -- 172.21.15.50:0/2316552764 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88005b70 con 0x7fee7c004180 2023-12-19T14:54:55.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 -- 172.21.15.50:0/2316552764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee7c004180 msgr2=0x7fee7c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 --2- 172.21.15.50:0/2316552764 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee7c004180 0x7fee7c004810 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fee8800c2d0 tx=0x7fee88001e10 comp rx=0 tx=0).stop 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee92d9d700 1 -- 172.21.15.50:0/2316552764 reap_dead start 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 -- 172.21.15.50:0/2316552764 shutdown_connections 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 -- 172.21.15.50:0/2316552764 >> 172.21.15.50:0/2316552764 conn(0x7fee94067bb0 msgr2=0x7fee940cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 -- 172.21.15.50:0/2316552764 shutdown_connections 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.842+0000 7fee988a5700 1 -- 172.21.15.50:0/2316552764 wait complete. 2023-12-19T14:54:55.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 Processor -- start 2023-12-19T14:54:55.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 -- start start 2023-12-19T14:54:55.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 0x7fee940b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 0x7fee94168720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940b1ef0 con 0x7fee94157a60 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee94157040 con 0x7fee940b1a50 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee988a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee94156430 con 0x7fee7c004180 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37868/0 (socket says 172.21.15.50:37868) 2023-12-19T14:54:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 -- 172.21.15.50:0/670779895 learned_addr learned my addr 172.21.15.50:0/670779895 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee92d9d700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 0x7fee94168720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee9259c700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 0x7fee940b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 -- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 msgr2=0x7fee940b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 0x7fee940b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 -- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 msgr2=0x7fee94168720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 0x7fee94168720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 -- 172.21.15.50:0/670779895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94156710 con 0x7fee94157a60 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee9259c700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 0x7fee940b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee92d9d700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 0x7fee94168720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:55.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee91d9b700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fee88007c90 tx=0x7fee88015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.846+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8800ce90 con 0x7fee94157a60 2023-12-19T14:54:55.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee88001060 con 0x7fee94157a60 2023-12-19T14:54:55.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee941569f0 con 0x7fee94157a60 2023-12-19T14:54:55.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88007530 con 0x7fee94157a60 2023-12-19T14:54:55.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee94156100 con 0x7fee94157a60 2023-12-19T14:54:55.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9416c3d0 con 0x7fee94157a60 2023-12-19T14:54:55.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.850+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fee88019030 con 0x7fee94157a60 2023-12-19T14:54:55.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.854+0000 7fee827fc700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 0x7fee74062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:55.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.854+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fee8807da80 con 0x7fee94157a60 2023-12-19T14:54:55.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.854+0000 7fee9259c700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 0x7fee74062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:55.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.854+0000 7fee9259c700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 0x7fee74062b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee84006dc0 tx=0x7fee84008040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:55.858+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fee9416c3d0 con 0x7fee94157a60 2023-12-19T14:54:56.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.030+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fee9416c3d0 con 0x7fee94157a60 2023-12-19T14:54:56.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.770+0000 7fee827fc700 1 -- 172.21.15.50:0/670779895 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v97) v1 ==== 138+0+0 (secure 0 0 0) 0x7fee9416c3d0 con 0x7fee94157a60 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 msgr2=0x7fee74062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 0x7fee74062b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee84006dc0 tx=0x7fee84008040 comp rx=0 tx=0).stop 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 msgr2=0x7fee940b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fee88007c90 tx=0x7fee88015000 comp rx=0 tx=0).stop 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 shutdown_connections 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee940b1a50 0x7fee94168720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fee740606a0 0x7fee74062b50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee94157a60 0x7fee940b1510 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 --2- 172.21.15.50:0/670779895 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fee7c004180 0x7fee940b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.778+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 >> 172.21.15.50:0/670779895 conn(0x7fee94067bb0 msgr2=0x7fee9414eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:56.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.782+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 shutdown_connections 2023-12-19T14:54:56.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.782+0000 7fee988a5700 1 -- 172.21.15.50:0/670779895 wait complete. 2023-12-19T14:54:56.787 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:54:56.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:54:56.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.978+0000 7f5af362e700 1 Processor -- start 2023-12-19T14:54:56.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.978+0000 7f5af362e700 1 -- start start 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5aec0afab0 0x7f5aec06b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5aec06b5a0 0x7f5aec15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec06ba20 con 0x7f5aec06b5a0 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec1598a0 con 0x7f5aec0afab0 2023-12-19T14:54:56.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.982+0000 7f5af362e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec159b80 con 0x7f5aec158e90 2023-12-19T14:54:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5aec06b5a0 0x7f5aec15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5aec06b5a0 0x7f5aec15cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54050/0 (socket says 172.21.15.50:54050) 2023-12-19T14:54:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 -- 172.21.15.50:0/144642050 learned_addr learned my addr 172.21.15.50:0/144642050 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af0bc9700 1 --2- 172.21.15.50:0/144642050 >> v2:172.21.15.125:3300/0 conn(0x7f5aec0afab0 0x7f5aec06b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af13ca700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 -- 172.21.15.50:0/144642050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 msgr2=0x7f5aec159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 -- 172.21.15.50:0/144642050 >> v2:172.21.15.125:3300/0 conn(0x7f5aec0afab0 msgr2=0x7f5aec06b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 --2- 172.21.15.50:0/144642050 >> v2:172.21.15.125:3300/0 conn(0x7f5aec0afab0 0x7f5aec06b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 -- 172.21.15.50:0/144642050 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec159e60 con 0x7f5aec06b5a0 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af13ca700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:54:56.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af1bcb700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec15cbe0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06a4b0 tx=0x7f5ae4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d80f02d823443c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:56.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5ae3fff700 1 -- 172.21.15.50:0/144642050 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400e070 con 0x7f5aec06b5a0 2023-12-19T14:54:56.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5ae3fff700 1 -- 172.21.15.50:0/144642050 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae400ba20 con 0x7f5aec06b5a0 2023-12-19T14:54:56.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5ae3fff700 1 -- 172.21.15.50:0/144642050 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40092e0 con 0x7f5aec06b5a0 2023-12-19T14:54:56.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 -- 172.21.15.50:0/144642050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 msgr2=0x7f5aec15cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:56.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec15cbe0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06a4b0 tx=0x7f5ae4009840 comp rx=0 tx=0).stop 2023-12-19T14:54:56.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 -- 172.21.15.50:0/144642050 shutdown_connections 2023-12-19T14:54:56.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec15cbe0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 --2- 172.21.15.50:0/144642050 >> v2:172.21.15.125:3300/0 conn(0x7f5aec0afab0 0x7f5aec06b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 --2- 172.21.15.50:0/144642050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 -- 172.21.15.50:0/144642050 >> 172.21.15.50:0/144642050 conn(0x7f5aec0cc580 msgr2=0x7f5aec0cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:56.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.986+0000 7f5af362e700 1 -- 172.21.15.50:0/144642050 shutdown_connections 2023-12-19T14:54:56.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.990+0000 7f5af362e700 1 -- 172.21.15.50:0/144642050 wait complete. 2023-12-19T14:54:56.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.990+0000 7f5af362e700 1 Processor -- start 2023-12-19T14:54:56.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.990+0000 7f5af362e700 1 -- start start 2023-12-19T14:54:56.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aec0afab0 0x7f5aec0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec0aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:56.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0cac80 con 0x7f5aec06b5a0 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0c8f80 con 0x7f5aec0afab0 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af362e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0c9260 con 0x7f5aec158e90 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54058/0 (socket says 172.21.15.50:54058) 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 -- 172.21.15.50:0/461119717 learned_addr learned my addr 172.21.15.50:0/461119717 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af0bc9700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aec0afab0 0x7f5aec0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af1bcb700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec0aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 -- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 msgr2=0x7f5aec0aa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec0aa9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 -- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aec0afab0 msgr2=0x7f5aec0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aec0afab0 0x7f5aec0ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 -- 172.21.15.50:0/461119717 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009ae0 con 0x7f5aec06b5a0 2023-12-19T14:54:56.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5af13ca700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800e8d0 tx=0x7f5ae8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:56.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.994+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8013410 con 0x7f5aec06b5a0 2023-12-19T14:54:56.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.998+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8013570 con 0x7f5aec06b5a0 2023-12-19T14:54:56.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.998+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0c9540 con 0x7f5aec06b5a0 2023-12-19T14:54:56.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.998+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8013910 con 0x7f5aec06b5a0 2023-12-19T14:54:56.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.998+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec0c9b00 con 0x7f5aec06b5a0 2023-12-19T14:54:57.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:56.998+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec071120 con 0x7f5aec06b5a0 2023-12-19T14:54:57.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.002+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5ae8013d40 con 0x7f5aec06b5a0 2023-12-19T14:54:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.002+0000 7f5ae17fa700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 0x7f5ad0062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.002+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5571+0+0 (secure 0 0 0) 0x7f5ae807c690 con 0x7f5aec06b5a0 2023-12-19T14:54:57.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.002+0000 7f5af0bc9700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 0x7f5ad0062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:57.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.006+0000 7f5af0bc9700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 0x7f5ad0062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06fda0 tx=0x7f5adc0091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:57.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.010+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ae804ca40 con 0x7f5aec06b5a0 2023-12-19T14:54:57.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.182+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5aec06f090 con 0x7f5aec06b5a0 2023-12-19T14:54:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.790+0000 7f5ae17fa700 1 -- 172.21.15.50:0/461119717 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v98) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5ae8050060 con 0x7f5aec06b5a0 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 msgr2=0x7f5ad0062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 0x7f5ad0062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06fda0 tx=0x7f5adc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 msgr2=0x7f5aec155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800e8d0 tx=0x7f5ae8008040 comp rx=0 tx=0).stop 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 shutdown_connections 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5aec0afab0 0x7f5aec0ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:57.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5ad0060740 0x7f5ad0062bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5aec158e90 0x7f5aec0aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 --2- 172.21.15.50:0/461119717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5aec06b5a0 0x7f5aec155910 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:57.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.798+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 >> 172.21.15.50:0/461119717 conn(0x7f5aec0cc580 msgr2=0x7f5aec0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:57.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.802+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 shutdown_connections 2023-12-19T14:54:57.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:57.802+0000 7f5af362e700 1 -- 172.21.15.50:0/461119717 wait complete. 2023-12-19T14:54:57.812 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:54:57.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:54:58.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.014+0000 7f8c5288e700 1 Processor -- start 2023-12-19T14:54:58.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.014+0000 7f8c5288e700 1 -- start start 2023-12-19T14:54:58.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5288e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8c4c0ce2d0 0x7f8c4c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5288e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c0cbbb0 con 0x7f8c4c0caa10 2023-12-19T14:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5288e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c06dc90 con 0x7f8c4c0cc040 2023-12-19T14:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5288e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0cafa0 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5188c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8c4c0ce2d0 0x7f8c4c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:58.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5108b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8c4c0caa10 0x7f8c4c158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35780/0 (socket says 172.21.15.50:35780) 2023-12-19T14:54:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5108b700 1 -- 172.21.15.50:0/1460434045 learned_addr learned my addr 172.21.15.50:0/1460434045 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:58.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5088a700 1 -- 172.21.15.50:0/1460434045 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513779926 0 0) 0x7f8c4c0cbbb0 con 0x7f8c4c0caa10 2023-12-19T14:54:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5088a700 1 -- 172.21.15.50:0/1460434045 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c38003570 con 0x7f8c4c0caa10 2023-12-19T14:54:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5188c700 1 -- 172.21.15.50:0/1460434045 >> v1:172.21.15.125:6789/0 conn(0x7f8c4c0cc040 legacy=0x7f8c4c06a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5188c700 1 -- 172.21.15.50:0/1460434045 >> v1:172.21.15.50:6789/0 conn(0x7f8c4c0caa10 legacy=0x7f8c4c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.018+0000 7f8c5188c700 1 -- 172.21.15.50:0/1460434045 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0cb280 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5188c700 1 --2- 172.21.15.50:0/1460434045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c0ce6a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8c480098e0 tx=0x7f8c480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999f02df46719c3a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5088a700 1 -- 172.21.15.50:0/1460434045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800b640 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5088a700 1 -- 172.21.15.50:0/1460434045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800b7a0 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5088a700 1 -- 172.21.15.50:0/1460434045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800bb40 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 -- 172.21.15.50:0/1460434045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0ce2d0 msgr2=0x7f8c4c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1460434045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c0ce6a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8c480098e0 tx=0x7f8c480091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 -- 172.21.15.50:0/1460434045 shutdown_connections 2023-12-19T14:54:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1460434045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c0ce6a0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 -- 172.21.15.50:0/1460434045 >> 172.21.15.50:0/1460434045 conn(0x7f8c4c067bb0 msgr2=0x7f8c4c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 -- 172.21.15.50:0/1460434045 shutdown_connections 2023-12-19T14:54:58.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.022+0000 7f8c5288e700 1 -- 172.21.15.50:0/1460434045 wait complete. 2023-12-19T14:54:58.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 Processor -- start 2023-12-19T14:54:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 -- start start 2023-12-19T14:54:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c4c0caa10 0x7f8c4c0b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:58.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1596c0 con 0x7f8c4c0cc040 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c158ab0 con 0x7f8c4c0ce2d0 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5288e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c158d90 con 0x7f8c4c0caa10 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54078/0 (socket says 172.21.15.50:54078) 2023-12-19T14:54:58.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 -- 172.21.15.50:0/1204238898 learned_addr learned my addr 172.21.15.50:0/1204238898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5188c700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c4c0caa10 0x7f8c4c0b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5208d700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 -- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c4c0caa10 msgr2=0x7f8c4c0b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c4c0caa10 0x7f8c4c0b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 -- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c4c0ce2d0 msgr2=0x7f8c4c156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.026+0000 7f8c5108b700 1 -- 172.21.15.50:0/1204238898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48009420 con 0x7f8c4c0cc040 2023-12-19T14:54:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c5108b700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c7c0 tx=0x7f8c3c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:58.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c005b30 con 0x7f8c4c0cc040 2023-12-19T14:54:58.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c005c90 con 0x7f8c4c0cc040 2023-12-19T14:54:58.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c159070 con 0x7f8c4c0cc040 2023-12-19T14:54:58.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c007340 con 0x7f8c4c0cc040 2023-12-19T14:54:58.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c159ec0 con 0x7f8c4c0cc040 2023-12-19T14:54:58.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.030+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c04be40 con 0x7f8c4c0cc040 2023-12-19T14:54:58.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.034+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8c3c01b400 con 0x7f8c4c0cc040 2023-12-19T14:54:58.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.034+0000 7f8c41ffb700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 0x7f8c30062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:58.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.034+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5953+0+0 (secure 0 0 0) 0x7f8c3c07c5a0 con 0x7f8c4c0cc040 2023-12-19T14:54:58.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.038+0000 7f8c5188c700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 0x7f8c30062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:58.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.038+0000 7f8c5188c700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 0x7f8c30062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c480093f0 tx=0x7f8c48017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:58.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.042+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c3c04c960 con 0x7f8c4c0cc040 2023-12-19T14:54:58.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.254+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8c4c0cb560 con 0x7f8c4c0cc040 2023-12-19T14:54:58.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.950+0000 7f8c41ffb700 1 -- 172.21.15.50:0/1204238898 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8c3c04ff80 con 0x7f8c4c0cc040 2023-12-19T14:54:58.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 msgr2=0x7f8c30062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 0x7f8c30062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c480093f0 tx=0x7f8c48017000 comp rx=0 tx=0).stop 2023-12-19T14:54:58.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 msgr2=0x7f8c4c14c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:58.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c7c0 tx=0x7f8c3c007e00 comp rx=0 tx=0).stop 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 shutdown_connections 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c4c0ce2d0 0x7f8c4c156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c30060670 0x7f8c30062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c4c0cc040 0x7f8c4c14c840 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 --2- 172.21.15.50:0/1204238898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c4c0caa10 0x7f8c4c0b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.954+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 >> 172.21.15.50:0/1204238898 conn(0x7f8c4c067bb0 msgr2=0x7f8c4c15bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.958+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 shutdown_connections 2023-12-19T14:54:58.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:58.958+0000 7f8c5288e700 1 -- 172.21.15.50:0/1204238898 wait complete. 2023-12-19T14:54:58.968 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 11 and data pool 12 2023-12-19T14:54:58.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:59.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 Processor -- start 2023-12-19T14:54:59.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 -- start start 2023-12-19T14:54:59.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b0158e20 0x7f02b006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b00ce810 0x7f02b00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b006b730 con 0x7f02b01591f0 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b006ab20 con 0x7f02b0158e20 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.158+0000 7f02b4c43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b006ae00 con 0x7f02b00ce810 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b0158e20 0x7f02b006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02b0158e20 0x7f02b006a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54080/0 (socket says 172.21.15.50:54080) 2023-12-19T14:54:59.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 -- 172.21.15.50:0/4235366983 learned_addr learned my addr 172.21.15.50:0/4235366983 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02affff700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b00ce810 0x7f02b00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 -- 172.21.15.50:0/4235366983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b00ce810 msgr2=0x7f02b00ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b00ce810 0x7f02b00ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 -- 172.21.15.50:0/4235366983 >> v1:172.21.15.125:6789/0 conn(0x7f02b01591f0 legacy=0x7f02b00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 -- 172.21.15.50:0/4235366983 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b006b0e0 con 0x7f02b0158e20 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02af7fe700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b006a540 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f02a00099b0 tx=0x7f02a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eda7e5b39d2c0d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02ae7fc700 1 -- 172.21.15.50:0/4235366983 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0009ea0 con 0x7f02b0158e20 2023-12-19T14:54:59.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02ae7fc700 1 -- 172.21.15.50:0/4235366983 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a000b860 con 0x7f02b0158e20 2023-12-19T14:54:59.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02ae7fc700 1 -- 172.21.15.50:0/4235366983 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a000bc00 con 0x7f02b0158e20 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 -- 172.21.15.50:0/4235366983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 msgr2=0x7f02b006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b006a540 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f02a00099b0 tx=0x7f02a0009250 comp rx=0 tx=0).stop 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 -- 172.21.15.50:0/4235366983 shutdown_connections 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b00ce810 0x7f02b00ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 --2- 172.21.15.50:0/4235366983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b006a540 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 -- 172.21.15.50:0/4235366983 >> 172.21.15.50:0/4235366983 conn(0x7f02b0067bb0 msgr2=0x7f02b00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:59.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 -- 172.21.15.50:0/4235366983 shutdown_connections 2023-12-19T14:54:59.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.162+0000 7f02b4c43700 1 -- 172.21.15.50:0/4235366983 wait complete. 2023-12-19T14:54:59.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.166+0000 7f02b4c43700 1 Processor -- start 2023-12-19T14:54:59.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.166+0000 7f02b4c43700 1 -- start start 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b00ce810 0x7f02b01417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 0x7f02b0153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00c6a90 con 0x7f02b0158e20 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01567e0 con 0x7f02b00ce810 2023-12-19T14:54:59.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0156ac0 con 0x7f02b01591f0 2023-12-19T14:54:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02aeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02aeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54092/0 (socket says 172.21.15.50:54092) 2023-12-19T14:54:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 0x7f02b0153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02aeffd700 1 -- 172.21.15.50:0/1630108113 learned_addr learned my addr 172.21.15.50:0/1630108113 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:59.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02af7fe700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b00ce810 0x7f02b01417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 -- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b00ce810 msgr2=0x7f02b01417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b00ce810 0x7f02b01417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 -- 172.21.15.50:0/1630108113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 msgr2=0x7f02b0153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 -- 172.21.15.50:0/1630108113 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00094f0 con 0x7f02b01591f0 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02aeffd700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02affff700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 0x7f02b0153680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009e50 tx=0x7f02a800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800caa0 con 0x7f02b01591f0 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a800f040 con 0x7f02b01591f0 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0156da0 con 0x7f02b01591f0 2023-12-19T14:54:59.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800ce50 con 0x7f02b01591f0 2023-12-19T14:54:59.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.170+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0141d20 con 0x7f02b01591f0 2023-12-19T14:54:59.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.174+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b01575a0 con 0x7f02b01591f0 2023-12-19T14:54:59.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.174+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f02a8014690 con 0x7f02b01591f0 2023-12-19T14:54:59.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.178+0000 7f02977fe700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 0x7f029c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.178+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f02a807d7a0 con 0x7f02b01591f0 2023-12-19T14:54:59.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.178+0000 7f02af7fe700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 0x7f029c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.178+0000 7f02af7fe700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 0x7f029c062bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02a00099b0 tx=0x7f02a0017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.186+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02b01575a0 con 0x7f02b01591f0 2023-12-19T14:54:59.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.362+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02b01575a0 con 0x7f02b01591f0 2023-12-19T14:54:59.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.366+0000 7f02977fe700 1 -- 172.21.15.50:0/1630108113 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+15614 (secure 0 0 0) 0x7f02b01575a0 con 0x7f02b01591f0 2023-12-19T14:54:59.369 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:59.369 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:58.258329+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 msgr2=0x7f029c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 0x7f029c062bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02a00099b0 tx=0x7f02a0017000 comp rx=0 tx=0).stop 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 msgr2=0x7f02b0153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 0x7f02b0153680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009e50 tx=0x7f02a800b040 comp rx=0 tx=0).stop 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 shutdown_connections 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f029c060740 0x7f029c062bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02b01591f0 0x7f02b0153680 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f02b0158e20 0x7f02b0153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 --2- 172.21.15.50:0/1630108113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02b00ce810 0x7f02b01417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 >> 172.21.15.50:0/1630108113 conn(0x7f02b0067bb0 msgr2=0x7f02b00ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:59.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 shutdown_connections 2023-12-19T14:54:59.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.370+0000 7f02b4c43700 1 -- 172.21.15.50:0/1630108113 wait complete. 2023-12-19T14:54:59.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:54:59.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.558+0000 7f21117c5700 1 Processor -- start 2023-12-19T14:54:59.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.558+0000 7f21117c5700 1 -- start start 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f210c158f70 0x7f210c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 0x7f210c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c068c70 con 0x7f210c159340 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c068060 con 0x7f210c158f70 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f21117c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c068340 con 0x7f210c069100 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54104/0 (socket says 172.21.15.50:54104) 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210b7fe700 1 -- 172.21.15.50:0/322054321 learned_addr learned my addr 172.21.15.50:0/322054321 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:59.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210affd700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.125:3300/0 conn(0x7f210c158f70 0x7f210c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 --2- 172.21.15.50:0/322054321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 0x7f210c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 -- 172.21.15.50:0/322054321 >> v2:172.21.15.125:3300/0 conn(0x7f210c158f70 msgr2=0x7f210c067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.125:3300/0 conn(0x7f210c158f70 0x7f210c067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 -- 172.21.15.50:0/322054321 >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 msgr2=0x7f210c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 -- 172.21.15.50:0/322054321 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c068620 con 0x7f210c069100 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210b7fe700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:54:59.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.562+0000 7f210a7fc700 1 --2- 172.21.15.50:0/322054321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 0x7f210c0cf7e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f210c0cdec0 tx=0x7f21000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdf8c42a364e4a6f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f2109ffb700 1 -- 172.21.15.50:0/322054321 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100011630 con 0x7f210c069100 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f2109ffb700 1 -- 172.21.15.50:0/322054321 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21000117b0 con 0x7f210c069100 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f2109ffb700 1 -- 172.21.15.50:0/322054321 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100011b70 con 0x7f210c069100 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- 172.21.15.50:0/322054321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 msgr2=0x7f210c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 --2- 172.21.15.50:0/322054321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 0x7f210c0cf7e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f210c0cdec0 tx=0x7f21000091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- 172.21.15.50:0/322054321 shutdown_connections 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.50:3300/0 conn(0x7f210c159340 0x7f210c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 --2- 172.21.15.50:0/322054321 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c069100 0x7f210c0cf7e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 --2- 172.21.15.50:0/322054321 >> v2:172.21.15.125:3300/0 conn(0x7f210c158f70 0x7f210c067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- 172.21.15.50:0/322054321 >> 172.21.15.50:0/322054321 conn(0x7f210c0ccc40 msgr2=0x7f210c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- 172.21.15.50:0/322054321 shutdown_connections 2023-12-19T14:54:59.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- 172.21.15.50:0/322054321 wait complete. 2023-12-19T14:54:59.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 Processor -- start 2023-12-19T14:54:59.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.566+0000 7f21117c5700 1 -- start start 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c158f70 0x7f210c14e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210c159340 0x7f210c14ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14eda0 con 0x7f210c069100 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14f080 con 0x7f210c159340 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c14f360 con 0x7f210c158f70 2023-12-19T14:54:59.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54106/0 (socket says 172.21.15.50:54106) 2023-12-19T14:54:59.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 -- 172.21.15.50:0/1014250114 learned_addr learned my addr 172.21.15.50:0/1014250114 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:59.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210b7fe700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210c159340 0x7f210c14ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210a7fc700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c158f70 0x7f210c14e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 -- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c158f70 msgr2=0x7f210c14e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c158f70 0x7f210c14e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 -- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210c159340 msgr2=0x7f210c14ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210c159340 0x7f210c14ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 -- 172.21.15.50:0/1014250114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104009ae0 con 0x7f210c069100 2023-12-19T14:54:59.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f210affd700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00ec80 tx=0x7f20fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc00c850 con 0x7f210c069100 2023-12-19T14:54:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc00c9d0 con 0x7f210c069100 2023-12-19T14:54:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc00cd90 con 0x7f210c069100 2023-12-19T14:54:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.570+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2100009420 con 0x7f210c069100 2023-12-19T14:54:59.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.574+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c14f920 con 0x7f210c069100 2023-12-19T14:54:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.574+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c14fb40 con 0x7f210c069100 2023-12-19T14:54:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.574+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f20fc013680 con 0x7f210c069100 2023-12-19T14:54:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.578+0000 7f20f2ffd700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 0x7f20f8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.578+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f20fc07cda0 con 0x7f210c069100 2023-12-19T14:54:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.578+0000 7f210a7fc700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 0x7f20f8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.578+0000 7f210a7fc700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 0x7f20f8062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f210c0cdec0 tx=0x7f2100017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.582+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f20fc04d080 con 0x7f210c069100 2023-12-19T14:54:59.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.758+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f210c071120 con 0x7f210c069100 2023-12-19T14:54:59.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.762+0000 7f20f2ffd700 1 -- 172.21.15.50:0/1014250114 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+15614 (secure 0 0 0) 0x7f20fc0506a0 con 0x7f210c069100 2023-12-19T14:54:59.764 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:54:59.764 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:58.258329+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 msgr2=0x7f20f8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 0x7f20f8062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f210c0cdec0 tx=0x7f2100017000 comp rx=0 tx=0).stop 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 msgr2=0x7f210c14d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00ec80 tx=0x7f20fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 shutdown_connections 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f210c159340 0x7f210c14ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f20f80606a0 0x7f20f8062b50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f210c158f70 0x7f210c14e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 --2- 172.21.15.50:0/1014250114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f210c069100 0x7f210c14d360 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 >> 172.21.15.50:0/1014250114 conn(0x7f210c0ccc40 msgr2=0x7f210c0cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:59.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 shutdown_connections 2023-12-19T14:54:59.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.766+0000 7f21117c5700 1 -- 172.21.15.50:0/1014250114 wait complete. 2023-12-19T14:54:59.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:54:59.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.966+0000 7f1562945700 1 Processor -- start 2023-12-19T14:54:59.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1562945700 1 -- start start 2023-12-19T14:54:59.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1562945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f155c158e20 0x7f155c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1562945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c06d5e0 con 0x7f155c0cb970 2023-12-19T14:54:59.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1562945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0cbdf0 con 0x7f155c0ca2c0 2023-12-19T14:54:59.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1562945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0cc0d0 con 0x7f155c158e20 2023-12-19T14:54:59.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155bfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f155c158e20 0x7f155c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f155c0cb970 0x7f155c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35792/0 (socket says 172.21.15.50:35792) 2023-12-19T14:54:59.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155b7fe700 1 -- 172.21.15.50:0/4166373808 learned_addr learned my addr 172.21.15.50:0/4166373808 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:54:59.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f1560ee2700 1 --1- 172.21.15.50:0/4166373808 >> v1:172.21.15.125:6789/0 conn(0x7f155c0ca2c0 0x7f155c06ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:44390/0 (socket says 172.21.15.50:44390) 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155bfff700 1 -- 172.21.15.50:0/4166373808 >> v1:172.21.15.125:6789/0 conn(0x7f155c0ca2c0 legacy=0x7f155c06ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155affd700 1 -- 172.21.15.50:0/4166373808 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447508128 0 0) 0x7f155c06d5e0 con 0x7f155c0cb970 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.970+0000 7f155bfff700 1 -- 172.21.15.50:0/4166373808 >> v1:172.21.15.50:6789/0 conn(0x7f155c0cb970 legacy=0x7f155c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f155bfff700 1 -- 172.21.15.50:0/4166373808 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0cc3b0 con 0x7f155c158e20 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f155bfff700 1 --2- 172.21.15.50:0/4166373808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0ce5c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f154c009910 tx=0x7f154c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6f3ead193bfc0f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f155affd700 1 -- 172.21.15.50:0/4166373808 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00b820 con 0x7f155c158e20 2023-12-19T14:54:59.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f155affd700 1 -- 172.21.15.50:0/4166373808 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c00b980 con 0x7f155c158e20 2023-12-19T14:54:59.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f155affd700 1 -- 172.21.15.50:0/4166373808 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00bd20 con 0x7f155c158e20 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 -- 172.21.15.50:0/4166373808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 msgr2=0x7f155c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 --2- 172.21.15.50:0/4166373808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0ce5c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f154c009910 tx=0x7f154c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 -- 172.21.15.50:0/4166373808 shutdown_connections 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 --2- 172.21.15.50:0/4166373808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0ce5c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 -- 172.21.15.50:0/4166373808 >> 172.21.15.50:0/4166373808 conn(0x7f155c067bb0 msgr2=0x7f155c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:54:59.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 -- 172.21.15.50:0/4166373808 shutdown_connections 2023-12-19T14:54:59.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.974+0000 7f1562945700 1 -- 172.21.15.50:0/4166373808 wait complete. 2023-12-19T14:54:59.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.978+0000 7f1562945700 1 Processor -- start 2023-12-19T14:54:59.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.978+0000 7f1562945700 1 -- start start 2023-12-19T14:54:59.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f155c0ca2c0 0x7f155c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b3ab0 con 0x7f155c0b5680 2023-12-19T14:54:59.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b3d90 con 0x7f155c158e20 2023-12-19T14:54:59.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0b4070 con 0x7f155c0ca2c0 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54114/0 (socket says 172.21.15.50:54114) 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 -- 172.21.15.50:0/672161 learned_addr learned my addr 172.21.15.50:0/672161 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f155b7fe700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f155bfff700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f155c0ca2c0 0x7f155c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 -- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f155c0ca2c0 msgr2=0x7f155c0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f155c0ca2c0 0x7f155c0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 -- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 msgr2=0x7f155c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 -- 172.21.15.50:0/672161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c009420 con 0x7f155c0b5680 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1560ee2700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f155400cc70 tx=0x7f1554001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554005df0 con 0x7f155c0b5680 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155400a640 con 0x7f155c0b5680 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 -- 172.21.15.50:0/672161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c0b4350 con 0x7f155c0b5680 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155400a9c0 con 0x7f155c0b5680 2023-12-19T14:54:59.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.982+0000 7f1562945700 1 -- 172.21.15.50:0/672161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c0b5cf0 con 0x7f155c0b5680 2023-12-19T14:54:59.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.986+0000 7f1562945700 1 -- 172.21.15.50:0/672161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c0b4850 con 0x7f155c0b5680 2023-12-19T14:54:59.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.986+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f15540070f0 con 0x7f155c0b5680 2023-12-19T14:54:59.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.986+0000 7f1543fff700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 0x7f1548062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:54:59.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.990+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f155407cb50 con 0x7f155c0b5680 2023-12-19T14:54:59.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.990+0000 7f155bfff700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 0x7f1548062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:54:59.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.990+0000 7f155bfff700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 0x7f1548062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f154c0093f0 tx=0x7f154c017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:54:59.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:54:59.994+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f155404ceb0 con 0x7f155c0b5680 2023-12-19T14:55:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.206+0000 7f1562945700 1 -- 172.21.15.50:0/672161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f155c0cc690 con 0x7f155c0b5680 2023-12-19T14:55:00.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.970+0000 7f1543fff700 1 -- 172.21.15.50:0/672161 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f15540504d0 con 0x7f155c0b5680 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 msgr2=0x7f1548062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 0x7f1548062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f154c0093f0 tx=0x7f154c017000 comp rx=0 tx=0).stop 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 msgr2=0x7f155c0c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f155400cc70 tx=0x7f1554001a70 comp rx=0 tx=0).stop 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 shutdown_connections 2023-12-19T14:55:00.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155c158e20 0x7f155c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1548060670 0x7f1548062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f155c0b5680 0x7f155c0c85e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 --2- 172.21.15.50:0/672161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f155c0ca2c0 0x7f155c0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 >> 172.21.15.50:0/672161 conn(0x7f155c067bb0 msgr2=0x7f155c14eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 shutdown_connections 2023-12-19T14:55:00.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:00.974+0000 7f1562945700 1 -- 172.21.15.50:0/672161 wait complete. 2023-12-19T14:55:01.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:55:01.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 Processor -- start 2023-12-19T14:55:01.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 -- start start 2023-12-19T14:55:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749006a830 0x7f749015d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749006b970 con 0x7f749006be00 2023-12-19T14:55:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749006ad60 con 0x7f749015d0e0 2023-12-19T14:55:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.174+0000 7f74980a5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749006b040 con 0x7f749006a830 2023-12-19T14:55:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749006a830 0x7f749015d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f749006a830 0x7f749015d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:46628/0 (socket says 172.21.15.50:46628) 2023-12-19T14:55:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 -- 172.21.15.50:0/1155573412 learned_addr learned my addr 172.21.15.50:0/1155573412 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 -- 172.21.15.50:0/1155573412 >> v1:172.21.15.125:6789/0 conn(0x7f749015d0e0 legacy=0x7f74900c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 -- 172.21.15.50:0/1155573412 >> v1:172.21.15.50:6789/0 conn(0x7f749006be00 legacy=0x7f749006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 -- 172.21.15.50:0/1155573412 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749006b320 con 0x7f749006a830 2023-12-19T14:55:01.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7495640700 1 --2- 172.21.15.50:0/1155573412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749006a830 0x7f749015d4b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f74800099c0 tx=0x7f7480009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4dc146f06160f81 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:01.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7494e3f700 1 -- 172.21.15.50:0/1155573412 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480005c10 con 0x7f749006a830 2023-12-19T14:55:01.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7494e3f700 1 -- 172.21.15.50:0/1155573412 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7480005d70 con 0x7f749006a830 2023-12-19T14:55:01.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f7494e3f700 1 -- 172.21.15.50:0/1155573412 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748000b840 con 0x7f749006a830 2023-12-19T14:55:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f74980a5700 1 -- 172.21.15.50:0/1155573412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749006a830 msgr2=0x7f749015d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.178+0000 7f74980a5700 1 --2- 172.21.15.50:0/1155573412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749006a830 0x7f749015d4b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f74800099c0 tx=0x7f7480009220 comp rx=0 tx=0).stop 2023-12-19T14:55:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- 172.21.15.50:0/1155573412 shutdown_connections 2023-12-19T14:55:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 --2- 172.21.15.50:0/1155573412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749006a830 0x7f749015d4b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- 172.21.15.50:0/1155573412 >> 172.21.15.50:0/1155573412 conn(0x7f7490067bb0 msgr2=0x7f74900cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:01.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- 172.21.15.50:0/1155573412 shutdown_connections 2023-12-19T14:55:01.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- 172.21.15.50:0/1155573412 wait complete. 2023-12-19T14:55:01.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 Processor -- start 2023-12-19T14:55:01.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- start start 2023-12-19T14:55:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 0x7f7490152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749015d0e0 0x7f74900ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490162770 con 0x7f749006be00 2023-12-19T14:55:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490161b60 con 0x7f749015d0e0 2023-12-19T14:55:01.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.182+0000 7f74980a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7490161e40 con 0x7f749006a830 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54122/0 (socket says 172.21.15.50:54122) 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 -- 172.21.15.50:0/3732800432 learned_addr learned my addr 172.21.15.50:0/3732800432 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495e41700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 0x7f7490152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7496642700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749015d0e0 0x7f74900ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 -- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 msgr2=0x7f7490152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 0x7f7490152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 -- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749015d0e0 msgr2=0x7f74900ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749015d0e0 0x7f74900ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 -- 172.21.15.50:0/3732800432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480009480 con 0x7f749006be00 2023-12-19T14:55:01.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495e41700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 0x7f7490152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:01.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7495640700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7480012000 tx=0x7f7480016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:01.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480011bb0 con 0x7f749006be00 2023-12-19T14:55:01.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7480011d10 con 0x7f749006be00 2023-12-19T14:55:01.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748001b680 con 0x7f749006be00 2023-12-19T14:55:01.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490162120 con 0x7f749006be00 2023-12-19T14:55:01.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.186+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74900acdb0 con 0x7f749006be00 2023-12-19T14:55:01.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.190+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7490162920 con 0x7f749006be00 2023-12-19T14:55:01.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.190+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f748001ba70 con 0x7f749006be00 2023-12-19T14:55:01.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.190+0000 7f7485ffb700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 0x7f7474062c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:01.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.194+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f748007e530 con 0x7f749006be00 2023-12-19T14:55:01.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.194+0000 7f7495e41700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 0x7f7474062c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:01.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.194+0000 7f7495e41700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 0x7f7474062c20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f748c007fd0 tx=0x7f748c009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:01.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.198+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7490162920 con 0x7f749006be00 2023-12-19T14:55:01.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.410+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7490071120 con 0x7f749006be00 2023-12-19T14:55:01.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.982+0000 7f7485ffb700 1 -- 172.21.15.50:0/3732800432 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f748004e890 con 0x7f749006be00 2023-12-19T14:55:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 msgr2=0x7f7474062c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 0x7f7474062c20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f748c007fd0 tx=0x7f748c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 msgr2=0x7f74900ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7480012000 tx=0x7f7480016000 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 shutdown_connections 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f749015d0e0 0x7f74900ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7474060770 0x7f7474062c20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f749006be00 0x7f74900ad180 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 --2- 172.21.15.50:0/3732800432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f749006a830 0x7f7490152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 >> 172.21.15.50:0/3732800432 conn(0x7f7490067bb0 msgr2=0x7f74901639e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 shutdown_connections 2023-12-19T14:55:01.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:01.986+0000 7f74980a5700 1 -- 172.21.15.50:0/3732800432 wait complete. 2023-12-19T14:55:02.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:02.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 Processor -- start 2023-12-19T14:55:02.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 -- start start 2023-12-19T14:55:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f680c6a50 0x7f3f680c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f68159b40 0x7f3f68159f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6806d790 con 0x7f3f6806dc20 2023-12-19T14:55:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f680c7360 con 0x7f3f68159b40 2023-12-19T14:55:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6f323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6806cb80 con 0x7f3f680c6a50 2023-12-19T14:55:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.218+0000 7f3f6eb22700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f68159b40 0x7f3f68159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6e321700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f680c6a50 0x7f3f680c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6db20700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3f6806dc20 0x7f3f6806b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:44404/0 (socket says 172.21.15.50:44404) 2023-12-19T14:55:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6db20700 1 -- 172.21.15.50:0/2250585589 learned_addr learned my addr 172.21.15.50:0/2250585589 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6eb22700 1 -- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f680c6a50 msgr2=0x7f3f680c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6eb22700 1 --2- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f680c6a50 0x7f3f680c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6eb22700 1 -- 172.21.15.50:0/2250585589 >> v1:172.21.15.125:6789/0 conn(0x7f3f6806dc20 legacy=0x7f3f6806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6eb22700 1 -- 172.21.15.50:0/2250585589 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6806ce60 con 0x7f3f68159b40 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6d31f700 1 -- 172.21.15.50:0/2250585589 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415938949 0 0) 0x7f3f6806d790 con 0x7f3f6806dc20 2023-12-19T14:55:02.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6eb22700 1 --2- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f68159f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009e60 tx=0x7f3f60009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda614d02eac3a07 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6d31f700 1 -- 172.21.15.50:0/2250585589 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000e070 con 0x7f3f68159b40 2023-12-19T14:55:02.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6d31f700 1 -- 172.21.15.50:0/2250585589 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6000bb60 con 0x7f3f68159b40 2023-12-19T14:55:02.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6d31f700 1 -- 172.21.15.50:0/2250585589 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60012430 con 0x7f3f68159b40 2023-12-19T14:55:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6f323700 1 -- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 msgr2=0x7f3f68159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.222+0000 7f3f6f323700 1 --2- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f68159f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009e60 tx=0x7f3f60009780 comp rx=0 tx=0).stop 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 -- 172.21.15.50:0/2250585589 shutdown_connections 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 --2- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f68159f90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 --2- 172.21.15.50:0/2250585589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f680c6a50 0x7f3f680c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 -- 172.21.15.50:0/2250585589 >> 172.21.15.50:0/2250585589 conn(0x7f3f68067bb0 msgr2=0x7f3f680cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 -- 172.21.15.50:0/2250585589 shutdown_connections 2023-12-19T14:55:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 -- 172.21.15.50:0/2250585589 wait complete. 2023-12-19T14:55:02.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 Processor -- start 2023-12-19T14:55:02.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.226+0000 7f3f6f323700 1 -- start start 2023-12-19T14:55:02.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 0x7f3f68151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f681511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6815cf70 con 0x7f3f680c6a50 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6815c360 con 0x7f3f68159b40 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6f323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6815c640 con 0x7f3f6806dc20 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6db20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 0x7f3f68151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6eb22700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f681511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6db20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54136/0 (socket says 172.21.15.50:54136) 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6db20700 1 -- 172.21.15.50:0/1224520296 learned_addr learned my addr 172.21.15.50:0/1224520296 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 -- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 msgr2=0x7f3f681511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f681511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 -- 172.21.15.50:0/1224520296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 msgr2=0x7f3f681524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 -- 172.21.15.50:0/1224520296 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f60009a20 con 0x7f3f6806dc20 2023-12-19T14:55:02.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6db20700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.230+0000 7f3f6e321700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 0x7f3f68151f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400ea50 tx=0x7f3f640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400ee90 con 0x7f3f6806dc20 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6400c7f0 con 0x7f3f6806dc20 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400cb90 con 0x7f3f6806dc20 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6815c920 con 0x7f3f6806dc20 2023-12-19T14:55:02.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6814f990 con 0x7f3f6806dc20 2023-12-19T14:55:02.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6804be40 con 0x7f3f6806dc20 2023-12-19T14:55:02.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.234+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3f64005a00 con 0x7f3f6806dc20 2023-12-19T14:55:02.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.238+0000 7f3f5e7fc700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 0x7f3f4c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.238+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3f6407cf50 con 0x7f3f6806dc20 2023-12-19T14:55:02.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.238+0000 7f3f6db20700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 0x7f3f4c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.238+0000 7f3f6db20700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 0x7f3f4c062bf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f6806be20 tx=0x7f3f58009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.242+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3f6404d330 con 0x7f3f6806dc20 2023-12-19T14:55:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.458+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f6806d140 con 0x7f3f6806dc20 2023-12-19T14:55:02.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f5e7fc700 1 -- 172.21.15.50:0/1224520296 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f3f64017090 con 0x7f3f6806dc20 2023-12-19T14:55:02.464 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:02.464 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 msgr2=0x7f3f4c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 0x7f3f4c062bf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f6806be20 tx=0x7f3f58009040 comp rx=0 tx=0).stop 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 msgr2=0x7f3f68151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 0x7f3f68151f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400ea50 tx=0x7f3f640091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 shutdown_connections 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f68159b40 0x7f3f681511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f4c060740 0x7f3f4c062bf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f680c6a50 0x7f3f681524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 --2- 172.21.15.50:0/1224520296 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f6806dc20 0x7f3f68151f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 >> 172.21.15.50:0/1224520296 conn(0x7f3f68067bb0 msgr2=0x7f3f680c14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.462+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 shutdown_connections 2023-12-19T14:55:02.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.466+0000 7f3f6f323700 1 -- 172.21.15.50:0/1224520296 wait complete. 2023-12-19T14:55:02.469 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:02.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:02.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 Processor -- start 2023-12-19T14:55:02.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 -- start start 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5120069100 0x7f5120067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5120068c70 con 0x7f5120158f70 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5120068060 con 0x7f5120159340 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f5125225700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5120068340 con 0x7f5120069100 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f511ed9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5120158f70 0x7f51200cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35798/0 (socket says 172.21.15.50:35798) 2023-12-19T14:55:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.658+0000 7f511ed9d700 1 -- 172.21.15.50:0/74555965 learned_addr learned my addr 172.21.15.50:0/74555965 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511e59c700 1 --2- 172.21.15.50:0/74555965 >> v2:172.21.15.125:3300/0 conn(0x7f5120069100 0x7f5120067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043770310 0 0) 0x7f5120068c70 con 0x7f5120158f70 2023-12-19T14:55:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5108003570 con 0x7f5120158f70 2023-12-19T14:55:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295937734 0 0) 0x7f5108003570 con 0x7f5120158f70 2023-12-19T14:55:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 >> v2:172.21.15.125:3300/0 conn(0x7f5120069100 msgr2=0x7f5120067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 --2- 172.21.15.50:0/74555965 >> v2:172.21.15.125:3300/0 conn(0x7f5120069100 0x7f5120067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 >> v1:172.21.15.125:6789/0 conn(0x7f5120159340 legacy=0x7f51200cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5120068620 con 0x7f5120158f70 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f5110001470 con 0x7f5120158f70 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 >> v1:172.21.15.50:6789/0 conn(0x7f5120158f70 legacy=0x7f51200cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5108003a60 0x7f5108003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108004810 0x7f5108004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 0x7f510800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120068340 con 0x7f5108005210 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510800c1b0 con 0x7f5108003a60 2023-12-19T14:55:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510800c370 con 0x7f5108004810 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 0x7f510800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511ed9d700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5108003a60 0x7f5108003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511e59c700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108004810 0x7f5108004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 -- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108004810 msgr2=0x7f5108004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108004810 0x7f5108004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 -- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5108003a60 msgr2=0x7f5108003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:02.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5108003a60 0x7f5108003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 -- 172.21.15.50:0/74555965 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5108003570 con 0x7f5108005210 2023-12-19T14:55:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 -- 172.21.15.50:0/74555965 reap_dead start 2023-12-19T14:55:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.662+0000 7f511f59e700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 0x7f510800ba60 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5120068c20 tx=0x7f5118001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c0588b8ce9a3d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5118001950 con 0x7f5108005210 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5118004030 con 0x7f5108005210 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f511dd9b700 1 -- 172.21.15.50:0/74555965 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511800cb00 con 0x7f5108005210 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 -- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 msgr2=0x7f510800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 0x7f510800ba60 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5120068c20 tx=0x7f5118001640 comp rx=0 tx=0).stop 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 -- 172.21.15.50:0/74555965 shutdown_connections 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 --2- 172.21.15.50:0/74555965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5108005210 0x7f510800ba60 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 -- 172.21.15.50:0/74555965 >> 172.21.15.50:0/74555965 conn(0x7f51200ccc40 msgr2=0x7f51200cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 -- 172.21.15.50:0/74555965 shutdown_connections 2023-12-19T14:55:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.666+0000 7f5125225700 1 -- 172.21.15.50:0/74555965 wait complete. 2023-12-19T14:55:02.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 Processor -- start 2023-12-19T14:55:02.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 -- start start 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 0x7f512015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f512015dff0 0x7f512016da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512016e3d0 con 0x7f512016df60 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512015bf90 con 0x7f512015dff0 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5125225700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51200b0630 con 0x7f5108005210 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54156/0 (socket says 172.21.15.50:54156) 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 -- 172.21.15.50:0/380877726 learned_addr learned my addr 172.21.15.50:0/380877726 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511e59c700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f512015dff0 0x7f512016da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511ed9d700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 0x7f512015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 -- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 msgr2=0x7f512015c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 0x7f512015c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 -- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f512015dff0 msgr2=0x7f512016da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f512015dff0 0x7f512016da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 -- 172.21.15.50:0/380877726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512016e8e0 con 0x7f512016df60 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511ed9d700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 0x7f512015c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f511f59e700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f51200ca790 tx=0x7f5118015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.670+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511800c6d0 con 0x7f512016df60 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511800cac0 con 0x7f512016df60 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511801d650 con 0x7f512016df60 2023-12-19T14:55:02.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512016e600 con 0x7f512016df60 2023-12-19T14:55:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512015bc20 con 0x7f512016df60 2023-12-19T14:55:02.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512015c140 con 0x7f512016df60 2023-12-19T14:55:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.674+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f511801da80 con 0x7f512016df60 2023-12-19T14:55:02.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.678+0000 7f5106ffd700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 0x7f510c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:02.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.678+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f511807e0a0 con 0x7f512016df60 2023-12-19T14:55:02.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.678+0000 7f511ed9d700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 0x7f510c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:02.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.678+0000 7f511ed9d700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 0x7f510c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51100015a0 tx=0x7f5110009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.682+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f512015c140 con 0x7f512016df60 2023-12-19T14:55:02.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.858+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f512015c140 con 0x7f512016df60 2023-12-19T14:55:02.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.862+0000 7f5106ffd700 1 -- 172.21.15.50:0/380877726 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+15603 (secure 0 0 0) 0x7f512015c140 con 0x7f512016df60 2023-12-19T14:55:02.864 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:02.864 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":100,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:54:59.795660+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 msgr2=0x7f510c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 0x7f510c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51100015a0 tx=0x7f5110009040 comp rx=0 tx=0).stop 2023-12-19T14:55:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 msgr2=0x7f512015a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:02.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f51200ca790 tx=0x7f5118015000 comp rx=0 tx=0).stop 2023-12-19T14:55:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 shutdown_connections 2023-12-19T14:55:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f512015dff0 0x7f512016da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f510c060670 0x7f510c062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f512016df60 0x7f512015a290 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 --2- 172.21.15.50:0/380877726 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5108005210 0x7f512015c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:02.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 >> 172.21.15.50:0/380877726 conn(0x7f51200ccc40 msgr2=0x7f51200c6360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:02.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 shutdown_connections 2023-12-19T14:55:02.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:02.866+0000 7f5125225700 1 -- 172.21.15.50:0/380877726 wait complete. 2023-12-19T14:55:02.890 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:55:02.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:55:03.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.050+0000 7f97916e3700 1 Processor -- start 2023-12-19T14:55:03.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.050+0000 7f97916e3700 1 -- start start 2023-12-19T14:55:03.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f97916e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f978c06aa00 0x7f978c06d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f97916e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c06c7c0 con 0x7f978c0b1410 2023-12-19T14:55:03.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f97916e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c06cd80 con 0x7f978c06a580 2023-12-19T14:55:03.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f97916e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c06b8f0 con 0x7f978c06aa00 2023-12-19T14:55:03.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f978bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f978c0b1410 0x7f978c0ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35814/0 (socket says 172.21.15.50:35814) 2023-12-19T14:55:03.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f978bfff700 1 -- 172.21.15.50:0/3680724498 learned_addr learned my addr 172.21.15.50:0/3680724498 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:03.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f9790ee2700 1 --2- 172.21.15.50:0/3680724498 >> v2:172.21.15.125:3300/0 conn(0x7f978c06aa00 0x7f978c06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f978b7fe700 1 -- 172.21.15.50:0/3680724498 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169411412 0 0) 0x7f978c06c7c0 con 0x7f978c0b1410 2023-12-19T14:55:03.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f9790ee2700 1 -- 172.21.15.50:0/3680724498 >> v1:172.21.15.125:6789/0 conn(0x7f978c06a580 legacy=0x7f978c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f9790ee2700 1 -- 172.21.15.50:0/3680724498 >> v1:172.21.15.50:6789/0 conn(0x7f978c0b1410 legacy=0x7f978c0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.054+0000 7f9790ee2700 1 -- 172.21.15.50:0/3680724498 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0ac1e0 con 0x7f978c06aa00 2023-12-19T14:55:03.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f9790ee2700 1 --2- 172.21.15.50:0/3680724498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06aa00 0x7f978c06d920 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f97840098e0 tx=0x7f97840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524ec49d9962c90d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f978b7fe700 1 -- 172.21.15.50:0/3680724498 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784009e80 con 0x7f978c06aa00 2023-12-19T14:55:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f978b7fe700 1 -- 172.21.15.50:0/3680724498 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978400b860 con 0x7f978c06aa00 2023-12-19T14:55:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f978b7fe700 1 -- 172.21.15.50:0/3680724498 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400bbe0 con 0x7f978c06aa00 2023-12-19T14:55:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 -- 172.21.15.50:0/3680724498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06aa00 msgr2=0x7f978c06d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 --2- 172.21.15.50:0/3680724498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06aa00 0x7f978c06d920 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f97840098e0 tx=0x7f97840091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 -- 172.21.15.50:0/3680724498 shutdown_connections 2023-12-19T14:55:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 --2- 172.21.15.50:0/3680724498 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06aa00 0x7f978c06d920 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 -- 172.21.15.50:0/3680724498 >> 172.21.15.50:0/3680724498 conn(0x7f978c067bb0 msgr2=0x7f978c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 -- 172.21.15.50:0/3680724498 shutdown_connections 2023-12-19T14:55:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.058+0000 7f97916e3700 1 -- 172.21.15.50:0/3680724498 wait complete. 2023-12-19T14:55:03.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 Processor -- start 2023-12-19T14:55:03.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 -- start start 2023-12-19T14:55:03.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 0x7f978c154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f978c0b1410 0x7f978c0c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c157320 con 0x7f978c06aa00 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1582c0 con 0x7f978c06a580 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97916e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c157e30 con 0x7f978c0b1410 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54164/0 (socket says 172.21.15.50:54164) 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 -- 172.21.15.50:0/454168408 learned_addr learned my addr 172.21.15.50:0/454168408 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f9790ee2700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f978c0b1410 0x7f978c0c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f978bfff700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 0x7f978c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 -- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f978c0b1410 msgr2=0x7f978c0c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f978c0b1410 0x7f978c0c6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 -- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 msgr2=0x7f978c154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 0x7f978c154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f97837fe700 1 -- 172.21.15.50:0/454168408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009420 con 0x7f978c06aa00 2023-12-19T14:55:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.062+0000 7f978bfff700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 0x7f978c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f97837fe700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f977400c590 tx=0x7f977400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774011590 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c157500 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9774011710 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740011f0 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c169cf0 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c1578f0 con 0x7f978c06aa00 2023-12-19T14:55:03.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.066+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9774011ca0 con 0x7f978c06aa00 2023-12-19T14:55:03.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.070+0000 7f9788ff9700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 0x7f976c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.070+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f977407be70 con 0x7f978c06aa00 2023-12-19T14:55:03.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.070+0000 7f978bfff700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 0x7f976c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.074+0000 7f978bfff700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 0x7f976c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f977c000f80 tx=0x7f977c009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.078+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f977404c2a0 con 0x7f978c06aa00 2023-12-19T14:55:03.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.246+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f978c157c60 con 0x7f976c060670 2023-12-19T14:55:03.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.270+0000 7f9788ff9700 1 -- 172.21.15.50:0/454168408 <== mgr.6089 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286841 (secure 0 0 0) 0x7f978c157c60 con 0x7f976c060670 2023-12-19T14:55:03.277 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 msgr2=0x7f976c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 0x7f976c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f977c000f80 tx=0x7f977c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 msgr2=0x7f978c15af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f977400c590 tx=0x7f977400c8a0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 shutdown_connections 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f976c060670 0x7f976c062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f978c0b1410 0x7f978c0c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f978c06aa00 0x7f978c15af20 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 --2- 172.21.15.50:0/454168408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f978c06a580 0x7f978c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.274+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 >> 172.21.15.50:0/454168408 conn(0x7f978c067bb0 msgr2=0x7f978c0c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.278+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 shutdown_connections 2023-12-19T14:55:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.278+0000 7f97916e3700 1 -- 172.21.15.50:0/454168408 wait complete. 2023-12-19T14:55:03.282 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:55:03.301 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:55:01.696594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2442200,"kb_used_data":10648,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752532520,"statfs":{"total":773094113280,"available":770593300480,"internally_reserved":0,"allocated":10903552,"data_stored":6020739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.922583"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817477+0000","last_change":"2023-12-19T14:54:58.817477+0000","last_active":"2023-12-19T14:54:58.817477+0000","last_peered":"2023-12-19T14:54:58.817477+0000","last_clean":"2023-12-19T14:54:58.817477+0000","last_became_active":"2023-12-19T14:54:58.816899+0000","last_became_peered":"2023-12-19T14:54:58.816899+0000","last_unstale":"2023-12-19T14:54:58.817477+0000","last_undegraded":"2023-12-19T14:54:58.817477+0000","last_fullsized":"2023-12-19T14:54:58.817477+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:55.388516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.836709+0000","last_change":"2023-12-19T14:54:58.836709+0000","last_active":"2023-12-19T14:54:58.836709+0000","last_peered":"2023-12-19T14:54:58.836709+0000","last_clean":"2023-12-19T14:54:58.836709+0000","last_became_active":"2023-12-19T14:54:58.836346+0000","last_became_peered":"2023-12-19T14:54:58.836346+0000","last_unstale":"2023-12-19T14:54:58.836709+0000","last_undegraded":"2023-12-19T14:54:58.836709+0000","last_fullsized":"2023-12-19T14:54:58.836709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:30:46.597215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832557+0000","last_change":"2023-12-19T14:54:58.832557+0000","last_active":"2023-12-19T14:54:58.832557+0000","last_peered":"2023-12-19T14:54:58.832557+0000","last_clean":"2023-12-19T14:54:58.832557+0000","last_became_active":"2023-12-19T14:54:58.832265+0000","last_became_peered":"2023-12-19T14:54:58.832265+0000","last_unstale":"2023-12-19T14:54:58.832557+0000","last_undegraded":"2023-12-19T14:54:58.832557+0000","last_fullsized":"2023-12-19T14:54:58.832557+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:00.765295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821001+0000","last_change":"2023-12-19T14:54:58.821001+0000","last_active":"2023-12-19T14:54:58.821001+0000","last_peered":"2023-12-19T14:54:58.821001+0000","last_clean":"2023-12-19T14:54:58.821001+0000","last_became_active":"2023-12-19T14:54:58.818846+0000","last_became_peered":"2023-12-19T14:54:58.818846+0000","last_unstale":"2023-12-19T14:54:58.821001+0000","last_undegraded":"2023-12-19T14:54:58.821001+0000","last_fullsized":"2023-12-19T14:54:58.821001+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:13:59.329747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818462+0000","last_change":"2023-12-19T14:54:58.818462+0000","last_active":"2023-12-19T14:54:58.818462+0000","last_peered":"2023-12-19T14:54:58.818462+0000","last_clean":"2023-12-19T14:54:58.818462+0000","last_became_active":"2023-12-19T14:54:58.818164+0000","last_became_peered":"2023-12-19T14:54:58.818164+0000","last_unstale":"2023-12-19T14:54:58.818462+0000","last_undegraded":"2023-12-19T14:54:58.818462+0000","last_fullsized":"2023-12-19T14:54:58.818462+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:20.258328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824850+0000","last_change":"2023-12-19T14:54:58.824850+0000","last_active":"2023-12-19T14:54:58.824850+0000","last_peered":"2023-12-19T14:54:58.824850+0000","last_clean":"2023-12-19T14:54:58.824850+0000","last_became_active":"2023-12-19T14:54:58.824231+0000","last_became_peered":"2023-12-19T14:54:58.824231+0000","last_unstale":"2023-12-19T14:54:58.824850+0000","last_undegraded":"2023-12-19T14:54:58.824850+0000","last_fullsized":"2023-12-19T14:54:58.824850+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:55.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.815416+0000","last_change":"2023-12-19T14:54:58.815416+0000","last_active":"2023-12-19T14:54:58.815416+0000","last_peered":"2023-12-19T14:54:58.815416+0000","last_clean":"2023-12-19T14:54:58.815416+0000","last_became_active":"2023-12-19T14:54:58.815120+0000","last_became_peered":"2023-12-19T14:54:58.815120+0000","last_unstale":"2023-12-19T14:54:58.815416+0000","last_undegraded":"2023-12-19T14:54:58.815416+0000","last_fullsized":"2023-12-19T14:54:58.815416+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:17.673794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821114+0000","last_change":"2023-12-19T14:54:58.821114+0000","last_active":"2023-12-19T14:54:58.821114+0000","last_peered":"2023-12-19T14:54:58.821114+0000","last_clean":"2023-12-19T14:54:58.821114+0000","last_became_active":"2023-12-19T14:54:58.820847+0000","last_became_peered":"2023-12-19T14:54:58.820847+0000","last_unstale":"2023-12-19T14:54:58.821114+0000","last_undegraded":"2023-12-19T14:54:58.821114+0000","last_fullsized":"2023-12-19T14:54:58.821114+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:15.615896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829972+0000","last_change":"2023-12-19T14:54:58.829972+0000","last_active":"2023-12-19T14:54:58.829972+0000","last_peered":"2023-12-19T14:54:58.829972+0000","last_clean":"2023-12-19T14:54:58.829972+0000","last_became_active":"2023-12-19T14:54:58.829713+0000","last_became_peered":"2023-12-19T14:54:58.829713+0000","last_unstale":"2023-12-19T14:54:58.829972+0000","last_undegraded":"2023-12-19T14:54:58.829972+0000","last_fullsized":"2023-12-19T14:54:58.829972+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:12.798653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824042+0000","last_change":"2023-12-19T14:54:58.824042+0000","last_active":"2023-12-19T14:54:58.824042+0000","last_peered":"2023-12-19T14:54:58.824042+0000","last_clean":"2023-12-19T14:54:58.824042+0000","last_became_active":"2023-12-19T14:54:58.822793+0000","last_became_peered":"2023-12-19T14:54:58.822793+0000","last_unstale":"2023-12-19T14:54:58.824042+0000","last_undegraded":"2023-12-19T14:54:58.824042+0000","last_fullsized":"2023-12-19T14:54:58.824042+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:22:13.417736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829542+0000","last_change":"2023-12-19T14:54:58.829542+0000","last_active":"2023-12-19T14:54:58.829542+0000","last_peered":"2023-12-19T14:54:58.829542+0000","last_clean":"2023-12-19T14:54:58.829542+0000","last_became_active":"2023-12-19T14:54:58.829256+0000","last_became_peered":"2023-12-19T14:54:58.829256+0000","last_unstale":"2023-12-19T14:54:58.829542+0000","last_undegraded":"2023-12-19T14:54:58.829542+0000","last_fullsized":"2023-12-19T14:54:58.829542+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:19.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.828661+0000","last_change":"2023-12-19T14:54:58.828661+0000","last_active":"2023-12-19T14:54:58.828661+0000","last_peered":"2023-12-19T14:54:58.828661+0000","last_clean":"2023-12-19T14:54:58.828661+0000","last_became_active":"2023-12-19T14:54:58.828392+0000","last_became_peered":"2023-12-19T14:54:58.828392+0000","last_unstale":"2023-12-19T14:54:58.828661+0000","last_undegraded":"2023-12-19T14:54:58.828661+0000","last_fullsized":"2023-12-19T14:54:58.828661+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:12.089349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823105+0000","last_change":"2023-12-19T14:54:58.823105+0000","last_active":"2023-12-19T14:54:58.823105+0000","last_peered":"2023-12-19T14:54:58.823105+0000","last_clean":"2023-12-19T14:54:58.823105+0000","last_became_active":"2023-12-19T14:54:58.822841+0000","last_became_peered":"2023-12-19T14:54:58.822841+0000","last_unstale":"2023-12-19T14:54:58.823105+0000","last_undegraded":"2023-12-19T14:54:58.823105+0000","last_fullsized":"2023-12-19T14:54:58.823105+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:39:10.741592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824858+0000","last_change":"2023-12-19T14:54:58.824858+0000","last_active":"2023-12-19T14:54:58.824858+0000","last_peered":"2023-12-19T14:54:58.824858+0000","last_clean":"2023-12-19T14:54:58.824858+0000","last_became_active":"2023-12-19T14:54:58.824596+0000","last_became_peered":"2023-12-19T14:54:58.824596+0000","last_unstale":"2023-12-19T14:54:58.824858+0000","last_undegraded":"2023-12-19T14:54:58.824858+0000","last_fullsized":"2023-12-19T14:54:58.824858+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:02.273426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832086+0000","last_change":"2023-12-19T14:54:58.832086+0000","last_active":"2023-12-19T14:54:58.832086+0000","last_peered":"2023-12-19T14:54:58.832086+0000","last_clean":"2023-12-19T14:54:58.832086+0000","last_became_active":"2023-12-19T14:54:58.831632+0000","last_became_peered":"2023-12-19T14:54:58.831632+0000","last_unstale":"2023-12-19T14:54:58.832086+0000","last_undegraded":"2023-12-19T14:54:58.832086+0000","last_fullsized":"2023-12-19T14:54:58.832086+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:32.377041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816649+0000","last_change":"2023-12-19T14:54:58.816649+0000","last_active":"2023-12-19T14:54:58.816649+0000","last_peered":"2023-12-19T14:54:58.816649+0000","last_clean":"2023-12-19T14:54:58.816649+0000","last_became_active":"2023-12-19T14:54:58.816245+0000","last_became_peered":"2023-12-19T14:54:58.816245+0000","last_unstale":"2023-12-19T14:54:58.816649+0000","last_undegraded":"2023-12-19T14:54:58.816649+0000","last_fullsized":"2023-12-19T14:54:58.816649+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:41.720028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824753+0000","last_change":"2023-12-19T14:54:58.824753+0000","last_active":"2023-12-19T14:54:58.824753+0000","last_peered":"2023-12-19T14:54:58.824753+0000","last_clean":"2023-12-19T14:54:58.824753+0000","last_became_active":"2023-12-19T14:54:58.820394+0000","last_became_peered":"2023-12-19T14:54:58.820394+0000","last_unstale":"2023-12-19T14:54:58.824753+0000","last_undegraded":"2023-12-19T14:54:58.824753+0000","last_fullsized":"2023-12-19T14:54:58.824753+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:24.574228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.822573+0000","last_change":"2023-12-19T14:54:58.822573+0000","last_active":"2023-12-19T14:54:58.822573+0000","last_peered":"2023-12-19T14:54:58.822573+0000","last_clean":"2023-12-19T14:54:58.822573+0000","last_became_active":"2023-12-19T14:54:58.822264+0000","last_became_peered":"2023-12-19T14:54:58.822264+0000","last_unstale":"2023-12-19T14:54:58.822573+0000","last_undegraded":"2023-12-19T14:54:58.822573+0000","last_fullsized":"2023-12-19T14:54:58.822573+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:07:18.713895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821174+0000","last_change":"2023-12-19T14:54:58.821174+0000","last_active":"2023-12-19T14:54:58.821174+0000","last_peered":"2023-12-19T14:54:58.821174+0000","last_clean":"2023-12-19T14:54:58.821174+0000","last_became_active":"2023-12-19T14:54:58.820872+0000","last_became_peered":"2023-12-19T14:54:58.820872+0000","last_unstale":"2023-12-19T14:54:58.821174+0000","last_undegraded":"2023-12-19T14:54:58.821174+0000","last_fullsized":"2023-12-19T14:54:58.821174+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:16:19.321573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829384+0000","last_change":"2023-12-19T14:54:58.829384+0000","last_active":"2023-12-19T14:54:58.829384+0000","last_peered":"2023-12-19T14:54:58.829384+0000","last_clean":"2023-12-19T14:54:58.829384+0000","last_became_active":"2023-12-19T14:54:58.828407+0000","last_became_peered":"2023-12-19T14:54:58.828407+0000","last_unstale":"2023-12-19T14:54:58.829384+0000","last_undegraded":"2023-12-19T14:54:58.829384+0000","last_fullsized":"2023-12-19T14:54:58.829384+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:23.055445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821376+0000","last_change":"2023-12-19T14:54:58.821376+0000","last_active":"2023-12-19T14:54:58.821376+0000","last_peered":"2023-12-19T14:54:58.821376+0000","last_clean":"2023-12-19T14:54:58.821376+0000","last_became_active":"2023-12-19T14:54:58.820014+0000","last_became_peered":"2023-12-19T14:54:58.820014+0000","last_unstale":"2023-12-19T14:54:58.821376+0000","last_undegraded":"2023-12-19T14:54:58.821376+0000","last_fullsized":"2023-12-19T14:54:58.821376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:02.081737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.826474+0000","last_change":"2023-12-19T14:54:58.826474+0000","last_active":"2023-12-19T14:54:58.826474+0000","last_peered":"2023-12-19T14:54:58.826474+0000","last_clean":"2023-12-19T14:54:58.826474+0000","last_became_active":"2023-12-19T14:54:58.826102+0000","last_became_peered":"2023-12-19T14:54:58.826102+0000","last_unstale":"2023-12-19T14:54:58.826474+0000","last_undegraded":"2023-12-19T14:54:58.826474+0000","last_fullsized":"2023-12-19T14:54:58.826474+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:12.801733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.819211+0000","last_change":"2023-12-19T14:54:58.819211+0000","last_active":"2023-12-19T14:54:58.819211+0000","last_peered":"2023-12-19T14:54:58.819211+0000","last_clean":"2023-12-19T14:54:58.819211+0000","last_became_active":"2023-12-19T14:54:58.816538+0000","last_became_peered":"2023-12-19T14:54:58.816538+0000","last_unstale":"2023-12-19T14:54:58.819211+0000","last_undegraded":"2023-12-19T14:54:58.819211+0000","last_fullsized":"2023-12-19T14:54:58.819211+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:24.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.835365+0000","last_change":"2023-12-19T14:54:58.835365+0000","last_active":"2023-12-19T14:54:58.835365+0000","last_peered":"2023-12-19T14:54:58.835365+0000","last_clean":"2023-12-19T14:54:58.835365+0000","last_became_active":"2023-12-19T14:54:58.835083+0000","last_became_peered":"2023-12-19T14:54:58.835083+0000","last_unstale":"2023-12-19T14:54:58.835365+0000","last_undegraded":"2023-12-19T14:54:58.835365+0000","last_fullsized":"2023-12-19T14:54:58.835365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:20:55.305099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.835650+0000","last_change":"2023-12-19T14:54:58.835650+0000","last_active":"2023-12-19T14:54:58.835650+0000","last_peered":"2023-12-19T14:54:58.835650+0000","last_clean":"2023-12-19T14:54:58.835650+0000","last_became_active":"2023-12-19T14:54:58.835373+0000","last_became_peered":"2023-12-19T14:54:58.835373+0000","last_unstale":"2023-12-19T14:54:58.835650+0000","last_undegraded":"2023-12-19T14:54:58.835650+0000","last_fullsized":"2023-12-19T14:54:58.835650+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:56.339188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.826662+0000","last_change":"2023-12-19T14:54:58.826662+0000","last_active":"2023-12-19T14:54:58.826662+0000","last_peered":"2023-12-19T14:54:58.826662+0000","last_clean":"2023-12-19T14:54:58.826662+0000","last_became_active":"2023-12-19T14:54:58.825722+0000","last_became_peered":"2023-12-19T14:54:58.825722+0000","last_unstale":"2023-12-19T14:54:58.826662+0000","last_undegraded":"2023-12-19T14:54:58.826662+0000","last_fullsized":"2023-12-19T14:54:58.826662+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:18.203830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818524+0000","last_change":"2023-12-19T14:54:58.818524+0000","last_active":"2023-12-19T14:54:58.818524+0000","last_peered":"2023-12-19T14:54:58.818524+0000","last_clean":"2023-12-19T14:54:58.818524+0000","last_became_active":"2023-12-19T14:54:58.817843+0000","last_became_peered":"2023-12-19T14:54:58.817843+0000","last_unstale":"2023-12-19T14:54:58.818524+0000","last_undegraded":"2023-12-19T14:54:58.818524+0000","last_fullsized":"2023-12-19T14:54:58.818524+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:02.273426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816936+0000","last_change":"2023-12-19T14:54:58.816936+0000","last_active":"2023-12-19T14:54:58.816936+0000","last_peered":"2023-12-19T14:54:58.816936+0000","last_clean":"2023-12-19T14:54:58.816936+0000","last_became_active":"2023-12-19T14:54:58.815613+0000","last_became_peered":"2023-12-19T14:54:58.815613+0000","last_unstale":"2023-12-19T14:54:58.816936+0000","last_undegraded":"2023-12-19T14:54:58.816936+0000","last_fullsized":"2023-12-19T14:54:58.816936+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:22:41.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.833987+0000","last_change":"2023-12-19T14:54:58.833987+0000","last_active":"2023-12-19T14:54:58.833987+0000","last_peered":"2023-12-19T14:54:58.833987+0000","last_clean":"2023-12-19T14:54:58.833987+0000","last_became_active":"2023-12-19T14:54:58.828696+0000","last_became_peered":"2023-12-19T14:54:58.828696+0000","last_unstale":"2023-12-19T14:54:58.833987+0000","last_undegraded":"2023-12-19T14:54:58.833987+0000","last_fullsized":"2023-12-19T14:54:58.833987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:28.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.825408+0000","last_change":"2023-12-19T14:54:58.825408+0000","last_active":"2023-12-19T14:54:58.825408+0000","last_peered":"2023-12-19T14:54:58.825408+0000","last_clean":"2023-12-19T14:54:58.825408+0000","last_became_active":"2023-12-19T14:54:58.825135+0000","last_became_peered":"2023-12-19T14:54:58.825135+0000","last_unstale":"2023-12-19T14:54:58.825408+0000","last_undegraded":"2023-12-19T14:54:58.825408+0000","last_fullsized":"2023-12-19T14:54:58.825408+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:23:58.910293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827681+0000","last_change":"2023-12-19T14:54:58.827681+0000","last_active":"2023-12-19T14:54:58.827681+0000","last_peered":"2023-12-19T14:54:58.827681+0000","last_clean":"2023-12-19T14:54:58.827681+0000","last_became_active":"2023-12-19T14:54:58.827375+0000","last_became_peered":"2023-12-19T14:54:58.827375+0000","last_unstale":"2023-12-19T14:54:58.827681+0000","last_undegraded":"2023-12-19T14:54:58.827681+0000","last_fullsized":"2023-12-19T14:54:58.827681+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:24.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817217+0000","last_change":"2023-12-19T14:54:58.817217+0000","last_active":"2023-12-19T14:54:58.817217+0000","last_peered":"2023-12-19T14:54:58.817217+0000","last_clean":"2023-12-19T14:54:58.817217+0000","last_became_active":"2023-12-19T14:54:58.816912+0000","last_became_peered":"2023-12-19T14:54:58.816912+0000","last_unstale":"2023-12-19T14:54:58.817217+0000","last_undegraded":"2023-12-19T14:54:58.817217+0000","last_fullsized":"2023-12-19T14:54:58.817217+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:12:20.811769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.820639+0000","last_change":"2023-12-19T14:54:58.820639+0000","last_active":"2023-12-19T14:54:58.820639+0000","last_peered":"2023-12-19T14:54:58.820639+0000","last_clean":"2023-12-19T14:54:58.820639+0000","last_became_active":"2023-12-19T14:54:58.820354+0000","last_became_peered":"2023-12-19T14:54:58.820354+0000","last_unstale":"2023-12-19T14:54:58.820639+0000","last_undegraded":"2023-12-19T14:54:58.820639+0000","last_fullsized":"2023-12-19T14:54:58.820639+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:12.798653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"99'4","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.972425+0000","last_change":"2023-12-19T14:54:57.820415+0000","last_active":"2023-12-19T14:54:58.972425+0000","last_peered":"2023-12-19T14:54:58.972425+0000","last_clean":"2023-12-19T14:54:58.972425+0000","last_became_active":"2023-12-19T14:54:57.820140+0000","last_became_peered":"2023-12-19T14:54:57.820140+0000","last_unstale":"2023-12-19T14:54:58.972425+0000","last_undegraded":"2023-12-19T14:54:58.972425+0000","last_fullsized":"2023-12-19T14:54:58.972425+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:36:13.451941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823633+0000","last_change":"2023-12-19T14:54:58.823633+0000","last_active":"2023-12-19T14:54:58.823633+0000","last_peered":"2023-12-19T14:54:58.823633+0000","last_clean":"2023-12-19T14:54:58.823633+0000","last_became_active":"2023-12-19T14:54:58.823347+0000","last_became_peered":"2023-12-19T14:54:58.823347+0000","last_unstale":"2023-12-19T14:54:58.823633+0000","last_undegraded":"2023-12-19T14:54:58.823633+0000","last_fullsized":"2023-12-19T14:54:58.823633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:51:28.154347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.809405+0000","last_change":"2023-12-19T14:54:57.816866+0000","last_active":"2023-12-19T14:54:58.809405+0000","last_peered":"2023-12-19T14:54:58.809405+0000","last_clean":"2023-12-19T14:54:58.809405+0000","last_became_active":"2023-12-19T14:54:57.816597+0000","last_became_peered":"2023-12-19T14:54:57.816597+0000","last_unstale":"2023-12-19T14:54:58.809405+0000","last_undegraded":"2023-12-19T14:54:58.809405+0000","last_fullsized":"2023-12-19T14:54:58.809405+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:11.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831120+0000","last_change":"2023-12-19T14:54:58.831120+0000","last_active":"2023-12-19T14:54:58.831120+0000","last_peered":"2023-12-19T14:54:58.831120+0000","last_clean":"2023-12-19T14:54:58.831120+0000","last_became_active":"2023-12-19T14:54:58.830775+0000","last_became_peered":"2023-12-19T14:54:58.830775+0000","last_unstale":"2023-12-19T14:54:58.831120+0000","last_undegraded":"2023-12-19T14:54:58.831120+0000","last_fullsized":"2023-12-19T14:54:58.831120+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:02.560851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.966589+0000","last_change":"2023-12-19T14:54:57.823340+0000","last_active":"2023-12-19T14:54:58.966589+0000","last_peered":"2023-12-19T14:54:58.966589+0000","last_clean":"2023-12-19T14:54:58.966589+0000","last_became_active":"2023-12-19T14:54:57.823136+0000","last_became_peered":"2023-12-19T14:54:57.823136+0000","last_unstale":"2023-12-19T14:54:58.966589+0000","last_undegraded":"2023-12-19T14:54:58.966589+0000","last_fullsized":"2023-12-19T14:54:58.966589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:21.061111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821565+0000","last_change":"2023-12-19T14:54:58.821565+0000","last_active":"2023-12-19T14:54:58.821565+0000","last_peered":"2023-12-19T14:54:58.821565+0000","last_clean":"2023-12-19T14:54:58.821565+0000","last_became_active":"2023-12-19T14:54:58.818354+0000","last_became_peered":"2023-12-19T14:54:58.818354+0000","last_unstale":"2023-12-19T14:54:58.821565+0000","last_undegraded":"2023-12-19T14:54:58.821565+0000","last_fullsized":"2023-12-19T14:54:58.821565+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:44:20.402010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.965244+0000","last_change":"2023-12-19T14:54:57.814698+0000","last_active":"2023-12-19T14:54:58.965244+0000","last_peered":"2023-12-19T14:54:58.965244+0000","last_clean":"2023-12-19T14:54:58.965244+0000","last_became_active":"2023-12-19T14:54:57.814219+0000","last_became_peered":"2023-12-19T14:54:57.814219+0000","last_unstale":"2023-12-19T14:54:58.965244+0000","last_undegraded":"2023-12-19T14:54:58.965244+0000","last_fullsized":"2023-12-19T14:54:58.965244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:23.558167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832183+0000","last_change":"2023-12-19T14:54:58.832183+0000","last_active":"2023-12-19T14:54:58.832183+0000","last_peered":"2023-12-19T14:54:58.832183+0000","last_clean":"2023-12-19T14:54:58.832183+0000","last_became_active":"2023-12-19T14:54:58.831884+0000","last_became_peered":"2023-12-19T14:54:58.831884+0000","last_unstale":"2023-12-19T14:54:58.832183+0000","last_undegraded":"2023-12-19T14:54:58.832183+0000","last_fullsized":"2023-12-19T14:54:58.832183+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:09:52.478598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.806834+0000","last_change":"2023-12-19T14:54:57.814122+0000","last_active":"2023-12-19T14:54:58.806834+0000","last_peered":"2023-12-19T14:54:58.806834+0000","last_clean":"2023-12-19T14:54:58.806834+0000","last_became_active":"2023-12-19T14:54:57.813795+0000","last_became_peered":"2023-12-19T14:54:57.813795+0000","last_unstale":"2023-12-19T14:54:58.806834+0000","last_undegraded":"2023-12-19T14:54:58.806834+0000","last_fullsized":"2023-12-19T14:54:58.806834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:36.745989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823111+0000","last_change":"2023-12-19T14:54:58.823111+0000","last_active":"2023-12-19T14:54:58.823111+0000","last_peered":"2023-12-19T14:54:58.823111+0000","last_clean":"2023-12-19T14:54:58.823111+0000","last_became_active":"2023-12-19T14:54:58.822801+0000","last_became_peered":"2023-12-19T14:54:58.822801+0000","last_unstale":"2023-12-19T14:54:58.823111+0000","last_undegraded":"2023-12-19T14:54:58.823111+0000","last_fullsized":"2023-12-19T14:54:58.823111+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:58.730300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.806030+0000","last_change":"2023-12-19T14:54:57.811480+0000","last_active":"2023-12-19T14:54:58.806030+0000","last_peered":"2023-12-19T14:54:58.806030+0000","last_clean":"2023-12-19T14:54:58.806030+0000","last_became_active":"2023-12-19T14:54:57.811196+0000","last_became_peered":"2023-12-19T14:54:57.811196+0000","last_unstale":"2023-12-19T14:54:58.806030+0000","last_undegraded":"2023-12-19T14:54:58.806030+0000","last_fullsized":"2023-12-19T14:54:58.806030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:35.098224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829937+0000","last_change":"2023-12-19T14:54:58.829937+0000","last_active":"2023-12-19T14:54:58.829937+0000","last_peered":"2023-12-19T14:54:58.829937+0000","last_clean":"2023-12-19T14:54:58.829937+0000","last_became_active":"2023-12-19T14:54:58.829616+0000","last_became_peered":"2023-12-19T14:54:58.829616+0000","last_unstale":"2023-12-19T14:54:58.829937+0000","last_undegraded":"2023-12-19T14:54:58.829937+0000","last_fullsized":"2023-12-19T14:54:58.829937+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:35.282225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826888+0000","last_change":"2023-12-19T14:54:57.819984+0000","last_active":"2023-12-19T14:54:57.826888+0000","last_peered":"2023-12-19T14:54:57.826888+0000","last_clean":"2023-12-19T14:54:57.826888+0000","last_became_active":"2023-12-19T14:54:57.819646+0000","last_became_peered":"2023-12-19T14:54:57.819646+0000","last_unstale":"2023-12-19T14:54:57.826888+0000","last_undegraded":"2023-12-19T14:54:57.826888+0000","last_fullsized":"2023-12-19T14:54:57.826888+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.814382+0000","last_change":"2023-12-19T14:54:58.814382+0000","last_active":"2023-12-19T14:54:58.814382+0000","last_peered":"2023-12-19T14:54:58.814382+0000","last_clean":"2023-12-19T14:54:58.814382+0000","last_became_active":"2023-12-19T14:54:58.813832+0000","last_became_peered":"2023-12-19T14:54:58.813832+0000","last_unstale":"2023-12-19T14:54:58.814382+0000","last_undegraded":"2023-12-19T14:54:58.814382+0000","last_fullsized":"2023-12-19T14:54:58.814382+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:37.762050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.824188+0000","last_change":"2023-12-19T14:54:57.819225+0000","last_active":"2023-12-19T14:54:57.824188+0000","last_peered":"2023-12-19T14:54:57.824188+0000","last_clean":"2023-12-19T14:54:57.824188+0000","last_became_active":"2023-12-19T14:54:57.818385+0000","last_became_peered":"2023-12-19T14:54:57.818385+0000","last_unstale":"2023-12-19T14:54:57.824188+0000","last_undegraded":"2023-12-19T14:54:57.824188+0000","last_fullsized":"2023-12-19T14:54:57.824188+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:52.422200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824006+0000","last_change":"2023-12-19T14:54:58.824006+0000","last_active":"2023-12-19T14:54:58.824006+0000","last_peered":"2023-12-19T14:54:58.824006+0000","last_clean":"2023-12-19T14:54:58.824006+0000","last_became_active":"2023-12-19T14:54:58.821857+0000","last_became_peered":"2023-12-19T14:54:58.821857+0000","last_unstale":"2023-12-19T14:54:58.824006+0000","last_undegraded":"2023-12-19T14:54:58.824006+0000","last_fullsized":"2023-12-19T14:54:58.824006+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:36.114285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822453+0000","last_change":"2023-12-19T14:54:57.819030+0000","last_active":"2023-12-19T14:54:57.822453+0000","last_peered":"2023-12-19T14:54:57.822453+0000","last_clean":"2023-12-19T14:54:57.822453+0000","last_became_active":"2023-12-19T14:54:57.818385+0000","last_became_peered":"2023-12-19T14:54:57.818385+0000","last_unstale":"2023-12-19T14:54:57.822453+0000","last_undegraded":"2023-12-19T14:54:57.822453+0000","last_fullsized":"2023-12-19T14:54:57.822453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:07:17.697834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.822439+0000","last_change":"2023-12-19T14:54:58.822439+0000","last_active":"2023-12-19T14:54:58.822439+0000","last_peered":"2023-12-19T14:54:58.822439+0000","last_clean":"2023-12-19T14:54:58.822439+0000","last_became_active":"2023-12-19T14:54:58.822104+0000","last_became_peered":"2023-12-19T14:54:58.822104+0000","last_unstale":"2023-12-19T14:54:58.822439+0000","last_undegraded":"2023-12-19T14:54:58.822439+0000","last_fullsized":"2023-12-19T14:54:58.822439+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:24:34.581860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.816567+0000","last_change":"2023-12-19T14:54:57.810499+0000","last_active":"2023-12-19T14:54:57.816567+0000","last_peered":"2023-12-19T14:54:57.816567+0000","last_clean":"2023-12-19T14:54:57.816567+0000","last_became_active":"2023-12-19T14:54:57.810181+0000","last_became_peered":"2023-12-19T14:54:57.810181+0000","last_unstale":"2023-12-19T14:54:57.816567+0000","last_undegraded":"2023-12-19T14:54:57.816567+0000","last_fullsized":"2023-12-19T14:54:57.816567+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:25.568467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824651+0000","last_change":"2023-12-19T14:54:58.824651+0000","last_active":"2023-12-19T14:54:58.824651+0000","last_peered":"2023-12-19T14:54:58.824651+0000","last_clean":"2023-12-19T14:54:58.824651+0000","last_became_active":"2023-12-19T14:54:58.823909+0000","last_became_peered":"2023-12-19T14:54:58.823909+0000","last_unstale":"2023-12-19T14:54:58.824651+0000","last_undegraded":"2023-12-19T14:54:58.824651+0000","last_fullsized":"2023-12-19T14:54:58.824651+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:41.720028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"99'3","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.971535+0000","last_change":"2023-12-19T14:54:57.825102+0000","last_active":"2023-12-19T14:54:58.971535+0000","last_peered":"2023-12-19T14:54:58.971535+0000","last_clean":"2023-12-19T14:54:58.971535+0000","last_became_active":"2023-12-19T14:54:57.824789+0000","last_became_peered":"2023-12-19T14:54:57.824789+0000","last_unstale":"2023-12-19T14:54:58.971535+0000","last_undegraded":"2023-12-19T14:54:58.971535+0000","last_fullsized":"2023-12-19T14:54:58.971535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:29:46.190041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829135+0000","last_change":"2023-12-19T14:54:58.829135+0000","last_active":"2023-12-19T14:54:58.829135+0000","last_peered":"2023-12-19T14:54:58.829135+0000","last_clean":"2023-12-19T14:54:58.829135+0000","last_became_active":"2023-12-19T14:54:58.828801+0000","last_became_peered":"2023-12-19T14:54:58.828801+0000","last_unstale":"2023-12-19T14:54:58.829135+0000","last_undegraded":"2023-12-19T14:54:58.829135+0000","last_fullsized":"2023-12-19T14:54:58.829135+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:35.282225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"99'6","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.967106+0000","last_change":"2023-12-19T14:54:57.813559+0000","last_active":"2023-12-19T14:54:58.967106+0000","last_peered":"2023-12-19T14:54:58.967106+0000","last_clean":"2023-12-19T14:54:58.967106+0000","last_became_active":"2023-12-19T14:54:57.812376+0000","last_became_peered":"2023-12-19T14:54:57.812376+0000","last_unstale":"2023-12-19T14:54:58.967106+0000","last_undegraded":"2023-12-19T14:54:58.967106+0000","last_fullsized":"2023-12-19T14:54:58.967106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:35.795935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.819662+0000","last_change":"2023-12-19T14:54:58.819662+0000","last_active":"2023-12-19T14:54:58.819662+0000","last_peered":"2023-12-19T14:54:58.819662+0000","last_clean":"2023-12-19T14:54:58.819662+0000","last_became_active":"2023-12-19T14:54:58.819379+0000","last_became_peered":"2023-12-19T14:54:58.819379+0000","last_unstale":"2023-12-19T14:54:58.819662+0000","last_undegraded":"2023-12-19T14:54:58.819662+0000","last_fullsized":"2023-12-19T14:54:58.819662+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:56.339188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.964976+0000","last_change":"2023-12-19T14:54:57.812060+0000","last_active":"2023-12-19T14:54:58.964976+0000","last_peered":"2023-12-19T14:54:58.964976+0000","last_clean":"2023-12-19T14:54:58.964976+0000","last_became_active":"2023-12-19T14:54:57.810171+0000","last_became_peered":"2023-12-19T14:54:57.810171+0000","last_unstale":"2023-12-19T14:54:58.964976+0000","last_undegraded":"2023-12-19T14:54:58.964976+0000","last_fullsized":"2023-12-19T14:54:58.964976+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:51:27.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823842+0000","last_change":"2023-12-19T14:54:58.823842+0000","last_active":"2023-12-19T14:54:58.823842+0000","last_peered":"2023-12-19T14:54:58.823842+0000","last_clean":"2023-12-19T14:54:58.823842+0000","last_became_active":"2023-12-19T14:54:58.823579+0000","last_became_peered":"2023-12-19T14:54:58.823579+0000","last_unstale":"2023-12-19T14:54:58.823842+0000","last_undegraded":"2023-12-19T14:54:58.823842+0000","last_fullsized":"2023-12-19T14:54:58.823842+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:41:34.846678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822501+0000","last_change":"2023-12-19T14:54:57.820105+0000","last_active":"2023-12-19T14:54:57.822501+0000","last_peered":"2023-12-19T14:54:57.822501+0000","last_clean":"2023-12-19T14:54:57.822501+0000","last_became_active":"2023-12-19T14:54:57.819771+0000","last_became_peered":"2023-12-19T14:54:57.819771+0000","last_unstale":"2023-12-19T14:54:57.822501+0000","last_undegraded":"2023-12-19T14:54:57.822501+0000","last_fullsized":"2023-12-19T14:54:57.822501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:16.657733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817282+0000","last_change":"2023-12-19T14:54:58.817282+0000","last_active":"2023-12-19T14:54:58.817282+0000","last_peered":"2023-12-19T14:54:58.817282+0000","last_clean":"2023-12-19T14:54:58.817282+0000","last_became_active":"2023-12-19T14:54:58.816947+0000","last_became_peered":"2023-12-19T14:54:58.816947+0000","last_unstale":"2023-12-19T14:54:58.817282+0000","last_undegraded":"2023-12-19T14:54:58.817282+0000","last_fullsized":"2023-12-19T14:54:58.817282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:52:24.807647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.823640+0000","last_change":"2023-12-19T14:54:57.819773+0000","last_active":"2023-12-19T14:54:57.823640+0000","last_peered":"2023-12-19T14:54:57.823640+0000","last_clean":"2023-12-19T14:54:57.823640+0000","last_became_active":"2023-12-19T14:54:57.819476+0000","last_became_peered":"2023-12-19T14:54:57.819476+0000","last_unstale":"2023-12-19T14:54:57.823640+0000","last_undegraded":"2023-12-19T14:54:57.823640+0000","last_fullsized":"2023-12-19T14:54:57.823640+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:37.105873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818724+0000","last_change":"2023-12-19T14:54:58.818724+0000","last_active":"2023-12-19T14:54:58.818724+0000","last_peered":"2023-12-19T14:54:58.818724+0000","last_clean":"2023-12-19T14:54:58.818724+0000","last_became_active":"2023-12-19T14:54:58.818445+0000","last_became_peered":"2023-12-19T14:54:58.818445+0000","last_unstale":"2023-12-19T14:54:58.818724+0000","last_undegraded":"2023-12-19T14:54:58.818724+0000","last_fullsized":"2023-12-19T14:54:58.818724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:29.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"99'8","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.971641+0000","last_change":"2023-12-19T14:54:57.822276+0000","last_active":"2023-12-19T14:54:58.971641+0000","last_peered":"2023-12-19T14:54:58.971641+0000","last_clean":"2023-12-19T14:54:58.971641+0000","last_became_active":"2023-12-19T14:54:57.821973+0000","last_became_peered":"2023-12-19T14:54:57.821973+0000","last_unstale":"2023-12-19T14:54:58.971641+0000","last_undegraded":"2023-12-19T14:54:58.971641+0000","last_fullsized":"2023-12-19T14:54:58.971641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:01.544789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.814453+0000","last_change":"2023-12-19T14:54:58.814453+0000","last_active":"2023-12-19T14:54:58.814453+0000","last_peered":"2023-12-19T14:54:58.814453+0000","last_clean":"2023-12-19T14:54:58.814453+0000","last_became_active":"2023-12-19T14:54:58.814154+0000","last_became_peered":"2023-12-19T14:54:58.814154+0000","last_unstale":"2023-12-19T14:54:58.814453+0000","last_undegraded":"2023-12-19T14:54:58.814453+0000","last_fullsized":"2023-12-19T14:54:58.814453+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:55.388516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.807749+0000","last_change":"2023-12-19T14:54:57.819221+0000","last_active":"2023-12-19T14:54:58.807749+0000","last_peered":"2023-12-19T14:54:58.807749+0000","last_clean":"2023-12-19T14:54:58.807749+0000","last_became_active":"2023-12-19T14:54:57.818757+0000","last_became_peered":"2023-12-19T14:54:57.818757+0000","last_unstale":"2023-12-19T14:54:58.807749+0000","last_undegraded":"2023-12-19T14:54:58.807749+0000","last_fullsized":"2023-12-19T14:54:58.807749+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:41:33.830617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823072+0000","last_change":"2023-12-19T14:54:58.823072+0000","last_active":"2023-12-19T14:54:58.823072+0000","last_peered":"2023-12-19T14:54:58.823072+0000","last_clean":"2023-12-19T14:54:58.823072+0000","last_became_active":"2023-12-19T14:54:58.820347+0000","last_became_peered":"2023-12-19T14:54:58.820347+0000","last_unstale":"2023-12-19T14:54:58.823072+0000","last_undegraded":"2023-12-19T14:54:58.823072+0000","last_fullsized":"2023-12-19T14:54:58.823072+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:36.811997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.965832+0000","last_change":"2023-12-19T14:54:57.815395+0000","last_active":"2023-12-19T14:54:58.965832+0000","last_peered":"2023-12-19T14:54:58.965832+0000","last_clean":"2023-12-19T14:54:58.965832+0000","last_became_active":"2023-12-19T14:54:57.815066+0000","last_became_peered":"2023-12-19T14:54:57.815066+0000","last_unstale":"2023-12-19T14:54:58.965832+0000","last_undegraded":"2023-12-19T14:54:58.965832+0000","last_fullsized":"2023-12-19T14:54:58.965832+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:48.798737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821764+0000","last_change":"2023-12-19T14:54:58.821764+0000","last_active":"2023-12-19T14:54:58.821764+0000","last_peered":"2023-12-19T14:54:58.821764+0000","last_clean":"2023-12-19T14:54:58.821764+0000","last_became_active":"2023-12-19T14:54:58.821497+0000","last_became_peered":"2023-12-19T14:54:58.821497+0000","last_unstale":"2023-12-19T14:54:58.821764+0000","last_undegraded":"2023-12-19T14:54:58.821764+0000","last_fullsized":"2023-12-19T14:54:58.821764+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:21.971472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.802169+0000","last_change":"2023-12-19T14:54:57.821220+0000","last_active":"2023-12-19T14:54:58.802169+0000","last_peered":"2023-12-19T14:54:58.802169+0000","last_clean":"2023-12-19T14:54:58.802169+0000","last_became_active":"2023-12-19T14:54:57.818528+0000","last_became_peered":"2023-12-19T14:54:57.818528+0000","last_unstale":"2023-12-19T14:54:58.802169+0000","last_undegraded":"2023-12-19T14:54:58.802169+0000","last_fullsized":"2023-12-19T14:54:58.802169+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:08.262253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816157+0000","last_change":"2023-12-19T14:54:58.816157+0000","last_active":"2023-12-19T14:54:58.816157+0000","last_peered":"2023-12-19T14:54:58.816157+0000","last_clean":"2023-12-19T14:54:58.816157+0000","last_became_active":"2023-12-19T14:54:58.815884+0000","last_became_peered":"2023-12-19T14:54:58.815884+0000","last_unstale":"2023-12-19T14:54:58.816157+0000","last_undegraded":"2023-12-19T14:54:58.816157+0000","last_fullsized":"2023-12-19T14:54:58.816157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:30:46.597215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.827514+0000","last_change":"2023-12-19T14:54:57.823532+0000","last_active":"2023-12-19T14:54:57.827514+0000","last_peered":"2023-12-19T14:54:57.827514+0000","last_clean":"2023-12-19T14:54:57.827514+0000","last_became_active":"2023-12-19T14:54:57.823250+0000","last_became_peered":"2023-12-19T14:54:57.823250+0000","last_unstale":"2023-12-19T14:54:57.827514+0000","last_undegraded":"2023-12-19T14:54:57.827514+0000","last_fullsized":"2023-12-19T14:54:57.827514+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:52:23.791586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831427+0000","last_change":"2023-12-19T14:54:58.831427+0000","last_active":"2023-12-19T14:54:58.831427+0000","last_peered":"2023-12-19T14:54:58.831427+0000","last_clean":"2023-12-19T14:54:58.831427+0000","last_became_active":"2023-12-19T14:54:58.831093+0000","last_became_peered":"2023-12-19T14:54:58.831093+0000","last_unstale":"2023-12-19T14:54:58.831427+0000","last_undegraded":"2023-12-19T14:54:58.831427+0000","last_fullsized":"2023-12-19T14:54:58.831427+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:18.629358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.970331+0000","last_change":"2023-12-19T14:54:57.824569+0000","last_active":"2023-12-19T14:54:58.970331+0000","last_peered":"2023-12-19T14:54:58.970331+0000","last_clean":"2023-12-19T14:54:58.970331+0000","last_became_active":"2023-12-19T14:54:57.824278+0000","last_became_peered":"2023-12-19T14:54:57.824278+0000","last_unstale":"2023-12-19T14:54:58.970331+0000","last_undegraded":"2023-12-19T14:54:58.970331+0000","last_fullsized":"2023-12-19T14:54:58.970331+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831017+0000","last_change":"2023-12-19T14:54:58.831017+0000","last_active":"2023-12-19T14:54:58.831017+0000","last_peered":"2023-12-19T14:54:58.831017+0000","last_clean":"2023-12-19T14:54:58.831017+0000","last_became_active":"2023-12-19T14:54:58.830711+0000","last_became_peered":"2023-12-19T14:54:58.830711+0000","last_unstale":"2023-12-19T14:54:58.831017+0000","last_undegraded":"2023-12-19T14:54:58.831017+0000","last_fullsized":"2023-12-19T14:54:58.831017+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:26.584529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.828208+0000","last_change":"2023-12-19T14:54:57.819079+0000","last_active":"2023-12-19T14:54:57.828208+0000","last_peered":"2023-12-19T14:54:57.828208+0000","last_clean":"2023-12-19T14:54:57.828208+0000","last_became_active":"2023-12-19T14:54:57.818645+0000","last_became_peered":"2023-12-19T14:54:57.818645+0000","last_unstale":"2023-12-19T14:54:57.828208+0000","last_undegraded":"2023-12-19T14:54:57.828208+0000","last_fullsized":"2023-12-19T14:54:57.828208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:34.266164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.815860+0000","last_change":"2023-12-19T14:54:58.815860+0000","last_active":"2023-12-19T14:54:58.815860+0000","last_peered":"2023-12-19T14:54:58.815860+0000","last_clean":"2023-12-19T14:54:58.815860+0000","last_became_active":"2023-12-19T14:54:58.815395+0000","last_became_peered":"2023-12-19T14:54:58.815395+0000","last_unstale":"2023-12-19T14:54:58.815860+0000","last_undegraded":"2023-12-19T14:54:58.815860+0000","last_fullsized":"2023-12-19T14:54:58.815860+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:44:38.311240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.828409+0000","last_change":"2023-12-19T14:54:57.820540+0000","last_active":"2023-12-19T14:54:57.828409+0000","last_peered":"2023-12-19T14:54:57.828409+0000","last_clean":"2023-12-19T14:54:57.828409+0000","last_became_active":"2023-12-19T14:54:57.818896+0000","last_became_peered":"2023-12-19T14:54:57.818896+0000","last_unstale":"2023-12-19T14:54:57.828409+0000","last_undegraded":"2023-12-19T14:54:57.828409+0000","last_fullsized":"2023-12-19T14:54:57.828409+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:01.544789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.825254+0000","last_change":"2023-12-19T14:54:58.825254+0000","last_active":"2023-12-19T14:54:58.825254+0000","last_peered":"2023-12-19T14:54:58.825254+0000","last_clean":"2023-12-19T14:54:58.825254+0000","last_became_active":"2023-12-19T14:54:58.824967+0000","last_became_peered":"2023-12-19T14:54:58.824967+0000","last_unstale":"2023-12-19T14:54:58.825254+0000","last_undegraded":"2023-12-19T14:54:58.825254+0000","last_fullsized":"2023-12-19T14:54:58.825254+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:03.999291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.964014+0000","last_change":"2023-12-19T14:54:57.818908+0000","last_active":"2023-12-19T14:54:58.964014+0000","last_peered":"2023-12-19T14:54:58.964014+0000","last_clean":"2023-12-19T14:54:58.964014+0000","last_became_active":"2023-12-19T14:54:57.818513+0000","last_became_peered":"2023-12-19T14:54:57.818513+0000","last_unstale":"2023-12-19T14:54:58.964014+0000","last_undegraded":"2023-12-19T14:54:58.964014+0000","last_fullsized":"2023-12-19T14:54:58.964014+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:45.338402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827728+0000","last_change":"2023-12-19T14:54:58.827728+0000","last_active":"2023-12-19T14:54:58.827728+0000","last_peered":"2023-12-19T14:54:58.827728+0000","last_clean":"2023-12-19T14:54:58.827728+0000","last_became_active":"2023-12-19T14:54:58.827246+0000","last_became_peered":"2023-12-19T14:54:58.827246+0000","last_unstale":"2023-12-19T14:54:58.827728+0000","last_undegraded":"2023-12-19T14:54:58.827728+0000","last_fullsized":"2023-12-19T14:54:58.827728+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:57.796572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826943+0000","last_change":"2023-12-19T14:54:57.820718+0000","last_active":"2023-12-19T14:54:57.826943+0000","last_peered":"2023-12-19T14:54:57.826943+0000","last_clean":"2023-12-19T14:54:57.826943+0000","last_became_active":"2023-12-19T14:54:57.820257+0000","last_became_peered":"2023-12-19T14:54:57.820257+0000","last_unstale":"2023-12-19T14:54:57.826943+0000","last_undegraded":"2023-12-19T14:54:57.826943+0000","last_fullsized":"2023-12-19T14:54:57.826943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:11.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824911+0000","last_change":"2023-12-19T14:54:58.824911+0000","last_active":"2023-12-19T14:54:58.824911+0000","last_peered":"2023-12-19T14:54:58.824911+0000","last_clean":"2023-12-19T14:54:58.824911+0000","last_became_active":"2023-12-19T14:54:58.823038+0000","last_became_peered":"2023-12-19T14:54:58.823038+0000","last_unstale":"2023-12-19T14:54:58.824911+0000","last_undegraded":"2023-12-19T14:54:58.824911+0000","last_fullsized":"2023-12-19T14:54:58.824911+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:00.765295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822461+0000","last_change":"2023-12-19T14:54:57.819815+0000","last_active":"2023-12-19T14:54:57.822461+0000","last_peered":"2023-12-19T14:54:57.822461+0000","last_clean":"2023-12-19T14:54:57.822461+0000","last_became_active":"2023-12-19T14:54:57.819511+0000","last_became_peered":"2023-12-19T14:54:57.819511+0000","last_unstale":"2023-12-19T14:54:57.822461+0000","last_undegraded":"2023-12-19T14:54:57.822461+0000","last_fullsized":"2023-12-19T14:54:57.822461+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.833729+0000","last_change":"2023-12-19T14:54:58.833729+0000","last_active":"2023-12-19T14:54:58.833729+0000","last_peered":"2023-12-19T14:54:58.833729+0000","last_clean":"2023-12-19T14:54:58.833729+0000","last_became_active":"2023-12-19T14:54:58.827596+0000","last_became_peered":"2023-12-19T14:54:58.827596+0000","last_unstale":"2023-12-19T14:54:58.833729+0000","last_undegraded":"2023-12-19T14:54:58.833729+0000","last_fullsized":"2023-12-19T14:54:58.833729+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:46.354463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.818069+0000","last_change":"2023-12-19T14:54:57.815372+0000","last_active":"2023-12-19T14:54:57.818069+0000","last_peered":"2023-12-19T14:54:57.818069+0000","last_clean":"2023-12-19T14:54:57.818069+0000","last_became_active":"2023-12-19T14:54:57.815069+0000","last_became_peered":"2023-12-19T14:54:57.815069+0000","last_unstale":"2023-12-19T14:54:57.818069+0000","last_undegraded":"2023-12-19T14:54:57.818069+0000","last_fullsized":"2023-12-19T14:54:57.818069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:49:34.468525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827508+0000","last_change":"2023-12-19T14:54:58.827508+0000","last_active":"2023-12-19T14:54:58.827508+0000","last_peered":"2023-12-19T14:54:58.827508+0000","last_clean":"2023-12-19T14:54:58.827508+0000","last_became_active":"2023-12-19T14:54:58.827238+0000","last_became_peered":"2023-12-19T14:54:58.827238+0000","last_unstale":"2023-12-19T14:54:58.827508+0000","last_undegraded":"2023-12-19T14:54:58.827508+0000","last_fullsized":"2023-12-19T14:54:58.827508+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:02:06.725392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.820349+0000","last_change":"2023-12-19T14:54:57.817044+0000","last_active":"2023-12-19T14:54:57.820349+0000","last_peered":"2023-12-19T14:54:57.820349+0000","last_clean":"2023-12-19T14:54:57.820349+0000","last_became_active":"2023-12-19T14:54:57.816580+0000","last_became_peered":"2023-12-19T14:54:57.816580+0000","last_unstale":"2023-12-19T14:54:57.820349+0000","last_undegraded":"2023-12-19T14:54:57.820349+0000","last_fullsized":"2023-12-19T14:54:57.820349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:14.599834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.812365+0000","last_change":"2023-12-19T14:54:58.812365+0000","last_active":"2023-12-19T14:54:58.812365+0000","last_peered":"2023-12-19T14:54:58.812365+0000","last_clean":"2023-12-19T14:54:58.812365+0000","last_became_active":"2023-12-19T14:54:58.812069+0000","last_became_peered":"2023-12-19T14:54:58.812069+0000","last_unstale":"2023-12-19T14:54:58.812365+0000","last_undegraded":"2023-12-19T14:54:58.812365+0000","last_fullsized":"2023-12-19T14:54:58.812365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:49.814798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826080+0000","last_change":"2023-12-19T14:54:57.819715+0000","last_active":"2023-12-19T14:54:57.826080+0000","last_peered":"2023-12-19T14:54:57.826080+0000","last_clean":"2023-12-19T14:54:57.826080+0000","last_became_active":"2023-12-19T14:54:57.819431+0000","last_became_peered":"2023-12-19T14:54:57.819431+0000","last_unstale":"2023-12-19T14:54:57.826080+0000","last_undegraded":"2023-12-19T14:54:57.826080+0000","last_fullsized":"2023-12-19T14:54:57.826080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:37.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.830224+0000","last_change":"2023-12-19T14:54:58.830224+0000","last_active":"2023-12-19T14:54:58.830224+0000","last_peered":"2023-12-19T14:54:58.830224+0000","last_clean":"2023-12-19T14:54:58.830224+0000","last_became_active":"2023-12-19T14:54:58.829850+0000","last_became_peered":"2023-12-19T14:54:58.829850+0000","last_unstale":"2023-12-19T14:54:58.830224+0000","last_undegraded":"2023-12-19T14:54:58.830224+0000","last_fullsized":"2023-12-19T14:54:58.830224+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:02.560851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.817480+0000","last_change":"2023-12-19T14:54:57.813153+0000","last_active":"2023-12-19T14:54:57.817480+0000","last_peered":"2023-12-19T14:54:57.817480+0000","last_clean":"2023-12-19T14:54:57.817480+0000","last_became_active":"2023-12-19T14:54:57.812154+0000","last_became_peered":"2023-12-19T14:54:57.812154+0000","last_unstale":"2023-12-19T14:54:57.817480+0000","last_undegraded":"2023-12-19T14:54:57.817480+0000","last_fullsized":"2023-12-19T14:54:57.817480+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:20.955411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823597+0000","last_change":"2023-12-19T14:54:58.823597+0000","last_active":"2023-12-19T14:54:58.823597+0000","last_peered":"2023-12-19T14:54:58.823597+0000","last_clean":"2023-12-19T14:54:58.823597+0000","last_became_active":"2023-12-19T14:54:58.821449+0000","last_became_peered":"2023-12-19T14:54:58.821449+0000","last_unstale":"2023-12-19T14:54:58.823597+0000","last_undegraded":"2023-12-19T14:54:58.823597+0000","last_fullsized":"2023-12-19T14:54:58.823597+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:49:35.484586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822631+0000","last_change":"2023-12-19T14:54:57.815024+0000","last_active":"2023-12-19T14:54:57.822631+0000","last_peered":"2023-12-19T14:54:57.822631+0000","last_clean":"2023-12-19T14:54:57.822631+0000","last_became_active":"2023-12-19T14:54:57.814715+0000","last_became_peered":"2023-12-19T14:54:57.814715+0000","last_unstale":"2023-12-19T14:54:57.822631+0000","last_undegraded":"2023-12-19T14:54:57.822631+0000","last_fullsized":"2023-12-19T14:54:57.822631+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:44:37.295179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821052+0000","last_change":"2023-12-19T14:54:58.821052+0000","last_active":"2023-12-19T14:54:58.821052+0000","last_peered":"2023-12-19T14:54:58.821052+0000","last_clean":"2023-12-19T14:54:58.821052+0000","last_became_active":"2023-12-19T14:54:58.820795+0000","last_became_peered":"2023-12-19T14:54:58.820795+0000","last_unstale":"2023-12-19T14:54:58.821052+0000","last_undegraded":"2023-12-19T14:54:58.821052+0000","last_fullsized":"2023-12-19T14:54:58.821052+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:17.108734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822313+0000","last_change":"2023-12-19T14:54:57.816532+0000","last_active":"2023-12-19T14:54:57.822313+0000","last_peered":"2023-12-19T14:54:57.822313+0000","last_clean":"2023-12-19T14:54:57.822313+0000","last_became_active":"2023-12-19T14:54:57.815581+0000","last_became_peered":"2023-12-19T14:54:57.815581+0000","last_unstale":"2023-12-19T14:54:57.822313+0000","last_undegraded":"2023-12-19T14:54:57.822313+0000","last_fullsized":"2023-12-19T14:54:57.822313+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:23.945969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'183","reported_seq":464,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.804182+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:54:57.804182+0000","last_peered":"2023-12-19T14:54:57.804182+0000","last_clean":"2023-12-19T14:54:57.804182+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:54:57.804182+0000","last_undegraded":"2023-12-19T14:54:57.804182+0000","last_fullsized":"2023-12-19T14:54:57.804182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311344,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060496,"statfs":{"total":96636764160,"available":96317947904,"internally_reserved":0,"allocated":1810432,"data_stored":1196241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.498,"5min":3.544,"15min":3.544},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81699999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5369999999999999,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.093},"min":{"1min":0.91700000000000004,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.4379999999999999,"5min":1.73,"15min":1.73},"last":1.115},{"interface":"front","average":{"1min":1.276,"5min":1.1950000000000001,"15min":1.1819999999999999},"min":{"1min":0.93400000000000005,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.984},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2729999999999999,"15min":1.2330000000000001},"min":{"1min":0.96699999999999997,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.571,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.274,"15min":1.278},"min":{"1min":1.1359999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.667,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":0.97399999999999998}]},{"osd":3,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.28,"15min":1.29},"min":{"1min":0.92500000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.5409999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.423},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.3109999999999999},"min":{"1min":1.038,"5min":0.95899999999999996,"15min":0.79700000000000004},"max":{"1min":1.6100000000000001,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.044}]},{"osd":4,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.1839999999999999,"15min":1.1699999999999999},"min":{"1min":0.68500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.752,"5min":1.752,"15min":2.1930000000000001},"last":1.091},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2410000000000001,"15min":1.222},"min":{"1min":0.90300000000000002,"5min":0.75700000000000001,"15min":0.67500000000000004},"max":{"1min":1.6000000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.4910000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2210000000000001,"15min":1.2270000000000001},"min":{"1min":1.034,"5min":0.496,"15min":0.496},"max":{"1min":1.6930000000000001,"5min":1.8109999999999999,"15min":1.976},"last":1.2},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3029999999999999,"15min":1.3180000000000001},"min":{"1min":0.90200000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.3620000000000001},"last":1.3580000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3899999999999999,"15min":1.3999999999999999},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.65600000000000003},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":3.1589999999999998},"last":1.274},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3620000000000001,"15min":1.361},"min":{"1min":1.0620000000000001,"5min":0.88500000000000001,"15min":0.70299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.5609999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302556,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069284,"statfs":{"total":96636764160,"available":96326946816,"internally_reserved":0,"allocated":1200128,"data_stored":598740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1220000000000001,"15min":1.1479999999999999},"min":{"1min":0.73699999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.466,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.1890000000000001},"min":{"1min":0.80800000000000005,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.577,"5min":1.845,"15min":2.335},"last":1.224}]},{"osd":1,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2090000000000001,"15min":1.2070000000000001},"min":{"1min":0.86899999999999999,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":1.4790000000000001,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.444},{"interface":"front","average":{"1min":1.194,"5min":1.2569999999999999,"15min":1.2649999999999999},"min":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.419,"5min":3.226,"15min":3.226},"last":1.1259999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.284,"15min":1.264},"min":{"1min":0.86699999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6519999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.2769999999999999,"15min":1.2569999999999999},"min":{"1min":0.98399999999999999,"5min":0.82399999999999995,"15min":0.67100000000000004},"max":{"1min":1.4159999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.292}]},{"osd":3,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3149999999999999,"15min":1.3200000000000001},"min":{"1min":0.88300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.587,"5min":1.9239999999999999,"15min":3.2330000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.353,"15min":1.355},"min":{"1min":0.94899999999999995,"5min":0.84299999999999997,"15min":0.69599999999999995},"max":{"1min":1.673,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.246,"15min":1.2330000000000001},"min":{"1min":0.83899999999999997,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":1.4419999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2809999999999999,"15min":1.3220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.57599999999999996},"max":{"1min":1.5960000000000001,"5min":3.3460000000000001,"15min":3.4399999999999999},"last":1.268}]},{"osd":5,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3109999999999999,"15min":1.327},"min":{"1min":0.89200000000000002,"5min":0.71799999999999997,"15min":0.63100000000000001},"max":{"1min":1.746,"5min":2.02,"15min":2.8620000000000001},"last":1.351},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.407,"15min":1.3839999999999999},"min":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.51000000000000001},"max":{"1min":1.5580000000000001,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.1950000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.4379999999999999},"min":{"1min":0.88300000000000001,"5min":0.85899999999999999,"15min":0.53000000000000003},"max":{"1min":1.8029999999999999,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.377},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4890000000000001,"15min":1.486},"min":{"1min":0.98299999999999998,"5min":0.67100000000000004,"15min":0.63100000000000001},"max":{"1min":1.8089999999999999,"5min":3.742,"15min":3.742},"last":1.411}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298472,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073368,"statfs":{"total":96636764160,"available":96331128832,"internally_reserved":0,"allocated":1212416,"data_stored":605851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0509999999999999,"15min":1.0600000000000001},"min":{"1min":0.71399999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4450000000000001,"5min":1.601,"15min":1.619},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.1379999999999999,"15min":1.1359999999999999},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.58699999999999997},"max":{"1min":1.417,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.4430000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.29},"min":{"1min":0.78500000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.508,"5min":1.8029999999999999,"15min":1.9650000000000001},"last":1.403},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.23,"15min":1.242},"min":{"1min":0.79000000000000004,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.677,"5min":1.677,"15min":1.7869999999999999},"last":1.2989999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2070000000000001,"15min":1.2350000000000001},"min":{"1min":0.99399999999999999,"5min":0.67000000000000004,"15min":0.47399999999999998},"max":{"1min":1.48,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.1599999999999999,"15min":1.2010000000000001},"min":{"1min":0.90000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.5720000000000001,"5min":1.609,"15min":2.0739999999999998},"last":0.94499999999999995}]},{"osd":3,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2869999999999999,"15min":1.2749999999999999},"min":{"1min":0.96099999999999997,"5min":0.76200000000000001,"15min":0.73199999999999998},"max":{"1min":1.6499999999999999,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.137},{"interface":"front","average":{"1min":1.246,"5min":1.2410000000000001,"15min":1.2729999999999999},"min":{"1min":0.97799999999999998,"5min":0.82299999999999995,"15min":0.68999999999999995},"max":{"1min":1.601,"5min":1.73,"15min":2.2149999999999999},"last":1.4990000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1140000000000001,"15min":1.1200000000000001},"min":{"1min":0.86899999999999999,"5min":0.72799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5820000000000001,"5min":1.603,"15min":2.1099999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2450000000000001,"15min":1.2470000000000001},"min":{"1min":0.80000000000000004,"5min":0.69099999999999995,"15min":0.49299999999999999},"max":{"1min":1.6950000000000001,"5min":2.27,"15min":2.27},"last":0.83999999999999997}]},{"osd":6,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3380000000000001},"min":{"1min":0.84399999999999997,"5min":0.79600000000000004,"15min":0.76000000000000001},"max":{"1min":1.982,"5min":1.982,"15min":2.4790000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.333},"min":{"1min":1.123,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.794,"5min":1.8540000000000001,"15min":2.0030000000000001},"last":1.3759999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":1.0649999999999999,"5min":0.76700000000000002,"15min":0.58899999999999997},"max":{"1min":1.7490000000000001,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.524},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3540000000000001,"15min":1.3919999999999999},"min":{"1min":1.054,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.411},"last":1.2390000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575040,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302588,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069252,"statfs":{"total":96636764160,"available":96326914048,"internally_reserved":0,"allocated":1232896,"data_stored":607491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0609999999999999,"15min":1.069},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.7869999999999999},"last":1.405},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.095,"15min":1.091},"min":{"1min":0.69499999999999995,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.5109999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.1559999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.236,"15min":1.2430000000000001},"min":{"1min":0.75,"5min":0.623,"15min":0.623},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":2.1259999999999999},"last":1.454},{"interface":"front","average":{"1min":1.222,"5min":1.244,"15min":1.276},"min":{"1min":0.73299999999999998,"5min":0.70299999999999996,"15min":0.65100000000000002},"max":{"1min":1.595,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.518}]},{"osd":2,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.165,"15min":1.1539999999999999},"min":{"1min":0.91600000000000004,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.625,"5min":1.625,"15min":1.891},"last":1.113},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1819999999999999,"15min":1.2070000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.893},"last":1.2969999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.163,"15min":1.1830000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.59599999999999997},"max":{"1min":1.359,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.256,"15min":1.2729999999999999},"min":{"1min":1.034,"5min":0.83799999999999997,"15min":0.60599999999999998},"max":{"1min":1.6259999999999999,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.329}]},{"osd":5,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1499999999999999,"15min":1.1619999999999999},"min":{"1min":0.72399999999999998,"5min":0.71099999999999997,"15min":0.57899999999999996},"max":{"1min":1.452,"5min":2.1800000000000002,"15min":2.3500000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1419999999999999,"15min":1.1679999999999999},"min":{"1min":0.79600000000000004,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.5840000000000001,"5min":2.077,"15min":2.161},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.276,"15min":1.323},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.6579999999999999,"5min":2.8119999999999998,"15min":3.27},"last":1.236},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.421},"min":{"1min":0.63400000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.843,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.5900000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.357},"min":{"1min":0.89200000000000002,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.637,"5min":2.9910000000000001,"15min":5.2060000000000004},"last":1.484},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4319999999999999,"15min":1.46},"min":{"1min":0.89900000000000002,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":2.0329999999999999,"5min":3.008,"15min":3.6739999999999999},"last":1.554}]}]},{"osd":3,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306668,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065172,"statfs":{"total":96636764160,"available":96322736128,"internally_reserved":0,"allocated":1216512,"data_stored":606405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84999999999999998,"15min":0.89000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2010000000000001,"5min":1.3640000000000001,"15min":4.1159999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.026,"5min":1.01,"15min":1.0129999999999999},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.542,"5min":1.542,"15min":3.3929999999999998},"last":1.111}]},{"osd":1,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":0.95799999999999996},"min":{"1min":0.69599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0149999999999999,"5min":1.611,"15min":2.4390000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.121,"15min":1.1220000000000001},"min":{"1min":0.749,"5min":0.63500000000000001,"15min":0.60199999999999998},"max":{"1min":1.5189999999999999,"5min":1.661,"15min":4.476},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1459999999999999,"15min":1.1419999999999999},"min":{"1min":0.89900000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.639,"5min":1.76,"15min":2.6989999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61699999999999999},"max":{"1min":1.5309999999999999,"5min":1.647,"15min":2.5110000000000001},"last":1.367}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.345},"min":{"1min":0.84899999999999998,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.526,"5min":1.768,"15min":3.46},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2829999999999999,"15min":1.3069999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.542,"5min":1.6970000000000001,"15min":2.9409999999999998},"last":1.2669999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.286,"15min":1.3280000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.629,"5min":1.7709999999999999,"15min":3.1019999999999999},"last":1.04},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.4159999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.833,"5min":2.1619999999999999,"15min":4.3650000000000002},"last":1.1499999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.446,"15min":1.4730000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9099999999999999,"5min":2.2709999999999999,"15min":4.5999999999999996},"last":1.3},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.419,"15min":1.444},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":3.4329999999999998},"last":1.325}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.516,"15min":1.5429999999999999},"min":{"1min":0.93300000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.7629999999999999,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.508},{"interface":"front","average":{"1min":1.429,"5min":1.45,"15min":1.488},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.728,"5min":2.0670000000000002,"15min":3.7240000000000002},"last":1.1930000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":1208320,"data_stored":604840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89400000000000002,"15min":0.90400000000000003},"min":{"1min":0.60599999999999998,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.946},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.056,"15min":1.0720000000000001},"min":{"1min":0.63300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.605,"5min":1.605,"15min":1.8560000000000001},"last":1.2250000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.82199999999999995,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.871}]},{"osd":3,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1.0009999999999999},"min":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.304,"5min":1.5669999999999999,"15min":1.9370000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.081,"15min":1.105},"min":{"1min":0.80100000000000005,"5min":0.66000000000000003,"15min":0.55000000000000004},"max":{"1min":1.603,"5min":1.603,"15min":1.6930000000000001},"last":0.99199999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":0.95199999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":1.893,"15min":3.4950000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.403,"15min":1.3999999999999999},"min":{"1min":1.2230000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.923,"5min":1.96,"15min":2.0409999999999999},"last":1.5209999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3740000000000001,"15min":1.3600000000000001},"min":{"1min":1.1040000000000001,"5min":0.876,"15min":0.876},"max":{"1min":1.7669999999999999,"5min":2.0760000000000001,"15min":3.8599999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.93899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.6759999999999999,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.4610000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.80600000000000005},"max":{"1min":1.6950000000000001,"5min":1.829,"15min":2.8759999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4370000000000001,"15min":1.45},"min":{"1min":1.2170000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8280000000000001,"5min":2.1200000000000001,"15min":3.7309999999999999},"last":1.395}]},{"osd":7,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.482,"15min":1.492},"min":{"1min":1.171,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.044,"5min":2.0699999999999998,"15min":2.5550000000000002},"last":1.651},{"interface":"front","average":{"1min":1.599,"5min":1.4710000000000001,"15min":1.4870000000000001},"min":{"1min":1.258,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.4049999999999998},"last":1.371}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306672,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065168,"statfs":{"total":96636764160,"available":96322732032,"internally_reserved":0,"allocated":1220608,"data_stored":605963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.877,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":1.238,"5min":1.464,"15min":1.871},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.98899999999999999,"15min":0.996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.50700000000000001},"max":{"1min":1.393,"5min":1.7070000000000001,"15min":2.468},"last":0.98499999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0169999999999999,"15min":1.0209999999999999},"min":{"1min":0.71699999999999997,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":1.2589999999999999,"5min":1.714,"15min":1.7829999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.66300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.5660000000000001,"5min":2.0230000000000001,"15min":3.0619999999999998},"last":1.2589999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.236,"15min":1.2350000000000001},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.0820000000000001},"min":{"1min":0.81200000000000006,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6439999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.0609999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2869999999999999,"15min":1.3109999999999999},"min":{"1min":0.93000000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.724,"5min":1.7370000000000001,"15min":2.6440000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.401,"15min":1.403},"min":{"1min":1.0009999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.677,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.5149999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.262,"15min":1.296},"min":{"1min":1.028,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.538,"5min":1.6719999999999999,"15min":2.1320000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3720000000000001,"15min":1.391},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.76,"5min":1.76,"15min":2.79},"last":1.409}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.333,"15min":1.3640000000000001},"min":{"1min":1.0129999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7070000000000001,"5min":1.78,"15min":2.4620000000000002},"last":1.343},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4319999999999999,"15min":1.456},"min":{"1min":1.073,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.8999999999999999},"last":1.242}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.474,"15min":1.488},"min":{"1min":1.0740000000000001,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":2.0470000000000002,"15min":3.782},"last":1.633},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":1.046,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":2.5390000000000001},"last":1.5609999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307240,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064600,"statfs":{"total":96636764160,"available":96322150400,"internally_reserved":0,"allocated":1802240,"data_stored":1195208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.97299999999999998},"min":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.53600000000000003},"max":{"1min":1.4810000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.22}]},{"osd":2,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.80000000000000004,"5min":0.69699999999999995,"15min":0.55200000000000005},"max":{"1min":1.5189999999999999,"5min":1.6339999999999999,"15min":1.8979999999999999},"last":1.462},{"interface":"front","average":{"1min":1.288,"5min":1.2410000000000001,"15min":1.2210000000000001},"min":{"1min":0.82199999999999995,"5min":0.70799999999999996,"15min":0.54300000000000004},"max":{"1min":1.728,"5min":1.982,"15min":1.982},"last":1.0680000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6359999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.091,"5min":1.1679999999999999,"15min":1.1819999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.361}]},{"osd":4,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3169999999999999,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.70199999999999996},"max":{"1min":1.754,"5min":2.153,"15min":2.298},"last":1.123},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.302,"15min":1.298},"min":{"1min":1.026,"5min":0.83999999999999997,"15min":0.80200000000000005},"max":{"1min":1.579,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.159}]},{"osd":5,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3600000000000001,"15min":1.375},"min":{"1min":0.89300000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9410000000000001,"5min":1.974,"15min":2.4630000000000001},"last":1.528},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":2.2189999999999999,"15min":2.6160000000000001},"last":1.3979999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.343,"15min":1.3460000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.88300000000000001},"max":{"1min":1.7130000000000001,"5min":2.0150000000000001,"15min":2.339},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4299999999999999,"15min":1.452},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.0590000000000002,"5min":2.113,"15min":2.5680000000000001},"last":1.5760000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.488,"15min":1.4710000000000001},"min":{"1min":1.083,"5min":1.0660000000000001,"15min":0.91500000000000004},"max":{"1min":2.0019999999999998,"5min":2.2629999999999999,"15min":2.5129999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":1.04,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.802,"5min":1.931,"15min":2.4500000000000002},"last":1.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:55:03.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.502+0000 7fc9b442e700 1 Processor -- start 2023-12-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b442e700 1 -- start start 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b442e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc9ac0cc610 0x7fc9ac0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b442e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac0c6630 con 0x7fc9ac1502b0 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b442e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ac0c6910 con 0x7fc9ac067c90 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b442e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ac0c6bf0 con 0x7fc9ac0cc610 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b29cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc9ac0cc610 0x7fc9ac0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b19c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc9ac1502b0 0x7fc9ac0c5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35818/0 (socket says 172.21.15.50:35818) 2023-12-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b19c9700 1 -- 172.21.15.50:0/947817637 learned_addr learned my addr 172.21.15.50:0/947817637 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b29cb700 1 -- 172.21.15.50:0/947817637 >> v1:172.21.15.125:6789/0 conn(0x7fc9ac067c90 legacy=0x7fc9ac14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b29cb700 1 -- 172.21.15.50:0/947817637 >> v1:172.21.15.50:6789/0 conn(0x7fc9ac1502b0 legacy=0x7fc9ac0c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b29cb700 1 -- 172.21.15.50:0/947817637 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac0c6ed0 con 0x7fc9ac0cc610 2023-12-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.506+0000 7fc9b29cb700 1 --2- 172.21.15.50:0/947817637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac0cc610 0x7fc9ac0c6010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009e60 tx=0x7fc9a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e422eb575089484b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b11c8700 1 -- 172.21.15.50:0/947817637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a400e070 con 0x7fc9ac0cc610 2023-12-19T14:55:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b11c8700 1 -- 172.21.15.50:0/947817637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a400bc30 con 0x7fc9ac0cc610 2023-12-19T14:55:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b11c8700 1 -- 172.21.15.50:0/947817637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4012430 con 0x7fc9ac0cc610 2023-12-19T14:55:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 -- 172.21.15.50:0/947817637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac0cc610 msgr2=0x7fc9ac0c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 --2- 172.21.15.50:0/947817637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac0cc610 0x7fc9ac0c6010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009e60 tx=0x7fc9a4009780 comp rx=0 tx=0).stop 2023-12-19T14:55:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 -- 172.21.15.50:0/947817637 shutdown_connections 2023-12-19T14:55:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 --2- 172.21.15.50:0/947817637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac0cc610 0x7fc9ac0c6010 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 -- 172.21.15.50:0/947817637 >> 172.21.15.50:0/947817637 conn(0x7fc9ac0cd6d0 msgr2=0x7fc9ac06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 -- 172.21.15.50:0/947817637 shutdown_connections 2023-12-19T14:55:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.510+0000 7fc9b442e700 1 -- 172.21.15.50:0/947817637 wait complete. 2023-12-19T14:55:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 Processor -- start 2023-12-19T14:55:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 -- start start 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9ac0cc610 0x7fc9ac14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9ac1502b0 0x7fc9ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac169610 con 0x7fc9ac067c90 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac1698f0 con 0x7fc9ac1502b0 2023-12-19T14:55:03.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b442e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac169bd0 con 0x7fc9ac0cc610 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54186/0 (socket says 172.21.15.50:54186) 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 -- 172.21.15.50:0/3122200960 learned_addr learned my addr 172.21.15.50:0/3122200960 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b19c9700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9ac0cc610 0x7fc9ac14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b29cb700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9ac1502b0 0x7fc9ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 -- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9ac0cc610 msgr2=0x7fc9ac14ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9ac0cc610 0x7fc9ac14ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 -- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9ac1502b0 msgr2=0x7fc9ac06a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9ac1502b0 0x7fc9ac06a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 -- 172.21.15.50:0/3122200960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4009a20 con 0x7fc9ac067c90 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.514+0000 7fc9b21ca700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800c790 tx=0x7fc9a8007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8011590 con 0x7fc9ac067c90 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a8011710 con 0x7fc9ac067c90 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac169eb0 con 0x7fc9ac067c90 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8011ad0 con 0x7fc9ac067c90 2023-12-19T14:55:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac1692c0 con 0x7fc9ac067c90 2023-12-19T14:55:03.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac071120 con 0x7fc9ac067c90 2023-12-19T14:55:03.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.518+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc9a801b3b0 con 0x7fc9ac067c90 2023-12-19T14:55:03.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.522+0000 7fc9a27fc700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.522+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc9a807bff0 con 0x7fc9ac067c90 2023-12-19T14:55:03.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.522+0000 7fc9b19c9700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.522+0000 7fc9b19c9700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 0x7fc990062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac06b350 tx=0x7fc99c009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.530+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc9a804c2d0 con 0x7fc9ac067c90 2023-12-19T14:55:03.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.698+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9ac169750 con 0x7fc990060670 2023-12-19T14:55:03.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.718+0000 7fc9a27fc700 1 -- 172.21.15.50:0/3122200960 <== mgr.6089 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286841 (secure 0 0 0) 0x7fc9ac169750 con 0x7fc990060670 2023-12-19T14:55:03.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 msgr2=0x7fc990062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 0x7fc990062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac06b350 tx=0x7fc99c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 msgr2=0x7fc9ac0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800c790 tx=0x7fc9a8007f40 comp rx=0 tx=0).stop 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 shutdown_connections 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9ac1502b0 0x7fc9ac06a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9ac0cc610 0x7fc9ac14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 --2- 172.21.15.50:0/3122200960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9ac067c90 0x7fc9ac0b41f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 >> 172.21.15.50:0/3122200960 conn(0x7fc9ac0cd6d0 msgr2=0x7fc9ac06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 shutdown_connections 2023-12-19T14:55:03.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.726+0000 7fc9b442e700 1 -- 172.21.15.50:0/3122200960 wait complete. 2023-12-19T14:55:03.731 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:55:03.749 INFO:tasks.ceph.mds.d.smithi125.stderr:2023-12-19T14:55:03.742+0000 7fe94c3d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:55:03.750 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-19T14:55:03.697279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2442200,"kb_used_data":10648,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752532520,"statfs":{"total":773094113280,"available":770593300480,"internally_reserved":0,"allocated":10903552,"data_stored":6020739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001971"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817477+0000","last_change":"2023-12-19T14:54:58.817477+0000","last_active":"2023-12-19T14:54:58.817477+0000","last_peered":"2023-12-19T14:54:58.817477+0000","last_clean":"2023-12-19T14:54:58.817477+0000","last_became_active":"2023-12-19T14:54:58.816899+0000","last_became_peered":"2023-12-19T14:54:58.816899+0000","last_unstale":"2023-12-19T14:54:58.817477+0000","last_undegraded":"2023-12-19T14:54:58.817477+0000","last_fullsized":"2023-12-19T14:54:58.817477+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:55.388516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.836709+0000","last_change":"2023-12-19T14:54:58.836709+0000","last_active":"2023-12-19T14:54:58.836709+0000","last_peered":"2023-12-19T14:54:58.836709+0000","last_clean":"2023-12-19T14:54:58.836709+0000","last_became_active":"2023-12-19T14:54:58.836346+0000","last_became_peered":"2023-12-19T14:54:58.836346+0000","last_unstale":"2023-12-19T14:54:58.836709+0000","last_undegraded":"2023-12-19T14:54:58.836709+0000","last_fullsized":"2023-12-19T14:54:58.836709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:30:46.597215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832557+0000","last_change":"2023-12-19T14:54:58.832557+0000","last_active":"2023-12-19T14:54:58.832557+0000","last_peered":"2023-12-19T14:54:58.832557+0000","last_clean":"2023-12-19T14:54:58.832557+0000","last_became_active":"2023-12-19T14:54:58.832265+0000","last_became_peered":"2023-12-19T14:54:58.832265+0000","last_unstale":"2023-12-19T14:54:58.832557+0000","last_undegraded":"2023-12-19T14:54:58.832557+0000","last_fullsized":"2023-12-19T14:54:58.832557+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:00.765295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821001+0000","last_change":"2023-12-19T14:54:58.821001+0000","last_active":"2023-12-19T14:54:58.821001+0000","last_peered":"2023-12-19T14:54:58.821001+0000","last_clean":"2023-12-19T14:54:58.821001+0000","last_became_active":"2023-12-19T14:54:58.818846+0000","last_became_peered":"2023-12-19T14:54:58.818846+0000","last_unstale":"2023-12-19T14:54:58.821001+0000","last_undegraded":"2023-12-19T14:54:58.821001+0000","last_fullsized":"2023-12-19T14:54:58.821001+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:13:59.329747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818462+0000","last_change":"2023-12-19T14:54:58.818462+0000","last_active":"2023-12-19T14:54:58.818462+0000","last_peered":"2023-12-19T14:54:58.818462+0000","last_clean":"2023-12-19T14:54:58.818462+0000","last_became_active":"2023-12-19T14:54:58.818164+0000","last_became_peered":"2023-12-19T14:54:58.818164+0000","last_unstale":"2023-12-19T14:54:58.818462+0000","last_undegraded":"2023-12-19T14:54:58.818462+0000","last_fullsized":"2023-12-19T14:54:58.818462+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:30:20.258328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824850+0000","last_change":"2023-12-19T14:54:58.824850+0000","last_active":"2023-12-19T14:54:58.824850+0000","last_peered":"2023-12-19T14:54:58.824850+0000","last_clean":"2023-12-19T14:54:58.824850+0000","last_became_active":"2023-12-19T14:54:58.824231+0000","last_became_peered":"2023-12-19T14:54:58.824231+0000","last_unstale":"2023-12-19T14:54:58.824850+0000","last_undegraded":"2023-12-19T14:54:58.824850+0000","last_fullsized":"2023-12-19T14:54:58.824850+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:55.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.815416+0000","last_change":"2023-12-19T14:54:58.815416+0000","last_active":"2023-12-19T14:54:58.815416+0000","last_peered":"2023-12-19T14:54:58.815416+0000","last_clean":"2023-12-19T14:54:58.815416+0000","last_became_active":"2023-12-19T14:54:58.815120+0000","last_became_peered":"2023-12-19T14:54:58.815120+0000","last_unstale":"2023-12-19T14:54:58.815416+0000","last_undegraded":"2023-12-19T14:54:58.815416+0000","last_fullsized":"2023-12-19T14:54:58.815416+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:17.673794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821114+0000","last_change":"2023-12-19T14:54:58.821114+0000","last_active":"2023-12-19T14:54:58.821114+0000","last_peered":"2023-12-19T14:54:58.821114+0000","last_clean":"2023-12-19T14:54:58.821114+0000","last_became_active":"2023-12-19T14:54:58.820847+0000","last_became_peered":"2023-12-19T14:54:58.820847+0000","last_unstale":"2023-12-19T14:54:58.821114+0000","last_undegraded":"2023-12-19T14:54:58.821114+0000","last_fullsized":"2023-12-19T14:54:58.821114+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:15.615896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829972+0000","last_change":"2023-12-19T14:54:58.829972+0000","last_active":"2023-12-19T14:54:58.829972+0000","last_peered":"2023-12-19T14:54:58.829972+0000","last_clean":"2023-12-19T14:54:58.829972+0000","last_became_active":"2023-12-19T14:54:58.829713+0000","last_became_peered":"2023-12-19T14:54:58.829713+0000","last_unstale":"2023-12-19T14:54:58.829972+0000","last_undegraded":"2023-12-19T14:54:58.829972+0000","last_fullsized":"2023-12-19T14:54:58.829972+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:12.798653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824042+0000","last_change":"2023-12-19T14:54:58.824042+0000","last_active":"2023-12-19T14:54:58.824042+0000","last_peered":"2023-12-19T14:54:58.824042+0000","last_clean":"2023-12-19T14:54:58.824042+0000","last_became_active":"2023-12-19T14:54:58.822793+0000","last_became_peered":"2023-12-19T14:54:58.822793+0000","last_unstale":"2023-12-19T14:54:58.824042+0000","last_undegraded":"2023-12-19T14:54:58.824042+0000","last_fullsized":"2023-12-19T14:54:58.824042+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:22:13.417736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829542+0000","last_change":"2023-12-19T14:54:58.829542+0000","last_active":"2023-12-19T14:54:58.829542+0000","last_peered":"2023-12-19T14:54:58.829542+0000","last_clean":"2023-12-19T14:54:58.829542+0000","last_became_active":"2023-12-19T14:54:58.829256+0000","last_became_peered":"2023-12-19T14:54:58.829256+0000","last_unstale":"2023-12-19T14:54:58.829542+0000","last_undegraded":"2023-12-19T14:54:58.829542+0000","last_fullsized":"2023-12-19T14:54:58.829542+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:19.557943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.828661+0000","last_change":"2023-12-19T14:54:58.828661+0000","last_active":"2023-12-19T14:54:58.828661+0000","last_peered":"2023-12-19T14:54:58.828661+0000","last_clean":"2023-12-19T14:54:58.828661+0000","last_became_active":"2023-12-19T14:54:58.828392+0000","last_became_peered":"2023-12-19T14:54:58.828392+0000","last_unstale":"2023-12-19T14:54:58.828661+0000","last_undegraded":"2023-12-19T14:54:58.828661+0000","last_fullsized":"2023-12-19T14:54:58.828661+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:12.089349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823105+0000","last_change":"2023-12-19T14:54:58.823105+0000","last_active":"2023-12-19T14:54:58.823105+0000","last_peered":"2023-12-19T14:54:58.823105+0000","last_clean":"2023-12-19T14:54:58.823105+0000","last_became_active":"2023-12-19T14:54:58.822841+0000","last_became_peered":"2023-12-19T14:54:58.822841+0000","last_unstale":"2023-12-19T14:54:58.823105+0000","last_undegraded":"2023-12-19T14:54:58.823105+0000","last_fullsized":"2023-12-19T14:54:58.823105+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:39:10.741592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824858+0000","last_change":"2023-12-19T14:54:58.824858+0000","last_active":"2023-12-19T14:54:58.824858+0000","last_peered":"2023-12-19T14:54:58.824858+0000","last_clean":"2023-12-19T14:54:58.824858+0000","last_became_active":"2023-12-19T14:54:58.824596+0000","last_became_peered":"2023-12-19T14:54:58.824596+0000","last_unstale":"2023-12-19T14:54:58.824858+0000","last_undegraded":"2023-12-19T14:54:58.824858+0000","last_fullsized":"2023-12-19T14:54:58.824858+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:02.273426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832086+0000","last_change":"2023-12-19T14:54:58.832086+0000","last_active":"2023-12-19T14:54:58.832086+0000","last_peered":"2023-12-19T14:54:58.832086+0000","last_clean":"2023-12-19T14:54:58.832086+0000","last_became_active":"2023-12-19T14:54:58.831632+0000","last_became_peered":"2023-12-19T14:54:58.831632+0000","last_unstale":"2023-12-19T14:54:58.832086+0000","last_undegraded":"2023-12-19T14:54:58.832086+0000","last_fullsized":"2023-12-19T14:54:58.832086+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:32.377041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816649+0000","last_change":"2023-12-19T14:54:58.816649+0000","last_active":"2023-12-19T14:54:58.816649+0000","last_peered":"2023-12-19T14:54:58.816649+0000","last_clean":"2023-12-19T14:54:58.816649+0000","last_became_active":"2023-12-19T14:54:58.816245+0000","last_became_peered":"2023-12-19T14:54:58.816245+0000","last_unstale":"2023-12-19T14:54:58.816649+0000","last_undegraded":"2023-12-19T14:54:58.816649+0000","last_fullsized":"2023-12-19T14:54:58.816649+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:41.720028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824753+0000","last_change":"2023-12-19T14:54:58.824753+0000","last_active":"2023-12-19T14:54:58.824753+0000","last_peered":"2023-12-19T14:54:58.824753+0000","last_clean":"2023-12-19T14:54:58.824753+0000","last_became_active":"2023-12-19T14:54:58.820394+0000","last_became_peered":"2023-12-19T14:54:58.820394+0000","last_unstale":"2023-12-19T14:54:58.824753+0000","last_undegraded":"2023-12-19T14:54:58.824753+0000","last_fullsized":"2023-12-19T14:54:58.824753+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:24.574228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.822573+0000","last_change":"2023-12-19T14:54:58.822573+0000","last_active":"2023-12-19T14:54:58.822573+0000","last_peered":"2023-12-19T14:54:58.822573+0000","last_clean":"2023-12-19T14:54:58.822573+0000","last_became_active":"2023-12-19T14:54:58.822264+0000","last_became_peered":"2023-12-19T14:54:58.822264+0000","last_unstale":"2023-12-19T14:54:58.822573+0000","last_undegraded":"2023-12-19T14:54:58.822573+0000","last_fullsized":"2023-12-19T14:54:58.822573+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:07:18.713895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821174+0000","last_change":"2023-12-19T14:54:58.821174+0000","last_active":"2023-12-19T14:54:58.821174+0000","last_peered":"2023-12-19T14:54:58.821174+0000","last_clean":"2023-12-19T14:54:58.821174+0000","last_became_active":"2023-12-19T14:54:58.820872+0000","last_became_peered":"2023-12-19T14:54:58.820872+0000","last_unstale":"2023-12-19T14:54:58.821174+0000","last_undegraded":"2023-12-19T14:54:58.821174+0000","last_fullsized":"2023-12-19T14:54:58.821174+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:16:19.321573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829384+0000","last_change":"2023-12-19T14:54:58.829384+0000","last_active":"2023-12-19T14:54:58.829384+0000","last_peered":"2023-12-19T14:54:58.829384+0000","last_clean":"2023-12-19T14:54:58.829384+0000","last_became_active":"2023-12-19T14:54:58.828407+0000","last_became_peered":"2023-12-19T14:54:58.828407+0000","last_unstale":"2023-12-19T14:54:58.829384+0000","last_undegraded":"2023-12-19T14:54:58.829384+0000","last_fullsized":"2023-12-19T14:54:58.829384+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:23.055445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821376+0000","last_change":"2023-12-19T14:54:58.821376+0000","last_active":"2023-12-19T14:54:58.821376+0000","last_peered":"2023-12-19T14:54:58.821376+0000","last_clean":"2023-12-19T14:54:58.821376+0000","last_became_active":"2023-12-19T14:54:58.820014+0000","last_became_peered":"2023-12-19T14:54:58.820014+0000","last_unstale":"2023-12-19T14:54:58.821376+0000","last_undegraded":"2023-12-19T14:54:58.821376+0000","last_fullsized":"2023-12-19T14:54:58.821376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:02.081737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.826474+0000","last_change":"2023-12-19T14:54:58.826474+0000","last_active":"2023-12-19T14:54:58.826474+0000","last_peered":"2023-12-19T14:54:58.826474+0000","last_clean":"2023-12-19T14:54:58.826474+0000","last_became_active":"2023-12-19T14:54:58.826102+0000","last_became_peered":"2023-12-19T14:54:58.826102+0000","last_unstale":"2023-12-19T14:54:58.826474+0000","last_undegraded":"2023-12-19T14:54:58.826474+0000","last_fullsized":"2023-12-19T14:54:58.826474+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:12.801733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.819211+0000","last_change":"2023-12-19T14:54:58.819211+0000","last_active":"2023-12-19T14:54:58.819211+0000","last_peered":"2023-12-19T14:54:58.819211+0000","last_clean":"2023-12-19T14:54:58.819211+0000","last_became_active":"2023-12-19T14:54:58.816538+0000","last_became_peered":"2023-12-19T14:54:58.816538+0000","last_unstale":"2023-12-19T14:54:58.819211+0000","last_undegraded":"2023-12-19T14:54:58.819211+0000","last_fullsized":"2023-12-19T14:54:58.819211+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:24.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.835365+0000","last_change":"2023-12-19T14:54:58.835365+0000","last_active":"2023-12-19T14:54:58.835365+0000","last_peered":"2023-12-19T14:54:58.835365+0000","last_clean":"2023-12-19T14:54:58.835365+0000","last_became_active":"2023-12-19T14:54:58.835083+0000","last_became_peered":"2023-12-19T14:54:58.835083+0000","last_unstale":"2023-12-19T14:54:58.835365+0000","last_undegraded":"2023-12-19T14:54:58.835365+0000","last_fullsized":"2023-12-19T14:54:58.835365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:20:55.305099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.835650+0000","last_change":"2023-12-19T14:54:58.835650+0000","last_active":"2023-12-19T14:54:58.835650+0000","last_peered":"2023-12-19T14:54:58.835650+0000","last_clean":"2023-12-19T14:54:58.835650+0000","last_became_active":"2023-12-19T14:54:58.835373+0000","last_became_peered":"2023-12-19T14:54:58.835373+0000","last_unstale":"2023-12-19T14:54:58.835650+0000","last_undegraded":"2023-12-19T14:54:58.835650+0000","last_fullsized":"2023-12-19T14:54:58.835650+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:56.339188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.826662+0000","last_change":"2023-12-19T14:54:58.826662+0000","last_active":"2023-12-19T14:54:58.826662+0000","last_peered":"2023-12-19T14:54:58.826662+0000","last_clean":"2023-12-19T14:54:58.826662+0000","last_became_active":"2023-12-19T14:54:58.825722+0000","last_became_peered":"2023-12-19T14:54:58.825722+0000","last_unstale":"2023-12-19T14:54:58.826662+0000","last_undegraded":"2023-12-19T14:54:58.826662+0000","last_fullsized":"2023-12-19T14:54:58.826662+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:18.203830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818524+0000","last_change":"2023-12-19T14:54:58.818524+0000","last_active":"2023-12-19T14:54:58.818524+0000","last_peered":"2023-12-19T14:54:58.818524+0000","last_clean":"2023-12-19T14:54:58.818524+0000","last_became_active":"2023-12-19T14:54:58.817843+0000","last_became_peered":"2023-12-19T14:54:58.817843+0000","last_unstale":"2023-12-19T14:54:58.818524+0000","last_undegraded":"2023-12-19T14:54:58.818524+0000","last_fullsized":"2023-12-19T14:54:58.818524+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:02.273426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816936+0000","last_change":"2023-12-19T14:54:58.816936+0000","last_active":"2023-12-19T14:54:58.816936+0000","last_peered":"2023-12-19T14:54:58.816936+0000","last_clean":"2023-12-19T14:54:58.816936+0000","last_became_active":"2023-12-19T14:54:58.815613+0000","last_became_peered":"2023-12-19T14:54:58.815613+0000","last_unstale":"2023-12-19T14:54:58.816936+0000","last_undegraded":"2023-12-19T14:54:58.816936+0000","last_fullsized":"2023-12-19T14:54:58.816936+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:22:41.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.833987+0000","last_change":"2023-12-19T14:54:58.833987+0000","last_active":"2023-12-19T14:54:58.833987+0000","last_peered":"2023-12-19T14:54:58.833987+0000","last_clean":"2023-12-19T14:54:58.833987+0000","last_became_active":"2023-12-19T14:54:58.828696+0000","last_became_peered":"2023-12-19T14:54:58.828696+0000","last_unstale":"2023-12-19T14:54:58.833987+0000","last_undegraded":"2023-12-19T14:54:58.833987+0000","last_fullsized":"2023-12-19T14:54:58.833987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:28.176162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.825408+0000","last_change":"2023-12-19T14:54:58.825408+0000","last_active":"2023-12-19T14:54:58.825408+0000","last_peered":"2023-12-19T14:54:58.825408+0000","last_clean":"2023-12-19T14:54:58.825408+0000","last_became_active":"2023-12-19T14:54:58.825135+0000","last_became_peered":"2023-12-19T14:54:58.825135+0000","last_unstale":"2023-12-19T14:54:58.825408+0000","last_undegraded":"2023-12-19T14:54:58.825408+0000","last_fullsized":"2023-12-19T14:54:58.825408+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:23:58.910293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827681+0000","last_change":"2023-12-19T14:54:58.827681+0000","last_active":"2023-12-19T14:54:58.827681+0000","last_peered":"2023-12-19T14:54:58.827681+0000","last_clean":"2023-12-19T14:54:58.827681+0000","last_became_active":"2023-12-19T14:54:58.827375+0000","last_became_peered":"2023-12-19T14:54:58.827375+0000","last_unstale":"2023-12-19T14:54:58.827681+0000","last_undegraded":"2023-12-19T14:54:58.827681+0000","last_fullsized":"2023-12-19T14:54:58.827681+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:24.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817217+0000","last_change":"2023-12-19T14:54:58.817217+0000","last_active":"2023-12-19T14:54:58.817217+0000","last_peered":"2023-12-19T14:54:58.817217+0000","last_clean":"2023-12-19T14:54:58.817217+0000","last_became_active":"2023-12-19T14:54:58.816912+0000","last_became_peered":"2023-12-19T14:54:58.816912+0000","last_unstale":"2023-12-19T14:54:58.817217+0000","last_undegraded":"2023-12-19T14:54:58.817217+0000","last_fullsized":"2023-12-19T14:54:58.817217+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:12:20.811769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.820639+0000","last_change":"2023-12-19T14:54:58.820639+0000","last_active":"2023-12-19T14:54:58.820639+0000","last_peered":"2023-12-19T14:54:58.820639+0000","last_clean":"2023-12-19T14:54:58.820639+0000","last_became_active":"2023-12-19T14:54:58.820354+0000","last_became_peered":"2023-12-19T14:54:58.820354+0000","last_unstale":"2023-12-19T14:54:58.820639+0000","last_undegraded":"2023-12-19T14:54:58.820639+0000","last_fullsized":"2023-12-19T14:54:58.820639+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:12.798653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"99'4","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.972425+0000","last_change":"2023-12-19T14:54:57.820415+0000","last_active":"2023-12-19T14:54:58.972425+0000","last_peered":"2023-12-19T14:54:58.972425+0000","last_clean":"2023-12-19T14:54:58.972425+0000","last_became_active":"2023-12-19T14:54:57.820140+0000","last_became_peered":"2023-12-19T14:54:57.820140+0000","last_unstale":"2023-12-19T14:54:58.972425+0000","last_undegraded":"2023-12-19T14:54:58.972425+0000","last_fullsized":"2023-12-19T14:54:58.972425+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:36:13.451941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823633+0000","last_change":"2023-12-19T14:54:58.823633+0000","last_active":"2023-12-19T14:54:58.823633+0000","last_peered":"2023-12-19T14:54:58.823633+0000","last_clean":"2023-12-19T14:54:58.823633+0000","last_became_active":"2023-12-19T14:54:58.823347+0000","last_became_peered":"2023-12-19T14:54:58.823347+0000","last_unstale":"2023-12-19T14:54:58.823633+0000","last_undegraded":"2023-12-19T14:54:58.823633+0000","last_fullsized":"2023-12-19T14:54:58.823633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:51:28.154347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.809405+0000","last_change":"2023-12-19T14:54:57.816866+0000","last_active":"2023-12-19T14:54:58.809405+0000","last_peered":"2023-12-19T14:54:58.809405+0000","last_clean":"2023-12-19T14:54:58.809405+0000","last_became_active":"2023-12-19T14:54:57.816597+0000","last_became_peered":"2023-12-19T14:54:57.816597+0000","last_unstale":"2023-12-19T14:54:58.809405+0000","last_undegraded":"2023-12-19T14:54:58.809405+0000","last_fullsized":"2023-12-19T14:54:58.809405+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:11.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831120+0000","last_change":"2023-12-19T14:54:58.831120+0000","last_active":"2023-12-19T14:54:58.831120+0000","last_peered":"2023-12-19T14:54:58.831120+0000","last_clean":"2023-12-19T14:54:58.831120+0000","last_became_active":"2023-12-19T14:54:58.830775+0000","last_became_peered":"2023-12-19T14:54:58.830775+0000","last_unstale":"2023-12-19T14:54:58.831120+0000","last_undegraded":"2023-12-19T14:54:58.831120+0000","last_fullsized":"2023-12-19T14:54:58.831120+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:02.560851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.966589+0000","last_change":"2023-12-19T14:54:57.823340+0000","last_active":"2023-12-19T14:54:58.966589+0000","last_peered":"2023-12-19T14:54:58.966589+0000","last_clean":"2023-12-19T14:54:58.966589+0000","last_became_active":"2023-12-19T14:54:57.823136+0000","last_became_peered":"2023-12-19T14:54:57.823136+0000","last_unstale":"2023-12-19T14:54:58.966589+0000","last_undegraded":"2023-12-19T14:54:58.966589+0000","last_fullsized":"2023-12-19T14:54:58.966589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:07:21.061111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821565+0000","last_change":"2023-12-19T14:54:58.821565+0000","last_active":"2023-12-19T14:54:58.821565+0000","last_peered":"2023-12-19T14:54:58.821565+0000","last_clean":"2023-12-19T14:54:58.821565+0000","last_became_active":"2023-12-19T14:54:58.818354+0000","last_became_peered":"2023-12-19T14:54:58.818354+0000","last_unstale":"2023-12-19T14:54:58.821565+0000","last_undegraded":"2023-12-19T14:54:58.821565+0000","last_fullsized":"2023-12-19T14:54:58.821565+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:44:20.402010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.965244+0000","last_change":"2023-12-19T14:54:57.814698+0000","last_active":"2023-12-19T14:54:58.965244+0000","last_peered":"2023-12-19T14:54:58.965244+0000","last_clean":"2023-12-19T14:54:58.965244+0000","last_became_active":"2023-12-19T14:54:57.814219+0000","last_became_peered":"2023-12-19T14:54:57.814219+0000","last_unstale":"2023-12-19T14:54:58.965244+0000","last_undegraded":"2023-12-19T14:54:58.965244+0000","last_fullsized":"2023-12-19T14:54:58.965244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:23.558167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.832183+0000","last_change":"2023-12-19T14:54:58.832183+0000","last_active":"2023-12-19T14:54:58.832183+0000","last_peered":"2023-12-19T14:54:58.832183+0000","last_clean":"2023-12-19T14:54:58.832183+0000","last_became_active":"2023-12-19T14:54:58.831884+0000","last_became_peered":"2023-12-19T14:54:58.831884+0000","last_unstale":"2023-12-19T14:54:58.832183+0000","last_undegraded":"2023-12-19T14:54:58.832183+0000","last_fullsized":"2023-12-19T14:54:58.832183+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:09:52.478598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.806834+0000","last_change":"2023-12-19T14:54:57.814122+0000","last_active":"2023-12-19T14:54:58.806834+0000","last_peered":"2023-12-19T14:54:58.806834+0000","last_clean":"2023-12-19T14:54:58.806834+0000","last_became_active":"2023-12-19T14:54:57.813795+0000","last_became_peered":"2023-12-19T14:54:57.813795+0000","last_unstale":"2023-12-19T14:54:58.806834+0000","last_undegraded":"2023-12-19T14:54:58.806834+0000","last_fullsized":"2023-12-19T14:54:58.806834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:36.745989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823111+0000","last_change":"2023-12-19T14:54:58.823111+0000","last_active":"2023-12-19T14:54:58.823111+0000","last_peered":"2023-12-19T14:54:58.823111+0000","last_clean":"2023-12-19T14:54:58.823111+0000","last_became_active":"2023-12-19T14:54:58.822801+0000","last_became_peered":"2023-12-19T14:54:58.822801+0000","last_unstale":"2023-12-19T14:54:58.823111+0000","last_undegraded":"2023-12-19T14:54:58.823111+0000","last_fullsized":"2023-12-19T14:54:58.823111+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:58.730300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.806030+0000","last_change":"2023-12-19T14:54:57.811480+0000","last_active":"2023-12-19T14:54:58.806030+0000","last_peered":"2023-12-19T14:54:58.806030+0000","last_clean":"2023-12-19T14:54:58.806030+0000","last_became_active":"2023-12-19T14:54:57.811196+0000","last_became_peered":"2023-12-19T14:54:57.811196+0000","last_unstale":"2023-12-19T14:54:58.806030+0000","last_undegraded":"2023-12-19T14:54:58.806030+0000","last_fullsized":"2023-12-19T14:54:58.806030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:35.098224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829937+0000","last_change":"2023-12-19T14:54:58.829937+0000","last_active":"2023-12-19T14:54:58.829937+0000","last_peered":"2023-12-19T14:54:58.829937+0000","last_clean":"2023-12-19T14:54:58.829937+0000","last_became_active":"2023-12-19T14:54:58.829616+0000","last_became_peered":"2023-12-19T14:54:58.829616+0000","last_unstale":"2023-12-19T14:54:58.829937+0000","last_undegraded":"2023-12-19T14:54:58.829937+0000","last_fullsized":"2023-12-19T14:54:58.829937+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:35.282225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826888+0000","last_change":"2023-12-19T14:54:57.819984+0000","last_active":"2023-12-19T14:54:57.826888+0000","last_peered":"2023-12-19T14:54:57.826888+0000","last_clean":"2023-12-19T14:54:57.826888+0000","last_became_active":"2023-12-19T14:54:57.819646+0000","last_became_peered":"2023-12-19T14:54:57.819646+0000","last_unstale":"2023-12-19T14:54:57.826888+0000","last_undegraded":"2023-12-19T14:54:57.826888+0000","last_fullsized":"2023-12-19T14:54:57.826888+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.814382+0000","last_change":"2023-12-19T14:54:58.814382+0000","last_active":"2023-12-19T14:54:58.814382+0000","last_peered":"2023-12-19T14:54:58.814382+0000","last_clean":"2023-12-19T14:54:58.814382+0000","last_became_active":"2023-12-19T14:54:58.813832+0000","last_became_peered":"2023-12-19T14:54:58.813832+0000","last_unstale":"2023-12-19T14:54:58.814382+0000","last_undegraded":"2023-12-19T14:54:58.814382+0000","last_fullsized":"2023-12-19T14:54:58.814382+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:37.762050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.824188+0000","last_change":"2023-12-19T14:54:57.819225+0000","last_active":"2023-12-19T14:54:57.824188+0000","last_peered":"2023-12-19T14:54:57.824188+0000","last_clean":"2023-12-19T14:54:57.824188+0000","last_became_active":"2023-12-19T14:54:57.818385+0000","last_became_peered":"2023-12-19T14:54:57.818385+0000","last_unstale":"2023-12-19T14:54:57.824188+0000","last_undegraded":"2023-12-19T14:54:57.824188+0000","last_fullsized":"2023-12-19T14:54:57.824188+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:52.422200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824006+0000","last_change":"2023-12-19T14:54:58.824006+0000","last_active":"2023-12-19T14:54:58.824006+0000","last_peered":"2023-12-19T14:54:58.824006+0000","last_clean":"2023-12-19T14:54:58.824006+0000","last_became_active":"2023-12-19T14:54:58.821857+0000","last_became_peered":"2023-12-19T14:54:58.821857+0000","last_unstale":"2023-12-19T14:54:58.824006+0000","last_undegraded":"2023-12-19T14:54:58.824006+0000","last_fullsized":"2023-12-19T14:54:58.824006+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:36.114285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822453+0000","last_change":"2023-12-19T14:54:57.819030+0000","last_active":"2023-12-19T14:54:57.822453+0000","last_peered":"2023-12-19T14:54:57.822453+0000","last_clean":"2023-12-19T14:54:57.822453+0000","last_became_active":"2023-12-19T14:54:57.818385+0000","last_became_peered":"2023-12-19T14:54:57.818385+0000","last_unstale":"2023-12-19T14:54:57.822453+0000","last_undegraded":"2023-12-19T14:54:57.822453+0000","last_fullsized":"2023-12-19T14:54:57.822453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:07:17.697834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.822439+0000","last_change":"2023-12-19T14:54:58.822439+0000","last_active":"2023-12-19T14:54:58.822439+0000","last_peered":"2023-12-19T14:54:58.822439+0000","last_clean":"2023-12-19T14:54:58.822439+0000","last_became_active":"2023-12-19T14:54:58.822104+0000","last_became_peered":"2023-12-19T14:54:58.822104+0000","last_unstale":"2023-12-19T14:54:58.822439+0000","last_undegraded":"2023-12-19T14:54:58.822439+0000","last_fullsized":"2023-12-19T14:54:58.822439+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:24:34.581860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.816567+0000","last_change":"2023-12-19T14:54:57.810499+0000","last_active":"2023-12-19T14:54:57.816567+0000","last_peered":"2023-12-19T14:54:57.816567+0000","last_clean":"2023-12-19T14:54:57.816567+0000","last_became_active":"2023-12-19T14:54:57.810181+0000","last_became_peered":"2023-12-19T14:54:57.810181+0000","last_unstale":"2023-12-19T14:54:57.816567+0000","last_undegraded":"2023-12-19T14:54:57.816567+0000","last_fullsized":"2023-12-19T14:54:57.816567+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:25.568467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824651+0000","last_change":"2023-12-19T14:54:58.824651+0000","last_active":"2023-12-19T14:54:58.824651+0000","last_peered":"2023-12-19T14:54:58.824651+0000","last_clean":"2023-12-19T14:54:58.824651+0000","last_became_active":"2023-12-19T14:54:58.823909+0000","last_became_peered":"2023-12-19T14:54:58.823909+0000","last_unstale":"2023-12-19T14:54:58.824651+0000","last_undegraded":"2023-12-19T14:54:58.824651+0000","last_fullsized":"2023-12-19T14:54:58.824651+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:41.720028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"99'3","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.971535+0000","last_change":"2023-12-19T14:54:57.825102+0000","last_active":"2023-12-19T14:54:58.971535+0000","last_peered":"2023-12-19T14:54:58.971535+0000","last_clean":"2023-12-19T14:54:58.971535+0000","last_became_active":"2023-12-19T14:54:57.824789+0000","last_became_peered":"2023-12-19T14:54:57.824789+0000","last_unstale":"2023-12-19T14:54:58.971535+0000","last_undegraded":"2023-12-19T14:54:58.971535+0000","last_fullsized":"2023-12-19T14:54:58.971535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:29:46.190041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.829135+0000","last_change":"2023-12-19T14:54:58.829135+0000","last_active":"2023-12-19T14:54:58.829135+0000","last_peered":"2023-12-19T14:54:58.829135+0000","last_clean":"2023-12-19T14:54:58.829135+0000","last_became_active":"2023-12-19T14:54:58.828801+0000","last_became_peered":"2023-12-19T14:54:58.828801+0000","last_unstale":"2023-12-19T14:54:58.829135+0000","last_undegraded":"2023-12-19T14:54:58.829135+0000","last_fullsized":"2023-12-19T14:54:58.829135+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:35.282225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"99'6","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.967106+0000","last_change":"2023-12-19T14:54:57.813559+0000","last_active":"2023-12-19T14:54:58.967106+0000","last_peered":"2023-12-19T14:54:58.967106+0000","last_clean":"2023-12-19T14:54:58.967106+0000","last_became_active":"2023-12-19T14:54:57.812376+0000","last_became_peered":"2023-12-19T14:54:57.812376+0000","last_unstale":"2023-12-19T14:54:58.967106+0000","last_undegraded":"2023-12-19T14:54:58.967106+0000","last_fullsized":"2023-12-19T14:54:58.967106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:35.795935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.819662+0000","last_change":"2023-12-19T14:54:58.819662+0000","last_active":"2023-12-19T14:54:58.819662+0000","last_peered":"2023-12-19T14:54:58.819662+0000","last_clean":"2023-12-19T14:54:58.819662+0000","last_became_active":"2023-12-19T14:54:58.819379+0000","last_became_peered":"2023-12-19T14:54:58.819379+0000","last_unstale":"2023-12-19T14:54:58.819662+0000","last_undegraded":"2023-12-19T14:54:58.819662+0000","last_fullsized":"2023-12-19T14:54:58.819662+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:56.339188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.964976+0000","last_change":"2023-12-19T14:54:57.812060+0000","last_active":"2023-12-19T14:54:58.964976+0000","last_peered":"2023-12-19T14:54:58.964976+0000","last_clean":"2023-12-19T14:54:58.964976+0000","last_became_active":"2023-12-19T14:54:57.810171+0000","last_became_peered":"2023-12-19T14:54:57.810171+0000","last_unstale":"2023-12-19T14:54:58.964976+0000","last_undegraded":"2023-12-19T14:54:58.964976+0000","last_fullsized":"2023-12-19T14:54:58.964976+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:51:27.138286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823842+0000","last_change":"2023-12-19T14:54:58.823842+0000","last_active":"2023-12-19T14:54:58.823842+0000","last_peered":"2023-12-19T14:54:58.823842+0000","last_clean":"2023-12-19T14:54:58.823842+0000","last_became_active":"2023-12-19T14:54:58.823579+0000","last_became_peered":"2023-12-19T14:54:58.823579+0000","last_unstale":"2023-12-19T14:54:58.823842+0000","last_undegraded":"2023-12-19T14:54:58.823842+0000","last_fullsized":"2023-12-19T14:54:58.823842+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:41:34.846678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822501+0000","last_change":"2023-12-19T14:54:57.820105+0000","last_active":"2023-12-19T14:54:57.822501+0000","last_peered":"2023-12-19T14:54:57.822501+0000","last_clean":"2023-12-19T14:54:57.822501+0000","last_became_active":"2023-12-19T14:54:57.819771+0000","last_became_peered":"2023-12-19T14:54:57.819771+0000","last_unstale":"2023-12-19T14:54:57.822501+0000","last_undegraded":"2023-12-19T14:54:57.822501+0000","last_fullsized":"2023-12-19T14:54:57.822501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:16.657733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.817282+0000","last_change":"2023-12-19T14:54:58.817282+0000","last_active":"2023-12-19T14:54:58.817282+0000","last_peered":"2023-12-19T14:54:58.817282+0000","last_clean":"2023-12-19T14:54:58.817282+0000","last_became_active":"2023-12-19T14:54:58.816947+0000","last_became_peered":"2023-12-19T14:54:58.816947+0000","last_unstale":"2023-12-19T14:54:58.817282+0000","last_undegraded":"2023-12-19T14:54:58.817282+0000","last_fullsized":"2023-12-19T14:54:58.817282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:52:24.807647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.823640+0000","last_change":"2023-12-19T14:54:57.819773+0000","last_active":"2023-12-19T14:54:57.823640+0000","last_peered":"2023-12-19T14:54:57.823640+0000","last_clean":"2023-12-19T14:54:57.823640+0000","last_became_active":"2023-12-19T14:54:57.819476+0000","last_became_peered":"2023-12-19T14:54:57.819476+0000","last_unstale":"2023-12-19T14:54:57.823640+0000","last_undegraded":"2023-12-19T14:54:57.823640+0000","last_fullsized":"2023-12-19T14:54:57.823640+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:37.105873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.818724+0000","last_change":"2023-12-19T14:54:58.818724+0000","last_active":"2023-12-19T14:54:58.818724+0000","last_peered":"2023-12-19T14:54:58.818724+0000","last_clean":"2023-12-19T14:54:58.818724+0000","last_became_active":"2023-12-19T14:54:58.818445+0000","last_became_peered":"2023-12-19T14:54:58.818445+0000","last_unstale":"2023-12-19T14:54:58.818724+0000","last_undegraded":"2023-12-19T14:54:58.818724+0000","last_fullsized":"2023-12-19T14:54:58.818724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:29.920508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"99'8","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.971641+0000","last_change":"2023-12-19T14:54:57.822276+0000","last_active":"2023-12-19T14:54:58.971641+0000","last_peered":"2023-12-19T14:54:58.971641+0000","last_clean":"2023-12-19T14:54:58.971641+0000","last_became_active":"2023-12-19T14:54:57.821973+0000","last_became_peered":"2023-12-19T14:54:57.821973+0000","last_unstale":"2023-12-19T14:54:58.971641+0000","last_undegraded":"2023-12-19T14:54:58.971641+0000","last_fullsized":"2023-12-19T14:54:58.971641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:01.544789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.814453+0000","last_change":"2023-12-19T14:54:58.814453+0000","last_active":"2023-12-19T14:54:58.814453+0000","last_peered":"2023-12-19T14:54:58.814453+0000","last_clean":"2023-12-19T14:54:58.814453+0000","last_became_active":"2023-12-19T14:54:58.814154+0000","last_became_peered":"2023-12-19T14:54:58.814154+0000","last_unstale":"2023-12-19T14:54:58.814453+0000","last_undegraded":"2023-12-19T14:54:58.814453+0000","last_fullsized":"2023-12-19T14:54:58.814453+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:55.388516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.807749+0000","last_change":"2023-12-19T14:54:57.819221+0000","last_active":"2023-12-19T14:54:58.807749+0000","last_peered":"2023-12-19T14:54:58.807749+0000","last_clean":"2023-12-19T14:54:58.807749+0000","last_became_active":"2023-12-19T14:54:57.818757+0000","last_became_peered":"2023-12-19T14:54:57.818757+0000","last_unstale":"2023-12-19T14:54:58.807749+0000","last_undegraded":"2023-12-19T14:54:58.807749+0000","last_fullsized":"2023-12-19T14:54:58.807749+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:41:33.830617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823072+0000","last_change":"2023-12-19T14:54:58.823072+0000","last_active":"2023-12-19T14:54:58.823072+0000","last_peered":"2023-12-19T14:54:58.823072+0000","last_clean":"2023-12-19T14:54:58.823072+0000","last_became_active":"2023-12-19T14:54:58.820347+0000","last_became_peered":"2023-12-19T14:54:58.820347+0000","last_unstale":"2023-12-19T14:54:58.823072+0000","last_undegraded":"2023-12-19T14:54:58.823072+0000","last_fullsized":"2023-12-19T14:54:58.823072+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:31:36.811997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.965832+0000","last_change":"2023-12-19T14:54:57.815395+0000","last_active":"2023-12-19T14:54:58.965832+0000","last_peered":"2023-12-19T14:54:58.965832+0000","last_clean":"2023-12-19T14:54:58.965832+0000","last_became_active":"2023-12-19T14:54:57.815066+0000","last_became_peered":"2023-12-19T14:54:57.815066+0000","last_unstale":"2023-12-19T14:54:58.965832+0000","last_undegraded":"2023-12-19T14:54:58.965832+0000","last_fullsized":"2023-12-19T14:54:58.965832+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:48.798737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821764+0000","last_change":"2023-12-19T14:54:58.821764+0000","last_active":"2023-12-19T14:54:58.821764+0000","last_peered":"2023-12-19T14:54:58.821764+0000","last_clean":"2023-12-19T14:54:58.821764+0000","last_became_active":"2023-12-19T14:54:58.821497+0000","last_became_peered":"2023-12-19T14:54:58.821497+0000","last_unstale":"2023-12-19T14:54:58.821764+0000","last_undegraded":"2023-12-19T14:54:58.821764+0000","last_fullsized":"2023-12-19T14:54:58.821764+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:21.971472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"99'1","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.802169+0000","last_change":"2023-12-19T14:54:57.821220+0000","last_active":"2023-12-19T14:54:58.802169+0000","last_peered":"2023-12-19T14:54:58.802169+0000","last_clean":"2023-12-19T14:54:58.802169+0000","last_became_active":"2023-12-19T14:54:57.818528+0000","last_became_peered":"2023-12-19T14:54:57.818528+0000","last_unstale":"2023-12-19T14:54:58.802169+0000","last_undegraded":"2023-12-19T14:54:58.802169+0000","last_fullsized":"2023-12-19T14:54:58.802169+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:08.262253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.816157+0000","last_change":"2023-12-19T14:54:58.816157+0000","last_active":"2023-12-19T14:54:58.816157+0000","last_peered":"2023-12-19T14:54:58.816157+0000","last_clean":"2023-12-19T14:54:58.816157+0000","last_became_active":"2023-12-19T14:54:58.815884+0000","last_became_peered":"2023-12-19T14:54:58.815884+0000","last_unstale":"2023-12-19T14:54:58.816157+0000","last_undegraded":"2023-12-19T14:54:58.816157+0000","last_fullsized":"2023-12-19T14:54:58.816157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:30:46.597215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.827514+0000","last_change":"2023-12-19T14:54:57.823532+0000","last_active":"2023-12-19T14:54:57.827514+0000","last_peered":"2023-12-19T14:54:57.827514+0000","last_clean":"2023-12-19T14:54:57.827514+0000","last_became_active":"2023-12-19T14:54:57.823250+0000","last_became_peered":"2023-12-19T14:54:57.823250+0000","last_unstale":"2023-12-19T14:54:57.827514+0000","last_undegraded":"2023-12-19T14:54:57.827514+0000","last_fullsized":"2023-12-19T14:54:57.827514+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:52:23.791586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831427+0000","last_change":"2023-12-19T14:54:58.831427+0000","last_active":"2023-12-19T14:54:58.831427+0000","last_peered":"2023-12-19T14:54:58.831427+0000","last_clean":"2023-12-19T14:54:58.831427+0000","last_became_active":"2023-12-19T14:54:58.831093+0000","last_became_peered":"2023-12-19T14:54:58.831093+0000","last_unstale":"2023-12-19T14:54:58.831427+0000","last_undegraded":"2023-12-19T14:54:58.831427+0000","last_fullsized":"2023-12-19T14:54:58.831427+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:18.629358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.970331+0000","last_change":"2023-12-19T14:54:57.824569+0000","last_active":"2023-12-19T14:54:58.970331+0000","last_peered":"2023-12-19T14:54:58.970331+0000","last_clean":"2023-12-19T14:54:58.970331+0000","last_became_active":"2023-12-19T14:54:57.824278+0000","last_became_peered":"2023-12-19T14:54:57.824278+0000","last_unstale":"2023-12-19T14:54:58.970331+0000","last_undegraded":"2023-12-19T14:54:58.970331+0000","last_fullsized":"2023-12-19T14:54:58.970331+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.831017+0000","last_change":"2023-12-19T14:54:58.831017+0000","last_active":"2023-12-19T14:54:58.831017+0000","last_peered":"2023-12-19T14:54:58.831017+0000","last_clean":"2023-12-19T14:54:58.831017+0000","last_became_active":"2023-12-19T14:54:58.830711+0000","last_became_peered":"2023-12-19T14:54:58.830711+0000","last_unstale":"2023-12-19T14:54:58.831017+0000","last_undegraded":"2023-12-19T14:54:58.831017+0000","last_fullsized":"2023-12-19T14:54:58.831017+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:26.584529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.828208+0000","last_change":"2023-12-19T14:54:57.819079+0000","last_active":"2023-12-19T14:54:57.828208+0000","last_peered":"2023-12-19T14:54:57.828208+0000","last_clean":"2023-12-19T14:54:57.828208+0000","last_became_active":"2023-12-19T14:54:57.818645+0000","last_became_peered":"2023-12-19T14:54:57.818645+0000","last_unstale":"2023-12-19T14:54:57.828208+0000","last_undegraded":"2023-12-19T14:54:57.828208+0000","last_fullsized":"2023-12-19T14:54:57.828208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:34.266164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.815860+0000","last_change":"2023-12-19T14:54:58.815860+0000","last_active":"2023-12-19T14:54:58.815860+0000","last_peered":"2023-12-19T14:54:58.815860+0000","last_clean":"2023-12-19T14:54:58.815860+0000","last_became_active":"2023-12-19T14:54:58.815395+0000","last_became_peered":"2023-12-19T14:54:58.815395+0000","last_unstale":"2023-12-19T14:54:58.815860+0000","last_undegraded":"2023-12-19T14:54:58.815860+0000","last_fullsized":"2023-12-19T14:54:58.815860+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:44:38.311240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.828409+0000","last_change":"2023-12-19T14:54:57.820540+0000","last_active":"2023-12-19T14:54:57.828409+0000","last_peered":"2023-12-19T14:54:57.828409+0000","last_clean":"2023-12-19T14:54:57.828409+0000","last_became_active":"2023-12-19T14:54:57.818896+0000","last_became_peered":"2023-12-19T14:54:57.818896+0000","last_unstale":"2023-12-19T14:54:57.828409+0000","last_undegraded":"2023-12-19T14:54:57.828409+0000","last_fullsized":"2023-12-19T14:54:57.828409+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:01.544789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.825254+0000","last_change":"2023-12-19T14:54:58.825254+0000","last_active":"2023-12-19T14:54:58.825254+0000","last_peered":"2023-12-19T14:54:58.825254+0000","last_clean":"2023-12-19T14:54:58.825254+0000","last_became_active":"2023-12-19T14:54:58.824967+0000","last_became_peered":"2023-12-19T14:54:58.824967+0000","last_unstale":"2023-12-19T14:54:58.825254+0000","last_undegraded":"2023-12-19T14:54:58.825254+0000","last_fullsized":"2023-12-19T14:54:58.825254+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:03.999291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"99'2","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.964014+0000","last_change":"2023-12-19T14:54:57.818908+0000","last_active":"2023-12-19T14:54:58.964014+0000","last_peered":"2023-12-19T14:54:58.964014+0000","last_clean":"2023-12-19T14:54:58.964014+0000","last_became_active":"2023-12-19T14:54:57.818513+0000","last_became_peered":"2023-12-19T14:54:57.818513+0000","last_unstale":"2023-12-19T14:54:58.964014+0000","last_undegraded":"2023-12-19T14:54:58.964014+0000","last_fullsized":"2023-12-19T14:54:58.964014+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:45.338402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827728+0000","last_change":"2023-12-19T14:54:58.827728+0000","last_active":"2023-12-19T14:54:58.827728+0000","last_peered":"2023-12-19T14:54:58.827728+0000","last_clean":"2023-12-19T14:54:58.827728+0000","last_became_active":"2023-12-19T14:54:58.827246+0000","last_became_peered":"2023-12-19T14:54:58.827246+0000","last_unstale":"2023-12-19T14:54:58.827728+0000","last_undegraded":"2023-12-19T14:54:58.827728+0000","last_fullsized":"2023-12-19T14:54:58.827728+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:57.796572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826943+0000","last_change":"2023-12-19T14:54:57.820718+0000","last_active":"2023-12-19T14:54:57.826943+0000","last_peered":"2023-12-19T14:54:57.826943+0000","last_clean":"2023-12-19T14:54:57.826943+0000","last_became_active":"2023-12-19T14:54:57.820257+0000","last_became_peered":"2023-12-19T14:54:57.820257+0000","last_unstale":"2023-12-19T14:54:57.826943+0000","last_undegraded":"2023-12-19T14:54:57.826943+0000","last_fullsized":"2023-12-19T14:54:57.826943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:11.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.824911+0000","last_change":"2023-12-19T14:54:58.824911+0000","last_active":"2023-12-19T14:54:58.824911+0000","last_peered":"2023-12-19T14:54:58.824911+0000","last_clean":"2023-12-19T14:54:58.824911+0000","last_became_active":"2023-12-19T14:54:58.823038+0000","last_became_peered":"2023-12-19T14:54:58.823038+0000","last_unstale":"2023-12-19T14:54:58.824911+0000","last_undegraded":"2023-12-19T14:54:58.824911+0000","last_fullsized":"2023-12-19T14:54:58.824911+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:00.765295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822461+0000","last_change":"2023-12-19T14:54:57.819815+0000","last_active":"2023-12-19T14:54:57.822461+0000","last_peered":"2023-12-19T14:54:57.822461+0000","last_clean":"2023-12-19T14:54:57.822461+0000","last_became_active":"2023-12-19T14:54:57.819511+0000","last_became_peered":"2023-12-19T14:54:57.819511+0000","last_unstale":"2023-12-19T14:54:57.822461+0000","last_undegraded":"2023-12-19T14:54:57.822461+0000","last_fullsized":"2023-12-19T14:54:57.822461+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:59.749234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.833729+0000","last_change":"2023-12-19T14:54:58.833729+0000","last_active":"2023-12-19T14:54:58.833729+0000","last_peered":"2023-12-19T14:54:58.833729+0000","last_clean":"2023-12-19T14:54:58.833729+0000","last_became_active":"2023-12-19T14:54:58.827596+0000","last_became_peered":"2023-12-19T14:54:58.827596+0000","last_unstale":"2023-12-19T14:54:58.833729+0000","last_undegraded":"2023-12-19T14:54:58.833729+0000","last_fullsized":"2023-12-19T14:54:58.833729+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:46.354463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.818069+0000","last_change":"2023-12-19T14:54:57.815372+0000","last_active":"2023-12-19T14:54:57.818069+0000","last_peered":"2023-12-19T14:54:57.818069+0000","last_clean":"2023-12-19T14:54:57.818069+0000","last_became_active":"2023-12-19T14:54:57.815069+0000","last_became_peered":"2023-12-19T14:54:57.815069+0000","last_unstale":"2023-12-19T14:54:57.818069+0000","last_undegraded":"2023-12-19T14:54:57.818069+0000","last_fullsized":"2023-12-19T14:54:57.818069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:49:34.468525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.827508+0000","last_change":"2023-12-19T14:54:58.827508+0000","last_active":"2023-12-19T14:54:58.827508+0000","last_peered":"2023-12-19T14:54:58.827508+0000","last_clean":"2023-12-19T14:54:58.827508+0000","last_became_active":"2023-12-19T14:54:58.827238+0000","last_became_peered":"2023-12-19T14:54:58.827238+0000","last_unstale":"2023-12-19T14:54:58.827508+0000","last_undegraded":"2023-12-19T14:54:58.827508+0000","last_fullsized":"2023-12-19T14:54:58.827508+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:02:06.725392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.820349+0000","last_change":"2023-12-19T14:54:57.817044+0000","last_active":"2023-12-19T14:54:57.820349+0000","last_peered":"2023-12-19T14:54:57.820349+0000","last_clean":"2023-12-19T14:54:57.820349+0000","last_became_active":"2023-12-19T14:54:57.816580+0000","last_became_peered":"2023-12-19T14:54:57.816580+0000","last_unstale":"2023-12-19T14:54:57.820349+0000","last_undegraded":"2023-12-19T14:54:57.820349+0000","last_fullsized":"2023-12-19T14:54:57.820349+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:14.599834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.812365+0000","last_change":"2023-12-19T14:54:58.812365+0000","last_active":"2023-12-19T14:54:58.812365+0000","last_peered":"2023-12-19T14:54:58.812365+0000","last_clean":"2023-12-19T14:54:58.812365+0000","last_became_active":"2023-12-19T14:54:58.812069+0000","last_became_peered":"2023-12-19T14:54:58.812069+0000","last_unstale":"2023-12-19T14:54:58.812365+0000","last_undegraded":"2023-12-19T14:54:58.812365+0000","last_fullsized":"2023-12-19T14:54:58.812365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:49.814798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.826080+0000","last_change":"2023-12-19T14:54:57.819715+0000","last_active":"2023-12-19T14:54:57.826080+0000","last_peered":"2023-12-19T14:54:57.826080+0000","last_clean":"2023-12-19T14:54:57.826080+0000","last_became_active":"2023-12-19T14:54:57.819431+0000","last_became_peered":"2023-12-19T14:54:57.819431+0000","last_unstale":"2023-12-19T14:54:57.826080+0000","last_undegraded":"2023-12-19T14:54:57.826080+0000","last_fullsized":"2023-12-19T14:54:57.826080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:37.719310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.830224+0000","last_change":"2023-12-19T14:54:58.830224+0000","last_active":"2023-12-19T14:54:58.830224+0000","last_peered":"2023-12-19T14:54:58.830224+0000","last_clean":"2023-12-19T14:54:58.830224+0000","last_became_active":"2023-12-19T14:54:58.829850+0000","last_became_peered":"2023-12-19T14:54:58.829850+0000","last_unstale":"2023-12-19T14:54:58.830224+0000","last_undegraded":"2023-12-19T14:54:58.830224+0000","last_fullsized":"2023-12-19T14:54:58.830224+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:02.560851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.817480+0000","last_change":"2023-12-19T14:54:57.813153+0000","last_active":"2023-12-19T14:54:57.817480+0000","last_peered":"2023-12-19T14:54:57.817480+0000","last_clean":"2023-12-19T14:54:57.817480+0000","last_became_active":"2023-12-19T14:54:57.812154+0000","last_became_peered":"2023-12-19T14:54:57.812154+0000","last_unstale":"2023-12-19T14:54:57.817480+0000","last_undegraded":"2023-12-19T14:54:57.817480+0000","last_fullsized":"2023-12-19T14:54:57.817480+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:20.955411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.823597+0000","last_change":"2023-12-19T14:54:58.823597+0000","last_active":"2023-12-19T14:54:58.823597+0000","last_peered":"2023-12-19T14:54:58.823597+0000","last_clean":"2023-12-19T14:54:58.823597+0000","last_became_active":"2023-12-19T14:54:58.821449+0000","last_became_peered":"2023-12-19T14:54:58.821449+0000","last_unstale":"2023-12-19T14:54:58.823597+0000","last_undegraded":"2023-12-19T14:54:58.823597+0000","last_fullsized":"2023-12-19T14:54:58.823597+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:49:35.484586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822631+0000","last_change":"2023-12-19T14:54:57.815024+0000","last_active":"2023-12-19T14:54:57.822631+0000","last_peered":"2023-12-19T14:54:57.822631+0000","last_clean":"2023-12-19T14:54:57.822631+0000","last_became_active":"2023-12-19T14:54:57.814715+0000","last_became_peered":"2023-12-19T14:54:57.814715+0000","last_unstale":"2023-12-19T14:54:57.822631+0000","last_undegraded":"2023-12-19T14:54:57.822631+0000","last_fullsized":"2023-12-19T14:54:57.822631+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:44:37.295179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":8,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-12-19T14:54:58.821052+0000","last_change":"2023-12-19T14:54:58.821052+0000","last_active":"2023-12-19T14:54:58.821052+0000","last_peered":"2023-12-19T14:54:58.821052+0000","last_clean":"2023-12-19T14:54:58.821052+0000","last_became_active":"2023-12-19T14:54:58.820795+0000","last_became_peered":"2023-12-19T14:54:58.820795+0000","last_unstale":"2023-12-19T14:54:58.821052+0000","last_undegraded":"2023-12-19T14:54:58.821052+0000","last_fullsized":"2023-12-19T14:54:58.821052+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:57.774736+0000","last_clean_scrub_stamp":"2023-12-19T14:54:57.774736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:17.108734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.822313+0000","last_change":"2023-12-19T14:54:57.816532+0000","last_active":"2023-12-19T14:54:57.822313+0000","last_peered":"2023-12-19T14:54:57.822313+0000","last_clean":"2023-12-19T14:54:57.822313+0000","last_became_active":"2023-12-19T14:54:57.815581+0000","last_became_peered":"2023-12-19T14:54:57.815581+0000","last_unstale":"2023-12-19T14:54:57.822313+0000","last_undegraded":"2023-12-19T14:54:57.822313+0000","last_fullsized":"2023-12-19T14:54:57.822313+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:54:56.758675+0000","last_clean_scrub_stamp":"2023-12-19T14:54:56.758675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:23.945969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'183","reported_seq":464,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T14:54:57.804182+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:54:57.804182+0000","last_peered":"2023-12-19T14:54:57.804182+0000","last_clean":"2023-12-19T14:54:57.804182+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:54:57.804182+0000","last_undegraded":"2023-12-19T14:54:57.804182+0000","last_fullsized":"2023-12-19T14:54:57.804182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311344,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060496,"statfs":{"total":96636764160,"available":96317947904,"internally_reserved":0,"allocated":1810432,"data_stored":1196241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.498,"5min":3.544,"15min":3.544},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81699999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5369999999999999,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.093},"min":{"1min":0.91700000000000004,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.4379999999999999,"5min":1.73,"15min":1.73},"last":1.115},{"interface":"front","average":{"1min":1.276,"5min":1.1950000000000001,"15min":1.1819999999999999},"min":{"1min":0.93400000000000005,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.984},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2729999999999999,"15min":1.2330000000000001},"min":{"1min":0.96699999999999997,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.571,"5min":1.6919999999999999,"15min":1.7609999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.274,"15min":1.278},"min":{"1min":1.1359999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.667,"5min":1.6890000000000001,"15min":2.1099999999999999},"last":0.97399999999999998}]},{"osd":3,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.28,"15min":1.29},"min":{"1min":0.92500000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.5409999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":1.423},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.3109999999999999},"min":{"1min":1.038,"5min":0.95899999999999996,"15min":0.79700000000000004},"max":{"1min":1.6100000000000001,"5min":1.9059999999999999,"15min":2.2280000000000002},"last":1.044}]},{"osd":4,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.1839999999999999,"15min":1.1699999999999999},"min":{"1min":0.68500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.752,"5min":1.752,"15min":2.1930000000000001},"last":1.091},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2410000000000001,"15min":1.222},"min":{"1min":0.90300000000000002,"5min":0.75700000000000001,"15min":0.67500000000000004},"max":{"1min":1.6000000000000001,"5min":1.6319999999999999,"15min":1.76},"last":1.4910000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2210000000000001,"15min":1.2270000000000001},"min":{"1min":1.034,"5min":0.496,"15min":0.496},"max":{"1min":1.6930000000000001,"5min":1.8109999999999999,"15min":1.976},"last":1.2},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3029999999999999,"15min":1.3180000000000001},"min":{"1min":0.90200000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":2.3620000000000001},"last":1.3580000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3899999999999999,"15min":1.3999999999999999},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.65600000000000003},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":3.1589999999999998},"last":1.274},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3620000000000001,"15min":1.361},"min":{"1min":1.0620000000000001,"5min":0.88500000000000001,"15min":0.70299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8720000000000001,"15min":2.3029999999999999},"last":1.5609999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302556,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069284,"statfs":{"total":96636764160,"available":96326946816,"internally_reserved":0,"allocated":1200128,"data_stored":598740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1220000000000001,"15min":1.1479999999999999},"min":{"1min":0.73699999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.466,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.1890000000000001},"min":{"1min":0.80800000000000005,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.577,"5min":1.845,"15min":2.335},"last":1.224}]},{"osd":1,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2090000000000001,"15min":1.2070000000000001},"min":{"1min":0.86899999999999999,"5min":0.77800000000000002,"15min":0.56999999999999995},"max":{"1min":1.4790000000000001,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.444},{"interface":"front","average":{"1min":1.194,"5min":1.2569999999999999,"15min":1.2649999999999999},"min":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.419,"5min":3.226,"15min":3.226},"last":1.1259999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.284,"15min":1.264},"min":{"1min":0.86699999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6519999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.2769999999999999,"15min":1.2569999999999999},"min":{"1min":0.98399999999999999,"5min":0.82399999999999995,"15min":0.67100000000000004},"max":{"1min":1.4159999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.292}]},{"osd":3,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3149999999999999,"15min":1.3200000000000001},"min":{"1min":0.88300000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.587,"5min":1.9239999999999999,"15min":3.2330000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.353,"15min":1.355},"min":{"1min":0.94899999999999995,"5min":0.84299999999999997,"15min":0.69599999999999995},"max":{"1min":1.673,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.246,"15min":1.2330000000000001},"min":{"1min":0.83899999999999997,"5min":0.621,"15min":0.55600000000000005},"max":{"1min":1.4419999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2809999999999999,"15min":1.3220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.57599999999999996},"max":{"1min":1.5960000000000001,"5min":3.3460000000000001,"15min":3.4399999999999999},"last":1.268}]},{"osd":5,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3109999999999999,"15min":1.327},"min":{"1min":0.89200000000000002,"5min":0.71799999999999997,"15min":0.63100000000000001},"max":{"1min":1.746,"5min":2.02,"15min":2.8620000000000001},"last":1.351},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.407,"15min":1.3839999999999999},"min":{"1min":0.92000000000000004,"5min":0.82799999999999996,"15min":0.51000000000000001},"max":{"1min":1.5580000000000001,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.1950000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.4379999999999999},"min":{"1min":0.88300000000000001,"5min":0.85899999999999999,"15min":0.53000000000000003},"max":{"1min":1.8029999999999999,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.377},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4890000000000001,"15min":1.486},"min":{"1min":0.98299999999999998,"5min":0.67100000000000004,"15min":0.63100000000000001},"max":{"1min":1.8089999999999999,"5min":3.742,"15min":3.742},"last":1.411}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298472,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073368,"statfs":{"total":96636764160,"available":96331128832,"internally_reserved":0,"allocated":1212416,"data_stored":605851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0509999999999999,"15min":1.0600000000000001},"min":{"1min":0.71399999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4450000000000001,"5min":1.601,"15min":1.619},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.1379999999999999,"15min":1.1359999999999999},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.58699999999999997},"max":{"1min":1.417,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.4430000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.29},"min":{"1min":0.78500000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.508,"5min":1.8029999999999999,"15min":1.9650000000000001},"last":1.403},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.23,"15min":1.242},"min":{"1min":0.79000000000000004,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.677,"5min":1.677,"15min":1.7869999999999999},"last":1.2989999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2070000000000001,"15min":1.2350000000000001},"min":{"1min":0.99399999999999999,"5min":0.67000000000000004,"15min":0.47399999999999998},"max":{"1min":1.48,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.1599999999999999,"15min":1.2010000000000001},"min":{"1min":0.90000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.5720000000000001,"5min":1.609,"15min":2.0739999999999998},"last":0.94499999999999995}]},{"osd":3,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2869999999999999,"15min":1.2749999999999999},"min":{"1min":0.96099999999999997,"5min":0.76200000000000001,"15min":0.73199999999999998},"max":{"1min":1.6499999999999999,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.137},{"interface":"front","average":{"1min":1.246,"5min":1.2410000000000001,"15min":1.2729999999999999},"min":{"1min":0.97799999999999998,"5min":0.82299999999999995,"15min":0.68999999999999995},"max":{"1min":1.601,"5min":1.73,"15min":2.2149999999999999},"last":1.4990000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1140000000000001,"15min":1.1200000000000001},"min":{"1min":0.86899999999999999,"5min":0.72799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5820000000000001,"5min":1.603,"15min":2.1099999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.2450000000000001,"15min":1.2470000000000001},"min":{"1min":0.80000000000000004,"5min":0.69099999999999995,"15min":0.49299999999999999},"max":{"1min":1.6950000000000001,"5min":2.27,"15min":2.27},"last":0.83999999999999997}]},{"osd":6,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3100000000000001,"15min":1.3380000000000001},"min":{"1min":0.84399999999999997,"5min":0.79600000000000004,"15min":0.76000000000000001},"max":{"1min":1.982,"5min":1.982,"15min":2.4790000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.333},"min":{"1min":1.123,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.794,"5min":1.8540000000000001,"15min":2.0030000000000001},"last":1.3759999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.361,"15min":1.3660000000000001},"min":{"1min":1.0649999999999999,"5min":0.76700000000000002,"15min":0.58899999999999997},"max":{"1min":1.7490000000000001,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.524},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3540000000000001,"15min":1.3919999999999999},"min":{"1min":1.054,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.411},"last":1.2390000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575040,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302588,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069252,"statfs":{"total":96636764160,"available":96326914048,"internally_reserved":0,"allocated":1232896,"data_stored":607491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0609999999999999,"15min":1.069},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.7869999999999999},"last":1.405},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.095,"15min":1.091},"min":{"1min":0.69499999999999995,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.5109999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.1559999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.236,"15min":1.2430000000000001},"min":{"1min":0.75,"5min":0.623,"15min":0.623},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":2.1259999999999999},"last":1.454},{"interface":"front","average":{"1min":1.222,"5min":1.244,"15min":1.276},"min":{"1min":0.73299999999999998,"5min":0.70299999999999996,"15min":0.65100000000000002},"max":{"1min":1.595,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.518}]},{"osd":2,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.165,"15min":1.1539999999999999},"min":{"1min":0.91600000000000004,"5min":0.71799999999999997,"15min":0.56999999999999995},"max":{"1min":1.625,"5min":1.625,"15min":1.891},"last":1.113},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1819999999999999,"15min":1.2070000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.893},"last":1.2969999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.163,"15min":1.1830000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.59599999999999997},"max":{"1min":1.359,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.256,"15min":1.2729999999999999},"min":{"1min":1.034,"5min":0.83799999999999997,"15min":0.60599999999999998},"max":{"1min":1.6259999999999999,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.329}]},{"osd":5,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1499999999999999,"15min":1.1619999999999999},"min":{"1min":0.72399999999999998,"5min":0.71099999999999997,"15min":0.57899999999999996},"max":{"1min":1.452,"5min":2.1800000000000002,"15min":2.3500000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1419999999999999,"15min":1.1679999999999999},"min":{"1min":0.79600000000000004,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.5840000000000001,"5min":2.077,"15min":2.161},"last":1.3640000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.276,"15min":1.323},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.6579999999999999,"5min":2.8119999999999998,"15min":3.27},"last":1.236},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.421},"min":{"1min":0.63400000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.843,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.5900000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:25 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.3260000000000001,"15min":1.357},"min":{"1min":0.89200000000000002,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.637,"5min":2.9910000000000001,"15min":5.2060000000000004},"last":1.484},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4319999999999999,"15min":1.46},"min":{"1min":0.89900000000000002,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":2.0329999999999999,"5min":3.008,"15min":3.6739999999999999},"last":1.554}]}]},{"osd":3,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306668,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065172,"statfs":{"total":96636764160,"available":96322736128,"internally_reserved":0,"allocated":1216512,"data_stored":606405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84999999999999998,"15min":0.89000000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2010000000000001,"5min":1.3640000000000001,"15min":4.1159999999999997},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.026,"5min":1.01,"15min":1.0129999999999999},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.542,"5min":1.542,"15min":3.3929999999999998},"last":1.111}]},{"osd":1,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":0.95799999999999996},"min":{"1min":0.69599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0149999999999999,"5min":1.611,"15min":2.4390000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.121,"15min":1.1220000000000001},"min":{"1min":0.749,"5min":0.63500000000000001,"15min":0.60199999999999998},"max":{"1min":1.5189999999999999,"5min":1.661,"15min":4.476},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1459999999999999,"15min":1.1419999999999999},"min":{"1min":0.89900000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.639,"5min":1.76,"15min":2.6989999999999998},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.61699999999999999},"max":{"1min":1.5309999999999999,"5min":1.647,"15min":2.5110000000000001},"last":1.367}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.345},"min":{"1min":0.84899999999999998,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.526,"5min":1.768,"15min":3.46},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2829999999999999,"15min":1.3069999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.542,"5min":1.6970000000000001,"15min":2.9409999999999998},"last":1.2669999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.286,"15min":1.3280000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.629,"5min":1.7709999999999999,"15min":3.1019999999999999},"last":1.04},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.4159999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.833,"5min":2.1619999999999999,"15min":4.3650000000000002},"last":1.1499999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.446,"15min":1.4730000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.9099999999999999,"5min":2.2709999999999999,"15min":4.5999999999999996},"last":1.3},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.419,"15min":1.444},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.599,"5min":2.0539999999999998,"15min":3.4329999999999998},"last":1.325}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.516,"15min":1.5429999999999999},"min":{"1min":0.93300000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.7629999999999999,"5min":2.1640000000000001,"15min":3.8109999999999999},"last":1.508},{"interface":"front","average":{"1min":1.429,"5min":1.45,"15min":1.488},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.728,"5min":2.0670000000000002,"15min":3.7240000000000002},"last":1.1930000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306660,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065180,"statfs":{"total":96636764160,"available":96322744320,"internally_reserved":0,"allocated":1208320,"data_stored":604840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89400000000000002,"15min":0.90400000000000003},"min":{"1min":0.60599999999999998,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.946},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.056,"15min":1.0720000000000001},"min":{"1min":0.63300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.605,"5min":1.605,"15min":1.8560000000000001},"last":1.2250000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.82199999999999995,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.69199999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.871}]},{"osd":3,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1.0009999999999999},"min":{"1min":0.61899999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.304,"5min":1.5669999999999999,"15min":1.9370000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.081,"15min":1.105},"min":{"1min":0.80100000000000005,"5min":0.66000000000000003,"15min":0.55000000000000004},"max":{"1min":1.603,"5min":1.603,"15min":1.6930000000000001},"last":0.99199999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":0.95199999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.7549999999999999,"5min":1.893,"15min":3.4950000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.403,"15min":1.3999999999999999},"min":{"1min":1.2230000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.923,"5min":1.96,"15min":2.0409999999999999},"last":1.5209999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3740000000000001,"15min":1.3600000000000001},"min":{"1min":1.1040000000000001,"5min":0.876,"15min":0.876},"max":{"1min":1.7669999999999999,"5min":2.0760000000000001,"15min":3.8599999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.3340000000000001,"15min":1.351},"min":{"1min":0.93899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.6759999999999999,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.4610000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.80600000000000005},"max":{"1min":1.6950000000000001,"5min":1.829,"15min":2.8759999999999999},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4370000000000001,"15min":1.45},"min":{"1min":1.2170000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.8280000000000001,"5min":2.1200000000000001,"15min":3.7309999999999999},"last":1.395}]},{"osd":7,"last update":"Tue Dec 19 14:54:17 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.482,"15min":1.492},"min":{"1min":1.171,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":2.044,"5min":2.0699999999999998,"15min":2.5550000000000002},"last":1.651},{"interface":"front","average":{"1min":1.599,"5min":1.4710000000000001,"15min":1.4870000000000001},"min":{"1min":1.258,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.4049999999999998},"last":1.371}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306672,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065168,"statfs":{"total":96636764160,"available":96322732032,"internally_reserved":0,"allocated":1220608,"data_stored":605963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.877,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":1.238,"5min":1.464,"15min":1.871},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.98899999999999999,"15min":0.996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.50700000000000001},"max":{"1min":1.393,"5min":1.7070000000000001,"15min":2.468},"last":0.98499999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0169999999999999,"15min":1.0209999999999999},"min":{"1min":0.71699999999999997,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":1.2589999999999999,"5min":1.714,"15min":1.7829999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0880000000000001,"15min":1.125},"min":{"1min":0.66300000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.5660000000000001,"5min":2.0230000000000001,"15min":3.0619999999999998},"last":1.2589999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.236,"15min":1.2350000000000001},"min":{"1min":0.871,"5min":0.70999999999999996,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.0820000000000001},"min":{"1min":0.81200000000000006,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6439999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.0609999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2869999999999999,"15min":1.3109999999999999},"min":{"1min":0.93000000000000005,"5min":0.624,"15min":0.624},"max":{"1min":1.724,"5min":1.7370000000000001,"15min":2.6440000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.401,"15min":1.403},"min":{"1min":1.0009999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.677,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.5149999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.262,"15min":1.296},"min":{"1min":1.028,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.538,"5min":1.6719999999999999,"15min":2.1320000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3720000000000001,"15min":1.391},"min":{"1min":0.86399999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.76,"5min":1.76,"15min":2.79},"last":1.409}]},{"osd":6,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.333,"15min":1.3640000000000001},"min":{"1min":1.0129999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7070000000000001,"5min":1.78,"15min":2.4620000000000002},"last":1.343},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.4319999999999999,"15min":1.456},"min":{"1min":1.073,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":2.8999999999999999},"last":1.242}]},{"osd":7,"last update":"Tue Dec 19 14:54:22 2023","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.474,"15min":1.488},"min":{"1min":1.0740000000000001,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":2.0470000000000002,"15min":3.782},"last":1.633},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":1.046,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":2.5390000000000001},"last":1.5609999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307240,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064600,"statfs":{"total":96636764160,"available":96322150400,"internally_reserved":0,"allocated":1802240,"data_stored":1195208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.85299999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.97299999999999998},"min":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.53600000000000003},"max":{"1min":1.4810000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.22}]},{"osd":2,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.80000000000000004,"5min":0.69699999999999995,"15min":0.55200000000000005},"max":{"1min":1.5189999999999999,"5min":1.6339999999999999,"15min":1.8979999999999999},"last":1.462},{"interface":"front","average":{"1min":1.288,"5min":1.2410000000000001,"15min":1.2210000000000001},"min":{"1min":0.82199999999999995,"5min":0.70799999999999996,"15min":0.54300000000000004},"max":{"1min":1.728,"5min":1.982,"15min":1.982},"last":1.0680000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0629999999999999,"15min":1.0309999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6359999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.091,"5min":1.1679999999999999,"15min":1.1819999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.829,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.361}]},{"osd":4,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3169999999999999,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.70199999999999996},"max":{"1min":1.754,"5min":2.153,"15min":2.298},"last":1.123},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.302,"15min":1.298},"min":{"1min":1.026,"5min":0.83999999999999997,"15min":0.80200000000000005},"max":{"1min":1.579,"5min":1.7689999999999999,"15min":2.5350000000000001},"last":1.159}]},{"osd":5,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3600000000000001,"15min":1.375},"min":{"1min":0.89300000000000002,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9410000000000001,"5min":1.974,"15min":2.4630000000000001},"last":1.528},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":1.0089999999999999,"5min":0.96399999999999997,"15min":0.58299999999999996},"max":{"1min":1.9339999999999999,"5min":2.2189999999999999,"15min":2.6160000000000001},"last":1.3979999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.343,"15min":1.3460000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.88300000000000001},"max":{"1min":1.7130000000000001,"5min":2.0150000000000001,"15min":2.339},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4299999999999999,"15min":1.452},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.0590000000000002,"5min":2.113,"15min":2.5680000000000001},"last":1.5760000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:54:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.488,"15min":1.4710000000000001},"min":{"1min":1.083,"5min":1.0660000000000001,"15min":0.91500000000000004},"max":{"1min":2.0019999999999998,"5min":2.2629999999999999,"15min":2.5129999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4159999999999999,"15min":1.444},"min":{"1min":1.04,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.802,"5min":1.931,"15min":2.4500000000000002},"last":1.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:55:03.755 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:55:03.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:55:03.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.974+0000 7f0f35d85700 1 Processor -- start 2023-12-19T14:55:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.974+0000 7f0f35d85700 1 -- start start 2023-12-19T14:55:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f35d85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f30158e20 0x7f0f300ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f35d85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 0x7f0f300ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f35d85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f300cbdf0 con 0x7f0f300cb970 2023-12-19T14:55:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f35d85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f300cc0d0 con 0x7f0f30158e20 2023-12-19T14:55:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f35d85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f300cc3b0 con 0x7f0f300ca2c0 2023-12-19T14:55:03.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f30158e20 0x7f0f300ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f30158e20 0x7f0f300ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54188/0 (socket says 172.21.15.50:54188) 2023-12-19T14:55:03.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/183602207 learned_addr learned my addr 172.21.15.50:0/183602207 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2ffff700 1 --2- 172.21.15.50:0/183602207 >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 0x7f0f300ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/183602207 >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 msgr2=0x7f0f300ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 --2- 172.21.15.50:0/183602207 >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 0x7f0f300ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/183602207 >> v1:172.21.15.125:6789/0 conn(0x7f0f300cb970 legacy=0x7f0f3006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/183602207 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300cc690 con 0x7f0f30158e20 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2ffff700 1 --2- 172.21.15.50:0/183602207 >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 0x7f0f300ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2f7fe700 1 --2- 172.21.15.50:0/183602207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f30158e20 0x7f0f300ce5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0f300cf780 tx=0x7f0f20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf45b5969615e61a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2e7fc700 1 -- 172.21.15.50:0/183602207 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000b7e0 con 0x7f0f30158e20 2023-12-19T14:55:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2e7fc700 1 -- 172.21.15.50:0/183602207 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2000b940 con 0x7f0f30158e20 2023-12-19T14:55:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.978+0000 7f0f2e7fc700 1 -- 172.21.15.50:0/183602207 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000bce0 con 0x7f0f30158e20 2023-12-19T14:55:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 -- 172.21.15.50:0/183602207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f30158e20 msgr2=0x7f0f300ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 --2- 172.21.15.50:0/183602207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f30158e20 0x7f0f300ce5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0f300cf780 tx=0x7f0f20009250 comp rx=0 tx=0).stop 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 -- 172.21.15.50:0/183602207 shutdown_connections 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 --2- 172.21.15.50:0/183602207 >> v2:172.21.15.125:3300/0 conn(0x7f0f300ca2c0 0x7f0f300ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 --2- 172.21.15.50:0/183602207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f30158e20 0x7f0f300ce5c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 -- 172.21.15.50:0/183602207 >> 172.21.15.50:0/183602207 conn(0x7f0f30067bb0 msgr2=0x7f0f300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 -- 172.21.15.50:0/183602207 shutdown_connections 2023-12-19T14:55:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.982+0000 7f0f35d85700 1 -- 172.21.15.50:0/183602207 wait complete. 2023-12-19T14:55:03.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 Processor -- start 2023-12-19T14:55:03.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 -- start start 2023-12-19T14:55:03.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f30158e20 0x7f0f30169100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f30169640 0x7f0f3015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f300b4130 con 0x7f0f300ca2c0 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3014f320 con 0x7f0f30158e20 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f35d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f300b3520 con 0x7f0f30169640 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54192/0 (socket says 172.21.15.50:54192) 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/2667866731 learned_addr learned my addr 172.21.15.50:0/2667866731 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2effd700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f30158e20 0x7f0f30169100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2ffff700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f30169640 0x7f0f3015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f30169640 msgr2=0x7f0f3015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f30169640 0x7f0f3015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f30158e20 msgr2=0x7f0f30169100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f30158e20 0x7f0f30169100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.986+0000 7f0f2f7fe700 1 -- 172.21.15.50:0/2667866731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200094f0 con 0x7f0f300ca2c0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f2f7fe700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0f30069a80 tx=0x7f0f20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001a420 con 0x7f0f300ca2c0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2001a580 con 0x7f0f300ca2c0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f300b3800 con 0x7f0f300ca2c0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001a920 con 0x7f0f300ca2c0 2023-12-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f300b3dc0 con 0x7f0f300ca2c0 2023-12-19T14:55:03.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.990+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30071120 con 0x7f0f300ca2c0 2023-12-19T14:55:03.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.994+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0f2001ad10 con 0x7f0f300ca2c0 2023-12-19T14:55:03.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.994+0000 7f0f177fe700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 0x7f0f1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:03.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.994+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0f2007e530 con 0x7f0f300ca2c0 2023-12-19T14:55:03.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.994+0000 7f0f2effd700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 0x7f0f1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:03.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:03.994+0000 7f0f2effd700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 0x7f0f1c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f301568b0 tx=0x7f0f24009040 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:04.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.002+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f2004e810 con 0x7f0f300ca2c0 2023-12-19T14:55:04.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.234+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0f3006a7a0 con 0x7f0f300ca2c0 2023-12-19T14:55:04.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.246+0000 7f0f177fe700 1 -- 172.21.15.50:0/2667866731 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0f20051e30 con 0x7f0f300ca2c0 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.246+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 msgr2=0x7f0f1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.246+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 0x7f0f1c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f301568b0 tx=0x7f0f24009040 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.246+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 msgr2=0x7f0f3014ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.246+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0f30069a80 tx=0x7f0f20017000 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 shutdown_connections 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f30158e20 0x7f0f30169100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f1c060670 0x7f0f1c062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f30169640 0x7f0f3015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 --2- 172.21.15.50:0/2667866731 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f300ca2c0 0x7f0f3014ede0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 >> 172.21.15.50:0/2667866731 conn(0x7f0f30067bb0 msgr2=0x7f0f3014fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:04.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 shutdown_connections 2023-12-19T14:55:04.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.250+0000 7f0f35d85700 1 -- 172.21.15.50:0/2667866731 wait complete. 2023-12-19T14:55:04.253 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:55:04.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.442+0000 7fe2358ae700 1 Processor -- start 2023-12-19T14:55:04.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.442+0000 7fe2358ae700 1 -- start start 2023-12-19T14:55:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.442+0000 7fe2358ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2300cc3b0 0x7fe2300cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.442+0000 7fe2358ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe230157060 0x7fe23015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.442+0000 7fe2358ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2300cd5a0 con 0x7fe230156c90 2023-12-19T14:55:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe2358ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe230157470 con 0x7fe2300cc3b0 2023-12-19T14:55:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe2358ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2300cc990 con 0x7fe230157060 2023-12-19T14:55:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2300cc3b0 0x7fe2300cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22effd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe230156c90 0x7fe2300c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35830/0 (socket says 172.21.15.50:35830) 2023-12-19T14:55:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22effd700 1 -- 172.21.15.50:0/1675968047 learned_addr learned my addr 172.21.15.50:0/1675968047 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22f7fe700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe230157060 0x7fe23015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 -- 172.21.15.50:0/1675968047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe230157060 msgr2=0x7fe23015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe230157060 0x7fe23015ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 -- 172.21.15.50:0/1675968047 >> v1:172.21.15.50:6789/0 conn(0x7fe230156c90 legacy=0x7fe2300c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 -- 172.21.15.50:0/1675968047 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2300ccc70 con 0x7fe2300cc3b0 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22e7fc700 1 -- 172.21.15.50:0/1675968047 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255741645 0 0) 0x7fe2300cd5a0 con 0x7fe230156c90 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe227fff700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2300cc3b0 0x7fe2300cda30 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe2180098e0 tx=0x7fe2180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7f7efc24d80587 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22e7fc700 1 -- 172.21.15.50:0/1675968047 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21800b640 con 0x7fe2300cc3b0 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22e7fc700 1 -- 172.21.15.50:0/1675968047 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21800b7a0 con 0x7fe2300cc3b0 2023-12-19T14:55:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe22e7fc700 1 -- 172.21.15.50:0/1675968047 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21800bb40 con 0x7fe2300cc3b0 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe2358ae700 1 -- 172.21.15.50:0/1675968047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2300cc3b0 msgr2=0x7fe2300cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.446+0000 7fe2358ae700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2300cc3b0 0x7fe2300cda30 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe2180098e0 tx=0x7fe2180091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 -- 172.21.15.50:0/1675968047 shutdown_connections 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe230157060 0x7fe23015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 --2- 172.21.15.50:0/1675968047 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2300cc3b0 0x7fe2300cda30 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 -- 172.21.15.50:0/1675968047 >> 172.21.15.50:0/1675968047 conn(0x7fe23006c7c0 msgr2=0x7fe23006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 -- 172.21.15.50:0/1675968047 shutdown_connections 2023-12-19T14:55:04.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 -- 172.21.15.50:0/1675968047 wait complete. 2023-12-19T14:55:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 Processor -- start 2023-12-19T14:55:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.450+0000 7fe2358ae700 1 -- start start 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 0x7fe2300acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe230157060 0x7fe230165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2301575b0 con 0x7fe230156c90 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe230157e50 con 0x7fe230157060 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23015a270 con 0x7fe2300cc3b0 2023-12-19T14:55:04.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe227fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 0x7fe2300acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:45806/0 (socket says 172.21.15.50:45806) 2023-12-19T14:55:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 -- 172.21.15.50:0/2565502426 learned_addr learned my addr 172.21.15.50:0/2565502426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:04.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22f7fe700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe230157060 0x7fe230165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 -- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe230157060 msgr2=0x7fe230165fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe230157060 0x7fe230165fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 -- 172.21.15.50:0/2565502426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 msgr2=0x7fe2300acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 0x7fe2300acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 -- 172.21.15.50:0/2565502426 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe218009420 con 0x7fe2300cc3b0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe227fff700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 0x7fe2300acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe22effd700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c810 tx=0x7fe220007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220005b30 con 0x7fe2300cc3b0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe220005c90 con 0x7fe2300cc3b0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220007300 con 0x7fe2300cc3b0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe230157800 con 0x7fe2300cc3b0 2023-12-19T14:55:04.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.454+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2300ad4f0 con 0x7fe2300cc3b0 2023-12-19T14:55:04.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.458+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2300ad770 con 0x7fe2300cc3b0 2023-12-19T14:55:04.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.458+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe22001b3b0 con 0x7fe2300cc3b0 2023-12-19T14:55:04.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.458+0000 7fe226ffd700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 0x7fe210062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:04.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.458+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe22007c090 con 0x7fe2300cc3b0 2023-12-19T14:55:04.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.458+0000 7fe227fff700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 0x7fe210062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:04.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.462+0000 7fe227fff700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 0x7fe210062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2180093f0 tx=0x7fe218017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:04.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.470+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe22004c370 con 0x7fe2300cc3b0 2023-12-19T14:55:04.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.686+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe230071120 con 0x7fe2300cc3b0 2023-12-19T14:55:04.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.686+0000 7fe226ffd700 1 -- 172.21.15.50:0/2565502426 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fe220001d60 con 0x7fe2300cc3b0 2023-12-19T14:55:04.690 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:04.691 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:04.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 msgr2=0x7fe210062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 0x7fe210062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2180093f0 tx=0x7fe218017000 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 msgr2=0x7fe2301582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c810 tx=0x7fe220007df0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 shutdown_connections 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe230157060 0x7fe230165fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe210060670 0x7fe210062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe230156c90 0x7fe2300acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 --2- 172.21.15.50:0/2565502426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2300cc3b0 0x7fe2301582e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 >> 172.21.15.50:0/2565502426 conn(0x7fe23006c7c0 msgr2=0x7fe2300c4520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 shutdown_connections 2023-12-19T14:55:04.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:04.694+0000 7fe2358ae700 1 -- 172.21.15.50:0/2565502426 wait complete. 2023-12-19T14:55:04.698 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:05.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:54:58.259130+0000', 'modified': '2023-12-19T14:55:01.972807+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6936}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6936': {'gid': 6936, 'name': 'd', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6836/2756457211', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6834', 'nonce': 2756457211}, {'type': 'v1', 'addr': '172.21.15.125:6836', 'nonce': 2756457211}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:55:05.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:55:05.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-19T14:55:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.282+0000 7fc816785700 1 Processor -- start 2023-12-19T14:55:05.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.286+0000 7fc816785700 1 -- start start 2023-12-19T14:55:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.286+0000 7fc816785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc81006dc00 0x7fc81006b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.286+0000 7fc816785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81006b0d0 con 0x7fc810158cc0 2023-12-19T14:55:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81006be90 con 0x7fc81006dc00 2023-12-19T14:55:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100b4790 con 0x7fc81006baa0 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc81006dc00 0x7fc81006b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc81006dc00 0x7fc81006b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54236/0 (socket says 172.21.15.50:54236) 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 -- 172.21.15.50:0/1948636818 learned_addr learned my addr 172.21.15.50:0/1948636818 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc80ffff700 1 --1- 172.21.15.50:0/1948636818 >> v1:172.21.15.50:6789/0 conn(0x7fc810158cc0 0x7fc81006d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:35846/0 (socket says 172.21.15.50:35846) 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc814d22700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 -- 172.21.15.50:0/1948636818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 msgr2=0x7fc8100b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 -- 172.21.15.50:0/1948636818 >> v1:172.21.15.50:6789/0 conn(0x7fc810158cc0 legacy=0x7fc81006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 -- 172.21.15.50:0/1948636818 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81006a7e0 con 0x7fc81006dc00 2023-12-19T14:55:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc807fff700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc81006dc00 0x7fc81006b560 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80099b0 tx=0x7fc7f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9d0632fd914c6a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc80f7fe700 1 -- 172.21.15.50:0/1948636818 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800b860 con 0x7fc81006dc00 2023-12-19T14:55:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc80f7fe700 1 -- 172.21.15.50:0/1948636818 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f800b9c0 con 0x7fc81006dc00 2023-12-19T14:55:05.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc80f7fe700 1 -- 172.21.15.50:0/1948636818 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800bd60 con 0x7fc81006dc00 2023-12-19T14:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- 172.21.15.50:0/1948636818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc81006dc00 msgr2=0x7fc81006b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc81006dc00 0x7fc81006b560 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80099b0 tx=0x7fc7f8009250 comp rx=0 tx=0).stop 2023-12-19T14:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- 172.21.15.50:0/1948636818 shutdown_connections 2023-12-19T14:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 --2- 172.21.15.50:0/1948636818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc81006dc00 0x7fc81006b560 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- 172.21.15.50:0/1948636818 >> 172.21.15.50:0/1948636818 conn(0x7fc810067bb0 msgr2=0x7fc8100cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- 172.21.15.50:0/1948636818 shutdown_connections 2023-12-19T14:55:05.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.290+0000 7fc816785700 1 -- 172.21.15.50:0/1948636818 wait complete. 2023-12-19T14:55:05.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.294+0000 7fc816785700 1 Processor -- start 2023-12-19T14:55:05.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.294+0000 7fc816785700 1 -- start start 2023-12-19T14:55:05.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 0x7fc810169370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100af150 con 0x7fc810158cc0 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100af430 con 0x7fc81006dc00 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8100af710 con 0x7fc81006baa0 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54244/0 (socket says 172.21.15.50:54244) 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 -- 172.21.15.50:0/2517629076 learned_addr learned my addr 172.21.15.50:0/2517629076 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc807fff700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 0x7fc810169370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc80ffff700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 -- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 msgr2=0x7fc8100b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 -- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 msgr2=0x7fc810169370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 0x7fc810169370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 -- 172.21.15.50:0/2517629076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80094f0 con 0x7fc810158cc0 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc807fff700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 0x7fc810169370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc814d22700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc808007f20 tx=0x7fc808009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc80cff9700 1 -- 172.21.15.50:0/2517629076 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80800eca0 con 0x7fc810158cc0 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc80cff9700 1 -- 172.21.15.50:0/2517629076 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80800ee20 con 0x7fc810158cc0 2023-12-19T14:55:05.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc80cff9700 1 -- 172.21.15.50:0/2517629076 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808014670 con 0x7fc810158cc0 2023-12-19T14:55:05.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8100af9f0 con 0x7fc810158cc0 2023-12-19T14:55:05.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.298+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8100b01f0 con 0x7fc810158cc0 2023-12-19T14:55:05.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.302+0000 7fc80cff9700 1 -- 172.21.15.50:0/2517629076 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc8080149e0 con 0x7fc810158cc0 2023-12-19T14:55:05.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.302+0000 7fc80cff9700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 0x7fc7f0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc80cff9700 1 -- 172.21.15.50:0/2517629076 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc80807d270 con 0x7fc810158cc0 2023-12-19T14:55:05.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc80ffff700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 0x7fc7f0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 Processor -- start 2023-12-19T14:55:05.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 -- start start 2023-12-19T14:55:05.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc80ffff700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 0x7fc7f0062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc800007fd0 tx=0x7fc800007d70 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27c2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x26e7f30 con 0x253c710 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x26e3ed0 con 0x253cae0 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x26aa0b0 con 0x2587790 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54260/0 (socket says 172.21.15.50:54260) 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 -- 172.21.15.50:0/4020118183 learned_addr learned my addr 172.21.15.50:0/4020118183 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc80ffff700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc814d22700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 -- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 msgr2=0x27c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 -- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 msgr2=0x27c2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27c2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc807fff700 1 -- 172.21.15.50:0/4020118183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25889e0 con 0x253c710 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.306+0000 7fc80ffff700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc807fff700 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc81006eb60 tx=0x7fc7f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3023738b8e607d80 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc8057fa700 1 -- 172.21.15.50:0/4020118183 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8020070 con 0x253c710 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc8057fa700 1 -- 172.21.15.50:0/4020118183 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f8000d10 con 0x253c710 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc8057fa700 1 -- 172.21.15.50:0/4020118183 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800bd60 con 0x253c710 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- 172.21.15.50:0/4020118183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 msgr2=0x27c2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc81006eb60 tx=0x7fc7f8017000 comp rx=0 tx=0).stop 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- 172.21.15.50:0/4020118183 shutdown_connections 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253c710 0x27c2c90 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 --2- 172.21.15.50:0/4020118183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- 172.21.15.50:0/4020118183 >> 172.21.15.50:0/4020118183 conn(0x253ceb0 msgr2=0x2776930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- 172.21.15.50:0/4020118183 shutdown_connections 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- 172.21.15.50:0/4020118183 wait complete. 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 -- start start 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.310+0000 7fc818a57740 1 Processor -- start 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x279e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 0x2780e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x26a9d50 con 0x2587790 2023-12-19T14:55:05.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2682970 con 0x253c710 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x26ea400 con 0x253cae0 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 10 client.0 _ll_register_callbacks cb 0x272d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54262/0 (socket says 172.21.15.50:54262) 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 -- 172.21.15.50:0/3470694372 learned_addr learned my addr 172.21.15.50:0/3470694372 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc80ffff700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x279e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc807fff700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 0x2780e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 -- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 msgr2=0x2780e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 0x2780e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 -- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 msgr2=0x279e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x279e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 -- 172.21.15.50:0/3470694372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81006a7e0 con 0x2587790 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc807fff700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 0x2780e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:05.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc814d22700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc80800e750 tx=0x7fc80807d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fd3d65fd9f99b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc808012e00 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc80804f5b0 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2682cc0 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc80806b410 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 10 client.7068 fetch_fsmap learned FSMap version 41 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x26b2370 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2682cc0 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 41) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc8080514b0 con 0x2587790 2023-12-19T14:55:05.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 10 client.7068 fetch_fsmap finished waiting for FSMap version 41 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 10 client.7068 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 20 client.7068 populate_metadata read hostname 'smithi050' 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x2782100 0x27d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 4 client.7068 mds_command: new command op to 6936 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 --> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x26863e0 con 0x2782100 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.314+0000 7fc80ffff700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x2782100 0x27d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc80ffff700 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x2782100 0x27d1ae0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e821e1b4707242c server_cookie=dd99911023a2d411 in_seq=0 out_seq=0 2023-12-19T14:55:05.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc7ee7fc700 10 client.7068 ms_handle_connect on v2:172.21.15.125:6834/2756457211 2023-12-19T14:55:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/3470694372 <== mds.0 v2:172.21.15.125:6834/2756457211 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x26863e0 con 0x2782100 2023-12-19T14:55:05.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc7ee7fc700 10 client.7068 handle_command_reply: tid=0 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 client.7068 shutdown 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 msgr2=0x2781370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc80800e750 tx=0x7fc80807d040 comp rx=0 tx=0).stop 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 shutdown_connections 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x2782100 0x27d1ae0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x2781370 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253cae0 0x2780e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc7ee7fc700 0 client.7068 ms_handle_reset on v2:172.21.15.125:6834/2756457211 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 --2- 172.21.15.50:0/3470694372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x279e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 >> 172.21.15.50:0/3470694372 conn(0x253ceb0 msgr2=0x276ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 shutdown_connections 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.318+0000 7fc818a57740 1 -- 172.21.15.50:0/3470694372 wait complete. 2023-12-19T14:55:05.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.322+0000 7fc818a57740 20 client.7068 trim_cache size 0 max 16384 2023-12-19T14:55:05.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.338+0000 7fc818a57740 1 Processor -- start 2023-12-19T14:55:05.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.338+0000 7fc818a57740 1 -- start start 2023-12-19T14:55:05.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x277fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x26e79d0 con 0x253cae0 2023-12-19T14:55:05.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x26b9550 con 0x253c710 2023-12-19T14:55:05.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x26b8610 con 0x2587790 2023-12-19T14:55:05.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54278/0 (socket says 172.21.15.50:54278) 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 -- 172.21.15.50:0/4045622904 learned_addr learned my addr 172.21.15.50:0/4045622904 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc80ffff700 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x277fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc814d22700 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 -- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 msgr2=0x27756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 -- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 msgr2=0x277fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x277fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 -- 172.21.15.50:0/4045622904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2698d80 con 0x253cae0 2023-12-19T14:55:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc807fff700 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009290 tx=0x7fc7f801d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72990ab4f0b9751d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc805ffb700 1 -- 172.21.15.50:0/4045622904 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8020040 con 0x253cae0 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc805ffb700 1 -- 172.21.15.50:0/4045622904 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f800b720 con 0x253cae0 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.342+0000 7fc805ffb700 1 -- 172.21.15.50:0/4045622904 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800bd80 con 0x253cae0 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- 172.21.15.50:0/4045622904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 msgr2=0x27803a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009290 tx=0x7fc7f801d070 comp rx=0 tx=0).stop 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- 172.21.15.50:0/4045622904 shutdown_connections 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2587790 0x27756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x253cae0 0x27803a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- 172.21.15.50:0/4045622904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253c710 0x277fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- 172.21.15.50:0/4045622904 >> 172.21.15.50:0/4045622904 conn(0x253ceb0 msgr2=0x2780fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- 172.21.15.50:0/4045622904 shutdown_connections 2023-12-19T14:55:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- 172.21.15.50:0/4045622904 wait complete. 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- start start 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 Processor -- start 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 0x281b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27be200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x26e6590 con 0x2587790 2023-12-19T14:55:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x26dfb50 con 0x253cae0 2023-12-19T14:55:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x26e8ef0 con 0x253c710 2023-12-19T14:55:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.346+0000 7fc818a57740 10 client.0 _ll_register_callbacks cb 0x25089d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 0x281b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc807fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27be200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54292/0 (socket says 172.21.15.50:54292) 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 0x281b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:45860/0 (socket says 172.21.15.50:45860) 2023-12-19T14:55:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 -- 172.21.15.50:0/2744594639 learned_addr learned my addr 172.21.15.50:0/2744594639 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 -- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 msgr2=0x281b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 0x281b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 -- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 msgr2=0x27be200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27be200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 -- 172.21.15.50:0/2744594639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x25889e0 con 0x2587790 2023-12-19T14:55:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc814d22700 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc80800efd0 tx=0x7fc808066000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf21b45c9f0cdd31 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:55:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80804f5b0 con 0x2587790 2023-12-19T14:55:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fc80806b410 con 0x2587790 2023-12-19T14:55:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7f8000d10 con 0x2587790 2023-12-19T14:55:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc80804d550 con 0x2587790 2023-12-19T14:55:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 10 client.7074 fetch_fsmap learned FSMap version 41 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x268d130 con 0x2587790 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f8000d10 con 0x2587790 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 41) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc8080514b0 con 0x2587790 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 10 client.7074 fetch_fsmap finished waiting for FSMap version 41 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 10 client.7074 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T14:55:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 20 client.7074 populate_metadata read hostname 'smithi050' 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x27bec80 0x27bf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 4 client.7074 mds_command: new command op to 6936 tid=0[{"prefix": "status"}] 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 --> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] -- command(tid 0: {"prefix": "status"}) v1 -- 0x26b8410 con 0x27bec80 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.350+0000 7fc80ffff700 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x27bec80 0x27bf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc7ee7fc700 10 client.7074 ms_handle_connect on v2:172.21.15.125:6834/2756457211 2023-12-19T14:55:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc80ffff700 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x27bec80 0x27bf050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b75422c70c4515e server_cookie=b27616d103d24fa9 in_seq=0 out_seq=0 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc7ee7fc700 1 -- 172.21.15.50:0/2744594639 <== mds.0 v2:172.21.15.125:6834/2756457211 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x26b8410 con 0x27bec80 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc7ee7fc700 10 client.7074 handle_command_reply: tid=0 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 client.7074 shutdown 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 msgr2=0x27be740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc80800efd0 tx=0x7fc808066000 comp rx=0 tx=0).stop 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 shutdown_connections 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:6834/2756457211,v1:172.21.15.125:6836/2756457211] conn(0x27bec80 0x27bf050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x253cae0 0x27be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x2587790 0x27be740 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc7ee7fc700 0 client.7074 ms_handle_reset on v2:172.21.15.125:6834/2756457211 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 --2- 172.21.15.50:0/2744594639 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x253c710 0x281b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 >> 172.21.15.50:0/2744594639 conn(0x253ceb0 msgr2=0x276f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 shutdown_connections 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 1 -- 172.21.15.50:0/2744594639 wait complete. 2023-12-19T14:55:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc818a57740 20 client.7074 trim_cache size 0 max 16384 2023-12-19T14:55:05.363 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:55:05.363 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:55:05.363 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:55:05.363 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 6936, 2023-12-19T14:55:05.363 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.405382007, 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 41, 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 100, 2023-12-19T14:55:05.364 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 100, 2023-12-19T14:55:05.365 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.450589597 2023-12-19T14:55:05.365 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:55:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 msgr2=0x7fc7f0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 0x7fc7f0062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc800007fd0 tx=0x7fc800007d70 comp rx=0 tx=0).stop 2023-12-19T14:55:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 msgr2=0x7fc810155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc808007f20 tx=0x7fc808009860 comp rx=0 tx=0).stop 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 shutdown_connections 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc81006dc00 0x7fc810169370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7f0060670 0x7fc7f0062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc810158cc0 0x7fc810155720 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 --2- 172.21.15.50:0/2517629076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc81006baa0 0x7fc8100b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 >> 172.21.15.50:0/2517629076 conn(0x7fc810067bb0 msgr2=0x7fc81015b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 shutdown_connections 2023-12-19T14:55:05.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.354+0000 7fc816785700 1 -- 172.21.15.50:0/2517629076 wait complete. 2023-12-19T14:55:05.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:55:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.550+0000 7f0095f85700 1 Processor -- start 2023-12-19T14:55:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.550+0000 7f0095f85700 1 -- start start 2023-12-19T14:55:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.550+0000 7f0095f85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f009006de30 0x7f00900cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.550+0000 7f0095f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00900cc5e0 0x7f0090162da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009006d9a0 con 0x7f00900c8ad0 2023-12-19T14:55:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900cce80 con 0x7f00900cc5e0 2023-12-19T14:55:05.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900ad250 con 0x7f009006de30 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00900cc5e0 0x7f0090162da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00900cc5e0 0x7f0090162da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54300/0 (socket says 172.21.15.50:54300) 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 -- 172.21.15.50:0/2492453242 learned_addr learned my addr 172.21.15.50:0/2492453242 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0086ffd700 1 --2- 172.21.15.50:0/2492453242 >> v2:172.21.15.125:3300/0 conn(0x7f009006de30 0x7f00900cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 -- 172.21.15.50:0/2492453242 >> v2:172.21.15.125:3300/0 conn(0x7f009006de30 msgr2=0x7f00900cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 --2- 172.21.15.50:0/2492453242 >> v2:172.21.15.125:3300/0 conn(0x7f009006de30 0x7f00900cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 -- 172.21.15.50:0/2492453242 >> v1:172.21.15.125:6789/0 conn(0x7f00900c8ad0 legacy=0x7f00900c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 -- 172.21.15.50:0/2492453242 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900ccba0 con 0x7f00900cc5e0 2023-12-19T14:55:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008ffff700 1 --2- 172.21.15.50:0/2492453242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00900cc5e0 0x7f0090162da0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f00900b4000 tx=0x7f0088009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ea4dcf718d0a2e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008effd700 1 -- 172.21.15.50:0/2492453242 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008800e070 con 0x7f00900cc5e0 2023-12-19T14:55:05.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008effd700 1 -- 172.21.15.50:0/2492453242 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008800ba20 con 0x7f00900cc5e0 2023-12-19T14:55:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f008effd700 1 -- 172.21.15.50:0/2492453242 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00880092e0 con 0x7f00900cc5e0 2023-12-19T14:55:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- 172.21.15.50:0/2492453242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00900cc5e0 msgr2=0x7f0090162da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 --2- 172.21.15.50:0/2492453242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00900cc5e0 0x7f0090162da0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f00900b4000 tx=0x7f0088009840 comp rx=0 tx=0).stop 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- 172.21.15.50:0/2492453242 shutdown_connections 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 --2- 172.21.15.50:0/2492453242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00900cc5e0 0x7f0090162da0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 --2- 172.21.15.50:0/2492453242 >> v2:172.21.15.125:3300/0 conn(0x7f009006de30 0x7f00900cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.554+0000 7f0095f85700 1 -- 172.21.15.50:0/2492453242 >> 172.21.15.50:0/2492453242 conn(0x7f00900ce000 msgr2=0x7f00900ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.558+0000 7f0095f85700 1 -- 172.21.15.50:0/2492453242 shutdown_connections 2023-12-19T14:55:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.558+0000 7f0095f85700 1 -- 172.21.15.50:0/2492453242 wait complete. 2023-12-19T14:55:05.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.558+0000 7f0095f85700 1 Processor -- start 2023-12-19T14:55:05.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 -- start start 2023-12-19T14:55:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00900c8ad0 0x7f00900b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00900cc5e0 0x7f00901691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900b14d0 con 0x7f009006de30 2023-12-19T14:55:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900b2010 con 0x7f00900c8ad0 2023-12-19T14:55:05.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900afef0 con 0x7f00900cc5e0 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54306/0 (socket says 172.21.15.50:54306) 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 -- 172.21.15.50:0/1526107450 learned_addr learned my addr 172.21.15.50:0/1526107450 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0086ffd700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00900c8ad0 0x7f00900b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008ffff700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00900cc5e0 0x7f00901691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 -- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00900cc5e0 msgr2=0x7f00901691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00900cc5e0 0x7f00901691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 -- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00900c8ad0 msgr2=0x7f00900b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00900c8ad0 0x7f00900b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 -- 172.21.15.50:0/1526107450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0088009ae0 con 0x7f009006de30 2023-12-19T14:55:05.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f008f7fe700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f008000c5c0 tx=0x7f008000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080011590 con 0x7f009006de30 2023-12-19T14:55:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.562+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00900b16e0 con 0x7f009006de30 2023-12-19T14:55:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.566+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0080011710 con 0x7f009006de30 2023-12-19T14:55:05.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.566+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080011ad0 con 0x7f009006de30 2023-12-19T14:55:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.566+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00900b1ca0 con 0x7f009006de30 2023-12-19T14:55:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.566+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0090071120 con 0x7f009006de30 2023-12-19T14:55:05.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.566+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f008001b3b0 con 0x7f009006de30 2023-12-19T14:55:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.570+0000 7f0087fff700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 0x7f0070062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.570+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f008007c060 con 0x7f009006de30 2023-12-19T14:55:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.570+0000 7f0086ffd700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 0x7f0070062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:05.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.570+0000 7f0086ffd700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 0x7f0070062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00900c4690 tx=0x7f00780091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:05.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.578+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00800abbe0 con 0x7f009006de30 2023-12-19T14:55:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.782+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f00900690d0 con 0x7f009006de30 2023-12-19T14:55:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.782+0000 7f0087fff700 1 -- 172.21.15.50:0/1526107450 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0080017020 con 0x7f009006de30 2023-12-19T14:55:05.785 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 msgr2=0x7f0070062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 0x7f0070062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00900c4690 tx=0x7f00780091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 msgr2=0x7f00900af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f008000c5c0 tx=0x7f008000c8d0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 shutdown_connections 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00900c8ad0 0x7f00900b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0070060670 0x7f0070062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00900cc5e0 0x7f00901691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 --2- 172.21.15.50:0/1526107450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f009006de30 0x7f00900af9b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:05.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 >> 172.21.15.50:0/1526107450 conn(0x7f00900ce000 msgr2=0x7f0090158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 shutdown_connections 2023-12-19T14:55:05.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:05.786+0000 7f0095f85700 1 -- 172.21.15.50:0/1526107450 wait complete. 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:55:05.812 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:05.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:05.813 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:55:05.829 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:55:05.830 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41444sec preferred_lft 41444sec 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:55:05.831 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:55:05.832 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:05.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:05.833 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:55:05.914 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:55:05.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:05.917 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:55:05.978 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:55:05.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:05.979 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:55:05.979 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:55:05.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:55:05.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:55:05.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:55:05.980 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:55:06.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:06.122 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:55:06.139 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:55:06.139 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:55:06.140 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:06.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:06.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:55:06.221 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:55:06.221 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:55:06.221 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:06.286 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:06.286 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:55:06.286 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:06.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:55:06.434 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:55:06.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:55:06.513 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:55:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:55:06.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:06.516 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:55:06.580 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:55:06.581 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:55:06.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:55:06.671 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:55:06.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:55:06.753 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:55:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:55:06.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:06.755 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:55:06.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:55:06.762+0000 7f02575cd380 -1 init, newargv = 0x560213fb3490 newargc=15 2023-12-19T14:55:06.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[49425]: starting ceph client 2023-12-19T14:55:06.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[49425]: starting fuse 2023-12-19T14:55:07.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:55:07.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:55:07.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:55:07.922 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:55:07.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:55:07.997 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:55:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:55:08.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:08.001 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:55:08.068 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:55:08.069 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:55:08.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:55:08.069 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:55:08.069 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:55:08.069 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:55:08.070 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:55:08.071 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:55:08.236 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.49425.asok 2023-12-19T14:55:08.421 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49425.asok 2023-12-19T14:55:08.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:08.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49425.asok status 2023-12-19T14:55:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:55:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:55:08.609 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "49425", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 7089, 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:55:08.610 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7089 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 173743702 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 173743702 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.7089 192.168.0.1:0/173743702", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/173743702", 2023-12-19T14:55:08.611 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 41, 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 100, 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:55:08.612 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:55:08.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:08.623 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:55:08.623 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:08.641 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:55:08.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:08.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:08.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:08.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:55:08.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.934+0000 7f46e96ae700 1 Processor -- start 2023-12-19T14:55:08.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.934+0000 7f46e96ae700 1 -- start start 2023-12-19T14:55:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e96ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46e4156d10 0x7f46e40cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e96ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46e406c7c0 0x7f46e406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e96ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e40d0880 con 0x7f46e4156940 2023-12-19T14:55:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e96ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e406d1a0 con 0x7f46e406c7c0 2023-12-19T14:55:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e96ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e40cfc70 con 0x7f46e4156d10 2023-12-19T14:55:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46e406c7c0 0x7f46e406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46e406c7c0 0x7f46e406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43942/0 (socket says 172.21.15.50:43942) 2023-12-19T14:55:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 -- 172.21.15.50:0/1037187772 learned_addr learned my addr 172.21.15.50:0/1037187772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e27fc700 1 --2- 172.21.15.50:0/1037187772 >> v2:172.21.15.125:3300/0 conn(0x7f46e4156d10 0x7f46e40cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 -- 172.21.15.50:0/1037187772 >> v2:172.21.15.125:3300/0 conn(0x7f46e4156d10 msgr2=0x7f46e40cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 --2- 172.21.15.50:0/1037187772 >> v2:172.21.15.125:3300/0 conn(0x7f46e4156d10 0x7f46e40cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 -- 172.21.15.50:0/1037187772 >> v1:172.21.15.125:6789/0 conn(0x7f46e4156940 legacy=0x7f46e4159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.938+0000 7f46e3fff700 1 -- 172.21.15.50:0/1037187772 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e40cff50 con 0x7f46e406c7c0 2023-12-19T14:55:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e3fff700 1 --2- 172.21.15.50:0/1037187772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e406cc60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f46e40639b0 tx=0x7f46dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e104fa1e9773ae8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e1ffb700 1 -- 172.21.15.50:0/1037187772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc00e070 con 0x7f46e406c7c0 2023-12-19T14:55:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e1ffb700 1 -- 172.21.15.50:0/1037187772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc00ba20 con 0x7f46e406c7c0 2023-12-19T14:55:08.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e1ffb700 1 -- 172.21.15.50:0/1037187772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0092e0 con 0x7f46e406c7c0 2023-12-19T14:55:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 -- 172.21.15.50:0/1037187772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 msgr2=0x7f46e406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:08.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 --2- 172.21.15.50:0/1037187772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e406cc60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f46e40639b0 tx=0x7f46dc009840 comp rx=0 tx=0).stop 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 -- 172.21.15.50:0/1037187772 shutdown_connections 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 --2- 172.21.15.50:0/1037187772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e406cc60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 --2- 172.21.15.50:0/1037187772 >> v2:172.21.15.125:3300/0 conn(0x7f46e4156d10 0x7f46e40cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 -- 172.21.15.50:0/1037187772 >> 172.21.15.50:0/1037187772 conn(0x7f46e40ce320 msgr2=0x7f46e40ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 -- 172.21.15.50:0/1037187772 shutdown_connections 2023-12-19T14:55:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.942+0000 7f46e96ae700 1 -- 172.21.15.50:0/1037187772 wait complete. 2023-12-19T14:55:08.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.946+0000 7f46e96ae700 1 Processor -- start 2023-12-19T14:55:08.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.946+0000 7f46e96ae700 1 -- start start 2023-12-19T14:55:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.946+0000 7f46e96ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.946+0000 7f46e96ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46e4156940 0x7f46e4151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e4156d10 0x7f46e4152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e4156060 con 0x7f46e406c7c0 2023-12-19T14:55:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e40c9b00 con 0x7f46e4156d10 2023-12-19T14:55:08.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e40c91d0 con 0x7f46e4156940 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43958/0 (socket says 172.21.15.50:43958) 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 -- 172.21.15.50:0/177715961 learned_addr learned my addr 172.21.15.50:0/177715961 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e27fc700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46e4156940 0x7f46e4151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e3fff700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e4156d10 0x7f46e4152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 -- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46e4156940 msgr2=0x7f46e4151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46e4156940 0x7f46e4151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 -- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e4156d10 msgr2=0x7f46e4152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e4156d10 0x7f46e4152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 -- 172.21.15.50:0/177715961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc009ae0 con 0x7f46e406c7c0 2023-12-19T14:55:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e2ffd700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f46d400c5c0 tx=0x7f46d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4011590 con 0x7f46e406c7c0 2023-12-19T14:55:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d4011710 con 0x7f46e406c7c0 2023-12-19T14:55:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4011ad0 con 0x7f46e406c7c0 2023-12-19T14:55:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e40c94b0 con 0x7f46e406c7c0 2023-12-19T14:55:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e413ef10 con 0x7f46e406c7c0 2023-12-19T14:55:08.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.950+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e4071120 con 0x7f46e406c7c0 2023-12-19T14:55:08.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.954+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f46d401b3b0 con 0x7f46e406c7c0 2023-12-19T14:55:08.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.954+0000 7f46caffd700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 0x7f46d0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:08.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.954+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f46d407bff0 con 0x7f46e406c7c0 2023-12-19T14:55:08.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.954+0000 7f46e27fc700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 0x7f46d0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:08.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.958+0000 7f46e27fc700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 0x7f46d0062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46e40cbe20 tx=0x7f46d80091a0 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:08.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:08.962+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f46d404c370 con 0x7f46e406c7c0 2023-12-19T14:55:09.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.154+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f46e40cda50 con 0x7f46d00606a0 2023-12-19T14:55:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46caffd700 1 -- 172.21.15.50:0/177715961 <== mgr.6089 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f46e40cda50 con 0x7f46d00606a0 2023-12-19T14:55:09.159 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:55:09.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:09.159 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:55:09.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:09.160 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 msgr2=0x7f46d0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 0x7f46d0062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46e40cbe20 tx=0x7f46d80091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 msgr2=0x7f46e413e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f46d400c5c0 tx=0x7f46d400c8d0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 shutdown_connections 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46e4156d10 0x7f46e4152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46d00606a0 0x7f46d0062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46e4156940 0x7f46e4151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 --2- 172.21.15.50:0/177715961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46e406c7c0 0x7f46e413e9d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.158+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 >> 172.21.15.50:0/177715961 conn(0x7f46e40ce320 msgr2=0x7f46e4067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:09.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.162+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 shutdown_connections 2023-12-19T14:55:09.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.162+0000 7f46e96ae700 1 -- 172.21.15.50:0/177715961 wait complete. 2023-12-19T14:55:09.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:55:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.350+0000 7fe34f00c700 1 Processor -- start 2023-12-19T14:55:09.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.350+0000 7fe34f00c700 1 -- start start 2023-12-19T14:55:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3480ce5c0 0x7fe3480ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe34815d4d0 0x7fe3480caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe348158a00 0x7fe348158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3480caf50 con 0x7fe3480ce5c0 2023-12-19T14:55:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3480cbd90 con 0x7fe348158a00 2023-12-19T14:55:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34f00c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480cb180 con 0x7fe34815d4d0 2023-12-19T14:55:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3480ce5c0 0x7fe3480ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3480ce5c0 0x7fe3480ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43972/0 (socket says 172.21.15.50:43972) 2023-12-19T14:55:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 -- 172.21.15.50:0/3802911944 learned_addr learned my addr 172.21.15.50:0/3802911944 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34d809700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe34815d4d0 0x7fe3480caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e80b700 1 --2- 172.21.15.50:0/3802911944 >> v2:172.21.15.125:3300/0 conn(0x7fe348158a00 0x7fe348158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 -- 172.21.15.50:0/3802911944 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe34815d4d0 msgr2=0x7fe3480caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe34815d4d0 0x7fe3480caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 -- 172.21.15.50:0/3802911944 >> v2:172.21.15.125:3300/0 conn(0x7fe348158a00 msgr2=0x7fe348158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3802911944 >> v2:172.21.15.125:3300/0 conn(0x7fe348158a00 0x7fe348158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 -- 172.21.15.50:0/3802911944 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480cb460 con 0x7fe3480ce5c0 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3480ce5c0 0x7fe3480ce990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe34806bf60 tx=0x7fe344009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab402b8156ef8f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:09.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.354+0000 7fe34d008700 1 -- 172.21.15.50:0/3802911944 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34400b640 con 0x7fe3480ce5c0 2023-12-19T14:55:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34d008700 1 -- 172.21.15.50:0/3802911944 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34400b7a0 con 0x7fe3480ce5c0 2023-12-19T14:55:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34d008700 1 -- 172.21.15.50:0/3802911944 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34400bb60 con 0x7fe3480ce5c0 2023-12-19T14:55:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- 172.21.15.50:0/3802911944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3480ce5c0 msgr2=0x7fe3480ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:09.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3480ce5c0 0x7fe3480ce990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe34806bf60 tx=0x7fe344009250 comp rx=0 tx=0).stop 2023-12-19T14:55:09.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- 172.21.15.50:0/3802911944 shutdown_connections 2023-12-19T14:55:09.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3802911944 >> v2:172.21.15.125:3300/0 conn(0x7fe348158a00 0x7fe348158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe34815d4d0 0x7fe3480caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3802911944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3480ce5c0 0x7fe3480ce990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- 172.21.15.50:0/3802911944 >> 172.21.15.50:0/3802911944 conn(0x7fe348067bb0 msgr2=0x7fe3480cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- 172.21.15.50:0/3802911944 shutdown_connections 2023-12-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- 172.21.15.50:0/3802911944 wait complete. 2023-12-19T14:55:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 Processor -- start 2023-12-19T14:55:09.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.358+0000 7fe34f00c700 1 -- start start 2023-12-19T14:55:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3480ce5c0 0x7fe3480c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 0x7fe348150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480c9fd0 con 0x7fe34815d4d0 2023-12-19T14:55:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480b5c10 con 0x7fe3480ce5c0 2023-12-19T14:55:09.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480b4b70 con 0x7fe348158a00 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43974/0 (socket says 172.21.15.50:43974) 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 -- 172.21.15.50:0/3132414561 learned_addr learned my addr 172.21.15.50:0/3132414561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34d809700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 0x7fe348150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3480ce5c0 0x7fe3480c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 -- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 msgr2=0x7fe348150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 0x7fe348150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 -- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3480ce5c0 msgr2=0x7fe3480c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3480ce5c0 0x7fe3480c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 -- 172.21.15.50:0/3132414561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440094f0 con 0x7fe34815d4d0 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34d809700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 0x7fe348150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34e80b700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe340009e90 tx=0x7fe34000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:09.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000cc20 con 0x7fe34815d4d0 2023-12-19T14:55:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34000f040 con 0x7fe34815d4d0 2023-12-19T14:55:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3480b4e50 con 0x7fe34815d4d0 2023-12-19T14:55:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340014480 con 0x7fe34815d4d0 2023-12-19T14:55:09.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.362+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3480b5410 con 0x7fe34815d4d0 2023-12-19T14:55:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.366+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34804be40 con 0x7fe34815d4d0 2023-12-19T14:55:09.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.366+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 33) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe340014830 con 0x7fe34815d4d0 2023-12-19T14:55:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.366+0000 7fe33e7fc700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 0x7fe32c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.366+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe34007d5a0 con 0x7fe34815d4d0 2023-12-19T14:55:09.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.370+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 0x7fe32c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:09.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.370+0000 7fe34e00a700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 0x7fe32c062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe34806bf60 tx=0x7fe344017000 comp rx=0 tx=0).ready entity=mgr.6089 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.374+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe3400512a0 con 0x7fe34815d4d0 2023-12-19T14:55:09.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:09.590+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe3480cb740 con 0x7fe34815d4d0 2023-12-19T14:55:10.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.042+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 34) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe34007b950 con 0x7fe34815d4d0 2023-12-19T14:55:10.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.046+0000 7fe33e7fc700 1 -- 172.21.15.50:0/3132414561 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v34) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe3400512a0 con 0x7fe34815d4d0 2023-12-19T14:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 msgr2=0x7fe32c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 0x7fe32c062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe34806bf60 tx=0x7fe344017000 comp rx=0 tx=0).stop 2023-12-19T14:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 msgr2=0x7fe3481521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe340009e90 tx=0x7fe34000b040 comp rx=0 tx=0).stop 2023-12-19T14:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 shutdown_connections 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe32c060670 0x7fe32c062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe34815d4d0 0x7fe3481521f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe348158a00 0x7fe348150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 --2- 172.21.15.50:0/3132414561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3480ce5c0 0x7fe3480c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 >> 172.21.15.50:0/3132414561 conn(0x7fe348067bb0 msgr2=0x7fe3480cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 shutdown_connections 2023-12-19T14:55:10.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.050+0000 7fe34f00c700 1 -- 172.21.15.50:0/3132414561 wait complete. 2023-12-19T14:55:10.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:55:10.232 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:10.225+0000 7fa6078eb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 Processor -- start 2023-12-19T14:55:10.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 -- start start 2023-12-19T14:55:10.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37ec0ca320 0x7f37ec15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec0cba00 0x7f37ec0cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0ac8f0 con 0x7f37ec156eb0 2023-12-19T14:55:10.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ec0ca750 con 0x7f37ec0ca320 2023-12-19T14:55:10.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f3ccd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec158290 con 0x7f37ec0cba00 2023-12-19T14:55:10.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f1268700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f37ec0ca320 0x7f37ec15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f1a69700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f37ec156eb0 0x7f37ec0acd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57330/0 (socket says 172.21.15.50:57330) 2023-12-19T14:55:10.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.230+0000 7f37f1a69700 1 -- 172.21.15.50:0/3257539787 learned_addr learned my addr 172.21.15.50:0/3257539787 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f226a700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec0cba00 0x7f37ec0cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f1268700 1 -- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec0cba00 msgr2=0x7f37ec0cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f1268700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec0cba00 0x7f37ec0cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f1268700 1 -- 172.21.15.50:0/3257539787 >> v1:172.21.15.50:6789/0 conn(0x7f37ec156eb0 legacy=0x7f37ec0acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f1268700 1 -- 172.21.15.50:0/3257539787 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec15b120 con 0x7f37ec0ca320 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f0a67700 1 -- 172.21.15.50:0/3257539787 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426749758 0 0) 0x7f37ec0ac8f0 con 0x7f37ec156eb0 2023-12-19T14:55:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f1268700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec15aa20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0098e0 tx=0x7f37dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f49a0f518c3317 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f0a67700 1 -- 172.21.15.50:0/3257539787 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc009e80 con 0x7f37ec0ca320 2023-12-19T14:55:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f0a67700 1 -- 172.21.15.50:0/3257539787 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00b860 con 0x7f37ec0ca320 2023-12-19T14:55:10.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f0a67700 1 -- 172.21.15.50:0/3257539787 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00bbe0 con 0x7f37ec0ca320 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 -- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 msgr2=0x7f37ec15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec15aa20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0098e0 tx=0x7f37dc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 -- 172.21.15.50:0/3257539787 shutdown_connections 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec0cba00 0x7f37ec0cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/3257539787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec15aa20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 -- 172.21.15.50:0/3257539787 >> 172.21.15.50:0/3257539787 conn(0x7f37ec0cd8f0 msgr2=0x7f37ec0c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:10.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 -- 172.21.15.50:0/3257539787 shutdown_connections 2023-12-19T14:55:10.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.234+0000 7f37f3ccd700 1 -- 172.21.15.50:0/3257539787 wait complete. 2023-12-19T14:55:10.240 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:10.238+0000 7f1b77b86000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:10.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.238+0000 7f37f3ccd700 1 Processor -- start 2023-12-19T14:55:10.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.238+0000 7f37f3ccd700 1 -- start start 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec0c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec156eb0 0x7f37ec150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec16e360 con 0x7f37ec0cba00 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec16e640 con 0x7f37ec0ca320 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec16e920 con 0x7f37ec156eb0 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43984/0 (socket says 172.21.15.50:43984) 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 -- 172.21.15.50:0/637488311 learned_addr learned my addr 172.21.15.50:0/637488311 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f226a700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec156eb0 0x7f37ec150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1a69700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec0c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 -- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec156eb0 msgr2=0x7f37ec150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec156eb0 0x7f37ec150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 -- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 msgr2=0x7f37ec0c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec0c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 -- 172.21.15.50:0/637488311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc009420 con 0x7f37ec0cba00 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f1268700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f37ec06ae00 tx=0x7f37dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00b9d0 con 0x7f37ec0cba00 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00bb50 con 0x7f37ec0cba00 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc011670 con 0x7f37ec0cba00 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec068390 con 0x7f37ec0cba00 2023-12-19T14:55:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.242+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec068040 con 0x7f37ec0cba00 2023-12-19T14:55:10.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.246+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec0685a0 con 0x7f37ec0cba00 2023-12-19T14:55:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.246+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 34) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f37dc0119e0 con 0x7f37ec0cba00 2023-12-19T14:55:10.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.250+0000 7f37e1ffb700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 0x7f37d0062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.250+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f37dc07df50 con 0x7f37ec0cba00 2023-12-19T14:55:10.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.250+0000 7f37f1a69700 1 -- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 msgr2=0x7f37d0062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:10.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.250+0000 7f37f1a69700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 0x7f37d0062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:10.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.254+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f37dc04e2b0 con 0x7f37ec0cba00 2023-12-19T14:55:10.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.426+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f37ec071120 con 0x7f37ec0cba00 2023-12-19T14:55:10.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.434+0000 7f37e1ffb700 1 -- 172.21.15.50:0/637488311 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f37dc0518d0 con 0x7f37ec0cba00 2023-12-19T14:55:10.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 msgr2=0x7f37d0062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 0x7f37d0062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 msgr2=0x7f37ec16db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f37ec06ae00 tx=0x7f37dc017000 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 shutdown_connections 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f37d00606f0 0x7f37d0062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f37ec156eb0 0x7f37ec150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37ec0cba00 0x7f37ec16db10 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 --2- 172.21.15.50:0/637488311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37ec0ca320 0x7f37ec0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 >> 172.21.15.50:0/637488311 conn(0x7f37ec0cd8f0 msgr2=0x7f37ec0c1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 shutdown_connections 2023-12-19T14:55:10.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.438+0000 7f37f3ccd700 1 -- 172.21.15.50:0/637488311 wait complete. 2023-12-19T14:55:10.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:55:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 Processor -- start 2023-12-19T14:55:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 -- start start 2023-12-19T14:55:10.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc900c69a0 0x7fcc900c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc9006de00 0x7fcc9006b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9006d970 con 0x7fcc9006c7c0 2023-12-19T14:55:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9006cd60 con 0x7fcc9006de00 2023-12-19T14:55:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.634+0000 7fcc96fcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9006d040 con 0x7fcc900c69a0 2023-12-19T14:55:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc9006de00 0x7fcc9006b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc967cb700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc9006c7c0 0x7fcc9015cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:51310/0 (socket says 172.21.15.50:51310) 2023-12-19T14:55:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc967cb700 1 -- 172.21.15.50:0/4175010454 learned_addr learned my addr 172.21.15.50:0/4175010454 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc95fca700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc900c69a0 0x7fcc900c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 -- 172.21.15.50:0/4175010454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc900c69a0 msgr2=0x7fcc900c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc900c69a0 0x7fcc900c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 -- 172.21.15.50:0/4175010454 >> v1:172.21.15.125:6789/0 conn(0x7fcc9006c7c0 legacy=0x7fcc9015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 -- 172.21.15.50:0/4175010454 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9006d320 con 0x7fcc9006de00 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc957c9700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc9006de00 0x7fcc9006b880 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009910 tx=0x7fcc800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d109d31a73e0fc12 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc94fc8700 1 -- 172.21.15.50:0/4175010454 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000b640 con 0x7fcc9006de00 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc94fc8700 1 -- 172.21.15.50:0/4175010454 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8000b7a0 con 0x7fcc9006de00 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc94fc8700 1 -- 172.21.15.50:0/4175010454 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000bb60 con 0x7fcc9006de00 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 -- 172.21.15.50:0/4175010454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc9006de00 msgr2=0x7fcc9006b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc9006de00 0x7fcc9006b880 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009910 tx=0x7fcc800091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 -- 172.21.15.50:0/4175010454 shutdown_connections 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc9006de00 0x7fcc9006b880 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/4175010454 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc900c69a0 0x7fcc900c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 -- 172.21.15.50:0/4175010454 >> 172.21.15.50:0/4175010454 conn(0x7fcc90067b50 msgr2=0x7fcc900cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 -- 172.21.15.50:0/4175010454 shutdown_connections 2023-12-19T14:55:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.638+0000 7fcc96fcc700 1 -- 172.21.15.50:0/4175010454 wait complete. 2023-12-19T14:55:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.642+0000 7fcc96fcc700 1 Processor -- start 2023-12-19T14:55:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.642+0000 7fcc96fcc700 1 -- start start 2023-12-19T14:55:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc9006c7c0 0x7fcc9015f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9006de00 0x7fcc9014abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9016d520 con 0x7fcc900c69a0 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc901718f0 con 0x7fcc9006de00 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9016d7e0 con 0x7fcc9006c7c0 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44000/0 (socket says 172.21.15.50:44000) 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 -- 172.21.15.50:0/2041491184 learned_addr learned my addr 172.21.15.50:0/2041491184 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc95fca700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc9006c7c0 0x7fcc9015f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc957c9700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9006de00 0x7fcc9014abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 -- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc9006c7c0 msgr2=0x7fcc9015f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc9006c7c0 0x7fcc9015f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 -- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9006de00 msgr2=0x7fcc9014abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9006de00 0x7fcc9014abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 -- 172.21.15.50:0/2041491184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80009420 con 0x7fcc900c69a0 2023-12-19T14:55:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc967cb700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800cc40 tx=0x7fcc88001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8800a8b0 con 0x7fcc900c69a0 2023-12-19T14:55:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8800aa10 con 0x7fcc900c69a0 2023-12-19T14:55:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8800add0 con 0x7fcc900c69a0 2023-12-19T14:55:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9016dac0 con 0x7fcc900c69a0 2023-12-19T14:55:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.646+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9014b110 con 0x7fcc900c69a0 2023-12-19T14:55:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.650+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9016dfc0 con 0x7fcc900c69a0 2023-12-19T14:55:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.650+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 34) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fcc88007300 con 0x7fcc900c69a0 2023-12-19T14:55:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.650+0000 7fcc85ffb700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 0x7fcc74062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.654+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fcc8807ce30 con 0x7fcc900c69a0 2023-12-19T14:55:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.654+0000 7fcc95fca700 1 -- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 msgr2=0x7fcc74062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:10.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.654+0000 7fcc95fca700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 0x7fcc74062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.658+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc8804d190 con 0x7fcc900c69a0 2023-12-19T14:55:10.764 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:10.757+0000 7fa6078eb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:10.812 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:10.810+0000 7f1b77b86000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:10.833 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:10.825+0000 7fa6078eb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:10.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.842+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcc9004be40 con 0x7fcc900c69a0 2023-12-19T14:55:10.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.850+0000 7fcc85ffb700 1 -- 172.21.15.50:0/2041491184 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcc880507b0 con 0x7fcc900c69a0 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.854+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 msgr2=0x7fcc74062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.854+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 0x7fcc74062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.854+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 msgr2=0x7fcc9015a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.854+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800cc40 tx=0x7fcc88001a50 comp rx=0 tx=0).stop 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 shutdown_connections 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9006de00 0x7fcc9014abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc740606c0 0x7fcc74062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc900c69a0 0x7fcc9015a950 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 --2- 172.21.15.50:0/2041491184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc9006c7c0 0x7fcc9015f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:10.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 >> 172.21.15.50:0/2041491184 conn(0x7fcc90067b50 msgr2=0x7fcc900b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:10.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 shutdown_connections 2023-12-19T14:55:10.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:10.858+0000 7fcc96fcc700 1 -- 172.21.15.50:0/2041491184 wait complete. 2023-12-19T14:55:10.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T14:55:10.893 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:10.890+0000 7f1b77b86000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:10.971 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:10.961+0000 7fa6078eb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.046+0000 7ff143a2e700 1 Processor -- start 2023-12-19T14:55:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.046+0000 7ff143a2e700 1 -- start start 2023-12-19T14:55:11.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff143a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 0x7ff13c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff143a2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 0x7ff13c156350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.053 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.050+0000 7f1b77b86000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff143a2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c1572d0 con 0x7ff13c156890 2023-12-19T14:55:11.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff143a2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13c156d10 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff143a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c156ff0 con 0x7ff13c1538c0 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff141fcb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff13c156890 0x7ff13c0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57336/0 (socket says 172.21.15.50:57336) 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff1417ca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 0x7ff13c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff140fc9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 0x7ff13c156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff141fcb700 1 -- 172.21.15.50:0/3918433044 learned_addr learned my addr 172.21.15.50:0/3918433044 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:11.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff133fff700 1 -- 172.21.15.50:0/3918433044 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021451459 0 0) 0x7ff13c1572d0 con 0x7ff13c156890 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.050+0000 7ff133fff700 1 -- 172.21.15.50:0/3918433044 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128003570 con 0x7ff13c156890 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff1417ca700 1 -- 172.21.15.50:0/3918433044 >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 msgr2=0x7ff13c156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff1417ca700 1 --2- 172.21.15.50:0/3918433044 >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 0x7ff13c156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff1417ca700 1 -- 172.21.15.50:0/3918433044 >> v1:172.21.15.50:6789/0 conn(0x7ff13c156890 legacy=0x7ff13c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff1417ca700 1 -- 172.21.15.50:0/3918433044 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0ace10 con 0x7ff13c1538c0 2023-12-19T14:55:11.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff140fc9700 1 --2- 172.21.15.50:0/3918433044 >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 0x7ff13c156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff1417ca700 1 --2- 172.21.15.50:0/3918433044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 0x7ff13c06c7c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1380098e0 tx=0x7ff1380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7ada1b25209245 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff133fff700 1 -- 172.21.15.50:0/3918433044 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138011630 con 0x7ff13c1538c0 2023-12-19T14:55:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff133fff700 1 -- 172.21.15.50:0/3918433044 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff138011790 con 0x7ff13c1538c0 2023-12-19T14:55:11.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff133fff700 1 -- 172.21.15.50:0/3918433044 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138011b30 con 0x7ff13c1538c0 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 -- 172.21.15.50:0/3918433044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 msgr2=0x7ff13c06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 --2- 172.21.15.50:0/3918433044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 0x7ff13c06c7c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1380098e0 tx=0x7ff1380091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 -- 172.21.15.50:0/3918433044 shutdown_connections 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 --2- 172.21.15.50:0/3918433044 >> v2:172.21.15.50:3300/0 conn(0x7ff13c0cc4e0 0x7ff13c156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 --2- 172.21.15.50:0/3918433044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c1538c0 0x7ff13c06c7c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 -- 172.21.15.50:0/3918433044 >> 172.21.15.50:0/3918433044 conn(0x7ff13c067bb0 msgr2=0x7ff13c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:11.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 -- 172.21.15.50:0/3918433044 shutdown_connections 2023-12-19T14:55:11.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.054+0000 7ff143a2e700 1 -- 172.21.15.50:0/3918433044 wait complete. 2023-12-19T14:55:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 Processor -- start 2023-12-19T14:55:11.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 -- start start 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c0cc4e0 0x7ff13c1408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13c1538c0 0x7ff13c16b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c148df0 0x7ff13c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c140e20 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c149f90 con 0x7ff13c1538c0 2023-12-19T14:55:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff143a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c14a420 con 0x7ff13c148df0 2023-12-19T14:55:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c0cc4e0 0x7ff13c1408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c0cc4e0 0x7ff13c1408e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44012/0 (socket says 172.21.15.50:44012) 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 -- 172.21.15.50:0/906340235 learned_addr learned my addr 172.21.15.50:0/906340235 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13c1538c0 0x7ff13c16b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff141fcb700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c148df0 0x7ff13c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c148df0 msgr2=0x7ff13c06d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff13c148df0 0x7ff13c06d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13c1538c0 msgr2=0x7ff13c16b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13c1538c0 0x7ff13c16b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.058+0000 7ff1417ca700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c009420 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13c1538c0 0x7ff13c16b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff1417ca700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff13c0cc4e0 0x7ff13c1408e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff13c157130 tx=0x7ff138017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138011b10 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380253f0 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380257b0 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff143a2e700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138009420 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff143a2e700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c1561b0 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff143a2e700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c071120 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.062+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 34) v1 ==== 80080+0+0 (secure 0 0 0) 0x7ff138025ba0 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.066+0000 7ff1317fa700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.066+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff13807e300 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.066+0000 7ff140fc9700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.066+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.070+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff13804e680 con 0x7ff13c0cc4e0 2023-12-19T14:55:11.228 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:11.221+0000 7fa6078eb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:11.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.266+0000 7ff143a2e700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff13c0ad0f0 con 0x7ff1200606c0 2023-12-19T14:55:11.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.270+0000 7ff140fc9700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:11.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.270+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:11.300 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:11.293+0000 7fa6078eb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:11.337 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.334+0000 7f1b77b86000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:11.422 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.418+0000 7f1b77b86000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:11.506 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:11.497+0000 7fa6078eb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:11.588 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:11.581+0000 7fa6078eb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:11.659 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.658+0000 7f1b77b86000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:11.670 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:11.661+0000 7fa6078eb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.674+0000 7ff140fc9700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:11.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:11.674+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:55:11.747 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.746+0000 7f1b77b86000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:11.836 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:11.834+0000 7f1b77b86000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:12.043 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.037+0000 7fa6078eb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:12.211 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.205+0000 7fa6078eb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:12.253 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:12.250+0000 7f1b77b86000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:12.423 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:12.422+0000 7f1b77b86000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:12.478+0000 7ff140fc9700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:12.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:12.478+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:55:12.581 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.573+0000 7fa6078eb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:12.792 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.785+0000 7fa6078eb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:12.882 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.873+0000 7fa6078eb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:12.955 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:12.945+0000 7fa6078eb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:13.003 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.002+0000 7f1b77b86000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:13.030 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.025+0000 7fa6078eb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:13.235 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.234+0000 7f1b77b86000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:13.252 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.245+0000 7fa6078eb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:13.325 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.317+0000 7fa6078eb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:13.337 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.334+0000 7f1b77b86000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:13.411 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.405+0000 7fa6078eb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:13.423 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.422+0000 7f1b77b86000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:13.508 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.506+0000 7f1b77b86000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:13.597 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.589+0000 7fa6078eb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:13.686 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:13.677+0000 7fa6078eb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:13.747 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.746+0000 7f1b77b86000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:13.833 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.830+0000 7f1b77b86000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:13.934 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:13.930+0000 7f1b77b86000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:14.082+0000 7ff140fc9700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:14.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:14.082+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:55:14.148 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:14.146+0000 7f1b77b86000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:14.245 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:14.242+0000 7f1b77b86000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:14.389 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.381+0000 7fa6078eb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:14.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:14.406+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 35) v1 ==== 79689+0+0 (secure 0 0 0) 0x7ff13804ea50 con 0x7ff13c0cc4e0 2023-12-19T14:55:14.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:14.406+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 msgr2=0x7ff120062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:14.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:14.406+0000 7ff1317fa700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:14.588 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5caeda700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5caeda700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5caeda700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5caeda700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5caeda700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5c6ed2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5c6ed2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5c6ed2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5c6ed2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:14.589 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:14.581+0000 7fa5c6ed2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:15.045 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:15.042+0000 7f1b77b86000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.418+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 36) v1 ==== 79816+0+0 (secure 0 0 0) 0x7ff138067660 con 0x7ff13c0cc4e0 2023-12-19T14:55:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.418+0000 7ff1317fa700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.418+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff13c0ad0f0 con 0x7ff1200606c0 2023-12-19T14:55:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.418+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.422+0000 7ff140fc9700 1 --2- 172.21.15.50:0/906340235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff1200606c0 0x7ff120062af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff13c156640 tx=0x7ff12c0091a0 comp rx=0 tx=0).ready entity=mgr.7110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:15.438 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:15.429+0000 7fa5d1ee8700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T14:55:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.434+0000 7ff1317fa700 1 -- 172.21.15.50:0/906340235 <== mgr.7110 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7ff13c0ad0f0 con 0x7ff1200606c0 2023-12-19T14:55:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T14:55:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T14:55:15.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:55:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.674+0000 7f174359e700 1 Processor -- start 2023-12-19T14:55:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.674+0000 7f174359e700 1 -- start start 2023-12-19T14:55:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17440cc040 0x7f1744159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17440cbbb0 con 0x7f17440caa10 2023-12-19T14:55:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174406dc90 con 0x7f17440ce2d0 2023-12-19T14:55:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17440cafa0 con 0x7f17440cc040 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1741d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f17440caa10 0x7f174406a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57338/0 (socket says 172.21.15.50:57338) 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1741d9b700 1 -- 172.21.15.50:0/2523699861 learned_addr learned my addr 172.21.15.50:0/2523699861 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1742d9d700 1 --2- 172.21.15.50:0/2523699861 >> v2:172.21.15.125:3300/0 conn(0x7f17440cc040 0x7f1744159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174159a700 1 -- 172.21.15.50:0/2523699861 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377789676 0 0) 0x7f17440cbbb0 con 0x7f17440caa10 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f174159a700 1 -- 172.21.15.50:0/2523699861 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f172c003570 con 0x7f17440caa10 2023-12-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1742d9d700 1 -- 172.21.15.50:0/2523699861 >> v1:172.21.15.125:6789/0 conn(0x7f17440ce2d0 legacy=0x7f1744158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1742d9d700 1 -- 172.21.15.50:0/2523699861 >> v1:172.21.15.50:6789/0 conn(0x7f17440caa10 legacy=0x7f174406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.678+0000 7f1742d9d700 1 -- 172.21.15.50:0/2523699861 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440cb280 con 0x7f17440cc040 2023-12-19T14:55:15.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f1742d9d700 1 --2- 172.21.15.50:0/2523699861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440cc040 0x7f1744159010 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f173c009e60 tx=0x7f173c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=248b0f56b90c5b49 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174159a700 1 -- 172.21.15.50:0/2523699861 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c00e070 con 0x7f17440cc040 2023-12-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174159a700 1 -- 172.21.15.50:0/2523699861 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c00bb60 con 0x7f17440cc040 2023-12-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174159a700 1 -- 172.21.15.50:0/2523699861 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c012430 con 0x7f17440cc040 2023-12-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 -- 172.21.15.50:0/2523699861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440cc040 msgr2=0x7f1744159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 --2- 172.21.15.50:0/2523699861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440cc040 0x7f1744159010 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f173c009e60 tx=0x7f173c009780 comp rx=0 tx=0).stop 2023-12-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 -- 172.21.15.50:0/2523699861 shutdown_connections 2023-12-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 --2- 172.21.15.50:0/2523699861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440cc040 0x7f1744159010 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:15.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 -- 172.21.15.50:0/2523699861 >> 172.21.15.50:0/2523699861 conn(0x7f1744067bb0 msgr2=0x7f17440cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:15.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 -- 172.21.15.50:0/2523699861 shutdown_connections 2023-12-19T14:55:15.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.682+0000 7f174359e700 1 -- 172.21.15.50:0/2523699861 wait complete. 2023-12-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 Processor -- start 2023-12-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 -- start start 2023-12-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440caa10 0x7f1744169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17440cc040 0x7f1744149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17441586d0 con 0x7f17440ce2d0 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174416a120 con 0x7f17440caa10 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17440c9480 con 0x7f17440cc040 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44020/0 (socket says 172.21.15.50:44020) 2023-12-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 -- 172.21.15.50:0/2754807119 learned_addr learned my addr 172.21.15.50:0/2754807119 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1741d9b700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17440cc040 0x7f1744149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f174259c700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440caa10 0x7f1744169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 -- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17440cc040 msgr2=0x7f1744149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17440cc040 0x7f1744149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 -- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440caa10 msgr2=0x7f1744169be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440caa10 0x7f1744169be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 -- 172.21.15.50:0/2754807119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c009a20 con 0x7f17440ce2d0 2023-12-19T14:55:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f1742d9d700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f173c00b6c0 tx=0x7f173c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.686+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c027470 con 0x7f17440ce2d0 2023-12-19T14:55:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c0275f0 con 0x7f17440ce2d0 2023-12-19T14:55:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0279b0 con 0x7f17440ce2d0 2023-12-19T14:55:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17440c8580 con 0x7f17440ce2d0 2023-12-19T14:55:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17440c8b40 con 0x7f17440ce2d0 2023-12-19T14:55:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1744158840 con 0x7f17440ce2d0 2023-12-19T14:55:15.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.690+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 36) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f173c024070 con 0x7f17440ce2d0 2023-12-19T14:55:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.694+0000 7f17327fc700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 0x7f17240628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.694+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f173c07f380 con 0x7f17440ce2d0 2023-12-19T14:55:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.694+0000 7f174259c700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 0x7f17240628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.694+0000 7f174259c700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 0x7f17240628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1744157ee0 tx=0x7f1734009040 comp rx=0 tx=0).ready entity=mgr.7110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.702+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f173c04f8a0 con 0x7f17440ce2d0 2023-12-19T14:55:15.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:15.906+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f174404be40 con 0x7f17440ce2d0 2023-12-19T14:55:16.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.462+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f173c003040 con 0x7f17440ce2d0 2023-12-19T14:55:16.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.462+0000 7f17327fc700 1 -- 172.21.15.50:0/2754807119 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v37) v1 ==== 93+0+0 (secure 0 0 0) 0x7f173c04f8a0 con 0x7f17440ce2d0 2023-12-19T14:55:16.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 msgr2=0x7f17240628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 0x7f17240628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1744157ee0 tx=0x7f1734009040 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 msgr2=0x7f1744156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f173c00b6c0 tx=0x7f173c019000 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 shutdown_connections 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17240603f0 0x7f17240628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17440ce2d0 0x7f1744156980 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17440cc040 0x7f1744149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 --2- 172.21.15.50:0/2754807119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17440caa10 0x7f1744169be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 >> 172.21.15.50:0/2754807119 conn(0x7f1744067bb0 msgr2=0x7f1744155400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 shutdown_connections 2023-12-19T14:55:16.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:16.466+0000 7f174359e700 1 -- 172.21.15.50:0/2754807119 wait complete. 2023-12-19T14:55:16.491 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:55:16.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:55:16.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:55:16.577 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:55:16.577 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:55:16.577 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:55:16.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:16.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[49425]: fuse finished with error 0 and tester_r 0 2023-12-19T14:55:16.610 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:55:16.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:16.611 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:55:16.665 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:16.657+0000 7f5d75bce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:16.676 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:16.674+0000 7f2fcc92c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:16.769 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:55:16.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:16.769 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:55:16.788 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:55:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:55:16.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:55:16.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:16.790 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:55:16.833 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:55:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:55:16.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:55:16.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:16.835 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:55:16.853 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:55:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:55:16.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:17.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.046+0000 7f0f1dc0c700 1 Processor -- start 2023-12-19T14:55:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 -- start start 2023-12-19T14:55:17.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f181591f0 0x7f0f1806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f180ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1806b730 con 0x7f0f18158e20 2023-12-19T14:55:17.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1806ab20 con 0x7f0f181591f0 2023-12-19T14:55:17.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1dc0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1806ae00 con 0x7f0f180ce810 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f181591f0 0x7f0f1806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f181591f0 0x7f0f1806a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:58820/0 (socket says 172.21.15.50:58820) 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 -- 172.21.15.50:0/770251355 learned_addr learned my addr 172.21.15.50:0/770251355 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1d40b700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f180ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 -- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 msgr2=0x7f0f180ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f180ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 -- 172.21.15.50:0/770251355 >> v1:172.21.15.125:6789/0 conn(0x7f0f18158e20 legacy=0x7f0f180ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f17fff700 1 -- 172.21.15.50:0/770251355 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1806b0e0 con 0x7f0f181591f0 2023-12-19T14:55:17.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.050+0000 7f0f1d40b700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f180ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:17.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f17fff700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f1806a540 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0f040099b0 tx=0x7f0f04009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a306a3738592500 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f177fe700 1 -- 172.21.15.50:0/770251355 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04009ea0 con 0x7f0f181591f0 2023-12-19T14:55:17.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f177fe700 1 -- 172.21.15.50:0/770251355 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0400b860 con 0x7f0f181591f0 2023-12-19T14:55:17.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f177fe700 1 -- 172.21.15.50:0/770251355 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400bc00 con 0x7f0f181591f0 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 msgr2=0x7f0f1806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f1806a540 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0f040099b0 tx=0x7f0f04009250 comp rx=0 tx=0).stop 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/770251355 shutdown_connections 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f180ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/770251355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f1806a540 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/770251355 >> 172.21.15.50:0/770251355 conn(0x7f0f18067bb0 msgr2=0x7f0f180cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/770251355 shutdown_connections 2023-12-19T14:55:17.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.054+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/770251355 wait complete. 2023-12-19T14:55:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 Processor -- start 2023-12-19T14:55:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 -- start start 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f1814a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f18153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18153f10 con 0x7f0f18158e20 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815b790 con 0x7f0f181591f0 2023-12-19T14:55:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1dc0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815c160 con 0x7f0f180ce810 2023-12-19T14:55:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58962/0 (socket says 172.21.15.50:58962) 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f17fff700 1 -- 172.21.15.50:0/3623795783 learned_addr learned my addr 172.21.15.50:0/3623795783 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f1814a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1d40b700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f18153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 msgr2=0x7f0f18153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f18153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 msgr2=0x7f0f1815bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f1cc0a700 1 -- 172.21.15.50:0/3623795783 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10009a20 con 0x7f0f180ce810 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.058+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f1cc0a700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f1814a640 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c590 tx=0x7f0f0c007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c011590 con 0x7f0f180ce810 2023-12-19T14:55:17.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c011710 con 0x7f0f180ce810 2023-12-19T14:55:17.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f040094f0 con 0x7f0f180ce810 2023-12-19T14:55:17.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c011ad0 con 0x7f0f180ce810 2023-12-19T14:55:17.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1815ae60 con 0x7f0f180ce810 2023-12-19T14:55:17.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1804be40 con 0x7f0f180ce810 2023-12-19T14:55:17.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.062+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f0f0c01b3b0 con 0x7f0f180ce810 2023-12-19T14:55:17.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.066+0000 7f0f14ff9700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 0x7f0efc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.066+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0f0c07c1a0 con 0x7f0f180ce810 2023-12-19T14:55:17.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.066+0000 7f0f17fff700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 msgr2=0x7f0efc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:17.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.066+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 0x7f0efc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:17.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.074+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f0c04c260 con 0x7f0f180ce810 2023-12-19T14:55:17.225 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:17.217+0000 7f5d75bce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:17.262 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:17.258+0000 7f2fcc92c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:17.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.270+0000 7f0f17fff700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 msgr2=0x7f0efc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:17.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.270+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 0x7f0efc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:17.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.278+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1806b3c0 con 0x7f0f180ce810 2023-12-19T14:55:17.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f14ff9700 1 -- 172.21.15.50:0/3623795783 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f0f0c04f880 con 0x7f0f180ce810 2023-12-19T14:55:17.284 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:17.284 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:17.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 msgr2=0x7f0efc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:17.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 0x7f0efc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 msgr2=0x7f0f1814a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f1814a640 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c590 tx=0x7f0f0c007ec0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 shutdown_connections 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f181591f0 0x7f0f18153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0efc060700 0x7f0efc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f18158e20 0x7f0f1815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 --2- 172.21.15.50:0/3623795783 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f180ce810 0x7f0f1814a640 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.282+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 >> 172.21.15.50:0/3623795783 conn(0x7f0f18067bb0 msgr2=0x7f0f18154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.286+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 shutdown_connections 2023-12-19T14:55:17.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.286+0000 7f0f1dc0c700 1 -- 172.21.15.50:0/3623795783 wait complete. 2023-12-19T14:55:17.290 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:17.307 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:17.301+0000 7f5d75bce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:17.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:17.343 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:17.342+0000 7f2fcc92c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:17.451 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:17.445+0000 7f5d75bce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.474+0000 7f7e16665700 1 Processor -- start 2023-12-19T14:55:17.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.474+0000 7f7e16665700 1 -- start start 2023-12-19T14:55:17.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e16665700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7e10158f70 0x7f7e10067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e16665700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10068c70 con 0x7f7e10069100 2023-12-19T14:55:17.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e16665700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10068060 con 0x7f7e10159340 2023-12-19T14:55:17.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e16665700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10068340 con 0x7f7e10158f70 2023-12-19T14:55:17.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0f7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7e10069100 0x7f7e100cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45620/0 (socket says 172.21.15.50:45620) 2023-12-19T14:55:17.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0f7fe700 1 -- 172.21.15.50:0/1383309192 learned_addr learned my addr 172.21.15.50:0/1383309192 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:17.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0ffff700 1 --2- 172.21.15.50:0/1383309192 >> v2:172.21.15.125:3300/0 conn(0x7f7e10158f70 0x7f7e10067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0ffff700 1 -- 172.21.15.50:0/1383309192 >> v1:172.21.15.125:6789/0 conn(0x7f7e10159340 legacy=0x7f7e100cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0ffff700 1 -- 172.21.15.50:0/1383309192 >> v1:172.21.15.50:6789/0 conn(0x7f7e10069100 legacy=0x7f7e100cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0ffff700 1 -- 172.21.15.50:0/1383309192 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10068620 con 0x7f7e10158f70 2023-12-19T14:55:17.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.478+0000 7f7e0ffff700 1 --2- 172.21.15.50:0/1383309192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10158f70 0x7f7e10067980 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009910 tx=0x7f7e000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d86c1517df4f6a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e0effd700 1 -- 172.21.15.50:0/1383309192 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000b820 con 0x7f7e10158f70 2023-12-19T14:55:17.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e0effd700 1 -- 172.21.15.50:0/1383309192 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0000b980 con 0x7f7e10158f70 2023-12-19T14:55:17.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e0effd700 1 -- 172.21.15.50:0/1383309192 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000bd20 con 0x7f7e10158f70 2023-12-19T14:55:17.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 -- 172.21.15.50:0/1383309192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10158f70 msgr2=0x7f7e10067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 --2- 172.21.15.50:0/1383309192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10158f70 0x7f7e10067980 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009910 tx=0x7f7e000091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 -- 172.21.15.50:0/1383309192 shutdown_connections 2023-12-19T14:55:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 --2- 172.21.15.50:0/1383309192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10158f70 0x7f7e10067980 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 -- 172.21.15.50:0/1383309192 >> 172.21.15.50:0/1383309192 conn(0x7f7e100ccc40 msgr2=0x7f7e100cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 -- 172.21.15.50:0/1383309192 shutdown_connections 2023-12-19T14:55:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.486+0000 7f7e16665700 1 -- 172.21.15.50:0/1383309192 wait complete. 2023-12-19T14:55:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.490+0000 7f7e16665700 1 Processor -- start 2023-12-19T14:55:17.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.490+0000 7f7e16665700 1 -- start start 2023-12-19T14:55:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 0x7f7e1014d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10159340 0x7f7e1015bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1016aad0 con 0x7f7e10158f70 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100b6180 con 0x7f7e10159340 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1015da60 con 0x7f7e10069100 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58964/0 (socket says 172.21.15.50:58964) 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 -- 172.21.15.50:0/2015380053 learned_addr learned my addr 172.21.15.50:0/2015380053 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0ffff700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 0x7f7e1014d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e14c02700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10159340 0x7f7e1015bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 -- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 msgr2=0x7f7e1014d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 0x7f7e1014d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 -- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10159340 msgr2=0x7f7e1015bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10159340 0x7f7e1015bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 -- 172.21.15.50:0/2015380053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009420 con 0x7f7e10158f70 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0ffff700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 0x7f7e1014d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e0f7fe700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c7f0 tx=0x7f7e04007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04005b30 con 0x7f7e10158f70 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04005cb0 con 0x7f7e10158f70 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1014dd90 con 0x7f7e10158f70 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04004a80 con 0x7f7e10158f70 2023-12-19T14:55:17.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.494+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1015d410 con 0x7f7e10158f70 2023-12-19T14:55:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.498+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1015dbd0 con 0x7f7e10158f70 2023-12-19T14:55:17.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.498+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f7e040070f0 con 0x7f7e10158f70 2023-12-19T14:55:17.502 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:17.502+0000 7f2fcc92c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.502+0000 7f7df7fff700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 0x7f7dfc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.502+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7e0407c290 con 0x7f7e10158f70 2023-12-19T14:55:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.502+0000 7f7e0ffff700 1 -- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 msgr2=0x7f7dfc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.502+0000 7f7e0ffff700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 0x7f7dfc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.506+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7e0404c330 con 0x7f7e10158f70 2023-12-19T14:55:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.678+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e10071120 con 0x7f7e10158f70 2023-12-19T14:55:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7df7fff700 1 -- 172.21.15.50:0/2015380053 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f7e0404f950 con 0x7f7e10158f70 2023-12-19T14:55:17.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:17.684 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":101,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:14.394534+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3252761303":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2020877475":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2604193437":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/812779881":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3729319264":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3000325649":"2023-12-20T14:55:14.394463+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 msgr2=0x7f7dfc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 0x7f7dfc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 msgr2=0x7f7e1015def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c7f0 tx=0x7f7e04007f70 comp rx=0 tx=0).stop 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 shutdown_connections 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e10159340 0x7f7e1015bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dfc060700 0x7f7dfc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e10158f70 0x7f7e1015def0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 --2- 172.21.15.50:0/2015380053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7e10069100 0x7f7e1014d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 >> 172.21.15.50:0/2015380053 conn(0x7f7e100ccc40 msgr2=0x7f7e100b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 shutdown_connections 2023-12-19T14:55:17.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.682+0000 7f7e16665700 1 -- 172.21.15.50:0/2015380053 wait complete. 2023-12-19T14:55:17.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:54:56.099031+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:54:57.248377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:55:17.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:55:17.730 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:17.721+0000 7f5d75bce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:17.793 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:17.790+0000 7f2fcc92c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:17.805 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:17.797+0000 7f5d75bce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:17.871 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:17.870+0000 7f2fcc92c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.918+0000 7f7d8c078700 1 Processor -- start 2023-12-19T14:55:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.918+0000 7f7d8c078700 1 -- start start 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d84158e20 0x7f7d840ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d840cb970 0x7f7d840ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 0x7f7d8406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d840cbdf0 con 0x7f7d840cb970 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d840cc0d0 con 0x7f7d84158e20 2023-12-19T14:55:17.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d8c078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840cc3b0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d89613700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d840cb970 0x7f7d840ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d89613700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d840cb970 0x7f7d840ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58966/0 (socket says 172.21.15.50:58966) 2023-12-19T14:55:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d89613700 1 -- 172.21.15.50:0/4072362345 learned_addr learned my addr 172.21.15.50:0/4072362345 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:17.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.922+0000 7f7d89e14700 1 --2- 172.21.15.50:0/4072362345 >> v2:172.21.15.125:3300/0 conn(0x7f7d84158e20 0x7f7d840ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 -- 172.21.15.50:0/4072362345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 msgr2=0x7f7d8406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8a615700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 0x7f7d8406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 0x7f7d8406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 -- 172.21.15.50:0/4072362345 >> v2:172.21.15.125:3300/0 conn(0x7f7d84158e20 msgr2=0x7f7d840ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 --2- 172.21.15.50:0/4072362345 >> v2:172.21.15.125:3300/0 conn(0x7f7d84158e20 0x7f7d840ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 -- 172.21.15.50:0/4072362345 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d840cc690 con 0x7f7d840cb970 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8a615700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 0x7f7d8406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d89613700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840cb970 0x7f7d840ceb00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7d740099b0 tx=0x7f7d74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf981574d51f8d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d88e12700 1 -- 172.21.15.50:0/4072362345 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74009ea0 con 0x7f7d840cb970 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d88e12700 1 -- 172.21.15.50:0/4072362345 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7400b820 con 0x7f7d840cb970 2023-12-19T14:55:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d88e12700 1 -- 172.21.15.50:0/4072362345 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400bbc0 con 0x7f7d840cb970 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 -- 172.21.15.50:0/4072362345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840cb970 msgr2=0x7f7d840ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840cb970 0x7f7d840ceb00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7d740099b0 tx=0x7f7d74009250 comp rx=0 tx=0).stop 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 -- 172.21.15.50:0/4072362345 shutdown_connections 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840cb970 0x7f7d840ceb00 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 --2- 172.21.15.50:0/4072362345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d840ca2c0 0x7f7d8406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 --2- 172.21.15.50:0/4072362345 >> v2:172.21.15.125:3300/0 conn(0x7f7d84158e20 0x7f7d840ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 -- 172.21.15.50:0/4072362345 >> 172.21.15.50:0/4072362345 conn(0x7f7d84067bb0 msgr2=0x7f7d840d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.926+0000 7f7d8c078700 1 -- 172.21.15.50:0/4072362345 shutdown_connections 2023-12-19T14:55:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.930+0000 7f7d8c078700 1 -- 172.21.15.50:0/4072362345 wait complete. 2023-12-19T14:55:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.930+0000 7f7d8c078700 1 Processor -- start 2023-12-19T14:55:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.930+0000 7f7d8c078700 1 -- start start 2023-12-19T14:55:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d84158e20 0x7f7d840b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 0x7f7d8414d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840b5190 con 0x7f7d840b4690 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840c7ac0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840b4bd0 con 0x7f7d84158e20 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:58888/0 (socket says 172.21.15.50:58888) 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8a615700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 0x7f7d8414d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 -- 172.21.15.50:0/499489331 learned_addr learned my addr 172.21.15.50:0/499489331 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89613700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d84158e20 0x7f7d840b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 -- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d84158e20 msgr2=0x7f7d840b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d84158e20 0x7f7d840b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 -- 172.21.15.50:0/499489331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 msgr2=0x7f7d8414d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 0x7f7d8414d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 -- 172.21.15.50:0/499489331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740094f0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8a615700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 0x7f7d8414d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d89e14700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e930 tx=0x7f7d800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80013680 con 0x7f7d840ca2c0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d800137e0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d840b4eb0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80013b80 con 0x7f7d840ca2c0 2023-12-19T14:55:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.934+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d84142940 con 0x7f7d840ca2c0 2023-12-19T14:55:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d84071120 con 0x7f7d840ca2c0 2023-12-19T14:55:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f7d8000c3e0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d79ffb700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 0x7f7d68062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7d8007cec0 con 0x7f7d840ca2c0 2023-12-19T14:55:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d89613700 1 -- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 msgr2=0x7f7d68062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.938+0000 7f7d89613700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 0x7f7d68062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:17.946+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7d8004cee0 con 0x7f7d840ca2c0 2023-12-19T14:55:18.037 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:18.029+0000 7f5d75bce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:18.109 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:18.106+0000 7f2fcc92c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:18.122 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:18.113+0000 7f5d75bce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.126+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d840d09b0 con 0x7f7d840ca2c0 2023-12-19T14:55:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.130+0000 7f7d79ffb700 1 -- 172.21.15.50:0/499489331 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d80050500 con 0x7f7d840ca2c0 2023-12-19T14:55:18.131 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:55:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 msgr2=0x7f7d68062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 0x7f7d68062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 msgr2=0x7f7d840c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e930 tx=0x7f7d800091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 shutdown_connections 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d68060730 0x7f7d68062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d840b4690 0x7f7d8414d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d84158e20 0x7f7d840b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 --2- 172.21.15.50:0/499489331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d840ca2c0 0x7f7d840c9a80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 >> 172.21.15.50:0/499489331 conn(0x7f7d84067bb0 msgr2=0x7f7d840cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 shutdown_connections 2023-12-19T14:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.134+0000 7f7d8c078700 1 -- 172.21.15.50:0/499489331 wait complete. 2023-12-19T14:55:18.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:55:18.201 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:18.198+0000 7f2fcc92c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:18.211 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:18.205+0000 7f5d75bce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:18.294 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:18.290+0000 7f2fcc92c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 Processor -- start 2023-12-19T14:55:18.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 -- start start 2023-12-19T14:55:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8410158e20 0x7f84100ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 0x7f84100ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100cbdf0 con 0x7f84100cb970 2023-12-19T14:55:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84100cc0d0 con 0x7f8410158e20 2023-12-19T14:55:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.366+0000 7f8416878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100cc3b0 con 0x7f84100ca2c0 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8410158e20 0x7f84100ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8410158e20 0x7f84100ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58992/0 (socket says 172.21.15.50:58992) 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 -- 172.21.15.50:0/1014669906 learned_addr learned my addr 172.21.15.50:0/1014669906 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8414e15700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 0x7f84100ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 -- 172.21.15.50:0/1014669906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 msgr2=0x7f84100ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 0x7f84100ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 -- 172.21.15.50:0/1014669906 >> v1:172.21.15.125:6789/0 conn(0x7f84100cb970 legacy=0x7f841006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 -- 172.21.15.50:0/1014669906 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84100cc690 con 0x7f8410158e20 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8414e15700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 0x7f84100ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:18.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840ffff700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8410158e20 0x7f84100ce5c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f84000099e0 tx=0x7f8400009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd5a75808650b64 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840effd700 1 -- 172.21.15.50:0/1014669906 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840000b820 con 0x7f8410158e20 2023-12-19T14:55:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840effd700 1 -- 172.21.15.50:0/1014669906 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840000b980 con 0x7f8410158e20 2023-12-19T14:55:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f840effd700 1 -- 172.21.15.50:0/1014669906 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840000bd20 con 0x7f8410158e20 2023-12-19T14:55:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 -- 172.21.15.50:0/1014669906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8410158e20 msgr2=0x7f84100ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8410158e20 0x7f84100ce5c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f84000099e0 tx=0x7f8400009250 comp rx=0 tx=0).stop 2023-12-19T14:55:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 -- 172.21.15.50:0/1014669906 shutdown_connections 2023-12-19T14:55:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84100ca2c0 0x7f84100ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 --2- 172.21.15.50:0/1014669906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8410158e20 0x7f84100ce5c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 -- 172.21.15.50:0/1014669906 >> 172.21.15.50:0/1014669906 conn(0x7f8410067bb0 msgr2=0x7f84100d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:18.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 -- 172.21.15.50:0/1014669906 shutdown_connections 2023-12-19T14:55:18.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.370+0000 7f8416878700 1 -- 172.21.15.50:0/1014669906 wait complete. 2023-12-19T14:55:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.374+0000 7f8416878700 1 Processor -- start 2023-12-19T14:55:18.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.374+0000 7f8416878700 1 -- start start 2023-12-19T14:55:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8410158e20 0x7f841015af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f841015b4b0 0x7f841006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841016d740 con 0x7f84100ca2c0 2023-12-19T14:55:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100b5350 con 0x7f8410158e20 2023-12-19T14:55:18.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841014c9d0 con 0x7f841015b4b0 2023-12-19T14:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59008/0 (socket says 172.21.15.50:59008) 2023-12-19T14:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 -- 172.21.15.50:0/3149730317 learned_addr learned my addr 172.21.15.50:0/3149730317 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8414e15700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f841015b4b0 0x7f841006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840f7fe700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8410158e20 0x7f841015af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 -- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f841015b4b0 msgr2=0x7f841006ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f841015b4b0 0x7f841006ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 -- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8410158e20 msgr2=0x7f841015af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8410158e20 0x7f841015af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 -- 172.21.15.50:0/3149730317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000094f0 con 0x7f84100ca2c0 2023-12-19T14:55:18.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f840ffff700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f84000090b0 tx=0x7f8400017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400011d30 con 0x7f84100ca2c0 2023-12-19T14:55:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400011e90 con 0x7f84100ca2c0 2023-12-19T14:55:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840001a6a0 con 0x7f84100ca2c0 2023-12-19T14:55:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841016cac0 con 0x7f84100ca2c0 2023-12-19T14:55:18.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.378+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841016d080 con 0x7f84100ca2c0 2023-12-19T14:55:18.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.382+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841016d880 con 0x7f84100ca2c0 2023-12-19T14:55:18.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.382+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f840001aa50 con 0x7f84100ca2c0 2023-12-19T14:55:18.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.386+0000 7f83f7fff700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 0x7f83fc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.386+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f840007e690 con 0x7f84100ca2c0 2023-12-19T14:55:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.386+0000 7f840f7fe700 1 -- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 msgr2=0x7f83fc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:18.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.386+0000 7f840f7fe700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 0x7f83fc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:18.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.394+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f840004e7d0 con 0x7f84100ca2c0 2023-12-19T14:55:18.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.570+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8410071120 con 0x7f84100ca2c0 2023-12-19T14:55:18.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.570+0000 7f83f7fff700 1 -- 172.21.15.50:0/3149730317 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8400051df0 con 0x7f84100ca2c0 2023-12-19T14:55:18.572 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:55:18.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.570+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 msgr2=0x7f83fc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.570+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 0x7f83fc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 msgr2=0x7f841014c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f84000090b0 tx=0x7f8400017000 comp rx=0 tx=0).stop 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 shutdown_connections 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8410158e20 0x7f841015af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f83fc060700 0x7f83fc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f841015b4b0 0x7f841006ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 --2- 172.21.15.50:0/3149730317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84100ca2c0 0x7f841014c490 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 >> 172.21.15.50:0/3149730317 conn(0x7f8410067bb0 msgr2=0x7f8410157e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 shutdown_connections 2023-12-19T14:55:18.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.574+0000 7f8416878700 1 -- 172.21.15.50:0/3149730317 wait complete. 2023-12-19T14:55:18.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:55:18.620 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:18.613+0000 7f5d75bce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:18.757 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:18.754+0000 7f2fcc92c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:18.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.770+0000 7f451359e700 1 Processor -- start 2023-12-19T14:55:18.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.770+0000 7f451359e700 1 -- start start 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f451406baa0 0x7f451415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45140adab0 0x7f45140adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451415ae30 con 0x7f4514154780 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140abf70 con 0x7f451406baa0 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4514154b50 con 0x7f45140adab0 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4514154780 0x7f4514157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45628/0 (socket says 172.21.15.50:45628) 2023-12-19T14:55:18.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f451259c700 1 -- 172.21.15.50:0/439780472 learned_addr learned my addr 172.21.15.50:0/439780472 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:18.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f4512d9d700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45140adab0 0x7f45140adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 --2- 172.21.15.50:0/439780472 >> v2:172.21.15.125:3300/0 conn(0x7f451406baa0 0x7f451415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f4511d9b700 1 -- 172.21.15.50:0/439780472 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853364039 0 0) 0x7f451415ae30 con 0x7f4514154780 2023-12-19T14:55:18.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f4511d9b700 1 -- 172.21.15.50:0/439780472 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003570 con 0x7f4514154780 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 -- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45140adab0 msgr2=0x7f45140adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45140adab0 0x7f45140adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 -- 172.21.15.50:0/439780472 >> v1:172.21.15.50:6789/0 conn(0x7f4514154780 legacy=0x7f4514157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 -- 172.21.15.50:0/439780472 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140ac210 con 0x7f451406baa0 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.774+0000 7f450bfff700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f451406baa0 0x7f451415b2c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098e0 tx=0x7f44fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced70ad7d4b90f18 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:18.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f4511d9b700 1 -- 172.21.15.50:0/439780472 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc011630 con 0x7f451406baa0 2023-12-19T14:55:18.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f4511d9b700 1 -- 172.21.15.50:0/439780472 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc011790 con 0x7f451406baa0 2023-12-19T14:55:18.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f4511d9b700 1 -- 172.21.15.50:0/439780472 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc011b30 con 0x7f451406baa0 2023-12-19T14:55:18.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 -- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f451406baa0 msgr2=0x7f451415b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f451406baa0 0x7f451415b2c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098e0 tx=0x7f44fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 -- 172.21.15.50:0/439780472 shutdown_connections 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45140adab0 0x7f45140adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 --2- 172.21.15.50:0/439780472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f451406baa0 0x7f451415b2c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 -- 172.21.15.50:0/439780472 >> 172.21.15.50:0/439780472 conn(0x7f4514067b50 msgr2=0x7f45140cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 -- 172.21.15.50:0/439780472 shutdown_connections 2023-12-19T14:55:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.778+0000 7f451359e700 1 -- 172.21.15.50:0/439780472 wait complete. 2023-12-19T14:55:18.783 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:18.777+0000 7f5d75bce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:18.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 Processor -- start 2023-12-19T14:55:18.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 -- start start 2023-12-19T14:55:18.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 0x7f45140ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45140adab0 0x7f4514148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140afe60 con 0x7f4514154780 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140af250 con 0x7f45140adab0 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140af530 con 0x7f451406baa0 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f4512d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f451259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 0x7f45140ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f4512d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59010/0 (socket says 172.21.15.50:59010) 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.782+0000 7f4512d9d700 1 -- 172.21.15.50:0/3665780057 learned_addr learned my addr 172.21.15.50:0/3665780057 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:18.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f450bfff700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45140adab0 0x7f4514148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:18.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 -- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45140adab0 msgr2=0x7f4514148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45140adab0 0x7f4514148520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 -- 172.21.15.50:0/3665780057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 msgr2=0x7f451415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 -- 172.21.15.50:0/3665780057 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc009420 con 0x7f451406baa0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f4512d9d700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451259c700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 0x7f45140ae950 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f450400c560 tx=0x7f4504007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504005b30 con 0x7f451406baa0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504005cb0 con 0x7f451406baa0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504007320 con 0x7f451406baa0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45140af810 con 0x7f451406baa0 2023-12-19T14:55:18.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.786+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140aee90 con 0x7f451406baa0 2023-12-19T14:55:18.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451404be40 con 0x7f451406baa0 2023-12-19T14:55:18.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f450401b3b0 con 0x7f451406baa0 2023-12-19T14:55:18.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f450a7fc700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 0x7f44f4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f450407c150 con 0x7f451406baa0 2023-12-19T14:55:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f450bfff700 1 -- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 msgr2=0x7f44f4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:18.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.790+0000 7f450bfff700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 0x7f44f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:18.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.798+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f450404c170 con 0x7f451406baa0 2023-12-19T14:55:18.943 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:18.942+0000 7f2fcc92c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:18.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.974+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45140af670 con 0x7f451406baa0 2023-12-19T14:55:18.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f450a7fc700 1 -- 172.21.15.50:0/3665780057 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7f45140af670 con 0x7f451406baa0 2023-12-19T14:55:18.979 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 msgr2=0x7f44f4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 0x7f44f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 msgr2=0x7f45140ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 0x7f45140ae950 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f450400c560 tx=0x7f4504007d50 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 shutdown_connections 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45140adab0 0x7f4514148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44f4060700 0x7f44f4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4514154780 0x7f451415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 --2- 172.21.15.50:0/3665780057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f451406baa0 0x7f45140ae950 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:18.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 >> 172.21.15.50:0/3665780057 conn(0x7f4514067b50 msgr2=0x7f451414a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 shutdown_connections 2023-12-19T14:55:18.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:18.978+0000 7f451359e700 1 -- 172.21.15.50:0/3665780057 wait complete. 2023-12-19T14:55:19.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:19.126 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.117+0000 7f5d75bce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:19.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.158+0000 7f2fad945700 1 Processor -- start 2023-12-19T14:55:19.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.158+0000 7f2fad945700 1 -- start start 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2fa8067c90 0x7f2fa8068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa814fba0 0x7f2fa814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 0x7f2fa80c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80c6630 con 0x7f2fa8067c90 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80c6910 con 0x7f2fa80c5900 2023-12-19T14:55:19.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fad945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80c6bf0 con 0x7f2fa814fba0 2023-12-19T14:55:19.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2fa8067c90 0x7f2fa8068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa77fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 0x7f2fa80c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2fa8067c90 0x7f2fa8068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59024/0 (socket says 172.21.15.50:59024) 2023-12-19T14:55:19.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/1680680906 learned_addr learned my addr 172.21.15.50:0/1680680906 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:19.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa67fc700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa814fba0 0x7f2fa814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/1680680906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa814fba0 msgr2=0x7f2fa814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:19.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa814fba0 0x7f2fa814ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/1680680906 >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 msgr2=0x7f2fa80c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:19.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/1680680906 >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 0x7f2fa80c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/1680680906 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80c6ed0 con 0x7f2fa8067c90 2023-12-19T14:55:19.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa77fe700 1 --2- 172.21.15.50:0/1680680906 >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 0x7f2fa80c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:19.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa8067c90 0x7f2fa8068060 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80c5680 tx=0x7f2f98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74a6ef69250068c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa5ffb700 1 -- 172.21.15.50:0/1680680906 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9800b640 con 0x7f2fa8067c90 2023-12-19T14:55:19.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa5ffb700 1 -- 172.21.15.50:0/1680680906 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9800b7a0 con 0x7f2fa8067c90 2023-12-19T14:55:19.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.162+0000 7f2fa5ffb700 1 -- 172.21.15.50:0/1680680906 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9800bb60 con 0x7f2fa8067c90 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 -- 172.21.15.50:0/1680680906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa8067c90 msgr2=0x7f2fa8068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa8067c90 0x7f2fa8068060 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80c5680 tx=0x7f2f98009250 comp rx=0 tx=0).stop 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 -- 172.21.15.50:0/1680680906 shutdown_connections 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 --2- 172.21.15.50:0/1680680906 >> v2:172.21.15.125:3300/0 conn(0x7f2fa80c5900 0x7f2fa80c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa814fba0 0x7f2fa814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 --2- 172.21.15.50:0/1680680906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa8067c90 0x7f2fa8068060 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 -- 172.21.15.50:0/1680680906 >> 172.21.15.50:0/1680680906 conn(0x7f2fa80cd6d0 msgr2=0x7f2fa806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 -- 172.21.15.50:0/1680680906 shutdown_connections 2023-12-19T14:55:19.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.166+0000 7f2fad945700 1 -- 172.21.15.50:0/1680680906 wait complete. 2023-12-19T14:55:19.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 Processor -- start 2023-12-19T14:55:19.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 -- start start 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fa80c5900 0x7f2fa8145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa80c8260 0x7f2fa8150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80b3b80 con 0x7f2fa814fba0 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80b2fb0 con 0x7f2fa80c5900 2023-12-19T14:55:19.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fad945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80b3250 con 0x7f2fa80c8260 2023-12-19T14:55:19.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59038/0 (socket says 172.21.15.50:59038) 2023-12-19T14:55:19.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 -- 172.21.15.50:0/2568246780 learned_addr learned my addr 172.21.15.50:0/2568246780 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:19.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fa80c5900 0x7f2fa8145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa77fe700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa80c8260 0x7f2fa8150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa80c8260 msgr2=0x7f2fa8150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa80c8260 0x7f2fa8150990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fa80c5900 msgr2=0x7f2fa8145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fa80c5900 0x7f2fa8145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.170+0000 7f2fa67fc700 1 -- 172.21.15.50:0/2568246780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0009a20 con 0x7f2fa814fba0 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2fa67fc700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e950 tx=0x7f2f9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c013680 con 0x7f2fa814fba0 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c0137e0 con 0x7f2fa814fba0 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c013ba0 con 0x7f2fa814fba0 2023-12-19T14:55:19.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f980094f0 con 0x7f2fa814fba0 2023-12-19T14:55:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa80b3810 con 0x7f2fa814fba0 2023-12-19T14:55:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa80b3cf0 con 0x7f2fa814fba0 2023-12-19T14:55:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.174+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f2f9c00c490 con 0x7f2fa814fba0 2023-12-19T14:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.178+0000 7f2f8effd700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 0x7f2f94062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.178+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2f9c07cf20 con 0x7f2fa814fba0 2023-12-19T14:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.178+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 msgr2=0x7f2f94062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:19.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.178+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 0x7f2f94062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:19.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.182+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f9c04d000 con 0x7f2fa814fba0 2023-12-19T14:55:19.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.337+0000 7f5d75bce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:19.351 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:19.350+0000 7f2fcc92c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.382+0000 7f2fa6ffd700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 msgr2=0x7f2f94062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.382+0000 7f2fa6ffd700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 0x7f2f94062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:19.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.394+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa8071120 con 0x7f2fa814fba0 2023-12-19T14:55:19.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.394+0000 7f2f8effd700 1 -- 172.21.15.50:0/2568246780 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f2f9c050620 con 0x7f2fa814fba0 2023-12-19T14:55:19.397 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:19.398 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 msgr2=0x7f2f94062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 0x7f2f94062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 msgr2=0x7f2fa80c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00e950 tx=0x7f2f9c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 shutdown_connections 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f94060700 0x7f2f94062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2fa80c8260 0x7f2fa8150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2fa814fba0 0x7f2fa80c7d20 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 --2- 172.21.15.50:0/2568246780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2fa80c5900 0x7f2fa8145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 >> 172.21.15.50:0/2568246780 conn(0x7f2fa80cd6d0 msgr2=0x7f2fa80cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:19.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 shutdown_connections 2023-12-19T14:55:19.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.398+0000 7f2fad945700 1 -- 172.21.15.50:0/2568246780 wait complete. 2023-12-19T14:55:19.405 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:19.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:19.434 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.425+0000 7f5d75bce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:19.514 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.505+0000 7f5d75bce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:19.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 Processor -- start 2023-12-19T14:55:19.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 -- start start 2023-12-19T14:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 0x7f1b04067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 0x7f1b040cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b04068c70 con 0x7f1b04159340 2023-12-19T14:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b04068060 con 0x7f1b04158f70 2023-12-19T14:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b09345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b04068340 con 0x7f1b04069100 2023-12-19T14:55:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b02ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 0x7f1b04067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b037fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1b04159340 0x7f1b040cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36516/0 (socket says 172.21.15.50:36516) 2023-12-19T14:55:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b037fe700 1 -- 172.21.15.50:0/2872911365 learned_addr learned my addr 172.21.15.50:0/2872911365 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2872911365 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 0x7f1b040cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b01ffb700 1 -- 172.21.15.50:0/2872911365 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613663842 0 0) 0x7f1b04068c70 con 0x7f1b04159340 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.582+0000 7f1b01ffb700 1 -- 172.21.15.50:0/2872911365 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aec003570 con 0x7f1b04159340 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b027fc700 1 -- 172.21.15.50:0/2872911365 >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 msgr2=0x7f1b04067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2872911365 >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 0x7f1b04067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b027fc700 1 -- 172.21.15.50:0/2872911365 >> v1:172.21.15.125:6789/0 conn(0x7f1b04159340 legacy=0x7f1b040cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b027fc700 1 -- 172.21.15.50:0/2872911365 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04068620 con 0x7f1b04069100 2023-12-19T14:55:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b02ffd700 1 --2- 172.21.15.50:0/2872911365 >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 0x7f1b04067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2872911365 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 0x7f1b040cf7e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1af80098e0 tx=0x7f1af80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46e4677c5efbad28 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b01ffb700 1 -- 172.21.15.50:0/2872911365 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8011630 con 0x7f1b04069100 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b01ffb700 1 -- 172.21.15.50:0/2872911365 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8011790 con 0x7f1b04069100 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b01ffb700 1 -- 172.21.15.50:0/2872911365 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8011b30 con 0x7f1b04069100 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 -- 172.21.15.50:0/2872911365 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 msgr2=0x7f1b040cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 --2- 172.21.15.50:0/2872911365 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 0x7f1b040cf7e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1af80098e0 tx=0x7f1af80091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 -- 172.21.15.50:0/2872911365 shutdown_connections 2023-12-19T14:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 --2- 172.21.15.50:0/2872911365 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04069100 0x7f1b040cf7e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 --2- 172.21.15.50:0/2872911365 >> v2:172.21.15.50:3300/0 conn(0x7f1b04158f70 0x7f1b04067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 -- 172.21.15.50:0/2872911365 >> 172.21.15.50:0/2872911365 conn(0x7f1b040ccc40 msgr2=0x7f1b040cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 -- 172.21.15.50:0/2872911365 shutdown_connections 2023-12-19T14:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.586+0000 7f1b09345700 1 -- 172.21.15.50:0/2872911365 wait complete. 2023-12-19T14:55:19.590 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.581+0000 7f5d75bce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:19.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 Processor -- start 2023-12-19T14:55:19.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- start start 2023-12-19T14:55:19.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 0x7f1b040c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 0x7f1b04156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040b5e60 con 0x7f1b04069100 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040b5690 con 0x7f1b04159340 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0415bf30 con 0x7f1b04158f70 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59064/0 (socket says 172.21.15.50:59064) 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 -- 172.21.15.50:0/2742205980 learned_addr learned my addr 172.21.15.50:0/2742205980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 0x7f1b040c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b037fe700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 0x7f1b04156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 -- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 msgr2=0x7f1b040c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 0x7f1b040c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 -- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 msgr2=0x7f1b04156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 0x7f1b04156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 -- 172.21.15.50:0/2742205980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4009420 con 0x7f1b04069100 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 0x7f1b040c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b037fe700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 0x7f1b04156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b02ffd700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1af400ec80 tx=0x7f1af40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af400c9d0 con 0x7f1b04069100 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af400cb30 con 0x7f1b04069100 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af400ceb0 con 0x7f1b04069100 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af8009420 con 0x7f1b04069100 2023-12-19T14:55:19.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.590+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b040b5b80 con 0x7f1b04069100 2023-12-19T14:55:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.594+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b04071120 con 0x7f1b04069100 2023-12-19T14:55:19.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.594+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f1af4013680 con 0x7f1b04069100 2023-12-19T14:55:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.594+0000 7f1aeaffd700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 0x7f1af0062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.598+0000 7f1b027fc700 1 -- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 msgr2=0x7f1af0062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:19.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.598+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1af407d1f0 con 0x7f1b04069100 2023-12-19T14:55:19.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.598+0000 7f1b027fc700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 0x7f1af0062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:19.601 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:19.598+0000 7f2fcc92c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:19.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.602+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1af404d210 con 0x7f1b04069100 2023-12-19T14:55:19.700 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:19.698+0000 7f2fcc92c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:19.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.774+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b04068900 con 0x7f1b04069100 2023-12-19T14:55:19.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.774+0000 7f1aeaffd700 1 -- 172.21.15.50:0/2742205980 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f1af4050830 con 0x7f1b04069100 2023-12-19T14:55:19.778 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:19.779 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":101,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:14.394534+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3252761303":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2020877475":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2604193437":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/812779881":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3729319264":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3000325649":"2023-12-20T14:55:14.394463+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 msgr2=0x7f1af0062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 0x7f1af0062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 msgr2=0x7f1b040b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f1af400ec80 tx=0x7f1af40091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 shutdown_connections 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b04159340 0x7f1b04156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1af0060730 0x7f1af0062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1b04158f70 0x7f1b040c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 --2- 172.21.15.50:0/2742205980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1b04069100 0x7f1b040b4ec0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 >> 172.21.15.50:0/2742205980 conn(0x7f1b040ccc40 msgr2=0x7f1b041579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 shutdown_connections 2023-12-19T14:55:19.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.778+0000 7f1b09345700 1 -- 172.21.15.50:0/2742205980 wait complete. 2023-12-19T14:55:19.786 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:19.782+0000 7f2fcc92c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:19.802 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:55:19.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:19.804 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.797+0000 7f5d75bce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:19.873 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:19.870+0000 7f2fcc92c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:19.886 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.877+0000 7f5d75bce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:19.969 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:19.961+0000 7f5d75bce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:19.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.970+0000 7fe4822f0700 1 Processor -- start 2023-12-19T14:55:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.970+0000 7fe4822f0700 1 -- start start 2023-12-19T14:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe4822f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 0x7fe47c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe4822f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe47c0c5900 0x7fe47c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe4822f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47c0c6630 con 0x7fe47c067c90 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe4822f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47c0c6910 con 0x7fe47c0c5900 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe4822f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47c0c6bf0 con 0x7fe47c1502b0 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe47c0c5900 0x7fe47c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe47bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe47c067c90 0x7fe47c14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45636/0 (socket says 172.21.15.50:45636) 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe47c0c5900 0x7fe47c0c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59066/0 (socket says 172.21.15.50:59066) 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe47bfff700 1 -- 172.21.15.50:0/1706499600 learned_addr learned my addr 172.21.15.50:0/1706499600 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe473fff700 1 --2- 172.21.15.50:0/1706499600 >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 0x7fe47c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 -- 172.21.15.50:0/1706499600 >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 msgr2=0x7fe47c068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 --2- 172.21.15.50:0/1706499600 >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 0x7fe47c068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 -- 172.21.15.50:0/1706499600 >> v1:172.21.15.50:6789/0 conn(0x7fe47c067c90 legacy=0x7fe47c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 -- 172.21.15.50:0/1706499600 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0c6ed0 con 0x7fe47c0c5900 2023-12-19T14:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe473fff700 1 --2- 172.21.15.50:0/1706499600 >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 0x7fe47c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe48088d700 1 --2- 172.21.15.50:0/1706499600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c0c5d60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0c56b0 tx=0x7fe474009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ede1437108cad0c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe47b7fe700 1 -- 172.21.15.50:0/1706499600 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47400e070 con 0x7fe47c0c5900 2023-12-19T14:55:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe47b7fe700 1 -- 172.21.15.50:0/1706499600 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47400bc30 con 0x7fe47c0c5900 2023-12-19T14:55:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.974+0000 7fe47b7fe700 1 -- 172.21.15.50:0/1706499600 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474012430 con 0x7fe47c0c5900 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 -- 172.21.15.50:0/1706499600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 msgr2=0x7fe47c0c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 --2- 172.21.15.50:0/1706499600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c0c5d60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0c56b0 tx=0x7fe474009780 comp rx=0 tx=0).stop 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 -- 172.21.15.50:0/1706499600 shutdown_connections 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 --2- 172.21.15.50:0/1706499600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c0c5d60 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 --2- 172.21.15.50:0/1706499600 >> v2:172.21.15.125:3300/0 conn(0x7fe47c1502b0 0x7fe47c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 -- 172.21.15.50:0/1706499600 >> 172.21.15.50:0/1706499600 conn(0x7fe47c0cd6d0 msgr2=0x7fe47c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 -- 172.21.15.50:0/1706499600 shutdown_connections 2023-12-19T14:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 -- 172.21.15.50:0/1706499600 wait complete. 2023-12-19T14:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.978+0000 7fe4822f0700 1 Processor -- start 2023-12-19T14:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- start start 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe47c067c90 0x7fe47c0b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47c1502b0 0x7fe47c0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c152ac0 con 0x7fe47c0c5900 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0b1da0 con 0x7fe47c1502b0 2023-12-19T14:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0b4740 con 0x7fe47c067c90 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59078/0 (socket says 172.21.15.50:59078) 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 -- 172.21.15.50:0/3881793912 learned_addr learned my addr 172.21.15.50:0/3881793912 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe48088d700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47c1502b0 0x7fe47c0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe47bfff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe47c067c90 0x7fe47c0b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe47c067c90 msgr2=0x7fe47c0b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe47c067c90 0x7fe47c0b1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47c1502b0 msgr2=0x7fe47c0b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47c1502b0 0x7fe47c0b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 -- 172.21.15.50:0/3881793912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474009a20 con 0x7fe47c0c5900 2023-12-19T14:55:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe473fff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe46400d860 tx=0x7fe464008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640125f0 con 0x7fe47c0c5900 2023-12-19T14:55:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464012750 con 0x7fe47c0c5900 2023-12-19T14:55:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47c152470 con 0x7fe47c0c5900 2023-12-19T14:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464012ad0 con 0x7fe47c0c5900 2023-12-19T14:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.982+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47c152160 con 0x7fe47c0c5900 2023-12-19T14:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47c152c30 con 0x7fe47c0c5900 2023-12-19T14:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fe464014430 con 0x7fe47c0c5900 2023-12-19T14:55:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe478ff9700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 0x7fe45c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe46407d1f0 con 0x7fe47c0c5900 2023-12-19T14:55:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe47bfff700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 msgr2=0x7fe45c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.986+0000 7fe47bfff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 0x7fe45c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:19.990+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe464050d40 con 0x7fe47c0c5900 2023-12-19T14:55:20.121 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:20.118+0000 7f2fcc92c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:20.159 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:20.153+0000 7f5d75bce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.190+0000 7fe47bfff700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 msgr2=0x7fe45c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.190+0000 7fe47bfff700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 0x7fe45c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:20.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.202+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe47c071120 con 0x7fe47c0c5900 2023-12-19T14:55:20.204 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:20.202+0000 7f2fcc92c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:20.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.202+0000 7fe478ff9700 1 -- 172.21.15.50:0/3881793912 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fe4640508d0 con 0x7fe47c0c5900 2023-12-19T14:55:20.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:20.205 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 msgr2=0x7fe45c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 0x7fe45c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 msgr2=0x7fe47c14ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe46400d860 tx=0x7fe464008040 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 shutdown_connections 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe47c1502b0 0x7fe47c0b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe45c060700 0x7fe45c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe47c0c5900 0x7fe47c14ed70 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 --2- 172.21.15.50:0/3881793912 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe47c067c90 0x7fe47c0b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 >> 172.21.15.50:0/3881793912 conn(0x7fe47c0cd6d0 msgr2=0x7fe47c0ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:20.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 shutdown_connections 2023-12-19T14:55:20.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.206+0000 7fe4822f0700 1 -- 172.21.15.50:0/3881793912 wait complete. 2023-12-19T14:55:20.212 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:20.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:20.244 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:20.237+0000 7f5d75bce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:20.299 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:20.294+0000 7f2fcc92c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 Processor -- start 2023-12-19T14:55:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 -- start start 2023-12-19T14:55:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf0c0c5900 0x7fbf0c0c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0c6630 con 0x7fbf0c14fba0 2023-12-19T14:55:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c0c6910 con 0x7fbf0c0c5900 2023-12-19T14:55:20.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.394+0000 7fbf10985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0c6bf0 con 0x7fbf0c067c90 2023-12-19T14:55:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf09d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbf0c14fba0 0x7fbf0c14ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45650/0 (socket says 172.21.15.50:45650) 2023-12-19T14:55:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf09d9b700 1 -- 172.21.15.50:0/3004001682 learned_addr learned my addr 172.21.15.50:0/3004001682 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:20.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0ad9d700 1 --2- 172.21.15.50:0/3004001682 >> v2:172.21.15.125:3300/0 conn(0x7fbf0c0c5900 0x7fbf0c0c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 -- 172.21.15.50:0/3004001682 >> v2:172.21.15.125:3300/0 conn(0x7fbf0c0c5900 msgr2=0x7fbf0c0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 --2- 172.21.15.50:0/3004001682 >> v2:172.21.15.125:3300/0 conn(0x7fbf0c0c5900 0x7fbf0c0c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 -- 172.21.15.50:0/3004001682 >> v1:172.21.15.50:6789/0 conn(0x7fbf0c14fba0 legacy=0x7fbf0c14ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 -- 172.21.15.50:0/3004001682 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c0c6ed0 con 0x7fbf0c067c90 2023-12-19T14:55:20.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0959a700 1 -- 172.21.15.50:0/3004001682 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756514338 0 0) 0x7fbf0c0c6630 con 0x7fbf0c14fba0 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0a59c700 1 --2- 172.21.15.50:0/3004001682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c068060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009e60 tx=0x7fbefc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4858ad8cca7260eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0959a700 1 -- 172.21.15.50:0/3004001682 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00e070 con 0x7fbf0c067c90 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0959a700 1 -- 172.21.15.50:0/3004001682 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc00bb60 con 0x7fbf0c067c90 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf0959a700 1 -- 172.21.15.50:0/3004001682 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc012430 con 0x7fbf0c067c90 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 -- 172.21.15.50:0/3004001682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 msgr2=0x7fbf0c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 --2- 172.21.15.50:0/3004001682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c068060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009e60 tx=0x7fbefc009780 comp rx=0 tx=0).stop 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 -- 172.21.15.50:0/3004001682 shutdown_connections 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 --2- 172.21.15.50:0/3004001682 >> v2:172.21.15.125:3300/0 conn(0x7fbf0c0c5900 0x7fbf0c0c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 --2- 172.21.15.50:0/3004001682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c068060 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 -- 172.21.15.50:0/3004001682 >> 172.21.15.50:0/3004001682 conn(0x7fbf0c0cd6d0 msgr2=0x7fbf0c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 -- 172.21.15.50:0/3004001682 shutdown_connections 2023-12-19T14:55:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.398+0000 7fbf10985700 1 -- 172.21.15.50:0/3004001682 wait complete. 2023-12-19T14:55:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.402+0000 7fbf10985700 1 Processor -- start 2023-12-19T14:55:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.402+0000 7fbf10985700 1 -- start start 2023-12-19T14:55:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 0x7fbf0c150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b44c0 con 0x7fbf0c14fba0 2023-12-19T14:55:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b4760 con 0x7fbf0c0c5900 2023-12-19T14:55:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b4a40 con 0x7fbf0c067c90 2023-12-19T14:55:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59084/0 (socket says 172.21.15.50:59084) 2023-12-19T14:55:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 -- 172.21.15.50:0/2627569169 learned_addr learned my addr 172.21.15.50:0/2627569169 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf09d9b700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 0x7fbf0c150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0a59c700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 msgr2=0x7fbf0c0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 msgr2=0x7fbf0c150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 0x7fbf0c150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 -- 172.21.15.50:0/2627569169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc009a20 con 0x7fbf0c14fba0 2023-12-19T14:55:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf09d9b700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 0x7fbf0c150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf0ad9d700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400ea50 tx=0x7fbf040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0400c920 con 0x7fbf0c14fba0 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c0b4d20 con 0x7fbf0c14fba0 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0400caa0 con 0x7fbf0c14fba0 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0400ce60 con 0x7fbf0c14fba0 2023-12-19T14:55:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.406+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c0b52e0 con 0x7fbf0c14fba0 2023-12-19T14:55:20.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c0b5500 con 0x7fbf0c14fba0 2023-12-19T14:55:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fbf04013680 con 0x7fbf0c14fba0 2023-12-19T14:55:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbefa7fc700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 0x7fbeec062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbf0407d2c0 con 0x7fbf0c14fba0 2023-12-19T14:55:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbf0a59c700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 msgr2=0x7fbeec062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:20.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.410+0000 7fbf0a59c700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 0x7fbeec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.418+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf0404d2e0 con 0x7fbf0c14fba0 2023-12-19T14:55:20.525 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:20.522+0000 7f2fcc92c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.614+0000 7fbf0a59c700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 msgr2=0x7fbeec062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:20.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.614+0000 7fbf0a59c700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 0x7fbeec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:20.672 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:20.618+0000 7f2fcc92c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:20.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.626+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0c071120 con 0x7fbf0c14fba0 2023-12-19T14:55:20.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbefa7fc700 1 -- 172.21.15.50:0/2627569169 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fbf04050900 con 0x7fbf0c14fba0 2023-12-19T14:55:20.673 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:20.673 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:54:58.259130+0000","modified":"2023-12-19T14:55:01.972807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6936},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6936":{"gid":6936,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/2756457211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2756457211},{"type":"v1","addr":"172.21.15.125:6836","nonce":2756457211}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T14:55:20.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 msgr2=0x7fbeec062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:20.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 0x7fbeec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 msgr2=0x7fbf0c150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400ea50 tx=0x7fbf040091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 shutdown_connections 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf0c0c5900 0x7fbf0c150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbeec060700 0x7fbeec062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf0c14fba0 0x7fbf0c150ca0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 --2- 172.21.15.50:0/2627569169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf0c067c90 0x7fbf0c0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 >> 172.21.15.50:0/2627569169 conn(0x7fbf0c0cd6d0 msgr2=0x7fbf0c06d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 shutdown_connections 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.630+0000 7fbf10985700 1 -- 172.21.15.50:0/2627569169 wait complete. 2023-12-19T14:55:20.675 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 41 2023-12-19T14:55:20.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 Processor -- start 2023-12-19T14:55:20.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 -- start start 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 0x7f3cac0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cac140ae0 0x7f3cac0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac06e150 con 0x7f3cac0c75e0 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac06d540 con 0x7f3cac140ae0 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cb162e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac06d820 con 0x7f3cac140710 2023-12-19T14:55:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cab7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3cac0c75e0 0x7f3cac06a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45664/0 (socket says 172.21.15.50:45664) 2023-12-19T14:55:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3cab7fe700 1 -- 172.21.15.50:0/1164693765 learned_addr learned my addr 172.21.15.50:0/1164693765 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3caaffd700 1 --2- 172.21.15.50:0/1164693765 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 0x7f3cac0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3caa7fc700 1 --2- 172.21.15.50:0/1164693765 >> v2:172.21.15.125:3300/0 conn(0x7f3cac140ae0 0x7f3cac0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3ca9ffb700 1 -- 172.21.15.50:0/1164693765 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905921105 0 0) 0x7f3cac06e150 con 0x7f3cac0c75e0 2023-12-19T14:55:20.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3ca9ffb700 1 -- 172.21.15.50:0/1164693765 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c94003570 con 0x7f3cac0c75e0 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3caaffd700 1 -- 172.21.15.50:0/1164693765 >> v2:172.21.15.125:3300/0 conn(0x7f3cac140ae0 msgr2=0x7f3cac0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.838+0000 7f3caaffd700 1 --2- 172.21.15.50:0/1164693765 >> v2:172.21.15.125:3300/0 conn(0x7f3cac140ae0 0x7f3cac0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3caaffd700 1 -- 172.21.15.50:0/1164693765 >> v1:172.21.15.50:6789/0 conn(0x7f3cac0c75e0 legacy=0x7f3cac06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3caaffd700 1 -- 172.21.15.50:0/1164693765 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac06db00 con 0x7f3cac140710 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3caaffd700 1 --2- 172.21.15.50:0/1164693765 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 0x7f3cac0c70a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c0098e0 tx=0x7f3c9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625c53652d5e7755 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:20.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3ca9ffb700 1 -- 172.21.15.50:0/1164693765 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c011630 con 0x7f3cac140710 2023-12-19T14:55:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3ca9ffb700 1 -- 172.21.15.50:0/1164693765 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9c011790 con 0x7f3cac140710 2023-12-19T14:55:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3ca9ffb700 1 -- 172.21.15.50:0/1164693765 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c011b30 con 0x7f3cac140710 2023-12-19T14:55:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 -- 172.21.15.50:0/1164693765 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 msgr2=0x7f3cac0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 --2- 172.21.15.50:0/1164693765 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 0x7f3cac0c70a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c0098e0 tx=0x7f3c9c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 -- 172.21.15.50:0/1164693765 shutdown_connections 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 --2- 172.21.15.50:0/1164693765 >> v2:172.21.15.125:3300/0 conn(0x7f3cac140ae0 0x7f3cac0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 --2- 172.21.15.50:0/1164693765 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac140710 0x7f3cac0c70a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 -- 172.21.15.50:0/1164693765 >> 172.21.15.50:0/1164693765 conn(0x7f3cac0c8850 msgr2=0x7f3cac0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 -- 172.21.15.50:0/1164693765 shutdown_connections 2023-12-19T14:55:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.842+0000 7f3cb162e700 1 -- 172.21.15.50:0/1164693765 wait complete. 2023-12-19T14:55:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 Processor -- start 2023-12-19T14:55:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 -- start start 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac0c75e0 0x7f3cac0aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cac140710 0x7f3cac156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0c1c70 con 0x7f3cac140ae0 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac158530 con 0x7f3cac140710 2023-12-19T14:55:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac1587f0 con 0x7f3cac0c75e0 2023-12-19T14:55:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59088/0 (socket says 172.21.15.50:59088) 2023-12-19T14:55:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 -- 172.21.15.50:0/2207053515 learned_addr learned my addr 172.21.15.50:0/2207053515 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3caaffd700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac0c75e0 0x7f3cac0aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3caa7fc700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cac140710 0x7f3cac156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 -- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac0c75e0 msgr2=0x7f3cac0aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac0c75e0 0x7f3cac0aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 -- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cac140710 msgr2=0x7f3cac156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cac140710 0x7f3cac156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 -- 172.21.15.50:0/2207053515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c009420 con 0x7f3cac140ae0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cab7fe700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400cc40 tx=0x7f3ca4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca400a830 con 0x7f3cac140ae0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca400a990 con 0x7f3cac140ae0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac157250 con 0x7f3cac140ae0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.846+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca400ad10 con 0x7f3cac140ae0 2023-12-19T14:55:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.850+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac0c16b0 con 0x7f3cac140ae0 2023-12-19T14:55:20.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.850+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac071120 con 0x7f3cac140ae0 2023-12-19T14:55:20.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.850+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 37) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f3ca4007300 con 0x7f3cac140ae0 2023-12-19T14:55:20.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.854+0000 7f3c92ffd700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 0x7f3c98062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:20.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.854+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3ca407d1d0 con 0x7f3cac140ae0 2023-12-19T14:55:20.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.854+0000 7f3caaffd700 1 -- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 msgr2=0x7f3c98062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:20.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.854+0000 7f3caaffd700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 0x7f3c98062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:20.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.858+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3ca404d1e0 con 0x7f3cac140ae0 2023-12-19T14:55:20.957 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:20.949+0000 7f5d75bce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:20.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.974+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 38) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f3ca404d5b0 con 0x7f3cac140ae0 2023-12-19T14:55:20.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.974+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 msgr2=0x7f3c98062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:20.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:20.974+0000 7f3c92ffd700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 0x7f3c98062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.046+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cac158930 con 0x7f3cac140ae0 2023-12-19T14:55:21.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.046+0000 7f3c92ffd700 1 -- 172.21.15.50:0/2207053515 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f3ca4078d40 con 0x7f3cac140ae0 2023-12-19T14:55:21.050 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:21.050 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":102,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:20.963479+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:54:56.099031+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T14:54:57.248377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3676580286":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/505904512":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/1693906057":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/1114598693":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/2297448116":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/4070416647":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/3252761303":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2020877475":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2604193437":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/812779881":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3729319264":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2885889354":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/3000325649":"2023-12-20T14:55:14.394463+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 msgr2=0x7f3cac0c0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400cc40 tx=0x7f3ca4001a40 comp rx=0 tx=0).stop 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 shutdown_connections 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cac140710 0x7f3cac156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c98060730 0x7f3c98062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3cac140ae0 0x7f3cac0c0b80 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 --2- 172.21.15.50:0/2207053515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3cac0c75e0 0x7f3cac0aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 >> 172.21.15.50:0/2207053515 conn(0x7f3cac0c8850 msgr2=0x7f3cac0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 shutdown_connections 2023-12-19T14:55:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.050+0000 7f3cb162e700 1 -- 172.21.15.50:0/2207053515 wait complete. 2023-12-19T14:55:21.067 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d40ee8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d40ee8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d40ee8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d40ee8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d40ee8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d3cee0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d3cee0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d3cee0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.068 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d3cee0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.069 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:21.061+0000 7f5d3cee0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:21.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:55:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.234+0000 7fbed1e78700 1 Processor -- start 2023-12-19T14:55:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.234+0000 7fbed1e78700 1 -- start start 2023-12-19T14:55:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbed1e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbed1e78700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbecc0b3870 0x7fbecc0b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbed1e78700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc06b0d0 con 0x7fbecc06dc00 2023-12-19T14:55:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbed1e78700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc0b4790 con 0x7fbecc0b3870 2023-12-19T14:55:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbed1e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc06d710 con 0x7fbecc158cc0 2023-12-19T14:55:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecb7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbecc0b3870 0x7fbecc0b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecb7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35940/0 (socket says 172.21.15.50:35940) 2023-12-19T14:55:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecb7fe700 1 -- 172.21.15.50:0/2826090917 learned_addr learned my addr 172.21.15.50:0/2826090917 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 -- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 msgr2=0x7fbecc06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 -- 172.21.15.50:0/2826090917 >> v1:172.21.15.125:6789/0 conn(0x7fbecc06dc00 legacy=0x7fbecc06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 -- 172.21.15.50:0/2826090917 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0b41c0 con 0x7fbecc0b3870 2023-12-19T14:55:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecb7fe700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbecbfff700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc0b3870 0x7fbecc0b3c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009e60 tx=0x7fbec4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aea077e897c4353 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbeca7fc700 1 -- 172.21.15.50:0/2826090917 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec400e070 con 0x7fbecc0b3870 2023-12-19T14:55:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbeca7fc700 1 -- 172.21.15.50:0/2826090917 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec400bb60 con 0x7fbecc0b3870 2023-12-19T14:55:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.238+0000 7fbeca7fc700 1 -- 172.21.15.50:0/2826090917 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4012430 con 0x7fbecc0b3870 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 -- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc0b3870 msgr2=0x7fbecc0b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc0b3870 0x7fbecc0b3c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009e60 tx=0x7fbec4009780 comp rx=0 tx=0).stop 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 -- 172.21.15.50:0/2826090917 shutdown_connections 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc0b3870 0x7fbecc0b3c80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 --2- 172.21.15.50:0/2826090917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc158cc0 0x7fbecc06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 -- 172.21.15.50:0/2826090917 >> 172.21.15.50:0/2826090917 conn(0x7fbecc067bb0 msgr2=0x7fbecc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:21.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 -- 172.21.15.50:0/2826090917 shutdown_connections 2023-12-19T14:55:21.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.242+0000 7fbed1e78700 1 -- 172.21.15.50:0/2826090917 wait complete. 2023-12-19T14:55:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 Processor -- start 2023-12-19T14:55:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 -- start start 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc06dc00 0x7fbecc15f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc0b3870 0x7fbecc14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15ee60 con 0x7fbecc158cc0 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15e2e0 con 0x7fbecc06dc00 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbed1e78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15d900 con 0x7fbecc0b3870 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59090/0 (socket says 172.21.15.50:59090) 2023-12-19T14:55:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 -- 172.21.15.50:0/3346127579 learned_addr learned my addr 172.21.15.50:0/3346127579 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:21.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecb7fe700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc06dc00 0x7fbecc15f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecaffd700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc0b3870 0x7fbecc14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 -- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc0b3870 msgr2=0x7fbecc14e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc0b3870 0x7fbecc14e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 -- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc06dc00 msgr2=0x7fbecc15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc06dc00 0x7fbecc15f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 -- 172.21.15.50:0/3346127579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0094f0 con 0x7fbecc158cc0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.246+0000 7fbecbfff700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbecc06eb60 tx=0x7fbec4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4012ea0 con 0x7fbecc158cc0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec4027470 con 0x7fbecc158cc0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4009a20 con 0x7fbecc158cc0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4027850 con 0x7fbecc158cc0 2023-12-19T14:55:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc15e810 con 0x7fbecc158cc0 2023-12-19T14:55:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 38) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fbec4024070 con 0x7fbecc158cc0 2023-12-19T14:55:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc15f010 con 0x7fbecc158cc0 2023-12-19T14:55:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.250+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6698+0+0 (secure 0 0 0) 0x7fbec407f0f0 con 0x7fbecc158cc0 2023-12-19T14:55:21.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.258+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbecc15f010 con 0x7fbecc158cc0 2023-12-19T14:55:21.498 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:21.494+0000 7f2fcc92c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:21.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.510+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbecc15f010 con 0x7fbecc158cc0 2023-12-19T14:55:21.513 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:55:21.510+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:55:21.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.518+0000 7fbeb37fe700 1 -- 172.21.15.50:0/3346127579 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbecc15f010 con 0x7fbecc158cc0 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 msgr2=0x7fbecc15d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbecc06eb60 tx=0x7fbec4019000 comp rx=0 tx=0).stop 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 shutdown_connections 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc158cc0 0x7fbecc15d3c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbecc0b3870 0x7fbecc14e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 --2- 172.21.15.50:0/3346127579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbecc06dc00 0x7fbecc15f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 >> 172.21.15.50:0/3346127579 conn(0x7fbecc067bb0 msgr2=0x7fbecc15bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 shutdown_connections 2023-12-19T14:55:21.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.522+0000 7fbed1e78700 1 -- 172.21.15.50:0/3346127579 wait complete. 2023-12-19T14:55:21.526 INFO:tasks.ceph.mds.d.smithi125.stderr: -2554> 2023-12-19T14:54:52.898+0000 7fe9523df700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6835/0: (98) Address already in use 2023-12-19T14:55:21.527 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:55:21.529 INFO:tasks.ceph.mds.d.smithi125.stderr: -1381> 2023-12-19T14:55:03.742+0000 7fe94c3d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:55:21.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:55:21.572 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:55:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.718+0000 7f1feec78700 1 Processor -- start 2023-12-19T14:55:21.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.718+0000 7f1feec78700 1 -- start start 2023-12-19T14:55:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fe8158e20 0x7f1fe80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1fe80cb970 0x7f1fe80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80cbdf0 con 0x7f1fe80cb970 2023-12-19T14:55:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80cc0d0 con 0x7f1fe8158e20 2023-12-19T14:55:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feec78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80cc3b0 con 0x7f1fe80ca2c0 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1fe80cb970 0x7f1fe80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1fe80cb970 0x7f1fe80ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59100/0 (socket says 172.21.15.50:59100) 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 -- 172.21.15.50:0/748969562 learned_addr learned my addr 172.21.15.50:0/748969562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fed215700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1feca14700 1 --2- 172.21.15.50:0/748969562 >> v2:172.21.15.125:3300/0 conn(0x7f1fe8158e20 0x7f1fe80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 -- 172.21.15.50:0/748969562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 msgr2=0x7f1fe806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 -- 172.21.15.50:0/748969562 >> v2:172.21.15.125:3300/0 conn(0x7f1fe8158e20 msgr2=0x7f1fe80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/748969562 >> v2:172.21.15.125:3300/0 conn(0x7f1fe8158e20 0x7f1fe80ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 -- 172.21.15.50:0/748969562 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80cc690 con 0x7f1fe80cb970 2023-12-19T14:55:21.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe80cb970 0x7f1fe80ceb00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40099b0 tx=0x7f1fd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc54761ac8dd153d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe77fe700 1 -- 172.21.15.50:0/748969562 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400b640 con 0x7f1fe80cb970 2023-12-19T14:55:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe77fe700 1 -- 172.21.15.50:0/748969562 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd400b7a0 con 0x7f1fe80cb970 2023-12-19T14:55:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.722+0000 7f1fe77fe700 1 -- 172.21.15.50:0/748969562 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400bb40 con 0x7f1fe80cb970 2023-12-19T14:55:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 -- 172.21.15.50:0/748969562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe80cb970 msgr2=0x7f1fe80ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe80cb970 0x7f1fe80ceb00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40099b0 tx=0x7f1fd4009250 comp rx=0 tx=0).stop 2023-12-19T14:55:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 -- 172.21.15.50:0/748969562 shutdown_connections 2023-12-19T14:55:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe80cb970 0x7f1fe80ceb00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 --2- 172.21.15.50:0/748969562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 --2- 172.21.15.50:0/748969562 >> v2:172.21.15.125:3300/0 conn(0x7f1fe8158e20 0x7f1fe80ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 -- 172.21.15.50:0/748969562 >> 172.21.15.50:0/748969562 conn(0x7f1fe8067bb0 msgr2=0x7f1fe80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 -- 172.21.15.50:0/748969562 shutdown_connections 2023-12-19T14:55:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.726+0000 7f1feec78700 1 -- 172.21.15.50:0/748969562 wait complete. 2023-12-19T14:55:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 Processor -- start 2023-12-19T14:55:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 -- start start 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe814b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe80c7f60 0x7f1fe80cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c93e0 con 0x7f1fe8158e20 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c87d0 con 0x7f1fe80c7f60 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feec78700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c8ab0 con 0x7f1fe80ca2c0 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59102/0 (socket says 172.21.15.50:59102) 2023-12-19T14:55:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 -- 172.21.15.50:0/2022923475 learned_addr learned my addr 172.21.15.50:0/2022923475 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feca14700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe814b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fed215700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe80c7f60 0x7f1fe80cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 -- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 msgr2=0x7f1fe814b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe814b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 -- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe80c7f60 msgr2=0x7f1fe80cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe80c7f60 0x7f1fe80cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1fe7fff700 1 -- 172.21.15.50:0/2022923475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd40094f0 con 0x7f1fe8158e20 2023-12-19T14:55:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.730+0000 7f1feca14700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe814b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1fe7fff700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009fd0 tx=0x7f1fd4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400bb20 con 0x7f1fe8158e20 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd4011420 con 0x7f1fe8158e20 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40117a0 con 0x7f1fe8158e20 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe80c8d90 con 0x7f1fe8158e20 2023-12-19T14:55:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe814b270 con 0x7f1fe8158e20 2023-12-19T14:55:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe80c9550 con 0x7f1fe8158e20 2023-12-19T14:55:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.734+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 38) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f1fd4011bd0 con 0x7f1fe8158e20 2023-12-19T14:55:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.738+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f1fd407dfe0 con 0x7f1fe8158e20 2023-12-19T14:55:21.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.742+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1fd404e4d0 con 0x7f1fe8158e20 2023-12-19T14:55:21.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.954+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1fe806a7a0 con 0x7f1fe8158e20 2023-12-19T14:55:21.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.978+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 39) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f1fd404df80 con 0x7f1fe8158e20 2023-12-19T14:55:21.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.982+0000 7f1fe4ff9700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 0x7f1fcc0687d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:21.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.982+0000 7f1feca14700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 0x7f1fcc0687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:21.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:21.986+0000 7f1feca14700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 0x7f1fcc0687d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009ce0 tx=0x7f1fdc008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:22.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.522+0000 7f1fe4ff9700 1 -- 172.21.15.50:0/2022923475 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1fd40253f0 con 0x7f1fe8158e20 2023-12-19T14:55:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 msgr2=0x7f1fcc0687d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 0x7f1fcc0687d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009ce0 tx=0x7f1fdc008040 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 msgr2=0x7f1fe814ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009fd0 tx=0x7f1fd4017000 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 shutdown_connections 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe80c7f60 0x7f1fe80cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1fcc0663f0 0x7f1fcc0687d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fe8158e20 0x7f1fe814ad30 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 --2- 172.21.15.50:0/2022923475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1fe80ca2c0 0x7f1fe814b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 >> 172.21.15.50:0/2022923475 conn(0x7f1fe8067bb0 msgr2=0x7f1fe80c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 shutdown_connections 2023-12-19T14:55:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.526+0000 7f1feec78700 1 -- 172.21.15.50:0/2022923475 wait complete. 2023-12-19T14:55:22.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:55:22.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 Processor -- start 2023-12-19T14:55:22.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 -- start start 2023-12-19T14:55:22.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f380af190 0x7f2f3806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f380af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f380cb3f0 con 0x7f2f380aedc0 2023-12-19T14:55:22.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f380cc140 con 0x7f2f380af190 2023-12-19T14:55:22.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.729+0000 7f2f37fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380af610 con 0x7f2f380cac50 2023-12-19T14:55:22.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f36ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2f380aedc0 0x7f2f3806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45680/0 (socket says 172.21.15.50:45680) 2023-12-19T14:55:22.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f36ffd700 1 -- 172.21.15.50:0/4048662870 learned_addr learned my addr 172.21.15.50:0/4048662870 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:22.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/4048662870 >> v2:172.21.15.125:3300/0 conn(0x7f2f380af190 0x7f2f3806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f377fe700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f380af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 -- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 msgr2=0x7f2f380af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:22.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f380af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 -- 172.21.15.50:0/4048662870 >> v1:172.21.15.50:6789/0 conn(0x7f2f380aedc0 legacy=0x7f2f3806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 -- 172.21.15.50:0/4048662870 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3806d220 con 0x7f2f380af190 2023-12-19T14:55:22.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f3806af50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009e60 tx=0x7f2f20009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8c2667d42a75805 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:22.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f367fc700 1 -- 172.21.15.50:0/4048662870 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000e070 con 0x7f2f380af190 2023-12-19T14:55:22.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f367fc700 1 -- 172.21.15.50:0/4048662870 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000bb60 con 0x7f2f380af190 2023-12-19T14:55:22.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f367fc700 1 -- 172.21.15.50:0/4048662870 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20012430 con 0x7f2f380af190 2023-12-19T14:55:22.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f37fff700 1 -- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 msgr2=0x7f2f3806af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.733+0000 7f2f37fff700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f3806af50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009e60 tx=0x7f2f20009780 comp rx=0 tx=0).stop 2023-12-19T14:55:22.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 -- 172.21.15.50:0/4048662870 shutdown_connections 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f380af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 --2- 172.21.15.50:0/4048662870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f3806af50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 -- 172.21.15.50:0/4048662870 >> 172.21.15.50:0/4048662870 conn(0x7f2f380cc760 msgr2=0x7f2f380ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 -- 172.21.15.50:0/4048662870 shutdown_connections 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 -- 172.21.15.50:0/4048662870 wait complete. 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 Processor -- start 2023-12-19T14:55:22.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.737+0000 7f2f37fff700 1 -- start start 2023-12-19T14:55:22.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f38169350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f38169890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380c74d0 con 0x7f2f380aedc0 2023-12-19T14:55:22.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380b5860 con 0x7f2f380af190 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f37fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f380c6e80 con 0x7f2f380cac50 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59116/0 (socket says 172.21.15.50:59116) 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 -- 172.21.15.50:0/1575724189 learned_addr learned my addr 172.21.15.50:0/1575724189 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f377fe700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f38169890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f38169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 -- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 msgr2=0x7f2f38169890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f38169890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 -- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 msgr2=0x7f2f38169350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f38169350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 -- 172.21.15.50:0/1575724189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20009a20 con 0x7f2f380aedc0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f36ffd700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800c560 tx=0x7f2f28001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.741+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28005b30 con 0x7f2f380aedc0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.745+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f28005cb0 con 0x7f2f380aedc0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.745+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28007320 con 0x7f2f380aedc0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.745+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f380c7160 con 0x7f2f380aedc0 2023-12-19T14:55:22.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.745+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f380c5750 con 0x7f2f380aedc0 2023-12-19T14:55:22.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.745+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3804be40 con 0x7f2f380aedc0 2023-12-19T14:55:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.749+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 39) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f2f2801b3b0 con 0x7f2f380aedc0 2023-12-19T14:55:22.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.749+0000 7f2f2f7fe700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 0x7f2f18062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:22.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.749+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f2f2807be90 con 0x7f2f380aedc0 2023-12-19T14:55:22.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.749+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 0x7f2f18062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:22.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.749+0000 7f2f2e7fc700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 0x7f2f18062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f200097c0 tx=0x7f2f20019000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.753+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f2804c250 con 0x7f2f380aedc0 2023-12-19T14:55:22.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:22.925+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f3806c8f0 con 0x7f2f380aedc0 2023-12-19T14:55:23.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.525+0000 7f2f2f7fe700 1 -- 172.21.15.50:0/1575724189 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2f280170d0 con 0x7f2f380aedc0 2023-12-19T14:55:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 msgr2=0x7f2f18062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 0x7f2f18062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f200097c0 tx=0x7f2f20019000 comp rx=0 tx=0).stop 2023-12-19T14:55:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 msgr2=0x7f2f380b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800c560 tx=0x7f2f28001020 comp rx=0 tx=0).stop 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 shutdown_connections 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f380af190 0x7f2f38169350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f180603a0 0x7f2f18062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f380cac50 0x7f2f38169890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 --2- 172.21.15.50:0/1575724189 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f380aedc0 0x7f2f380b4f60 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 >> 172.21.15.50:0/1575724189 conn(0x7f2f380cc760 msgr2=0x7f2f3815c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 shutdown_connections 2023-12-19T14:55:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.533+0000 7f2f37fff700 1 -- 172.21.15.50:0/1575724189 wait complete. 2023-12-19T14:55:23.538 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:55:23.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:55:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.733+0000 7f85e9545700 1 Processor -- start 2023-12-19T14:55:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.733+0000 7f85e9545700 1 -- start start 2023-12-19T14:55:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e9545700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85e40acd20 0x7f85e40cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e9545700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85e40af3d0 0x7f85e406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e9545700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e40adf30 con 0x7f85e40ad0f0 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e9545700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e40cd6f0 con 0x7f85e40af3d0 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e9545700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e40cd9d0 con 0x7f85e40acd20 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85e40af3d0 0x7f85e406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85e40af3d0 0x7f85e406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59124/0 (socket says 172.21.15.50:59124) 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 -- 172.21.15.50:0/3304401087 learned_addr learned my addr 172.21.15.50:0/3304401087 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e2ffd700 1 --2- 172.21.15.50:0/3304401087 >> v2:172.21.15.125:3300/0 conn(0x7f85e40acd20 0x7f85e40cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 -- 172.21.15.50:0/3304401087 >> v2:172.21.15.125:3300/0 conn(0x7f85e40acd20 msgr2=0x7f85e40cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 --2- 172.21.15.50:0/3304401087 >> v2:172.21.15.125:3300/0 conn(0x7f85e40acd20 0x7f85e40cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 -- 172.21.15.50:0/3304401087 >> v1:172.21.15.125:6789/0 conn(0x7f85e40ad0f0 legacy=0x7f85e40cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.737+0000 7f85e37fe700 1 -- 172.21.15.50:0/3304401087 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e40adbc0 con 0x7f85e40af3d0 2023-12-19T14:55:23.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e37fe700 1 --2- 172.21.15.50:0/3304401087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e406c7c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f85e40c4d40 tx=0x7f85dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ad9dcf51f12358 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:23.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e27fc700 1 -- 172.21.15.50:0/3304401087 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00e070 con 0x7f85e40af3d0 2023-12-19T14:55:23.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e27fc700 1 -- 172.21.15.50:0/3304401087 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc00ba20 con 0x7f85e40af3d0 2023-12-19T14:55:23.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e27fc700 1 -- 172.21.15.50:0/3304401087 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0092e0 con 0x7f85e40af3d0 2023-12-19T14:55:23.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 -- 172.21.15.50:0/3304401087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 msgr2=0x7f85e406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 --2- 172.21.15.50:0/3304401087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e406c7c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f85e40c4d40 tx=0x7f85dc009840 comp rx=0 tx=0).stop 2023-12-19T14:55:23.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 -- 172.21.15.50:0/3304401087 shutdown_connections 2023-12-19T14:55:23.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 --2- 172.21.15.50:0/3304401087 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e406c7c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 --2- 172.21.15.50:0/3304401087 >> v2:172.21.15.125:3300/0 conn(0x7f85e40acd20 0x7f85e40cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 -- 172.21.15.50:0/3304401087 >> 172.21.15.50:0/3304401087 conn(0x7f85e40cdce0 msgr2=0x7f85e40ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:23.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 -- 172.21.15.50:0/3304401087 shutdown_connections 2023-12-19T14:55:23.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.741+0000 7f85e9545700 1 -- 172.21.15.50:0/3304401087 wait complete. 2023-12-19T14:55:23.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 Processor -- start 2023-12-19T14:55:23.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 -- start start 2023-12-19T14:55:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85e40acd20 0x7f85e40ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85e40ad0f0 0x7f85e415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e4150dd0 con 0x7f85e40af3d0 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e41504e0 con 0x7f85e40ad0f0 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e40ac5a0 con 0x7f85e40acd20 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59138/0 (socket says 172.21.15.50:59138) 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 -- 172.21.15.50:0/1535484676 learned_addr learned my addr 172.21.15.50:0/1535484676 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e2ffd700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85e40acd20 0x7f85e40ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85dbfff700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85e40ad0f0 0x7f85e415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 -- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85e40acd20 msgr2=0x7f85e40ac060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85e40acd20 0x7f85e40ac060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 -- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85e40ad0f0 msgr2=0x7f85e415bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85e40ad0f0 0x7f85e415bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 -- 172.21.15.50:0/1535484676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc009ae0 con 0x7f85e40af3d0 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e37fe700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009880 tx=0x7f85dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc008b30 con 0x7f85e40af3d0 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc008c90 con 0x7f85e40af3d0 2023-12-19T14:55:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0266b0 con 0x7f85e40af3d0 2023-12-19T14:55:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.745+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e4150780 con 0x7f85e40af3d0 2023-12-19T14:55:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.749+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e415b080 con 0x7f85e40af3d0 2023-12-19T14:55:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.749+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e4150f80 con 0x7f85e40af3d0 2023-12-19T14:55:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.749+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 39) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f85dc003040 con 0x7f85e40af3d0 2023-12-19T14:55:23.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.749+0000 7f85daffd700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 0x7f85c4062920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:23.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.749+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6346+0+0 (secure 0 0 0) 0x7f85dc07e5a0 con 0x7f85e40af3d0 2023-12-19T14:55:23.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.753+0000 7f85e2ffd700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 0x7f85c4062920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.753+0000 7f85e2ffd700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 0x7f85c4062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009d10 tx=0x7f85d40091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:23.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.757+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f85e4150f80 con 0x7f85e40af3d0 2023-12-19T14:55:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.937+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f85e4071120 con 0x7f85e40af3d0 2023-12-19T14:55:23.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:23.965+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f85dc008e00 con 0x7f85e40af3d0 2023-12-19T14:55:24.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.537+0000 7f85daffd700 1 -- 172.21.15.50:0/1535484676 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7f85dc052190 con 0x7f85e40af3d0 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 msgr2=0x7f85c4062920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 0x7f85c4062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009d10 tx=0x7f85d40091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 msgr2=0x7f85e40b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009880 tx=0x7f85dc018000 comp rx=0 tx=0).stop 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 shutdown_connections 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85e40ad0f0 0x7f85e415bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c4060470 0x7f85c4062920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85e40af3d0 0x7f85e40b15e0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 --2- 172.21.15.50:0/1535484676 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85e40acd20 0x7f85e40ac060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 >> 172.21.15.50:0/1535484676 conn(0x7f85e40cdce0 msgr2=0x7f85e4155a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:24.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 shutdown_connections 2023-12-19T14:55:24.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.549+0000 7f85e9545700 1 -- 172.21.15.50:0/1535484676 wait complete. 2023-12-19T14:55:24.559 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:55:24.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:55:24.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.745+0000 7f85bf59e700 1 Processor -- start 2023-12-19T14:55:24.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.745+0000 7f85bf59e700 1 -- start start 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bf59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85c015a010 0x7f85c00c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bf59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c0159b80 con 0x7f85c0142690 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bf59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00c8220 con 0x7f85c0141e70 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bf59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c00c7f40 con 0x7f85c015a010 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bed9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f85c0142690 0x7f85c015c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45682/0 (socket says 172.21.15.50:45682) 2023-12-19T14:55:24.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bed9d700 1 -- 172.21.15.50:0/170878382 learned_addr learned my addr 172.21.15.50:0/170878382 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:24.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bdd9b700 1 --2- 172.21.15.50:0/170878382 >> v2:172.21.15.125:3300/0 conn(0x7f85c015a010 0x7f85c00c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:24.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826566946 0 0) 0x7f85c0159b80 con 0x7f85c0142690 2023-12-19T14:55:24.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a8003570 con 0x7f85c0142690 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335313982 0 0) 0x7f85c00c8220 con 0x7f85c0141e70 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c0159b80 con 0x7f85c0141e70 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1972740984 0 0) 0x7f85a8003570 con 0x7f85c0142690 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bdd9b700 1 -- 172.21.15.50:0/170878382 >> v1:172.21.15.125:6789/0 conn(0x7f85c0141e70 legacy=0x7f85c00c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bdd9b700 1 -- 172.21.15.50:0/170878382 >> v1:172.21.15.50:6789/0 conn(0x7f85c0142690 legacy=0x7f85c015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.749+0000 7f85bdd9b700 1 -- 172.21.15.50:0/170878382 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0159530 con 0x7f85c015a010 2023-12-19T14:55:24.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bdd9b700 1 --2- 172.21.15.50:0/170878382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c015a010 0x7f85c00c4ce0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098e0 tx=0x7f85b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c007399334a09c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:24.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80017f0 con 0x7f85c015a010 2023-12-19T14:55:24.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4011420 con 0x7f85c015a010 2023-12-19T14:55:24.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bd59a700 1 -- 172.21.15.50:0/170878382 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40117c0 con 0x7f85c015a010 2023-12-19T14:55:24.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 -- 172.21.15.50:0/170878382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c015a010 msgr2=0x7f85c00c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 --2- 172.21.15.50:0/170878382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c015a010 0x7f85c00c4ce0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098e0 tx=0x7f85b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 -- 172.21.15.50:0/170878382 shutdown_connections 2023-12-19T14:55:24.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 --2- 172.21.15.50:0/170878382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c015a010 0x7f85c00c4ce0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 -- 172.21.15.50:0/170878382 >> 172.21.15.50:0/170878382 conn(0x7f85c0067b50 msgr2=0x7f85c00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:24.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 -- 172.21.15.50:0/170878382 shutdown_connections 2023-12-19T14:55:24.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.753+0000 7f85bf59e700 1 -- 172.21.15.50:0/170878382 wait complete. 2023-12-19T14:55:24.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.757+0000 7f85bf59e700 1 Processor -- start 2023-12-19T14:55:24.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.757+0000 7f85bf59e700 1 -- start start 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c0141e70 0x7f85c00aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85c015a010 0x7f85c013e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c013cae0 con 0x7f85c0142690 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c013eb50 con 0x7f85c0141e70 2023-12-19T14:55:24.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c013bed0 con 0x7f85c015a010 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59154/0 (socket says 172.21.15.50:59154) 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 -- 172.21.15.50:0/394984464 learned_addr learned my addr 172.21.15.50:0/394984464 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85be59c700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c0141e70 0x7f85c00aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bed9d700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85c015a010 0x7f85c013e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 -- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85c015a010 msgr2=0x7f85c013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85c015a010 0x7f85c013e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 -- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c0141e70 msgr2=0x7f85c00aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c0141e70 0x7f85c00aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 -- 172.21.15.50:0/394984464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4009420 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bdd9b700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f85c006b910 tx=0x7f85b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b401cbe0 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b401cd40 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b401a6a0 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c013c1b0 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.761+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c013c770 con 0x7f85c0142690 2023-12-19T14:55:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.765+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c004be40 con 0x7f85c0142690 2023-12-19T14:55:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.765+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f85b401aa90 con 0x7f85c0142690 2023-12-19T14:55:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.765+0000 7f85ae7fc700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 0x7f85a0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.765+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f85b407e620 con 0x7f85c0142690 2023-12-19T14:55:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.765+0000 7f85be59c700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 0x7f85a0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:24.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.769+0000 7f85be59c700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 0x7f85a0062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f85b0007fd0 tx=0x7f85b0007d50 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:24.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.773+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f85b404eaa0 con 0x7f85c0142690 2023-12-19T14:55:24.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.949+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f85c0159810 con 0x7f85c0142690 2023-12-19T14:55:24.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85ae7fc700 1 -- 172.21.15.50:0/394984464 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f85b40520c0 con 0x7f85c0142690 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 msgr2=0x7f85a0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 0x7f85a0062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f85b0007fd0 tx=0x7f85b0007d50 comp rx=0 tx=0).stop 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 msgr2=0x7f85c013e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f85c006b910 tx=0x7f85b4016000 comp rx=0 tx=0).stop 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 shutdown_connections 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85a0060670 0x7f85a0062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85c015a010 0x7f85c013e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85c0142690 0x7f85c013e0d0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 --2- 172.21.15.50:0/394984464 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85c0141e70 0x7f85c00aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 >> 172.21.15.50:0/394984464 conn(0x7f85c0067b50 msgr2=0x7f85c0141140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 shutdown_connections 2023-12-19T14:55:24.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:24.961+0000 7f85bf59e700 1 -- 172.21.15.50:0/394984464 wait complete. 2023-12-19T14:55:24.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:55:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.149+0000 7fa4c042e700 1 Processor -- start 2023-12-19T14:55:25.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.149+0000 7fa4c042e700 1 -- start start 2023-12-19T14:55:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4c042e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4c042e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 0x7fa4b806b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4c042e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b806a7a0 con 0x7fa4b80c6d20 2023-12-19T14:55:25.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4c042e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b806d4b0 con 0x7fa4b806ac30 2023-12-19T14:55:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4c042e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b806d790 con 0x7fa4b806c7c0 2023-12-19T14:55:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4be9cb700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 0x7fa4b806b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:36018/0 (socket says 172.21.15.50:36018) 2023-12-19T14:55:25.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/4060669130 learned_addr learned my addr 172.21.15.50:0/4060669130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/4060669130 >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 msgr2=0x7fa4b806b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 --2- 172.21.15.50:0/4060669130 >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 0x7fa4b806b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/4060669130 >> v1:172.21.15.125:6789/0 conn(0x7fa4b80c6d20 legacy=0x7fa4b80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/4060669130 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b806da70 con 0x7fa4b806c7c0 2023-12-19T14:55:25.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd1c8700 1 -- 172.21.15.50:0/4060669130 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456315208 0 0) 0x7fa4b806a7a0 con 0x7fa4b80c6d20 2023-12-19T14:55:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4be9cb700 1 --2- 172.21.15.50:0/4060669130 >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 0x7fa4b806b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:25.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.153+0000 7fa4bd9c9700 1 --2- 172.21.15.50:0/4060669130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80099b0 tx=0x7fa4a8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01385ef4331bc11 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4bd1c8700 1 -- 172.21.15.50:0/4060669130 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8009ea0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4bd1c8700 1 -- 172.21.15.50:0/4060669130 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a800b860 con 0x7fa4b806c7c0 2023-12-19T14:55:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4bd1c8700 1 -- 172.21.15.50:0/4060669130 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800bc00 con 0x7fa4b806c7c0 2023-12-19T14:55:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- 172.21.15.50:0/4060669130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 msgr2=0x7fa4b806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 --2- 172.21.15.50:0/4060669130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80099b0 tx=0x7fa4a8009250 comp rx=0 tx=0).stop 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- 172.21.15.50:0/4060669130 shutdown_connections 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 --2- 172.21.15.50:0/4060669130 >> v2:172.21.15.125:3300/0 conn(0x7fa4b806ac30 0x7fa4b806b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 --2- 172.21.15.50:0/4060669130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b806c7c0 0x7fa4b806cc60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- 172.21.15.50:0/4060669130 >> 172.21.15.50:0/4060669130 conn(0x7fa4b8067bb0 msgr2=0x7fa4b80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- 172.21.15.50:0/4060669130 shutdown_connections 2023-12-19T14:55:25.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- 172.21.15.50:0/4060669130 wait complete. 2023-12-19T14:55:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 Processor -- start 2023-12-19T14:55:25.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.157+0000 7fa4c042e700 1 -- start start 2023-12-19T14:55:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 0x7fa4b80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b80c6d20 0x7fa4b8144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80af0e0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80a9ba0 con 0x7fa4b806ac30 2023-12-19T14:55:25.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b80a9e80 con 0x7fa4b80c6d20 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59170/0 (socket says 172.21.15.50:59170) 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/73571288 learned_addr learned my addr 172.21.15.50:0/73571288 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4be9cb700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b80c6d20 0x7fa4b8144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4be1ca700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 0x7fa4b80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b80c6d20 msgr2=0x7fa4b8144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b80c6d20 0x7fa4b8144740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 msgr2=0x7fa4b80aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 0x7fa4b80aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 -- 172.21.15.50:0/73571288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009a20 con 0x7fa4b806c7c0 2023-12-19T14:55:25.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4be1ca700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 0x7fa4b80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4bd9c9700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa4b806d8d0 tx=0x7fa4a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800bbe0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a8011420 con 0x7fa4b806c7c0 2023-12-19T14:55:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a80117e0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a80094f0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.161+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b80aa440 con 0x7fa4b806c7c0 2023-12-19T14:55:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.165+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b8071120 con 0x7fa4b806c7c0 2023-12-19T14:55:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.165+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa4a8011c10 con 0x7fa4b806c7c0 2023-12-19T14:55:25.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.169+0000 7fa4ae7fc700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 0x7fa49c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.169+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7fa4a807e3d0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.169+0000 7fa4be1ca700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 0x7fa49c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.169+0000 7fa4be1ca700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 0x7fa49c062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4007fd0 tx=0x7fa4b4009040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.173+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa4a804e7f0 con 0x7fa4b806c7c0 2023-12-19T14:55:25.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.345+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fa4b806dd50 con 0x7fa4b806c7c0 2023-12-19T14:55:25.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.357+0000 7fa4ae7fc700 1 -- 172.21.15.50:0/73571288 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa4a8051e10 con 0x7fa4b806c7c0 2023-12-19T14:55:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 msgr2=0x7fa49c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 0x7fa49c062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4007fd0 tx=0x7fa4b4009040 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 msgr2=0x7fa4b80ad170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa4b806d8d0 tx=0x7fa4a8017000 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 shutdown_connections 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa49c060670 0x7fa49c062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa4b80c6d20 0x7fa4b8144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4b806c7c0 0x7fa4b80ad170 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 --2- 172.21.15.50:0/73571288 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4b806ac30 0x7fa4b80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 >> 172.21.15.50:0/73571288 conn(0x7fa4b8067bb0 msgr2=0x7fa4b8158530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 shutdown_connections 2023-12-19T14:55:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.361+0000 7fa4c042e700 1 -- 172.21.15.50:0/73571288 wait complete. 2023-12-19T14:55:25.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:55:25.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.553+0000 7febd648e700 1 Processor -- start 2023-12-19T14:55:25.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.553+0000 7febd648e700 1 -- start start 2023-12-19T14:55:25.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd648e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7febd0158c90 0x7febd00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd648e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd006c850 con 0x7febd0159770 2023-12-19T14:55:25.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd648e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd006b470 con 0x7febd006aeb0 2023-12-19T14:55:25.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd648e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd006b750 con 0x7febd0158c90 2023-12-19T14:55:25.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd5c8d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7febd0159770 0x7febd006cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45696/0 (socket says 172.21.15.50:45696) 2023-12-19T14:55:25.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd5c8d700 1 -- 172.21.15.50:0/3537097481 learned_addr learned my addr 172.21.15.50:0/3537097481 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:25.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd548c700 1 --2- 172.21.15.50:0/3537097481 >> v2:172.21.15.125:3300/0 conn(0x7febd0158c90 0x7febd00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364965057 0 0) 0x7febd006c850 con 0x7febd0159770 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc003570 con 0x7febd0159770 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd548c700 1 -- 172.21.15.50:0/3537097481 >> v1:172.21.15.125:6789/0 conn(0x7febd006aeb0 legacy=0x7febd0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216143976 0 0) 0x7febd006b470 con 0x7febd006aeb0 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd548c700 1 -- 172.21.15.50:0/3537097481 >> v1:172.21.15.50:6789/0 conn(0x7febd0159770 legacy=0x7febd006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd548c700 1 -- 172.21.15.50:0/3537097481 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd006ba30 con 0x7febd0158c90 2023-12-19T14:55:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.557+0000 7febd548c700 1 --2- 172.21.15.50:0/3537097481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0158c90 0x7febd00caf40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7febcc0098e0 tx=0x7febcc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f1794f9ce27258 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc00b640 con 0x7febd0158c90 2023-12-19T14:55:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc00b7a0 con 0x7febd0158c90 2023-12-19T14:55:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febc7fff700 1 -- 172.21.15.50:0/3537097481 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc00bb40 con 0x7febd0158c90 2023-12-19T14:55:25.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 -- 172.21.15.50:0/3537097481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0158c90 msgr2=0x7febd00caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 --2- 172.21.15.50:0/3537097481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0158c90 0x7febd00caf40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7febcc0098e0 tx=0x7febcc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 -- 172.21.15.50:0/3537097481 shutdown_connections 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 --2- 172.21.15.50:0/3537097481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0158c90 0x7febd00caf40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 -- 172.21.15.50:0/3537097481 >> 172.21.15.50:0/3537097481 conn(0x7febd0067bb0 msgr2=0x7febd00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 -- 172.21.15.50:0/3537097481 shutdown_connections 2023-12-19T14:55:25.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 -- 172.21.15.50:0/3537097481 wait complete. 2023-12-19T14:55:25.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.561+0000 7febd648e700 1 Processor -- start 2023-12-19T14:55:25.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 -- start start 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd006aeb0 0x7febd0156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 0x7febd006cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd016d9a0 con 0x7febd0158c90 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd016cef0 con 0x7febd006aeb0 2023-12-19T14:55:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd00c4f10 con 0x7febd0159770 2023-12-19T14:55:25.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59184/0 (socket says 172.21.15.50:59184) 2023-12-19T14:55:25.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 -- 172.21.15.50:0/3220000242 learned_addr learned my addr 172.21.15.50:0/3220000242 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:25.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd548c700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd006aeb0 0x7febd0156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd5c8d700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 0x7febd006cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 -- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 msgr2=0x7febd006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 0x7febd006cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 -- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd006aeb0 msgr2=0x7febd0156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd006aeb0 0x7febd0156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 -- 172.21.15.50:0/3220000242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc009420 con 0x7febd0158c90 2023-12-19T14:55:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd5c8d700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 0x7febd006cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd4c8b700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7febc000c7e0 tx=0x7febc0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0011590 con 0x7febd0158c90 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc0011710 con 0x7febd0158c90 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0011ad0 con 0x7febd0158c90 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.565+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd016d100 con 0x7febd0158c90 2023-12-19T14:55:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.569+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd016d6c0 con 0x7febd0158c90 2023-12-19T14:55:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.569+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd00c4d40 con 0x7febd0158c90 2023-12-19T14:55:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.569+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7febc001b3b0 con 0x7febd0158c90 2023-12-19T14:55:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.573+0000 7febc57fa700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 0x7febb4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.573+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7febc007bff0 con 0x7febd0158c90 2023-12-19T14:55:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.573+0000 7febd548c700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 0x7febb4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.573+0000 7febd548c700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 0x7febb4062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febcc0093f0 tx=0x7febcc017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.577+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7febc004c350 con 0x7febd0158c90 2023-12-19T14:55:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.749+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7febd00cefa0 con 0x7febd0158c90 2023-12-19T14:55:25.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.761+0000 7febc57fa700 1 -- 172.21.15.50:0/3220000242 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7febc00170b0 con 0x7febd0158c90 2023-12-19T14:55:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 msgr2=0x7febb4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 0x7febb4062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febcc0093f0 tx=0x7febcc017000 comp rx=0 tx=0).stop 2023-12-19T14:55:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 msgr2=0x7febd016c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7febc000c7e0 tx=0x7febc0007df0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 shutdown_connections 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7febb4060670 0x7febb4062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0159770 0x7febd006cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0158c90 0x7febd016c9b0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 --2- 172.21.15.50:0/3220000242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd006aeb0 0x7febd0156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 >> 172.21.15.50:0/3220000242 conn(0x7febd0067bb0 msgr2=0x7febd0159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 shutdown_connections 2023-12-19T14:55:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.765+0000 7febd648e700 1 -- 172.21.15.50:0/3220000242 wait complete. 2023-12-19T14:55:25.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-12-19T14:55:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.953+0000 7fe2cbd45700 1 Processor -- start 2023-12-19T14:55:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.953+0000 7fe2cbd45700 1 -- start start 2023-12-19T14:55:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2cbd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2cbd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2c414fba0 0x7fe2c414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2cbd45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c40c6630 con 0x7fe2c40c5900 2023-12-19T14:55:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2cbd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c40c6440 con 0x7fe2c414fba0 2023-12-19T14:55:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2cbd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40c6910 con 0x7fe2c4067c90 2023-12-19T14:55:25.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2ca2e2700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe2c40c5900 0x7fe2c40c5d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36560/0 (socket says 172.21.15.50:36560) 2023-12-19T14:55:25.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2ca2e2700 1 -- 172.21.15.50:0/3853959620 learned_addr learned my addr 172.21.15.50:0/3853959620 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:25.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c9ae1700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/3853959620 >> v2:172.21.15.50:3300/0 conn(0x7fe2c414fba0 0x7fe2c414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c92e0700 1 -- 172.21.15.50:0/3853959620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 msgr2=0x7fe2c4068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c4068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c92e0700 1 -- 172.21.15.50:0/3853959620 >> v1:172.21.15.125:6789/0 conn(0x7fe2c40c5900 legacy=0x7fe2c40c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c92e0700 1 -- 172.21.15.50:0/3853959620 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c40c6bf0 con 0x7fe2c414fba0 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.957+0000 7fe2c9ae1700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c4068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.961+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c414ffd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40098e0 tx=0x7fe2b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e1a1beff00e910 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.961+0000 7fe2c8adf700 1 -- 172.21.15.50:0/3853959620 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400b640 con 0x7fe2c414fba0 2023-12-19T14:55:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.961+0000 7fe2c8adf700 1 -- 172.21.15.50:0/3853959620 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b400b7a0 con 0x7fe2c414fba0 2023-12-19T14:55:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.961+0000 7fe2c8adf700 1 -- 172.21.15.50:0/3853959620 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400bb40 con 0x7fe2c414fba0 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- 172.21.15.50:0/3853959620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 msgr2=0x7fe2c414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c414ffd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40098e0 tx=0x7fe2b40091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- 172.21.15.50:0/3853959620 shutdown_connections 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c414ffd0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/3853959620 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c4068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- 172.21.15.50:0/3853959620 >> 172.21.15.50:0/3853959620 conn(0x7fe2c40cd6d0 msgr2=0x7fe2c406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:25.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- 172.21.15.50:0/3853959620 shutdown_connections 2023-12-19T14:55:25.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- 172.21.15.50:0/3853959620 wait complete. 2023-12-19T14:55:25.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 Processor -- start 2023-12-19T14:55:25.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.965+0000 7fe2cbd45700 1 -- start start 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c41504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c4151de0 con 0x7fe2c414fba0 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c41509f0 con 0x7fe2c40c5900 2023-12-19T14:55:25.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c40b58f0 con 0x7fe2c4067c90 2023-12-19T14:55:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2ca2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c41504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c9ae1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:59116/0 (socket says 172.21.15.50:59116) 2023-12-19T14:55:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 -- 172.21.15.50:0/1426984408 learned_addr learned my addr 172.21.15.50:0/1426984408 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 -- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 msgr2=0x7fe2c413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c413fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 -- 172.21.15.50:0/1426984408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 msgr2=0x7fe2c41504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c41504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 -- 172.21.15.50:0/1426984408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0009420 con 0x7fe2c40c5900 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2ca2e2700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c41504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:25.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2c92e0700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400a010 tx=0x7fe2b4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b401bcc0 con 0x7fe2c40c5900 2023-12-19T14:55:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b401be60 con 0x7fe2c40c5900 2023-12-19T14:55:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b4009420 con 0x7fe2c40c5900 2023-12-19T14:55:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4011670 con 0x7fe2c40c5900 2023-12-19T14:55:25.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.969+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c4140450 con 0x7fe2c40c5900 2023-12-19T14:55:25.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.973+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c4071120 con 0x7fe2c40c5900 2023-12-19T14:55:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.973+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe2b4011a60 con 0x7fe2c40c5900 2023-12-19T14:55:25.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.973+0000 7fe2b9ffb700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 0x7fe2a8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:25.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.973+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7fe2b407e100 con 0x7fe2c40c5900 2023-12-19T14:55:25.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.977+0000 7fe2c9ae1700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 0x7fe2a8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:25.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.977+0000 7fe2c9ae1700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 0x7fe2a8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009ea0 tx=0x7fe2c00091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:25.981+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2b404e500 con 0x7fe2c40c5900 2023-12-19T14:55:26.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.213+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fe2c4140450 con 0x7fe2c40c5900 2023-12-19T14:55:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.549+0000 7fe2b9ffb700 1 -- 172.21.15.50:0/1426984408 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v619) v1 ==== 166+0+0 (secure 0 0 0) 0x7fe2c4140450 con 0x7fe2c40c5900 2023-12-19T14:55:26.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 msgr2=0x7fe2a8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 0x7fe2a8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009ea0 tx=0x7fe2c00091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 msgr2=0x7fe2c414ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400a010 tx=0x7fe2b4017000 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 shutdown_connections 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2c40c5900 0x7fe2c414ff70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe2a8060670 0x7fe2a8062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2c414fba0 0x7fe2c41504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 --2- 172.21.15.50:0/1426984408 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2c4067c90 0x7fe2c413fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 >> 172.21.15.50:0/1426984408 conn(0x7fe2c40cd6d0 msgr2=0x7fe2c40ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 shutdown_connections 2023-12-19T14:55:26.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.553+0000 7fe2cbd45700 1 -- 172.21.15.50:0/1426984408 wait complete. 2023-12-19T14:55:26.582 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-12-19T14:55:26.582 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-12-19T14:55:26.583 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:55:26.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.753+0000 7f8d03fff700 1 Processor -- start 2023-12-19T14:55:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.753+0000 7f8d03fff700 1 -- start start 2023-12-19T14:55:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d04158e70 0x7f8d040afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 0x7f8d0406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8d040cc470 0x7f8d040cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d040ca5a0 con 0x7f8d040cc470 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0406dac0 con 0x7f8d04158e70 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d03fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b0260 con 0x7f8d0406a540 2023-12-19T14:55:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8d040cc470 0x7f8d040cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d027fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 0x7f8d0406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d027fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 0x7f8d0406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53324/0 (socket says 172.21.15.50:53324) 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d027fc700 1 -- 172.21.15.50:0/3050630057 learned_addr learned my addr 172.21.15.50:0/3050630057 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d02ffd700 1 --2- 172.21.15.50:0/3050630057 >> v2:172.21.15.125:3300/0 conn(0x7f8d04158e70 0x7f8d040afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 -- 172.21.15.50:0/3050630057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 msgr2=0x7f8d0406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 --2- 172.21.15.50:0/3050630057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 0x7f8d0406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 -- 172.21.15.50:0/3050630057 >> v2:172.21.15.125:3300/0 conn(0x7f8d04158e70 msgr2=0x7f8d040afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 --2- 172.21.15.50:0/3050630057 >> v2:172.21.15.125:3300/0 conn(0x7f8d04158e70 0x7f8d040afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.757+0000 7f8d037fe700 1 -- 172.21.15.50:0/3050630057 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0406d220 con 0x7f8d040cc470 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d037fe700 1 --2- 172.21.15.50:0/3050630057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d040cc470 0x7f8d040cc8d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8d0406a210 tx=0x7f8cfc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b700fcd8691c9a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d01ffb700 1 -- 172.21.15.50:0/3050630057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc00e070 con 0x7f8d040cc470 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d01ffb700 1 -- 172.21.15.50:0/3050630057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc00bab0 con 0x7f8d040cc470 2023-12-19T14:55:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d01ffb700 1 -- 172.21.15.50:0/3050630057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc00be50 con 0x7f8d040cc470 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 -- 172.21.15.50:0/3050630057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d040cc470 msgr2=0x7f8d040cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 --2- 172.21.15.50:0/3050630057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d040cc470 0x7f8d040cc8d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8d0406a210 tx=0x7f8cfc009780 comp rx=0 tx=0).stop 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 -- 172.21.15.50:0/3050630057 shutdown_connections 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 --2- 172.21.15.50:0/3050630057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d040cc470 0x7f8d040cc8d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 --2- 172.21.15.50:0/3050630057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d0406a540 0x7f8d0406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 --2- 172.21.15.50:0/3050630057 >> v2:172.21.15.125:3300/0 conn(0x7f8d04158e70 0x7f8d040afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 -- 172.21.15.50:0/3050630057 >> 172.21.15.50:0/3050630057 conn(0x7f8d040670e0 msgr2=0x7f8d040674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 -- 172.21.15.50:0/3050630057 shutdown_connections 2023-12-19T14:55:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.761+0000 7f8d03fff700 1 -- 172.21.15.50:0/3050630057 wait complete. 2023-12-19T14:55:26.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 Processor -- start 2023-12-19T14:55:26.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 -- start start 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d040cc470 0x7f8d041563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 0x7f8d04159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040c5790 con 0x7f8d0406a540 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040c4ea0 con 0x7f8d04158e70 2023-12-19T14:55:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d03fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040c6500 con 0x7f8d040cc470 2023-12-19T14:55:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d02ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56558/0 (socket says 172.21.15.50:56558) 2023-12-19T14:55:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d02ffd700 1 -- 172.21.15.50:0/1205348988 learned_addr learned my addr 172.21.15.50:0/1205348988 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.765+0000 7f8d027fc700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d040cc470 0x7f8d041563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d037fe700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 0x7f8d04159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 -- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d040cc470 msgr2=0x7f8d041563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d040cc470 0x7f8d041563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 -- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 msgr2=0x7f8d04159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 0x7f8d04159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 -- 172.21.15.50:0/1205348988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc009a20 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d037fe700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 0x7f8d04159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d02ffd700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400ec80 tx=0x7f8cf40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400c9d0 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf400cb30 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4013470 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0415ba70 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d040c5420 con 0x7f8d0406a540 2023-12-19T14:55:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.769+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0404be40 con 0x7f8d0406a540 2023-12-19T14:55:26.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.773+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8cf4013820 con 0x7f8d0406a540 2023-12-19T14:55:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.773+0000 7f8ceaffd700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 0x7f8cf0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.773+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f8cf407ce00 con 0x7f8d0406a540 2023-12-19T14:55:26.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.773+0000 7f8d027fc700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 0x7f8cf0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.773+0000 7f8d027fc700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 0x7f8cf0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8006fa0 tx=0x7f8cf8008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:26.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.781+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8cf404d160 con 0x7f8d0406a540 2023-12-19T14:55:26.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.953+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d0406d8a0 con 0x7f8d0406a540 2023-12-19T14:55:26.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.953+0000 7f8ceaffd700 1 -- 172.21.15.50:0/1205348988 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+13454 (secure 0 0 0) 0x7f8cf4050780 con 0x7f8d0406a540 2023-12-19T14:55:26.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:26.957 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":105,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:24.527191+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6836/2756457211":"2023-12-20T14:55:21.515383+0000","172.21.15.125:6834/2756457211":"2023-12-20T14:55:21.515383+0000","172.21.15.125:0/3676580286":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/505904512":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/1693906057":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/1114598693":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/2297448116":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/4070416647":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/3252761303":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2020877475":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2604193437":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/812779881":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/3729319264":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:14.394463+0000","172.21.15.125:0/2885889354":"2023-12-20T14:55:20.963401+0000","172.21.15.125:0/3000325649":"2023-12-20T14:55:14.394463+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 msgr2=0x7f8cf0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 0x7f8cf0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8006fa0 tx=0x7f8cf8008040 comp rx=0 tx=0).stop 2023-12-19T14:55:26.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 msgr2=0x7f8d040c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400ec80 tx=0x7f8cf40091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 shutdown_connections 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d04158e70 0x7f8d04159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8cf00606a0 0x7f8cf0062b50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d040cc470 0x7f8d041563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 --2- 172.21.15.50:0/1205348988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d0406a540 0x7f8d040c5c20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 >> 172.21.15.50:0/1205348988 conn(0x7f8d040670e0 msgr2=0x7f8d0406a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 shutdown_connections 2023-12-19T14:55:26.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:26.957+0000 7f8d03fff700 1 -- 172.21.15.50:0/1205348988 wait complete. 2023-12-19T14:55:26.982 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:55:26.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:55:27.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.157+0000 7fd530078700 1 Processor -- start 2023-12-19T14:55:27.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.157+0000 7fd530078700 1 -- start start 2023-12-19T14:55:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd530078700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd528158e20 0x7fd5280ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd530078700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52806d5e0 con 0x7fd5280ca2c0 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd530078700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5280cbdf0 con 0x7fd5280cb970 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd530078700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280cc0d0 con 0x7fd528158e20 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd52d613700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd5280cb970 0x7fd52806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:59482/0 (socket says 172.21.15.50:59482) 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd52d613700 1 -- 172.21.15.50:0/3468562036 learned_addr learned my addr 172.21.15.50:0/3468562036 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd52e615700 1 --1- 172.21.15.50:0/3468562036 >> v1:172.21.15.50:6789/0 conn(0x7fd5280ca2c0 0x7fd52806ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47196/0 (socket says 172.21.15.50:47196) 2023-12-19T14:55:27.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.161+0000 7fd52de14700 1 --2- 172.21.15.50:0/3468562036 >> v2:172.21.15.125:3300/0 conn(0x7fd528158e20 0x7fd5280ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283222087 0 0) 0x7fd5280cbdf0 con 0x7fd5280cb970 2023-12-19T14:55:27.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd514003570 con 0x7fd5280cb970 2023-12-19T14:55:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647680101 0 0) 0x7fd52806d5e0 con 0x7fd5280ca2c0 2023-12-19T14:55:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5280cbdf0 con 0x7fd5280ca2c0 2023-12-19T14:55:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52de14700 1 -- 172.21.15.50:0/3468562036 >> v1:172.21.15.125:6789/0 conn(0x7fd5280cb970 legacy=0x7fd52806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52de14700 1 -- 172.21.15.50:0/3468562036 >> v1:172.21.15.50:6789/0 conn(0x7fd5280ca2c0 legacy=0x7fd52806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52de14700 1 -- 172.21.15.50:0/3468562036 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280cc3b0 con 0x7fd528158e20 2023-12-19T14:55:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52de14700 1 --2- 172.21.15.50:0/3468562036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd528158e20 0x7fd5280ce5c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd524009e90 tx=0x7fd524009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc469be50e320d82 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52400e070 con 0x7fd528158e20 2023-12-19T14:55:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52400bb90 con 0x7fd528158e20 2023-12-19T14:55:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd52ce12700 1 -- 172.21.15.50:0/3468562036 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524012430 con 0x7fd528158e20 2023-12-19T14:55:27.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd530078700 1 -- 172.21.15.50:0/3468562036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd528158e20 msgr2=0x7fd5280ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd530078700 1 --2- 172.21.15.50:0/3468562036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd528158e20 0x7fd5280ce5c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd524009e90 tx=0x7fd524009780 comp rx=0 tx=0).stop 2023-12-19T14:55:27.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd530078700 1 -- 172.21.15.50:0/3468562036 shutdown_connections 2023-12-19T14:55:27.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd530078700 1 --2- 172.21.15.50:0/3468562036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd528158e20 0x7fd5280ce5c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.165+0000 7fd530078700 1 -- 172.21.15.50:0/3468562036 >> 172.21.15.50:0/3468562036 conn(0x7fd528067bb0 msgr2=0x7fd5280d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- 172.21.15.50:0/3468562036 shutdown_connections 2023-12-19T14:55:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- 172.21.15.50:0/3468562036 wait complete. 2023-12-19T14:55:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 Processor -- start 2023-12-19T14:55:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- start start 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5280ca2c0 0x7fd52815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52815b960 0x7fd52814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52815a250 con 0x7fd528158e20 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52815a530 con 0x7fd52815b960 2023-12-19T14:55:27.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.169+0000 7fd530078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52815a810 con 0x7fd5280ca2c0 2023-12-19T14:55:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56570/0 (socket says 172.21.15.50:56570) 2023-12-19T14:55:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 -- 172.21.15.50:0/404098978 learned_addr learned my addr 172.21.15.50:0/404098978 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52e615700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52815b960 0x7fd52814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52de14700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5280ca2c0 0x7fd52815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 -- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5280ca2c0 msgr2=0x7fd52815b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5280ca2c0 0x7fd52815b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 -- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52815b960 msgr2=0x7fd52814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52815b960 0x7fd52814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 -- 172.21.15.50:0/404098978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524009a20 con 0x7fd528158e20 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd52d613700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c790 tx=0x7fd518007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518005b30 con 0x7fd528158e20 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518005c90 con 0x7fd528158e20 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5280c7330 con 0x7fd528158e20 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518007300 con 0x7fd528158e20 2023-12-19T14:55:27.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.173+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd528159720 con 0x7fd528158e20 2023-12-19T14:55:27.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.177+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5280c7170 con 0x7fd528158e20 2023-12-19T14:55:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.177+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd51801b3b0 con 0x7fd528158e20 2023-12-19T14:55:27.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.177+0000 7fd51dffb700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 0x7fd50c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.177+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7fd51807bff0 con 0x7fd528158e20 2023-12-19T14:55:27.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.181+0000 7fd52de14700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 0x7fd50c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.181+0000 7fd52de14700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 0x7fd50c062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd524009e60 tx=0x7fd524019000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.185+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd51804fe20 con 0x7fd528158e20 2023-12-19T14:55:27.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.353+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5280c7520 con 0x7fd528158e20 2023-12-19T14:55:27.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.357+0000 7fd51dffb700 1 -- 172.21.15.50:0/404098978 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd518017020 con 0x7fd528158e20 2023-12-19T14:55:27.359 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 msgr2=0x7fd50c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 0x7fd50c062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd524009e60 tx=0x7fd524019000 comp rx=0 tx=0).stop 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 msgr2=0x7fd52814e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c790 tx=0x7fd518007e40 comp rx=0 tx=0).stop 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 shutdown_connections 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd52815b960 0x7fd52814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd50c0606a0 0x7fd50c062b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd528158e20 0x7fd52814e9a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 --2- 172.21.15.50:0/404098978 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5280ca2c0 0x7fd52815b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 >> 172.21.15.50:0/404098978 conn(0x7fd528067bb0 msgr2=0x7fd52814fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:27.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 shutdown_connections 2023-12-19T14:55:27.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.361+0000 7fd530078700 1 -- 172.21.15.50:0/404098978 wait complete. 2023-12-19T14:55:27.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-12-19T14:55:27.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.545+0000 7f65e759e700 1 Processor -- start 2023-12-19T14:55:27.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.545+0000 7f65e759e700 1 -- start start 2023-12-19T14:55:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e80c6900 0x7f65e80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e806b900 0x7f65e806da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 0x7f65e815d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e806b470 con 0x7f65e806b900 2023-12-19T14:55:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e8161d10 con 0x7f65e815cc20 2023-12-19T14:55:27.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e806bd70 con 0x7f65e80c6900 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e806b900 0x7f65e806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65e806b900 0x7f65e806da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56584/0 (socket says 172.21.15.50:56584) 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 -- 172.21.15.50:0/1190431435 learned_addr learned my addr 172.21.15.50:0/1190431435 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e659c700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e80c6900 0x7f65e80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e6d9d700 1 --2- 172.21.15.50:0/1190431435 >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 0x7f65e815d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 -- 172.21.15.50:0/1190431435 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e80c6900 msgr2=0x7f65e80c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e80c6900 0x7f65e80c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 -- 172.21.15.50:0/1190431435 >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 msgr2=0x7f65e815d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/1190431435 >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 0x7f65e815d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 -- 172.21.15.50:0/1190431435 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e806a580 con 0x7f65e806b900 2023-12-19T14:55:27.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e6d9d700 1 --2- 172.21.15.50:0/1190431435 >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 0x7f65e815d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:27.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e806b900 0x7f65e806da00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0099b0 tx=0x7f65dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55ca36ed46cd8e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e559a700 1 -- 172.21.15.50:0/1190431435 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc009ea0 con 0x7f65e806b900 2023-12-19T14:55:27.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e559a700 1 -- 172.21.15.50:0/1190431435 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00b820 con 0x7f65e806b900 2023-12-19T14:55:27.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.549+0000 7f65e559a700 1 -- 172.21.15.50:0/1190431435 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00bbc0 con 0x7f65e806b900 2023-12-19T14:55:27.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 -- 172.21.15.50:0/1190431435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e806b900 msgr2=0x7f65e806da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e806b900 0x7f65e806da00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0099b0 tx=0x7f65dc009250 comp rx=0 tx=0).stop 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 -- 172.21.15.50:0/1190431435 shutdown_connections 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e806b900 0x7f65e806da00 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 --2- 172.21.15.50:0/1190431435 >> v2:172.21.15.125:3300/0 conn(0x7f65e815cc20 0x7f65e815d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 --2- 172.21.15.50:0/1190431435 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e80c6900 0x7f65e80c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 -- 172.21.15.50:0/1190431435 >> 172.21.15.50:0/1190431435 conn(0x7f65e8067bb0 msgr2=0x7f65e80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 -- 172.21.15.50:0/1190431435 shutdown_connections 2023-12-19T14:55:27.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.553+0000 7f65e759e700 1 -- 172.21.15.50:0/1190431435 wait complete. 2023-12-19T14:55:27.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 Processor -- start 2023-12-19T14:55:27.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 -- start start 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e815cc20 0x7f65e816d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b07d0 con 0x7f65e80c6900 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e815dc80 con 0x7f65e806b900 2023-12-19T14:55:27.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80b09c0 con 0x7f65e815cc20 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e659c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e659c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56884/0 (socket says 172.21.15.50:56884) 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56594/0 (socket says 172.21.15.50:56594) 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e659c700 1 -- 172.21.15.50:0/3136697998 learned_addr learned my addr 172.21.15.50:0/3136697998 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:27.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e6d9d700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e815cc20 0x7f65e816d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 -- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e815cc20 msgr2=0x7f65e816d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e815cc20 0x7f65e816d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 -- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 msgr2=0x7f65e8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 -- 172.21.15.50:0/3136697998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0094f0 con 0x7f65e80c6900 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e659c700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65e5d9b700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f65e8064070 tx=0x7f65dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.557+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc000d10 con 0x7f65e80c6900 2023-12-19T14:55:27.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc01f070 con 0x7f65e80c6900 2023-12-19T14:55:27.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc0266d0 con 0x7f65e80c6900 2023-12-19T14:55:27.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e80b0ca0 con 0x7f65e80c6900 2023-12-19T14:55:27.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e80b1260 con 0x7f65e80c6900 2023-12-19T14:55:27.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e804be40 con 0x7f65e80c6900 2023-12-19T14:55:27.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f65dc026b00 con 0x7f65e80c6900 2023-12-19T14:55:27.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65d67fc700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 0x7f65c8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:27.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f65dc021070 con 0x7f65e80c6900 2023-12-19T14:55:27.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.561+0000 7f65e659c700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 0x7f65c8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:27.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.565+0000 7f65e659c700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 0x7f65c8062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65e80b1a80 tx=0x7f65d80091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:27.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.569+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f65dc04ea50 con 0x7f65e80c6900 2023-12-19T14:55:27.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:27.801+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f65e806a860 con 0x7f65e80c6900 2023-12-19T14:55:28.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.561+0000 7f65d67fc700 1 -- 172.21.15.50:0/3136697998 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v621) v1 ==== 186+0+0 (secure 0 0 0) 0x7f65dc026db0 con 0x7f65e80c6900 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 msgr2=0x7f65c8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 0x7f65c8062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65e80b1a80 tx=0x7f65d80091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 msgr2=0x7f65e80b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f65e8064070 tx=0x7f65dc017000 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 shutdown_connections 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f65c8060670 0x7f65c8062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65e815cc20 0x7f65e816d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65e80c6900 0x7f65e80b0290 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 --2- 172.21.15.50:0/3136697998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65e806b900 0x7f65e8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 >> 172.21.15.50:0/3136697998 conn(0x7f65e8067bb0 msgr2=0x7f65e8159b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 shutdown_connections 2023-12-19T14:55:28.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.565+0000 7f65e759e700 1 -- 172.21.15.50:0/3136697998 wait complete. 2023-12-19T14:55:28.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:55:28.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.761+0000 7f1a5e6e3700 1 Processor -- start 2023-12-19T14:55:28.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 -- start start 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a580c69a0 0x7f1a580c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a5806de00 0x7f1a5806b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a5806c7c0 0x7f1a5815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5806d970 con 0x7f1a5806de00 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5806cd60 con 0x7f1a5806c7c0 2023-12-19T14:55:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5e6e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5806d040 con 0x7f1a580c69a0 2023-12-19T14:55:28.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5cee0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a5806de00 0x7f1a5806b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5dee2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a5806c7c0 0x7f1a5815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5cee0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1a5806de00 0x7f1a5806b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56604/0 (socket says 172.21.15.50:56604) 2023-12-19T14:55:28.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5cee0700 1 -- 172.21.15.50:0/1258276801 learned_addr learned my addr 172.21.15.50:0/1258276801 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:28.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.765+0000 7f1a5d6e1700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a580c69a0 0x7f1a580c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 -- 172.21.15.50:0/1258276801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a580c69a0 msgr2=0x7f1a580c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:28.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a580c69a0 0x7f1a580c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 -- 172.21.15.50:0/1258276801 >> v2:172.21.15.125:3300/0 conn(0x7f1a5806c7c0 msgr2=0x7f1a5815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:28.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 --2- 172.21.15.50:0/1258276801 >> v2:172.21.15.125:3300/0 conn(0x7f1a5806c7c0 0x7f1a5815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 -- 172.21.15.50:0/1258276801 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5806d320 con 0x7f1a5806de00 2023-12-19T14:55:28.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5cee0700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a5806de00 0x7f1a5806b880 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1a480099b0 tx=0x7f1a48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb4c432caaa97c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:28.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a4ffff700 1 -- 172.21.15.50:0/1258276801 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4800b640 con 0x7f1a5806de00 2023-12-19T14:55:28.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a4ffff700 1 -- 172.21.15.50:0/1258276801 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4800b7a0 con 0x7f1a5806de00 2023-12-19T14:55:28.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a4ffff700 1 -- 172.21.15.50:0/1258276801 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4800bb40 con 0x7f1a5806de00 2023-12-19T14:55:28.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/1258276801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a5806de00 msgr2=0x7f1a5806b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a5806de00 0x7f1a5806b880 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1a480099b0 tx=0x7f1a48009250 comp rx=0 tx=0).stop 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/1258276801 shutdown_connections 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a5806de00 0x7f1a5806b880 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/1258276801 >> v2:172.21.15.125:3300/0 conn(0x7f1a5806c7c0 0x7f1a5815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/1258276801 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a580c69a0 0x7f1a580c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/1258276801 >> 172.21.15.50:0/1258276801 conn(0x7f1a58067b50 msgr2=0x7f1a580cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/1258276801 shutdown_connections 2023-12-19T14:55:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.769+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/1258276801 wait complete. 2023-12-19T14:55:28.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 Processor -- start 2023-12-19T14:55:28.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 -- start start 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5806c7c0 0x7f1a5815be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a5806de00 0x7f1a5815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.773+0000 7f1a5e6e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a58170d70 con 0x7f1a580c69a0 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5e6e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5815b770 con 0x7f1a5806c7c0 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5e6e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5816dc40 con 0x7f1a5806de00 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56608/0 (socket says 172.21.15.50:56608) 2023-12-19T14:55:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 -- 172.21.15.50:0/732084603 learned_addr learned my addr 172.21.15.50:0/732084603 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5cee0700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a5806de00 0x7f1a5815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5d6e1700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5806c7c0 0x7f1a5815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 -- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a5806de00 msgr2=0x7f1a5815b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a5806de00 0x7f1a5815b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 -- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5806c7c0 msgr2=0x7f1a5815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5806c7c0 0x7f1a5815be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 -- 172.21.15.50:0/732084603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480094f0 con 0x7f1a580c69a0 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5dee2700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009ec0 tx=0x7f1a5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5000cb70 con 0x7f1a580c69a0 2023-12-19T14:55:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5000f040 con 0x7f1a580c69a0 2023-12-19T14:55:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50014480 con 0x7f1a580c69a0 2023-12-19T14:55:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5816df20 con 0x7f1a580c69a0 2023-12-19T14:55:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.777+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a58170a00 con 0x7f1a580c69a0 2023-12-19T14:55:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.781+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58170f20 con 0x7f1a580c69a0 2023-12-19T14:55:28.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.781+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1a50014830 con 0x7f1a580c69a0 2023-12-19T14:55:28.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.781+0000 7f1a4d7fa700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 0x7f1a3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:28.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.781+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f1a5007d4b0 con 0x7f1a580c69a0 2023-12-19T14:55:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.785+0000 7f1a5d6e1700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 0x7f1a3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:28.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.785+0000 7f1a5d6e1700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 0x7f1a3c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009d10 tx=0x7f1a540091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:28.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:28.789+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a5004d700 con 0x7f1a580c69a0 2023-12-19T14:55:29.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.029+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1a5804be40 con 0x7f1a580c69a0 2023-12-19T14:55:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.037+0000 7f1a4d7fa700 1 -- 172.21.15.50:0/732084603 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1a50050d20 con 0x7f1a580c69a0 2023-12-19T14:55:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 msgr2=0x7f1a3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 0x7f1a3c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009d10 tx=0x7f1a540091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 msgr2=0x7f1a58171200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009ec0 tx=0x7f1a5000b040 comp rx=0 tx=0).stop 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 shutdown_connections 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a3c060670 0x7f1a3c062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a580c69a0 0x7f1a58171200 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a5806de00 0x7f1a5815b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 --2- 172.21.15.50:0/732084603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5806c7c0 0x7f1a5815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 >> 172.21.15.50:0/732084603 conn(0x7f1a58067b50 msgr2=0x7f1a580acb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 shutdown_connections 2023-12-19T14:55:29.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.041+0000 7f1a5e6e3700 1 -- 172.21.15.50:0/732084603 wait complete. 2023-12-19T14:55:29.068 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:55:29.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:55:29.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:55:29.160 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:55:29.160 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:55:29.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.329+0000 7fcc7166e700 1 Processor -- start 2023-12-19T14:55:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.329+0000 7fcc7166e700 1 -- start start 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 0x7fcc6c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c06a9d0 con 0x7fcc6c158e40 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6c06acb0 con 0x7fcc6c06dee0 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c06af90 con 0x7fcc6c158a70 2023-12-19T14:55:29.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56622/0 (socket says 172.21.15.50:56622) 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/117967493 learned_addr learned my addr 172.21.15.50:0/117967493 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6affd700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 0x7fcc6c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/117967493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 msgr2=0x7fcc6c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 0x7fcc6c06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/117967493 >> v1:172.21.15.125:6789/0 conn(0x7fcc6c158e40 legacy=0x7fcc6c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/117967493 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c06b270 con 0x7fcc6c06dee0 2023-12-19T14:55:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6affd700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 0x7fcc6c06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc6b7fe700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fcc640099b0 tx=0x7fcc64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d6460433c6a10f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc69ffb700 1 -- 172.21.15.50:0/117967493 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400b860 con 0x7fcc6c06dee0 2023-12-19T14:55:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc69ffb700 1 -- 172.21.15.50:0/117967493 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6400b9c0 con 0x7fcc6c06dee0 2023-12-19T14:55:29.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc69ffb700 1 -- 172.21.15.50:0/117967493 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400bd60 con 0x7fcc6c06dee0 2023-12-19T14:55:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- 172.21.15.50:0/117967493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c06dee0 msgr2=0x7fcc6c0c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fcc640099b0 tx=0x7fcc64009250 comp rx=0 tx=0).stop 2023-12-19T14:55:29.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- 172.21.15.50:0/117967493 shutdown_connections 2023-12-19T14:55:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c06dee0 0x7fcc6c0c9d80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 --2- 172.21.15.50:0/117967493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158a70 0x7fcc6c06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- 172.21.15.50:0/117967493 >> 172.21.15.50:0/117967493 conn(0x7fcc6c067bb0 msgr2=0x7fcc6c0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.333+0000 7fcc7166e700 1 -- 172.21.15.50:0/117967493 shutdown_connections 2023-12-19T14:55:29.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.337+0000 7fcc7166e700 1 -- 172.21.15.50:0/117967493 wait complete. 2023-12-19T14:55:29.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.337+0000 7fcc7166e700 1 Processor -- start 2023-12-19T14:55:29.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.337+0000 7fcc7166e700 1 -- start start 2023-12-19T14:55:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc6c158a70 0x7fcc6c0c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158e40 0x7fcc6c0c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c14a910 con 0x7fcc6c1694f0 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c149d00 con 0x7fcc6c158a70 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c149fe0 con 0x7fcc6c158e40 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56632/0 (socket says 172.21.15.50:56632) 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/4248433084 learned_addr learned my addr 172.21.15.50:0/4248433084 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:29.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6affd700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc6c158a70 0x7fcc6c0c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6a7fc700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158e40 0x7fcc6c0c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158e40 msgr2=0x7fcc6c0c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158e40 0x7fcc6c0c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc6c158a70 msgr2=0x7fcc6c0c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc6c158a70 0x7fcc6c0c1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 -- 172.21.15.50:0/4248433084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640094f0 con 0x7fcc6c1694f0 2023-12-19T14:55:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc6b7fe700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcc640090b0 tx=0x7fcc64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400b860 con 0x7fcc6c1694f0 2023-12-19T14:55:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6401f070 con 0x7fcc6c1694f0 2023-12-19T14:55:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6401a870 con 0x7fcc6c1694f0 2023-12-19T14:55:29.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c14a2c0 con 0x7fcc6c1694f0 2023-12-19T14:55:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.341+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c149990 con 0x7fcc6c1694f0 2023-12-19T14:55:29.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.345+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c14aa80 con 0x7fcc6c1694f0 2023-12-19T14:55:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.345+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcc6401aca0 con 0x7fcc6c1694f0 2023-12-19T14:55:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.345+0000 7fcc52ffd700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 0x7fcc58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.345+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7fcc64021070 con 0x7fcc6c1694f0 2023-12-19T14:55:29.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.349+0000 7fcc6affd700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 0x7fcc58062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.349+0000 7fcc6affd700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 0x7fcc58062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0c5a50 tx=0x7fcc5c0099d0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.353+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc6404ed50 con 0x7fcc6c1694f0 2023-12-19T14:55:29.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.565+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc6c04be40 con 0x7fcc6c1694f0 2023-12-19T14:55:29.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.565+0000 7fcc52ffd700 1 -- 172.21.15.50:0/4248433084 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fcc6402bde0 con 0x7fcc6c1694f0 2023-12-19T14:55:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:29.568 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/586787549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":586787549},{"type":"v1","addr":"172.21.15.125:6837","nonce":586787549}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6927,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3266616281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3266616281},{"type":"v1","addr":"172.21.15.50:6833","nonce":3266616281}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6933,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3167545349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3167545349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3167545349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/4081157198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4081157198},{"type":"v1","addr":"172.21.15.125:6836","nonce":4081157198}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[]} 2023-12-19T14:55:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 msgr2=0x7fcc58062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 0x7fcc58062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0c5a50 tx=0x7fcc5c0099d0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 msgr2=0x7fcc6c169990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcc640090b0 tx=0x7fcc64017000 comp rx=0 tx=0).stop 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 shutdown_connections 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcc58060670 0x7fcc58062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc6c1694f0 0x7fcc6c169990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcc6c158e40 0x7fcc6c0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 --2- 172.21.15.50:0/4248433084 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc6c158a70 0x7fcc6c0c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 >> 172.21.15.50:0/4248433084 conn(0x7fcc6c067bb0 msgr2=0x7fcc6c0c0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 shutdown_connections 2023-12-19T14:55:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.569+0000 7fcc7166e700 1 -- 172.21.15.50:0/4248433084 wait complete. 2023-12-19T14:55:29.573 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 43 2023-12-19T14:55:29.590 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:55:29.590 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:55:29.591 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:55:29.591 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:55:29.591 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:55:29.591 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:55:29.591 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:55:29.591 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:55:29.591 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:55:29.591 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:55:29.592 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:55:29.592 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:55:29.592 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:55:29.592 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:55:29.592 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:55:29.592 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:55:29.612 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:55:29.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:55:29.614 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:55:29.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:55:29.615 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:55:29.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:55:29.616 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:55:29.616 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:55:29.661 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:55:29.661 DEBUG:teuthology.parallel:result is None 2023-12-19T14:55:29.664 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:55:29.665 DEBUG:teuthology.parallel:result is None 2023-12-19T14:55:29.666 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:55:29.666 DEBUG:teuthology.parallel:result is None 2023-12-19T14:55:29.670 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:55:29.671 DEBUG:teuthology.parallel:result is None 2023-12-19T14:55:29.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:55:29.772 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:55:29.772 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:55:29.775 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:55:29.782 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:55:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 Processor -- start 2023-12-19T14:55:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 -- start start 2023-12-19T14:55:29.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 0x7f4e6413f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 0x7f4e640ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6406dc00 con 0x7f4e640c7d90 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6413fb30 con 0x7f4e6413f6b0 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e69145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6406e090 con 0x7f4e640c8160 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e6359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 0x7f4e640ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e5bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 0x7f4e6413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e62d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4e640c7d90 0x7f4e6406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47212/0 (socket says 172.21.15.50:47212) 2023-12-19T14:55:29.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.833+0000 7f4e62d9d700 1 -- 172.21.15.50:0/1949748176 learned_addr learned my addr 172.21.15.50:0/1949748176 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e5bfff700 1 -- 172.21.15.50:0/1949748176 >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 msgr2=0x7f4e640ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e5bfff700 1 --2- 172.21.15.50:0/1949748176 >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 0x7f4e640ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e5bfff700 1 -- 172.21.15.50:0/1949748176 >> v1:172.21.15.50:6789/0 conn(0x7f4e640c7d90 legacy=0x7f4e6406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e5bfff700 1 -- 172.21.15.50:0/1949748176 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6406cff0 con 0x7f4e640c8160 2023-12-19T14:55:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e6359e700 1 --2- 172.21.15.50:0/1949748176 >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 0x7f4e640ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e5bfff700 1 --2- 172.21.15.50:0/1949748176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 0x7f4e6413f170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009910 tx=0x7f4e4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d146f6edcf341f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e6259c700 1 -- 172.21.15.50:0/1949748176 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00b640 con 0x7f4e640c8160 2023-12-19T14:55:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e6259c700 1 -- 172.21.15.50:0/1949748176 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c00b7a0 con 0x7f4e640c8160 2023-12-19T14:55:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e6259c700 1 -- 172.21.15.50:0/1949748176 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00bb60 con 0x7f4e640c8160 2023-12-19T14:55:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 -- 172.21.15.50:0/1949748176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 msgr2=0x7f4e6413f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 --2- 172.21.15.50:0/1949748176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 0x7f4e6413f170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009910 tx=0x7f4e4c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 -- 172.21.15.50:0/1949748176 shutdown_connections 2023-12-19T14:55:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 --2- 172.21.15.50:0/1949748176 >> v2:172.21.15.50:3300/0 conn(0x7f4e6413f6b0 0x7f4e640ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 --2- 172.21.15.50:0/1949748176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c8160 0x7f4e6413f170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.837+0000 7f4e69145700 1 -- 172.21.15.50:0/1949748176 >> 172.21.15.50:0/1949748176 conn(0x7f4e64067b60 msgr2=0x7f4e640ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- 172.21.15.50:0/1949748176 shutdown_connections 2023-12-19T14:55:29.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- 172.21.15.50:0/1949748176 wait complete. 2023-12-19T14:55:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 Processor -- start 2023-12-19T14:55:29.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- start start 2023-12-19T14:55:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 0x7f4e64154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e640c8160 0x7f4e640c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 0x7f4e640c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e640c24a0 con 0x7f4e6413f6b0 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64154ca0 con 0x7f4e640c8160 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e69145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e641597e0 con 0x7f4e640c7d90 2023-12-19T14:55:29.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 0x7f4e64154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e5bfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e640c8160 0x7f4e640c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e6359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 0x7f4e640c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.841+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 0x7f4e64154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53436/0 (socket says 172.21.15.50:53436) 2023-12-19T14:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e62d9d700 1 -- 172.21.15.50:0/3084690824 learned_addr learned my addr 172.21.15.50:0/3084690824 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 -- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 msgr2=0x7f4e64154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 0x7f4e64154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 -- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e640c8160 msgr2=0x7f4e640c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e640c8160 0x7f4e640c1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 -- 172.21.15.50:0/3084690824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c009a20 con 0x7f4e6413f6b0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e6359e700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 0x7f4e640c5060 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009e70 tx=0x7f4e5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00caa0 con 0x7f4e6413f6b0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00f040 con 0x7f4e6413f6b0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00ce90 con 0x7f4e6413f6b0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c009420 con 0x7f4e6413f6b0 2023-12-19T14:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e64159da0 con 0x7f4e6413f6b0 2023-12-19T14:55:29.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.845+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e641590c0 con 0x7f4e6413f6b0 2023-12-19T14:55:29.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.849+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4e5c003020 con 0x7f4e6413f6b0 2023-12-19T14:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.849+0000 7f4e5affd700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 0x7f4e44062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.849+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5932+0+0 (secure 0 0 0) 0x7f4e5c07d2d0 con 0x7f4e6413f6b0 2023-12-19T14:55:29.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.849+0000 7f4e62d9d700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 0x7f4e44062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.853+0000 7f4e62d9d700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 0x7f4e44062bf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e54004fc0 tx=0x7f4e54007f70 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:29.857+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e641590c0 con 0x7f4e6413f6b0 2023-12-19T14:55:30.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.049+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4e641590c0 con 0x7f4e6413f6b0 2023-12-19T14:55:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.593+0000 7f4e5affd700 1 -- 172.21.15.50:0/3084690824 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4e641590c0 con 0x7f4e6413f6b0 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 msgr2=0x7f4e44062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 0x7f4e44062bf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e54004fc0 tx=0x7f4e54007f70 comp rx=0 tx=0).stop 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 msgr2=0x7f4e640c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 0x7f4e640c5060 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009e70 tx=0x7f4e5c00b040 comp rx=0 tx=0).stop 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 shutdown_connections 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e640c8160 0x7f4e640c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e44060740 0x7f4e44062bf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e6413f6b0 0x7f4e640c5060 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 --2- 172.21.15.50:0/3084690824 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e640c7d90 0x7f4e64154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 >> 172.21.15.50:0/3084690824 conn(0x7f4e64067b60 msgr2=0x7f4e641589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 shutdown_connections 2023-12-19T14:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.597+0000 7f4e69145700 1 -- 172.21.15.50:0/3084690824 wait complete. 2023-12-19T14:55:30.601 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:55:30.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:55:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.785+0000 7ff584e6e700 1 Processor -- start 2023-12-19T14:55:30.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.785+0000 7ff584e6e700 1 -- start start 2023-12-19T14:55:30.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff584e6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5800ae140 0x7ff5800a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff584e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff584e6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58006dc20 con 0x7ff5800ae510 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff584e6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5800c2200 con 0x7ff5800ae140 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff584e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58006d940 con 0x7ff5800c1d60 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff57ed9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.789+0000 7ff57e59c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5800ae140 0x7ff5800a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53450/0 (socket says 172.21.15.50:53450) 2023-12-19T14:55:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 -- 172.21.15.50:0/1948886648 learned_addr learned my addr 172.21.15.50:0/1948886648 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 -- 172.21.15.50:0/1948886648 >> v2:172.21.15.125:3300/0 conn(0x7ff5800ae140 msgr2=0x7ff5800a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 --2- 172.21.15.50:0/1948886648 >> v2:172.21.15.125:3300/0 conn(0x7ff5800ae140 0x7ff5800a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 -- 172.21.15.50:0/1948886648 >> v1:172.21.15.125:6789/0 conn(0x7ff5800ae510 legacy=0x7ff58006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 -- 172.21.15.50:0/1948886648 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800c6e20 con 0x7ff5800c1d60 2023-12-19T14:55:30.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57ed9d700 1 --2- 172.21.15.50:0/1948886648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff578009f20 tx=0x7ff578009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc72f8cbce8f1d76 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57d59a700 1 -- 172.21.15.50:0/1948886648 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57800e070 con 0x7ff5800c1d60 2023-12-19T14:55:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57d59a700 1 -- 172.21.15.50:0/1948886648 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57800b9d0 con 0x7ff5800c1d60 2023-12-19T14:55:30.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff57d59a700 1 -- 172.21.15.50:0/1948886648 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5780092e0 con 0x7ff5800c1d60 2023-12-19T14:55:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff584e6e700 1 -- 172.21.15.50:0/1948886648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 msgr2=0x7ff5800680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.793+0000 7ff584e6e700 1 --2- 172.21.15.50:0/1948886648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff578009f20 tx=0x7ff578009840 comp rx=0 tx=0).stop 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- 172.21.15.50:0/1948886648 shutdown_connections 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 --2- 172.21.15.50:0/1948886648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff5800680f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 --2- 172.21.15.50:0/1948886648 >> v2:172.21.15.125:3300/0 conn(0x7ff5800ae140 0x7ff5800a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- 172.21.15.50:0/1948886648 >> 172.21.15.50:0/1948886648 conn(0x7ff5800c8850 msgr2=0x7ff5800c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- 172.21.15.50:0/1948886648 shutdown_connections 2023-12-19T14:55:30.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- 172.21.15.50:0/1948886648 wait complete. 2023-12-19T14:55:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 Processor -- start 2023-12-19T14:55:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- start start 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800ae140 0x7ff58014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff580159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58015cc90 con 0x7ff5800ae510 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5801572a0 con 0x7ff5800ae140 2023-12-19T14:55:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.797+0000 7ff584e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff580157580 con 0x7ff5800c1d60 2023-12-19T14:55:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56696/0 (socket says 172.21.15.50:56696) 2023-12-19T14:55:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 -- 172.21.15.50:0/4050413913 learned_addr learned my addr 172.21.15.50:0/4050413913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57e59c700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800ae140 0x7ff58014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57ed9d700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff580159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 -- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 msgr2=0x7ff580159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff580159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 -- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800ae140 msgr2=0x7ff58014f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800ae140 0x7ff58014f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 -- 172.21.15.50:0/4050413913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578009ae0 con 0x7ff5800ae510 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff57dd9b700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c7c0 tx=0x7ff574007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011590 con 0x7ff5800ae510 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5740116f0 con 0x7ff5800ae510 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011ab0 con 0x7ff5800ae510 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff580157860 con 0x7ff5800ae510 2023-12-19T14:55:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58015b010 con 0x7ff5800ae510 2023-12-19T14:55:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.801+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58015ca60 con 0x7ff5800ae510 2023-12-19T14:55:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.805+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff57401b3b0 con 0x7ff5800ae510 2023-12-19T14:55:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.805+0000 7ff56e7fc700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 0x7ff560062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.805+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5201+0+0 (secure 0 0 0) 0x7ff57407bc20 con 0x7ff5800ae510 2023-12-19T14:55:30.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.805+0000 7ff57e59c700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 0x7ff560062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.809+0000 7ff57e59c700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 0x7ff560062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff58016c8d0 tx=0x7ff5700091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:30.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:30.813+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff58015ca60 con 0x7ff5800ae510 2023-12-19T14:55:31.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.045+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff58015ca60 con 0x7ff5800ae510 2023-12-19T14:55:31.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.045+0000 7ff56e7fc700 1 -- 172.21.15.50:0/4050413913 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff58015ca60 con 0x7ff5800ae510 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.049 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:55:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.053 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.055 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:55:31.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:55:31.058 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:55:31.059 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.060 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:55:31.061 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:55:31.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:31.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:55:31.062 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:55:31.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 msgr2=0x7ff560062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 0x7ff560062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff58016c8d0 tx=0x7ff5700091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 msgr2=0x7ff58015aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c7c0 tx=0x7ff574007f50 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 shutdown_connections 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff560060670 0x7ff560062b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff5800c1d60 0x7ff580159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5800ae510 0x7ff58015aad0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 --2- 172.21.15.50:0/4050413913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5800ae140 0x7ff58014f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 >> 172.21.15.50:0/4050413913 conn(0x7ff5800c8850 msgr2=0x7ff5800684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 shutdown_connections 2023-12-19T14:55:31.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.049+0000 7ff584e6e700 1 -- 172.21.15.50:0/4050413913 wait complete. 2023-12-19T14:55:31.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:31.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.237+0000 7fa92e8f0700 1 Processor -- start 2023-12-19T14:55:31.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.237+0000 7fa92e8f0700 1 -- start start 2023-12-19T14:55:31.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa92e8f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9280c2980 0x7fa9280aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa92e8f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92806c9f0 con 0x7fa92813f890 2023-12-19T14:55:31.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa92e8f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280c7f20 con 0x7fa9280680f0 2023-12-19T14:55:31.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa92e8f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280c1c50 con 0x7fa9280c2980 2023-12-19T14:55:31.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa927fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa92813f890 0x7fa9280c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47220/0 (socket says 172.21.15.50:47220) 2023-12-19T14:55:31.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa927fff700 1 -- 172.21.15.50:0/105547 learned_addr learned my addr 172.21.15.50:0/105547 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:31.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa9277fe700 1 --2- 172.21.15.50:0/105547 >> v2:172.21.15.125:3300/0 conn(0x7fa9280c2980 0x7fa9280aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543761019 0 0) 0x7fa92806c9f0 con 0x7fa92813f890 2023-12-19T14:55:31.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa910003570 con 0x7fa92813f890 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.241+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547765761 0 0) 0x7fa9280c7f20 con 0x7fa9280680f0 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92806c9f0 con 0x7fa9280680f0 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291721100 0 0) 0x7fa910003570 con 0x7fa92813f890 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 >> v2:172.21.15.125:3300/0 conn(0x7fa9280c2980 msgr2=0x7fa9280aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 --2- 172.21.15.50:0/105547 >> v2:172.21.15.125:3300/0 conn(0x7fa9280c2980 0x7fa9280aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 >> v1:172.21.15.125:6789/0 conn(0x7fa9280680f0 legacy=0x7fa928068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280c8190 con 0x7fa92813f890 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fa918001470 con 0x7fa92813f890 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 >> v1:172.21.15.50:6789/0 conn(0x7fa92813f890 legacy=0x7fa9280c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa910003bf0 0x7fa910003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 0x7fa910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9100051c0 0x7fa91000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280c1c50 con 0x7fa910004810 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91000c0b0 con 0x7fa910003bf0 2023-12-19T14:55:31.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91000c270 con 0x7fa9100051c0 2023-12-19T14:55:31.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 0x7fa910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa92ce8d700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9100051c0 0x7fa91000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa927fff700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa910003bf0 0x7fa910003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 -- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9100051c0 msgr2=0x7fa91000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9100051c0 0x7fa91000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 -- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa910003bf0 msgr2=0x7fa910003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa910003bf0 0x7fa910003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 -- 172.21.15.50:0/105547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910003570 con 0x7fa910004810 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa927fff700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa910003bf0 0x7fa910003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa92ce8d700 1 -- 172.21.15.50:0/105547 reap_dead start 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa9277fe700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 0x7fa910004c80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa9280c2dc0 tx=0x7fa91c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f3f2386d7b814 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00e710 con 0x7fa910004810 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c00e870 con 0x7fa910004810 2023-12-19T14:55:31.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.245+0000 7fa926ffd700 1 -- 172.21.15.50:0/105547 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00ec30 con 0x7fa910004810 2023-12-19T14:55:31.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 msgr2=0x7fa910004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 0x7fa910004c80 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa9280c2dc0 tx=0x7fa91c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- 172.21.15.50:0/105547 shutdown_connections 2023-12-19T14:55:31.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/105547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa910004810 0x7fa910004c80 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- 172.21.15.50:0/105547 >> 172.21.15.50:0/105547 conn(0x7fa9280c8850 msgr2=0x7fa9280c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:31.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- 172.21.15.50:0/105547 shutdown_connections 2023-12-19T14:55:31.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- 172.21.15.50:0/105547 wait complete. 2023-12-19T14:55:31.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 Processor -- start 2023-12-19T14:55:31.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.249+0000 7fa92e8f0700 1 -- start start 2023-12-19T14:55:31.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa910004810 0x7fa92815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9280c5610 0x7fa92813b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280c6720 con 0x7fa9280c5b00 2023-12-19T14:55:31.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280c6050 con 0x7fa9280c5610 2023-12-19T14:55:31.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928170ae0 con 0x7fa910004810 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56708/0 (socket says 172.21.15.50:56708) 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 -- 172.21.15.50:0/686702898 learned_addr learned my addr 172.21.15.50:0/686702898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92ce8d700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9280c5610 0x7fa92813b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa927fff700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa910004810 0x7fa92815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 -- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa910004810 msgr2=0x7fa92815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa910004810 0x7fa92815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 -- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9280c5610 msgr2=0x7fa92813b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9280c5610 0x7fa92813b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 -- 172.21.15.50:0/686702898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928170dc0 con 0x7fa9280c5b00 2023-12-19T14:55:31.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa9277fe700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00e4b0 tx=0x7fa91c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c022070 con 0x7fa9280c5b00 2023-12-19T14:55:31.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c013c50 con 0x7fa9280c5b00 2023-12-19T14:55:31.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9281710a0 con 0x7fa9280c5b00 2023-12-19T14:55:31.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00e710 con 0x7fa9280c5b00 2023-12-19T14:55:31.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.253+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281718a0 con 0x7fa9280c5b00 2023-12-19T14:55:31.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.257+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928171520 con 0x7fa9280c5b00 2023-12-19T14:55:31.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.257+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa91c01c660 con 0x7fa9280c5b00 2023-12-19T14:55:31.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.261+0000 7fa90ffff700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 0x7fa914062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.261+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa91c07dd70 con 0x7fa9280c5b00 2023-12-19T14:55:31.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.261+0000 7fa927fff700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 0x7fa914062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.261+0000 7fa927fff700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 0x7fa914062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa918000f60 tx=0x7fa918007040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.265+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa91c04eb00 con 0x7fa9280c5b00 2023-12-19T14:55:31.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.437+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9280c6720 con 0x7fa9280c5b00 2023-12-19T14:55:31.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.437+0000 7fa90ffff700 1 -- 172.21.15.50:0/686702898 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+12384 (secure 0 0 0) 0x7fa91c017090 con 0x7fa9280c5b00 2023-12-19T14:55:31.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:31.441 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":109,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:30.604828+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 msgr2=0x7fa914062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 0x7fa914062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa918000f60 tx=0x7fa918007040 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 msgr2=0x7fa9280c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00e4b0 tx=0x7fa91c01a000 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 shutdown_connections 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9280c5610 0x7fa92813b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa914060670 0x7fa914062b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9280c5b00 0x7fa9280c50d0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 --2- 172.21.15.50:0/686702898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa910004810 0x7fa92815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.441+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 >> 172.21.15.50:0/686702898 conn(0x7fa9280c8850 msgr2=0x7fa9280af140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.445+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 shutdown_connections 2023-12-19T14:55:31.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.445+0000 7fa92e8f0700 1 -- 172.21.15.50:0/686702898 wait complete. 2023-12-19T14:55:31.468 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:55:31.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:55:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.657+0000 7fbb67f45700 1 Processor -- start 2023-12-19T14:55:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.657+0000 7fbb67f45700 1 -- start start 2023-12-19T14:55:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb67f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb60158e20 0x7fbb600ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb67f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb600ca2c0 0x7fbb600ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb67f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb600cbdf0 con 0x7fbb600cb970 2023-12-19T14:55:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb67f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb600cc0d0 con 0x7fbb60158e20 2023-12-19T14:55:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb67f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb600cc3b0 con 0x7fbb600ca2c0 2023-12-19T14:55:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb60158e20 0x7fbb600ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb60158e20 0x7fbb600ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56988/0 (socket says 172.21.15.50:56988) 2023-12-19T14:55:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 -- 172.21.15.50:0/818436647 learned_addr learned my addr 172.21.15.50:0/818436647 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb664e2700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb600ca2c0 0x7fbb600ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 -- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb600ca2c0 msgr2=0x7fbb600ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb600ca2c0 0x7fbb600ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 -- 172.21.15.50:0/818436647 >> v1:172.21.15.125:6789/0 conn(0x7fbb600cb970 legacy=0x7fbb6006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 -- 172.21.15.50:0/818436647 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb600cc690 con 0x7fbb60158e20 2023-12-19T14:55:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb65ce1700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb60158e20 0x7fbb600ce5c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0099e0 tx=0x7fbb5c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ada329e32f9c03ea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb64cdf700 1 -- 172.21.15.50:0/818436647 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00b820 con 0x7fbb60158e20 2023-12-19T14:55:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb64cdf700 1 -- 172.21.15.50:0/818436647 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c00b980 con 0x7fbb60158e20 2023-12-19T14:55:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.661+0000 7fbb64cdf700 1 -- 172.21.15.50:0/818436647 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00bd20 con 0x7fbb60158e20 2023-12-19T14:55:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 -- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb60158e20 msgr2=0x7fbb600ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb60158e20 0x7fbb600ce5c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0099e0 tx=0x7fbb5c009250 comp rx=0 tx=0).stop 2023-12-19T14:55:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 -- 172.21.15.50:0/818436647 shutdown_connections 2023-12-19T14:55:31.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb600ca2c0 0x7fbb600ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 --2- 172.21.15.50:0/818436647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb60158e20 0x7fbb600ce5c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 -- 172.21.15.50:0/818436647 >> 172.21.15.50:0/818436647 conn(0x7fbb60067bb0 msgr2=0x7fbb600d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:31.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 -- 172.21.15.50:0/818436647 shutdown_connections 2023-12-19T14:55:31.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.665+0000 7fbb67f45700 1 -- 172.21.15.50:0/818436647 wait complete. 2023-12-19T14:55:31.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 Processor -- start 2023-12-19T14:55:31.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 -- start start 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb600ca2c0 0x7fbb60169700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb60169c40 0x7fbb600c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb600c8160 con 0x7fbb60158e20 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb60150930 con 0x7fbb600ca2c0 2023-12-19T14:55:31.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb67f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb600c7e80 con 0x7fbb60169c40 2023-12-19T14:55:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb654e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb654e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56712/0 (socket says 172.21.15.50:56712) 2023-12-19T14:55:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb654e0700 1 -- 172.21.15.50:0/2781767067 learned_addr learned my addr 172.21.15.50:0/2781767067 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb664e2700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb60169c40 0x7fbb600c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.669+0000 7fbb65ce1700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb600ca2c0 0x7fbb60169700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 -- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb60169c40 msgr2=0x7fbb600c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb60169c40 0x7fbb600c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 -- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb600ca2c0 msgr2=0x7fbb60169700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb600ca2c0 0x7fbb60169700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 -- 172.21.15.50:0/2781767067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c0094f0 con 0x7fbb60158e20 2023-12-19T14:55:31.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb654e0700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000c7c0 tx=0x7fbb50007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50011590 con 0x7fbb60158e20 2023-12-19T14:55:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb500116f0 con 0x7fbb60158e20 2023-12-19T14:55:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6016a120 con 0x7fbb60158e20 2023-12-19T14:55:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50011ab0 con 0x7fbb60158e20 2023-12-19T14:55:31.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6016a6e0 con 0x7fbb60158e20 2023-12-19T14:55:31.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.673+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6016a930 con 0x7fbb60158e20 2023-12-19T14:55:31.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.677+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbb5001b3b0 con 0x7fbb60158e20 2023-12-19T14:55:31.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.677+0000 7fbb55ffb700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 0x7fbb44062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:31.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.677+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fbb5007bc50 con 0x7fbb60158e20 2023-12-19T14:55:31.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.677+0000 7fbb65ce1700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 0x7fbb44062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:31.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.681+0000 7fbb65ce1700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 0x7fbb44062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0099e0 tx=0x7fbb5c017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:31.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.685+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbb5004c280 con 0x7fbb60158e20 2023-12-19T14:55:31.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.857+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb6006a7a0 con 0x7fbb60158e20 2023-12-19T14:55:31.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.857+0000 7fbb55ffb700 1 -- 172.21.15.50:0/2781767067 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb50017020 con 0x7fbb60158e20 2023-12-19T14:55:31.860 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:55:31.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 msgr2=0x7fbb44062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 0x7fbb44062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0099e0 tx=0x7fbb5c017000 comp rx=0 tx=0).stop 2023-12-19T14:55:31.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 msgr2=0x7fbb600c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000c7c0 tx=0x7fbb50007f50 comp rx=0 tx=0).stop 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 shutdown_connections 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbb44060670 0x7fbb44062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbb60169c40 0x7fbb600c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb60158e20 0x7fbb600c8320 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 --2- 172.21.15.50:0/2781767067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb600ca2c0 0x7fbb60169700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 >> 172.21.15.50:0/2781767067 conn(0x7fbb60067bb0 msgr2=0x7fbb600ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 shutdown_connections 2023-12-19T14:55:31.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:31.861+0000 7fbb67f45700 1 -- 172.21.15.50:0/2781767067 wait complete. 2023-12-19T14:55:31.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:55:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.053+0000 7f456a6e3700 1 Processor -- start 2023-12-19T14:55:32.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.053+0000 7f456a6e3700 1 -- start start 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f456a6e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4564158cb0 0x7f45640b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f456a6e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 0x7f456406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f456a6e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456406d470 con 0x7f45640a8850 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f456a6e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456406c860 con 0x7f4564158cb0 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f456a6e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456406cb40 con 0x7f456406b4f0 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f4568ee0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f45640a8850 0x7f456406ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47222/0 (socket says 172.21.15.50:47222) 2023-12-19T14:55:32.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f4568ee0700 1 -- 172.21.15.50:0/1422031791 learned_addr learned my addr 172.21.15.50:0/1422031791 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 --2- 172.21.15.50:0/1422031791 >> v2:172.21.15.50:3300/0 conn(0x7f4564158cb0 0x7f45640b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f4569ee2700 1 --2- 172.21.15.50:0/1422031791 >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 0x7f456406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f455bfff700 1 -- 172.21.15.50:0/1422031791 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470898178 0 0) 0x7f456406d470 con 0x7f45640a8850 2023-12-19T14:55:32.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f455bfff700 1 -- 172.21.15.50:0/1422031791 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4550003570 con 0x7f45640a8850 2023-12-19T14:55:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 -- 172.21.15.50:0/1422031791 >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 msgr2=0x7f456406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 --2- 172.21.15.50:0/1422031791 >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 0x7f456406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 -- 172.21.15.50:0/1422031791 >> v1:172.21.15.50:6789/0 conn(0x7f45640a8850 legacy=0x7f456406ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 -- 172.21.15.50:0/1422031791 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456406ce20 con 0x7f4564158cb0 2023-12-19T14:55:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f4569ee2700 1 --2- 172.21.15.50:0/1422031791 >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 0x7f456406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.057+0000 7f45696e1700 1 --2- 172.21.15.50:0/1422031791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564158cb0 0x7f45640b1d40 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f45640af130 tx=0x7f45600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5205513272140a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f455bfff700 1 -- 172.21.15.50:0/1422031791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560011630 con 0x7f4564158cb0 2023-12-19T14:55:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f455bfff700 1 -- 172.21.15.50:0/1422031791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4560011790 con 0x7f4564158cb0 2023-12-19T14:55:32.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f455bfff700 1 -- 172.21.15.50:0/1422031791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560011b30 con 0x7f4564158cb0 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 -- 172.21.15.50:0/1422031791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564158cb0 msgr2=0x7f45640b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 --2- 172.21.15.50:0/1422031791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564158cb0 0x7f45640b1d40 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f45640af130 tx=0x7f45600091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 -- 172.21.15.50:0/1422031791 shutdown_connections 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 --2- 172.21.15.50:0/1422031791 >> v2:172.21.15.125:3300/0 conn(0x7f456406b4f0 0x7f456406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 --2- 172.21.15.50:0/1422031791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4564158cb0 0x7f45640b1d40 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 -- 172.21.15.50:0/1422031791 >> 172.21.15.50:0/1422031791 conn(0x7f45640684d0 msgr2=0x7f45640688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 -- 172.21.15.50:0/1422031791 shutdown_connections 2023-12-19T14:55:32.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.061+0000 7f456a6e3700 1 -- 172.21.15.50:0/1422031791 wait complete. 2023-12-19T14:55:32.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 Processor -- start 2023-12-19T14:55:32.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 -- start start 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f456406b4f0 0x7f4564148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4564158cb0 0x7f456415b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456416c570 con 0x7f45640a8850 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456416ce10 con 0x7f456406b4f0 2023-12-19T14:55:32.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f456a6e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4564149220 con 0x7f4564158cb0 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f4568ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f4568ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56728/0 (socket says 172.21.15.50:56728) 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.065+0000 7f4568ee0700 1 -- 172.21.15.50:0/2788457294 learned_addr learned my addr 172.21.15.50:0/2788457294 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4569ee2700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4564158cb0 0x7f456415b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f45696e1700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f456406b4f0 0x7f4564148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 -- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4564158cb0 msgr2=0x7f456415b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4564158cb0 0x7f456415b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 -- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f456406b4f0 msgr2=0x7f4564148ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f456406b4f0 0x7f4564148ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 -- 172.21.15.50:0/2788457294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560009420 con 0x7f45640a8850 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f4568ee0700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f455400c7f0 tx=0x7f4554007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554005cd0 con 0x7f45640a8850 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554005e50 con 0x7f45640a8850 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45640c1d30 con 0x7f45640a8850 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554007320 con 0x7f45640a8850 2023-12-19T14:55:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.069+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456416caa0 con 0x7f45640a8850 2023-12-19T14:55:32.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.073+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456404be40 con 0x7f45640a8850 2023-12-19T14:55:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.073+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f455401b3b0 con 0x7f45640a8850 2023-12-19T14:55:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.073+0000 7f45597fa700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 0x7f4548062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.073+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f455407be60 con 0x7f45640a8850 2023-12-19T14:55:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.073+0000 7f45696e1700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 0x7f4548062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.077+0000 7f45696e1700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 0x7f4548062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45600093f0 tx=0x7f4560017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.081+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f455404c490 con 0x7f45640a8850 2023-12-19T14:55:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.253+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f456416cf50 con 0x7f45640a8850 2023-12-19T14:55:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f45597fa700 1 -- 172.21.15.50:0/2788457294 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4554017020 con 0x7f45640a8850 2023-12-19T14:55:32.258 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:32.259 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:55:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 msgr2=0x7f4548062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 0x7f4548062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45600093f0 tx=0x7f4560017000 comp rx=0 tx=0).stop 2023-12-19T14:55:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 msgr2=0x7f456415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f455400c7f0 tx=0x7f4554007e00 comp rx=0 tx=0).stop 2023-12-19T14:55:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 shutdown_connections 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4548060670 0x7f4548062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4564158cb0 0x7f456415b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45640a8850 0x7f456415b1d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 --2- 172.21.15.50:0/2788457294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f456406b4f0 0x7f4564148ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 >> 172.21.15.50:0/2788457294 conn(0x7f45640684d0 msgr2=0x7f456406d900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 shutdown_connections 2023-12-19T14:55:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.257+0000 7f456a6e3700 1 -- 172.21.15.50:0/2788457294 wait complete. 2023-12-19T14:55:32.280 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:55:32.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:55:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.453+0000 7fa9c24ae700 1 Processor -- start 2023-12-19T14:55:32.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.453+0000 7fa9c24ae700 1 -- start start 2023-12-19T14:55:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c24ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa9bc0cb130 0x7fa9bc0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c24ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c24ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0b0ac0 con 0x7fa9bc06a6a0 2023-12-19T14:55:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c24ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9bc06b030 con 0x7fa9bc0cb130 2023-12-19T14:55:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c24ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc06b210 con 0x7fa9bc15c930 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bb7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa9bc06a6a0 0x7fa9bc15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47224/0 (socket says 172.21.15.50:47224) 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bb7fe700 1 -- 172.21.15.50:0/1696804423 learned_addr learned my addr 172.21.15.50:0/1696804423 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/1696804423 >> v2:172.21.15.50:3300/0 conn(0x7fa9bc0cb130 0x7fa9bc0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9c0a4b700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9baffd700 1 -- 172.21.15.50:0/1696804423 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412373985 0 0) 0x7fa9bc0b0ac0 con 0x7fa9bc06a6a0 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9baffd700 1 -- 172.21.15.50:0/1696804423 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4003570 con 0x7fa9bc06a6a0 2023-12-19T14:55:32.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 -- 172.21.15.50:0/1696804423 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 msgr2=0x7fa9bc06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 -- 172.21.15.50:0/1696804423 >> v1:172.21.15.50:6789/0 conn(0x7fa9bc06a6a0 legacy=0x7fa9bc15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 -- 172.21.15.50:0/1696804423 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc0ca810 con 0x7fa9bc0cb130 2023-12-19T14:55:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc0cb500 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0098e0 tx=0x7fa9ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7272b13e45dbf535 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.457+0000 7fa9baffd700 1 -- 172.21.15.50:0/1696804423 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac00b640 con 0x7fa9bc0cb130 2023-12-19T14:55:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9baffd700 1 -- 172.21.15.50:0/1696804423 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac00b7a0 con 0x7fa9bc0cb130 2023-12-19T14:55:32.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9baffd700 1 -- 172.21.15.50:0/1696804423 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac00bb40 con 0x7fa9bc0cb130 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 -- 172.21.15.50:0/1696804423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 msgr2=0x7fa9bc0cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc0cb500 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0098e0 tx=0x7fa9ac0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 -- 172.21.15.50:0/1696804423 shutdown_connections 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/1696804423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc0cb500 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 -- 172.21.15.50:0/1696804423 >> 172.21.15.50:0/1696804423 conn(0x7fa9bc067bb0 msgr2=0x7fa9bc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 -- 172.21.15.50:0/1696804423 shutdown_connections 2023-12-19T14:55:32.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.461+0000 7fa9c24ae700 1 -- 172.21.15.50:0/1696804423 wait complete. 2023-12-19T14:55:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 Processor -- start 2023-12-19T14:55:32.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 -- start start 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 0x7fa9bc0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc0b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc146450 con 0x7fa9bc0cb130 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc155e90 con 0x7fa9bc06a6a0 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c24ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0b3d30 con 0x7fa9bc15c930 2023-12-19T14:55:32.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56754/0 (socket says 172.21.15.50:56754) 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 -- 172.21.15.50:0/353041889 learned_addr learned my addr 172.21.15.50:0/353041889 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 -- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 msgr2=0x7fa9bc0b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9c0a4b700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc0b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 0x7fa9bc0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc0b1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 -- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 msgr2=0x7fa9bc0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 0x7fa9bc0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 -- 172.21.15.50:0/353041889 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac009420 con 0x7fa9bc0cb130 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 0x7fa9bc0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.465+0000 7fa9bb7fe700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000c5c0 tx=0x7fa9b0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0005b30 con 0x7fa9bc0cb130 2023-12-19T14:55:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b0005cb0 con 0x7fa9bc0cb130 2023-12-19T14:55:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9bc0b2e40 con 0x7fa9bc0cb130 2023-12-19T14:55:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0007320 con 0x7fa9bc0cb130 2023-12-19T14:55:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc0b3400 con 0x7fa9bc0cb130 2023-12-19T14:55:32.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9bc071120 con 0x7fa9bc0cb130 2023-12-19T14:55:32.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.469+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa9b001b3b0 con 0x7fa9bc0cb130 2023-12-19T14:55:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.473+0000 7fa9a3fff700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 0x7fa9a8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.473+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa9b007bd60 con 0x7fa9bc0cb130 2023-12-19T14:55:32.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.473+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 0x7fa9a8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:32.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.473+0000 7fa9bbfff700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 0x7fa9a8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac003080 tx=0x7fa9ac017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:32.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.477+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa9b004c390 con 0x7fa9bc0cb130 2023-12-19T14:55:32.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:32.649+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa9bc0cbe90 con 0x7fa9bc0cb130 2023-12-19T14:55:33.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.617+0000 7fa9a3fff700 1 -- 172.21.15.50:0/353041889 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa9b0017020 con 0x7fa9bc0cb130 2023-12-19T14:55:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 msgr2=0x7fa9a8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 0x7fa9a8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac003080 tx=0x7fa9ac017000 comp rx=0 tx=0).stop 2023-12-19T14:55:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 msgr2=0x7fa9bc15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000c5c0 tx=0x7fa9b0007e00 comp rx=0 tx=0).stop 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 shutdown_connections 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa9a8060670 0x7fa9a8062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa9bc15c930 0x7fa9bc0b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9bc0cb130 0x7fa9bc15a280 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 --2- 172.21.15.50:0/353041889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9bc06a6a0 0x7fa9bc0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 >> 172.21.15.50:0/353041889 conn(0x7fa9bc067bb0 msgr2=0x7fa9bc158450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 shutdown_connections 2023-12-19T14:55:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.625+0000 7fa9c24ae700 1 -- 172.21.15.50:0/353041889 wait complete. 2023-12-19T14:55:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:55:33.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:55:33.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.829+0000 7fbd71a8e700 1 Processor -- start 2023-12-19T14:55:33.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.829+0000 7fbd71a8e700 1 -- start start 2023-12-19T14:55:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.829+0000 7fbd71a8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd6c0ae6c0 0x7fbd6c155520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.829+0000 7fbd71a8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c06b530 0x7fbd6c0c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.829+0000 7fbd71a8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6c155a60 con 0x7fbd6c0c7130 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6c0ad020 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c0c81e0 con 0x7fbd6c06b530 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd6c0ae6c0 0x7fbd6c155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbd6c0ae6c0 0x7fbd6c155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56768/0 (socket says 172.21.15.50:56768) 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 -- 172.21.15.50:0/1470624649 learned_addr learned my addr 172.21.15.50:0/1470624649 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:33.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd7128d700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c06b530 0x7fbd6c0c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 -- 172.21.15.50:0/1470624649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c06b530 msgr2=0x7fbd6c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:33.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c06b530 0x7fbd6c0c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 -- 172.21.15.50:0/1470624649 >> v1:172.21.15.125:6789/0 conn(0x7fbd6c0c7130 legacy=0x7fbd6c06ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 -- 172.21.15.50:0/1470624649 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c0aca10 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd70a8c700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c155520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fbd680099b0 tx=0x7fbd68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491d1f3489643445 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd637fe700 1 -- 172.21.15.50:0/1470624649 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800b8f0 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd637fe700 1 -- 172.21.15.50:0/1470624649 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6800ba50 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd637fe700 1 -- 172.21.15.50:0/1470624649 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800bdf0 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- 172.21.15.50:0/1470624649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c0ae6c0 msgr2=0x7fbd6c155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c155520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fbd680099b0 tx=0x7fbd68009250 comp rx=0 tx=0).stop 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- 172.21.15.50:0/1470624649 shutdown_connections 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c06b530 0x7fbd6c0c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/1470624649 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c155520 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- 172.21.15.50:0/1470624649 >> 172.21.15.50:0/1470624649 conn(0x7fbd6c0c8cd0 msgr2=0x7fbd6c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- 172.21.15.50:0/1470624649 shutdown_connections 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.833+0000 7fbd71a8e700 1 -- 172.21.15.50:0/1470624649 wait complete. 2023-12-19T14:55:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 Processor -- start 2023-12-19T14:55:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 -- start start 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c0ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 0x7fbd6c0cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c0c3960 con 0x7fbd6c06b530 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c0afe90 con 0x7fbd6c0ae6c0 2023-12-19T14:55:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd71a8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c0b0170 con 0x7fbd6c0c7130 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56778/0 (socket says 172.21.15.50:56778) 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 -- 172.21.15.50:0/4203862537 learned_addr learned my addr 172.21.15.50:0/4203862537 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd7128d700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 0x7fbd6c0cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd63fff700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c0ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 -- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 msgr2=0x7fbd6c0cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 0x7fbd6c0cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 -- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c0ae6c0 msgr2=0x7fbd6c0ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c0ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd70a8c700 1 -- 172.21.15.50:0/4203862537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd680094f0 con 0x7fbd6c06b530 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.837+0000 7fbd7128d700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 0x7fbd6c0cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd70a8c700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0680c0 tx=0x7fbd68017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68024440 con 0x7fbd6c06b530 2023-12-19T14:55:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd680245a0 con 0x7fbd6c06b530 2023-12-19T14:55:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68024960 con 0x7fbd6c06b530 2023-12-19T14:55:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c0c3310 con 0x7fbd6c06b530 2023-12-19T14:55:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c14c520 con 0x7fbd6c06b530 2023-12-19T14:55:33.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.841+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c04be40 con 0x7fbd6c06b530 2023-12-19T14:55:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.845+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbd6801a420 con 0x7fbd6c06b530 2023-12-19T14:55:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.845+0000 7fbd60ff9700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 0x7fbd58062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.845+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5571+0+0 (secure 0 0 0) 0x7fbd6807e9b0 con 0x7fbd6c06b530 2023-12-19T14:55:33.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.845+0000 7fbd63fff700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 0x7fbd58062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:33.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.849+0000 7fbd63fff700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 0x7fbd58062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0c2b80 tx=0x7fbd5c009040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:33.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:33.853+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbd6804eef0 con 0x7fbd6c06b530 2023-12-19T14:55:34.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.025+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbd6c0c7c20 con 0x7fbd6c06b530 2023-12-19T14:55:34.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.633+0000 7fbd60ff9700 1 -- 172.21.15.50:0/4203862537 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbd68052510 con 0x7fbd6c06b530 2023-12-19T14:55:34.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.645+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 msgr2=0x7fbd58062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.645+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 0x7fbd58062bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0c2b80 tx=0x7fbd5c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.645+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 msgr2=0x7fbd6c14bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.645+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0680c0 tx=0x7fbd68017000 comp rx=0 tx=0).stop 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 shutdown_connections 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c0ae6c0 0x7fbd6c0ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbd58060740 0x7fbd58062bf0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0c2b80 tx=0x7fbd5c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:34.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbd6c0c7130 0x7fbd6c0cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 --2- 172.21.15.50:0/4203862537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd6c06b530 0x7fbd6c14bfe0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 >> 172.21.15.50:0/4203862537 conn(0x7fbd6c0c8cd0 msgr2=0x7fbd6c06e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:34.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 shutdown_connections 2023-12-19T14:55:34.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.649+0000 7fbd71a8e700 1 -- 172.21.15.50:0/4203862537 wait complete. 2023-12-19T14:55:34.657 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:55:34.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:55:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 Processor -- start 2023-12-19T14:55:34.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 -- start start 2023-12-19T14:55:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc0c7450 con 0x7f17cc15d410 2023-12-19T14:55:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc06bea0 con 0x7f17cc0cab10 2023-12-19T14:55:34.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.853+0000 7f17d2e25700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0c6e00 con 0x7f17cc06b8c0 2023-12-19T14:55:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cbfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f17cc0cab10 0x7f17cc06b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:59548/0 (socket says 172.21.15.50:59548) 2023-12-19T14:55:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cbfff700 1 -- 172.21.15.50:0/554812532 learned_addr learned my addr 172.21.15.50:0/554812532 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:34.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56788/0 (socket says 172.21.15.50:56788) 2023-12-19T14:55:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 -- 172.21.15.50:0/554812532 >> v1:172.21.15.125:6789/0 conn(0x7f17cc0cab10 legacy=0x7f17cc06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 -- 172.21.15.50:0/554812532 >> v1:172.21.15.50:6789/0 conn(0x7f17cc15d410 legacy=0x7f17cc0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cb7fe700 1 -- 172.21.15.50:0/554812532 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661475916 0 0) 0x7f17cc06bea0 con 0x7f17cc0cab10 2023-12-19T14:55:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 -- 172.21.15.50:0/554812532 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0c70e0 con 0x7f17cc06b8c0 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d13c2700 1 --2- 172.21.15.50:0/554812532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f17c4009e60 tx=0x7f17c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef65d7e350ed99f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cb7fe700 1 -- 172.21.15.50:0/554812532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c400e070 con 0x7f17cc06b8c0 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cb7fe700 1 -- 172.21.15.50:0/554812532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c400bc30 con 0x7f17cc06b8c0 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17cb7fe700 1 -- 172.21.15.50:0/554812532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4012430 con 0x7f17cc06b8c0 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d2e25700 1 -- 172.21.15.50:0/554812532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc06b8c0 msgr2=0x7f17cc0ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d2e25700 1 --2- 172.21.15.50:0/554812532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f17c4009e60 tx=0x7f17c4009780 comp rx=0 tx=0).stop 2023-12-19T14:55:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d2e25700 1 -- 172.21.15.50:0/554812532 shutdown_connections 2023-12-19T14:55:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d2e25700 1 --2- 172.21.15.50:0/554812532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0ab6c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.857+0000 7f17d2e25700 1 -- 172.21.15.50:0/554812532 >> 172.21.15.50:0/554812532 conn(0x7f17cc067bb0 msgr2=0x7f17cc0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.861+0000 7f17d2e25700 1 -- 172.21.15.50:0/554812532 shutdown_connections 2023-12-19T14:55:34.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.861+0000 7f17d2e25700 1 -- 172.21.15.50:0/554812532 wait complete. 2023-12-19T14:55:34.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.861+0000 7f17d2e25700 1 Processor -- start 2023-12-19T14:55:34.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.861+0000 7f17d2e25700 1 -- start start 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 0x7f17cc1585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc158b30 con 0x7f17cc15d410 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc15ec20 con 0x7f17cc06b8c0 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc15e010 con 0x7f17cc0cab10 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56802/0 (socket says 172.21.15.50:56802) 2023-12-19T14:55:34.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 -- 172.21.15.50:0/904579547 learned_addr learned my addr 172.21.15.50:0/904579547 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d0bc1700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17cbfff700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 0x7f17cc1585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 -- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 msgr2=0x7f17cc1585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 0x7f17cc1585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 -- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17cc06b8c0 msgr2=0x7f17cc0af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:34.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 -- 172.21.15.50:0/904579547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c4009a20 con 0x7f17cc15d410 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17cbfff700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 0x7f17cc1585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d13c2700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f17c4013000 tx=0x7f17c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4012e60 con 0x7f17cc15d410 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c401b630 con 0x7f17cc15d410 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc15e2f0 con 0x7f17cc15d410 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c401b9d0 con 0x7f17cc15d410 2023-12-19T14:55:34.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.865+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc15e8b0 con 0x7f17cc15d410 2023-12-19T14:55:34.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.869+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc071120 con 0x7f17cc15d410 2023-12-19T14:55:34.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.869+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f17c4003020 con 0x7f17cc15d410 2023-12-19T14:55:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.873+0000 7f17c8ff9700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 0x7f17b0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.873+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5953+0+0 (secure 0 0 0) 0x7f17c407eb60 con 0x7f17cc15d410 2023-12-19T14:55:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.873+0000 7f17d0bc1700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 0x7f17b0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.873+0000 7f17d0bc1700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 0x7f17b0062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17c00039e0 tx=0x7f17c0007ec0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:34.877+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f17c404ee10 con 0x7f17cc15d410 2023-12-19T14:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.089+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f17cc15ed60 con 0x7f17cc15d410 2023-12-19T14:55:35.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.665+0000 7f17c8ff9700 1 -- 172.21.15.50:0/904579547 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v46) v1 ==== 174+0+0 (secure 0 0 0) 0x7f17c402a0c0 con 0x7f17cc15d410 2023-12-19T14:55:35.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 msgr2=0x7f17b0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:35.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 0x7f17b0062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17c00039e0 tx=0x7f17c0007ec0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 msgr2=0x7f17cc15d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:35.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f17c4013000 tx=0x7f17c4019000 comp rx=0 tx=0).stop 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 shutdown_connections 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f17b00606a0 0x7f17b0062b50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17cc15d410 0x7f17cc15d7e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f17cc0cab10 0x7f17cc1585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 --2- 172.21.15.50:0/904579547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17cc06b8c0 0x7f17cc0af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.669+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 >> 172.21.15.50:0/904579547 conn(0x7f17cc067bb0 msgr2=0x7f17cc0c1900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.677+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 shutdown_connections 2023-12-19T14:55:35.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.677+0000 7f17d2e25700 1 -- 172.21.15.50:0/904579547 wait complete. 2023-12-19T14:55:35.691 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 13 and data pool 14 2023-12-19T14:55:35.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.861+0000 7f6f5eaf0700 1 Processor -- start 2023-12-19T14:55:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.861+0000 7f6f5eaf0700 1 -- start start 2023-12-19T14:55:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f580cc610 0x7f6f580c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f580c6630 con 0x7f6f581502b0 2023-12-19T14:55:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f580c6910 con 0x7f6f58067c90 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f580c6bf0 con 0x7f6f580cc610 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f580cc610 0x7f6f580c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f580cc610 0x7f6f580c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56818/0 (socket says 172.21.15.50:56818) 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 -- 172.21.15.50:0/3432898652 learned_addr learned my addr 172.21.15.50:0/3432898652 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 -- 172.21.15.50:0/3432898652 >> v1:172.21.15.125:6789/0 conn(0x7f6f58067c90 legacy=0x7f6f5814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:35.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 -- 172.21.15.50:0/3432898652 >> v1:172.21.15.50:6789/0 conn(0x7f6f581502b0 legacy=0x7f6f580c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 -- 172.21.15.50:0/3432898652 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f580c6ed0 con 0x7f6f580cc610 2023-12-19T14:55:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5d08d700 1 --2- 172.21.15.50:0/3432898652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580c6010 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009f20 tx=0x7f6f50009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed23cb15dd6bdda server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f4f7fe700 1 -- 172.21.15.50:0/3432898652 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000e070 con 0x7f6f580cc610 2023-12-19T14:55:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f4f7fe700 1 -- 172.21.15.50:0/3432898652 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5000ba40 con 0x7f6f580cc610 2023-12-19T14:55:35.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f4f7fe700 1 -- 172.21.15.50:0/3432898652 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500092e0 con 0x7f6f580cc610 2023-12-19T14:55:35.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/3432898652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 msgr2=0x7f6f580c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.865+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/3432898652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580c6010 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009f20 tx=0x7f6f50009840 comp rx=0 tx=0).stop 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/3432898652 shutdown_connections 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/3432898652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580c6010 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/3432898652 >> 172.21.15.50:0/3432898652 conn(0x7f6f580cd6d0 msgr2=0x7f6f5806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/3432898652 shutdown_connections 2023-12-19T14:55:35.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/3432898652 wait complete. 2023-12-19T14:55:35.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.869+0000 7f6f5eaf0700 1 Processor -- start 2023-12-19T14:55:35.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 -- start start 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f58067c90 0x7f6f581527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 0x7f6f5814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580b3730 con 0x7f6f580cc610 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580b39d0 con 0x7f6f581502b0 2023-12-19T14:55:35.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58152d10 con 0x7f6f58067c90 2023-12-19T14:55:35.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:35.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56834/0 (socket says 172.21.15.50:56834) 2023-12-19T14:55:35.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 -- 172.21.15.50:0/2066980232 learned_addr learned my addr 172.21.15.50:0/2066980232 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:35.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5c88c700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f58067c90 0x7f6f581527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5d08d700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 0x7f6f5814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 -- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f58067c90 msgr2=0x7f6f581527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f58067c90 0x7f6f581527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 -- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 msgr2=0x7f6f5814d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 0x7f6f5814d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 -- 172.21.15.50:0/2066980232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50009ae0 con 0x7f6f580cc610 2023-12-19T14:55:35.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5d08d700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 0x7f6f5814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4ffff700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800c4e0 tx=0x7f6f4800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011590 con 0x7f6f580cc610 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48011710 con 0x7f6f580cc610 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f580b27d0 con 0x7f6f580cc610 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.873+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48011ad0 con 0x7f6f580cc610 2023-12-19T14:55:35.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.877+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f58153240 con 0x7f6f580cc610 2023-12-19T14:55:35.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.877+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58071120 con 0x7f6f580cc610 2023-12-19T14:55:35.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.877+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6f4801b3b0 con 0x7f6f580cc610 2023-12-19T14:55:35.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.881+0000 7f6f4cff9700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 0x7f6f44062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:35.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.881+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f6f4807bfa0 con 0x7f6f580cc610 2023-12-19T14:55:35.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.881+0000 7f6f5c88c700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 0x7f6f44062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:35.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.881+0000 7f6f5c88c700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 0x7f6f44062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f54003a30 tx=0x7f6f54007ea0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:35.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:35.889+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f4804c280 con 0x7f6f580cc610 2023-12-19T14:55:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.061+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f580b3b10 con 0x7f6f580cc610 2023-12-19T14:55:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.065+0000 7f6f4cff9700 1 -- 172.21.15.50:0/2066980232 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+15625 (secure 0 0 0) 0x7f6f48017090 con 0x7f6f580cc610 2023-12-19T14:55:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:36.070 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":112,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:35.097736+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 msgr2=0x7f6f44062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 0x7f6f44062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f54003a30 tx=0x7f6f54007ea0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 msgr2=0x7f6f580b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800c4e0 tx=0x7f6f4800c7f0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 shutdown_connections 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f581502b0 0x7f6f5814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f44060670 0x7f6f44062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f580cc610 0x7f6f580b2290 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 --2- 172.21.15.50:0/2066980232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f58067c90 0x7f6f581527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 >> 172.21.15.50:0/2066980232 conn(0x7f6f580cd6d0 msgr2=0x7f6f5814eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 shutdown_connections 2023-12-19T14:55:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.069+0000 7f6f5eaf0700 1 -- 172.21.15.50:0/2066980232 wait complete. 2023-12-19T14:55:36.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:36.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 Processor -- start 2023-12-19T14:55:36.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 -- start start 2023-12-19T14:55:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f97440b1410 0x7f97440ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f974406aa00 0x7f974415af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974406c7c0 con 0x7f974406a580 2023-12-19T14:55:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974406cd80 con 0x7f97440b1410 2023-12-19T14:55:36.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.253+0000 7f974946b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974406b8f0 con 0x7f974406aa00 2023-12-19T14:55:36.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f97440b1410 0x7f97440ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f97440b1410 0x7f97440ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56840/0 (socket says 172.21.15.50:56840) 2023-12-19T14:55:36.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 -- 172.21.15.50:0/631512918 learned_addr learned my addr 172.21.15.50:0/631512918 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:36.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9748c6a700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f974406aa00 0x7f974415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 -- 172.21.15.50:0/631512918 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f974406aa00 msgr2=0x7f974415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f974406aa00 0x7f974415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 -- 172.21.15.50:0/631512918 >> v1:172.21.15.125:6789/0 conn(0x7f974406a580 legacy=0x7f974415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 -- 172.21.15.50:0/631512918 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97440ac1e0 con 0x7f97440b1410 2023-12-19T14:55:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f9743fff700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97440b1410 0x7f97440ab440 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f97340099b0 tx=0x7f9734009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78743058465506e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f97437fe700 1 -- 172.21.15.50:0/631512918 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734009ea0 con 0x7f97440b1410 2023-12-19T14:55:36.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f97437fe700 1 -- 172.21.15.50:0/631512918 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973400b860 con 0x7f97440b1410 2023-12-19T14:55:36.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f97437fe700 1 -- 172.21.15.50:0/631512918 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400bc00 con 0x7f97440b1410 2023-12-19T14:55:36.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 -- 172.21.15.50:0/631512918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97440b1410 msgr2=0x7f97440ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97440b1410 0x7f97440ab440 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f97340099b0 tx=0x7f9734009250 comp rx=0 tx=0).stop 2023-12-19T14:55:36.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 -- 172.21.15.50:0/631512918 shutdown_connections 2023-12-19T14:55:36.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f974406aa00 0x7f974415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 --2- 172.21.15.50:0/631512918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97440b1410 0x7f97440ab440 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 -- 172.21.15.50:0/631512918 >> 172.21.15.50:0/631512918 conn(0x7f9744067bb0 msgr2=0x7f97440cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:36.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.257+0000 7f974946b700 1 -- 172.21.15.50:0/631512918 shutdown_connections 2023-12-19T14:55:36.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 -- 172.21.15.50:0/631512918 wait complete. 2023-12-19T14:55:36.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 Processor -- start 2023-12-19T14:55:36.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 -- start start 2023-12-19T14:55:36.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f974406a580 0x7f974414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97440b1410 0x7f9744154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97441580d0 con 0x7f974406aa00 2023-12-19T14:55:36.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97441574c0 con 0x7f974406a580 2023-12-19T14:55:36.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.261+0000 7f974946b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97441577a0 con 0x7f97440b1410 2023-12-19T14:55:36.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56854/0 (socket says 172.21.15.50:56854) 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 -- 172.21.15.50:0/2534439840 learned_addr learned my addr 172.21.15.50:0/2534439840 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f9748c6a700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97440b1410 0x7f9744154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f9743fff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f974406a580 0x7f974414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 -- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97440b1410 msgr2=0x7f9744154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97440b1410 0x7f9744154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 -- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f974406a580 msgr2=0x7f974414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f974406a580 0x7f974414faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 -- 172.21.15.50:0/2534439840 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340094f0 con 0x7f974406aa00 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f973bfff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c4e0 tx=0x7f972c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011590 con 0x7f974406aa00 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c011710 con 0x7f974406aa00 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c011ad0 con 0x7f974406aa00 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9744157a80 con 0x7f974406aa00 2023-12-19T14:55:36.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.265+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97440c7640 con 0x7f974406aa00 2023-12-19T14:55:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.269+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974404be40 con 0x7f974406aa00 2023-12-19T14:55:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.269+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f972c01b3b0 con 0x7f974406aa00 2023-12-19T14:55:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.269+0000 7f9740ff9700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 0x7f9724062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.269+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f972c07c270 con 0x7f974406aa00 2023-12-19T14:55:36.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.273+0000 7f9743fff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 0x7f9724062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.273+0000 7f9743fff700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 0x7f9724062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97340099b0 tx=0x7f9734017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.277+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f972c04c5d0 con 0x7f974406aa00 2023-12-19T14:55:36.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.445+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97440cc340 con 0x7f974406aa00 2023-12-19T14:55:36.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.445+0000 7f9740ff9700 1 -- 172.21.15.50:0/2534439840 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+15625 (secure 0 0 0) 0x7f972c017090 con 0x7f974406aa00 2023-12-19T14:55:36.451 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:36.451 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":112,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:35.097736+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 msgr2=0x7f9724062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 0x7f9724062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97340099b0 tx=0x7f9734017000 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 msgr2=0x7f97440c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c4e0 tx=0x7f972c00c7f0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 shutdown_connections 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9724060670 0x7f9724062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97440b1410 0x7f9744154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f974406aa00 0x7f97440c7100 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 --2- 172.21.15.50:0/2534439840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f974406a580 0x7f974414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 >> 172.21.15.50:0/2534439840 conn(0x7f9744067bb0 msgr2=0x7f974406ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 shutdown_connections 2023-12-19T14:55:36.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.449+0000 7f974946b700 1 -- 172.21.15.50:0/2534439840 wait complete. 2023-12-19T14:55:36.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:55:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.649+0000 7f5a5db85700 1 Processor -- start 2023-12-19T14:55:36.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.649+0000 7f5a5db85700 1 -- start start 2023-12-19T14:55:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a5db85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5a580c6d20 0x7f5a580c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a5db85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a5806ac30 0x7f5a5806b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a5db85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5806a7a0 con 0x7f5a5806c7c0 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a5db85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a580c7630 con 0x7f5a5806ac30 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a5db85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5806d4b0 con 0x7f5a580c6d20 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a5806ac30 0x7f5a5806b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5a5806ac30 0x7f5a5806b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48746/0 (socket says 172.21.15.50:48746) 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 -- 172.21.15.50:0/3841917582 learned_addr learned my addr 172.21.15.50:0/3841917582 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a577fe700 1 --2- 172.21.15.50:0/3841917582 >> v2:172.21.15.125:3300/0 conn(0x7f5a580c6d20 0x7f5a580c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 -- 172.21.15.50:0/3841917582 >> v2:172.21.15.125:3300/0 conn(0x7f5a580c6d20 msgr2=0x7f5a580c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 --2- 172.21.15.50:0/3841917582 >> v2:172.21.15.125:3300/0 conn(0x7f5a580c6d20 0x7f5a580c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 -- 172.21.15.50:0/3841917582 >> v1:172.21.15.50:6789/0 conn(0x7f5a5806c7c0 legacy=0x7f5a5806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 -- 172.21.15.50:0/3841917582 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5806d790 con 0x7f5a5806ac30 2023-12-19T14:55:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.653+0000 7f5a57fff700 1 --2- 172.21.15.50:0/3841917582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a5806b000 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5a58068030 tx=0x7f5a50009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ccdaa9091e7ba0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a567fc700 1 -- 172.21.15.50:0/3841917582 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5000e070 con 0x7f5a5806ac30 2023-12-19T14:55:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a567fc700 1 -- 172.21.15.50:0/3841917582 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5000ba20 con 0x7f5a5806ac30 2023-12-19T14:55:36.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a567fc700 1 -- 172.21.15.50:0/3841917582 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a500092e0 con 0x7f5a5806ac30 2023-12-19T14:55:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 -- 172.21.15.50:0/3841917582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 msgr2=0x7f5a5806b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3841917582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a5806b000 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5a58068030 tx=0x7f5a50009840 comp rx=0 tx=0).stop 2023-12-19T14:55:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 -- 172.21.15.50:0/3841917582 shutdown_connections 2023-12-19T14:55:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3841917582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a5806b000 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3841917582 >> v2:172.21.15.125:3300/0 conn(0x7f5a580c6d20 0x7f5a580c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 -- 172.21.15.50:0/3841917582 >> 172.21.15.50:0/3841917582 conn(0x7f5a58067bb0 msgr2=0x7f5a580cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 -- 172.21.15.50:0/3841917582 shutdown_connections 2023-12-19T14:55:36.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.657+0000 7f5a5db85700 1 -- 172.21.15.50:0/3841917582 wait complete. 2023-12-19T14:55:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 Processor -- start 2023-12-19T14:55:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 -- start start 2023-12-19T14:55:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 0x7f5a580acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a580c6d20 0x7f5a580ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5816a3b0 con 0x7f5a5806ac30 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a58154370 con 0x7f5a580c6d20 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a5db85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5816a0d0 con 0x7f5a5806c7c0 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a577fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a577fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48750/0 (socket says 172.21.15.50:48750) 2023-12-19T14:55:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a577fe700 1 -- 172.21.15.50:0/3898538547 learned_addr learned my addr 172.21.15.50:0/3898538547 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.661+0000 7f5a57fff700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a580c6d20 0x7f5a580ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a56ffd700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 0x7f5a580acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 -- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 msgr2=0x7f5a580acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 0x7f5a580acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 -- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a580c6d20 msgr2=0x7f5a580ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a580c6d20 0x7f5a580ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 -- 172.21.15.50:0/3898538547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a50009ae0 con 0x7f5a5806ac30 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a56ffd700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 0x7f5a580acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:36.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a577fe700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800e960 tx=0x7f5a480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48013680 con 0x7f5a5806ac30 2023-12-19T14:55:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a480137e0 con 0x7f5a5806ac30 2023-12-19T14:55:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.665+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48013b80 con 0x7f5a5806ac30 2023-12-19T14:55:36.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.669+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5816a6c0 con 0x7f5a5806ac30 2023-12-19T14:55:36.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.669+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5816ac80 con 0x7f5a5806ac30 2023-12-19T14:55:36.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.669+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a580ab6c0 con 0x7f5a5806ac30 2023-12-19T14:55:36.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.669+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5a4800c490 con 0x7f5a5806ac30 2023-12-19T14:55:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.669+0000 7f5a3f7fe700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 0x7f5a44062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:36.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.673+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5a4807cc50 con 0x7f5a5806ac30 2023-12-19T14:55:36.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.673+0000 7f5a56ffd700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 0x7f5a44062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:36.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.673+0000 7f5a56ffd700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 0x7f5a44062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c007fd0 tx=0x7f5a4c009040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:36.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.677+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5a580ab6c0 con 0x7f5a5806ac30 2023-12-19T14:55:36.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:36.881+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5a58071120 con 0x7f5a5806ac30 2023-12-19T14:55:37.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.677+0000 7f5a3f7fe700 1 -- 172.21.15.50:0/3898538547 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5a480509c0 con 0x7f5a5806ac30 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 msgr2=0x7f5a44062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 0x7f5a44062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c007fd0 tx=0x7f5a4c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 msgr2=0x7f5a580c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800e960 tx=0x7f5a480091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 shutdown_connections 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5a580c6d20 0x7f5a580ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5a44060670 0x7f5a44062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5a5806c7c0 0x7f5a580acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 --2- 172.21.15.50:0/3898538547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a5806ac30 0x7f5a580c64d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 >> 172.21.15.50:0/3898538547 conn(0x7f5a58067bb0 msgr2=0x7f5a58158570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 shutdown_connections 2023-12-19T14:55:37.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.681+0000 7f5a5db85700 1 -- 172.21.15.50:0/3898538547 wait complete. 2023-12-19T14:55:37.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:55:37.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:55:37.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:55:37.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.881+0000 7ff4bb363700 1 Processor -- start 2023-12-19T14:55:37.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.881+0000 7ff4bb363700 1 -- start start 2023-12-19T14:55:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bb363700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40caa10 0x7ff4b4159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bb363700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff4b40cc040 0x7ff4b40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bb363700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40cbbb0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bb363700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b406dc90 con 0x7ff4b40cc040 2023-12-19T14:55:37.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bb363700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40cafa0 con 0x7ff4b40caa10 2023-12-19T14:55:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bab62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff4b40cc040 0x7ff4b40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bab62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff4b40cc040 0x7ff4b40ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48766/0 (socket says 172.21.15.50:48766) 2023-12-19T14:55:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4bab62700 1 -- 172.21.15.50:0/3760508720 learned_addr learned my addr 172.21.15.50:0/3760508720 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4ba361700 1 --1- 172.21.15.50:0/3760508720 >> v1:172.21.15.50:6789/0 conn(0x7ff4b40ce2d0 0x7ff4b4158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44964/0 (socket says 172.21.15.50:44964) 2023-12-19T14:55:37.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.885+0000 7ff4b9b60700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40caa10 0x7ff4b4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bab62700 1 -- 172.21.15.50:0/3760508720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40caa10 msgr2=0x7ff4b4159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bab62700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40caa10 0x7ff4b4159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bab62700 1 -- 172.21.15.50:0/3760508720 >> v1:172.21.15.50:6789/0 conn(0x7ff4b40ce2d0 legacy=0x7ff4b4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bab62700 1 -- 172.21.15.50:0/3760508720 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40cb280 con 0x7ff4b40cc040 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4b935f700 1 -- 172.21.15.50:0/3760508720 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636173355 0 0) 0x7ff4b40cbbb0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bab62700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40cc040 0x7ff4b40ce6a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009f20 tx=0x7ff4ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e819492807c25dc6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4b935f700 1 -- 172.21.15.50:0/3760508720 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac00e070 con 0x7ff4b40cc040 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4b935f700 1 -- 172.21.15.50:0/3760508720 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac00b9d0 con 0x7ff4b40cc040 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4b935f700 1 -- 172.21.15.50:0/3760508720 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac0092e0 con 0x7ff4b40cc040 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 -- 172.21.15.50:0/3760508720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40cc040 msgr2=0x7ff4b40ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40cc040 0x7ff4b40ce6a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009f20 tx=0x7ff4ac009840 comp rx=0 tx=0).stop 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 -- 172.21.15.50:0/3760508720 shutdown_connections 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40cc040 0x7ff4b40ce6a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3760508720 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40caa10 0x7ff4b4159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 -- 172.21.15.50:0/3760508720 >> 172.21.15.50:0/3760508720 conn(0x7ff4b4067bb0 msgr2=0x7ff4b40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 -- 172.21.15.50:0/3760508720 shutdown_connections 2023-12-19T14:55:37.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.889+0000 7ff4bb363700 1 -- 172.21.15.50:0/3760508720 wait complete. 2023-12-19T14:55:37.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 Processor -- start 2023-12-19T14:55:37.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 -- start start 2023-12-19T14:55:37.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4b40caa10 0x7ff4b416ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 0x7ff4b4155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b41678e0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40c85f0 con 0x7ff4b40caa10 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bb363700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40c9430 con 0x7ff4b40cc040 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48770/0 (socket says 172.21.15.50:48770) 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 -- 172.21.15.50:0/3706591200 learned_addr learned my addr 172.21.15.50:0/3706591200 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4b9b60700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 0x7ff4b4155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4ba361700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4b40caa10 0x7ff4b416ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 -- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 msgr2=0x7ff4b4155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 0x7ff4b4155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 -- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4b40caa10 msgr2=0x7ff4b416ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:37.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4b40caa10 0x7ff4b416ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4bab62700 1 -- 172.21.15.50:0/3706591200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac009ae0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.893+0000 7ff4b9b60700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 0x7ff4b4155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:37.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4bab62700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40634d0 tx=0x7ff4ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac008b10 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac008c90 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b416e330 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac0266b0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b40c8b00 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.897+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b404be40 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.901+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff4ac003040 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.901+0000 7ff4aa7fc700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 0x7ff498062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:37.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.901+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff4ac07f0a0 con 0x7ff4b40ce2d0 2023-12-19T14:55:37.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.901+0000 7ff4ba361700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 0x7ff498062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:37.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.905+0000 7ff4ba361700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 0x7ff498062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415c2b0 tx=0x7ff4b0007ef0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:37.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:37.905+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff4ac04f2f0 con 0x7ff4b40ce2d0 2023-12-19T14:55:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.125+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff4b40681f0 con 0x7ff4b40ce2d0 2023-12-19T14:55:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.973+0000 7ff4aa7fc700 1 -- 172.21.15.50:0/3706591200 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff4ac052910 con 0x7ff4b40ce2d0 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 msgr2=0x7ff498062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 0x7ff498062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4b415c2b0 tx=0x7ff4b0007ef0 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 msgr2=0x7ff4b40c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40634d0 tx=0x7ff4ac018000 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 shutdown_connections 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff498060670 0x7ff498062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4b40ce2d0 0x7ff4b40c98c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4b40cc040 0x7ff4b4155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 --2- 172.21.15.50:0/3706591200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4b40caa10 0x7ff4b416ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 >> 172.21.15.50:0/3706591200 conn(0x7ff4b4067bb0 msgr2=0x7ff4b4062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 shutdown_connections 2023-12-19T14:55:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:38.977+0000 7ff4bb363700 1 -- 172.21.15.50:0/3706591200 wait complete. 2023-12-19T14:55:39.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.189+0000 7fd350fc5700 1 Processor -- start 2023-12-19T14:55:39.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.189+0000 7fd350fc5700 1 -- start start 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c15d320 0x7fd34c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd34c0c15b0 0x7fd34c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd34c06d690 0x7fd34c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0c7190 con 0x7fd34c06d690 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0c1b50 con 0x7fd34c0c15b0 2023-12-19T14:55:39.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd350fc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0c1e30 con 0x7fd34c15d320 2023-12-19T14:55:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd34c06d690 0x7fd34c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd34c06d690 0x7fd34c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48778/0 (socket says 172.21.15.50:48778) 2023-12-19T14:55:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 -- 172.21.15.50:0/241666247 learned_addr learned my addr 172.21.15.50:0/241666247 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd349d9b700 1 --2- 172.21.15.50:0/241666247 >> v2:172.21.15.125:3300/0 conn(0x7fd34c0c15b0 0x7fd34c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34a59c700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c15d320 0x7fd34c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 -- 172.21.15.50:0/241666247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c15d320 msgr2=0x7fd34c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c15d320 0x7fd34c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 -- 172.21.15.50:0/241666247 >> v2:172.21.15.125:3300/0 conn(0x7fd34c0c15b0 msgr2=0x7fd34c0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 --2- 172.21.15.50:0/241666247 >> v2:172.21.15.125:3300/0 conn(0x7fd34c0c15b0 0x7fd34c0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 -- 172.21.15.50:0/241666247 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0c6e20 con 0x7fd34c06d690 2023-12-19T14:55:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34ad9d700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0680f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd34c068cc0 tx=0x7fd344009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8480959819e3a2d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34959a700 1 -- 172.21.15.50:0/241666247 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400e070 con 0x7fd34c06d690 2023-12-19T14:55:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34959a700 1 -- 172.21.15.50:0/241666247 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34400b970 con 0x7fd34c06d690 2023-12-19T14:55:39.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.193+0000 7fd34959a700 1 -- 172.21.15.50:0/241666247 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440092e0 con 0x7fd34c06d690 2023-12-19T14:55:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- 172.21.15.50:0/241666247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 msgr2=0x7fd34c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0680f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd34c068cc0 tx=0x7fd344009840 comp rx=0 tx=0).stop 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- 172.21.15.50:0/241666247 shutdown_connections 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0680f0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 --2- 172.21.15.50:0/241666247 >> v2:172.21.15.125:3300/0 conn(0x7fd34c0c15b0 0x7fd34c0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 --2- 172.21.15.50:0/241666247 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c15d320 0x7fd34c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- 172.21.15.50:0/241666247 >> 172.21.15.50:0/241666247 conn(0x7fd34c0c8850 msgr2=0x7fd34c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- 172.21.15.50:0/241666247 shutdown_connections 2023-12-19T14:55:39.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- 172.21.15.50:0/241666247 wait complete. 2023-12-19T14:55:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 Processor -- start 2023-12-19T14:55:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.197+0000 7fd350fc5700 1 -- start start 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c0c15b0 0x7fd34c15e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 0x7fd34c14b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c15e840 con 0x7fd34c06d690 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c14b610 con 0x7fd34c15d320 2023-12-19T14:55:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd350fc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0c55c0 con 0x7fd34c0c15b0 2023-12-19T14:55:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48790/0 (socket says 172.21.15.50:48790) 2023-12-19T14:55:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 -- 172.21.15.50:0/1920854440 learned_addr learned my addr 172.21.15.50:0/1920854440 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd349d9b700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c0c15b0 0x7fd34c15e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34ad9d700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 0x7fd34c14b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 -- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c0c15b0 msgr2=0x7fd34c15e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c0c15b0 0x7fd34c15e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 -- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 msgr2=0x7fd34c14b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 0x7fd34c14b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 -- 172.21.15.50:0/1920854440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344009ae0 con 0x7fd34c06d690 2023-12-19T14:55:39.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34ad9d700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 0x7fd34c14b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.201+0000 7fd34a59c700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e980 tx=0x7fd33c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00ee70 con 0x7fd34c06d690 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00c850 con 0x7fd34c06d690 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c0c52e0 con 0x7fd34c06d690 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00cc10 con 0x7fd34c06d690 2023-12-19T14:55:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c15db00 con 0x7fd34c06d690 2023-12-19T14:55:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c071120 con 0x7fd34c06d690 2023-12-19T14:55:39.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.205+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd33c013470 con 0x7fd34c06d690 2023-12-19T14:55:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.209+0000 7fd33a7fc700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 0x7fd32c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.209+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fd33c07d000 con 0x7fd34c06d690 2023-12-19T14:55:39.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.209+0000 7fd349d9b700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 0x7fd32c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.209+0000 7fd349d9b700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 0x7fd32c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd34c155020 tx=0x7fd3400091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.213+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd33c04d400 con 0x7fd34c06d690 2023-12-19T14:55:39.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.425+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34c0642a0 con 0x7fd34c06d690 2023-12-19T14:55:39.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.425+0000 7fd33a7fc700 1 -- 172.21.15.50:0/1920854440 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fd33c050a20 con 0x7fd34c06d690 2023-12-19T14:55:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:39.428 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 msgr2=0x7fd32c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 0x7fd32c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd34c155020 tx=0x7fd3400091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 msgr2=0x7fd34c0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e980 tx=0x7fd33c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 shutdown_connections 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd34c15d320 0x7fd34c14b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd32c060670 0x7fd32c062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd34c0c15b0 0x7fd34c15e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 --2- 172.21.15.50:0/1920854440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd34c06d690 0x7fd34c0c58e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 >> 172.21.15.50:0/1920854440 conn(0x7fd34c0c8850 msgr2=0x7fd34c0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 shutdown_connections 2023-12-19T14:55:39.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.429+0000 7fd350fc5700 1 -- 172.21.15.50:0/1920854440 wait complete. 2023-12-19T14:55:39.435 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:55:39.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:55:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.613+0000 7f250a785700 1 Processor -- start 2023-12-19T14:55:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.613+0000 7f250a785700 1 -- start start 2023-12-19T14:55:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f25041556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25040ae520 0x7f25040c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f250406c7c0 0x7f250406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250406d120 con 0x7f250406c7c0 2023-12-19T14:55:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250406a9d0 con 0x7f25040ae520 2023-12-19T14:55:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f250a785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250406acb0 con 0x7f25041552f0 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f250406c7c0 0x7f250406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f250406c7c0 0x7f250406cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48798/0 (socket says 172.21.15.50:48798) 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 -- 172.21.15.50:0/25677566 learned_addr learned my addr 172.21.15.50:0/25677566 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f24fbfff700 1 --2- 172.21.15.50:0/25677566 >> v2:172.21.15.125:3300/0 conn(0x7f25040ae520 0x7f25040c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2503fff700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f25041556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 -- 172.21.15.50:0/25677566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 msgr2=0x7f25041556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f25041556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 -- 172.21.15.50:0/25677566 >> v2:172.21.15.125:3300/0 conn(0x7f25040ae520 msgr2=0x7f25040c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:39.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 --2- 172.21.15.50:0/25677566 >> v2:172.21.15.125:3300/0 conn(0x7f25040ae520 0x7f25040c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.617+0000 7f2508d22700 1 -- 172.21.15.50:0/25677566 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250406af90 con 0x7f250406c7c0 2023-12-19T14:55:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f2508d22700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f250406cbe0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f250406a160 tx=0x7f24fc00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f824f24ca32a18 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f25037fe700 1 -- 172.21.15.50:0/25677566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc010070 con 0x7f250406c7c0 2023-12-19T14:55:39.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f25037fe700 1 -- 172.21.15.50:0/25677566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc00dbd0 con 0x7f250406c7c0 2023-12-19T14:55:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f25037fe700 1 -- 172.21.15.50:0/25677566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc016400 con 0x7f250406c7c0 2023-12-19T14:55:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 -- 172.21.15.50:0/25677566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 msgr2=0x7f250406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f250406cbe0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f250406a160 tx=0x7f24fc00ba50 comp rx=0 tx=0).stop 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 -- 172.21.15.50:0/25677566 shutdown_connections 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f250406cbe0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 --2- 172.21.15.50:0/25677566 >> v2:172.21.15.125:3300/0 conn(0x7f25040ae520 0x7f25040c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 --2- 172.21.15.50:0/25677566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f25041556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 -- 172.21.15.50:0/25677566 >> 172.21.15.50:0/25677566 conn(0x7f25040c8850 msgr2=0x7f25040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 -- 172.21.15.50:0/25677566 shutdown_connections 2023-12-19T14:55:39.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.621+0000 7f250a785700 1 -- 172.21.15.50:0/25677566 wait complete. 2023-12-19T14:55:39.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 Processor -- start 2023-12-19T14:55:39.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- start start 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25040ae520 0x7f2504169bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f2504157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040a9a20 con 0x7f250406c7c0 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040a8c00 con 0x7f25040ae520 2023-12-19T14:55:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2504145f70 con 0x7f25041552f0 2023-12-19T14:55:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48800/0 (socket says 172.21.15.50:48800) 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 -- 172.21.15.50:0/3942252074 learned_addr learned my addr 172.21.15.50:0/3942252074 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2508d22700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f2504157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f24fbfff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25040ae520 0x7f2504169bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 -- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 msgr2=0x7f2504157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f2504157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 -- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25040ae520 msgr2=0x7f2504169bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25040ae520 0x7f2504169bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 -- 172.21.15.50:0/3942252074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc00bcf0 con 0x7f250406c7c0 2023-12-19T14:55:39.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2503fff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f24f400d890 tx=0x7f24f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f400b600 con 0x7f250406c7c0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040c46d0 con 0x7f250406c7c0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f400b760 con 0x7f250406c7c0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f400bae0 con 0x7f250406c7c0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.625+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25040c4080 con 0x7f250406c7c0 2023-12-19T14:55:39.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.629+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2504071120 con 0x7f250406c7c0 2023-12-19T14:55:39.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.629+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f24f401e460 con 0x7f250406c7c0 2023-12-19T14:55:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.633+0000 7f2500ff9700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 0x7f24e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.633+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f24f407d2b0 con 0x7f250406c7c0 2023-12-19T14:55:39.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.633+0000 7f24fbfff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 0x7f24e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:39.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.633+0000 7f24fbfff700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 0x7f24e4062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2504068580 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:39.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.637+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f24f404d590 con 0x7f250406c7c0 2023-12-19T14:55:39.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.817+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25040642a0 con 0x7f250406c7c0 2023-12-19T14:55:39.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.817+0000 7f2500ff9700 1 -- 172.21.15.50:0/3942252074 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+15612 (secure 0 0 0) 0x7f24f4050bb0 con 0x7f250406c7c0 2023-12-19T14:55:39.822 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:39.823 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":113,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:55:36.653960+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 msgr2=0x7f24e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 0x7f24e4062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2504068580 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 msgr2=0x7f2504145a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f24f400d890 tx=0x7f24f4008040 comp rx=0 tx=0).stop 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 shutdown_connections 2023-12-19T14:55:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25040ae520 0x7f2504169bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f24e4060670 0x7f24e4062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f25041552f0 0x7f2504157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 --2- 172.21.15.50:0/3942252074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f250406c7c0 0x7f2504145a30 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 >> 172.21.15.50:0/3942252074 conn(0x7f25040c8850 msgr2=0x7f25040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 shutdown_connections 2023-12-19T14:55:39.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:39.821+0000 7f250a785700 1 -- 172.21.15.50:0/3942252074 wait complete. 2023-12-19T14:55:39.849 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:55:39.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:55:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.009+0000 7f58c5a8e700 1 Processor -- start 2023-12-19T14:55:40.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.009+0000 7f58c5a8e700 1 -- start start 2023-12-19T14:55:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c5a8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58c0159530 0x7f58c00c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c5a8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f58c0158200 0x7f58c0158660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c5a8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c00accb0 con 0x7f58c0159160 2023-12-19T14:55:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c5a8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c006a7b0 con 0x7f58c0158200 2023-12-19T14:55:40.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c5a8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c00ad550 con 0x7f58c0159530 2023-12-19T14:55:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f58c0158200 0x7f58c0158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f58c0158200 0x7f58c0158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48812/0 (socket says 172.21.15.50:48812) 2023-12-19T14:55:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 -- 172.21.15.50:0/3214006445 learned_addr learned my addr 172.21.15.50:0/3214006445 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58b7fff700 1 --2- 172.21.15.50:0/3214006445 >> v2:172.21.15.125:3300/0 conn(0x7f58c0159530 0x7f58c00c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 -- 172.21.15.50:0/3214006445 >> v2:172.21.15.125:3300/0 conn(0x7f58c0159530 msgr2=0x7f58c00c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 --2- 172.21.15.50:0/3214006445 >> v2:172.21.15.125:3300/0 conn(0x7f58c0159530 0x7f58c00c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 -- 172.21.15.50:0/3214006445 >> v1:172.21.15.125:6789/0 conn(0x7f58c0159160 legacy=0x7f58c00adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 -- 172.21.15.50:0/3214006445 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00acf90 con 0x7f58c0158200 2023-12-19T14:55:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.013+0000 7f58c528d700 1 --2- 172.21.15.50:0/3214006445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0158660 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f58c006eb00 tx=0x7f58b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9082dacabc08c97 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58b77fe700 1 -- 172.21.15.50:0/3214006445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b800e070 con 0x7f58c0158200 2023-12-19T14:55:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58b77fe700 1 -- 172.21.15.50:0/3214006445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800ba20 con 0x7f58c0158200 2023-12-19T14:55:40.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58b77fe700 1 -- 172.21.15.50:0/3214006445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b80092e0 con 0x7f58c0158200 2023-12-19T14:55:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 -- 172.21.15.50:0/3214006445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 msgr2=0x7f58c0158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/3214006445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0158660 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f58c006eb00 tx=0x7f58b8009840 comp rx=0 tx=0).stop 2023-12-19T14:55:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 -- 172.21.15.50:0/3214006445 shutdown_connections 2023-12-19T14:55:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/3214006445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0158660 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/3214006445 >> v2:172.21.15.125:3300/0 conn(0x7f58c0159530 0x7f58c00c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 -- 172.21.15.50:0/3214006445 >> 172.21.15.50:0/3214006445 conn(0x7f58c0067bb0 msgr2=0x7f58c00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 -- 172.21.15.50:0/3214006445 shutdown_connections 2023-12-19T14:55:40.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.017+0000 7f58c5a8e700 1 -- 172.21.15.50:0/3214006445 wait complete. 2023-12-19T14:55:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 Processor -- start 2023-12-19T14:55:40.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 -- start start 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58c0159160 0x7f58c01690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c0159530 0x7f58c0169600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c0155d80 con 0x7f58c0158200 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c01543a0 con 0x7f58c0159530 2023-12-19T14:55:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c5a8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c014c560 con 0x7f58c0159160 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.021+0000 7f58c4a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48828/0 (socket says 172.21.15.50:48828) 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 -- 172.21.15.50:0/261780597 learned_addr learned my addr 172.21.15.50:0/261780597 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58b7fff700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58c0159160 0x7f58c01690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c528d700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c0159530 0x7f58c0169600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 -- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58c0159160 msgr2=0x7f58c01690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58c0159160 0x7f58c01690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 -- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c0159530 msgr2=0x7f58c0169600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c0159530 0x7f58c0169600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 -- 172.21.15.50:0/261780597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b8009ae0 con 0x7f58c0158200 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c4a8c700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c590 tx=0x7f58bc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc011590 con 0x7f58c0158200 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc011710 con 0x7f58c0158200 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc011ad0 con 0x7f58c0158200 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c0154640 con 0x7f58c0158200 2023-12-19T14:55:40.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c0154c00 con 0x7f58c0158200 2023-12-19T14:55:40.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.025+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c004be40 con 0x7f58c0158200 2023-12-19T14:55:40.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.029+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f58bc01b3b0 con 0x7f58c0158200 2023-12-19T14:55:40.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.029+0000 7f58b4ff9700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 0x7f58ac062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.029+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f58bc07c2f0 con 0x7f58c0158200 2023-12-19T14:55:40.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.033+0000 7f58b7fff700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 0x7f58ac062c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.033+0000 7f58b7fff700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 0x7f58ac062c70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58c00aa5e0 tx=0x7f58b00091a0 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.037+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f58bc04c6d0 con 0x7f58c0158200 2023-12-19T14:55:40.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.205+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58c00aaed0 con 0x7f58ac0607c0 2023-12-19T14:55:40.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.229+0000 7f58b4ff9700 1 -- 172.21.15.50:0/261780597 <== mgr.7057 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287086 (secure 0 0 0) 0x7f58c00aaed0 con 0x7f58ac0607c0 2023-12-19T14:55:40.235 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 msgr2=0x7f58ac062c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 0x7f58ac062c70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58c00aa5e0 tx=0x7f58b00091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 msgr2=0x7f58c0151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c590 tx=0x7f58bc00c8a0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 shutdown_connections 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c0159530 0x7f58c0169600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f58ac0607c0 0x7f58ac062c70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58c0159160 0x7f58c01690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 --2- 172.21.15.50:0/261780597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f58c0158200 0x7f58c0151df0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 >> 172.21.15.50:0/261780597 conn(0x7f58c0067bb0 msgr2=0x7f58c014deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 shutdown_connections 2023-12-19T14:55:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.233+0000 7f58c5a8e700 1 -- 172.21.15.50:0/261780597 wait complete. 2023-12-19T14:55:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:55:40.257 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T14:55:38.994250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443564,"kb_used_data":12012,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752531156,"statfs":{"total":773094113280,"available":770591903744,"internally_reserved":0,"allocated":12300288,"data_stored":6710458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.377588"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682552+0000","last_change":"2023-12-19T14:55:35.677709+0000","last_active":"2023-12-19T14:55:35.682552+0000","last_peered":"2023-12-19T14:55:35.682552+0000","last_clean":"2023-12-19T14:55:35.682552+0000","last_became_active":"2023-12-19T14:55:35.677426+0000","last_became_peered":"2023-12-19T14:55:35.677426+0000","last_unstale":"2023-12-19T14:55:35.682552+0000","last_undegraded":"2023-12-19T14:55:35.682552+0000","last_fullsized":"2023-12-19T14:55:35.682552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.667166+0000","last_change":"2023-12-19T14:55:35.667166+0000","last_active":"2023-12-19T14:55:35.667166+0000","last_peered":"2023-12-19T14:55:35.667166+0000","last_clean":"2023-12-19T14:55:35.667166+0000","last_became_active":"2023-12-19T14:55:35.666851+0000","last_became_peered":"2023-12-19T14:55:35.666851+0000","last_unstale":"2023-12-19T14:55:35.667166+0000","last_undegraded":"2023-12-19T14:55:35.667166+0000","last_fullsized":"2023-12-19T14:55:35.667166+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:34:31.573812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678369+0000","last_change":"2023-12-19T14:55:35.676784+0000","last_active":"2023-12-19T14:55:35.678369+0000","last_peered":"2023-12-19T14:55:35.678369+0000","last_clean":"2023-12-19T14:55:35.678369+0000","last_became_active":"2023-12-19T14:55:35.676448+0000","last_became_peered":"2023-12-19T14:55:35.676448+0000","last_unstale":"2023-12-19T14:55:35.678369+0000","last_undegraded":"2023-12-19T14:55:35.678369+0000","last_fullsized":"2023-12-19T14:55:35.678369+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:22:17.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685813+0000","last_change":"2023-12-19T14:55:35.667151+0000","last_active":"2023-12-19T14:55:35.685813+0000","last_peered":"2023-12-19T14:55:35.685813+0000","last_clean":"2023-12-19T14:55:35.685813+0000","last_became_active":"2023-12-19T14:55:35.666870+0000","last_became_peered":"2023-12-19T14:55:35.666870+0000","last_unstale":"2023-12-19T14:55:35.685813+0000","last_undegraded":"2023-12-19T14:55:35.685813+0000","last_fullsized":"2023-12-19T14:55:35.685813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:49:22.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.665276+0000","last_change":"2023-12-19T14:55:35.665276+0000","last_active":"2023-12-19T14:55:35.665276+0000","last_peered":"2023-12-19T14:55:35.665276+0000","last_clean":"2023-12-19T14:55:35.665276+0000","last_became_active":"2023-12-19T14:55:35.664975+0000","last_became_peered":"2023-12-19T14:55:35.664975+0000","last_unstale":"2023-12-19T14:55:35.665276+0000","last_undegraded":"2023-12-19T14:55:35.665276+0000","last_fullsized":"2023-12-19T14:55:35.665276+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.515354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.684168+0000","last_change":"2023-12-19T14:55:35.684168+0000","last_active":"2023-12-19T14:55:35.684168+0000","last_peered":"2023-12-19T14:55:35.684168+0000","last_clean":"2023-12-19T14:55:35.684168+0000","last_became_active":"2023-12-19T14:55:35.683852+0000","last_became_peered":"2023-12-19T14:55:35.683852+0000","last_unstale":"2023-12-19T14:55:35.684168+0000","last_undegraded":"2023-12-19T14:55:35.684168+0000","last_fullsized":"2023-12-19T14:55:35.684168+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:38.736254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.666095+0000","last_change":"2023-12-19T14:55:35.666095+0000","last_active":"2023-12-19T14:55:35.666095+0000","last_peered":"2023-12-19T14:55:35.666095+0000","last_clean":"2023-12-19T14:55:35.666095+0000","last_became_active":"2023-12-19T14:55:35.665699+0000","last_became_peered":"2023-12-19T14:55:35.665699+0000","last_unstale":"2023-12-19T14:55:35.666095+0000","last_undegraded":"2023-12-19T14:55:35.666095+0000","last_fullsized":"2023-12-19T14:55:35.666095+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:45:11.056272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.689084+0000","last_change":"2023-12-19T14:55:35.668252+0000","last_active":"2023-12-19T14:55:35.689084+0000","last_peered":"2023-12-19T14:55:35.689084+0000","last_clean":"2023-12-19T14:55:35.689084+0000","last_became_active":"2023-12-19T14:55:35.667925+0000","last_became_peered":"2023-12-19T14:55:35.667925+0000","last_unstale":"2023-12-19T14:55:35.689084+0000","last_undegraded":"2023-12-19T14:55:35.689084+0000","last_fullsized":"2023-12-19T14:55:35.689084+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:39.382342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.663513+0000","last_change":"2023-12-19T14:55:35.663513+0000","last_active":"2023-12-19T14:55:35.663513+0000","last_peered":"2023-12-19T14:55:35.663513+0000","last_clean":"2023-12-19T14:55:35.663513+0000","last_became_active":"2023-12-19T14:55:35.663215+0000","last_became_peered":"2023-12-19T14:55:35.663215+0000","last_unstale":"2023-12-19T14:55:35.663513+0000","last_undegraded":"2023-12-19T14:55:35.663513+0000","last_fullsized":"2023-12-19T14:55:35.663513+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:24:30.746073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.686392+0000","last_change":"2023-12-19T14:55:35.686392+0000","last_active":"2023-12-19T14:55:35.686392+0000","last_peered":"2023-12-19T14:55:35.686392+0000","last_clean":"2023-12-19T14:55:35.686392+0000","last_became_active":"2023-12-19T14:55:35.685922+0000","last_became_peered":"2023-12-19T14:55:35.685922+0000","last_unstale":"2023-12-19T14:55:35.686392+0000","last_undegraded":"2023-12-19T14:55:35.686392+0000","last_fullsized":"2023-12-19T14:55:35.686392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:16.569653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678001+0000","last_change":"2023-12-19T14:55:35.678001+0000","last_active":"2023-12-19T14:55:35.678001+0000","last_peered":"2023-12-19T14:55:35.678001+0000","last_clean":"2023-12-19T14:55:35.678001+0000","last_became_active":"2023-12-19T14:55:35.677727+0000","last_became_peered":"2023-12-19T14:55:35.677727+0000","last_unstale":"2023-12-19T14:55:35.678001+0000","last_undegraded":"2023-12-19T14:55:35.678001+0000","last_fullsized":"2023-12-19T14:55:35.678001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:17.506894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.675479+0000","last_change":"2023-12-19T14:55:35.675479+0000","last_active":"2023-12-19T14:55:35.675479+0000","last_peered":"2023-12-19T14:55:35.675479+0000","last_clean":"2023-12-19T14:55:35.675479+0000","last_became_active":"2023-12-19T14:55:35.675169+0000","last_became_peered":"2023-12-19T14:55:35.675169+0000","last_unstale":"2023-12-19T14:55:35.675479+0000","last_undegraded":"2023-12-19T14:55:35.675479+0000","last_fullsized":"2023-12-19T14:55:35.675479+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:35:12.576101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677866+0000","last_change":"2023-12-19T14:55:35.659592+0000","last_active":"2023-12-19T14:55:35.677866+0000","last_peered":"2023-12-19T14:55:35.677866+0000","last_clean":"2023-12-19T14:55:35.677866+0000","last_became_active":"2023-12-19T14:55:35.659257+0000","last_became_peered":"2023-12-19T14:55:35.659257+0000","last_unstale":"2023-12-19T14:55:35.677866+0000","last_undegraded":"2023-12-19T14:55:35.677866+0000","last_fullsized":"2023-12-19T14:55:35.677866+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:43.880340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.684342+0000","last_change":"2023-12-19T14:55:35.684342+0000","last_active":"2023-12-19T14:55:35.684342+0000","last_peered":"2023-12-19T14:55:35.684342+0000","last_clean":"2023-12-19T14:55:35.684342+0000","last_became_active":"2023-12-19T14:55:35.683988+0000","last_became_peered":"2023-12-19T14:55:35.683988+0000","last_unstale":"2023-12-19T14:55:35.684342+0000","last_undegraded":"2023-12-19T14:55:35.684342+0000","last_fullsized":"2023-12-19T14:55:35.684342+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:18:53.585873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669072+0000","last_change":"2023-12-19T14:55:35.658197+0000","last_active":"2023-12-19T14:55:35.669072+0000","last_peered":"2023-12-19T14:55:35.669072+0000","last_clean":"2023-12-19T14:55:35.669072+0000","last_became_active":"2023-12-19T14:55:35.656578+0000","last_became_peered":"2023-12-19T14:55:35.656578+0000","last_unstale":"2023-12-19T14:55:35.669072+0000","last_undegraded":"2023-12-19T14:55:35.669072+0000","last_fullsized":"2023-12-19T14:55:35.669072+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678693+0000","last_change":"2023-12-19T14:55:35.678693+0000","last_active":"2023-12-19T14:55:35.678693+0000","last_peered":"2023-12-19T14:55:35.678693+0000","last_clean":"2023-12-19T14:55:35.678693+0000","last_became_active":"2023-12-19T14:55:35.678341+0000","last_became_peered":"2023-12-19T14:55:35.678341+0000","last_unstale":"2023-12-19T14:55:35.678693+0000","last_undegraded":"2023-12-19T14:55:35.678693+0000","last_fullsized":"2023-12-19T14:55:35.678693+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:05:58.727283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679694+0000","last_change":"2023-12-19T14:55:35.679694+0000","last_active":"2023-12-19T14:55:35.679694+0000","last_peered":"2023-12-19T14:55:35.679694+0000","last_clean":"2023-12-19T14:55:35.679694+0000","last_became_active":"2023-12-19T14:55:35.679385+0000","last_became_peered":"2023-12-19T14:55:35.679385+0000","last_unstale":"2023-12-19T14:55:35.679694+0000","last_undegraded":"2023-12-19T14:55:35.679694+0000","last_fullsized":"2023-12-19T14:55:35.679694+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:28:04.118006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669346+0000","last_change":"2023-12-19T14:55:35.660392+0000","last_active":"2023-12-19T14:55:35.669346+0000","last_peered":"2023-12-19T14:55:35.669346+0000","last_clean":"2023-12-19T14:55:35.669346+0000","last_became_active":"2023-12-19T14:55:35.660062+0000","last_became_peered":"2023-12-19T14:55:35.660062+0000","last_unstale":"2023-12-19T14:55:35.669346+0000","last_undegraded":"2023-12-19T14:55:35.669346+0000","last_fullsized":"2023-12-19T14:55:35.669346+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680296+0000","last_change":"2023-12-19T14:55:35.680296+0000","last_active":"2023-12-19T14:55:35.680296+0000","last_peered":"2023-12-19T14:55:35.680296+0000","last_clean":"2023-12-19T14:55:35.680296+0000","last_became_active":"2023-12-19T14:55:35.680005+0000","last_became_peered":"2023-12-19T14:55:35.680005+0000","last_unstale":"2023-12-19T14:55:35.680296+0000","last_undegraded":"2023-12-19T14:55:35.680296+0000","last_fullsized":"2023-12-19T14:55:35.680296+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:40.305557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685835+0000","last_change":"2023-12-19T14:55:35.668486+0000","last_active":"2023-12-19T14:55:35.685835+0000","last_peered":"2023-12-19T14:55:35.685835+0000","last_clean":"2023-12-19T14:55:35.685835+0000","last_became_active":"2023-12-19T14:55:35.667976+0000","last_became_peered":"2023-12-19T14:55:35.667976+0000","last_unstale":"2023-12-19T14:55:35.685835+0000","last_undegraded":"2023-12-19T14:55:35.685835+0000","last_fullsized":"2023-12-19T14:55:35.685835+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:16.774265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669752+0000","last_change":"2023-12-19T14:55:35.669752+0000","last_active":"2023-12-19T14:55:35.669752+0000","last_peered":"2023-12-19T14:55:35.669752+0000","last_clean":"2023-12-19T14:55:35.669752+0000","last_became_active":"2023-12-19T14:55:35.669386+0000","last_became_peered":"2023-12-19T14:55:35.669386+0000","last_unstale":"2023-12-19T14:55:35.669752+0000","last_undegraded":"2023-12-19T14:55:35.669752+0000","last_fullsized":"2023-12-19T14:55:35.669752+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:07.306981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680231+0000","last_change":"2023-12-19T14:55:35.680231+0000","last_active":"2023-12-19T14:55:35.680231+0000","last_peered":"2023-12-19T14:55:35.680231+0000","last_clean":"2023-12-19T14:55:35.680231+0000","last_became_active":"2023-12-19T14:55:35.679952+0000","last_became_peered":"2023-12-19T14:55:35.679952+0000","last_unstale":"2023-12-19T14:55:35.680231+0000","last_undegraded":"2023-12-19T14:55:35.680231+0000","last_fullsized":"2023-12-19T14:55:35.680231+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:47.582564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683923+0000","last_change":"2023-12-19T14:55:35.683923+0000","last_active":"2023-12-19T14:55:35.683923+0000","last_peered":"2023-12-19T14:55:35.683923+0000","last_clean":"2023-12-19T14:55:35.683923+0000","last_became_active":"2023-12-19T14:55:35.683400+0000","last_became_peered":"2023-12-19T14:55:35.683400+0000","last_unstale":"2023-12-19T14:55:35.683923+0000","last_undegraded":"2023-12-19T14:55:35.683923+0000","last_fullsized":"2023-12-19T14:55:35.683923+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:57:17.889446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677746+0000","last_change":"2023-12-19T14:55:35.675846+0000","last_active":"2023-12-19T14:55:35.677746+0000","last_peered":"2023-12-19T14:55:35.677746+0000","last_clean":"2023-12-19T14:55:35.677746+0000","last_became_active":"2023-12-19T14:55:35.675533+0000","last_became_peered":"2023-12-19T14:55:35.675533+0000","last_unstale":"2023-12-19T14:55:35.677746+0000","last_undegraded":"2023-12-19T14:55:35.677746+0000","last_fullsized":"2023-12-19T14:55:35.677746+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:57:03.667084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664002+0000","last_change":"2023-12-19T14:55:35.664002+0000","last_active":"2023-12-19T14:55:35.664002+0000","last_peered":"2023-12-19T14:55:35.664002+0000","last_clean":"2023-12-19T14:55:35.664002+0000","last_became_active":"2023-12-19T14:55:35.663721+0000","last_became_peered":"2023-12-19T14:55:35.663721+0000","last_unstale":"2023-12-19T14:55:35.664002+0000","last_undegraded":"2023-12-19T14:55:35.664002+0000","last_fullsized":"2023-12-19T14:55:35.664002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:13.153722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680430+0000","last_change":"2023-12-19T14:55:35.680430+0000","last_active":"2023-12-19T14:55:35.680430+0000","last_peered":"2023-12-19T14:55:35.680430+0000","last_clean":"2023-12-19T14:55:35.680430+0000","last_became_active":"2023-12-19T14:55:35.680171+0000","last_became_peered":"2023-12-19T14:55:35.680171+0000","last_unstale":"2023-12-19T14:55:35.680430+0000","last_undegraded":"2023-12-19T14:55:35.680430+0000","last_fullsized":"2023-12-19T14:55:35.680430+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:56:51.429164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678607+0000","last_change":"2023-12-19T14:55:35.678607+0000","last_active":"2023-12-19T14:55:35.678607+0000","last_peered":"2023-12-19T14:55:35.678607+0000","last_clean":"2023-12-19T14:55:35.678607+0000","last_became_active":"2023-12-19T14:55:35.678239+0000","last_became_peered":"2023-12-19T14:55:35.678239+0000","last_unstale":"2023-12-19T14:55:35.678607+0000","last_undegraded":"2023-12-19T14:55:35.678607+0000","last_fullsized":"2023-12-19T14:55:35.678607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:32.624382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685853+0000","last_change":"2023-12-19T14:55:35.669577+0000","last_active":"2023-12-19T14:55:35.685853+0000","last_peered":"2023-12-19T14:55:35.685853+0000","last_clean":"2023-12-19T14:55:35.685853+0000","last_became_active":"2023-12-19T14:55:35.668390+0000","last_became_peered":"2023-12-19T14:55:35.668390+0000","last_unstale":"2023-12-19T14:55:35.685853+0000","last_undegraded":"2023-12-19T14:55:35.685853+0000","last_fullsized":"2023-12-19T14:55:35.685853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.670576+0000","last_change":"2023-12-19T14:55:35.670576+0000","last_active":"2023-12-19T14:55:35.670576+0000","last_peered":"2023-12-19T14:55:35.670576+0000","last_clean":"2023-12-19T14:55:35.670576+0000","last_became_active":"2023-12-19T14:55:35.670105+0000","last_became_peered":"2023-12-19T14:55:35.670105+0000","last_unstale":"2023-12-19T14:55:35.670576+0000","last_undegraded":"2023-12-19T14:55:35.670576+0000","last_fullsized":"2023-12-19T14:55:35.670576+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.663956+0000","last_change":"2023-12-19T14:55:35.663956+0000","last_active":"2023-12-19T14:55:35.663956+0000","last_peered":"2023-12-19T14:55:35.663956+0000","last_clean":"2023-12-19T14:55:35.663956+0000","last_became_active":"2023-12-19T14:55:35.663666+0000","last_became_peered":"2023-12-19T14:55:35.663666+0000","last_unstale":"2023-12-19T14:55:35.663956+0000","last_undegraded":"2023-12-19T14:55:35.663956+0000","last_fullsized":"2023-12-19T14:55:35.663956+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:21:45.290974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.681429+0000","last_change":"2023-12-19T14:55:35.681429+0000","last_active":"2023-12-19T14:55:35.681429+0000","last_peered":"2023-12-19T14:55:35.681429+0000","last_clean":"2023-12-19T14:55:35.681429+0000","last_became_active":"2023-12-19T14:55:35.681072+0000","last_became_peered":"2023-12-19T14:55:35.681072+0000","last_unstale":"2023-12-19T14:55:35.681429+0000","last_undegraded":"2023-12-19T14:55:35.681429+0000","last_fullsized":"2023-12-19T14:55:35.681429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.917699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678662+0000","last_change":"2023-12-19T14:55:35.678662+0000","last_active":"2023-12-19T14:55:35.678662+0000","last_peered":"2023-12-19T14:55:35.678662+0000","last_clean":"2023-12-19T14:55:35.678662+0000","last_became_active":"2023-12-19T14:55:35.677985+0000","last_became_peered":"2023-12-19T14:55:35.677985+0000","last_unstale":"2023-12-19T14:55:35.678662+0000","last_undegraded":"2023-12-19T14:55:35.678662+0000","last_fullsized":"2023-12-19T14:55:35.678662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:53:56.104030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679260+0000","last_change":"2023-12-19T14:55:35.679260+0000","last_active":"2023-12-19T14:55:35.679260+0000","last_peered":"2023-12-19T14:55:35.679260+0000","last_clean":"2023-12-19T14:55:35.679260+0000","last_became_active":"2023-12-19T14:55:35.676157+0000","last_became_peered":"2023-12-19T14:55:35.676157+0000","last_unstale":"2023-12-19T14:55:35.679260+0000","last_undegraded":"2023-12-19T14:55:35.679260+0000","last_fullsized":"2023-12-19T14:55:35.679260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:33.080441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"112'4","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.691675+0000","last_change":"2023-12-19T14:55:34.663860+0000","last_active":"2023-12-19T14:55:35.691675+0000","last_peered":"2023-12-19T14:55:35.691675+0000","last_clean":"2023-12-19T14:55:35.691675+0000","last_became_active":"2023-12-19T14:55:34.663331+0000","last_became_peered":"2023-12-19T14:55:34.663331+0000","last_unstale":"2023-12-19T14:55:35.691675+0000","last_undegraded":"2023-12-19T14:55:35.691675+0000","last_fullsized":"2023-12-19T14:55:35.691675+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:16.479157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685141+0000","last_change":"2023-12-19T14:55:35.685141+0000","last_active":"2023-12-19T14:55:35.685141+0000","last_peered":"2023-12-19T14:55:35.685141+0000","last_clean":"2023-12-19T14:55:35.685141+0000","last_became_active":"2023-12-19T14:55:35.684738+0000","last_became_peered":"2023-12-19T14:55:35.684738+0000","last_unstale":"2023-12-19T14:55:35.685141+0000","last_undegraded":"2023-12-19T14:55:35.685141+0000","last_fullsized":"2023-12-19T14:55:35.685141+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.917699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.655484+0000","last_change":"2023-12-19T14:55:34.661920+0000","last_active":"2023-12-19T14:55:35.655484+0000","last_peered":"2023-12-19T14:55:35.655484+0000","last_clean":"2023-12-19T14:55:35.655484+0000","last_became_active":"2023-12-19T14:55:34.661307+0000","last_became_peered":"2023-12-19T14:55:34.661307+0000","last_unstale":"2023-12-19T14:55:35.655484+0000","last_undegraded":"2023-12-19T14:55:35.655484+0000","last_fullsized":"2023-12-19T14:55:35.655484+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:10.719896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680454+0000","last_change":"2023-12-19T14:55:35.680454+0000","last_active":"2023-12-19T14:55:35.680454+0000","last_peered":"2023-12-19T14:55:35.680454+0000","last_clean":"2023-12-19T14:55:35.680454+0000","last_became_active":"2023-12-19T14:55:35.680190+0000","last_became_peered":"2023-12-19T14:55:35.680190+0000","last_unstale":"2023-12-19T14:55:35.680454+0000","last_undegraded":"2023-12-19T14:55:35.680454+0000","last_fullsized":"2023-12-19T14:55:35.680454+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:07.306981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682933+0000","last_change":"2023-12-19T14:55:34.661290+0000","last_active":"2023-12-19T14:55:35.682933+0000","last_peered":"2023-12-19T14:55:35.682933+0000","last_clean":"2023-12-19T14:55:35.682933+0000","last_became_active":"2023-12-19T14:55:34.660961+0000","last_became_peered":"2023-12-19T14:55:34.660961+0000","last_unstale":"2023-12-19T14:55:35.682933+0000","last_undegraded":"2023-12-19T14:55:35.682933+0000","last_fullsized":"2023-12-19T14:55:35.682933+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:41.162344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679028+0000","last_change":"2023-12-19T14:55:35.679028+0000","last_active":"2023-12-19T14:55:35.679028+0000","last_peered":"2023-12-19T14:55:35.679028+0000","last_clean":"2023-12-19T14:55:35.679028+0000","last_became_active":"2023-12-19T14:55:35.673824+0000","last_became_peered":"2023-12-19T14:55:35.673824+0000","last_unstale":"2023-12-19T14:55:35.679028+0000","last_undegraded":"2023-12-19T14:55:35.679028+0000","last_fullsized":"2023-12-19T14:55:35.679028+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:37:15.646881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685468+0000","last_change":"2023-12-19T14:55:34.666585+0000","last_active":"2023-12-19T14:55:35.685468+0000","last_peered":"2023-12-19T14:55:35.685468+0000","last_clean":"2023-12-19T14:55:35.685468+0000","last_became_active":"2023-12-19T14:55:34.665522+0000","last_became_peered":"2023-12-19T14:55:34.665522+0000","last_unstale":"2023-12-19T14:55:35.685468+0000","last_undegraded":"2023-12-19T14:55:35.685468+0000","last_fullsized":"2023-12-19T14:55:35.685468+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:25.682306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.659753+0000","last_change":"2023-12-19T14:55:35.655324+0000","last_active":"2023-12-19T14:55:35.659753+0000","last_peered":"2023-12-19T14:55:35.659753+0000","last_clean":"2023-12-19T14:55:35.659753+0000","last_became_active":"2023-12-19T14:55:35.654964+0000","last_became_peered":"2023-12-19T14:55:35.654964+0000","last_unstale":"2023-12-19T14:55:35.659753+0000","last_undegraded":"2023-12-19T14:55:35.659753+0000","last_fullsized":"2023-12-19T14:55:35.659753+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:22:17.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658517+0000","last_change":"2023-12-19T14:55:34.667483+0000","last_active":"2023-12-19T14:55:35.658517+0000","last_peered":"2023-12-19T14:55:35.658517+0000","last_clean":"2023-12-19T14:55:35.658517+0000","last_became_active":"2023-12-19T14:55:34.666862+0000","last_became_peered":"2023-12-19T14:55:34.666862+0000","last_unstale":"2023-12-19T14:55:35.658517+0000","last_undegraded":"2023-12-19T14:55:35.658517+0000","last_fullsized":"2023-12-19T14:55:35.658517+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:49:56.446441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679688+0000","last_change":"2023-12-19T14:55:35.679688+0000","last_active":"2023-12-19T14:55:35.679688+0000","last_peered":"2023-12-19T14:55:35.679688+0000","last_clean":"2023-12-19T14:55:35.679688+0000","last_became_active":"2023-12-19T14:55:35.677391+0000","last_became_peered":"2023-12-19T14:55:35.677391+0000","last_unstale":"2023-12-19T14:55:35.679688+0000","last_undegraded":"2023-12-19T14:55:35.679688+0000","last_fullsized":"2023-12-19T14:55:35.679688+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:49:57.460332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.694269+0000","last_change":"2023-12-19T14:55:34.674680+0000","last_active":"2023-12-19T14:55:35.694269+0000","last_peered":"2023-12-19T14:55:35.694269+0000","last_clean":"2023-12-19T14:55:35.694269+0000","last_became_active":"2023-12-19T14:55:34.674379+0000","last_became_peered":"2023-12-19T14:55:34.674379+0000","last_unstale":"2023-12-19T14:55:35.694269+0000","last_undegraded":"2023-12-19T14:55:35.694269+0000","last_fullsized":"2023-12-19T14:55:35.694269+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:56:19.598180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.696714+0000","last_change":"2023-12-19T14:55:35.696714+0000","last_active":"2023-12-19T14:55:35.696714+0000","last_peered":"2023-12-19T14:55:35.696714+0000","last_clean":"2023-12-19T14:55:35.696714+0000","last_became_active":"2023-12-19T14:55:35.696395+0000","last_became_peered":"2023-12-19T14:55:35.696395+0000","last_unstale":"2023-12-19T14:55:35.696714+0000","last_undegraded":"2023-12-19T14:55:35.696714+0000","last_fullsized":"2023-12-19T14:55:35.696714+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:46.844201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.651512+0000","last_change":"2023-12-19T14:55:34.658231+0000","last_active":"2023-12-19T14:55:35.651512+0000","last_peered":"2023-12-19T14:55:35.651512+0000","last_clean":"2023-12-19T14:55:35.651512+0000","last_became_active":"2023-12-19T14:55:34.657931+0000","last_became_peered":"2023-12-19T14:55:34.657931+0000","last_unstale":"2023-12-19T14:55:35.651512+0000","last_undegraded":"2023-12-19T14:55:35.651512+0000","last_fullsized":"2023-12-19T14:55:35.651512+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:06.195525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685272+0000","last_change":"2023-12-19T14:55:35.667105+0000","last_active":"2023-12-19T14:55:35.685272+0000","last_peered":"2023-12-19T14:55:35.685272+0000","last_clean":"2023-12-19T14:55:35.685272+0000","last_became_active":"2023-12-19T14:55:35.666746+0000","last_became_peered":"2023-12-19T14:55:35.666746+0000","last_unstale":"2023-12-19T14:55:35.685272+0000","last_undegraded":"2023-12-19T14:55:35.685272+0000","last_fullsized":"2023-12-19T14:55:35.685272+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:57:03.667084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.674062+0000","last_change":"2023-12-19T14:55:34.673102+0000","last_active":"2023-12-19T14:55:34.674062+0000","last_peered":"2023-12-19T14:55:34.674062+0000","last_clean":"2023-12-19T14:55:34.674062+0000","last_became_active":"2023-12-19T14:55:34.672821+0000","last_became_peered":"2023-12-19T14:55:34.672821+0000","last_unstale":"2023-12-19T14:55:34.674062+0000","last_undegraded":"2023-12-19T14:55:34.674062+0000","last_fullsized":"2023-12-19T14:55:34.674062+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:39.166215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.668175+0000","last_change":"2023-12-19T14:55:35.668175+0000","last_active":"2023-12-19T14:55:35.668175+0000","last_peered":"2023-12-19T14:55:35.668175+0000","last_clean":"2023-12-19T14:55:35.668175+0000","last_became_active":"2023-12-19T14:55:35.667356+0000","last_became_peered":"2023-12-19T14:55:35.667356+0000","last_unstale":"2023-12-19T14:55:35.668175+0000","last_undegraded":"2023-12-19T14:55:35.668175+0000","last_fullsized":"2023-12-19T14:55:35.668175+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.670819+0000","last_change":"2023-12-19T14:55:34.668513+0000","last_active":"2023-12-19T14:55:34.670819+0000","last_peered":"2023-12-19T14:55:34.670819+0000","last_clean":"2023-12-19T14:55:34.670819+0000","last_became_active":"2023-12-19T14:55:34.667808+0000","last_became_peered":"2023-12-19T14:55:34.667808+0000","last_unstale":"2023-12-19T14:55:34.670819+0000","last_undegraded":"2023-12-19T14:55:34.670819+0000","last_fullsized":"2023-12-19T14:55:34.670819+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:21:44.277083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.670877+0000","last_change":"2023-12-19T14:55:35.670877+0000","last_active":"2023-12-19T14:55:35.670877+0000","last_peered":"2023-12-19T14:55:35.670877+0000","last_clean":"2023-12-19T14:55:35.670877+0000","last_became_active":"2023-12-19T14:55:35.670519+0000","last_became_peered":"2023-12-19T14:55:35.670519+0000","last_unstale":"2023-12-19T14:55:35.670877+0000","last_undegraded":"2023-12-19T14:55:35.670877+0000","last_fullsized":"2023-12-19T14:55:35.670877+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.653831+0000","last_change":"2023-12-19T14:55:34.669270+0000","last_active":"2023-12-19T14:55:35.653831+0000","last_peered":"2023-12-19T14:55:35.653831+0000","last_clean":"2023-12-19T14:55:35.653831+0000","last_became_active":"2023-12-19T14:55:34.668954+0000","last_became_peered":"2023-12-19T14:55:34.668954+0000","last_unstale":"2023-12-19T14:55:35.653831+0000","last_undegraded":"2023-12-19T14:55:35.653831+0000","last_fullsized":"2023-12-19T14:55:35.653831+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:59.931922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683396+0000","last_change":"2023-12-19T14:55:35.678518+0000","last_active":"2023-12-19T14:55:35.683396+0000","last_peered":"2023-12-19T14:55:35.683396+0000","last_clean":"2023-12-19T14:55:35.683396+0000","last_became_active":"2023-12-19T14:55:35.678234+0000","last_became_peered":"2023-12-19T14:55:35.678234+0000","last_unstale":"2023-12-19T14:55:35.683396+0000","last_undegraded":"2023-12-19T14:55:35.683396+0000","last_fullsized":"2023-12-19T14:55:35.683396+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"112'3","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687023+0000","last_change":"2023-12-19T14:55:34.671274+0000","last_active":"2023-12-19T14:55:35.687023+0000","last_peered":"2023-12-19T14:55:35.687023+0000","last_clean":"2023-12-19T14:55:35.687023+0000","last_became_active":"2023-12-19T14:55:34.671005+0000","last_became_peered":"2023-12-19T14:55:34.671005+0000","last_unstale":"2023-12-19T14:55:35.687023+0000","last_undegraded":"2023-12-19T14:55:35.687023+0000","last_fullsized":"2023-12-19T14:55:35.687023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:37:29.341509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677671+0000","last_change":"2023-12-19T14:55:35.661438+0000","last_active":"2023-12-19T14:55:35.677671+0000","last_peered":"2023-12-19T14:55:35.677671+0000","last_clean":"2023-12-19T14:55:35.677671+0000","last_became_active":"2023-12-19T14:55:35.661134+0000","last_became_peered":"2023-12-19T14:55:35.661134+0000","last_unstale":"2023-12-19T14:55:35.677671+0000","last_undegraded":"2023-12-19T14:55:35.677671+0000","last_fullsized":"2023-12-19T14:55:35.677671+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.697340+0000","last_change":"2023-12-19T14:55:34.663318+0000","last_active":"2023-12-19T14:55:35.697340+0000","last_peered":"2023-12-19T14:55:35.697340+0000","last_clean":"2023-12-19T14:55:35.697340+0000","last_became_active":"2023-12-19T14:55:34.663042+0000","last_became_peered":"2023-12-19T14:55:34.663042+0000","last_unstale":"2023-12-19T14:55:35.697340+0000","last_undegraded":"2023-12-19T14:55:35.697340+0000","last_fullsized":"2023-12-19T14:55:35.697340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:27.950808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677568+0000","last_change":"2023-12-19T14:55:35.662672+0000","last_active":"2023-12-19T14:55:35.677568+0000","last_peered":"2023-12-19T14:55:35.677568+0000","last_clean":"2023-12-19T14:55:35.677568+0000","last_became_active":"2023-12-19T14:55:35.662381+0000","last_became_peered":"2023-12-19T14:55:35.662381+0000","last_unstale":"2023-12-19T14:55:35.677568+0000","last_undegraded":"2023-12-19T14:55:35.677568+0000","last_fullsized":"2023-12-19T14:55:35.677568+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:40.180106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.690023+0000","last_change":"2023-12-19T14:55:34.658626+0000","last_active":"2023-12-19T14:55:35.690023+0000","last_peered":"2023-12-19T14:55:35.690023+0000","last_clean":"2023-12-19T14:55:35.690023+0000","last_became_active":"2023-12-19T14:55:34.658333+0000","last_became_peered":"2023-12-19T14:55:34.658333+0000","last_unstale":"2023-12-19T14:55:35.690023+0000","last_undegraded":"2023-12-19T14:55:35.690023+0000","last_fullsized":"2023-12-19T14:55:35.690023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:38.368451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664583+0000","last_change":"2023-12-19T14:55:35.664583+0000","last_active":"2023-12-19T14:55:35.664583+0000","last_peered":"2023-12-19T14:55:35.664583+0000","last_clean":"2023-12-19T14:55:35.664583+0000","last_became_active":"2023-12-19T14:55:35.664303+0000","last_became_peered":"2023-12-19T14:55:35.664303+0000","last_unstale":"2023-12-19T14:55:35.664583+0000","last_undegraded":"2023-12-19T14:55:35.664583+0000","last_fullsized":"2023-12-19T14:55:35.664583+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:09:41.563795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.676775+0000","last_change":"2023-12-19T14:55:34.666726+0000","last_active":"2023-12-19T14:55:34.676775+0000","last_peered":"2023-12-19T14:55:34.676775+0000","last_clean":"2023-12-19T14:55:34.676775+0000","last_became_active":"2023-12-19T14:55:34.666447+0000","last_became_peered":"2023-12-19T14:55:34.666447+0000","last_unstale":"2023-12-19T14:55:34.676775+0000","last_undegraded":"2023-12-19T14:55:34.676775+0000","last_fullsized":"2023-12-19T14:55:34.676775+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:16.687042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687942+0000","last_change":"2023-12-19T14:55:35.687942+0000","last_active":"2023-12-19T14:55:35.687942+0000","last_peered":"2023-12-19T14:55:35.687942+0000","last_clean":"2023-12-19T14:55:35.687942+0000","last_became_active":"2023-12-19T14:55:35.687611+0000","last_became_peered":"2023-12-19T14:55:35.687611+0000","last_unstale":"2023-12-19T14:55:35.687942+0000","last_undegraded":"2023-12-19T14:55:35.687942+0000","last_fullsized":"2023-12-19T14:55:35.687942+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:28:58.905253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.647534+0000","last_change":"2023-12-19T14:55:34.664745+0000","last_active":"2023-12-19T14:55:35.647534+0000","last_peered":"2023-12-19T14:55:35.647534+0000","last_clean":"2023-12-19T14:55:35.647534+0000","last_became_active":"2023-12-19T14:55:34.664179+0000","last_became_peered":"2023-12-19T14:55:34.664179+0000","last_unstale":"2023-12-19T14:55:35.647534+0000","last_undegraded":"2023-12-19T14:55:35.647534+0000","last_fullsized":"2023-12-19T14:55:35.647534+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:07:37.333779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679041+0000","last_change":"2023-12-19T14:55:35.679041+0000","last_active":"2023-12-19T14:55:35.679041+0000","last_peered":"2023-12-19T14:55:35.679041+0000","last_clean":"2023-12-19T14:55:35.679041+0000","last_became_active":"2023-12-19T14:55:35.678377+0000","last_became_peered":"2023-12-19T14:55:35.678377+0000","last_unstale":"2023-12-19T14:55:35.679041+0000","last_undegraded":"2023-12-19T14:55:35.679041+0000","last_fullsized":"2023-12-19T14:55:35.679041+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:10.189861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.698415+0000","last_change":"2023-12-19T14:55:34.672667+0000","last_active":"2023-12-19T14:55:35.698415+0000","last_peered":"2023-12-19T14:55:35.698415+0000","last_clean":"2023-12-19T14:55:35.698415+0000","last_became_active":"2023-12-19T14:55:34.672366+0000","last_became_peered":"2023-12-19T14:55:34.672366+0000","last_unstale":"2023-12-19T14:55:35.698415+0000","last_undegraded":"2023-12-19T14:55:35.698415+0000","last_fullsized":"2023-12-19T14:55:35.698415+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:35.898230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678585+0000","last_change":"2023-12-19T14:55:35.678585+0000","last_active":"2023-12-19T14:55:35.678585+0000","last_peered":"2023-12-19T14:55:35.678585+0000","last_clean":"2023-12-19T14:55:35.678585+0000","last_became_active":"2023-12-19T14:55:35.674892+0000","last_became_peered":"2023-12-19T14:55:35.674892+0000","last_unstale":"2023-12-19T14:55:35.678585+0000","last_undegraded":"2023-12-19T14:55:35.678585+0000","last_fullsized":"2023-12-19T14:55:35.678585+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:15:24.018068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658255+0000","last_change":"2023-12-19T14:55:34.654926+0000","last_active":"2023-12-19T14:55:35.658255+0000","last_peered":"2023-12-19T14:55:35.658255+0000","last_clean":"2023-12-19T14:55:35.658255+0000","last_became_active":"2023-12-19T14:55:34.654257+0000","last_became_peered":"2023-12-19T14:55:34.654257+0000","last_unstale":"2023-12-19T14:55:35.658255+0000","last_undegraded":"2023-12-19T14:55:35.658255+0000","last_fullsized":"2023-12-19T14:55:35.658255+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:15.523614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664416+0000","last_change":"2023-12-19T14:55:35.664416+0000","last_active":"2023-12-19T14:55:35.664416+0000","last_peered":"2023-12-19T14:55:35.664416+0000","last_clean":"2023-12-19T14:55:35.664416+0000","last_became_active":"2023-12-19T14:55:35.664053+0000","last_became_peered":"2023-12-19T14:55:35.664053+0000","last_unstale":"2023-12-19T14:55:35.664416+0000","last_undegraded":"2023-12-19T14:55:35.664416+0000","last_fullsized":"2023-12-19T14:55:35.664416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:11.733787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682252+0000","last_change":"2023-12-19T14:55:34.665593+0000","last_active":"2023-12-19T14:55:35.682252+0000","last_peered":"2023-12-19T14:55:35.682252+0000","last_clean":"2023-12-19T14:55:35.682252+0000","last_became_active":"2023-12-19T14:55:34.664712+0000","last_became_peered":"2023-12-19T14:55:34.664712+0000","last_unstale":"2023-12-19T14:55:35.682252+0000","last_undegraded":"2023-12-19T14:55:35.682252+0000","last_fullsized":"2023-12-19T14:55:35.682252+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:07.297099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679098+0000","last_change":"2023-12-19T14:55:35.679098+0000","last_active":"2023-12-19T14:55:35.679098+0000","last_peered":"2023-12-19T14:55:35.679098+0000","last_clean":"2023-12-19T14:55:35.679098+0000","last_became_active":"2023-12-19T14:55:35.678806+0000","last_became_peered":"2023-12-19T14:55:35.678806+0000","last_unstale":"2023-12-19T14:55:35.679098+0000","last_undegraded":"2023-12-19T14:55:35.679098+0000","last_fullsized":"2023-12-19T14:55:35.679098+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:23.549232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658740+0000","last_change":"2023-12-19T14:55:34.667154+0000","last_active":"2023-12-19T14:55:35.658740+0000","last_peered":"2023-12-19T14:55:35.658740+0000","last_clean":"2023-12-19T14:55:35.658740+0000","last_became_active":"2023-12-19T14:55:34.666819+0000","last_became_peered":"2023-12-19T14:55:34.666819+0000","last_unstale":"2023-12-19T14:55:35.658740+0000","last_undegraded":"2023-12-19T14:55:35.658740+0000","last_fullsized":"2023-12-19T14:55:35.658740+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:59.931922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677704+0000","last_change":"2023-12-19T14:55:35.677704+0000","last_active":"2023-12-19T14:55:35.677704+0000","last_peered":"2023-12-19T14:55:35.677704+0000","last_clean":"2023-12-19T14:55:35.677704+0000","last_became_active":"2023-12-19T14:55:35.675256+0000","last_became_peered":"2023-12-19T14:55:35.675256+0000","last_unstale":"2023-12-19T14:55:35.677704+0000","last_undegraded":"2023-12-19T14:55:35.677704+0000","last_fullsized":"2023-12-19T14:55:35.677704+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:21.964252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.681645+0000","last_change":"2023-12-19T14:55:34.666257+0000","last_active":"2023-12-19T14:55:35.681645+0000","last_peered":"2023-12-19T14:55:35.681645+0000","last_clean":"2023-12-19T14:55:35.681645+0000","last_became_active":"2023-12-19T14:55:34.665972+0000","last_became_peered":"2023-12-19T14:55:34.665972+0000","last_unstale":"2023-12-19T14:55:35.681645+0000","last_undegraded":"2023-12-19T14:55:35.681645+0000","last_fullsized":"2023-12-19T14:55:35.681645+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:16.493003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.667345+0000","last_change":"2023-12-19T14:55:35.667345+0000","last_active":"2023-12-19T14:55:35.667345+0000","last_peered":"2023-12-19T14:55:35.667345+0000","last_clean":"2023-12-19T14:55:35.667345+0000","last_became_active":"2023-12-19T14:55:35.667005+0000","last_became_peered":"2023-12-19T14:55:35.667005+0000","last_unstale":"2023-12-19T14:55:35.667345+0000","last_undegraded":"2023-12-19T14:55:35.667345+0000","last_fullsized":"2023-12-19T14:55:35.667345+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:46.844201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.673553+0000","last_change":"2023-12-19T14:55:34.661994+0000","last_active":"2023-12-19T14:55:34.673553+0000","last_peered":"2023-12-19T14:55:34.673553+0000","last_clean":"2023-12-19T14:55:34.673553+0000","last_became_active":"2023-12-19T14:55:34.661633+0000","last_became_peered":"2023-12-19T14:55:34.661633+0000","last_unstale":"2023-12-19T14:55:34.673553+0000","last_undegraded":"2023-12-19T14:55:34.673553+0000","last_fullsized":"2023-12-19T14:55:34.673553+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:22:49.249620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680955+0000","last_change":"2023-12-19T14:55:35.680955+0000","last_active":"2023-12-19T14:55:35.680955+0000","last_peered":"2023-12-19T14:55:35.680955+0000","last_clean":"2023-12-19T14:55:35.680955+0000","last_became_active":"2023-12-19T14:55:35.680668+0000","last_became_peered":"2023-12-19T14:55:35.680668+0000","last_unstale":"2023-12-19T14:55:35.680955+0000","last_undegraded":"2023-12-19T14:55:35.680955+0000","last_fullsized":"2023-12-19T14:55:35.680955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:37.133407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.674524+0000","last_change":"2023-12-19T14:55:34.673641+0000","last_active":"2023-12-19T14:55:34.674524+0000","last_peered":"2023-12-19T14:55:34.674524+0000","last_clean":"2023-12-19T14:55:34.674524+0000","last_became_active":"2023-12-19T14:55:34.673267+0000","last_became_peered":"2023-12-19T14:55:34.673267+0000","last_unstale":"2023-12-19T14:55:34.674524+0000","last_undegraded":"2023-12-19T14:55:34.674524+0000","last_fullsized":"2023-12-19T14:55:34.674524+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:24:29.732182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677681+0000","last_change":"2023-12-19T14:55:35.677681+0000","last_active":"2023-12-19T14:55:35.677681+0000","last_peered":"2023-12-19T14:55:35.677681+0000","last_clean":"2023-12-19T14:55:35.677681+0000","last_became_active":"2023-12-19T14:55:35.674674+0000","last_became_peered":"2023-12-19T14:55:35.674674+0000","last_unstale":"2023-12-19T14:55:35.677681+0000","last_undegraded":"2023-12-19T14:55:35.677681+0000","last_fullsized":"2023-12-19T14:55:35.677681+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:48.155868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.671232+0000","last_change":"2023-12-19T14:55:34.667695+0000","last_active":"2023-12-19T14:55:34.671232+0000","last_peered":"2023-12-19T14:55:34.671232+0000","last_clean":"2023-12-19T14:55:34.671232+0000","last_became_active":"2023-12-19T14:55:34.667408+0000","last_became_peered":"2023-12-19T14:55:34.667408+0000","last_unstale":"2023-12-19T14:55:34.671232+0000","last_undegraded":"2023-12-19T14:55:34.671232+0000","last_fullsized":"2023-12-19T14:55:34.671232+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:25:10.413744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.661471+0000","last_change":"2023-12-19T14:55:35.661471+0000","last_active":"2023-12-19T14:55:35.661471+0000","last_peered":"2023-12-19T14:55:35.661471+0000","last_clean":"2023-12-19T14:55:35.661471+0000","last_became_active":"2023-12-19T14:55:35.661132+0000","last_became_peered":"2023-12-19T14:55:35.661132+0000","last_unstale":"2023-12-19T14:55:35.661471+0000","last_undegraded":"2023-12-19T14:55:35.661471+0000","last_fullsized":"2023-12-19T14:55:35.661471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:52:38.467131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682925+0000","last_change":"2023-12-19T14:55:34.661337+0000","last_active":"2023-12-19T14:55:35.682925+0000","last_peered":"2023-12-19T14:55:35.682925+0000","last_clean":"2023-12-19T14:55:35.682925+0000","last_became_active":"2023-12-19T14:55:34.661027+0000","last_became_peered":"2023-12-19T14:55:34.661027+0000","last_unstale":"2023-12-19T14:55:35.682925+0000","last_undegraded":"2023-12-19T14:55:35.682925+0000","last_fullsized":"2023-12-19T14:55:35.682925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:37:29.341509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677528+0000","last_change":"2023-12-19T14:55:35.677528+0000","last_active":"2023-12-19T14:55:35.677528+0000","last_peered":"2023-12-19T14:55:35.677528+0000","last_clean":"2023-12-19T14:55:35.677528+0000","last_became_active":"2023-12-19T14:55:35.674797+0000","last_became_peered":"2023-12-19T14:55:35.674797+0000","last_unstale":"2023-12-19T14:55:35.677528+0000","last_undegraded":"2023-12-19T14:55:35.677528+0000","last_fullsized":"2023-12-19T14:55:35.677528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:37.133407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.669853+0000","last_change":"2023-12-19T14:55:34.665684+0000","last_active":"2023-12-19T14:55:34.669853+0000","last_peered":"2023-12-19T14:55:34.669853+0000","last_clean":"2023-12-19T14:55:34.669853+0000","last_became_active":"2023-12-19T14:55:34.665293+0000","last_became_peered":"2023-12-19T14:55:34.665293+0000","last_unstale":"2023-12-19T14:55:34.669853+0000","last_undegraded":"2023-12-19T14:55:34.669853+0000","last_fullsized":"2023-12-19T14:55:34.669853+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:57:16.875555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687696+0000","last_change":"2023-12-19T14:55:35.687696+0000","last_active":"2023-12-19T14:55:35.687696+0000","last_peered":"2023-12-19T14:55:35.687696+0000","last_clean":"2023-12-19T14:55:35.687696+0000","last_became_active":"2023-12-19T14:55:35.687319+0000","last_became_peered":"2023-12-19T14:55:35.687319+0000","last_unstale":"2023-12-19T14:55:35.687696+0000","last_undegraded":"2023-12-19T14:55:35.687696+0000","last_fullsized":"2023-12-19T14:55:35.687696+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:13.153722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660758+0000","last_change":"2023-12-19T14:55:34.659331+0000","last_active":"2023-12-19T14:55:34.660758+0000","last_peered":"2023-12-19T14:55:34.660758+0000","last_clean":"2023-12-19T14:55:34.660758+0000","last_became_active":"2023-12-19T14:55:34.658955+0000","last_became_peered":"2023-12-19T14:55:34.658955+0000","last_unstale":"2023-12-19T14:55:34.660758+0000","last_undegraded":"2023-12-19T14:55:34.660758+0000","last_fullsized":"2023-12-19T14:55:34.660758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:06.195525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669984+0000","last_change":"2023-12-19T14:55:35.669984+0000","last_active":"2023-12-19T14:55:35.669984+0000","last_peered":"2023-12-19T14:55:35.669984+0000","last_clean":"2023-12-19T14:55:35.669984+0000","last_became_active":"2023-12-19T14:55:35.669701+0000","last_became_peered":"2023-12-19T14:55:35.669701+0000","last_unstale":"2023-12-19T14:55:35.669984+0000","last_undegraded":"2023-12-19T14:55:35.669984+0000","last_fullsized":"2023-12-19T14:55:35.669984+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:48.155868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660809+0000","last_change":"2023-12-19T14:55:34.657561+0000","last_active":"2023-12-19T14:55:34.660809+0000","last_peered":"2023-12-19T14:55:34.660809+0000","last_clean":"2023-12-19T14:55:34.660809+0000","last_became_active":"2023-12-19T14:55:34.655549+0000","last_became_peered":"2023-12-19T14:55:34.655549+0000","last_unstale":"2023-12-19T14:55:34.660809+0000","last_undegraded":"2023-12-19T14:55:34.660809+0000","last_fullsized":"2023-12-19T14:55:34.660809+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:23.212689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687772+0000","last_change":"2023-12-19T14:55:35.687772+0000","last_active":"2023-12-19T14:55:35.687772+0000","last_peered":"2023-12-19T14:55:35.687772+0000","last_clean":"2023-12-19T14:55:35.687772+0000","last_became_active":"2023-12-19T14:55:35.687187+0000","last_became_peered":"2023-12-19T14:55:35.687187+0000","last_unstale":"2023-12-19T14:55:35.687772+0000","last_undegraded":"2023-12-19T14:55:35.687772+0000","last_fullsized":"2023-12-19T14:55:35.687772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:54.406543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.671963+0000","last_change":"2023-12-19T14:55:34.662850+0000","last_active":"2023-12-19T14:55:34.671963+0000","last_peered":"2023-12-19T14:55:34.671963+0000","last_clean":"2023-12-19T14:55:34.671963+0000","last_became_active":"2023-12-19T14:55:34.662521+0000","last_became_peered":"2023-12-19T14:55:34.662521+0000","last_unstale":"2023-12-19T14:55:34.671963+0000","last_undegraded":"2023-12-19T14:55:34.671963+0000","last_fullsized":"2023-12-19T14:55:34.671963+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:57.677388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669626+0000","last_change":"2023-12-19T14:55:35.663500+0000","last_active":"2023-12-19T14:55:35.669626+0000","last_peered":"2023-12-19T14:55:35.669626+0000","last_clean":"2023-12-19T14:55:35.669626+0000","last_became_active":"2023-12-19T14:55:35.663194+0000","last_became_peered":"2023-12-19T14:55:35.663194+0000","last_unstale":"2023-12-19T14:55:35.669626+0000","last_undegraded":"2023-12-19T14:55:35.669626+0000","last_fullsized":"2023-12-19T14:55:35.669626+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:16.774265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.669297+0000","last_change":"2023-12-19T14:55:34.665492+0000","last_active":"2023-12-19T14:55:34.669297+0000","last_peered":"2023-12-19T14:55:34.669297+0000","last_clean":"2023-12-19T14:55:34.669297+0000","last_became_active":"2023-12-19T14:55:34.665182+0000","last_became_peered":"2023-12-19T14:55:34.665182+0000","last_unstale":"2023-12-19T14:55:34.669297+0000","last_undegraded":"2023-12-19T14:55:34.669297+0000","last_fullsized":"2023-12-19T14:55:34.669297+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:46.568673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683746+0000","last_change":"2023-12-19T14:55:35.683746+0000","last_active":"2023-12-19T14:55:35.683746+0000","last_peered":"2023-12-19T14:55:35.683746+0000","last_clean":"2023-12-19T14:55:35.683746+0000","last_became_active":"2023-12-19T14:55:35.683407+0000","last_became_peered":"2023-12-19T14:55:35.683407+0000","last_unstale":"2023-12-19T14:55:35.683746+0000","last_undegraded":"2023-12-19T14:55:35.683746+0000","last_fullsized":"2023-12-19T14:55:35.683746+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:57.419174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660217+0000","last_change":"2023-12-19T14:55:34.656987+0000","last_active":"2023-12-19T14:55:34.660217+0000","last_peered":"2023-12-19T14:55:34.660217+0000","last_clean":"2023-12-19T14:55:34.660217+0000","last_became_active":"2023-12-19T14:55:34.656349+0000","last_became_peered":"2023-12-19T14:55:34.656349+0000","last_unstale":"2023-12-19T14:55:34.660217+0000","last_undegraded":"2023-12-19T14:55:34.660217+0000","last_fullsized":"2023-12-19T14:55:34.660217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:15.698174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678774+0000","last_change":"2023-12-19T14:55:35.678774+0000","last_active":"2023-12-19T14:55:35.678774+0000","last_peered":"2023-12-19T14:55:35.678774+0000","last_clean":"2023-12-19T14:55:35.678774+0000","last_became_active":"2023-12-19T14:55:35.677281+0000","last_became_peered":"2023-12-19T14:55:35.677281+0000","last_unstale":"2023-12-19T14:55:35.678774+0000","last_undegraded":"2023-12-19T14:55:35.678774+0000","last_fullsized":"2023-12-19T14:55:35.678774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:24.642145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.668457+0000","last_change":"2023-12-19T14:55:34.666545+0000","last_active":"2023-12-19T14:55:34.668457+0000","last_peered":"2023-12-19T14:55:34.668457+0000","last_clean":"2023-12-19T14:55:34.668457+0000","last_became_active":"2023-12-19T14:55:34.666259+0000","last_became_peered":"2023-12-19T14:55:34.666259+0000","last_unstale":"2023-12-19T14:55:34.668457+0000","last_undegraded":"2023-12-19T14:55:34.668457+0000","last_fullsized":"2023-12-19T14:55:34.668457+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:18.582399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.666155+0000","last_change":"2023-12-19T14:55:35.666155+0000","last_active":"2023-12-19T14:55:35.666155+0000","last_peered":"2023-12-19T14:55:35.666155+0000","last_clean":"2023-12-19T14:55:35.666155+0000","last_became_active":"2023-12-19T14:55:35.665860+0000","last_became_peered":"2023-12-19T14:55:35.665860+0000","last_unstale":"2023-12-19T14:55:35.666155+0000","last_undegraded":"2023-12-19T14:55:35.666155+0000","last_fullsized":"2023-12-19T14:55:35.666155+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:58.691279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.650397+0000","last_change":"2023-12-19T14:55:34.672154+0000","last_active":"2023-12-19T14:55:35.650397+0000","last_peered":"2023-12-19T14:55:35.650397+0000","last_clean":"2023-12-19T14:55:35.650397+0000","last_became_active":"2023-12-19T14:55:34.670150+0000","last_became_peered":"2023-12-19T14:55:34.670150+0000","last_unstale":"2023-12-19T14:55:35.650397+0000","last_undegraded":"2023-12-19T14:55:35.650397+0000","last_fullsized":"2023-12-19T14:55:35.650397+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:40.803441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"102'205","reported_seq":552,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.648185+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:55:34.648185+0000","last_peered":"2023-12-19T14:55:34.648185+0000","last_clean":"2023-12-19T14:55:34.648185+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:55:34.648185+0000","last_undegraded":"2023-12-19T14:55:34.648185+0000","last_fullsized":"2023-12-19T14:55:34.648185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1280486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.107,"15min":1.085},"min":{"1min":0.71299999999999997,"5min":0.59699999999999998,"15min":0.52300000000000002},"max":{"1min":1.466,"5min":3.544,"15min":3.544},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.999,"5min":1.139,"15min":1.131},"min":{"1min":0.629,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.4730000000000001,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.276}]},{"osd":1,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0820000000000001,"15min":1.095},"min":{"1min":0.74399999999999999,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.73},"last":1.22},{"interface":"front","average":{"1min":1.171,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.75,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6080000000000001,"5min":1.6759999999999999,"15min":1.984},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.258,"15min":1.23},"min":{"1min":0.77900000000000003,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.4590000000000001,"5min":1.5980000000000001,"15min":1.6919999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2529999999999999,"15min":1.2689999999999999},"min":{"1min":0.61099999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5620000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.26,"15min":1.282},"min":{"1min":0.96599999999999997,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.649,"5min":1.649,"15min":1.7969999999999999},"last":1.161},{"interface":"front","average":{"1min":1.292,"5min":1.302,"15min":1.3069999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.3640000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.173,"15min":1.155},"min":{"1min":0.66900000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.677,"5min":1.752,"15min":2.0259999999999998},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.234,"15min":1.208},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.67500000000000004},"max":{"1min":1.5249999999999999,"5min":1.6180000000000001,"15min":1.7290000000000001},"last":0.95799999999999996}]},{"osd":5,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2150000000000001,"15min":1.212},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.6459999999999999,"5min":1.6930000000000001,"15min":1.8109999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3080000000000001,"15min":1.3049999999999999},"min":{"1min":0.81200000000000006,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.806,"5min":1.8919999999999999,"15min":2.2549999999999999},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.383},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.65600000000000003},"max":{"1min":1.744,"5min":1.9419999999999999,"15min":2.1930000000000001},"last":1.196},{"interface":"front","average":{"1min":1.379,"5min":1.3660000000000001,"15min":1.351},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.70299999999999996},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.948},"last":1.242}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":692174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1140000000000001,"15min":1.131},"min":{"1min":0.68500000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5960000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1699999999999999},"min":{"1min":0.83199999999999996,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.528,"5min":1.845,"15min":1.845},"last":1.512}]},{"osd":1,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.173,"15min":1.1890000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.444,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.141},{"interface":"front","average":{"1min":1.212,"5min":1.2330000000000001,"15min":1.246},"min":{"1min":0.82599999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.573,"5min":3.226,"15min":3.226},"last":1.236}]},{"osd":2,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.276,"15min":1.2470000000000001},"min":{"1min":0.74399999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7649999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.159},{"interface":"front","average":{"1min":1.133,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.67100000000000004},"max":{"1min":1.4359999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.1859999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.296,"15min":1.298},"min":{"1min":0.91200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5880000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.341},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.3200000000000001,"15min":1.331},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6799999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.202}]},{"osd":4,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.2110000000000001,"15min":1.2050000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.55600000000000005},"max":{"1min":1.569,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.266,"15min":1.2969999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.57599999999999996},"max":{"1min":1.6619999999999999,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":1.603}]},{"osd":5,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2809999999999999,"15min":1.2949999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.671,"5min":2.02,"15min":2.0230000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3720000000000001,"15min":1.357},"min":{"1min":0.88700000000000001,"5min":0.82799999999999996,"15min":0.51000000000000001},"max":{"1min":1.5429999999999999,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":0.89100000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.431,"15min":1.413},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.53000000000000003},"max":{"1min":1.716,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.4490000000000001,"15min":1.454},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":1.8440000000000001,"5min":3.742,"15min":3.742},"last":1.0980000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298636,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073204,"statfs":{"total":96636764160,"available":96330960896,"internally_reserved":0,"allocated":1380352,"data_stored":690062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.044,"15min":1.0640000000000001},"min":{"1min":0.63700000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.52,"5min":1.601,"15min":1.619},"last":1.369},{"interface":"front","average":{"1min":1.03,"5min":1.119,"15min":1.1299999999999999},"min":{"1min":0.69199999999999995,"5min":0.63900000000000001,"15min":0.58699999999999997},"max":{"1min":1.4430000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.456}]},{"osd":1,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.6100000000000001,"5min":1.8029999999999999,"15min":1.9650000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.2150000000000001,"15min":1.2350000000000001},"min":{"1min":0.77300000000000002,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.456,"5min":1.677,"15min":1.7869999999999999},"last":1.4950000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1679999999999999,"15min":1.23},"min":{"1min":0.72699999999999998,"5min":0.67000000000000004,"15min":0.47399999999999998},"max":{"1min":1.756,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.137,"15min":1.1870000000000001},"min":{"1min":0.82099999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.45,"5min":1.589,"15min":1.954},"last":0.92800000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.2689999999999999,"15min":1.264},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6200000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.242,"15min":1.2629999999999999},"min":{"1min":0.84999999999999998,"5min":0.82299999999999995,"15min":0.68999999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.105}]},{"osd":4,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.5489999999999999,"5min":1.603,"15min":1.867},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.224,"15min":1.2370000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7529999999999999,"5min":2.27,"15min":2.27},"last":1.6559999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3049999999999999,"15min":1.3260000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.7869999999999999,"5min":1.982,"15min":2.0630000000000002},"last":1.252},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.319,"15min":1.321},"min":{"1min":0.80000000000000004,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.6499999999999999,"5min":1.8540000000000001,"15min":1.9219999999999999},"last":1.615}]},{"osd":7,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.347,"15min":1.353},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.5900000000000001,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.554},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.353,"15min":1.377},"min":{"1min":0.99399999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.95,"5min":1.95,"15min":2.2080000000000002},"last":1.4139999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302740,"kb_used_data":1356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069100,"statfs":{"total":96636764160,"available":96326758400,"internally_reserved":0,"allocated":1388544,"data_stored":690222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.0660000000000001},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.625,"5min":1.625,"15min":1.7869999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.083,"15min":1.083},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.7490000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0509999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2290000000000001,"15min":1.24},"min":{"1min":0.80600000000000005,"5min":0.623,"15min":0.623},"max":{"1min":1.726,"5min":1.8100000000000001,"15min":1.966},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.234,"15min":1.2689999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5229999999999999,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.23}]},{"osd":2,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.56999999999999995},"max":{"1min":1.5700000000000001,"5min":1.625,"15min":1.7330000000000001},"last":1.008},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.157,"15min":1.1990000000000001},"min":{"1min":0.73499999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.893},"last":1.0820000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1639999999999999,"15min":1.1799999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.6100000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.2390000000000001,"15min":1.262},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.60599999999999998},"max":{"1min":1.635,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.133}]},{"osd":5,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1619999999999999,"15min":1.151},"min":{"1min":0.82699999999999996,"5min":0.72399999999999998,"15min":0.57899999999999996},"max":{"1min":1.6559999999999999,"5min":2.1800000000000002,"15min":2.3300000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.6830000000000001,"5min":2.077,"15min":2.161},"last":1.196}]},{"osd":6,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.2969999999999999,"15min":1.304},"min":{"1min":0.63900000000000001,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.8069999999999999,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4059999999999999},"min":{"1min":0.89900000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.1549999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.339,"15min":1.3280000000000001},"min":{"1min":0.88200000000000001,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.5760000000000001,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.448,"15min":1.4399999999999999},"min":{"1min":0.94999999999999996,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":1.9370000000000001,"5min":3.008,"15min":3.008},"last":1.363}]}]},{"osd":3,"up_from":13,"seq":55834575046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306840,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065000,"statfs":{"total":96636764160,"available":96322560000,"internally_reserved":0,"allocated":1392640,"data_stored":696698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.86199999999999999},"min":{"1min":0.54900000000000004,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2889999999999999,"5min":1.3640000000000001,"15min":1.444},"last":1.276},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.97799999999999998,"15min":0.98999999999999999},"min":{"1min":0.59499999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4750000000000001,"5min":1.542,"15min":1.7270000000000001},"last":0.77400000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.90200000000000002,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.345,"5min":1.611,"15min":1.611},"last":1.171},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.1020000000000001,"15min":1.099},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5780000000000001,"5min":1.661,"15min":1.758},"last":1.381}]},{"osd":2,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.56100000000000005},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.23,"15min":1.2370000000000001},"min":{"1min":0.79100000000000004,"5min":0.67900000000000005,"15min":0.63500000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.8520000000000001},"last":1.5029999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3160000000000001,"15min":1.3260000000000001},"min":{"1min":1.071,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.6799999999999999,"5min":1.736,"15min":1.899},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.29,"15min":1.296},"min":{"1min":1.075,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.599,"5min":1.6879999999999999,"15min":1.7989999999999999},"last":1.2490000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":1.04,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.518,"5min":1.7709999999999999,"15min":1.893},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.371,"15min":1.389},"min":{"1min":1.056,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.827,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.4490000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8520000000000001,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":1.014,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7629999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.139}]},{"osd":7,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5189999999999999,"15min":1.5129999999999999},"min":{"1min":1.0249999999999999,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.8340000000000001,"5min":1.99,"15min":2.1640000000000001},"last":1.345},{"interface":"front","average":{"1min":1.502,"5min":1.464,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.704,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.415}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306824,"kb_used_data":1344,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065016,"statfs":{"total":96636764160,"available":96322576384,"internally_reserved":0,"allocated":1376256,"data_stored":689051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.61299999999999999,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1},{"interface":"front","average":{"1min":1.03,"5min":1.0429999999999999,"15min":1.0660000000000001},"min":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":1.2689999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1140000000000001,"15min":1.1299999999999999},"min":{"1min":0.64300000000000002,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.544,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.411},{"interface":"front","average":{"1min":1.002,"5min":1.0660000000000001,"15min":1.0920000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4119999999999999,"5min":1.853,"15min":1.853},"last":0.93000000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.0029999999999999,"15min":0.99199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.401,"5min":1.5669999999999999,"15min":1.623},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.073,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0549999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3700000000000001,"15min":1.3759999999999999},"min":{"1min":0.88,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.044},"last":1.607},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3859999999999999,"15min":1.389},"min":{"1min":0.85299999999999998,"5min":0.746,"15min":0.746},"max":{"1min":1.633,"5min":1.96,"15min":1.96},"last":1.6699999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.363,"15min":1.3440000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.395},"last":1.238},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3320000000000001,"15min":1.337},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.718,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.46}]},{"osd":6,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3240000000000001,"15min":1.3600000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80600000000000005},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.8759999999999999},"last":1.389},{"interface":"front","average":{"1min":1.431,"5min":1.4279999999999999,"15min":1.4359999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.5390000000000001},"last":1.5309999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.486,"15min":1.48},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8260000000000001,"5min":2.0699999999999998,"15min":2.2679999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4690000000000001,"15min":1.4750000000000001},"min":{"1min":1.079,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.875,"5min":2.1190000000000002,"15min":2.1819999999999999},"last":1.504}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306832,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065008,"statfs":{"total":96636764160,"available":96322568192,"internally_reserved":0,"allocated":1384448,"data_stored":690132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.89800000000000002,"15min":0.86799999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96999999999999997,"15min":0.97999999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.50700000000000001},"max":{"1min":1.2669999999999999,"5min":1.7070000000000001,"15min":1.802},"last":0.84999999999999998}]},{"osd":2,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.028,"15min":1.018},"min":{"1min":0.65800000000000003,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0860000000000001,"15min":1.117},"min":{"1min":0.66000000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.702,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.74199999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.222,"15min":1.2330000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.59199999999999997},"max":{"1min":1.53,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.083,"15min":1.073},"min":{"1min":0.64500000000000002,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.5249999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.2909999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2969999999999999,"15min":1.298},"min":{"1min":0.89600000000000002,"5min":0.624,"15min":0.624},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.381,"15min":1.391},"min":{"1min":0.92800000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6719999999999999,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.0549999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.268,"15min":1.2909999999999999},"min":{"1min":1.0069999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3939999999999999,"15min":1.385},"min":{"1min":1.042,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":1.26}]},{"osd":6,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3360000000000001,"15min":1.3580000000000001},"min":{"1min":0.97899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.6990000000000001,"5min":1.78,"15min":1.9179999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4399999999999999,"15min":1.446},"min":{"1min":1.0800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.335}]},{"osd":7,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4370000000000001,"15min":1.466},"min":{"1min":1.046,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7,"5min":2.0470000000000002,"15min":2.0590000000000002},"last":1.429},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":1.171,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.7649999999999999,"5min":1.8129999999999999,"15min":2.0609999999999999},"last":1.804}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307428,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064412,"statfs":{"total":96636764160,"available":96321957888,"internally_reserved":0,"allocated":1994752,"data_stored":1281633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83499999999999996,"15min":0.84399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3109999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.325,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.107,"15min":1.095},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":1.639},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.228,"15min":1.214},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.54300000000000004},"max":{"1min":1.7669999999999999,"5min":1.982,"15min":1.982},"last":0.85899999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.024,"15min":1.0209999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.5029999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.133,"5min":1.167,"15min":1.1759999999999999},"min":{"1min":0.621,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.756}]},{"osd":4,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2949999999999999,"15min":1.2909999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.631,"5min":2.153,"15min":2.153},"last":1.117},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2889999999999999,"15min":1.288},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.7170000000000001,"5min":1.7689999999999999,"15min":1.9359999999999999},"last":1.081}]},{"osd":5,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.377,"15min":1.371},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":2.4630000000000001},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.25,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.58299999999999996},"max":{"1min":1.825,"5min":2.2189999999999999,"15min":2.2229999999999999},"last":1.252}]},{"osd":6,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3200000000000001,"15min":1.3300000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.589,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.4390000000000001,"15min":1.4410000000000001},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.887,"5min":2.113,"15min":2.3799999999999999},"last":1.9710000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4670000000000001,"15min":1.452},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.893,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.425},"min":{"1min":0.98699999999999999,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.7470000000000001,"5min":1.931,"15min":2.0800000000000001},"last":1.1890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:55:40.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:55:40.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.461+0000 7feaf0078700 1 Processor -- start 2023-12-19T14:55:40.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.461+0000 7feaf0078700 1 -- start start 2023-12-19T14:55:40.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaf0078700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feae80c15b0 0x7feae815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaf0078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80aeba0 0x7feae80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaf0078700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae80c7190 con 0x7feae815d320 2023-12-19T14:55:40.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaf0078700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80aefd0 con 0x7feae80c15b0 2023-12-19T14:55:40.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaf0078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80c1b50 con 0x7feae80aeba0 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feae80c15b0 0x7feae815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feae80c15b0 0x7feae815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42578/0 (socket says 172.21.15.50:42578) 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 -- 172.21.15.50:0/3917022568 learned_addr learned my addr 172.21.15.50:0/3917022568 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaee615700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80aeba0 0x7feae80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaede14700 1 --1- 172.21.15.50:0/3917022568 >> v1:172.21.15.125:6789/0 conn(0x7feae815d320 0x7feae806d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60710/0 (socket says 172.21.15.50:60710) 2023-12-19T14:55:40.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 -- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80aeba0 msgr2=0x7feae80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:40.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 --2- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80aeba0 0x7feae80680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 -- 172.21.15.50:0/3917022568 >> v1:172.21.15.125:6789/0 conn(0x7feae815d320 legacy=0x7feae806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 -- 172.21.15.50:0/3917022568 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80c1e30 con 0x7feae80c15b0 2023-12-19T14:55:40.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaed613700 1 --2- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae80c15b0 0x7feae815d6f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fead80099b0 tx=0x7fead8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=498bc05fc5d7cc24 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaece12700 1 -- 172.21.15.50:0/3917022568 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8009ea0 con 0x7feae80c15b0 2023-12-19T14:55:40.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaece12700 1 -- 172.21.15.50:0/3917022568 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead800b860 con 0x7feae80c15b0 2023-12-19T14:55:40.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.465+0000 7feaece12700 1 -- 172.21.15.50:0/3917022568 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead800bc00 con 0x7feae80c15b0 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae80c15b0 msgr2=0x7feae815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 --2- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae80c15b0 0x7feae815d6f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fead80099b0 tx=0x7fead8009250 comp rx=0 tx=0).stop 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- 172.21.15.50:0/3917022568 shutdown_connections 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 --2- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80aeba0 0x7feae80680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 --2- 172.21.15.50:0/3917022568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae80c15b0 0x7feae815d6f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- 172.21.15.50:0/3917022568 >> 172.21.15.50:0/3917022568 conn(0x7feae80c8850 msgr2=0x7feae80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:40.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- 172.21.15.50:0/3917022568 shutdown_connections 2023-12-19T14:55:40.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- 172.21.15.50:0/3917022568 wait complete. 2023-12-19T14:55:40.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 Processor -- start 2023-12-19T14:55:40.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.469+0000 7feaf0078700 1 -- start start 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80c15b0 0x7feae80c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae815d320 0x7feae8153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8155180 con 0x7feae80aeba0 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8155310 con 0x7feae815d320 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80c3e80 con 0x7feae80c15b0 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48836/0 (socket says 172.21.15.50:48836) 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 -- 172.21.15.50:0/1109036069 learned_addr learned my addr 172.21.15.50:0/1109036069 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaed613700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80c15b0 0x7feae80c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaee615700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae815d320 0x7feae8153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 -- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80c15b0 msgr2=0x7feae80c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80c15b0 0x7feae80c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 -- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae815d320 msgr2=0x7feae8153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae815d320 0x7feae8153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 -- 172.21.15.50:0/1109036069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead80094f0 con 0x7feae80aeba0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaede14700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7feae400c560 tx=0x7feae4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae4005a70 con 0x7feae80aeba0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae4005bd0 con 0x7feae80aeba0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae40070d0 con 0x7feae80aeba0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae80c4160 con 0x7feae80aeba0 2023-12-19T14:55:40.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.473+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae80c4720 con 0x7feae80aeba0 2023-12-19T14:55:40.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.477+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae8071120 con 0x7feae80aeba0 2023-12-19T14:55:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.477+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7feae4007480 con 0x7feae80aeba0 2023-12-19T14:55:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.477+0000 7feaddffb700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 0x7feacc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.477+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7feae407bef0 con 0x7feae80aeba0 2023-12-19T14:55:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.481+0000 7feaed613700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 0x7feacc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.481+0000 7feaed613700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 0x7feacc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fead80099b0 tx=0x7fead8017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.485+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feae404c1d0 con 0x7feae80aeba0 2023-12-19T14:55:40.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.653+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feae80642a0 con 0x7feacc060670 2023-12-19T14:55:40.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.669+0000 7feaddffb700 1 -- 172.21.15.50:0/1109036069 <== mgr.7057 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287086 (secure 0 0 0) 0x7feae80642a0 con 0x7feacc060670 2023-12-19T14:55:40.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 msgr2=0x7feacc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 0x7feacc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fead80099b0 tx=0x7fead8017000 comp rx=0 tx=0).stop 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 msgr2=0x7feae80c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7feae400c560 tx=0x7feae4007f70 comp rx=0 tx=0).stop 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 shutdown_connections 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feae815d320 0x7feae8153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7feacc060670 0x7feacc062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7feae80c15b0 0x7feae80c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 --2- 172.21.15.50:0/1109036069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feae80aeba0 0x7feae80c3850 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 >> 172.21.15.50:0/1109036069 conn(0x7feae80c8850 msgr2=0x7feae81548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:40.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 shutdown_connections 2023-12-19T14:55:40.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.681+0000 7feaf0078700 1 -- 172.21.15.50:0/1109036069 wait complete. 2023-12-19T14:55:40.685 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:55:40.707 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T14:55:38.994250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443564,"kb_used_data":12012,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752531156,"statfs":{"total":773094113280,"available":770591903744,"internally_reserved":0,"allocated":12300288,"data_stored":6710458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.377588"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682552+0000","last_change":"2023-12-19T14:55:35.677709+0000","last_active":"2023-12-19T14:55:35.682552+0000","last_peered":"2023-12-19T14:55:35.682552+0000","last_clean":"2023-12-19T14:55:35.682552+0000","last_became_active":"2023-12-19T14:55:35.677426+0000","last_became_peered":"2023-12-19T14:55:35.677426+0000","last_unstale":"2023-12-19T14:55:35.682552+0000","last_undegraded":"2023-12-19T14:55:35.682552+0000","last_fullsized":"2023-12-19T14:55:35.682552+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.667166+0000","last_change":"2023-12-19T14:55:35.667166+0000","last_active":"2023-12-19T14:55:35.667166+0000","last_peered":"2023-12-19T14:55:35.667166+0000","last_clean":"2023-12-19T14:55:35.667166+0000","last_became_active":"2023-12-19T14:55:35.666851+0000","last_became_peered":"2023-12-19T14:55:35.666851+0000","last_unstale":"2023-12-19T14:55:35.667166+0000","last_undegraded":"2023-12-19T14:55:35.667166+0000","last_fullsized":"2023-12-19T14:55:35.667166+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:34:31.573812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678369+0000","last_change":"2023-12-19T14:55:35.676784+0000","last_active":"2023-12-19T14:55:35.678369+0000","last_peered":"2023-12-19T14:55:35.678369+0000","last_clean":"2023-12-19T14:55:35.678369+0000","last_became_active":"2023-12-19T14:55:35.676448+0000","last_became_peered":"2023-12-19T14:55:35.676448+0000","last_unstale":"2023-12-19T14:55:35.678369+0000","last_undegraded":"2023-12-19T14:55:35.678369+0000","last_fullsized":"2023-12-19T14:55:35.678369+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:22:17.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685813+0000","last_change":"2023-12-19T14:55:35.667151+0000","last_active":"2023-12-19T14:55:35.685813+0000","last_peered":"2023-12-19T14:55:35.685813+0000","last_clean":"2023-12-19T14:55:35.685813+0000","last_became_active":"2023-12-19T14:55:35.666870+0000","last_became_peered":"2023-12-19T14:55:35.666870+0000","last_unstale":"2023-12-19T14:55:35.685813+0000","last_undegraded":"2023-12-19T14:55:35.685813+0000","last_fullsized":"2023-12-19T14:55:35.685813+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:49:22.129269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.665276+0000","last_change":"2023-12-19T14:55:35.665276+0000","last_active":"2023-12-19T14:55:35.665276+0000","last_peered":"2023-12-19T14:55:35.665276+0000","last_clean":"2023-12-19T14:55:35.665276+0000","last_became_active":"2023-12-19T14:55:35.664975+0000","last_became_peered":"2023-12-19T14:55:35.664975+0000","last_unstale":"2023-12-19T14:55:35.665276+0000","last_undegraded":"2023-12-19T14:55:35.665276+0000","last_fullsized":"2023-12-19T14:55:35.665276+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:22:31.515354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.684168+0000","last_change":"2023-12-19T14:55:35.684168+0000","last_active":"2023-12-19T14:55:35.684168+0000","last_peered":"2023-12-19T14:55:35.684168+0000","last_clean":"2023-12-19T14:55:35.684168+0000","last_became_active":"2023-12-19T14:55:35.683852+0000","last_became_peered":"2023-12-19T14:55:35.683852+0000","last_unstale":"2023-12-19T14:55:35.684168+0000","last_undegraded":"2023-12-19T14:55:35.684168+0000","last_fullsized":"2023-12-19T14:55:35.684168+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:38.736254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.666095+0000","last_change":"2023-12-19T14:55:35.666095+0000","last_active":"2023-12-19T14:55:35.666095+0000","last_peered":"2023-12-19T14:55:35.666095+0000","last_clean":"2023-12-19T14:55:35.666095+0000","last_became_active":"2023-12-19T14:55:35.665699+0000","last_became_peered":"2023-12-19T14:55:35.665699+0000","last_unstale":"2023-12-19T14:55:35.666095+0000","last_undegraded":"2023-12-19T14:55:35.666095+0000","last_fullsized":"2023-12-19T14:55:35.666095+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:45:11.056272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.689084+0000","last_change":"2023-12-19T14:55:35.668252+0000","last_active":"2023-12-19T14:55:35.689084+0000","last_peered":"2023-12-19T14:55:35.689084+0000","last_clean":"2023-12-19T14:55:35.689084+0000","last_became_active":"2023-12-19T14:55:35.667925+0000","last_became_peered":"2023-12-19T14:55:35.667925+0000","last_unstale":"2023-12-19T14:55:35.689084+0000","last_undegraded":"2023-12-19T14:55:35.689084+0000","last_fullsized":"2023-12-19T14:55:35.689084+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:39.382342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.663513+0000","last_change":"2023-12-19T14:55:35.663513+0000","last_active":"2023-12-19T14:55:35.663513+0000","last_peered":"2023-12-19T14:55:35.663513+0000","last_clean":"2023-12-19T14:55:35.663513+0000","last_became_active":"2023-12-19T14:55:35.663215+0000","last_became_peered":"2023-12-19T14:55:35.663215+0000","last_unstale":"2023-12-19T14:55:35.663513+0000","last_undegraded":"2023-12-19T14:55:35.663513+0000","last_fullsized":"2023-12-19T14:55:35.663513+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:24:30.746073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.686392+0000","last_change":"2023-12-19T14:55:35.686392+0000","last_active":"2023-12-19T14:55:35.686392+0000","last_peered":"2023-12-19T14:55:35.686392+0000","last_clean":"2023-12-19T14:55:35.686392+0000","last_became_active":"2023-12-19T14:55:35.685922+0000","last_became_peered":"2023-12-19T14:55:35.685922+0000","last_unstale":"2023-12-19T14:55:35.686392+0000","last_undegraded":"2023-12-19T14:55:35.686392+0000","last_fullsized":"2023-12-19T14:55:35.686392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:16.569653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678001+0000","last_change":"2023-12-19T14:55:35.678001+0000","last_active":"2023-12-19T14:55:35.678001+0000","last_peered":"2023-12-19T14:55:35.678001+0000","last_clean":"2023-12-19T14:55:35.678001+0000","last_became_active":"2023-12-19T14:55:35.677727+0000","last_became_peered":"2023-12-19T14:55:35.677727+0000","last_unstale":"2023-12-19T14:55:35.678001+0000","last_undegraded":"2023-12-19T14:55:35.678001+0000","last_fullsized":"2023-12-19T14:55:35.678001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:17.506894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.675479+0000","last_change":"2023-12-19T14:55:35.675479+0000","last_active":"2023-12-19T14:55:35.675479+0000","last_peered":"2023-12-19T14:55:35.675479+0000","last_clean":"2023-12-19T14:55:35.675479+0000","last_became_active":"2023-12-19T14:55:35.675169+0000","last_became_peered":"2023-12-19T14:55:35.675169+0000","last_unstale":"2023-12-19T14:55:35.675479+0000","last_undegraded":"2023-12-19T14:55:35.675479+0000","last_fullsized":"2023-12-19T14:55:35.675479+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:35:12.576101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677866+0000","last_change":"2023-12-19T14:55:35.659592+0000","last_active":"2023-12-19T14:55:35.677866+0000","last_peered":"2023-12-19T14:55:35.677866+0000","last_clean":"2023-12-19T14:55:35.677866+0000","last_became_active":"2023-12-19T14:55:35.659257+0000","last_became_peered":"2023-12-19T14:55:35.659257+0000","last_unstale":"2023-12-19T14:55:35.677866+0000","last_undegraded":"2023-12-19T14:55:35.677866+0000","last_fullsized":"2023-12-19T14:55:35.677866+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:43.880340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.684342+0000","last_change":"2023-12-19T14:55:35.684342+0000","last_active":"2023-12-19T14:55:35.684342+0000","last_peered":"2023-12-19T14:55:35.684342+0000","last_clean":"2023-12-19T14:55:35.684342+0000","last_became_active":"2023-12-19T14:55:35.683988+0000","last_became_peered":"2023-12-19T14:55:35.683988+0000","last_unstale":"2023-12-19T14:55:35.684342+0000","last_undegraded":"2023-12-19T14:55:35.684342+0000","last_fullsized":"2023-12-19T14:55:35.684342+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:18:53.585873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669072+0000","last_change":"2023-12-19T14:55:35.658197+0000","last_active":"2023-12-19T14:55:35.669072+0000","last_peered":"2023-12-19T14:55:35.669072+0000","last_clean":"2023-12-19T14:55:35.669072+0000","last_became_active":"2023-12-19T14:55:35.656578+0000","last_became_peered":"2023-12-19T14:55:35.656578+0000","last_unstale":"2023-12-19T14:55:35.669072+0000","last_undegraded":"2023-12-19T14:55:35.669072+0000","last_fullsized":"2023-12-19T14:55:35.669072+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678693+0000","last_change":"2023-12-19T14:55:35.678693+0000","last_active":"2023-12-19T14:55:35.678693+0000","last_peered":"2023-12-19T14:55:35.678693+0000","last_clean":"2023-12-19T14:55:35.678693+0000","last_became_active":"2023-12-19T14:55:35.678341+0000","last_became_peered":"2023-12-19T14:55:35.678341+0000","last_unstale":"2023-12-19T14:55:35.678693+0000","last_undegraded":"2023-12-19T14:55:35.678693+0000","last_fullsized":"2023-12-19T14:55:35.678693+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:05:58.727283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679694+0000","last_change":"2023-12-19T14:55:35.679694+0000","last_active":"2023-12-19T14:55:35.679694+0000","last_peered":"2023-12-19T14:55:35.679694+0000","last_clean":"2023-12-19T14:55:35.679694+0000","last_became_active":"2023-12-19T14:55:35.679385+0000","last_became_peered":"2023-12-19T14:55:35.679385+0000","last_unstale":"2023-12-19T14:55:35.679694+0000","last_undegraded":"2023-12-19T14:55:35.679694+0000","last_fullsized":"2023-12-19T14:55:35.679694+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:28:04.118006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669346+0000","last_change":"2023-12-19T14:55:35.660392+0000","last_active":"2023-12-19T14:55:35.669346+0000","last_peered":"2023-12-19T14:55:35.669346+0000","last_clean":"2023-12-19T14:55:35.669346+0000","last_became_active":"2023-12-19T14:55:35.660062+0000","last_became_peered":"2023-12-19T14:55:35.660062+0000","last_unstale":"2023-12-19T14:55:35.669346+0000","last_undegraded":"2023-12-19T14:55:35.669346+0000","last_fullsized":"2023-12-19T14:55:35.669346+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680296+0000","last_change":"2023-12-19T14:55:35.680296+0000","last_active":"2023-12-19T14:55:35.680296+0000","last_peered":"2023-12-19T14:55:35.680296+0000","last_clean":"2023-12-19T14:55:35.680296+0000","last_became_active":"2023-12-19T14:55:35.680005+0000","last_became_peered":"2023-12-19T14:55:35.680005+0000","last_unstale":"2023-12-19T14:55:35.680296+0000","last_undegraded":"2023-12-19T14:55:35.680296+0000","last_fullsized":"2023-12-19T14:55:35.680296+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:53:40.305557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685835+0000","last_change":"2023-12-19T14:55:35.668486+0000","last_active":"2023-12-19T14:55:35.685835+0000","last_peered":"2023-12-19T14:55:35.685835+0000","last_clean":"2023-12-19T14:55:35.685835+0000","last_became_active":"2023-12-19T14:55:35.667976+0000","last_became_peered":"2023-12-19T14:55:35.667976+0000","last_unstale":"2023-12-19T14:55:35.685835+0000","last_undegraded":"2023-12-19T14:55:35.685835+0000","last_fullsized":"2023-12-19T14:55:35.685835+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:16.774265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669752+0000","last_change":"2023-12-19T14:55:35.669752+0000","last_active":"2023-12-19T14:55:35.669752+0000","last_peered":"2023-12-19T14:55:35.669752+0000","last_clean":"2023-12-19T14:55:35.669752+0000","last_became_active":"2023-12-19T14:55:35.669386+0000","last_became_peered":"2023-12-19T14:55:35.669386+0000","last_unstale":"2023-12-19T14:55:35.669752+0000","last_undegraded":"2023-12-19T14:55:35.669752+0000","last_fullsized":"2023-12-19T14:55:35.669752+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:07.306981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680231+0000","last_change":"2023-12-19T14:55:35.680231+0000","last_active":"2023-12-19T14:55:35.680231+0000","last_peered":"2023-12-19T14:55:35.680231+0000","last_clean":"2023-12-19T14:55:35.680231+0000","last_became_active":"2023-12-19T14:55:35.679952+0000","last_became_peered":"2023-12-19T14:55:35.679952+0000","last_unstale":"2023-12-19T14:55:35.680231+0000","last_undegraded":"2023-12-19T14:55:35.680231+0000","last_fullsized":"2023-12-19T14:55:35.680231+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:47.582564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683923+0000","last_change":"2023-12-19T14:55:35.683923+0000","last_active":"2023-12-19T14:55:35.683923+0000","last_peered":"2023-12-19T14:55:35.683923+0000","last_clean":"2023-12-19T14:55:35.683923+0000","last_became_active":"2023-12-19T14:55:35.683400+0000","last_became_peered":"2023-12-19T14:55:35.683400+0000","last_unstale":"2023-12-19T14:55:35.683923+0000","last_undegraded":"2023-12-19T14:55:35.683923+0000","last_fullsized":"2023-12-19T14:55:35.683923+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:57:17.889446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677746+0000","last_change":"2023-12-19T14:55:35.675846+0000","last_active":"2023-12-19T14:55:35.677746+0000","last_peered":"2023-12-19T14:55:35.677746+0000","last_clean":"2023-12-19T14:55:35.677746+0000","last_became_active":"2023-12-19T14:55:35.675533+0000","last_became_peered":"2023-12-19T14:55:35.675533+0000","last_unstale":"2023-12-19T14:55:35.677746+0000","last_undegraded":"2023-12-19T14:55:35.677746+0000","last_fullsized":"2023-12-19T14:55:35.677746+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:57:03.667084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664002+0000","last_change":"2023-12-19T14:55:35.664002+0000","last_active":"2023-12-19T14:55:35.664002+0000","last_peered":"2023-12-19T14:55:35.664002+0000","last_clean":"2023-12-19T14:55:35.664002+0000","last_became_active":"2023-12-19T14:55:35.663721+0000","last_became_peered":"2023-12-19T14:55:35.663721+0000","last_unstale":"2023-12-19T14:55:35.664002+0000","last_undegraded":"2023-12-19T14:55:35.664002+0000","last_fullsized":"2023-12-19T14:55:35.664002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:13.153722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680430+0000","last_change":"2023-12-19T14:55:35.680430+0000","last_active":"2023-12-19T14:55:35.680430+0000","last_peered":"2023-12-19T14:55:35.680430+0000","last_clean":"2023-12-19T14:55:35.680430+0000","last_became_active":"2023-12-19T14:55:35.680171+0000","last_became_peered":"2023-12-19T14:55:35.680171+0000","last_unstale":"2023-12-19T14:55:35.680430+0000","last_undegraded":"2023-12-19T14:55:35.680430+0000","last_fullsized":"2023-12-19T14:55:35.680430+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:56:51.429164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678607+0000","last_change":"2023-12-19T14:55:35.678607+0000","last_active":"2023-12-19T14:55:35.678607+0000","last_peered":"2023-12-19T14:55:35.678607+0000","last_clean":"2023-12-19T14:55:35.678607+0000","last_became_active":"2023-12-19T14:55:35.678239+0000","last_became_peered":"2023-12-19T14:55:35.678239+0000","last_unstale":"2023-12-19T14:55:35.678607+0000","last_undegraded":"2023-12-19T14:55:35.678607+0000","last_fullsized":"2023-12-19T14:55:35.678607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:32.624382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685853+0000","last_change":"2023-12-19T14:55:35.669577+0000","last_active":"2023-12-19T14:55:35.685853+0000","last_peered":"2023-12-19T14:55:35.685853+0000","last_clean":"2023-12-19T14:55:35.685853+0000","last_became_active":"2023-12-19T14:55:35.668390+0000","last_became_peered":"2023-12-19T14:55:35.668390+0000","last_unstale":"2023-12-19T14:55:35.685853+0000","last_undegraded":"2023-12-19T14:55:35.685853+0000","last_fullsized":"2023-12-19T14:55:35.685853+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.670576+0000","last_change":"2023-12-19T14:55:35.670576+0000","last_active":"2023-12-19T14:55:35.670576+0000","last_peered":"2023-12-19T14:55:35.670576+0000","last_clean":"2023-12-19T14:55:35.670576+0000","last_became_active":"2023-12-19T14:55:35.670105+0000","last_became_peered":"2023-12-19T14:55:35.670105+0000","last_unstale":"2023-12-19T14:55:35.670576+0000","last_undegraded":"2023-12-19T14:55:35.670576+0000","last_fullsized":"2023-12-19T14:55:35.670576+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:28.964698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.663956+0000","last_change":"2023-12-19T14:55:35.663956+0000","last_active":"2023-12-19T14:55:35.663956+0000","last_peered":"2023-12-19T14:55:35.663956+0000","last_clean":"2023-12-19T14:55:35.663956+0000","last_became_active":"2023-12-19T14:55:35.663666+0000","last_became_peered":"2023-12-19T14:55:35.663666+0000","last_unstale":"2023-12-19T14:55:35.663956+0000","last_undegraded":"2023-12-19T14:55:35.663956+0000","last_fullsized":"2023-12-19T14:55:35.663956+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:21:45.290974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.681429+0000","last_change":"2023-12-19T14:55:35.681429+0000","last_active":"2023-12-19T14:55:35.681429+0000","last_peered":"2023-12-19T14:55:35.681429+0000","last_clean":"2023-12-19T14:55:35.681429+0000","last_became_active":"2023-12-19T14:55:35.681072+0000","last_became_peered":"2023-12-19T14:55:35.681072+0000","last_unstale":"2023-12-19T14:55:35.681429+0000","last_undegraded":"2023-12-19T14:55:35.681429+0000","last_fullsized":"2023-12-19T14:55:35.681429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.917699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678662+0000","last_change":"2023-12-19T14:55:35.678662+0000","last_active":"2023-12-19T14:55:35.678662+0000","last_peered":"2023-12-19T14:55:35.678662+0000","last_clean":"2023-12-19T14:55:35.678662+0000","last_became_active":"2023-12-19T14:55:35.677985+0000","last_became_peered":"2023-12-19T14:55:35.677985+0000","last_unstale":"2023-12-19T14:55:35.678662+0000","last_undegraded":"2023-12-19T14:55:35.678662+0000","last_fullsized":"2023-12-19T14:55:35.678662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:53:56.104030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679260+0000","last_change":"2023-12-19T14:55:35.679260+0000","last_active":"2023-12-19T14:55:35.679260+0000","last_peered":"2023-12-19T14:55:35.679260+0000","last_clean":"2023-12-19T14:55:35.679260+0000","last_became_active":"2023-12-19T14:55:35.676157+0000","last_became_peered":"2023-12-19T14:55:35.676157+0000","last_unstale":"2023-12-19T14:55:35.679260+0000","last_undegraded":"2023-12-19T14:55:35.679260+0000","last_fullsized":"2023-12-19T14:55:35.679260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:33.080441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"112'4","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.691675+0000","last_change":"2023-12-19T14:55:34.663860+0000","last_active":"2023-12-19T14:55:35.691675+0000","last_peered":"2023-12-19T14:55:35.691675+0000","last_clean":"2023-12-19T14:55:35.691675+0000","last_became_active":"2023-12-19T14:55:34.663331+0000","last_became_peered":"2023-12-19T14:55:34.663331+0000","last_unstale":"2023-12-19T14:55:35.691675+0000","last_undegraded":"2023-12-19T14:55:35.691675+0000","last_fullsized":"2023-12-19T14:55:35.691675+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:16.479157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685141+0000","last_change":"2023-12-19T14:55:35.685141+0000","last_active":"2023-12-19T14:55:35.685141+0000","last_peered":"2023-12-19T14:55:35.685141+0000","last_clean":"2023-12-19T14:55:35.685141+0000","last_became_active":"2023-12-19T14:55:35.684738+0000","last_became_peered":"2023-12-19T14:55:35.684738+0000","last_unstale":"2023-12-19T14:55:35.685141+0000","last_undegraded":"2023-12-19T14:55:35.685141+0000","last_fullsized":"2023-12-19T14:55:35.685141+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.917699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.655484+0000","last_change":"2023-12-19T14:55:34.661920+0000","last_active":"2023-12-19T14:55:35.655484+0000","last_peered":"2023-12-19T14:55:35.655484+0000","last_clean":"2023-12-19T14:55:35.655484+0000","last_became_active":"2023-12-19T14:55:34.661307+0000","last_became_peered":"2023-12-19T14:55:34.661307+0000","last_unstale":"2023-12-19T14:55:35.655484+0000","last_undegraded":"2023-12-19T14:55:35.655484+0000","last_fullsized":"2023-12-19T14:55:35.655484+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:10.719896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680454+0000","last_change":"2023-12-19T14:55:35.680454+0000","last_active":"2023-12-19T14:55:35.680454+0000","last_peered":"2023-12-19T14:55:35.680454+0000","last_clean":"2023-12-19T14:55:35.680454+0000","last_became_active":"2023-12-19T14:55:35.680190+0000","last_became_peered":"2023-12-19T14:55:35.680190+0000","last_unstale":"2023-12-19T14:55:35.680454+0000","last_undegraded":"2023-12-19T14:55:35.680454+0000","last_fullsized":"2023-12-19T14:55:35.680454+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:07.306981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682933+0000","last_change":"2023-12-19T14:55:34.661290+0000","last_active":"2023-12-19T14:55:35.682933+0000","last_peered":"2023-12-19T14:55:35.682933+0000","last_clean":"2023-12-19T14:55:35.682933+0000","last_became_active":"2023-12-19T14:55:34.660961+0000","last_became_peered":"2023-12-19T14:55:34.660961+0000","last_unstale":"2023-12-19T14:55:35.682933+0000","last_undegraded":"2023-12-19T14:55:35.682933+0000","last_fullsized":"2023-12-19T14:55:35.682933+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:41.162344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679028+0000","last_change":"2023-12-19T14:55:35.679028+0000","last_active":"2023-12-19T14:55:35.679028+0000","last_peered":"2023-12-19T14:55:35.679028+0000","last_clean":"2023-12-19T14:55:35.679028+0000","last_became_active":"2023-12-19T14:55:35.673824+0000","last_became_peered":"2023-12-19T14:55:35.673824+0000","last_unstale":"2023-12-19T14:55:35.679028+0000","last_undegraded":"2023-12-19T14:55:35.679028+0000","last_fullsized":"2023-12-19T14:55:35.679028+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:37:15.646881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685468+0000","last_change":"2023-12-19T14:55:34.666585+0000","last_active":"2023-12-19T14:55:35.685468+0000","last_peered":"2023-12-19T14:55:35.685468+0000","last_clean":"2023-12-19T14:55:35.685468+0000","last_became_active":"2023-12-19T14:55:34.665522+0000","last_became_peered":"2023-12-19T14:55:34.665522+0000","last_unstale":"2023-12-19T14:55:35.685468+0000","last_undegraded":"2023-12-19T14:55:35.685468+0000","last_fullsized":"2023-12-19T14:55:35.685468+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:25.682306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.659753+0000","last_change":"2023-12-19T14:55:35.655324+0000","last_active":"2023-12-19T14:55:35.659753+0000","last_peered":"2023-12-19T14:55:35.659753+0000","last_clean":"2023-12-19T14:55:35.659753+0000","last_became_active":"2023-12-19T14:55:35.654964+0000","last_became_peered":"2023-12-19T14:55:35.654964+0000","last_unstale":"2023-12-19T14:55:35.659753+0000","last_undegraded":"2023-12-19T14:55:35.659753+0000","last_fullsized":"2023-12-19T14:55:35.659753+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:22:17.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658517+0000","last_change":"2023-12-19T14:55:34.667483+0000","last_active":"2023-12-19T14:55:35.658517+0000","last_peered":"2023-12-19T14:55:35.658517+0000","last_clean":"2023-12-19T14:55:35.658517+0000","last_became_active":"2023-12-19T14:55:34.666862+0000","last_became_peered":"2023-12-19T14:55:34.666862+0000","last_unstale":"2023-12-19T14:55:35.658517+0000","last_undegraded":"2023-12-19T14:55:35.658517+0000","last_fullsized":"2023-12-19T14:55:35.658517+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:49:56.446441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679688+0000","last_change":"2023-12-19T14:55:35.679688+0000","last_active":"2023-12-19T14:55:35.679688+0000","last_peered":"2023-12-19T14:55:35.679688+0000","last_clean":"2023-12-19T14:55:35.679688+0000","last_became_active":"2023-12-19T14:55:35.677391+0000","last_became_peered":"2023-12-19T14:55:35.677391+0000","last_unstale":"2023-12-19T14:55:35.679688+0000","last_undegraded":"2023-12-19T14:55:35.679688+0000","last_fullsized":"2023-12-19T14:55:35.679688+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:49:57.460332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.694269+0000","last_change":"2023-12-19T14:55:34.674680+0000","last_active":"2023-12-19T14:55:35.694269+0000","last_peered":"2023-12-19T14:55:35.694269+0000","last_clean":"2023-12-19T14:55:35.694269+0000","last_became_active":"2023-12-19T14:55:34.674379+0000","last_became_peered":"2023-12-19T14:55:34.674379+0000","last_unstale":"2023-12-19T14:55:35.694269+0000","last_undegraded":"2023-12-19T14:55:35.694269+0000","last_fullsized":"2023-12-19T14:55:35.694269+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:56:19.598180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.696714+0000","last_change":"2023-12-19T14:55:35.696714+0000","last_active":"2023-12-19T14:55:35.696714+0000","last_peered":"2023-12-19T14:55:35.696714+0000","last_clean":"2023-12-19T14:55:35.696714+0000","last_became_active":"2023-12-19T14:55:35.696395+0000","last_became_peered":"2023-12-19T14:55:35.696395+0000","last_unstale":"2023-12-19T14:55:35.696714+0000","last_undegraded":"2023-12-19T14:55:35.696714+0000","last_fullsized":"2023-12-19T14:55:35.696714+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:46.844201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.651512+0000","last_change":"2023-12-19T14:55:34.658231+0000","last_active":"2023-12-19T14:55:35.651512+0000","last_peered":"2023-12-19T14:55:35.651512+0000","last_clean":"2023-12-19T14:55:35.651512+0000","last_became_active":"2023-12-19T14:55:34.657931+0000","last_became_peered":"2023-12-19T14:55:34.657931+0000","last_unstale":"2023-12-19T14:55:35.651512+0000","last_undegraded":"2023-12-19T14:55:35.651512+0000","last_fullsized":"2023-12-19T14:55:35.651512+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:06.195525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.685272+0000","last_change":"2023-12-19T14:55:35.667105+0000","last_active":"2023-12-19T14:55:35.685272+0000","last_peered":"2023-12-19T14:55:35.685272+0000","last_clean":"2023-12-19T14:55:35.685272+0000","last_became_active":"2023-12-19T14:55:35.666746+0000","last_became_peered":"2023-12-19T14:55:35.666746+0000","last_unstale":"2023-12-19T14:55:35.685272+0000","last_undegraded":"2023-12-19T14:55:35.685272+0000","last_fullsized":"2023-12-19T14:55:35.685272+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:57:03.667084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.674062+0000","last_change":"2023-12-19T14:55:34.673102+0000","last_active":"2023-12-19T14:55:34.674062+0000","last_peered":"2023-12-19T14:55:34.674062+0000","last_clean":"2023-12-19T14:55:34.674062+0000","last_became_active":"2023-12-19T14:55:34.672821+0000","last_became_peered":"2023-12-19T14:55:34.672821+0000","last_unstale":"2023-12-19T14:55:34.674062+0000","last_undegraded":"2023-12-19T14:55:34.674062+0000","last_fullsized":"2023-12-19T14:55:34.674062+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:39.166215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.668175+0000","last_change":"2023-12-19T14:55:35.668175+0000","last_active":"2023-12-19T14:55:35.668175+0000","last_peered":"2023-12-19T14:55:35.668175+0000","last_clean":"2023-12-19T14:55:35.668175+0000","last_became_active":"2023-12-19T14:55:35.667356+0000","last_became_peered":"2023-12-19T14:55:35.667356+0000","last_unstale":"2023-12-19T14:55:35.668175+0000","last_undegraded":"2023-12-19T14:55:35.668175+0000","last_fullsized":"2023-12-19T14:55:35.668175+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.670819+0000","last_change":"2023-12-19T14:55:34.668513+0000","last_active":"2023-12-19T14:55:34.670819+0000","last_peered":"2023-12-19T14:55:34.670819+0000","last_clean":"2023-12-19T14:55:34.670819+0000","last_became_active":"2023-12-19T14:55:34.667808+0000","last_became_peered":"2023-12-19T14:55:34.667808+0000","last_unstale":"2023-12-19T14:55:34.670819+0000","last_undegraded":"2023-12-19T14:55:34.670819+0000","last_fullsized":"2023-12-19T14:55:34.670819+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:21:44.277083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.670877+0000","last_change":"2023-12-19T14:55:35.670877+0000","last_active":"2023-12-19T14:55:35.670877+0000","last_peered":"2023-12-19T14:55:35.670877+0000","last_clean":"2023-12-19T14:55:35.670877+0000","last_became_active":"2023-12-19T14:55:35.670519+0000","last_became_peered":"2023-12-19T14:55:35.670519+0000","last_unstale":"2023-12-19T14:55:35.670877+0000","last_undegraded":"2023-12-19T14:55:35.670877+0000","last_fullsized":"2023-12-19T14:55:35.670877+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.653831+0000","last_change":"2023-12-19T14:55:34.669270+0000","last_active":"2023-12-19T14:55:35.653831+0000","last_peered":"2023-12-19T14:55:35.653831+0000","last_clean":"2023-12-19T14:55:35.653831+0000","last_became_active":"2023-12-19T14:55:34.668954+0000","last_became_peered":"2023-12-19T14:55:34.668954+0000","last_unstale":"2023-12-19T14:55:35.653831+0000","last_undegraded":"2023-12-19T14:55:35.653831+0000","last_fullsized":"2023-12-19T14:55:35.653831+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:59.931922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683396+0000","last_change":"2023-12-19T14:55:35.678518+0000","last_active":"2023-12-19T14:55:35.683396+0000","last_peered":"2023-12-19T14:55:35.683396+0000","last_clean":"2023-12-19T14:55:35.683396+0000","last_became_active":"2023-12-19T14:55:35.678234+0000","last_became_peered":"2023-12-19T14:55:35.678234+0000","last_unstale":"2023-12-19T14:55:35.683396+0000","last_undegraded":"2023-12-19T14:55:35.683396+0000","last_fullsized":"2023-12-19T14:55:35.683396+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:25.625227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"112'3","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687023+0000","last_change":"2023-12-19T14:55:34.671274+0000","last_active":"2023-12-19T14:55:35.687023+0000","last_peered":"2023-12-19T14:55:35.687023+0000","last_clean":"2023-12-19T14:55:35.687023+0000","last_became_active":"2023-12-19T14:55:34.671005+0000","last_became_peered":"2023-12-19T14:55:34.671005+0000","last_unstale":"2023-12-19T14:55:35.687023+0000","last_undegraded":"2023-12-19T14:55:35.687023+0000","last_fullsized":"2023-12-19T14:55:35.687023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:37:29.341509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677671+0000","last_change":"2023-12-19T14:55:35.661438+0000","last_active":"2023-12-19T14:55:35.677671+0000","last_peered":"2023-12-19T14:55:35.677671+0000","last_clean":"2023-12-19T14:55:35.677671+0000","last_became_active":"2023-12-19T14:55:35.661134+0000","last_became_peered":"2023-12-19T14:55:35.661134+0000","last_unstale":"2023-12-19T14:55:35.677671+0000","last_undegraded":"2023-12-19T14:55:35.677671+0000","last_fullsized":"2023-12-19T14:55:35.677671+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:17.700933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.697340+0000","last_change":"2023-12-19T14:55:34.663318+0000","last_active":"2023-12-19T14:55:35.697340+0000","last_peered":"2023-12-19T14:55:35.697340+0000","last_clean":"2023-12-19T14:55:35.697340+0000","last_became_active":"2023-12-19T14:55:34.663042+0000","last_became_peered":"2023-12-19T14:55:34.663042+0000","last_unstale":"2023-12-19T14:55:35.697340+0000","last_undegraded":"2023-12-19T14:55:35.697340+0000","last_fullsized":"2023-12-19T14:55:35.697340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:35:27.950808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677568+0000","last_change":"2023-12-19T14:55:35.662672+0000","last_active":"2023-12-19T14:55:35.677568+0000","last_peered":"2023-12-19T14:55:35.677568+0000","last_clean":"2023-12-19T14:55:35.677568+0000","last_became_active":"2023-12-19T14:55:35.662381+0000","last_became_peered":"2023-12-19T14:55:35.662381+0000","last_unstale":"2023-12-19T14:55:35.677568+0000","last_undegraded":"2023-12-19T14:55:35.677568+0000","last_fullsized":"2023-12-19T14:55:35.677568+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:40.180106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.690023+0000","last_change":"2023-12-19T14:55:34.658626+0000","last_active":"2023-12-19T14:55:35.690023+0000","last_peered":"2023-12-19T14:55:35.690023+0000","last_clean":"2023-12-19T14:55:35.690023+0000","last_became_active":"2023-12-19T14:55:34.658333+0000","last_became_peered":"2023-12-19T14:55:34.658333+0000","last_unstale":"2023-12-19T14:55:35.690023+0000","last_undegraded":"2023-12-19T14:55:35.690023+0000","last_fullsized":"2023-12-19T14:55:35.690023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:38.368451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664583+0000","last_change":"2023-12-19T14:55:35.664583+0000","last_active":"2023-12-19T14:55:35.664583+0000","last_peered":"2023-12-19T14:55:35.664583+0000","last_clean":"2023-12-19T14:55:35.664583+0000","last_became_active":"2023-12-19T14:55:35.664303+0000","last_became_peered":"2023-12-19T14:55:35.664303+0000","last_unstale":"2023-12-19T14:55:35.664583+0000","last_undegraded":"2023-12-19T14:55:35.664583+0000","last_fullsized":"2023-12-19T14:55:35.664583+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:09:41.563795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.676775+0000","last_change":"2023-12-19T14:55:34.666726+0000","last_active":"2023-12-19T14:55:34.676775+0000","last_peered":"2023-12-19T14:55:34.676775+0000","last_clean":"2023-12-19T14:55:34.676775+0000","last_became_active":"2023-12-19T14:55:34.666447+0000","last_became_peered":"2023-12-19T14:55:34.666447+0000","last_unstale":"2023-12-19T14:55:34.676775+0000","last_undegraded":"2023-12-19T14:55:34.676775+0000","last_fullsized":"2023-12-19T14:55:34.676775+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:24:16.687042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687942+0000","last_change":"2023-12-19T14:55:35.687942+0000","last_active":"2023-12-19T14:55:35.687942+0000","last_peered":"2023-12-19T14:55:35.687942+0000","last_clean":"2023-12-19T14:55:35.687942+0000","last_became_active":"2023-12-19T14:55:35.687611+0000","last_became_peered":"2023-12-19T14:55:35.687611+0000","last_unstale":"2023-12-19T14:55:35.687942+0000","last_undegraded":"2023-12-19T14:55:35.687942+0000","last_fullsized":"2023-12-19T14:55:35.687942+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:28:58.905253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.647534+0000","last_change":"2023-12-19T14:55:34.664745+0000","last_active":"2023-12-19T14:55:35.647534+0000","last_peered":"2023-12-19T14:55:35.647534+0000","last_clean":"2023-12-19T14:55:35.647534+0000","last_became_active":"2023-12-19T14:55:34.664179+0000","last_became_peered":"2023-12-19T14:55:34.664179+0000","last_unstale":"2023-12-19T14:55:35.647534+0000","last_undegraded":"2023-12-19T14:55:35.647534+0000","last_fullsized":"2023-12-19T14:55:35.647534+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:07:37.333779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679041+0000","last_change":"2023-12-19T14:55:35.679041+0000","last_active":"2023-12-19T14:55:35.679041+0000","last_peered":"2023-12-19T14:55:35.679041+0000","last_clean":"2023-12-19T14:55:35.679041+0000","last_became_active":"2023-12-19T14:55:35.678377+0000","last_became_peered":"2023-12-19T14:55:35.678377+0000","last_unstale":"2023-12-19T14:55:35.679041+0000","last_undegraded":"2023-12-19T14:55:35.679041+0000","last_fullsized":"2023-12-19T14:55:35.679041+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:10.189861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.698415+0000","last_change":"2023-12-19T14:55:34.672667+0000","last_active":"2023-12-19T14:55:35.698415+0000","last_peered":"2023-12-19T14:55:35.698415+0000","last_clean":"2023-12-19T14:55:35.698415+0000","last_became_active":"2023-12-19T14:55:34.672366+0000","last_became_peered":"2023-12-19T14:55:34.672366+0000","last_unstale":"2023-12-19T14:55:35.698415+0000","last_undegraded":"2023-12-19T14:55:35.698415+0000","last_fullsized":"2023-12-19T14:55:35.698415+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:35.898230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678585+0000","last_change":"2023-12-19T14:55:35.678585+0000","last_active":"2023-12-19T14:55:35.678585+0000","last_peered":"2023-12-19T14:55:35.678585+0000","last_clean":"2023-12-19T14:55:35.678585+0000","last_became_active":"2023-12-19T14:55:35.674892+0000","last_became_peered":"2023-12-19T14:55:35.674892+0000","last_unstale":"2023-12-19T14:55:35.678585+0000","last_undegraded":"2023-12-19T14:55:35.678585+0000","last_fullsized":"2023-12-19T14:55:35.678585+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:15:24.018068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658255+0000","last_change":"2023-12-19T14:55:34.654926+0000","last_active":"2023-12-19T14:55:35.658255+0000","last_peered":"2023-12-19T14:55:35.658255+0000","last_clean":"2023-12-19T14:55:35.658255+0000","last_became_active":"2023-12-19T14:55:34.654257+0000","last_became_peered":"2023-12-19T14:55:34.654257+0000","last_unstale":"2023-12-19T14:55:35.658255+0000","last_undegraded":"2023-12-19T14:55:35.658255+0000","last_fullsized":"2023-12-19T14:55:35.658255+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:15.523614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.664416+0000","last_change":"2023-12-19T14:55:35.664416+0000","last_active":"2023-12-19T14:55:35.664416+0000","last_peered":"2023-12-19T14:55:35.664416+0000","last_clean":"2023-12-19T14:55:35.664416+0000","last_became_active":"2023-12-19T14:55:35.664053+0000","last_became_peered":"2023-12-19T14:55:35.664053+0000","last_unstale":"2023-12-19T14:55:35.664416+0000","last_undegraded":"2023-12-19T14:55:35.664416+0000","last_fullsized":"2023-12-19T14:55:35.664416+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:11.733787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682252+0000","last_change":"2023-12-19T14:55:34.665593+0000","last_active":"2023-12-19T14:55:35.682252+0000","last_peered":"2023-12-19T14:55:35.682252+0000","last_clean":"2023-12-19T14:55:35.682252+0000","last_became_active":"2023-12-19T14:55:34.664712+0000","last_became_peered":"2023-12-19T14:55:34.664712+0000","last_unstale":"2023-12-19T14:55:35.682252+0000","last_undegraded":"2023-12-19T14:55:35.682252+0000","last_fullsized":"2023-12-19T14:55:35.682252+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:07.297099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.679098+0000","last_change":"2023-12-19T14:55:35.679098+0000","last_active":"2023-12-19T14:55:35.679098+0000","last_peered":"2023-12-19T14:55:35.679098+0000","last_clean":"2023-12-19T14:55:35.679098+0000","last_became_active":"2023-12-19T14:55:35.678806+0000","last_became_peered":"2023-12-19T14:55:35.678806+0000","last_unstale":"2023-12-19T14:55:35.679098+0000","last_undegraded":"2023-12-19T14:55:35.679098+0000","last_fullsized":"2023-12-19T14:55:35.679098+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:23.549232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"112'1","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.658740+0000","last_change":"2023-12-19T14:55:34.667154+0000","last_active":"2023-12-19T14:55:35.658740+0000","last_peered":"2023-12-19T14:55:35.658740+0000","last_clean":"2023-12-19T14:55:35.658740+0000","last_became_active":"2023-12-19T14:55:34.666819+0000","last_became_peered":"2023-12-19T14:55:34.666819+0000","last_unstale":"2023-12-19T14:55:35.658740+0000","last_undegraded":"2023-12-19T14:55:35.658740+0000","last_fullsized":"2023-12-19T14:55:35.658740+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:53:59.931922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677704+0000","last_change":"2023-12-19T14:55:35.677704+0000","last_active":"2023-12-19T14:55:35.677704+0000","last_peered":"2023-12-19T14:55:35.677704+0000","last_clean":"2023-12-19T14:55:35.677704+0000","last_became_active":"2023-12-19T14:55:35.675256+0000","last_became_peered":"2023-12-19T14:55:35.675256+0000","last_unstale":"2023-12-19T14:55:35.677704+0000","last_undegraded":"2023-12-19T14:55:35.677704+0000","last_fullsized":"2023-12-19T14:55:35.677704+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:21.964252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.681645+0000","last_change":"2023-12-19T14:55:34.666257+0000","last_active":"2023-12-19T14:55:35.681645+0000","last_peered":"2023-12-19T14:55:35.681645+0000","last_clean":"2023-12-19T14:55:35.681645+0000","last_became_active":"2023-12-19T14:55:34.665972+0000","last_became_peered":"2023-12-19T14:55:34.665972+0000","last_unstale":"2023-12-19T14:55:35.681645+0000","last_undegraded":"2023-12-19T14:55:35.681645+0000","last_fullsized":"2023-12-19T14:55:35.681645+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:16.493003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.667345+0000","last_change":"2023-12-19T14:55:35.667345+0000","last_active":"2023-12-19T14:55:35.667345+0000","last_peered":"2023-12-19T14:55:35.667345+0000","last_clean":"2023-12-19T14:55:35.667345+0000","last_became_active":"2023-12-19T14:55:35.667005+0000","last_became_peered":"2023-12-19T14:55:35.667005+0000","last_unstale":"2023-12-19T14:55:35.667345+0000","last_undegraded":"2023-12-19T14:55:35.667345+0000","last_fullsized":"2023-12-19T14:55:35.667345+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:36:46.844201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.673553+0000","last_change":"2023-12-19T14:55:34.661994+0000","last_active":"2023-12-19T14:55:34.673553+0000","last_peered":"2023-12-19T14:55:34.673553+0000","last_clean":"2023-12-19T14:55:34.673553+0000","last_became_active":"2023-12-19T14:55:34.661633+0000","last_became_peered":"2023-12-19T14:55:34.661633+0000","last_unstale":"2023-12-19T14:55:34.673553+0000","last_undegraded":"2023-12-19T14:55:34.673553+0000","last_fullsized":"2023-12-19T14:55:34.673553+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:22:49.249620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.680955+0000","last_change":"2023-12-19T14:55:35.680955+0000","last_active":"2023-12-19T14:55:35.680955+0000","last_peered":"2023-12-19T14:55:35.680955+0000","last_clean":"2023-12-19T14:55:35.680955+0000","last_became_active":"2023-12-19T14:55:35.680668+0000","last_became_peered":"2023-12-19T14:55:35.680668+0000","last_unstale":"2023-12-19T14:55:35.680955+0000","last_undegraded":"2023-12-19T14:55:35.680955+0000","last_fullsized":"2023-12-19T14:55:35.680955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:37.133407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.674524+0000","last_change":"2023-12-19T14:55:34.673641+0000","last_active":"2023-12-19T14:55:34.674524+0000","last_peered":"2023-12-19T14:55:34.674524+0000","last_clean":"2023-12-19T14:55:34.674524+0000","last_became_active":"2023-12-19T14:55:34.673267+0000","last_became_peered":"2023-12-19T14:55:34.673267+0000","last_unstale":"2023-12-19T14:55:34.674524+0000","last_undegraded":"2023-12-19T14:55:34.674524+0000","last_fullsized":"2023-12-19T14:55:34.674524+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:24:29.732182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677681+0000","last_change":"2023-12-19T14:55:35.677681+0000","last_active":"2023-12-19T14:55:35.677681+0000","last_peered":"2023-12-19T14:55:35.677681+0000","last_clean":"2023-12-19T14:55:35.677681+0000","last_became_active":"2023-12-19T14:55:35.674674+0000","last_became_peered":"2023-12-19T14:55:35.674674+0000","last_unstale":"2023-12-19T14:55:35.677681+0000","last_undegraded":"2023-12-19T14:55:35.677681+0000","last_fullsized":"2023-12-19T14:55:35.677681+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:48.155868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.671232+0000","last_change":"2023-12-19T14:55:34.667695+0000","last_active":"2023-12-19T14:55:34.671232+0000","last_peered":"2023-12-19T14:55:34.671232+0000","last_clean":"2023-12-19T14:55:34.671232+0000","last_became_active":"2023-12-19T14:55:34.667408+0000","last_became_peered":"2023-12-19T14:55:34.667408+0000","last_unstale":"2023-12-19T14:55:34.671232+0000","last_undegraded":"2023-12-19T14:55:34.671232+0000","last_fullsized":"2023-12-19T14:55:34.671232+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:25:10.413744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.661471+0000","last_change":"2023-12-19T14:55:35.661471+0000","last_active":"2023-12-19T14:55:35.661471+0000","last_peered":"2023-12-19T14:55:35.661471+0000","last_clean":"2023-12-19T14:55:35.661471+0000","last_became_active":"2023-12-19T14:55:35.661132+0000","last_became_peered":"2023-12-19T14:55:35.661132+0000","last_unstale":"2023-12-19T14:55:35.661471+0000","last_undegraded":"2023-12-19T14:55:35.661471+0000","last_fullsized":"2023-12-19T14:55:35.661471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:52:38.467131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"112'2","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.682925+0000","last_change":"2023-12-19T14:55:34.661337+0000","last_active":"2023-12-19T14:55:35.682925+0000","last_peered":"2023-12-19T14:55:35.682925+0000","last_clean":"2023-12-19T14:55:35.682925+0000","last_became_active":"2023-12-19T14:55:34.661027+0000","last_became_peered":"2023-12-19T14:55:34.661027+0000","last_unstale":"2023-12-19T14:55:35.682925+0000","last_undegraded":"2023-12-19T14:55:35.682925+0000","last_fullsized":"2023-12-19T14:55:35.682925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:37:29.341509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.677528+0000","last_change":"2023-12-19T14:55:35.677528+0000","last_active":"2023-12-19T14:55:35.677528+0000","last_peered":"2023-12-19T14:55:35.677528+0000","last_clean":"2023-12-19T14:55:35.677528+0000","last_became_active":"2023-12-19T14:55:35.674797+0000","last_became_peered":"2023-12-19T14:55:35.674797+0000","last_unstale":"2023-12-19T14:55:35.677528+0000","last_undegraded":"2023-12-19T14:55:35.677528+0000","last_fullsized":"2023-12-19T14:55:35.677528+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:37.133407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.669853+0000","last_change":"2023-12-19T14:55:34.665684+0000","last_active":"2023-12-19T14:55:34.669853+0000","last_peered":"2023-12-19T14:55:34.669853+0000","last_clean":"2023-12-19T14:55:34.669853+0000","last_became_active":"2023-12-19T14:55:34.665293+0000","last_became_peered":"2023-12-19T14:55:34.665293+0000","last_unstale":"2023-12-19T14:55:34.669853+0000","last_undegraded":"2023-12-19T14:55:34.669853+0000","last_fullsized":"2023-12-19T14:55:34.669853+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:57:16.875555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687696+0000","last_change":"2023-12-19T14:55:35.687696+0000","last_active":"2023-12-19T14:55:35.687696+0000","last_peered":"2023-12-19T14:55:35.687696+0000","last_clean":"2023-12-19T14:55:35.687696+0000","last_became_active":"2023-12-19T14:55:35.687319+0000","last_became_peered":"2023-12-19T14:55:35.687319+0000","last_unstale":"2023-12-19T14:55:35.687696+0000","last_undegraded":"2023-12-19T14:55:35.687696+0000","last_fullsized":"2023-12-19T14:55:35.687696+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:13.153722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660758+0000","last_change":"2023-12-19T14:55:34.659331+0000","last_active":"2023-12-19T14:55:34.660758+0000","last_peered":"2023-12-19T14:55:34.660758+0000","last_clean":"2023-12-19T14:55:34.660758+0000","last_became_active":"2023-12-19T14:55:34.658955+0000","last_became_peered":"2023-12-19T14:55:34.658955+0000","last_unstale":"2023-12-19T14:55:34.660758+0000","last_undegraded":"2023-12-19T14:55:34.660758+0000","last_fullsized":"2023-12-19T14:55:34.660758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:06.195525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669984+0000","last_change":"2023-12-19T14:55:35.669984+0000","last_active":"2023-12-19T14:55:35.669984+0000","last_peered":"2023-12-19T14:55:35.669984+0000","last_clean":"2023-12-19T14:55:35.669984+0000","last_became_active":"2023-12-19T14:55:35.669701+0000","last_became_peered":"2023-12-19T14:55:35.669701+0000","last_unstale":"2023-12-19T14:55:35.669984+0000","last_undegraded":"2023-12-19T14:55:35.669984+0000","last_fullsized":"2023-12-19T14:55:35.669984+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:47:48.155868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660809+0000","last_change":"2023-12-19T14:55:34.657561+0000","last_active":"2023-12-19T14:55:34.660809+0000","last_peered":"2023-12-19T14:55:34.660809+0000","last_clean":"2023-12-19T14:55:34.660809+0000","last_became_active":"2023-12-19T14:55:34.655549+0000","last_became_peered":"2023-12-19T14:55:34.655549+0000","last_unstale":"2023-12-19T14:55:34.660809+0000","last_undegraded":"2023-12-19T14:55:34.660809+0000","last_fullsized":"2023-12-19T14:55:34.660809+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:23.212689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.687772+0000","last_change":"2023-12-19T14:55:35.687772+0000","last_active":"2023-12-19T14:55:35.687772+0000","last_peered":"2023-12-19T14:55:35.687772+0000","last_clean":"2023-12-19T14:55:35.687772+0000","last_became_active":"2023-12-19T14:55:35.687187+0000","last_became_peered":"2023-12-19T14:55:35.687187+0000","last_unstale":"2023-12-19T14:55:35.687772+0000","last_undegraded":"2023-12-19T14:55:35.687772+0000","last_fullsized":"2023-12-19T14:55:35.687772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:54:54.406543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.671963+0000","last_change":"2023-12-19T14:55:34.662850+0000","last_active":"2023-12-19T14:55:34.671963+0000","last_peered":"2023-12-19T14:55:34.671963+0000","last_clean":"2023-12-19T14:55:34.671963+0000","last_became_active":"2023-12-19T14:55:34.662521+0000","last_became_peered":"2023-12-19T14:55:34.662521+0000","last_unstale":"2023-12-19T14:55:34.671963+0000","last_undegraded":"2023-12-19T14:55:34.671963+0000","last_fullsized":"2023-12-19T14:55:34.671963+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:57.677388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.669626+0000","last_change":"2023-12-19T14:55:35.663500+0000","last_active":"2023-12-19T14:55:35.669626+0000","last_peered":"2023-12-19T14:55:35.669626+0000","last_clean":"2023-12-19T14:55:35.669626+0000","last_became_active":"2023-12-19T14:55:35.663194+0000","last_became_peered":"2023-12-19T14:55:35.663194+0000","last_unstale":"2023-12-19T14:55:35.669626+0000","last_undegraded":"2023-12-19T14:55:35.669626+0000","last_fullsized":"2023-12-19T14:55:35.669626+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:16.774265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.669297+0000","last_change":"2023-12-19T14:55:34.665492+0000","last_active":"2023-12-19T14:55:34.669297+0000","last_peered":"2023-12-19T14:55:34.669297+0000","last_clean":"2023-12-19T14:55:34.669297+0000","last_became_active":"2023-12-19T14:55:34.665182+0000","last_became_peered":"2023-12-19T14:55:34.665182+0000","last_unstale":"2023-12-19T14:55:34.669297+0000","last_undegraded":"2023-12-19T14:55:34.669297+0000","last_fullsized":"2023-12-19T14:55:34.669297+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:46.568673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.683746+0000","last_change":"2023-12-19T14:55:35.683746+0000","last_active":"2023-12-19T14:55:35.683746+0000","last_peered":"2023-12-19T14:55:35.683746+0000","last_clean":"2023-12-19T14:55:35.683746+0000","last_became_active":"2023-12-19T14:55:35.683407+0000","last_became_peered":"2023-12-19T14:55:35.683407+0000","last_unstale":"2023-12-19T14:55:35.683746+0000","last_undegraded":"2023-12-19T14:55:35.683746+0000","last_fullsized":"2023-12-19T14:55:35.683746+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:57.419174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.660217+0000","last_change":"2023-12-19T14:55:34.656987+0000","last_active":"2023-12-19T14:55:34.660217+0000","last_peered":"2023-12-19T14:55:34.660217+0000","last_clean":"2023-12-19T14:55:34.660217+0000","last_became_active":"2023-12-19T14:55:34.656349+0000","last_became_peered":"2023-12-19T14:55:34.656349+0000","last_unstale":"2023-12-19T14:55:34.660217+0000","last_undegraded":"2023-12-19T14:55:34.660217+0000","last_fullsized":"2023-12-19T14:55:34.660217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:15.698174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.678774+0000","last_change":"2023-12-19T14:55:35.678774+0000","last_active":"2023-12-19T14:55:35.678774+0000","last_peered":"2023-12-19T14:55:35.678774+0000","last_clean":"2023-12-19T14:55:35.678774+0000","last_became_active":"2023-12-19T14:55:35.677281+0000","last_became_peered":"2023-12-19T14:55:35.677281+0000","last_unstale":"2023-12-19T14:55:35.678774+0000","last_undegraded":"2023-12-19T14:55:35.678774+0000","last_fullsized":"2023-12-19T14:55:35.678774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:24.642145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.668457+0000","last_change":"2023-12-19T14:55:34.666545+0000","last_active":"2023-12-19T14:55:34.668457+0000","last_peered":"2023-12-19T14:55:34.668457+0000","last_clean":"2023-12-19T14:55:34.668457+0000","last_became_active":"2023-12-19T14:55:34.666259+0000","last_became_peered":"2023-12-19T14:55:34.666259+0000","last_unstale":"2023-12-19T14:55:34.668457+0000","last_undegraded":"2023-12-19T14:55:34.668457+0000","last_fullsized":"2023-12-19T14:55:34.668457+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:18.582399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":8,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.666155+0000","last_change":"2023-12-19T14:55:35.666155+0000","last_active":"2023-12-19T14:55:35.666155+0000","last_peered":"2023-12-19T14:55:35.666155+0000","last_clean":"2023-12-19T14:55:35.666155+0000","last_became_active":"2023-12-19T14:55:35.665860+0000","last_became_peered":"2023-12-19T14:55:35.665860+0000","last_unstale":"2023-12-19T14:55:35.666155+0000","last_undegraded":"2023-12-19T14:55:35.666155+0000","last_fullsized":"2023-12-19T14:55:35.666155+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:34.620511+0000","last_clean_scrub_stamp":"2023-12-19T14:55:34.620511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:58.691279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T14:55:35.650397+0000","last_change":"2023-12-19T14:55:34.672154+0000","last_active":"2023-12-19T14:55:35.650397+0000","last_peered":"2023-12-19T14:55:35.650397+0000","last_clean":"2023-12-19T14:55:35.650397+0000","last_became_active":"2023-12-19T14:55:34.670150+0000","last_became_peered":"2023-12-19T14:55:34.670150+0000","last_unstale":"2023-12-19T14:55:35.650397+0000","last_undegraded":"2023-12-19T14:55:35.650397+0000","last_fullsized":"2023-12-19T14:55:35.650397+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:55:33.606620+0000","last_clean_scrub_stamp":"2023-12-19T14:55:33.606620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:40.803441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"102'205","reported_seq":552,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-12-19T14:55:34.648185+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:55:34.648185+0000","last_peered":"2023-12-19T14:55:34.648185+0000","last_clean":"2023-12-19T14:55:34.648185+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:55:34.648185+0000","last_undegraded":"2023-12-19T14:55:34.648185+0000","last_fullsized":"2023-12-19T14:55:34.648185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1280486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.107,"15min":1.085},"min":{"1min":0.71299999999999997,"5min":0.59699999999999998,"15min":0.52300000000000002},"max":{"1min":1.466,"5min":3.544,"15min":3.544},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.999,"5min":1.139,"15min":1.131},"min":{"1min":0.629,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.4730000000000001,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.276}]},{"osd":1,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0820000000000001,"15min":1.095},"min":{"1min":0.74399999999999999,"5min":0.627,"15min":0.59299999999999997},"max":{"1min":1.722,"5min":1.722,"15min":1.73},"last":1.22},{"interface":"front","average":{"1min":1.171,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.75,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6080000000000001,"5min":1.6759999999999999,"15min":1.984},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.258,"15min":1.23},"min":{"1min":0.77900000000000003,"5min":0.68400000000000005,"15min":0.623},"max":{"1min":1.4590000000000001,"5min":1.5980000000000001,"15min":1.6919999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2529999999999999,"15min":1.2689999999999999},"min":{"1min":0.61099999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.5620000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.26,"15min":1.282},"min":{"1min":0.96599999999999997,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.649,"5min":1.649,"15min":1.7969999999999999},"last":1.161},{"interface":"front","average":{"1min":1.292,"5min":1.302,"15min":1.3069999999999999},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.3640000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.173,"15min":1.155},"min":{"1min":0.66900000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.677,"5min":1.752,"15min":2.0259999999999998},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.234,"15min":1.208},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.67500000000000004},"max":{"1min":1.5249999999999999,"5min":1.6180000000000001,"15min":1.7290000000000001},"last":0.95799999999999996}]},{"osd":5,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2150000000000001,"15min":1.212},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.6459999999999999,"5min":1.6930000000000001,"15min":1.8109999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3080000000000001,"15min":1.3049999999999999},"min":{"1min":0.81200000000000006,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.806,"5min":1.8919999999999999,"15min":2.2549999999999999},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:55:22 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3759999999999999,"15min":1.383},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.65600000000000003},"max":{"1min":1.744,"5min":1.9419999999999999,"15min":2.1930000000000001},"last":1.196},{"interface":"front","average":{"1min":1.379,"5min":1.3660000000000001,"15min":1.351},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.70299999999999996},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.948},"last":1.242}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":692174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1140000000000001,"15min":1.131},"min":{"1min":0.68500000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.5960000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1459999999999999,"15min":1.1699999999999999},"min":{"1min":0.83199999999999996,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.528,"5min":1.845,"15min":1.845},"last":1.512}]},{"osd":1,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.173,"15min":1.1890000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.444,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.141},{"interface":"front","average":{"1min":1.212,"5min":1.2330000000000001,"15min":1.246},"min":{"1min":0.82599999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.573,"5min":3.226,"15min":3.226},"last":1.236}]},{"osd":2,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.276,"15min":1.2470000000000001},"min":{"1min":0.74399999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7649999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":1.159},{"interface":"front","average":{"1min":1.133,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.67100000000000004},"max":{"1min":1.4359999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.1859999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.296,"15min":1.298},"min":{"1min":0.91200000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5880000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.341},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.3200000000000001,"15min":1.331},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6799999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.202}]},{"osd":4,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.2110000000000001,"15min":1.2050000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.55600000000000005},"max":{"1min":1.569,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.266,"15min":1.2969999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.57599999999999996},"max":{"1min":1.6619999999999999,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":1.603}]},{"osd":5,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.2809999999999999,"15min":1.2949999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.671,"5min":2.02,"15min":2.0230000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3720000000000001,"15min":1.357},"min":{"1min":0.88700000000000001,"5min":0.82799999999999996,"15min":0.51000000000000001},"max":{"1min":1.5429999999999999,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":0.89100000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:55:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.431,"15min":1.413},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.53000000000000003},"max":{"1min":1.716,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.4490000000000001,"15min":1.454},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":1.8440000000000001,"5min":3.742,"15min":3.742},"last":1.0980000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298636,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073204,"statfs":{"total":96636764160,"available":96330960896,"internally_reserved":0,"allocated":1380352,"data_stored":690062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.044,"15min":1.0640000000000001},"min":{"1min":0.63700000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.52,"5min":1.601,"15min":1.619},"last":1.369},{"interface":"front","average":{"1min":1.03,"5min":1.119,"15min":1.1299999999999999},"min":{"1min":0.69199999999999995,"5min":0.63900000000000001,"15min":0.58699999999999997},"max":{"1min":1.4430000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.456}]},{"osd":1,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.262,"15min":1.2849999999999999},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.6100000000000001,"5min":1.8029999999999999,"15min":1.9650000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.2150000000000001,"15min":1.2350000000000001},"min":{"1min":0.77300000000000002,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.456,"5min":1.677,"15min":1.7869999999999999},"last":1.4950000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1679999999999999,"15min":1.23},"min":{"1min":0.72699999999999998,"5min":0.67000000000000004,"15min":0.47399999999999998},"max":{"1min":1.756,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.137,"15min":1.1870000000000001},"min":{"1min":0.82099999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.45,"5min":1.589,"15min":1.954},"last":0.92800000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.2689999999999999,"15min":1.264},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6200000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.3069999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.242,"15min":1.2629999999999999},"min":{"1min":0.84999999999999998,"5min":0.82299999999999995,"15min":0.68999999999999995},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.105}]},{"osd":4,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.5489999999999999,"5min":1.603,"15min":1.867},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.224,"15min":1.2370000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.7529999999999999,"5min":2.27,"15min":2.27},"last":1.6559999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3049999999999999,"15min":1.3260000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.7869999999999999,"5min":1.982,"15min":2.0630000000000002},"last":1.252},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.319,"15min":1.321},"min":{"1min":0.80000000000000004,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.6499999999999999,"5min":1.8540000000000001,"15min":1.9219999999999999},"last":1.615}]},{"osd":7,"last update":"Tue Dec 19 14:55:29 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.347,"15min":1.353},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.58899999999999997},"max":{"1min":1.5900000000000001,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.554},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.353,"15min":1.377},"min":{"1min":0.99399999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.95,"5min":1.95,"15min":2.2080000000000002},"last":1.4139999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302740,"kb_used_data":1356,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069100,"statfs":{"total":96636764160,"available":96326758400,"internally_reserved":0,"allocated":1388544,"data_stored":690222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0549999999999999,"15min":1.0660000000000001},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.625,"5min":1.625,"15min":1.7869999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.083,"15min":1.083},"min":{"1min":0.68500000000000005,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.7490000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0509999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2290000000000001,"15min":1.24},"min":{"1min":0.80600000000000005,"5min":0.623,"15min":0.623},"max":{"1min":1.726,"5min":1.8100000000000001,"15min":1.966},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.234,"15min":1.2689999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5229999999999999,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.23}]},{"osd":2,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.56999999999999995},"max":{"1min":1.5700000000000001,"5min":1.625,"15min":1.7330000000000001},"last":1.008},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.157,"15min":1.1990000000000001},"min":{"1min":0.73499999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.893},"last":1.0820000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1639999999999999,"15min":1.1799999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.6100000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.2390000000000001,"15min":1.262},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.60599999999999998},"max":{"1min":1.635,"5min":1.7749999999999999,"15min":2.0190000000000001},"last":1.133}]},{"osd":5,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1619999999999999,"15min":1.151},"min":{"1min":0.82699999999999996,"5min":0.72399999999999998,"15min":0.57899999999999996},"max":{"1min":1.6559999999999999,"5min":2.1800000000000002,"15min":2.3300000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.1559999999999999,"15min":1.1579999999999999},"min":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.6830000000000001,"5min":2.077,"15min":2.161},"last":1.196}]},{"osd":6,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.2969999999999999,"15min":1.304},"min":{"1min":0.63900000000000001,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.8069999999999999,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4139999999999999,"15min":1.4059999999999999},"min":{"1min":0.89900000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.1549999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:55:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.339,"15min":1.3280000000000001},"min":{"1min":0.88200000000000001,"5min":0.78200000000000003,"15min":0.60899999999999999},"max":{"1min":1.5760000000000001,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.448,"15min":1.4399999999999999},"min":{"1min":0.94999999999999996,"5min":0.83299999999999996,"15min":0.70599999999999996},"max":{"1min":1.9370000000000001,"5min":3.008,"15min":3.008},"last":1.363}]}]},{"osd":3,"up_from":13,"seq":55834575046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306840,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065000,"statfs":{"total":96636764160,"available":96322560000,"internally_reserved":0,"allocated":1392640,"data_stored":696698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83899999999999997,"15min":0.86199999999999999},"min":{"1min":0.54900000000000004,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.2889999999999999,"5min":1.3640000000000001,"15min":1.444},"last":1.276},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.97799999999999998,"15min":0.98999999999999999},"min":{"1min":0.59499999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4750000000000001,"5min":1.542,"15min":1.7270000000000001},"last":0.77400000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.90200000000000002,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.345,"5min":1.611,"15min":1.611},"last":1.171},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.1020000000000001,"15min":1.099},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5780000000000001,"5min":1.661,"15min":1.758},"last":1.381}]},{"osd":2,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1439999999999999,"15min":1.127},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.56100000000000005},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.23,"15min":1.2370000000000001},"min":{"1min":0.79100000000000004,"5min":0.67900000000000005,"15min":0.63500000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.8520000000000001},"last":1.5029999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.3160000000000001,"15min":1.3260000000000001},"min":{"1min":1.071,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.6799999999999999,"5min":1.736,"15min":1.899},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.29,"15min":1.296},"min":{"1min":1.075,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.599,"5min":1.6879999999999999,"15min":1.7989999999999999},"last":1.2490000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":1.04,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.518,"5min":1.7709999999999999,"15min":1.893},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.371,"15min":1.389},"min":{"1min":1.056,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.827,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4399999999999999,"15min":1.4490000000000001},"min":{"1min":1.0640000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8520000000000001,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.4239999999999999,"15min":1.4159999999999999},"min":{"1min":1.014,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7629999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.139}]},{"osd":7,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5189999999999999,"15min":1.5129999999999999},"min":{"1min":1.0249999999999999,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.8340000000000001,"5min":1.99,"15min":2.1640000000000001},"last":1.345},{"interface":"front","average":{"1min":1.502,"5min":1.464,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.704,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.415}]}]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306824,"kb_used_data":1344,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065016,"statfs":{"total":96636764160,"available":96322576384,"internally_reserved":0,"allocated":1376256,"data_stored":689051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.61299999999999999,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1},{"interface":"front","average":{"1min":1.03,"5min":1.0429999999999999,"15min":1.0660000000000001},"min":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":1.2689999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1140000000000001,"15min":1.1299999999999999},"min":{"1min":0.64300000000000002,"5min":0.56599999999999995,"15min":0.46500000000000002},"max":{"1min":1.544,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.411},{"interface":"front","average":{"1min":1.002,"5min":1.0660000000000001,"15min":1.0920000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4119999999999999,"5min":1.853,"15min":1.853},"last":0.93000000000000005}]},{"osd":3,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.0029999999999999,"15min":0.99199999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.401,"5min":1.5669999999999999,"15min":1.623},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.073,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.0549999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3700000000000001,"15min":1.3759999999999999},"min":{"1min":0.88,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.044},"last":1.607},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3859999999999999,"15min":1.389},"min":{"1min":0.85299999999999998,"5min":0.746,"15min":0.746},"max":{"1min":1.633,"5min":1.96,"15min":1.96},"last":1.6699999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.363,"15min":1.3440000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.395},"last":1.238},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3320000000000001,"15min":1.337},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.718,"5min":2.1200000000000001,"15min":2.9009999999999998},"last":1.46}]},{"osd":6,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3240000000000001,"15min":1.3600000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80600000000000005},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":2.8759999999999999},"last":1.389},{"interface":"front","average":{"1min":1.431,"5min":1.4279999999999999,"15min":1.4359999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.5390000000000001},"last":1.5309999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:55:18 2023","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.486,"15min":1.48},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8260000000000001,"5min":2.0699999999999998,"15min":2.2679999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4690000000000001,"15min":1.4750000000000001},"min":{"1min":1.079,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.875,"5min":2.1190000000000002,"15min":2.1819999999999999},"last":1.504}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306832,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065008,"statfs":{"total":96636764160,"available":96322568192,"internally_reserved":0,"allocated":1384448,"data_stored":690132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.89800000000000002,"15min":0.86799999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.443},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96999999999999997,"15min":0.97999999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.50700000000000001},"max":{"1min":1.2669999999999999,"5min":1.7070000000000001,"15min":1.802},"last":0.84999999999999998}]},{"osd":2,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.028,"15min":1.018},"min":{"1min":0.65800000000000003,"5min":0.54600000000000004,"15min":0.46500000000000002},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0860000000000001,"15min":1.117},"min":{"1min":0.66000000000000003,"5min":0.442,"15min":0.442},"max":{"1min":1.702,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.74199999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.222,"15min":1.2330000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.59199999999999997},"max":{"1min":1.53,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.083,"15min":1.073},"min":{"1min":0.64500000000000002,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.5249999999999999,"5min":1.7250000000000001,"15min":1.7270000000000001},"last":1.2909999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.2969999999999999,"15min":1.298},"min":{"1min":0.89600000000000002,"5min":0.624,"15min":0.624},"max":{"1min":2.5670000000000002,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.381,"15min":1.391},"min":{"1min":0.92800000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6719999999999999,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.0549999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.268,"15min":1.2909999999999999},"min":{"1min":1.0069999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.3939999999999999,"15min":1.385},"min":{"1min":1.042,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":1.26}]},{"osd":6,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3360000000000001,"15min":1.3580000000000001},"min":{"1min":0.97899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.6990000000000001,"5min":1.78,"15min":1.9179999999999999},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4399999999999999,"15min":1.446},"min":{"1min":1.0800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.335}]},{"osd":7,"last update":"Tue Dec 19 14:55:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.4370000000000001,"15min":1.466},"min":{"1min":1.046,"5min":0.89500000000000002,"15min":0.73599999999999999},"max":{"1min":1.7,"5min":2.0470000000000002,"15min":2.0590000000000002},"last":1.429},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":1.171,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":1.7649999999999999,"5min":1.8129999999999999,"15min":2.0609999999999999},"last":1.804}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307428,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064412,"statfs":{"total":96636764160,"available":96321957888,"internally_reserved":0,"allocated":1994752,"data_stored":1281633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83499999999999996,"15min":0.84399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3109999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96899999999999997,"15min":0.96399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.325,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.97999999999999998}]},{"osd":2,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.107,"15min":1.095},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":1.639},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.228,"15min":1.214},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.54300000000000004},"max":{"1min":1.7669999999999999,"5min":1.982,"15min":1.982},"last":0.85899999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.024,"15min":1.0209999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.5029999999999999,"5min":1.726,"15min":2.0299999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.133,"5min":1.167,"15min":1.1759999999999999},"min":{"1min":0.621,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.756}]},{"osd":4,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2949999999999999,"15min":1.2909999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.631,"5min":2.153,"15min":2.153},"last":1.117},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2889999999999999,"15min":1.288},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.7170000000000001,"5min":1.7689999999999999,"15min":1.9359999999999999},"last":1.081}]},{"osd":5,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.377,"15min":1.371},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":2.4630000000000001},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.25,"5min":1.3720000000000001,"15min":1.3839999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.58299999999999996},"max":{"1min":1.825,"5min":2.2189999999999999,"15min":2.2229999999999999},"last":1.252}]},{"osd":6,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3200000000000001,"15min":1.3300000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.589,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.4390000000000001,"15min":1.4410000000000001},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.887,"5min":2.113,"15min":2.3799999999999999},"last":1.9710000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4670000000000001,"15min":1.452},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.893,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.405,"15min":1.425},"min":{"1min":0.98699999999999999,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.7470000000000001,"5min":1.931,"15min":2.0800000000000001},"last":1.1890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:55:40.712 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:55:40.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:55:40.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.933+0000 7f3c33345700 1 Processor -- start 2023-12-19T14:55:40.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.933+0000 7f3c33345700 1 -- start start 2023-12-19T14:55:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c33345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c2c06b7a0 0x7f3c2c161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c33345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0cbee0 con 0x7f3c2c0cae60 2023-12-19T14:55:40.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c33345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0cc370 con 0x7f3c2c1514a0 2023-12-19T14:55:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c33345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c0cb2d0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c318e2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3c2c06b7a0 0x7f3c2c161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c308e0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3c2c0cae60 0x7f3c2c06b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44968/0 (socket says 172.21.15.50:44968) 2023-12-19T14:55:40.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c308e0700 1 -- 172.21.15.50:0/2885565244 learned_addr learned my addr 172.21.15.50:0/2885565244 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c23fff700 1 -- 172.21.15.50:0/2885565244 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555075532 0 0) 0x7f3c2c0cbee0 con 0x7f3c2c0cae60 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c23fff700 1 -- 172.21.15.50:0/2885565244 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c18003570 con 0x7f3c2c0cae60 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c318e2700 1 -- 172.21.15.50:0/2885565244 >> v1:172.21.15.125:6789/0 conn(0x7f3c2c1514a0 legacy=0x7f3c2c157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c318e2700 1 -- 172.21.15.50:0/2885565244 >> v1:172.21.15.50:6789/0 conn(0x7f3c2c0cae60 legacy=0x7f3c2c06b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c318e2700 1 -- 172.21.15.50:0/2885565244 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c0cb5b0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.937+0000 7f3c318e2700 1 --2- 172.21.15.50:0/2885565244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c161f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009e60 tx=0x7f3c24009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83817d49468ce489 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c23fff700 1 -- 172.21.15.50:0/2885565244 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400e070 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c23fff700 1 -- 172.21.15.50:0/2885565244 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2400bb60 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c23fff700 1 -- 172.21.15.50:0/2885565244 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24012430 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 -- 172.21.15.50:0/2885565244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c06b7a0 msgr2=0x7f3c2c161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 --2- 172.21.15.50:0/2885565244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c161f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009e60 tx=0x7f3c24009780 comp rx=0 tx=0).stop 2023-12-19T14:55:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 -- 172.21.15.50:0/2885565244 shutdown_connections 2023-12-19T14:55:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 --2- 172.21.15.50:0/2885565244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c161f10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 -- 172.21.15.50:0/2885565244 >> 172.21.15.50:0/2885565244 conn(0x7f3c2c0cc580 msgr2=0x7f3c2c0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 -- 172.21.15.50:0/2885565244 shutdown_connections 2023-12-19T14:55:40.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 -- 172.21.15.50:0/2885565244 wait complete. 2023-12-19T14:55:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.941+0000 7f3c33345700 1 Processor -- start 2023-12-19T14:55:40.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 -- start start 2023-12-19T14:55:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c0cae60 0x7f3c2c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3c2c1514a0 0x7f3c2c152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c74d0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c15c1d0 con 0x7f3c2c0cae60 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c0c6690 con 0x7f3c2c1514a0 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48840/0 (socket says 172.21.15.50:48840) 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 -- 172.21.15.50:0/2853866524 learned_addr learned my addr 172.21.15.50:0/2853866524 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c308e0700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c0cae60 0x7f3c2c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c318e2700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3c2c1514a0 0x7f3c2c152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 -- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3c2c1514a0 msgr2=0x7f3c2c152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3c2c1514a0 0x7f3c2c152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 -- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c0cae60 msgr2=0x7f3c2c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c0cae60 0x7f3c2c154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 -- 172.21.15.50:0/2853866524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009a20 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c310e1700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c790 tx=0x7f3c28007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28011590 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c0c6ba0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c28011710 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.945+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28011ad0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.949+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c0c7160 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.949+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c071120 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.949+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3c2801b3b0 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.953+0000 7f3c217fa700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 0x7f3c10062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.953+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f3c2807bf70 con 0x7f3c2c06b7a0 2023-12-19T14:55:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.953+0000 7f3c308e0700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 0x7f3c10062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.953+0000 7f3c308e0700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 0x7f3c10062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0b0c80 tx=0x7f3c1c009040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:40.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:40.957+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c2804c250 con 0x7f3c2c06b7a0 2023-12-19T14:55:41.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.185+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3c2c0afd30 con 0x7f3c2c06b7a0 2023-12-19T14:55:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.197+0000 7f3c217fa700 1 -- 172.21.15.50:0/2853866524 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3c28017020 con 0x7f3c2c06b7a0 2023-12-19T14:55:41.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 msgr2=0x7f3c10062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 0x7f3c10062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0b0c80 tx=0x7f3c1c009040 comp rx=0 tx=0).stop 2023-12-19T14:55:41.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 msgr2=0x7f3c2c15bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c790 tx=0x7f3c28007f40 comp rx=0 tx=0).stop 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 shutdown_connections 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3c2c0cae60 0x7f3c2c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3c10060670 0x7f3c10062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3c2c1514a0 0x7f3c2c152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 --2- 172.21.15.50:0/2853866524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c2c06b7a0 0x7f3c2c15bc90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 >> 172.21.15.50:0/2853866524 conn(0x7f3c2c0cc580 msgr2=0x7f3c2c154020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 shutdown_connections 2023-12-19T14:55:41.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.201+0000 7f3c33345700 1 -- 172.21.15.50:0/2853866524 wait complete. 2023-12-19T14:55:41.206 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:55:41.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:55:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.389+0000 7f1f5da43700 1 Processor -- start 2023-12-19T14:55:41.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.389+0000 7f1f5da43700 1 -- start start 2023-12-19T14:55:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5da43700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f58155960 0x7f1f5815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:41.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5da43700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5806d990 con 0x7f1f580c7f00 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5da43700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5815a570 con 0x7f1f5806de20 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5da43700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5806cd80 con 0x7f1f58155960 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f58155960 0x7f1f5815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1f58155960 0x7f1f5815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48850/0 (socket says 172.21.15.50:48850) 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 -- 172.21.15.50:0/686507628 learned_addr learned my addr 172.21.15.50:0/686507628 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:41.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5d242700 1 --1- 172.21.15.50:0/686507628 >> v1:172.21.15.50:6789/0 conn(0x7f1f580c7f00 0x7f1f580aca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44980/0 (socket says 172.21.15.50:44980) 2023-12-19T14:55:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 -- 172.21.15.50:0/686507628 >> v1:172.21.15.125:6789/0 conn(0x7f1f5806de20 legacy=0x7f1f580c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 -- 172.21.15.50:0/686507628 >> v1:172.21.15.50:6789/0 conn(0x7f1f580c7f00 legacy=0x7f1f580aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 -- 172.21.15.50:0/686507628 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5806d060 con 0x7f1f58155960 2023-12-19T14:55:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f5ca41700 1 --2- 172.21.15.50:0/686507628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f58155960 0x7f1f5815a030 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0099b0 tx=0x7f1f4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1167659270dda75a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:41.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.393+0000 7f1f577fe700 1 -- 172.21.15.50:0/686507628 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c00b860 con 0x7f1f58155960 2023-12-19T14:55:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f577fe700 1 -- 172.21.15.50:0/686507628 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c00b9c0 con 0x7f1f58155960 2023-12-19T14:55:41.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f577fe700 1 -- 172.21.15.50:0/686507628 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c00bd60 con 0x7f1f58155960 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 -- 172.21.15.50:0/686507628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f58155960 msgr2=0x7f1f5815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 --2- 172.21.15.50:0/686507628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f58155960 0x7f1f5815a030 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0099b0 tx=0x7f1f4c009250 comp rx=0 tx=0).stop 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 -- 172.21.15.50:0/686507628 shutdown_connections 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 --2- 172.21.15.50:0/686507628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f58155960 0x7f1f5815a030 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 -- 172.21.15.50:0/686507628 >> 172.21.15.50:0/686507628 conn(0x7f1f58067bb0 msgr2=0x7f1f580cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 -- 172.21.15.50:0/686507628 shutdown_connections 2023-12-19T14:55:41.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.397+0000 7f1f5da43700 1 -- 172.21.15.50:0/686507628 wait complete. 2023-12-19T14:55:41.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 Processor -- start 2023-12-19T14:55:41.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 -- start start 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 0x7f1f58153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 0x7f1f58148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581522d0 con 0x7f1f580c7f00 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58154d00 con 0x7f1f5806de20 2023-12-19T14:55:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5da43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581540f0 con 0x7f1f58155960 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48854/0 (socket says 172.21.15.50:48854) 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 -- 172.21.15.50:0/1382636118 learned_addr learned my addr 172.21.15.50:0/1382636118 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5ca41700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 0x7f1f58153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5d242700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 0x7f1f58148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 -- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 msgr2=0x7f1f58148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 0x7f1f58148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 -- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 msgr2=0x7f1f58153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 0x7f1f58153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f57fff700 1 -- 172.21.15.50:0/1382636118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c0094f0 con 0x7f1f580c7f00 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5ca41700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 0x7f1f58153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.401+0000 7f1f5d242700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 0x7f1f58148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f57fff700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c590 tx=0x7f1f4400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44011590 con 0x7f1f580c7f00 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f440116f0 con 0x7f1f580c7f00 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44011ab0 con 0x7f1f580c7f00 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f581543d0 con 0x7f1f580c7f00 2023-12-19T14:55:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f58154990 con 0x7f1f580c7f00 2023-12-19T14:55:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.405+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5815bca0 con 0x7f1f580c7f00 2023-12-19T14:55:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.409+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1f4401b3b0 con 0x7f1f580c7f00 2023-12-19T14:55:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.409+0000 7f1f54ff9700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 0x7f1f3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.409+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f1f4407c090 con 0x7f1f580c7f00 2023-12-19T14:55:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.409+0000 7f1f5ca41700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 0x7f1f3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.409+0000 7f1f5ca41700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 0x7f1f3c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0099b0 tx=0x7f1f4c017000 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:41.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.417+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1f4404c370 con 0x7f1f580c7f00 2023-12-19T14:55:41.614 INFO:tasks.ceph.mds.a.smithi050.stderr:2023-12-19T14:55:41.609+0000 7f6cbd088700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:55:41.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.625+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f5804be90 con 0x7f1f580c7f00 2023-12-19T14:55:41.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.629+0000 7f1f54ff9700 1 -- 172.21.15.50:0/1382636118 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f1f44017090 con 0x7f1f580c7f00 2023-12-19T14:55:41.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:41.632 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 msgr2=0x7f1f3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 0x7f1f3c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0099b0 tx=0x7f1f4c017000 comp rx=0 tx=0).stop 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 msgr2=0x7f1f58152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c590 tx=0x7f1f4400c950 comp rx=0 tx=0).stop 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 shutdown_connections 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1f3c060670 0x7f1f3c062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1f58155960 0x7f1f58148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f580c7f00 0x7f1f58152e90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 --2- 172.21.15.50:0/1382636118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f5806de20 0x7f1f58153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 >> 172.21.15.50:0/1382636118 conn(0x7f1f58067bb0 msgr2=0x7f1f5814a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 shutdown_connections 2023-12-19T14:55:41.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:41.633+0000 7f1f5da43700 1 -- 172.21.15.50:0/1382636118 wait complete. 2023-12-19T14:55:41.639 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:55:42.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:55:35.098455+0000', 'modified': '2023-12-19T14:55:38.968315+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7377}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7377': {'gid': 7377, 'name': 'a', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/3096035349', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6833', 'nonce': 3096035349}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 3096035349}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:55:42.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:55:42.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T14:55:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.241+0000 7f06283c5700 1 Processor -- start 2023-12-19T14:55:42.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.241+0000 7f06283c5700 1 -- start start 2023-12-19T14:55:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0620158ff0 0x7f062006b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06200b3dd0 0x7f062006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f06200b41e0 0x7f06200b4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06200b4b80 con 0x7f0620158ff0 2023-12-19T14:55:42.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f062006d570 con 0x7f06200b41e0 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f06283c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062006c960 con 0x7f06200b3dd0 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0620158ff0 0x7f062006b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0620158ff0 0x7f062006b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48868/0 (socket says 172.21.15.50:48868) 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 -- 172.21.15.50:0/1135337796 learned_addr learned my addr 172.21.15.50:0/1135337796 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0625960700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06200b3dd0 0x7f062006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626962700 1 --2- 172.21.15.50:0/1135337796 >> v2:172.21.15.125:3300/0 conn(0x7f06200b41e0 0x7f06200b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 -- 172.21.15.50:0/1135337796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06200b3dd0 msgr2=0x7f062006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06200b3dd0 0x7f062006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 -- 172.21.15.50:0/1135337796 >> v2:172.21.15.125:3300/0 conn(0x7f06200b41e0 msgr2=0x7f06200b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 --2- 172.21.15.50:0/1135337796 >> v2:172.21.15.125:3300/0 conn(0x7f06200b41e0 0x7f06200b4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.245+0000 7f0626161700 1 -- 172.21.15.50:0/1135337796 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062006cc40 con 0x7f0620158ff0 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f0626161700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0620158ff0 0x7f062006b310 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f062006abd0 tx=0x7f061c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37e7e489d33665b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f062515f700 1 -- 172.21.15.50:0/1135337796 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c00b860 con 0x7f0620158ff0 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f062515f700 1 -- 172.21.15.50:0/1135337796 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c00b9c0 con 0x7f0620158ff0 2023-12-19T14:55:42.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f062515f700 1 -- 172.21.15.50:0/1135337796 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c00bd60 con 0x7f0620158ff0 2023-12-19T14:55:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 -- 172.21.15.50:0/1135337796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0620158ff0 msgr2=0x7f062006b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0620158ff0 0x7f062006b310 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f062006abd0 tx=0x7f061c009250 comp rx=0 tx=0).stop 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 -- 172.21.15.50:0/1135337796 shutdown_connections 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 --2- 172.21.15.50:0/1135337796 >> v2:172.21.15.125:3300/0 conn(0x7f06200b41e0 0x7f06200b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f06200b3dd0 0x7f062006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 --2- 172.21.15.50:0/1135337796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0620158ff0 0x7f062006b310 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 -- 172.21.15.50:0/1135337796 >> 172.21.15.50:0/1135337796 conn(0x7f06200670e0 msgr2=0x7f06200674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 -- 172.21.15.50:0/1135337796 shutdown_connections 2023-12-19T14:55:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.249+0000 7f06283c5700 1 -- 172.21.15.50:0/1135337796 wait complete. 2023-12-19T14:55:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 Processor -- start 2023-12-19T14:55:42.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 -- start start 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06200b41e0 0x7f0620154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0620158ff0 0x7f0620154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06201377d0 con 0x7f06200b3dd0 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0620150820 con 0x7f06200b41e0 2023-12-19T14:55:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f06283c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062014cd80 con 0x7f0620158ff0 2023-12-19T14:55:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48874/0 (socket says 172.21.15.50:48874) 2023-12-19T14:55:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 -- 172.21.15.50:0/1793407323 learned_addr learned my addr 172.21.15.50:0/1793407323 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0625960700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06200b41e0 0x7f0620154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626962700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0620158ff0 0x7f0620154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 -- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0620158ff0 msgr2=0x7f0620154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0620158ff0 0x7f0620154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 -- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06200b41e0 msgr2=0x7f0620154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06200b41e0 0x7f0620154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 -- 172.21.15.50:0/1793407323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c0094f0 con 0x7f06200b3dd0 2023-12-19T14:55:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.253+0000 7f0626161700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0620069d10 tx=0x7f061c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.257+0000 7f06167fc700 1 -- 172.21.15.50:0/1793407323 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c011dc0 con 0x7f06200b3dd0 2023-12-19T14:55:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.257+0000 7f06167fc700 1 -- 172.21.15.50:0/1793407323 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c024440 con 0x7f06200b3dd0 2023-12-19T14:55:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.257+0000 7f06167fc700 1 -- 172.21.15.50:0/1793407323 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c024800 con 0x7f06200b3dd0 2023-12-19T14:55:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.257+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062014d060 con 0x7f06200b3dd0 2023-12-19T14:55:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.257+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062014e1f0 con 0x7f06200b3dd0 2023-12-19T14:55:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f06167fc700 1 -- 172.21.15.50:0/1793407323 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f061c01a420 con 0x7f06200b3dd0 2023-12-19T14:55:42.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f06167fc700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 0x7f0604062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f06167fc700 1 -- 172.21.15.50:0/1793407323 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f061c07e960 con 0x7f06200b3dd0 2023-12-19T14:55:42.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f0625960700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 0x7f0604062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f062a699740 1 Processor -- start 2023-12-19T14:55:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.261+0000 7f062a699740 1 -- start start 2023-12-19T14:55:42.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0625960700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 0x7f0604062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f06200b4860 tx=0x7f0610008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 0x10fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1029f30 con 0xe7eae0 2023-12-19T14:55:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1025ed0 con 0xec9790 2023-12-19T14:55:42.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0xfec0b0 con 0xe7e710 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48876/0 (socket says 172.21.15.50:48876) 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 -- 172.21.15.50:0/944596496 learned_addr learned my addr 172.21.15.50:0/944596496 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626962700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0625960700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 0x10fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 -- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 msgr2=0x10fc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 0x10fc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 -- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 msgr2=0x10fc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10fc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 -- 172.21.15.50:0/944596496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xeca9e0 con 0xe7eae0 2023-12-19T14:55:42.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0625960700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 0x10fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f0626161700 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f061c014040 tx=0x7f061c003040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba5c71cf2d15dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f06037fe700 1 -- 172.21.15.50:0/944596496 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c055b90 con 0xe7eae0 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f06037fe700 1 -- 172.21.15.50:0/944596496 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c0598e0 con 0xe7eae0 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.265+0000 7f06037fe700 1 -- 172.21.15.50:0/944596496 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c050ca0 con 0xe7eae0 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- 172.21.15.50:0/944596496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 msgr2=0x10fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f061c014040 tx=0x7f061c003040 comp rx=0 tx=0).stop 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- 172.21.15.50:0/944596496 shutdown_connections 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10fc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7e710 0x10fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 --2- 172.21.15.50:0/944596496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7eae0 0x10fbf00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- 172.21.15.50:0/944596496 >> 172.21.15.50:0/944596496 conn(0xe7eeb0 msgr2=0x10b8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- 172.21.15.50:0/944596496 shutdown_connections 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- 172.21.15.50:0/944596496 wait complete. 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 -- start start 2023-12-19T14:55:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 Processor -- start 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.269+0000 7f062a699740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7eae0 0x1100a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xec9790 0x10e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0xfebd50 con 0xe7e710 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0xfc4970 con 0xe7eae0 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0xfee420 con 0xec9790 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 10 client.0 _ll_register_callbacks cb 0x106f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48886/0 (socket says 172.21.15.50:48886) 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 -- 172.21.15.50:0/414826789 learned_addr learned my addr 172.21.15.50:0/414826789 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0625960700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7eae0 0x1100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 -- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xec9790 msgr2=0x10e1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xec9790 0x10e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 -- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7eae0 msgr2=0x1100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7eae0 0x1100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 -- 172.21.15.50:0/414826789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062014e1f0 con 0xe7e710 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0626161700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f061c052890 tx=0x7f061c069000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb2fb26a2424fd6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c055b90 con 0xe7e710 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f061c07c880 con 0xe7e710 2023-12-19T14:55:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xfc4cc0 con 0xe7e710 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f061c02b340 con 0xe7e710 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 10 client.7518 fetch_fsmap learned FSMap version 50 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0xff4370 con 0xe7e710 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xfc4cc0 con 0xe7e710 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 50) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f061c078070 con 0xe7e710 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 10 client.7518 fetch_fsmap finished waiting for FSMap version 50 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 10 client.7518 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 20 client.7518 populate_metadata read hostname 'smithi050' 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x10e3360 0x110b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 4 client.7518 mds_command: new command op to 7377 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:55:42.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 --> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xfc83e0 con 0x10e3360 2023-12-19T14:55:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.273+0000 7f0625960700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x10e3360 0x110b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f0600ff9700 10 client.7518 ms_handle_connect on v2:172.21.15.50:6833/3096035349 2023-12-19T14:55:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f0625960700 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x10e3360 0x110b6b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4991f70bf59ddf88 server_cookie=31410af28567f76b in_seq=0 out_seq=0 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f0600ff9700 1 -- 172.21.15.50:0/414826789 <== mds.0 v2:172.21.15.50:6833/3096035349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0xfc83e0 con 0x10e3360 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f0600ff9700 10 client.7518 handle_command_reply: tid=0 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 client.7518 shutdown 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 msgr2=0x1100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f061c052890 tx=0x7f061c069000 comp rx=0 tx=0).stop 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 shutdown_connections 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x10e3360 0x110b6b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7eae0 0x1100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xec9790 0x10e1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f0600ff9700 0 client.7518 ms_handle_reset on v2:172.21.15.50:6833/3096035349 2023-12-19T14:55:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 --2- 172.21.15.50:0/414826789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1100510 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 >> 172.21.15.50:0/414826789 conn(0xe7eeb0 msgr2=0x10b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 shutdown_connections 2023-12-19T14:55:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 1 -- 172.21.15.50:0/414826789 wait complete. 2023-12-19T14:55:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.277+0000 7f062a699740 20 client.7518 trim_cache size 0 max 16384 2023-12-19T14:55:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 Processor -- start 2023-12-19T14:55:42.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 -- start start 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 0x10b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x10b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x10b3ea0 con 0xec9790 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1021b50 con 0xe7e710 2023-12-19T14:55:42.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x102c290 con 0xe7eae0 2023-12-19T14:55:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48896/0 (socket says 172.21.15.50:48896) 2023-12-19T14:55:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 -- 172.21.15.50:0/3725039909 learned_addr learned my addr 172.21.15.50:0/3725039909 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0625960700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x10b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626161700 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 0x10b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 -- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 msgr2=0x10b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x10b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 -- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 msgr2=0x10b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 0x10b8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626962700 1 -- 172.21.15.50:0/3725039909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xfdad80 con 0xec9790 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.301+0000 7f0626161700 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 0x10b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f0626962700 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0618007370 tx=0x7f0618009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66df95cb2649df72 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f0603fff700 1 -- 172.21.15.50:0/3725039909 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618007d70 con 0xec9790 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f0603fff700 1 -- 172.21.15.50:0/3725039909 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0618017470 con 0xec9790 2023-12-19T14:55:42.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f0603fff700 1 -- 172.21.15.50:0/3725039909 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618017830 con 0xec9790 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 -- 172.21.15.50:0/3725039909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 msgr2=0x10b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0618007370 tx=0x7f0618009850 comp rx=0 tx=0).stop 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 -- 172.21.15.50:0/3725039909 shutdown_connections 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xec9790 0x10b14c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x10b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 --2- 172.21.15.50:0/3725039909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xe7e710 0x10b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 -- 172.21.15.50:0/3725039909 >> 172.21.15.50:0/3725039909 conn(0xe7eeb0 msgr2=0x10fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 -- 172.21.15.50:0/3725039909 shutdown_connections 2023-12-19T14:55:42.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.305+0000 7f062a699740 1 -- 172.21.15.50:0/3725039909 wait complete. 2023-12-19T14:55:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 -- start start 2023-12-19T14:55:42.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 Processor -- start 2023-12-19T14:55:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1155880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x1155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1023890 con 0xe7e710 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0xff9040 con 0xec9790 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0xfd7550 con 0xe7eae0 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f062a699740 10 client.0 _ll_register_callbacks cb 0xe4a9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0626962700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0625960700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x1155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0626962700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42648/0 (socket says 172.21.15.50:42648) 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0626161700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48900/0 (socket says 172.21.15.50:48900) 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.309+0000 7f0626962700 1 -- 172.21.15.50:0/730284035 learned_addr learned my addr 172.21.15.50:0/730284035 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 -- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 msgr2=0x1155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x1155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 -- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 msgr2=0x1155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 -- 172.21.15.50:0/730284035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xff4370 con 0xec9790 2023-12-19T14:55:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626962700 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0xfd2750 tx=0x7f061801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=179572a0cadbdf62 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-12-19T14:55:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618001040 con 0xec9790 2023-12-19T14:55:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mon.1 v2:172.21.15.125:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f06180177f0 con 0xec9790 2023-12-19T14:55:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f061c0598e0 con 0xec9790 2023-12-19T14:55:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0618020660 con 0xec9790 2023-12-19T14:55:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 10 client.7408 fetch_fsmap learned FSMap version 50 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x7f0618011040 con 0xec9790 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mon.1 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c0598e0 con 0xec9790 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mon.1 v2:172.21.15.125:3300/0 5 ==== fsmap(e 50) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f06180207c0 con 0xec9790 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 10 client.7408 fetch_fsmap finished waiting for FSMap version 50 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 10 client.7408 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 20 client.7408 populate_metadata read hostname 'smithi050' 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x115c710 0x115eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 4 client.7408 mds_command: new command op to 7377 tid=0[{"prefix": "status"}] 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 --> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] -- command(tid 0: {"prefix": "status"}) v1 -- 0xffa410 con 0x115c710 2023-12-19T14:55:42.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.313+0000 7f0626161700 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x115c710 0x115eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f0600ff9700 10 client.7408 ms_handle_connect on v2:172.21.15.50:6833/3096035349 2023-12-19T14:55:42.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f0626161700 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x115c710 0x115eaf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f27a6678b9e1625 server_cookie=3cc5d81d4ec852aa in_seq=0 out_seq=0 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f0600ff9700 1 -- 172.21.15.50:0/730284035 <== mds.0 v2:172.21.15.50:6833/3096035349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0xffa410 con 0x115c710 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f0600ff9700 10 client.7408 handle_command_reply: tid=0 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 client.7408 shutdown 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 msgr2=0x10b54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0xfd2750 tx=0x7f061801d000 comp rx=0 tx=0).stop 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 shutdown_connections 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:6833/3096035349,v1:172.21.15.50:6835/3096035349] conn(0x115c710 0x115eaf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0xec9790 0x10b54c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0xe7eae0 0x1155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 --2- 172.21.15.50:0/730284035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0xe7e710 0x1155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f0600ff9700 0 client.7408 ms_handle_reset on v2:172.21.15.50:6833/3096035349 2023-12-19T14:55:42.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 >> 172.21.15.50:0/730284035 conn(0xe7eeb0 msgr2=0x10ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 shutdown_connections 2023-12-19T14:55:42.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 1 -- 172.21.15.50:0/730284035 wait complete. 2023-12-19T14:55:42.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f062a699740 20 client.7408 trim_cache size 0 max 16384 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 7377, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.66719259, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 50, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 113, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 113, 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.543296974 2023-12-19T14:55:42.322 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:55:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 msgr2=0x7f0604062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 0x7f0604062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f06200b4860 tx=0x7f0610008040 comp rx=0 tx=0).stop 2023-12-19T14:55:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 msgr2=0x7f06201502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0620069d10 tx=0x7f061c017000 comp rx=0 tx=0).stop 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 shutdown_connections 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06200b41e0 0x7f0620154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0604060670 0x7f0604062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0620158ff0 0x7f0620154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 --2- 172.21.15.50:0/1793407323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06200b3dd0 0x7f06201502e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 >> 172.21.15.50:0/1793407323 conn(0x7f06200670e0 msgr2=0x7f0620149530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 shutdown_connections 2023-12-19T14:55:42.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.317+0000 7f06283c5700 1 -- 172.21.15.50:0/1793407323 wait complete. 2023-12-19T14:55:42.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:55:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 Processor -- start 2023-12-19T14:55:42.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 -- start start 2023-12-19T14:55:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93ec0c6900 0x7f93ec0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 0x7f93ec06da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec06b470 con 0x7f93ec15cc20 2023-12-19T14:55:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec15d7b0 con 0x7f93ec0c6900 2023-12-19T14:55:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec161d10 con 0x7f93ec06b900 2023-12-19T14:55:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2122700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f93ec15cc20 0x7f93ec15d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44986/0 (socket says 172.21.15.50:44986) 2023-12-19T14:55:42.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f2122700 1 -- 172.21.15.50:0/1960730347 learned_addr learned my addr 172.21.15.50:0/1960730347 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.517+0000 7f93f1120700 1 --2- 172.21.15.50:0/1960730347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 0x7f93ec06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1921700 1 --2- 172.21.15.50:0/1960730347 >> v2:172.21.15.125:3300/0 conn(0x7f93ec0c6900 0x7f93ec0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f091f700 1 -- 172.21.15.50:0/1960730347 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228624185 0 0) 0x7f93ec06b470 con 0x7f93ec15cc20 2023-12-19T14:55:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f091f700 1 -- 172.21.15.50:0/1960730347 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d8003570 con 0x7f93ec15cc20 2023-12-19T14:55:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1120700 1 -- 172.21.15.50:0/1960730347 >> v2:172.21.15.125:3300/0 conn(0x7f93ec0c6900 msgr2=0x7f93ec0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1120700 1 --2- 172.21.15.50:0/1960730347 >> v2:172.21.15.125:3300/0 conn(0x7f93ec0c6900 0x7f93ec0c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1120700 1 -- 172.21.15.50:0/1960730347 >> v1:172.21.15.50:6789/0 conn(0x7f93ec15cc20 legacy=0x7f93ec15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1120700 1 -- 172.21.15.50:0/1960730347 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec06bd70 con 0x7f93ec06b900 2023-12-19T14:55:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f1120700 1 --2- 172.21.15.50:0/1960730347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 0x7f93ec06da00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009910 tx=0x7f93dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df54af30e425336 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f091f700 1 -- 172.21.15.50:0/1960730347 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc011630 con 0x7f93ec06b900 2023-12-19T14:55:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f091f700 1 -- 172.21.15.50:0/1960730347 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc011790 con 0x7f93ec06b900 2023-12-19T14:55:42.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f091f700 1 -- 172.21.15.50:0/1960730347 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc011b30 con 0x7f93ec06b900 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 -- 172.21.15.50:0/1960730347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 msgr2=0x7f93ec06da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 --2- 172.21.15.50:0/1960730347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 0x7f93ec06da00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009910 tx=0x7f93dc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 -- 172.21.15.50:0/1960730347 shutdown_connections 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 --2- 172.21.15.50:0/1960730347 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec06b900 0x7f93ec06da00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 --2- 172.21.15.50:0/1960730347 >> v2:172.21.15.125:3300/0 conn(0x7f93ec0c6900 0x7f93ec0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 -- 172.21.15.50:0/1960730347 >> 172.21.15.50:0/1960730347 conn(0x7f93ec067bb0 msgr2=0x7f93ec0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 -- 172.21.15.50:0/1960730347 shutdown_connections 2023-12-19T14:55:42.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.521+0000 7f93f2923700 1 -- 172.21.15.50:0/1960730347 wait complete. 2023-12-19T14:55:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 Processor -- start 2023-12-19T14:55:42.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 -- start start 2023-12-19T14:55:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 0x7f93ec159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec15cc20 0x7f93ec0ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b0b50 con 0x7f93ec0c6900 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0aff40 con 0x7f93ec06b900 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec0b0220 con 0x7f93ec15cc20 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f1120700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f1120700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48906/0 (socket says 172.21.15.50:48906) 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f1120700 1 -- 172.21.15.50:0/2729141972 learned_addr learned my addr 172.21.15.50:0/2729141972 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f1921700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 0x7f93ec159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.525+0000 7f93f2122700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec15cc20 0x7f93ec0ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 -- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec15cc20 msgr2=0x7f93ec0ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec15cc20 0x7f93ec0ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 -- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 msgr2=0x7f93ec159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 0x7f93ec159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 -- 172.21.15.50:0/2729141972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc009420 con 0x7f93ec0c6900 2023-12-19T14:55:42.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1921700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 0x7f93ec159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:42.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f1120700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f93ec161e50 tx=0x7f93dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc011b10 con 0x7f93ec0c6900 2023-12-19T14:55:42.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc025400 con 0x7f93ec0c6900 2023-12-19T14:55:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc025780 con 0x7f93ec0c6900 2023-12-19T14:55:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec0b0500 con 0x7f93ec0c6900 2023-12-19T14:55:42.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec15a360 con 0x7f93ec0c6900 2023-12-19T14:55:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.529+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04be40 con 0x7f93ec0c6900 2023-12-19T14:55:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.533+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f93dc02e430 con 0x7f93ec0c6900 2023-12-19T14:55:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.533+0000 7f93e1ffb700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 0x7f93d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.533+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f93dc07e340 con 0x7f93ec0c6900 2023-12-19T14:55:42.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.537+0000 7f93f1921700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 0x7f93d0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:42.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.537+0000 7f93f1921700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 0x7f93d0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009d80 tx=0x7f93e8008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:42.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.541+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f93dc04e6c0 con 0x7f93ec0c6900 2023-12-19T14:55:42.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.753+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93ec06a580 con 0x7f93ec0c6900 2023-12-19T14:55:42.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.753+0000 7f93e1ffb700 1 -- 172.21.15.50:0/2729141972 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7f93dc051ce0 con 0x7f93ec0c6900 2023-12-19T14:55:42.756 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 msgr2=0x7f93d0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 0x7f93d0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009d80 tx=0x7f93e8008040 comp rx=0 tx=0).stop 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 msgr2=0x7f93ec0ad480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f93ec161e50 tx=0x7f93dc017000 comp rx=0 tx=0).stop 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 shutdown_connections 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f93d0060670 0x7f93d0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f93ec15cc20 0x7f93ec0ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f93ec0c6900 0x7f93ec0ad480 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 --2- 172.21.15.50:0/2729141972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec06b900 0x7f93ec159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 >> 172.21.15.50:0/2729141972 conn(0x7f93ec067bb0 msgr2=0x7f93ec15baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 shutdown_connections 2023-12-19T14:55:42.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:42.757+0000 7f93f2923700 1 -- 172.21.15.50:0/2729141972 wait complete. 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:55:42.784 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:42.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:42.785 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:55:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41407sec preferred_lft 41407sec 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.806 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:55:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:55:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:55:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:55:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:55:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:55:42.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:42.809 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:55:42.887 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:55:42.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:42.891 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:55:42.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:55:42.957 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:55:43.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:43.099 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:55:43.120 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:55:43.120 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:55:43.120 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:43.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:55:43.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:55:43.201 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:55:43.201 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:55:43.201 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:43.260 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:55:43.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:55:43.260 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:43.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:55:43.402 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:55:43.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:55:43.484 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:55:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:55:43.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:43.487 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:55:43.553 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:55:43.554 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:55:43.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:55:43.636 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:55:43.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:55:43.707 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:55:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:55:43.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:43.710 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:55:43.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:55:43.745+0000 7f9aa02db380 -1 init, newargv = 0x563aeb8dd380 newargc=15 2023-12-19T14:55:43.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[51041]: starting ceph client 2023-12-19T14:55:43.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[51041]: starting fuse 2023-12-19T14:55:43.763 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:55:43.764 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:55:43.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:55:43.764 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:55:43.764 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:55:43.764 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:55:43.765 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:55:43.766 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:55:43.929 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.51041.asok 2023-12-19T14:55:44.114 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51041.asok 2023-12-19T14:55:44.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:44.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51041.asok status 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:55:44.303 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "51041", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 7539, 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 7539 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:55:44.304 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2531494806 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2531494806 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.7539 192.168.0.1:0/2531494806", 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/2531494806", 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 50, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 113, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:55:44.305 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:55:44.306 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:55:44.306 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:55:44.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:44.317 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:55:44.317 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:44.334 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:55:44.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:44.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:55:44.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:55:44.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:55:44.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.617+0000 7f0c07ef0700 1 Processor -- start 2023-12-19T14:55:44.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.617+0000 7f0c07ef0700 1 -- start start 2023-12-19T14:55:44.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c07ef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0c00156ad0 0x7f0c000cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c07ef0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c00156ea0 0x7f0c000cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c07ef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000b3ca0 con 0x7f0c000b37a0 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c07ef0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000b25f0 con 0x7f0c00156ea0 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c07ef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0006d300 con 0x7f0c00156ad0 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c00156ea0 0x7f0c000cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c00156ea0 0x7f0c000cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48912/0 (socket says 172.21.15.50:48912) 2023-12-19T14:55:44.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 -- 172.21.15.50:0/2757573009 learned_addr learned my addr 172.21.15.50:0/2757573009 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c05c8c700 1 --2- 172.21.15.50:0/2757573009 >> v2:172.21.15.125:3300/0 conn(0x7f0c00156ad0 0x7f0c000cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 -- 172.21.15.50:0/2757573009 >> v2:172.21.15.125:3300/0 conn(0x7f0c00156ad0 msgr2=0x7f0c000cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 --2- 172.21.15.50:0/2757573009 >> v2:172.21.15.125:3300/0 conn(0x7f0c00156ad0 0x7f0c000cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 -- 172.21.15.50:0/2757573009 >> v1:172.21.15.50:6789/0 conn(0x7f0c000b37a0 legacy=0x7f0c0015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 -- 172.21.15.50:0/2757573009 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0006d5e0 con 0x7f0c00156ea0 2023-12-19T14:55:44.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.621+0000 7f0c0548b700 1 --2- 172.21.15.50:0/2757573009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ea0 0x7f0c000cc720 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0c000ceeb0 tx=0x7f0bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e333530af2a5ec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:44.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c04c8a700 1 -- 172.21.15.50:0/2757573009 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000b860 con 0x7f0c00156ea0 2023-12-19T14:55:44.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c04c8a700 1 -- 172.21.15.50:0/2757573009 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf000b9c0 con 0x7f0c00156ea0 2023-12-19T14:55:44.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c04c8a700 1 -- 172.21.15.50:0/2757573009 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000bd60 con 0x7f0c00156ea0 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- 172.21.15.50:0/2757573009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ea0 msgr2=0x7f0c000cc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/2757573009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ea0 0x7f0c000cc720 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0c000ceeb0 tx=0x7f0bf0009250 comp rx=0 tx=0).stop 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- 172.21.15.50:0/2757573009 shutdown_connections 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/2757573009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ea0 0x7f0c000cc720 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/2757573009 >> v2:172.21.15.125:3300/0 conn(0x7f0c00156ad0 0x7f0c000cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- 172.21.15.50:0/2757573009 >> 172.21.15.50:0/2757573009 conn(0x7f0c000cdda0 msgr2=0x7f0c000ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- 172.21.15.50:0/2757573009 shutdown_connections 2023-12-19T14:55:44.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- 172.21.15.50:0/2757573009 wait complete. 2023-12-19T14:55:44.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 Processor -- start 2023-12-19T14:55:44.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.625+0000 7f0c07ef0700 1 -- start start 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000b37a0 0x7f0c0014aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c00156ea0 0x7f0c000ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0015a950 con 0x7f0c00156ad0 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0014b0e0 con 0x7f0c000b37a0 2023-12-19T14:55:44.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c07ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000a8b20 con 0x7f0c00156ea0 2023-12-19T14:55:44.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48922/0 (socket says 172.21.15.50:48922) 2023-12-19T14:55:44.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c05c8c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000b37a0 0x7f0c0014aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 -- 172.21.15.50:0/780175988 learned_addr learned my addr 172.21.15.50:0/780175988 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0648d700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c00156ea0 0x7f0c000ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 -- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c00156ea0 msgr2=0x7f0c000ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c00156ea0 0x7f0c000ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 -- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000b37a0 msgr2=0x7f0c0014aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000b37a0 0x7f0c0014aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 -- 172.21.15.50:0/780175988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00094f0 con 0x7f0c00156ad0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0c0548b700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0c0006f820 tx=0x7f0bf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.629+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0024440 con 0x7f0c00156ad0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.633+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00245a0 con 0x7f0c00156ad0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.633+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0024960 con 0x7f0c00156ad0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.633+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000a8e00 con 0x7f0c00156ad0 2023-12-19T14:55:44.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.633+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000a93c0 con 0x7f0c00156ad0 2023-12-19T14:55:44.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.633+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00071120 con 0x7f0c00156ad0 2023-12-19T14:55:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.637+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0bf001a420 con 0x7f0c00156ad0 2023-12-19T14:55:44.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.637+0000 7f0bf5ffb700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 0x7f0be4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.637+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0bf007e8c0 con 0x7f0c00156ad0 2023-12-19T14:55:44.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.637+0000 7f0c05c8c700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 0x7f0be4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:44.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.637+0000 7f0c05c8c700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 0x7f0be4062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc008040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:44.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.645+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0bf004eba0 con 0x7f0c00156ad0 2023-12-19T14:55:44.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.845+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c0006d8c0 con 0x7f0be40606a0 2023-12-19T14:55:44.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.849+0000 7f0bf5ffb700 1 -- 172.21.15.50:0/780175988 <== mgr.7057 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0c0006d8c0 con 0x7f0be40606a0 2023-12-19T14:55:44.852 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:55:44.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:55:44.853 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:55:44.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:55:44.853 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:55:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 msgr2=0x7f0be4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 0x7f0be4062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc008040 comp rx=0 tx=0).stop 2023-12-19T14:55:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 msgr2=0x7f0c000ab310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:44.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0c0006f820 tx=0x7f0bf0017000 comp rx=0 tx=0).stop 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 shutdown_connections 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0be40606a0 0x7f0be4062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c00156ea0 0x7f0c000ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c00156ad0 0x7f0c000ab310 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 --2- 172.21.15.50:0/780175988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000b37a0 0x7f0c0014aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 >> 172.21.15.50:0/780175988 conn(0x7f0c000cdda0 msgr2=0x7f0c000af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.853+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 shutdown_connections 2023-12-19T14:55:44.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:44.857+0000 7f0c07ef0700 1 -- 172.21.15.50:0/780175988 wait complete. 2023-12-19T14:55:44.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:55:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.041+0000 7f0f1c8a3700 1 Processor -- start 2023-12-19T14:55:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.041+0000 7f0f1c8a3700 1 -- start start 2023-12-19T14:55:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f1c8a3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f1806de00 0x7f0f180c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f1c8a3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f1806c7c0 0x7f0f1815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f1c8a3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1806d970 con 0x7f0f180c69a0 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f1c8a3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1806cd60 con 0x7f0f1806c7c0 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f1c8a3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1806d040 con 0x7f0f1806de00 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f17fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f1806c7c0 0x7f0f1815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f1806de00 0x7f0f180c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f1806de00 0x7f0f180c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42686/0 (socket says 172.21.15.50:42686) 2023-12-19T14:55:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 -- 172.21.15.50:0/1202751772 learned_addr learned my addr 172.21.15.50:0/1202751772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 -- 172.21.15.50:0/1202751772 >> v2:172.21.15.50:3300/0 conn(0x7f0f1806c7c0 msgr2=0x7f0f1815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 --2- 172.21.15.50:0/1202751772 >> v2:172.21.15.50:3300/0 conn(0x7f0f1806c7c0 0x7f0f1815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 -- 172.21.15.50:0/1202751772 >> v1:172.21.15.125:6789/0 conn(0x7f0f180c69a0 legacy=0x7f0f1806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:45.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.045+0000 7f0f16ffd700 1 -- 172.21.15.50:0/1202751772 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1806d320 con 0x7f0f1806de00 2023-12-19T14:55:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f16ffd700 1 --2- 172.21.15.50:0/1202751772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806de00 0x7f0f180c6d70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0f180cb990 tx=0x7f0f0c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27e75eddee6db267 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f167fc700 1 -- 172.21.15.50:0/1202751772 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c009e70 con 0x7f0f1806de00 2023-12-19T14:55:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f167fc700 1 -- 172.21.15.50:0/1202751772 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c00b860 con 0x7f0f1806de00 2023-12-19T14:55:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f167fc700 1 -- 172.21.15.50:0/1202751772 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00bc00 con 0x7f0f1806de00 2023-12-19T14:55:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/1202751772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806de00 msgr2=0x7f0f180c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/1202751772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806de00 0x7f0f180c6d70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0f180cb990 tx=0x7f0f0c009250 comp rx=0 tx=0).stop 2023-12-19T14:55:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/1202751772 shutdown_connections 2023-12-19T14:55:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/1202751772 >> v2:172.21.15.50:3300/0 conn(0x7f0f1806c7c0 0x7f0f1815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/1202751772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806de00 0x7f0f180c6d70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/1202751772 >> 172.21.15.50:0/1202751772 conn(0x7f0f18067b50 msgr2=0x7f0f180cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/1202751772 shutdown_connections 2023-12-19T14:55:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.049+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/1202751772 wait complete. 2023-12-19T14:55:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 Processor -- start 2023-12-19T14:55:45.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 -- start start 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 0x7f0f180b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f1806de00 0x7f0f18149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815d860 con 0x7f0f180c69a0 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815db40 con 0x7f0f1806c7c0 2023-12-19T14:55:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f1c8a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815de20 con 0x7f0f1806de00 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f17fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48926/0 (socket says 172.21.15.50:48926) 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.053+0000 7f0f17fff700 1 -- 172.21.15.50:0/3107627508 learned_addr learned my addr 172.21.15.50:0/3107627508 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f16ffd700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f1806de00 0x7f0f18149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f177fe700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 0x7f0f180b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 -- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f1806de00 msgr2=0x7f0f18149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f1806de00 0x7f0f18149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 -- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 msgr2=0x7f0f180b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 0x7f0f180b1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 -- 172.21.15.50:0/3107627508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c0094f0 con 0x7f0f180c69a0 2023-12-19T14:55:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f177fe700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 0x7f0f180b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f17fff700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009e90 tx=0x7f0f1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000cc20 con 0x7f0f180c69a0 2023-12-19T14:55:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1000f040 con 0x7f0f180c69a0 2023-12-19T14:55:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10014480 con 0x7f0f180c69a0 2023-12-19T14:55:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f180ae950 con 0x7f0f180c69a0 2023-12-19T14:55:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f180b1680 con 0x7f0f180c69a0 2023-12-19T14:55:45.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.057+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f180aee50 con 0x7f0f180c69a0 2023-12-19T14:55:45.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.061+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 40) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0f10014830 con 0x7f0f180c69a0 2023-12-19T14:55:45.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.061+0000 7f0eff7fe700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 0x7f0f04062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:45.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.061+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f0f1007d680 con 0x7f0f180c69a0 2023-12-19T14:55:45.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.061+0000 7f0f177fe700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 0x7f0f04062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:45.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.065+0000 7f0f177fe700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 0x7f0f04062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08009040 comp rx=0 tx=0).ready entity=mgr.7057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:45.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.069+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f100512e0 con 0x7f0f180c69a0 2023-12-19T14:55:45.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:45.281+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0f1804be40 con 0x7f0f180c69a0 2023-12-19T14:55:46.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.037+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 41) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f0f1004dc20 con 0x7f0f180c69a0 2023-12-19T14:55:46.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.037+0000 7f0eff7fe700 1 -- 172.21.15.50:0/3107627508 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v41) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0f10051510 con 0x7f0f180c69a0 2023-12-19T14:55:46.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 msgr2=0x7f0f04062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 0x7f0f04062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08009040 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 msgr2=0x7f0f1815e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009e90 tx=0x7f0f1000b040 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 shutdown_connections 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f04060670 0x7f0f04062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f180c69a0 0x7f0f1815e5e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f1806de00 0x7f0f18149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 --2- 172.21.15.50:0/3107627508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f1806c7c0 0x7f0f180b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 >> 172.21.15.50:0/3107627508 conn(0x7f0f18067b50 msgr2=0x7f0f1815abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 shutdown_connections 2023-12-19T14:55:46.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.041+0000 7f0f1c8a3700 1 -- 172.21.15.50:0/3107627508 wait complete. 2023-12-19T14:55:46.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:55:46.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.217+0000 7ff80e46b700 1 Processor -- start 2023-12-19T14:55:46.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.217+0000 7ff80e46b700 1 -- start start 2023-12-19T14:55:46.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80e46b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff808158c90 0x7ff8080caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80e46b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808159770 0x7ff80806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80e46b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80806c850 con 0x7ff80806aeb0 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80e46b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80806b470 con 0x7ff808158c90 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80e46b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80806b750 con 0x7ff808159770 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff808158c90 0x7ff8080caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff808158c90 0x7ff8080caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48948/0 (socket says 172.21.15.50:48948) 2023-12-19T14:55:46.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 -- 172.21.15.50:0/1299539827 learned_addr learned my addr 172.21.15.50:0/1299539827 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80dc6a700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808159770 0x7ff80806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 -- 172.21.15.50:0/1299539827 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808159770 msgr2=0x7ff80806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808159770 0x7ff80806cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 -- 172.21.15.50:0/1299539827 >> v1:172.21.15.50:6789/0 conn(0x7ff80806aeb0 legacy=0x7ff808159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 -- 172.21.15.50:0/1299539827 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80806ba30 con 0x7ff808158c90 2023-12-19T14:55:46.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff80d469700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808158c90 0x7ff8080caf40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff8040099b0 tx=0x7ff804009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e7cef6e7ed95f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:46.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff7fffff700 1 -- 172.21.15.50:0/1299539827 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80400b860 con 0x7ff808158c90 2023-12-19T14:55:46.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff7fffff700 1 -- 172.21.15.50:0/1299539827 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80400b9c0 con 0x7ff808158c90 2023-12-19T14:55:46.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.221+0000 7ff7fffff700 1 -- 172.21.15.50:0/1299539827 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80400bd60 con 0x7ff808158c90 2023-12-19T14:55:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- 172.21.15.50:0/1299539827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808158c90 msgr2=0x7ff8080caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808158c90 0x7ff8080caf40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff8040099b0 tx=0x7ff804009250 comp rx=0 tx=0).stop 2023-12-19T14:55:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- 172.21.15.50:0/1299539827 shutdown_connections 2023-12-19T14:55:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808159770 0x7ff80806cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 --2- 172.21.15.50:0/1299539827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808158c90 0x7ff8080caf40 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- 172.21.15.50:0/1299539827 >> 172.21.15.50:0/1299539827 conn(0x7ff808067bb0 msgr2=0x7ff8080cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:46.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- 172.21.15.50:0/1299539827 shutdown_connections 2023-12-19T14:55:46.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- 172.21.15.50:0/1299539827 wait complete. 2023-12-19T14:55:46.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 Processor -- start 2023-12-19T14:55:46.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.225+0000 7ff80e46b700 1 -- start start 2023-12-19T14:55:46.229 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:46.221+0000 7fe0ff5cc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80806aeb0 0x7ff808155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808158c90 0x7ff8080caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808156e30 con 0x7ff808159770 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808154bd0 con 0x7ff80806aeb0 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff808154eb0 con 0x7ff808158c90 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48962/0 (socket says 172.21.15.50:48962) 2023-12-19T14:55:46.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 -- 172.21.15.50:0/326376303 learned_addr learned my addr 172.21.15.50:0/326376303 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80cc68700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808158c90 0x7ff8080caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80d469700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80806aeb0 0x7ff808155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 -- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808158c90 msgr2=0x7ff8080caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808158c90 0x7ff8080caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 -- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80806aeb0 msgr2=0x7ff808155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80806aeb0 0x7ff808155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 -- 172.21.15.50:0/326376303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8040094f0 con 0x7ff808159770 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80dc6a700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff800007f20 tx=0x7ff800009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000eb90 con 0x7ff808159770 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000ed10 con 0x7ff808159770 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800014670 con 0x7ff808159770 2023-12-19T14:55:46.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808155190 con 0x7ff808159770 2023-12-19T14:55:46.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808150290 con 0x7ff808159770 2023-12-19T14:55:46.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.229+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff808155610 con 0x7ff808159770 2023-12-19T14:55:46.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.233+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 41) v1 ==== 80080+0+0 (secure 0 0 0) 0x7ff8000149e0 con 0x7ff808159770 2023-12-19T14:55:46.236 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:46.233+0000 7f296a95a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:55:46.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.233+0000 7ff7fd7fa700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 0x7ff7ec062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.233+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7ff80007d150 con 0x7ff808159770 2023-12-19T14:55:46.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.233+0000 7ff80d469700 1 -- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 msgr2=0x7ff7ec062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:46.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.233+0000 7ff80d469700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 0x7ff7ec062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:46.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.241+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff80004d430 con 0x7ff808159770 2023-12-19T14:55:46.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.417+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff80804be70 con 0x7ff808159770 2023-12-19T14:55:46.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.425+0000 7ff7fd7fa700 1 -- 172.21.15.50:0/326376303 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff800050a50 con 0x7ff808159770 2023-12-19T14:55:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 msgr2=0x7ff7ec062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 0x7ff7ec062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 msgr2=0x7ff808156370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff800007f20 tx=0x7ff800009860 comp rx=0 tx=0).stop 2023-12-19T14:55:46.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 shutdown_connections 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7ec0606c0 0x7ff7ec062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff808159770 0x7ff808156370 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff808158c90 0x7ff8080caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 --2- 172.21.15.50:0/326376303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80806aeb0 0x7ff808155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 >> 172.21.15.50:0/326376303 conn(0x7ff808067bb0 msgr2=0x7ff8080aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 shutdown_connections 2023-12-19T14:55:46.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.429+0000 7ff80e46b700 1 -- 172.21.15.50:0/326376303 wait complete. 2023-12-19T14:55:46.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:55:46.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.621+0000 7f403b945700 1 Processor -- start 2023-12-19T14:55:46.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.621+0000 7f403b945700 1 -- start start 2023-12-19T14:55:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f403b945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 0x7f403406b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f403b945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340cbdf0 con 0x7f40340cb970 2023-12-19T14:55:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f403b945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340cc0d0 con 0x7f4034158e20 2023-12-19T14:55:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f403b945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cc3b0 con 0x7f40340cecd0 2023-12-19T14:55:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f4038ee0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f40340cb970 0x7f403406acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33296/0 (socket says 172.21.15.50:33296) 2023-12-19T14:55:46.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f4038ee0700 1 -- 172.21.15.50:0/1019580995 learned_addr learned my addr 172.21.15.50:0/1019580995 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:46.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f4039ee2700 1 --2- 172.21.15.50:0/1019580995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 0x7f403406b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720216571 0 0) 0x7f40340cbdf0 con 0x7f40340cb970 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003570 con 0x7f40340cb970 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389353108 0 0) 0x7f40340cc0d0 con 0x7f4034158e20 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40340cbdf0 con 0x7f4034158e20 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.625+0000 7f4039ee2700 1 -- 172.21.15.50:0/1019580995 >> v1:172.21.15.125:6789/0 conn(0x7f4034158e20 legacy=0x7f40340ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f4039ee2700 1 -- 172.21.15.50:0/1019580995 >> v1:172.21.15.50:6789/0 conn(0x7f40340cb970 legacy=0x7f403406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f4039ee2700 1 -- 172.21.15.50:0/1019580995 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340cc690 con 0x7f40340cecd0 2023-12-19T14:55:46.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f4039ee2700 1 --2- 172.21.15.50:0/1019580995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 0x7f403406b3d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f402c009ec0 tx=0x7f402c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae20c82f9becec08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:46.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c00e070 con 0x7f40340cecd0 2023-12-19T14:55:46.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c00bb40 con 0x7f40340cecd0 2023-12-19T14:55:46.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f402bfff700 1 -- 172.21.15.50:0/1019580995 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c012430 con 0x7f40340cecd0 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 -- 172.21.15.50:0/1019580995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 msgr2=0x7f403406b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 --2- 172.21.15.50:0/1019580995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 0x7f403406b3d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f402c009ec0 tx=0x7f402c009780 comp rx=0 tx=0).stop 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 -- 172.21.15.50:0/1019580995 shutdown_connections 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 --2- 172.21.15.50:0/1019580995 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40340cecd0 0x7f403406b3d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 -- 172.21.15.50:0/1019580995 >> 172.21.15.50:0/1019580995 conn(0x7f4034067bb0 msgr2=0x7f40340d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:46.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 -- 172.21.15.50:0/1019580995 shutdown_connections 2023-12-19T14:55:46.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.629+0000 7f403b945700 1 -- 172.21.15.50:0/1019580995 wait complete. 2023-12-19T14:55:46.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.633+0000 7f403b945700 1 Processor -- start 2023-12-19T14:55:46.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.633+0000 7f403b945700 1 -- start start 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 0x7f403414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f403414e980 0x7f40340c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403414fa00 con 0x7f4034158e20 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403414edf0 con 0x7f40340cecd0 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403414f0d0 con 0x7f403414e980 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38998/0 (socket says 172.21.15.50:38998) 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 -- 172.21.15.50:0/3120038098 learned_addr learned my addr 172.21.15.50:0/3120038098 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4039ee2700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f403414e980 0x7f40340c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f40396e1700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 0x7f403414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 -- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f403414e980 msgr2=0x7f40340c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f403414e980 0x7f40340c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 -- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 msgr2=0x7f403414fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 0x7f403414fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 -- 172.21.15.50:0/3120038098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009a20 con 0x7f4034158e20 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f40396e1700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 0x7f403414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:55:46.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f4038ee0700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f402400c790 tx=0x7f4024007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024005b30 con 0x7f4034158e20 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4024005c90 con 0x7f4034158e20 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403414f3b0 con 0x7f4034158e20 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024007300 con 0x7f4034158e20 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4034155d80 con 0x7f4034158e20 2023-12-19T14:55:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.637+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403414fb70 con 0x7f4034158e20 2023-12-19T14:55:46.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.641+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 41) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f402401b400 con 0x7f4034158e20 2023-12-19T14:55:46.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.641+0000 7f40297fa700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 0x7f4018062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:46.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.641+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f402407c630 con 0x7f4034158e20 2023-12-19T14:55:46.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.641+0000 7f40396e1700 1 -- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 msgr2=0x7f4018062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:46.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.641+0000 7f40396e1700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 0x7f4018062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:46.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.653+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4024050420 con 0x7f4034158e20 2023-12-19T14:55:46.785 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:46.777+0000 7fe0ff5cc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.825+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f403406a7a0 con 0x7f4034158e20 2023-12-19T14:55:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.833+0000 7f40297fa700 1 -- 172.21.15.50:0/3120038098 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f40240076f0 con 0x7f4034158e20 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 msgr2=0x7f4018062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 0x7f4018062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 msgr2=0x7f4034155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f402400c790 tx=0x7f4024007e40 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 shutdown_connections 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40180606f0 0x7f4018062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f403414e980 0x7f40340c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4034158e20 0x7f4034155840 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 --2- 172.21.15.50:0/3120038098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40340cecd0 0x7f403414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 >> 172.21.15.50:0/3120038098 conn(0x7f4034067bb0 msgr2=0x7f40340cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 shutdown_connections 2023-12-19T14:55:46.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f403b945700 1 -- 172.21.15.50:0/3120038098 wait complete. 2023-12-19T14:55:46.841 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:46.837+0000 7f296a95a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:55:46.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:46.853+0000 7fe0ff5cc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:46.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /bad-path 1M 2023-12-19T14:55:46.912 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:46.909+0000 7f296a95a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:55:46.999 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:46.993+0000 7fe0ff5cc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:47.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.025+0000 7fa1784cd700 1 Processor -- start 2023-12-19T14:55:47.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.025+0000 7fa1784cd700 1 -- start start 2023-12-19T14:55:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.029+0000 7fa1784cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa1700cb130 0x7fa1700cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.029+0000 7fa1784cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa17015c930 0x7fa17006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.029+0000 7fa1784cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700b0ac0 con 0x7fa17006a6a0 2023-12-19T14:55:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.029+0000 7fa1784cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006b030 con 0x7fa1700cb130 2023-12-19T14:55:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.029+0000 7fa1784cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006b210 con 0x7fa17015c930 2023-12-19T14:55:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa1700cb130 0x7fa1700cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa1700cb130 0x7fa1700cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39006/0 (socket says 172.21.15.50:39006) 2023-12-19T14:55:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 -- 172.21.15.50:0/818200040 learned_addr learned my addr 172.21.15.50:0/818200040 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176a6a700 1 --2- 172.21.15.50:0/818200040 >> v2:172.21.15.125:3300/0 conn(0x7fa17015c930 0x7fa17006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 -- 172.21.15.50:0/818200040 >> v2:172.21.15.125:3300/0 conn(0x7fa17015c930 msgr2=0x7fa17006aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 --2- 172.21.15.50:0/818200040 >> v2:172.21.15.125:3300/0 conn(0x7fa17015c930 0x7fa17006aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 -- 172.21.15.50:0/818200040 >> v1:172.21.15.50:6789/0 conn(0x7fa17006a6a0 legacy=0x7fa17015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 -- 172.21.15.50:0/818200040 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1700ca810 con 0x7fa1700cb130 2023-12-19T14:55:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.033+0000 7fa176269700 1 --2- 172.21.15.50:0/818200040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cb500 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa1700cfc60 tx=0x7fa16c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc79d21b71c9379 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.037+0000 7fa175267700 1 -- 172.21.15.50:0/818200040 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c009e70 con 0x7fa1700cb130 2023-12-19T14:55:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.037+0000 7fa175267700 1 -- 172.21.15.50:0/818200040 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c00b860 con 0x7fa1700cb130 2023-12-19T14:55:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.037+0000 7fa175267700 1 -- 172.21.15.50:0/818200040 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00bc00 con 0x7fa1700cb130 2023-12-19T14:55:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- 172.21.15.50:0/818200040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 msgr2=0x7fa1700cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 --2- 172.21.15.50:0/818200040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cb500 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa1700cfc60 tx=0x7fa16c009250 comp rx=0 tx=0).stop 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- 172.21.15.50:0/818200040 shutdown_connections 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 --2- 172.21.15.50:0/818200040 >> v2:172.21.15.125:3300/0 conn(0x7fa17015c930 0x7fa17006aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 --2- 172.21.15.50:0/818200040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cb500 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- 172.21.15.50:0/818200040 >> 172.21.15.50:0/818200040 conn(0x7fa170067bb0 msgr2=0x7fa1700cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- 172.21.15.50:0/818200040 shutdown_connections 2023-12-19T14:55:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- 172.21.15.50:0/818200040 wait complete. 2023-12-19T14:55:47.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 Processor -- start 2023-12-19T14:55:47.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.041+0000 7fa1784cd700 1 -- start start 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa17015c930 0x7fa1700cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701435b0 con 0x7fa1700cb130 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170146220 con 0x7fa17006a6a0 2023-12-19T14:55:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701466b0 con 0x7fa17015c930 2023-12-19T14:55:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa176269700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa176269700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:38486/0 (socket says 172.21.15.50:38486) 2023-12-19T14:55:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 learned_addr learned my addr 172.21.15.50:0/1079293137 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa176a6a700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa17015c930 0x7fa1700cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa17015c930 msgr2=0x7fa1700cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa17015c930 0x7fa1700cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 msgr2=0x7fa1700c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c0094f0 con 0x7fa1700cb130 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa175a68700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cefc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c490 tx=0x7fa160007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000ce40 con 0x7fa1700cb130 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160011590 con 0x7fa1700cb130 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa160011950 con 0x7fa1700cb130 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa170145610 con 0x7fa1700cb130 2023-12-19T14:55:47.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.045+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170145bd0 con 0x7fa1700cb130 2023-12-19T14:55:47.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.049+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1701463d0 con 0x7fa1700cb130 2023-12-19T14:55:47.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.049+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 41) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fa160011cc0 con 0x7fa1700cb130 2023-12-19T14:55:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.053+0000 7fa1667fc700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.053+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fa16007be40 con 0x7fa1700cb130 2023-12-19T14:55:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.053+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:47.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.053+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:55:47.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.061+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa1701463d0 con 0x7fa1700cb130 2023-12-19T14:55:47.067 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.065+0000 7f296a95a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:47.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.257+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:47.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.257+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:55:47.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.261+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa170146390 con 0x7fa1540606c0 2023-12-19T14:55:47.270 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:47.261+0000 7fe0ff5cc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:47.347 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.345+0000 7f296a95a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:55:47.350 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:47.341+0000 7fe0ff5cc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:47.426 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.421+0000 7f296a95a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:47.576 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:47.569+0000 7fe0ff5cc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:47.660 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:47.653+0000 7fe0ff5cc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.661+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:47.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:47.661+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:55:47.663 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.661+0000 7f296a95a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:55:47.747 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:47.741+0000 7fe0ff5cc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:47.750 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.745+0000 7f296a95a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:55:47.845 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:47.841+0000 7f296a95a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:55:48.160 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:48.153+0000 7fe0ff5cc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:48.277 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:48.273+0000 7f296a95a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:55:48.329 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:48.321+0000 7fe0ff5cc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:48.455 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:48.453+0000 7f296a95a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:55:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:48.465+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:48.465+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:55:48.697 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:48.689+0000 7fe0ff5cc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:48.846 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:48.841+0000 7f296a95a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:55:48.909 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:48.901+0000 7fe0ff5cc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:49.002 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:48.993+0000 7fe0ff5cc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:49.075 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.069+0000 7f296a95a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:55:49.075 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.069+0000 7fe0ff5cc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:49.153 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.145+0000 7fe0ff5cc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:49.172 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.169+0000 7f296a95a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:55:49.255 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.253+0000 7f296a95a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:55:49.337 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.333+0000 7f296a95a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:55:49.383 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.377+0000 7fe0ff5cc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:49.464 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.461+0000 7fe0ff5cc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:49.558 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.549+0000 7fe0ff5cc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:49.583 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.581+0000 7f296a95a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:55:49.670 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.665+0000 7f296a95a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:55:49.759 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.753+0000 7fe0ff5cc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:49.765 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.761+0000 7f296a95a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:55:49.852 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:49.849+0000 7fe0ff5cc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:49.972 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:49.969+0000 7f296a95a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:55:50.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:50.069+0000 7fa176269700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:55:50.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:50.069+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:55:50.087 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:50.085+0000 7f296a95a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:55:50.583 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.577+0000 7fe0ff5cc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:50.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:50.601+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 42) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fa16004c4f0 con 0x7fa1700cb130 2023-12-19T14:55:50.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:50.601+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:55:50.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:50.601+0000 7fa1667fc700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0bf49d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0bf49d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0bf49d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0bf49d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0bf49d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.773 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0c24a3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.774 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0c24a3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.774 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0c24a3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.774 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0c24a3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.774 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:55:50.765+0000 7fe0c24a3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:55:50.895 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:55:50.893+0000 7f296a95a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:55:51.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.609+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 43) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fa16007a150 con 0x7fa1700cb130 2023-12-19T14:55:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.609+0000 7fa1667fc700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.609+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa170146390 con 0x7fa1540606c0 2023-12-19T14:55:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.609+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.613+0000 7fa176269700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006e40 tx=0x7fa16c017000 comp rx=0 tx=0).ready entity=mgr.7447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:51.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.697+0000 7fa1667fc700 1 -- 172.21.15.50:0/1079293137 <== mgr.7447 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa170146390 con 0x7fa1540606c0 2023-12-19T14:55:51.702 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /bad-path 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 msgr2=0x7fa154062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006e40 tx=0x7fa16c017000 comp rx=0 tx=0).stop 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 msgr2=0x7fa1700cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cefc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c490 tx=0x7fa160007ec0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 shutdown_connections 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.701+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa1540606c0 0x7fa154062b70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa17015c930 0x7fa1700cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa1700cb130 0x7fa1700cefc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 --2- 172.21.15.50:0/1079293137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa17006a6a0 0x7fa1700c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 >> 172.21.15.50:0/1079293137 conn(0x7fa170067bb0 msgr2=0x7fa1701579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 shutdown_connections 2023-12-19T14:55:51.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.705+0000 7fa1784cd700 1 -- 172.21.15.50:0/1079293137 wait complete. 2023-12-19T14:55:51.732 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-12-19T14:55:51.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-12-19T14:55:51.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.909+0000 7f994f42e700 1 Processor -- start 2023-12-19T14:55:51.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.909+0000 7f994f42e700 1 -- start start 2023-12-19T14:55:51.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f99480cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994f42e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f994815c930 0x7f994806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994f42e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480b0ac0 con 0x7f994806a6a0 2023-12-19T14:55:51.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994f42e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994806b030 con 0x7f994815c930 2023-12-19T14:55:51.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994806b210 con 0x7f99480cb130 2023-12-19T14:55:51.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994c9c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f994806a6a0 0x7f994815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33324/0 (socket says 172.21.15.50:33324) 2023-12-19T14:55:51.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994c9c9700 1 -- 172.21.15.50:0/2260481304 learned_addr learned my addr 172.21.15.50:0/2260481304 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:55:51.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d1ca700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f99480cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> v2:172.21.15.125:3300/0 conn(0x7f994815c930 0x7f994806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322387461 0 0) 0x7f99480b0ac0 con 0x7f994806a6a0 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9934003570 con 0x7f994806a6a0 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 609511237 0 0) 0x7f9934003570 con 0x7f994806a6a0 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 msgr2=0x7f99480cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f99480cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 >> v2:172.21.15.125:3300/0 conn(0x7f994815c930 msgr2=0x7f994806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 --2- 172.21.15.50:0/2260481304 >> v2:172.21.15.125:3300/0 conn(0x7f994815c930 0x7f994806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480ca810 con 0x7f994806a6a0 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f9938001470 con 0x7f994806a6a0 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 >> v1:172.21.15.50:6789/0 conn(0x7f994806a6a0 legacy=0x7f994815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9934003a60 0x7f9934003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f9934004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f99340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994806b030 con 0x7f9934004d50 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994806b210 con 0x7f9934003a60 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993400ba60 con 0x7f99480cb130 2023-12-19T14:55:51.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> v2:172.21.15.125:3300/0 conn(0x7f994815c930 0x7f994806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:55:51.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f99340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994c9c9700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f9934004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d1ca700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9934003a60 0x7f9934003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 -- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 msgr2=0x7f9934004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:51.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99480cb130 0x7f9934004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 -- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9934003a60 msgr2=0x7f9934003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.913+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9934003a60 0x7f9934003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994d9cb700 1 -- 172.21.15.50:0/2260481304 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9934003570 con 0x7f9934004d50 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994d1ca700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9934003a60 0x7f9934003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994d9cb700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f99340051c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f994815cd30 tx=0x7f994000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7c80965bcae0f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000ebe0 con 0x7f9934004d50 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994000f040 con 0x7f9934004d50 2023-12-19T14:55:51.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f993ffff700 1 -- 172.21.15.50:0/2260481304 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940014480 con 0x7f9934004d50 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 -- 172.21.15.50:0/2260481304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 msgr2=0x7f99340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 --2- 172.21.15.50:0/2260481304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f99340051c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f994815cd30 tx=0x7f994000b040 comp rx=0 tx=0).stop 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994d9cb700 1 -- 172.21.15.50:0/2260481304 reap_dead start 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 -- 172.21.15.50:0/2260481304 shutdown_connections 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 -- 172.21.15.50:0/2260481304 >> 172.21.15.50:0/2260481304 conn(0x7f9948067bb0 msgr2=0x7f99480cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 -- 172.21.15.50:0/2260481304 shutdown_connections 2023-12-19T14:55:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.917+0000 7f994f42e700 1 -- 172.21.15.50:0/2260481304 wait complete. 2023-12-19T14:55:51.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 Processor -- start 2023-12-19T14:55:51.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 -- start start 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 0x7f99480b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994814caa0 0x7f99480cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994815cdf0 con 0x7f9934004d50 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994816cb80 con 0x7f994814caa0 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994f42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994816ce60 con 0x7f994816c610 2023-12-19T14:55:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994d1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994d1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39034/0 (socket says 172.21.15.50:39034) 2023-12-19T14:55:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994d1ca700 1 -- 172.21.15.50:0/3918228235 learned_addr learned my addr 172.21.15.50:0/3918228235 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:55:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994c9c9700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 0x7f99480b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.921+0000 7f994d9cb700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994814caa0 0x7f99480cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 -- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 msgr2=0x7f99480b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 0x7f99480b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 -- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994814caa0 msgr2=0x7f99480cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994814caa0 0x7f99480cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 -- 172.21.15.50:0/3918228235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994816d140 con 0x7f9934004d50 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994c9c9700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 0x7f99480b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994d1ca700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9944011960 tx=0x7f9944008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944011ea0 con 0x7f9934004d50 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994400f8e0 con 0x7f9934004d50 2023-12-19T14:55:51.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994816d420 con 0x7f9934004d50 2023-12-19T14:55:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400fc60 con 0x7f9934004d50 2023-12-19T14:55:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480b4e50 con 0x7f9934004d50 2023-12-19T14:55:51.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.925+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948071120 con 0x7f9934004d50 2023-12-19T14:55:51.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.929+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 43) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f9944015460 con 0x7f9934004d50 2023-12-19T14:55:51.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.929+0000 7f993d7fa700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 0x7f992c0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:55:51.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.929+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f994407d050 con 0x7f9934004d50 2023-12-19T14:55:51.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.929+0000 7f994c9c9700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 0x7f992c0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:55:51.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.929+0000 7f994c9c9700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 0x7f992c0628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99380015a0 tx=0x7f9938009040 comp rx=0 tx=0).ready entity=mgr.7447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:55:51.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:51.933+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f99440acbe0 con 0x7f9934004d50 2023-12-19T14:55:52.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.133+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f994815cf30 con 0x7f992c0603f0 2023-12-19T14:55:52.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.145+0000 7f993d7fa700 1 -- 172.21.15.50:0/3918228235 <== mgr.7447 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f9938007220 con 0x7f992c0603f0 2023-12-19T14:55:52.150 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:55:52.150 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:55:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-12-19T14:55:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 msgr2=0x7f992c0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 0x7f992c0628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f99380015a0 tx=0x7f9938009040 comp rx=0 tx=0).stop 2023-12-19T14:55:52.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 msgr2=0x7f994815b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:55:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f9944011960 tx=0x7f9944008040 comp rx=0 tx=0).stop 2023-12-19T14:55:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 shutdown_connections 2023-12-19T14:55:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f994814caa0 0x7f99480cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f992c0603f0 0x7f992c0628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:52.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f994816c610 0x7f99480b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 --2- 172.21.15.50:0/3918228235 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9934004d50 0x7f994815b250 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:55:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 >> 172.21.15.50:0/3918228235 conn(0x7f9948067bb0 msgr2=0x7f99480cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:55:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 shutdown_connections 2023-12-19T14:55:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:55:52.149+0000 7f994f42e700 1 -- 172.21.15.50:0/3918228235 wait complete. 2023-12-19T14:55:52.179 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:55:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-12-19T14:56:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:00.716+0000 7fe0b2483700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:00.716+0000 7fe0b2483700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:00.716+0000 7fe0b2483700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:00.716+0000 7fe0b2483700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:00.716+0000 7fe0b2483700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:07.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:56:07.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:56:37.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:56:37.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:56:52.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-12-19T14:56:52.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.359+0000 7f15b90f0700 1 Processor -- start 2023-12-19T14:56:52.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.359+0000 7f15b90f0700 1 -- start start 2023-12-19T14:56:52.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b90f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f15b4161da0 0x7f15b406a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b90f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b40c6560 con 0x7f15b415d720 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b90f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b40c70e0 con 0x7f15b415d350 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b90f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b40cc900 con 0x7f15b4161da0 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f15b4161da0 0x7f15b406a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b359e700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f15b415d720 0x7f15b40abc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50422/0 (socket says 172.21.15.50:50422) 2023-12-19T14:56:52.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b359e700 1 -- 172.21.15.50:0/1346329875 learned_addr learned my addr 172.21.15.50:0/1346329875 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b259c700 1 -- 172.21.15.50:0/1346329875 >> v1:172.21.15.125:6789/0 conn(0x7f15b415d350 legacy=0x7f15b40cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b259c700 1 -- 172.21.15.50:0/1346329875 >> v1:172.21.15.50:6789/0 conn(0x7f15b415d720 legacy=0x7f15b40abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b1d9b700 1 -- 172.21.15.50:0/1346329875 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543666125 0 0) 0x7f15b40c6560 con 0x7f15b415d720 2023-12-19T14:56:52.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b259c700 1 -- 172.21.15.50:0/1346329875 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b40c67b0 con 0x7f15b4161da0 2023-12-19T14:56:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b259c700 1 --2- 172.21.15.50:0/1346329875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b406a6a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f15a80098e0 tx=0x7f15a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d3fdcb1629970a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.363+0000 7f15b1d9b700 1 -- 172.21.15.50:0/1346329875 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009e80 con 0x7f15b4161da0 2023-12-19T14:56:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b1d9b700 1 -- 172.21.15.50:0/1346329875 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800b860 con 0x7f15b4161da0 2023-12-19T14:56:52.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b1d9b700 1 -- 172.21.15.50:0/1346329875 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800bbe0 con 0x7f15b4161da0 2023-12-19T14:56:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- 172.21.15.50:0/1346329875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 msgr2=0x7f15b406a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 --2- 172.21.15.50:0/1346329875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b406a6a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f15a80098e0 tx=0x7f15a80091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- 172.21.15.50:0/1346329875 shutdown_connections 2023-12-19T14:56:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 --2- 172.21.15.50:0/1346329875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b406a6a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- 172.21.15.50:0/1346329875 >> 172.21.15.50:0/1346329875 conn(0x7f15b4067bb0 msgr2=0x7f15b40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- 172.21.15.50:0/1346329875 shutdown_connections 2023-12-19T14:56:52.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- 172.21.15.50:0/1346329875 wait complete. 2023-12-19T14:56:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 Processor -- start 2023-12-19T14:56:52.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.367+0000 7f15b90f0700 1 -- start start 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15b415d350 0x7f15b41561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15b415d720 0x7f15b4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b41566e0 con 0x7f15b4161da0 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40b19c0 con 0x7f15b415d350 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b41583f0 con 0x7f15b415d720 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42504/0 (socket says 172.21.15.50:42504) 2023-12-19T14:56:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 -- 172.21.15.50:0/3010126331 learned_addr learned my addr 172.21.15.50:0/3010126331 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b259c700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15b415d720 0x7f15b4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b2d9d700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15b415d350 0x7f15b41561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 -- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15b415d720 msgr2=0x7f15b4157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15b415d720 0x7f15b4157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 -- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15b415d350 msgr2=0x7f15b41561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15b415d350 0x7f15b41561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 -- 172.21.15.50:0/3010126331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a8009420 con 0x7f15b4161da0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b359e700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00cc90 tx=0x7f15ac001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00cea0 con 0x7f15b4161da0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac00ab30 con 0x7f15b4161da0 2023-12-19T14:56:52.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0070f0 con 0x7f15b4161da0 2023-12-19T14:56:52.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b40b16e0 con 0x7f15b4161da0 2023-12-19T14:56:52.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b41576b0 con 0x7f15b4161da0 2023-12-19T14:56:52.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.371+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b415cf50 con 0x7f15b4161da0 2023-12-19T14:56:52.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.375+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 45) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f15ac0074a0 con 0x7f15b4161da0 2023-12-19T14:56:52.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.375+0000 7f159affd700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 0x7f15a0062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.375+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f15ac07d3e0 con 0x7f15b4161da0 2023-12-19T14:56:52.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.375+0000 7f15b2d9d700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 0x7f15a0062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.379+0000 7f15b2d9d700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 0x7f15a0062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15a4003860 tx=0x7f15a4009040 comp rx=0 tx=0).ready entity=mgr.7447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.383+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f15b415cf50 con 0x7f15b4161da0 2023-12-19T14:56:52.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.587+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15b415cf50 con 0x7f15a00607e0 2023-12-19T14:56:52.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.603+0000 7f159affd700 1 -- 172.21.15.50:0/3010126331 <== mgr.7447 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7f15b415cf50 con 0x7f15a00607e0 2023-12-19T14:56:52.605 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:52.605 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:55:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 msgr2=0x7f15a0062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 0x7f15a0062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15a4003860 tx=0x7f15a4009040 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 msgr2=0x7f15b40b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00cc90 tx=0x7f15ac001a20 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 shutdown_connections 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f15a00607e0 0x7f15a0062c90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15b4161da0 0x7f15b40b0c60 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15b415d720 0x7f15b4157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 --2- 172.21.15.50:0/3010126331 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15b415d350 0x7f15b41561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 >> 172.21.15.50:0/3010126331 conn(0x7f15b4067bb0 msgr2=0x7f15b40ac010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 shutdown_connections 2023-12-19T14:56:52.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.607+0000 7f15b90f0700 1 -- 172.21.15.50:0/3010126331 wait complete. 2023-12-19T14:56:52.630 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:55:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-12-19T14:56:52.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /bad-path 2023-12-19T14:56:52.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.799+0000 7ffafe225700 1 Processor -- start 2023-12-19T14:56:52.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.799+0000 7ffafe225700 1 -- start start 2023-12-19T14:56:52.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffafe225700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffaf815c490 0x7ffaf81596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffafe225700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80add70 con 0x7ffaf815c860 2023-12-19T14:56:52.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffafe225700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf8159fa0 con 0x7ffaf806d760 2023-12-19T14:56:52.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffafe225700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf815a280 con 0x7ffaf815c490 2023-12-19T14:56:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffaf6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffaf815c860 0x7ffaf806d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50430/0 (socket says 172.21.15.50:50430) 2023-12-19T14:56:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffaf6ffd700 1 -- 172.21.15.50:0/3277887082 learned_addr learned my addr 172.21.15.50:0/3277887082 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.803+0000 7ffaf77fe700 1 --2- 172.21.15.50:0/3277887082 >> v2:172.21.15.50:3300/0 conn(0x7ffaf815c490 0x7ffaf81596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf77fe700 1 -- 172.21.15.50:0/3277887082 >> v1:172.21.15.125:6789/0 conn(0x7ffaf806d760 legacy=0x7ffaf80ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf67fc700 1 -- 172.21.15.50:0/3277887082 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908469137 0 0) 0x7ffaf80add70 con 0x7ffaf815c860 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf77fe700 1 -- 172.21.15.50:0/3277887082 >> v1:172.21.15.50:6789/0 conn(0x7ffaf815c860 legacy=0x7ffaf806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf77fe700 1 -- 172.21.15.50:0/3277887082 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf815a560 con 0x7ffaf815c490 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf77fe700 1 --2- 172.21.15.50:0/3277887082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf815c490 0x7ffaf81596b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009e90 tx=0x7ffae8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e01b11d6a58476 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf67fc700 1 -- 172.21.15.50:0/3277887082 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800e070 con 0x7ffaf815c490 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf67fc700 1 -- 172.21.15.50:0/3277887082 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae800bbf0 con 0x7ffaf815c490 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffaf67fc700 1 -- 172.21.15.50:0/3277887082 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8012430 con 0x7ffaf815c490 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 -- 172.21.15.50:0/3277887082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf815c490 msgr2=0x7ffaf81596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 --2- 172.21.15.50:0/3277887082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf815c490 0x7ffaf81596b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009e90 tx=0x7ffae8009780 comp rx=0 tx=0).stop 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 -- 172.21.15.50:0/3277887082 shutdown_connections 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 --2- 172.21.15.50:0/3277887082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf815c490 0x7ffaf81596b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 -- 172.21.15.50:0/3277887082 >> 172.21.15.50:0/3277887082 conn(0x7ffaf80ccc20 msgr2=0x7ffaf80cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 -- 172.21.15.50:0/3277887082 shutdown_connections 2023-12-19T14:56:52.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.807+0000 7ffafe225700 1 -- 172.21.15.50:0/3277887082 wait complete. 2023-12-19T14:56:52.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 Processor -- start 2023-12-19T14:56:52.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 -- start start 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaf815c490 0x7ffaf8151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaf815c860 0x7ffaf8151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b2170 con 0x7ffaf806d760 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b1560 con 0x7ffaf815c860 2023-12-19T14:56:52.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.811+0000 7ffafe225700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b1840 con 0x7ffaf815c490 2023-12-19T14:56:52.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42528/0 (socket says 172.21.15.50:42528) 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 -- 172.21.15.50:0/1348351059 learned_addr learned my addr 172.21.15.50:0/1348351059 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf6ffd700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaf815c490 0x7ffaf8151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf7fff700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaf815c860 0x7ffaf8151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 -- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaf815c490 msgr2=0x7ffaf8151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaf815c490 0x7ffaf8151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 -- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaf815c860 msgr2=0x7ffaf8151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaf815c860 0x7ffaf8151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:52.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 -- 172.21.15.50:0/1348351059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009a20 con 0x7ffaf806d760 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffaf77fe700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ffae800b6c0 tx=0x7ffae8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8026480 con 0x7ffaf806d760 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae80265e0 con 0x7ffaf806d760 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80b1b20 con 0x7ffaf806d760 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80269a0 con 0x7ffaf806d760 2023-12-19T14:56:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.815+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80b1080 con 0x7ffaf806d760 2023-12-19T14:56:52.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.819+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf80b1330 con 0x7ffaf806d760 2023-12-19T14:56:52.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.819+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 45) v1 ==== 80194+0+0 (secure 0 0 0) 0x7ffae8026d10 con 0x7ffaf806d760 2023-12-19T14:56:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.819+0000 7ffadf7fe700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 0x7ffae4062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.819+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7ffae807efd0 con 0x7ffaf806d760 2023-12-19T14:56:52.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.823+0000 7ffaf6ffd700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 0x7ffae4062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:52.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.823+0000 7ffaf6ffd700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 0x7ffae4062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80ab9c0 tx=0x7ffaec007f30 comp rx=0 tx=0).ready entity=mgr.7447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:52.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:52.827+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffae804eef0 con 0x7ffaf806d760 2023-12-19T14:56:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.023+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaf80b22b0 con 0x7ffae40607e0 2023-12-19T14:56:53.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.119+0000 7ffadf7fe700 1 -- 172.21.15.50:0/1348351059 <== mgr.7447 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7ffaec006d20 con 0x7ffae40607e0 2023-12-19T14:56:53.121 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /bad-path 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 msgr2=0x7ffae4062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 0x7ffae4062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80ab9c0 tx=0x7ffaec007f30 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 msgr2=0x7ffaf80cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ffae800b6c0 tx=0x7ffae8019000 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 shutdown_connections 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffaf815c860 0x7ffaf8151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffae40607e0 0x7ffae4062c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffaf815c490 0x7ffaf8151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 --2- 172.21.15.50:0/1348351059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffaf806d760 0x7ffaf80cbf20 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 >> 172.21.15.50:0/1348351059 conn(0x7ffaf80ccc20 msgr2=0x7ffaf8158090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 shutdown_connections 2023-12-19T14:56:53.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.123+0000 7ffafe225700 1 -- 172.21.15.50:0/1348351059 wait complete. 2023-12-19T14:56:53.152 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-12-19T14:56:53.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:56:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 Processor -- start 2023-12-19T14:56:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 -- start start 2023-12-19T14:56:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 0x7f091c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c15ae30 con 0x7f091c06baa0 2023-12-19T14:56:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0abf70 con 0x7f091c0adab0 2023-12-19T14:56:53.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.319+0000 7f091b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c154b50 con 0x7f091c154780 2023-12-19T14:56:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f0919d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f091c06baa0 0x7f091c157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50440/0 (socket says 172.21.15.50:50440) 2023-12-19T14:56:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f0919d9b700 1 -- 172.21.15.50:0/4140836168 learned_addr learned my addr 172.21.15.50:0/4140836168 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091a59c700 1 --2- 172.21.15.50:0/4140836168 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 0x7f091c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790120265 0 0) 0x7f091c15ae30 con 0x7f091c06baa0 2023-12-19T14:56:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904003570 con 0x7f091c06baa0 2023-12-19T14:56:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392452564 0 0) 0x7f091c0abf70 con 0x7f091c0adab0 2023-12-19T14:56:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c15ae30 con 0x7f091c0adab0 2023-12-19T14:56:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091a59c700 1 -- 172.21.15.50:0/4140836168 >> v1:172.21.15.125:6789/0 conn(0x7f091c0adab0 legacy=0x7f091c0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091a59c700 1 -- 172.21.15.50:0/4140836168 >> v1:172.21.15.50:6789/0 conn(0x7f091c06baa0 legacy=0x7f091c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091a59c700 1 -- 172.21.15.50:0/4140836168 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0ac210 con 0x7f091c154780 2023-12-19T14:56:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091a59c700 1 --2- 172.21.15.50:0/4140836168 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 0x7f091c15b2c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f090c009910 tx=0x7f090c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b71bcf702d05a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c011630 con 0x7f091c154780 2023-12-19T14:56:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c011790 con 0x7f091c154780 2023-12-19T14:56:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.323+0000 7f091959a700 1 -- 172.21.15.50:0/4140836168 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c011b30 con 0x7f091c154780 2023-12-19T14:56:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 -- 172.21.15.50:0/4140836168 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 msgr2=0x7f091c15b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 --2- 172.21.15.50:0/4140836168 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 0x7f091c15b2c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f090c009910 tx=0x7f090c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 -- 172.21.15.50:0/4140836168 shutdown_connections 2023-12-19T14:56:53.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 --2- 172.21.15.50:0/4140836168 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c154780 0x7f091c15b2c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 -- 172.21.15.50:0/4140836168 >> 172.21.15.50:0/4140836168 conn(0x7f091c067b50 msgr2=0x7f091c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 -- 172.21.15.50:0/4140836168 shutdown_connections 2023-12-19T14:56:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.327+0000 7f091b59e700 1 -- 172.21.15.50:0/4140836168 wait complete. 2023-12-19T14:56:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 Processor -- start 2023-12-19T14:56:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 -- start start 2023-12-19T14:56:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c06baa0 0x7f091c168790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c0adab0 0x7f091c1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c1685d0 con 0x7f091c154780 2023-12-19T14:56:53.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0aee60 con 0x7f091c0adab0 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c168010 con 0x7f091c06baa0 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42542/0 (socket says 172.21.15.50:42542) 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 -- 172.21.15.50:0/1313552991 learned_addr learned my addr 172.21.15.50:0/1313552991 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f0919d9b700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c0adab0 0x7f091c1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:53.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091a59c700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c06baa0 0x7f091c168790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 -- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c06baa0 msgr2=0x7f091c168790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c06baa0 0x7f091c168790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 -- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c0adab0 msgr2=0x7f091c1513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c0adab0 0x7f091c1513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 -- 172.21.15.50:0/1313552991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c009420 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.331+0000 7f091ad9d700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f091400c860 tx=0x7f0914002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914002030 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09140021b0 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c1682f0 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914012630 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c151930 con 0x7f091c154780 2023-12-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.335+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c0ae950 con 0x7f091c154780 2023-12-19T14:56:53.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.339+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 45) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f0914012a20 con 0x7f091c154780 2023-12-19T14:56:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.339+0000 7f090a7fc700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 0x7f08fc062cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:53.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.339+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f091407cf00 con 0x7f091c154780 2023-12-19T14:56:53.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.339+0000 7f091a59c700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 0x7f08fc062cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:53.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.343+0000 7f091a59c700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 0x7f08fc062cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f091c0caf80 tx=0x7f090c017000 comp rx=0 tx=0).ready entity=mgr.7447 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:53.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.347+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f091c0ae950 con 0x7f091c154780 2023-12-19T14:56:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:53.559+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f091c0ae950 con 0x7f091c154780 2023-12-19T14:56:54.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.075+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f091404d190 con 0x7f091c154780 2023-12-19T14:56:54.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.075+0000 7f090a7fc700 1 -- 172.21.15.50:0/1313552991 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v46) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0914047c60 con 0x7f091c154780 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 msgr2=0x7f08fc062cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 0x7f08fc062cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f091c0caf80 tx=0x7f090c017000 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 msgr2=0x7f091c167570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f091400c860 tx=0x7f0914002700 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 shutdown_connections 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f091c0adab0 0x7f091c1513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08fc060810 0x7f08fc062cc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f091c154780 0x7f091c167570 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 --2- 172.21.15.50:0/1313552991 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f091c06baa0 0x7f091c168790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 >> 172.21.15.50:0/1313552991 conn(0x7f091c067b50 msgr2=0x7f091c153420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:54.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 shutdown_connections 2023-12-19T14:56:54.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.079+0000 7f091b59e700 1 -- 172.21.15.50:0/1313552991 wait complete. 2023-12-19T14:56:54.104 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:56:54.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:56:54.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:56:54.180 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:56:54.180 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:56:54.180 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:56:54.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:56:54.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[51041]: fuse finished with error 0 and tester_r 0 2023-12-19T14:56:54.223 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:56:54.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:56:54.223 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:56:54.274 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:54.267+0000 7f7645d62000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:56:54.284 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:54.283+0000 7fe6b6f82000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:56:54.303 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:56:54.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:56:54.303 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:56:54.323 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:56:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:56:54.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:56:54.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:56:54.325 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:56:54.361 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:56:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:56:54.362 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:56:54.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:56:54.363 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:56:54.385 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:56:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:56:54.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:56:54.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 Processor -- start 2023-12-19T14:56:54.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 -- start start 2023-12-19T14:56:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 0x7fdea406cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea415d770 con 0x7fdea4156b20 2023-12-19T14:56:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea4156ef0 con 0x7fdea40b0ce0 2023-12-19T14:56:54.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdeaa678700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea415cb60 con 0x7fdea40b5650 2023-12-19T14:56:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdea37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdea8c15700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 0x7fdea406cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdea8c15700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 0x7fdea406cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48096/0 (socket says 172.21.15.50:48096) 2023-12-19T14:56:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdea8c15700 1 -- 172.21.15.50:0/3891326784 learned_addr learned my addr 172.21.15.50:0/3891326784 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.571+0000 7fdea37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:59016/0 (socket says 172.21.15.50:59016) 2023-12-19T14:56:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea37fe700 1 -- 172.21.15.50:0/3891326784 >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 msgr2=0x7fdea406cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea37fe700 1 --2- 172.21.15.50:0/3891326784 >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 0x7fdea406cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea37fe700 1 -- 172.21.15.50:0/3891326784 >> v1:172.21.15.125:6789/0 conn(0x7fdea4156b20 legacy=0x7fdea40b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea37fe700 1 -- 172.21.15.50:0/3891326784 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea415ce40 con 0x7fdea40b5650 2023-12-19T14:56:54.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea37fe700 1 --2- 172.21.15.50:0/3891326784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0098e0 tx=0x7fde8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=777a93f8c328f3c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:54.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea2ffd700 1 -- 172.21.15.50:0/3891326784 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c00b640 con 0x7fdea40b5650 2023-12-19T14:56:54.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea2ffd700 1 -- 172.21.15.50:0/3891326784 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c00b7a0 con 0x7fdea40b5650 2023-12-19T14:56:54.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdea2ffd700 1 -- 172.21.15.50:0/3891326784 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c00bb40 con 0x7fdea40b5650 2023-12-19T14:56:54.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 -- 172.21.15.50:0/3891326784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 msgr2=0x7fdea40b07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 --2- 172.21.15.50:0/3891326784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0098e0 tx=0x7fde8c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 -- 172.21.15.50:0/3891326784 shutdown_connections 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 --2- 172.21.15.50:0/3891326784 >> v2:172.21.15.125:3300/0 conn(0x7fdea40b0ce0 0x7fdea406cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 --2- 172.21.15.50:0/3891326784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea40b07a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 -- 172.21.15.50:0/3891326784 >> 172.21.15.50:0/3891326784 conn(0x7fdea40670e0 msgr2=0x7fdea40674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 -- 172.21.15.50:0/3891326784 shutdown_connections 2023-12-19T14:56:54.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.575+0000 7fdeaa678700 1 -- 172.21.15.50:0/3891326784 wait complete. 2023-12-19T14:56:54.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 Processor -- start 2023-12-19T14:56:54.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 -- start start 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 0x7fdea40cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 0x7fdea40b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea40ccfa0 con 0x7fdea40b0ce0 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea40cd280 con 0x7fdea4156b20 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdeaa678700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea40cd560 con 0x7fdea40b5650 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 0x7fdea40cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:54.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 0x7fdea40cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42566/0 (socket says 172.21.15.50:42566) 2023-12-19T14:56:54.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea3fff700 1 -- 172.21.15.50:0/2889739668 learned_addr learned my addr 172.21.15.50:0/2889739668 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:54.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 0x7fdea40b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:54.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea37fe700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:54.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 msgr2=0x7fdea414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea414dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 msgr2=0x7fdea40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 0x7fdea40cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 -- 172.21.15.50:0/2889739668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c009420 con 0x7fdea4156b20 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea37fe700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:56:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.579+0000 7fdea8c15700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 0x7fdea40b2e30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00ecc0 tx=0x7fde9c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00c6b0 con 0x7fdea4156b20 2023-12-19T14:56:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c00f040 con 0x7fdea4156b20 2023-12-19T14:56:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea40cd840 con 0x7fdea4156b20 2023-12-19T14:56:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00ca60 con 0x7fdea4156b20 2023-12-19T14:56:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea414e2c0 con 0x7fdea4156b20 2023-12-19T14:56:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea4071120 con 0x7fdea4156b20 2023-12-19T14:56:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.583+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fde9c014440 con 0x7fdea4156b20 2023-12-19T14:56:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.587+0000 7fde9bfff700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 0x7fde84062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.587+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fde9c07d570 con 0x7fdea4156b20 2023-12-19T14:56:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.587+0000 7fdea3fff700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 msgr2=0x7fde84062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.587+0000 7fdea3fff700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 0x7fde84062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:54.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.595+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde9c04d320 con 0x7fdea4156b20 2023-12-19T14:56:54.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.791+0000 7fdea3fff700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 msgr2=0x7fde84062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:54.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.791+0000 7fdea3fff700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 0x7fde84062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:56:54.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.807+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdea40b0500 con 0x7fdea4156b20 2023-12-19T14:56:54.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.811+0000 7fde9bfff700 1 -- 172.21.15.50:0/2889739668 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fde9c050940 con 0x7fdea4156b20 2023-12-19T14:56:54.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:54.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:56:54.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 msgr2=0x7fde84062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 0x7fde84062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 msgr2=0x7fdea40b2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 0x7fdea40b2e30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00ecc0 tx=0x7fde9c00b040 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 shutdown_connections 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdea4156b20 0x7fdea40b2e30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde84060790 0x7fde84062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdea40b5650 0x7fdea414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 --2- 172.21.15.50:0/2889739668 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdea40b0ce0 0x7fdea40cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 >> 172.21.15.50:0/2889739668 conn(0x7fdea40670e0 msgr2=0x7fdea4150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 shutdown_connections 2023-12-19T14:56:54.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:54.815+0000 7fdeaa678700 1 -- 172.21.15.50:0/2889739668 wait complete. 2023-12-19T14:56:54.817 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:56:54.829 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:54.823+0000 7f7645d62000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:56:54.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:56:54.867 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:54.863+0000 7fe6b6f82000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:56:54.906 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:54.899+0000 7f7645d62000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:56:54.952 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:54.951+0000 7fe6b6f82000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:56:55.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 Processor -- start 2023-12-19T14:56:55.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 -- start start 2023-12-19T14:56:55.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1041502b0 0x7ff104068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff1040c5900 0x7ff1040c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1040c6630 con 0x7ff104067c90 2023-12-19T14:56:55.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1040c6910 con 0x7ff1040c5900 2023-12-19T14:56:55.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.011+0000 7ff10882e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1040c6bf0 con 0x7ff1041502b0 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff1040c5900 0x7ff1040c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff1040c5900 0x7ff1040c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42578/0 (socket says 172.21.15.50:42578) 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 -- 172.21.15.50:0/3960500735 learned_addr learned my addr 172.21.15.50:0/3960500735 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff0fbfff700 1 --2- 172.21.15.50:0/3960500735 >> v2:172.21.15.125:3300/0 conn(0x7ff1041502b0 0x7ff104068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 -- 172.21.15.50:0/3960500735 >> v2:172.21.15.125:3300/0 conn(0x7ff1041502b0 msgr2=0x7ff104068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 --2- 172.21.15.50:0/3960500735 >> v2:172.21.15.125:3300/0 conn(0x7ff1041502b0 0x7ff104068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 -- 172.21.15.50:0/3960500735 >> v1:172.21.15.125:6789/0 conn(0x7ff104067c90 legacy=0x7ff10414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:55.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 -- 172.21.15.50:0/3960500735 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1040c6ed0 con 0x7ff1040c5900 2023-12-19T14:56:55.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff102d9d700 1 --2- 172.21.15.50:0/3960500735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1040c5900 0x7ff1040c5d60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff1040c56b0 tx=0x7ff0fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57081d7fcf7266df server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff101d9b700 1 -- 172.21.15.50:0/3960500735 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00e070 con 0x7ff1040c5900 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff101d9b700 1 -- 172.21.15.50:0/3960500735 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc00ba10 con 0x7ff1040c5900 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff101d9b700 1 -- 172.21.15.50:0/3960500735 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0092e0 con 0x7ff1040c5900 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 -- 172.21.15.50:0/3960500735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1040c5900 msgr2=0x7ff1040c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 --2- 172.21.15.50:0/3960500735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1040c5900 0x7ff1040c5d60 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff1040c56b0 tx=0x7ff0fc009840 comp rx=0 tx=0).stop 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 -- 172.21.15.50:0/3960500735 shutdown_connections 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 --2- 172.21.15.50:0/3960500735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1040c5900 0x7ff1040c5d60 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 --2- 172.21.15.50:0/3960500735 >> v2:172.21.15.125:3300/0 conn(0x7ff1041502b0 0x7ff104068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 -- 172.21.15.50:0/3960500735 >> 172.21.15.50:0/3960500735 conn(0x7ff1040cd6d0 msgr2=0x7ff10406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 -- 172.21.15.50:0/3960500735 shutdown_connections 2023-12-19T14:56:55.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.015+0000 7ff10882e700 1 -- 172.21.15.50:0/3960500735 wait complete. 2023-12-19T14:56:55.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 Processor -- start 2023-12-19T14:56:55.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 -- start start 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff104067c90 0x7ff104150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 0x7ff1040b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff104151630 con 0x7ff1041502b0 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1041697b0 con 0x7ff1040c5900 2023-12-19T14:56:55.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10882e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff104152a20 con 0x7ff104067c90 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42588/0 (socket says 172.21.15.50:42588) 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 -- 172.21.15.50:0/1790262110 learned_addr learned my addr 172.21.15.50:0/1790262110 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff10259c700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff104067c90 0x7ff104150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff0fbfff700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 0x7ff1040b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 -- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff104067c90 msgr2=0x7ff104150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff104067c90 0x7ff104150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 -- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 msgr2=0x7ff1040b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 0x7ff1040b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 -- 172.21.15.50:0/1790262110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc009ae0 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff0fbfff700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 0x7ff1040b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.019+0000 7ff102d9d700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00bee0 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc0094e0 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff104150d90 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0032f0 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff104151350 con 0x7ff1041502b0 2023-12-19T14:56:55.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff104071120 con 0x7ff1041502b0 2023-12-19T14:56:55.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.023+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7ff0fc01d630 con 0x7ff1041502b0 2023-12-19T14:56:55.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.027+0000 7ff0fa7fc700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 0x7ff0e4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.027+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7ff0fc07e6d0 con 0x7ff1041502b0 2023-12-19T14:56:55.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.027+0000 7ff10259c700 1 -- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 msgr2=0x7ff0e4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:55.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.027+0000 7ff10259c700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 0x7ff0e4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:55.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.031+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff0fc04e550 con 0x7ff1041502b0 2023-12-19T14:56:55.055 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.047+0000 7f7645d62000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:56:55.106 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.103+0000 7fe6b6f82000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:56:55.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.211+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1040642a0 con 0x7ff1041502b0 2023-12-19T14:56:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.211+0000 7ff0fa7fc700 1 -- 172.21.15.50:0/1790262110 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16304 (secure 0 0 0) 0x7ff0fc051b70 con 0x7ff1041502b0 2023-12-19T14:56:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:55.215 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":116,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:56:52.034648+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.1a","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/738581743":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/1068162751":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3184659104":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3685761614":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2498915371":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2764835356":"2023-12-20T14:55:50.589350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:56:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 msgr2=0x7ff0e4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 0x7ff0e4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 msgr2=0x7ff10414f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc017000 comp rx=0 tx=0).stop 2023-12-19T14:56:55.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 shutdown_connections 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1040c5900 0x7ff1040b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff0e4060790 0x7ff0e4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1041502b0 0x7ff10414f110 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 --2- 172.21.15.50:0/1790262110 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff104067c90 0x7ff104150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 >> 172.21.15.50:0/1790262110 conn(0x7ff1040cd6d0 msgr2=0x7ff1040cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 shutdown_connections 2023-12-19T14:56:55.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.215+0000 7ff10882e700 1 -- 172.21.15.50:0/1790262110 wait complete. 2023-12-19T14:56:55.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:55:32.714035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:55:34.094095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:56:55.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:56:55.324 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.319+0000 7f7645d62000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:56:55.400 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.395+0000 7f7645d62000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:56:55.410 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.407+0000 7fe6b6f82000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:56:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.459+0000 7fd006363700 1 Processor -- start 2023-12-19T14:56:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.459+0000 7fd006363700 1 -- start start 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd006363700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd0000c9b50 0x7fd00006ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd006363700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0000c86b0 con 0x7fd000158f40 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd006363700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0000c8380 con 0x7fd000158b70 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd006363700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd000068880 con 0x7fd0000c9b50 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd0000c9b50 0x7fd00006ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd0000c9b50 0x7fd00006ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42604/0 (socket says 172.21.15.50:42604) 2023-12-19T14:56:55.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 -- 172.21.15.50:0/3532085905 learned_addr learned my addr 172.21.15.50:0/3532085905 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd004b60700 1 --1- 172.21.15.50:0/3532085905 >> v1:172.21.15.50:6789/0 conn(0x7fd000158f40 0x7fd00006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50448/0 (socket says 172.21.15.50:50448) 2023-12-19T14:56:55.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 -- 172.21.15.50:0/3532085905 >> v1:172.21.15.125:6789/0 conn(0x7fd000158b70 legacy=0x7fd0000cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 -- 172.21.15.50:0/3532085905 >> v1:172.21.15.50:6789/0 conn(0x7fd000158f40 legacy=0x7fd00006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 -- 172.21.15.50:0/3532085905 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd000068b60 con 0x7fd0000c9b50 2023-12-19T14:56:55.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fd005b62700 1 --2- 172.21.15.50:0/3532085905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd0000c9b50 0x7fd00006ced0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009f20 tx=0x7fcff8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd27080366658ed9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fcff7fff700 1 -- 172.21.15.50:0/3532085905 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800e070 con 0x7fd0000c9b50 2023-12-19T14:56:55.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fcff7fff700 1 -- 172.21.15.50:0/3532085905 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff800bb10 con 0x7fd0000c9b50 2023-12-19T14:56:55.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.463+0000 7fcff7fff700 1 -- 172.21.15.50:0/3532085905 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff80092e0 con 0x7fd0000c9b50 2023-12-19T14:56:55.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 -- 172.21.15.50:0/3532085905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd0000c9b50 msgr2=0x7fd00006ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 --2- 172.21.15.50:0/3532085905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd0000c9b50 0x7fd00006ced0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009f20 tx=0x7fcff8009840 comp rx=0 tx=0).stop 2023-12-19T14:56:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 -- 172.21.15.50:0/3532085905 shutdown_connections 2023-12-19T14:56:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 --2- 172.21.15.50:0/3532085905 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd0000c9b50 0x7fd00006ced0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 -- 172.21.15.50:0/3532085905 >> 172.21.15.50:0/3532085905 conn(0x7fd0000cdc00 msgr2=0x7fd0000ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 -- 172.21.15.50:0/3532085905 shutdown_connections 2023-12-19T14:56:55.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.467+0000 7fd006363700 1 -- 172.21.15.50:0/3532085905 wait complete. 2023-12-19T14:56:55.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 Processor -- start 2023-12-19T14:56:55.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 -- start start 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0000c9b50 0x7fd0000c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 0x7fd0000b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 0x7fd000153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c5290 con 0x7fd000158b70 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c5570 con 0x7fd000158f40 2023-12-19T14:56:55.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd006363700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c5850 con 0x7fd0000c9b50 2023-12-19T14:56:55.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 0x7fd0000b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd005b62700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 0x7fd000153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd005361700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0000c9b50 0x7fd0000c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd005b62700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 0x7fd000153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48132/0 (socket says 172.21.15.50:48132) 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd005b62700 1 -- 172.21.15.50:0/2102006975 learned_addr learned my addr 172.21.15.50:0/2102006975 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 -- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0000c9b50 msgr2=0x7fd0000c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0000c9b50 0x7fd0000c4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 -- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 msgr2=0x7fd000153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 0x7fd000153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.471+0000 7fd004b60700 1 -- 172.21.15.50:0/2102006975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8009ae0 con 0x7fd000158b70 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fd004b60700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 0x7fd0000b2050 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c590 tx=0x7fcff0007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0005a70 con 0x7fd000158b70 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0005bf0 con 0x7fd000158b70 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00070d0 con 0x7fd000158b70 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0000c5b30 con 0x7fd000158b70 2023-12-19T14:56:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd000168e30 con 0x7fd000158b70 2023-12-19T14:56:55.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.475+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00004be40 con 0x7fd000158b70 2023-12-19T14:56:55.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.479+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fcff0007480 con 0x7fd000158b70 2023-12-19T14:56:55.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.479+0000 7fcff57fa700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 0x7fcfe4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.479+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fcff007c260 con 0x7fd000158b70 2023-12-19T14:56:55.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.479+0000 7fd005361700 1 -- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 msgr2=0x7fcfe4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:55.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.479+0000 7fd005361700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 0x7fcfe4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:55.485 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.483+0000 7fe6b6f82000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:56:55.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.487+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcff004c1f0 con 0x7fd000158b70 2023-12-19T14:56:55.622 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.615+0000 7f7645d62000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:56:55.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.663+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd000068e40 con 0x7fd000158b70 2023-12-19T14:56:55.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.663+0000 7fcff57fa700 1 -- 172.21.15.50:0/2102006975 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcff004f810 con 0x7fd000158b70 2023-12-19T14:56:55.665 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 msgr2=0x7fcfe4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 0x7fcfe4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 msgr2=0x7fd0000b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 0x7fd0000b2050 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c590 tx=0x7fcff0007eb0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 shutdown_connections 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd000158f40 0x7fd000153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcfe4060790 0x7fcfe4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd000158b70 0x7fd0000b2050 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 --2- 172.21.15.50:0/2102006975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd0000c9b50 0x7fd0000c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 >> 172.21.15.50:0/2102006975 conn(0x7fd0000cdc00 msgr2=0x7fd0000c44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 shutdown_connections 2023-12-19T14:56:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.667+0000 7fd006363700 1 -- 172.21.15.50:0/2102006975 wait complete. 2023-12-19T14:56:55.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:56:55.702 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.699+0000 7f7645d62000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:56:55.727 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.727+0000 7fe6b6f82000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:56:55.796 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:55.791+0000 7f7645d62000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:56:55.815 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.815+0000 7fe6b6f82000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:56:55.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.859+0000 7f0e60345700 1 Processor -- start 2023-12-19T14:56:55.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 -- start start 2023-12-19T14:56:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e58158e20 0x7f0e580ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580cbdf0 con 0x7f0e580cb970 2023-12-19T14:56:55.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e580cc0d0 con 0x7f0e58158e20 2023-12-19T14:56:55.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e60345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580cc3b0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e5e8e2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e5e0e1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e58158e20 0x7f0e580ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e5e8e2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:59052/0 (socket says 172.21.15.50:59052) 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.863+0000 7f0e5e8e2700 1 -- 172.21.15.50:0/1468779504 learned_addr learned my addr 172.21.15.50:0/1468779504 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5e8e2700 1 -- 172.21.15.50:0/1468779504 >> v2:172.21.15.125:3300/0 conn(0x7f0e58158e20 msgr2=0x7f0e580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5e8e2700 1 --2- 172.21.15.50:0/1468779504 >> v2:172.21.15.125:3300/0 conn(0x7f0e58158e20 0x7f0e580ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5e8e2700 1 -- 172.21.15.50:0/1468779504 >> v1:172.21.15.125:6789/0 conn(0x7f0e580cb970 legacy=0x7f0e5806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5e8e2700 1 -- 172.21.15.50:0/1468779504 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580cc690 con 0x7f0e580ca2c0 2023-12-19T14:56:55.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5e8e2700 1 --2- 172.21.15.50:0/1468779504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009f50 tx=0x7f0e50009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6049ad39892573e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5d0df700 1 -- 172.21.15.50:0/1468779504 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5000e070 con 0x7f0e580ca2c0 2023-12-19T14:56:55.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5d0df700 1 -- 172.21.15.50:0/1468779504 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5000b9c0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e5d0df700 1 -- 172.21.15.50:0/1468779504 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e500092e0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 -- 172.21.15.50:0/1468779504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 msgr2=0x7f0e580ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 --2- 172.21.15.50:0/1468779504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009f50 tx=0x7f0e50009840 comp rx=0 tx=0).stop 2023-12-19T14:56:55.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 -- 172.21.15.50:0/1468779504 shutdown_connections 2023-12-19T14:56:55.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 --2- 172.21.15.50:0/1468779504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580ceb00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 --2- 172.21.15.50:0/1468779504 >> v2:172.21.15.125:3300/0 conn(0x7f0e58158e20 0x7f0e580ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 -- 172.21.15.50:0/1468779504 >> 172.21.15.50:0/1468779504 conn(0x7f0e58067bb0 msgr2=0x7f0e580d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:55.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 -- 172.21.15.50:0/1468779504 shutdown_connections 2023-12-19T14:56:55.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.867+0000 7f0e60345700 1 -- 172.21.15.50:0/1468779504 wait complete. 2023-12-19T14:56:55.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 Processor -- start 2023-12-19T14:56:55.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 -- start start 2023-12-19T14:56:55.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e58158e20 0x7f0e580c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580b56b0 0x7f0e5815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5814c6c0 con 0x7f0e58158e20 2023-12-19T14:56:55.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580b5220 con 0x7f0e580b56b0 2023-12-19T14:56:55.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.871+0000 7f0e60345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5814d090 con 0x7f0e580ca2c0 2023-12-19T14:56:55.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:59060/0 (socket says 172.21.15.50:59060) 2023-12-19T14:56:55.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 -- 172.21.15.50:0/1235564348 learned_addr learned my addr 172.21.15.50:0/1235564348 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5d8e0700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e58158e20 0x7f0e580c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e8e2700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580b56b0 0x7f0e5815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 -- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580b56b0 msgr2=0x7f0e5815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580b56b0 0x7f0e5815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 -- 172.21.15.50:0/1235564348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e58158e20 msgr2=0x7f0e580c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e58158e20 0x7f0e580c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 -- 172.21.15.50:0/1235564348 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50009ae0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e5e0e1700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400e930 tx=0x7f0e540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54013680 con 0x7f0e580ca2c0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e540137e0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54013b80 con 0x7f0e580ca2c0 2023-12-19T14:56:55.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e580b4610 con 0x7f0e580ca2c0 2023-12-19T14:56:55.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.875+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e580b4bd0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e58071120 con 0x7f0e580ca2c0 2023-12-19T14:56:55.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f0e5400c3e0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e4e7fc700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 0x7f0e3c062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f0e5407cfa0 con 0x7f0e580ca2c0 2023-12-19T14:56:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e5d8e0700 1 -- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 msgr2=0x7f0e3c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:55.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.879+0000 7f0e5d8e0700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 0x7f0e3c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:55.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:55.887+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e5404cf30 con 0x7f0e580ca2c0 2023-12-19T14:56:55.910 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:55.907+0000 7fe6b6f82000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:56:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.059+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e5806a7a0 con 0x7f0e580ca2c0 2023-12-19T14:56:56.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.059+0000 7f0e4e7fc700 1 -- 172.21.15.50:0/1235564348 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e54050550 con 0x7f0e580ca2c0 2023-12-19T14:56:56.061 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:56:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 msgr2=0x7f0e3c062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:56.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 0x7f0e3c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 msgr2=0x7f0e580b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400e930 tx=0x7f0e540091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 shutdown_connections 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580b56b0 0x7f0e5815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0e3c060790 0x7f0e3c062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e58158e20 0x7f0e580c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 --2- 172.21.15.50:0/1235564348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e580ca2c0 0x7f0e580b3f80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 >> 172.21.15.50:0/1235564348 conn(0x7f0e58067bb0 msgr2=0x7f0e5814f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 shutdown_connections 2023-12-19T14:56:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.063+0000 7f0e60345700 1 -- 172.21.15.50:0/1235564348 wait complete. 2023-12-19T14:56:56.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:56:56.203 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:56.199+0000 7f7645d62000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:56:56.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 Processor -- start 2023-12-19T14:56:56.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 -- start start 2023-12-19T14:56:56.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fd0069a20 0x7f6fd006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd00b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00ab6c0 con 0x7f6fd006cfa0 2023-12-19T14:56:56.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00ac240 con 0x7f6fd0069a20 2023-12-19T14:56:56.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd63c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00ab910 con 0x7f6fd01591e0 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fcffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6fd006cfa0 0x7f6fd0158ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50450/0 (socket says 172.21.15.50:50450) 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fcffff700 1 -- 172.21.15.50:0/3162947682 learned_addr learned my addr 172.21.15.50:0/3162947682 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.299+0000 7f6fd4962700 1 --2- 172.21.15.50:0/3162947682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd00b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fcf7fe700 1 --2- 172.21.15.50:0/3162947682 >> v2:172.21.15.125:3300/0 conn(0x7f6fd0069a20 0x7f6fd006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fceffd700 1 -- 172.21.15.50:0/3162947682 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138997601 0 0) 0x7f6fd00ab6c0 con 0x7f6fd006cfa0 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fceffd700 1 -- 172.21.15.50:0/3162947682 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb8003570 con 0x7f6fd006cfa0 2023-12-19T14:56:56.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd4962700 1 -- 172.21.15.50:0/3162947682 >> v2:172.21.15.125:3300/0 conn(0x7f6fd0069a20 msgr2=0x7f6fd006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:56.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd4962700 1 --2- 172.21.15.50:0/3162947682 >> v2:172.21.15.125:3300/0 conn(0x7f6fd0069a20 0x7f6fd006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd4962700 1 -- 172.21.15.50:0/3162947682 >> v1:172.21.15.50:6789/0 conn(0x7f6fd006cfa0 legacy=0x7f6fd0158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd4962700 1 -- 172.21.15.50:0/3162947682 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00abbf0 con 0x7f6fd01591e0 2023-12-19T14:56:56.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd4962700 1 --2- 172.21.15.50:0/3162947682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd00b28d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e60 tx=0x7f6fc8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cadcb9fe16fe1dff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:56.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fceffd700 1 -- 172.21.15.50:0/3162947682 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc800e070 con 0x7f6fd01591e0 2023-12-19T14:56:56.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fceffd700 1 -- 172.21.15.50:0/3162947682 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc800bab0 con 0x7f6fd01591e0 2023-12-19T14:56:56.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fceffd700 1 -- 172.21.15.50:0/3162947682 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc800be70 con 0x7f6fd01591e0 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 -- 172.21.15.50:0/3162947682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 msgr2=0x7f6fd00b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/3162947682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd00b28d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e60 tx=0x7f6fc8009780 comp rx=0 tx=0).stop 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 -- 172.21.15.50:0/3162947682 shutdown_connections 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/3162947682 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd00b28d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/3162947682 >> v2:172.21.15.125:3300/0 conn(0x7f6fd0069a20 0x7f6fd006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 -- 172.21.15.50:0/3162947682 >> 172.21.15.50:0/3162947682 conn(0x7f6fd00cd870 msgr2=0x7f6fd0069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 -- 172.21.15.50:0/3162947682 shutdown_connections 2023-12-19T14:56:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.303+0000 7f6fd63c5700 1 -- 172.21.15.50:0/3162947682 wait complete. 2023-12-19T14:56:56.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.307+0000 7f6fd63c5700 1 Processor -- start 2023-12-19T14:56:56.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.307+0000 7f6fd63c5700 1 -- start start 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd0069a20 0x7f6fd015ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd015e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00c7580 con 0x7f6fd006cfa0 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00c6970 con 0x7f6fd0069a20 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00c6c50 con 0x7f6fd01591e0 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42624/0 (socket says 172.21.15.50:42624) 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 -- 172.21.15.50:0/2168021471 learned_addr learned my addr 172.21.15.50:0/2168021471 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcffff700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd0069a20 0x7f6fd015ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd4962700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 -- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 msgr2=0x7f6fd015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 -- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd0069a20 msgr2=0x7f6fd015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd0069a20 0x7f6fd015ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 -- 172.21.15.50:0/2168021471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc8009a20 con 0x7f6fd006cfa0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fcf7fe700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400ec50 tx=0x7f6fc40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400c850 con 0x7f6fd006cfa0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc400c9b0 con 0x7f6fd006cfa0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd00c6f30 con 0x7f6fd006cfa0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400cd70 con 0x7f6fd006cfa0 2023-12-19T14:56:56.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.311+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd015e330 con 0x7f6fd006cfa0 2023-12-19T14:56:56.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd00ad750 con 0x7f6fd006cfa0 2023-12-19T14:56:56.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f6fc4013680 con 0x7f6fd006cfa0 2023-12-19T14:56:56.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fb7fff700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 0x7f6fbc062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f6fc407d0e0 con 0x7f6fd006cfa0 2023-12-19T14:56:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fcffff700 1 -- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 msgr2=0x7f6fbc062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.315+0000 7f6fcffff700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 0x7f6fbc062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:56.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.323+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6fd00ad750 con 0x7f6fd006cfa0 2023-12-19T14:56:56.338 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:56.335+0000 7fe6b6f82000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:56:56.377 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:56.371+0000 7f7645d62000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:56:56.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.491+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6fd00ad750 con 0x7f6fd006cfa0 2023-12-19T14:56:56.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fb7fff700 1 -- 172.21.15.50:0/2168021471 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6fd00ad750 con 0x7f6fd006cfa0 2023-12-19T14:56:56.495 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 msgr2=0x7f6fbc062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 0x7f6fbc062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 msgr2=0x7f6fd00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400ec50 tx=0x7f6fc40091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 shutdown_connections 2023-12-19T14:56:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6fbc060790 0x7f6fbc062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fd01591e0 0x7f6fd015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fd006cfa0 0x7f6fd00cbd00 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 --2- 172.21.15.50:0/2168021471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fd0069a20 0x7f6fd015ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 >> 172.21.15.50:0/2168021471 conn(0x7f6fd00cd870 msgr2=0x7f6fd0069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 shutdown_connections 2023-12-19T14:56:56.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.495+0000 7f6fd63c5700 1 -- 172.21.15.50:0/2168021471 wait complete. 2023-12-19T14:56:56.517 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:56.515+0000 7fe6b6f82000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:56:56.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:56:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.683+0000 7f73d686e700 1 Processor -- start 2023-12-19T14:56:56.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.683+0000 7f73d686e700 1 -- start start 2023-12-19T14:56:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d686e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d00680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d686e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73d0068630 0x7f73d00ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d686e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d006e150 con 0x7f73d0140710 2023-12-19T14:56:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d686e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d0068ab0 con 0x7f73d0068630 2023-12-19T14:56:56.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d686e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d006d540 con 0x7f73d0140ae0 2023-12-19T14:56:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73d0068630 0x7f73d00ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73d0068630 0x7f73d00ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:40546/0 (socket says 172.21.15.50:40546) 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 -- 172.21.15.50:0/321901004 learned_addr learned my addr 172.21.15.50:0/321901004 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73c7fff700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d00680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 -- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 msgr2=0x7f73d00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d00680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 -- 172.21.15.50:0/321901004 >> v1:172.21.15.125:6789/0 conn(0x7f73d0140710 legacy=0x7f73d00c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 -- 172.21.15.50:0/321901004 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d006d820 con 0x7f73d0068630 2023-12-19T14:56:56.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73d4e0b700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0068630 0x7f73d00ac250 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009f20 tx=0x7f73c8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf21baa0224bef4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.687+0000 7f73cf7fe700 1 -- 172.21.15.50:0/321901004 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800e070 con 0x7f73d0068630 2023-12-19T14:56:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73cf7fe700 1 -- 172.21.15.50:0/321901004 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800b9d0 con 0x7f73d0068630 2023-12-19T14:56:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73cf7fe700 1 -- 172.21.15.50:0/321901004 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80092e0 con 0x7f73d0068630 2023-12-19T14:56:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 -- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0068630 msgr2=0x7f73d00ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0068630 0x7f73d00ac250 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009f20 tx=0x7f73c8009840 comp rx=0 tx=0).stop 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 -- 172.21.15.50:0/321901004 shutdown_connections 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0068630 0x7f73d00ac250 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 --2- 172.21.15.50:0/321901004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d00680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 -- 172.21.15.50:0/321901004 >> 172.21.15.50:0/321901004 conn(0x7f73d00c8850 msgr2=0x7f73d00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 -- 172.21.15.50:0/321901004 shutdown_connections 2023-12-19T14:56:56.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.691+0000 7f73d686e700 1 -- 172.21.15.50:0/321901004 wait complete. 2023-12-19T14:56:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 Processor -- start 2023-12-19T14:56:56.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 -- start start 2023-12-19T14:56:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 0x7f73d00c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00b19c0 con 0x7f73d0068630 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00c1a30 con 0x7f73d0140710 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d686e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00a9480 con 0x7f73d0140ae0 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73d4e0b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d01569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73c7fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 0x7f73d00c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49940/0 (socket says 172.21.15.50:49940) 2023-12-19T14:56:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 -- 172.21.15.50:0/630525386 learned_addr learned my addr 172.21.15.50:0/630525386 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 msgr2=0x7f73d01569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d01569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 msgr2=0x7f73d00c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 0x7f73d00c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 -- 172.21.15.50:0/630525386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8009ae0 con 0x7f73d0068630 2023-12-19T14:56:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73c7fff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 0x7f73d00c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:56:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.695+0000 7f73cffff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f73c000c4e0 tx=0x7f73c000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011590 con 0x7f73d0068630 2023-12-19T14:56:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c0011710 con 0x7f73d0068630 2023-12-19T14:56:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d00a9760 con 0x7f73d0068630 2023-12-19T14:56:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011ad0 con 0x7f73d0068630 2023-12-19T14:56:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d00c2400 con 0x7f73d0068630 2023-12-19T14:56:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.699+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0071120 con 0x7f73d0068630 2023-12-19T14:56:56.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.703+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f73c001b3b0 con 0x7f73d0068630 2023-12-19T14:56:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.703+0000 7f73ccff9700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 0x7f73b0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.703+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f73c007c670 con 0x7f73d0068630 2023-12-19T14:56:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.703+0000 7f73c7fff700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 msgr2=0x7f73b0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:56.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.703+0000 7f73c7fff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 0x7f73b0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.707+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f73c004c680 con 0x7f73d0068630 2023-12-19T14:56:56.754 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:56.747+0000 7f7645d62000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.907+0000 7f73c7fff700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 msgr2=0x7f73b0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.907+0000 7f73c7fff700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 0x7f73b0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:56:56.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.915+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73d00c1860 con 0x7f73d0068630 2023-12-19T14:56:56.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.915+0000 7f73ccff9700 1 -- 172.21.15.50:0/630525386 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f73c0017020 con 0x7f73d0068630 2023-12-19T14:56:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:56.918 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 msgr2=0x7f73b0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 0x7f73b0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 msgr2=0x7f73d00b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f73c000c4e0 tx=0x7f73c000c7f0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 shutdown_connections 2023-12-19T14:56:56.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73d0140710 0x7f73d00c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f73b0060790 0x7f73b0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73d0140ae0 0x7f73d01569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 --2- 172.21.15.50:0/630525386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f73d0068630 0x7f73d00b1480 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 >> 172.21.15.50:0/630525386 conn(0x7f73d00c8850 msgr2=0x7f73d013e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 shutdown_connections 2023-12-19T14:56:56.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:56.919+0000 7f73d686e700 1 -- 172.21.15.50:0/630525386 wait complete. 2023-12-19T14:56:56.925 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:56:56.932 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:56.931+0000 7fe6b6f82000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:56:56.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:56:56.977 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:56.971+0000 7f7645d62000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:56:57.066 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.059+0000 7f7645d62000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:56:57.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.115+0000 7f7718c8e700 1 Processor -- start 2023-12-19T14:56:57.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.115+0000 7f7718c8e700 1 -- start start 2023-12-19T14:56:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7718c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 0x7f771406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7718c8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77140cc470 0x7f77140cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7718c8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77140ca5a0 con 0x7f7714158e70 2023-12-19T14:56:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7718c8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771406dac0 con 0x7f77140cc470 2023-12-19T14:56:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7718c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771406d220 con 0x7f771406a540 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77140cc470 0x7f77140cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77140cc470 0x7f77140cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49944/0 (socket says 172.21.15.50:49944) 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 -- 172.21.15.50:0/2714219997 learned_addr learned my addr 172.21.15.50:0/2714219997 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 -- 172.21.15.50:0/2714219997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 msgr2=0x7f771406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f770bfff700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 0x7f771406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 0x7f771406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 -- 172.21.15.50:0/2714219997 >> v1:172.21.15.50:6789/0 conn(0x7f7714158e70 legacy=0x7f77140afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 -- 172.21.15.50:0/2714219997 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771406d500 con 0x7f77140cc470 2023-12-19T14:56:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f770bfff700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 0x7f771406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:56:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7713fff700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77140cc470 0x7f77140cc8d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f770c009f20 tx=0x7f770c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b3a00c32d5bfac server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7712ffd700 1 -- 172.21.15.50:0/2714219997 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c00e070 con 0x7f77140cc470 2023-12-19T14:56:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7712ffd700 1 -- 172.21.15.50:0/2714219997 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c00ba80 con 0x7f77140cc470 2023-12-19T14:56:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.119+0000 7f7712ffd700 1 -- 172.21.15.50:0/2714219997 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0092e0 con 0x7f77140cc470 2023-12-19T14:56:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- 172.21.15.50:0/2714219997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77140cc470 msgr2=0x7f77140cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77140cc470 0x7f77140cc8d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f770c009f20 tx=0x7f770c009840 comp rx=0 tx=0).stop 2023-12-19T14:56:57.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- 172.21.15.50:0/2714219997 shutdown_connections 2023-12-19T14:56:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77140cc470 0x7f77140cc8d0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 --2- 172.21.15.50:0/2714219997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f771406a540 0x7f771406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- 172.21.15.50:0/2714219997 >> 172.21.15.50:0/2714219997 conn(0x7f77140670e0 msgr2=0x7f77140674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- 172.21.15.50:0/2714219997 shutdown_connections 2023-12-19T14:56:57.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- 172.21.15.50:0/2714219997 wait complete. 2023-12-19T14:56:57.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 Processor -- start 2023-12-19T14:56:57.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 -- start start 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.123+0000 7f7718c8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77140cc470 0x7f7714155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7714158e70 0x7f771415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7714159240 con 0x7f771406a540 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7714156520 con 0x7f7714158e70 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77140c38c0 con 0x7f77140cc470 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49948/0 (socket says 172.21.15.50:49948) 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 -- 172.21.15.50:0/684973919 learned_addr learned my addr 172.21.15.50:0/684973919 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:57.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f770bfff700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77140cc470 0x7f7714155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7713fff700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7714158e70 0x7f771415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 -- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77140cc470 msgr2=0x7f7714155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77140cc470 0x7f7714155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 -- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7714158e70 msgr2=0x7f771415baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7714158e70 0x7f771415baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 -- 172.21.15.50:0/684973919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c009ae0 con 0x7f771406a540 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f77137fe700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f770400c5c0 tx=0x7f770400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704011590 con 0x7f771406a540 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7704011710 con 0x7f771406a540 2023-12-19T14:56:57.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704011ad0 con 0x7f771406a540 2023-12-19T14:56:57.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7714159790 con 0x7f771406a540 2023-12-19T14:56:57.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.127+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77140b4e50 con 0x7f771406a540 2023-12-19T14:56:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77140b5070 con 0x7f771406a540 2023-12-19T14:56:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f770401b3b0 con 0x7f771406a540 2023-12-19T14:56:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f770b7fe700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 0x7f76f4062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f770407c310 con 0x7f771406a540 2023-12-19T14:56:57.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f770bfff700 1 -- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 msgr2=0x7f76f4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:57.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.131+0000 7f770bfff700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 0x7f76f4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:57.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.139+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f770404c2a0 con 0x7f771406a540 2023-12-19T14:56:57.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.139+0000 7f7645d62000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:56:57.178 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.175+0000 7fe6b6f82000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:56:57.225 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.219+0000 7f7645d62000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:56:57.279 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.279+0000 7fe6b6f82000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:56:57.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.311+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771404be40 con 0x7f771406a540 2023-12-19T14:56:57.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.311+0000 7f770b7fe700 1 -- 172.21.15.50:0/684973919 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16304 (secure 0 0 0) 0x7f770404f8c0 con 0x7f771406a540 2023-12-19T14:56:57.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:57.315 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":116,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:56:52.034648+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.1a","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/738581743":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/1068162751":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3184659104":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3685761614":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2498915371":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2764835356":"2023-12-20T14:55:50.589350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 msgr2=0x7f76f4062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 0x7f76f4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 msgr2=0x7f77140b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f770400c5c0 tx=0x7f770400c8d0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 shutdown_connections 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7714158e70 0x7f771415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f76f40607c0 0x7f76f4062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f77140cc470 0x7f7714155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 --2- 172.21.15.50:0/684973919 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f771406a540 0x7f77140b4520 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 >> 172.21.15.50:0/684973919 conn(0x7f77140670e0 msgr2=0x7f77140cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 shutdown_connections 2023-12-19T14:56:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.315+0000 7f7718c8e700 1 -- 172.21.15.50:0/684973919 wait complete. 2023-12-19T14:56:57.338 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:56:57.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:56:57.360 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.359+0000 7fe6b6f82000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:56:57.457 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.451+0000 7f7645d62000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:56:57.457 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.455+0000 7fe6b6f82000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:56:57.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.499+0000 7f31e47c5700 1 Processor -- start 2023-12-19T14:56:57.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.499+0000 7f31e47c5700 1 -- start start 2023-12-19T14:56:57.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e47c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31dc0c9fc0 0x7f31dc0afc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e47c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0b01c0 con 0x7f31dc0c8b00 2023-12-19T14:56:57.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e47c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0ca3c0 con 0x7f31dc153860 2023-12-19T14:56:57.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e47c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0ca650 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e1d60700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f31dc0c8b00 0x7f31dc0b16e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52934/0 (socket says 172.21.15.50:52934) 2023-12-19T14:56:57.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e1d60700 1 -- 172.21.15.50:0/675983231 learned_addr learned my addr 172.21.15.50:0/675983231 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:57.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.503+0000 7f31e2d62700 1 --2- 172.21.15.50:0/675983231 >> v2:172.21.15.125:3300/0 conn(0x7f31dc0c9fc0 0x7f31dc0afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e155f700 1 -- 172.21.15.50:0/675983231 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361711357 0 0) 0x7f31cc001110 con 0x7f31dc0c8b00 2023-12-19T14:56:57.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e155f700 1 -- 172.21.15.50:0/675983231 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c8003570 con 0x7f31dc0c8b00 2023-12-19T14:56:57.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e2d62700 1 -- 172.21.15.50:0/675983231 >> v1:172.21.15.125:6789/0 conn(0x7f31dc153860 legacy=0x7f31dc0c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e2d62700 1 -- 172.21.15.50:0/675983231 >> v1:172.21.15.50:6789/0 conn(0x7f31dc0c8b00 legacy=0x7f31dc0b16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e2d62700 1 -- 172.21.15.50:0/675983231 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc156700 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e2d62700 1 --2- 172.21.15.50:0/675983231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc0afc80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009e60 tx=0x7f31d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b92d42bc303e8b20 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e155f700 1 -- 172.21.15.50:0/675983231 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d400bb60 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e155f700 1 -- 172.21.15.50:0/675983231 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d400bcc0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e155f700 1 -- 172.21.15.50:0/675983231 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d40116b0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 -- 172.21.15.50:0/675983231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c9fc0 msgr2=0x7f31dc0afc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 --2- 172.21.15.50:0/675983231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc0afc80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009e60 tx=0x7f31d4009780 comp rx=0 tx=0).stop 2023-12-19T14:56:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 -- 172.21.15.50:0/675983231 shutdown_connections 2023-12-19T14:56:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 --2- 172.21.15.50:0/675983231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc0afc80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 -- 172.21.15.50:0/675983231 >> 172.21.15.50:0/675983231 conn(0x7f31dc0cd8e0 msgr2=0x7f31dc0c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 -- 172.21.15.50:0/675983231 shutdown_connections 2023-12-19T14:56:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.507+0000 7f31e47c5700 1 -- 172.21.15.50:0/675983231 wait complete. 2023-12-19T14:56:57.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.511+0000 7f31e47c5700 1 Processor -- start 2023-12-19T14:56:57.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.511+0000 7f31e47c5700 1 -- start start 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.511+0000 7f31e47c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 0x7f31dc0699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.511+0000 7f31e47c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.511+0000 7f31e47c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 0x7f31dc14f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15a5a0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0c1c40 con 0x7f31dc0c8b00 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0c1350 con 0x7f31dc153860 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49962/0 (socket says 172.21.15.50:49962) 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 -- 172.21.15.50:0/1078340345 learned_addr learned my addr 172.21.15.50:0/1078340345 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:57.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e2561700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 0x7f31dc0699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e2d62700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 0x7f31dc14f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 msgr2=0x7f31dc14f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 0x7f31dc14f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 msgr2=0x7f31dc0699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 0x7f31dc0699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 -- 172.21.15.50:0/1078340345 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4009a20 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e2d62700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 0x7f31dc14f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e2561700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 0x7f31dc0699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e1d60700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c790 tx=0x7f31cc007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc005b30 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc005c90 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc0c15f0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc007320 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc15a7d0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc0c1df0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.515+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f31cc007710 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.519+0000 7f31d27fc700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 0x7f31c0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.519+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f31cc07c2d0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.519+0000 7f31e2561700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 msgr2=0x7f31c0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:57.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.519+0000 7f31e2561700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 0x7f31c0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:57.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.523+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f31dc0c1df0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.538 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.531+0000 7f7645d62000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:56:57.627 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.623+0000 7f7645d62000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:56:57.709 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.707+0000 7fe6b6f82000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:56:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.723+0000 7f31e2561700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 msgr2=0x7f31c0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.723+0000 7f31e2561700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 0x7f31c0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:56:57.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.747+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31dc0b01c0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.747+0000 7f31d27fc700 1 -- 172.21.15.50:0/1078340345 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f31dc0b01c0 con 0x7f31dc0c9fc0 2023-12-19T14:56:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:57.750 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.751+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 msgr2=0x7f31c0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.751+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 0x7f31c0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.751+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 msgr2=0x7f31dc15a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.751+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c790 tx=0x7f31cc007c50 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.751+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 shutdown_connections 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f31c0060790 0x7f31c0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f31dc153860 0x7f31dc14f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31dc0c9fc0 0x7f31dc15a060 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 --2- 172.21.15.50:0/1078340345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0c8b00 0x7f31dc0699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 >> 172.21.15.50:0/1078340345 conn(0x7f31dc0cd8e0 msgr2=0x7f31dc0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 shutdown_connections 2023-12-19T14:56:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.755+0000 7f31e47c5700 1 -- 172.21.15.50:0/1078340345 wait complete. 2023-12-19T14:56:57.757 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:56:57.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:56:57.791 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.791+0000 7fe6b6f82000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:56:57.828 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.823+0000 7f7645d62000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:56:57.889 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:57.887+0000 7fe6b6f82000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:56:57.916 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:57.911+0000 7f7645d62000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:56:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.951+0000 7f11cd66e700 1 Processor -- start 2023-12-19T14:56:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.951+0000 7f11cd66e700 1 -- start start 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11cd66e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f11c80c2980 0x7f11c80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11cd66e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11c80680f0 0x7f11c8068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11cd66e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c806c9f0 con 0x7f11c813f890 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11cd66e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80c7f20 con 0x7f11c80c2980 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11cd66e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c8068a50 con 0x7f11c80680f0 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f11c80c2980 0x7f11c80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f11c80c2980 0x7f11c80aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49972/0 (socket says 172.21.15.50:49972) 2023-12-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 -- 172.21.15.50:0/1096758401 learned_addr learned my addr 172.21.15.50:0/1096758401 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c77fe700 1 --2- 172.21.15.50:0/1096758401 >> v2:172.21.15.125:3300/0 conn(0x7f11c80680f0 0x7f11c8068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 -- 172.21.15.50:0/1096758401 >> v2:172.21.15.125:3300/0 conn(0x7f11c80680f0 msgr2=0x7f11c8068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 --2- 172.21.15.50:0/1096758401 >> v2:172.21.15.125:3300/0 conn(0x7f11c80680f0 0x7f11c8068510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 -- 172.21.15.50:0/1096758401 >> v1:172.21.15.125:6789/0 conn(0x7f11c813f890 legacy=0x7f11c80c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 -- 172.21.15.50:0/1096758401 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80c1c50 con 0x7f11c80c2980 2023-12-19T14:56:57.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c67fc700 1 --2- 172.21.15.50:0/1096758401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80c2980 0x7f11c80aeba0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f11c806a6a0 tx=0x7f11bc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8612d2c9e3505fd0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c5ffb700 1 -- 172.21.15.50:0/1096758401 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00dc50 con 0x7f11c80c2980 2023-12-19T14:56:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c5ffb700 1 -- 172.21.15.50:0/1096758401 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc00ddb0 con 0x7f11c80c2980 2023-12-19T14:56:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.955+0000 7f11c5ffb700 1 -- 172.21.15.50:0/1096758401 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0196d0 con 0x7f11c80c2980 2023-12-19T14:56:57.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- 172.21.15.50:0/1096758401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80c2980 msgr2=0x7f11c80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1096758401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80c2980 0x7f11c80aeba0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f11c806a6a0 tx=0x7f11bc00b4f0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- 172.21.15.50:0/1096758401 shutdown_connections 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1096758401 >> v2:172.21.15.125:3300/0 conn(0x7f11c80680f0 0x7f11c8068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1096758401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80c2980 0x7f11c80aeba0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- 172.21.15.50:0/1096758401 >> 172.21.15.50:0/1096758401 conn(0x7f11c80c8850 msgr2=0x7f11c80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- 172.21.15.50:0/1096758401 shutdown_connections 2023-12-19T14:56:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- 172.21.15.50:0/1096758401 wait complete. 2023-12-19T14:56:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 Processor -- start 2023-12-19T14:56:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- start start 2023-12-19T14:56:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 0x7f11c8136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 0x7f11c8138870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11c813f890 0x7f11c813cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c8136f90 con 0x7f11c80680f0 2023-12-19T14:56:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c8138db0 con 0x7f11c80c2980 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.959+0000 7f11cd66e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c8139050 con 0x7f11c813f890 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 0x7f11c8136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c77fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11c813f890 0x7f11c813cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c67fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 0x7f11c8138870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c67fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 0x7f11c8138870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:40594/0 (socket says 172.21.15.50:40594) 2023-12-19T14:56:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c67fc700 1 -- 172.21.15.50:0/1339245156 learned_addr learned my addr 172.21.15.50:0/1339245156 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11c813f890 msgr2=0x7f11c813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11c813f890 0x7f11c813cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 msgr2=0x7f11c8138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 0x7f11c8138870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 -- 172.21.15.50:0/1339245156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc00b790 con 0x7f11c80680f0 2023-12-19T14:56:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11c6ffd700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 0x7f11c8136a50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c490 tx=0x7f11b8007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b800ce40 con 0x7f11c80680f0 2023-12-19T14:56:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b8011590 con 0x7f11c80680f0 2023-12-19T14:56:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c8139330 con 0x7f11c80680f0 2023-12-19T14:56:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8011950 con 0x7f11c80680f0 2023-12-19T14:56:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c81398f0 con 0x7f11c80680f0 2023-12-19T14:56:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.963+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c8071120 con 0x7f11c80680f0 2023-12-19T14:56:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.967+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f11b8011cc0 con 0x7f11c80680f0 2023-12-19T14:56:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.967+0000 7f11aeffd700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 0x7f11b4062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.967+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f11b807c160 con 0x7f11c80680f0 2023-12-19T14:56:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.967+0000 7f11c67fc700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 msgr2=0x7f11b4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.967+0000 7f11c67fc700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 0x7f11b4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:57.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:57.975+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f11b804c190 con 0x7f11c80680f0 2023-12-19T14:56:58.111 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:58.111+0000 7fe6b6f82000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:56:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.171+0000 7f11c67fc700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 msgr2=0x7f11b4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:58.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.171+0000 7f11c67fc700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 0x7f11b4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:56:58.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.187+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11c80642a0 con 0x7f11c80680f0 2023-12-19T14:56:58.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.187+0000 7f11aeffd700 1 -- 172.21.15.50:0/1339245156 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f11b804f7b0 con 0x7f11c80680f0 2023-12-19T14:56:58.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:58.188 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:55:35.098455+0000","modified":"2023-12-19T14:55:38.968315+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7377},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7377":{"gid":7377,"name":"a","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3096035349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3096035349},{"type":"v1","addr":"172.21.15.50:6835","nonce":3096035349}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T14:56:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 msgr2=0x7f11b4062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 0x7f11b4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 msgr2=0x7f11c8136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 0x7f11c8136a50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c490 tx=0x7f11b8007ec0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 shutdown_connections 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11c80c2980 0x7f11c8138870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11b40607c0 0x7f11b4062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11c813f890 0x7f11c813cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 --2- 172.21.15.50:0/1339245156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11c80680f0 0x7f11c8136a50 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 >> 172.21.15.50:0/1339245156 conn(0x7f11c80c8850 msgr2=0x7f11c813e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 shutdown_connections 2023-12-19T14:56:58.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.191+0000 7f11cd66e700 1 -- 172.21.15.50:0/1339245156 wait complete. 2023-12-19T14:56:58.194 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 50 2023-12-19T14:56:58.207 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:58.207+0000 7fe6b6f82000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:56:58.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:56:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 Processor -- start 2023-12-19T14:56:58.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 -- start start 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f89081592b0 0x7f890806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080688d0 con 0x7f890806d590 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080b1510 con 0x7f8908158ee0 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f890cf45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890806d9b0 con 0x7f89081592b0 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f8906d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f890806d590 0x7f89080b19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52940/0 (socket says 172.21.15.50:52940) 2023-12-19T14:56:58.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f8906d9d700 1 -- 172.21.15.50:0/914859937 learned_addr learned my addr 172.21.15.50:0/914859937 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.371+0000 7f8905d9b700 1 --2- 172.21.15.50:0/914859937 >> v2:172.21.15.125:3300/0 conn(0x7f89081592b0 0x7f890806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403287656 0 0) 0x7f89080688d0 con 0x7f890806d590 2023-12-19T14:56:58.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f0003570 con 0x7f890806d590 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32752354 0 0) 0x7f89080b1510 con 0x7f8908158ee0 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89080688d0 con 0x7f8908158ee0 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 433970110 0 0) 0x7f88f0003570 con 0x7f890806d590 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f8905d9b700 1 -- 172.21.15.50:0/914859937 >> v1:172.21.15.125:6789/0 conn(0x7f8908158ee0 legacy=0x7f8908067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f8905d9b700 1 -- 172.21.15.50:0/914859937 >> v1:172.21.15.50:6789/0 conn(0x7f890806d590 legacy=0x7f89080b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f8905d9b700 1 -- 172.21.15.50:0/914859937 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890806a180 con 0x7f89081592b0 2023-12-19T14:56:58.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f8905d9b700 1 --2- 172.21.15.50:0/914859937 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89081592b0 0x7f890806d050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0098e0 tx=0x7f88fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca68f23e3651b09 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000017f0 con 0x7f89081592b0 2023-12-19T14:56:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc011420 con 0x7f89081592b0 2023-12-19T14:56:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890559a700 1 -- 172.21.15.50:0/914859937 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0117c0 con 0x7f89081592b0 2023-12-19T14:56:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 -- 172.21.15.50:0/914859937 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89081592b0 msgr2=0x7f890806d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 --2- 172.21.15.50:0/914859937 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89081592b0 0x7f890806d050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0098e0 tx=0x7f88fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 -- 172.21.15.50:0/914859937 shutdown_connections 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 --2- 172.21.15.50:0/914859937 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89081592b0 0x7f890806d050 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 -- 172.21.15.50:0/914859937 >> 172.21.15.50:0/914859937 conn(0x7f89080cd990 msgr2=0x7f89080ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 -- 172.21.15.50:0/914859937 shutdown_connections 2023-12-19T14:56:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.375+0000 7f890cf45700 1 -- 172.21.15.50:0/914859937 wait complete. 2023-12-19T14:56:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.379+0000 7f890cf45700 1 Processor -- start 2023-12-19T14:56:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.379+0000 7f890cf45700 1 -- start start 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890806d590 0x7f8908152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89081592b0 0x7f890815a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890815da10 con 0x7f8908158ee0 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89080a4950 con 0x7f890806d590 2023-12-19T14:56:58.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890815c540 con 0x7f89081592b0 2023-12-19T14:56:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49976/0 (socket says 172.21.15.50:49976) 2023-12-19T14:56:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 -- 172.21.15.50:0/4225280019 learned_addr learned my addr 172.21.15.50:0/4225280019 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:58.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890659c700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890806d590 0x7f8908152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8906d9d700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89081592b0 0x7f890815a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 -- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89081592b0 msgr2=0x7f890815a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89081592b0 0x7f890815a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 -- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890806d590 msgr2=0x7f8908152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890806d590 0x7f8908152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 -- 172.21.15.50:0/4225280019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc009420 con 0x7f8908158ee0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f8905d9b700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009ed0 tx=0x7f88fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01cb20 con 0x7f8908158ee0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc01cc80 con 0x7f8908158ee0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01a450 con 0x7f8908158ee0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890815c820 con 0x7f8908158ee0 2023-12-19T14:56:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.383+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8908152fb0 con 0x7f8908158ee0 2023-12-19T14:56:58.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.387+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890815ca60 con 0x7f8908158ee0 2023-12-19T14:56:58.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.387+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 46) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f88fc01a840 con 0x7f8908158ee0 2023-12-19T14:56:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.391+0000 7f88f67fc700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 0x7f88e8062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.391+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f88fc07e660 con 0x7f8908158ee0 2023-12-19T14:56:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.391+0000 7f890659c700 1 -- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 msgr2=0x7f88e8062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:56:58.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.391+0000 7f890659c700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 0x7f88e8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:56:58.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.395+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f890815ca60 con 0x7f8908158ee0 2023-12-19T14:56:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.567+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89080a4440 con 0x7f8908158ee0 2023-12-19T14:56:58.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.567+0000 7f88f67fc700 1 -- 172.21.15.50:0/4225280019 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16304 (secure 0 0 0) 0x7f89080a4440 con 0x7f8908158ee0 2023-12-19T14:56:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:56:58.569 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":116,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:56:52.034648+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:55:32.714035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T14:55:34.094095+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.1a","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/738581743":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/1068162751":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3184659104":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3685761614":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2498915371":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/2764835356":"2023-12-20T14:55:50.589350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 msgr2=0x7f88e8062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 0x7f88e8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 msgr2=0x7f8908169380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009ed0 tx=0x7f88fc016000 comp rx=0 tx=0).stop 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 shutdown_connections 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f88e8060790 0x7f88e8062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89081592b0 0x7f890815a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8908158ee0 0x7f8908169380 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 --2- 172.21.15.50:0/4225280019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f890806d590 0x7f8908152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 >> 172.21.15.50:0/4225280019 conn(0x7f89080cd990 msgr2=0x7f8908154990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 shutdown_connections 2023-12-19T14:56:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.571+0000 7f890cf45700 1 -- 172.21.15.50:0/4225280019 wait complete. 2023-12-19T14:56:58.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:56:58.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.643+0000 7f7645d62000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:56:58.754 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760f922700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760f922700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760f922700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760f922700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760f922700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760d11d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760d11d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760d11d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760d11d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.755 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:56:58.747+0000 7f760d11d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:56:58.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.763+0000 7fe53a985700 1 Processor -- start 2023-12-19T14:56:58.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.763+0000 7fe53a985700 1 -- start start 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe53406dc00 0x7fe53406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53406b0d0 con 0x7fe53406baa0 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53406beb0 con 0x7fe53406dc00 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340b4790 con 0x7fe534158cc0 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55774/0 (socket says 172.21.15.50:55774) 2023-12-19T14:56:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 -- 172.21.15.50:0/3996563687 learned_addr learned my addr 172.21.15.50:0/3996563687 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe5337fe700 1 --2- 172.21.15.50:0/3996563687 >> v2:172.21.15.125:3300/0 conn(0x7fe53406dc00 0x7fe53406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 -- 172.21.15.50:0/3996563687 >> v2:172.21.15.125:3300/0 conn(0x7fe53406dc00 msgr2=0x7fe53406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 --2- 172.21.15.50:0/3996563687 >> v2:172.21.15.125:3300/0 conn(0x7fe53406dc00 0x7fe53406b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 -- 172.21.15.50:0/3996563687 >> v1:172.21.15.125:6789/0 conn(0x7fe53406baa0 legacy=0x7fe5340b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 -- 172.21.15.50:0/3996563687 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53406d710 con 0x7fe534158cc0 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe533fff700 1 --2- 172.21.15.50:0/3996563687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe5240099b0 tx=0x7fe524009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=586544cbdf7ba83d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe532ffd700 1 -- 172.21.15.50:0/3996563687 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524009ea0 con 0x7fe534158cc0 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe532ffd700 1 -- 172.21.15.50:0/3996563687 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400b860 con 0x7fe534158cc0 2023-12-19T14:56:58.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe532ffd700 1 -- 172.21.15.50:0/3996563687 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52400bc00 con 0x7fe534158cc0 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 -- 172.21.15.50:0/3996563687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 msgr2=0x7fe53406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 --2- 172.21.15.50:0/3996563687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe5240099b0 tx=0x7fe524009250 comp rx=0 tx=0).stop 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 -- 172.21.15.50:0/3996563687 shutdown_connections 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 --2- 172.21.15.50:0/3996563687 >> v2:172.21.15.125:3300/0 conn(0x7fe53406dc00 0x7fe53406b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.767+0000 7fe53a985700 1 --2- 172.21.15.50:0/3996563687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe534158cc0 0x7fe53406d1d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.771+0000 7fe53a985700 1 -- 172.21.15.50:0/3996563687 >> 172.21.15.50:0/3996563687 conn(0x7fe534067bb0 msgr2=0x7fe5340cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.771+0000 7fe53a985700 1 -- 172.21.15.50:0/3996563687 shutdown_connections 2023-12-19T14:56:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.771+0000 7fe53a985700 1 -- 172.21.15.50:0/3996563687 wait complete. 2023-12-19T14:56:58.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.771+0000 7fe53a985700 1 Processor -- start 2023-12-19T14:56:58.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.771+0000 7fe53a985700 1 -- start start 2023-12-19T14:56:58.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe53406baa0 0x7fe53416d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534158cc0 0x7fe5340b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53416d760 con 0x7fe53406dc00 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340af720 con 0x7fe534158cc0 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340afa00 con 0x7fe53406baa0 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49984/0 (socket says 172.21.15.50:49984) 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 -- 172.21.15.50:0/2730689772 learned_addr learned my addr 172.21.15.50:0/2730689772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe533fff700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe53406baa0 0x7fe53416d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe538f22700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534158cc0 0x7fe5340b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 -- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe53406baa0 msgr2=0x7fe53416d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe53406baa0 0x7fe53416d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 -- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534158cc0 msgr2=0x7fe5340b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534158cc0 0x7fe5340b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 -- 172.21.15.50:0/2730689772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240094f0 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe5337fe700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec80 tx=0x7fe5280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800c850 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52800c9b0 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5340afce0 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800cd50 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340b02a0 con 0x7fe53406dc00 2023-12-19T14:56:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.775+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534071120 con 0x7fe53406dc00 2023-12-19T14:56:58.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.779+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 47) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fe528013680 con 0x7fe53406dc00 2023-12-19T14:56:58.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.779+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6857+0+0 (secure 0 0 0) 0x7fe52807c980 con 0x7fe53406dc00 2023-12-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.787+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe52804ce50 con 0x7fe53406dc00 2023-12-19T14:56:58.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.987+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe53406b880 con 0x7fe53406dc00 2023-12-19T14:56:58.992 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:56:58.991+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:56:59.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:58.999+0000 7fe51bfff700 1 -- 172.21.15.50:0/2730689772 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe528050470 con 0x7fe53406dc00 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 msgr2=0x7fe5340abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec80 tx=0x7fe5280091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 shutdown_connections 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe534158cc0 0x7fe5340b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe53406dc00 0x7fe5340abe70 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 --2- 172.21.15.50:0/2730689772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe53406baa0 0x7fe53416d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 >> 172.21.15.50:0/2730689772 conn(0x7fe534067bb0 msgr2=0x7fe5340b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 shutdown_connections 2023-12-19T14:56:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe53a985700 1 -- 172.21.15.50:0/2730689772 wait complete. 2023-12-19T14:56:59.005 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:56:59.003+0000 7fe6b6f82000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:56:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:56:59.016 INFO:tasks.ceph.mds.a.smithi050.stderr: -6141> 2023-12-19T14:55:41.609+0000 7f6cbd088700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:56:59.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:56:59.068 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:56:59.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.179+0000 7ffb1bfff700 1 Processor -- start 2023-12-19T14:56:59.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.179+0000 7ffb1bfff700 1 -- start start 2023-12-19T14:56:59.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1bfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffb1c154780 0x7ffb1c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:59.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1bfff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c15ae30 con 0x7ffb1c0adab0 2023-12-19T14:56:59.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1bfff700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0abf70 con 0x7ffb1c06baa0 2023-12-19T14:56:59.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1bfff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c154b50 con 0x7ffb1c154780 2023-12-19T14:56:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffb1c0adab0 0x7ffb1c0adeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52948/0 (socket says 172.21.15.50:52948) 2023-12-19T14:56:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1b7fe700 1 -- 172.21.15.50:0/691560928 learned_addr learned my addr 172.21.15.50:0/691560928 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:56:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1affd700 1 --2- 172.21.15.50:0/691560928 >> v2:172.21.15.125:3300/0 conn(0x7ffb1c154780 0x7ffb1c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb19ffb700 1 -- 172.21.15.50:0/691560928 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097704000 0 0) 0x7ffb1c15ae30 con 0x7ffb1c0adab0 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb19ffb700 1 -- 172.21.15.50:0/691560928 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003570 con 0x7ffb1c0adab0 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1affd700 1 -- 172.21.15.50:0/691560928 >> v1:172.21.15.125:6789/0 conn(0x7ffb1c06baa0 legacy=0x7ffb1c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1affd700 1 -- 172.21.15.50:0/691560928 >> v1:172.21.15.50:6789/0 conn(0x7ffb1c0adab0 legacy=0x7ffb1c0adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1affd700 1 -- 172.21.15.50:0/691560928 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0ac210 con 0x7ffb1c154780 2023-12-19T14:56:59.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.183+0000 7ffb1affd700 1 --2- 172.21.15.50:0/691560928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c15b2c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0098e0 tx=0x7ffb0c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8ec49031dad5483 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb19ffb700 1 -- 172.21.15.50:0/691560928 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00b640 con 0x7ffb1c154780 2023-12-19T14:56:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb19ffb700 1 -- 172.21.15.50:0/691560928 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c00b7a0 con 0x7ffb1c154780 2023-12-19T14:56:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb19ffb700 1 -- 172.21.15.50:0/691560928 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c00bb40 con 0x7ffb1c154780 2023-12-19T14:56:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- 172.21.15.50:0/691560928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 msgr2=0x7ffb1c15b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/691560928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c15b2c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0098e0 tx=0x7ffb0c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- 172.21.15.50:0/691560928 shutdown_connections 2023-12-19T14:56:59.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/691560928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c15b2c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- 172.21.15.50:0/691560928 >> 172.21.15.50:0/691560928 conn(0x7ffb1c067b50 msgr2=0x7ffb1c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:56:59.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- 172.21.15.50:0/691560928 shutdown_connections 2023-12-19T14:56:59.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- 172.21.15.50:0/691560928 wait complete. 2023-12-19T14:56:59.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 Processor -- start 2023-12-19T14:56:59.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.187+0000 7ffb1bfff700 1 -- start start 2023-12-19T14:56:59.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 0x7ffb1c169370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c169c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1587e0 con 0x7ffb1c0adab0 2023-12-19T14:56:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c158a60 con 0x7ffb1c154780 2023-12-19T14:56:59.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c158d40 con 0x7ffb1c06baa0 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50012/0 (socket says 172.21.15.50:50012) 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1a7fc700 1 -- 172.21.15.50:0/2283485621 learned_addr learned my addr 172.21.15.50:0/2283485621 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 0x7ffb1c169370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1b7fe700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c169c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 -- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 msgr2=0x7ffb1c169c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c169c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 -- 172.21.15.50:0/2283485621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 msgr2=0x7ffb1c1582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:56:59.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 -- 172.21.15.50:0/2283485621 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c009420 con 0x7ffb1c06baa0 2023-12-19T14:56:59.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1b7fe700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c169c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:56:59.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1a7fc700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:56:59.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1affd700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 0x7ffb1c169370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c016000 tx=0x7ffb0c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:56:59.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c000d10 con 0x7ffb1c06baa0 2023-12-19T14:56:59.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c00bb20 con 0x7ffb1c06baa0 2023-12-19T14:56:59.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c159020 con 0x7ffb1c06baa0 2023-12-19T14:56:59.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c011650 con 0x7ffb1c06baa0 2023-12-19T14:56:59.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.191+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1595e0 con 0x7ffb1c06baa0 2023-12-19T14:56:59.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.195+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c04be40 con 0x7ffb1c06baa0 2023-12-19T14:56:59.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.195+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 47) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ffb0c011a00 con 0x7ffb1c06baa0 2023-12-19T14:56:59.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.199+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6943+0+0 (secure 0 0 0) 0x7ffb0c07dce0 con 0x7ffb1c06baa0 2023-12-19T14:56:59.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.207+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb0c04e150 con 0x7ffb1c06baa0 2023-12-19T14:56:59.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.407+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffb1c158e80 con 0x7ffb1c06baa0 2023-12-19T14:56:59.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.675+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 48) v1 ==== 79799+0+0 (secure 0 0 0) 0x7ffb0c04dc00 con 0x7ffb1c06baa0 2023-12-19T14:56:59.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.675+0000 7ffb02ffd700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 0x7ffb08068ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:56:59.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.679+0000 7ffb1a7fc700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 0x7ffb08068ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:56:59.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:56:59.683+0000 7ffb1a7fc700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 0x7ffb08068ad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c151f60 tx=0x7ffb10009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.007+0000 7ffb02ffd700 1 -- 172.21.15.50:0/2283485621 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffb0c013440 con 0x7ffb1c06baa0 2023-12-19T14:57:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 msgr2=0x7ffb08068ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 0x7ffb08068ad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c151f60 tx=0x7ffb10009040 comp rx=0 tx=0).stop 2023-12-19T14:57:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 msgr2=0x7ffb1c169370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 0x7ffb1c169370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c016000 tx=0x7ffb0c017000 comp rx=0 tx=0).stop 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 shutdown_connections 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb1c154780 0x7ffb1c169c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ffb080666f0 0x7ffb08068ad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb1c0adab0 0x7ffb1c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 --2- 172.21.15.50:0/2283485621 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffb1c06baa0 0x7ffb1c169370 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 >> 172.21.15.50:0/2283485621 conn(0x7ffb1c067b50 msgr2=0x7ffb1c157140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 shutdown_connections 2023-12-19T14:57:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.011+0000 7ffb1bfff700 1 -- 172.21.15.50:0/2283485621 wait complete. 2023-12-19T14:57:00.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:57:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.251+0000 7fdb5359e700 1 Processor -- start 2023-12-19T14:57:00.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.255+0000 7fdb5359e700 1 -- start start 2023-12-19T14:57:00.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.255+0000 7fdb5359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.255+0000 7fdb5359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdb5415b140 0x7fdb540b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.255+0000 7fdb5359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5406da90 con 0x7fdb540c8db0 2023-12-19T14:57:00.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5406a620 con 0x7fdb5415b140 2023-12-19T14:57:00.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540c9200 con 0x7fdb540ad9f0 2023-12-19T14:57:00.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55818/0 (socket says 172.21.15.50:55818) 2023-12-19T14:57:00.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 -- 172.21.15.50:0/1281993880 learned_addr learned my addr 172.21.15.50:0/1281993880 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:00.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb52d9d700 1 --2- 172.21.15.50:0/1281993880 >> v2:172.21.15.125:3300/0 conn(0x7fdb5415b140 0x7fdb540b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb51d9b700 1 --1- 172.21.15.50:0/1281993880 >> v1:172.21.15.125:6789/0 conn(0x7fdb540c8db0 0x7fdb5406ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60890/0 (socket says 172.21.15.50:60890) 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5159a700 1 -- 172.21.15.50:0/1281993880 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843183482 0 0) 0x7fdb5406da90 con 0x7fdb540c8db0 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5159a700 1 -- 172.21.15.50:0/1281993880 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c003570 con 0x7fdb540c8db0 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 -- 172.21.15.50:0/1281993880 >> v2:172.21.15.125:3300/0 conn(0x7fdb5415b140 msgr2=0x7fdb540b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 --2- 172.21.15.50:0/1281993880 >> v2:172.21.15.125:3300/0 conn(0x7fdb5415b140 0x7fdb540b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 -- 172.21.15.50:0/1281993880 >> v1:172.21.15.125:6789/0 conn(0x7fdb540c8db0 legacy=0x7fdb5406ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 -- 172.21.15.50:0/1281993880 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5406dd30 con 0x7fdb540ad9f0 2023-12-19T14:57:00.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5259c700 1 --2- 172.21.15.50:0/1281993880 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb440099b0 tx=0x7fdb44009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382e6200798f5292 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:00.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5159a700 1 -- 172.21.15.50:0/1281993880 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4400b680 con 0x7fdb540ad9f0 2023-12-19T14:57:00.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5159a700 1 -- 172.21.15.50:0/1281993880 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4400b7e0 con 0x7fdb540ad9f0 2023-12-19T14:57:00.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5159a700 1 -- 172.21.15.50:0/1281993880 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4400bb60 con 0x7fdb540ad9f0 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5359e700 1 -- 172.21.15.50:0/1281993880 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 msgr2=0x7fdb540addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.259+0000 7fdb5359e700 1 --2- 172.21.15.50:0/1281993880 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb440099b0 tx=0x7fdb44009250 comp rx=0 tx=0).stop 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 -- 172.21.15.50:0/1281993880 shutdown_connections 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 --2- 172.21.15.50:0/1281993880 >> v2:172.21.15.125:3300/0 conn(0x7fdb5415b140 0x7fdb540b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 --2- 172.21.15.50:0/1281993880 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540ad9f0 0x7fdb540addc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 -- 172.21.15.50:0/1281993880 >> 172.21.15.50:0/1281993880 conn(0x7fdb54067bb0 msgr2=0x7fdb540c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:00.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 -- 172.21.15.50:0/1281993880 shutdown_connections 2023-12-19T14:57:00.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 -- 172.21.15.50:0/1281993880 wait complete. 2023-12-19T14:57:00.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.263+0000 7fdb5359e700 1 Processor -- start 2023-12-19T14:57:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 -- start start 2023-12-19T14:57:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb540ad9f0 0x7fdb541398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540c8db0 0x7fdb540c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541574f0 con 0x7fdb5415b140 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540c1970 con 0x7fdb540ad9f0 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540c1c50 con 0x7fdb540c8db0 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50028/0 (socket says 172.21.15.50:50028) 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 -- 172.21.15.50:0/2659121892 learned_addr learned my addr 172.21.15.50:0/2659121892 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb51d9b700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540c8db0 0x7fdb540c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb5259c700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb540ad9f0 0x7fdb541398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 -- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540c8db0 msgr2=0x7fdb540c1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540c8db0 0x7fdb540c1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 -- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb540ad9f0 msgr2=0x7fdb541398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb540ad9f0 0x7fdb541398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.267+0000 7fdb52d9d700 1 -- 172.21.15.50:0/2659121892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb440094f0 con 0x7fdb5415b140 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb52d9d700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009df0 tx=0x7fdb4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00caa0 con 0x7fdb5415b140 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c00f040 con 0x7fdb5415b140 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb540c1f30 con 0x7fdb5415b140 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ce70 con 0x7fdb5415b140 2023-12-19T14:57:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb54164be0 con 0x7fdb5415b140 2023-12-19T14:57:00.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.271+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb54157660 con 0x7fdb5415b140 2023-12-19T14:57:00.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.275+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 48) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fdb4c014690 con 0x7fdb5415b140 2023-12-19T14:57:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.275+0000 7fdb427fc700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 0x7fdb34062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.275+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6943+0+0 (secure 0 0 0) 0x7fdb4c07d210 con 0x7fdb5415b140 2023-12-19T14:57:00.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.279+0000 7fdb5259c700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 0x7fdb34062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:00.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.279+0000 7fdb5259c700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 0x7fdb34062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdb440099b0 tx=0x7fdb44017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:00.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.283+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdb4c050fa0 con 0x7fdb5415b140 2023-12-19T14:57:00.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:00.455+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb5404be40 con 0x7fdb5415b140 2023-12-19T14:57:01.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.107+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdb4c04d8e0 con 0x7fdb5415b140 2023-12-19T14:57:01.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.111+0000 7fdb427fc700 1 -- 172.21.15.50:0/2659121892 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdb4c050d10 con 0x7fdb5415b140 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 msgr2=0x7fdb34062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 0x7fdb34062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdb440099b0 tx=0x7fdb44017000 comp rx=0 tx=0).stop 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 msgr2=0x7fdb54159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009df0 tx=0x7fdb4c00b040 comp rx=0 tx=0).stop 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 shutdown_connections 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdb340603d0 0x7fdb34062880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb5415b140 0x7fdb54159f40 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb540c8db0 0x7fdb540c1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 --2- 172.21.15.50:0/2659121892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb540ad9f0 0x7fdb541398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 >> 172.21.15.50:0/2659121892 conn(0x7fdb54067bb0 msgr2=0x7fdb540b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:01.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 shutdown_connections 2023-12-19T14:57:01.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.119+0000 7fdb5359e700 1 -- 172.21.15.50:0/2659121892 wait complete. 2023-12-19T14:57:01.121 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:57:01.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:57:01.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.315+0000 7f8f5edcc700 1 Processor -- start 2023-12-19T14:57:01.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.315+0000 7f8f5edcc700 1 -- start start 2023-12-19T14:57:01.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5edcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 0x7f8f5806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:01.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5edcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580cb3f0 con 0x7f8f580af190 2023-12-19T14:57:01.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5edcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5806d220 con 0x7f8f580aedc0 2023-12-19T14:57:01.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5edcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580cc140 con 0x7f8f580b0040 2023-12-19T14:57:01.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5d5c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f580af190 0x7f8f580af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52964/0 (socket says 172.21.15.50:52964) 2023-12-19T14:57:01.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5d5c9700 1 -- 172.21.15.50:0/943593841 learned_addr learned my addr 172.21.15.50:0/943593841 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:01.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5e5cb700 1 --2- 172.21.15.50:0/943593841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 0x7f8f5806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788161208 0 0) 0x7f8f580cb3f0 con 0x7f8f580af190 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f44003570 con 0x7f8f580af190 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142036082 0 0) 0x7f8f5806d220 con 0x7f8f580aedc0 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f580cb3f0 con 0x7f8f580aedc0 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5e5cb700 1 -- 172.21.15.50:0/943593841 >> v1:172.21.15.125:6789/0 conn(0x7f8f580aedc0 legacy=0x7f8f5806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5e5cb700 1 -- 172.21.15.50:0/943593841 >> v1:172.21.15.50:6789/0 conn(0x7f8f580af190 legacy=0x7f8f580af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5e5cb700 1 -- 172.21.15.50:0/943593841 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5806c8f0 con 0x7f8f580b0040 2023-12-19T14:57:01.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.319+0000 7f8f5e5cb700 1 --2- 172.21.15.50:0/943593841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 0x7f8f5806af50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009e90 tx=0x7f8f50009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74704b0a1edd8e92 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:01.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5000e070 con 0x7f8f580b0040 2023-12-19T14:57:01.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000bb80 con 0x7f8f580b0040 2023-12-19T14:57:01.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5cdc8700 1 -- 172.21.15.50:0/943593841 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50012430 con 0x7f8f580b0040 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 -- 172.21.15.50:0/943593841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 msgr2=0x7f8f5806af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/943593841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 0x7f8f5806af50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009e90 tx=0x7f8f50009780 comp rx=0 tx=0).stop 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 -- 172.21.15.50:0/943593841 shutdown_connections 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/943593841 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580b0040 0x7f8f5806af50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 -- 172.21.15.50:0/943593841 >> 172.21.15.50:0/943593841 conn(0x7f8f580cc760 msgr2=0x7f8f580ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 -- 172.21.15.50:0/943593841 shutdown_connections 2023-12-19T14:57:01.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 -- 172.21.15.50:0/943593841 wait complete. 2023-12-19T14:57:01.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.323+0000 7f8f5edcc700 1 Processor -- start 2023-12-19T14:57:01.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 -- start start 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 0x7f8f580b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f580b0040 0x7f8f581553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580af610 con 0x7f8f580af190 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580c75b0 con 0x7f8f580b0040 2023-12-19T14:57:01.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5edcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580c7890 con 0x7f8f580aedc0 2023-12-19T14:57:01.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50044/0 (socket says 172.21.15.50:50044) 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 -- 172.21.15.50:0/1442205881 learned_addr learned my addr 172.21.15.50:0/1442205881 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5e5cb700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f580b0040 0x7f8f581553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5ddca700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 0x7f8f580b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 -- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 msgr2=0x7f8f580b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 0x7f8f580b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 -- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f580b0040 msgr2=0x7f8f581553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f580b0040 0x7f8f581553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 -- 172.21.15.50:0/1442205881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f50009a20 con 0x7f8f580af190 2023-12-19T14:57:01.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5ddca700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 0x7f8f580b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f5d5c9700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c7c0 tx=0x7f8f48007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f48005b30 con 0x7f8f580af190 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f48005cb0 con 0x7f8f580af190 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.327+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f48007320 con 0x7f8f580af190 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.331+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f580c7b70 con 0x7f8f580af190 2023-12-19T14:57:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.331+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f580c8130 con 0x7f8f580af190 2023-12-19T14:57:01.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.331+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f580c8350 con 0x7f8f580af190 2023-12-19T14:57:01.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.331+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8f4801b3b0 con 0x7f8f580af190 2023-12-19T14:57:01.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.335+0000 7f8f4dffb700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 0x7f8f3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:01.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.335+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6505+0+0 (secure 0 0 0) 0x7f8f4807c110 con 0x7f8f580af190 2023-12-19T14:57:01.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.339+0000 7f8f5ddca700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 0x7f8f3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:01.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.339+0000 7f8f5ddca700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 0x7f8f3c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f54003990 tx=0x7f8f54009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:01.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.343+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f4804c2d0 con 0x7f8f580af190 2023-12-19T14:57:01.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:01.511+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8f5804be40 con 0x7f8f580af190 2023-12-19T14:57:02.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.123+0000 7f8f4dffb700 1 -- 172.21.15.50:0/1442205881 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8f4804f8f0 con 0x7f8f580af190 2023-12-19T14:57:02.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 msgr2=0x7f8f3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 0x7f8f3c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f54003990 tx=0x7f8f54009040 comp rx=0 tx=0).stop 2023-12-19T14:57:02.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 msgr2=0x7f8f58154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c7c0 tx=0x7f8f48007ea0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 shutdown_connections 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f580b0040 0x7f8f581553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8f3c060670 0x7f8f3c062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f580af190 0x7f8f58154e60 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 --2- 172.21.15.50:0/1442205881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f580aedc0 0x7f8f580b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 >> 172.21.15.50:0/1442205881 conn(0x7f8f580cc760 msgr2=0x7f8f580cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 shutdown_connections 2023-12-19T14:57:02.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.127+0000 7f8f5edcc700 1 -- 172.21.15.50:0/1442205881 wait complete. 2023-12-19T14:57:02.132 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:57:02.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:57:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.327+0000 7fc3ec785700 1 Processor -- start 2023-12-19T14:57:02.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.327+0000 7fc3ec785700 1 -- start start 2023-12-19T14:57:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ec785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3e40c6d20 0x7fc3e40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ec785700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e406a7a0 con 0x7fc3e406ac30 2023-12-19T14:57:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ec785700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40c7630 con 0x7fc3e406c7c0 2023-12-19T14:57:02.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ec785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e406d4b0 con 0x7fc3e40c6d20 2023-12-19T14:57:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ead22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc3e406ac30 0x7fc3e406b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52970/0 (socket says 172.21.15.50:52970) 2023-12-19T14:57:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ead22700 1 -- 172.21.15.50:0/3862920565 learned_addr learned my addr 172.21.15.50:0/3862920565 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:02.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ea521700 1 --2- 172.21.15.50:0/3862920565 >> v2:172.21.15.125:3300/0 conn(0x7fc3e40c6d20 0x7fc3e40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3e951f700 1 -- 172.21.15.50:0/3862920565 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950040177 0 0) 0x7fc3e406a7a0 con 0x7fc3e406ac30 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3e951f700 1 -- 172.21.15.50:0/3862920565 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d0003570 con 0x7fc3e406ac30 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ea521700 1 -- 172.21.15.50:0/3862920565 >> v1:172.21.15.125:6789/0 conn(0x7fc3e406c7c0 legacy=0x7fc3e406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ea521700 1 -- 172.21.15.50:0/3862920565 >> v1:172.21.15.50:6789/0 conn(0x7fc3e406ac30 legacy=0x7fc3e406b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ea521700 1 -- 172.21.15.50:0/3862920565 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e406d790 con 0x7fc3e40c6d20 2023-12-19T14:57:02.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.331+0000 7fc3ea521700 1 --2- 172.21.15.50:0/3862920565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40c70f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009e60 tx=0x7fc3e0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a448960ea1c78165 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3e951f700 1 -- 172.21.15.50:0/3862920565 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000e070 con 0x7fc3e40c6d20 2023-12-19T14:57:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3e951f700 1 -- 172.21.15.50:0/3862920565 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e000bb60 con 0x7fc3e40c6d20 2023-12-19T14:57:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3e951f700 1 -- 172.21.15.50:0/3862920565 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0012430 con 0x7fc3e40c6d20 2023-12-19T14:57:02.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 -- 172.21.15.50:0/3862920565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e40c6d20 msgr2=0x7fc3e40c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 --2- 172.21.15.50:0/3862920565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40c70f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009e60 tx=0x7fc3e0009780 comp rx=0 tx=0).stop 2023-12-19T14:57:02.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 -- 172.21.15.50:0/3862920565 shutdown_connections 2023-12-19T14:57:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 --2- 172.21.15.50:0/3862920565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40c70f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 -- 172.21.15.50:0/3862920565 >> 172.21.15.50:0/3862920565 conn(0x7fc3e4067bb0 msgr2=0x7fc3e40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 -- 172.21.15.50:0/3862920565 shutdown_connections 2023-12-19T14:57:02.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.335+0000 7fc3ec785700 1 -- 172.21.15.50:0/3862920565 wait complete. 2023-12-19T14:57:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.339+0000 7fc3ec785700 1 Processor -- start 2023-12-19T14:57:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.339+0000 7fc3ec785700 1 -- start start 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3e406ac30 0x7fc3e41440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 0x7fc3e40ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40cb3a0 con 0x7fc3e40c6d20 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40cb830 con 0x7fc3e406c7c0 2023-12-19T14:57:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40ca790 con 0x7fc3e406ac30 2023-12-19T14:57:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50048/0 (socket says 172.21.15.50:50048) 2023-12-19T14:57:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 -- 172.21.15.50:0/184258858 learned_addr learned my addr 172.21.15.50:0/184258858 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3e9d20700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 0x7fc3e40ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ea521700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3e406ac30 0x7fc3e41440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 -- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3e406ac30 msgr2=0x7fc3e41440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3e406ac30 0x7fc3e41440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 -- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 msgr2=0x7fc3e40ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 0x7fc3e40ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 -- 172.21.15.50:0/184258858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e0009a20 con 0x7fc3e40c6d20 2023-12-19T14:57:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3e9d20700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 0x7fc3e40ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ead22700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0010e0 tx=0x7fc3dc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc005c00 con 0x7fc3e40c6d20 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc005d60 con 0x7fc3e40c6d20 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0072e0 con 0x7fc3e40c6d20 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e40caa70 con 0x7fc3e40c6d20 2023-12-19T14:57:02.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.343+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e40cb030 con 0x7fc3e40c6d20 2023-12-19T14:57:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.347+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc3dc012400 con 0x7fc3e40c6d20 2023-12-19T14:57:02.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.347+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e40cb550 con 0x7fc3e40c6d20 2023-12-19T14:57:02.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.347+0000 7fc3da7fc700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 0x7fc3c8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.347+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fc3dc07bde0 con 0x7fc3e40c6d20 2023-12-19T14:57:02.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.347+0000 7fc3ea521700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 0x7fc3c8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.351+0000 7fc3ea521700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 0x7fc3c8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00097c0 tx=0x7fc3e0019000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.355+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc3dc049be0 con 0x7fc3e40c6d20 2023-12-19T14:57:02.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.527+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc3e406da70 con 0x7fc3e40c6d20 2023-12-19T14:57:02.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.535+0000 7fc3da7fc700 1 -- 172.21.15.50:0/184258858 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc3e40cb550 con 0x7fc3e40c6d20 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 msgr2=0x7fc3c8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 0x7fc3c8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00097c0 tx=0x7fc3e0019000 comp rx=0 tx=0).stop 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 msgr2=0x7fc3e40ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0010e0 tx=0x7fc3dc007dd0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 shutdown_connections 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3e406c7c0 0x7fc3e40ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc3c8060670 0x7fc3c8062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3e40c6d20 0x7fc3e40ad360 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 --2- 172.21.15.50:0/184258858 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3e406ac30 0x7fc3e41440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 >> 172.21.15.50:0/184258858 conn(0x7fc3e4067bb0 msgr2=0x7fc3e40c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:02.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 shutdown_connections 2023-12-19T14:57:02.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.539+0000 7fc3ec785700 1 -- 172.21.15.50:0/184258858 wait complete. 2023-12-19T14:57:02.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:57:02.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.735+0000 7f34fa785700 1 Processor -- start 2023-12-19T14:57:02.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.735+0000 7f34fa785700 1 -- start start 2023-12-19T14:57:02.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34fa785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f34f40cb970 0x7f34f406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34fa785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 0x7f34f406b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34fa785700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40cbdf0 con 0x7f34f4158e20 2023-12-19T14:57:02.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34fa785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f40cc0d0 con 0x7f34f40cb970 2023-12-19T14:57:02.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34fa785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f40cc3b0 con 0x7f34f40cecd0 2023-12-19T14:57:02.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f34f4158e20 0x7f34f40ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52984/0 (socket says 172.21.15.50:52984) 2023-12-19T14:57:02.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f3fff700 1 -- 172.21.15.50:0/3945714306 learned_addr learned my addr 172.21.15.50:0/3945714306 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:02.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3945714306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 0x7f34f406b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f37fe700 1 --2- 172.21.15.50:0/3945714306 >> v2:172.21.15.125:3300/0 conn(0x7f34f40cb970 0x7f34f406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f8d22700 1 -- 172.21.15.50:0/3945714306 >> v2:172.21.15.125:3300/0 conn(0x7f34f40cb970 msgr2=0x7f34f406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3945714306 >> v2:172.21.15.125:3300/0 conn(0x7f34f40cb970 0x7f34f406acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f8d22700 1 -- 172.21.15.50:0/3945714306 >> v1:172.21.15.50:6789/0 conn(0x7f34f4158e20 legacy=0x7f34f40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f2ffd700 1 -- 172.21.15.50:0/3945714306 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158017217 0 0) 0x7f34f40cbdf0 con 0x7f34f4158e20 2023-12-19T14:57:02.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.739+0000 7f34f8d22700 1 -- 172.21.15.50:0/3945714306 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f40cc690 con 0x7f34f40cecd0 2023-12-19T14:57:02.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3945714306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 0x7f34f406b200 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009e90 tx=0x7f34ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127bc8912f6a8186 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34f2ffd700 1 -- 172.21.15.50:0/3945714306 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00e070 con 0x7f34f40cecd0 2023-12-19T14:57:02.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34f2ffd700 1 -- 172.21.15.50:0/3945714306 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec00bbf0 con 0x7f34f40cecd0 2023-12-19T14:57:02.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34f2ffd700 1 -- 172.21.15.50:0/3945714306 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec012430 con 0x7f34f40cecd0 2023-12-19T14:57:02.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 -- 172.21.15.50:0/3945714306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 msgr2=0x7f34f406b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 --2- 172.21.15.50:0/3945714306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 0x7f34f406b200 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009e90 tx=0x7f34ec009780 comp rx=0 tx=0).stop 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 -- 172.21.15.50:0/3945714306 shutdown_connections 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 --2- 172.21.15.50:0/3945714306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40cecd0 0x7f34f406b200 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 --2- 172.21.15.50:0/3945714306 >> v2:172.21.15.125:3300/0 conn(0x7f34f40cb970 0x7f34f406acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 -- 172.21.15.50:0/3945714306 >> 172.21.15.50:0/3945714306 conn(0x7f34f4067bb0 msgr2=0x7f34f40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 -- 172.21.15.50:0/3945714306 shutdown_connections 2023-12-19T14:57:02.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.743+0000 7f34fa785700 1 -- 172.21.15.50:0/3945714306 wait complete. 2023-12-19T14:57:02.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 Processor -- start 2023-12-19T14:57:02.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 -- start start 2023-12-19T14:57:02.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 0x7f34f4152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34f4158e20 0x7f34f4150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4151d90 con 0x7f34f40cecd0 2023-12-19T14:57:02.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4151180 con 0x7f34f4158e20 2023-12-19T14:57:02.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34fa785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4151460 con 0x7f34f40b5a70 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 0x7f34f4152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55884/0 (socket says 172.21.15.50:55884) 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 -- 172.21.15.50:0/3070556626 learned_addr learned my addr 172.21.15.50:0/3070556626 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 0x7f34f4152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50064/0 (socket says 172.21.15.50:50064) 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f37fe700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34f4158e20 0x7f34f4150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 -- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34f4158e20 msgr2=0x7f34f4150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34f4158e20 0x7f34f4150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 -- 172.21.15.50:0/3070556626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 msgr2=0x7f34f4152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 0x7f34f4152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 -- 172.21.15.50:0/3070556626 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec009a20 con 0x7f34f40b5a70 2023-12-19T14:57:02.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.747+0000 7f34f8d22700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f34f40cc4f0 tx=0x7f34ec019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec027470 con 0x7f34f40b5a70 2023-12-19T14:57:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec0275d0 con 0x7f34f40b5a70 2023-12-19T14:57:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec027990 con 0x7f34f40b5a70 2023-12-19T14:57:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f4151740 con 0x7f34f40b5a70 2023-12-19T14:57:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f4150e70 con 0x7f34f40b5a70 2023-12-19T14:57:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.751+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f41565e0 con 0x7f34f40b5a70 2023-12-19T14:57:02.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.755+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f34ec024070 con 0x7f34f40b5a70 2023-12-19T14:57:02.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.755+0000 7f34dbfff700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 0x7f34e0062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:02.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.755+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f34ec07f240 con 0x7f34f40b5a70 2023-12-19T14:57:02.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.755+0000 7f34f3fff700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 0x7f34e0062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:02.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.759+0000 7f34f3fff700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 0x7f34e0062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34e4007fd0 tx=0x7f34e4007d30 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:02.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.763+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f34f41565e0 con 0x7f34f40b5a70 2023-12-19T14:57:02.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.939+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f34f41565e0 con 0x7f34f40b5a70 2023-12-19T14:57:02.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.951+0000 7f34dbfff700 1 -- 172.21.15.50:0/3070556626 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v38) v1 ==== 115+0+0 (secure 0 0 0) 0x7f34f41565e0 con 0x7f34f40b5a70 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.955+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 msgr2=0x7f34e0062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.955+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 0x7f34e0062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34e4007fd0 tx=0x7f34e4007d30 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.955+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 msgr2=0x7f34f414cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.955+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f34f40cc4f0 tx=0x7f34ec019000 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 shutdown_connections 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34f4158e20 0x7f34f4150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f34e0060740 0x7f34e0062bf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f34f40b5a70 0x7f34f414cb20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 --2- 172.21.15.50:0/3070556626 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34f40cecd0 0x7f34f4152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 >> 172.21.15.50:0/3070556626 conn(0x7f34f4067bb0 msgr2=0x7f34f4155ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 shutdown_connections 2023-12-19T14:57:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:02.959+0000 7f34fa785700 1 -- 172.21.15.50:0/3070556626 wait complete. 2023-12-19T14:57:02.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:57:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.171+0000 7f8b8a26b700 1 Processor -- start 2023-12-19T14:57:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.171+0000 7f8b8a26b700 1 -- start start 2023-12-19T14:57:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b8a26b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b84158c90 0x7f8b840caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b8a26b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8406c850 con 0x7f8b84159770 2023-12-19T14:57:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b8a26b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8406b470 con 0x7f8b8406aeb0 2023-12-19T14:57:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b8a26b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8406b750 con 0x7f8b84158c90 2023-12-19T14:57:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89269700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b84158c90 0x7f8b840caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89a6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8b84159770 0x7f8b8406cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52998/0 (socket says 172.21.15.50:52998) 2023-12-19T14:57:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89a6a700 1 -- 172.21.15.50:0/1681624474 learned_addr learned my addr 172.21.15.50:0/1681624474 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89269700 1 -- 172.21.15.50:0/1681624474 >> v1:172.21.15.125:6789/0 conn(0x7f8b8406aeb0 legacy=0x7f8b84159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b7bfff700 1 -- 172.21.15.50:0/1681624474 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257877977 0 0) 0x7f8b8406b470 con 0x7f8b8406aeb0 2023-12-19T14:57:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89269700 1 -- 172.21.15.50:0/1681624474 >> v1:172.21.15.50:6789/0 conn(0x7f8b84159770 legacy=0x7f8b8406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89269700 1 -- 172.21.15.50:0/1681624474 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8406ba30 con 0x7f8b84158c90 2023-12-19T14:57:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b89269700 1 --2- 172.21.15.50:0/1681624474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84158c90 0x7f8b840caf40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8b800098e0 tx=0x7f8b800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a936c149eed6094 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b7bfff700 1 -- 172.21.15.50:0/1681624474 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80009e80 con 0x7f8b84158c90 2023-12-19T14:57:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b7bfff700 1 -- 172.21.15.50:0/1681624474 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8000b760 con 0x7f8b84158c90 2023-12-19T14:57:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.175+0000 7f8b7bfff700 1 -- 172.21.15.50:0/1681624474 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8000bae0 con 0x7f8b84158c90 2023-12-19T14:57:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 -- 172.21.15.50:0/1681624474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84158c90 msgr2=0x7f8b840caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/1681624474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84158c90 0x7f8b840caf40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8b800098e0 tx=0x7f8b800091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 -- 172.21.15.50:0/1681624474 shutdown_connections 2023-12-19T14:57:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/1681624474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84158c90 0x7f8b840caf40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 -- 172.21.15.50:0/1681624474 >> 172.21.15.50:0/1681624474 conn(0x7f8b84067bb0 msgr2=0x7f8b840cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 -- 172.21.15.50:0/1681624474 shutdown_connections 2023-12-19T14:57:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 -- 172.21.15.50:0/1681624474 wait complete. 2023-12-19T14:57:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.179+0000 7f8b8a26b700 1 Processor -- start 2023-12-19T14:57:03.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 -- start start 2023-12-19T14:57:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b8406aeb0 0x7f8b84152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b84158c90 0x7f8b84154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84156c70 con 0x7f8b84159770 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84154f60 con 0x7f8b84158c90 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b8a26b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84144d60 con 0x7f8b8406aeb0 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50090/0 (socket says 172.21.15.50:50090) 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 -- 172.21.15.50:0/670310638 learned_addr learned my addr 172.21.15.50:0/670310638 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89269700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b8406aeb0 0x7f8b84152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b88a68700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b84158c90 0x7f8b84154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 -- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b8406aeb0 msgr2=0x7f8b84152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b8406aeb0 0x7f8b84152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 -- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b84158c90 msgr2=0x7f8b84154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b84158c90 0x7f8b84154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.183+0000 7f8b89a6a700 1 -- 172.21.15.50:0/670310638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b80009420 con 0x7f8b84159770 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b89a6a700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00cd50 tx=0x7f8b7c001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00ce50 con 0x7f8b84159770 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b84155200 con 0x7f8b84159770 2023-12-19T14:57:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00aac0 con 0x7f8b84159770 2023-12-19T14:57:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00ae80 con 0x7f8b84159770 2023-12-19T14:57:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b841557c0 con 0x7f8b84159770 2023-12-19T14:57:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.187+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b84156e20 con 0x7f8b84159770 2023-12-19T14:57:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.191+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8b7c007300 con 0x7f8b84159770 2023-12-19T14:57:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.191+0000 7f8b797fa700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 0x7f8b68062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.191+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f8b7c07ce90 con 0x7f8b84159770 2023-12-19T14:57:03.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.191+0000 7f8b89269700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 0x7f8b68062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.195+0000 7f8b89269700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 0x7f8b68062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b800030c0 tx=0x7f8b80016000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.199+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8b84156e20 con 0x7f8b84159770 2023-12-19T14:57:03.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.371+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f8b8404be40 con 0x7f8b84159770 2023-12-19T14:57:03.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.383+0000 7f8b797fa700 1 -- 172.21.15.50:0/670310638 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8b7c050c50 con 0x7f8b84159770 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.383+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 msgr2=0x7f8b68062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.383+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 0x7f8b68062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b800030c0 tx=0x7f8b80016000 comp rx=0 tx=0).stop 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 msgr2=0x7f8b84144820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00cd50 tx=0x7f8b7c001ab0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 shutdown_connections 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b84158c90 0x7f8b84154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b68060670 0x7f8b68062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b84159770 0x7f8b84144820 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 --2- 172.21.15.50:0/670310638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b8406aeb0 0x7f8b84152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 >> 172.21.15.50:0/670310638 conn(0x7f8b84067bb0 msgr2=0x7f8b840aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 shutdown_connections 2023-12-19T14:57:03.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.387+0000 7f8b8a26b700 1 -- 172.21.15.50:0/670310638 wait complete. 2023-12-19T14:57:03.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-12-19T14:57:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 Processor -- start 2023-12-19T14:57:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 -- start start 2023-12-19T14:57:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f50c4067c90 0x7f50c4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f50c414fba0 0x7f50c4150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c40c5900 0x7f50c40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40c6630 con 0x7f50c414fba0 2023-12-19T14:57:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40c6910 con 0x7f50c4067c90 2023-12-19T14:57:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.575+0000 7f50cb878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c40c6bf0 con 0x7f50c40c5900 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f50c414fba0 0x7f50c4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f50c414fba0 0x7f50c4150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50092/0 (socket says 172.21.15.50:50092) 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 -- 172.21.15.50:0/2930314775 learned_addr learned my addr 172.21.15.50:0/2930314775 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c9614700 1 --2- 172.21.15.50:0/2930314775 >> v2:172.21.15.125:3300/0 conn(0x7f50c4067c90 0x7f50c4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c9e15700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c40c5900 0x7f50c40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 -- 172.21.15.50:0/2930314775 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c40c5900 msgr2=0x7f50c40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c40c5900 0x7f50c40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 -- 172.21.15.50:0/2930314775 >> v2:172.21.15.125:3300/0 conn(0x7f50c4067c90 msgr2=0x7f50c4068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2930314775 >> v2:172.21.15.125:3300/0 conn(0x7f50c4067c90 0x7f50c4068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 -- 172.21.15.50:0/2930314775 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40c6ed0 con 0x7f50c414fba0 2023-12-19T14:57:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c414fba0 0x7f50c4150000 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f50c406a0c0 tx=0x7f50b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a2dbbc3587c492 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50bbfff700 1 -- 172.21.15.50:0/2930314775 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b400b860 con 0x7f50c414fba0 2023-12-19T14:57:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50bbfff700 1 -- 172.21.15.50:0/2930314775 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b400b9c0 con 0x7f50c414fba0 2023-12-19T14:57:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50bbfff700 1 -- 172.21.15.50:0/2930314775 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b400bd60 con 0x7f50c414fba0 2023-12-19T14:57:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 -- 172.21.15.50:0/2930314775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c414fba0 msgr2=0x7f50c4150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c414fba0 0x7f50c4150000 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f50c406a0c0 tx=0x7f50b4009250 comp rx=0 tx=0).stop 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 -- 172.21.15.50:0/2930314775 shutdown_connections 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c414fba0 0x7f50c4150000 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 --2- 172.21.15.50:0/2930314775 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c40c5900 0x7f50c40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 --2- 172.21.15.50:0/2930314775 >> v2:172.21.15.125:3300/0 conn(0x7f50c4067c90 0x7f50c4068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 -- 172.21.15.50:0/2930314775 >> 172.21.15.50:0/2930314775 conn(0x7f50c40cd6d0 msgr2=0x7f50c406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 -- 172.21.15.50:0/2930314775 shutdown_connections 2023-12-19T14:57:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.579+0000 7f50cb878700 1 -- 172.21.15.50:0/2930314775 wait complete. 2023-12-19T14:57:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.583+0000 7f50cb878700 1 Processor -- start 2023-12-19T14:57:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.583+0000 7f50cb878700 1 -- start start 2023-12-19T14:57:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c4067c90 0x7f50c40c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c414fba0 0x7f50c40c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c40b24d0 con 0x7f50c40c5900 2023-12-19T14:57:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c416d4d0 con 0x7f50c4067c90 2023-12-19T14:57:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c416d7b0 con 0x7f50c414fba0 2023-12-19T14:57:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50098/0 (socket says 172.21.15.50:50098) 2023-12-19T14:57:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 -- 172.21.15.50:0/2926293226 learned_addr learned my addr 172.21.15.50:0/2926293226 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c9614700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c4067c90 0x7f50c40c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c9e15700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c414fba0 0x7f50c40c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 -- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c414fba0 msgr2=0x7f50c40c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c414fba0 0x7f50c40c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 -- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c4067c90 msgr2=0x7f50c40c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c4067c90 0x7f50c40c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:03.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 -- 172.21.15.50:0/2926293226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b40094f0 con 0x7f50c40c5900 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50c8e13700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f50c406a0f0 tx=0x7f50b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4011e70 con 0x7f50c40c5900 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b4024440 con 0x7f50c40c5900 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4024800 con 0x7f50c40c5900 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c40c9410 con 0x7f50c40c5900 2023-12-19T14:57:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.587+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c40c9030 con 0x7f50c40c5900 2023-12-19T14:57:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.591+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c40c9910 con 0x7f50c40c5900 2023-12-19T14:57:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.591+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f50b401a420 con 0x7f50c40c5900 2023-12-19T14:57:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.591+0000 7f50b97fa700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 0x7f50a8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.591+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f50b407e930 con 0x7f50c40c5900 2023-12-19T14:57:03.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.591+0000 7f50c9614700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 0x7f50a8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:03.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.595+0000 7f50c9614700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 0x7f50a8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f50c0006fd0 tx=0x7f50c0008040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.599+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f50b404ece0 con 0x7f50c40c5900 2023-12-19T14:57:03.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:03.827+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f50c4071120 con 0x7f50c40c5900 2023-12-19T14:57:04.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.131+0000 7f50b97fa700 1 -- 172.21.15.50:0/2926293226 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v684) v1 ==== 183+0+0 (secure 0 0 0) 0x7f50b4052300 con 0x7f50c40c5900 2023-12-19T14:57:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 msgr2=0x7f50a8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 0x7f50a8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f50c0006fd0 tx=0x7f50c0008040 comp rx=0 tx=0).stop 2023-12-19T14:57:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 msgr2=0x7f50c416a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f50c406a0f0 tx=0x7f50b4017000 comp rx=0 tx=0).stop 2023-12-19T14:57:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 shutdown_connections 2023-12-19T14:57:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f50a8060670 0x7f50a8062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f50c414fba0 0x7f50c40c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50c40c5900 0x7f50c416a530 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 --2- 172.21.15.50:0/2926293226 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f50c4067c90 0x7f50c40c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.135+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 >> 172.21.15.50:0/2926293226 conn(0x7f50c40cd6d0 msgr2=0x7f50c40cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.139+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 shutdown_connections 2023-12-19T14:57:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.139+0000 7f50cb878700 1 -- 172.21.15.50:0/2926293226 wait complete. 2023-12-19T14:57:04.162 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-12-19T14:57:04.162 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-12-19T14:57:04.162 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:57:04.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:57:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 Processor -- start 2023-12-19T14:57:04.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 -- start start 2023-12-19T14:57:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00f00b1410 0x7f00f00ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f006a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f006c7c0 con 0x7f00f015a860 2023-12-19T14:57:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f006cd80 con 0x7f00f00b1410 2023-12-19T14:57:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00f4944700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f006af10 con 0x7f00f006a580 2023-12-19T14:57:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00ef59e700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f00f015a860 0x7f00f015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53004/0 (socket says 172.21.15.50:53004) 2023-12-19T14:57:04.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00ef59e700 1 -- 172.21.15.50:0/1514513156 learned_addr learned my addr 172.21.15.50:0/1514513156 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00eed9d700 1 --2- 172.21.15.50:0/1514513156 >> v2:172.21.15.50:3300/0 conn(0x7f00f00b1410 0x7f00f00ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.327+0000 7f00ee59c700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f006a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00eed9d700 1 -- 172.21.15.50:0/1514513156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 msgr2=0x7f00f006a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00eed9d700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f006a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00eed9d700 1 -- 172.21.15.50:0/1514513156 >> v1:172.21.15.50:6789/0 conn(0x7f00f015a860 legacy=0x7f00f015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00eed9d700 1 -- 172.21.15.50:0/1514513156 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f006b8f0 con 0x7f00f00b1410 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00eed9d700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00ab440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f00e00098e0 tx=0x7f00e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f253c05ee7bc384 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00edd9b700 1 -- 172.21.15.50:0/1514513156 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000b640 con 0x7f00f00b1410 2023-12-19T14:57:04.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00edd9b700 1 -- 172.21.15.50:0/1514513156 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e000b7a0 con 0x7f00f00b1410 2023-12-19T14:57:04.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00edd9b700 1 -- 172.21.15.50:0/1514513156 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000bb40 con 0x7f00f00b1410 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 -- 172.21.15.50:0/1514513156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 msgr2=0x7f00f00ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00ab440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f00e00098e0 tx=0x7f00e00091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 -- 172.21.15.50:0/1514513156 shutdown_connections 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f006a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 --2- 172.21.15.50:0/1514513156 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00ab440 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 -- 172.21.15.50:0/1514513156 >> 172.21.15.50:0/1514513156 conn(0x7f00f0067bb0 msgr2=0x7f00f00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 -- 172.21.15.50:0/1514513156 shutdown_connections 2023-12-19T14:57:04.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.331+0000 7f00f4944700 1 -- 172.21.15.50:0/1514513156 wait complete. 2023-12-19T14:57:04.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.335+0000 7f00f4944700 1 Processor -- start 2023-12-19T14:57:04.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.335+0000 7f00f4944700 1 -- start start 2023-12-19T14:57:04.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f0154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f015a860 0x7f00f00ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00c6940 con 0x7f00f00b1410 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00c6c20 con 0x7f00f015a860 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00c6f00 con 0x7f00f006a580 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50102/0 (socket says 172.21.15.50:50102) 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ef59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f015a860 0x7f00f00ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 -- 172.21.15.50:0/3547521437 learned_addr learned my addr 172.21.15.50:0/3547521437 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00eed9d700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f0154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 -- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 msgr2=0x7f00f0154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f0154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 -- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f015a860 msgr2=0x7f00f00ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f015a860 0x7f00f00ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 -- 172.21.15.50:0/3547521437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0009420 con 0x7f00f00b1410 2023-12-19T14:57:04.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00ee59c700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f00e400e930 tx=0x7f00e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4013680 con 0x7f00f00b1410 2023-12-19T14:57:04.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e40137e0 con 0x7f00f00b1410 2023-12-19T14:57:04.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4013b80 con 0x7f00f00b1410 2023-12-19T14:57:04.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f00c71e0 con 0x7f00f00b1410 2023-12-19T14:57:04.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.339+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f0154fa0 con 0x7f00f00b1410 2023-12-19T14:57:04.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.343+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f00c7660 con 0x7f00f00b1410 2023-12-19T14:57:04.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.343+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f00e400c3e0 con 0x7f00f00b1410 2023-12-19T14:57:04.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.347+0000 7f00d6ffd700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 0x7f00dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.347+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f00e407cb80 con 0x7f00f00b1410 2023-12-19T14:57:04.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.347+0000 7f00eed9d700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 0x7f00dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.347+0000 7f00eed9d700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 0x7f00dc062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00e00093f0 tx=0x7f00e0017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.351+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00e404ceb0 con 0x7f00f00b1410 2023-12-19T14:57:04.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.539+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00f004be40 con 0x7f00f00b1410 2023-12-19T14:57:04.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.539+0000 7f00d6ffd700 1 -- 172.21.15.50:0/3547521437 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+13522 (secure 0 0 0) 0x7f00e4017090 con 0x7f00f00b1410 2023-12-19T14:57:04.541 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:04.541 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":120,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:02.107559+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/3096035349":"2023-12-20T14:56:58.993818+0000","172.21.15.125:0/3729048763":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/4072641689":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/103015057":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/1374097553":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/1406531692":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/658284133":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/3923833256":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/738581743":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/1068162751":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3184659104":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3685761614":"2023-12-20T14:55:50.589350+0000","172.21.15.125:0/3174298071":"2023-12-20T14:56:58.654619+0000","172.21.15.125:0/2498915371":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6833/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.125:6832/33827":"2023-12-20T14:55:50.589350+0000","172.21.15.50:6833/3096035349":"2023-12-20T14:56:58.993818+0000","172.21.15.125:0/2764835356":"2023-12-20T14:55:50.589350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 msgr2=0x7f00dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 0x7f00dc062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00e00093f0 tx=0x7f00e0017000 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 msgr2=0x7f00f00c5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f00e400e930 tx=0x7f00e40091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 shutdown_connections 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f015a860 0x7f00f00ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f00dc060670 0x7f00dc062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f00b1410 0x7f00f00c5fd0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 --2- 172.21.15.50:0/3547521437 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006a580 0x7f00f0154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 >> 172.21.15.50:0/3547521437 conn(0x7f00f0067bb0 msgr2=0x7f00f015aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:04.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 shutdown_connections 2023-12-19T14:57:04.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.543+0000 7f00f4944700 1 -- 172.21.15.50:0/3547521437 wait complete. 2023-12-19T14:57:04.564 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:57:04.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:57:04.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.735+0000 7f75b1a2e700 1 Processor -- start 2023-12-19T14:57:04.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.735+0000 7f75b1a2e700 1 -- start start 2023-12-19T14:57:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75b1a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 0x7f75ac06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75b1a2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75ac06ac30 0x7f75ac06b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75b1a2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac06a7a0 con 0x7f75ac0c6d20 2023-12-19T14:57:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75b1a2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac06d4b0 con 0x7f75ac06ac30 2023-12-19T14:57:04.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75b1a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac06d790 con 0x7f75ac06c7c0 2023-12-19T14:57:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aaffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75ac0c6d20 0x7f75ac0c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53010/0 (socket says 172.21.15.50:53010) 2023-12-19T14:57:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aaffd700 1 -- 172.21.15.50:0/3296104791 learned_addr learned my addr 172.21.15.50:0/3296104791 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/3296104791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 0x7f75ac06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75ab7fe700 1 --2- 172.21.15.50:0/3296104791 >> v2:172.21.15.125:3300/0 conn(0x7f75ac06ac30 0x7f75ac06b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 -- 172.21.15.50:0/3296104791 >> v2:172.21.15.125:3300/0 conn(0x7f75ac06ac30 msgr2=0x7f75ac06b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/3296104791 >> v2:172.21.15.125:3300/0 conn(0x7f75ac06ac30 0x7f75ac06b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 -- 172.21.15.50:0/3296104791 >> v1:172.21.15.50:6789/0 conn(0x7f75ac0c6d20 legacy=0x7f75ac0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 -- 172.21.15.50:0/3296104791 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac06da70 con 0x7f75ac06c7c0 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75a9ffb700 1 -- 172.21.15.50:0/3296104791 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436310370 0 0) 0x7f75ac06a7a0 con 0x7f75ac0c6d20 2023-12-19T14:57:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/3296104791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 0x7f75ac06cc60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098e0 tx=0x7f75a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cc4a75af27d570f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75a9ffb700 1 -- 172.21.15.50:0/3296104791 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000b640 con 0x7f75ac06c7c0 2023-12-19T14:57:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75a9ffb700 1 -- 172.21.15.50:0/3296104791 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a000b7a0 con 0x7f75ac06c7c0 2023-12-19T14:57:04.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.739+0000 7f75a9ffb700 1 -- 172.21.15.50:0/3296104791 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000bb40 con 0x7f75ac06c7c0 2023-12-19T14:57:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- 172.21.15.50:0/3296104791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 msgr2=0x7f75ac06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/3296104791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 0x7f75ac06cc60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f75a00098e0 tx=0x7f75a00091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- 172.21.15.50:0/3296104791 shutdown_connections 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/3296104791 >> v2:172.21.15.125:3300/0 conn(0x7f75ac06ac30 0x7f75ac06b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/3296104791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac06c7c0 0x7f75ac06cc60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- 172.21.15.50:0/3296104791 >> 172.21.15.50:0/3296104791 conn(0x7f75ac067bb0 msgr2=0x7f75ac0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- 172.21.15.50:0/3296104791 shutdown_connections 2023-12-19T14:57:04.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- 172.21.15.50:0/3296104791 wait complete. 2023-12-19T14:57:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 Processor -- start 2023-12-19T14:57:04.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.743+0000 7f75b1a2e700 1 -- start start 2023-12-19T14:57:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 0x7f75ac0caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac0c6d20 0x7f75ac1697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0adb80 con 0x7f75ac06c7c0 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0ca1d0 con 0x7f75ac06ac30 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75b1a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac16a560 con 0x7f75ac0c6d20 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50114/0 (socket says 172.21.15.50:50114) 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 -- 172.21.15.50:0/2176846515 learned_addr learned my addr 172.21.15.50:0/2176846515 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aaffd700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 0x7f75ac0caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75ab7fe700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac0c6d20 0x7f75ac1697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 -- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac0c6d20 msgr2=0x7f75ac1697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac0c6d20 0x7f75ac1697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 -- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 msgr2=0x7f75ac0caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 0x7f75ac0caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 -- 172.21.15.50:0/2176846515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0009420 con 0x7f75ac06c7c0 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aaffd700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 0x7f75ac0caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:04.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f75aa7fc700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009f80 tx=0x7f75a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a000bb20 con 0x7f75ac06c7c0 2023-12-19T14:57:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.751+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac0c98a0 con 0x7f75ac06c7c0 2023-12-19T14:57:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.747+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a0000c90 con 0x7f75ac06c7c0 2023-12-19T14:57:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.751+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0011630 con 0x7f75ac06c7c0 2023-12-19T14:57:04.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.751+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0c9e60 con 0x7f75ac06c7c0 2023-12-19T14:57:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.751+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac0ca360 con 0x7f75ac06c7c0 2023-12-19T14:57:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.751+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f75a00119a0 con 0x7f75ac06c7c0 2023-12-19T14:57:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.755+0000 7f7592ffd700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 0x7f7598062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:04.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.755+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f75a007dd50 con 0x7f75ac06c7c0 2023-12-19T14:57:04.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.755+0000 7f75aaffd700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 0x7f7598062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:04.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.755+0000 7f75aaffd700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 0x7f7598062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0b12b0 tx=0x7f759c007f50 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.759+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f75a004e080 con 0x7f75ac06c7c0 2023-12-19T14:57:04.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.939+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75ac06dd50 con 0x7f75ac06c7c0 2023-12-19T14:57:04.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.939+0000 7f7592ffd700 1 -- 172.21.15.50:0/2176846515 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75a00516a0 con 0x7f75ac06c7c0 2023-12-19T14:57:04.942 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:57:04.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 msgr2=0x7f7598062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 0x7f7598062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0b12b0 tx=0x7f759c007f50 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 msgr2=0x7f75ac0cb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009f80 tx=0x7f75a0017000 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 shutdown_connections 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f75980606a0 0x7f7598062b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75ac0c6d20 0x7f75ac1697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75ac06c7c0 0x7f75ac0cb450 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 --2- 172.21.15.50:0/2176846515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac06ac30 0x7f75ac0caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.943+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 >> 172.21.15.50:0/2176846515 conn(0x7f75ac067bb0 msgr2=0x7f75ac157990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.947+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 shutdown_connections 2023-12-19T14:57:04.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:04.947+0000 7f75b1a2e700 1 -- 172.21.15.50:0/2176846515 wait complete. 2023-12-19T14:57:04.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-12-19T14:57:05.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.135+0000 7f0ce0878700 1 Processor -- start 2023-12-19T14:57:05.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 -- start start 2023-12-19T14:57:05.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc067c90 0x7f0cdc068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cdc14fba0 0x7f0cdc14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc0c6630 con 0x7f0cdc0c5900 2023-12-19T14:57:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc0c6440 con 0x7f0cdc14fba0 2023-12-19T14:57:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0ce0878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0c6910 con 0x7f0cdc067c90 2023-12-19T14:57:05.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0cda59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc067c90 0x7f0cdc068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0cdad9d700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0cdc0c5900 0x7f0cdc0c5d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:60932/0 (socket says 172.21.15.50:60932) 2023-12-19T14:57:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0cdad9d700 1 -- 172.21.15.50:0/3221817549 learned_addr learned my addr 172.21.15.50:0/3221817549 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.139+0000 7f0cd9d9b700 1 --2- 172.21.15.50:0/3221817549 >> v2:172.21.15.125:3300/0 conn(0x7f0cdc14fba0 0x7f0cdc14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.143+0000 7f0cd9d9b700 1 -- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc067c90 msgr2=0x7f0cdc068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.143+0000 7f0cd9d9b700 1 --2- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc067c90 0x7f0cdc068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.143+0000 7f0cd9d9b700 1 -- 172.21.15.50:0/3221817549 >> v1:172.21.15.125:6789/0 conn(0x7f0cdc0c5900 legacy=0x7f0cdc0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.143+0000 7f0cd9d9b700 1 -- 172.21.15.50:0/3221817549 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0c6bf0 con 0x7f0cdc14fba0 2023-12-19T14:57:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.143+0000 7f0cd9d9b700 1 --2- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc14fba0 0x7f0cdc14ffd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00098e0 tx=0x7f0cd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df63a23a8ed48bb0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:05.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0cd959a700 1 -- 172.21.15.50:0/3221817549 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd000b640 con 0x7f0cdc14fba0 2023-12-19T14:57:05.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0cd959a700 1 -- 172.21.15.50:0/3221817549 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd000b7a0 con 0x7f0cdc14fba0 2023-12-19T14:57:05.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0cd959a700 1 -- 172.21.15.50:0/3221817549 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd000bb40 con 0x7f0cdc14fba0 2023-12-19T14:57:05.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 -- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc14fba0 msgr2=0x7f0cdc14ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:05.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 --2- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc14fba0 0x7f0cdc14ffd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00098e0 tx=0x7f0cd00091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 -- 172.21.15.50:0/3221817549 shutdown_connections 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 --2- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc14fba0 0x7f0cdc14ffd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 --2- 172.21.15.50:0/3221817549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc067c90 0x7f0cdc068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 -- 172.21.15.50:0/3221817549 >> 172.21.15.50:0/3221817549 conn(0x7f0cdc0cd6d0 msgr2=0x7f0cdc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 -- 172.21.15.50:0/3221817549 shutdown_connections 2023-12-19T14:57:05.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.147+0000 7f0ce0878700 1 -- 172.21.15.50:0/3221817549 wait complete. 2023-12-19T14:57:05.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 Processor -- start 2023-12-19T14:57:05.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 -- start start 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cdc0c5900 0x7f0cdc1532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 0x7f0cdc0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0b3cf0 con 0x7f0cdc0c5900 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc152500 con 0x7f0cdc067c90 2023-12-19T14:57:05.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0ce0878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1527e0 con 0x7f0cdc14fba0 2023-12-19T14:57:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.151+0000 7f0cda59c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cd9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cdc0c5900 0x7f0cdc1532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:40822/0 (socket says 172.21.15.50:40822) 2023-12-19T14:57:05.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 -- 172.21.15.50:0/1309329443 learned_addr learned my addr 172.21.15.50:0/1309329443 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cdad9d700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 0x7f0cdc0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 -- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 msgr2=0x7f0cdc0c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 0x7f0cdc0c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 -- 172.21.15.50:0/1309329443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cdc0c5900 msgr2=0x7f0cdc1532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cdc0c5900 0x7f0cdc1532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 -- 172.21.15.50:0/1309329443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0009420 con 0x7f0cdc067c90 2023-12-19T14:57:05.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cdad9d700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 0x7f0cdc0c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cda59c700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e930 tx=0x7f0ccc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc013680 con 0x7f0cdc067c90 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc0137e0 con 0x7f0cdc067c90 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc013b80 con 0x7f0cdc067c90 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc152ac0 con 0x7f0cdc067c90 2023-12-19T14:57:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.155+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc152120 con 0x7f0cdc067c90 2023-12-19T14:57:05.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.159+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc071120 con 0x7f0cdc067c90 2023-12-19T14:57:05.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.159+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0ccc00c460 con 0x7f0cdc067c90 2023-12-19T14:57:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.159+0000 7f0cca7fc700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 0x7f0cbc062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:05.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.159+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f0ccc07cba0 con 0x7f0cdc067c90 2023-12-19T14:57:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.159+0000 7f0cd9d9b700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 0x7f0cbc062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.163+0000 7f0cd9d9b700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 0x7f0cbc062bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00093f0 tx=0x7f0cd0017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:05.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.167+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0ccc04cf50 con 0x7f0cdc067c90 2023-12-19T14:57:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:05.407+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f0cdc0642a0 con 0x7f0cdc067c90 2023-12-19T14:57:06.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.147+0000 7f0cca7fc700 1 -- 172.21.15.50:0/1309329443 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v686) v1 ==== 151+0+0 (secure 0 0 0) 0x7f0ccc050570 con 0x7f0cdc067c90 2023-12-19T14:57:06.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 msgr2=0x7f0cbc062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 0x7f0cbc062bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00093f0 tx=0x7f0cd0017000 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 msgr2=0x7f0cdc1501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e930 tx=0x7f0ccc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 shutdown_connections 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0cbc060740 0x7f0cbc062bf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cdc14fba0 0x7f0cdc0c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cdc0c5900 0x7f0cdc1532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 --2- 172.21.15.50:0/1309329443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cdc067c90 0x7f0cdc1501a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 >> 172.21.15.50:0/1309329443 conn(0x7f0cdc0cd6d0 msgr2=0x7f0cdc0cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 shutdown_connections 2023-12-19T14:57:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.151+0000 7f0ce0878700 1 -- 172.21.15.50:0/1309329443 wait complete. 2023-12-19T14:57:06.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:57:06.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.351+0000 7faac06ae700 1 Processor -- start 2023-12-19T14:57:06.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.351+0000 7faac06ae700 1 -- start start 2023-12-19T14:57:06.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faac06ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faab81592a0 0x7faab80cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faac06ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80ca280 con 0x7faab8158ed0 2023-12-19T14:57:06.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faac06ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab815d7c0 con 0x7faab806d050 2023-12-19T14:57:06.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faac06ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab80cca20 con 0x7faab81592a0 2023-12-19T14:57:06.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabe44a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7faab8158ed0 0x7faab815cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53018/0 (socket says 172.21.15.50:53018) 2023-12-19T14:57:06.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabe44a700 1 -- 172.21.15.50:0/3959894157 learned_addr learned my addr 172.21.15.50:0/3959894157 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:06.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabdc49700 1 --2- 172.21.15.50:0/3959894157 >> v2:172.21.15.50:3300/0 conn(0x7faab81592a0 0x7faab80cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006826054 0 0) 0x7faab80ca280 con 0x7faab8158ed0 2023-12-19T14:57:06.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa4003570 con 0x7faab8158ed0 2023-12-19T14:57:06.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222828163 0 0) 0x7faab815d7c0 con 0x7faab806d050 2023-12-19T14:57:06.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab80ca280 con 0x7faab806d050 2023-12-19T14:57:06.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabdc49700 1 -- 172.21.15.50:0/3959894157 >> v1:172.21.15.125:6789/0 conn(0x7faab806d050 legacy=0x7faab80adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabdc49700 1 -- 172.21.15.50:0/3959894157 >> v1:172.21.15.50:6789/0 conn(0x7faab8158ed0 legacy=0x7faab815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390228381 0 0) 0x7faaa4003570 con 0x7faab8158ed0 2023-12-19T14:57:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.355+0000 7faabdc49700 1 -- 172.21.15.50:0/3959894157 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab815daa0 con 0x7faab81592a0 2023-12-19T14:57:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faabdc49700 1 --2- 172.21.15.50:0/3959894157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab81592a0 0x7faab80cc4e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098e0 tx=0x7faaa80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63233fe0b1e20654 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4001470 con 0x7faab81592a0 2023-12-19T14:57:06.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa8011420 con 0x7faab81592a0 2023-12-19T14:57:06.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faabd448700 1 -- 172.21.15.50:0/3959894157 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa80117a0 con 0x7faab81592a0 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- 172.21.15.50:0/3959894157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab81592a0 msgr2=0x7faab80cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 --2- 172.21.15.50:0/3959894157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab81592a0 0x7faab80cc4e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098e0 tx=0x7faaa80091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- 172.21.15.50:0/3959894157 shutdown_connections 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 --2- 172.21.15.50:0/3959894157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab81592a0 0x7faab80cc4e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- 172.21.15.50:0/3959894157 >> 172.21.15.50:0/3959894157 conn(0x7faab80cd8f0 msgr2=0x7faab80c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:06.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- 172.21.15.50:0/3959894157 shutdown_connections 2023-12-19T14:57:06.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- 172.21.15.50:0/3959894157 wait complete. 2023-12-19T14:57:06.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 Processor -- start 2023-12-19T14:57:06.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.359+0000 7faac06ae700 1 -- start start 2023-12-19T14:57:06.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faab806d050 0x7faab8151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faab81592a0 0x7faab814b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab815f670 con 0x7faab8158ed0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab815ebe0 con 0x7faab81592a0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab815edd0 con 0x7faab806d050 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50152/0 (socket says 172.21.15.50:50152) 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 -- 172.21.15.50:0/2295611138 learned_addr learned my addr 172.21.15.50:0/2295611138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabec4b700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faab81592a0 0x7faab814b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabe44a700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faab806d050 0x7faab8151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 -- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faab806d050 msgr2=0x7faab8151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faab806d050 0x7faab8151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 -- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faab81592a0 msgr2=0x7faab814b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faab81592a0 0x7faab814b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 -- 172.21.15.50:0/2295611138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8009420 con 0x7faab8158ed0 2023-12-19T14:57:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faabdc49700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7faaa80093f0 tx=0x7faaa8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa801cbe0 con 0x7faab8158ed0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa801cd80 con 0x7faab8158ed0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa801a6a0 con 0x7faab8158ed0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab815f0b0 con 0x7faab8158ed0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.363+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab8067210 con 0x7faab8158ed0 2023-12-19T14:57:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.367+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8067430 con 0x7faab8158ed0 2023-12-19T14:57:06.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.367+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7faaa801aa50 con 0x7faab8158ed0 2023-12-19T14:57:06.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.371+0000 7faaae7fc700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 0x7faa9c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.371+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7faaa807e4e0 con 0x7faab8158ed0 2023-12-19T14:57:06.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.371+0000 7faabe44a700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 0x7faa9c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.371+0000 7faabe44a700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 0x7faa9c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faab4005c20 tx=0x7faab4000bc0 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.375+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faaa804e8b0 con 0x7faab8158ed0 2023-12-19T14:57:06.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.547+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7faab8071120 con 0x7faab8158ed0 2023-12-19T14:57:06.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.559+0000 7faaae7fc700 1 -- 172.21.15.50:0/2295611138 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7faaa8051ed0 con 0x7faab8158ed0 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 msgr2=0x7faa9c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 0x7faa9c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faab4005c20 tx=0x7faab4000bc0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 msgr2=0x7faab815e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7faaa80093f0 tx=0x7faaa8016000 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 shutdown_connections 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faab81592a0 0x7faab814b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa9c060670 0x7faa9c062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faab8158ed0 0x7faab815e6a0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 --2- 172.21.15.50:0/2295611138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faab806d050 0x7faab8151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 >> 172.21.15.50:0/2295611138 conn(0x7faab80cd8f0 msgr2=0x7faab80c2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:06.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 shutdown_connections 2023-12-19T14:57:06.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.563+0000 7faac06ae700 1 -- 172.21.15.50:0/2295611138 wait complete. 2023-12-19T14:57:06.585 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:57:06.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:57:06.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:57:06.670 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:57:06.670 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:57:06.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:57:06.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 Processor -- start 2023-12-19T14:57:06.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 -- start start 2023-12-19T14:57:06.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6806e150 con 0x7f8a68140ae0 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6806d540 con 0x7f8a68068800 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a6d945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6806d820 con 0x7f8a68140710 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a677fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a66ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a66ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:41472/0 (socket says 172.21.15.50:41472) 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a677fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32850/0 (socket says 172.21.15.50:32850) 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a66ffd700 1 -- 172.21.15.50:0/727915111 learned_addr learned my addr 172.21.15.50:0/727915111 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.847+0000 7f8a667fc700 1 --1- 172.21.15.50:0/727915111 >> v1:172.21.15.50:6789/0 conn(0x7f8a68140ae0 0x7f8a680680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55582/0 (socket says 172.21.15.50:55582) 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a66ffd700 1 -- 172.21.15.50:0/727915111 >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 msgr2=0x7f8a680ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a66ffd700 1 --2- 172.21.15.50:0/727915111 >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a66ffd700 1 -- 172.21.15.50:0/727915111 >> v1:172.21.15.50:6789/0 conn(0x7f8a68140ae0 legacy=0x7f8a680680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a66ffd700 1 -- 172.21.15.50:0/727915111 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6806db00 con 0x7f8a68140710 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a677fe700 1 --2- 172.21.15.50:0/727915111 >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a66ffd700 1 --2- 172.21.15.50:0/727915111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009f20 tx=0x7f8a58009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=632a31fac1c74fdf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a65ffb700 1 -- 172.21.15.50:0/727915111 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5800e070 con 0x7f8a68140710 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a65ffb700 1 -- 172.21.15.50:0/727915111 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5800b9d0 con 0x7f8a68140710 2023-12-19T14:57:06.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a65ffb700 1 -- 172.21.15.50:0/727915111 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580092e0 con 0x7f8a68140710 2023-12-19T14:57:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 -- 172.21.15.50:0/727915111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 msgr2=0x7f8a680c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 --2- 172.21.15.50:0/727915111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009f20 tx=0x7f8a58009840 comp rx=0 tx=0).stop 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 -- 172.21.15.50:0/727915111 shutdown_connections 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 --2- 172.21.15.50:0/727915111 >> v2:172.21.15.50:3300/0 conn(0x7f8a68068800 0x7f8a680ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 --2- 172.21.15.50:0/727915111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140710 0x7f8a680c70a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 -- 172.21.15.50:0/727915111 >> 172.21.15.50:0/727915111 conn(0x7f8a680c8850 msgr2=0x7f8a680c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 -- 172.21.15.50:0/727915111 shutdown_connections 2023-12-19T14:57:06.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.851+0000 7f8a6d945700 1 -- 172.21.15.50:0/727915111 wait complete. 2023-12-19T14:57:06.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 Processor -- start 2023-12-19T14:57:06.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 -- start start 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a68140710 0x7f8a6813eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140ae0 0x7f8a6816c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6816d850 con 0x7f8a68068800 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a68169830 con 0x7f8a68140710 2023-12-19T14:57:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a6d945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a68169260 con 0x7f8a68140ae0 2023-12-19T14:57:06.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a66ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a66ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32862/0 (socket says 172.21.15.50:32862) 2023-12-19T14:57:06.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a66ffd700 1 -- 172.21.15.50:0/1487149684 learned_addr learned my addr 172.21.15.50:0/1487149684 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:06.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a677fe700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140ae0 0x7f8a6816c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.855+0000 7f8a667fc700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a68140710 0x7f8a6813eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 -- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140ae0 msgr2=0x7f8a6816c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140ae0 0x7f8a6816c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 -- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a68140710 msgr2=0x7f8a6813eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a68140710 0x7f8a6813eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 -- 172.21.15.50:0/1487149684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60009420 con 0x7f8a68068800 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a66ffd700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800b610 tx=0x7f8a58018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580088f0 con 0x7f8a68068800 2023-12-19T14:57:06.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a58008a50 con 0x7f8a68068800 2023-12-19T14:57:06.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58008df0 con 0x7f8a68068800 2023-12-19T14:57:06.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a58009ae0 con 0x7f8a68068800 2023-12-19T14:57:06.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a681694c0 con 0x7f8a68068800 2023-12-19T14:57:06.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.859+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a68071120 con 0x7f8a68068800 2023-12-19T14:57:06.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.863+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8a58003040 con 0x7f8a68068800 2023-12-19T14:57:06.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.863+0000 7f8a4effd700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 0x7f8a54062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:06.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.863+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7f8a5807e890 con 0x7f8a68068800 2023-12-19T14:57:06.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.863+0000 7f8a667fc700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 0x7f8a54062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:06.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.867+0000 7f8a667fc700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 0x7f8a54062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8a6813e8e0 tx=0x7f8a5c007eb0 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:06.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:06.871+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a5804eab0 con 0x7f8a68068800 2023-12-19T14:57:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.075+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a680642a0 con 0x7f8a68068800 2023-12-19T14:57:07.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a4effd700 1 -- 172.21.15.50:0/1487149684 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f8a5802bde0 con 0x7f8a68068800 2023-12-19T14:57:07.079 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1475380687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6836","nonce":1475380687},{"type":"v1","addr":"172.21.15.125:6837","nonce":1475380687}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7276,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2360910106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2360910106},{"type":"v1","addr":"172.21.15.125:6835","nonce":2360910106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7374,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/474350797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":474350797},{"type":"v1","addr":"172.21.15.50:6834","nonce":474350797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7779,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3273373998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":3273373998},{"type":"v1","addr":"172.21.15.50:6835","nonce":3273373998}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[]} 2023-12-19T14:57:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 msgr2=0x7f8a54062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 0x7f8a54062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8a6813e8e0 tx=0x7f8a5c007eb0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 msgr2=0x7f8a68169cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800b610 tx=0x7f8a58018000 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 shutdown_connections 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a68140710 0x7f8a6813eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a540606a0 0x7f8a54062b50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a68140ae0 0x7f8a6816c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 --2- 172.21.15.50:0/1487149684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a68068800 0x7f8a68169cc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 >> 172.21.15.50:0/1487149684 conn(0x7f8a680c8850 msgr2=0x7f8a680aacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 shutdown_connections 2023-12-19T14:57:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.079+0000 7f8a6d945700 1 -- 172.21.15.50:0/1487149684 wait complete. 2023-12-19T14:57:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 52 2023-12-19T14:57:07.102 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:57:07.102 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:57:07.102 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:57:07.102 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:57:07.102 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:57:07.103 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:57:07.103 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:57:07.103 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:57:07.103 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:57:07.103 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:57:07.104 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:57:07.104 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:57:07.104 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:57:07.104 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:57:07.104 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:57:07.104 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:57:07.122 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:57:07.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:57:07.123 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:57:07.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:57:07.127 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:57:07.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:57:07.128 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:57:07.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:57:07.178 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:57:07.179 DEBUG:teuthology.parallel:result is None 2023-12-19T14:57:07.179 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:57:07.180 DEBUG:teuthology.parallel:result is None 2023-12-19T14:57:07.183 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:57:07.183 DEBUG:teuthology.parallel:result is None 2023-12-19T14:57:07.184 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:57:07.184 DEBUG:teuthology.parallel:result is None 2023-12-19T14:57:07.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:57:07.288 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:57:07.295 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:57:07.296 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:57:07.296 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:57:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.343+0000 7fc624af0700 1 Processor -- start 2023-12-19T14:57:07.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.343+0000 7fc624af0700 1 -- start start 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc6200b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6201407f0 0x7fc6200b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62006d7d0 con 0x7fc620140420 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6200ae280 con 0x7fc6201407f0 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cb8f0 con 0x7fc62006c7c0 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61e59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc620140420 0x7fc62015b220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55584/0 (socket says 172.21.15.50:55584) 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61e59c700 1 -- 172.21.15.50:0/2850327795 learned_addr learned my addr 172.21.15.50:0/2850327795 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:07.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2850327795 >> v2:172.21.15.50:3300/0 conn(0x7fc6201407f0 0x7fc6200b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc615d9b700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc6200b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2850327795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 msgr2=0x7fc6200b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:07.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc6200b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2850327795 >> v1:172.21.15.50:6789/0 conn(0x7fc620140420 legacy=0x7fc62015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2850327795 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62006cc50 con 0x7fc6201407f0 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc615d9b700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc6200b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc6200b1210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc6180098e0 tx=0x7fc6180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51651dda4582cac0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61dd9b700 1 -- 172.21.15.50:0/2850327795 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618009e80 con 0x7fc6201407f0 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61dd9b700 1 -- 172.21.15.50:0/2850327795 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61800b860 con 0x7fc6201407f0 2023-12-19T14:57:07.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc61dd9b700 1 -- 172.21.15.50:0/2850327795 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800bbe0 con 0x7fc6201407f0 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- 172.21.15.50:0/2850327795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 msgr2=0x7fc6200b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc6200b1210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc6180098e0 tx=0x7fc6180091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- 172.21.15.50:0/2850327795 shutdown_connections 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc6200b1210 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 --2- 172.21.15.50:0/2850327795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc6200b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.347+0000 7fc624af0700 1 -- 172.21.15.50:0/2850327795 >> 172.21.15.50:0/2850327795 conn(0x7fc620067b60 msgr2=0x7fc6200ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.351+0000 7fc624af0700 1 -- 172.21.15.50:0/2850327795 shutdown_connections 2023-12-19T14:57:07.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.351+0000 7fc624af0700 1 -- 172.21.15.50:0/2850327795 wait complete. 2023-12-19T14:57:07.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.351+0000 7fc624af0700 1 Processor -- start 2023-12-19T14:57:07.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.351+0000 7fc624af0700 1 -- start start 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc620136690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 0x7fc620158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620136bd0 con 0x7fc6201407f0 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620137f90 con 0x7fc620140420 2023-12-19T14:57:07.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc620138270 con 0x7fc62006c7c0 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32916/0 (socket says 172.21.15.50:32916) 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2297294779 learned_addr learned my addr 172.21.15.50:0/2297294779 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61e59c700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc620136690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc615d9b700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 0x7fc620158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 msgr2=0x7fc620136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc620136690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 msgr2=0x7fc620158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:07.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 0x7fc620158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:07.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 -- 172.21.15.50:0/2297294779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618009420 con 0x7fc6201407f0 2023-12-19T14:57:07.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc615d9b700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 0x7fc620158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:07.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc61ed9d700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc620069d50 tx=0x7fc618017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:07.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800bb50 con 0x7fc6201407f0 2023-12-19T14:57:07.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618011630 con 0x7fc6201407f0 2023-12-19T14:57:07.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc620138550 con 0x7fc6201407f0 2023-12-19T14:57:07.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.355+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6180119d0 con 0x7fc6201407f0 2023-12-19T14:57:07.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.359+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc620138b10 con 0x7fc6201407f0 2023-12-19T14:57:07.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.359+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc620071120 con 0x7fc6201407f0 2023-12-19T14:57:07.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.359+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc6180233f0 con 0x7fc6201407f0 2023-12-19T14:57:07.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.363+0000 7fc616ffd700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 0x7fc600062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:07.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.363+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5975+0+0 (secure 0 0 0) 0x7fc61807e0c0 con 0x7fc6201407f0 2023-12-19T14:57:07.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.363+0000 7fc61e59c700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 0x7fc600062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:07.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.363+0000 7fc61e59c700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 0x7fc600062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc6100020c0 tx=0x7fc610001a10 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:07.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.371+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc61804e510 con 0x7fc6201407f0 2023-12-19T14:57:07.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:07.559+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc62006cf30 con 0x7fc6201407f0 2023-12-19T14:57:08.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:57:08.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:57:08.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.599+0000 7fc616ffd700 1 -- 172.21.15.50:0/2297294779 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v125) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc618051b30 con 0x7fc6201407f0 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 msgr2=0x7fc600062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 0x7fc600062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc6100020c0 tx=0x7fc610001a10 comp rx=0 tx=0).stop 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 msgr2=0x7fc620155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc620069d50 tx=0x7fc618017000 comp rx=0 tx=0).stop 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 shutdown_connections 2023-12-19T14:57:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc620140420 0x7fc620158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc600060670 0x7fc600062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6201407f0 0x7fc620155ee0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 --2- 172.21.15.50:0/2297294779 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc62006c7c0 0x7fc620136690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 >> 172.21.15.50:0/2297294779 conn(0x7fc620067b60 msgr2=0x7fc6200cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 shutdown_connections 2023-12-19T14:57:08.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.603+0000 7fc624af0700 1 -- 172.21.15.50:0/2297294779 wait complete. 2023-12-19T14:57:08.607 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:57:08.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:57:08.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.803+0000 7f547abc5700 1 Processor -- start 2023-12-19T14:57:08.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.803+0000 7f547abc5700 1 -- start start 2023-12-19T14:57:08.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f547abc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f547413f890 0x7f54740aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:08.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f547abc5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547406c9f0 con 0x7f54740680f0 2023-12-19T14:57:08.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f547abc5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54740c7f20 con 0x7f54740c2980 2023-12-19T14:57:08.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f547abc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54740c1c50 con 0x7f547413f890 2023-12-19T14:57:08.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f5478961700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f547413f890 0x7f54740aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:08.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546bfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f54740c2980 0x7f54740c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33820/0 (socket says 172.21.15.50:33820) 2023-12-19T14:57:08.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546bfff700 1 -- 172.21.15.50:0/3438133933 learned_addr learned my addr 172.21.15.50:0/3438133933 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366185074 0 0) 0x7f54740c7f20 con 0x7f54740c2980 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c003570 con 0x7f54740c2980 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144205239 0 0) 0x7f547406c9f0 con 0x7f54740680f0 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f5478961700 1 -- 172.21.15.50:0/3438133933 >> v1:172.21.15.125:6789/0 conn(0x7f54740c2980 legacy=0x7f54740c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f5478961700 1 -- 172.21.15.50:0/3438133933 >> v1:172.21.15.50:6789/0 conn(0x7f54740680f0 legacy=0x7f5474068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f5478961700 1 -- 172.21.15.50:0/3438133933 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54740c8190 con 0x7f547413f890 2023-12-19T14:57:08.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f5478961700 1 --2- 172.21.15.50:0/3438133933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740aeba0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f5470009e60 tx=0x7f5470009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563a01889b8eaa00 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:08.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547000e070 con 0x7f547413f890 2023-12-19T14:57:08.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547000bb60 con 0x7f547413f890 2023-12-19T14:57:08.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.807+0000 7f546b7fe700 1 -- 172.21.15.50:0/3438133933 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470012430 con 0x7f547413f890 2023-12-19T14:57:08.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 -- 172.21.15.50:0/3438133933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 msgr2=0x7f54740aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 --2- 172.21.15.50:0/3438133933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740aeba0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f5470009e60 tx=0x7f5470009780 comp rx=0 tx=0).stop 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 -- 172.21.15.50:0/3438133933 shutdown_connections 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 --2- 172.21.15.50:0/3438133933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740aeba0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 -- 172.21.15.50:0/3438133933 >> 172.21.15.50:0/3438133933 conn(0x7f54740c8850 msgr2=0x7f54740c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 -- 172.21.15.50:0/3438133933 shutdown_connections 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 -- 172.21.15.50:0/3438133933 wait complete. 2023-12-19T14:57:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.811+0000 7f547abc5700 1 Processor -- start 2023-12-19T14:57:08.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 -- start start 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54740680f0 0x7f547416c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54740c2980 0x7f5474169250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547416c5b0 con 0x7f547413f890 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474168a70 con 0x7f54740c2980 2023-12-19T14:57:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547416c7e0 con 0x7f54740680f0 2023-12-19T14:57:08.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32942/0 (socket says 172.21.15.50:32942) 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 -- 172.21.15.50:0/4042601986 learned_addr learned my addr 172.21.15.50:0/4042601986 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5478961700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54740680f0 0x7f547416c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f546bfff700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54740c2980 0x7f5474169250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 -- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54740680f0 msgr2=0x7f547416c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54740680f0 0x7f547416c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 -- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54740c2980 msgr2=0x7f5474169250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54740c2980 0x7f5474169250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 -- 172.21.15.50:0/4042601986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470009a20 con 0x7f547413f890 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5479162700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c5d0 tx=0x7f546c007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c011590 con 0x7f547413f890 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c0116f0 con 0x7f547413f890 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547416cac0 con 0x7f547413f890 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.815+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c011ab0 con 0x7f547413f890 2023-12-19T14:57:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.819+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5474168d30 con 0x7f547413f890 2023-12-19T14:57:08.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.819+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5474071120 con 0x7f547413f890 2023-12-19T14:57:08.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.819+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f546c01b3b0 con 0x7f547413f890 2023-12-19T14:57:08.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.823+0000 7f5468ff9700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 0x7f5460062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:08.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.823+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f546c07bc50 con 0x7f547413f890 2023-12-19T14:57:08.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.823+0000 7f5478961700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 0x7f5460062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:08.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.823+0000 7f5478961700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 0x7f5460062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54700099f0 tx=0x7f5470019000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:08.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:08.827+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f546c04c280 con 0x7f547413f890 2023-12-19T14:57:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.055+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f54740642a0 con 0x7f547413f890 2023-12-19T14:57:09.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.059+0000 7f5468ff9700 1 -- 172.21.15.50:0/4042601986 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f546c017020 con 0x7f547413f890 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:57:09.060 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:57:09.063 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.065 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:57:09.067 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:57:09.073 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:57:09.074 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:57:09.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:09.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:57:09.074 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 msgr2=0x7f5460062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 0x7f5460062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54700099f0 tx=0x7f5470019000 comp rx=0 tx=0).stop 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 msgr2=0x7f54740c11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c5d0 tx=0x7f546c007d70 comp rx=0 tx=0).stop 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 shutdown_connections 2023-12-19T14:57:09.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54740c2980 0x7f5474169250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5460060670 0x7f5460062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f547413f890 0x7f54740c11b0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 --2- 172.21.15.50:0/4042601986 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54740680f0 0x7f547416c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 >> 172.21.15.50:0/4042601986 conn(0x7f54740c8850 msgr2=0x7f547406aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 shutdown_connections 2023-12-19T14:57:09.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.063+0000 7f547abc5700 1 -- 172.21.15.50:0/4042601986 wait complete. 2023-12-19T14:57:09.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:57:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.283+0000 7f110d2e3700 1 Processor -- start 2023-12-19T14:57:09.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.283+0000 7f110d2e3700 1 -- start start 2023-12-19T14:57:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110d2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 0x7f110806d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110d2e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110806c7c0 con 0x7f110806a580 2023-12-19T14:57:09.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110d2e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110806cd80 con 0x7f11080b1410 2023-12-19T14:57:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110d2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110806b8f0 con 0x7f110806aa00 2023-12-19T14:57:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f10ff7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f110806a580 0x7f110806d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55596/0 (socket says 172.21.15.50:55596) 2023-12-19T14:57:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f10ff7fe700 1 -- 172.21.15.50:0/2276378837 learned_addr learned my addr 172.21.15.50:0/2276378837 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110cae2700 1 --2- 172.21.15.50:0/2276378837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 0x7f110806d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491263527 0 0) 0x7f110806c7c0 con 0x7f110806a580 2023-12-19T14:57:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003570 con 0x7f110806a580 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970104039 0 0) 0x7f110806cd80 con 0x7f11080b1410 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110806c7c0 con 0x7f11080b1410 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110cae2700 1 -- 172.21.15.50:0/2276378837 >> v1:172.21.15.125:6789/0 conn(0x7f11080b1410 legacy=0x7f11080ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f110cae2700 1 -- 172.21.15.50:0/2276378837 >> v1:172.21.15.50:6789/0 conn(0x7f110806a580 legacy=0x7f110806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.287+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42260878 0 0) 0x7f10f4003570 con 0x7f110806a580 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110cae2700 1 -- 172.21.15.50:0/2276378837 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080ac1e0 con 0x7f110806aa00 2023-12-19T14:57:09.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110cae2700 1 --2- 172.21.15.50:0/2276378837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 0x7f110806d920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1100009e90 tx=0x7f1100009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5f06305fa2c6fbb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110000e070 con 0x7f110806aa00 2023-12-19T14:57:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110000b870 con 0x7f110806aa00 2023-12-19T14:57:09.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f11077fe700 1 -- 172.21.15.50:0/2276378837 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0001470 con 0x7f110806aa00 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 -- 172.21.15.50:0/2276378837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 msgr2=0x7f110806d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 --2- 172.21.15.50:0/2276378837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 0x7f110806d920 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1100009e90 tx=0x7f1100009780 comp rx=0 tx=0).stop 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 -- 172.21.15.50:0/2276378837 shutdown_connections 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 --2- 172.21.15.50:0/2276378837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f110806aa00 0x7f110806d920 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 -- 172.21.15.50:0/2276378837 >> 172.21.15.50:0/2276378837 conn(0x7f1108067bb0 msgr2=0x7f11080cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 -- 172.21.15.50:0/2276378837 shutdown_connections 2023-12-19T14:57:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.291+0000 7f110d2e3700 1 -- 172.21.15.50:0/2276378837 wait complete. 2023-12-19T14:57:09.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 Processor -- start 2023-12-19T14:57:09.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 -- start start 2023-12-19T14:57:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f110806a580 0x7f11080a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11080b1410 0x7f11080a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108159a60 con 0x7f110806aa00 2023-12-19T14:57:09.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11080a9e10 con 0x7f110806a580 2023-12-19T14:57:09.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.295+0000 7f110d2e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1108158e70 con 0x7f11080b1410 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32958/0 (socket says 172.21.15.50:32958) 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 -- 172.21.15.50:0/649665470 learned_addr learned my addr 172.21.15.50:0/649665470 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f110cae2700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11080b1410 0x7f11080a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f1107fff700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f110806a580 0x7f11080a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 -- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11080b1410 msgr2=0x7f11080a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11080b1410 0x7f11080a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 -- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f110806a580 msgr2=0x7f11080a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f110806a580 0x7f11080a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 -- 172.21.15.50:0/649665470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100009a20 con 0x7f110806aa00 2023-12-19T14:57:09.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f10ff7fe700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f10f000cba0 tx=0x7f10f0001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0007360 con 0x7f110806aa00 2023-12-19T14:57:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f00074e0 con 0x7f110806aa00 2023-12-19T14:57:09.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1108159130 con 0x7f110806aa00 2023-12-19T14:57:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0004830 con 0x7f110806aa00 2023-12-19T14:57:09.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.299+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11081596f0 con 0x7f110806aa00 2023-12-19T14:57:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.303+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1108159bd0 con 0x7f110806aa00 2023-12-19T14:57:09.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.303+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f10f0004bc0 con 0x7f110806aa00 2023-12-19T14:57:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.303+0000 7f1104ff9700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 0x7f10e8062c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.303+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f10f007c0a0 con 0x7f110806aa00 2023-12-19T14:57:09.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.303+0000 7f1107fff700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 0x7f10e8062c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.307+0000 7f1107fff700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 0x7f10e8062c20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10f80037e0 tx=0x7f10f8009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.311+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10f004c7d0 con 0x7f110806aa00 2023-12-19T14:57:09.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.495+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f110804be40 con 0x7f110806aa00 2023-12-19T14:57:09.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.495+0000 7f1104ff9700 1 -- 172.21.15.50:0/649665470 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+12392 (secure 0 0 0) 0x7f10f004fdf0 con 0x7f110806aa00 2023-12-19T14:57:09.497 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:09.497 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":125,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:08.591306+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 msgr2=0x7f10e8062c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 0x7f10e8062c20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10f80037e0 tx=0x7f10f8009040 comp rx=0 tx=0).stop 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 msgr2=0x7f1108159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f10f000cba0 tx=0x7f10f0001b50 comp rx=0 tx=0).stop 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 shutdown_connections 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10e8060770 0x7f10e8062c20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f11080b1410 0x7f11080a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f110806aa00 0x7f1108159ef0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 --2- 172.21.15.50:0/649665470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f110806a580 0x7f11080a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 >> 172.21.15.50:0/649665470 conn(0x7f1108067bb0 msgr2=0x7f11080ab440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 shutdown_connections 2023-12-19T14:57:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.499+0000 7f110d2e3700 1 -- 172.21.15.50:0/649665470 wait complete. 2023-12-19T14:57:09.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:57:09.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:57:09.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.747+0000 7f5caa585700 1 Processor -- start 2023-12-19T14:57:09.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.747+0000 7f5caa585700 1 -- start start 2023-12-19T14:57:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5caa585700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ca4158e20 0x7f5ca40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5caa585700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca406d5e0 con 0x7f5ca40cb970 2023-12-19T14:57:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5caa585700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40cbdf0 con 0x7f5ca40ca2c0 2023-12-19T14:57:09.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5caa585700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca40cc0d0 con 0x7f5ca4158e20 2023-12-19T14:57:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ca4158e20 0x7f5ca40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ca4158e20 0x7f5ca40ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32962/0 (socket says 172.21.15.50:32962) 2023-12-19T14:57:09.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 -- 172.21.15.50:0/3169555592 learned_addr learned my addr 172.21.15.50:0/3169555592 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca37fe700 1 -- 172.21.15.50:0/3169555592 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165289229 0 0) 0x7f5ca406d5e0 con 0x7f5ca40cb970 2023-12-19T14:57:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 -- 172.21.15.50:0/3169555592 >> v1:172.21.15.125:6789/0 conn(0x7f5ca40ca2c0 legacy=0x7f5ca406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 -- 172.21.15.50:0/3169555592 >> v1:172.21.15.50:6789/0 conn(0x7f5ca40cb970 legacy=0x7f5ca406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 -- 172.21.15.50:0/3169555592 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40cc3b0 con 0x7f5ca4158e20 2023-12-19T14:57:09.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca3fff700 1 --2- 172.21.15.50:0/3169555592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40ce5c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5c940099e0 tx=0x7f5c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8aecc2f3506d365 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca37fe700 1 -- 172.21.15.50:0/3169555592 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9400b820 con 0x7f5ca4158e20 2023-12-19T14:57:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca37fe700 1 -- 172.21.15.50:0/3169555592 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9400b980 con 0x7f5ca4158e20 2023-12-19T14:57:09.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.751+0000 7f5ca37fe700 1 -- 172.21.15.50:0/3169555592 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9400bd20 con 0x7f5ca4158e20 2023-12-19T14:57:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- 172.21.15.50:0/3169555592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 msgr2=0x7f5ca40ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 --2- 172.21.15.50:0/3169555592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40ce5c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5c940099e0 tx=0x7f5c94009250 comp rx=0 tx=0).stop 2023-12-19T14:57:09.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- 172.21.15.50:0/3169555592 shutdown_connections 2023-12-19T14:57:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 --2- 172.21.15.50:0/3169555592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40ce5c0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- 172.21.15.50:0/3169555592 >> 172.21.15.50:0/3169555592 conn(0x7f5ca4067bb0 msgr2=0x7f5ca40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- 172.21.15.50:0/3169555592 shutdown_connections 2023-12-19T14:57:09.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- 172.21.15.50:0/3169555592 wait complete. 2023-12-19T14:57:09.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 Processor -- start 2023-12-19T14:57:09.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.755+0000 7f5caa585700 1 -- start start 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ca40ca2c0 0x7f5ca40b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca40b5680 0x7f5ca40c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b3ab0 con 0x7f5ca4158e20 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b3d90 con 0x7f5ca40b5680 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b4070 con 0x7f5ca40ca2c0 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32966/0 (socket says 172.21.15.50:32966) 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 -- 172.21.15.50:0/2236418075 learned_addr learned my addr 172.21.15.50:0/2236418075 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5ca3fff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ca40ca2c0 0x7f5ca40b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5ca8b22700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca40b5680 0x7f5ca40c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 -- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ca40ca2c0 msgr2=0x7f5ca40b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ca40ca2c0 0x7f5ca40b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 -- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca40b5680 msgr2=0x7f5ca40c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca40b5680 0x7f5ca40c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 -- 172.21.15.50:0/2236418075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940094f0 con 0x7f5ca4158e20 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5c9bfff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c5d0 tx=0x7f5c8c007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c011590 con 0x7f5ca4158e20 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c011710 con 0x7f5ca4158e20 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca40b4350 con 0x7f5ca4158e20 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.759+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c011ad0 con 0x7f5ca4158e20 2023-12-19T14:57:09.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.763+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca40b5cf0 con 0x7f5ca4158e20 2023-12-19T14:57:09.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.763+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca40b4850 con 0x7f5ca4158e20 2023-12-19T14:57:09.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.763+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5c8c01b3b0 con 0x7f5ca4158e20 2023-12-19T14:57:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.767+0000 7f5ca0ff9700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 0x7f5c84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:09.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.767+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f5c8c07bc30 con 0x7f5ca4158e20 2023-12-19T14:57:09.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.767+0000 7f5ca3fff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 0x7f5c84062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:09.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.767+0000 7f5ca3fff700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 0x7f5c84062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5c940099e0 tx=0x7f5c94017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:09.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.775+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c8c04c300 con 0x7f5ca4158e20 2023-12-19T14:57:09.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.947+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ca4071120 con 0x7f5ca4158e20 2023-12-19T14:57:09.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.947+0000 7f5ca0ff9700 1 -- 172.21.15.50:0/2236418075 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v125) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c8c017020 con 0x7f5ca4158e20 2023-12-19T14:57:09.948 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:57:09.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 msgr2=0x7f5c84062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 0x7f5c84062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5c940099e0 tx=0x7f5c94017000 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 msgr2=0x7f5ca40b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c5d0 tx=0x7f5c8c007d70 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 shutdown_connections 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca40b5680 0x7f5ca40c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5c84060670 0x7f5c84062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ca4158e20 0x7f5ca40b2fb0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 --2- 172.21.15.50:0/2236418075 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ca40ca2c0 0x7f5ca40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 >> 172.21.15.50:0/2236418075 conn(0x7f5ca4067bb0 msgr2=0x7f5ca414eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 shutdown_connections 2023-12-19T14:57:09.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:09.951+0000 7f5caa585700 1 -- 172.21.15.50:0/2236418075 wait complete. 2023-12-19T14:57:09.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:57:10.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.138+0000 7fde30e0c700 1 Processor -- start 2023-12-19T14:57:10.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.138+0000 7fde30e0c700 1 -- start start 2023-12-19T14:57:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fde2c158e20 0x7fde2c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde2c1591f0 0x7fde2c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c06b730 con 0x7fde2c15ca80 2023-12-19T14:57:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c06ab20 con 0x7fde2c1591f0 2023-12-19T14:57:10.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c06ae00 con 0x7fde2c158e20 2023-12-19T14:57:10.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde2c1591f0 0x7fde2c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fde2c15ca80 0x7fde2c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55614/0 (socket says 172.21.15.50:55614) 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2bfff700 1 -- 172.21.15.50:0/2539447549 learned_addr learned my addr 172.21.15.50:0/2539447549 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2b7fe700 1 --2- 172.21.15.50:0/2539447549 >> v2:172.21.15.125:3300/0 conn(0x7fde2c158e20 0x7fde2c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 -- 172.21.15.50:0/2539447549 >> v2:172.21.15.125:3300/0 conn(0x7fde2c158e20 msgr2=0x7fde2c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 --2- 172.21.15.50:0/2539447549 >> v2:172.21.15.125:3300/0 conn(0x7fde2c158e20 0x7fde2c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 -- 172.21.15.50:0/2539447549 >> v1:172.21.15.50:6789/0 conn(0x7fde2c15ca80 legacy=0x7fde2c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 -- 172.21.15.50:0/2539447549 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c06b0e0 con 0x7fde2c1591f0 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2affd700 1 -- 172.21.15.50:0/2539447549 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047163355 0 0) 0x7fde2c06b730 con 0x7fde2c15ca80 2023-12-19T14:57:10.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde22ffd700 1 --2- 172.21.15.50:0/2539447549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c0ce100 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06e510 tx=0x7fde14009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e1aaea0db4f3fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2affd700 1 -- 172.21.15.50:0/2539447549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400e070 con 0x7fde2c1591f0 2023-12-19T14:57:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2affd700 1 -- 172.21.15.50:0/2539447549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1400bb60 con 0x7fde2c1591f0 2023-12-19T14:57:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde2affd700 1 -- 172.21.15.50:0/2539447549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14012430 con 0x7fde2c1591f0 2023-12-19T14:57:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 -- 172.21.15.50:0/2539447549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 msgr2=0x7fde2c0ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.142+0000 7fde30e0c700 1 --2- 172.21.15.50:0/2539447549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c0ce100 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fde2c06e510 tx=0x7fde14009780 comp rx=0 tx=0).stop 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 -- 172.21.15.50:0/2539447549 shutdown_connections 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 --2- 172.21.15.50:0/2539447549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c0ce100 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 --2- 172.21.15.50:0/2539447549 >> v2:172.21.15.125:3300/0 conn(0x7fde2c158e20 0x7fde2c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 -- 172.21.15.50:0/2539447549 >> 172.21.15.50:0/2539447549 conn(0x7fde2c067bb0 msgr2=0x7fde2c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 -- 172.21.15.50:0/2539447549 shutdown_connections 2023-12-19T14:57:10.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 -- 172.21.15.50:0/2539447549 wait complete. 2023-12-19T14:57:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 Processor -- start 2023-12-19T14:57:10.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.146+0000 7fde30e0c700 1 -- start start 2023-12-19T14:57:10.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c158e20 0x7fde2c150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 0x7fde2c0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c151240 con 0x7fde2c1591f0 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c15e5f0 con 0x7fde2c158e20 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c15d9e0 con 0x7fde2c15ca80 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32976/0 (socket says 172.21.15.50:32976) 2023-12-19T14:57:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 -- 172.21.15.50:0/4039541122 learned_addr learned my addr 172.21.15.50:0/4039541122 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde2b7fe700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c158e20 0x7fde2c150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde2bfff700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 0x7fde2c0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 -- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 msgr2=0x7fde2c0b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 0x7fde2c0b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 -- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c158e20 msgr2=0x7fde2c150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c158e20 0x7fde2c150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 -- 172.21.15.50:0/4039541122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14009a20 con 0x7fde2c1591f0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde2bfff700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 0x7fde2c0b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde22ffd700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fde14009fd0 tx=0x7fde14019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14012df0 con 0x7fde2c1591f0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14027470 con 0x7fde2c1591f0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c15dcc0 con 0x7fde2c1591f0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde14027830 con 0x7fde2c1591f0 2023-12-19T14:57:10.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.150+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c15e280 con 0x7fde2c1591f0 2023-12-19T14:57:10.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.154+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c04be40 con 0x7fde2c1591f0 2023-12-19T14:57:10.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.154+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fde14024070 con 0x7fde2c1591f0 2023-12-19T14:57:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.158+0000 7fde23fff700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 0x7fde0c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.158+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fde1407f170 con 0x7fde2c1591f0 2023-12-19T14:57:10.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.158+0000 7fde2b7fe700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 0x7fde0c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.158+0000 7fde2b7fe700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 0x7fde0c062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009d40 tx=0x7fde1c008040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.162+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde1404f7a0 con 0x7fde2c1591f0 2023-12-19T14:57:10.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.338+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fde2c06b3c0 con 0x7fde2c1591f0 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde23fff700 1 -- 172.21.15.50:0/4039541122 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v125) v1 ==== 84+0+65 (secure 0 0 0) 0x7fde14052dc0 con 0x7fde2c1591f0 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:57:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 msgr2=0x7fde0c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 0x7fde0c062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009d40 tx=0x7fde1c008040 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 msgr2=0x7fde2c15cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fde14009fd0 tx=0x7fde14019000 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 shutdown_connections 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde0c060670 0x7fde0c062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde2c15ca80 0x7fde2c0b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde2c1591f0 0x7fde2c15cf60 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 --2- 172.21.15.50:0/4039541122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde2c158e20 0x7fde2c150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 >> 172.21.15.50:0/4039541122 conn(0x7fde2c067bb0 msgr2=0x7fde2c0cc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:10.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 shutdown_connections 2023-12-19T14:57:10.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.342+0000 7fde30e0c700 1 -- 172.21.15.50:0/4039541122 wait complete. 2023-12-19T14:57:10.369 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:57:10.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:57:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.534+0000 7fd37de8e700 1 Processor -- start 2023-12-19T14:57:10.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.534+0000 7fd37de8e700 1 -- start start 2023-12-19T14:57:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd37de8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3780af190 0x7fd37806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd37de8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780cb3f0 con 0x7fd3780cac50 2023-12-19T14:57:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd37de8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37806d220 con 0x7fd3780aedc0 2023-12-19T14:57:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd37de8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780cc140 con 0x7fd3780af190 2023-12-19T14:57:10.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3780af190 0x7fd37806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd3780af190 0x7fd37806af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32992/0 (socket says 172.21.15.50:32992) 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 -- 172.21.15.50:0/1220911772 learned_addr learned my addr 172.21.15.50:0/1220911772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 -- 172.21.15.50:0/1220911772 >> v1:172.21.15.125:6789/0 conn(0x7fd3780aedc0 legacy=0x7fd37806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 -- 172.21.15.50:0/1220911772 >> v1:172.21.15.50:6789/0 conn(0x7fd3780cac50 legacy=0x7fd3780af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 -- 172.21.15.50:0/1220911772 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37806c8f0 con 0x7fd3780af190 2023-12-19T14:57:10.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd377fff700 1 --2- 172.21.15.50:0/1220911772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780af190 0x7fd37806af50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce0024cf9ef771f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd3777fe700 1 -- 172.21.15.50:0/1220911772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36400b860 con 0x7fd3780af190 2023-12-19T14:57:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd3777fe700 1 -- 172.21.15.50:0/1220911772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36400b9c0 con 0x7fd3780af190 2023-12-19T14:57:10.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.538+0000 7fd3777fe700 1 -- 172.21.15.50:0/1220911772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36400bd60 con 0x7fd3780af190 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 -- 172.21.15.50:0/1220911772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780af190 msgr2=0x7fd37806af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 --2- 172.21.15.50:0/1220911772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780af190 0x7fd37806af50 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fd3640099b0 tx=0x7fd364009250 comp rx=0 tx=0).stop 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 -- 172.21.15.50:0/1220911772 shutdown_connections 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 --2- 172.21.15.50:0/1220911772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780af190 0x7fd37806af50 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 -- 172.21.15.50:0/1220911772 >> 172.21.15.50:0/1220911772 conn(0x7fd3780cc760 msgr2=0x7fd3780ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 -- 172.21.15.50:0/1220911772 shutdown_connections 2023-12-19T14:57:10.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 -- 172.21.15.50:0/1220911772 wait complete. 2023-12-19T14:57:10.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.542+0000 7fd37de8e700 1 Processor -- start 2023-12-19T14:57:10.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 -- start start 2023-12-19T14:57:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3780aedc0 0x7fd3780b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd3780af190 0x7fd3780b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780af610 con 0x7fd3780cac50 2023-12-19T14:57:10.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780b5590 con 0x7fd3780aedc0 2023-12-19T14:57:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37de8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37815a400 con 0x7fd3780af190 2023-12-19T14:57:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33006/0 (socket says 172.21.15.50:33006) 2023-12-19T14:57:10.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 -- 172.21.15.50:0/2883122006 learned_addr learned my addr 172.21.15.50:0/2883122006 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd377fff700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd3780af190 0x7fd3780b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37ce8c700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3780aedc0 0x7fd3780b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 -- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd3780af190 msgr2=0x7fd3780b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd3780af190 0x7fd3780b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 -- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3780aedc0 msgr2=0x7fd3780b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3780aedc0 0x7fd3780b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.546+0000 7fd37d68d700 1 -- 172.21.15.50:0/2883122006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640094f0 con 0x7fd3780cac50 2023-12-19T14:57:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd37d68d700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c8e0 tx=0x7fd370001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370012630 con 0x7fd3780cac50 2023-12-19T14:57:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd370012790 con 0x7fd3780cac50 2023-12-19T14:57:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd370012b50 con 0x7fd3780cac50 2023-12-19T14:57:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd378147c00 con 0x7fd3780cac50 2023-12-19T14:57:10.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3780b4bf0 con 0x7fd3780cac50 2023-12-19T14:57:10.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.550+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37804be40 con 0x7fd3780cac50 2023-12-19T14:57:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.554+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd37001a3a0 con 0x7fd3780cac50 2023-12-19T14:57:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.554+0000 7fd374ff9700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 0x7fd35c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.554+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fd37007c8c0 con 0x7fd3780cac50 2023-12-19T14:57:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.554+0000 7fd37ce8c700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 0x7fd35c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.554+0000 7fd37ce8c700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 0x7fd35c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0039e0 tx=0x7fd36c007ec0 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.562+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd370050a20 con 0x7fd3780cac50 2023-12-19T14:57:10.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:10.746+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd37806cbd0 con 0x7fd3780cac50 2023-12-19T14:57:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.182+0000 7fd374ff9700 1 -- 172.21.15.50:0/2883122006 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v126) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd3700505b0 con 0x7fd3780cac50 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.186+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 msgr2=0x7fd35c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.186+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 0x7fd35c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0039e0 tx=0x7fd36c007ec0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 msgr2=0x7fd3780c54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c8e0 tx=0x7fd370001710 comp rx=0 tx=0).stop 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 shutdown_connections 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd35c060670 0x7fd35c062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd3780cac50 0x7fd3780c54e0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd3780af190 0x7fd3780b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 --2- 172.21.15.50:0/2883122006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3780aedc0 0x7fd3780b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 >> 172.21.15.50:0/2883122006 conn(0x7fd3780cc760 msgr2=0x7fd37815c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:11.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 shutdown_connections 2023-12-19T14:57:11.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.190+0000 7fd37de8e700 1 -- 172.21.15.50:0/2883122006 wait complete. 2023-12-19T14:57:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:57:11.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:57:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.394+0000 7f521660c700 1 Processor -- start 2023-12-19T14:57:11.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.394+0000 7f521660c700 1 -- start start 2023-12-19T14:57:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f521660c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f521006d760 0x7f52100ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f521660c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100cc7e0 con 0x7f5210142a50 2023-12-19T14:57:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f521660c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100cbbf0 con 0x7f5210142680 2023-12-19T14:57:11.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f521660c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100cbeb0 con 0x7f521006d760 2023-12-19T14:57:11.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5214e09700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5210142a50 0x7f521006d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55636/0 (socket says 172.21.15.50:55636) 2023-12-19T14:57:11.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5214e09700 1 -- 172.21.15.50:0/1896685685 learned_addr learned my addr 172.21.15.50:0/1896685685 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:11.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5215e0b700 1 --2- 172.21.15.50:0/1896685685 >> v2:172.21.15.125:3300/0 conn(0x7f521006d760 0x7f52100ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:11.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954814352 0 0) 0x7f52100cbbf0 con 0x7f5210142680 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc003570 con 0x7f5210142680 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029030642 0 0) 0x7f52100cc7e0 con 0x7f5210142a50 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52100cbbf0 con 0x7f5210142a50 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5215e0b700 1 -- 172.21.15.50:0/1896685685 >> v1:172.21.15.125:6789/0 conn(0x7f5210142680 legacy=0x7f52100c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5215e0b700 1 -- 172.21.15.50:0/1896685685 >> v1:172.21.15.50:6789/0 conn(0x7f5210142a50 legacy=0x7f521006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5215e0b700 1 -- 172.21.15.50:0/1896685685 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100cc190 con 0x7f521006d760 2023-12-19T14:57:11.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.398+0000 7f5215e0b700 1 --2- 172.21.15.50:0/1896685685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100ca7e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5208009e60 tx=0x7f5208009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e35d8ebc99197358 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520800e070 con 0x7f521006d760 2023-12-19T14:57:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520800bab0 con 0x7f521006d760 2023-12-19T14:57:11.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f5207fff700 1 -- 172.21.15.50:0/1896685685 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520800be70 con 0x7f521006d760 2023-12-19T14:57:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 -- 172.21.15.50:0/1896685685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 msgr2=0x7f52100ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 --2- 172.21.15.50:0/1896685685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100ca7e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5208009e60 tx=0x7f5208009780 comp rx=0 tx=0).stop 2023-12-19T14:57:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 -- 172.21.15.50:0/1896685685 shutdown_connections 2023-12-19T14:57:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 --2- 172.21.15.50:0/1896685685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100ca7e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 -- 172.21.15.50:0/1896685685 >> 172.21.15.50:0/1896685685 conn(0x7f52100ccb30 msgr2=0x7f52100ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 -- 172.21.15.50:0/1896685685 shutdown_connections 2023-12-19T14:57:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.402+0000 7f521660c700 1 -- 172.21.15.50:0/1896685685 wait complete. 2023-12-19T14:57:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 Processor -- start 2023-12-19T14:57:11.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 -- start start 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5210142680 0x7f52100ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100aa170 con 0x7f5210142a50 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100a9560 con 0x7f521006d760 2023-12-19T14:57:11.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f521660c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100a9840 con 0x7f5210142680 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f5215e0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f5215e0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33008/0 (socket says 172.21.15.50:33008) 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.406+0000 7f5215e0b700 1 -- 172.21.15.50:0/382072736 learned_addr learned my addr 172.21.15.50:0/382072736 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5214e09700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5210142680 0x7f52100ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f521560a700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 -- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5210142680 msgr2=0x7f52100ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5210142680 0x7f52100ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 -- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 msgr2=0x7f52100c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:11.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 -- 172.21.15.50:0/382072736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208009a20 con 0x7f5210142a50 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f5215e0b700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5208009460 tx=0x7f5208019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208012dd0 con 0x7f5210142a50 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208026480 con 0x7f5210142a50 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52100a9b20 con 0x7f5210142a50 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208026840 con 0x7f5210142a50 2023-12-19T14:57:11.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.410+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100c4580 con 0x7f5210142a50 2023-12-19T14:57:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.414+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52100aa320 con 0x7f5210142a50 2023-12-19T14:57:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.414+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f520801c400 con 0x7f5210142a50 2023-12-19T14:57:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.418+0000 7f52057fa700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 0x7f51f4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.418+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5571+0+0 (secure 0 0 0) 0x7f520807ef90 con 0x7f5210142a50 2023-12-19T14:57:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.418+0000 7f521560a700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 0x7f51f4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.418+0000 7f521560a700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 0x7f51f4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f520c007fd0 tx=0x7f520c007d70 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.422+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f520804f400 con 0x7f5210142a50 2023-12-19T14:57:11.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:11.594+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f521004be40 con 0x7f5210142a50 2023-12-19T14:57:12.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.194+0000 7f52057fa700 1 -- 172.21.15.50:0/382072736 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v127) v1 ==== 170+0+0 (secure 0 0 0) 0x7f520802b0d0 con 0x7f5210142a50 2023-12-19T14:57:12.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.202+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 msgr2=0x7f51f4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.202+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 0x7f51f4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f520c007fd0 tx=0x7f520c007d70 comp rx=0 tx=0).stop 2023-12-19T14:57:12.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.202+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 msgr2=0x7f52100cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.202+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5208009460 tx=0x7f5208019000 comp rx=0 tx=0).stop 2023-12-19T14:57:12.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 shutdown_connections 2023-12-19T14:57:12.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51f4060670 0x7f51f4062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5210142a50 0x7f52100cd440 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5210142680 0x7f52100ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 --2- 172.21.15.50:0/382072736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f521006d760 0x7f52100c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.210+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 >> 172.21.15.50:0/382072736 conn(0x7f52100ccb30 msgr2=0x7f52100b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.214+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 shutdown_connections 2023-12-19T14:57:12.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.214+0000 7f521660c700 1 -- 172.21.15.50:0/382072736 wait complete. 2023-12-19T14:57:12.222 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:57:12.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:57:12.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.410+0000 7fc92e0e3700 1 Processor -- start 2023-12-19T14:57:12.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.410+0000 7fc92e0e3700 1 -- start start 2023-12-19T14:57:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92e0e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc92806aeb0 0x7fc928159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92e0e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc928159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92e0e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92806c850 con 0x7fc928158c90 2023-12-19T14:57:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92e0e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9280cb650 con 0x7fc92806aeb0 2023-12-19T14:57:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92e0e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806b470 con 0x7fc9281595a0 2023-12-19T14:57:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92d0e1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc928158c90 0x7fc9280caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55642/0 (socket says 172.21.15.50:55642) 2023-12-19T14:57:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92d0e1700 1 -- 172.21.15.50:0/3819518785 learned_addr learned my addr 172.21.15.50:0/3819518785 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92c8e0700 1 --2- 172.21.15.50:0/3819518785 >> v2:172.21.15.125:3300/0 conn(0x7fc92806aeb0 0x7fc928159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc928159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360467940 0 0) 0x7fc92806c850 con 0x7fc928158c90 2023-12-19T14:57:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc914003570 con 0x7fc928158c90 2023-12-19T14:57:12.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33537560 0 0) 0x7fc914003570 con 0x7fc928158c90 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 msgr2=0x7fc928159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc928159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 >> v2:172.21.15.125:3300/0 conn(0x7fc92806aeb0 msgr2=0x7fc928159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 --2- 172.21.15.50:0/3819518785 >> v2:172.21.15.125:3300/0 conn(0x7fc92806aeb0 0x7fc928159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92806b750 con 0x7fc928158c90 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fc924001470 con 0x7fc928158c90 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 >> v1:172.21.15.50:6789/0 conn(0x7fc928158c90 legacy=0x7fc9280caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc914003a60 0x7fc914003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc914004810 0x7fc914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc9140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280cb650 con 0x7fc914003a60 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806b470 con 0x7fc914004810 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.414+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91400ba60 con 0x7fc9281595a0 2023-12-19T14:57:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc914003a60 0x7fc914003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92c8e0700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc914004810 0x7fc914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc9140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 -- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 msgr2=0x7fc9140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc9140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 -- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc914004810 msgr2=0x7fc914004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc914004810 0x7fc914004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 -- 172.21.15.50:0/3819518785 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc914003570 con 0x7fc914003a60 2023-12-19T14:57:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc9281595a0 0x7fc9140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc914003a60 0x7fc914003e70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c3f0 tx=0x7fc924001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113f7af8784a9fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92400cc80 con 0x7fc914003a60 2023-12-19T14:57:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92400cde0 con 0x7fc914003a60 2023-12-19T14:57:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc91ffff700 1 -- 172.21.15.50:0/3819518785 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924005b50 con 0x7fc914003a60 2023-12-19T14:57:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 -- 172.21.15.50:0/3819518785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc914003a60 msgr2=0x7fc914003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/3819518785 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc914003a60 0x7fc914003e70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c3f0 tx=0x7fc924001e90 comp rx=0 tx=0).stop 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92d8e2700 1 -- 172.21.15.50:0/3819518785 reap_dead start 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 -- 172.21.15.50:0/3819518785 shutdown_connections 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 -- 172.21.15.50:0/3819518785 >> 172.21.15.50:0/3819518785 conn(0x7fc928067bb0 msgr2=0x7fc9280cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 -- 172.21.15.50:0/3819518785 shutdown_connections 2023-12-19T14:57:12.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.418+0000 7fc92e0e3700 1 -- 172.21.15.50:0/3819518785 wait complete. 2023-12-19T14:57:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.422+0000 7fc92e0e3700 1 Processor -- start 2023-12-19T14:57:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.422+0000 7fc92e0e3700 1 -- start start 2023-12-19T14:57:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92814b760 0x7fc928154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc928155060 0x7fc928157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928156180 con 0x7fc92814bb30 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928156610 con 0x7fc92814b760 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc928155570 con 0x7fc928155060 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33036/0 (socket says 172.21.15.50:33036) 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 -- 172.21.15.50:0/4038707201 learned_addr learned my addr 172.21.15.50:0/4038707201 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92c8e0700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc928155060 0x7fc928157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92814b760 0x7fc928154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 -- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc928155060 msgr2=0x7fc928157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc928155060 0x7fc928157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 -- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92814b760 msgr2=0x7fc928154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92814b760 0x7fc928154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 -- 172.21.15.50:0/4038707201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928155850 con 0x7fc92814bb30 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92d8e2700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc92806e830 tx=0x7fc92000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92001b070 con 0x7fc92814bb30 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92000f040 con 0x7fc92814bb30 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc928155b30 con 0x7fc92814bb30 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92000e050 con 0x7fc92814bb30 2023-12-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.426+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280c8530 con 0x7fc92814bb30 2023-12-19T14:57:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.430+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc928156330 con 0x7fc92814bb30 2023-12-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.430+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc920003020 con 0x7fc92814bb30 2023-12-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.434+0000 7fc91d7fa700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 0x7fc90c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.434+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5953+0+0 (secure 0 0 0) 0x7fc92007dad0 con 0x7fc92814bb30 2023-12-19T14:57:12.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.434+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 0x7fc90c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:12.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.434+0000 7fc92d0e1700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 0x7fc90c062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b2b20 tx=0x7fc924015000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:12.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.438+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc92004de60 con 0x7fc92814bb30 2023-12-19T14:57:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:12.662+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc928155fb0 con 0x7fc92814bb30 2023-12-19T14:57:13.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.214+0000 7fc91d7fa700 1 -- 172.21.15.50:0/4038707201 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc920051480 con 0x7fc92814bb30 2023-12-19T14:57:13.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 msgr2=0x7fc90c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 0x7fc90c062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b2b20 tx=0x7fc924015000 comp rx=0 tx=0).stop 2023-12-19T14:57:13.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 msgr2=0x7fc928157760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc92806e830 tx=0x7fc92000b040 comp rx=0 tx=0).stop 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 shutdown_connections 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc90c0606a0 0x7fc90c062b50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc92814bb30 0x7fc928157760 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc928155060 0x7fc928157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 --2- 172.21.15.50:0/4038707201 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc92814b760 0x7fc928154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 >> 172.21.15.50:0/4038707201 conn(0x7fc928067bb0 msgr2=0x7fc9280afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 shutdown_connections 2023-12-19T14:57:13.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.222+0000 7fc92e0e3700 1 -- 172.21.15.50:0/4038707201 wait complete. 2023-12-19T14:57:13.230 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 15 and data pool 16 2023-12-19T14:57:13.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:57:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 Processor -- start 2023-12-19T14:57:13.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 -- start start 2023-12-19T14:57:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5df80b1410 0x7f5df80ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5df806aa00 0x7f5df815af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df806c7c0 con 0x7f5df806a580 2023-12-19T14:57:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df806cd80 con 0x7f5df806aa00 2023-12-19T14:57:13.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5dfcc16700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df806b8f0 con 0x7f5df80b1410 2023-12-19T14:57:13.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5df806a580 0x7f5df815a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55656/0 (socket says 172.21.15.50:55656) 2023-12-19T14:57:13.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df6ffd700 1 -- 172.21.15.50:0/2319568715 learned_addr learned my addr 172.21.15.50:0/2319568715 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:13.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df7fff700 1 --2- 172.21.15.50:0/2319568715 >> v2:172.21.15.50:3300/0 conn(0x7f5df806aa00 0x7f5df815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df77fe700 1 --2- 172.21.15.50:0/2319568715 >> v2:172.21.15.125:3300/0 conn(0x7f5df80b1410 0x7f5df80ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df67fc700 1 -- 172.21.15.50:0/2319568715 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396214611 0 0) 0x7f5df806c7c0 con 0x7f5df806a580 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df67fc700 1 -- 172.21.15.50:0/2319568715 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de0003570 con 0x7f5df806a580 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df7fff700 1 -- 172.21.15.50:0/2319568715 >> v2:172.21.15.125:3300/0 conn(0x7f5df80b1410 msgr2=0x7f5df80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.438+0000 7f5df7fff700 1 --2- 172.21.15.50:0/2319568715 >> v2:172.21.15.125:3300/0 conn(0x7f5df80b1410 0x7f5df80ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df7fff700 1 -- 172.21.15.50:0/2319568715 >> v1:172.21.15.50:6789/0 conn(0x7f5df806a580 legacy=0x7f5df815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df7fff700 1 -- 172.21.15.50:0/2319568715 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df80ac1e0 con 0x7f5df806aa00 2023-12-19T14:57:13.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df7fff700 1 --2- 172.21.15.50:0/2319568715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806aa00 0x7f5df815af70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5df8069f90 tx=0x7f5df0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02daeaf0d9b719d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df67fc700 1 -- 172.21.15.50:0/2319568715 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df000e070 con 0x7f5df806aa00 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df67fc700 1 -- 172.21.15.50:0/2319568715 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df000bab0 con 0x7f5df806aa00 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5df67fc700 1 -- 172.21.15.50:0/2319568715 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df000be30 con 0x7f5df806aa00 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 -- 172.21.15.50:0/2319568715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806aa00 msgr2=0x7f5df815af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/2319568715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806aa00 0x7f5df815af70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5df8069f90 tx=0x7f5df0009780 comp rx=0 tx=0).stop 2023-12-19T14:57:13.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 -- 172.21.15.50:0/2319568715 shutdown_connections 2023-12-19T14:57:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/2319568715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806aa00 0x7f5df815af70 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/2319568715 >> v2:172.21.15.125:3300/0 conn(0x7f5df80b1410 0x7f5df80ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 -- 172.21.15.50:0/2319568715 >> 172.21.15.50:0/2319568715 conn(0x7f5df8067bb0 msgr2=0x7f5df80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 -- 172.21.15.50:0/2319568715 shutdown_connections 2023-12-19T14:57:13.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.442+0000 7f5dfcc16700 1 -- 172.21.15.50:0/2319568715 wait complete. 2023-12-19T14:57:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 Processor -- start 2023-12-19T14:57:13.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- start start 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5df806aa00 0x7f5df815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5df80b1410 0x7f5df815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81546b0 con 0x7f5df806a580 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df815e040 con 0x7f5df80b1410 2023-12-19T14:57:13.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df815d4a0 con 0x7f5df806aa00 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33042/0 (socket says 172.21.15.50:33042) 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 -- 172.21.15.50:0/1492589142 learned_addr learned my addr 172.21.15.50:0/1492589142 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df7fff700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5df80b1410 0x7f5df815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df6ffd700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5df806aa00 0x7f5df815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 -- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5df806aa00 msgr2=0x7f5df815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5df806aa00 0x7f5df815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 -- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5df80b1410 msgr2=0x7f5df815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5df80b1410 0x7f5df815b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 -- 172.21.15.50:0/1492589142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df0009a20 con 0x7f5df806a580 2023-12-19T14:57:13.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5df77fe700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e960 tx=0x7f5de80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8013680 con 0x7f5df806a580 2023-12-19T14:57:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de80137e0 con 0x7f5df806a580 2023-12-19T14:57:13.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df815d710 con 0x7f5df806a580 2023-12-19T14:57:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8013b80 con 0x7f5df806a580 2023-12-19T14:57:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.446+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df815dcd0 con 0x7f5df806a580 2023-12-19T14:57:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.450+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df804be40 con 0x7f5df806a580 2023-12-19T14:57:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.450+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5de80123d0 con 0x7f5df806a580 2023-12-19T14:57:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.454+0000 7f5ddf7fe700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 0x7f5de4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.454+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5de807cb10 con 0x7f5df806a580 2023-12-19T14:57:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.454+0000 7f5df6ffd700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 0x7f5de4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.454+0000 7f5df6ffd700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 0x7f5de4062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5df8158500 tx=0x7f5dec009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.458+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5de804ce90 con 0x7f5df806a580 2023-12-19T14:57:13.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.630+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5df8154830 con 0x7f5df806a580 2023-12-19T14:57:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.630+0000 7f5ddf7fe700 1 -- 172.21.15.50:0/1492589142 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+15625 (secure 0 0 0) 0x7f5de8017090 con 0x7f5df806a580 2023-12-19T14:57:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:13.634 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":128,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:12.669480+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:57:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 msgr2=0x7f5de4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 0x7f5de4062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5df8158500 tx=0x7f5dec009040 comp rx=0 tx=0).stop 2023-12-19T14:57:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 msgr2=0x7f5df8154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5de800e960 tx=0x7f5de80091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 shutdown_connections 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5df80b1410 0x7f5df815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5de4060670 0x7f5de4062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5df806aa00 0x7f5df815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 --2- 172.21.15.50:0/1492589142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5df806a580 0x7f5df8154170 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 >> 172.21.15.50:0/1492589142 conn(0x7f5df8067bb0 msgr2=0x7f5df80af330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 shutdown_connections 2023-12-19T14:57:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.634+0000 7f5dfcc16700 1 -- 172.21.15.50:0/1492589142 wait complete. 2023-12-19T14:57:13.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:57:13.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 Processor -- start 2023-12-19T14:57:13.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 -- start start 2023-12-19T14:57:13.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 0x7f3554159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355406c9f0 con 0x7f35540ca010 2023-12-19T14:57:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35540ca860 con 0x7f35540ca3e0 2023-12-19T14:57:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.822+0000 7f355942e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35540c8d20 con 0x7f35540ad150 2023-12-19T14:57:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f3553fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 0x7f3554159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:41634/0 (socket says 172.21.15.50:41634) 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f3553fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 0x7f3554159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33044/0 (socket says 172.21.15.50:33044) 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 -- 172.21.15.50:0/2249753468 learned_addr learned my addr 172.21.15.50:0/2249753468 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 -- 172.21.15.50:0/2249753468 >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 msgr2=0x7f3554159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 --2- 172.21.15.50:0/2249753468 >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 0x7f3554159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 -- 172.21.15.50:0/2249753468 >> v1:172.21.15.125:6789/0 conn(0x7f35540ca010 legacy=0x7f35540c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 -- 172.21.15.50:0/2249753468 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540c8480 con 0x7f35540ad150 2023-12-19T14:57:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f354bfff700 1 --2- 172.21.15.50:0/2249753468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f353c0099b0 tx=0x7f353c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34b882c7a86e777 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f35527fc700 1 -- 172.21.15.50:0/2249753468 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c009ea0 con 0x7f35540ad150 2023-12-19T14:57:13.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f35527fc700 1 -- 172.21.15.50:0/2249753468 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c00b860 con 0x7f35540ad150 2023-12-19T14:57:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f35527fc700 1 -- 172.21.15.50:0/2249753468 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c00bc00 con 0x7f35540ad150 2023-12-19T14:57:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f355942e700 1 -- 172.21.15.50:0/2249753468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 msgr2=0x7f355406a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.826+0000 7f355942e700 1 --2- 172.21.15.50:0/2249753468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f353c0099b0 tx=0x7f353c009250 comp rx=0 tx=0).stop 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 -- 172.21.15.50:0/2249753468 shutdown_connections 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 --2- 172.21.15.50:0/2249753468 >> v2:172.21.15.50:3300/0 conn(0x7f35540ca3e0 0x7f3554159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 --2- 172.21.15.50:0/2249753468 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ad150 0x7f355406a3f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 -- 172.21.15.50:0/2249753468 >> 172.21.15.50:0/2249753468 conn(0x7f3554068500 msgr2=0x7f35540688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 -- 172.21.15.50:0/2249753468 shutdown_connections 2023-12-19T14:57:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 -- 172.21.15.50:0/2249753468 wait complete. 2023-12-19T14:57:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 Processor -- start 2023-12-19T14:57:13.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.830+0000 7f355942e700 1 -- start start 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 0x7f355413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ca3e0 0x7f35540c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355415b440 con 0x7f35540ad150 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355415a3a0 con 0x7f35540ca010 2023-12-19T14:57:13.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f355942e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355415a680 con 0x7f35540ca3e0 2023-12-19T14:57:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33050/0 (socket says 172.21.15.50:33050) 2023-12-19T14:57:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f3552ffd700 1 -- 172.21.15.50:0/3775912180 learned_addr learned my addr 172.21.15.50:0/3775912180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 0x7f355413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f3553fff700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ca3e0 0x7f35540c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 -- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ca3e0 msgr2=0x7f35540c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ca3e0 0x7f35540c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 -- 172.21.15.50:0/3775912180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 msgr2=0x7f3554159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 -- 172.21.15.50:0/3775912180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c0094f0 con 0x7f35540ca010 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f3552ffd700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:13.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.834+0000 7f354bfff700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 0x7f355413ecf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f353c013000 tx=0x7f353c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c000d10 con 0x7f35540ca010 2023-12-19T14:57:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c00bbe0 con 0x7f35540ca010 2023-12-19T14:57:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355415a960 con 0x7f35540ca010 2023-12-19T14:57:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c011670 con 0x7f35540ca010 2023-12-19T14:57:13.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355415af20 con 0x7f35540ca010 2023-12-19T14:57:13.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.838+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355415b140 con 0x7f35540ca010 2023-12-19T14:57:13.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.842+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f353c046ad0 con 0x7f35540ca010 2023-12-19T14:57:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.842+0000 7f354affd700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 0x7f3534062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.842+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f353c07e2c0 con 0x7f35540ca010 2023-12-19T14:57:13.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.842+0000 7f3552ffd700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 0x7f3534062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:13.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.846+0000 7f3552ffd700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 0x7f3534062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3544007fd0 tx=0x7f3544009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:13.850+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f353c04e6a0 con 0x7f35540ca010 2023-12-19T14:57:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.022+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3554071120 con 0x7f35540ca010 2023-12-19T14:57:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.034+0000 7f354affd700 1 -- 172.21.15.50:0/3775912180 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+15612 (secure 0 0 0) 0x7f353c051c40 con 0x7f35540ca010 2023-12-19T14:57:14.036 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:14.037 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":129,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:14.026434+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 msgr2=0x7f3534062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 0x7f3534062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3544007fd0 tx=0x7f3544009040 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 msgr2=0x7f355413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 0x7f355413ecf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f353c013000 tx=0x7f353c017000 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 shutdown_connections 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35540ca010 0x7f355413ecf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3534060670 0x7f3534062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35540ca3e0 0x7f35540c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 --2- 172.21.15.50:0/3775912180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35540ad150 0x7f3554159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 >> 172.21.15.50:0/3775912180 conn(0x7f3554068500 msgr2=0x7f35540c2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 shutdown_connections 2023-12-19T14:57:14.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.038+0000 7f355942e700 1 -- 172.21.15.50:0/3775912180 wait complete. 2023-12-19T14:57:14.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:57:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.246+0000 7f57bc8e3700 1 Processor -- start 2023-12-19T14:57:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.246+0000 7f57bc8e3700 1 -- start start 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.246+0000 7f57bc8e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f57b806c7c0 0x7f57b815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b815da20 con 0x7f57b806b420 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b806da40 con 0x7f57b806b050 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b806cc00 con 0x7f57b806c7c0 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b77fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f57b806b050 0x7f57b81594b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33872/0 (socket says 172.21.15.50:33872) 2023-12-19T14:57:14.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b77fe700 1 -- 172.21.15.50:0/2975461671 learned_addr learned my addr 172.21.15.50:0/2975461671 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b7fff700 1 --2- 172.21.15.50:0/2975461671 >> v2:172.21.15.50:3300/0 conn(0x7f57b806c7c0 0x7f57b815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b7fff700 1 -- 172.21.15.50:0/2975461671 >> v1:172.21.15.125:6789/0 conn(0x7f57b806b050 legacy=0x7f57b81594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b7fff700 1 -- 172.21.15.50:0/2975461671 >> v1:172.21.15.50:6789/0 conn(0x7f57b806b420 legacy=0x7f57b8159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b67fc700 1 -- 172.21.15.50:0/2975461671 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773886411 0 0) 0x7f57b806da40 con 0x7f57b806b050 2023-12-19T14:57:14.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b7fff700 1 -- 172.21.15.50:0/2975461671 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b806cec0 con 0x7f57b806c7c0 2023-12-19T14:57:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b7fff700 1 --2- 172.21.15.50:0/2975461671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806c7c0 0x7f57b815d4e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e60 tx=0x7f57b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc7d2c1889fc4d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b67fc700 1 -- 172.21.15.50:0/2975461671 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b000e070 con 0x7f57b806c7c0 2023-12-19T14:57:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b67fc700 1 -- 172.21.15.50:0/2975461671 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b000bc30 con 0x7f57b806c7c0 2023-12-19T14:57:14.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57b67fc700 1 -- 172.21.15.50:0/2975461671 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0012430 con 0x7f57b806c7c0 2023-12-19T14:57:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2975461671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806c7c0 msgr2=0x7f57b815d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2975461671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806c7c0 0x7f57b815d4e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e60 tx=0x7f57b0009780 comp rx=0 tx=0).stop 2023-12-19T14:57:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2975461671 shutdown_connections 2023-12-19T14:57:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2975461671 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806c7c0 0x7f57b815d4e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2975461671 >> 172.21.15.50:0/2975461671 conn(0x7f57b80c8cd0 msgr2=0x7f57b80c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2975461671 shutdown_connections 2023-12-19T14:57:14.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.250+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2975461671 wait complete. 2023-12-19T14:57:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 Processor -- start 2023-12-19T14:57:14.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 -- start start 2023-12-19T14:57:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b806b050 0x7f57b8155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f57b806c7c0 0x7f57b80aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:14.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b816da90 con 0x7f57b806b420 2023-12-19T14:57:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80b0840 con 0x7f57b806b050 2023-12-19T14:57:14.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57bc8e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b816cba0 con 0x7f57b806c7c0 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57b6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57b6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33064/0 (socket says 172.21.15.50:33064) 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57b6ffd700 1 -- 172.21.15.50:0/2852096631 learned_addr learned my addr 172.21.15.50:0/2852096631 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57b7fff700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f57b806c7c0 0x7f57b80aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.254+0000 7f57b77fe700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b806b050 0x7f57b8155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 -- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f57b806c7c0 msgr2=0x7f57b80aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f57b806c7c0 0x7f57b80aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 -- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b806b050 msgr2=0x7f57b8155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b806b050 0x7f57b8155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 -- 172.21.15.50:0/2852096631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0009a20 con 0x7f57b806b420 2023-12-19T14:57:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57b6ffd700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c590 tx=0x7f57ac00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac011590 con 0x7f57b806b420 2023-12-19T14:57:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b816ce80 con 0x7f57b806b420 2023-12-19T14:57:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac011710 con 0x7f57b806b420 2023-12-19T14:57:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac011ad0 con 0x7f57b806b420 2023-12-19T14:57:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b816d440 con 0x7f57b806b420 2023-12-19T14:57:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.258+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b804be40 con 0x7f57b806b420 2023-12-19T14:57:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.262+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f57ac01b3b0 con 0x7f57b806b420 2023-12-19T14:57:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.262+0000 7f579f7fe700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 0x7f57a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.262+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f57ac07bff0 con 0x7f57b806b420 2023-12-19T14:57:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.262+0000 7f57b77fe700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 0x7f57a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.262+0000 7f57b77fe700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 0x7f57a4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57a8007fd0 tx=0x7f57a8007d90 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:14.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.270+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f57ac0abbe0 con 0x7f57b806b420 2023-12-19T14:57:14.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:14.478+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f57b806d1a0 con 0x7f57b806b420 2023-12-19T14:57:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.238+0000 7f579f7fe700 1 -- 172.21.15.50:0/2852096631 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v57) v1 ==== 118+0+0 (secure 0 0 0) 0x7f57ac017020 con 0x7f57b806b420 2023-12-19T14:57:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 msgr2=0x7f57a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 0x7f57a4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57a8007fd0 tx=0x7f57a8007d90 comp rx=0 tx=0).stop 2023-12-19T14:57:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 msgr2=0x7f57b80b0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00c590 tx=0x7f57ac00c8a0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 shutdown_connections 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f57a4060670 0x7f57a4062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f57b806c7c0 0x7f57b80aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f57b806b420 0x7f57b80b0300 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 --2- 172.21.15.50:0/2852096631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57b806b050 0x7f57b8155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 >> 172.21.15.50:0/2852096631 conn(0x7f57b80c8cd0 msgr2=0x7f57b80c8330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 shutdown_connections 2023-12-19T14:57:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.242+0000 7f57bc8e3700 1 -- 172.21.15.50:0/2852096631 wait complete. 2023-12-19T14:57:15.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:57:15.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.442+0000 7f2e87416700 1 Processor -- start 2023-12-19T14:57:15.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.442+0000 7f2e87416700 1 -- start start 2023-12-19T14:57:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e87416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 0x7f2e8015d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e87416700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e800c7820 con 0x7f2e800ac170 2023-12-19T14:57:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e87416700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e800ac5c0 con 0x7f2e8006ae20 2023-12-19T14:57:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e87416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e800c6c10 con 0x7f2e8015d140 2023-12-19T14:57:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85c13700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2e800ac170 0x7f2e8006d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55680/0 (socket says 172.21.15.50:55680) 2023-12-19T14:57:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85c13700 1 -- 172.21.15.50:0/1817173294 learned_addr learned my addr 172.21.15.50:0/1817173294 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:15.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e86414700 1 --2- 172.21.15.50:0/1817173294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 0x7f2e8015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718733972 0 0) 0x7f2e800c7820 con 0x7f2e800ac170 2023-12-19T14:57:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c003570 con 0x7f2e800ac170 2023-12-19T14:57:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259757233 0 0) 0x7f2e800ac5c0 con 0x7f2e8006ae20 2023-12-19T14:57:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e800c7820 con 0x7f2e8006ae20 2023-12-19T14:57:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e86414700 1 -- 172.21.15.50:0/1817173294 >> v1:172.21.15.125:6789/0 conn(0x7f2e8006ae20 legacy=0x7f2e800c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e86414700 1 -- 172.21.15.50:0/1817173294 >> v1:172.21.15.50:6789/0 conn(0x7f2e800ac170 legacy=0x7f2e8006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.446+0000 7f2e86414700 1 -- 172.21.15.50:0/1817173294 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e800c6ef0 con 0x7f2e8015d140 2023-12-19T14:57:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e86414700 1 --2- 172.21.15.50:0/1817173294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 0x7f2e8015d510 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009910 tx=0x7f2e7c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=528c4081218e038f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c011630 con 0x7f2e8015d140 2023-12-19T14:57:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c011790 con 0x7f2e8015d140 2023-12-19T14:57:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e85412700 1 -- 172.21.15.50:0/1817173294 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c011b30 con 0x7f2e8015d140 2023-12-19T14:57:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 -- 172.21.15.50:0/1817173294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 msgr2=0x7f2e8015d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 --2- 172.21.15.50:0/1817173294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 0x7f2e8015d510 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009910 tx=0x7f2e7c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 -- 172.21.15.50:0/1817173294 shutdown_connections 2023-12-19T14:57:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 --2- 172.21.15.50:0/1817173294 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e8015d140 0x7f2e8015d510 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 -- 172.21.15.50:0/1817173294 >> 172.21.15.50:0/1817173294 conn(0x7f2e800c8cd0 msgr2=0x7f2e800c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 -- 172.21.15.50:0/1817173294 shutdown_connections 2023-12-19T14:57:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.450+0000 7f2e87416700 1 -- 172.21.15.50:0/1817173294 wait complete. 2023-12-19T14:57:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 Processor -- start 2023-12-19T14:57:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 -- start start 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8006ae20 0x7f2e8016e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e800ac170 0x7f2e80171520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8016eea0 con 0x7f2e8015d140 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8015ee00 con 0x7f2e8006ae20 2023-12-19T14:57:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e87416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8015eb00 con 0x7f2e800ac170 2023-12-19T14:57:15.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e86c15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:15.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e86c15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33066/0 (socket says 172.21.15.50:33066) 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e86c15700 1 -- 172.21.15.50:0/3786975549 learned_addr learned my addr 172.21.15.50:0/3786975549 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.454+0000 7f2e85c13700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e800ac170 0x7f2e80171520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86414700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8006ae20 0x7f2e8016e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 -- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e800ac170 msgr2=0x7f2e80171520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e800ac170 0x7f2e80171520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 -- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8006ae20 msgr2=0x7f2e8016e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8006ae20 0x7f2e8016e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 -- 172.21.15.50:0/3786975549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c009420 con 0x7f2e8015d140 2023-12-19T14:57:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e86c15700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c970 tx=0x7f2e780027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78012630 con 0x7f2e8015d140 2023-12-19T14:57:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e780127b0 con 0x7f2e8015d140 2023-12-19T14:57:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e80171d70 con 0x7f2e8015d140 2023-12-19T14:57:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78012b70 con 0x7f2e8015d140 2023-12-19T14:57:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e80171a60 con 0x7f2e8015d140 2023-12-19T14:57:15.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.458+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8015ef90 con 0x7f2e8015d140 2023-12-19T14:57:15.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.462+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2e7801b400 con 0x7f2e8015d140 2023-12-19T14:57:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.462+0000 7f2e767fc700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 0x7f2e64062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.462+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f2e7807cd20 con 0x7f2e8015d140 2023-12-19T14:57:15.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.466+0000 7f2e86414700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 0x7f2e64062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:15.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.466+0000 7f2e86414700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 0x7f2e64062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0093f0 tx=0x7f2e7c017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:15.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.470+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e7804d0a0 con 0x7f2e8015d140 2023-12-19T14:57:15.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:15.674+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2e8004be40 con 0x7f2e8015d140 2023-12-19T14:57:16.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.238+0000 7f2e767fc700 1 -- 172.21.15.50:0/3786975549 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2e780160c0 con 0x7f2e8015d140 2023-12-19T14:57:16.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 msgr2=0x7f2e64062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 0x7f2e64062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c0093f0 tx=0x7f2e7c017000 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 msgr2=0x7f2e8015f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c970 tx=0x7f2e780027a0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 shutdown_connections 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2e64060670 0x7f2e64062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e8015d140 0x7f2e8015f290 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e800ac170 0x7f2e80171520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 --2- 172.21.15.50:0/3786975549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8006ae20 0x7f2e8016e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.242+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 >> 172.21.15.50:0/3786975549 conn(0x7f2e800c8cd0 msgr2=0x7f2e800c90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.246+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 shutdown_connections 2023-12-19T14:57:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.246+0000 7f2e87416700 1 -- 172.21.15.50:0/3786975549 wait complete. 2023-12-19T14:57:16.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:57:16.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.430+0000 7f7ad9b85700 1 Processor -- start 2023-12-19T14:57:16.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.430+0000 7f7ad9b85700 1 -- start start 2023-12-19T14:57:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad9b85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ad4140710 0x7f7ad40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad9b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad9b85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad406e150 con 0x7f7ad4140ae0 2023-12-19T14:57:16.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad9b85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad406d540 con 0x7f7ad4140710 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad9b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad406d820 con 0x7f7ad4068800 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ad4140710 0x7f7ad40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ad4140710 0x7f7ad40c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56138/0 (socket says 172.21.15.50:56138) 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39446/0 (socket says 172.21.15.50:39446) 2023-12-19T14:57:16.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 -- 172.21.15.50:0/2545425184 learned_addr learned my addr 172.21.15.50:0/2545425184 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:16.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 -- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 msgr2=0x7f7ad40ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:16.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 -- 172.21.15.50:0/2545425184 >> v1:172.21.15.125:6789/0 conn(0x7f7ad4140ae0 legacy=0x7f7ad40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad37fe700 1 -- 172.21.15.50:0/2545425184 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad406db00 con 0x7f7ad4140710 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.434+0000 7f7ad3fff700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad37fe700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4140710 0x7f7ad40c70a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009f20 tx=0x7f7ac4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f896c75f03b24b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad2ffd700 1 -- 172.21.15.50:0/2545425184 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400e070 con 0x7f7ad4140710 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad2ffd700 1 -- 172.21.15.50:0/2545425184 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac400b9d0 con 0x7f7ad4140710 2023-12-19T14:57:16.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad2ffd700 1 -- 172.21.15.50:0/2545425184 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40092e0 con 0x7f7ad4140710 2023-12-19T14:57:16.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 -- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4140710 msgr2=0x7f7ad40c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4140710 0x7f7ad40c70a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009f20 tx=0x7f7ac4009840 comp rx=0 tx=0).stop 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 -- 172.21.15.50:0/2545425184 shutdown_connections 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4068800 0x7f7ad40ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/2545425184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4140710 0x7f7ad40c70a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 -- 172.21.15.50:0/2545425184 >> 172.21.15.50:0/2545425184 conn(0x7f7ad40c8850 msgr2=0x7f7ad40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 -- 172.21.15.50:0/2545425184 shutdown_connections 2023-12-19T14:57:16.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.438+0000 7f7ad9b85700 1 -- 172.21.15.50:0/2545425184 wait complete. 2023-12-19T14:57:16.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 Processor -- start 2023-12-19T14:57:16.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 -- start start 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4068800 0x7f7ad413eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4140710 0x7f7ad4157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40b1a20 con 0x7f7ad4140ae0 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40aefb0 con 0x7f7ad4068800 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad9b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40b13b0 con 0x7f7ad4140710 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45086/0 (socket says 172.21.15.50:45086) 2023-12-19T14:57:16.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad3fff700 1 -- 172.21.15.50:0/3035514463 learned_addr learned my addr 172.21.15.50:0/3035514463 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7ad37fe700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4068800 0x7f7ad413eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.442+0000 7f7acaffd700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4140710 0x7f7ad4157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 -- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4140710 msgr2=0x7f7ad4157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4140710 0x7f7ad4157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 -- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4068800 msgr2=0x7f7ad413eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4068800 0x7f7ad413eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 -- 172.21.15.50:0/3035514463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc009420 con 0x7f7ad4140ae0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad3fff700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00ea20 tx=0x7f7acc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00eeb0 con 0x7f7ad4140ae0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc00c850 con 0x7f7ad4140ae0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac4009ae0 con 0x7f7ad4140ae0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00cc30 con 0x7f7ad4140ae0 2023-12-19T14:57:16.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad40b1150 con 0x7f7ad4140ae0 2023-12-19T14:57:16.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.446+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad40aed60 con 0x7f7ad4140ae0 2023-12-19T14:57:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.450+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7acc013470 con 0x7f7ad4140ae0 2023-12-19T14:57:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.450+0000 7f7acbfff700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 0x7f7ab4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.450+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7acc07cff0 con 0x7f7ad4140ae0 2023-12-19T14:57:16.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.450+0000 7f7ad37fe700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 0x7f7ab4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.450+0000 7f7ad37fe700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 0x7f7ab4062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ad415ec80 tx=0x7f7ac4018000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.458+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7acc0acbe0 con 0x7f7ad4140ae0 2023-12-19T14:57:16.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.658+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ad40af120 con 0x7f7ad4140ae0 2023-12-19T14:57:16.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.658+0000 7f7acbfff700 1 -- 172.21.15.50:0/3035514463 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f7acc04d310 con 0x7f7ad4140ae0 2023-12-19T14:57:16.662 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:16.662 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 msgr2=0x7f7ab4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 0x7f7ab4062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ad415ec80 tx=0x7f7ac4018000 comp rx=0 tx=0).stop 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 msgr2=0x7f7ad40c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00ea20 tx=0x7f7acc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 shutdown_connections 2023-12-19T14:57:16.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ab40606a0 0x7f7ab4062b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ad4140ae0 0x7f7ad40c5400 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ad4140710 0x7f7ad4157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 --2- 172.21.15.50:0/3035514463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ad4068800 0x7f7ad413eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 >> 172.21.15.50:0/3035514463 conn(0x7f7ad40c8850 msgr2=0x7f7ad40ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 shutdown_connections 2023-12-19T14:57:16.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.666+0000 7f7ad9b85700 1 -- 172.21.15.50:0/3035514463 wait complete. 2023-12-19T14:57:16.670 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:57:16.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:57:16.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.846+0000 7fe8fb945700 1 Processor -- start 2023-12-19T14:57:16.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 -- start start 2023-12-19T14:57:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 0x7fe8f4068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe8f4068fb0 0x7fe8f4142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f40cbea0 con 0x7fe8f40cea90 2023-12-19T14:57:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f40cf2e0 con 0x7fe8f4068fb0 2023-12-19T14:57:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8fb945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f40cca40 con 0x7fe8f40cee60 2023-12-19T14:57:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f9ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe8f4068fb0 0x7fe8f4142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f8ee0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 0x7fe8f4068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f9ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe8f4068fb0 0x7fe8f4142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45102/0 (socket says 172.21.15.50:45102) 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2656378703 learned_addr learned my addr 172.21.15.50:0/2656378703 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f8ee0700 1 --2- 172.21.15.50:0/2656378703 >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 0x7fe8f4068a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56146/0 (socket says 172.21.15.50:56146) 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.850+0000 7fe8f96e1700 1 --1- 172.21.15.50:0/2656378703 >> v1:172.21.15.125:6789/0 conn(0x7fe8f40cea90 0x7fe8f40cc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39694/0 (socket says 172.21.15.50:39694) 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2656378703 >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 msgr2=0x7fe8f4068a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8f9ee2700 1 --2- 172.21.15.50:0/2656378703 >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 0x7fe8f4068a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2656378703 >> v1:172.21.15.125:6789/0 conn(0x7fe8f40cea90 legacy=0x7fe8f40cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2656378703 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f40cb030 con 0x7fe8f4068fb0 2023-12-19T14:57:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8f9ee2700 1 --2- 172.21.15.50:0/2656378703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4142720 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe8f406a330 tx=0x7fe8ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf93d2a605b3bca server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8ebfff700 1 -- 172.21.15.50:0/2656378703 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00e070 con 0x7fe8f4068fb0 2023-12-19T14:57:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8ebfff700 1 -- 172.21.15.50:0/2656378703 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec00b990 con 0x7fe8f4068fb0 2023-12-19T14:57:16.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8ebfff700 1 -- 172.21.15.50:0/2656378703 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0092e0 con 0x7fe8f4068fb0 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 -- 172.21.15.50:0/2656378703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f4068fb0 msgr2=0x7fe8f4142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2656378703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4142720 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fe8f406a330 tx=0x7fe8ec009840 comp rx=0 tx=0).stop 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 -- 172.21.15.50:0/2656378703 shutdown_connections 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2656378703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4142720 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2656378703 >> v2:172.21.15.125:3300/0 conn(0x7fe8f40cee60 0x7fe8f4068a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 -- 172.21.15.50:0/2656378703 >> 172.21.15.50:0/2656378703 conn(0x7fe8f40cd650 msgr2=0x7fe8f406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 -- 172.21.15.50:0/2656378703 shutdown_connections 2023-12-19T14:57:16.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.854+0000 7fe8fb945700 1 -- 172.21.15.50:0/2656378703 wait complete. 2023-12-19T14:57:16.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 Processor -- start 2023-12-19T14:57:16.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 -- start start 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f40cea90 0x7fe8f40c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40c8ce0 con 0x7fe8f40cee60 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f413fa40 con 0x7fe8f4068fb0 2023-12-19T14:57:16.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8fb945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40c8310 con 0x7fe8f40cea90 2023-12-19T14:57:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45118/0 (socket says 172.21.15.50:45118) 2023-12-19T14:57:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2617154524 learned_addr learned my addr 172.21.15.50:0/2617154524 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f96e1700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f8ee0700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f40cea90 0x7fe8f40c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f40cea90 msgr2=0x7fe8f40c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f40cea90 0x7fe8f40c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f4068fb0 msgr2=0x7fe8f4158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.858+0000 7fe8f9ee2700 1 -- 172.21.15.50:0/2617154524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009ae0 con 0x7fe8f40cee60 2023-12-19T14:57:16.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8f9ee2700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009880 tx=0x7fe8ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec008ba0 con 0x7fe8f40cee60 2023-12-19T14:57:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec008d00 con 0x7fe8f40cee60 2023-12-19T14:57:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0266b0 con 0x7fe8f40cee60 2023-12-19T14:57:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f40c7700 con 0x7fe8f40cee60 2023-12-19T14:57:16.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f40c7cc0 con 0x7fe8f40cee60 2023-12-19T14:57:16.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.862+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f415d6d0 con 0x7fe8f40cee60 2023-12-19T14:57:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.866+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe8ec003040 con 0x7fe8f40cee60 2023-12-19T14:57:16.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.866+0000 7fe8e97fa700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 0x7fe8d8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:16.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.866+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe8ec07e3c0 con 0x7fe8f40cee60 2023-12-19T14:57:16.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.866+0000 7fe8f96e1700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 0x7fe8d8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:16.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.870+0000 7fe8f96e1700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 0x7fe8d8062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0004fd0 tx=0x7fe8f0007eb0 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:16.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:16.870+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe8f415d6d0 con 0x7fe8f40cee60 2023-12-19T14:57:17.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.042+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8f415d6d0 con 0x7fe8f40cee60 2023-12-19T14:57:17.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.046+0000 7fe8e97fa700 1 -- 172.21.15.50:0/2617154524 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+15612 (secure 0 0 0) 0x7fe8f415d6d0 con 0x7fe8f40cee60 2023-12-19T14:57:17.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:17.048 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":129,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:14.026434+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:57:17.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.046+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 msgr2=0x7fe8d8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.046+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 0x7fe8d8062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0004fd0 tx=0x7fe8f0007eb0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.046+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 msgr2=0x7fe8f413f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.046+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009880 tx=0x7fe8ec018000 comp rx=0 tx=0).stop 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 shutdown_connections 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe8d80606a0 0x7fe8d8062b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f40cee60 0x7fe8f413f500 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f40cea90 0x7fe8f40c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 --2- 172.21.15.50:0/2617154524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f4068fb0 0x7fe8f4158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 >> 172.21.15.50:0/2617154524 conn(0x7fe8f40cd650 msgr2=0x7fe8f40c55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:17.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 shutdown_connections 2023-12-19T14:57:17.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.050+0000 7fe8fb945700 1 -- 172.21.15.50:0/2617154524 wait complete. 2023-12-19T14:57:17.074 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:57:17.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:57:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.234+0000 7f39cf48e700 1 Processor -- start 2023-12-19T14:57:17.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.234+0000 7f39cf48e700 1 -- start start 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cf48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 0x7f39c8163ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cf48e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c806c7c0 con 0x7f39c8158d60 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cf48e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c806caa0 con 0x7f39c806b7a0 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cf48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c815d130 con 0x7f39c80c5380 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39ce48c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f39c8158d60 0x7f39c815c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47596/0 (socket says 172.21.15.50:47596) 2023-12-19T14:57:17.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39ce48c700 1 -- 172.21.15.50:0/4041559533 learned_addr learned my addr 172.21.15.50:0/4041559533 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cec8d700 1 --2- 172.21.15.50:0/4041559533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 0x7f39c8163ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cd48a700 1 -- 172.21.15.50:0/4041559533 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540071626 0 0) 0x7f39c806c7c0 con 0x7f39c8158d60 2023-12-19T14:57:17.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cd48a700 1 -- 172.21.15.50:0/4041559533 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b4003570 con 0x7f39c8158d60 2023-12-19T14:57:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cec8d700 1 -- 172.21.15.50:0/4041559533 >> v1:172.21.15.125:6789/0 conn(0x7f39c806b7a0 legacy=0x7f39c80c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cec8d700 1 -- 172.21.15.50:0/4041559533 >> v1:172.21.15.50:6789/0 conn(0x7f39c8158d60 legacy=0x7f39c815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cec8d700 1 -- 172.21.15.50:0/4041559533 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c815d410 con 0x7f39c80c5380 2023-12-19T14:57:17.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.238+0000 7f39cec8d700 1 --2- 172.21.15.50:0/4041559533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 0x7f39c8163ca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009e90 tx=0x7f39c0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29994b3177cabec7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cd48a700 1 -- 172.21.15.50:0/4041559533 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c000e070 con 0x7f39c80c5380 2023-12-19T14:57:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cd48a700 1 -- 172.21.15.50:0/4041559533 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c000bc30 con 0x7f39c80c5380 2023-12-19T14:57:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cd48a700 1 -- 172.21.15.50:0/4041559533 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0012430 con 0x7f39c80c5380 2023-12-19T14:57:17.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 -- 172.21.15.50:0/4041559533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 msgr2=0x7f39c8163ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 --2- 172.21.15.50:0/4041559533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 0x7f39c8163ca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009e90 tx=0x7f39c0009780 comp rx=0 tx=0).stop 2023-12-19T14:57:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 -- 172.21.15.50:0/4041559533 shutdown_connections 2023-12-19T14:57:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 --2- 172.21.15.50:0/4041559533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c80c5380 0x7f39c8163ca0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 -- 172.21.15.50:0/4041559533 >> 172.21.15.50:0/4041559533 conn(0x7f39c8067bb0 msgr2=0x7f39c80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 -- 172.21.15.50:0/4041559533 shutdown_connections 2023-12-19T14:57:17.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.246+0000 7f39cf48e700 1 -- 172.21.15.50:0/4041559533 wait complete. 2023-12-19T14:57:17.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.250+0000 7f39cf48e700 1 Processor -- start 2023-12-19T14:57:17.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.250+0000 7f39cf48e700 1 -- start start 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 0x7f39c815acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c8158d60 0x7f39c815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c81518d0 con 0x7f39c806b7a0 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c816a8d0 con 0x7f39c80c5380 2023-12-19T14:57:17.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8169f80 con 0x7f39c8158d60 2023-12-19T14:57:17.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45120/0 (socket says 172.21.15.50:45120) 2023-12-19T14:57:17.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 -- 172.21.15.50:0/124619449 learned_addr learned my addr 172.21.15.50:0/124619449 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cdc8b700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 0x7f39c815acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cec8d700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c8158d60 0x7f39c815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 -- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c8158d60 msgr2=0x7f39c815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c8158d60 0x7f39c815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 -- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 msgr2=0x7f39c815acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 0x7f39c815acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 -- 172.21.15.50:0/124619449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c0009a20 con 0x7f39c806b7a0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cdc8b700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 0x7f39c815acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39ce48c700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f39c400c590 tx=0x7f39c4007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4005b30 con 0x7f39c806b7a0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c4005cb0 con 0x7f39c806b7a0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4007320 con 0x7f39c806b7a0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c815b890 con 0x7f39c806b7a0 2023-12-19T14:57:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.254+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c816a560 con 0x7f39c806b7a0 2023-12-19T14:57:17.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.258+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c804be40 con 0x7f39c806b7a0 2023-12-19T14:57:17.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.258+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f39c401b3b0 con 0x7f39c806b7a0 2023-12-19T14:57:17.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.258+0000 7f39be7fc700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 0x7f39ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.258+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f39c407c0b0 con 0x7f39c806b7a0 2023-12-19T14:57:17.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.258+0000 7f39cdc8b700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 0x7f39ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.262+0000 7f39cdc8b700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 0x7f39ac062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39b8007fd0 tx=0x7f39b8009040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.266+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39c404c390 con 0x7f39c806b7a0 2023-12-19T14:57:17.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.430+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39c815d6f0 con 0x7f39ac060670 2023-12-19T14:57:17.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.454+0000 7f39be7fc700 1 -- 172.21.15.50:0/124619449 <== mgr.6833 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286926 (secure 0 0 0) 0x7f39c815d6f0 con 0x7f39ac060670 2023-12-19T14:57:17.459 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 msgr2=0x7f39ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 0x7f39ac062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39b8007fd0 tx=0x7f39b8009040 comp rx=0 tx=0).stop 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 msgr2=0x7f39c813fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f39c400c590 tx=0x7f39c4007e00 comp rx=0 tx=0).stop 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 shutdown_connections 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f39c80c5380 0x7f39c815acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f39ac060670 0x7f39ac062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f39c8158d60 0x7f39c815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 --2- 172.21.15.50:0/124619449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39c806b7a0 0x7f39c813fb20 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 >> 172.21.15.50:0/124619449 conn(0x7f39c8067bb0 msgr2=0x7f39c8153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:17.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 shutdown_connections 2023-12-19T14:57:17.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.462+0000 7f39cf48e700 1 -- 172.21.15.50:0/124619449 wait complete. 2023-12-19T14:57:17.468 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:57:17.487 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:57:16.683482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449284,"kb_used_data":13636,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525436,"statfs":{"total":773094113280,"available":770586046464,"internally_reserved":0,"allocated":13963264,"data_stored":7573274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.503226"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061856+0000","last_change":"2023-12-19T14:57:13.057390+0000","last_active":"2023-12-19T14:57:13.061856+0000","last_peered":"2023-12-19T14:57:13.061856+0000","last_clean":"2023-12-19T14:57:13.061856+0000","last_became_active":"2023-12-19T14:57:13.056910+0000","last_became_peered":"2023-12-19T14:57:13.056910+0000","last_unstale":"2023-12-19T14:57:13.061856+0000","last_undegraded":"2023-12-19T14:57:13.061856+0000","last_fullsized":"2023-12-19T14:57:13.061856+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:41.333180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060354+0000","last_change":"2023-12-19T14:57:13.054541+0000","last_active":"2023-12-19T14:57:13.060354+0000","last_peered":"2023-12-19T14:57:13.060354+0000","last_clean":"2023-12-19T14:57:13.060354+0000","last_became_active":"2023-12-19T14:57:13.052986+0000","last_became_peered":"2023-12-19T14:57:13.052986+0000","last_unstale":"2023-12-19T14:57:13.060354+0000","last_undegraded":"2023-12-19T14:57:13.060354+0000","last_fullsized":"2023-12-19T14:57:13.060354+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067838+0000","last_change":"2023-12-19T14:57:13.063457+0000","last_active":"2023-12-19T14:57:13.067838+0000","last_peered":"2023-12-19T14:57:13.067838+0000","last_clean":"2023-12-19T14:57:13.067838+0000","last_became_active":"2023-12-19T14:57:13.062932+0000","last_became_peered":"2023-12-19T14:57:13.062932+0000","last_unstale":"2023-12-19T14:57:13.067838+0000","last_undegraded":"2023-12-19T14:57:13.067838+0000","last_fullsized":"2023-12-19T14:57:13.067838+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:22.183822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058113+0000","last_change":"2023-12-19T14:57:13.051915+0000","last_active":"2023-12-19T14:57:13.058113+0000","last_peered":"2023-12-19T14:57:13.058113+0000","last_clean":"2023-12-19T14:57:13.058113+0000","last_became_active":"2023-12-19T14:57:13.051590+0000","last_became_peered":"2023-12-19T14:57:13.051590+0000","last_unstale":"2023-12-19T14:57:13.058113+0000","last_undegraded":"2023-12-19T14:57:13.058113+0000","last_fullsized":"2023-12-19T14:57:13.058113+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:20.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062033+0000","last_change":"2023-12-19T14:57:13.051286+0000","last_active":"2023-12-19T14:57:13.062033+0000","last_peered":"2023-12-19T14:57:13.062033+0000","last_clean":"2023-12-19T14:57:13.062033+0000","last_became_active":"2023-12-19T14:57:13.050979+0000","last_became_peered":"2023-12-19T14:57:13.050979+0000","last_unstale":"2023-12-19T14:57:13.062033+0000","last_undegraded":"2023-12-19T14:57:13.062033+0000","last_fullsized":"2023-12-19T14:57:13.062033+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067129+0000","last_change":"2023-12-19T14:57:13.066300+0000","last_active":"2023-12-19T14:57:13.067129+0000","last_peered":"2023-12-19T14:57:13.067129+0000","last_clean":"2023-12-19T14:57:13.067129+0000","last_became_active":"2023-12-19T14:57:13.065974+0000","last_became_peered":"2023-12-19T14:57:13.065974+0000","last_unstale":"2023-12-19T14:57:13.067129+0000","last_undegraded":"2023-12-19T14:57:13.067129+0000","last_fullsized":"2023-12-19T14:57:13.067129+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:43:03.428835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066106+0000","last_change":"2023-12-19T14:57:13.065390+0000","last_active":"2023-12-19T14:57:13.066106+0000","last_peered":"2023-12-19T14:57:13.066106+0000","last_clean":"2023-12-19T14:57:13.066106+0000","last_became_active":"2023-12-19T14:57:13.065105+0000","last_became_peered":"2023-12-19T14:57:13.065105+0000","last_unstale":"2023-12-19T14:57:13.066106+0000","last_undegraded":"2023-12-19T14:57:13.066106+0000","last_fullsized":"2023-12-19T14:57:13.066106+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:34.379221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055669+0000","last_change":"2023-12-19T14:57:13.049509+0000","last_active":"2023-12-19T14:57:13.055669+0000","last_peered":"2023-12-19T14:57:13.055669+0000","last_clean":"2023-12-19T14:57:13.055669+0000","last_became_active":"2023-12-19T14:57:13.049215+0000","last_became_peered":"2023-12-19T14:57:13.049215+0000","last_unstale":"2023-12-19T14:57:13.055669+0000","last_undegraded":"2023-12-19T14:57:13.055669+0000","last_fullsized":"2023-12-19T14:57:13.055669+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:21.578627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055684+0000","last_change":"2023-12-19T14:57:13.050408+0000","last_active":"2023-12-19T14:57:13.055684+0000","last_peered":"2023-12-19T14:57:13.055684+0000","last_clean":"2023-12-19T14:57:13.055684+0000","last_became_active":"2023-12-19T14:57:13.050072+0000","last_became_peered":"2023-12-19T14:57:13.050072+0000","last_unstale":"2023-12-19T14:57:13.055684+0000","last_undegraded":"2023-12-19T14:57:13.055684+0000","last_fullsized":"2023-12-19T14:57:13.055684+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:51.886340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.069674+0000","last_change":"2023-12-19T14:57:13.068520+0000","last_active":"2023-12-19T14:57:13.069674+0000","last_peered":"2023-12-19T14:57:13.069674+0000","last_clean":"2023-12-19T14:57:13.069674+0000","last_became_active":"2023-12-19T14:57:13.068146+0000","last_became_peered":"2023-12-19T14:57:13.068146+0000","last_unstale":"2023-12-19T14:57:13.069674+0000","last_undegraded":"2023-12-19T14:57:13.069674+0000","last_fullsized":"2023-12-19T14:57:13.069674+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060664+0000","last_change":"2023-12-19T14:57:13.052831+0000","last_active":"2023-12-19T14:57:13.060664+0000","last_peered":"2023-12-19T14:57:13.060664+0000","last_clean":"2023-12-19T14:57:13.060664+0000","last_became_active":"2023-12-19T14:57:13.052458+0000","last_became_peered":"2023-12-19T14:57:13.052458+0000","last_unstale":"2023-12-19T14:57:13.060664+0000","last_undegraded":"2023-12-19T14:57:13.060664+0000","last_fullsized":"2023-12-19T14:57:13.060664+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:18.331281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.051612+0000","last_change":"2023-12-19T14:57:13.047253+0000","last_active":"2023-12-19T14:57:13.051612+0000","last_peered":"2023-12-19T14:57:13.051612+0000","last_clean":"2023-12-19T14:57:13.051612+0000","last_became_active":"2023-12-19T14:57:13.046667+0000","last_became_peered":"2023-12-19T14:57:13.046667+0000","last_unstale":"2023-12-19T14:57:13.051612+0000","last_undegraded":"2023-12-19T14:57:13.051612+0000","last_fullsized":"2023-12-19T14:57:13.051612+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:16:50.173814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056697+0000","last_change":"2023-12-19T14:57:13.050982+0000","last_active":"2023-12-19T14:57:13.056697+0000","last_peered":"2023-12-19T14:57:13.056697+0000","last_clean":"2023-12-19T14:57:13.056697+0000","last_became_active":"2023-12-19T14:57:13.050449+0000","last_became_peered":"2023-12-19T14:57:13.050449+0000","last_unstale":"2023-12-19T14:57:13.056697+0000","last_undegraded":"2023-12-19T14:57:13.056697+0000","last_fullsized":"2023-12-19T14:57:13.056697+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:02:34.060332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.052516+0000","last_change":"2023-12-19T14:57:13.046092+0000","last_active":"2023-12-19T14:57:13.052516+0000","last_peered":"2023-12-19T14:57:13.052516+0000","last_clean":"2023-12-19T14:57:13.052516+0000","last_became_active":"2023-12-19T14:57:13.045741+0000","last_became_peered":"2023-12-19T14:57:13.045741+0000","last_unstale":"2023-12-19T14:57:13.052516+0000","last_undegraded":"2023-12-19T14:57:13.052516+0000","last_fullsized":"2023-12-19T14:57:13.052516+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062399+0000","last_change":"2023-12-19T14:57:13.058119+0000","last_active":"2023-12-19T14:57:13.062399+0000","last_peered":"2023-12-19T14:57:13.062399+0000","last_clean":"2023-12-19T14:57:13.062399+0000","last_became_active":"2023-12-19T14:57:13.057815+0000","last_became_peered":"2023-12-19T14:57:13.057815+0000","last_unstale":"2023-12-19T14:57:13.062399+0000","last_undegraded":"2023-12-19T14:57:13.062399+0000","last_fullsized":"2023-12-19T14:57:13.062399+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.054554+0000","last_change":"2023-12-19T14:57:13.048883+0000","last_active":"2023-12-19T14:57:13.054554+0000","last_peered":"2023-12-19T14:57:13.054554+0000","last_clean":"2023-12-19T14:57:13.054554+0000","last_became_active":"2023-12-19T14:57:13.048608+0000","last_became_peered":"2023-12-19T14:57:13.048608+0000","last_unstale":"2023-12-19T14:57:13.054554+0000","last_undegraded":"2023-12-19T14:57:13.054554+0000","last_fullsized":"2023-12-19T14:57:13.054554+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066279+0000","last_change":"2023-12-19T14:57:13.061740+0000","last_active":"2023-12-19T14:57:13.066279+0000","last_peered":"2023-12-19T14:57:13.066279+0000","last_clean":"2023-12-19T14:57:13.066279+0000","last_became_active":"2023-12-19T14:57:13.061454+0000","last_became_peered":"2023-12-19T14:57:13.061454+0000","last_unstale":"2023-12-19T14:57:13.066279+0000","last_undegraded":"2023-12-19T14:57:13.066279+0000","last_fullsized":"2023-12-19T14:57:13.066279+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:13:32.026843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067141+0000","last_change":"2023-12-19T14:57:13.066328+0000","last_active":"2023-12-19T14:57:13.067141+0000","last_peered":"2023-12-19T14:57:13.067141+0000","last_clean":"2023-12-19T14:57:13.067141+0000","last_became_active":"2023-12-19T14:57:13.065978+0000","last_became_peered":"2023-12-19T14:57:13.065978+0000","last_unstale":"2023-12-19T14:57:13.067141+0000","last_undegraded":"2023-12-19T14:57:13.067141+0000","last_fullsized":"2023-12-19T14:57:13.067141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067907+0000","last_change":"2023-12-19T14:57:13.063498+0000","last_active":"2023-12-19T14:57:13.067907+0000","last_peered":"2023-12-19T14:57:13.067907+0000","last_clean":"2023-12-19T14:57:13.067907+0000","last_became_active":"2023-12-19T14:57:13.063184+0000","last_became_peered":"2023-12-19T14:57:13.063184+0000","last_unstale":"2023-12-19T14:57:13.067907+0000","last_undegraded":"2023-12-19T14:57:13.067907+0000","last_fullsized":"2023-12-19T14:57:13.067907+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:30.268380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059486+0000","last_change":"2023-12-19T14:57:13.054866+0000","last_active":"2023-12-19T14:57:13.059486+0000","last_peered":"2023-12-19T14:57:13.059486+0000","last_clean":"2023-12-19T14:57:13.059486+0000","last_became_active":"2023-12-19T14:57:13.053860+0000","last_became_peered":"2023-12-19T14:57:13.053860+0000","last_unstale":"2023-12-19T14:57:13.059486+0000","last_undegraded":"2023-12-19T14:57:13.059486+0000","last_fullsized":"2023-12-19T14:57:13.059486+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:14:07.850723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063197+0000","last_change":"2023-12-19T14:57:13.061649+0000","last_active":"2023-12-19T14:57:13.063197+0000","last_peered":"2023-12-19T14:57:13.063197+0000","last_clean":"2023-12-19T14:57:13.063197+0000","last_became_active":"2023-12-19T14:57:13.061301+0000","last_became_peered":"2023-12-19T14:57:13.061301+0000","last_unstale":"2023-12-19T14:57:13.063197+0000","last_undegraded":"2023-12-19T14:57:13.063197+0000","last_fullsized":"2023-12-19T14:57:13.063197+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:49.473135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063617+0000","last_change":"2023-12-19T14:57:13.058318+0000","last_active":"2023-12-19T14:57:13.063617+0000","last_peered":"2023-12-19T14:57:13.063617+0000","last_clean":"2023-12-19T14:57:13.063617+0000","last_became_active":"2023-12-19T14:57:13.055248+0000","last_became_peered":"2023-12-19T14:57:13.055248+0000","last_unstale":"2023-12-19T14:57:13.063617+0000","last_undegraded":"2023-12-19T14:57:13.063617+0000","last_fullsized":"2023-12-19T14:57:13.063617+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:31.625153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067700+0000","last_change":"2023-12-19T14:57:13.063046+0000","last_active":"2023-12-19T14:57:13.067700+0000","last_peered":"2023-12-19T14:57:13.067700+0000","last_clean":"2023-12-19T14:57:13.067700+0000","last_became_active":"2023-12-19T14:57:13.062647+0000","last_became_peered":"2023-12-19T14:57:13.062647+0000","last_unstale":"2023-12-19T14:57:13.067700+0000","last_undegraded":"2023-12-19T14:57:13.067700+0000","last_fullsized":"2023-12-19T14:57:13.067700+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:23.005310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062349+0000","last_change":"2023-12-19T14:57:13.055881+0000","last_active":"2023-12-19T14:57:13.062349+0000","last_peered":"2023-12-19T14:57:13.062349+0000","last_clean":"2023-12-19T14:57:13.062349+0000","last_became_active":"2023-12-19T14:57:13.055592+0000","last_became_peered":"2023-12-19T14:57:13.055592+0000","last_unstale":"2023-12-19T14:57:13.062349+0000","last_undegraded":"2023-12-19T14:57:13.062349+0000","last_fullsized":"2023-12-19T14:57:13.062349+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:24.572601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061019+0000","last_change":"2023-12-19T14:57:13.055059+0000","last_active":"2023-12-19T14:57:13.061019+0000","last_peered":"2023-12-19T14:57:13.061019+0000","last_clean":"2023-12-19T14:57:13.061019+0000","last_became_active":"2023-12-19T14:57:13.054747+0000","last_became_peered":"2023-12-19T14:57:13.054747+0000","last_unstale":"2023-12-19T14:57:13.061019+0000","last_undegraded":"2023-12-19T14:57:13.061019+0000","last_fullsized":"2023-12-19T14:57:13.061019+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:29.761636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061703+0000","last_change":"2023-12-19T14:57:13.057919+0000","last_active":"2023-12-19T14:57:13.061703+0000","last_peered":"2023-12-19T14:57:13.061703+0000","last_clean":"2023-12-19T14:57:13.061703+0000","last_became_active":"2023-12-19T14:57:13.057595+0000","last_became_peered":"2023-12-19T14:57:13.057595+0000","last_unstale":"2023-12-19T14:57:13.061703+0000","last_undegraded":"2023-12-19T14:57:13.061703+0000","last_fullsized":"2023-12-19T14:57:13.061703+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:18.331281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056599+0000","last_change":"2023-12-19T14:57:13.050863+0000","last_active":"2023-12-19T14:57:13.056599+0000","last_peered":"2023-12-19T14:57:13.056599+0000","last_clean":"2023-12-19T14:57:13.056599+0000","last_became_active":"2023-12-19T14:57:13.050358+0000","last_became_peered":"2023-12-19T14:57:13.050358+0000","last_unstale":"2023-12-19T14:57:13.056599+0000","last_undegraded":"2023-12-19T14:57:13.056599+0000","last_fullsized":"2023-12-19T14:57:13.056599+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:14:07.850723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061997+0000","last_change":"2023-12-19T14:57:13.058217+0000","last_active":"2023-12-19T14:57:13.061997+0000","last_peered":"2023-12-19T14:57:13.061997+0000","last_clean":"2023-12-19T14:57:13.061997+0000","last_became_active":"2023-12-19T14:57:13.057931+0000","last_became_peered":"2023-12-19T14:57:13.057931+0000","last_unstale":"2023-12-19T14:57:13.061997+0000","last_undegraded":"2023-12-19T14:57:13.061997+0000","last_fullsized":"2023-12-19T14:57:13.061997+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061508+0000","last_change":"2023-12-19T14:57:13.057666+0000","last_active":"2023-12-19T14:57:13.061508+0000","last_peered":"2023-12-19T14:57:13.061508+0000","last_clean":"2023-12-19T14:57:13.061508+0000","last_became_active":"2023-12-19T14:57:13.057338+0000","last_became_peered":"2023-12-19T14:57:13.057338+0000","last_unstale":"2023-12-19T14:57:13.061508+0000","last_undegraded":"2023-12-19T14:57:13.061508+0000","last_fullsized":"2023-12-19T14:57:13.061508+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:03:49.948787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058608+0000","last_change":"2023-12-19T14:57:13.054390+0000","last_active":"2023-12-19T14:57:13.058608+0000","last_peered":"2023-12-19T14:57:13.058608+0000","last_clean":"2023-12-19T14:57:13.058608+0000","last_became_active":"2023-12-19T14:57:13.054109+0000","last_became_peered":"2023-12-19T14:57:13.054109+0000","last_unstale":"2023-12-19T14:57:13.058608+0000","last_undegraded":"2023-12-19T14:57:13.058608+0000","last_fullsized":"2023-12-19T14:57:13.058608+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:15.738373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.064038+0000","last_change":"2023-12-19T14:57:13.061294+0000","last_active":"2023-12-19T14:57:13.064038+0000","last_peered":"2023-12-19T14:57:13.064038+0000","last_clean":"2023-12-19T14:57:13.064038+0000","last_became_active":"2023-12-19T14:57:13.060693+0000","last_became_peered":"2023-12-19T14:57:13.060693+0000","last_unstale":"2023-12-19T14:57:13.064038+0000","last_undegraded":"2023-12-19T14:57:13.064038+0000","last_fullsized":"2023-12-19T14:57:13.064038+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:20.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062227+0000","last_change":"2023-12-19T14:57:13.052410+0000","last_active":"2023-12-19T14:57:13.062227+0000","last_peered":"2023-12-19T14:57:13.062227+0000","last_clean":"2023-12-19T14:57:13.062227+0000","last_became_active":"2023-12-19T14:57:13.052070+0000","last_became_peered":"2023-12-19T14:57:13.052070+0000","last_unstale":"2023-12-19T14:57:13.062227+0000","last_undegraded":"2023-12-19T14:57:13.062227+0000","last_fullsized":"2023-12-19T14:57:13.062227+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:57.340483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058646+0000","last_change":"2023-12-19T14:57:13.054165+0000","last_active":"2023-12-19T14:57:13.058646+0000","last_peered":"2023-12-19T14:57:13.058646+0000","last_clean":"2023-12-19T14:57:13.058646+0000","last_became_active":"2023-12-19T14:57:13.053800+0000","last_became_peered":"2023-12-19T14:57:13.053800+0000","last_unstale":"2023-12-19T14:57:13.058646+0000","last_undegraded":"2023-12-19T14:57:13.058646+0000","last_fullsized":"2023-12-19T14:57:13.058646+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:01:51.423899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237505+0000","last_change":"2023-12-19T14:57:12.224820+0000","last_active":"2023-12-19T14:57:13.237505+0000","last_peered":"2023-12-19T14:57:13.237505+0000","last_clean":"2023-12-19T14:57:13.237505+0000","last_became_active":"2023-12-19T14:57:12.224511+0000","last_became_peered":"2023-12-19T14:57:12.224511+0000","last_unstale":"2023-12-19T14:57:13.237505+0000","last_undegraded":"2023-12-19T14:57:13.237505+0000","last_fullsized":"2023-12-19T14:57:13.237505+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:40.508592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055484+0000","last_change":"2023-12-19T14:57:13.048726+0000","last_active":"2023-12-19T14:57:13.055484+0000","last_peered":"2023-12-19T14:57:13.055484+0000","last_clean":"2023-12-19T14:57:13.055484+0000","last_became_active":"2023-12-19T14:57:13.048406+0000","last_became_peered":"2023-12-19T14:57:13.048406+0000","last_unstale":"2023-12-19T14:57:13.055484+0000","last_undegraded":"2023-12-19T14:57:13.055484+0000","last_fullsized":"2023-12-19T14:57:13.055484+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:32.137185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.235799+0000","last_change":"2023-12-19T14:57:12.222347+0000","last_active":"2023-12-19T14:57:13.235799+0000","last_peered":"2023-12-19T14:57:13.235799+0000","last_clean":"2023-12-19T14:57:13.235799+0000","last_became_active":"2023-12-19T14:57:12.222087+0000","last_became_peered":"2023-12-19T14:57:12.222087+0000","last_unstale":"2023-12-19T14:57:13.235799+0000","last_undegraded":"2023-12-19T14:57:13.235799+0000","last_fullsized":"2023-12-19T14:57:13.235799+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:36.584909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.048529+0000","last_change":"2023-12-19T14:57:13.046939+0000","last_active":"2023-12-19T14:57:13.048529+0000","last_peered":"2023-12-19T14:57:13.048529+0000","last_clean":"2023-12-19T14:57:13.048529+0000","last_became_active":"2023-12-19T14:57:13.046617+0000","last_became_peered":"2023-12-19T14:57:13.046617+0000","last_unstale":"2023-12-19T14:57:13.048529+0000","last_undegraded":"2023-12-19T14:57:13.048529+0000","last_fullsized":"2023-12-19T14:57:13.048529+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:46.498057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.233314+0000","last_change":"2023-12-19T14:57:12.214186+0000","last_active":"2023-12-19T14:57:13.233314+0000","last_peered":"2023-12-19T14:57:13.233314+0000","last_clean":"2023-12-19T14:57:13.233314+0000","last_became_active":"2023-12-19T14:57:12.213819+0000","last_became_peered":"2023-12-19T14:57:12.213819+0000","last_unstale":"2023-12-19T14:57:13.233314+0000","last_undegraded":"2023-12-19T14:57:13.233314+0000","last_fullsized":"2023-12-19T14:57:13.233314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:43.547892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056552+0000","last_change":"2023-12-19T14:57:13.048651+0000","last_active":"2023-12-19T14:57:13.056552+0000","last_peered":"2023-12-19T14:57:13.056552+0000","last_clean":"2023-12-19T14:57:13.056552+0000","last_became_active":"2023-12-19T14:57:13.048319+0000","last_became_peered":"2023-12-19T14:57:13.048319+0000","last_unstale":"2023-12-19T14:57:13.056552+0000","last_undegraded":"2023-12-19T14:57:13.056552+0000","last_fullsized":"2023-12-19T14:57:13.056552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:10.953482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.238517+0000","last_change":"2023-12-19T14:57:12.218816+0000","last_active":"2023-12-19T14:57:13.238517+0000","last_peered":"2023-12-19T14:57:13.238517+0000","last_clean":"2023-12-19T14:57:13.238517+0000","last_became_active":"2023-12-19T14:57:12.218512+0000","last_became_peered":"2023-12-19T14:57:12.218512+0000","last_unstale":"2023-12-19T14:57:13.238517+0000","last_undegraded":"2023-12-19T14:57:13.238517+0000","last_fullsized":"2023-12-19T14:57:13.238517+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:06.151098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066472+0000","last_change":"2023-12-19T14:57:13.065371+0000","last_active":"2023-12-19T14:57:13.066472+0000","last_peered":"2023-12-19T14:57:13.066472+0000","last_clean":"2023-12-19T14:57:13.066472+0000","last_became_active":"2023-12-19T14:57:13.065072+0000","last_became_peered":"2023-12-19T14:57:13.065072+0000","last_unstale":"2023-12-19T14:57:13.066472+0000","last_undegraded":"2023-12-19T14:57:13.066472+0000","last_fullsized":"2023-12-19T14:57:13.066472+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:10:05.441581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.232747+0000","last_change":"2023-12-19T14:57:12.219505+0000","last_active":"2023-12-19T14:57:13.232747+0000","last_peered":"2023-12-19T14:57:13.232747+0000","last_clean":"2023-12-19T14:57:13.232747+0000","last_became_active":"2023-12-19T14:57:12.219163+0000","last_became_peered":"2023-12-19T14:57:12.219163+0000","last_unstale":"2023-12-19T14:57:13.232747+0000","last_undegraded":"2023-12-19T14:57:13.232747+0000","last_fullsized":"2023-12-19T14:57:13.232747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:31.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067725+0000","last_change":"2023-12-19T14:57:13.063241+0000","last_active":"2023-12-19T14:57:13.067725+0000","last_peered":"2023-12-19T14:57:13.067725+0000","last_clean":"2023-12-19T14:57:13.067725+0000","last_became_active":"2023-12-19T14:57:13.062292+0000","last_became_peered":"2023-12-19T14:57:13.062292+0000","last_unstale":"2023-12-19T14:57:13.067725+0000","last_undegraded":"2023-12-19T14:57:13.067725+0000","last_fullsized":"2023-12-19T14:57:13.067725+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:12.208855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234308+0000","last_change":"2023-12-19T14:57:12.214250+0000","last_active":"2023-12-19T14:57:13.234308+0000","last_peered":"2023-12-19T14:57:13.234308+0000","last_clean":"2023-12-19T14:57:13.234308+0000","last_became_active":"2023-12-19T14:57:12.213038+0000","last_became_peered":"2023-12-19T14:57:12.213038+0000","last_unstale":"2023-12-19T14:57:13.234308+0000","last_undegraded":"2023-12-19T14:57:13.234308+0000","last_fullsized":"2023-12-19T14:57:13.234308+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:43.547892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062031+0000","last_change":"2023-12-19T14:57:13.058393+0000","last_active":"2023-12-19T14:57:13.062031+0000","last_peered":"2023-12-19T14:57:13.062031+0000","last_clean":"2023-12-19T14:57:13.062031+0000","last_became_active":"2023-12-19T14:57:13.058037+0000","last_became_peered":"2023-12-19T14:57:13.058037+0000","last_unstale":"2023-12-19T14:57:13.062031+0000","last_undegraded":"2023-12-19T14:57:13.062031+0000","last_fullsized":"2023-12-19T14:57:13.062031+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042200+0000","last_change":"2023-12-19T14:57:12.212301+0000","last_active":"2023-12-19T14:57:13.042200+0000","last_peered":"2023-12-19T14:57:13.042200+0000","last_clean":"2023-12-19T14:57:13.042200+0000","last_became_active":"2023-12-19T14:57:12.212024+0000","last_became_peered":"2023-12-19T14:57:12.212024+0000","last_unstale":"2023-12-19T14:57:13.042200+0000","last_undegraded":"2023-12-19T14:57:13.042200+0000","last_fullsized":"2023-12-19T14:57:13.042200+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:42.039352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058475+0000","last_change":"2023-12-19T14:57:13.052752+0000","last_active":"2023-12-19T14:57:13.058475+0000","last_peered":"2023-12-19T14:57:13.058475+0000","last_clean":"2023-12-19T14:57:13.058475+0000","last_became_active":"2023-12-19T14:57:13.050548+0000","last_became_peered":"2023-12-19T14:57:13.050548+0000","last_unstale":"2023-12-19T14:57:13.058475+0000","last_undegraded":"2023-12-19T14:57:13.058475+0000","last_fullsized":"2023-12-19T14:57:13.058475+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:33:30.650455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036739+0000","last_change":"2023-12-19T14:57:12.201356+0000","last_active":"2023-12-19T14:57:13.036739+0000","last_peered":"2023-12-19T14:57:13.036739+0000","last_clean":"2023-12-19T14:57:13.036739+0000","last_became_active":"2023-12-19T14:57:12.200902+0000","last_became_peered":"2023-12-19T14:57:12.200902+0000","last_unstale":"2023-12-19T14:57:13.036739+0000","last_undegraded":"2023-12-19T14:57:13.036739+0000","last_fullsized":"2023-12-19T14:57:13.036739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:52:46.558984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062209+0000","last_change":"2023-12-19T14:57:13.059482+0000","last_active":"2023-12-19T14:57:13.062209+0000","last_peered":"2023-12-19T14:57:13.062209+0000","last_clean":"2023-12-19T14:57:13.062209+0000","last_became_active":"2023-12-19T14:57:13.058446+0000","last_became_peered":"2023-12-19T14:57:13.058446+0000","last_unstale":"2023-12-19T14:57:13.062209+0000","last_undegraded":"2023-12-19T14:57:13.062209+0000","last_fullsized":"2023-12-19T14:57:13.062209+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:00.226492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.046398+0000","last_change":"2023-12-19T14:57:12.212456+0000","last_active":"2023-12-19T14:57:13.046398+0000","last_peered":"2023-12-19T14:57:13.046398+0000","last_clean":"2023-12-19T14:57:13.046398+0000","last_became_active":"2023-12-19T14:57:12.212105+0000","last_became_peered":"2023-12-19T14:57:12.212105+0000","last_unstale":"2023-12-19T14:57:13.046398+0000","last_undegraded":"2023-12-19T14:57:13.046398+0000","last_fullsized":"2023-12-19T14:57:13.046398+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:30.579427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.057920+0000","last_change":"2023-12-19T14:57:13.050453+0000","last_active":"2023-12-19T14:57:13.057920+0000","last_peered":"2023-12-19T14:57:13.057920+0000","last_clean":"2023-12-19T14:57:13.057920+0000","last_became_active":"2023-12-19T14:57:13.049743+0000","last_became_peered":"2023-12-19T14:57:13.049743+0000","last_unstale":"2023-12-19T14:57:13.057920+0000","last_undegraded":"2023-12-19T14:57:13.057920+0000","last_fullsized":"2023-12-19T14:57:13.057920+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042643+0000","last_change":"2023-12-19T14:57:12.222441+0000","last_active":"2023-12-19T14:57:13.042643+0000","last_peered":"2023-12-19T14:57:13.042643+0000","last_clean":"2023-12-19T14:57:13.042643+0000","last_became_active":"2023-12-19T14:57:12.221879+0000","last_became_peered":"2023-12-19T14:57:12.221879+0000","last_unstale":"2023-12-19T14:57:13.042643+0000","last_undegraded":"2023-12-19T14:57:13.042643+0000","last_fullsized":"2023-12-19T14:57:13.042643+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:45.485326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058084+0000","last_change":"2023-12-19T14:57:13.051257+0000","last_active":"2023-12-19T14:57:13.058084+0000","last_peered":"2023-12-19T14:57:13.058084+0000","last_clean":"2023-12-19T14:57:13.058084+0000","last_became_active":"2023-12-19T14:57:13.050957+0000","last_became_peered":"2023-12-19T14:57:13.050957+0000","last_unstale":"2023-12-19T14:57:13.058084+0000","last_undegraded":"2023-12-19T14:57:13.058084+0000","last_fullsized":"2023-12-19T14:57:13.058084+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:48:54.891330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.233627+0000","last_change":"2023-12-19T14:57:12.221225+0000","last_active":"2023-12-19T14:57:13.233627+0000","last_peered":"2023-12-19T14:57:13.233627+0000","last_clean":"2023-12-19T14:57:13.233627+0000","last_became_active":"2023-12-19T14:57:12.220859+0000","last_became_peered":"2023-12-19T14:57:12.220859+0000","last_unstale":"2023-12-19T14:57:13.233627+0000","last_undegraded":"2023-12-19T14:57:13.233627+0000","last_fullsized":"2023-12-19T14:57:13.233627+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:02:33.047600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058802+0000","last_change":"2023-12-19T14:57:13.050125+0000","last_active":"2023-12-19T14:57:13.058802+0000","last_peered":"2023-12-19T14:57:13.058802+0000","last_clean":"2023-12-19T14:57:13.058802+0000","last_became_active":"2023-12-19T14:57:13.049404+0000","last_became_peered":"2023-12-19T14:57:13.049404+0000","last_unstale":"2023-12-19T14:57:13.058802+0000","last_undegraded":"2023-12-19T14:57:13.058802+0000","last_fullsized":"2023-12-19T14:57:13.058802+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.236626+0000","last_change":"2023-12-19T14:57:12.217230+0000","last_active":"2023-12-19T14:57:13.236626+0000","last_peered":"2023-12-19T14:57:13.236626+0000","last_clean":"2023-12-19T14:57:13.236626+0000","last_became_active":"2023-12-19T14:57:12.216913+0000","last_became_peered":"2023-12-19T14:57:12.216913+0000","last_unstale":"2023-12-19T14:57:13.236626+0000","last_undegraded":"2023-12-19T14:57:13.236626+0000","last_fullsized":"2023-12-19T14:57:13.236626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:11:37.366869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059634+0000","last_change":"2023-12-19T14:57:13.053548+0000","last_active":"2023-12-19T14:57:13.059634+0000","last_peered":"2023-12-19T14:57:13.059634+0000","last_clean":"2023-12-19T14:57:13.059634+0000","last_became_active":"2023-12-19T14:57:13.053242+0000","last_became_peered":"2023-12-19T14:57:13.053242+0000","last_unstale":"2023-12-19T14:57:13.059634+0000","last_undegraded":"2023-12-19T14:57:13.059634+0000","last_fullsized":"2023-12-19T14:57:13.059634+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:29.761636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237957+0000","last_change":"2023-12-19T14:57:12.221298+0000","last_active":"2023-12-19T14:57:13.237957+0000","last_peered":"2023-12-19T14:57:13.237957+0000","last_clean":"2023-12-19T14:57:13.237957+0000","last_became_active":"2023-12-19T14:57:12.220988+0000","last_became_peered":"2023-12-19T14:57:12.220988+0000","last_unstale":"2023-12-19T14:57:13.237957+0000","last_undegraded":"2023-12-19T14:57:13.237957+0000","last_fullsized":"2023-12-19T14:57:13.237957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:32.522139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059891+0000","last_change":"2023-12-19T14:57:13.054401+0000","last_active":"2023-12-19T14:57:13.059891+0000","last_peered":"2023-12-19T14:57:13.059891+0000","last_clean":"2023-12-19T14:57:13.059891+0000","last_became_active":"2023-12-19T14:57:13.052265+0000","last_became_peered":"2023-12-19T14:57:13.052265+0000","last_unstale":"2023-12-19T14:57:13.059891+0000","last_undegraded":"2023-12-19T14:57:13.059891+0000","last_fullsized":"2023-12-19T14:57:13.059891+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:33:30.650455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036714+0000","last_change":"2023-12-19T14:57:12.222138+0000","last_active":"2023-12-19T14:57:13.036714+0000","last_peered":"2023-12-19T14:57:13.036714+0000","last_clean":"2023-12-19T14:57:13.036714+0000","last_became_active":"2023-12-19T14:57:12.221427+0000","last_became_peered":"2023-12-19T14:57:12.221427+0000","last_unstale":"2023-12-19T14:57:13.036714+0000","last_undegraded":"2023-12-19T14:57:13.036714+0000","last_fullsized":"2023-12-19T14:57:13.036714+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:53.586375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060258+0000","last_change":"2023-12-19T14:57:13.052115+0000","last_active":"2023-12-19T14:57:13.060258+0000","last_peered":"2023-12-19T14:57:13.060258+0000","last_clean":"2023-12-19T14:57:13.060258+0000","last_became_active":"2023-12-19T14:57:13.051764+0000","last_became_peered":"2023-12-19T14:57:13.051764+0000","last_unstale":"2023-12-19T14:57:13.060258+0000","last_undegraded":"2023-12-19T14:57:13.060258+0000","last_fullsized":"2023-12-19T14:57:13.060258+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:59:13.690764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040682+0000","last_change":"2023-12-19T14:57:12.219172+0000","last_active":"2023-12-19T14:57:13.040682+0000","last_peered":"2023-12-19T14:57:13.040682+0000","last_clean":"2023-12-19T14:57:13.040682+0000","last_became_active":"2023-12-19T14:57:12.218873+0000","last_became_peered":"2023-12-19T14:57:12.218873+0000","last_unstale":"2023-12-19T14:57:13.040682+0000","last_undegraded":"2023-12-19T14:57:13.040682+0000","last_fullsized":"2023-12-19T14:57:13.040682+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:59.600265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062246+0000","last_change":"2023-12-19T14:57:13.059382+0000","last_active":"2023-12-19T14:57:13.062246+0000","last_peered":"2023-12-19T14:57:13.062246+0000","last_clean":"2023-12-19T14:57:13.062246+0000","last_became_active":"2023-12-19T14:57:13.058744+0000","last_became_peered":"2023-12-19T14:57:13.058744+0000","last_unstale":"2023-12-19T14:57:13.062246+0000","last_undegraded":"2023-12-19T14:57:13.062246+0000","last_fullsized":"2023-12-19T14:57:13.062246+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:54.672177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.236449+0000","last_change":"2023-12-19T14:57:12.212082+0000","last_active":"2023-12-19T14:57:13.236449+0000","last_peered":"2023-12-19T14:57:13.236449+0000","last_clean":"2023-12-19T14:57:13.236449+0000","last_became_active":"2023-12-19T14:57:12.211611+0000","last_became_peered":"2023-12-19T14:57:12.211611+0000","last_unstale":"2023-12-19T14:57:13.236449+0000","last_undegraded":"2023-12-19T14:57:13.236449+0000","last_fullsized":"2023-12-19T14:57:13.236449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:30.612421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062248+0000","last_change":"2023-12-19T14:57:13.052430+0000","last_active":"2023-12-19T14:57:13.062248+0000","last_peered":"2023-12-19T14:57:13.062248+0000","last_clean":"2023-12-19T14:57:13.062248+0000","last_became_active":"2023-12-19T14:57:13.052051+0000","last_became_peered":"2023-12-19T14:57:13.052051+0000","last_unstale":"2023-12-19T14:57:13.062248+0000","last_undegraded":"2023-12-19T14:57:13.062248+0000","last_fullsized":"2023-12-19T14:57:13.062248+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:52:05.162666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.232822+0000","last_change":"2023-12-19T14:57:12.217199+0000","last_active":"2023-12-19T14:57:13.232822+0000","last_peered":"2023-12-19T14:57:13.232822+0000","last_clean":"2023-12-19T14:57:13.232822+0000","last_became_active":"2023-12-19T14:57:12.216830+0000","last_became_peered":"2023-12-19T14:57:12.216830+0000","last_unstale":"2023-12-19T14:57:13.232822+0000","last_undegraded":"2023-12-19T14:57:13.232822+0000","last_fullsized":"2023-12-19T14:57:13.232822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:31.265320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061907+0000","last_change":"2023-12-19T14:57:13.057158+0000","last_active":"2023-12-19T14:57:13.061907+0000","last_peered":"2023-12-19T14:57:13.061907+0000","last_clean":"2023-12-19T14:57:13.061907+0000","last_became_active":"2023-12-19T14:57:13.056859+0000","last_became_peered":"2023-12-19T14:57:13.056859+0000","last_unstale":"2023-12-19T14:57:13.061907+0000","last_undegraded":"2023-12-19T14:57:13.061907+0000","last_fullsized":"2023-12-19T14:57:13.061907+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:31.625153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234428+0000","last_change":"2023-12-19T14:57:12.216557+0000","last_active":"2023-12-19T14:57:13.234428+0000","last_peered":"2023-12-19T14:57:13.234428+0000","last_clean":"2023-12-19T14:57:13.234428+0000","last_became_active":"2023-12-19T14:57:12.216244+0000","last_became_peered":"2023-12-19T14:57:12.216244+0000","last_unstale":"2023-12-19T14:57:13.234428+0000","last_undegraded":"2023-12-19T14:57:13.234428+0000","last_fullsized":"2023-12-19T14:57:13.234428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:10:04.428849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063582+0000","last_change":"2023-12-19T14:57:13.058968+0000","last_active":"2023-12-19T14:57:13.063582+0000","last_peered":"2023-12-19T14:57:13.063582+0000","last_clean":"2023-12-19T14:57:13.063582+0000","last_became_active":"2023-12-19T14:57:13.058608+0000","last_became_peered":"2023-12-19T14:57:13.058608+0000","last_unstale":"2023-12-19T14:57:13.063582+0000","last_undegraded":"2023-12-19T14:57:13.063582+0000","last_fullsized":"2023-12-19T14:57:13.063582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:44.560623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.039560+0000","last_change":"2023-12-19T14:57:12.218553+0000","last_active":"2023-12-19T14:57:13.039560+0000","last_peered":"2023-12-19T14:57:13.039560+0000","last_clean":"2023-12-19T14:57:13.039560+0000","last_became_active":"2023-12-19T14:57:12.218138+0000","last_became_peered":"2023-12-19T14:57:12.218138+0000","last_unstale":"2023-12-19T14:57:13.039560+0000","last_undegraded":"2023-12-19T14:57:13.039560+0000","last_fullsized":"2023-12-19T14:57:13.039560+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:30.612421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061316+0000","last_change":"2023-12-19T14:57:13.057204+0000","last_active":"2023-12-19T14:57:13.061316+0000","last_peered":"2023-12-19T14:57:13.061316+0000","last_clean":"2023-12-19T14:57:13.061316+0000","last_became_active":"2023-12-19T14:57:13.056845+0000","last_became_peered":"2023-12-19T14:57:13.056845+0000","last_unstale":"2023-12-19T14:57:13.061316+0000","last_undegraded":"2023-12-19T14:57:13.061316+0000","last_fullsized":"2023-12-19T14:57:13.061316+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:12.493809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.235653+0000","last_change":"2023-12-19T14:57:12.220138+0000","last_active":"2023-12-19T14:57:13.235653+0000","last_peered":"2023-12-19T14:57:13.235653+0000","last_clean":"2023-12-19T14:57:13.235653+0000","last_became_active":"2023-12-19T14:57:12.218313+0000","last_became_peered":"2023-12-19T14:57:12.218313+0000","last_unstale":"2023-12-19T14:57:13.235653+0000","last_undegraded":"2023-12-19T14:57:13.235653+0000","last_fullsized":"2023-12-19T14:57:13.235653+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:20.412744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067475+0000","last_change":"2023-12-19T14:57:13.062603+0000","last_active":"2023-12-19T14:57:13.067475+0000","last_peered":"2023-12-19T14:57:13.067475+0000","last_clean":"2023-12-19T14:57:13.067475+0000","last_became_active":"2023-12-19T14:57:13.060694+0000","last_became_peered":"2023-12-19T14:57:13.060694+0000","last_unstale":"2023-12-19T14:57:13.067475+0000","last_undegraded":"2023-12-19T14:57:13.067475+0000","last_fullsized":"2023-12-19T14:57:13.067475+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:08:19.636446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.041990+0000","last_change":"2023-12-19T14:57:12.216347+0000","last_active":"2023-12-19T14:57:13.041990+0000","last_peered":"2023-12-19T14:57:13.041990+0000","last_clean":"2023-12-19T14:57:13.041990+0000","last_became_active":"2023-12-19T14:57:12.216011+0000","last_became_peered":"2023-12-19T14:57:12.216011+0000","last_unstale":"2023-12-19T14:57:13.041990+0000","last_undegraded":"2023-12-19T14:57:13.041990+0000","last_fullsized":"2023-12-19T14:57:13.041990+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:57.871908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060474+0000","last_change":"2023-12-19T14:57:13.057401+0000","last_active":"2023-12-19T14:57:13.060474+0000","last_peered":"2023-12-19T14:57:13.060474+0000","last_clean":"2023-12-19T14:57:13.060474+0000","last_became_active":"2023-12-19T14:57:13.056897+0000","last_became_peered":"2023-12-19T14:57:13.056897+0000","last_unstale":"2023-12-19T14:57:13.060474+0000","last_undegraded":"2023-12-19T14:57:13.060474+0000","last_fullsized":"2023-12-19T14:57:13.060474+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:51.886340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.037952+0000","last_change":"2023-12-19T14:57:12.213984+0000","last_active":"2023-12-19T14:57:13.037952+0000","last_peered":"2023-12-19T14:57:13.037952+0000","last_clean":"2023-12-19T14:57:13.037952+0000","last_became_active":"2023-12-19T14:57:12.213496+0000","last_became_peered":"2023-12-19T14:57:12.213496+0000","last_unstale":"2023-12-19T14:57:13.037952+0000","last_undegraded":"2023-12-19T14:57:13.037952+0000","last_fullsized":"2023-12-19T14:57:13.037952+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:40.508592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058440+0000","last_change":"2023-12-19T14:57:13.051432+0000","last_active":"2023-12-19T14:57:13.058440+0000","last_peered":"2023-12-19T14:57:13.058440+0000","last_clean":"2023-12-19T14:57:13.058440+0000","last_became_active":"2023-12-19T14:57:13.051082+0000","last_became_peered":"2023-12-19T14:57:13.051082+0000","last_unstale":"2023-12-19T14:57:13.058440+0000","last_undegraded":"2023-12-19T14:57:13.058440+0000","last_fullsized":"2023-12-19T14:57:13.058440+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:38:12.466924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040102+0000","last_change":"2023-12-19T14:57:12.214861+0000","last_active":"2023-12-19T14:57:13.040102+0000","last_peered":"2023-12-19T14:57:13.040102+0000","last_clean":"2023-12-19T14:57:13.040102+0000","last_became_active":"2023-12-19T14:57:12.214093+0000","last_became_peered":"2023-12-19T14:57:12.214093+0000","last_unstale":"2023-12-19T14:57:13.040102+0000","last_undegraded":"2023-12-19T14:57:13.040102+0000","last_fullsized":"2023-12-19T14:57:13.040102+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:27.389000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066592+0000","last_change":"2023-12-19T14:57:13.061141+0000","last_active":"2023-12-19T14:57:13.066592+0000","last_peered":"2023-12-19T14:57:13.066592+0000","last_clean":"2023-12-19T14:57:13.066592+0000","last_became_active":"2023-12-19T14:57:13.058882+0000","last_became_peered":"2023-12-19T14:57:13.058882+0000","last_unstale":"2023-12-19T14:57:13.066592+0000","last_undegraded":"2023-12-19T14:57:13.066592+0000","last_fullsized":"2023-12-19T14:57:13.066592+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:10.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237722+0000","last_change":"2023-12-19T14:57:12.219381+0000","last_active":"2023-12-19T14:57:13.237722+0000","last_peered":"2023-12-19T14:57:13.237722+0000","last_clean":"2023-12-19T14:57:13.237722+0000","last_became_active":"2023-12-19T14:57:12.217783+0000","last_became_peered":"2023-12-19T14:57:12.217783+0000","last_unstale":"2023-12-19T14:57:13.237722+0000","last_undegraded":"2023-12-19T14:57:13.237722+0000","last_fullsized":"2023-12-19T14:57:13.237722+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:06:36.734033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.048407+0000","last_change":"2023-12-19T14:57:13.046757+0000","last_active":"2023-12-19T14:57:13.048407+0000","last_peered":"2023-12-19T14:57:13.048407+0000","last_clean":"2023-12-19T14:57:13.048407+0000","last_became_active":"2023-12-19T14:57:13.046399+0000","last_became_peered":"2023-12-19T14:57:13.046399+0000","last_unstale":"2023-12-19T14:57:13.048407+0000","last_undegraded":"2023-12-19T14:57:13.048407+0000","last_fullsized":"2023-12-19T14:57:13.048407+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:10.953482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042767+0000","last_change":"2023-12-19T14:57:12.222206+0000","last_active":"2023-12-19T14:57:13.042767+0000","last_peered":"2023-12-19T14:57:13.042767+0000","last_clean":"2023-12-19T14:57:13.042767+0000","last_became_active":"2023-12-19T14:57:12.221481+0000","last_became_peered":"2023-12-19T14:57:12.221481+0000","last_unstale":"2023-12-19T14:57:13.042767+0000","last_undegraded":"2023-12-19T14:57:13.042767+0000","last_fullsized":"2023-12-19T14:57:13.042767+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:09.940751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058281+0000","last_change":"2023-12-19T14:57:13.053321+0000","last_active":"2023-12-19T14:57:13.058281+0000","last_peered":"2023-12-19T14:57:13.058281+0000","last_clean":"2023-12-19T14:57:13.058281+0000","last_became_active":"2023-12-19T14:57:13.052585+0000","last_became_peered":"2023-12-19T14:57:13.052585+0000","last_unstale":"2023-12-19T14:57:13.058281+0000","last_undegraded":"2023-12-19T14:57:13.058281+0000","last_fullsized":"2023-12-19T14:57:13.058281+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:32.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042865+0000","last_change":"2023-12-19T14:57:12.210379+0000","last_active":"2023-12-19T14:57:13.042865+0000","last_peered":"2023-12-19T14:57:13.042865+0000","last_clean":"2023-12-19T14:57:13.042865+0000","last_became_active":"2023-12-19T14:57:12.209728+0000","last_became_peered":"2023-12-19T14:57:12.209728+0000","last_unstale":"2023-12-19T14:57:13.042865+0000","last_undegraded":"2023-12-19T14:57:13.042865+0000","last_fullsized":"2023-12-19T14:57:13.042865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:17:55.429946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056534+0000","last_change":"2023-12-19T14:57:13.045359+0000","last_active":"2023-12-19T14:57:13.056534+0000","last_peered":"2023-12-19T14:57:13.056534+0000","last_clean":"2023-12-19T14:57:13.056534+0000","last_became_active":"2023-12-19T14:57:13.045015+0000","last_became_peered":"2023-12-19T14:57:13.045015+0000","last_unstale":"2023-12-19T14:57:13.056534+0000","last_undegraded":"2023-12-19T14:57:13.056534+0000","last_fullsized":"2023-12-19T14:57:13.056534+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:16:50.173814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.031971+0000","last_change":"2023-12-19T14:57:12.223438+0000","last_active":"2023-12-19T14:57:13.031971+0000","last_peered":"2023-12-19T14:57:13.031971+0000","last_clean":"2023-12-19T14:57:13.031971+0000","last_became_active":"2023-12-19T14:57:12.223129+0000","last_became_peered":"2023-12-19T14:57:12.223129+0000","last_unstale":"2023-12-19T14:57:13.031971+0000","last_undegraded":"2023-12-19T14:57:13.031971+0000","last_fullsized":"2023-12-19T14:57:13.031971+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:22.599848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055533+0000","last_change":"2023-12-19T14:57:13.049393+0000","last_active":"2023-12-19T14:57:13.055533+0000","last_peered":"2023-12-19T14:57:13.055533+0000","last_clean":"2023-12-19T14:57:13.055533+0000","last_became_active":"2023-12-19T14:57:13.049073+0000","last_became_peered":"2023-12-19T14:57:13.049073+0000","last_unstale":"2023-12-19T14:57:13.055533+0000","last_undegraded":"2023-12-19T14:57:13.055533+0000","last_fullsized":"2023-12-19T14:57:13.055533+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:49:51.805454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040697+0000","last_change":"2023-12-19T14:57:12.218609+0000","last_active":"2023-12-19T14:57:13.040697+0000","last_peered":"2023-12-19T14:57:13.040697+0000","last_clean":"2023-12-19T14:57:13.040697+0000","last_became_active":"2023-12-19T14:57:12.218315+0000","last_became_peered":"2023-12-19T14:57:12.218315+0000","last_unstale":"2023-12-19T14:57:13.040697+0000","last_undegraded":"2023-12-19T14:57:13.040697+0000","last_fullsized":"2023-12-19T14:57:13.040697+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:53.321470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060602+0000","last_change":"2023-12-19T14:57:13.052171+0000","last_active":"2023-12-19T14:57:13.060602+0000","last_peered":"2023-12-19T14:57:13.060602+0000","last_clean":"2023-12-19T14:57:13.060602+0000","last_became_active":"2023-12-19T14:57:13.051822+0000","last_became_peered":"2023-12-19T14:57:13.051822+0000","last_unstale":"2023-12-19T14:57:13.060602+0000","last_undegraded":"2023-12-19T14:57:13.060602+0000","last_fullsized":"2023-12-19T14:57:13.060602+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:12.839228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042127+0000","last_change":"2023-12-19T14:57:12.224789+0000","last_active":"2023-12-19T14:57:13.042127+0000","last_peered":"2023-12-19T14:57:13.042127+0000","last_clean":"2023-12-19T14:57:13.042127+0000","last_became_active":"2023-12-19T14:57:12.224486+0000","last_became_peered":"2023-12-19T14:57:12.224486+0000","last_unstale":"2023-12-19T14:57:13.042127+0000","last_undegraded":"2023-12-19T14:57:13.042127+0000","last_fullsized":"2023-12-19T14:57:13.042127+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:10.271647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062446+0000","last_change":"2023-12-19T14:57:13.058194+0000","last_active":"2023-12-19T14:57:13.062446+0000","last_peered":"2023-12-19T14:57:13.062446+0000","last_clean":"2023-12-19T14:57:13.062446+0000","last_became_active":"2023-12-19T14:57:13.057832+0000","last_became_peered":"2023-12-19T14:57:13.057832+0000","last_unstale":"2023-12-19T14:57:13.062446+0000","last_undegraded":"2023-12-19T14:57:13.062446+0000","last_fullsized":"2023-12-19T14:57:13.062446+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.038052+0000","last_change":"2023-12-19T14:57:12.213799+0000","last_active":"2023-12-19T14:57:13.038052+0000","last_peered":"2023-12-19T14:57:13.038052+0000","last_clean":"2023-12-19T14:57:13.038052+0000","last_became_active":"2023-12-19T14:57:12.213158+0000","last_became_peered":"2023-12-19T14:57:12.213158+0000","last_unstale":"2023-12-19T14:57:13.038052+0000","last_undegraded":"2023-12-19T14:57:13.038052+0000","last_fullsized":"2023-12-19T14:57:13.038052+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:52:46.558984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056430+0000","last_change":"2023-12-19T14:57:13.050530+0000","last_active":"2023-12-19T14:57:13.056430+0000","last_peered":"2023-12-19T14:57:13.056430+0000","last_clean":"2023-12-19T14:57:13.056430+0000","last_became_active":"2023-12-19T14:57:13.050165+0000","last_became_peered":"2023-12-19T14:57:13.050165+0000","last_unstale":"2023-12-19T14:57:13.056430+0000","last_undegraded":"2023-12-19T14:57:13.056430+0000","last_fullsized":"2023-12-19T14:57:13.056430+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:15.738373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040184+0000","last_change":"2023-12-19T14:57:12.220869+0000","last_active":"2023-12-19T14:57:13.040184+0000","last_peered":"2023-12-19T14:57:13.040184+0000","last_clean":"2023-12-19T14:57:13.040184+0000","last_became_active":"2023-12-19T14:57:12.219736+0000","last_became_peered":"2023-12-19T14:57:12.219736+0000","last_unstale":"2023-12-19T14:57:13.040184+0000","last_undegraded":"2023-12-19T14:57:13.040184+0000","last_fullsized":"2023-12-19T14:57:13.040184+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:21.868897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056626+0000","last_change":"2023-12-19T14:57:13.047550+0000","last_active":"2023-12-19T14:57:13.056626+0000","last_peered":"2023-12-19T14:57:13.056626+0000","last_clean":"2023-12-19T14:57:13.056626+0000","last_became_active":"2023-12-19T14:57:13.047283+0000","last_became_peered":"2023-12-19T14:57:13.047283+0000","last_unstale":"2023-12-19T14:57:13.056626+0000","last_undegraded":"2023-12-19T14:57:13.056626+0000","last_fullsized":"2023-12-19T14:57:13.056626+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:46.498057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234853+0000","last_change":"2023-12-19T14:57:12.217729+0000","last_active":"2023-12-19T14:57:13.234853+0000","last_peered":"2023-12-19T14:57:13.234853+0000","last_clean":"2023-12-19T14:57:13.234853+0000","last_became_active":"2023-12-19T14:57:12.217442+0000","last_became_peered":"2023-12-19T14:57:12.217442+0000","last_unstale":"2023-12-19T14:57:13.234853+0000","last_undegraded":"2023-12-19T14:57:13.234853+0000","last_fullsized":"2023-12-19T14:57:13.234853+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:02.298531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'227","reported_seq":643,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036205+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:57:13.036205+0000","last_peered":"2023-12-19T14:57:13.036205+0000","last_clean":"2023-12-19T14:57:13.036205+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:57:13.036205+0000","last_undegraded":"2023-12-19T14:57:13.036205+0000","last_fullsized":"2023-12-19T14:57:13.036205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575065,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1386783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.103,"15min":1.093},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.52300000000000002},"max":{"1min":1.4730000000000001,"5min":3.544,"15min":3.544},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.159,"15min":1.143},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5780000000000001,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.0349999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.82799999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.6759999999999999,"5min":1.722,"15min":1.73},"last":1.081},{"interface":"front","average":{"1min":1.222,"5min":1.1839999999999999,"15min":1.1879999999999999},"min":{"1min":0.59999999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.548,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.147}]},{"osd":2,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.90100000000000002,"5min":0.68400000000000005,"15min":0.67900000000000005},"max":{"1min":1.583,"5min":1.5980000000000001,"15min":1.6919999999999999},"last":1.006},{"interface":"front","average":{"1min":1.329,"5min":1.254,"15min":1.2769999999999999},"min":{"1min":0.96599999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6890000000000001},"last":1.3600000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.238,"15min":1.282},"min":{"1min":0.79700000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.601,"5min":1.649,"15min":1.7969999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3080000000000001,"15min":1.3109999999999999},"min":{"1min":0.99299999999999999,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.6339999999999999,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.234}]},{"osd":4,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.1559999999999999,"15min":1.153},"min":{"1min":0.72699999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6930000000000001,"5min":1.752,"15min":1.752},"last":1.385},{"interface":"front","average":{"1min":1.208,"5min":1.2310000000000001,"15min":1.208},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.67500000000000004},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.7290000000000001},"last":1.115}]},{"osd":5,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2210000000000001,"15min":1.2150000000000001},"min":{"1min":0.875,"5min":0.496,"15min":0.496},"max":{"1min":1.73,"5min":1.73,"15min":1.8109999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.3089999999999999,"15min":1.3029999999999999},"min":{"1min":0.754,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6679999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.389,"15min":1.3859999999999999},"min":{"1min":0.78200000000000003,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.833,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.379,"15min":1.3520000000000001},"min":{"1min":0.80700000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.2130000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302964,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068876,"statfs":{"total":96636764160,"available":96326529024,"internally_reserved":0,"allocated":1617920,"data_stored":805003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.131,"15min":1.131},"min":{"1min":0.68400000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.611,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.137,"15min":1.1659999999999999},"min":{"1min":0.73499999999999999,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.5629999999999999,"5min":1.845,"15min":1.845},"last":1.012}]},{"osd":1,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1679999999999999,"15min":1.1879999999999999},"min":{"1min":0.78800000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.403,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.242,"15min":1.2509999999999999},"min":{"1min":0.95999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.601,"5min":3.226,"15min":3.226},"last":1.286}]},{"osd":2,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.266,"15min":1.244},"min":{"1min":0.88300000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6299999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.218,"5min":1.2350000000000001,"15min":1.2430000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.70799999999999996},"max":{"1min":1.6379999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.0549999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2989999999999999,"15min":1.3},"min":{"1min":1.036,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.536,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.468},{"interface":"front","average":{"1min":1.327,"5min":1.3340000000000001,"15min":1.3240000000000001},"min":{"1min":0.89200000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8779999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.4410000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.206},"min":{"1min":0.82099999999999995,"5min":0.68300000000000005,"15min":0.621},"max":{"1min":1.7,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.282,"5min":1.3009999999999999,"15min":1.296},"min":{"1min":0.85999999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6479999999999999,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":1.1659999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.262,"15min":1.286},"min":{"1min":0.78000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7070000000000001,"5min":2.02,"15min":2.0219999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.3740000000000001,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.82799999999999996,"15min":0.76300000000000001},"max":{"1min":1.754,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.383}]},{"osd":7,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4419999999999999,"15min":1.413},"min":{"1min":0.97799999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.847,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.4510000000000001,"15min":1.45},"min":{"1min":1.0940000000000001,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":2.0910000000000002,"5min":3.742,"15min":3.742},"last":1.3200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302944,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068896,"statfs":{"total":96636764160,"available":96326549504,"internally_reserved":0,"allocated":1597440,"data_stored":802891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0640000000000001,"15min":1.0680000000000001},"min":{"1min":0.82499999999999996,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5409999999999999,"5min":1.601,"15min":1.601},"last":1.083},{"interface":"front","average":{"1min":1.22,"5min":1.1259999999999999,"15min":1.137},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.518,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.754}]},{"osd":1,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.264,"15min":1.2929999999999999},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.863,"5min":1.863,"15min":1.9650000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2270000000000001,"15min":1.238},"min":{"1min":0.82299999999999995,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.5940000000000001,"5min":1.677,"15min":1.677},"last":1.2689999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.147,"15min":1.2210000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.47399999999999998},"max":{"1min":1.5109999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.359},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.159,"15min":1.1890000000000001},"min":{"1min":0.90000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.7,"5min":1.7,"15min":1.954},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2569999999999999,"15min":1.27},"min":{"1min":0.70699999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.637,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.254,"15min":1.2649999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.68999999999999995},"max":{"1min":1.724,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.123,"15min":1.103},"min":{"1min":0.86499999999999999,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.238,"15min":1.2350000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6559999999999999,"5min":2.27,"15min":2.27},"last":1.1399999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.335,"15min":1.329},"min":{"1min":0.98099999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9099999999999999,"5min":1.982,"15min":2.0630000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.327,"15min":1.321},"min":{"1min":0.90300000000000002,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.726,"5min":1.8540000000000001,"15min":1.9219999999999999},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3600000000000001,"15min":1.3500000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.58899999999999997},"max":{"1min":1.7509999999999999,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.387,"15min":1.377},"min":{"1min":0.871,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.048,"5min":2.048,"15min":2.2080000000000002},"last":1.393}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068900,"statfs":{"total":96636764160,"available":96326553600,"internally_reserved":0,"allocated":1593344,"data_stored":796347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.034,"15min":1.0629999999999999},"min":{"1min":0.627,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.49,"5min":1.625,"15min":1.7869999999999999},"last":1.454},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.78500000000000003,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.5189999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2729999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2210000000000001,"15min":1.2350000000000001},"min":{"1min":0.65800000000000003,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.224,"15min":1.268},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.677,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.3089999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.157,"15min":1.143},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1479999999999999,"15min":1.1990000000000001},"min":{"1min":0.81100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.625,"5min":1.7649999999999999,"15min":1.893},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.161,"15min":1.1739999999999999},"min":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.6000000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.23,"15min":1.2549999999999999},"min":{"1min":0.86799999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.534,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.4950000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.165,"15min":1.1379999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.57899999999999996},"max":{"1min":1.4650000000000001,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.149,"15min":1.1439999999999999},"min":{"1min":0.73899999999999999,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.4930000000000001,"5min":2.077,"15min":2.077},"last":1.089}]},{"osd":6,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.278,"15min":1.294},"min":{"1min":0.79700000000000004,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.6439999999999999,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.349},{"interface":"front","average":{"1min":1.367,"5min":1.405,"15min":1.399},"min":{"1min":0.81599999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7190000000000001,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.6040000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3109999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.60899999999999999},"max":{"1min":1.639,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.4330000000000001,"15min":1.427},"min":{"1min":0.755,"5min":0.755,"15min":0.72599999999999998},"max":{"1min":1.7110000000000001,"5min":3.008,"15min":3.008},"last":1.6439999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":803445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.68200000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.026},{"interface":"front","average":{"1min":0.999,"5min":0.97499999999999998,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3520000000000001,"5min":1.542,"15min":1.7270000000000001},"last":1.1619999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.90300000000000002,"15min":0.93899999999999995},"min":{"1min":0.621,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.54,"5min":1.611,"15min":1.611},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0900000000000001,"15min":1.095},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.446,"5min":1.661,"15min":1.758},"last":1.3280000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1379999999999999,"15min":1.1319999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.56100000000000005},"max":{"1min":1.748,"5min":1.778,"15min":1.778},"last":1.075},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.222,"15min":1.2370000000000001},"min":{"1min":0.81499999999999995,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.575,"5min":1.72,"15min":1.8520000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3280000000000001,"15min":1.327},"min":{"1min":1.0820000000000001,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.899},"last":1.194},{"interface":"front","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.2929999999999999},"min":{"1min":0.92800000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.504,"5min":1.6879999999999999,"15min":1.7989999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2829999999999999,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.75,"5min":1.7709999999999999,"15min":1.893},"last":1},{"interface":"front","average":{"1min":1.413,"5min":1.371,"15min":1.3879999999999999},"min":{"1min":1.123,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.8120000000000001,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.5429999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4390000000000001,"15min":1.45},"min":{"1min":1.2629999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8839999999999999,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4139999999999999,"15min":1.415},"min":{"1min":1.123,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.9059999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.5269999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.512,"15min":1.51},"min":{"1min":1.097,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.1640000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4510000000000001,"15min":1.456},"min":{"1min":1.1499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.26}]}]},{"osd":2,"up_from":13,"seq":55834575066,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307028,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064812,"statfs":{"total":96636764160,"available":96322367488,"internally_reserved":0,"allocated":1585152,"data_stored":795278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91500000000000004,"15min":0.90600000000000003},"min":{"1min":0.60199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.258,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.022,"5min":1.0169999999999999,"15min":1.0669999999999999},"min":{"1min":0.70199999999999996,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3,"5min":1.629,"15min":1.629},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.109,"15min":1.133},"min":{"1min":0.69499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4419999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.048,"15min":1.093},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4490000000000001,"5min":1.853,"15min":1.853},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.998,"15min":0.98999999999999999},"min":{"1min":0.58599999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.4419999999999999,"5min":1.5669999999999999,"15min":1.623},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0740000000000001,"15min":1.095},"min":{"1min":0.63,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.6080000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.012}]},{"osd":4,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3620000000000001,"15min":1.3759999999999999},"min":{"1min":0.747,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.716,"5min":2.0270000000000001,"15min":2.044},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.371,"15min":1.391},"min":{"1min":0.91000000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.7410000000000001,"5min":1.96,"15min":1.96},"last":1.431}]},{"osd":5,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3480000000000001,"15min":1.3400000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.679,"5min":2.0979999999999999,"15min":2.395},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.3160000000000001,"15min":1.3280000000000001},"min":{"1min":0.98499999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":2.1200000000000001},"last":1.2949999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":0.99199999999999999,"5min":0.85699999999999998,"15min":0.83299999999999996},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.069},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4379999999999999,"15min":1.431},"min":{"1min":0.95699999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.921,"5min":2.1840000000000002,"15min":2.4729999999999999},"last":1.698}]},{"osd":7,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.472,"15min":1.4730000000000001},"min":{"1min":0.95199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9399999999999999,"5min":2.0699999999999998,"15min":2.2080000000000002},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.48,"15min":1.4730000000000001},"min":{"1min":1.05,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":796289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.88300000000000001,"15min":0.86599999999999999},"min":{"1min":0.54400000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.206,"5min":2.238,"15min":2.238},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94599999999999995,"15min":0.97699999999999998},"min":{"1min":0.65600000000000003,"5min":0.57199999999999995,"15min":0.53100000000000003},"max":{"1min":1.49,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.486}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.02,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.506,"5min":2.444,"15min":2.444},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.075,"15min":1.1140000000000001},"min":{"1min":0.68300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.3740000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.829}]},{"osd":3,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.204,"15min":1.23},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.573,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.369},{"interface":"front","average":{"1min":1.153,"5min":1.081,"15min":1.0860000000000001},"min":{"1min":0.628,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6000000000000001,"5min":1.6890000000000001,"15min":1.7270000000000001},"last":0.92500000000000004}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.2949999999999999,"15min":1.304},"min":{"1min":0.93799999999999994,"5min":0.624,"15min":0.624},"max":{"1min":1.974,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.3660000000000001,"15min":1.3939999999999999},"min":{"1min":0.95899999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7529999999999999,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.7190000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.27,"15min":1.292},"min":{"1min":1.0109999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8160000000000001,"5min":1.857,"15min":1.857},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3959999999999999,"15min":1.3859999999999999},"min":{"1min":1.0089999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.2029999999999998,"5min":2.665,"15min":2.665},"last":2.1339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3420000000000001,"15min":1.365},"min":{"1min":1.071,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4319999999999999,"15min":1.446},"min":{"1min":1.0109999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.129,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.331}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4239999999999999,"15min":1.468},"min":{"1min":1.0349999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.923},{"interface":"front","average":{"1min":1.51,"5min":1.4379999999999999,"15min":1.4359999999999999},"min":{"1min":1.125,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0609999999999999},"last":1.863}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1387238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.84799999999999998,"15min":0.84999999999999998},"min":{"1min":0.65800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3759999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.98199999999999998,"15min":0.96999999999999997},"min":{"1min":0.748,"5min":0.433,"15min":0.433},"max":{"1min":1.577,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.1140000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1240000000000001,"15min":1.1080000000000001},"min":{"1min":0.81599999999999995,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.6639999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.282,"5min":1.2210000000000001,"15min":1.2250000000000001},"min":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.55400000000000005},"max":{"1min":1.821,"5min":1.982,"15min":1.982},"last":1.7909999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.012,"15min":1.0229999999999999},"min":{"1min":0.76100000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.5,"5min":1.726,"15min":2.0299999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.175,"15min":1.1870000000000001},"min":{"1min":0.88700000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.756,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.0309999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2909999999999999,"15min":1.2969999999999999},"min":{"1min":0.80400000000000005,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.605,"5min":2.153,"15min":2.153},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.288,"15min":1.2909999999999999},"min":{"1min":1.0229999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.613,"5min":1.7170000000000001,"15min":1.7929999999999999},"last":1.2609999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.383,"15min":1.3819999999999999},"min":{"1min":1.1839999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.4630000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3640000000000001,"15min":1.3899999999999999},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7210000000000001,"5min":2.2189999999999999,"15min":2.2229999999999999},"last":1.552}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3140000000000001,"15min":1.3300000000000001},"min":{"1min":1.0209999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.5960000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4490000000000001,"15min":1.4470000000000001},"min":{"1min":1.1399999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.9710000000000001,"5min":2.113,"15min":2.3799999999999999},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4339999999999999,"15min":1.4550000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7649999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":1.0920000000000001,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.746,"5min":1.931,"15min":2.0379999999999998},"last":1.613}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:57:17.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:57:17.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.694+0000 7f89a68cd700 1 Processor -- start 2023-12-19T14:57:17.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.698+0000 7f89a68cd700 1 -- start start 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f89a015d320 0x7f89a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a00c15b0 0x7f89a00aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 0x7f89a00680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a00c7190 con 0x7f89a006d690 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a00c1b50 con 0x7f89a015d320 2023-12-19T14:57:17.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a00c1e30 con 0x7f89a00c15b0 2023-12-19T14:57:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a4e6a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 0x7f89a00680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f89a015d320 0x7f89a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a4e6a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 0x7f89a00680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45124/0 (socket says 172.21.15.50:45124) 2023-12-19T14:57:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a4e6a700 1 -- 172.21.15.50:0/3115038896 learned_addr learned my addr 172.21.15.50:0/3115038896 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:17.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f8997fff700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a00c15b0 0x7f89a00aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 -- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a00c15b0 msgr2=0x7f89a00aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a00c15b0 0x7f89a00aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 -- 172.21.15.50:0/3115038896 >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 msgr2=0x7f89a00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 --2- 172.21.15.50:0/3115038896 >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 0x7f89a00680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 -- 172.21.15.50:0/3115038896 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a00c6e20 con 0x7f89a015d320 2023-12-19T14:57:17.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899ffff700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a015d320 0x7f89a015d6f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cbd40 tx=0x7f89900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf7903e1e5fb2aa2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899f7fe700 1 -- 172.21.15.50:0/3115038896 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990011630 con 0x7f89a015d320 2023-12-19T14:57:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899f7fe700 1 -- 172.21.15.50:0/3115038896 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89900117b0 con 0x7f89a015d320 2023-12-19T14:57:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f899f7fe700 1 -- 172.21.15.50:0/3115038896 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990011b70 con 0x7f89a015d320 2023-12-19T14:57:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 -- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a015d320 msgr2=0x7f89a015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.702+0000 7f89a68cd700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a015d320 0x7f89a015d6f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cbd40 tx=0x7f89900091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 -- 172.21.15.50:0/3115038896 shutdown_connections 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 --2- 172.21.15.50:0/3115038896 >> v2:172.21.15.50:3300/0 conn(0x7f89a006d690 0x7f89a00680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a00c15b0 0x7f89a00aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 --2- 172.21.15.50:0/3115038896 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a015d320 0x7f89a015d6f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 -- 172.21.15.50:0/3115038896 >> 172.21.15.50:0/3115038896 conn(0x7f89a00c8850 msgr2=0x7f89a00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 -- 172.21.15.50:0/3115038896 shutdown_connections 2023-12-19T14:57:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 -- 172.21.15.50:0/3115038896 wait complete. 2023-12-19T14:57:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 Processor -- start 2023-12-19T14:57:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.706+0000 7f89a68cd700 1 -- start start 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 0x7f89a015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 0x7f89a0161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a00c4400 con 0x7f89a015d320 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a0162790 con 0x7f89a00c15b0 2023-12-19T14:57:17.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a0170de0 con 0x7f89a006d690 2023-12-19T14:57:17.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45132/0 (socket says 172.21.15.50:45132) 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 -- 172.21.15.50:0/695790819 learned_addr learned my addr 172.21.15.50:0/695790819 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f899ffff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 0x7f89a015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f8997fff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 0x7f89a0161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 -- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 msgr2=0x7f89a015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 0x7f89a015efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 -- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 msgr2=0x7f89a0161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 0x7f89a0161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 -- 172.21.15.50:0/695790819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8990009420 con 0x7f89a015d320 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f899ffff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 0x7f89a015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f8997fff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 0x7f89a0161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:17.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a4e6a700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8998009db0 tx=0x7f899800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899800c940 con 0x7f89a015d320 2023-12-19T14:57:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899800f040 con 0x7f89a015d320 2023-12-19T14:57:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899800cd10 con 0x7f89a015d320 2023-12-19T14:57:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a01710c0 con 0x7f89a015d320 2023-12-19T14:57:17.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.710+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a0171680 con 0x7f89a015d320 2023-12-19T14:57:17.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.714+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a00c2740 con 0x7f89a015d320 2023-12-19T14:57:17.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.714+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8998014650 con 0x7f89a015d320 2023-12-19T14:57:17.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.714+0000 7f899cff9700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 0x7f8980062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.718+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f899807d330 con 0x7f89a015d320 2023-12-19T14:57:17.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.718+0000 7f899ffff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 0x7f8980062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:17.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.718+0000 7f899ffff700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 0x7f8980062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cbd40 tx=0x7f8990017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.722+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f89a00c2740 con 0x7f89a015d320 2023-12-19T14:57:17.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.890+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89a00c2740 con 0x7f8980060670 2023-12-19T14:57:17.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.906+0000 7f899cff9700 1 -- 172.21.15.50:0/695790819 <== mgr.6833 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286926 (secure 0 0 0) 0x7f89a00c2740 con 0x7f8980060670 2023-12-19T14:57:17.913 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 msgr2=0x7f8980062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 0x7f8980062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89a00cbd40 tx=0x7f8990017000 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 msgr2=0x7f89a01620f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8998009db0 tx=0x7f899800b040 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 shutdown_connections 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89a00c15b0 0x7f89a0161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8980060670 0x7f8980062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f89a015d320 0x7f89a01620f0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 --2- 172.21.15.50:0/695790819 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f89a006d690 0x7f89a015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 >> 172.21.15.50:0/695790819 conn(0x7f89a00c8850 msgr2=0x7f89a00c2030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 shutdown_connections 2023-12-19T14:57:17.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:17.914+0000 7f89a68cd700 1 -- 172.21.15.50:0/695790819 wait complete. 2023-12-19T14:57:17.919 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:57:17.942 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:57:16.683482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449284,"kb_used_data":13636,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525436,"statfs":{"total":773094113280,"available":770586046464,"internally_reserved":0,"allocated":13963264,"data_stored":7573274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.503226"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061856+0000","last_change":"2023-12-19T14:57:13.057390+0000","last_active":"2023-12-19T14:57:13.061856+0000","last_peered":"2023-12-19T14:57:13.061856+0000","last_clean":"2023-12-19T14:57:13.061856+0000","last_became_active":"2023-12-19T14:57:13.056910+0000","last_became_peered":"2023-12-19T14:57:13.056910+0000","last_unstale":"2023-12-19T14:57:13.061856+0000","last_undegraded":"2023-12-19T14:57:13.061856+0000","last_fullsized":"2023-12-19T14:57:13.061856+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:41.333180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060354+0000","last_change":"2023-12-19T14:57:13.054541+0000","last_active":"2023-12-19T14:57:13.060354+0000","last_peered":"2023-12-19T14:57:13.060354+0000","last_clean":"2023-12-19T14:57:13.060354+0000","last_became_active":"2023-12-19T14:57:13.052986+0000","last_became_peered":"2023-12-19T14:57:13.052986+0000","last_unstale":"2023-12-19T14:57:13.060354+0000","last_undegraded":"2023-12-19T14:57:13.060354+0000","last_fullsized":"2023-12-19T14:57:13.060354+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067838+0000","last_change":"2023-12-19T14:57:13.063457+0000","last_active":"2023-12-19T14:57:13.067838+0000","last_peered":"2023-12-19T14:57:13.067838+0000","last_clean":"2023-12-19T14:57:13.067838+0000","last_became_active":"2023-12-19T14:57:13.062932+0000","last_became_peered":"2023-12-19T14:57:13.062932+0000","last_unstale":"2023-12-19T14:57:13.067838+0000","last_undegraded":"2023-12-19T14:57:13.067838+0000","last_fullsized":"2023-12-19T14:57:13.067838+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:22.183822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058113+0000","last_change":"2023-12-19T14:57:13.051915+0000","last_active":"2023-12-19T14:57:13.058113+0000","last_peered":"2023-12-19T14:57:13.058113+0000","last_clean":"2023-12-19T14:57:13.058113+0000","last_became_active":"2023-12-19T14:57:13.051590+0000","last_became_peered":"2023-12-19T14:57:13.051590+0000","last_unstale":"2023-12-19T14:57:13.058113+0000","last_undegraded":"2023-12-19T14:57:13.058113+0000","last_fullsized":"2023-12-19T14:57:13.058113+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:20.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062033+0000","last_change":"2023-12-19T14:57:13.051286+0000","last_active":"2023-12-19T14:57:13.062033+0000","last_peered":"2023-12-19T14:57:13.062033+0000","last_clean":"2023-12-19T14:57:13.062033+0000","last_became_active":"2023-12-19T14:57:13.050979+0000","last_became_peered":"2023-12-19T14:57:13.050979+0000","last_unstale":"2023-12-19T14:57:13.062033+0000","last_undegraded":"2023-12-19T14:57:13.062033+0000","last_fullsized":"2023-12-19T14:57:13.062033+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067129+0000","last_change":"2023-12-19T14:57:13.066300+0000","last_active":"2023-12-19T14:57:13.067129+0000","last_peered":"2023-12-19T14:57:13.067129+0000","last_clean":"2023-12-19T14:57:13.067129+0000","last_became_active":"2023-12-19T14:57:13.065974+0000","last_became_peered":"2023-12-19T14:57:13.065974+0000","last_unstale":"2023-12-19T14:57:13.067129+0000","last_undegraded":"2023-12-19T14:57:13.067129+0000","last_fullsized":"2023-12-19T14:57:13.067129+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:43:03.428835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066106+0000","last_change":"2023-12-19T14:57:13.065390+0000","last_active":"2023-12-19T14:57:13.066106+0000","last_peered":"2023-12-19T14:57:13.066106+0000","last_clean":"2023-12-19T14:57:13.066106+0000","last_became_active":"2023-12-19T14:57:13.065105+0000","last_became_peered":"2023-12-19T14:57:13.065105+0000","last_unstale":"2023-12-19T14:57:13.066106+0000","last_undegraded":"2023-12-19T14:57:13.066106+0000","last_fullsized":"2023-12-19T14:57:13.066106+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:34.379221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055669+0000","last_change":"2023-12-19T14:57:13.049509+0000","last_active":"2023-12-19T14:57:13.055669+0000","last_peered":"2023-12-19T14:57:13.055669+0000","last_clean":"2023-12-19T14:57:13.055669+0000","last_became_active":"2023-12-19T14:57:13.049215+0000","last_became_peered":"2023-12-19T14:57:13.049215+0000","last_unstale":"2023-12-19T14:57:13.055669+0000","last_undegraded":"2023-12-19T14:57:13.055669+0000","last_fullsized":"2023-12-19T14:57:13.055669+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:21.578627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055684+0000","last_change":"2023-12-19T14:57:13.050408+0000","last_active":"2023-12-19T14:57:13.055684+0000","last_peered":"2023-12-19T14:57:13.055684+0000","last_clean":"2023-12-19T14:57:13.055684+0000","last_became_active":"2023-12-19T14:57:13.050072+0000","last_became_peered":"2023-12-19T14:57:13.050072+0000","last_unstale":"2023-12-19T14:57:13.055684+0000","last_undegraded":"2023-12-19T14:57:13.055684+0000","last_fullsized":"2023-12-19T14:57:13.055684+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:51.886340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.069674+0000","last_change":"2023-12-19T14:57:13.068520+0000","last_active":"2023-12-19T14:57:13.069674+0000","last_peered":"2023-12-19T14:57:13.069674+0000","last_clean":"2023-12-19T14:57:13.069674+0000","last_became_active":"2023-12-19T14:57:13.068146+0000","last_became_peered":"2023-12-19T14:57:13.068146+0000","last_unstale":"2023-12-19T14:57:13.069674+0000","last_undegraded":"2023-12-19T14:57:13.069674+0000","last_fullsized":"2023-12-19T14:57:13.069674+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060664+0000","last_change":"2023-12-19T14:57:13.052831+0000","last_active":"2023-12-19T14:57:13.060664+0000","last_peered":"2023-12-19T14:57:13.060664+0000","last_clean":"2023-12-19T14:57:13.060664+0000","last_became_active":"2023-12-19T14:57:13.052458+0000","last_became_peered":"2023-12-19T14:57:13.052458+0000","last_unstale":"2023-12-19T14:57:13.060664+0000","last_undegraded":"2023-12-19T14:57:13.060664+0000","last_fullsized":"2023-12-19T14:57:13.060664+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:18.331281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.051612+0000","last_change":"2023-12-19T14:57:13.047253+0000","last_active":"2023-12-19T14:57:13.051612+0000","last_peered":"2023-12-19T14:57:13.051612+0000","last_clean":"2023-12-19T14:57:13.051612+0000","last_became_active":"2023-12-19T14:57:13.046667+0000","last_became_peered":"2023-12-19T14:57:13.046667+0000","last_unstale":"2023-12-19T14:57:13.051612+0000","last_undegraded":"2023-12-19T14:57:13.051612+0000","last_fullsized":"2023-12-19T14:57:13.051612+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:16:50.173814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056697+0000","last_change":"2023-12-19T14:57:13.050982+0000","last_active":"2023-12-19T14:57:13.056697+0000","last_peered":"2023-12-19T14:57:13.056697+0000","last_clean":"2023-12-19T14:57:13.056697+0000","last_became_active":"2023-12-19T14:57:13.050449+0000","last_became_peered":"2023-12-19T14:57:13.050449+0000","last_unstale":"2023-12-19T14:57:13.056697+0000","last_undegraded":"2023-12-19T14:57:13.056697+0000","last_fullsized":"2023-12-19T14:57:13.056697+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:02:34.060332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.052516+0000","last_change":"2023-12-19T14:57:13.046092+0000","last_active":"2023-12-19T14:57:13.052516+0000","last_peered":"2023-12-19T14:57:13.052516+0000","last_clean":"2023-12-19T14:57:13.052516+0000","last_became_active":"2023-12-19T14:57:13.045741+0000","last_became_peered":"2023-12-19T14:57:13.045741+0000","last_unstale":"2023-12-19T14:57:13.052516+0000","last_undegraded":"2023-12-19T14:57:13.052516+0000","last_fullsized":"2023-12-19T14:57:13.052516+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062399+0000","last_change":"2023-12-19T14:57:13.058119+0000","last_active":"2023-12-19T14:57:13.062399+0000","last_peered":"2023-12-19T14:57:13.062399+0000","last_clean":"2023-12-19T14:57:13.062399+0000","last_became_active":"2023-12-19T14:57:13.057815+0000","last_became_peered":"2023-12-19T14:57:13.057815+0000","last_unstale":"2023-12-19T14:57:13.062399+0000","last_undegraded":"2023-12-19T14:57:13.062399+0000","last_fullsized":"2023-12-19T14:57:13.062399+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.054554+0000","last_change":"2023-12-19T14:57:13.048883+0000","last_active":"2023-12-19T14:57:13.054554+0000","last_peered":"2023-12-19T14:57:13.054554+0000","last_clean":"2023-12-19T14:57:13.054554+0000","last_became_active":"2023-12-19T14:57:13.048608+0000","last_became_peered":"2023-12-19T14:57:13.048608+0000","last_unstale":"2023-12-19T14:57:13.054554+0000","last_undegraded":"2023-12-19T14:57:13.054554+0000","last_fullsized":"2023-12-19T14:57:13.054554+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066279+0000","last_change":"2023-12-19T14:57:13.061740+0000","last_active":"2023-12-19T14:57:13.066279+0000","last_peered":"2023-12-19T14:57:13.066279+0000","last_clean":"2023-12-19T14:57:13.066279+0000","last_became_active":"2023-12-19T14:57:13.061454+0000","last_became_peered":"2023-12-19T14:57:13.061454+0000","last_unstale":"2023-12-19T14:57:13.066279+0000","last_undegraded":"2023-12-19T14:57:13.066279+0000","last_fullsized":"2023-12-19T14:57:13.066279+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:13:32.026843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067141+0000","last_change":"2023-12-19T14:57:13.066328+0000","last_active":"2023-12-19T14:57:13.067141+0000","last_peered":"2023-12-19T14:57:13.067141+0000","last_clean":"2023-12-19T14:57:13.067141+0000","last_became_active":"2023-12-19T14:57:13.065978+0000","last_became_peered":"2023-12-19T14:57:13.065978+0000","last_unstale":"2023-12-19T14:57:13.067141+0000","last_undegraded":"2023-12-19T14:57:13.067141+0000","last_fullsized":"2023-12-19T14:57:13.067141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067907+0000","last_change":"2023-12-19T14:57:13.063498+0000","last_active":"2023-12-19T14:57:13.067907+0000","last_peered":"2023-12-19T14:57:13.067907+0000","last_clean":"2023-12-19T14:57:13.067907+0000","last_became_active":"2023-12-19T14:57:13.063184+0000","last_became_peered":"2023-12-19T14:57:13.063184+0000","last_unstale":"2023-12-19T14:57:13.067907+0000","last_undegraded":"2023-12-19T14:57:13.067907+0000","last_fullsized":"2023-12-19T14:57:13.067907+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:30.268380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059486+0000","last_change":"2023-12-19T14:57:13.054866+0000","last_active":"2023-12-19T14:57:13.059486+0000","last_peered":"2023-12-19T14:57:13.059486+0000","last_clean":"2023-12-19T14:57:13.059486+0000","last_became_active":"2023-12-19T14:57:13.053860+0000","last_became_peered":"2023-12-19T14:57:13.053860+0000","last_unstale":"2023-12-19T14:57:13.059486+0000","last_undegraded":"2023-12-19T14:57:13.059486+0000","last_fullsized":"2023-12-19T14:57:13.059486+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:14:07.850723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063197+0000","last_change":"2023-12-19T14:57:13.061649+0000","last_active":"2023-12-19T14:57:13.063197+0000","last_peered":"2023-12-19T14:57:13.063197+0000","last_clean":"2023-12-19T14:57:13.063197+0000","last_became_active":"2023-12-19T14:57:13.061301+0000","last_became_peered":"2023-12-19T14:57:13.061301+0000","last_unstale":"2023-12-19T14:57:13.063197+0000","last_undegraded":"2023-12-19T14:57:13.063197+0000","last_fullsized":"2023-12-19T14:57:13.063197+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:49.473135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063617+0000","last_change":"2023-12-19T14:57:13.058318+0000","last_active":"2023-12-19T14:57:13.063617+0000","last_peered":"2023-12-19T14:57:13.063617+0000","last_clean":"2023-12-19T14:57:13.063617+0000","last_became_active":"2023-12-19T14:57:13.055248+0000","last_became_peered":"2023-12-19T14:57:13.055248+0000","last_unstale":"2023-12-19T14:57:13.063617+0000","last_undegraded":"2023-12-19T14:57:13.063617+0000","last_fullsized":"2023-12-19T14:57:13.063617+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:31.625153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067700+0000","last_change":"2023-12-19T14:57:13.063046+0000","last_active":"2023-12-19T14:57:13.067700+0000","last_peered":"2023-12-19T14:57:13.067700+0000","last_clean":"2023-12-19T14:57:13.067700+0000","last_became_active":"2023-12-19T14:57:13.062647+0000","last_became_peered":"2023-12-19T14:57:13.062647+0000","last_unstale":"2023-12-19T14:57:13.067700+0000","last_undegraded":"2023-12-19T14:57:13.067700+0000","last_fullsized":"2023-12-19T14:57:13.067700+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:23.005310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062349+0000","last_change":"2023-12-19T14:57:13.055881+0000","last_active":"2023-12-19T14:57:13.062349+0000","last_peered":"2023-12-19T14:57:13.062349+0000","last_clean":"2023-12-19T14:57:13.062349+0000","last_became_active":"2023-12-19T14:57:13.055592+0000","last_became_peered":"2023-12-19T14:57:13.055592+0000","last_unstale":"2023-12-19T14:57:13.062349+0000","last_undegraded":"2023-12-19T14:57:13.062349+0000","last_fullsized":"2023-12-19T14:57:13.062349+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:24.572601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061019+0000","last_change":"2023-12-19T14:57:13.055059+0000","last_active":"2023-12-19T14:57:13.061019+0000","last_peered":"2023-12-19T14:57:13.061019+0000","last_clean":"2023-12-19T14:57:13.061019+0000","last_became_active":"2023-12-19T14:57:13.054747+0000","last_became_peered":"2023-12-19T14:57:13.054747+0000","last_unstale":"2023-12-19T14:57:13.061019+0000","last_undegraded":"2023-12-19T14:57:13.061019+0000","last_fullsized":"2023-12-19T14:57:13.061019+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:29.761636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061703+0000","last_change":"2023-12-19T14:57:13.057919+0000","last_active":"2023-12-19T14:57:13.061703+0000","last_peered":"2023-12-19T14:57:13.061703+0000","last_clean":"2023-12-19T14:57:13.061703+0000","last_became_active":"2023-12-19T14:57:13.057595+0000","last_became_peered":"2023-12-19T14:57:13.057595+0000","last_unstale":"2023-12-19T14:57:13.061703+0000","last_undegraded":"2023-12-19T14:57:13.061703+0000","last_fullsized":"2023-12-19T14:57:13.061703+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:13:18.331281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056599+0000","last_change":"2023-12-19T14:57:13.050863+0000","last_active":"2023-12-19T14:57:13.056599+0000","last_peered":"2023-12-19T14:57:13.056599+0000","last_clean":"2023-12-19T14:57:13.056599+0000","last_became_active":"2023-12-19T14:57:13.050358+0000","last_became_peered":"2023-12-19T14:57:13.050358+0000","last_unstale":"2023-12-19T14:57:13.056599+0000","last_undegraded":"2023-12-19T14:57:13.056599+0000","last_fullsized":"2023-12-19T14:57:13.056599+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:14:07.850723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061997+0000","last_change":"2023-12-19T14:57:13.058217+0000","last_active":"2023-12-19T14:57:13.061997+0000","last_peered":"2023-12-19T14:57:13.061997+0000","last_clean":"2023-12-19T14:57:13.061997+0000","last_became_active":"2023-12-19T14:57:13.057931+0000","last_became_peered":"2023-12-19T14:57:13.057931+0000","last_unstale":"2023-12-19T14:57:13.061997+0000","last_undegraded":"2023-12-19T14:57:13.061997+0000","last_fullsized":"2023-12-19T14:57:13.061997+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:54.334202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061508+0000","last_change":"2023-12-19T14:57:13.057666+0000","last_active":"2023-12-19T14:57:13.061508+0000","last_peered":"2023-12-19T14:57:13.061508+0000","last_clean":"2023-12-19T14:57:13.061508+0000","last_became_active":"2023-12-19T14:57:13.057338+0000","last_became_peered":"2023-12-19T14:57:13.057338+0000","last_unstale":"2023-12-19T14:57:13.061508+0000","last_undegraded":"2023-12-19T14:57:13.061508+0000","last_fullsized":"2023-12-19T14:57:13.061508+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:03:49.948787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058608+0000","last_change":"2023-12-19T14:57:13.054390+0000","last_active":"2023-12-19T14:57:13.058608+0000","last_peered":"2023-12-19T14:57:13.058608+0000","last_clean":"2023-12-19T14:57:13.058608+0000","last_became_active":"2023-12-19T14:57:13.054109+0000","last_became_peered":"2023-12-19T14:57:13.054109+0000","last_unstale":"2023-12-19T14:57:13.058608+0000","last_undegraded":"2023-12-19T14:57:13.058608+0000","last_fullsized":"2023-12-19T14:57:13.058608+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:15.738373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.064038+0000","last_change":"2023-12-19T14:57:13.061294+0000","last_active":"2023-12-19T14:57:13.064038+0000","last_peered":"2023-12-19T14:57:13.064038+0000","last_clean":"2023-12-19T14:57:13.064038+0000","last_became_active":"2023-12-19T14:57:13.060693+0000","last_became_peered":"2023-12-19T14:57:13.060693+0000","last_unstale":"2023-12-19T14:57:13.064038+0000","last_undegraded":"2023-12-19T14:57:13.064038+0000","last_fullsized":"2023-12-19T14:57:13.064038+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:20.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062227+0000","last_change":"2023-12-19T14:57:13.052410+0000","last_active":"2023-12-19T14:57:13.062227+0000","last_peered":"2023-12-19T14:57:13.062227+0000","last_clean":"2023-12-19T14:57:13.062227+0000","last_became_active":"2023-12-19T14:57:13.052070+0000","last_became_peered":"2023-12-19T14:57:13.052070+0000","last_unstale":"2023-12-19T14:57:13.062227+0000","last_undegraded":"2023-12-19T14:57:13.062227+0000","last_fullsized":"2023-12-19T14:57:13.062227+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:57.340483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058646+0000","last_change":"2023-12-19T14:57:13.054165+0000","last_active":"2023-12-19T14:57:13.058646+0000","last_peered":"2023-12-19T14:57:13.058646+0000","last_clean":"2023-12-19T14:57:13.058646+0000","last_became_active":"2023-12-19T14:57:13.053800+0000","last_became_peered":"2023-12-19T14:57:13.053800+0000","last_unstale":"2023-12-19T14:57:13.058646+0000","last_undegraded":"2023-12-19T14:57:13.058646+0000","last_fullsized":"2023-12-19T14:57:13.058646+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:01:51.423899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":14,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237505+0000","last_change":"2023-12-19T14:57:12.224820+0000","last_active":"2023-12-19T14:57:13.237505+0000","last_peered":"2023-12-19T14:57:13.237505+0000","last_clean":"2023-12-19T14:57:13.237505+0000","last_became_active":"2023-12-19T14:57:12.224511+0000","last_became_peered":"2023-12-19T14:57:12.224511+0000","last_unstale":"2023-12-19T14:57:13.237505+0000","last_undegraded":"2023-12-19T14:57:13.237505+0000","last_fullsized":"2023-12-19T14:57:13.237505+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:40.508592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055484+0000","last_change":"2023-12-19T14:57:13.048726+0000","last_active":"2023-12-19T14:57:13.055484+0000","last_peered":"2023-12-19T14:57:13.055484+0000","last_clean":"2023-12-19T14:57:13.055484+0000","last_became_active":"2023-12-19T14:57:13.048406+0000","last_became_peered":"2023-12-19T14:57:13.048406+0000","last_unstale":"2023-12-19T14:57:13.055484+0000","last_undegraded":"2023-12-19T14:57:13.055484+0000","last_fullsized":"2023-12-19T14:57:13.055484+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:32.137185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.235799+0000","last_change":"2023-12-19T14:57:12.222347+0000","last_active":"2023-12-19T14:57:13.235799+0000","last_peered":"2023-12-19T14:57:13.235799+0000","last_clean":"2023-12-19T14:57:13.235799+0000","last_became_active":"2023-12-19T14:57:12.222087+0000","last_became_peered":"2023-12-19T14:57:12.222087+0000","last_unstale":"2023-12-19T14:57:13.235799+0000","last_undegraded":"2023-12-19T14:57:13.235799+0000","last_fullsized":"2023-12-19T14:57:13.235799+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:36.584909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.048529+0000","last_change":"2023-12-19T14:57:13.046939+0000","last_active":"2023-12-19T14:57:13.048529+0000","last_peered":"2023-12-19T14:57:13.048529+0000","last_clean":"2023-12-19T14:57:13.048529+0000","last_became_active":"2023-12-19T14:57:13.046617+0000","last_became_peered":"2023-12-19T14:57:13.046617+0000","last_unstale":"2023-12-19T14:57:13.048529+0000","last_undegraded":"2023-12-19T14:57:13.048529+0000","last_fullsized":"2023-12-19T14:57:13.048529+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:46.498057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.233314+0000","last_change":"2023-12-19T14:57:12.214186+0000","last_active":"2023-12-19T14:57:13.233314+0000","last_peered":"2023-12-19T14:57:13.233314+0000","last_clean":"2023-12-19T14:57:13.233314+0000","last_became_active":"2023-12-19T14:57:12.213819+0000","last_became_peered":"2023-12-19T14:57:12.213819+0000","last_unstale":"2023-12-19T14:57:13.233314+0000","last_undegraded":"2023-12-19T14:57:13.233314+0000","last_fullsized":"2023-12-19T14:57:13.233314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:43.547892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056552+0000","last_change":"2023-12-19T14:57:13.048651+0000","last_active":"2023-12-19T14:57:13.056552+0000","last_peered":"2023-12-19T14:57:13.056552+0000","last_clean":"2023-12-19T14:57:13.056552+0000","last_became_active":"2023-12-19T14:57:13.048319+0000","last_became_peered":"2023-12-19T14:57:13.048319+0000","last_unstale":"2023-12-19T14:57:13.056552+0000","last_undegraded":"2023-12-19T14:57:13.056552+0000","last_fullsized":"2023-12-19T14:57:13.056552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:10.953482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.238517+0000","last_change":"2023-12-19T14:57:12.218816+0000","last_active":"2023-12-19T14:57:13.238517+0000","last_peered":"2023-12-19T14:57:13.238517+0000","last_clean":"2023-12-19T14:57:13.238517+0000","last_became_active":"2023-12-19T14:57:12.218512+0000","last_became_peered":"2023-12-19T14:57:12.218512+0000","last_unstale":"2023-12-19T14:57:13.238517+0000","last_undegraded":"2023-12-19T14:57:13.238517+0000","last_fullsized":"2023-12-19T14:57:13.238517+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:06.151098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066472+0000","last_change":"2023-12-19T14:57:13.065371+0000","last_active":"2023-12-19T14:57:13.066472+0000","last_peered":"2023-12-19T14:57:13.066472+0000","last_clean":"2023-12-19T14:57:13.066472+0000","last_became_active":"2023-12-19T14:57:13.065072+0000","last_became_peered":"2023-12-19T14:57:13.065072+0000","last_unstale":"2023-12-19T14:57:13.066472+0000","last_undegraded":"2023-12-19T14:57:13.066472+0000","last_fullsized":"2023-12-19T14:57:13.066472+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:10:05.441581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.232747+0000","last_change":"2023-12-19T14:57:12.219505+0000","last_active":"2023-12-19T14:57:13.232747+0000","last_peered":"2023-12-19T14:57:13.232747+0000","last_clean":"2023-12-19T14:57:13.232747+0000","last_became_active":"2023-12-19T14:57:12.219163+0000","last_became_peered":"2023-12-19T14:57:12.219163+0000","last_unstale":"2023-12-19T14:57:13.232747+0000","last_undegraded":"2023-12-19T14:57:13.232747+0000","last_fullsized":"2023-12-19T14:57:13.232747+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:31.342414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067725+0000","last_change":"2023-12-19T14:57:13.063241+0000","last_active":"2023-12-19T14:57:13.067725+0000","last_peered":"2023-12-19T14:57:13.067725+0000","last_clean":"2023-12-19T14:57:13.067725+0000","last_became_active":"2023-12-19T14:57:13.062292+0000","last_became_peered":"2023-12-19T14:57:13.062292+0000","last_unstale":"2023-12-19T14:57:13.067725+0000","last_undegraded":"2023-12-19T14:57:13.067725+0000","last_fullsized":"2023-12-19T14:57:13.067725+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:12.208855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234308+0000","last_change":"2023-12-19T14:57:12.214250+0000","last_active":"2023-12-19T14:57:13.234308+0000","last_peered":"2023-12-19T14:57:13.234308+0000","last_clean":"2023-12-19T14:57:13.234308+0000","last_became_active":"2023-12-19T14:57:12.213038+0000","last_became_peered":"2023-12-19T14:57:12.213038+0000","last_unstale":"2023-12-19T14:57:13.234308+0000","last_undegraded":"2023-12-19T14:57:13.234308+0000","last_fullsized":"2023-12-19T14:57:13.234308+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:43.547892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062031+0000","last_change":"2023-12-19T14:57:13.058393+0000","last_active":"2023-12-19T14:57:13.062031+0000","last_peered":"2023-12-19T14:57:13.062031+0000","last_clean":"2023-12-19T14:57:13.062031+0000","last_became_active":"2023-12-19T14:57:13.058037+0000","last_became_peered":"2023-12-19T14:57:13.058037+0000","last_unstale":"2023-12-19T14:57:13.062031+0000","last_undegraded":"2023-12-19T14:57:13.062031+0000","last_fullsized":"2023-12-19T14:57:13.062031+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042200+0000","last_change":"2023-12-19T14:57:12.212301+0000","last_active":"2023-12-19T14:57:13.042200+0000","last_peered":"2023-12-19T14:57:13.042200+0000","last_clean":"2023-12-19T14:57:13.042200+0000","last_became_active":"2023-12-19T14:57:12.212024+0000","last_became_peered":"2023-12-19T14:57:12.212024+0000","last_unstale":"2023-12-19T14:57:13.042200+0000","last_undegraded":"2023-12-19T14:57:13.042200+0000","last_fullsized":"2023-12-19T14:57:13.042200+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:42.039352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058475+0000","last_change":"2023-12-19T14:57:13.052752+0000","last_active":"2023-12-19T14:57:13.058475+0000","last_peered":"2023-12-19T14:57:13.058475+0000","last_clean":"2023-12-19T14:57:13.058475+0000","last_became_active":"2023-12-19T14:57:13.050548+0000","last_became_peered":"2023-12-19T14:57:13.050548+0000","last_unstale":"2023-12-19T14:57:13.058475+0000","last_undegraded":"2023-12-19T14:57:13.058475+0000","last_fullsized":"2023-12-19T14:57:13.058475+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:33:30.650455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036739+0000","last_change":"2023-12-19T14:57:12.201356+0000","last_active":"2023-12-19T14:57:13.036739+0000","last_peered":"2023-12-19T14:57:13.036739+0000","last_clean":"2023-12-19T14:57:13.036739+0000","last_became_active":"2023-12-19T14:57:12.200902+0000","last_became_peered":"2023-12-19T14:57:12.200902+0000","last_unstale":"2023-12-19T14:57:13.036739+0000","last_undegraded":"2023-12-19T14:57:13.036739+0000","last_fullsized":"2023-12-19T14:57:13.036739+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:52:46.558984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062209+0000","last_change":"2023-12-19T14:57:13.059482+0000","last_active":"2023-12-19T14:57:13.062209+0000","last_peered":"2023-12-19T14:57:13.062209+0000","last_clean":"2023-12-19T14:57:13.062209+0000","last_became_active":"2023-12-19T14:57:13.058446+0000","last_became_peered":"2023-12-19T14:57:13.058446+0000","last_unstale":"2023-12-19T14:57:13.062209+0000","last_undegraded":"2023-12-19T14:57:13.062209+0000","last_fullsized":"2023-12-19T14:57:13.062209+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:00.226492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.046398+0000","last_change":"2023-12-19T14:57:12.212456+0000","last_active":"2023-12-19T14:57:13.046398+0000","last_peered":"2023-12-19T14:57:13.046398+0000","last_clean":"2023-12-19T14:57:13.046398+0000","last_became_active":"2023-12-19T14:57:12.212105+0000","last_became_peered":"2023-12-19T14:57:12.212105+0000","last_unstale":"2023-12-19T14:57:13.046398+0000","last_undegraded":"2023-12-19T14:57:13.046398+0000","last_fullsized":"2023-12-19T14:57:13.046398+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:30.579427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.057920+0000","last_change":"2023-12-19T14:57:13.050453+0000","last_active":"2023-12-19T14:57:13.057920+0000","last_peered":"2023-12-19T14:57:13.057920+0000","last_clean":"2023-12-19T14:57:13.057920+0000","last_became_active":"2023-12-19T14:57:13.049743+0000","last_became_peered":"2023-12-19T14:57:13.049743+0000","last_unstale":"2023-12-19T14:57:13.057920+0000","last_undegraded":"2023-12-19T14:57:13.057920+0000","last_fullsized":"2023-12-19T14:57:13.057920+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:43.052084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042643+0000","last_change":"2023-12-19T14:57:12.222441+0000","last_active":"2023-12-19T14:57:13.042643+0000","last_peered":"2023-12-19T14:57:13.042643+0000","last_clean":"2023-12-19T14:57:13.042643+0000","last_became_active":"2023-12-19T14:57:12.221879+0000","last_became_peered":"2023-12-19T14:57:12.221879+0000","last_unstale":"2023-12-19T14:57:13.042643+0000","last_undegraded":"2023-12-19T14:57:13.042643+0000","last_fullsized":"2023-12-19T14:57:13.042643+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:45.485326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058084+0000","last_change":"2023-12-19T14:57:13.051257+0000","last_active":"2023-12-19T14:57:13.058084+0000","last_peered":"2023-12-19T14:57:13.058084+0000","last_clean":"2023-12-19T14:57:13.058084+0000","last_became_active":"2023-12-19T14:57:13.050957+0000","last_became_peered":"2023-12-19T14:57:13.050957+0000","last_unstale":"2023-12-19T14:57:13.058084+0000","last_undegraded":"2023-12-19T14:57:13.058084+0000","last_fullsized":"2023-12-19T14:57:13.058084+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:48:54.891330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.233627+0000","last_change":"2023-12-19T14:57:12.221225+0000","last_active":"2023-12-19T14:57:13.233627+0000","last_peered":"2023-12-19T14:57:13.233627+0000","last_clean":"2023-12-19T14:57:13.233627+0000","last_became_active":"2023-12-19T14:57:12.220859+0000","last_became_peered":"2023-12-19T14:57:12.220859+0000","last_unstale":"2023-12-19T14:57:13.233627+0000","last_undegraded":"2023-12-19T14:57:13.233627+0000","last_fullsized":"2023-12-19T14:57:13.233627+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:02:33.047600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058802+0000","last_change":"2023-12-19T14:57:13.050125+0000","last_active":"2023-12-19T14:57:13.058802+0000","last_peered":"2023-12-19T14:57:13.058802+0000","last_clean":"2023-12-19T14:57:13.058802+0000","last_became_active":"2023-12-19T14:57:13.049404+0000","last_became_peered":"2023-12-19T14:57:13.049404+0000","last_unstale":"2023-12-19T14:57:13.058802+0000","last_undegraded":"2023-12-19T14:57:13.058802+0000","last_fullsized":"2023-12-19T14:57:13.058802+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:31.592159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":16,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.236626+0000","last_change":"2023-12-19T14:57:12.217230+0000","last_active":"2023-12-19T14:57:13.236626+0000","last_peered":"2023-12-19T14:57:13.236626+0000","last_clean":"2023-12-19T14:57:13.236626+0000","last_became_active":"2023-12-19T14:57:12.216913+0000","last_became_peered":"2023-12-19T14:57:12.216913+0000","last_unstale":"2023-12-19T14:57:13.236626+0000","last_undegraded":"2023-12-19T14:57:13.236626+0000","last_fullsized":"2023-12-19T14:57:13.236626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:11:37.366869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059634+0000","last_change":"2023-12-19T14:57:13.053548+0000","last_active":"2023-12-19T14:57:13.059634+0000","last_peered":"2023-12-19T14:57:13.059634+0000","last_clean":"2023-12-19T14:57:13.059634+0000","last_became_active":"2023-12-19T14:57:13.053242+0000","last_became_peered":"2023-12-19T14:57:13.053242+0000","last_unstale":"2023-12-19T14:57:13.059634+0000","last_undegraded":"2023-12-19T14:57:13.059634+0000","last_fullsized":"2023-12-19T14:57:13.059634+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:29.761636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237957+0000","last_change":"2023-12-19T14:57:12.221298+0000","last_active":"2023-12-19T14:57:13.237957+0000","last_peered":"2023-12-19T14:57:13.237957+0000","last_clean":"2023-12-19T14:57:13.237957+0000","last_became_active":"2023-12-19T14:57:12.220988+0000","last_became_peered":"2023-12-19T14:57:12.220988+0000","last_unstale":"2023-12-19T14:57:13.237957+0000","last_undegraded":"2023-12-19T14:57:13.237957+0000","last_fullsized":"2023-12-19T14:57:13.237957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:32.522139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.059891+0000","last_change":"2023-12-19T14:57:13.054401+0000","last_active":"2023-12-19T14:57:13.059891+0000","last_peered":"2023-12-19T14:57:13.059891+0000","last_clean":"2023-12-19T14:57:13.059891+0000","last_became_active":"2023-12-19T14:57:13.052265+0000","last_became_peered":"2023-12-19T14:57:13.052265+0000","last_unstale":"2023-12-19T14:57:13.059891+0000","last_undegraded":"2023-12-19T14:57:13.059891+0000","last_fullsized":"2023-12-19T14:57:13.059891+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:33:30.650455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036714+0000","last_change":"2023-12-19T14:57:12.222138+0000","last_active":"2023-12-19T14:57:13.036714+0000","last_peered":"2023-12-19T14:57:13.036714+0000","last_clean":"2023-12-19T14:57:13.036714+0000","last_became_active":"2023-12-19T14:57:12.221427+0000","last_became_peered":"2023-12-19T14:57:12.221427+0000","last_unstale":"2023-12-19T14:57:13.036714+0000","last_undegraded":"2023-12-19T14:57:13.036714+0000","last_fullsized":"2023-12-19T14:57:13.036714+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:53.586375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060258+0000","last_change":"2023-12-19T14:57:13.052115+0000","last_active":"2023-12-19T14:57:13.060258+0000","last_peered":"2023-12-19T14:57:13.060258+0000","last_clean":"2023-12-19T14:57:13.060258+0000","last_became_active":"2023-12-19T14:57:13.051764+0000","last_became_peered":"2023-12-19T14:57:13.051764+0000","last_unstale":"2023-12-19T14:57:13.060258+0000","last_undegraded":"2023-12-19T14:57:13.060258+0000","last_fullsized":"2023-12-19T14:57:13.060258+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:59:13.690764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040682+0000","last_change":"2023-12-19T14:57:12.219172+0000","last_active":"2023-12-19T14:57:13.040682+0000","last_peered":"2023-12-19T14:57:13.040682+0000","last_clean":"2023-12-19T14:57:13.040682+0000","last_became_active":"2023-12-19T14:57:12.218873+0000","last_became_peered":"2023-12-19T14:57:12.218873+0000","last_unstale":"2023-12-19T14:57:13.040682+0000","last_undegraded":"2023-12-19T14:57:13.040682+0000","last_fullsized":"2023-12-19T14:57:13.040682+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:59.600265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062246+0000","last_change":"2023-12-19T14:57:13.059382+0000","last_active":"2023-12-19T14:57:13.062246+0000","last_peered":"2023-12-19T14:57:13.062246+0000","last_clean":"2023-12-19T14:57:13.062246+0000","last_became_active":"2023-12-19T14:57:13.058744+0000","last_became_peered":"2023-12-19T14:57:13.058744+0000","last_unstale":"2023-12-19T14:57:13.062246+0000","last_undegraded":"2023-12-19T14:57:13.062246+0000","last_fullsized":"2023-12-19T14:57:13.062246+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:54.672177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.236449+0000","last_change":"2023-12-19T14:57:12.212082+0000","last_active":"2023-12-19T14:57:13.236449+0000","last_peered":"2023-12-19T14:57:13.236449+0000","last_clean":"2023-12-19T14:57:13.236449+0000","last_became_active":"2023-12-19T14:57:12.211611+0000","last_became_peered":"2023-12-19T14:57:12.211611+0000","last_unstale":"2023-12-19T14:57:13.236449+0000","last_undegraded":"2023-12-19T14:57:13.236449+0000","last_fullsized":"2023-12-19T14:57:13.236449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:30.612421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062248+0000","last_change":"2023-12-19T14:57:13.052430+0000","last_active":"2023-12-19T14:57:13.062248+0000","last_peered":"2023-12-19T14:57:13.062248+0000","last_clean":"2023-12-19T14:57:13.062248+0000","last_became_active":"2023-12-19T14:57:13.052051+0000","last_became_peered":"2023-12-19T14:57:13.052051+0000","last_unstale":"2023-12-19T14:57:13.062248+0000","last_undegraded":"2023-12-19T14:57:13.062248+0000","last_fullsized":"2023-12-19T14:57:13.062248+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:52:05.162666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.232822+0000","last_change":"2023-12-19T14:57:12.217199+0000","last_active":"2023-12-19T14:57:13.232822+0000","last_peered":"2023-12-19T14:57:13.232822+0000","last_clean":"2023-12-19T14:57:13.232822+0000","last_became_active":"2023-12-19T14:57:12.216830+0000","last_became_peered":"2023-12-19T14:57:12.216830+0000","last_unstale":"2023-12-19T14:57:13.232822+0000","last_undegraded":"2023-12-19T14:57:13.232822+0000","last_fullsized":"2023-12-19T14:57:13.232822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:31.265320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061907+0000","last_change":"2023-12-19T14:57:13.057158+0000","last_active":"2023-12-19T14:57:13.061907+0000","last_peered":"2023-12-19T14:57:13.061907+0000","last_clean":"2023-12-19T14:57:13.061907+0000","last_became_active":"2023-12-19T14:57:13.056859+0000","last_became_peered":"2023-12-19T14:57:13.056859+0000","last_unstale":"2023-12-19T14:57:13.061907+0000","last_undegraded":"2023-12-19T14:57:13.061907+0000","last_fullsized":"2023-12-19T14:57:13.061907+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:31.625153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234428+0000","last_change":"2023-12-19T14:57:12.216557+0000","last_active":"2023-12-19T14:57:13.234428+0000","last_peered":"2023-12-19T14:57:13.234428+0000","last_clean":"2023-12-19T14:57:13.234428+0000","last_became_active":"2023-12-19T14:57:12.216244+0000","last_became_peered":"2023-12-19T14:57:12.216244+0000","last_unstale":"2023-12-19T14:57:13.234428+0000","last_undegraded":"2023-12-19T14:57:13.234428+0000","last_fullsized":"2023-12-19T14:57:13.234428+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:10:04.428849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.063582+0000","last_change":"2023-12-19T14:57:13.058968+0000","last_active":"2023-12-19T14:57:13.063582+0000","last_peered":"2023-12-19T14:57:13.063582+0000","last_clean":"2023-12-19T14:57:13.063582+0000","last_became_active":"2023-12-19T14:57:13.058608+0000","last_became_peered":"2023-12-19T14:57:13.058608+0000","last_unstale":"2023-12-19T14:57:13.063582+0000","last_undegraded":"2023-12-19T14:57:13.063582+0000","last_fullsized":"2023-12-19T14:57:13.063582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:44.560623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.039560+0000","last_change":"2023-12-19T14:57:12.218553+0000","last_active":"2023-12-19T14:57:13.039560+0000","last_peered":"2023-12-19T14:57:13.039560+0000","last_clean":"2023-12-19T14:57:13.039560+0000","last_became_active":"2023-12-19T14:57:12.218138+0000","last_became_peered":"2023-12-19T14:57:12.218138+0000","last_unstale":"2023-12-19T14:57:13.039560+0000","last_undegraded":"2023-12-19T14:57:13.039560+0000","last_fullsized":"2023-12-19T14:57:13.039560+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:30.612421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.061316+0000","last_change":"2023-12-19T14:57:13.057204+0000","last_active":"2023-12-19T14:57:13.061316+0000","last_peered":"2023-12-19T14:57:13.061316+0000","last_clean":"2023-12-19T14:57:13.061316+0000","last_became_active":"2023-12-19T14:57:13.056845+0000","last_became_peered":"2023-12-19T14:57:13.056845+0000","last_unstale":"2023-12-19T14:57:13.061316+0000","last_undegraded":"2023-12-19T14:57:13.061316+0000","last_fullsized":"2023-12-19T14:57:13.061316+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:12.493809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.235653+0000","last_change":"2023-12-19T14:57:12.220138+0000","last_active":"2023-12-19T14:57:13.235653+0000","last_peered":"2023-12-19T14:57:13.235653+0000","last_clean":"2023-12-19T14:57:13.235653+0000","last_became_active":"2023-12-19T14:57:12.218313+0000","last_became_peered":"2023-12-19T14:57:12.218313+0000","last_unstale":"2023-12-19T14:57:13.235653+0000","last_undegraded":"2023-12-19T14:57:13.235653+0000","last_fullsized":"2023-12-19T14:57:13.235653+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:20.412744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.067475+0000","last_change":"2023-12-19T14:57:13.062603+0000","last_active":"2023-12-19T14:57:13.067475+0000","last_peered":"2023-12-19T14:57:13.067475+0000","last_clean":"2023-12-19T14:57:13.067475+0000","last_became_active":"2023-12-19T14:57:13.060694+0000","last_became_peered":"2023-12-19T14:57:13.060694+0000","last_unstale":"2023-12-19T14:57:13.067475+0000","last_undegraded":"2023-12-19T14:57:13.067475+0000","last_fullsized":"2023-12-19T14:57:13.067475+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:08:19.636446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.041990+0000","last_change":"2023-12-19T14:57:12.216347+0000","last_active":"2023-12-19T14:57:13.041990+0000","last_peered":"2023-12-19T14:57:13.041990+0000","last_clean":"2023-12-19T14:57:13.041990+0000","last_became_active":"2023-12-19T14:57:12.216011+0000","last_became_peered":"2023-12-19T14:57:12.216011+0000","last_unstale":"2023-12-19T14:57:13.041990+0000","last_undegraded":"2023-12-19T14:57:13.041990+0000","last_fullsized":"2023-12-19T14:57:13.041990+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:57.871908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060474+0000","last_change":"2023-12-19T14:57:13.057401+0000","last_active":"2023-12-19T14:57:13.060474+0000","last_peered":"2023-12-19T14:57:13.060474+0000","last_clean":"2023-12-19T14:57:13.060474+0000","last_became_active":"2023-12-19T14:57:13.056897+0000","last_became_peered":"2023-12-19T14:57:13.056897+0000","last_unstale":"2023-12-19T14:57:13.060474+0000","last_undegraded":"2023-12-19T14:57:13.060474+0000","last_fullsized":"2023-12-19T14:57:13.060474+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:51.886340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.037952+0000","last_change":"2023-12-19T14:57:12.213984+0000","last_active":"2023-12-19T14:57:13.037952+0000","last_peered":"2023-12-19T14:57:13.037952+0000","last_clean":"2023-12-19T14:57:13.037952+0000","last_became_active":"2023-12-19T14:57:12.213496+0000","last_became_peered":"2023-12-19T14:57:12.213496+0000","last_unstale":"2023-12-19T14:57:13.037952+0000","last_undegraded":"2023-12-19T14:57:13.037952+0000","last_fullsized":"2023-12-19T14:57:13.037952+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:40.508592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058440+0000","last_change":"2023-12-19T14:57:13.051432+0000","last_active":"2023-12-19T14:57:13.058440+0000","last_peered":"2023-12-19T14:57:13.058440+0000","last_clean":"2023-12-19T14:57:13.058440+0000","last_became_active":"2023-12-19T14:57:13.051082+0000","last_became_peered":"2023-12-19T14:57:13.051082+0000","last_unstale":"2023-12-19T14:57:13.058440+0000","last_undegraded":"2023-12-19T14:57:13.058440+0000","last_fullsized":"2023-12-19T14:57:13.058440+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:38:12.466924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040102+0000","last_change":"2023-12-19T14:57:12.214861+0000","last_active":"2023-12-19T14:57:13.040102+0000","last_peered":"2023-12-19T14:57:13.040102+0000","last_clean":"2023-12-19T14:57:13.040102+0000","last_became_active":"2023-12-19T14:57:12.214093+0000","last_became_peered":"2023-12-19T14:57:12.214093+0000","last_unstale":"2023-12-19T14:57:13.040102+0000","last_undegraded":"2023-12-19T14:57:13.040102+0000","last_fullsized":"2023-12-19T14:57:13.040102+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:27.389000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.066592+0000","last_change":"2023-12-19T14:57:13.061141+0000","last_active":"2023-12-19T14:57:13.066592+0000","last_peered":"2023-12-19T14:57:13.066592+0000","last_clean":"2023-12-19T14:57:13.066592+0000","last_became_active":"2023-12-19T14:57:13.058882+0000","last_became_peered":"2023-12-19T14:57:13.058882+0000","last_unstale":"2023-12-19T14:57:13.066592+0000","last_undegraded":"2023-12-19T14:57:13.066592+0000","last_fullsized":"2023-12-19T14:57:13.066592+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:10.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.237722+0000","last_change":"2023-12-19T14:57:12.219381+0000","last_active":"2023-12-19T14:57:13.237722+0000","last_peered":"2023-12-19T14:57:13.237722+0000","last_clean":"2023-12-19T14:57:13.237722+0000","last_became_active":"2023-12-19T14:57:12.217783+0000","last_became_peered":"2023-12-19T14:57:12.217783+0000","last_unstale":"2023-12-19T14:57:13.237722+0000","last_undegraded":"2023-12-19T14:57:13.237722+0000","last_fullsized":"2023-12-19T14:57:13.237722+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:06:36.734033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.048407+0000","last_change":"2023-12-19T14:57:13.046757+0000","last_active":"2023-12-19T14:57:13.048407+0000","last_peered":"2023-12-19T14:57:13.048407+0000","last_clean":"2023-12-19T14:57:13.048407+0000","last_became_active":"2023-12-19T14:57:13.046399+0000","last_became_peered":"2023-12-19T14:57:13.046399+0000","last_unstale":"2023-12-19T14:57:13.048407+0000","last_undegraded":"2023-12-19T14:57:13.048407+0000","last_fullsized":"2023-12-19T14:57:13.048407+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:10.953482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042767+0000","last_change":"2023-12-19T14:57:12.222206+0000","last_active":"2023-12-19T14:57:13.042767+0000","last_peered":"2023-12-19T14:57:13.042767+0000","last_clean":"2023-12-19T14:57:13.042767+0000","last_became_active":"2023-12-19T14:57:12.221481+0000","last_became_peered":"2023-12-19T14:57:12.221481+0000","last_unstale":"2023-12-19T14:57:13.042767+0000","last_undegraded":"2023-12-19T14:57:13.042767+0000","last_fullsized":"2023-12-19T14:57:13.042767+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:09.940751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.058281+0000","last_change":"2023-12-19T14:57:13.053321+0000","last_active":"2023-12-19T14:57:13.058281+0000","last_peered":"2023-12-19T14:57:13.058281+0000","last_clean":"2023-12-19T14:57:13.058281+0000","last_became_active":"2023-12-19T14:57:13.052585+0000","last_became_peered":"2023-12-19T14:57:13.052585+0000","last_unstale":"2023-12-19T14:57:13.058281+0000","last_undegraded":"2023-12-19T14:57:13.058281+0000","last_fullsized":"2023-12-19T14:57:13.058281+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:32.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042865+0000","last_change":"2023-12-19T14:57:12.210379+0000","last_active":"2023-12-19T14:57:13.042865+0000","last_peered":"2023-12-19T14:57:13.042865+0000","last_clean":"2023-12-19T14:57:13.042865+0000","last_became_active":"2023-12-19T14:57:12.209728+0000","last_became_peered":"2023-12-19T14:57:12.209728+0000","last_unstale":"2023-12-19T14:57:13.042865+0000","last_undegraded":"2023-12-19T14:57:13.042865+0000","last_fullsized":"2023-12-19T14:57:13.042865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:17:55.429946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056534+0000","last_change":"2023-12-19T14:57:13.045359+0000","last_active":"2023-12-19T14:57:13.056534+0000","last_peered":"2023-12-19T14:57:13.056534+0000","last_clean":"2023-12-19T14:57:13.056534+0000","last_became_active":"2023-12-19T14:57:13.045015+0000","last_became_peered":"2023-12-19T14:57:13.045015+0000","last_unstale":"2023-12-19T14:57:13.056534+0000","last_undegraded":"2023-12-19T14:57:13.056534+0000","last_fullsized":"2023-12-19T14:57:13.056534+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:16:50.173814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.031971+0000","last_change":"2023-12-19T14:57:12.223438+0000","last_active":"2023-12-19T14:57:13.031971+0000","last_peered":"2023-12-19T14:57:13.031971+0000","last_clean":"2023-12-19T14:57:13.031971+0000","last_became_active":"2023-12-19T14:57:12.223129+0000","last_became_peered":"2023-12-19T14:57:12.223129+0000","last_unstale":"2023-12-19T14:57:13.031971+0000","last_undegraded":"2023-12-19T14:57:13.031971+0000","last_fullsized":"2023-12-19T14:57:13.031971+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:00:22.599848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.055533+0000","last_change":"2023-12-19T14:57:13.049393+0000","last_active":"2023-12-19T14:57:13.055533+0000","last_peered":"2023-12-19T14:57:13.055533+0000","last_clean":"2023-12-19T14:57:13.055533+0000","last_became_active":"2023-12-19T14:57:13.049073+0000","last_became_peered":"2023-12-19T14:57:13.049073+0000","last_unstale":"2023-12-19T14:57:13.055533+0000","last_undegraded":"2023-12-19T14:57:13.055533+0000","last_fullsized":"2023-12-19T14:57:13.055533+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:49:51.805454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040697+0000","last_change":"2023-12-19T14:57:12.218609+0000","last_active":"2023-12-19T14:57:13.040697+0000","last_peered":"2023-12-19T14:57:13.040697+0000","last_clean":"2023-12-19T14:57:13.040697+0000","last_became_active":"2023-12-19T14:57:12.218315+0000","last_became_peered":"2023-12-19T14:57:12.218315+0000","last_unstale":"2023-12-19T14:57:13.040697+0000","last_undegraded":"2023-12-19T14:57:13.040697+0000","last_fullsized":"2023-12-19T14:57:13.040697+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:41:53.321470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.060602+0000","last_change":"2023-12-19T14:57:13.052171+0000","last_active":"2023-12-19T14:57:13.060602+0000","last_peered":"2023-12-19T14:57:13.060602+0000","last_clean":"2023-12-19T14:57:13.060602+0000","last_became_active":"2023-12-19T14:57:13.051822+0000","last_became_peered":"2023-12-19T14:57:13.051822+0000","last_unstale":"2023-12-19T14:57:13.060602+0000","last_undegraded":"2023-12-19T14:57:13.060602+0000","last_fullsized":"2023-12-19T14:57:13.060602+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:50:12.839228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.042127+0000","last_change":"2023-12-19T14:57:12.224789+0000","last_active":"2023-12-19T14:57:13.042127+0000","last_peered":"2023-12-19T14:57:13.042127+0000","last_clean":"2023-12-19T14:57:13.042127+0000","last_became_active":"2023-12-19T14:57:12.224486+0000","last_became_peered":"2023-12-19T14:57:12.224486+0000","last_unstale":"2023-12-19T14:57:13.042127+0000","last_undegraded":"2023-12-19T14:57:13.042127+0000","last_fullsized":"2023-12-19T14:57:13.042127+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:10.271647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.062446+0000","last_change":"2023-12-19T14:57:13.058194+0000","last_active":"2023-12-19T14:57:13.062446+0000","last_peered":"2023-12-19T14:57:13.062446+0000","last_clean":"2023-12-19T14:57:13.062446+0000","last_became_active":"2023-12-19T14:57:13.057832+0000","last_became_peered":"2023-12-19T14:57:13.057832+0000","last_unstale":"2023-12-19T14:57:13.062446+0000","last_undegraded":"2023-12-19T14:57:13.062446+0000","last_fullsized":"2023-12-19T14:57:13.062446+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:30.935155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.038052+0000","last_change":"2023-12-19T14:57:12.213799+0000","last_active":"2023-12-19T14:57:13.038052+0000","last_peered":"2023-12-19T14:57:13.038052+0000","last_clean":"2023-12-19T14:57:13.038052+0000","last_became_active":"2023-12-19T14:57:12.213158+0000","last_became_peered":"2023-12-19T14:57:12.213158+0000","last_unstale":"2023-12-19T14:57:13.038052+0000","last_undegraded":"2023-12-19T14:57:13.038052+0000","last_fullsized":"2023-12-19T14:57:13.038052+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:52:46.558984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056430+0000","last_change":"2023-12-19T14:57:13.050530+0000","last_active":"2023-12-19T14:57:13.056430+0000","last_peered":"2023-12-19T14:57:13.056430+0000","last_clean":"2023-12-19T14:57:13.056430+0000","last_became_active":"2023-12-19T14:57:13.050165+0000","last_became_peered":"2023-12-19T14:57:13.050165+0000","last_unstale":"2023-12-19T14:57:13.056430+0000","last_undegraded":"2023-12-19T14:57:13.056430+0000","last_fullsized":"2023-12-19T14:57:13.056430+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:26:15.738373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.040184+0000","last_change":"2023-12-19T14:57:12.220869+0000","last_active":"2023-12-19T14:57:13.040184+0000","last_peered":"2023-12-19T14:57:13.040184+0000","last_clean":"2023-12-19T14:57:13.040184+0000","last_became_active":"2023-12-19T14:57:12.219736+0000","last_became_peered":"2023-12-19T14:57:12.219736+0000","last_unstale":"2023-12-19T14:57:13.040184+0000","last_undegraded":"2023-12-19T14:57:13.040184+0000","last_fullsized":"2023-12-19T14:57:13.040184+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:44:21.868897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.056626+0000","last_change":"2023-12-19T14:57:13.047550+0000","last_active":"2023-12-19T14:57:13.056626+0000","last_peered":"2023-12-19T14:57:13.056626+0000","last_clean":"2023-12-19T14:57:13.056626+0000","last_became_active":"2023-12-19T14:57:13.047283+0000","last_became_peered":"2023-12-19T14:57:13.047283+0000","last_unstale":"2023-12-19T14:57:13.056626+0000","last_undegraded":"2023-12-19T14:57:13.056626+0000","last_fullsized":"2023-12-19T14:57:13.056626+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:12.182204+0000","last_clean_scrub_stamp":"2023-12-19T14:57:12.182204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:46.498057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.234853+0000","last_change":"2023-12-19T14:57:12.217729+0000","last_active":"2023-12-19T14:57:13.234853+0000","last_peered":"2023-12-19T14:57:13.234853+0000","last_clean":"2023-12-19T14:57:13.234853+0000","last_became_active":"2023-12-19T14:57:12.217442+0000","last_became_peered":"2023-12-19T14:57:12.217442+0000","last_unstale":"2023-12-19T14:57:13.234853+0000","last_undegraded":"2023-12-19T14:57:13.234853+0000","last_fullsized":"2023-12-19T14:57:13.234853+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:57:11.169473+0000","last_clean_scrub_stamp":"2023-12-19T14:57:11.169473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:02.298531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'227","reported_seq":643,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T14:57:13.036205+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:57:13.036205+0000","last_peered":"2023-12-19T14:57:13.036205+0000","last_clean":"2023-12-19T14:57:13.036205+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:57:13.036205+0000","last_undegraded":"2023-12-19T14:57:13.036205+0000","last_fullsized":"2023-12-19T14:57:13.036205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575065,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1386783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.103,"15min":1.093},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.52300000000000002},"max":{"1min":1.4730000000000001,"5min":3.544,"15min":3.544},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.159,"15min":1.143},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5780000000000001,"5min":7.8239999999999998,"15min":7.8239999999999998},"last":1.0349999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.82799999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.6759999999999999,"5min":1.722,"15min":1.73},"last":1.081},{"interface":"front","average":{"1min":1.222,"5min":1.1839999999999999,"15min":1.1879999999999999},"min":{"1min":0.59999999999999998,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.548,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.147}]},{"osd":2,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.2589999999999999,"15min":1.242},"min":{"1min":0.90100000000000002,"5min":0.68400000000000005,"15min":0.67900000000000005},"max":{"1min":1.583,"5min":1.5980000000000001,"15min":1.6919999999999999},"last":1.006},{"interface":"front","average":{"1min":1.329,"5min":1.254,"15min":1.2769999999999999},"min":{"1min":0.96599999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6890000000000001},"last":1.3600000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.238,"15min":1.282},"min":{"1min":0.79700000000000004,"5min":0.78000000000000003,"15min":0.76200000000000001},"max":{"1min":1.601,"5min":1.649,"15min":1.7969999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3080000000000001,"15min":1.3109999999999999},"min":{"1min":0.99299999999999999,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.6339999999999999,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.234}]},{"osd":4,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.1559999999999999,"15min":1.153},"min":{"1min":0.72699999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6930000000000001,"5min":1.752,"15min":1.752},"last":1.385},{"interface":"front","average":{"1min":1.208,"5min":1.2310000000000001,"15min":1.208},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.67500000000000004},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.7290000000000001},"last":1.115}]},{"osd":5,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2210000000000001,"15min":1.2150000000000001},"min":{"1min":0.875,"5min":0.496,"15min":0.496},"max":{"1min":1.73,"5min":1.73,"15min":1.8109999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.3089999999999999,"15min":1.3029999999999999},"min":{"1min":0.754,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6679999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.2729999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:26 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.389,"15min":1.3859999999999999},"min":{"1min":0.78200000000000003,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.833,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.379,"15min":1.3520000000000001},"min":{"1min":0.80700000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.2130000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302964,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068876,"statfs":{"total":96636764160,"available":96326529024,"internally_reserved":0,"allocated":1617920,"data_stored":805003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.131,"15min":1.131},"min":{"1min":0.68400000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.611,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.137,"15min":1.1659999999999999},"min":{"1min":0.73499999999999999,"5min":0.72799999999999998,"15min":0.64000000000000001},"max":{"1min":1.5629999999999999,"5min":1.845,"15min":1.845},"last":1.012}]},{"osd":1,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1679999999999999,"15min":1.1879999999999999},"min":{"1min":0.78800000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.403,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.242,"15min":1.2509999999999999},"min":{"1min":0.95999999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.601,"5min":3.226,"15min":3.226},"last":1.286}]},{"osd":2,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.266,"15min":1.244},"min":{"1min":0.88300000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6299999999999999,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.218,"5min":1.2350000000000001,"15min":1.2430000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.70799999999999996},"max":{"1min":1.6379999999999999,"5min":3.1619999999999999,"15min":3.1619999999999999},"last":1.0549999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2989999999999999,"15min":1.3},"min":{"1min":1.036,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.536,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.468},{"interface":"front","average":{"1min":1.327,"5min":1.3340000000000001,"15min":1.3240000000000001},"min":{"1min":0.89200000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8779999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.4410000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.206},"min":{"1min":0.82099999999999995,"5min":0.68300000000000005,"15min":0.621},"max":{"1min":1.7,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.282,"5min":1.3009999999999999,"15min":1.296},"min":{"1min":0.85999999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6479999999999999,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":1.1659999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.262,"15min":1.286},"min":{"1min":0.78000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7070000000000001,"5min":2.02,"15min":2.0219999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.3740000000000001,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.82799999999999996,"15min":0.76300000000000001},"max":{"1min":1.754,"5min":5.9610000000000003,"15min":5.9610000000000003},"last":1.383}]},{"osd":7,"last update":"Tue Dec 19 14:56:31 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4419999999999999,"15min":1.413},"min":{"1min":0.97799999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.847,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.4510000000000001,"15min":1.45},"min":{"1min":1.0940000000000001,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":2.0910000000000002,"5min":3.742,"15min":3.742},"last":1.3200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302944,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068896,"statfs":{"total":96636764160,"available":96326549504,"internally_reserved":0,"allocated":1597440,"data_stored":802891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0640000000000001,"15min":1.0680000000000001},"min":{"1min":0.82499999999999996,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5409999999999999,"5min":1.601,"15min":1.601},"last":1.083},{"interface":"front","average":{"1min":1.22,"5min":1.1259999999999999,"15min":1.137},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.518,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.754}]},{"osd":1,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.264,"15min":1.2929999999999999},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.863,"5min":1.863,"15min":1.9650000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2270000000000001,"15min":1.238},"min":{"1min":0.82299999999999995,"5min":0.73899999999999999,"15min":0.55700000000000005},"max":{"1min":1.5940000000000001,"5min":1.677,"15min":1.677},"last":1.2689999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.147,"15min":1.2210000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.47399999999999998},"max":{"1min":1.5109999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.359},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.159,"15min":1.1890000000000001},"min":{"1min":0.90000000000000002,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.7,"5min":1.7,"15min":1.954},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2569999999999999,"15min":1.27},"min":{"1min":0.70699999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.637,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.254,"15min":1.2649999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.68999999999999995},"max":{"1min":1.724,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.123,"15min":1.103},"min":{"1min":0.86499999999999999,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.238,"15min":1.2350000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6559999999999999,"5min":2.27,"15min":2.27},"last":1.1399999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.335,"15min":1.329},"min":{"1min":0.98099999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.9099999999999999,"5min":1.982,"15min":2.0630000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.327,"15min":1.321},"min":{"1min":0.90300000000000002,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.726,"5min":1.8540000000000001,"15min":1.9219999999999999},"last":1.4119999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3600000000000001,"15min":1.3500000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.58899999999999997},"max":{"1min":1.7509999999999999,"5min":2.1720000000000002,"15min":2.4470000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.387,"15min":1.377},"min":{"1min":0.871,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.048,"5min":2.048,"15min":2.2080000000000002},"last":1.393}]}]},{"osd":4,"up_from":13,"seq":55834575065,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068900,"statfs":{"total":96636764160,"available":96326553600,"internally_reserved":0,"allocated":1593344,"data_stored":796347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.034,"15min":1.0629999999999999},"min":{"1min":0.627,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.49,"5min":1.625,"15min":1.7869999999999999},"last":1.454},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.78500000000000003,"5min":0.59199999999999997,"15min":0.54700000000000004},"max":{"1min":1.5189999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2729999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2210000000000001,"15min":1.2350000000000001},"min":{"1min":0.65800000000000003,"5min":0.623,"15min":0.623},"max":{"1min":1.552,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.224,"15min":1.268},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.677,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.3089999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.157,"15min":1.143},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1479999999999999,"15min":1.1990000000000001},"min":{"1min":0.81100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.625,"5min":1.7649999999999999,"15min":1.893},"last":1.1279999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.161,"15min":1.1739999999999999},"min":{"1min":0.80200000000000005,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.6000000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.23,"15min":1.2549999999999999},"min":{"1min":0.86799999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.534,"5min":1.7749999999999999,"15min":1.7749999999999999},"last":1.4950000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.165,"15min":1.1379999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.57899999999999996},"max":{"1min":1.4650000000000001,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.149,"15min":1.1439999999999999},"min":{"1min":0.73899999999999999,"5min":0.67000000000000004,"15min":0.51300000000000001},"max":{"1min":1.4930000000000001,"5min":2.077,"15min":2.077},"last":1.089}]},{"osd":6,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.278,"15min":1.294},"min":{"1min":0.79700000000000004,"5min":0.621,"15min":0.55000000000000004},"max":{"1min":1.6439999999999999,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":1.349},{"interface":"front","average":{"1min":1.367,"5min":1.405,"15min":1.399},"min":{"1min":0.81599999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7190000000000001,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.6040000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:56:29 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.3100000000000001,"15min":1.3109999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.60899999999999999},"max":{"1min":1.639,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.4330000000000001,"15min":1.427},"min":{"1min":0.755,"5min":0.755,"15min":0.72599999999999998},"max":{"1min":1.7110000000000001,"5min":3.008,"15min":3.008},"last":1.6439999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":803445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.68200000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.026},{"interface":"front","average":{"1min":0.999,"5min":0.97499999999999998,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3520000000000001,"5min":1.542,"15min":1.7270000000000001},"last":1.1619999999999999}]},{"osd":1,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.90300000000000002,"15min":0.93899999999999995},"min":{"1min":0.621,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.54,"5min":1.611,"15min":1.611},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0900000000000001,"15min":1.095},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.446,"5min":1.661,"15min":1.758},"last":1.3280000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1379999999999999,"15min":1.1319999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.56100000000000005},"max":{"1min":1.748,"5min":1.778,"15min":1.778},"last":1.075},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.222,"15min":1.2370000000000001},"min":{"1min":0.81499999999999995,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.575,"5min":1.72,"15min":1.8520000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3280000000000001,"15min":1.327},"min":{"1min":1.0820000000000001,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.899},"last":1.194},{"interface":"front","average":{"1min":1.246,"5min":1.2749999999999999,"15min":1.2929999999999999},"min":{"1min":0.92800000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.504,"5min":1.6879999999999999,"15min":1.7989999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2829999999999999,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.75,"5min":1.7709999999999999,"15min":1.893},"last":1},{"interface":"front","average":{"1min":1.413,"5min":1.371,"15min":1.3879999999999999},"min":{"1min":1.123,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.8120000000000001,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.5429999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4390000000000001,"15min":1.45},"min":{"1min":1.2629999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8839999999999999,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4139999999999999,"15min":1.415},"min":{"1min":1.123,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.9059999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.5269999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.512,"15min":1.51},"min":{"1min":1.097,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":2.1640000000000001},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4510000000000001,"15min":1.456},"min":{"1min":1.1499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.677,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.26}]}]},{"osd":2,"up_from":13,"seq":55834575066,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307028,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064812,"statfs":{"total":96636764160,"available":96322367488,"internally_reserved":0,"allocated":1585152,"data_stored":795278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91500000000000004,"15min":0.90600000000000003},"min":{"1min":0.60199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.258,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.022,"5min":1.0169999999999999,"15min":1.0669999999999999},"min":{"1min":0.70199999999999996,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3,"5min":1.629,"15min":1.629},"last":1.0880000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.109,"15min":1.133},"min":{"1min":0.69499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4419999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.048,"15min":1.093},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4490000000000001,"5min":1.853,"15min":1.853},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.998,"15min":0.98999999999999999},"min":{"1min":0.58599999999999997,"5min":0.505,"15min":0.505},"max":{"1min":1.4419999999999999,"5min":1.5669999999999999,"15min":1.623},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0740000000000001,"15min":1.095},"min":{"1min":0.63,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.6080000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.012}]},{"osd":4,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3620000000000001,"15min":1.3759999999999999},"min":{"1min":0.747,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.716,"5min":2.0270000000000001,"15min":2.044},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.371,"15min":1.391},"min":{"1min":0.91000000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.7410000000000001,"5min":1.96,"15min":1.96},"last":1.431}]},{"osd":5,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3480000000000001,"15min":1.3400000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.679,"5min":2.0979999999999999,"15min":2.395},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.3160000000000001,"15min":1.3280000000000001},"min":{"1min":0.98499999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":2.1200000000000001},"last":1.2949999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":0.99199999999999999,"5min":0.85699999999999998,"15min":0.83299999999999996},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.069},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4379999999999999,"15min":1.431},"min":{"1min":0.95699999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.921,"5min":2.1840000000000002,"15min":2.4729999999999999},"last":1.698}]},{"osd":7,"last update":"Tue Dec 19 14:56:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.472,"15min":1.4730000000000001},"min":{"1min":0.95199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9399999999999999,"5min":2.0699999999999998,"15min":2.2080000000000002},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.48,"15min":1.4730000000000001},"min":{"1min":1.05,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":796289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.88300000000000001,"15min":0.86599999999999999},"min":{"1min":0.54400000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.206,"5min":2.238,"15min":2.238},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94599999999999995,"15min":0.97699999999999998},"min":{"1min":0.65600000000000003,"5min":0.57199999999999995,"15min":0.53100000000000003},"max":{"1min":1.49,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.486}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.02,"15min":1.022},"min":{"1min":0.71399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.506,"5min":2.444,"15min":2.444},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.075,"15min":1.1140000000000001},"min":{"1min":0.68300000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.3740000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.829}]},{"osd":3,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.204,"15min":1.23},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.573,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.369},{"interface":"front","average":{"1min":1.153,"5min":1.081,"15min":1.0860000000000001},"min":{"1min":0.628,"5min":0.624,"15min":0.56899999999999995},"max":{"1min":1.6000000000000001,"5min":1.6890000000000001,"15min":1.7270000000000001},"last":0.92500000000000004}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.2949999999999999,"15min":1.304},"min":{"1min":0.93799999999999994,"5min":0.624,"15min":0.624},"max":{"1min":1.974,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.3660000000000001,"15min":1.3939999999999999},"min":{"1min":0.95899999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7529999999999999,"5min":3.5630000000000002,"15min":3.5630000000000002},"last":1.7190000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.27,"15min":1.292},"min":{"1min":1.0109999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8160000000000001,"5min":1.857,"15min":1.857},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3959999999999999,"15min":1.3859999999999999},"min":{"1min":1.0089999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.2029999999999998,"5min":2.665,"15min":2.665},"last":2.1339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3420000000000001,"15min":1.365},"min":{"1min":1.071,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4319999999999999,"15min":1.446},"min":{"1min":1.0109999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.129,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.331}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4239999999999999,"15min":1.468},"min":{"1min":1.0349999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.923},{"interface":"front","average":{"1min":1.51,"5min":1.4379999999999999,"15min":1.4359999999999999},"min":{"1min":1.125,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.0609999999999999},"last":1.863}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1387238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.84799999999999998,"15min":0.84999999999999998},"min":{"1min":0.65800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3759999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.98199999999999998,"15min":0.96999999999999997},"min":{"1min":0.748,"5min":0.433,"15min":0.433},"max":{"1min":1.577,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":1.1140000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1240000000000001,"15min":1.1080000000000001},"min":{"1min":0.81599999999999995,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.6639999999999999,"5min":1.7729999999999999,"15min":1.8979999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.282,"5min":1.2210000000000001,"15min":1.2250000000000001},"min":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.55400000000000005},"max":{"1min":1.821,"5min":1.982,"15min":1.982},"last":1.7909999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.012,"15min":1.0229999999999999},"min":{"1min":0.76100000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.5,"5min":1.726,"15min":2.0299999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.175,"15min":1.1870000000000001},"min":{"1min":0.88700000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.756,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.0309999999999999}]},{"osd":4,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2909999999999999,"15min":1.2969999999999999},"min":{"1min":0.80400000000000005,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.605,"5min":2.153,"15min":2.153},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.288,"15min":1.2909999999999999},"min":{"1min":1.0229999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.613,"5min":1.7170000000000001,"15min":1.7929999999999999},"last":1.2609999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.383,"15min":1.3819999999999999},"min":{"1min":1.1839999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.4630000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3640000000000001,"15min":1.3899999999999999},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7210000000000001,"5min":2.2189999999999999,"15min":2.2229999999999999},"last":1.552}]},{"osd":6,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3140000000000001,"15min":1.3300000000000001},"min":{"1min":1.0209999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.5960000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4490000000000001,"15min":1.4470000000000001},"min":{"1min":1.1399999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.9710000000000001,"5min":2.113,"15min":2.3799999999999999},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:56:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4339999999999999,"15min":1.4550000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7649999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":1.0920000000000001,"5min":0.93899999999999995,"15min":0.88200000000000001},"max":{"1min":1.746,"5min":1.931,"15min":2.0379999999999998},"last":1.613}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:57:17.947 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:57:17.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:57:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.158+0000 7f7aca923700 1 Processor -- start 2023-12-19T14:57:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.158+0000 7f7aca923700 1 -- start start 2023-12-19T14:57:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca923700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7ac406aeb0 0x7f7ac4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac41595a0 0x7f7ac4159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca923700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac406c850 con 0x7f7ac4158c90 2023-12-19T14:57:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca923700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac40cb650 con 0x7f7ac406aeb0 2023-12-19T14:57:18.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac406b470 con 0x7f7ac41595a0 2023-12-19T14:57:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9921700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7ac4158c90 0x7f7ac40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47610/0 (socket says 172.21.15.50:47610) 2023-12-19T14:57:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9921700 1 -- 172.21.15.50:0/31183831 learned_addr learned my addr 172.21.15.50:0/31183831 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 --2- 172.21.15.50:0/31183831 >> v2:172.21.15.125:3300/0 conn(0x7f7ac406aeb0 0x7f7ac4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7aca122700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac41595a0 0x7f7ac4159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac891f700 1 -- 172.21.15.50:0/31183831 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885279629 0 0) 0x7f7ac406c850 con 0x7f7ac4158c90 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac891f700 1 -- 172.21.15.50:0/31183831 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab0003570 con 0x7f7ac4158c90 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 -- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac41595a0 msgr2=0x7f7ac4159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac41595a0 0x7f7ac4159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 -- 172.21.15.50:0/31183831 >> v1:172.21.15.50:6789/0 conn(0x7f7ac4158c90 legacy=0x7f7ac40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 -- 172.21.15.50:0/31183831 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac406b750 con 0x7f7ac406aeb0 2023-12-19T14:57:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.162+0000 7f7ac9120700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4159060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40098e0 tx=0x7f7ab40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d4ce1c92fc57b95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7ac891f700 1 -- 172.21.15.50:0/31183831 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4011630 con 0x7f7ac406aeb0 2023-12-19T14:57:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7ac891f700 1 -- 172.21.15.50:0/31183831 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab4011790 con 0x7f7ac406aeb0 2023-12-19T14:57:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7ac891f700 1 -- 172.21.15.50:0/31183831 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4011b30 con 0x7f7ac406aeb0 2023-12-19T14:57:18.167 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f6a11399700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 -- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac406aeb0 msgr2=0x7f7ac4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4159060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40098e0 tx=0x7f7ab40091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 -- 172.21.15.50:0/31183831 shutdown_connections 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac41595a0 0x7f7ac4159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 --2- 172.21.15.50:0/31183831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4159060 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 -- 172.21.15.50:0/31183831 >> 172.21.15.50:0/31183831 conn(0x7f7ac4067bb0 msgr2=0x7f7ac40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 -- 172.21.15.50:0/31183831 shutdown_connections 2023-12-19T14:57:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.166+0000 7f7aca923700 1 -- 172.21.15.50:0/31183831 wait complete. 2023-12-19T14:57:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 Processor -- start 2023-12-19T14:57:18.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 -- start start 2023-12-19T14:57:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 0x7f7ac41488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac41595a0 0x7f7ac416a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac416a840 con 0x7f7ac406aeb0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac4148e30 con 0x7f7ac41595a0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac40c9390 con 0x7f7ac4158c90 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45138/0 (socket says 172.21.15.50:45138) 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 -- 172.21.15.50:0/1867414340 learned_addr learned my addr 172.21.15.50:0/1867414340 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7aca122700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac41595a0 0x7f7ac416a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9120700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 0x7f7ac41488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 -- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 msgr2=0x7f7ac41488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 0x7f7ac41488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 -- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac41595a0 msgr2=0x7f7ac416a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac41595a0 0x7f7ac416a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9921700 1 -- 172.21.15.50:0/1867414340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4009420 con 0x7f7ac406aeb0 2023-12-19T14:57:18.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.170+0000 7f7ac9120700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 0x7f7ac41488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7ac9921700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c7f0 tx=0x7f7ac0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0005b30 con 0x7f7ac406aeb0 2023-12-19T14:57:18.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac0005c90 con 0x7f7ac406aeb0 2023-12-19T14:57:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0007340 con 0x7f7ac406aeb0 2023-12-19T14:57:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac40c87c0 con 0x7f7ac406aeb0 2023-12-19T14:57:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac40c8a60 con 0x7f7ac406aeb0 2023-12-19T14:57:18.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.174+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac404be40 con 0x7f7ac406aeb0 2023-12-19T14:57:18.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.178+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7ac001b400 con 0x7f7ac406aeb0 2023-12-19T14:57:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.178+0000 7f7ab9ffb700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 0x7f7aa8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.178+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7ac007c5a0 con 0x7f7ac406aeb0 2023-12-19T14:57:18.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.178+0000 7f7ac9120700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 0x7f7aa8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.178+0000 7f7ac9120700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 0x7f7aa8062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ac40cf180 tx=0x7f7ab4017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.182+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7ac004c9a0 con 0x7f7ac406aeb0 2023-12-19T14:57:18.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.410+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7ac406ba60 con 0x7f7ac406aeb0 2023-12-19T14:57:18.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.418+0000 7f7ab9ffb700 1 -- 172.21.15.50:0/1867414340 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7ac00076f0 con 0x7f7ac406aeb0 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.418+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 msgr2=0x7f7aa8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.418+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 0x7f7aa8062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ac40cf180 tx=0x7f7ab4017000 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.418+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 msgr2=0x7f7ac4158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.418+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c7f0 tx=0x7f7ac0007e00 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 shutdown_connections 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7ac41595a0 0x7f7ac416a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7aa8060670 0x7f7aa8062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7ac4158c90 0x7f7ac41488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 --2- 172.21.15.50:0/1867414340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ac406aeb0 0x7f7ac4158070 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 >> 172.21.15.50:0/1867414340 conn(0x7f7ac4067bb0 msgr2=0x7f7ac4062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 shutdown_connections 2023-12-19T14:57:18.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.422+0000 7f7aca923700 1 -- 172.21.15.50:0/1867414340 wait complete. 2023-12-19T14:57:18.425 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:57:18.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:57:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.610+0000 7f5f36dc5700 1 Processor -- start 2023-12-19T14:57:18.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 -- start start 2023-12-19T14:57:18.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f3015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f300c15b0 0x7f5f300aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f300c7190 con 0x7f5f3006d690 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f300c1b50 con 0x7f5f300c15b0 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f36dc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300c1e30 con 0x7f5f3015d320 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f27fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f300c15b0 0x7f5f300aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f27fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f300c15b0 0x7f5f300aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45142/0 (socket says 172.21.15.50:45142) 2023-12-19T14:57:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f27fff700 1 -- 172.21.15.50:0/2573202763 learned_addr learned my addr 172.21.15.50:0/2573202763 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f27fff700 1 -- 172.21.15.50:0/2573202763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 msgr2=0x7f5f3015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.614+0000 7f5f34b61700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f3015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f27fff700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f3015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f27fff700 1 -- 172.21.15.50:0/2573202763 >> v1:172.21.15.50:6789/0 conn(0x7f5f3006d690 legacy=0x7f5f300680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f27fff700 1 -- 172.21.15.50:0/2573202763 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300c6e20 con 0x7f5f300c15b0 2023-12-19T14:57:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f34b61700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f3015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f27fff700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f300aeba0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5f200099b0 tx=0x7f5f20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4897b9dca0cbed server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f277fe700 1 -- 172.21.15.50:0/2573202763 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2000b860 con 0x7f5f300c15b0 2023-12-19T14:57:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f277fe700 1 -- 172.21.15.50:0/2573202763 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2000b9c0 con 0x7f5f300c15b0 2023-12-19T14:57:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f277fe700 1 -- 172.21.15.50:0/2573202763 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2000bd60 con 0x7f5f300c15b0 2023-12-19T14:57:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 -- 172.21.15.50:0/2573202763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 msgr2=0x7f5f300aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f300aeba0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5f200099b0 tx=0x7f5f20009250 comp rx=0 tx=0).stop 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 -- 172.21.15.50:0/2573202763 shutdown_connections 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f300aeba0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/2573202763 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f3015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 -- 172.21.15.50:0/2573202763 >> 172.21.15.50:0/2573202763 conn(0x7f5f300c8850 msgr2=0x7f5f300c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 -- 172.21.15.50:0/2573202763 shutdown_connections 2023-12-19T14:57:18.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.618+0000 7f5f36dc5700 1 -- 172.21.15.50:0/2573202763 wait complete. 2023-12-19T14:57:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 Processor -- start 2023-12-19T14:57:18.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 -- start start 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f3006d690 0x7f5f30155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f30153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3015d6f0 con 0x7f5f300c15b0 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f300c40b0 con 0x7f5f3006d690 2023-12-19T14:57:18.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.622+0000 7f5f36dc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f30155cf0 con 0x7f5f3015d320 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45150/0 (socket says 172.21.15.50:45150) 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 -- 172.21.15.50:0/592990664 learned_addr learned my addr 172.21.15.50:0/592990664 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f34b61700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f3006d690 0x7f5f30155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f35362700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f30153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 -- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 msgr2=0x7f5f30153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f30153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 -- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f3006d690 msgr2=0x7f5f30155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f3006d690 0x7f5f30155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 -- 172.21.15.50:0/592990664 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200094f0 con 0x7f5f300c15b0 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f35362700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f30153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:18.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f27fff700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009290 tx=0x7f5f20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20024440 con 0x7f5f300c15b0 2023-12-19T14:57:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f200245a0 con 0x7f5f300c15b0 2023-12-19T14:57:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f20024960 con 0x7f5f300c15b0 2023-12-19T14:57:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f300c4300 con 0x7f5f300c15b0 2023-12-19T14:57:18.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.626+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f300c48c0 con 0x7f5f300c15b0 2023-12-19T14:57:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.630+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f300c4b10 con 0x7f5f300c15b0 2023-12-19T14:57:18.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.630+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5f2001a420 con 0x7f5f300c15b0 2023-12-19T14:57:18.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.630+0000 7f5f24ff9700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 0x7f5f1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.630+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5f2007e990 con 0x7f5f300c15b0 2023-12-19T14:57:18.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.630+0000 7f5f34b61700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 0x7f5f1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:18.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.634+0000 7f5f34b61700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 0x7f5f1c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c006fd0 tx=0x7f5f2c008040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:18.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.638+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5f2004ed20 con 0x7f5f300c15b0 2023-12-19T14:57:18.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.846+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f30071120 con 0x7f5f300c15b0 2023-12-19T14:57:18.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.846+0000 7f5f24ff9700 1 -- 172.21.15.50:0/592990664 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f5f20052340 con 0x7f5f300c15b0 2023-12-19T14:57:18.849 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:18.849 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:57:18.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 msgr2=0x7f5f1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 0x7f5f1c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c006fd0 tx=0x7f5f2c008040 comp rx=0 tx=0).stop 2023-12-19T14:57:18.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 msgr2=0x7f5f3006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009290 tx=0x7f5f20017000 comp rx=0 tx=0).stop 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 shutdown_connections 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f1c060670 0x7f5f1c062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f3015d320 0x7f5f30153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f300c15b0 0x7f5f3006b260 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 --2- 172.21.15.50:0/592990664 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f3006d690 0x7f5f30155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:18.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 >> 172.21.15.50:0/592990664 conn(0x7f5f300c8850 msgr2=0x7f5f3006a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:18.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 shutdown_connections 2023-12-19T14:57:18.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:18.850+0000 7f5f36dc5700 1 -- 172.21.15.50:0/592990664 wait complete. 2023-12-19T14:57:18.855 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:57:19.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:57:12.670233+0000', 'modified': '2023-12-19T14:57:16.239398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7878}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7878': {'gid': 7878, 'name': 'c', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/1300068976', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1300068976}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1300068976}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:57:19.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:57:19.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T14:57:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 Processor -- start 2023-12-19T14:57:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 -- start start 2023-12-19T14:57:19.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 0x7f712c0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f712c06c7c0 0x7f712c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712c06dc00 con 0x7f712c13f6b0 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712c13faf0 con 0x7f712c06c7c0 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.446+0000 7f7131f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712c06e090 con 0x7f712c0c7d90 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f712c06c7c0 0x7f712c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f712c06c7c0 0x7f712c13f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45166/0 (socket says 172.21.15.50:45166) 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 -- 172.21.15.50:0/2476314955 learned_addr learned my addr 172.21.15.50:0/2476314955 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2476314955 >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 0x7f712c0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 -- 172.21.15.50:0/2476314955 >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 msgr2=0x7f712c0c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 --2- 172.21.15.50:0/2476314955 >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 0x7f712c0c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 -- 172.21.15.50:0/2476314955 >> v1:172.21.15.125:6789/0 conn(0x7f712c13f6b0 legacy=0x7f712c0c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 -- 172.21.15.50:0/2476314955 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c06cff0 con 0x7f712c06c7c0 2023-12-19T14:57:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2476314955 >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 0x7f712c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:19.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712affd700 1 --2- 172.21.15.50:0/2476314955 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c06c7c0 0x7f712c13f170 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f712c069860 tx=0x7f7120009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968f00b332b4745e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712a7fc700 1 -- 172.21.15.50:0/2476314955 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712000b860 con 0x7f712c06c7c0 2023-12-19T14:57:19.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712a7fc700 1 -- 172.21.15.50:0/2476314955 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712000b9c0 con 0x7f712c06c7c0 2023-12-19T14:57:19.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f712a7fc700 1 -- 172.21.15.50:0/2476314955 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712000bd60 con 0x7f712c06c7c0 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 -- 172.21.15.50:0/2476314955 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c06c7c0 msgr2=0x7f712c13f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 --2- 172.21.15.50:0/2476314955 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c06c7c0 0x7f712c13f170 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f712c069860 tx=0x7f7120009250 comp rx=0 tx=0).stop 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 -- 172.21.15.50:0/2476314955 shutdown_connections 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 --2- 172.21.15.50:0/2476314955 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c06c7c0 0x7f712c13f170 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 --2- 172.21.15.50:0/2476314955 >> v2:172.21.15.125:3300/0 conn(0x7f712c0c7d90 0x7f712c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 -- 172.21.15.50:0/2476314955 >> 172.21.15.50:0/2476314955 conn(0x7f712c067b60 msgr2=0x7f712c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 -- 172.21.15.50:0/2476314955 shutdown_connections 2023-12-19T14:57:19.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.450+0000 7f7131f45700 1 -- 172.21.15.50:0/2476314955 wait complete. 2023-12-19T14:57:19.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.454+0000 7f7131f45700 1 Processor -- start 2023-12-19T14:57:19.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.454+0000 7f7131f45700 1 -- start start 2023-12-19T14:57:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 0x7f712c0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 0x7f712c13bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c169d30 con 0x7f712c0c7d90 2023-12-19T14:57:19.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c0a5c60 con 0x7f712c06c7c0 2023-12-19T14:57:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712c0a5f40 con 0x7f712c13f6b0 2023-12-19T14:57:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45180/0 (socket says 172.21.15.50:45180) 2023-12-19T14:57:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 -- 172.21.15.50:0/4070603094 learned_addr learned my addr 172.21.15.50:0/4070603094 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712b7fe700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 0x7f712c0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712bfff700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 0x7f712c13bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 -- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 msgr2=0x7f712c13bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 0x7f712c13bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 -- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 msgr2=0x7f712c0a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 0x7f712c0a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 -- 172.21.15.50:0/4070603094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71200094f0 con 0x7f712c0c7d90 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712b7fe700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 0x7f712c0a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712bfff700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 0x7f712c13bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:19.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f712affd700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7120009290 tx=0x7f7120017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f71137fe700 1 -- 172.21.15.50:0/4070603094 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120011e60 con 0x7f712c0c7d90 2023-12-19T14:57:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f71137fe700 1 -- 172.21.15.50:0/4070603094 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712001f070 con 0x7f712c0c7d90 2023-12-19T14:57:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f71137fe700 1 -- 172.21.15.50:0/4070603094 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712001a870 con 0x7f712c0c7d90 2023-12-19T14:57:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c0a6220 con 0x7f712c0c7d90 2023-12-19T14:57:19.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.458+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c0aa530 con 0x7f712c0c7d90 2023-12-19T14:57:19.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.462+0000 7f71137fe700 1 -- 172.21.15.50:0/4070603094 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f712001abe0 con 0x7f712c0c7d90 2023-12-19T14:57:19.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.462+0000 7f71137fe700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 0x7f7118062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f71137fe700 1 -- 172.21.15.50:0/4070603094 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f7120021070 con 0x7f712c0c7d90 2023-12-19T14:57:19.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f712b7fe700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 0x7f7118062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f712b7fe700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 0x7f7118062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f711c00bd10 tx=0x7f711c008040 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 Processor -- start 2023-12-19T14:57:19.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 -- start start 2023-12-19T14:57:19.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2a6e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a6e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x299bf30 con 0x27f0ae0 2023-12-19T14:57:19.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2997ed0 con 0x27f0710 2023-12-19T14:57:19.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.466+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x295e0b0 con 0x283b790 2023-12-19T14:57:19.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45182/0 (socket says 172.21.15.50:45182) 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 -- 172.21.15.50:0/47458043 learned_addr learned my addr 172.21.15.50:0/47458043 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712affd700 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2a6e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712bfff700 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a6e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 -- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 msgr2=0x2a6e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a6e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 -- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 msgr2=0x2a6e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2a6e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 -- 172.21.15.50:0/47458043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x283c9e0 con 0x27f0ae0 2023-12-19T14:57:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f712b7fe700 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f711c00fee0 tx=0x7f711c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70e2307e606a565 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7110ff9700 1 -- 172.21.15.50:0/47458043 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c00b9d0 con 0x27f0ae0 2023-12-19T14:57:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7110ff9700 1 -- 172.21.15.50:0/47458043 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c011dd0 con 0x27f0ae0 2023-12-19T14:57:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7110ff9700 1 -- 172.21.15.50:0/47458043 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c01e660 con 0x27f0ae0 2023-12-19T14:57:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 -- 172.21.15.50:0/47458043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 msgr2=0x2a6df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f711c00fee0 tx=0x7f711c016000 comp rx=0 tx=0).stop 2023-12-19T14:57:19.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 -- 172.21.15.50:0/47458043 shutdown_connections 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2a6e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a6e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 --2- 172.21.15.50:0/47458043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a6df00 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 -- 172.21.15.50:0/47458043 >> 172.21.15.50:0/47458043 conn(0x27f0eb0 msgr2=0x2a2a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 -- 172.21.15.50:0/47458043 shutdown_connections 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.470+0000 7f7134228740 1 -- 172.21.15.50:0/47458043 wait complete. 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 -- start start 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 Processor -- start 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0ae0 0x2a72ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a53cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x295dd50 con 0x27f0710 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2936970 con 0x27f0ae0 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x2946e00 con 0x283b790 2023-12-19T14:57:19.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f7134228740 10 client.0 _ll_register_callbacks cb 0x29e16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:57:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45190/0 (socket says 172.21.15.50:45190) 2023-12-19T14:57:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f712b7fe700 1 -- 172.21.15.50:0/249441873 learned_addr learned my addr 172.21.15.50:0/249441873 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f712affd700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0ae0 0x2a72ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.474+0000 7f712bfff700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a53cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 -- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 msgr2=0x2a53cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a53cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 -- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0ae0 msgr2=0x2a72ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0ae0 0x2a72ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 -- 172.21.15.50:0/249441873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711c00fc00 con 0x27f0710 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712b7fe700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f711c00f9e0 tx=0x7f711c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4910093c57521d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c02e070 con 0x27f0710 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2936cc0 con 0x27f0710 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f711c01edf0 con 0x27f0710 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f711c027660 con 0x27f0710 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 10 client.8019 fetch_fsmap learned FSMap version 58 2023-12-19T14:57:19.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x2966370 con 0x27f0710 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2936cc0 con 0x27f0710 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 58) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f711c0277c0 con 0x27f0710 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 10 client.8019 fetch_fsmap finished waiting for FSMap version 58 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 10 client.8019 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 20 client.8019 populate_metadata read hostname 'smithi050' 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a553c0 0x2a7d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 4 client.8019 mds_command: new command op to 7878 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 --> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x293a3e0 con 0x2a553c0 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.478+0000 7f712affd700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a553c0 0x2a7d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f712affd700 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a553c0 0x2a7d6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee43b1d783967554 server_cookie=b80409a4421cc0ff in_seq=0 out_seq=0 2023-12-19T14:57:19.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7109ffb700 10 client.8019 ms_handle_connect on v2:172.21.15.50:6834/1300068976 2023-12-19T14:57:19.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7109ffb700 1 -- 172.21.15.50:0/249441873 <== mds.0 v2:172.21.15.50:6834/1300068976 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x293a3e0 con 0x2a553c0 2023-12-19T14:57:19.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7109ffb700 10 client.8019 handle_command_reply: tid=0 2023-12-19T14:57:19.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 client.8019 shutdown 2023-12-19T14:57:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 msgr2=0x2a72590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f711c00f9e0 tx=0x7f711c020000 comp rx=0 tx=0).stop 2023-12-19T14:57:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 shutdown_connections 2023-12-19T14:57:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a553c0 0x2a7d6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0ae0 0x2a72ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a53cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7109ffb700 0 client.8019 ms_handle_reset on v2:172.21.15.50:6834/1300068976 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 --2- 172.21.15.50:0/249441873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0710 0x2a72590 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 >> 172.21.15.50:0/249441873 conn(0x27f0eb0 msgr2=0x2a22f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 shutdown_connections 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 1 -- 172.21.15.50:0/249441873 wait complete. 2023-12-19T14:57:19.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.482+0000 7f7134228740 20 client.8019 trim_cache size 0 max 16384 2023-12-19T14:57:19.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.502+0000 7f7134228740 1 Processor -- start 2023-12-19T14:57:19.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.502+0000 7f7134228740 1 -- start start 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x27f0710 0x2a24ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 0x2a6d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x299a590 con 0x27f0ae0 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x286d650 con 0x283b790 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x28697c0 con 0x27f0710 2023-12-19T14:57:19.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45192/0 (socket says 172.21.15.50:45192) 2023-12-19T14:57:19.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 -- 172.21.15.50:0/2708072099 learned_addr learned my addr 172.21.15.50:0/2708072099 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x27f0710 0x2a24ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712bfff700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 0x2a6d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 -- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x27f0710 msgr2=0x2a24ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x27f0710 0x2a24ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 -- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 msgr2=0x2a6d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 0x2a6d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 -- 172.21.15.50:0/2708072099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x294cd80 con 0x27f0ae0 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712bfff700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 0x2a6d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f712affd700 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f71200526b0 tx=0x7f712000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b2b7f7b9cf7ac9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f71117fa700 1 -- 172.21.15.50:0/2708072099 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120052780 con 0x27f0ae0 2023-12-19T14:57:19.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f71117fa700 1 -- 172.21.15.50:0/2708072099 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712002ee70 con 0x27f0ae0 2023-12-19T14:57:19.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f71117fa700 1 -- 172.21.15.50:0/2708072099 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71200530f0 con 0x27f0ae0 2023-12-19T14:57:19.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- 172.21.15.50:0/2708072099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 msgr2=0x2a25010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f71200526b0 tx=0x7f712000b040 comp rx=0 tx=0).stop 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- 172.21.15.50:0/2708072099 shutdown_connections 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x283b790 0x2a6d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2a25010 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 --2- 172.21.15.50:0/2708072099 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x27f0710 0x2a24ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- 172.21.15.50:0/2708072099 >> 172.21.15.50:0/2708072099 conn(0x27f0eb0 msgr2=0x2a241a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- 172.21.15.50:0/2708072099 shutdown_connections 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.506+0000 7f7134228740 1 -- 172.21.15.50:0/2708072099 wait complete. 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 -- start start 2023-12-19T14:57:19.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 Processor -- start 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2ac7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x296b040 con 0x27f0ae0 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2993b50 con 0x27f0710 2023-12-19T14:57:19.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x296d550 con 0x283b790 2023-12-19T14:57:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 10 client.0 _ll_register_callbacks cb 0x27bc9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:57:19.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45208/0 (socket says 172.21.15.50:45208) 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 -- 172.21.15.50:0/2917663602 learned_addr learned my addr 172.21.15.50:0/2917663602 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2ac7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 -- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 msgr2=0x2a274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a274c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 -- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 msgr2=0x2ac7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2ac7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 -- 172.21.15.50:0/2917663602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c0aa530 con 0x27f0ae0 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2ac7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f712affd700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f71200526b0 tx=0x7f7120079000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f803f58dba2bdd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7109ffb700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120055c40 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f712007e040 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f711c011dd0 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.510+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7120059990 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 10 client.8025 fetch_fsmap learned FSMap version 58 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) v3 -- 0x2941130 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c011dd0 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 58) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f712007cd20 con 0x27f0ae0 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 10 client.8025 fetch_fsmap finished waiting for FSMap version 58 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 10 client.8025 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T14:57:19.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 20 client.8025 populate_metadata read hostname 'smithi050' 2023-12-19T14:57:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a27e80 0x2a28250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 4 client.8025 mds_command: new command op to 7878 tid=0[{"prefix": "status"}] 2023-12-19T14:57:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 --> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] -- command(tid 0: {"prefix": "status"}) v1 -- 0x296c410 con 0x2a27e80 2023-12-19T14:57:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a27e80 0x2a28250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 10 client.8025 ms_handle_connect on v2:172.21.15.50:6834/1300068976 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f712b7fe700 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a27e80 0x2a28250 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d68fa32c86085cf0 server_cookie=5467fb549bac411a in_seq=0 out_seq=0 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 1 -- 172.21.15.50:0/2917663602 <== mds.0 v2:172.21.15.50:6834/1300068976 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x296c410 con 0x2a27e80 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 10 client.8025 handle_command_reply: tid=0 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 client.8025 shutdown 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 msgr2=0x2ac83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f71200526b0 tx=0x7f7120079000 comp rx=0 tx=0).stop 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 shutdown_connections 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:6834/1300068976,v1:172.21.15.50:6835/1300068976] conn(0x2a27e80 0x2a28250 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x283b790 0x2a274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x27f0ae0 0x2ac83f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 --2- 172.21.15.50:0/2917663602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x27f0710 0x2ac7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7109ffb700 0 client.8025 ms_handle_reset on v2:172.21.15.50:6834/1300068976 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 >> 172.21.15.50:0/2917663602 conn(0x27f0eb0 msgr2=0x2a25570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 shutdown_connections 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 1 -- 172.21.15.50:0/2917663602 wait complete. 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.514+0000 7f7134228740 20 client.8025 trim_cache size 0 max 16384 2023-12-19T14:57:19.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 7878, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3013741899999998, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 58, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 129, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 129, 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.215900261 2023-12-19T14:57:19.520 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:57:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 msgr2=0x7f7118062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 0x7f7118062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f711c00bd10 tx=0x7f711c008040 comp rx=0 tx=0).stop 2023-12-19T14:57:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 msgr2=0x7f712c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7120009290 tx=0x7f7120017000 comp rx=0 tx=0).stop 2023-12-19T14:57:19.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 shutdown_connections 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f71180606a0 0x7f7118062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f712c13f6b0 0x7f712c13bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f712c0c7d90 0x7f712c0a5410 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 --2- 172.21.15.50:0/4070603094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f712c06c7c0 0x7f712c0a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 >> 172.21.15.50:0/4070603094 conn(0x7f712c067b60 msgr2=0x7f712c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 shutdown_connections 2023-12-19T14:57:19.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.518+0000 7f7131f45700 1 -- 172.21.15.50:0/4070603094 wait complete. 2023-12-19T14:57:19.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:57:19.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.706+0000 7f5693bc5700 1 Processor -- start 2023-12-19T14:57:19.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.706+0000 7f5693bc5700 1 -- start start 2023-12-19T14:57:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5693bc5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f568c158d40 0x7f568c067960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5693bc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5693bc5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c068be0 con 0x7f568c069070 2023-12-19T14:57:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5693bc5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c159530 con 0x7f568c158d40 2023-12-19T14:57:19.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5693bc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c067fd0 con 0x7f568c159110 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5691961700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f568c158d40 0x7f568c067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5692162700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5691961700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f568c158d40 0x7f568c067960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56278/0 (socket says 172.21.15.50:56278) 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5691961700 1 -- 172.21.15.50:0/2743170768 learned_addr learned my addr 172.21.15.50:0/2743170768 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.710+0000 7f5691160700 1 --1- 172.21.15.50:0/2743170768 >> v1:172.21.15.125:6789/0 conn(0x7f568c069070 0x7f568c0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39726/0 (socket says 172.21.15.50:39726) 2023-12-19T14:57:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5691961700 1 -- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 msgr2=0x7f568c0cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5691961700 1 --2- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5691961700 1 -- 172.21.15.50:0/2743170768 >> v1:172.21.15.125:6789/0 conn(0x7f568c069070 legacy=0x7f568c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5691961700 1 -- 172.21.15.50:0/2743170768 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0682b0 con 0x7f568c158d40 2023-12-19T14:57:19.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5691961700 1 --2- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c158d40 0x7f568c067960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f56880099b0 tx=0x7f5688009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c11c5806533fda4e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f569095f700 1 -- 172.21.15.50:0/2743170768 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688009ea0 con 0x7f568c158d40 2023-12-19T14:57:19.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f569095f700 1 -- 172.21.15.50:0/2743170768 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568800b860 con 0x7f568c158d40 2023-12-19T14:57:19.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f569095f700 1 -- 172.21.15.50:0/2743170768 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568800bc00 con 0x7f568c158d40 2023-12-19T14:57:19.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 -- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c158d40 msgr2=0x7f568c067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 --2- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c158d40 0x7f568c067960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f56880099b0 tx=0x7f5688009250 comp rx=0 tx=0).stop 2023-12-19T14:57:19.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 -- 172.21.15.50:0/2743170768 shutdown_connections 2023-12-19T14:57:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 --2- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 --2- 172.21.15.50:0/2743170768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c158d40 0x7f568c067960 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 -- 172.21.15.50:0/2743170768 >> 172.21.15.50:0/2743170768 conn(0x7f568c0ccc40 msgr2=0x7f568c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 -- 172.21.15.50:0/2743170768 shutdown_connections 2023-12-19T14:57:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.714+0000 7f5693bc5700 1 -- 172.21.15.50:0/2743170768 wait complete. 2023-12-19T14:57:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 Processor -- start 2023-12-19T14:57:19.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 -- start start 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 0x7f568c15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c0ca120 con 0x7f568c158d40 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c15cea0 con 0x7f568c069070 2023-12-19T14:57:19.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5693bc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c15d180 con 0x7f568c159110 2023-12-19T14:57:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45220/0 (socket says 172.21.15.50:45220) 2023-12-19T14:57:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 -- 172.21.15.50:0/4178561163 learned_addr learned my addr 172.21.15.50:0/4178561163 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691961700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 0x7f568c15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5692162700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 -- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 msgr2=0x7f568c0afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 -- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 msgr2=0x7f568c15c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 0x7f568c15c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691160700 1 -- 172.21.15.50:0/4178561163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56880094f0 con 0x7f568c158d40 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.718+0000 7f5691961700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 0x7f568c15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:57:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5691160700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c7c0 tx=0x7f567c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c005a70 con 0x7f568c158d40 2023-12-19T14:57:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567c005bd0 con 0x7f568c158d40 2023-12-19T14:57:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c15d460 con 0x7f568c158d40 2023-12-19T14:57:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c0070d0 con 0x7f568c158d40 2023-12-19T14:57:19.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c15da20 con 0x7f568c158d40 2023-12-19T14:57:19.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.722+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c15dc40 con 0x7f568c158d40 2023-12-19T14:57:19.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.726+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f567c007480 con 0x7f568c158d40 2023-12-19T14:57:19.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.726+0000 7f5681ffb700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 0x7f5670062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:19.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.726+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f567c07be90 con 0x7f568c158d40 2023-12-19T14:57:19.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.726+0000 7f5691961700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 0x7f5670062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:19.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.726+0000 7f5691961700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 0x7f5670062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56880099b0 tx=0x7f5688017000 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:19.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.734+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f567c04c210 con 0x7f568c158d40 2023-12-19T14:57:19.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.942+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f568c071120 con 0x7f568c158d40 2023-12-19T14:57:19.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.942+0000 7f5681ffb700 1 -- 172.21.15.50:0/4178561163 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v58) v1 ==== 53+0+73 (secure 0 0 0) 0x7f567c017070 con 0x7f568c158d40 2023-12-19T14:57:19.946 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 msgr2=0x7f5670062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 0x7f5670062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56880099b0 tx=0x7f5688017000 comp rx=0 tx=0).stop 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 msgr2=0x7f568c14e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c7c0 tx=0x7f567c007f70 comp rx=0 tx=0).stop 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 shutdown_connections 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5670060670 0x7f5670062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f568c159110 0x7f568c0afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f568c158d40 0x7f568c14e090 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 --2- 172.21.15.50:0/4178561163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f568c069070 0x7f568c15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 >> 172.21.15.50:0/4178561163 conn(0x7f568c0ccc40 msgr2=0x7f568c0c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 shutdown_connections 2023-12-19T14:57:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:19.950+0000 7f5693bc5700 1 -- 172.21.15.50:0/4178561163 wait complete. 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:57:19.974 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:57:19.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:19.975 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:57:19.993 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41310sec preferred_lft 41310sec 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.994 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:57:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:57:19.996 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:57:19.996 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:57:19.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:19.997 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:57:20.076 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:57:20.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:20.079 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:57:20.143 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:57:20.143 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:57:20.143 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:57:20.144 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:57:20.144 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:57:20.144 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:57:20.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:57:20.145 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:57:20.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:20.274 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:57:20.286 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:57:20.286 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:57:20.287 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:57:20.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:57:20.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:57:20.366 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:57:20.366 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:57:20.366 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:20.429 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:57:20.430 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:57:20.430 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:20.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:57:20.574 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:57:20.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:57:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:57:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:57:20.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:20.659 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:57:20.721 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:57:20.721 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:57:20.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:57:20.807 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:57:20.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:57:20.887 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:57:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:57:20.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:20.891 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:57:20.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:57:20.914+0000 7f2a03546380 -1 init, newargv = 0x558fa161be30 newargc=15 2023-12-19T14:57:20.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[52755]: starting ceph client 2023-12-19T14:57:20.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[52755]: starting fuse 2023-12-19T14:57:20.935 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:57:20.936 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:57:20.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:57:20.936 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:57:20.936 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:57:20.937 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:57:20.938 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:57:21.108 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.52755.asok 2023-12-19T14:57:21.292 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52755.asok 2023-12-19T14:57:21.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:21.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52755.asok status 2023-12-19T14:57:21.481 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:57:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:57:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:57:21.481 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "52755", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 8034, 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8034 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:21.482 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3938342586 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3938342586 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.8034 192.168.0.1:0/3938342586", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/3938342586", 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 58, 2023-12-19T14:57:21.483 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 129, 2023-12-19T14:57:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:57:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:57:21.484 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:57:21.484 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:57:21.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:21.494 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:57:21.494 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:21.515 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:57:21.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:21.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:21.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:57:21.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:57:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.814+0000 7f5cda6ae700 1 Processor -- start 2023-12-19T14:57:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.814+0000 7f5cda6ae700 1 -- start start 2023-12-19T14:57:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cda6ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cda6ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cda6ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd406d7d0 con 0x7f5cd4140420 2023-12-19T14:57:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cda6ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd40ae280 con 0x7f5cd406c7c0 2023-12-19T14:57:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cda6ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd40cb8f0 con 0x7f5cd41407f0 2023-12-19T14:57:21.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56288/0 (socket says 172.21.15.50:56288) 2023-12-19T14:57:21.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/262625919 learned_addr learned my addr 172.21.15.50:0/262625919 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:21.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cd8c4b700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5cd3fff700 1 --1- 172.21.15.50:0/262625919 >> v1:172.21.15.125:6789/0 conn(0x7f5cd4140420 0x7f5cd415b220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39742/0 (socket says 172.21.15.50:39742) 2023-12-19T14:57:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 msgr2=0x7f5cd40b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/262625919 >> v1:172.21.15.125:6789/0 conn(0x7f5cd4140420 legacy=0x7f5cd415b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.818+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/262625919 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd406cc50 con 0x7f5cd406c7c0 2023-12-19T14:57:21.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5ccb7fe700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0099b0 tx=0x7f5cbc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574a12de2312aff4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cd37fe700 1 -- 172.21.15.50:0/262625919 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc009ea0 con 0x7f5cd406c7c0 2023-12-19T14:57:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cd37fe700 1 -- 172.21.15.50:0/262625919 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00b860 con 0x7f5cd406c7c0 2023-12-19T14:57:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cd37fe700 1 -- 172.21.15.50:0/262625919 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00bc00 con 0x7f5cd406c7c0 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 -- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 msgr2=0x7f5cd40b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0099b0 tx=0x7f5cbc009250 comp rx=0 tx=0).stop 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 -- 172.21.15.50:0/262625919 shutdown_connections 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/262625919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd40b0cd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 -- 172.21.15.50:0/262625919 >> 172.21.15.50:0/262625919 conn(0x7f5cd4067b60 msgr2=0x7f5cd40ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 -- 172.21.15.50:0/262625919 shutdown_connections 2023-12-19T14:57:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.822+0000 7f5cda6ae700 1 -- 172.21.15.50:0/262625919 wait complete. 2023-12-19T14:57:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 Processor -- start 2023-12-19T14:57:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 -- start start 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd414c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40c2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd416a570 con 0x7f5cd4140420 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd416add0 con 0x7f5cd406c7c0 2023-12-19T14:57:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cda6ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd414cbd0 con 0x7f5cd41407f0 2023-12-19T14:57:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45228/0 (socket says 172.21.15.50:45228) 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/2978895642 learned_addr learned my addr 172.21.15.50:0/2978895642 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cd3fff700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd414c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5cd8c4b700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 msgr2=0x7f5cd40c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40c2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 msgr2=0x7f5cd414c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd414c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.826+0000 7f5ccb7fe700 1 -- 172.21.15.50:0/2978895642 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0094f0 con 0x7f5cd4140420 2023-12-19T14:57:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5ccb7fe700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4069d50 tx=0x7f5cbc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00bbe0 con 0x7f5cd4140420 2023-12-19T14:57:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc011420 con 0x7f5cd4140420 2023-12-19T14:57:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0117c0 con 0x7f5cd4140420 2023-12-19T14:57:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd4167cf0 con 0x7f5cd4140420 2023-12-19T14:57:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd416aa60 con 0x7f5cd4140420 2023-12-19T14:57:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.830+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd4071120 con 0x7f5cd4140420 2023-12-19T14:57:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.834+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5cbc011bf0 con 0x7f5cd4140420 2023-12-19T14:57:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.834+0000 7f5cd0ff9700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 0x7f5cb4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.834+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f5cbc07dff0 con 0x7f5cd4140420 2023-12-19T14:57:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.834+0000 7f5cd3fff700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 0x7f5cb4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:21.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.834+0000 7f5cd3fff700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 0x7f5cb4062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4001f80 tx=0x7f5cc4005cd0 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:21.842+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5cbc04e3f0 con 0x7f5cd4140420 2023-12-19T14:57:22.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.038+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cd406cf30 con 0x7f5cb40606a0 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cd0ff9700 1 -- 172.21.15.50:0/2978895642 <== mgr.6833 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5cd406cf30 con 0x7f5cb40606a0 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:57:22.043 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:57:22.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 msgr2=0x7f5cb4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 0x7f5cb4062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4001f80 tx=0x7f5cc4005cd0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 msgr2=0x7f5cd40c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4069d50 tx=0x7f5cbc017000 comp rx=0 tx=0).stop 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 shutdown_connections 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5cb40606a0 0x7f5cb4062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5cd41407f0 0x7f5cd40c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5cd4140420 0x7f5cd40c22d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 --2- 172.21.15.50:0/2978895642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5cd406c7c0 0x7f5cd414c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 >> 172.21.15.50:0/2978895642 conn(0x7f5cd4067b60 msgr2=0x7f5cd41584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 shutdown_connections 2023-12-19T14:57:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.042+0000 7f5cda6ae700 1 -- 172.21.15.50:0/2978895642 wait complete. 2023-12-19T14:57:22.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:57:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.230+0000 7fe6b9d23700 1 Processor -- start 2023-12-19T14:57:22.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.230+0000 7fe6b9d23700 1 -- start start 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b4158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9d23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b406b6c0 con 0x7fe6b406bb50 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9d23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b406aab0 con 0x7fe6b4158b00 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b406ad90 con 0x7fe6b40cb290 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9522700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe6b406bb50 0x7fe6b40afd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47628/0 (socket says 172.21.15.50:47628) 2023-12-19T14:57:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b8d21700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe6b4158b00 0x7fe6b40cab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39756/0 (socket says 172.21.15.50:39756) 2023-12-19T14:57:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b9522700 1 -- 172.21.15.50:0/2712973156 learned_addr learned my addr 172.21.15.50:0/2712973156 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.234+0000 7fe6b3fff700 1 --2- 172.21.15.50:0/2712973156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b4158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276471455 0 0) 0x7fe6b406b6c0 con 0x7fe6b406bb50 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a4003570 con 0x7fe6b406bb50 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368115754 0 0) 0x7fe6b406aab0 con 0x7fe6b4158b00 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b406b6c0 con 0x7fe6b4158b00 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b3fff700 1 -- 172.21.15.50:0/2712973156 >> v1:172.21.15.125:6789/0 conn(0x7fe6b4158b00 legacy=0x7fe6b40cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b3fff700 1 -- 172.21.15.50:0/2712973156 >> v1:172.21.15.50:6789/0 conn(0x7fe6b406bb50 legacy=0x7fe6b40afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 559970219 0 0) 0x7fe6a4003570 con 0x7fe6b406bb50 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b3fff700 1 -- 172.21.15.50:0/2712973156 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b406b070 con 0x7fe6b40cb290 2023-12-19T14:57:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b3fff700 1 --2- 172.21.15.50:0/2712973156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b4158ed0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009910 tx=0x7fe6a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7cc2973fa7995e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac001720 con 0x7fe6b40cb290 2023-12-19T14:57:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0011420 con 0x7fe6b40cb290 2023-12-19T14:57:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b37fe700 1 -- 172.21.15.50:0/2712973156 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00117c0 con 0x7fe6b40cb290 2023-12-19T14:57:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b9d23700 1 -- 172.21.15.50:0/2712973156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 msgr2=0x7fe6b4158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.238+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/2712973156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b4158ed0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009910 tx=0x7fe6a00091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 -- 172.21.15.50:0/2712973156 shutdown_connections 2023-12-19T14:57:22.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/2712973156 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b4158ed0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 -- 172.21.15.50:0/2712973156 >> 172.21.15.50:0/2712973156 conn(0x7fe6b4067bb0 msgr2=0x7fe6b40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 -- 172.21.15.50:0/2712973156 shutdown_connections 2023-12-19T14:57:22.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 -- 172.21.15.50:0/2712973156 wait complete. 2023-12-19T14:57:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 Processor -- start 2023-12-19T14:57:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.242+0000 7fe6b9d23700 1 -- start start 2023-12-19T14:57:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6b406bb50 0x7fe6b4150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b40b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 0x7fe6b414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:22.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40b5810 con 0x7fe6b4158b00 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b416cfb0 con 0x7fe6b406bb50 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4151500 con 0x7fe6b40cb290 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9522700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 0x7fe6b414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b40b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b8d21700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6b406bb50 0x7fe6b4150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9522700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 0x7fe6b414dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45234/0 (socket says 172.21.15.50:45234) 2023-12-19T14:57:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9522700 1 -- 172.21.15.50:0/669534539 learned_addr learned my addr 172.21.15.50:0/669534539 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 -- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6b406bb50 msgr2=0x7fe6b4150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6b406bb50 0x7fe6b4150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 -- 172.21.15.50:0/669534539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 msgr2=0x7fe6b414dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 0x7fe6b414dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 -- 172.21.15.50:0/669534539 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a0009420 con 0x7fe6b40cb290 2023-12-19T14:57:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b3fff700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b40b52d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0015000 tx=0x7fe6a0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a001cb30 con 0x7fe6b40cb290 2023-12-19T14:57:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a001ccd0 con 0x7fe6b40cb290 2023-12-19T14:57:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b416c680 con 0x7fe6b40cb290 2023-12-19T14:57:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a001a6a0 con 0x7fe6b40cb290 2023-12-19T14:57:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.246+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b416cc40 con 0x7fe6b40cb290 2023-12-19T14:57:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.250+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b416d120 con 0x7fe6b40cb290 2023-12-19T14:57:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.250+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 49) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe6a001aa90 con 0x7fe6b40cb290 2023-12-19T14:57:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.250+0000 7fe6b0ff9700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 0x7fe698062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:22.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.254+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fe6a007e800 con 0x7fe6b40cb290 2023-12-19T14:57:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.254+0000 7fe6b8d21700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 0x7fe698062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.254+0000 7fe6b8d21700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 0x7fe698062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8004f70 tx=0x7fe6a8007d40 comp rx=0 tx=0).ready entity=mgr.6833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.258+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe6a004eb20 con 0x7fe6b40cb290 2023-12-19T14:57:22.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:22.470+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe6b404be40 con 0x7fe6b40cb290 2023-12-19T14:57:23.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.058+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 50) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe6a001ad40 con 0x7fe6b40cb290 2023-12-19T14:57:23.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.058+0000 7fe6b0ff9700 1 -- 172.21.15.50:0/669534539 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v50) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe6a0064040 con 0x7fe6b40cb290 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 msgr2=0x7fe698062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 0x7fe698062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8004f70 tx=0x7fe6a8007d40 comp rx=0 tx=0).stop 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 msgr2=0x7fe6b40b52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b40b52d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0015000 tx=0x7fe6a0016000 comp rx=0 tx=0).stop 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 shutdown_connections 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe698060670 0x7fe698062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6b4158b00 0x7fe6b414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6b40cb290 0x7fe6b40b52d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 --2- 172.21.15.50:0/669534539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6b406bb50 0x7fe6b4150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 >> 172.21.15.50:0/669534539 conn(0x7fe6b4067bb0 msgr2=0x7fe6b40d0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:23.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 shutdown_connections 2023-12-19T14:57:23.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.062+0000 7fe6b9d23700 1 -- 172.21.15.50:0/669534539 wait complete. 2023-12-19T14:57:23.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:57:23.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 Processor -- start 2023-12-19T14:57:23.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 -- start start 2023-12-19T14:57:23.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed4158e20 0x7f4ed40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4ed40ca2c0 0x7f4ed40ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40cbdf0 con 0x7f4ed40cb970 2023-12-19T14:57:23.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed40cc0d0 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.254+0000 7f4edaef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40cc3b0 con 0x7f4ed4158e20 2023-12-19T14:57:23.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed3fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4ed40cb970 0x7f4ed406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39758/0 (socket says 172.21.15.50:39758) 2023-12-19T14:57:23.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed3fff700 1 -- 172.21.15.50:0/2786103221 learned_addr learned my addr 172.21.15.50:0/2786103221 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:23.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed8c8c700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed4158e20 0x7f4ed40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 --2- 172.21.15.50:0/2786103221 >> v2:172.21.15.125:3300/0 conn(0x7f4ed40ca2c0 0x7f4ed40ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed37fe700 1 -- 172.21.15.50:0/2786103221 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586064769 0 0) 0x7f4ed40cbdf0 con 0x7f4ed40cb970 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed37fe700 1 -- 172.21.15.50:0/2786103221 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ebc003570 con 0x7f4ed40cb970 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 -- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed4158e20 msgr2=0x7f4ed40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed4158e20 0x7f4ed40ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 -- 172.21.15.50:0/2786103221 >> v1:172.21.15.125:6789/0 conn(0x7f4ed40cb970 legacy=0x7f4ed406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 -- 172.21.15.50:0/2786103221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40cc690 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed948d700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed40ceb00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009e60 tx=0x7f4ecc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc85f47aaad6bd30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:23.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed37fe700 1 -- 172.21.15.50:0/2786103221 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00e070 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed37fe700 1 -- 172.21.15.50:0/2786103221 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc00bab0 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4ed37fe700 1 -- 172.21.15.50:0/2786103221 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00be70 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.261 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:23.255+0000 7fa31bf92000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:57:23.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 -- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed40ca2c0 msgr2=0x7f4ed40ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed40ceb00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009e60 tx=0x7f4ecc009780 comp rx=0 tx=0).stop 2023-12-19T14:57:23.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 -- 172.21.15.50:0/2786103221 shutdown_connections 2023-12-19T14:57:23.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed40ceb00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 --2- 172.21.15.50:0/2786103221 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed4158e20 0x7f4ed40ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 -- 172.21.15.50:0/2786103221 >> 172.21.15.50:0/2786103221 conn(0x7f4ed4067bb0 msgr2=0x7f4ed40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:23.263 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f264275a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:57:23.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.258+0000 7f4edaef0700 1 -- 172.21.15.50:0/2786103221 shutdown_connections 2023-12-19T14:57:23.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.262+0000 7f4edaef0700 1 -- 172.21.15.50:0/2786103221 wait complete. 2023-12-19T14:57:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.262+0000 7f4edaef0700 1 Processor -- start 2023-12-19T14:57:23.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.262+0000 7f4edaef0700 1 -- start start 2023-12-19T14:57:23.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 0x7f4ed41522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed416c890 0x7f4ed416cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c58f0 con 0x7f4ed40ca2c0 2023-12-19T14:57:23.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed416d240 con 0x7f4ed4158e20 2023-12-19T14:57:23.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c5ae0 con 0x7f4ed416c890 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed8c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 0x7f4ed41522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed8c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed41559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45254/0 (socket says 172.21.15.50:45254) 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed8c8c700 1 -- 172.21.15.50:0/845787196 learned_addr learned my addr 172.21.15.50:0/845787196 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed948d700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed416c890 0x7f4ed416cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed416c890 msgr2=0x7f4ed416cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed416c890 0x7f4ed416cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 msgr2=0x7f4ed41559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed41559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 -- 172.21.15.50:0/845787196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc009a20 con 0x7f4ed4158e20 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed3fff700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 0x7f4ed41522c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40cf0d0 tx=0x7f4ec4007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4005b30 con 0x7f4ed4158e20 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4005c90 con 0x7f4ed4158e20 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed40c5dc0 con 0x7f4ed4158e20 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4007320 con 0x7f4ed4158e20 2023-12-19T14:57:23.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.266+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40c6380 con 0x7f4ed4158e20 2023-12-19T14:57:23.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed4071120 con 0x7f4ed4158e20 2023-12-19T14:57:23.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 50) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f4ec401b3b0 con 0x7f4ed4158e20 2023-12-19T14:57:23.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4ed0ff9700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 0x7f4ec0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f4ec407bf30 con 0x7f4ed4158e20 2023-12-19T14:57:23.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4ed8c8c700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 msgr2=0x7f4ec0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:23.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.270+0000 7f4ed8c8c700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 0x7f4ec0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:57:23.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.278+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4ec404c210 con 0x7f4ed4158e20 2023-12-19T14:57:23.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.454+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f4ed406a7a0 con 0x7f4ed4158e20 2023-12-19T14:57:23.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.470+0000 7f4ed0ff9700 1 -- 172.21.15.50:0/845787196 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4ec40170d0 con 0x7f4ed4158e20 2023-12-19T14:57:23.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4ed8c8c700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 msgr2=0x7f4ec0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:23.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4ed8c8c700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 0x7f4ec0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 msgr2=0x7f4ec0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 0x7f4ec0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 msgr2=0x7f4ed41522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 0x7f4ed41522c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40cf0d0 tx=0x7f4ec4007d50 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 shutdown_connections 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ed4158e20 0x7f4ed41522c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4ec00606c0 0x7f4ec0062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4ed416c890 0x7f4ed416cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 --2- 172.21.15.50:0/845787196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ed40ca2c0 0x7f4ed41559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 >> 172.21.15.50:0/845787196 conn(0x7f4ed4067bb0 msgr2=0x7f4ed414c980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:23.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 shutdown_connections 2023-12-19T14:57:23.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.474+0000 7f4edaef0700 1 -- 172.21.15.50:0/845787196 wait complete. 2023-12-19T14:57:23.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:57:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.690+0000 7f48caacd700 1 Processor -- start 2023-12-19T14:57:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.690+0000 7f48caacd700 1 -- start start 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f48c4158e20 0x7f48c40ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48c40cb970 0x7f48c40ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c40ca2c0 0x7f48c406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40cbdf0 con 0x7f48c4158e20 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40cc0d0 con 0x7f48c40cb970 2023-12-19T14:57:23.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48caacd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40cc3b0 con 0x7f48c40ca2c0 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f48c4158e20 0x7f48c40ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f48c4158e20 0x7f48c40ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45266/0 (socket says 172.21.15.50:45266) 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 -- 172.21.15.50:0/991745643 learned_addr learned my addr 172.21.15.50:0/991745643 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c906a700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c40ca2c0 0x7f48c406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c3fff700 1 --2- 172.21.15.50:0/991745643 >> v2:172.21.15.125:3300/0 conn(0x7f48c40cb970 0x7f48c40ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 -- 172.21.15.50:0/991745643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c40ca2c0 msgr2=0x7f48c406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c40ca2c0 0x7f48c406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 -- 172.21.15.50:0/991745643 >> v2:172.21.15.125:3300/0 conn(0x7f48c40cb970 msgr2=0x7f48c40ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 --2- 172.21.15.50:0/991745643 >> v2:172.21.15.125:3300/0 conn(0x7f48c40cb970 0x7f48c40ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 -- 172.21.15.50:0/991745643 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c40cc690 con 0x7f48c4158e20 2023-12-19T14:57:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c8869700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4158e20 0x7f48c40ce5c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f48b80099b0 tx=0x7f48b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518ebe629eb0ca3a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c37fe700 1 -- 172.21.15.50:0/991745643 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b800b820 con 0x7f48c4158e20 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c37fe700 1 -- 172.21.15.50:0/991745643 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b800b980 con 0x7f48c4158e20 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.694+0000 7f48c37fe700 1 -- 172.21.15.50:0/991745643 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b800bd20 con 0x7f48c4158e20 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- 172.21.15.50:0/991745643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4158e20 msgr2=0x7f48c40ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4158e20 0x7f48c40ce5c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f48b80099b0 tx=0x7f48b8009250 comp rx=0 tx=0).stop 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- 172.21.15.50:0/991745643 shutdown_connections 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c40ca2c0 0x7f48c406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 --2- 172.21.15.50:0/991745643 >> v2:172.21.15.125:3300/0 conn(0x7f48c40cb970 0x7f48c40ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 --2- 172.21.15.50:0/991745643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4158e20 0x7f48c40ce5c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- 172.21.15.50:0/991745643 >> 172.21.15.50:0/991745643 conn(0x7f48c4067bb0 msgr2=0x7f48c40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- 172.21.15.50:0/991745643 shutdown_connections 2023-12-19T14:57:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- 172.21.15.50:0/991745643 wait complete. 2023-12-19T14:57:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 Processor -- start 2023-12-19T14:57:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.698+0000 7f48caacd700 1 -- start start 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c40ca2c0 0x7f48c416c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c4158e20 0x7f48c4169840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c416cbf0 con 0x7f48c4159c30 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40c8360 con 0x7f48c40ca2c0 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48caacd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40c8640 con 0x7f48c4158e20 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45270/0 (socket says 172.21.15.50:45270) 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 -- 172.21.15.50:0/1247480696 learned_addr learned my addr 172.21.15.50:0/1247480696 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c3fff700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c4158e20 0x7f48c4169840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c8869700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c40ca2c0 0x7f48c416c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:23.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 -- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c4158e20 msgr2=0x7f48c4169840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c4158e20 0x7f48c4169840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 -- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c40ca2c0 msgr2=0x7f48c416c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c40ca2c0 0x7f48c416c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 -- 172.21.15.50:0/1247480696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80094f0 con 0x7f48c4159c30 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c906a700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009fa0 tx=0x7f48bc009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc00ebc0 con 0x7f48c4159c30 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc00ed20 con 0x7f48c4159c30 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.702+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc014670 con 0x7f48c4159c30 2023-12-19T14:57:23.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.706+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c40c8920 con 0x7f48c4159c30 2023-12-19T14:57:23.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.706+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c40c8ee0 con 0x7f48c4159c30 2023-12-19T14:57:23.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.706+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c4071120 con 0x7f48c4159c30 2023-12-19T14:57:23.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.706+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 50) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f48bc0149e0 con 0x7f48c4159c30 2023-12-19T14:57:23.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.710+0000 7f48c0ff9700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 0x7f48a8062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:23.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.710+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f48bc07d1c0 con 0x7f48c4159c30 2023-12-19T14:57:23.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.710+0000 7f48c8869700 1 -- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 msgr2=0x7f48a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:23.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.710+0000 7f48c8869700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 0x7f48a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:57:23.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.718+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f48bc04d4a0 con 0x7f48c4159c30 2023-12-19T14:57:23.805 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:23.799+0000 7fa31bf92000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:57:23.847 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:23.846+0000 7f264275a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:57:23.884 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:23.879+0000 7fa31bf92000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:57:23.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.894+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f48c406a7a0 con 0x7f48c4159c30 2023-12-19T14:57:23.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48c0ff9700 1 -- 172.21.15.50:0/1247480696 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f48bc050ac0 con 0x7f48c4159c30 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 msgr2=0x7f48a8062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 0x7f48a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 msgr2=0x7f48c415a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009fa0 tx=0x7f48bc009900 comp rx=0 tx=0).stop 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 shutdown_connections 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f48a80606c0 0x7f48a8062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48c4159c30 0x7f48c415a0d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48c4158e20 0x7f48c4169840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 --2- 172.21.15.50:0/1247480696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48c40ca2c0 0x7f48c416c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 >> 172.21.15.50:0/1247480696 conn(0x7f48c4067bb0 msgr2=0x7f48c406afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 shutdown_connections 2023-12-19T14:57:23.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:23.902+0000 7f48caacd700 1 -- 172.21.15.50:0/1247480696 wait complete. 2023-12-19T14:57:23.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:23.926 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T14:57:23.933 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:23.926+0000 7f264275a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:57:23.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T14:57:24.044 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.039+0000 7fa31bf92000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:57:24.086 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.082+0000 7f264275a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:57:24.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.170+0000 7f64b359e700 1 Processor -- start 2023-12-19T14:57:24.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.170+0000 7f64b359e700 1 -- start start 2023-12-19T14:57:24.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b40aedc0 0x7f64b40af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64b406af50 0x7f64b40cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40cb3f0 con 0x7f64b406d6b0 2023-12-19T14:57:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b359e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b40cc140 con 0x7f64b406af50 2023-12-19T14:57:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b406d220 con 0x7f64b40aedc0 2023-12-19T14:57:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b2d9d700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64b406d6b0 0x7f64b40af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39774/0 (socket says 172.21.15.50:39774) 2023-12-19T14:57:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b2d9d700 1 -- 172.21.15.50:0/3655592922 learned_addr learned my addr 172.21.15.50:0/3655592922 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:57:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b1d9b700 1 --2- 172.21.15.50:0/3655592922 >> v2:172.21.15.50:3300/0 conn(0x7f64b406af50 0x7f64b40cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b259c700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b40aedc0 0x7f64b40af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b159a700 1 -- 172.21.15.50:0/3655592922 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572677328 0 0) 0x7f64b40cb3f0 con 0x7f64b406d6b0 2023-12-19T14:57:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b1d9b700 1 -- 172.21.15.50:0/3655592922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b40aedc0 msgr2=0x7f64b40af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b1d9b700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b40aedc0 0x7f64b40af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b1d9b700 1 -- 172.21.15.50:0/3655592922 >> v1:172.21.15.125:6789/0 conn(0x7f64b406d6b0 legacy=0x7f64b40af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.174+0000 7f64b1d9b700 1 -- 172.21.15.50:0/3655592922 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40cb190 con 0x7f64b406af50 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b1d9b700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b406af50 0x7f64b40cac50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f64a80098e0 tx=0x7f64a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb9913e40f870c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b159a700 1 -- 172.21.15.50:0/3655592922 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800b640 con 0x7f64b406af50 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b159a700 1 -- 172.21.15.50:0/3655592922 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a800b7a0 con 0x7f64b406af50 2023-12-19T14:57:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b159a700 1 -- 172.21.15.50:0/3655592922 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800bb40 con 0x7f64b406af50 2023-12-19T14:57:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- 172.21.15.50:0/3655592922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b406af50 msgr2=0x7f64b40cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b406af50 0x7f64b40cac50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f64a80098e0 tx=0x7f64a80091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- 172.21.15.50:0/3655592922 shutdown_connections 2023-12-19T14:57:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b406af50 0x7f64b40cac50 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 --2- 172.21.15.50:0/3655592922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b40aedc0 0x7f64b40af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- 172.21.15.50:0/3655592922 >> 172.21.15.50:0/3655592922 conn(0x7f64b40cc760 msgr2=0x7f64b40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- 172.21.15.50:0/3655592922 shutdown_connections 2023-12-19T14:57:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- 172.21.15.50:0/3655592922 wait complete. 2023-12-19T14:57:24.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 Processor -- start 2023-12-19T14:57:24.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.178+0000 7f64b359e700 1 -- start start 2023-12-19T14:57:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b406af50 0x7f64b4158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64b406d6b0 0x7f64b40b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40c4890 con 0x7f64b40aedc0 2023-12-19T14:57:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b40c4400 con 0x7f64b406d6b0 2023-12-19T14:57:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b416d920 con 0x7f64b406af50 2023-12-19T14:57:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b406af50 0x7f64b4158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:24.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45286/0 (socket says 172.21.15.50:45286) 2023-12-19T14:57:24.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b1d9b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64b406d6b0 0x7f64b40b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:24.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 -- 172.21.15.50:0/1233577117 learned_addr learned my addr 172.21.15.50:0/1233577117 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:24.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b406af50 msgr2=0x7f64b4158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b406af50 0x7f64b4158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64b406d6b0 msgr2=0x7f64b40b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64b406d6b0 0x7f64b40b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8009420 con 0x7f64b40aedc0 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b2d9d700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00cbf0 tx=0x7f64ac0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:24.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00a8b0 con 0x7f64b40aedc0 2023-12-19T14:57:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac00aa10 con 0x7f64b40aedc0 2023-12-19T14:57:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00add0 con 0x7f64b40aedc0 2023-12-19T14:57:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b40c37f0 con 0x7f64b40aedc0 2023-12-19T14:57:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.182+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b40c3db0 con 0x7f64b40aedc0 2023-12-19T14:57:24.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.186+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b404be40 con 0x7f64b40aedc0 2023-12-19T14:57:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.190+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 50) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f64ac007300 con 0x7f64b40aedc0 2023-12-19T14:57:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.190+0000 7f64a27fc700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.190+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f64ac07ce80 con 0x7f64b40aedc0 2023-12-19T14:57:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.190+0000 7f64b259c700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.190+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:57:24.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.194+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f64ac04d1e0 con 0x7f64b40aedc0 2023-12-19T14:57:24.326 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.319+0000 7fa31bf92000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:57:24.389 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.382+0000 7f264275a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:57:24.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.386+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f64b40b0f60 con 0x7f64940606c0 2023-12-19T14:57:24.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.394+0000 7f64b259c700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:24.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.394+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:57:24.403 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.399+0000 7fa31bf92000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:57:24.463 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.462+0000 7f264275a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:57:24.622 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.615+0000 7fa31bf92000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:57:24.690 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.686+0000 7f264275a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:57:24.712 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.707+0000 7fa31bf92000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:57:24.782 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.778+0000 7f264275a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:57:24.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.798+0000 7f64b259c700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:24.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:24.798+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:57:24.801 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:24.795+0000 7fa31bf92000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:57:24.877 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:24.874+0000 7f264275a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:57:25.206 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:25.199+0000 7fa31bf92000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:57:25.289 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:25.286+0000 7f264275a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:57:25.380 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:25.375+0000 7fa31bf92000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:57:25.473 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:25.470+0000 7f264275a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:57:25.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:25.602+0000 7f64b259c700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:25.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:25.602+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:57:25.751 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:25.747+0000 7fa31bf92000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:57:25.863 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:25.862+0000 7f264275a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:57:25.958 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:25.951+0000 7fa31bf92000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:57:26.050 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.043+0000 7fa31bf92000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:57:26.105 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.102+0000 7f264275a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:57:26.128 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.123+0000 7fa31bf92000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:57:26.201 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.198+0000 7f264275a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:57:26.207 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.203+0000 7fa31bf92000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:57:26.287 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.286+0000 7f264275a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:57:26.372 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.370+0000 7f264275a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:57:26.447 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.443+0000 7fa31bf92000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:57:26.530 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.523+0000 7fa31bf92000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:57:26.617 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.614+0000 7f264275a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:57:26.627 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.619+0000 7fa31bf92000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:57:26.702 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.698+0000 7f264275a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:57:26.798 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:26.794+0000 7f264275a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:57:26.831 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.827+0000 7fa31bf92000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:57:26.931 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:26.927+0000 7fa31bf92000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:57:27.009 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:27.006+0000 7f264275a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:57:27.102 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:27.098+0000 7f264275a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:57:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:27.206+0000 7f64b259c700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:57:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:27.206+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:57:27.683 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.675+0000 7fa31bf92000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:57:27.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:27.698+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 51) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f64ac0560f0 con 0x7f64b40aedc0 2023-12-19T14:57:27.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:27.698+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:57:27.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:27.698+0000 7f64a27fc700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:27.887 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dee67700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dee67700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dee67700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dee67700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dee67700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dc662700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dc662700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dc662700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dc662700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:57:27.883+0000 7fa2dc662700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:57:27.927 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:57:27.926+0000 7f264275a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:57:28.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.706+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 52) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f64ac04d5b0 con 0x7f64b40aedc0 2023-12-19T14:57:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.710+0000 7f64a27fc700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:28.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.710+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f64b40b0f60 con 0x7f64940606c0 2023-12-19T14:57:28.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.710+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:28.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.710+0000 7f64b259c700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062af0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f64a40097c0 tx=0x7f64a40091a0 comp rx=0 tx=0).ready entity=mgr.7942 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:28.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.818+0000 7f64a27fc700 1 -- 172.21.15.50:0/1233577117 <== mgr.7942 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f64b40b0f60 con 0x7f64940606c0 2023-12-19T14:57:28.820 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T14:57:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 msgr2=0x7f6494062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062af0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f64a40097c0 tx=0x7f64a40091a0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 msgr2=0x7f64b40b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00cbf0 tx=0x7f64ac0019b0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 shutdown_connections 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64b406d6b0 0x7f64b40b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64940606c0 0x7f6494062af0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64b40aedc0 0x7f64b40b5600 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 --2- 172.21.15.50:0/1233577117 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64b406af50 0x7f64b4158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 >> 172.21.15.50:0/1233577117 conn(0x7f64b40cc760 msgr2=0x7f64b40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 shutdown_connections 2023-12-19T14:57:28.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:28.822+0000 7f64b359e700 1 -- 172.21.15.50:0/1233577117 wait complete. 2023-12-19T14:57:28.850 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T14:57:28.851 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-14_58_00 in ~32s... 2023-12-19T14:57:28.851 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-12-19T14:57:28.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T14:57:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.022+0000 7efecd4e3700 1 Processor -- start 2023-12-19T14:57:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.022+0000 7efecd4e3700 1 -- start start 2023-12-19T14:57:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efecd4e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec806bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efecd4e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efec8159d60 0x7efec806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efecd4e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec815a1e0 con 0x7efec80b2160 2023-12-19T14:57:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efecd4e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec80cd3d0 con 0x7efec8159d60 2023-12-19T14:57:29.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efecd4e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec81598d0 con 0x7efec806b6f0 2023-12-19T14:57:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efec8159d60 0x7efec806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efec8159d60 0x7efec806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54600/0 (socket says 172.21.15.50:54600) 2023-12-19T14:57:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 -- 172.21.15.50:0/3998434663 learned_addr learned my addr 172.21.15.50:0/3998434663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:29.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efec7fff700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec806bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 -- 172.21.15.50:0/3998434663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 msgr2=0x7efec806bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec806bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 -- 172.21.15.50:0/3998434663 >> v1:172.21.15.125:6789/0 conn(0x7efec80b2160 legacy=0x7efec806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:57:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 -- 172.21.15.50:0/3998434663 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec80b2590 con 0x7efec8159d60 2023-12-19T14:57:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efeccce2700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec8159d60 0x7efec806ac90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efec0009f50 tx=0x7efec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad27f4217f7c74fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:29.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.026+0000 7efec6ffd700 1 -- 172.21.15.50:0/3998434663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000e070 con 0x7efec8159d60 2023-12-19T14:57:29.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efec6ffd700 1 -- 172.21.15.50:0/3998434663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec000ba30 con 0x7efec8159d60 2023-12-19T14:57:29.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efec6ffd700 1 -- 172.21.15.50:0/3998434663 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00092e0 con 0x7efec8159d60 2023-12-19T14:57:29.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 -- 172.21.15.50:0/3998434663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec8159d60 msgr2=0x7efec806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:29.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec8159d60 0x7efec806ac90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efec0009f50 tx=0x7efec0009840 comp rx=0 tx=0).stop 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 -- 172.21.15.50:0/3998434663 shutdown_connections 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec8159d60 0x7efec806ac90 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3998434663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec806bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 -- 172.21.15.50:0/3998434663 >> 172.21.15.50:0/3998434663 conn(0x7efec8067bb0 msgr2=0x7efec80cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 -- 172.21.15.50:0/3998434663 shutdown_connections 2023-12-19T14:57:29.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.030+0000 7efecd4e3700 1 -- 172.21.15.50:0/3998434663 wait complete. 2023-12-19T14:57:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 Processor -- start 2023-12-19T14:57:29.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 -- start start 2023-12-19T14:57:29.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec8150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 0x7efec8151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec8151a00 con 0x7efec80b2160 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec8150df0 con 0x7efec8159d60 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efecd4e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec81510d0 con 0x7efec806b6f0 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54602/0 (socket says 172.21.15.50:54602) 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 -- 172.21.15.50:0/3746586379 learned_addr learned my addr 172.21.15.50:0/3746586379 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:57:29.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec7fff700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec8150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efeccce2700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 0x7efec8151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 -- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 msgr2=0x7efec8150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec8150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 -- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 msgr2=0x7efec8151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 0x7efec8151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efec77fe700 1 -- 172.21.15.50:0/3746586379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0009ae0 con 0x7efec80b2160 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.034+0000 7efeccce2700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 0x7efec8151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efec77fe700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c5d0 tx=0x7efebc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc011590 con 0x7efec80b2160 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc011710 con 0x7efec80b2160 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec81513b0 con 0x7efec80b2160 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc011ad0 con 0x7efec80b2160 2023-12-19T14:57:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec8153600 con 0x7efec80b2160 2023-12-19T14:57:29.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.038+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec806c3b0 con 0x7efec80b2160 2023-12-19T14:57:29.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.042+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 52) v1 ==== 79816+0+0 (secure 0 0 0) 0x7efebc01c460 con 0x7efec80b2160 2023-12-19T14:57:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.042+0000 7efeaffff700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 0x7efeb40628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:57:29.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.042+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efebc07c570 con 0x7efec80b2160 2023-12-19T14:57:29.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.042+0000 7efec7fff700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 0x7efeb40628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:57:29.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.046+0000 7efec7fff700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 0x7efeb40628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006fd0 tx=0x7efeb8008040 comp rx=0 tx=0).ready entity=mgr.7942 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:57:29.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.050+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efec806c3b0 con 0x7efec80b2160 2023-12-19T14:57:29.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.246+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efec806c3b0 con 0x7efeb40603f0 2023-12-19T14:57:29.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.262+0000 7efeaffff700 1 -- 172.21.15.50:0/3746586379 <== mgr.7942 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7efec806c3b0 con 0x7efeb40603f0 2023-12-19T14:57:29.267 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:57:29.267 INFO:teuthology.orchestra.run.smithi050.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-12-19T14:57:29.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 msgr2=0x7efeb40628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:29.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 0x7efeb40628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006fd0 tx=0x7efeb8008040 comp rx=0 tx=0).stop 2023-12-19T14:57:29.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 msgr2=0x7efec81530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:57:29.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c5d0 tx=0x7efebc007f60 comp rx=0 tx=0).stop 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 shutdown_connections 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efec8159d60 0x7efec8151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7efeb40603f0 0x7efeb40628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efec80b2160 0x7efec81530c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 --2- 172.21.15.50:0/3746586379 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efec806b6f0 0x7efec8150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.266+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 >> 172.21.15.50:0/3746586379 conn(0x7efec8067bb0 msgr2=0x7efec80b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.270+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 shutdown_connections 2023-12-19T14:57:29.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:57:29.270+0000 7efecd4e3700 1 -- 172.21.15.50:0/3746586379 wait complete. 2023-12-19T14:57:29.295 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-12-19T14:57:29.295 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{}]} 2023-12-19T14:57:29.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:29.295 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:29.326 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:29.326 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:29.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:30.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:30.328 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:30.363 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:30.363 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:30.363 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:31.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:31.365 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:31.402 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:31.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:31.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:32.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:32.403 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:32.440 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:32.441 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:32.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:33.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:33.442 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:33.479 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:33.480 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:33.480 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:34.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:34.481 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:34.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:34.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:34.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:35.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:35.518 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:35.551 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:35.551 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:35.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:36.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:36.553 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:36.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:36.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:36.589 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:37.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:37.590 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:37.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:37.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:37.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:38.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:57:38.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:57:38.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:38.627 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:38.658 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:38.658 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:38.658 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:39.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:39.659 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:39.696 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:39.696 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:39.696 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:40.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:40.698 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:40.732 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:40.732 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:40.732 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:41.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:41.734 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:41.770 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:41.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:41.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:42.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:42.771 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:42.808 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:42.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:42.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:43.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:43.810 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:43.847 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:43.847 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:43.847 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:44.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:44.848 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:44.884 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:44.884 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:44.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:45.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:45.885 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:45.921 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:45.921 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:45.921 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:46.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:46.922 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:46.958 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:46.959 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:46.959 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:47.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:47.960 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:47.994 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:47.994 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:47.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:48.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:48.995 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:49.029 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:49.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:49.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:50.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:50.030 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:50.062 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:50.062 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:50.062 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:51.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:51.063 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:51.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:51.098 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:51.098 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:52.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:52.099 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:52.134 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:52.134 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:52.134 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:53.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:53.136 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:53.170 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:53.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:53.171 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:54.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:54.172 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:54.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:54.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:54.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:55.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:55.210 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:55.248 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:55.248 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:55.248 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:56.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:56.249 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:56.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:56.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:56.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:57.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:57.284 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:57.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:57.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:57.322 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:58.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:58.322 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:58.356 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:58.357 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:58.357 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:57:59.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:57:59.360 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:57:59.394 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:57:59.394 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:57:59.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:58:00.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:00.395 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:58:00.430 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T14:58:00.430 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T14:58:00.430 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:58:00.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:00.834+0000 7fa2cee47700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:00.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:00.834+0000 7fa2cee47700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:00.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:00.834+0000 7fa2cee47700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:00.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:00.834+0000 7fa2cee47700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:00.840 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:00.834+0000 7fa2cee47700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:01.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:01.431 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:58:01.463 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_58_00_UTC 2023-12-19T14:58:01.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-14_58_00_UTC'} 2023-12-19T14:58:01.465 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-14_58_00_UTC'} 2023-12-19T14:58:01.465 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T14:58:01.465 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-14_58_00_UTC'] 2023-12-19T14:58:01.465 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702997848.8511136, now = 1702997881.4657123, timo = 32 2023-12-19T14:58:02.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T14:58:02.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 Processor -- start 2023-12-19T14:58:02.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 -- start start 2023-12-19T14:58:02.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f601569e0 0x7f6f60156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 0x7f6f600ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6006beb0 con 0x7f6f600aef60 2023-12-19T14:58:02.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6006cb80 con 0x7f6f601569e0 2023-12-19T14:58:02.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f65e8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6006ce60 con 0x7f6f600adf40 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.645+0000 7f6f64e8c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f601569e0 0x7f6f60156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5ffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6f600aef60 0x7f6f600b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51832/0 (socket says 172.21.15.50:51832) 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5ffff700 1 -- 172.21.15.50:0/3046907101 learned_addr learned my addr 172.21.15.50:0/3046907101 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f6568d700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 0x7f6f600ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f64e8c700 1 -- 172.21.15.50:0/3046907101 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 msgr2=0x7f6f600ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f64e8c700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 0x7f6f600ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f64e8c700 1 -- 172.21.15.50:0/3046907101 >> v1:172.21.15.50:6789/0 conn(0x7f6f600aef60 legacy=0x7f6f600b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5f7fe700 1 -- 172.21.15.50:0/3046907101 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314110076 0 0) 0x7f6f6006beb0 con 0x7f6f600aef60 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f64e8c700 1 -- 172.21.15.50:0/3046907101 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6006d140 con 0x7f6f601569e0 2023-12-19T14:58:02.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f6568d700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 0x7f6f600ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:02.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f64e8c700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f601569e0 0x7f6f60156db0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098e0 tx=0x7f6f540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5f1907334e8dff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:02.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5f7fe700 1 -- 172.21.15.50:0/3046907101 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f54009e80 con 0x7f6f601569e0 2023-12-19T14:58:02.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5f7fe700 1 -- 172.21.15.50:0/3046907101 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5400b860 con 0x7f6f601569e0 2023-12-19T14:58:02.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f5f7fe700 1 -- 172.21.15.50:0/3046907101 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5400bbe0 con 0x7f6f601569e0 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 -- 172.21.15.50:0/3046907101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f601569e0 msgr2=0x7f6f60156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f601569e0 0x7f6f60156db0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098e0 tx=0x7f6f540091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 -- 172.21.15.50:0/3046907101 shutdown_connections 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f600adf40 0x7f6f600ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/3046907101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f601569e0 0x7f6f60156db0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 -- 172.21.15.50:0/3046907101 >> 172.21.15.50:0/3046907101 conn(0x7f6f600c8c00 msgr2=0x7f6f600c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:02.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 -- 172.21.15.50:0/3046907101 shutdown_connections 2023-12-19T14:58:02.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.649+0000 7f6f65e8e700 1 -- 172.21.15.50:0/3046907101 wait complete. 2023-12-19T14:58:02.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.653+0000 7f6f65e8e700 1 Processor -- start 2023-12-19T14:58:02.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.653+0000 7f6f65e8e700 1 -- start start 2023-12-19T14:58:02.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f600adf40 0x7f6f60154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f601569e0 0x7f6f601584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600c4090 con 0x7f6f600aef60 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600c31a0 con 0x7f6f600adf40 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600c3480 con 0x7f6f601569e0 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37490/0 (socket says 172.21.15.50:37490) 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 -- 172.21.15.50:0/463443537 learned_addr learned my addr 172.21.15.50:0/463443537 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f64e8c700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f600adf40 0x7f6f60154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f6568d700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f601569e0 0x7f6f601584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 -- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f601569e0 msgr2=0x7f6f601584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f601569e0 0x7f6f601584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 -- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f600adf40 msgr2=0x7f6f60154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f600adf40 0x7f6f60154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 -- 172.21.15.50:0/463443537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54009420 con 0x7f6f600aef60 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5ffff700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c7e0 tx=0x7f6f4c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c005b30 con 0x7f6f600aef60 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c005c90 con 0x7f6f600aef60 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f600c3760 con 0x7f6f600aef60 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c007300 con 0x7f6f600aef60 2023-12-19T14:58:02.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.657+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f600c3d20 con 0x7f6f600aef60 2023-12-19T14:58:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.661+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f600c4240 con 0x7f6f600aef60 2023-12-19T14:58:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.661+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 54) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f6f4c01b3b0 con 0x7f6f600aef60 2023-12-19T14:58:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.665+0000 7f6f5cff9700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 0x7f6f44062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.665+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f4c07c4b0 con 0x7f6f600aef60 2023-12-19T14:58:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.665+0000 7f6f64e8c700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 0x7f6f44062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:02.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.665+0000 7f6f64e8c700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 0x7f6f44062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6f540091e0 tx=0x7f6f54017000 comp rx=0 tx=0).ready entity=mgr.7942 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:02.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.669+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f600c4240 con 0x7f6f600aef60 2023-12-19T14:58:02.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.865+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f6004be40 con 0x7f6f440607e0 2023-12-19T14:58:02.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:02.997+0000 7f6f5cff9700 1 -- 172.21.15.50:0/463443537 <== mgr.7942 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6f6004be40 con 0x7f6f440607e0 2023-12-19T14:58:02.999 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path snap_test_dir1 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 msgr2=0x7f6f44062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 0x7f6f44062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6f540091e0 tx=0x7f6f54017000 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 msgr2=0x7f6f60157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c7e0 tx=0x7f6f4c007df0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 shutdown_connections 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f440607e0 0x7f6f44062c90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f601569e0 0x7f6f601584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f600aef60 0x7f6f60157fa0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 --2- 172.21.15.50:0/463443537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f600adf40 0x7f6f60154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 >> 172.21.15.50:0/463443537 conn(0x7f6f600c8c00 msgr2=0x7f6f60155730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:03.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 shutdown_connections 2023-12-19T14:58:03.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.001+0000 7f6f65e8e700 1 -- 172.21.15.50:0/463443537 wait complete. 2023-12-19T14:58:03.029 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T14:58:03.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:03.029 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T14:58:03.060 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-14_58_00_UTC 2023-12-19T14:58:03.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-14_58_00_UTC 2023-12-19T14:58:03.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:03.062 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-14_58_00_UTC) 2023-12-19T14:58:03.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:03.147 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T14:58:03.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T14:58:03.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.429+0000 7f09116f0700 1 Processor -- start 2023-12-19T14:58:03.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.429+0000 7f09116f0700 1 -- start start 2023-12-19T14:58:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f09116f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cb970 0x7f090c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f09116f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f090c0cecd0 0x7f090c06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f09116f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c0cbdf0 con 0x7f090c158e20 2023-12-19T14:58:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f09116f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090c0cc0d0 con 0x7f090c0cecd0 2023-12-19T14:58:03.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f09116f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0cc3b0 con 0x7f090c0cb970 2023-12-19T14:58:03.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090affd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f090c158e20 0x7f090c0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51834/0 (socket says 172.21.15.50:51834) 2023-12-19T14:58:03.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090affd700 1 -- 172.21.15.50:0/2038479255 learned_addr learned my addr 172.21.15.50:0/2038479255 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:03.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090b7fe700 1 --2- 172.21.15.50:0/2038479255 >> v2:172.21.15.125:3300/0 conn(0x7f090c0cecd0 0x7f090c06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f0903fff700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cb970 0x7f090c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090a7fc700 1 -- 172.21.15.50:0/2038479255 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152971119 0 0) 0x7f090c0cbdf0 con 0x7f090c158e20 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090a7fc700 1 -- 172.21.15.50:0/2038479255 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8003570 con 0x7f090c158e20 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090b7fe700 1 -- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cb970 msgr2=0x7f090c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090b7fe700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cb970 0x7f090c06acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090b7fe700 1 -- 172.21.15.50:0/2038479255 >> v1:172.21.15.50:6789/0 conn(0x7f090c158e20 legacy=0x7f090c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.433+0000 7f090b7fe700 1 -- 172.21.15.50:0/2038479255 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c0cc690 con 0x7f090c0cecd0 2023-12-19T14:58:03.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f090b7fe700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c0cecd0 0x7f090c06b200 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0904009e60 tx=0x7f0904009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ad6742dbf71def3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:03.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f090a7fc700 1 -- 172.21.15.50:0/2038479255 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400e070 con 0x7f090c0cecd0 2023-12-19T14:58:03.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f090a7fc700 1 -- 172.21.15.50:0/2038479255 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090400bab0 con 0x7f090c0cecd0 2023-12-19T14:58:03.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f090a7fc700 1 -- 172.21.15.50:0/2038479255 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400be70 con 0x7f090c0cecd0 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 -- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c0cecd0 msgr2=0x7f090c06b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c0cecd0 0x7f090c06b200 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0904009e60 tx=0x7f0904009780 comp rx=0 tx=0).stop 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 -- 172.21.15.50:0/2038479255 shutdown_connections 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c0cecd0 0x7f090c06b200 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 --2- 172.21.15.50:0/2038479255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cb970 0x7f090c06acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 -- 172.21.15.50:0/2038479255 >> 172.21.15.50:0/2038479255 conn(0x7f090c067bb0 msgr2=0x7f090c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 -- 172.21.15.50:0/2038479255 shutdown_connections 2023-12-19T14:58:03.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 -- 172.21.15.50:0/2038479255 wait complete. 2023-12-19T14:58:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.437+0000 7f09116f0700 1 Processor -- start 2023-12-19T14:58:03.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 -- start start 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cecd0 0x7f090c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c15ba30 0x7f090c0b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0c7c20 con 0x7f090c158e20 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c1502c0 con 0x7f090c15ba30 2023-12-19T14:58:03.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0b4fe0 con 0x7f090c0cecd0 2023-12-19T14:58:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37500/0 (socket says 172.21.15.50:37500) 2023-12-19T14:58:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 -- 172.21.15.50:0/1993050521 learned_addr learned my addr 172.21.15.50:0/1993050521 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f090affd700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cecd0 0x7f090c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f090b7fe700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c15ba30 0x7f090c0b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 -- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cecd0 msgr2=0x7f090c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cecd0 0x7f090c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 -- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c15ba30 msgr2=0x7f090c0b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c15ba30 0x7f090c0b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 -- 172.21.15.50:0/1993050521 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0904009a20 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0903fff700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f08f400ec80 tx=0x7f08f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f400c9d0 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f400cb30 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c0b52c0 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.441+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4013470 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.445+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c0b5880 con 0x7f090c158e20 2023-12-19T14:58:03.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.445+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c0c7dd0 con 0x7f090c158e20 2023-12-19T14:58:03.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.445+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 54) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f08f4013820 con 0x7f090c158e20 2023-12-19T14:58:03.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.449+0000 7f0902ffd700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 0x7f08ec062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:03.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.449+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f08f407d1d0 con 0x7f090c158e20 2023-12-19T14:58:03.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.449+0000 7f090affd700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 0x7f08ec062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:03.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.449+0000 7f090affd700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 0x7f08ec062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0058d0 tx=0x7f08fc009040 comp rx=0 tx=0).ready entity=mgr.7942 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:03.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.453+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f090c0c7dd0 con 0x7f090c158e20 2023-12-19T14:58:03.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:03.669+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f090c1574c0 con 0x7f090c158e20 2023-12-19T14:58:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.157+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f08f40560d0 con 0x7f090c158e20 2023-12-19T14:58:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.161+0000 7f0902ffd700 1 -- 172.21.15.50:0/1993050521 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v55) v1 ==== 93+0+0 (secure 0 0 0) 0x7f08f4050ea0 con 0x7f090c158e20 2023-12-19T14:58:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 msgr2=0x7f08ec062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 0x7f08ec062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0058d0 tx=0x7f08fc009040 comp rx=0 tx=0).stop 2023-12-19T14:58:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 msgr2=0x7f090c15b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f08f400ec80 tx=0x7f08f40091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 shutdown_connections 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f090c15ba30 0x7f090c0b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f08ec0607e0 0x7f08ec062c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f090c158e20 0x7f090c15b4f0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 --2- 172.21.15.50:0/1993050521 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f090c0cecd0 0x7f090c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 >> 172.21.15.50:0/1993050521 conn(0x7f090c067bb0 msgr2=0x7f090c0ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 shutdown_connections 2023-12-19T14:58:04.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.165+0000 7f09116f0700 1 -- 172.21.15.50:0/1993050521 wait complete. 2023-12-19T14:58:04.190 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:58:04.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:58:04.233 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:58:04.278 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:58:04.278 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T14:58:04.278 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T14:58:04.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:04.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[52755]: fuse finished with error 0 and tester_r 0 2023-12-19T14:58:04.318 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:58:04.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:04.319 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T14:58:04.356 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:04.350+0000 7f106c1c5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:58:04.358 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:04.353+0000 7fd5a0b86000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:58:04.504 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:58:04.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:04.504 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T14:58:04.523 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T14:58:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:58:04.524 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T14:58:04.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:04.524 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T14:58:04.567 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T14:58:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:58:04.568 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T14:58:04.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:04.569 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T14:58:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T14:58:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T14:58:04.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 Processor -- start 2023-12-19T14:58:04.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 -- start start 2023-12-19T14:58:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a7406a580 0x7f9a7406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9a7406d210 0x7f9a7406d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7406c7c0 con 0x7f9a740b1410 2023-12-19T14:58:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406af40 con 0x7f9a7406d210 2023-12-19T14:58:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a7924c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7406cd80 con 0x7f9a7406a580 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a78a4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9a7406d210 0x7f9a7406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a73fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9a740b1410 0x7f9a740ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51856/0 (socket says 172.21.15.50:51856) 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a73fff700 1 -- 172.21.15.50:0/1179308029 learned_addr learned my addr 172.21.15.50:0/1179308029 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a6bfff700 1 --2- 172.21.15.50:0/1179308029 >> v2:172.21.15.125:3300/0 conn(0x7f9a7406a580 0x7f9a7406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a78a4b700 1 -- 172.21.15.50:0/1179308029 >> v2:172.21.15.125:3300/0 conn(0x7f9a7406a580 msgr2=0x7f9a7406aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a78a4b700 1 --2- 172.21.15.50:0/1179308029 >> v2:172.21.15.125:3300/0 conn(0x7f9a7406a580 0x7f9a7406aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a78a4b700 1 -- 172.21.15.50:0/1179308029 >> v1:172.21.15.50:6789/0 conn(0x7f9a740b1410 legacy=0x7f9a740ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a78a4b700 1 -- 172.21.15.50:0/1179308029 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7406b8f0 con 0x7f9a7406d210 2023-12-19T14:58:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.773+0000 7f9a737fe700 1 -- 172.21.15.50:0/1179308029 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068578122 0 0) 0x7f9a7406c7c0 con 0x7f9a740b1410 2023-12-19T14:58:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a78a4b700 1 --2- 172.21.15.50:0/1179308029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a7406d210 0x7f9a7406d620 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9a74068040 tx=0x7f9a6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c0ce893d6d6fd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a737fe700 1 -- 172.21.15.50:0/1179308029 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00e070 con 0x7f9a7406d210 2023-12-19T14:58:04.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a737fe700 1 -- 172.21.15.50:0/1179308029 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00bb60 con 0x7f9a7406d210 2023-12-19T14:58:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a737fe700 1 -- 172.21.15.50:0/1179308029 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c012430 con 0x7f9a7406d210 2023-12-19T14:58:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 -- 172.21.15.50:0/1179308029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a7406d210 msgr2=0x7f9a7406d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 --2- 172.21.15.50:0/1179308029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a7406d210 0x7f9a7406d620 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f9a74068040 tx=0x7f9a6c009780 comp rx=0 tx=0).stop 2023-12-19T14:58:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 -- 172.21.15.50:0/1179308029 shutdown_connections 2023-12-19T14:58:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 --2- 172.21.15.50:0/1179308029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a7406d210 0x7f9a7406d620 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 --2- 172.21.15.50:0/1179308029 >> v2:172.21.15.125:3300/0 conn(0x7f9a7406a580 0x7f9a7406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 -- 172.21.15.50:0/1179308029 >> 172.21.15.50:0/1179308029 conn(0x7f9a74067bb0 msgr2=0x7f9a740cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 -- 172.21.15.50:0/1179308029 shutdown_connections 2023-12-19T14:58:04.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.777+0000 7f9a7924c700 1 -- 172.21.15.50:0/1179308029 wait complete. 2023-12-19T14:58:04.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 Processor -- start 2023-12-19T14:58:04.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 -- start start 2023-12-19T14:58:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9a7406a580 0x7f9a740a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7406d210 0x7f9a740c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740c6f30 con 0x7f9a740b1410 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740c6320 con 0x7f9a7406d210 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a7924c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740c6600 con 0x7f9a7406a580 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37536/0 (socket says 172.21.15.50:37536) 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 -- 172.21.15.50:0/3620877892 learned_addr learned my addr 172.21.15.50:0/3620877892 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a6bfff700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7406d210 0x7f9a740c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a73fff700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9a7406a580 0x7f9a740a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9a7406a580 msgr2=0x7f9a740a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9a7406a580 0x7f9a740a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7406d210 msgr2=0x7f9a740c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:04.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7406d210 0x7f9a740c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:04.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.781+0000 7f9a78a4b700 1 -- 172.21.15.50:0/3620877892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c009a20 con 0x7f9a740b1410 2023-12-19T14:58:04.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a78a4b700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0097c0 tx=0x7f9a6c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:04.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c027470 con 0x7f9a740b1410 2023-12-19T14:58:04.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c027610 con 0x7f9a740b1410 2023-12-19T14:58:04.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0279d0 con 0x7f9a740b1410 2023-12-19T14:58:04.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a740c68e0 con 0x7f9a740b1410 2023-12-19T14:58:04.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740c7900 con 0x7f9a740b1410 2023-12-19T14:58:04.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.785+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a740c70e0 con 0x7f9a740b1410 2023-12-19T14:58:04.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.789+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f9a6c024070 con 0x7f9a740b1410 2023-12-19T14:58:04.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.789+0000 7f9a70ff9700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 0x7f9a54062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:04.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.789+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9a6c07f470 con 0x7f9a740b1410 2023-12-19T14:58:04.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.789+0000 7f9a73fff700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 msgr2=0x7f9a54062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:04.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.789+0000 7f9a73fff700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 0x7f9a54062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.797+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9a740c70e0 con 0x7f9a740b1410 2023-12-19T14:58:04.905 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:04.898+0000 7f106c1c5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:58:04.922 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:04.917+0000 7fd5a0b86000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:58:04.979 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:04.974+0000 7f106c1c5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:58:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.993+0000 7f9a73fff700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 msgr2=0x7f9a54062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:04.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:04.993+0000 7f9a73fff700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 0x7f9a54062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:04.996 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:04.993+0000 7fd5a0b86000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:58:05.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.001+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7404be40 con 0x7f9a740b1410 2023-12-19T14:58:05.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.005+0000 7f9a70ff9700 1 -- 172.21.15.50:0/3620877892 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f9a6c04f520 con 0x7f9a740b1410 2023-12-19T14:58:05.008 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:05.008 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:58:05.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 msgr2=0x7f9a54062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 0x7f9a54062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 msgr2=0x7f9a740c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0097c0 tx=0x7f9a6c019000 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 shutdown_connections 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a7406d210 0x7f9a740c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9a54060790 0x7f9a54062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a740b1410 0x7f9a740c9fb0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 --2- 172.21.15.50:0/3620877892 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9a7406a580 0x7f9a740a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 >> 172.21.15.50:0/3620877892 conn(0x7f9a74067bb0 msgr2=0x7f9a740adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 shutdown_connections 2023-12-19T14:58:05.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.009+0000 7f9a7924c700 1 -- 172.21.15.50:0/3620877892 wait complete. 2023-12-19T14:58:05.014 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:58:05.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:05.128 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.122+0000 7f106c1c5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:05.153 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:05.149+0000 7fd5a0b86000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:05.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.201+0000 7f1cdb345700 1 Processor -- start 2023-12-19T14:58:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.201+0000 7f1cdb345700 1 -- start start 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cdb345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cd41502b0 0x7f1cd4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cdb345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 0x7f1cd40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cdb345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40c6630 con 0x7f1cd4067c90 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cdb345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40c6910 con 0x7f1cd41502b0 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cdb345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c6bf0 con 0x7f1cd40c5900 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd90e1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1cd4067c90 0x7f1cd414fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51868/0 (socket says 172.21.15.50:51868) 2023-12-19T14:58:05.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd90e1700 1 -- 172.21.15.50:0/387485552 learned_addr learned my addr 172.21.15.50:0/387485552 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/387485552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 0x7f1cd40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd88e0700 1 --2- 172.21.15.50:0/387485552 >> v2:172.21.15.125:3300/0 conn(0x7f1cd41502b0 0x7f1cd4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1ccbfff700 1 -- 172.21.15.50:0/387485552 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172228302 0 0) 0x7f1cd40c6630 con 0x7f1cd4067c90 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1ccbfff700 1 -- 172.21.15.50:0/387485552 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc0003570 con 0x7f1cd4067c90 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd98e2700 1 -- 172.21.15.50:0/387485552 >> v2:172.21.15.125:3300/0 conn(0x7f1cd41502b0 msgr2=0x7f1cd4068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/387485552 >> v2:172.21.15.125:3300/0 conn(0x7f1cd41502b0 0x7f1cd4068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd98e2700 1 -- 172.21.15.50:0/387485552 >> v1:172.21.15.50:6789/0 conn(0x7f1cd4067c90 legacy=0x7f1cd414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.205+0000 7f1cd98e2700 1 -- 172.21.15.50:0/387485552 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40c6ed0 con 0x7f1cd40c5900 2023-12-19T14:58:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/387485552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 0x7f1cd40c5d60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009e60 tx=0x7f1ccc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdea71ba7e0b5e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1ccbfff700 1 -- 172.21.15.50:0/387485552 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00e070 con 0x7f1cd40c5900 2023-12-19T14:58:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1ccbfff700 1 -- 172.21.15.50:0/387485552 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc00bab0 con 0x7f1cd40c5900 2023-12-19T14:58:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1ccbfff700 1 -- 172.21.15.50:0/387485552 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00be30 con 0x7f1cd40c5900 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 -- 172.21.15.50:0/387485552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 msgr2=0x7f1cd40c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 --2- 172.21.15.50:0/387485552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 0x7f1cd40c5d60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009e60 tx=0x7f1ccc009780 comp rx=0 tx=0).stop 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 -- 172.21.15.50:0/387485552 shutdown_connections 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 --2- 172.21.15.50:0/387485552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd40c5900 0x7f1cd40c5d60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 --2- 172.21.15.50:0/387485552 >> v2:172.21.15.125:3300/0 conn(0x7f1cd41502b0 0x7f1cd4068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 -- 172.21.15.50:0/387485552 >> 172.21.15.50:0/387485552 conn(0x7f1cd40cd6d0 msgr2=0x7f1cd406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 -- 172.21.15.50:0/387485552 shutdown_connections 2023-12-19T14:58:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.209+0000 7f1cdb345700 1 -- 172.21.15.50:0/387485552 wait complete. 2023-12-19T14:58:05.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 Processor -- start 2023-12-19T14:58:05.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 -- start start 2023-12-19T14:58:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd4067c90 0x7f1cd4169ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 0x7f1cd40cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd414c820 con 0x7f1cd41502b0 2023-12-19T14:58:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c8630 con 0x7f1cd40c5900 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cdb345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd416a0e0 con 0x7f1cd4067c90 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd90e1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd4067c90 0x7f1cd4169ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37548/0 (socket says 172.21.15.50:37548) 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 -- 172.21.15.50:0/3836277234 learned_addr learned my addr 172.21.15.50:0/3836277234 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd88e0700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 0x7f1cd40cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 -- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd4067c90 msgr2=0x7f1cd4169ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd4067c90 0x7f1cd4169ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 -- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 msgr2=0x7f1cd40cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 0x7f1cd40cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd98e2700 1 -- 172.21.15.50:0/3836277234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc009a20 con 0x7f1cd41502b0 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.213+0000 7f1cd88e0700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 0x7f1cd40cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:05.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cd98e2700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40c6d30 tx=0x7f1ccc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc027470 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc027610 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc0279d0 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40b3520 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd40b3ae0 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4071120 con 0x7f1cd41502b0 2023-12-19T14:58:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.217+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f1ccc024070 con 0x7f1cd41502b0 2023-12-19T14:58:05.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.221+0000 7f1cc97fa700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 0x7f1cb8062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.221+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1ccc07f730 con 0x7f1cd41502b0 2023-12-19T14:58:05.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.221+0000 7f1cd90e1700 1 -- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 msgr2=0x7f1cb8062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:05.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.221+0000 7f1cd90e1700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 0x7f1cb8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:05.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.225+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1ccc04f740 con 0x7f1cd41502b0 2023-12-19T14:58:05.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.393+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cd40642a0 con 0x7f1cd41502b0 2023-12-19T14:58:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.397+0000 7f1cc97fa700 1 -- 172.21.15.50:0/3836277234 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16111 (secure 0 0 0) 0x7f1ccc052d60 con 0x7f1cd41502b0 2023-12-19T14:58:05.400 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:05.400 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":130,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:27.688637+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/2353401000":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2989287076":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1628872297":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2571945050":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/4156266535":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6833/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6832/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1370626584":"2023-12-20T14:57:27.688521+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 msgr2=0x7f1cb8062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:05.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 0x7f1cb8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 msgr2=0x7f1cd414b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40c6d30 tx=0x7f1ccc019000 comp rx=0 tx=0).stop 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 shutdown_connections 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cd40c5900 0x7f1cd40cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb8060790 0x7f1cb8062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cd41502b0 0x7f1cd414b3f0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 --2- 172.21.15.50:0/3836277234 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cd4067c90 0x7f1cd4169ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 >> 172.21.15.50:0/3836277234 conn(0x7f1cd40cd6d0 msgr2=0x7f1cd414f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 shutdown_connections 2023-12-19T14:58:05.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.401+0000 7f1cdb345700 1 -- 172.21.15.50:0/3836277234 wait complete. 2023-12-19T14:58:05.410 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.402+0000 7f106c1c5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:58:05.418 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:05.413+0000 7fd5a0b86000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:58:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:57:10.809944+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:57:11.656478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T14:58:05.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:58:05.483 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.478+0000 7f106c1c5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:05.492 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:05.489+0000 7fd5a0b86000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:05.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.593+0000 7f17a2d23700 1 Processor -- start 2023-12-19T14:58:05.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.593+0000 7f17a2d23700 1 -- start start 2023-12-19T14:58:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2d23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f179c06aeb0 0x7f179c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2d23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c06c850 con 0x7f179c158c90 2023-12-19T14:58:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2d23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179c0cb650 con 0x7f179c06aeb0 2023-12-19T14:58:05.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c06b470 con 0x7f179c1595a0 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f179c06aeb0 0x7f179c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f179c06aeb0 0x7f179c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37554/0 (socket says 172.21.15.50:37554) 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 -- 172.21.15.50:0/2796950209 learned_addr learned my addr 172.21.15.50:0/2796950209 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a2522700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 -- 172.21.15.50:0/2796950209 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 msgr2=0x7f179c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 -- 172.21.15.50:0/2796950209 >> v1:172.21.15.125:6789/0 conn(0x7f179c158c90 legacy=0x7f179c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 -- 172.21.15.50:0/2796950209 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c06b750 con 0x7f179c06aeb0 2023-12-19T14:58:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a1520700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c159060 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f178c0099b0 tx=0x7f178c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2ab9157512e4ae server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a0d1f700 1 -- 172.21.15.50:0/2796950209 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00b860 con 0x7f179c06aeb0 2023-12-19T14:58:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a0d1f700 1 -- 172.21.15.50:0/2796950209 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c00b9c0 con 0x7f179c06aeb0 2023-12-19T14:58:05.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.597+0000 7f17a0d1f700 1 -- 172.21.15.50:0/2796950209 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00bd60 con 0x7f179c06aeb0 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 -- 172.21.15.50:0/2796950209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 msgr2=0x7f179c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c159060 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f178c0099b0 tx=0x7f178c009250 comp rx=0 tx=0).stop 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 -- 172.21.15.50:0/2796950209 shutdown_connections 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 --2- 172.21.15.50:0/2796950209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c159060 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 -- 172.21.15.50:0/2796950209 >> 172.21.15.50:0/2796950209 conn(0x7f179c067bb0 msgr2=0x7f179c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 -- 172.21.15.50:0/2796950209 shutdown_connections 2023-12-19T14:58:05.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 -- 172.21.15.50:0/2796950209 wait complete. 2023-12-19T14:58:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.601+0000 7f17a2d23700 1 Processor -- start 2023-12-19T14:58:05.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 -- start start 2023-12-19T14:58:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f179c158c90 0x7f179c0a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c16c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0c7bd0 con 0x7f179c06aeb0 2023-12-19T14:58:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0c7e20 con 0x7f179c158c90 2023-12-19T14:58:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0c8100 con 0x7f179c1595a0 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37566/0 (socket says 172.21.15.50:37566) 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 -- 172.21.15.50:0/1504627873 learned_addr learned my addr 172.21.15.50:0/1504627873 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2522700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c16c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1520700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f179c158c90 0x7f179c0a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 msgr2=0x7f179c16c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c16c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f179c158c90 msgr2=0x7f179c0a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f179c158c90 0x7f179c0a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 -- 172.21.15.50:0/1504627873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0094f0 con 0x7f179c06aeb0 2023-12-19T14:58:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a1d21700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f179800c590 tx=0x7f179800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798011590 con 0x7f179c06aeb0 2023-12-19T14:58:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1798011710 con 0x7f179c06aeb0 2023-12-19T14:58:05.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c0c83e0 con 0x7f179c06aeb0 2023-12-19T14:58:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.605+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798011ad0 con 0x7f179c06aeb0 2023-12-19T14:58:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c1492e0 con 0x7f179c06aeb0 2023-12-19T14:58:05.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179c062ee0 con 0x7f179c06aeb0 2023-12-19T14:58:05.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f179801b3b0 con 0x7f179c06aeb0 2023-12-19T14:58:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f1791ffb700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 0x7f1780062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f179807c2f0 con 0x7f179c06aeb0 2023-12-19T14:58:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f17a1520700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 msgr2=0x7f1780062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:05.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.609+0000 7f17a1520700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 0x7f1780062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:05.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.617+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f179c062ee0 con 0x7f179c06aeb0 2023-12-19T14:58:05.709 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.702+0000 7f106c1c5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:58:05.752 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:05.749+0000 7fd5a0b86000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:58:05.798 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.790+0000 7f106c1c5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:58:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.813+0000 7f17a1520700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 msgr2=0x7f1780062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:05.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.813+0000 7f17a1520700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 0x7f1780062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:05.889 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:05.885+0000 7fd5a0b86000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:58:05.891 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:05.886+0000 7f106c1c5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:58:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.945+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f179c062ee0 con 0x7f179c06aeb0 2023-12-19T14:58:05.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.949+0000 7f1791ffb700 1 -- 172.21.15.50:0/1504627873 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1798017020 con 0x7f179c06aeb0 2023-12-19T14:58:05.951 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:58:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 msgr2=0x7f1780062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:05.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 0x7f1780062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 msgr2=0x7f179c0a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f179800c590 tx=0x7f179800c8a0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 shutdown_connections 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f179c158c90 0x7f179c0a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1780060790 0x7f1780062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f179c1595a0 0x7f179c16c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 --2- 172.21.15.50:0/1504627873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f179c06aeb0 0x7f179c0a8850 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 >> 172.21.15.50:0/1504627873 conn(0x7f179c067bb0 msgr2=0x7f179c0afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 shutdown_connections 2023-12-19T14:58:05.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:05.953+0000 7f17a2d23700 1 -- 172.21.15.50:0/1504627873 wait complete. 2023-12-19T14:58:05.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T14:58:06.019 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:06.017+0000 7fd5a0b86000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:58:06.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.141+0000 7fc518185700 1 Processor -- start 2023-12-19T14:58:06.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.141+0000 7fc518185700 1 -- start start 2023-12-19T14:58:06.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc518185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc5100cecd0 0x7fc51006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc518185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100cbdf0 con 0x7fc510158e20 2023-12-19T14:58:06.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc518185700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100cc0d0 con 0x7fc5100cb970 2023-12-19T14:58:06.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc518185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5100cc3b0 con 0x7fc5100cecd0 2023-12-19T14:58:06.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc5100cecd0 0x7fc51006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc5100cecd0 0x7fc51006b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37578/0 (socket says 172.21.15.50:37578) 2023-12-19T14:58:06.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 -- 172.21.15.50:0/2628507685 learned_addr learned my addr 172.21.15.50:0/2628507685 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:06.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc515f21700 1 --1- 172.21.15.50:0/2628507685 >> v1:172.21.15.50:6789/0 conn(0x7fc510158e20 0x7fc5100ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51878/0 (socket says 172.21.15.50:51878) 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 -- 172.21.15.50:0/2628507685 >> v1:172.21.15.125:6789/0 conn(0x7fc5100cb970 legacy=0x7fc51006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 -- 172.21.15.50:0/2628507685 >> v1:172.21.15.50:6789/0 conn(0x7fc510158e20 legacy=0x7fc5100ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 -- 172.21.15.50:0/2628507685 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100cc690 con 0x7fc5100cecd0 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.145+0000 7fc516722700 1 --2- 172.21.15.50:0/2628507685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5100cecd0 0x7fc51006b3d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc508009f50 tx=0x7fc508009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2c3486adaac133 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc514f1f700 1 -- 172.21.15.50:0/2628507685 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50800e070 con 0x7fc5100cecd0 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc514f1f700 1 -- 172.21.15.50:0/2628507685 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800ba70 con 0x7fc5100cecd0 2023-12-19T14:58:06.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc514f1f700 1 -- 172.21.15.50:0/2628507685 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5080092e0 con 0x7fc5100cecd0 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- 172.21.15.50:0/2628507685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5100cecd0 msgr2=0x7fc51006b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 --2- 172.21.15.50:0/2628507685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5100cecd0 0x7fc51006b3d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc508009f50 tx=0x7fc508009840 comp rx=0 tx=0).stop 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- 172.21.15.50:0/2628507685 shutdown_connections 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 --2- 172.21.15.50:0/2628507685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5100cecd0 0x7fc51006b3d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- 172.21.15.50:0/2628507685 >> 172.21.15.50:0/2628507685 conn(0x7fc510067bb0 msgr2=0x7fc5100d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- 172.21.15.50:0/2628507685 shutdown_connections 2023-12-19T14:58:06.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- 172.21.15.50:0/2628507685 wait complete. 2023-12-19T14:58:06.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 Processor -- start 2023-12-19T14:58:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.149+0000 7fc518185700 1 -- start start 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5100cecd0 0x7fc510153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 0x7fc51014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510154f10 con 0x7fc5101549a0 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101551f0 con 0x7fc5100cecd0 2023-12-19T14:58:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101554d0 con 0x7fc510158e20 2023-12-19T14:58:06.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37586/0 (socket says 172.21.15.50:37586) 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 -- 172.21.15.50:0/3736968611 learned_addr learned my addr 172.21.15.50:0/3736968611 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc515f21700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5100cecd0 0x7fc510153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc515720700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 0x7fc51014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 -- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 msgr2=0x7fc51014db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 0x7fc51014db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 -- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5100cecd0 msgr2=0x7fc510153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5100cecd0 0x7fc510153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 -- 172.21.15.50:0/3736968611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc508009ae0 con 0x7fc5101549a0 2023-12-19T14:58:06.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc515720700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 0x7fc51014db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:06.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc516722700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc508012000 tx=0x7fc508018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508008d90 con 0x7fc5101549a0 2023-12-19T14:58:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc508026460 con 0x7fc5101549a0 2023-12-19T14:58:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5101557b0 con 0x7fc5101549a0 2023-12-19T14:58:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508026820 con 0x7fc5101549a0 2023-12-19T14:58:06.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.153+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51014fbe0 con 0x7fc5101549a0 2023-12-19T14:58:06.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.157+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510071120 con 0x7fc5101549a0 2023-12-19T14:58:06.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.157+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fc508003040 con 0x7fc5101549a0 2023-12-19T14:58:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.161+0000 7fc505ffb700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 0x7fc4f4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.161+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc50807efc0 con 0x7fc5101549a0 2023-12-19T14:58:06.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.161+0000 7fc515f21700 1 -- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 msgr2=0x7fc4f4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:06.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.161+0000 7fc515f21700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 0x7fc4f4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.169+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc50804eec0 con 0x7fc5101549a0 2023-12-19T14:58:06.305 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:06.298+0000 7f106c1c5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:58:06.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.341+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc51006bb00 con 0x7fc5101549a0 2023-12-19T14:58:06.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.341+0000 7fc505ffb700 1 -- 172.21.15.50:0/3736968611 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc5080524e0 con 0x7fc5101549a0 2023-12-19T14:58:06.346 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T14:58:06.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 msgr2=0x7fc4f4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:06.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 0x7fc4f4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 msgr2=0x7fc51014f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc508012000 tx=0x7fc508018000 comp rx=0 tx=0).stop 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 shutdown_connections 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4f4060790 0x7fc4f4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5101549a0 0x7fc51014f6a0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc510158e20 0x7fc51014db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 --2- 172.21.15.50:0/3736968611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc5100cecd0 0x7fc510153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 >> 172.21.15.50:0/3736968611 conn(0x7fc510067bb0 msgr2=0x7fc51006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 shutdown_connections 2023-12-19T14:58:06.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.349+0000 7fc518185700 1 -- 172.21.15.50:0/3736968611 wait complete. 2023-12-19T14:58:06.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T14:58:06.454 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:06.449+0000 7fd5a0b86000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:58:06.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:06.470+0000 7f106c1c5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:58:06.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.581+0000 7fa791a0c700 1 Processor -- start 2023-12-19T14:58:06.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.581+0000 7fa791a0c700 1 -- start start 2023-12-19T14:58:06.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa791a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c158c90 0x7fa78c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa791a0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa78c06aeb0 0x7fa78c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa791a0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c06c850 con 0x7fa78c1595a0 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa791a0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78c06b470 con 0x7fa78c06aeb0 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa791a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c06b750 con 0x7fa78c158c90 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa79120b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa78c1595a0 0x7fa78c15c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42870/0 (socket says 172.21.15.50:42870) 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa79120b700 1 -- 172.21.15.50:0/3583184396 learned_addr learned my addr 172.21.15.50:0/3583184396 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78bfff700 1 --2- 172.21.15.50:0/3583184396 >> v2:172.21.15.125:3300/0 conn(0x7fa78c06aeb0 0x7fa78c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa790a0a700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c158c90 0x7fa78c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78b7fe700 1 -- 172.21.15.50:0/3583184396 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778961597 0 0) 0x7fa78c06c850 con 0x7fa78c1595a0 2023-12-19T14:58:06.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78b7fe700 1 -- 172.21.15.50:0/3583184396 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c003570 con 0x7fa78c1595a0 2023-12-19T14:58:06.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78bfff700 1 -- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c158c90 msgr2=0x7fa78c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:06.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78bfff700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c158c90 0x7fa78c0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78bfff700 1 -- 172.21.15.50:0/3583184396 >> v1:172.21.15.50:6789/0 conn(0x7fa78c1595a0 legacy=0x7fa78c15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.585+0000 7fa78bfff700 1 -- 172.21.15.50:0/3583184396 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c06ba30 con 0x7fa78c06aeb0 2023-12-19T14:58:06.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa78bfff700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c06aeb0 0x7fa78c159060 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa7780098e0 tx=0x7fa7780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8f492f11de6d938 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:06.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa78b7fe700 1 -- 172.21.15.50:0/3583184396 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778011630 con 0x7fa78c06aeb0 2023-12-19T14:58:06.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa78b7fe700 1 -- 172.21.15.50:0/3583184396 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778011790 con 0x7fa78c06aeb0 2023-12-19T14:58:06.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa78b7fe700 1 -- 172.21.15.50:0/3583184396 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778011b30 con 0x7fa78c06aeb0 2023-12-19T14:58:06.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 -- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c06aeb0 msgr2=0x7fa78c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c06aeb0 0x7fa78c159060 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa7780098e0 tx=0x7fa7780091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 -- 172.21.15.50:0/3583184396 shutdown_connections 2023-12-19T14:58:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c06aeb0 0x7fa78c159060 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 --2- 172.21.15.50:0/3583184396 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c158c90 0x7fa78c0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 -- 172.21.15.50:0/3583184396 >> 172.21.15.50:0/3583184396 conn(0x7fa78c067bb0 msgr2=0x7fa78c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 -- 172.21.15.50:0/3583184396 shutdown_connections 2023-12-19T14:58:06.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.589+0000 7fa791a0c700 1 -- 172.21.15.50:0/3583184396 wait complete. 2023-12-19T14:58:06.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 Processor -- start 2023-12-19T14:58:06.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 -- start start 2023-12-19T14:58:06.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c06aeb0 0x7fa78c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c1595a0 0x7fa78c153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c155920 con 0x7fa78c158c90 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c155bf0 con 0x7fa78c1595a0 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa791a0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c155ed0 con 0x7fa78c06aeb0 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33058/0 (socket says 172.21.15.50:33058) 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 -- 172.21.15.50:0/2289091975 learned_addr learned my addr 172.21.15.50:0/2289091975 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa790a0a700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c06aeb0 0x7fa78c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa79120b700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c1595a0 0x7fa78c153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 -- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c06aeb0 msgr2=0x7fa78c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c06aeb0 0x7fa78c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 -- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c1595a0 msgr2=0x7fa78c153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c1595a0 0x7fa78c153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 -- 172.21.15.50:0/2289091975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa778009420 con 0x7fa78c158c90 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.593+0000 7fa78bfff700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa78c069b70 tx=0x7fa778017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:06.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778000d10 con 0x7fa78c158c90 2023-12-19T14:58:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778011900 con 0x7fa78c158c90 2023-12-19T14:58:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c1561b0 con 0x7fa78c158c90 2023-12-19T14:58:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778025420 con 0x7fa78c158c90 2023-12-19T14:58:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c153870 con 0x7fa78c158c90 2023-12-19T14:58:06.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c086f70 con 0x7fa78c158c90 2023-12-19T14:58:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.597+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fa778023070 con 0x7fa78c158c90 2023-12-19T14:58:06.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.601+0000 7fa788ff9700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 0x7fa770062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:06.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.601+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa77807e310 con 0x7fa78c158c90 2023-12-19T14:58:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.601+0000 7fa790a0a700 1 -- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 msgr2=0x7fa770062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.601+0000 7fa790a0a700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 0x7fa770062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.605+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa78c086f70 con 0x7fa78c158c90 2023-12-19T14:58:06.638 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:06.633+0000 7fd5a0b86000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:58:06.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.777+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa78c086f70 con 0x7fa78c158c90 2023-12-19T14:58:06.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.777+0000 7fa788ff9700 1 -- 172.21.15.50:0/2289091975 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa78c086f70 con 0x7fa78c158c90 2023-12-19T14:58:06.782 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 msgr2=0x7fa770062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 0x7fa770062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 msgr2=0x7fa78c1553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa78c069b70 tx=0x7fa778017000 comp rx=0 tx=0).stop 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 shutdown_connections 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa78c1595a0 0x7fa78c153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7700607c0 0x7fa770062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa78c158c90 0x7fa78c1553e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 --2- 172.21.15.50:0/2289091975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa78c06aeb0 0x7fa78c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:06.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 >> 172.21.15.50:0/2289091975 conn(0x7fa78c067bb0 msgr2=0x7fa78c0cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:06.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 shutdown_connections 2023-12-19T14:58:06.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:06.781+0000 7fa791a0c700 1 -- 172.21.15.50:0/2289091975 wait complete. 2023-12-19T14:58:06.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:06.869 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:06.862+0000 7f106c1c5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:58:07.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 Processor -- start 2023-12-19T14:58:07.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 -- start start 2023-12-19T14:58:07.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c8406dce0 0x7f9c840ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840c7810 0x7f9c8406a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c840c6580 con 0x7f9c840ca000 2023-12-19T14:58:07.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840c67b0 con 0x7f9c840c7810 2023-12-19T14:58:07.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.009+0000 7f9c8886d700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840c6a50 con 0x7f9c8406dce0 2023-12-19T14:58:07.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840c7810 0x7f9c8406a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c840c7810 0x7f9c8406a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33060/0 (socket says 172.21.15.50:33060) 2023-12-19T14:58:07.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 -- 172.21.15.50:0/928807643 learned_addr learned my addr 172.21.15.50:0/928807643 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8259c700 1 --2- 172.21.15.50:0/928807643 >> v2:172.21.15.125:3300/0 conn(0x7f9c8406dce0 0x7f9c840ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 -- 172.21.15.50:0/928807643 >> v2:172.21.15.125:3300/0 conn(0x7f9c8406dce0 msgr2=0x7f9c840ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:07.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 --2- 172.21.15.50:0/928807643 >> v2:172.21.15.125:3300/0 conn(0x7f9c8406dce0 0x7f9c840ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 -- 172.21.15.50:0/928807643 >> v1:172.21.15.50:6789/0 conn(0x7f9c840ca000 legacy=0x7f9c84161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 -- 172.21.15.50:0/928807643 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840c7380 con 0x7f9c840c7810 2023-12-19T14:58:07.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8359e700 1 --2- 172.21.15.50:0/928807643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840c7810 0x7f9c8406a580 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f9c840c9b80 tx=0x7f9c7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526f0872256c221b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c81d9b700 1 -- 172.21.15.50:0/928807643 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00e070 con 0x7f9c840c7810 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c81d9b700 1 -- 172.21.15.50:0/928807643 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c00ba50 con 0x7f9c840c7810 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c81d9b700 1 -- 172.21.15.50:0/928807643 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0092e0 con 0x7f9c840c7810 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 -- 172.21.15.50:0/928807643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840c7810 msgr2=0x7f9c8406a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 --2- 172.21.15.50:0/928807643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840c7810 0x7f9c8406a580 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f9c840c9b80 tx=0x7f9c7c009840 comp rx=0 tx=0).stop 2023-12-19T14:58:07.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 -- 172.21.15.50:0/928807643 shutdown_connections 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 --2- 172.21.15.50:0/928807643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840c7810 0x7f9c8406a580 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 --2- 172.21.15.50:0/928807643 >> v2:172.21.15.125:3300/0 conn(0x7f9c8406dce0 0x7f9c840ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 -- 172.21.15.50:0/928807643 >> 172.21.15.50:0/928807643 conn(0x7f9c84067bb0 msgr2=0x7f9c840cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 -- 172.21.15.50:0/928807643 shutdown_connections 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.013+0000 7f9c8886d700 1 -- 172.21.15.50:0/928807643 wait complete. 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.017+0000 7f9c8886d700 1 Processor -- start 2023-12-19T14:58:07.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.017+0000 7f9c8886d700 1 -- start start 2023-12-19T14:58:07.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.017+0000 7f9c8886d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c8406dce0 0x7f9c8415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.017+0000 7f9c8886d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 0x7f9c8415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84157030 con 0x7f9c840ca000 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84156420 con 0x7f9c840c7810 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84156700 con 0x7f9c8406dce0 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33068/0 (socket says 172.21.15.50:33068) 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 -- 172.21.15.50:0/3940662043 learned_addr learned my addr 172.21.15.50:0/3940662043 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8259c700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 0x7f9c8415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c82d9d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c8406dce0 0x7f9c8415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c8406dce0 msgr2=0x7f9c8415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c8406dce0 0x7f9c8415cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 msgr2=0x7f9c8415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 0x7f9c8415d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 -- 172.21.15.50:0/3940662043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c009ae0 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8259c700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 0x7f9c8415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8359e700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9c84069810 tx=0x7f9c7c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00be90 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c008e40 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c841569e0 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0266b0 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c840b16b0 con 0x7f9c840ca000 2023-12-19T14:58:07.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.021+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c841571e0 con 0x7f9c840ca000 2023-12-19T14:58:07.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.025+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f9c7c003040 con 0x7f9c840ca000 2023-12-19T14:58:07.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.025+0000 7f9c6affd700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 0x7f9c70062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.025+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9c7c07ef80 con 0x7f9c840ca000 2023-12-19T14:58:07.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.025+0000 7f9c82d9d700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 msgr2=0x7f9c70062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:07.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.025+0000 7f9c82d9d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 0x7f9c70062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:07.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.033+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c841571e0 con 0x7f9c840ca000 2023-12-19T14:58:07.039 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.037+0000 7fd5a0b86000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:58:07.088 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.082+0000 7f106c1c5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:07.176 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.170+0000 7f106c1c5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:58:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.229+0000 7f9c82d9d700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 msgr2=0x7f9c70062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:07.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.229+0000 7f9c82d9d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 0x7f9c70062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:07.247 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.242+0000 7f106c1c5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:58:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.245+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c84064040 con 0x7f9c840ca000 2023-12-19T14:58:07.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.245+0000 7f9c6affd700 1 -- 172.21.15.50:0/3940662043 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f9c7c04ef30 con 0x7f9c840ca000 2023-12-19T14:58:07.249 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:07.249 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:58:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 msgr2=0x7f9c70062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 0x7f9c70062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 msgr2=0x7f9c841522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9c84069810 tx=0x7f9c7c018000 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 shutdown_connections 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c840c7810 0x7f9c8415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f9c70060790 0x7f9c70062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c840ca000 0x7f9c841522b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 --2- 172.21.15.50:0/3940662043 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9c8406dce0 0x7f9c8415cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 >> 172.21.15.50:0/3940662043 conn(0x7f9c84067bb0 msgr2=0x7f9c840c0860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 shutdown_connections 2023-12-19T14:58:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.249+0000 7f9c8886d700 1 -- 172.21.15.50:0/3940662043 wait complete. 2023-12-19T14:58:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:58:07.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:07.281 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.277+0000 7fd5a0b86000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:07.330 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.322+0000 7f106c1c5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:58:07.385 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.381+0000 7fd5a0b86000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:58:07.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.429+0000 7fbfaa66e700 1 Processor -- start 2023-12-19T14:58:07.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.429+0000 7fbfaa66e700 1 -- start start 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbfa40cea90 0x7fbfa40cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cc330 0x7fbfa4068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbfa40cc780 0x7fbfa4142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40cbea0 con 0x7fbfa40cea90 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40cb030 con 0x7fbfa40cc780 2023-12-19T14:58:07.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfaa66e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40cb2d0 con 0x7fbfa40cc330 2023-12-19T14:58:07.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbfa40cea90 0x7fbfa40cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbfa40cea90 0x7fbfa40cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33070/0 (socket says 172.21.15.50:33070) 2023-12-19T14:58:07.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3089627405 learned_addr learned my addr 172.21.15.50:0/3089627405 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa37fe700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cc330 0x7fbfa4068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa8c0b700 1 --2- 172.21.15.50:0/3089627405 >> v2:172.21.15.125:3300/0 conn(0x7fbfa40cc780 0x7fbfa4142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3089627405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cc330 msgr2=0x7fbfa4068a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cc330 0x7fbfa4068a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3089627405 >> v2:172.21.15.125:3300/0 conn(0x7fbfa40cc780 msgr2=0x7fbfa4142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3089627405 >> v2:172.21.15.125:3300/0 conn(0x7fbfa40cc780 0x7fbfa4142720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.433+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3089627405 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa4067e00 con 0x7fbfa40cea90 2023-12-19T14:58:07.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cea90 0x7fbfa40cee60 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40cae00 tx=0x7fbf94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f5b22b313280cd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfa2ffd700 1 -- 172.21.15.50:0/3089627405 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400b640 con 0x7fbfa40cea90 2023-12-19T14:58:07.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfa2ffd700 1 -- 172.21.15.50:0/3089627405 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400b7a0 con 0x7fbfa40cea90 2023-12-19T14:58:07.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfa2ffd700 1 -- 172.21.15.50:0/3089627405 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400bb60 con 0x7fbfa40cea90 2023-12-19T14:58:07.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3089627405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cea90 msgr2=0x7fbfa40cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cea90 0x7fbfa40cee60 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40cae00 tx=0x7fbf94009250 comp rx=0 tx=0).stop 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3089627405 shutdown_connections 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3089627405 >> v2:172.21.15.125:3300/0 conn(0x7fbfa40cc780 0x7fbfa4142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cc330 0x7fbfa4068a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3089627405 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cea90 0x7fbfa40cee60 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3089627405 >> 172.21.15.50:0/3089627405 conn(0x7fbfa40cd650 msgr2=0x7fbfa406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3089627405 shutdown_connections 2023-12-19T14:58:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.437+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3089627405 wait complete. 2023-12-19T14:58:07.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 Processor -- start 2023-12-19T14:58:07.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 -- start start 2023-12-19T14:58:07.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfa40cc780 0x7fbfa40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cea90 0x7fbfa40c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4140350 con 0x7fbfa40cc330 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4087c30 con 0x7fbfa40cc780 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfaa66e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4087f10 con 0x7fbfa40cea90 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfa3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfa3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33082/0 (socket says 172.21.15.50:33082) 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3185898474 learned_addr learned my addr 172.21.15.50:0/3185898474 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfa37fe700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfa40cc780 0x7fbfa40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.441+0000 7fbfa8c0b700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cea90 0x7fbfa40c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cea90 msgr2=0x7fbfa40c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cea90 0x7fbfa40c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfa40cc780 msgr2=0x7fbfa40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfa40cc780 0x7fbfa40b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 -- 172.21.15.50:0/3185898474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf940094f0 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfa3fff700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbfa406a0f0 tx=0x7fbf94017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400b910 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400ba70 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa40881f0 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94011630 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa4088840 con 0x7fbfa40cc330 2023-12-19T14:58:07.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.445+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa4071120 con 0x7fbfa40cc330 2023-12-19T14:58:07.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.449+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fbf940119a0 con 0x7fbfa40cc330 2023-12-19T14:58:07.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.449+0000 7fbf8bfff700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 0x7fbf90062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.449+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbf9407e430 con 0x7fbfa40cc330 2023-12-19T14:58:07.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.449+0000 7fbfa37fe700 1 -- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 msgr2=0x7fbf90062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:07.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.449+0000 7fbfa37fe700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 0x7fbf90062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:07.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.457+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf9404e4e0 con 0x7fbfa40cc330 2023-12-19T14:58:07.466 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.461+0000 7fd5a0b86000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:58:07.553 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.549+0000 7fd5a0b86000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:58:07.564 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.558+0000 7f106c1c5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:07.644 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.638+0000 7f106c1c5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:58:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.641+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfa4089d30 con 0x7fbfa40cc330 2023-12-19T14:58:07.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.645+0000 7fbf8bfff700 1 -- 172.21.15.50:0/3185898474 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16111 (secure 0 0 0) 0x7fbf94057070 con 0x7fbfa40cc330 2023-12-19T14:58:07.649 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:07.649 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":130,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:57:27.688637+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/2353401000":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2989287076":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1628872297":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2571945050":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/4156266535":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6833/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6832/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1370626584":"2023-12-20T14:57:27.688521+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 msgr2=0x7fbf90062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 0x7fbf90062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 msgr2=0x7fbfa413fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbfa406a0f0 tx=0x7fbf94017000 comp rx=0 tx=0).stop 2023-12-19T14:58:07.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 shutdown_connections 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfa40cc780 0x7fbfa40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbf90060790 0x7fbf90062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfa40cea90 0x7fbfa40c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 --2- 172.21.15.50:0/3185898474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa40cc330 0x7fbfa413fe10 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 >> 172.21.15.50:0/3185898474 conn(0x7fbfa40cd650 msgr2=0x7fbfa4140c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 shutdown_connections 2023-12-19T14:58:07.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.649+0000 7fbfaa66e700 1 -- 172.21.15.50:0/3185898474 wait complete. 2023-12-19T14:58:07.681 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T14:58:07.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:07.734 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.726+0000 7f106c1c5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:58:07.810 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.805+0000 7fd5a0b86000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.881+0000 7f345bca5700 1 Processor -- start 2023-12-19T14:58:07.885 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.881+0000 7f345bca5700 1 -- start start 2023-12-19T14:58:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345bca5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f345406b990 0x7f34540c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345bca5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345bca5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540cb030 con 0x7f345415a210 2023-12-19T14:58:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345bca5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34541599a0 con 0x7f345415a5e0 2023-12-19T14:58:07.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345bca5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34540cb290 con 0x7f345406b990 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345a242700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345a242700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33090/0 (socket says 172.21.15.50:33090) 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345a242700 1 -- 172.21.15.50:0/3608150301 learned_addr learned my addr 172.21.15.50:0/3608150301 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 --2- 172.21.15.50:0/3608150301 >> v2:172.21.15.125:3300/0 conn(0x7f345406b990 0x7f34540c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 -- 172.21.15.50:0/3608150301 >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 msgr2=0x7f34540c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 --2- 172.21.15.50:0/3608150301 >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 -- 172.21.15.50:0/3608150301 >> v1:172.21.15.125:6789/0 conn(0x7f345415a210 legacy=0x7f345406b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 -- 172.21.15.50:0/3608150301 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540cb570 con 0x7f345406b990 2023-12-19T14:58:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f345a242700 1 --2- 172.21.15.50:0/3608150301 >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.885+0000 7f3459240700 1 --2- 172.21.15.50:0/3608150301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34540c90b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f34540b2760 tx=0x7f34440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f531b229ce8ccbc9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f3458a3f700 1 -- 172.21.15.50:0/3608150301 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011630 con 0x7f345406b990 2023-12-19T14:58:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f3458a3f700 1 -- 172.21.15.50:0/3608150301 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444011790 con 0x7f345406b990 2023-12-19T14:58:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f3458a3f700 1 -- 172.21.15.50:0/3608150301 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011b30 con 0x7f345406b990 2023-12-19T14:58:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 -- 172.21.15.50:0/3608150301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 msgr2=0x7f34540c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 --2- 172.21.15.50:0/3608150301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34540c90b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f34540b2760 tx=0x7f34440091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 -- 172.21.15.50:0/3608150301 shutdown_connections 2023-12-19T14:58:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 --2- 172.21.15.50:0/3608150301 >> v2:172.21.15.50:3300/0 conn(0x7f345415a5e0 0x7f34540c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 --2- 172.21.15.50:0/3608150301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34540c90b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 -- 172.21.15.50:0/3608150301 >> 172.21.15.50:0/3608150301 conn(0x7f3454067410 msgr2=0x7f34540677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 -- 172.21.15.50:0/3608150301 shutdown_connections 2023-12-19T14:58:07.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.889+0000 7f345bca5700 1 -- 172.21.15.50:0/3608150301 wait complete. 2023-12-19T14:58:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 Processor -- start 2023-12-19T14:58:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 -- start start 2023-12-19T14:58:07.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34541463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f345415a5e0 0x7f34541565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345406d470 con 0x7f345415a210 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345406c860 con 0x7f345406b990 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345406cb40 con 0x7f345415a5e0 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33096/0 (socket says 172.21.15.50:33096) 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 -- 172.21.15.50:0/1487754847 learned_addr learned my addr 172.21.15.50:0/1487754847 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459a41700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34541463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345a242700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f345415a5e0 0x7f34541565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f345415a5e0 msgr2=0x7f34541565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f345415a5e0 0x7f34541565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 msgr2=0x7f34541463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34541463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 -- 172.21.15.50:0/1487754847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c009ae0 con 0x7f345415a210 2023-12-19T14:58:07.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3459240700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f3454069c10 tx=0x7f3444017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:07.900 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.893+0000 7fd5a0b86000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:58:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444000d10 con 0x7f345415a210 2023-12-19T14:58:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444011b10 con 0x7f345415a210 2023-12-19T14:58:07.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.893+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3444009420 con 0x7f345415a210 2023-12-19T14:58:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.897+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444025640 con 0x7f345415a210 2023-12-19T14:58:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.897+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345406d100 con 0x7f345415a210 2023-12-19T14:58:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.897+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345406d5e0 con 0x7f345415a210 2023-12-19T14:58:07.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.897+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f3444025a30 con 0x7f345415a210 2023-12-19T14:58:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.901+0000 7f3449ffb700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 0x7f3438062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.901+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f344407e660 con 0x7f345415a210 2023-12-19T14:58:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.901+0000 7f3459a41700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 msgr2=0x7f3438062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:07.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.901+0000 7f3459a41700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 0x7f3438062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:07.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:07.905+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f344404e670 con 0x7f345415a210 2023-12-19T14:58:07.933 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:07.926+0000 7f106c1c5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:58:08.002 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:07.997+0000 7fd5a0b86000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:58:08.021 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.014+0000 7f106c1c5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:58:08.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.105+0000 7f3459a41700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 msgr2=0x7f3438062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:08.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.105+0000 7f3459a41700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 0x7f3438062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:08.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.117+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3454071120 con 0x7f345415a210 2023-12-19T14:58:08.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.117+0000 7f3449ffb700 1 -- 172.21.15.50:0/1487754847 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f3444051c90 con 0x7f345415a210 2023-12-19T14:58:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:08.122 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:58:08.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 msgr2=0x7f3438062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:08.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 0x7f3438062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 msgr2=0x7f3454157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f3454069c10 tx=0x7f3444017000 comp rx=0 tx=0).stop 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 shutdown_connections 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3438060790 0x7f3438062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f345415a5e0 0x7f34541565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f345415a210 0x7f3454157d30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 --2- 172.21.15.50:0/1487754847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f345406b990 0x7f34541463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 >> 172.21.15.50:0/1487754847 conn(0x7f3454067410 msgr2=0x7f3454150520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 shutdown_connections 2023-12-19T14:58:08.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.121+0000 7f345bca5700 1 -- 172.21.15.50:0/1487754847 wait complete. 2023-12-19T14:58:08.127 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:58:08.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:08.229 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:08.225+0000 7fd5a0b86000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:58:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.305+0000 7f6b246cd700 1 Processor -- start 2023-12-19T14:58:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.305+0000 7f6b246cd700 1 -- start start 2023-12-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b246cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b246cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 0x7f6b1c14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b246cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1c0c6630 con 0x7f6b1c0c5900 2023-12-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b246cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c6440 con 0x7f6b1c14fba0 2023-12-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b246cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c6910 con 0x7f6b1c067c90 2023-12-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b21c68700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 0x7f6b1c14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55228/0 (socket says 172.21.15.50:55228) 2023-12-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 -- 172.21.15.50:0/3811874687 learned_addr learned my addr 172.21.15.50:0/3811874687 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:08.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 -- 172.21.15.50:0/3811874687 >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 msgr2=0x7f6b1c14ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:08.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 --2- 172.21.15.50:0/3811874687 >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 0x7f6b1c14ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 -- 172.21.15.50:0/3811874687 >> v1:172.21.15.125:6789/0 conn(0x7f6b1c0c5900 legacy=0x7f6b1c0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 -- 172.21.15.50:0/3811874687 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0c6bf0 con 0x7f6b1c067c90 2023-12-19T14:58:08.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b21c68700 1 --2- 172.21.15.50:0/3811874687 >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 0x7f6b1c14ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b22469700 1 --2- 172.21.15.50:0/3811874687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b180099b0 tx=0x7f6b18009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce4d331fa37d49b5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b21467700 1 -- 172.21.15.50:0/3811874687 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18009ea0 con 0x7f6b1c067c90 2023-12-19T14:58:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b21467700 1 -- 172.21.15.50:0/3811874687 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800b860 con 0x7f6b1c067c90 2023-12-19T14:58:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.309+0000 7f6b21467700 1 -- 172.21.15.50:0/3811874687 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800bc00 con 0x7f6b1c067c90 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- 172.21.15.50:0/3811874687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 msgr2=0x7f6b1c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 --2- 172.21.15.50:0/3811874687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b180099b0 tx=0x7f6b18009250 comp rx=0 tx=0).stop 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- 172.21.15.50:0/3811874687 shutdown_connections 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 --2- 172.21.15.50:0/3811874687 >> v2:172.21.15.125:3300/0 conn(0x7f6b1c14fba0 0x7f6b1c14ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 --2- 172.21.15.50:0/3811874687 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c067c90 0x7f6b1c068060 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- 172.21.15.50:0/3811874687 >> 172.21.15.50:0/3811874687 conn(0x7f6b1c0cd6d0 msgr2=0x7f6b1c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- 172.21.15.50:0/3811874687 shutdown_connections 2023-12-19T14:58:08.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- 172.21.15.50:0/3811874687 wait complete. 2023-12-19T14:58:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 Processor -- start 2023-12-19T14:58:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.313+0000 7f6b246cd700 1 -- start start 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c067c90 0x7f6b1c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c14fba0 0x7f6b1c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c16cdc0 con 0x7f6b1c067c90 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c16d0a0 con 0x7f6b1c0c5900 2023-12-19T14:58:08.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c0c8be0 con 0x7f6b1c14fba0 2023-12-19T14:58:08.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b22469700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c067c90 0x7f6b1c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:49610/0 (socket says 172.21.15.50:49610) 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 -- 172.21.15.50:0/2084052503 learned_addr learned my addr 172.21.15.50:0/2084052503 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b22c6a700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c14fba0 0x7f6b1c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c14fba0 msgr2=0x7f6b1c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c14fba0 0x7f6b1c141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c067c90 msgr2=0x7f6b1c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c067c90 0x7f6b1c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 -- 172.21.15.50:0/2084052503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c009420 con 0x7f6b1c0c5900 2023-12-19T14:58:08.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b21c68700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00ec80 tx=0x7f6b0c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:08.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00c920 con 0x7f6b1c0c5900 2023-12-19T14:58:08.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c00ca80 con 0x7f6b1c0c5900 2023-12-19T14:58:08.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.317+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b180094f0 con 0x7f6b1c0c5900 2023-12-19T14:58:08.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.321+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00ce00 con 0x7f6b1c0c5900 2023-12-19T14:58:08.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.321+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c16ca20 con 0x7f6b1c0c5900 2023-12-19T14:58:08.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.321+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c0c9060 con 0x7f6b1c0c5900 2023-12-19T14:58:08.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.321+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f6b0c013680 con 0x7f6b1c0c5900 2023-12-19T14:58:08.327 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:08.325+0000 7fd5a0b86000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:58:08.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.325+0000 7f6b127fc700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 0x7f6b00062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.325+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6b0c07d260 con 0x7f6b1c0c5900 2023-12-19T14:58:08.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.325+0000 7f6b22469700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 msgr2=0x7f6b00062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:08.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.325+0000 7f6b22469700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 0x7f6b00062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:08.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.333+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6b0c04d230 con 0x7f6b1c0c5900 2023-12-19T14:58:08.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:58:08.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:58:08.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.529+0000 7f6b22469700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 msgr2=0x7f6b00062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:08.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.529+0000 7f6b22469700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 0x7f6b00062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.541+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b1c071120 con 0x7f6b1c0c5900 2023-12-19T14:58:08.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.541+0000 7f6b127fc700 1 -- 172.21.15.50:0/2084052503 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f6b0c050850 con 0x7f6b1c0c5900 2023-12-19T14:58:08.546 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:08.547 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:57:12.670233+0000","modified":"2023-12-19T14:57:16.239398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7878":{"gid":7878,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1300068976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1300068976},{"type":"v1","addr":"172.21.15.50:6835","nonce":1300068976}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T14:58:08.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 msgr2=0x7f6b00062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:08.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 0x7f6b00062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 msgr2=0x7f6b1c16c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00ec80 tx=0x7f6b0c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 shutdown_connections 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b1c0c5900 0x7f6b1c16c4e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6b00060790 0x7f6b00062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b1c14fba0 0x7f6b1c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 --2- 172.21.15.50:0/2084052503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b1c067c90 0x7f6b1c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 >> 172.21.15.50:0/2084052503 conn(0x7f6b1c0cd6d0 msgr2=0x7f6b1c0cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 shutdown_connections 2023-12-19T14:58:08.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.545+0000 7f6b246cd700 1 -- 172.21.15.50:0/2084052503 wait complete. 2023-12-19T14:58:08.552 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 58 2023-12-19T14:58:08.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:08.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.729+0000 7f4e156cd700 1 Processor -- start 2023-12-19T14:58:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.729+0000 7f4e156cd700 1 -- start start 2023-12-19T14:58:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e100ccd80 0x7f4e1015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1006cfb0 con 0x7f4e100cc9b0 2023-12-19T14:58:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100cb660 con 0x7f4e1015c3e0 2023-12-19T14:58:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1015c840 con 0x7f4e100ccd80 2023-12-19T14:58:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0e7fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e100ccd80 0x7f4e1015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0effd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4e100cc9b0 0x7f4e100afc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42892/0 (socket says 172.21.15.50:42892) 2023-12-19T14:58:08.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0effd700 1 -- 172.21.15.50:0/4044234618 learned_addr learned my addr 172.21.15.50:0/4044234618 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:08.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0e7fc700 1 -- 172.21.15.50:0/4044234618 >> v1:172.21.15.125:6789/0 conn(0x7f4e1015c3e0 legacy=0x7f4e100ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0e7fc700 1 -- 172.21.15.50:0/4044234618 >> v1:172.21.15.50:6789/0 conn(0x7f4e100cc9b0 legacy=0x7f4e100afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0e7fc700 1 -- 172.21.15.50:0/4044234618 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100cb380 con 0x7f4e100ccd80 2023-12-19T14:58:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0e7fc700 1 --2- 172.21.15.50:0/4044234618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100ccd80 0x7f4e1015bea0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4e040098e0 tx=0x7f4e040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b76ee9664781b47a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0dffb700 1 -- 172.21.15.50:0/4044234618 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04009e80 con 0x7f4e100ccd80 2023-12-19T14:58:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0dffb700 1 -- 172.21.15.50:0/4044234618 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0400b860 con 0x7f4e100ccd80 2023-12-19T14:58:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e0dffb700 1 -- 172.21.15.50:0/4044234618 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400bbe0 con 0x7f4e100ccd80 2023-12-19T14:58:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 -- 172.21.15.50:0/4044234618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100ccd80 msgr2=0x7f4e1015bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.733+0000 7f4e156cd700 1 --2- 172.21.15.50:0/4044234618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100ccd80 0x7f4e1015bea0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4e040098e0 tx=0x7f4e040091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 -- 172.21.15.50:0/4044234618 shutdown_connections 2023-12-19T14:58:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 --2- 172.21.15.50:0/4044234618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100ccd80 0x7f4e1015bea0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 -- 172.21.15.50:0/4044234618 >> 172.21.15.50:0/4044234618 conn(0x7f4e100670e0 msgr2=0x7f4e100674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 -- 172.21.15.50:0/4044234618 shutdown_connections 2023-12-19T14:58:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 -- 172.21.15.50:0/4044234618 wait complete. 2023-12-19T14:58:08.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 Processor -- start 2023-12-19T14:58:08.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.737+0000 7f4e156cd700 1 -- start start 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100cc9b0 0x7f4e100c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100c30e0 con 0x7f4e1015c3e0 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1016a100 con 0x7f4e100cc9b0 2023-12-19T14:58:08.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10169230 con 0x7f4e100ccd80 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0effd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100cc9b0 0x7f4e100c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33112/0 (socket says 172.21.15.50:33112) 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55254/0 (socket says 172.21.15.50:55254) 2023-12-19T14:58:08.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 -- 172.21.15.50:0/2742250449 learned_addr learned my addr 172.21.15.50:0/2742250449 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 -- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 msgr2=0x7f4e100c4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 -- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100cc9b0 msgr2=0x7f4e100c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100cc9b0 0x7f4e100c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 -- 172.21.15.50:0/2742250449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04009420 con 0x7f4e1015c3e0 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0e7fc700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:08.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e0f7fe700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c870 tx=0x7f4e08001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08012630 con 0x7f4e1015c3e0 2023-12-19T14:58:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e080127b0 con 0x7f4e1015c3e0 2023-12-19T14:58:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08012b70 con 0x7f4e1015c3e0 2023-12-19T14:58:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.741+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e101694f0 con 0x7f4e1015c3e0 2023-12-19T14:58:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.745+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e10169ab0 con 0x7f4e1015c3e0 2023-12-19T14:58:08.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.745+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1004be40 con 0x7f4e1015c3e0 2023-12-19T14:58:08.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.745+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 55) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f4e08003030 con 0x7f4e1015c3e0 2023-12-19T14:58:08.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.749+0000 7f4df6ffd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 0x7f4dfc062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:08.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.749+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4e0807cd10 con 0x7f4e1015c3e0 2023-12-19T14:58:08.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.749+0000 7f4e0effd700 1 -- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 msgr2=0x7f4dfc062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:08.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.749+0000 7f4e0effd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 0x7f4dfc062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:08.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.753+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e08082020 con 0x7f4e1015c3e0 2023-12-19T14:58:08.797 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.790+0000 7f106c1c5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:58:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.813+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 56) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f4e0804d0b0 con 0x7f4e1015c3e0 2023-12-19T14:58:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.813+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 msgr2=0x7f4dfc062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:08.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.813+0000 7f4df6ffd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 0x7f4dfc062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.900 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1037627700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.900 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1037627700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.900 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1037627700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1037627700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1037627700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1034621700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1034621700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1034621700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1034621700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.901 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:08.894+0000 7f1034621700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:08.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.945+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e1006c960 con 0x7f4e1015c3e0 2023-12-19T14:58:08.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.949+0000 7f4df6ffd700 1 -- 172.21.15.50:0/2742250449 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16683 (secure 0 0 0) 0x7f4e08068a70 con 0x7f4e1015c3e0 2023-12-19T14:58:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:08.953 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":132,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:08.802894+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:57:10.809944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T14:57:11.656478+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/2574956284":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/3845778139":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1048412071":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1114305366":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1072406585":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/4291635214":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/2353401000":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2989287076":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1628872297":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2571945050":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/4156266535":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1080158370":"2023-12-20T14:58:08.802830+0000","172.21.15.125:6833/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6832/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/3396238407":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1370626584":"2023-12-20T14:57:27.688521+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 msgr2=0x7f4e100c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c870 tx=0x7f4e08001670 comp rx=0 tx=0).stop 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 shutdown_connections 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4dfc060790 0x7f4dfc062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e1015c3e0 0x7f4e100c4bc0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e100ccd80 0x7f4e100c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 --2- 172.21.15.50:0/2742250449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e100cc9b0 0x7f4e100c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 >> 172.21.15.50:0/2742250449 conn(0x7f4e100670e0 msgr2=0x7f4e10159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:08.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 shutdown_connections 2023-12-19T14:58:08.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:08.953+0000 7f4e156cd700 1 -- 172.21.15.50:0/2742250449 wait complete. 2023-12-19T14:58:08.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T14:58:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.149+0000 7f800e585700 1 Processor -- start 2023-12-19T14:58:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.149+0000 7f800e585700 1 -- start start 2023-12-19T14:58:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800e585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8008158cc0 0x7f800806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800e585700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f800806dc00 0x7f800806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800e585700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800806b0d0 con 0x7f800806baa0 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800e585700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800806beb0 con 0x7f800806dc00 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800e585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080b4790 con 0x7f8008158cc0 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f800806dc00 0x7f800806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800cb22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f800806baa0 0x7f80080b3870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42898/0 (socket says 172.21.15.50:42898) 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f800cb22700 1 -- 172.21.15.50:0/513420157 learned_addr learned my addr 172.21.15.50:0/513420157 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f8007fff700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8008158cc0 0x7f800806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594260925 0 0) 0x7f800806b0d0 con 0x7f800806baa0 2023-12-19T14:58:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0003570 con 0x7f800806baa0 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 -- 172.21.15.50:0/513420157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8008158cc0 msgr2=0x7f800806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8008158cc0 0x7f800806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 -- 172.21.15.50:0/513420157 >> v1:172.21.15.50:6789/0 conn(0x7f800806baa0 legacy=0x7f80080b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669731034 0 0) 0x7f7ff0003570 con 0x7f800806baa0 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 -- 172.21.15.50:0/513420157 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800806d710 con 0x7f800806dc00 2023-12-19T14:58:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.153+0000 7f80077fe700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806dc00 0x7f800806b560 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0098e0 tx=0x7f7ffc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b245c21aba80b0dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000017f0 con 0x7f800806dc00 2023-12-19T14:58:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc011420 con 0x7f800806dc00 2023-12-19T14:58:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f8006ffd700 1 -- 172.21.15.50:0/513420157 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0117c0 con 0x7f800806dc00 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 -- 172.21.15.50:0/513420157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806dc00 msgr2=0x7f800806b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806dc00 0x7f800806b560 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0098e0 tx=0x7f7ffc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 -- 172.21.15.50:0/513420157 shutdown_connections 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806dc00 0x7f800806b560 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 --2- 172.21.15.50:0/513420157 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8008158cc0 0x7f800806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 -- 172.21.15.50:0/513420157 >> 172.21.15.50:0/513420157 conn(0x7f8008067bb0 msgr2=0x7f80080cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 -- 172.21.15.50:0/513420157 shutdown_connections 2023-12-19T14:58:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.157+0000 7f800e585700 1 -- 172.21.15.50:0/513420157 wait complete. 2023-12-19T14:58:09.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 Processor -- start 2023-12-19T14:58:09.163 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:09.161+0000 7fd5a0b86000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:58:09.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 -- start start 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 0x7f800815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8008158cc0 0x7f8008168fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800815eb80 con 0x7f800806baa0 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800815df70 con 0x7f8008158cc0 2023-12-19T14:58:09.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f800e585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800815e250 con 0x7f800806dc00 2023-12-19T14:58:09.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33140/0 (socket says 172.21.15.50:33140) 2023-12-19T14:58:09.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 -- 172.21.15.50:0/3902685844 learned_addr learned my addr 172.21.15.50:0/3902685844 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f80077fe700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 0x7f800815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 -- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 msgr2=0x7f800815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 0x7f800815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 -- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8008158cc0 msgr2=0x7f8008168fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8008158cc0 0x7f8008168fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 -- 172.21.15.50:0/3902685844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009420 con 0x7f800806baa0 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f80077fe700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 0x7f800815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.161+0000 7f8007fff700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d890 tx=0x7f7ff8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800b600 con 0x7f800806baa0 2023-12-19T14:58:09.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff800b760 con 0x7f800806baa0 2023-12-19T14:58:09.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800815e530 con 0x7f800806baa0 2023-12-19T14:58:09.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800bae0 con 0x7f800806baa0 2023-12-19T14:58:09.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8008151070 con 0x7f800806baa0 2023-12-19T14:58:09.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.165+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8008071120 con 0x7f800806baa0 2023-12-19T14:58:09.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.169+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 56) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f7ff801e460 con 0x7f800806baa0 2023-12-19T14:58:09.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.169+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6769+0+0 (secure 0 0 0) 0x7f7ff807cf00 con 0x7f800806baa0 2023-12-19T14:58:09.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.173+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7ff804d390 con 0x7f800806baa0 2023-12-19T14:58:09.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.389+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f80080cf930 con 0x7f800806baa0 2023-12-19T14:58:09.394 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T14:58:09.389+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T14:58:09.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.401+0000 7f7feffff700 1 -- 172.21.15.50:0/3902685844 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7ff80509b0 con 0x7f800806baa0 2023-12-19T14:58:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 msgr2=0x7f8008150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d890 tx=0x7f7ff8008040 comp rx=0 tx=0).stop 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 shutdown_connections 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8008158cc0 0x7f8008168fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f800806dc00 0x7f800815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 --2- 172.21.15.50:0/3902685844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800806baa0 0x7f8008150b30 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 >> 172.21.15.50:0/3902685844 conn(0x7f8008067bb0 msgr2=0x7f80080b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 shutdown_connections 2023-12-19T14:58:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.405+0000 7f800e585700 1 -- 172.21.15.50:0/3902685844 wait complete. 2023-12-19T14:58:09.409 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T14:58:09.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T14:58:09.475 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:58:09.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.589+0000 7f0165c8e700 1 Processor -- start 2023-12-19T14:58:09.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.589+0000 7f0165c8e700 1 -- start start 2023-12-19T14:58:09.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600ce2d0 0x7f01600ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f01600caa10 0x7f016015d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 0x7f0160158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016015d960 con 0x7f01600caa10 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600cbbb0 con 0x7f01600cc040 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0165c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016006dc90 con 0x7f01600ce2d0 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f01600caa10 0x7f016015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f01600caa10 0x7f016015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33166/0 (socket says 172.21.15.50:33166) 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 -- 172.21.15.50:0/3530704009 learned_addr learned my addr 172.21.15.50:0/3530704009 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:09.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f0164c8c700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600ce2d0 0x7f01600ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f016548d700 1 --2- 172.21.15.50:0/3530704009 >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 0x7f0160158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 -- 172.21.15.50:0/3530704009 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600ce2d0 msgr2=0x7f01600ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600ce2d0 0x7f01600ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 -- 172.21.15.50:0/3530704009 >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 msgr2=0x7f0160158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 --2- 172.21.15.50:0/3530704009 >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 0x7f0160158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 -- 172.21.15.50:0/3530704009 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600cafa0 con 0x7f01600caa10 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f016548d700 1 --2- 172.21.15.50:0/3530704009 >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 0x7f0160158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015ffff700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f016015d420 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2ab0 tx=0x7f014c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def31d965a6e062f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015f7fe700 1 -- 172.21.15.50:0/3530704009 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c009e70 con 0x7f01600caa10 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015f7fe700 1 -- 172.21.15.50:0/3530704009 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00b860 con 0x7f01600caa10 2023-12-19T14:58:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.593+0000 7f015f7fe700 1 -- 172.21.15.50:0/3530704009 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00bc00 con 0x7f01600caa10 2023-12-19T14:58:09.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- 172.21.15.50:0/3530704009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 msgr2=0x7f016015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f016015d420 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2ab0 tx=0x7f014c009250 comp rx=0 tx=0).stop 2023-12-19T14:58:09.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- 172.21.15.50:0/3530704009 shutdown_connections 2023-12-19T14:58:09.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f016015d420 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 --2- 172.21.15.50:0/3530704009 >> v2:172.21.15.125:3300/0 conn(0x7f01600cc040 0x7f0160158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 --2- 172.21.15.50:0/3530704009 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600ce2d0 0x7f01600ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- 172.21.15.50:0/3530704009 >> 172.21.15.50:0/3530704009 conn(0x7f0160067bb0 msgr2=0x7f01600cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:09.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- 172.21.15.50:0/3530704009 shutdown_connections 2023-12-19T14:58:09.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- 172.21.15.50:0/3530704009 wait complete. 2023-12-19T14:58:09.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 Processor -- start 2023-12-19T14:58:09.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.597+0000 7f0165c8e700 1 -- start start 2023-12-19T14:58:09.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600cc040 0x7f01600b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01600ce2d0 0x7f01600b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01601592a0 con 0x7f01600caa10 2023-12-19T14:58:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160171b70 con 0x7f01600ce2d0 2023-12-19T14:58:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016016ea60 con 0x7f01600cc040 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33180/0 (socket says 172.21.15.50:33180) 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 -- 172.21.15.50:0/2652272743 learned_addr learned my addr 172.21.15.50:0/2652272743 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f015ffff700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600cc040 0x7f01600b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f016548d700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01600ce2d0 0x7f01600b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 -- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600cc040 msgr2=0x7f01600b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600cc040 0x7f01600b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 -- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01600ce2d0 msgr2=0x7f01600b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01600ce2d0 0x7f01600b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 -- 172.21.15.50:0/2652272743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0094f0 con 0x7f01600caa10 2023-12-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0164c8c700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f015400e930 tx=0x7f01540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154013680 con 0x7f01600caa10 2023-12-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01540137e0 con 0x7f01600caa10 2023-12-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0160171240 con 0x7f01600caa10 2023-12-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154013b80 con 0x7f01600caa10 2023-12-19T14:58:09.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.601+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0160171800 con 0x7f01600caa10 2023-12-19T14:58:09.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.605+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016004be40 con 0x7f01600caa10 2023-12-19T14:58:09.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.605+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 56) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f015400c3e0 con 0x7f01600caa10 2023-12-19T14:58:09.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.605+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f015407c8b0 con 0x7f01600caa10 2023-12-19T14:58:09.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.613+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f015404ccf0 con 0x7f01600caa10 2023-12-19T14:58:09.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.817+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 57) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f015404c7a0 con 0x7f01600caa10 2023-12-19T14:58:09.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.817+0000 7f015cff9700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 0x7f0144068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:09.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.821+0000 7f015ffff700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 0x7f0144068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:09.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.825+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f01600cb280 con 0x7f01600caa10 2023-12-19T14:58:09.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:09.829+0000 7f015ffff700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 0x7f0144068900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2ab0 tx=0x7f014c017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:10.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.405+0000 7f015cff9700 1 -- 172.21.15.50:0/2652272743 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0154068550 con 0x7f01600caa10 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 msgr2=0x7f0144068900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 0x7f0144068900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2ab0 tx=0x7f014c017000 comp rx=0 tx=0).stop 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 msgr2=0x7f0160158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f015400e930 tx=0x7f01540091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 shutdown_connections 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01600ce2d0 0x7f01600b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0144066520 0x7f0144068900 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f01600cc040 0x7f01600b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 --2- 172.21.15.50:0/2652272743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01600caa10 0x7f0160158d60 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 >> 172.21.15.50:0/2652272743 conn(0x7f0160067bb0 msgr2=0x7f0160155570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 shutdown_connections 2023-12-19T14:58:10.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.409+0000 7f0165c8e700 1 -- 172.21.15.50:0/2652272743 wait complete. 2023-12-19T14:58:10.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T14:58:10.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.609+0000 7fc4644f0700 1 Processor -- start 2023-12-19T14:58:10.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.613+0000 7fc4644f0700 1 -- start start 2023-12-19T14:58:10.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc4644f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c0c9ac0 0x7fc45c0672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc4644f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc45c067830 0x7fc45c162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc4644f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c06d570 con 0x7fc45c0c86e0 2023-12-19T14:58:10.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc4644f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c067cb0 con 0x7fc45c067830 2023-12-19T14:58:10.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc4644f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c06e110 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46228c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc45c0c86e0 0x7fc45c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42908/0 (socket says 172.21.15.50:42908) 2023-12-19T14:58:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46228c700 1 -- 172.21.15.50:0/3217141711 learned_addr learned my addr 172.21.15.50:0/3217141711 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3217141711 >> v2:172.21.15.50:3300/0 conn(0x7fc45c067830 0x7fc45c162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc461a8b700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c0c9ac0 0x7fc45c0672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 -- 172.21.15.50:0/3217141711 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c0c9ac0 msgr2=0x7fc45c0672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:10.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c0c9ac0 0x7fc45c0672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 -- 172.21.15.50:0/3217141711 >> v1:172.21.15.50:6789/0 conn(0x7fc45c0c86e0 legacy=0x7fc45c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 -- 172.21.15.50:0/3217141711 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c06c960 con 0x7fc45c067830 2023-12-19T14:58:10.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46128a700 1 -- 172.21.15.50:0/3217141711 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745615127 0 0) 0x7fc45c06d570 con 0x7fc45c0c86e0 2023-12-19T14:58:10.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c067830 0x7fc45c162ae0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc454009e60 tx=0x7fc454009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91faf8cf9fe07bc3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:10.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46128a700 1 -- 172.21.15.50:0/3217141711 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45400e070 con 0x7fc45c067830 2023-12-19T14:58:10.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46128a700 1 -- 172.21.15.50:0/3217141711 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45400bb60 con 0x7fc45c067830 2023-12-19T14:58:10.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.617+0000 7fc46128a700 1 -- 172.21.15.50:0/3217141711 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454012430 con 0x7fc45c067830 2023-12-19T14:58:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- 172.21.15.50:0/3217141711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c067830 msgr2=0x7fc45c162ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c067830 0x7fc45c162ae0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc454009e60 tx=0x7fc454009780 comp rx=0 tx=0).stop 2023-12-19T14:58:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- 172.21.15.50:0/3217141711 shutdown_connections 2023-12-19T14:58:10.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c067830 0x7fc45c162ae0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3217141711 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c0c9ac0 0x7fc45c0672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- 172.21.15.50:0/3217141711 >> 172.21.15.50:0/3217141711 conn(0x7fc45c0cdce0 msgr2=0x7fc45c0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- 172.21.15.50:0/3217141711 shutdown_connections 2023-12-19T14:58:10.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- 172.21.15.50:0/3217141711 wait complete. 2023-12-19T14:58:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 Processor -- start 2023-12-19T14:58:10.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.621+0000 7fc4644f0700 1 -- start start 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c067830 0x7fc45c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0c86e0 0x7fc45c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0af950 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c1567b0 con 0x7fc45c0c86e0 2023-12-19T14:58:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc4644f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15a1f0 con 0x7fc45c067830 2023-12-19T14:58:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33204/0 (socket says 172.21.15.50:33204) 2023-12-19T14:58:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 -- 172.21.15.50:0/3909321130 learned_addr learned my addr 172.21.15.50:0/3909321130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc461a8b700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0c86e0 0x7fc45c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc46228c700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c067830 0x7fc45c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 -- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c067830 msgr2=0x7fc45c1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c067830 0x7fc45c1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 -- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0c86e0 msgr2=0x7fc45c155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0c86e0 0x7fc45c155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 -- 172.21.15.50:0/3909321130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454009a20 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.625+0000 7fc462a8d700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc45400b6c0 tx=0x7fc454019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454012e90 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc454027470 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454027830 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c15b4a0 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c159e80 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c071120 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.629+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 57) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fc454024070 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.633+0000 7fc4527fc700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 0x7fc440062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.633+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fc45407f010 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.633+0000 7fc46228c700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 0x7fc440062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.633+0000 7fc46228c700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 0x7fc440062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc45c063680 tx=0x7fc458007d30 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.637+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc45404f430 con 0x7fc45c0c9ac0 2023-12-19T14:58:10.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:10.813+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc45c15a330 con 0x7fc45c0c9ac0 2023-12-19T14:58:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.409+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc45402c0d0 con 0x7fc45c0c9ac0 2023-12-19T14:58:11.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.417+0000 7fc4527fc700 1 -- 172.21.15.50:0/3909321130 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc454003040 con 0x7fc45c0c9ac0 2023-12-19T14:58:11.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.417+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 msgr2=0x7fc440062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.417+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 0x7fc440062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc45c063680 tx=0x7fc458007d30 comp rx=0 tx=0).stop 2023-12-19T14:58:11.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.417+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 msgr2=0x7fc45c156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.417+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc45400b6c0 tx=0x7fc454019000 comp rx=0 tx=0).stop 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 shutdown_connections 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc45c0c86e0 0x7fc45c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4400603a0 0x7fc440062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc45c0c9ac0 0x7fc45c156270 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 --2- 172.21.15.50:0/3909321130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc45c067830 0x7fc45c1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 >> 172.21.15.50:0/3909321130 conn(0x7fc45c0cdce0 msgr2=0x7fc45c0c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 shutdown_connections 2023-12-19T14:58:11.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.421+0000 7fc4644f0700 1 -- 172.21.15.50:0/3909321130 wait complete. 2023-12-19T14:58:11.425 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T14:58:11.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T14:58:11.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.625+0000 7f4887145700 1 Processor -- start 2023-12-19T14:58:11.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 -- start start 2023-12-19T14:58:11.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4880155590 0x7f4880159b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48800ac9d0 0x7f48800acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4880069900 con 0x7f48801551c0 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48801559f0 con 0x7f4880155590 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.629+0000 7f4887145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48800c7340 con 0x7f48800ac9d0 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4884ee1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f48801551c0 0x7f48800690d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42914/0 (socket says 172.21.15.50:42914) 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4884ee1700 1 -- 172.21.15.50:0/2865895065 learned_addr learned my addr 172.21.15.50:0/2865895065 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 --2- 172.21.15.50:0/2865895065 >> v2:172.21.15.50:3300/0 conn(0x7f4880155590 0x7f4880159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f48856e2700 1 --2- 172.21.15.50:0/2865895065 >> v2:172.21.15.125:3300/0 conn(0x7f48800ac9d0 0x7f48800acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487f7fe700 1 -- 172.21.15.50:0/2865895065 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562459786 0 0) 0x7f4880069900 con 0x7f48801551c0 2023-12-19T14:58:11.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487f7fe700 1 -- 172.21.15.50:0/2865895065 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870003570 con 0x7f48801551c0 2023-12-19T14:58:11.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 -- 172.21.15.50:0/2865895065 >> v2:172.21.15.125:3300/0 conn(0x7f48800ac9d0 msgr2=0x7f48800acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:11.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 --2- 172.21.15.50:0/2865895065 >> v2:172.21.15.125:3300/0 conn(0x7f48800ac9d0 0x7f48800acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 -- 172.21.15.50:0/2865895065 >> v1:172.21.15.50:6789/0 conn(0x7f48801551c0 legacy=0x7f48800690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 -- 172.21.15.50:0/2865895065 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800c6a10 con 0x7f4880155590 2023-12-19T14:58:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487ffff700 1 --2- 172.21.15.50:0/2865895065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4880155590 0x7f4880159b30 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f48800c32d0 tx=0x7f486c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f40c03acb177c6b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487f7fe700 1 -- 172.21.15.50:0/2865895065 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c011630 con 0x7f4880155590 2023-12-19T14:58:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487f7fe700 1 -- 172.21.15.50:0/2865895065 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c0117b0 con 0x7f4880155590 2023-12-19T14:58:11.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f487f7fe700 1 -- 172.21.15.50:0/2865895065 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c011b70 con 0x7f4880155590 2023-12-19T14:58:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 -- 172.21.15.50:0/2865895065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4880155590 msgr2=0x7f4880159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 --2- 172.21.15.50:0/2865895065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4880155590 0x7f4880159b30 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f48800c32d0 tx=0x7f486c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 -- 172.21.15.50:0/2865895065 shutdown_connections 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 --2- 172.21.15.50:0/2865895065 >> v2:172.21.15.125:3300/0 conn(0x7f48800ac9d0 0x7f48800acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 --2- 172.21.15.50:0/2865895065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4880155590 0x7f4880159b30 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.633+0000 7f4887145700 1 -- 172.21.15.50:0/2865895065 >> 172.21.15.50:0/2865895065 conn(0x7f48800c8db0 msgr2=0x7f48800c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- 172.21.15.50:0/2865895065 shutdown_connections 2023-12-19T14:58:11.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- 172.21.15.50:0/2865895065 wait complete. 2023-12-19T14:58:11.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 Processor -- start 2023-12-19T14:58:11.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- start start 2023-12-19T14:58:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48801551c0 0x7f48801470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4880155590 0x7f48800bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48800c1410 con 0x7f48800ac9d0 2023-12-19T14:58:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48800c0800 con 0x7f4880155590 2023-12-19T14:58:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.637+0000 7f4887145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800c0ae0 con 0x7f48801551c0 2023-12-19T14:58:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33216/0 (socket says 172.21.15.50:33216) 2023-12-19T14:58:11.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 -- 172.21.15.50:0/1882904094 learned_addr learned my addr 172.21.15.50:0/1882904094 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:11.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f48856e2700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4880155590 0x7f48800bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f487ffff700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48801551c0 0x7f48801470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 -- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48801551c0 msgr2=0x7f48801470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48801551c0 0x7f48801470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 -- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4880155590 msgr2=0x7f48800bfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4880155590 0x7f48800bfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 -- 172.21.15.50:0/1882904094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c009420 con 0x7f48800ac9d0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4884ee1700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f487400c7c0 tx=0x7f4874007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874005cd0 con 0x7f48800ac9d0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874005e30 con 0x7f48800ac9d0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874007320 con 0x7f48800ac9d0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48800c0dc0 con 0x7f48800ac9d0 2023-12-19T14:58:11.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.641+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4880147620 con 0x7f48800ac9d0 2023-12-19T14:58:11.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.645+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4880071120 con 0x7f48800ac9d0 2023-12-19T14:58:11.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.645+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f487401b3b0 con 0x7f48800ac9d0 2023-12-19T14:58:11.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.645+0000 7f487cff9700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 0x7f4864062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.645+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f487407c340 con 0x7f48800ac9d0 2023-12-19T14:58:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.645+0000 7f487ffff700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 0x7f4864062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.649+0000 7f487ffff700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 0x7f4864062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48800c32d0 tx=0x7f486c017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:11.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.653+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f487404c4b0 con 0x7f48800ac9d0 2023-12-19T14:58:11.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:11.829+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48800c6cf0 con 0x7f48800ac9d0 2023-12-19T14:58:12.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.425+0000 7f487cff9700 1 -- 172.21.15.50:0/1882904094 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7f487404fad0 con 0x7f48800ac9d0 2023-12-19T14:58:12.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 msgr2=0x7f4864062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 0x7f4864062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48800c32d0 tx=0x7f486c017000 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 msgr2=0x7f48800c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f487400c7c0 tx=0x7f4874007e00 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 shutdown_connections 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4880155590 0x7f48800bfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4864060670 0x7f4864062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48801551c0 0x7f48801470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 --2- 172.21.15.50:0/1882904094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48800ac9d0 0x7f48800c18a0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 >> 172.21.15.50:0/1882904094 conn(0x7f48800c8db0 msgr2=0x7f488014a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 shutdown_connections 2023-12-19T14:58:12.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.429+0000 7f4887145700 1 -- 172.21.15.50:0/1882904094 wait complete. 2023-12-19T14:58:12.435 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T14:58:12.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T14:58:12.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.625+0000 7f64ccae3700 1 Processor -- start 2023-12-19T14:58:12.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.625+0000 7f64ccae3700 1 -- start start 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64ccae3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f64c806dcf0 0x7f64c8159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64ccae3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c806c850 con 0x7f64c806aeb0 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64ccae3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80cb650 con 0x7f64c8158c90 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64ccae3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c806b470 con 0x7f64c806dcf0 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c6ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f64c806aeb0 0x7f64c8159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42916/0 (socket says 172.21.15.50:42916) 2023-12-19T14:58:12.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c6ffd700 1 -- 172.21.15.50:0/4026050842 learned_addr learned my addr 172.21.15.50:0/4026050842 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:12.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c7fff700 1 --2- 172.21.15.50:0/4026050842 >> v2:172.21.15.125:3300/0 conn(0x7f64c806dcf0 0x7f64c8159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443003757 0 0) 0x7f64c806c850 con 0x7f64c806aeb0 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0003570 con 0x7f64c806aeb0 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356291156 0 0) 0x7f64c80cb650 con 0x7f64c8158c90 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c806c850 con 0x7f64c8158c90 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c7fff700 1 -- 172.21.15.50:0/4026050842 >> v1:172.21.15.125:6789/0 conn(0x7f64c8158c90 legacy=0x7f64c80caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c7fff700 1 -- 172.21.15.50:0/4026050842 >> v1:172.21.15.50:6789/0 conn(0x7f64c806aeb0 legacy=0x7f64c8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.629+0000 7f64c7fff700 1 -- 172.21.15.50:0/4026050842 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c806b750 con 0x7f64c806dcf0 2023-12-19T14:58:12.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64c7fff700 1 --2- 172.21.15.50:0/4026050842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806dcf0 0x7f64c8159770 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009e60 tx=0x7f64c0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46189381d697366e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c000e070 con 0x7f64c806dcf0 2023-12-19T14:58:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c000bab0 con 0x7f64c806dcf0 2023-12-19T14:58:12.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64c67fc700 1 -- 172.21.15.50:0/4026050842 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c000be70 con 0x7f64c806dcf0 2023-12-19T14:58:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 -- 172.21.15.50:0/4026050842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806dcf0 msgr2=0x7f64c8159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 --2- 172.21.15.50:0/4026050842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806dcf0 0x7f64c8159770 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009e60 tx=0x7f64c0009780 comp rx=0 tx=0).stop 2023-12-19T14:58:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 -- 172.21.15.50:0/4026050842 shutdown_connections 2023-12-19T14:58:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 --2- 172.21.15.50:0/4026050842 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806dcf0 0x7f64c8159770 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 -- 172.21.15.50:0/4026050842 >> 172.21.15.50:0/4026050842 conn(0x7f64c8067bb0 msgr2=0x7f64c80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 -- 172.21.15.50:0/4026050842 shutdown_connections 2023-12-19T14:58:12.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.633+0000 7f64ccae3700 1 -- 172.21.15.50:0/4026050842 wait complete. 2023-12-19T14:58:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 Processor -- start 2023-12-19T14:58:12.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 -- start start 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 0x7f64c8156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64c8158c90 0x7f64c80c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80b3890 con 0x7f64c806dcf0 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80b4410 con 0x7f64c806aeb0 2023-12-19T14:58:12.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64ccae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c80b3ae0 con 0x7f64c8158c90 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.637+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33220/0 (socket says 172.21.15.50:33220) 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 -- 172.21.15.50:0/3335707092 learned_addr learned my addr 172.21.15.50:0/3335707092 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c77fe700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 0x7f64c8156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c7fff700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64c8158c90 0x7f64c80c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 -- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64c8158c90 msgr2=0x7f64c80c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64c8158c90 0x7f64c80c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 -- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 msgr2=0x7f64c8156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 0x7f64c8156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 -- 172.21.15.50:0/3335707092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0009a20 con 0x7f64c806dcf0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c77fe700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 0x7f64c8156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64c6ffd700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c790 tx=0x7f64bc007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc005b30 con 0x7f64c806dcf0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc005c90 con 0x7f64c806dcf0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c80b3dc0 con 0x7f64c806dcf0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc007300 con 0x7f64c806dcf0 2023-12-19T14:58:12.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c81565d0 con 0x7f64c806dcf0 2023-12-19T14:58:12.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.641+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c80b4580 con 0x7f64c806dcf0 2023-12-19T14:58:12.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.645+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f64bc01b400 con 0x7f64c806dcf0 2023-12-19T14:58:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.645+0000 7f64af7fe700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 0x7f64b4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.645+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f64bc07c5a0 con 0x7f64c806dcf0 2023-12-19T14:58:12.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.645+0000 7f64c77fe700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 0x7f64b4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.649+0000 7f64c77fe700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 0x7f64b4062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64b80038f0 tx=0x7f64b8009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:12.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.653+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f64bc0503c0 con 0x7f64c806dcf0 2023-12-19T14:58:12.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.829+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f64c804be40 con 0x7f64c806dcf0 2023-12-19T14:58:12.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.837+0000 7f64af7fe700 1 -- 172.21.15.50:0/3335707092 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f64bc0076f0 con 0x7f64c806dcf0 2023-12-19T14:58:12.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 msgr2=0x7f64b4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 0x7f64b4062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f64b80038f0 tx=0x7f64b8009040 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 msgr2=0x7f64c815cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c790 tx=0x7f64bc007ea0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 shutdown_connections 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f64b40606a0 0x7f64b4062b50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64c8158c90 0x7f64c80c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64c806dcf0 0x7f64c815cda0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 --2- 172.21.15.50:0/3335707092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64c806aeb0 0x7f64c8156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 >> 172.21.15.50:0/3335707092 conn(0x7f64c8067bb0 msgr2=0x7f64c8157f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 shutdown_connections 2023-12-19T14:58:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:12.841+0000 7f64ccae3700 1 -- 172.21.15.50:0/3335707092 wait complete. 2023-12-19T14:58:12.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T14:58:13.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.041+0000 7f621d8a5700 1 Processor -- start 2023-12-19T14:58:13.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.041+0000 7f621d8a5700 1 -- start start 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f621d8a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 0x7f62180a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f621d8a5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180c6ab0 con 0x7f621815c520 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f621d8a5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621806b240 con 0x7f62180ca240 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f621d8a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621806b520 con 0x7f621815c8f0 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f6216ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f621815c520 0x7f621806a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42918/0 (socket says 172.21.15.50:42918) 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f6216ffd700 1 -- 172.21.15.50:0/1842920440 learned_addr learned my addr 172.21.15.50:0/1842920440 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f620e7fc700 1 --2- 172.21.15.50:0/1842920440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 0x7f62180a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427796119 0 0) 0x7f62180c6ab0 con 0x7f621815c520 2023-12-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6204003570 con 0x7f621815c520 2023-12-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f620e7fc700 1 -- 172.21.15.50:0/1842920440 >> v1:172.21.15.125:6789/0 conn(0x7f62180ca240 legacy=0x7f62180c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296495000 0 0) 0x7f621806b240 con 0x7f62180ca240 2023-12-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f620e7fc700 1 -- 172.21.15.50:0/1842920440 >> v1:172.21.15.50:6789/0 conn(0x7f621815c520 legacy=0x7f621806a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f620e7fc700 1 -- 172.21.15.50:0/1842920440 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180c6d50 con 0x7f621815c8f0 2023-12-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.045+0000 7f620e7fc700 1 --2- 172.21.15.50:0/1842920440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 0x7f62180a8850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6200009910 tx=0x7f62000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d6881b29a958d91 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000b710 con 0x7f621815c8f0 2023-12-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620000b870 con 0x7f621815c8f0 2023-12-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f62167fc700 1 -- 172.21.15.50:0/1842920440 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000bc30 con 0x7f621815c8f0 2023-12-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 -- 172.21.15.50:0/1842920440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 msgr2=0x7f62180a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 --2- 172.21.15.50:0/1842920440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 0x7f62180a8850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6200009910 tx=0x7f62000091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 -- 172.21.15.50:0/1842920440 shutdown_connections 2023-12-19T14:58:13.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 --2- 172.21.15.50:0/1842920440 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f621815c8f0 0x7f62180a8850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 -- 172.21.15.50:0/1842920440 >> 172.21.15.50:0/1842920440 conn(0x7f6218067bb0 msgr2=0x7f62180cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:13.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 -- 172.21.15.50:0/1842920440 shutdown_connections 2023-12-19T14:58:13.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.049+0000 7f621d8a5700 1 -- 172.21.15.50:0/1842920440 wait complete. 2023-12-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 Processor -- start 2023-12-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 -- start start 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62180ca240 0x7f62180afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f621815c520 0x7f6218157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b04c0 con 0x7f621815c8f0 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b07a0 con 0x7f621815c520 2023-12-19T14:58:13.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f621d8a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b0a80 con 0x7f62180ca240 2023-12-19T14:58:13.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f620e7fc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f621815c520 0x7f6218157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33232/0 (socket says 172.21.15.50:33232) 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 -- 172.21.15.50:0/3343149178 learned_addr learned my addr 172.21.15.50:0/3343149178 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f6216ffd700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62180ca240 0x7f62180afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 -- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62180ca240 msgr2=0x7f62180afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62180ca240 0x7f62180afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 -- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f621815c520 msgr2=0x7f6218157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f621815c520 0x7f6218157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.053+0000 7f62177fe700 1 -- 172.21.15.50:0/3343149178 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200009420 con 0x7f621815c8f0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f62177fe700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f6210001560 tx=0x7f6210001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210004160 con 0x7f621815c8f0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621000a7e0 con 0x7f621815c8f0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621815cd90 con 0x7f621815c8f0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621000aba0 con 0x7f621815c8f0 2023-12-19T14:58:13.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180b00b0 con 0x7f621815c8f0 2023-12-19T14:58:13.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621815c0a0 con 0x7f621815c8f0 2023-12-19T14:58:13.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.057+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6210012420 con 0x7f621815c8f0 2023-12-19T14:58:13.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.061+0000 7f620f7fe700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 0x7f61f8062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.061+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f621007c8f0 con 0x7f621815c8f0 2023-12-19T14:58:13.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.061+0000 7f6216ffd700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 0x7f61f8062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.061+0000 7f6216ffd700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 0x7f61f8062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6208007fd0 tx=0x7f6208009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.069+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f621004cc80 con 0x7f621815c8f0 2023-12-19T14:58:13.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.245+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f6218071120 con 0x7f621815c8f0 2023-12-19T14:58:13.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.253+0000 7f620f7fe700 1 -- 172.21.15.50:0/3343149178 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v44) v1 ==== 115+0+0 (secure 0 0 0) 0x7f62100502a0 con 0x7f621815c8f0 2023-12-19T14:58:13.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 msgr2=0x7f61f8062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 0x7f61f8062bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6208007fd0 tx=0x7f6208009040 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 msgr2=0x7f621815d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f6210001560 tx=0x7f6210001770 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 shutdown_connections 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f621815c520 0x7f6218157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f61f8060740 0x7f61f8062bf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621815c8f0 0x7f621815d590 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 --2- 172.21.15.50:0/3343149178 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f62180ca240 0x7f62180afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 >> 172.21.15.50:0/3343149178 conn(0x7f6218067bb0 msgr2=0x7f62180cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 shutdown_connections 2023-12-19T14:58:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.257+0000 7f621d8a5700 1 -- 172.21.15.50:0/3343149178 wait complete. 2023-12-19T14:58:13.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T14:58:13.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.449+0000 7f0eda6e3700 1 Processor -- start 2023-12-19T14:58:13.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.449+0000 7f0eda6e3700 1 -- start start 2023-12-19T14:58:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0eda6e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ed406aeb0 0x7f0ed4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0eda6e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ed41595a0 0x7f0ed41599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0eda6e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed406c850 con 0x7f0ed4158c90 2023-12-19T14:58:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0eda6e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40cb650 con 0x7f0ed406aeb0 2023-12-19T14:58:13.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0eda6e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed406b470 con 0x7f0ed41595a0 2023-12-19T14:58:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed96e1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0ed4158c90 0x7f0ed40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42934/0 (socket says 172.21.15.50:42934) 2023-12-19T14:58:13.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0ed406aeb0 0x7f0ed4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed96e1700 1 -- 172.21.15.50:0/1513631342 learned_addr learned my addr 172.21.15.50:0/1513631342 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed9ee2700 1 --2- 172.21.15.50:0/1513631342 >> v2:172.21.15.125:3300/0 conn(0x7f0ed41595a0 0x7f0ed41599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ecbfff700 1 -- 172.21.15.50:0/1513631342 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689788882 0 0) 0x7f0ed406c850 con 0x7f0ed4158c90 2023-12-19T14:58:13.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ecbfff700 1 -- 172.21.15.50:0/1513631342 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec0003570 con 0x7f0ed4158c90 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1513631342 >> v2:172.21.15.125:3300/0 conn(0x7f0ed41595a0 msgr2=0x7f0ed41599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 --2- 172.21.15.50:0/1513631342 >> v2:172.21.15.125:3300/0 conn(0x7f0ed41595a0 0x7f0ed41599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1513631342 >> v1:172.21.15.50:6789/0 conn(0x7f0ed4158c90 legacy=0x7f0ed40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1513631342 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed406b750 con 0x7f0ed406aeb0 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ed8ee0700 1 --2- 172.21.15.50:0/1513631342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed406aeb0 0x7f0ed4159060 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40cc7f0 tx=0x7f0ec40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e3c1bd01fc19d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ecbfff700 1 -- 172.21.15.50:0/1513631342 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4011630 con 0x7f0ed406aeb0 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ecbfff700 1 -- 172.21.15.50:0/1513631342 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec40117b0 con 0x7f0ed406aeb0 2023-12-19T14:58:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.453+0000 7f0ecbfff700 1 -- 172.21.15.50:0/1513631342 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4011b70 con 0x7f0ed406aeb0 2023-12-19T14:58:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1513631342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed406aeb0 msgr2=0x7f0ed4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1513631342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed406aeb0 0x7f0ed4159060 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40cc7f0 tx=0x7f0ec40091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1513631342 shutdown_connections 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1513631342 >> v2:172.21.15.125:3300/0 conn(0x7f0ed41595a0 0x7f0ed41599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1513631342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed406aeb0 0x7f0ed4159060 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1513631342 >> 172.21.15.50:0/1513631342 conn(0x7f0ed4067bb0 msgr2=0x7f0ed40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1513631342 shutdown_connections 2023-12-19T14:58:13.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1513631342 wait complete. 2023-12-19T14:58:13.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.457+0000 7f0eda6e3700 1 Processor -- start 2023-12-19T14:58:13.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 -- start start 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ed406aeb0 0x7f0ed40b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ed41595a0 0x7f0ed40b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b3f50 con 0x7f0ed4158c90 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b3340 con 0x7f0ed41595a0 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0eda6e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b3620 con 0x7f0ed406aeb0 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33252/0 (socket says 172.21.15.50:33252) 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1121696635 learned_addr learned my addr 172.21.15.50:0/1121696635 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:13.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed96e1700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ed406aeb0 0x7f0ed40b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed9ee2700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ed41595a0 0x7f0ed40b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ed406aeb0 msgr2=0x7f0ed40b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ed406aeb0 0x7f0ed40b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ed41595a0 msgr2=0x7f0ed40b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ed41595a0 0x7f0ed40b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 -- 172.21.15.50:0/1121696635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4009420 con 0x7f0ed4158c90 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.461+0000 7f0ed8ee0700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4013000 tx=0x7f0ec4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec401f070 con 0x7f0ed4158c90 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec4000bc0 con 0x7f0ed4158c90 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4011b50 con 0x7f0ed4158c90 2023-12-19T14:58:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed40b3900 con 0x7f0ed4158c90 2023-12-19T14:58:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed4154090 con 0x7f0ed4158c90 2023-12-19T14:58:13.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed40b40c0 con 0x7f0ed4158c90 2023-12-19T14:58:13.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.465+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0ec4023070 con 0x7f0ed4158c90 2023-12-19T14:58:13.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.469+0000 7f0ec97fa700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 0x7f0eb8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.469+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f0ec407dda0 con 0x7f0ed4158c90 2023-12-19T14:58:13.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.469+0000 7f0ed96e1700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 0x7f0eb8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.469+0000 7f0ed96e1700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 0x7f0eb8062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007fd0 tx=0x7f0ed0009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.477+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0ec404e0b0 con 0x7f0ed4158c90 2023-12-19T14:58:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.657+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0ed404be40 con 0x7f0ed4158c90 2023-12-19T14:58:13.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.661+0000 7f0ec97fa700 1 -- 172.21.15.50:0/1121696635 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0ec40516d0 con 0x7f0ed4158c90 2023-12-19T14:58:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 msgr2=0x7f0eb8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 0x7f0eb8062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007fd0 tx=0x7f0ed0009040 comp rx=0 tx=0).stop 2023-12-19T14:58:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 msgr2=0x7f0ed40b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4013000 tx=0x7f0ec4017000 comp rx=0 tx=0).stop 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 shutdown_connections 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ed41595a0 0x7f0ed40b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0eb8060670 0x7f0eb8062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ed4158c90 0x7f0ed40b0a50 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 --2- 172.21.15.50:0/1121696635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ed406aeb0 0x7f0ed40b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.665+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 >> 172.21.15.50:0/1121696635 conn(0x7f0ed4067bb0 msgr2=0x7f0ed406a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.669+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 shutdown_connections 2023-12-19T14:58:13.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.669+0000 7f0eda6e3700 1 -- 172.21.15.50:0/1121696635 wait complete. 2023-12-19T14:58:13.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-12-19T14:58:13.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.857+0000 7f97958cd700 1 Processor -- start 2023-12-19T14:58:13.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.857+0000 7f97958cd700 1 -- start start 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9790159ba0 0x7f97900670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f979015a4b0 0x7f97901558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900cb8a0 con 0x7f979015a4b0 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9790067620 con 0x7f9790159ba0 2023-12-19T14:58:13.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f97958cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97900cac90 con 0x7f97900ca420 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f979015a4b0 0x7f97901558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978e7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55408/0 (socket says 172.21.15.50:55408) 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f979015a4b0 0x7f97901558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33254/0 (socket says 172.21.15.50:33254) 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978e7fc700 1 -- 172.21.15.50:0/41389794 learned_addr learned my addr 172.21.15.50:0/41389794 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:13.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978effd700 1 --2- 172.21.15.50:0/41389794 >> v2:172.21.15.125:3300/0 conn(0x7f9790159ba0 0x7f97900670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 -- 172.21.15.50:0/41389794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 msgr2=0x7f9790159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 -- 172.21.15.50:0/41389794 >> v2:172.21.15.125:3300/0 conn(0x7f9790159ba0 msgr2=0x7f97900670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 --2- 172.21.15.50:0/41389794 >> v2:172.21.15.125:3300/0 conn(0x7f9790159ba0 0x7f97900670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 -- 172.21.15.50:0/41389794 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900caf70 con 0x7f979015a4b0 2023-12-19T14:58:13.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978e7fc700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978f7fe700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f979015a4b0 0x7f97901558b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f979006a940 tx=0x7f9788009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149a2da559ed3920 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978dffb700 1 -- 172.21.15.50:0/41389794 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978800e070 con 0x7f979015a4b0 2023-12-19T14:58:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978dffb700 1 -- 172.21.15.50:0/41389794 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978800bab0 con 0x7f979015a4b0 2023-12-19T14:58:13.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.861+0000 7f978dffb700 1 -- 172.21.15.50:0/41389794 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978800be70 con 0x7f979015a4b0 2023-12-19T14:58:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- 172.21.15.50:0/41389794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f979015a4b0 msgr2=0x7f97901558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f979015a4b0 0x7f97901558b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f979006a940 tx=0x7f9788009780 comp rx=0 tx=0).stop 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- 172.21.15.50:0/41389794 shutdown_connections 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f979015a4b0 0x7f97901558b0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 --2- 172.21.15.50:0/41389794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97900ca420 0x7f9790159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 --2- 172.21.15.50:0/41389794 >> v2:172.21.15.125:3300/0 conn(0x7f9790159ba0 0x7f97900670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- 172.21.15.50:0/41389794 >> 172.21.15.50:0/41389794 conn(0x7f97900cd990 msgr2=0x7f97900ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- 172.21.15.50:0/41389794 shutdown_connections 2023-12-19T14:58:13.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- 172.21.15.50:0/41389794 wait complete. 2023-12-19T14:58:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 Processor -- start 2023-12-19T14:58:13.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.865+0000 7f97958cd700 1 -- start start 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9790159ba0 0x7f9790156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f979015a4b0 0x7f97901571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97900c6300 con 0x7f97900ca420 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97901576e0 con 0x7f979015a4b0 2023-12-19T14:58:13.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9790152b50 con 0x7f9790159ba0 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33262/0 (socket says 172.21.15.50:33262) 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 -- 172.21.15.50:0/1605668056 learned_addr learned my addr 172.21.15.50:0/1605668056 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978f7fe700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f979015a4b0 0x7f97901571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978e7fc700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9790159ba0 0x7f9790156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 -- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9790159ba0 msgr2=0x7f9790156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9790159ba0 0x7f9790156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 -- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f979015a4b0 msgr2=0x7f97901571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f979015a4b0 0x7f97901571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 -- 172.21.15.50:0/1605668056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97840094f0 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f978effd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f978000ec80 tx=0x7f97800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978000c9d0 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978000cb30 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9788009a20 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780013470 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.869+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9790153110 con 0x7f97900ca420 2023-12-19T14:58:13.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.873+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9790071120 con 0x7f97900ca420 2023-12-19T14:58:13.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.873+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9780013820 con 0x7f97900ca420 2023-12-19T14:58:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.873+0000 7f9776ffd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 0x7f977c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:13.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.873+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f978007ce50 con 0x7f97900ca420 2023-12-19T14:58:13.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.873+0000 7f978e7fc700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 0x7f977c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:13.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.877+0000 7f978e7fc700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 0x7f977c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97900c8f40 tx=0x7f9784008040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:13.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:13.877+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f978004d160 con 0x7f97900ca420 2023-12-19T14:58:14.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.113+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f97900cb250 con 0x7f97900ca420 2023-12-19T14:58:14.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.441+0000 7f9776ffd700 1 -- 172.21.15.50:0/1605668056 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v733) v1 ==== 148+0+0 (secure 0 0 0) 0x7f9780050780 con 0x7f97900ca420 2023-12-19T14:58:14.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 msgr2=0x7f977c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 0x7f977c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97900c8f40 tx=0x7f9784008040 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 msgr2=0x7f9790153bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f978000ec80 tx=0x7f97800091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 shutdown_connections 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f979015a4b0 0x7f97901571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f977c060670 0x7f977c062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9790159ba0 0x7f9790156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 --2- 172.21.15.50:0/1605668056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97900ca420 0x7f9790153bf0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 >> 172.21.15.50:0/1605668056 conn(0x7f97900cd990 msgr2=0x7f97900acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 shutdown_connections 2023-12-19T14:58:14.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.445+0000 7f97958cd700 1 -- 172.21.15.50:0/1605668056 wait complete. 2023-12-19T14:58:14.473 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-12-19T14:58:14.473 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-12-19T14:58:14.473 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T14:58:14.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:14.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.641+0000 7f453b8f0700 1 Processor -- start 2023-12-19T14:58:14.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 -- start start 2023-12-19T14:58:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4534159ba0 0x7f45340670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 0x7f4534155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45340cb8a0 con 0x7f45340ca420 2023-12-19T14:58:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4534067620 con 0x7f4534159ba0 2023-12-19T14:58:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453b8f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4534156250 con 0x7f45341558b0 2023-12-19T14:58:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4534159ba0 0x7f45340670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4534159ba0 0x7f45340670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33268/0 (socket says 172.21.15.50:33268) 2023-12-19T14:58:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 -- 172.21.15.50:0/3764149254 learned_addr learned my addr 172.21.15.50:0/3764149254 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f4539e8d700 1 --2- 172.21.15.50:0/3764149254 >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 0x7f4534155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 -- 172.21.15.50:0/3764149254 >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 msgr2=0x7f4534155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 --2- 172.21.15.50:0/3764149254 >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 0x7f4534155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 -- 172.21.15.50:0/3764149254 >> v1:172.21.15.125:6789/0 conn(0x7f45340ca420 legacy=0x7f4534159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f453968c700 1 -- 172.21.15.50:0/3764149254 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340cac90 con 0x7f4534159ba0 2023-12-19T14:58:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.645+0000 7f4539e8d700 1 --2- 172.21.15.50:0/3764149254 >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 0x7f4534155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453968c700 1 --2- 172.21.15.50:0/3764149254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4534159ba0 0x7f45340670e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f45300099b0 tx=0x7f4530009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1faa185f358a5c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f452bfff700 1 -- 172.21.15.50:0/3764149254 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453000b820 con 0x7f4534159ba0 2023-12-19T14:58:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f452bfff700 1 -- 172.21.15.50:0/3764149254 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453000b980 con 0x7f4534159ba0 2023-12-19T14:58:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f452bfff700 1 -- 172.21.15.50:0/3764149254 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453000bd20 con 0x7f4534159ba0 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 -- 172.21.15.50:0/3764149254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4534159ba0 msgr2=0x7f45340670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 --2- 172.21.15.50:0/3764149254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4534159ba0 0x7f45340670e0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f45300099b0 tx=0x7f4530009250 comp rx=0 tx=0).stop 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 -- 172.21.15.50:0/3764149254 shutdown_connections 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 --2- 172.21.15.50:0/3764149254 >> v2:172.21.15.125:3300/0 conn(0x7f45341558b0 0x7f4534155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 --2- 172.21.15.50:0/3764149254 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4534159ba0 0x7f45340670e0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 -- 172.21.15.50:0/3764149254 >> 172.21.15.50:0/3764149254 conn(0x7f45340cd990 msgr2=0x7f45340ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 -- 172.21.15.50:0/3764149254 shutdown_connections 2023-12-19T14:58:14.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.649+0000 7f453b8f0700 1 -- 172.21.15.50:0/3764149254 wait complete. 2023-12-19T14:58:14.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 Processor -- start 2023-12-19T14:58:14.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 -- start start 2023-12-19T14:58:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45341558b0 0x7f45341448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4534159ba0 0x7f45340ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4534158930 con 0x7f45340ca420 2023-12-19T14:58:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4534159300 con 0x7f4534159ba0 2023-12-19T14:58:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453b8f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4534157d20 con 0x7f45341558b0 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33282/0 (socket says 172.21.15.50:33282) 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 -- 172.21.15.50:0/1651798929 learned_addr learned my addr 172.21.15.50:0/1651798929 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f4539e8d700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4534159ba0 0x7f45340ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f4538e8b700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45341558b0 0x7f45341448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 -- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45341558b0 msgr2=0x7f45341448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45341558b0 0x7f45341448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 -- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4534159ba0 msgr2=0x7f45340ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4534159ba0 0x7f45340ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.653+0000 7f453968c700 1 -- 172.21.15.50:0/1651798929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45300094f0 con 0x7f45340ca420 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f453968c700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4530009290 tx=0x7f4530017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530026480 con 0x7f45340ca420 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453001f070 con 0x7f45340ca420 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4534158000 con 0x7f45340ca420 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530026870 con 0x7f45340ca420 2023-12-19T14:58:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45341585c0 con 0x7f45340ca420 2023-12-19T14:58:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.657+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4534071120 con 0x7f45340ca420 2023-12-19T14:58:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.661+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4530026be0 con 0x7f45340ca420 2023-12-19T14:58:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.661+0000 7f45297fa700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 0x7f4518062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.661+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f4530021070 con 0x7f45340ca420 2023-12-19T14:58:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.661+0000 7f4538e8b700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 0x7f4518062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.665+0000 7f4538e8b700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 0x7f4518062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f45340c95e0 tx=0x7f4524009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.665+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f453004ec10 con 0x7f45340ca420 2023-12-19T14:58:14.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.841+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45340caf70 con 0x7f45340ca420 2023-12-19T14:58:14.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.841+0000 7f45297fa700 1 -- 172.21.15.50:0/1651798929 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+13570 (secure 0 0 0) 0x7f453002b090 con 0x7f45340ca420 2023-12-19T14:58:14.846 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:14.846 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":135,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:12.418427+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/1300068976":"2023-12-20T14:58:09.395960+0000","172.21.15.125:0/2574956284":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/3845778139":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1048412071":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1114305366":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1072406585":"2023-12-20T14:58:08.802830+0000","172.21.15.50:6834/1300068976":"2023-12-20T14:58:09.395960+0000","172.21.15.125:0/4291635214":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/2353401000":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2989287076":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1628872297":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/2571945050":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/4156266535":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/1080158370":"2023-12-20T14:58:08.802830+0000","172.21.15.125:6833/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:6832/33827":"2023-12-20T14:57:27.688521+0000","172.21.15.125:0/3396238407":"2023-12-20T14:58:08.802830+0000","172.21.15.125:0/1370626584":"2023-12-20T14:57:27.688521+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:14.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 msgr2=0x7f4518062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 0x7f4518062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f45340c95e0 tx=0x7f4524009040 comp rx=0 tx=0).stop 2023-12-19T14:58:14.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 msgr2=0x7f4534158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:14.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4530009290 tx=0x7f4530017000 comp rx=0 tx=0).stop 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 shutdown_connections 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4534159ba0 0x7f45340ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4518060670 0x7f4518062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45341558b0 0x7f45341448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 --2- 172.21.15.50:0/1651798929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45340ca420 0x7f4534158dc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 >> 172.21.15.50:0/1651798929 conn(0x7f45340cd990 msgr2=0x7f45340b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 shutdown_connections 2023-12-19T14:58:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:14.845+0000 7f453b8f0700 1 -- 172.21.15.50:0/1651798929 wait complete. 2023-12-19T14:58:14.872 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:58:14.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:58:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.041+0000 7fc7c9f45700 1 Processor -- start 2023-12-19T14:58:15.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.041+0000 7fc7c9f45700 1 -- start start 2023-12-19T14:58:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c9f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7c40cb970 0x7fc7c406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c9f45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c40cbdf0 con 0x7fc7c40cecd0 2023-12-19T14:58:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c9f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c40cc0d0 con 0x7fc7c4158e20 2023-12-19T14:58:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c9f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c40cc3b0 con 0x7fc7c40cb970 2023-12-19T14:58:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc7c40cecd0 0x7fc7c406b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42938/0 (socket says 172.21.15.50:42938) 2023-12-19T14:58:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c3fff700 1 -- 172.21.15.50:0/2093258794 learned_addr learned my addr 172.21.15.50:0/2093258794 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:15.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/2093258794 >> v2:172.21.15.125:3300/0 conn(0x7fc7c40cb970 0x7fc7c406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456292354 0 0) 0x7fc7c40cbdf0 con 0x7fc7c40cecd0 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac003570 con 0x7fc7c40cecd0 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496407906 0 0) 0x7fc7c40cc0d0 con 0x7fc7c4158e20 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c40cbdf0 con 0x7fc7c4158e20 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2676506292 0 0) 0x7fc7ac003570 con 0x7fc7c40cecd0 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 >> v2:172.21.15.125:3300/0 conn(0x7fc7c40cb970 msgr2=0x7fc7c406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 --2- 172.21.15.50:0/2093258794 >> v2:172.21.15.125:3300/0 conn(0x7fc7c40cb970 0x7fc7c406acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 >> v1:172.21.15.125:6789/0 conn(0x7fc7c4158e20 legacy=0x7fc7c40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c40cc690 con 0x7fc7c40cecd0 2023-12-19T14:58:15.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fc7bc0017f0 con 0x7fc7c40cecd0 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 >> v1:172.21.15.50:6789/0 conn(0x7fc7c40cecd0 legacy=0x7fc7c406b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 0x7fc7ac004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac004810 0x7fc7ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7ac0051c0 0x7fc7ac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40cc3b0 con 0x7fc7ac003c20 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac00c1e0 con 0x7fc7ac004810 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac00c3a0 con 0x7fc7ac0051c0 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.045+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/2093258794 >> v2:172.21.15.125:3300/0 conn(0x7fc7c40cb970 0x7fc7c406acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 0x7fc7ac004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac004810 0x7fc7ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c3fff700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7ac0051c0 0x7fc7ac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 -- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7ac0051c0 msgr2=0x7fc7ac00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7ac0051c0 0x7fc7ac00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 -- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac004810 msgr2=0x7fc7ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac004810 0x7fc7ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 -- 172.21.15.50:0/2093258794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac003570 con 0x7fc7ac003c20 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac004810 0x7fc7ac004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c3fff700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7ac0051c0 0x7fc7ac00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c3fff700 1 -- 172.21.15.50:0/2093258794 reap_dead start 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 0x7fc7ac004010 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40019e0 tx=0x7fc7b4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19898518c111321 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400c5d0 con 0x7fc7ac003c20 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b400c730 con 0x7fc7ac003c20 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c27fc700 1 -- 172.21.15.50:0/2093258794 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400cad0 con 0x7fc7ac003c20 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 -- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 msgr2=0x7fc7ac004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 0x7fc7ac004010 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40019e0 tx=0x7fc7b4007d30 comp rx=0 tx=0).stop 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 -- 172.21.15.50:0/2093258794 shutdown_connections 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/2093258794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7ac003c20 0x7fc7ac004010 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 -- 172.21.15.50:0/2093258794 >> 172.21.15.50:0/2093258794 conn(0x7fc7c4067bb0 msgr2=0x7fc7c40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 -- 172.21.15.50:0/2093258794 shutdown_connections 2023-12-19T14:58:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.049+0000 7fc7c9f45700 1 -- 172.21.15.50:0/2093258794 wait complete. 2023-12-19T14:58:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 Processor -- start 2023-12-19T14:58:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 -- start start 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac003c20 0x7fc7c4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7c40c80a0 0x7fc7c4159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415ceb0 con 0x7fc7c40c71d0 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40c8510 con 0x7fc7ac003c20 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c9f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c415a840 con 0x7fc7c40c80a0 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c2ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33308/0 (socket says 172.21.15.50:33308) 2023-12-19T14:58:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c2ffd700 1 -- 172.21.15.50:0/405271897 learned_addr learned my addr 172.21.15.50:0/405271897 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c3fff700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7c40c80a0 0x7fc7c4159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac003c20 0x7fc7c4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c2ffd700 1 -- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7c40c80a0 msgr2=0x7fc7c4159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.053+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7c40c80a0 0x7fc7c4159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c2ffd700 1 -- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac003c20 msgr2=0x7fc7c4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac003c20 0x7fc7c4150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c2ffd700 1 -- 172.21.15.50:0/405271897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c415ab20 con 0x7fc7c40c71d0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c2ffd700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8011ab0 tx=0x7fc7b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b800f9b0 con 0x7fc7c40c71d0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b800fb10 con 0x7fc7c40c71d0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c415ae00 con 0x7fc7c40c71d0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b800fed0 con 0x7fc7c40c71d0 2023-12-19T14:58:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c4169400 con 0x7fc7c40c71d0 2023-12-19T14:58:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.057+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c4071120 con 0x7fc7c40c71d0 2023-12-19T14:58:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.061+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc7b8015670 con 0x7fc7c40c71d0 2023-12-19T14:58:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.061+0000 7fc7ab7fe700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 0x7fc7b0062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.061+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7fc7b807d630 con 0x7fc7c40c71d0 2023-12-19T14:58:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.061+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 0x7fc7b0062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.061+0000 7fc7c37fe700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 0x7fc7b0062bf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40019e0 tx=0x7fc7b4015000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.065+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc7b804d9c0 con 0x7fc7c40c71d0 2023-12-19T14:58:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.241+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7c40cd430 con 0x7fc7c40c71d0 2023-12-19T14:58:15.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7ab7fe700 1 -- 172.21.15.50:0/405271897 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7b8050fe0 con 0x7fc7c40c71d0 2023-12-19T14:58:15.247 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 msgr2=0x7fc7b0062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 0x7fc7b0062bf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40019e0 tx=0x7fc7b4015000 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 msgr2=0x7fc7c40c7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8011ab0 tx=0x7fc7b8008040 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 shutdown_connections 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc7b0060740 0x7fc7b0062bf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc7c40c80a0 0x7fc7c4159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7c40c71d0 0x7fc7c40c7b60 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 --2- 172.21.15.50:0/405271897 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7ac003c20 0x7fc7c4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 >> 172.21.15.50:0/405271897 conn(0x7fc7c4067bb0 msgr2=0x7fc7c406b400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 shutdown_connections 2023-12-19T14:58:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.245+0000 7fc7c9f45700 1 -- 172.21.15.50:0/405271897 wait complete. 2023-12-19T14:58:15.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-12-19T14:58:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 Processor -- start 2023-12-19T14:58:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 -- start start 2023-12-19T14:58:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2a74158f70 0x7f2a74067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a74069100 0x7f2a740cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a74068c70 con 0x7f2a74159340 2023-12-19T14:58:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a74068060 con 0x7f2a74069100 2023-12-19T14:58:15.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a7a185700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a74068340 con 0x7f2a74158f70 2023-12-19T14:58:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a73fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2a74159340 0x7f2a740cb3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42948/0 (socket says 172.21.15.50:42948) 2023-12-19T14:58:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a73fff700 1 -- 172.21.15.50:0/2754609905 learned_addr learned my addr 172.21.15.50:0/2754609905 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a72ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2a74069100 0x7f2a740cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.433+0000 7f2a737fe700 1 --2- 172.21.15.50:0/2754609905 >> v2:172.21.15.125:3300/0 conn(0x7f2a74158f70 0x7f2a74067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.437+0000 7f2a737fe700 1 -- 172.21.15.50:0/2754609905 >> v2:172.21.15.50:3300/0 conn(0x7f2a74069100 msgr2=0x7f2a740cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.437+0000 7f2a737fe700 1 --2- 172.21.15.50:0/2754609905 >> v2:172.21.15.50:3300/0 conn(0x7f2a74069100 0x7f2a740cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.437+0000 7f2a737fe700 1 -- 172.21.15.50:0/2754609905 >> v1:172.21.15.50:6789/0 conn(0x7f2a74159340 legacy=0x7f2a740cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.437+0000 7f2a737fe700 1 -- 172.21.15.50:0/2754609905 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74068620 con 0x7f2a74158f70 2023-12-19T14:58:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.437+0000 7f2a737fe700 1 --2- 172.21.15.50:0/2754609905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a74067980 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098e0 tx=0x7f2a640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8000b07fda869232 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.441+0000 7f2a727fc700 1 -- 172.21.15.50:0/2754609905 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64011630 con 0x7f2a74158f70 2023-12-19T14:58:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.441+0000 7f2a727fc700 1 -- 172.21.15.50:0/2754609905 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64011790 con 0x7f2a74158f70 2023-12-19T14:58:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.441+0000 7f2a727fc700 1 -- 172.21.15.50:0/2754609905 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64011b30 con 0x7f2a74158f70 2023-12-19T14:58:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 -- 172.21.15.50:0/2754609905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 msgr2=0x7f2a74067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 --2- 172.21.15.50:0/2754609905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a74067980 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a640098e0 tx=0x7f2a640091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 -- 172.21.15.50:0/2754609905 shutdown_connections 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 --2- 172.21.15.50:0/2754609905 >> v2:172.21.15.50:3300/0 conn(0x7f2a74069100 0x7f2a740cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 --2- 172.21.15.50:0/2754609905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a74067980 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 -- 172.21.15.50:0/2754609905 >> 172.21.15.50:0/2754609905 conn(0x7f2a740ccc40 msgr2=0x7f2a740cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 -- 172.21.15.50:0/2754609905 shutdown_connections 2023-12-19T14:58:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 -- 172.21.15.50:0/2754609905 wait complete. 2023-12-19T14:58:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.445+0000 7f2a7a185700 1 Processor -- start 2023-12-19T14:58:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 -- start start 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a74069100 0x7f2a740b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a7414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740b59e0 con 0x7f2a74159340 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7416d3a0 con 0x7f2a74158f70 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740b5700 con 0x7f2a74069100 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a72ffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a7414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33312/0 (socket says 172.21.15.50:33312) 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 -- 172.21.15.50:0/1598398477 learned_addr learned my addr 172.21.15.50:0/1598398477 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a737fe700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a74069100 0x7f2a740b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 -- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a74069100 msgr2=0x7f2a740b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a74069100 0x7f2a740b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 -- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 msgr2=0x7f2a7414c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a7414c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 -- 172.21.15.50:0/1598398477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64009420 con 0x7f2a74159340 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a73fff700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00cbc0 tx=0x7f2a6c001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c00a7a0 con 0x7f2a74159340 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c00a900 con 0x7f2a74159340 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7416cd50 con 0x7f2a74159340 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.449+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c00ac80 con 0x7f2a74159340 2023-12-19T14:58:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.453+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a740b53f0 con 0x7f2a74159340 2023-12-19T14:58:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.453+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7416cbb0 con 0x7f2a74159340 2023-12-19T14:58:15.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.453+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2a6c007300 con 0x7f2a74159340 2023-12-19T14:58:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.453+0000 7f2a5b7fe700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 0x7f2a60062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.453+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f2a6c07cd50 con 0x7f2a74159340 2023-12-19T14:58:15.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.457+0000 7f2a737fe700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 0x7f2a60062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.457+0000 7f2a737fe700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 0x7f2a60062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a64017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.461+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2a7416cbb0 con 0x7f2a74159340 2023-12-19T14:58:15.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:15.693+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f2a7416cbb0 con 0x7f2a74159340 2023-12-19T14:58:16.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.453+0000 7f2a5b7fe700 1 -- 172.21.15.50:0/1598398477 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v735) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2a7416cbb0 con 0x7f2a74159340 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 msgr2=0x7f2a60062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 0x7f2a60062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a64017000 comp rx=0 tx=0).stop 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 msgr2=0x7f2a7415ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00cbc0 tx=0x7f2a6c001990 comp rx=0 tx=0).stop 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 shutdown_connections 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a74158f70 0x7f2a7414c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2a600606a0 0x7f2a60062b50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a74159340 0x7f2a7415ee60 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 --2- 172.21.15.50:0/1598398477 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2a74069100 0x7f2a740b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 >> 172.21.15.50:0/1598398477 conn(0x7f2a740ccc40 msgr2=0x7f2a740c6010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:16.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 shutdown_connections 2023-12-19T14:58:16.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.457+0000 7f2a7a185700 1 -- 172.21.15.50:0/1598398477 wait complete. 2023-12-19T14:58:16.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T14:58:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.665+0000 7ff76c88e700 1 Processor -- start 2023-12-19T14:58:16.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.665+0000 7ff76c88e700 1 -- start start 2023-12-19T14:58:16.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff76c88e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff7680c53f0 0x7ff76815c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:16.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff76c88e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff768159b80 con 0x7ff768141e70 2023-12-19T14:58:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff76c88e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7680c8220 con 0x7ff76815a010 2023-12-19T14:58:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff76c88e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7680c7f40 con 0x7ff7680c53f0 2023-12-19T14:58:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff7677fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff768141e70 0x7ff7680c7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42162/0 (socket says 172.21.15.50:42162) 2023-12-19T14:58:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff7677fe700 1 -- 172.21.15.50:0/216533529 learned_addr learned my addr 172.21.15.50:0/216533529 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:16.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.669+0000 7ff767fff700 1 --2- 172.21.15.50:0/216533529 >> v2:172.21.15.125:3300/0 conn(0x7ff7680c53f0 0x7ff76815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff766ffd700 1 -- 172.21.15.50:0/216533529 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143391893 0 0) 0x7ff768159b80 con 0x7ff768141e70 2023-12-19T14:58:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff766ffd700 1 -- 172.21.15.50:0/216533529 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff754003570 con 0x7ff768141e70 2023-12-19T14:58:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff767fff700 1 -- 172.21.15.50:0/216533529 >> v1:172.21.15.125:6789/0 conn(0x7ff76815a010 legacy=0x7ff7680c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff767fff700 1 -- 172.21.15.50:0/216533529 >> v1:172.21.15.50:6789/0 conn(0x7ff768141e70 legacy=0x7ff7680c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff767fff700 1 -- 172.21.15.50:0/216533529 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff768159530 con 0x7ff7680c53f0 2023-12-19T14:58:16.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff767fff700 1 --2- 172.21.15.50:0/216533529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7680c53f0 0x7ff76815c970 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff760009e60 tx=0x7ff760009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=468172f61f406c07 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:16.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff766ffd700 1 -- 172.21.15.50:0/216533529 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76000e070 con 0x7ff7680c53f0 2023-12-19T14:58:16.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff766ffd700 1 -- 172.21.15.50:0/216533529 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76000bb60 con 0x7ff7680c53f0 2023-12-19T14:58:16.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff766ffd700 1 -- 172.21.15.50:0/216533529 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760012430 con 0x7ff7680c53f0 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 -- 172.21.15.50:0/216533529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7680c53f0 msgr2=0x7ff76815c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 --2- 172.21.15.50:0/216533529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7680c53f0 0x7ff76815c970 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff760009e60 tx=0x7ff760009780 comp rx=0 tx=0).stop 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 -- 172.21.15.50:0/216533529 shutdown_connections 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 --2- 172.21.15.50:0/216533529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7680c53f0 0x7ff76815c970 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 -- 172.21.15.50:0/216533529 >> 172.21.15.50:0/216533529 conn(0x7ff768067b50 msgr2=0x7ff7680cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 -- 172.21.15.50:0/216533529 shutdown_connections 2023-12-19T14:58:16.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.673+0000 7ff76c88e700 1 -- 172.21.15.50:0/216533529 wait complete. 2023-12-19T14:58:16.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.677+0000 7ff76c88e700 1 Processor -- start 2023-12-19T14:58:16.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.677+0000 7ff76c88e700 1 -- start start 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 0x7ff7680afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 0x7ff7680aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76815dc00 con 0x7ff7680c53f0 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76813de80 con 0x7ff768141e70 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76813d5e0 con 0x7ff76815a010 2023-12-19T14:58:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39122/0 (socket says 172.21.15.50:39122) 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 -- 172.21.15.50:0/789410838 learned_addr learned my addr 172.21.15.50:0/789410838 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff767fff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 0x7ff7680aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff75ffff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 0x7ff7680afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 -- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 msgr2=0x7ff7680aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 0x7ff7680aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 -- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 msgr2=0x7ff7680afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 0x7ff7680afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 -- 172.21.15.50:0/789410838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff760009a20 con 0x7ff7680c53f0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff767fff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 0x7ff7680aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff75ffff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 0x7ff7680afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff7677fe700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff75800c7c0 tx=0x7ff758007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758005b30 con 0x7ff7680c53f0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff758005c90 con 0x7ff7680c53f0 2023-12-19T14:58:16.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76813d8c0 con 0x7ff7680c53f0 2023-12-19T14:58:16.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758007340 con 0x7ff7680c53f0 2023-12-19T14:58:16.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.681+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff768158aa0 con 0x7ff7680c53f0 2023-12-19T14:58:16.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.685+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff768141a70 con 0x7ff7680c53f0 2023-12-19T14:58:16.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.685+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff75801b400 con 0x7ff7680c53f0 2023-12-19T14:58:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.689+0000 7ff75f7fe700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 0x7ff748062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.689+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7ff75807c580 con 0x7ff7680c53f0 2023-12-19T14:58:16.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.689+0000 7ff75ffff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 0x7ff748062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:16.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.689+0000 7ff75ffff700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 0x7ff748062bd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff76815f8e0 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:16.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.693+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff768141a70 con 0x7ff7680c53f0 2023-12-19T14:58:16.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.869+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff768141a70 con 0x7ff7680c53f0 2023-12-19T14:58:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.877+0000 7ff75f7fe700 1 -- 172.21.15.50:0/789410838 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff75804c910 con 0x7ff7680c53f0 2023-12-19T14:58:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.877+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 msgr2=0x7ff748062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.877+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 0x7ff748062bd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff76815f8e0 tx=0x7ff750009040 comp rx=0 tx=0).stop 2023-12-19T14:58:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.877+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 msgr2=0x7ff768158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.877+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff75800c7c0 tx=0x7ff758007e00 comp rx=0 tx=0).stop 2023-12-19T14:58:16.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 shutdown_connections 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff768141e70 0x7ff7680afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff748060720 0x7ff748062bd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff76815a010 0x7ff7680aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 --2- 172.21.15.50:0/789410838 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7680c53f0 0x7ff768158560 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 >> 172.21.15.50:0/789410838 conn(0x7ff768067b50 msgr2=0x7ff76806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 shutdown_connections 2023-12-19T14:58:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:16.881+0000 7ff76c88e700 1 -- 172.21.15.50:0/789410838 wait complete. 2023-12-19T14:58:16.906 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T14:58:16.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:58:16.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T14:58:17.000 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T14:58:17.000 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T14:58:17.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 Processor -- start 2023-12-19T14:58:17.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 -- start start 2023-12-19T14:58:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc564140710 0x7fc5640c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 0x7fc5640680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56406e150 con 0x7fc5640c75e0 2023-12-19T14:58:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56406d540 con 0x7fc564140710 2023-12-19T14:58:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc569d45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56406d820 con 0x7fc564140ae0 2023-12-19T14:58:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc5637fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc564140710 0x7fc5640c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc563fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc5640c75e0 0x7fc56406a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42174/0 (socket says 172.21.15.50:42174) 2023-12-19T14:58:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc563fff700 1 -- 172.21.15.50:0/2868226750 learned_addr learned my addr 172.21.15.50:0/2868226750 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.161+0000 7fc562ffd700 1 --2- 172.21.15.50:0/2868226750 >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 0x7fc5640680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5637fe700 1 -- 172.21.15.50:0/2868226750 >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 msgr2=0x7fc5640680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5637fe700 1 --2- 172.21.15.50:0/2868226750 >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 0x7fc5640680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5637fe700 1 -- 172.21.15.50:0/2868226750 >> v1:172.21.15.50:6789/0 conn(0x7fc5640c75e0 legacy=0x7fc56406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5637fe700 1 -- 172.21.15.50:0/2868226750 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56406db00 con 0x7fc564140710 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc562ffd700 1 --2- 172.21.15.50:0/2868226750 >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 0x7fc5640680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:17.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5627fc700 1 -- 172.21.15.50:0/2868226750 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657031060 0 0) 0x7fc56406e150 con 0x7fc5640c75e0 2023-12-19T14:58:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5637fe700 1 --2- 172.21.15.50:0/2868226750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc564140710 0x7fc5640c70a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc5640c9960 tx=0x7fc5540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9e1e85e3572ba2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5627fc700 1 -- 172.21.15.50:0/2868226750 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554011630 con 0x7fc564140710 2023-12-19T14:58:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5627fc700 1 -- 172.21.15.50:0/2868226750 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc554011790 con 0x7fc564140710 2023-12-19T14:58:17.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc5627fc700 1 -- 172.21.15.50:0/2868226750 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554011b30 con 0x7fc564140710 2023-12-19T14:58:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 -- 172.21.15.50:0/2868226750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc564140710 msgr2=0x7fc5640c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 --2- 172.21.15.50:0/2868226750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc564140710 0x7fc5640c70a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc5640c9960 tx=0x7fc5540091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 -- 172.21.15.50:0/2868226750 shutdown_connections 2023-12-19T14:58:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 --2- 172.21.15.50:0/2868226750 >> v2:172.21.15.125:3300/0 conn(0x7fc564140ae0 0x7fc5640680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 --2- 172.21.15.50:0/2868226750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc564140710 0x7fc5640c70a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 -- 172.21.15.50:0/2868226750 >> 172.21.15.50:0/2868226750 conn(0x7fc5640c8850 msgr2=0x7fc5640c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 -- 172.21.15.50:0/2868226750 shutdown_connections 2023-12-19T14:58:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.165+0000 7fc569d45700 1 -- 172.21.15.50:0/2868226750 wait complete. 2023-12-19T14:58:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 Processor -- start 2023-12-19T14:58:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 -- start start 2023-12-19T14:58:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc564140710 0x7fc5640b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc564140ae0 0x7fc5640aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5641697c0 con 0x7fc5640c75e0 2023-12-19T14:58:17.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc564168e90 con 0x7fc564140ae0 2023-12-19T14:58:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc569d45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc564169170 con 0x7fc564140710 2023-12-19T14:58:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc5637fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc5637fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39136/0 (socket says 172.21.15.50:39136) 2023-12-19T14:58:17.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.169+0000 7fc5637fe700 1 -- 172.21.15.50:0/2256903054 learned_addr learned my addr 172.21.15.50:0/2256903054 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc562ffd700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc564140710 0x7fc5640b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 -- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc564140710 msgr2=0x7fc5640b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc564140710 0x7fc5640b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 -- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc564140ae0 msgr2=0x7fc5640aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc564140ae0 0x7fc5640aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 -- 172.21.15.50:0/2256903054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558009420 con 0x7fc5640c75e0 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc5637fe700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc56406a280 tx=0x7fc554017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554011b10 con 0x7fc5640c75e0 2023-12-19T14:58:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5540253f0 con 0x7fc5640c75e0 2023-12-19T14:58:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554025790 con 0x7fc5640c75e0 2023-12-19T14:58:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc554009420 con 0x7fc5640c75e0 2023-12-19T14:58:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56415a600 con 0x7fc5640c75e0 2023-12-19T14:58:17.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.173+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564071120 con 0x7fc5640c75e0 2023-12-19T14:58:17.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.177+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc554025b80 con 0x7fc5640c75e0 2023-12-19T14:58:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.177+0000 7fc54b7fe700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 0x7fc550062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.177+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7fc55407e3f0 con 0x7fc5640c75e0 2023-12-19T14:58:17.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.177+0000 7fc562ffd700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 0x7fc550062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.181+0000 7fc562ffd700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 0x7fc550062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5580093f0 tx=0x7fc558008040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.185+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc55404e700 con 0x7fc5640c75e0 2023-12-19T14:58:17.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.389+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5640642a0 con 0x7fc5640c75e0 2023-12-19T14:58:17.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc54b7fe700 1 -- 172.21.15.50:0/2256903054 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3532 (secure 0 0 0) 0x7fc554015de0 con 0x7fc5640c75e0 2023-12-19T14:58:17.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:17.396 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/354033464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":354033464},{"type":"v1","addr":"172.21.15.125:6836","nonce":354033464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":6998,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2019383748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2019383748},{"type":"v1","addr":"172.21.15.125:6837","nonce":2019383748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/4132279064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":4132279064},{"type":"v1","addr":"172.21.15.50:6833","nonce":4132279064}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53},{"gid":8268,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3763554924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3763554924},{"type":"v1","addr":"172.21.15.50:6835","nonce":3763554924}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[]} 2023-12-19T14:58:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 msgr2=0x7fc550062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 0x7fc550062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5580093f0 tx=0x7fc558008040 comp rx=0 tx=0).stop 2023-12-19T14:58:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 msgr2=0x7fc56415a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc56406a280 tx=0x7fc554017000 comp rx=0 tx=0).stop 2023-12-19T14:58:17.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 shutdown_connections 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc564140ae0 0x7fc5640aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc550060670 0x7fc550062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc564140710 0x7fc5640b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 --2- 172.21.15.50:0/2256903054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc5640c75e0 0x7fc56415a0c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 >> 172.21.15.50:0/2256903054 conn(0x7fc5640c8850 msgr2=0x7fc5640684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 shutdown_connections 2023-12-19T14:58:17.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.393+0000 7fc569d45700 1 -- 172.21.15.50:0/2256903054 wait complete. 2023-12-19T14:58:17.402 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 60 2023-12-19T14:58:17.420 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T14:58:17.420 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T14:58:17.420 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T14:58:17.420 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:58:17.421 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T14:58:17.421 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T14:58:17.421 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T14:58:17.421 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:58:17.421 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T14:58:17.421 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T14:58:17.421 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T14:58:17.421 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:58:17.422 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T14:58:17.422 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T14:58:17.422 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T14:58:17.422 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T14:58:17.438 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T14:58:17.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T14:58:17.445 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T14:58:17.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T14:58:17.446 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T14:58:17.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T14:58:17.448 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T14:58:17.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T14:58:17.492 INFO:tasks.ceph.mds.a:Started 2023-12-19T14:58:17.493 DEBUG:teuthology.parallel:result is None 2023-12-19T14:58:17.496 INFO:tasks.ceph.mds.c:Started 2023-12-19T14:58:17.496 DEBUG:teuthology.parallel:result is None 2023-12-19T14:58:17.499 INFO:tasks.ceph.mds.d:Started 2023-12-19T14:58:17.500 DEBUG:teuthology.parallel:result is None 2023-12-19T14:58:17.506 INFO:tasks.ceph.mds.b:Started 2023-12-19T14:58:17.506 DEBUG:teuthology.parallel:result is None 2023-12-19T14:58:17.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T14:58:17.596 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T14:58:17.603 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T14:58:17.614 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T14:58:17.614 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T14:58:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.669+0000 7f0a7e8e3700 1 Processor -- start 2023-12-19T14:58:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.669+0000 7f0a7e8e3700 1 -- start start 2023-12-19T14:58:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7e8e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0a780c6900 0x7f0a780c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7e8e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806b470 con 0x7f0a7806b900 2023-12-19T14:58:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7e8e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806e110 con 0x7f0a7815cc20 2023-12-19T14:58:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7e8e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7815d790 con 0x7f0a780c6900 2023-12-19T14:58:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d0e0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0a7806b900 0x7f0a7806da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42180/0 (socket says 172.21.15.50:42180) 2023-12-19T14:58:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d0e0700 1 -- 172.21.15.50:0/2359202726 learned_addr learned my addr 172.21.15.50:0/2359202726 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:17.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/2359202726 >> v2:172.21.15.50:3300/0 conn(0x7f0a780c6900 0x7f0a780c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831377150 0 0) 0x7f0a7806b470 con 0x7f0a7806b900 2023-12-19T14:58:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a64003570 con 0x7f0a7806b900 2023-12-19T14:58:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833991681 0 0) 0x7f0a7806e110 con 0x7f0a7815cc20 2023-12-19T14:58:17.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7806b470 con 0x7f0a7815cc20 2023-12-19T14:58:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d8e1700 1 -- 172.21.15.50:0/2359202726 >> v1:172.21.15.125:6789/0 conn(0x7f0a7815cc20 legacy=0x7f0a7815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d8e1700 1 -- 172.21.15.50:0/2359202726 >> v1:172.21.15.50:6789/0 conn(0x7f0a7806b900 legacy=0x7f0a7806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.673+0000 7f0a7d8e1700 1 -- 172.21.15.50:0/2359202726 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78161d10 con 0x7f0a780c6900 2023-12-19T14:58:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/2359202726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a780c6cd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0a740098e0 tx=0x7f0a740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22aac81ce9c51498 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74011630 con 0x7f0a780c6900 2023-12-19T14:58:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74011790 con 0x7f0a780c6900 2023-12-19T14:58:17.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7c8df700 1 -- 172.21.15.50:0/2359202726 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74011b30 con 0x7f0a780c6900 2023-12-19T14:58:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/2359202726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 msgr2=0x7f0a780c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/2359202726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a780c6cd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0a740098e0 tx=0x7f0a740091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/2359202726 shutdown_connections 2023-12-19T14:58:17.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/2359202726 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a780c6cd0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/2359202726 >> 172.21.15.50:0/2359202726 conn(0x7f0a78067bb0 msgr2=0x7f0a780cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/2359202726 shutdown_connections 2023-12-19T14:58:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.677+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/2359202726 wait complete. 2023-12-19T14:58:17.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 Processor -- start 2023-12-19T14:58:17.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 -- start start 2023-12-19T14:58:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 0x7f0a78160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a7815cc20 0x7f0a78153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780b0300 con 0x7f0a780c6900 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780b0b50 con 0x7f0a7806b900 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e8e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78160870 con 0x7f0a7815cc20 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7d0e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7d0e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39192/0 (socket says 172.21.15.50:39192) 2023-12-19T14:58:17.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7d0e0700 1 -- 172.21.15.50:0/3512885378 learned_addr learned my addr 172.21.15.50:0/3512885378 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 0x7f0a78160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.681+0000 7f0a7e0e2700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a7815cc20 0x7f0a78153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 -- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a7815cc20 msgr2=0x7f0a78153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a7815cc20 0x7f0a78153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 -- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 msgr2=0x7f0a78160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 0x7f0a78160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 -- 172.21.15.50:0/3512885378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74009420 con 0x7f0a780c6900 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 0x7f0a78160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:17.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7d0e0700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c790 tx=0x7f0a68007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005b30 con 0x7f0a780c6900 2023-12-19T14:58:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a68005c90 con 0x7f0a780c6900 2023-12-19T14:58:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68007300 con 0x7f0a780c6900 2023-12-19T14:58:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a780b0500 con 0x7f0a780c6900 2023-12-19T14:58:17.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a78150340 con 0x7f0a780c6900 2023-12-19T14:58:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.685+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7804be40 con 0x7f0a780c6900 2023-12-19T14:58:17.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.689+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0a6801b3b0 con 0x7f0a780c6900 2023-12-19T14:58:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.689+0000 7f0a6dffb700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 0x7f0a5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.689+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f0a6807bfe0 con 0x7f0a780c6900 2023-12-19T14:58:17.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.689+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 0x7f0a5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:17.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.693+0000 7f0a7d8e1700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 0x7f0a5c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a740093f0 tx=0x7f0a74017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:17.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.697+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a6804c2f0 con 0x7f0a780c6900 2023-12-19T14:58:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:17.881+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0a780b0c90 con 0x7f0a780c6900 2023-12-19T14:58:19.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.085+0000 7f0a6dffb700 1 -- 172.21.15.50:0/3512885378 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0a68017080 con 0x7f0a780c6900 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 msgr2=0x7f0a5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 0x7f0a5c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a740093f0 tx=0x7f0a74017000 comp rx=0 tx=0).stop 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 msgr2=0x7f0a7814fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c790 tx=0x7f0a68007ea0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 shutdown_connections 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0a5c060670 0x7f0a5c062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a7815cc20 0x7f0a78153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a780c6900 0x7f0a7814fe00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 --2- 172.21.15.50:0/3512885378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a7806b900 0x7f0a78160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 >> 172.21.15.50:0/3512885378 conn(0x7f0a78067bb0 msgr2=0x7f0a78159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 shutdown_connections 2023-12-19T14:58:19.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.089+0000 7f0a7e8e3700 1 -- 172.21.15.50:0/3512885378 wait complete. 2023-12-19T14:58:19.095 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T14:58:19.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T14:58:19.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.281+0000 7f7fcd4cd700 1 Processor -- start 2023-12-19T14:58:19.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.281+0000 7f7fcd4cd700 1 -- start start 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fcd4cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7fc806b8b0 0x7f7fc8159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fcd4cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fc80ab960 0x7f7fc80b0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fcd4cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80b0080 con 0x7f7fc8155350 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fcd4cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc815a010 con 0x7f7fc806b8b0 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fcd4cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc806e740 con 0x7f7fc80ab960 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7fc806b8b0 0x7f7fc8159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc6ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7fc8155350 0x7f7fc806b1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:54586/0 (socket says 172.21.15.50:54586) 2023-12-19T14:58:19.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc6ffd700 1 -- 172.21.15.50:0/3435449090 learned_addr learned my addr 172.21.15.50:0/3435449090 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:19.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc77fe700 1 --2- 172.21.15.50:0/3435449090 >> v2:172.21.15.125:3300/0 conn(0x7f7fc80ab960 0x7f7fc80b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc5ffb700 1 -- 172.21.15.50:0/3435449090 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064868755 0 0) 0x7f7fc80b0080 con 0x7f7fc8155350 2023-12-19T14:58:19.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc5ffb700 1 -- 172.21.15.50:0/3435449090 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003570 con 0x7f7fc8155350 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 -- 172.21.15.50:0/3435449090 >> v2:172.21.15.125:3300/0 conn(0x7f7fc80ab960 msgr2=0x7f7fc80b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 --2- 172.21.15.50:0/3435449090 >> v2:172.21.15.125:3300/0 conn(0x7f7fc80ab960 0x7f7fc80b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 -- 172.21.15.50:0/3435449090 >> v1:172.21.15.125:6789/0 conn(0x7f7fc8155350 legacy=0x7f7fc806b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 -- 172.21.15.50:0/3435449090 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc80af470 con 0x7f7fc806b8b0 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.285+0000 7f7fc67fc700 1 --2- 172.21.15.50:0/3435449090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc806b8b0 0x7f7fc8159ad0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806e490 tx=0x7f7fbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab9e7e81b6ac8d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fc5ffb700 1 -- 172.21.15.50:0/3435449090 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc011630 con 0x7f7fc806b8b0 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fc5ffb700 1 -- 172.21.15.50:0/3435449090 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc011790 con 0x7f7fc806b8b0 2023-12-19T14:58:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fc5ffb700 1 -- 172.21.15.50:0/3435449090 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc011b30 con 0x7f7fc806b8b0 2023-12-19T14:58:19.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/3435449090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc806b8b0 msgr2=0x7f7fc8159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/3435449090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc806b8b0 0x7f7fc8159ad0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806e490 tx=0x7f7fbc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/3435449090 shutdown_connections 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/3435449090 >> v2:172.21.15.125:3300/0 conn(0x7f7fc80ab960 0x7f7fc80b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/3435449090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc806b8b0 0x7f7fc8159ad0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/3435449090 >> 172.21.15.50:0/3435449090 conn(0x7f7fc8067410 msgr2=0x7f7fc80677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/3435449090 shutdown_connections 2023-12-19T14:58:19.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.289+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/3435449090 wait complete. 2023-12-19T14:58:19.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 Processor -- start 2023-12-19T14:58:19.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- start start 2023-12-19T14:58:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 0x7f7fc80ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 0x7f7fc816ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814f4b0 con 0x7f7fc80ab960 2023-12-19T14:58:19.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814e600 con 0x7f7fc806b8b0 2023-12-19T14:58:19.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814e8a0 con 0x7f7fc8155350 2023-12-19T14:58:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39220/0 (socket says 172.21.15.50:39220) 2023-12-19T14:58:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 -- 172.21.15.50:0/2196603181 learned_addr learned my addr 172.21.15.50:0/2196603181 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc77fe700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 0x7f7fc816ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc6ffd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 0x7f7fc80ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 -- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 msgr2=0x7f7fc816ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 0x7f7fc816ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 -- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 msgr2=0x7f7fc80ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 0x7f7fc80ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 -- 172.21.15.50:0/2196603181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc009420 con 0x7f7fc80ab960 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc77fe700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 0x7f7fc816ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc6ffd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 0x7f7fc80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:19.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fc67fc700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8069c10 tx=0x7f7fbc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc011900 con 0x7f7fc80ab960 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc814eb80 con 0x7f7fc80ab960 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc011a60 con 0x7f7fc80ab960 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc026680 con 0x7f7fc80ab960 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc814f140 con 0x7f7fc80ab960 2023-12-19T14:58:19.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.297+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc814f620 con 0x7f7fc80ab960 2023-12-19T14:58:19.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.301+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7fbc0193b0 con 0x7f7fc80ab960 2023-12-19T14:58:19.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.301+0000 7f7faeffd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 0x7f7fb4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.301+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f7fbc07db00 con 0x7f7fc80ab960 2023-12-19T14:58:19.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.301+0000 7f7fc6ffd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 0x7f7fb4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.305+0000 7f7fc6ffd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 0x7f7fb4062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8007fd0 tx=0x7f7fb8009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.313+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7fbc07fe60 con 0x7f7fc80ab960 2023-12-19T14:58:19.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.537+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7fc8071120 con 0x7f7fc80ab960 2023-12-19T14:58:19.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.537+0000 7f7faeffd700 1 -- 172.21.15.50:0/2196603181 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7fbc015090 con 0x7f7fc80ab960 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:58:19.543 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T14:58:19.545 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.546 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.547 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.548 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.550 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T14:58:19.551 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.553 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T14:58:19.554 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.555 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T14:58:19.556 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T14:58:19.557 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T14:58:19.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:19.557 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T14:58:19.557 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:58:19.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 msgr2=0x7f7fb4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 0x7f7fb4062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8007fd0 tx=0x7f7fb8009040 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 msgr2=0x7f7fc816a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8069c10 tx=0x7f7fbc017000 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 shutdown_connections 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7fb40606a0 0x7f7fb4062b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7fc8155350 0x7f7fc816ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7fc80ab960 0x7f7fc816a850 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 --2- 172.21.15.50:0/2196603181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fc806b8b0 0x7f7fc80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 >> 172.21.15.50:0/2196603181 conn(0x7f7fc8067410 msgr2=0x7f7fc80c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 shutdown_connections 2023-12-19T14:58:19.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.541+0000 7f7fcd4cd700 1 -- 172.21.15.50:0/2196603181 wait complete. 2023-12-19T14:58:19.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:19.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.737+0000 7fd5c264c700 1 Processor -- start 2023-12-19T14:58:19.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.741+0000 7fd5c264c700 1 -- start start 2023-12-19T14:58:19.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.741+0000 7fd5c264c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc1591f0 0x7fd5bc06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c264c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5bc06b730 con 0x7fd5bc158e20 2023-12-19T14:58:19.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c264c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5bc06ab20 con 0x7fd5bc0ce810 2023-12-19T14:58:19.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c264c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc06ae00 con 0x7fd5bc1591f0 2023-12-19T14:58:19.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd5bc158e20 0x7fd5bc0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42194/0 (socket says 172.21.15.50:42194) 2023-12-19T14:58:19.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 -- 172.21.15.50:0/403885959 learned_addr learned my addr 172.21.15.50:0/403885959 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c0e49700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc1591f0 0x7fd5bc06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922798869 0 0) 0x7fd5bc06b730 con 0x7fd5bc158e20 2023-12-19T14:58:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a8003570 con 0x7fd5bc158e20 2023-12-19T14:58:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947349074 0 0) 0x7fd5bc06ab20 con 0x7fd5bc0ce810 2023-12-19T14:58:19.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5bc06b730 con 0x7fd5bc0ce810 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644489951 0 0) 0x7fd5a8003570 con 0x7fd5bc158e20 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc1591f0 msgr2=0x7fd5bc06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc1591f0 0x7fd5bc06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 >> v1:172.21.15.125:6789/0 conn(0x7fd5bc0ce810 legacy=0x7fd5bc0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc06b0e0 con 0x7fd5bc158e20 2023-12-19T14:58:19.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fd5b8001470 con 0x7fd5bc158e20 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 >> v1:172.21.15.50:6789/0 conn(0x7fd5bc158e20 legacy=0x7fd5bc0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5a8003c20 0x7fd5a8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8004810 0x7fd5a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5a80051c0 0x7fd5a800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc06ae00 con 0x7fd5a8004810 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a800c020 con 0x7fd5a80051c0 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a800c1e0 con 0x7fd5a8003c20 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5a8003c20 0x7fd5a8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c0e49700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8004810 0x7fd5a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c1e4b700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5a80051c0 0x7fd5a800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 -- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5a80051c0 msgr2=0x7fd5a800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5a80051c0 0x7fd5a800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 -- 172.21.15.50:0/403885959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8004810 msgr2=0x7fd5a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8004810 0x7fd5a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 -- 172.21.15.50:0/403885959 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8003570 con 0x7fd5a8003c20 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5c164a700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5a8003c20 0x7fd5a8004010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c2d0 tx=0x7fd5b8001e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613889050876d66a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b800cbc0 con 0x7fd5a8003c20 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b800cd20 con 0x7fd5a8003c20 2023-12-19T14:58:19.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.745+0000 7fd5b3fff700 1 -- 172.21.15.50:0/403885959 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8005b70 con 0x7fd5a8003c20 2023-12-19T14:58:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 -- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5a8003c20 msgr2=0x7fd5a8004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 --2- 172.21.15.50:0/403885959 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5a8003c20 0x7fd5a8004010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c2d0 tx=0x7fd5b8001e10 comp rx=0 tx=0).stop 2023-12-19T14:58:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c1e4b700 1 -- 172.21.15.50:0/403885959 reap_dead start 2023-12-19T14:58:19.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 -- 172.21.15.50:0/403885959 shutdown_connections 2023-12-19T14:58:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 -- 172.21.15.50:0/403885959 >> 172.21.15.50:0/403885959 conn(0x7fd5bc067bb0 msgr2=0x7fd5bc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 -- 172.21.15.50:0/403885959 shutdown_connections 2023-12-19T14:58:19.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.749+0000 7fd5c264c700 1 -- 172.21.15.50:0/403885959 wait complete. 2023-12-19T14:58:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 Processor -- start 2023-12-19T14:58:19.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- start start 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc0c9970 0x7fd5bc08a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 0x7fd5bc0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc170c90 con 0x7fd5a8003c20 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1584a0 con 0x7fd5bc08aa60 2023-12-19T14:58:19.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc170ee0 con 0x7fd5bc0c9970 2023-12-19T14:58:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39244/0 (socket says 172.21.15.50:39244) 2023-12-19T14:58:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 -- 172.21.15.50:0/1292292253 learned_addr learned my addr 172.21.15.50:0/1292292253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c1e4b700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 0x7fd5bc0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c0e49700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc0c9970 0x7fd5bc08a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 -- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc0c9970 msgr2=0x7fd5bc08a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc0c9970 0x7fd5bc08a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 -- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 msgr2=0x7fd5bc0c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 0x7fd5bc0c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 -- 172.21.15.50:0/1292292253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc1711c0 con 0x7fd5a8003c20 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c1e4b700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 0x7fd5bc0c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c164a700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c8c0 tx=0x7fd5b8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80072e0 con 0x7fd5a8003c20 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8007440 con 0x7fd5a8003c20 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8007820 con 0x7fd5a8003c20 2023-12-19T14:58:19.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc1714a0 con 0x7fd5a8003c20 2023-12-19T14:58:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc1719c0 con 0x7fd5a8003c20 2023-12-19T14:58:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.757+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc171be0 con 0x7fd5a8003c20 2023-12-19T14:58:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.765+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd5b8019030 con 0x7fd5a8003c20 2023-12-19T14:58:19.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.765+0000 7fd5b17fa700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 0x7fd5a0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.765+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fd5b807df00 con 0x7fd5a8003c20 2023-12-19T14:58:19.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.765+0000 7fd5c0e49700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 0x7fd5a0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:19.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.769+0000 7fd5c0e49700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 0x7fd5a0062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc0c6490 tx=0x7fd5ac008040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:19.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.773+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd5b804e620 con 0x7fd5a8003c20 2023-12-19T14:58:19.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.941+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5bc0b5b80 con 0x7fd5a8003c20 2023-12-19T14:58:19.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.945+0000 7fd5b17fa700 1 -- 172.21.15.50:0/1292292253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+12437 (secure 0 0 0) 0x7fd5bc0b5b80 con 0x7fd5a8003c20 2023-12-19T14:58:19.949 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:19.949 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":140,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:19.080925+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 msgr2=0x7fd5a0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 0x7fd5a0062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc0c6490 tx=0x7fd5ac008040 comp rx=0 tx=0).stop 2023-12-19T14:58:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 msgr2=0x7fd5bc157ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:19.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c8c0 tx=0x7fd5b8015000 comp rx=0 tx=0).stop 2023-12-19T14:58:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 shutdown_connections 2023-12-19T14:58:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5bc08aa60 0x7fd5bc0c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd5a00606a0 0x7fd5a0062b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5bc0c9970 0x7fd5bc08a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 --2- 172.21.15.50:0/1292292253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd5a8003c20 0x7fd5bc157ea0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 >> 172.21.15.50:0/1292292253 conn(0x7fd5bc067bb0 msgr2=0x7fd5bc0b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 shutdown_connections 2023-12-19T14:58:19.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:19.949+0000 7fd5c264c700 1 -- 172.21.15.50:0/1292292253 wait complete. 2023-12-19T14:58:19.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T14:58:19.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T14:58:20.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.145+0000 7fcd3eae3700 1 Processor -- start 2023-12-19T14:58:20.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.145+0000 7fcd3eae3700 1 -- start start 2023-12-19T14:58:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3eae3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd3806de00 0x7fcd380c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3eae3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcd3806c7c0 0x7fcd3815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3eae3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3806d970 con 0x7fcd380c69a0 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3eae3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3806cd60 con 0x7fcd3806de00 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3eae3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3806d040 con 0x7fcd3806c7c0 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd3806de00 0x7fcd380c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd3806de00 0x7fcd380c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39260/0 (socket says 172.21.15.50:39260) 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/3834721647 learned_addr learned my addr 172.21.15.50:0/3834721647 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3e2e2700 1 --2- 172.21.15.50:0/3834721647 >> v2:172.21.15.125:3300/0 conn(0x7fcd3806c7c0 0x7fcd3815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/3834721647 >> v2:172.21.15.125:3300/0 conn(0x7fcd3806c7c0 msgr2=0x7fcd3815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 --2- 172.21.15.50:0/3834721647 >> v2:172.21.15.125:3300/0 conn(0x7fcd3806c7c0 0x7fcd3815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/3834721647 >> v1:172.21.15.125:6789/0 conn(0x7fcd380c69a0 legacy=0x7fcd3806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.149+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/3834721647 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3806d320 con 0x7fcd3806de00 2023-12-19T14:58:20.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3d2e0700 1 --2- 172.21.15.50:0/3834721647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd380c6d70 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcd380c8360 tx=0x7fcd28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377c599dabce7de3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3cadf700 1 -- 172.21.15.50:0/3834721647 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28009e70 con 0x7fcd3806de00 2023-12-19T14:58:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3cadf700 1 -- 172.21.15.50:0/3834721647 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2800b860 con 0x7fcd3806de00 2023-12-19T14:58:20.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3cadf700 1 -- 172.21.15.50:0/3834721647 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800bc00 con 0x7fcd3806de00 2023-12-19T14:58:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 -- 172.21.15.50:0/3834721647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 msgr2=0x7fcd380c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/3834721647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd380c6d70 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcd380c8360 tx=0x7fcd28009250 comp rx=0 tx=0).stop 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 -- 172.21.15.50:0/3834721647 shutdown_connections 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/3834721647 >> v2:172.21.15.125:3300/0 conn(0x7fcd3806c7c0 0x7fcd3815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/3834721647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd380c6d70 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 -- 172.21.15.50:0/3834721647 >> 172.21.15.50:0/3834721647 conn(0x7fcd38067b50 msgr2=0x7fcd380cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 -- 172.21.15.50:0/3834721647 shutdown_connections 2023-12-19T14:58:20.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.153+0000 7fcd3eae3700 1 -- 172.21.15.50:0/3834721647 wait complete. 2023-12-19T14:58:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 Processor -- start 2023-12-19T14:58:20.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 -- start start 2023-12-19T14:58:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd3806c7c0 0x7fcd38161c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd380c69a0 0x7fcd3815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3816dd80 con 0x7fcd3806de00 2023-12-19T14:58:20.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd380aea80 con 0x7fcd380c69a0 2023-12-19T14:58:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.157+0000 7fcd3eae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd380aed60 con 0x7fcd3806c7c0 2023-12-19T14:58:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39268/0 (socket says 172.21.15.50:39268) 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/640046618 learned_addr learned my addr 172.21.15.50:0/640046618 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3dae1700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd3806c7c0 0x7fcd38161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3e2e2700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd380c69a0 0x7fcd3815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd3806c7c0 msgr2=0x7fcd38161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd3806c7c0 0x7fcd38161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd380c69a0 msgr2=0x7fcd3815df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd380c69a0 0x7fcd3815df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 -- 172.21.15.50:0/640046618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280094f0 con 0x7fcd3806de00 2023-12-19T14:58:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3d2e0700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcd3806a0d0 tx=0x7fcd28017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800bbe0 con 0x7fcd3806de00 2023-12-19T14:58:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28011420 con 0x7fcd3806de00 2023-12-19T14:58:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd280117e0 con 0x7fcd3806de00 2023-12-19T14:58:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd380af040 con 0x7fcd3806de00 2023-12-19T14:58:20.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.161+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd380af9e0 con 0x7fcd3806de00 2023-12-19T14:58:20.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.165+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd380af560 con 0x7fcd3806de00 2023-12-19T14:58:20.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.165+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcd28011bd0 con 0x7fcd3806de00 2023-12-19T14:58:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.169+0000 7fcd2dffb700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 0x7fcd1c062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.169+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fcd2807dc40 con 0x7fcd3806de00 2023-12-19T14:58:20.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.169+0000 7fcd3dae1700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 0x7fcd1c062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.169+0000 7fcd3dae1700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 0x7fcd1c062ba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcd340039e0 tx=0x7fcd34007ec0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.173+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcd380af560 con 0x7fcd3806de00 2023-12-19T14:58:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.345+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd380af560 con 0x7fcd3806de00 2023-12-19T14:58:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.349+0000 7fcd2dffb700 1 -- 172.21.15.50:0/640046618 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd380af560 con 0x7fcd3806de00 2023-12-19T14:58:20.348 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T14:58:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 msgr2=0x7fcd1c062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 0x7fcd1c062ba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcd340039e0 tx=0x7fcd34007ec0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 msgr2=0x7fcd3815da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcd3806a0d0 tx=0x7fcd28017000 comp rx=0 tx=0).stop 2023-12-19T14:58:20.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 shutdown_connections 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd380c69a0 0x7fcd3815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fcd1c0606f0 0x7fcd1c062ba0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd3806de00 0x7fcd3815da30 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 --2- 172.21.15.50:0/640046618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd3806c7c0 0x7fcd38161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 >> 172.21.15.50:0/640046618 conn(0x7fcd38067b50 msgr2=0x7fcd3815ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 shutdown_connections 2023-12-19T14:58:20.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.353+0000 7fcd3eae3700 1 -- 172.21.15.50:0/640046618 wait complete. 2023-12-19T14:58:20.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T14:58:20.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 Processor -- start 2023-12-19T14:58:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 -- start start 2023-12-19T14:58:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 0x7f7d1415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 0x7f7d140c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1406d990 con 0x7f7d1406c7c0 2023-12-19T14:58:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1415a570 con 0x7f7d14155960 2023-12-19T14:58:20.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1406cd80 con 0x7f7d1406de20 2023-12-19T14:58:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.549+0000 7f7d1259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 0x7f7d1415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 0x7f7d140c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1259c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 0x7f7d1415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39270/0 (socket says 172.21.15.50:39270) 2023-12-19T14:58:20.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1259c700 1 -- 172.21.15.50:0/2336266085 learned_addr learned my addr 172.21.15.50:0/2336266085 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 -- 172.21.15.50:0/2336266085 >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 msgr2=0x7f7d1415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 --2- 172.21.15.50:0/2336266085 >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 0x7f7d1415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 -- 172.21.15.50:0/2336266085 >> v1:172.21.15.50:6789/0 conn(0x7f7d1406c7c0 legacy=0x7f7d140c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 -- 172.21.15.50:0/2336266085 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1406d060 con 0x7f7d1406de20 2023-12-19T14:58:20.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d11d9b700 1 --2- 172.21.15.50:0/2336266085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 0x7f7d140c77f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098e0 tx=0x7f7d080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d764238d0126db2f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1159a700 1 -- 172.21.15.50:0/2336266085 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800b640 con 0x7f7d1406de20 2023-12-19T14:58:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1159a700 1 -- 172.21.15.50:0/2336266085 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0800b7a0 con 0x7f7d1406de20 2023-12-19T14:58:20.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1159a700 1 -- 172.21.15.50:0/2336266085 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800bb40 con 0x7f7d1406de20 2023-12-19T14:58:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 -- 172.21.15.50:0/2336266085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 msgr2=0x7f7d140c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 --2- 172.21.15.50:0/2336266085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 0x7f7d140c77f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098e0 tx=0x7f7d080091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 -- 172.21.15.50:0/2336266085 shutdown_connections 2023-12-19T14:58:20.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 --2- 172.21.15.50:0/2336266085 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406de20 0x7f7d140c77f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 --2- 172.21.15.50:0/2336266085 >> v2:172.21.15.50:3300/0 conn(0x7f7d14155960 0x7f7d1415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 -- 172.21.15.50:0/2336266085 >> 172.21.15.50:0/2336266085 conn(0x7f7d14067bb0 msgr2=0x7f7d140cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 -- 172.21.15.50:0/2336266085 shutdown_connections 2023-12-19T14:58:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.553+0000 7f7d1359e700 1 -- 172.21.15.50:0/2336266085 wait complete. 2023-12-19T14:58:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 Processor -- start 2023-12-19T14:58:20.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 -- start start 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406c7c0 0x7f7d14153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 0x7f7d1416d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141551a0 con 0x7f7d1406de20 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14155d30 con 0x7f7d14155960 2023-12-19T14:58:20.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.557+0000 7f7d1359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14154be0 con 0x7f7d1406c7c0 2023-12-19T14:58:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39284/0 (socket says 172.21.15.50:39284) 2023-12-19T14:58:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 -- 172.21.15.50:0/812915426 learned_addr learned my addr 172.21.15.50:0/812915426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d1259c700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406c7c0 0x7f7d14153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d12d9d700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 0x7f7d1416d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 -- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406c7c0 msgr2=0x7f7d14153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406c7c0 0x7f7d14153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 -- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 msgr2=0x7f7d1416d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 0x7f7d1416d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 -- 172.21.15.50:0/812915426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08009420 con 0x7f7d1406de20 2023-12-19T14:58:20.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d12d9d700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 0x7f7d1416d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d11d9b700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7d1406cec0 tx=0x7f7d08017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0801bcf0 con 0x7f7d1406de20 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0801be90 con 0x7f7d1406de20 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08011670 con 0x7f7d1406de20 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d14154ec0 con 0x7f7d1406de20 2023-12-19T14:58:20.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.561+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d141516c0 con 0x7f7d1406de20 2023-12-19T14:58:20.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.565+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1404be40 con 0x7f7d1406de20 2023-12-19T14:58:20.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.565+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7d080119e0 con 0x7f7d1406de20 2023-12-19T14:58:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.569+0000 7f7d027fc700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 0x7f7cf4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.569+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f7d0807de60 con 0x7f7d1406de20 2023-12-19T14:58:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.569+0000 7f7d1259c700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 0x7f7cf4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.569+0000 7f7d1259c700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 0x7f7cf4062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d1415a540 tx=0x7f7d04009250 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.573+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7d0804e480 con 0x7f7d1406de20 2023-12-19T14:58:20.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.749+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7d140cc350 con 0x7f7d1406de20 2023-12-19T14:58:20.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.749+0000 7f7d027fc700 1 -- 172.21.15.50:0/812915426 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v140) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7d08015060 con 0x7f7d1406de20 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:20.750 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 msgr2=0x7f7cf4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 0x7f7cf4062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d1415a540 tx=0x7f7d04009250 comp rx=0 tx=0).stop 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 msgr2=0x7f7d14154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7d1406cec0 tx=0x7f7d08017000 comp rx=0 tx=0).stop 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 shutdown_connections 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d14155960 0x7f7d1416d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7cf4060670 0x7f7cf4062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d1406de20 0x7f7d14154380 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 --2- 172.21.15.50:0/812915426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d1406c7c0 0x7f7d14153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 >> 172.21.15.50:0/812915426 conn(0x7f7d14067bb0 msgr2=0x7f7d140b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:20.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 shutdown_connections 2023-12-19T14:58:20.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.753+0000 7f7d1359e700 1 -- 172.21.15.50:0/812915426 wait complete. 2023-12-19T14:58:20.776 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T14:58:20.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T14:58:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.953+0000 7ffa0d6cd700 1 Processor -- start 2023-12-19T14:58:20.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.953+0000 7ffa0d6cd700 1 -- start start 2023-12-19T14:58:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa0d6cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 0x7ffa080c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa0d6cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa0d6cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0806d7b0 con 0x7ffa0806dc40 2023-12-19T14:58:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa0d6cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa0806e090 con 0x7ffa080c6af0 2023-12-19T14:58:20.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa0d6cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0806cba0 con 0x7ffa081566a0 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa06ffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 0x7ffa080c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:38604/0 (socket says 172.21.15.50:38604) 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 -- 172.21.15.50:0/3490161525 learned_addr learned my addr 172.21.15.50:0/3490161525 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ff9fe7fc700 1 --1- 172.21.15.50:0/3490161525 >> v1:172.21.15.50:6789/0 conn(0x7ffa0806dc40 0x7ffa0806b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42220/0 (socket says 172.21.15.50:42220) 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 -- 172.21.15.50:0/3490161525 >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 msgr2=0x7ffa080c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 --2- 172.21.15.50:0/3490161525 >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 0x7ffa080c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 -- 172.21.15.50:0/3490161525 >> v1:172.21.15.50:6789/0 conn(0x7ffa0806dc40 legacy=0x7ffa0806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa077fe700 1 -- 172.21.15.50:0/3490161525 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0806ce80 con 0x7ffa081566a0 2023-12-19T14:58:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa067fc700 1 -- 172.21.15.50:0/3490161525 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878574267 0 0) 0x7ffa0806d7b0 con 0x7ffa0806dc40 2023-12-19T14:58:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.957+0000 7ffa06ffd700 1 --2- 172.21.15.50:0/3490161525 >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 0x7ffa080c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa077fe700 1 --2- 172.21.15.50:0/3490161525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa000099b0 tx=0x7ffa00009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb76fea0b4d7386 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa067fc700 1 -- 172.21.15.50:0/3490161525 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00009ea0 con 0x7ffa081566a0 2023-12-19T14:58:20.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa067fc700 1 -- 172.21.15.50:0/3490161525 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0000b860 con 0x7ffa081566a0 2023-12-19T14:58:20.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa067fc700 1 -- 172.21.15.50:0/3490161525 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000bc00 con 0x7ffa081566a0 2023-12-19T14:58:20.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3490161525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 msgr2=0x7ffa08156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3490161525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa000099b0 tx=0x7ffa00009250 comp rx=0 tx=0).stop 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3490161525 shutdown_connections 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3490161525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa081566a0 0x7ffa08156b20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3490161525 >> v2:172.21.15.125:3300/0 conn(0x7ffa080c6af0 0x7ffa080c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3490161525 >> 172.21.15.50:0/3490161525 conn(0x7ffa08067b10 msgr2=0x7ffa080cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3490161525 shutdown_connections 2023-12-19T14:58:20.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.961+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3490161525 wait complete. 2023-12-19T14:58:20.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 Processor -- start 2023-12-19T14:58:20.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 -- start start 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa0806dc40 0x7ffa080c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa080c6af0 0x7ffa080c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0816b450 con 0x7ffa081566a0 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa08167e90 con 0x7ffa0806dc40 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa0d6cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0816afc0 con 0x7ffa080c6af0 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39288/0 (socket says 172.21.15.50:39288) 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa077fe700 1 -- 172.21.15.50:0/3013535316 learned_addr learned my addr 172.21.15.50:0/3013535316 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:20.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ffa06ffd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa0806dc40 0x7ffa080c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.965+0000 7ff9fe7fc700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa080c6af0 0x7ffa080c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 -- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa080c6af0 msgr2=0x7ffa080c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa080c6af0 0x7ffa080c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 -- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa0806dc40 msgr2=0x7ffa080c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa0806dc40 0x7ffa080c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 -- 172.21.15.50:0/3013535316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8009420 con 0x7ffa081566a0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa077fe700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009290 tx=0x7ffa00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0001a450 con 0x7ffa081566a0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0001a5b0 con 0x7ffa081566a0 2023-12-19T14:58:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa000094f0 con 0x7ffa081566a0 2023-12-19T14:58:20.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0001a970 con 0x7ffa081566a0 2023-12-19T14:58:20.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0816a690 con 0x7ffa081566a0 2023-12-19T14:58:20.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.969+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0816b130 con 0x7ffa081566a0 2023-12-19T14:58:20.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.973+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ffa00011420 con 0x7ffa081566a0 2023-12-19T14:58:20.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.973+0000 7ff9ff7fe700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 0x7ff9e8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:20.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.973+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5229+0+0 (secure 0 0 0) 0x7ffa0007e1b0 con 0x7ffa081566a0 2023-12-19T14:58:20.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.973+0000 7ffa06ffd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 0x7ff9e8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:20.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.977+0000 7ffa06ffd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 0x7ff9e8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009f40 tx=0x7ff9f80091a0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:20.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:20.977+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffa0004e7d0 con 0x7ffa081566a0 2023-12-19T14:58:21.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.153+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffa0804be40 con 0x7ffa081566a0 2023-12-19T14:58:21.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.505+0000 7ff9ff7fe700 1 -- 172.21.15.50:0/3013535316 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v141) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffa00051df0 con 0x7ffa081566a0 2023-12-19T14:58:21.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 msgr2=0x7ff9e8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 0x7ff9e8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009f40 tx=0x7ff9f80091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 msgr2=0x7ffa080c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009290 tx=0x7ffa00017000 comp rx=0 tx=0).stop 2023-12-19T14:58:21.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 shutdown_connections 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff9e8060670 0x7ff9e8062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa081566a0 0x7ffa080c86a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa080c6af0 0x7ffa080c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 --2- 172.21.15.50:0/3013535316 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa0806dc40 0x7ffa080c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 >> 172.21.15.50:0/3013535316 conn(0x7ffa08067b10 msgr2=0x7ffa080ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 shutdown_connections 2023-12-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.513+0000 7ffa0d6cd700 1 -- 172.21.15.50:0/3013535316 wait complete. 2023-12-19T14:58:21.518 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T14:58:21.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T14:58:21.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.721+0000 7f10ee745700 1 Processor -- start 2023-12-19T14:58:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 -- start start 2023-12-19T14:58:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f10e80cb970 0x7f10e806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10e80cecd0 0x7f10e806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e80cbdf0 con 0x7f10e8158e20 2023-12-19T14:58:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e80cc0d0 con 0x7f10e80cecd0 2023-12-19T14:58:21.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.725+0000 7f10ee745700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e80cc3b0 con 0x7f10e80cb970 2023-12-19T14:58:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10e80cecd0 0x7f10e806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f10e80cecd0 0x7f10e806b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39302/0 (socket says 172.21.15.50:39302) 2023-12-19T14:58:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 -- 172.21.15.50:0/2612768967 learned_addr learned my addr 172.21.15.50:0/2612768967 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e77fe700 1 --2- 172.21.15.50:0/2612768967 >> v2:172.21.15.125:3300/0 conn(0x7f10e80cb970 0x7f10e806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e6ffd700 1 -- 172.21.15.50:0/2612768967 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703658269 0 0) 0x7f10e80cbdf0 con 0x7f10e8158e20 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e6ffd700 1 -- 172.21.15.50:0/2612768967 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d0003570 con 0x7f10e8158e20 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 -- 172.21.15.50:0/2612768967 >> v2:172.21.15.125:3300/0 conn(0x7f10e80cb970 msgr2=0x7f10e806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 --2- 172.21.15.50:0/2612768967 >> v2:172.21.15.125:3300/0 conn(0x7f10e80cb970 0x7f10e806acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 -- 172.21.15.50:0/2612768967 >> v1:172.21.15.125:6789/0 conn(0x7f10e8158e20 legacy=0x7f10e80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 -- 172.21.15.50:0/2612768967 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e80cc690 con 0x7f10e80cecd0 2023-12-19T14:58:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ecce2700 1 --2- 172.21.15.50:0/2612768967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e806b200 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009f20 tx=0x7f10e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acd1baa74758ef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e6ffd700 1 -- 172.21.15.50:0/2612768967 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e000e070 con 0x7f10e80cecd0 2023-12-19T14:58:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e6ffd700 1 -- 172.21.15.50:0/2612768967 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e000b930 con 0x7f10e80cecd0 2023-12-19T14:58:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10e6ffd700 1 -- 172.21.15.50:0/2612768967 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e00092e0 con 0x7f10e80cecd0 2023-12-19T14:58:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 -- 172.21.15.50:0/2612768967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 msgr2=0x7f10e806b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 --2- 172.21.15.50:0/2612768967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e806b200 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009f20 tx=0x7f10e0009840 comp rx=0 tx=0).stop 2023-12-19T14:58:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 -- 172.21.15.50:0/2612768967 shutdown_connections 2023-12-19T14:58:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 --2- 172.21.15.50:0/2612768967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e806b200 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 --2- 172.21.15.50:0/2612768967 >> v2:172.21.15.125:3300/0 conn(0x7f10e80cb970 0x7f10e806acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 -- 172.21.15.50:0/2612768967 >> 172.21.15.50:0/2612768967 conn(0x7f10e8067bb0 msgr2=0x7f10e80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 -- 172.21.15.50:0/2612768967 shutdown_connections 2023-12-19T14:58:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.729+0000 7f10ee745700 1 -- 172.21.15.50:0/2612768967 wait complete. 2023-12-19T14:58:21.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.733+0000 7f10ee745700 1 Processor -- start 2023-12-19T14:58:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.733+0000 7f10ee745700 1 -- start start 2023-12-19T14:58:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.733+0000 7f10ee745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.733+0000 7f10ee745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10e8158e20 0x7f10e814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 0x7f10e814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e814d9f0 con 0x7f10e80cecd0 2023-12-19T14:58:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e80b1ce0 con 0x7f10e8158e20 2023-12-19T14:58:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e80b1fc0 con 0x7f10e814af30 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39316/0 (socket says 172.21.15.50:39316) 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 -- 172.21.15.50:0/3898684562 learned_addr learned my addr 172.21.15.50:0/3898684562 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ecce2700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 0x7f10e814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e77fe700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10e8158e20 0x7f10e814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 -- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 msgr2=0x7f10e814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 0x7f10e814c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 -- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10e8158e20 msgr2=0x7f10e814bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10e8158e20 0x7f10e814bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 -- 172.21.15.50:0/3898684562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0009ae0 con 0x7f10e80cecd0 2023-12-19T14:58:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ecce2700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 0x7f10e814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10e7fff700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f10d800c790 tx=0x7f10d8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8010510 con 0x7f10e80cecd0 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d8010670 con 0x7f10e80cecd0 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e814d3a0 con 0x7f10e80cecd0 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8010a30 con 0x7f10e80cecd0 2023-12-19T14:58:21.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.737+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e814f240 con 0x7f10e80cecd0 2023-12-19T14:58:21.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.741+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e80cb7a0 con 0x7f10e80cecd0 2023-12-19T14:58:21.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.741+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f10d801b3b0 con 0x7f10e80cecd0 2023-12-19T14:58:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.741+0000 7f10cffff700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 0x7f10d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.745+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5599+0+0 (secure 0 0 0) 0x7f10d807c200 con 0x7f10e80cecd0 2023-12-19T14:58:21.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.745+0000 7f10e77fe700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 0x7f10d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:21.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.745+0000 7f10e77fe700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 0x7f10d4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10e808b130 tx=0x7f10dc0091a0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:21.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.753+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10e80cb7a0 con 0x7f10e80cecd0 2023-12-19T14:58:21.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:21.933+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f10e8071120 con 0x7f10e80cecd0 2023-12-19T14:58:22.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.525+0000 7f10cffff700 1 -- 172.21.15.50:0/3898684562 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v142) v1 ==== 170+0+0 (secure 0 0 0) 0x7f10d80170d0 con 0x7f10e80cecd0 2023-12-19T14:58:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 msgr2=0x7f10d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 0x7f10d4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10e808b130 tx=0x7f10dc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 msgr2=0x7f10e814ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f10d800c790 tx=0x7f10d8007e10 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 shutdown_connections 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10e8158e20 0x7f10e814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f10d4060670 0x7f10d4062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10e814af30 0x7f10e814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 --2- 172.21.15.50:0/3898684562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10e80cecd0 0x7f10e814ed00 secure :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f10d800c790 tx=0x7f10d8007e10 comp rx=0 tx=0).stop 2023-12-19T14:58:22.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 >> 172.21.15.50:0/3898684562 conn(0x7f10e8067bb0 msgr2=0x7f10e80c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.541+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 shutdown_connections 2023-12-19T14:58:22.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.545+0000 7f10ee745700 1 -- 172.21.15.50:0/3898684562 wait complete. 2023-12-19T14:58:22.553 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T14:58:22.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T14:58:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 Processor -- start 2023-12-19T14:58:22.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 -- start start 2023-12-19T14:58:22.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde60154780 0x7fde6015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde600adab0 0x7fde600adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6015ae30 con 0x7fde6006baa0 2023-12-19T14:58:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde600abf70 con 0x7fde600adab0 2023-12-19T14:58:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde64e6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde60154b50 con 0x7fde60154780 2023-12-19T14:58:22.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde600adab0 0x7fde600adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5f7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde60154780 0x7fde6015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fde600adab0 0x7fde600adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39318/0 (socket says 172.21.15.50:39318) 2023-12-19T14:58:22.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 -- 172.21.15.50:0/697934857 learned_addr learned my addr 172.21.15.50:0/697934857 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:22.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 -- 172.21.15.50:0/697934857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde60154780 msgr2=0x7fde6015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:22.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 --2- 172.21.15.50:0/697934857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde60154780 0x7fde6015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 -- 172.21.15.50:0/697934857 >> v1:172.21.15.125:6789/0 conn(0x7fde6006baa0 legacy=0x7fde60157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.749+0000 7fde5ffff700 1 -- 172.21.15.50:0/697934857 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde600ac210 con 0x7fde600adab0 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde5ffff700 1 --2- 172.21.15.50:0/697934857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde600adab0 0x7fde600adf10 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fde58009f20 tx=0x7fde58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76efe2a4225331a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde5effd700 1 -- 172.21.15.50:0/697934857 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5800e070 con 0x7fde600adab0 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde5effd700 1 -- 172.21.15.50:0/697934857 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800b9d0 con 0x7fde600adab0 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde5effd700 1 -- 172.21.15.50:0/697934857 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde580092e0 con 0x7fde600adab0 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 -- 172.21.15.50:0/697934857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde600adab0 msgr2=0x7fde600adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 --2- 172.21.15.50:0/697934857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde600adab0 0x7fde600adf10 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fde58009f20 tx=0x7fde58009840 comp rx=0 tx=0).stop 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 -- 172.21.15.50:0/697934857 shutdown_connections 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 --2- 172.21.15.50:0/697934857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde600adab0 0x7fde600adf10 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 --2- 172.21.15.50:0/697934857 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde60154780 0x7fde6015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 -- 172.21.15.50:0/697934857 >> 172.21.15.50:0/697934857 conn(0x7fde60067b50 msgr2=0x7fde600cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:22.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 -- 172.21.15.50:0/697934857 shutdown_connections 2023-12-19T14:58:22.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.753+0000 7fde64e6b700 1 -- 172.21.15.50:0/697934857 wait complete. 2023-12-19T14:58:22.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 Processor -- start 2023-12-19T14:58:22.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 -- start start 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde6006baa0 0x7fde600c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde600adab0 0x7fde600c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600ae950 con 0x7fde60154780 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600c8310 con 0x7fde600adab0 2023-12-19T14:58:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde64e6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde600aeb60 con 0x7fde6006baa0 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39320/0 (socket says 172.21.15.50:39320) 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 -- 172.21.15.50:0/2015577097 learned_addr learned my addr 172.21.15.50:0/2015577097 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5f7fe700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde6006baa0 0x7fde600c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde56ffd700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde600adab0 0x7fde600c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 -- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde6006baa0 msgr2=0x7fde600c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde6006baa0 0x7fde600c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 -- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde600adab0 msgr2=0x7fde600c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:22.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde600adab0 0x7fde600c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:22.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.757+0000 7fde5ffff700 1 -- 172.21.15.50:0/2015577097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58009ae0 con 0x7fde60154780 2023-12-19T14:58:22.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde5ffff700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde5800b610 tx=0x7fde58018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58008b80 con 0x7fde60154780 2023-12-19T14:58:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58008ce0 con 0x7fde60154780 2023-12-19T14:58:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde600aee40 con 0x7fde60154780 2023-12-19T14:58:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58026690 con 0x7fde60154780 2023-12-19T14:58:22.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde600b11f0 con 0x7fde60154780 2023-12-19T14:58:22.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.761+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6004be40 con 0x7fde60154780 2023-12-19T14:58:22.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.765+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fde58003040 con 0x7fde60154780 2023-12-19T14:58:22.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.765+0000 7fde57fff700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 0x7fde40062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:22.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.765+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5981+0+0 (secure 0 0 0) 0x7fde5807f210 con 0x7fde60154780 2023-12-19T14:58:22.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.765+0000 7fde5f7fe700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 0x7fde40062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:22.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.769+0000 7fde5f7fe700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 0x7fde40062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde50009d10 tx=0x7fde500091a0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.773+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde5804f550 con 0x7fde60154780 2023-12-19T14:58:22.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:22.981+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fde600c9060 con 0x7fde60154780 2023-12-19T14:58:23.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.533+0000 7fde57fff700 1 -- 172.21.15.50:0/2015577097 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7fde58052b70 con 0x7fde60154780 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.533+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 msgr2=0x7fde40062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.533+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 0x7fde40062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde50009d10 tx=0x7fde500091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.533+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 msgr2=0x7fde60153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.533+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde5800b610 tx=0x7fde58018000 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 shutdown_connections 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde600adab0 0x7fde600c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde40060670 0x7fde40062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde60154780 0x7fde60153370 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 --2- 172.21.15.50:0/2015577097 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fde6006baa0 0x7fde600c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 >> 172.21.15.50:0/2015577097 conn(0x7fde60067b50 msgr2=0x7fde60151030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 shutdown_connections 2023-12-19T14:58:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.537+0000 7fde64e6b700 1 -- 172.21.15.50:0/2015577097 wait complete. 2023-12-19T14:58:23.544 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 17 and data pool 18 2023-12-19T14:58:23.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:23.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 Processor -- start 2023-12-19T14:58:23.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 -- start start 2023-12-19T14:58:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa7dc15d330 0x7fa7dc06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc06b470 con 0x7fa7dc0c6900 2023-12-19T14:58:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc06e110 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1a43700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc06be40 con 0x7fa7dc15d330 2023-12-19T14:58:23.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e0a41700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa7dc0c6900 0x7fa7dc06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42232/0 (socket says 172.21.15.50:42232) 2023-12-19T14:58:23.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e0a41700 1 -- 172.21.15.50:0/1030071020 learned_addr learned my addr 172.21.15.50:0/1030071020 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:23.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.737+0000 7fa7e1242700 1 --2- 172.21.15.50:0/1030071020 >> v2:172.21.15.50:3300/0 conn(0x7fa7dc15d330 0x7fa7dc06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:23.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1242700 1 -- 172.21.15.50:0/1030071020 >> v1:172.21.15.125:6789/0 conn(0x7fa7dc0c6cd0 legacy=0x7fa7dc15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1242700 1 -- 172.21.15.50:0/1030071020 >> v1:172.21.15.50:6789/0 conn(0x7fa7dc0c6900 legacy=0x7fa7dc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1242700 1 -- 172.21.15.50:0/1030071020 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc161d10 con 0x7fa7dc15d330 2023-12-19T14:58:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1242700 1 --2- 172.21.15.50:0/1030071020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc15d330 0x7fa7dc06b900 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009e60 tx=0x7fa7d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f379b9bbae759b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7d37fe700 1 -- 172.21.15.50:0/1030071020 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400e070 con 0x7fa7dc15d330 2023-12-19T14:58:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7d37fe700 1 -- 172.21.15.50:0/1030071020 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d400bc30 con 0x7fa7dc15d330 2023-12-19T14:58:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7d37fe700 1 -- 172.21.15.50:0/1030071020 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4012430 con 0x7fa7dc15d330 2023-12-19T14:58:23.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 -- 172.21.15.50:0/1030071020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc15d330 msgr2=0x7fa7dc06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/1030071020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc15d330 0x7fa7dc06b900 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009e60 tx=0x7fa7d4009780 comp rx=0 tx=0).stop 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 -- 172.21.15.50:0/1030071020 shutdown_connections 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/1030071020 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc15d330 0x7fa7dc06b900 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 -- 172.21.15.50:0/1030071020 >> 172.21.15.50:0/1030071020 conn(0x7fa7dc067bb0 msgr2=0x7fa7dc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 -- 172.21.15.50:0/1030071020 shutdown_connections 2023-12-19T14:58:23.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.741+0000 7fa7e1a43700 1 -- 172.21.15.50:0/1030071020 wait complete. 2023-12-19T14:58:23.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.745+0000 7fa7e1a43700 1 Processor -- start 2023-12-19T14:58:23.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.745+0000 7fa7e1a43700 1 -- start start 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 0x7fa7dc14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7dc15d330 0x7fa7dc0aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc1502a0 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0afa10 con 0x7fa7dc15d330 2023-12-19T14:58:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc162870 con 0x7fa7dc0c6900 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39342/0 (socket says 172.21.15.50:39342) 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 -- 172.21.15.50:0/2947189390 learned_addr learned my addr 172.21.15.50:0/2947189390 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e0a41700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 0x7fa7dc14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1242700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7dc15d330 0x7fa7dc0aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 -- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 msgr2=0x7fa7dc14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 0x7fa7dc14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 -- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7dc15d330 msgr2=0x7fa7dc0aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7dc15d330 0x7fa7dc0aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 -- 172.21.15.50:0/2947189390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4009a20 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e0a41700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 0x7fa7dc14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d3fff700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00c7c0 tx=0x7fa7cc007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc011590 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc011710 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc011ad0 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc1581e0 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.749+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc0af730 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.753+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc04be40 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.753+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa7cc01b3b0 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.757+0000 7fa7d0ff9700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 0x7fa7c8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.757+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fa7cc07bf70 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.757+0000 7fa7e0a41700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 0x7fa7c8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.757+0000 7fa7e0a41700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 0x7fa7c8062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80018c0 tx=0x7fa7d8009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:23.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.761+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa7cc04c230 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.937+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7dc1629b0 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.941+0000 7fa7d0ff9700 1 -- 172.21.15.50:0/2947189390 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+15670 (secure 0 0 0) 0x7fa7cc017020 con 0x7fa7dc0c6cd0 2023-12-19T14:58:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:23.941 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":143,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:22.982803+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:23.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 msgr2=0x7fa7c8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 0x7fa7c8062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80018c0 tx=0x7fa7d8009040 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 msgr2=0x7fa7dc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00c7c0 tx=0x7fa7cc007f40 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 shutdown_connections 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7dc15d330 0x7fa7dc0aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa7c8060670 0x7fa7c8062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc0c6cd0 0x7fa7dc0ae8e0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 --2- 172.21.15.50:0/2947189390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa7dc0c6900 0x7fa7dc14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:23.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 >> 172.21.15.50:0/2947189390 conn(0x7fa7dc067bb0 msgr2=0x7fa7dc160d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 shutdown_connections 2023-12-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:23.945+0000 7fa7e1a43700 1 -- 172.21.15.50:0/2947189390 wait complete. 2023-12-19T14:58:23.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.125+0000 7f43dd416700 1 Processor -- start 2023-12-19T14:58:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.125+0000 7f43dd416700 1 -- start start 2023-12-19T14:58:24.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43dd416700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f43d8155960 0x7f43d815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43dd416700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d806d990 con 0x7f43d806de20 2023-12-19T14:58:24.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43dd416700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d815a570 con 0x7f43d80c7f00 2023-12-19T14:58:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43dd416700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d806cd80 con 0x7f43d8155960 2023-12-19T14:58:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d77fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f43d806de20 0x7f43d80c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42240/0 (socket says 172.21.15.50:42240) 2023-12-19T14:58:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d77fe700 1 -- 172.21.15.50:0/2632142232 learned_addr learned my addr 172.21.15.50:0/2632142232 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d7fff700 1 --2- 172.21.15.50:0/2632142232 >> v2:172.21.15.125:3300/0 conn(0x7f43d8155960 0x7f43d815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d6ffd700 1 -- 172.21.15.50:0/2632142232 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770568050 0 0) 0x7f43d806d990 con 0x7f43d806de20 2023-12-19T14:58:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d6ffd700 1 -- 172.21.15.50:0/2632142232 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0003570 con 0x7f43d806de20 2023-12-19T14:58:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d7fff700 1 -- 172.21.15.50:0/2632142232 >> v1:172.21.15.125:6789/0 conn(0x7f43d80c7f00 legacy=0x7f43d80aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d7fff700 1 -- 172.21.15.50:0/2632142232 >> v1:172.21.15.50:6789/0 conn(0x7f43d806de20 legacy=0x7f43d80c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.129+0000 7f43d7fff700 1 -- 172.21.15.50:0/2632142232 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d806d060 con 0x7f43d8155960 2023-12-19T14:58:24.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43d7fff700 1 --2- 172.21.15.50:0/2632142232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d8155960 0x7f43d815a030 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009e60 tx=0x7f43c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb4d45754f09414 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43d6ffd700 1 -- 172.21.15.50:0/2632142232 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800e070 con 0x7f43d8155960 2023-12-19T14:58:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43d6ffd700 1 -- 172.21.15.50:0/2632142232 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800bb60 con 0x7f43d8155960 2023-12-19T14:58:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43d6ffd700 1 -- 172.21.15.50:0/2632142232 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8012430 con 0x7f43d8155960 2023-12-19T14:58:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 -- 172.21.15.50:0/2632142232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d8155960 msgr2=0x7f43d815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 --2- 172.21.15.50:0/2632142232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d8155960 0x7f43d815a030 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009e60 tx=0x7f43c8009780 comp rx=0 tx=0).stop 2023-12-19T14:58:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 -- 172.21.15.50:0/2632142232 shutdown_connections 2023-12-19T14:58:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 --2- 172.21.15.50:0/2632142232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d8155960 0x7f43d815a030 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 -- 172.21.15.50:0/2632142232 >> 172.21.15.50:0/2632142232 conn(0x7f43d8067bb0 msgr2=0x7f43d80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 -- 172.21.15.50:0/2632142232 shutdown_connections 2023-12-19T14:58:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.133+0000 7f43dd416700 1 -- 172.21.15.50:0/2632142232 wait complete. 2023-12-19T14:58:24.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 Processor -- start 2023-12-19T14:58:24.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 -- start start 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d80c7f00 0x7f43d80c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43d8155960 0x7f43d8148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8148ee0 con 0x7f43d806de20 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8149a20 con 0x7f43d80c7f00 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43dd416700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81490f0 con 0x7f43d8155960 2023-12-19T14:58:24.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43d7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43d7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39358/0 (socket says 172.21.15.50:39358) 2023-12-19T14:58:24.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.137+0000 7f43d7fff700 1 -- 172.21.15.50:0/2065568654 learned_addr learned my addr 172.21.15.50:0/2065568654 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:24.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43dcc15700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43d8155960 0x7f43d8148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d77fe700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d80c7f00 0x7f43d80c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 -- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43d8155960 msgr2=0x7f43d8148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43d8155960 0x7f43d8148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 -- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d80c7f00 msgr2=0x7f43d80c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d80c7f00 0x7f43d80c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 -- 172.21.15.50:0/2065568654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8009a20 con 0x7f43d806de20 2023-12-19T14:58:24.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43d7fff700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f43c8012560 tx=0x7f43c8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8012cd0 con 0x7f43d806de20 2023-12-19T14:58:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8012e30 con 0x7f43d806de20 2023-12-19T14:58:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d81493d0 con 0x7f43d806de20 2023-12-19T14:58:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80276c0 con 0x7f43d806de20 2023-12-19T14:58:24.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d8149eb0 con 0x7f43d806de20 2023-12-19T14:58:24.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.141+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d804be40 con 0x7f43d806de20 2023-12-19T14:58:24.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.145+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f43c8024070 con 0x7f43d806de20 2023-12-19T14:58:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.149+0000 7f43bffff700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 0x7f43c4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.149+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f43c807f0f0 con 0x7f43d806de20 2023-12-19T14:58:24.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.149+0000 7f43d77fe700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 0x7f43c4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.149+0000 7f43d77fe700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 0x7f43c4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0058d0 tx=0x7f43cc009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.153+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43c804f3b0 con 0x7f43d806de20 2023-12-19T14:58:24.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.329+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43d806d340 con 0x7f43d806de20 2023-12-19T14:58:24.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.329+0000 7f43bffff700 1 -- 172.21.15.50:0/2065568654 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+15657 (secure 0 0 0) 0x7f43c80529d0 con 0x7f43d806de20 2023-12-19T14:58:24.331 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:24.331 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":144,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:24.062904+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:24.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 msgr2=0x7f43c4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 0x7f43c4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0058d0 tx=0x7f43cc009040 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 msgr2=0x7f43d80c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f43c8012560 tx=0x7f43c8019000 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 shutdown_connections 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43d80c7f00 0x7f43d80c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f43c4060670 0x7f43c4062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f43d8155960 0x7f43d8148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 --2- 172.21.15.50:0/2065568654 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43d806de20 0x7f43d80c9970 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.333+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 >> 172.21.15.50:0/2065568654 conn(0x7f43d8067bb0 msgr2=0x7f43d80b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:24.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.337+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 shutdown_connections 2023-12-19T14:58:24.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.337+0000 7f43dd416700 1 -- 172.21.15.50:0/2065568654 wait complete. 2023-12-19T14:58:24.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T14:58:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.521+0000 7fe5feb45700 1 Processor -- start 2023-12-19T14:58:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.521+0000 7fe5feb45700 1 -- start start 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5feb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 0x7fe5f80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5feb45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5f80ca2c0 0x7fe5f80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5feb45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f80cbdf0 con 0x7fe5f80cb970 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5feb45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f80cc0d0 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5feb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80cc3b0 con 0x7fe5f8158e20 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5f7fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe5f80cb970 0x7fe5f806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:42250/0 (socket says 172.21.15.50:42250) 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5f7fff700 1 -- 172.21.15.50:0/1699224078 learned_addr learned my addr 172.21.15.50:0/1699224078 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:24.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5fd0e2700 1 --2- 172.21.15.50:0/1699224078 >> v2:172.21.15.125:3300/0 conn(0x7fe5f80ca2c0 0x7fe5f80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.525+0000 7fe5fc8e1700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 0x7fe5f80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fd0e2700 1 -- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 msgr2=0x7fe5f80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:24.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fd0e2700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 0x7fe5f80ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fd0e2700 1 -- 172.21.15.50:0/1699224078 >> v1:172.21.15.50:6789/0 conn(0x7fe5f80cb970 legacy=0x7fe5f806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fd0e2700 1 -- 172.21.15.50:0/1699224078 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80cc690 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fc8e1700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 0x7fe5f80ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:24.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5fd0e2700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f80ceb00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009e60 tx=0x7fe5f0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b60f5f97fe5f31e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5f77fe700 1 -- 172.21.15.50:0/1699224078 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f000e070 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5f77fe700 1 -- 172.21.15.50:0/1699224078 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f000bb60 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5f77fe700 1 -- 172.21.15.50:0/1699224078 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0012430 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 -- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f80ca2c0 msgr2=0x7fe5f80ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f80ceb00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009e60 tx=0x7fe5f0009780 comp rx=0 tx=0).stop 2023-12-19T14:58:24.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 -- 172.21.15.50:0/1699224078 shutdown_connections 2023-12-19T14:58:24.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f80ceb00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 --2- 172.21.15.50:0/1699224078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f8158e20 0x7fe5f80ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 -- 172.21.15.50:0/1699224078 >> 172.21.15.50:0/1699224078 conn(0x7fe5f8067bb0 msgr2=0x7fe5f80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:24.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 -- 172.21.15.50:0/1699224078 shutdown_connections 2023-12-19T14:58:24.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.529+0000 7fe5feb45700 1 -- 172.21.15.50:0/1699224078 wait complete. 2023-12-19T14:58:24.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 Processor -- start 2023-12-19T14:58:24.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 -- start start 2023-12-19T14:58:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f8158e20 0x7fe5f80d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f80c9e50 0x7fe5f80d0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c99c0 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f81501a0 con 0x7fe5f8158e20 2023-12-19T14:58:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5feb45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c8db0 con 0x7fe5f80c9e50 2023-12-19T14:58:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5fc8e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5fc8e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39370/0 (socket says 172.21.15.50:39370) 2023-12-19T14:58:24.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.533+0000 7fe5fc8e1700 1 -- 172.21.15.50:0/3065613721 learned_addr learned my addr 172.21.15.50:0/3065613721 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5f7fff700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f8158e20 0x7fe5f80d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fd0e2700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f80c9e50 0x7fe5f80d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 -- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f80c9e50 msgr2=0x7fe5f80d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f80c9e50 0x7fe5f80d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 -- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f8158e20 msgr2=0x7fe5f80d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:24.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f8158e20 0x7fe5f80d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 -- 172.21.15.50:0/3065613721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec009420 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5fc8e1700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00ec80 tx=0x7fe5ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec00c850 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec00c9b0 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec00cd30 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0009a20 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80c9370 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.537+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f8071120 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.541+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe5ec013680 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.541+0000 7fe5f4ff9700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 0x7fe5dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:24.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.541+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe5ec07d010 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.541+0000 7fe5f7fff700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 0x7fe5dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:24.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.545+0000 7fe5f7fff700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 0x7fe5dc062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80d0060 tx=0x7fe5e4007ed0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.549+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe5ec050e00 con 0x7fe5f80ca2c0 2023-12-19T14:58:24.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:24.756+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe5f815f4c0 con 0x7fe5f80ca2c0 2023-12-19T14:58:25.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.556+0000 7fe5f4ff9700 1 -- 172.21.15.50:0/3065613721 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe5ec050990 con 0x7fe5f80ca2c0 2023-12-19T14:58:25.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 msgr2=0x7fe5dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 0x7fe5dc062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80d0060 tx=0x7fe5e4007ed0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 msgr2=0x7fe5f814fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00ec80 tx=0x7fe5ec0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 shutdown_connections 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5f8158e20 0x7fe5f80d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5dc060670 0x7fe5dc062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5f80c9e50 0x7fe5f80d0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 --2- 172.21.15.50:0/3065613721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5f80ca2c0 0x7fe5f814fc60 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 >> 172.21.15.50:0/3065613721 conn(0x7fe5f8067bb0 msgr2=0x7fe5f806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 shutdown_connections 2023-12-19T14:58:25.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.560+0000 7fe5feb45700 1 -- 172.21.15.50:0/3065613721 wait complete. 2023-12-19T14:58:25.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T14:58:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.752+0000 7f3f5fa2e700 1 Processor -- start 2023-12-19T14:58:25.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 -- start start 2023-12-19T14:58:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3f5815d410 0x7f3f580ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f5806b1b0 0x7f3f5806b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580c7450 con 0x7f3f580ca400 2023-12-19T14:58:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5806bea0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5fa2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580c6e00 con 0x7f3f5815d410 2023-12-19T14:58:25.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5dfcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f5806b1b0 0x7f3f5806b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5dfcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f5806b1b0 0x7f3f5806b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39386/0 (socket says 172.21.15.50:39386) 2023-12-19T14:58:25.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5dfcb700 1 -- 172.21.15.50:0/400778806 learned_addr learned my addr 172.21.15.50:0/400778806 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:25.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.756+0000 7f3f5d7ca700 1 --2- 172.21.15.50:0/400778806 >> v2:172.21.15.125:3300/0 conn(0x7f3f5815d410 0x7f3f580ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5dfcb700 1 -- 172.21.15.50:0/400778806 >> v2:172.21.15.125:3300/0 conn(0x7f3f5815d410 msgr2=0x7f3f580ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5dfcb700 1 --2- 172.21.15.50:0/400778806 >> v2:172.21.15.125:3300/0 conn(0x7f3f5815d410 0x7f3f580ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5dfcb700 1 -- 172.21.15.50:0/400778806 >> v1:172.21.15.125:6789/0 conn(0x7f3f580ca400 legacy=0x7f3f580ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5dfcb700 1 -- 172.21.15.50:0/400778806 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580c70e0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5dfcb700 1 --2- 172.21.15.50:0/400778806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f5806b1b0 0x7f3f5806b650 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3f58068030 tx=0x7f3f50009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5deaaeb83e23c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f4ffff700 1 -- 172.21.15.50:0/400778806 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5000e070 con 0x7f3f5806b1b0 2023-12-19T14:58:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f4ffff700 1 -- 172.21.15.50:0/400778806 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5000ba20 con 0x7f3f5806b1b0 2023-12-19T14:58:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f4ffff700 1 -- 172.21.15.50:0/400778806 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f500092e0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/400778806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f5806b1b0 msgr2=0x7f3f5806b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/400778806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f5806b1b0 0x7f3f5806b650 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3f58068030 tx=0x7f3f50009840 comp rx=0 tx=0).stop 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/400778806 shutdown_connections 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/400778806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f5806b1b0 0x7f3f5806b650 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/400778806 >> v2:172.21.15.125:3300/0 conn(0x7f3f5815d410 0x7f3f580ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/400778806 >> 172.21.15.50:0/400778806 conn(0x7f3f58067bb0 msgr2=0x7f3f580cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/400778806 shutdown_connections 2023-12-19T14:58:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.760+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/400778806 wait complete. 2023-12-19T14:58:25.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 Processor -- start 2023-12-19T14:58:25.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 -- start start 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 0x7f3f580afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f5815d410 0x7f3f580b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58161890 con 0x7f3f580ca400 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58163130 con 0x7f3f5806b1b0 2023-12-19T14:58:25.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5fa2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58162400 con 0x7f3f5815d410 2023-12-19T14:58:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5d7ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5d7ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:43430/0 (socket says 172.21.15.50:43430) 2023-12-19T14:58:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5d7ca700 1 -- 172.21.15.50:0/1476751499 learned_addr learned my addr 172.21.15.50:0/1476751499 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5cfc9700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 0x7f3f580afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5dfcb700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f5815d410 0x7f3f580b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.764+0000 7f3f5d7ca700 1 -- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f5815d410 msgr2=0x7f3f580b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5d7ca700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f5815d410 0x7f3f580b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5d7ca700 1 -- 172.21.15.50:0/1476751499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 msgr2=0x7f3f580afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5d7ca700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 0x7f3f580afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5d7ca700 1 -- 172.21.15.50:0/1476751499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50009ae0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5cfc9700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 0x7f3f580afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5d7ca700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400ec80 tx=0x7f3f540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400c850 con 0x7f3f5806b1b0 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5400c9d0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400cd90 con 0x7f3f5806b1b0 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f580c5ee0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f58162c10 con 0x7f3f5806b1b0 2023-12-19T14:58:25.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.768+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f58162e30 con 0x7f3f5806b1b0 2023-12-19T14:58:25.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.772+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3f54013680 con 0x7f3f5806b1b0 2023-12-19T14:58:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.772+0000 7f3f4d7fa700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 0x7f3f3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:25.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.772+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f3f5407d1c0 con 0x7f3f5806b1b0 2023-12-19T14:58:25.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.772+0000 7f3f5cfc9700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 0x7f3f3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:25.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.776+0000 7f3f5cfc9700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 0x7f3f3c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f58161210 tx=0x7f3f48009040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:25.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.784+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3f5404d480 con 0x7f3f5806b1b0 2023-12-19T14:58:25.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:25.996+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3f580ae770 con 0x7f3f5806b1b0 2023-12-19T14:58:26.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.560+0000 7f3f4d7fa700 1 -- 172.21.15.50:0/1476751499 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3f54050aa0 con 0x7f3f5806b1b0 2023-12-19T14:58:26.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 msgr2=0x7f3f3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 0x7f3f3c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f58161210 tx=0x7f3f48009040 comp rx=0 tx=0).stop 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 msgr2=0x7f3f580c2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400ec80 tx=0x7f3f540091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 shutdown_connections 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f3c060670 0x7f3f3c062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f5815d410 0x7f3f580b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f580ca400 0x7f3f580afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 --2- 172.21.15.50:0/1476751499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f5806b1b0 0x7f3f580c2d80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 >> 172.21.15.50:0/1476751499 conn(0x7f3f58067bb0 msgr2=0x7f3f580ca810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:26.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 shutdown_connections 2023-12-19T14:58:26.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.564+0000 7f3f5fa2e700 1 -- 172.21.15.50:0/1476751499 wait complete. 2023-12-19T14:58:26.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:26.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.756+0000 7fd86eef0700 1 Processor -- start 2023-12-19T14:58:26.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.756+0000 7fd86eef0700 1 -- start start 2023-12-19T14:58:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86eef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140710 0x7fd8680c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86eef0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd868140ae0 0x7fd8680680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86eef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86806e150 con 0x7fd8680c75e0 2023-12-19T14:58:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86eef0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86806d540 con 0x7fd868140ae0 2023-12-19T14:58:26.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86eef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86806d820 con 0x7fd868140710 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd868140ae0 0x7fd8680680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd868140ae0 0x7fd8680680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55266/0 (socket says 172.21.15.50:55266) 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 -- 172.21.15.50:0/2453658268 learned_addr learned my addr 172.21.15.50:0/2453658268 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd86cc8c700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140710 0x7fd8680c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 -- 172.21.15.50:0/2453658268 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140710 msgr2=0x7fd8680c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:26.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140710 0x7fd8680c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 -- 172.21.15.50:0/2453658268 >> v1:172.21.15.125:6789/0 conn(0x7fd8680c75e0 legacy=0x7fd86806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 -- 172.21.15.50:0/2453658268 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86806db00 con 0x7fd868140ae0 2023-12-19T14:58:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd867fff700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd868140ae0 0x7fd8680680f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd8540099b0 tx=0x7fd854009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bafbc774e48b3f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:26.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.760+0000 7fd8677fe700 1 -- 172.21.15.50:0/2453658268 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85400b860 con 0x7fd868140ae0 2023-12-19T14:58:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd8677fe700 1 -- 172.21.15.50:0/2453658268 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85400b9c0 con 0x7fd868140ae0 2023-12-19T14:58:26.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd8677fe700 1 -- 172.21.15.50:0/2453658268 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85400bd60 con 0x7fd868140ae0 2023-12-19T14:58:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 -- 172.21.15.50:0/2453658268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd868140ae0 msgr2=0x7fd8680680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd868140ae0 0x7fd8680680f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd8540099b0 tx=0x7fd854009250 comp rx=0 tx=0).stop 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 -- 172.21.15.50:0/2453658268 shutdown_connections 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd868140ae0 0x7fd8680680f0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2453658268 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140710 0x7fd8680c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 -- 172.21.15.50:0/2453658268 >> 172.21.15.50:0/2453658268 conn(0x7fd8680c8850 msgr2=0x7fd8680c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 -- 172.21.15.50:0/2453658268 shutdown_connections 2023-12-19T14:58:26.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.764+0000 7fd86eef0700 1 -- 172.21.15.50:0/2453658268 wait complete. 2023-12-19T14:58:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 Processor -- start 2023-12-19T14:58:26.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 -- start start 2023-12-19T14:58:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 0x7fd8680b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 0x7fd86813f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680af8c0 con 0x7fd8680c75e0 2023-12-19T14:58:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680c55b0 con 0x7fd868140710 2023-12-19T14:58:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86eef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680b1250 con 0x7fd868140ae0 2023-12-19T14:58:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86cc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86cc8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55276/0 (socket says 172.21.15.50:55276) 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86cc8c700 1 -- 172.21.15.50:0/2700016262 learned_addr learned my addr 172.21.15.50:0/2700016262 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd867fff700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 0x7fd8680b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.768+0000 7fd86d48d700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 0x7fd86813f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 -- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 msgr2=0x7fd86813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 0x7fd86813f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 -- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 msgr2=0x7fd8680b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 0x7fd8680b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 -- 172.21.15.50:0/2700016262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540094f0 con 0x7fd8680c75e0 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd867fff700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 0x7fd8680b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:26.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86d48d700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 0x7fd86813f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86cc8c700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00e190 tx=0x7fd85c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00ee30 con 0x7fd8680c75e0 2023-12-19T14:58:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c014420 con 0x7fd8680c75e0 2023-12-19T14:58:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0147e0 con 0x7fd8680c75e0 2023-12-19T14:58:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86813f6e0 con 0x7fd8680c75e0 2023-12-19T14:58:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8680aef20 con 0x7fd8680c75e0 2023-12-19T14:58:26.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.772+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8681581a0 con 0x7fd8680c75e0 2023-12-19T14:58:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.776+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd85c014b90 con 0x7fd8680c75e0 2023-12-19T14:58:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.776+0000 7fd864ff9700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 0x7fd84c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.776+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd85c07d360 con 0x7fd8680c75e0 2023-12-19T14:58:26.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.776+0000 7fd867fff700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 0x7fd84c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.780+0000 7fd867fff700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 0x7fd84c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd8540099b0 tx=0x7fd854017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:26.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.784+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd85c04d620 con 0x7fd8680c75e0 2023-12-19T14:58:26.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.992+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd868071120 con 0x7fd8680c75e0 2023-12-19T14:58:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.992+0000 7fd864ff9700 1 -- 172.21.15.50:0/2700016262 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7fd85c050c40 con 0x7fd8680c75e0 2023-12-19T14:58:26.993 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:26.993 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T14:58:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 msgr2=0x7fd84c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 0x7fd84c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd8540099b0 tx=0x7fd854017000 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 msgr2=0x7fd8680c5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00e190 tx=0x7fd85c009920 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 shutdown_connections 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd868140710 0x7fd8680b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd84c060670 0x7fd84c062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd868140ae0 0x7fd86813f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 --2- 172.21.15.50:0/2700016262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8680c75e0 0x7fd8680c5070 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 >> 172.21.15.50:0/2700016262 conn(0x7fd8680c8850 msgr2=0x7fd8680670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 shutdown_connections 2023-12-19T14:58:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:26.996+0000 7fd86eef0700 1 -- 172.21.15.50:0/2700016262 wait complete. 2023-12-19T14:58:26.999 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T14:58:27.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T14:58:27.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.188+0000 7f51e06ae700 1 Processor -- start 2023-12-19T14:58:27.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.188+0000 7f51e06ae700 1 -- start start 2023-12-19T14:58:27.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51e06ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f51d8140710 0x7f51d80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51e06ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51d8068800 0x7f51d80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51e06ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d806e150 con 0x7f51d8140ae0 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51e06ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d806d540 con 0x7f51d8140710 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51e06ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d806d820 con 0x7f51d8068800 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f51d8140710 0x7f51d80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f51d8140710 0x7f51d80c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55288/0 (socket says 172.21.15.50:55288) 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 -- 172.21.15.50:0/1141500969 learned_addr learned my addr 172.21.15.50:0/1141500969 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51dec4b700 1 --2- 172.21.15.50:0/1141500969 >> v2:172.21.15.125:3300/0 conn(0x7f51d8068800 0x7f51d80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 -- 172.21.15.50:0/1141500969 >> v2:172.21.15.125:3300/0 conn(0x7f51d8068800 msgr2=0x7f51d80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 --2- 172.21.15.50:0/1141500969 >> v2:172.21.15.125:3300/0 conn(0x7f51d8068800 0x7f51d80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 -- 172.21.15.50:0/1141500969 >> v1:172.21.15.50:6789/0 conn(0x7f51d8140ae0 legacy=0x7f51d80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.192+0000 7f51de44a700 1 -- 172.21.15.50:0/1141500969 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d806db00 con 0x7f51d8140710 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51de44a700 1 --2- 172.21.15.50:0/1141500969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c70a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f51d806bfc0 tx=0x7f51d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41860908879a7c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51dd448700 1 -- 172.21.15.50:0/1141500969 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d400b860 con 0x7f51d8140710 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51dd448700 1 -- 172.21.15.50:0/1141500969 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d400b9c0 con 0x7f51d8140710 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51dd448700 1 -- 172.21.15.50:0/1141500969 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d400bd60 con 0x7f51d8140710 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 -- 172.21.15.50:0/1141500969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 msgr2=0x7f51d80c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1141500969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c70a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f51d806bfc0 tx=0x7f51d4009250 comp rx=0 tx=0).stop 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 -- 172.21.15.50:0/1141500969 shutdown_connections 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1141500969 >> v2:172.21.15.125:3300/0 conn(0x7f51d8068800 0x7f51d80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1141500969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c70a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 -- 172.21.15.50:0/1141500969 >> 172.21.15.50:0/1141500969 conn(0x7f51d80c8850 msgr2=0x7f51d80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 -- 172.21.15.50:0/1141500969 shutdown_connections 2023-12-19T14:58:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.196+0000 7f51e06ae700 1 -- 172.21.15.50:0/1141500969 wait complete. 2023-12-19T14:58:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 Processor -- start 2023-12-19T14:58:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 -- start start 2023-12-19T14:58:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f51d8068800 0x7f51d815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 0x7f51d816c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80b07d0 con 0x7f51d8140710 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80b0ab0 con 0x7f51d8140ae0 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51e06ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80b0d90 con 0x7f51d8068800 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51ddc49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51ddc49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55300/0 (socket says 172.21.15.50:55300) 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.200+0000 7f51ddc49700 1 -- 172.21.15.50:0/1908496909 learned_addr learned my addr 172.21.15.50:0/1908496909 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51de44a700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f51d8068800 0x7f51d815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51dec4b700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 0x7f51d816c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 -- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f51d8068800 msgr2=0x7f51d815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f51d8068800 0x7f51d815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 -- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 msgr2=0x7f51d816c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 0x7f51d816c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 -- 172.21.15.50:0/1908496909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40094f0 con 0x7f51d8140710 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51dec4b700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 0x7f51d816c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ddc49700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c790 tx=0x7f51c8007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8011590 con 0x7f51d8140710 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c80116f0 con 0x7f51d8140710 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80b1070 con 0x7f51d8140710 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8011ab0 con 0x7f51d8140710 2023-12-19T14:58:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d816d2f0 con 0x7f51d8140710 2023-12-19T14:58:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.204+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d8071120 con 0x7f51d8140710 2023-12-19T14:58:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.208+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f51c801b3b0 con 0x7f51d8140710 2023-12-19T14:58:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.208+0000 7f51ce7fc700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 0x7f51bc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.208+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f51c807bf30 con 0x7f51d8140710 2023-12-19T14:58:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.208+0000 7f51de44a700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 0x7f51bc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.208+0000 7f51de44a700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 0x7f51bc062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d806bfc0 tx=0x7f51d4017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.216+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f51c804c290 con 0x7f51d8140710 2023-12-19T14:58:27.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.392+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51d80642a0 con 0x7f51d8140710 2023-12-19T14:58:27.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.392+0000 7f51ce7fc700 1 -- 172.21.15.50:0/1908496909 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+15657 (secure 0 0 0) 0x7f51c8017020 con 0x7f51d8140710 2023-12-19T14:58:27.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:27.394 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":144,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T14:58:24.062904+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T14:58:27.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 msgr2=0x7f51bc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 0x7f51bc062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d806bfc0 tx=0x7f51d4017000 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 msgr2=0x7f51d80c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c790 tx=0x7f51c8007f40 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 shutdown_connections 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51d8140ae0 0x7f51d816c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f51bc060670 0x7f51bc062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f51d8140710 0x7f51d80c1480 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 --2- 172.21.15.50:0/1908496909 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f51d8068800 0x7f51d815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 >> 172.21.15.50:0/1908496909 conn(0x7f51d80c8850 msgr2=0x7f51d80ca0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 shutdown_connections 2023-12-19T14:58:27.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.396+0000 7f51e06ae700 1 -- 172.21.15.50:0/1908496909 wait complete. 2023-12-19T14:58:27.415 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T14:58:27.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:58:27.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.572+0000 7f7df11cc700 1 Processor -- start 2023-12-19T14:58:27.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.572+0000 7f7df11cc700 1 -- start start 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7dec15aec0 0x7f7dec0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec0cca10 con 0x7f7dec0c8200 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0cc0e0 con 0x7f7dec15aec0 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0cc3c0 con 0x7f7dec0acd30 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7de3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7dec0c8200 0x7f7dec15a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34500/0 (socket says 172.21.15.50:34500) 2023-12-19T14:58:27.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7de3fff700 1 -- 172.21.15.50:0/3572190037 learned_addr learned my addr 172.21.15.50:0/3572190037 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:27.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 --2- 172.21.15.50:0/3572190037 >> v2:172.21.15.50:3300/0 conn(0x7f7dec15aec0 0x7f7dec0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7debfff700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7deb7fe700 1 -- 172.21.15.50:0/3572190037 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247195307 0 0) 0x7f7dec0cca10 con 0x7f7dec0c8200 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7deb7fe700 1 -- 172.21.15.50:0/3572190037 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8003570 con 0x7f7dec0c8200 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 -- 172.21.15.50:0/3572190037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 msgr2=0x7f7dec0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 -- 172.21.15.50:0/3572190037 >> v1:172.21.15.50:6789/0 conn(0x7f7dec0c8200 legacy=0x7f7dec15a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 -- 172.21.15.50:0/3572190037 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0cc6a0 con 0x7f7dec15aec0 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df09cb700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0ccea0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009e60 tx=0x7f7de4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d053ac66096b6daa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7deb7fe700 1 -- 172.21.15.50:0/3572190037 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de400e070 con 0x7f7dec15aec0 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7deb7fe700 1 -- 172.21.15.50:0/3572190037 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de400bb40 con 0x7f7dec15aec0 2023-12-19T14:58:27.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7deb7fe700 1 -- 172.21.15.50:0/3572190037 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4012430 con 0x7f7dec15aec0 2023-12-19T14:58:27.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.576+0000 7f7df11cc700 1 -- 172.21.15.50:0/3572190037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 msgr2=0x7f7dec0ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0ccea0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009e60 tx=0x7f7de4009780 comp rx=0 tx=0).stop 2023-12-19T14:58:27.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 -- 172.21.15.50:0/3572190037 shutdown_connections 2023-12-19T14:58:27.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0ccea0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 --2- 172.21.15.50:0/3572190037 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 -- 172.21.15.50:0/3572190037 >> 172.21.15.50:0/3572190037 conn(0x7f7dec0cdbc0 msgr2=0x7f7dec0cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:27.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 -- 172.21.15.50:0/3572190037 shutdown_connections 2023-12-19T14:58:27.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 -- 172.21.15.50:0/3572190037 wait complete. 2023-12-19T14:58:27.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 Processor -- start 2023-12-19T14:58:27.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.580+0000 7f7df11cc700 1 -- start start 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b3960 con 0x7f7dec15aec0 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0c7620 con 0x7f7dec0c8200 2023-12-19T14:58:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0c6710 con 0x7f7dec0acd30 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df09cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7debfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:53262/0 (socket says 172.21.15.50:53262) 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 -- 172.21.15.50:0/2568588403 learned_addr learned my addr 172.21.15.50:0/2568588403 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 -- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 msgr2=0x7f7dec0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 -- 172.21.15.50:0/2568588403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 msgr2=0x7f7dec0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 -- 172.21.15.50:0/2568588403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4009a20 con 0x7f7dec0c8200 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de3fff700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c7c0 tx=0x7f7dd4007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4005cd0 con 0x7f7dec0c8200 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd4005e30 con 0x7f7dec0c8200 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec0c5de0 con 0x7f7dec0c8200 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4007320 con 0x7f7dec0c8200 2023-12-19T14:58:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.584+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec0c63a0 con 0x7f7dec0c8200 2023-12-19T14:58:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.588+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec0c6880 con 0x7f7dec0c8200 2023-12-19T14:58:27.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.588+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7dd401b3b0 con 0x7f7dec0c8200 2023-12-19T14:58:27.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.592+0000 7f7de8ff9700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 0x7f7dcc062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:27.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.592+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f7dd407c1f0 con 0x7f7dec0c8200 2023-12-19T14:58:27.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.592+0000 7f7debfff700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 0x7f7dcc062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:27.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.592+0000 7f7debfff700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 0x7f7dcc062ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0c35c0 tx=0x7f7ddc0091a0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:27.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.600+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7dd404c4b0 con 0x7f7dec0c8200 2023-12-19T14:58:27.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.772+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7dec04be40 con 0x7f7dcc0606f0 2023-12-19T14:58:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.788+0000 7f7de8ff9700 1 -- 172.21.15.50:0/2568588403 <== mgr.7202 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287193 (secure 0 0 0) 0x7f7dec04be40 con 0x7f7dcc0606f0 2023-12-19T14:58:27.796 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 msgr2=0x7f7dcc062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 0x7f7dcc062ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0c35c0 tx=0x7f7ddc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 msgr2=0x7f7dec0c70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c7c0 tx=0x7f7dd4007e00 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 shutdown_connections 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7dec0c8200 0x7f7dec0c70e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7dcc0606f0 0x7f7dcc062ba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dec15aec0 0x7f7dec0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 --2- 172.21.15.50:0/2568588403 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7dec0acd30 0x7f7dec0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 >> 172.21.15.50:0/2568588403 conn(0x7f7dec0cdbc0 msgr2=0x7f7dec0cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:27.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 shutdown_connections 2023-12-19T14:58:27.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:27.796+0000 7f7df11cc700 1 -- 172.21.15.50:0/2568588403 wait complete. 2023-12-19T14:58:27.800 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:58:27.816 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:58:26.831649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2450788,"kb_used_data":15140,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752523932,"statfs":{"total":773094113280,"available":770584506368,"internally_reserved":0,"allocated":15503360,"data_stored":8370310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.329703"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088646+0000","last_change":"2023-12-19T14:58:23.088646+0000","last_active":"2023-12-19T14:58:23.088646+0000","last_peered":"2023-12-19T14:58:23.088646+0000","last_clean":"2023-12-19T14:58:23.088646+0000","last_became_active":"2023-12-19T14:58:23.087353+0000","last_became_peered":"2023-12-19T14:58:23.087353+0000","last_unstale":"2023-12-19T14:58:23.088646+0000","last_undegraded":"2023-12-19T14:58:23.088646+0000","last_fullsized":"2023-12-19T14:58:23.088646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:36.808951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102171+0000","last_change":"2023-12-19T14:58:23.100467+0000","last_active":"2023-12-19T14:58:23.102171+0000","last_peered":"2023-12-19T14:58:23.102171+0000","last_clean":"2023-12-19T14:58:23.102171+0000","last_became_active":"2023-12-19T14:58:23.099318+0000","last_became_peered":"2023-12-19T14:58:23.099318+0000","last_unstale":"2023-12-19T14:58:23.102171+0000","last_undegraded":"2023-12-19T14:58:23.102171+0000","last_fullsized":"2023-12-19T14:58:23.102171+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.097127+0000","last_change":"2023-12-19T14:58:23.093179+0000","last_active":"2023-12-19T14:58:23.097127+0000","last_peered":"2023-12-19T14:58:23.097127+0000","last_clean":"2023-12-19T14:58:23.097127+0000","last_became_active":"2023-12-19T14:58:23.092561+0000","last_became_peered":"2023-12-19T14:58:23.092561+0000","last_unstale":"2023-12-19T14:58:23.097127+0000","last_undegraded":"2023-12-19T14:58:23.097127+0000","last_fullsized":"2023-12-19T14:58:23.097127+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094247+0000","last_change":"2023-12-19T14:58:23.094247+0000","last_active":"2023-12-19T14:58:23.094247+0000","last_peered":"2023-12-19T14:58:23.094247+0000","last_clean":"2023-12-19T14:58:23.094247+0000","last_became_active":"2023-12-19T14:58:23.093176+0000","last_became_peered":"2023-12-19T14:58:23.093176+0000","last_unstale":"2023-12-19T14:58:23.094247+0000","last_undegraded":"2023-12-19T14:58:23.094247+0000","last_fullsized":"2023-12-19T14:58:23.094247+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086509+0000","last_change":"2023-12-19T14:58:23.086509+0000","last_active":"2023-12-19T14:58:23.086509+0000","last_peered":"2023-12-19T14:58:23.086509+0000","last_clean":"2023-12-19T14:58:23.086509+0000","last_became_active":"2023-12-19T14:58:23.086160+0000","last_became_peered":"2023-12-19T14:58:23.086160+0000","last_unstale":"2023-12-19T14:58:23.086509+0000","last_undegraded":"2023-12-19T14:58:23.086509+0000","last_fullsized":"2023-12-19T14:58:23.086509+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:48.754592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.097647+0000","last_change":"2023-12-19T14:58:23.097647+0000","last_active":"2023-12-19T14:58:23.097647+0000","last_peered":"2023-12-19T14:58:23.097647+0000","last_clean":"2023-12-19T14:58:23.097647+0000","last_became_active":"2023-12-19T14:58:23.097365+0000","last_became_peered":"2023-12-19T14:58:23.097365+0000","last_unstale":"2023-12-19T14:58:23.097647+0000","last_undegraded":"2023-12-19T14:58:23.097647+0000","last_fullsized":"2023-12-19T14:58:23.097647+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:55:25.092911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090829+0000","last_change":"2023-12-19T14:58:23.090829+0000","last_active":"2023-12-19T14:58:23.090829+0000","last_peered":"2023-12-19T14:58:23.090829+0000","last_clean":"2023-12-19T14:58:23.090829+0000","last_became_active":"2023-12-19T14:58:23.090557+0000","last_became_peered":"2023-12-19T14:58:23.090557+0000","last_unstale":"2023-12-19T14:58:23.090829+0000","last_undegraded":"2023-12-19T14:58:23.090829+0000","last_fullsized":"2023-12-19T14:58:23.090829+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:54:22.065431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098521+0000","last_change":"2023-12-19T14:58:23.093680+0000","last_active":"2023-12-19T14:58:23.098521+0000","last_peered":"2023-12-19T14:58:23.098521+0000","last_clean":"2023-12-19T14:58:23.098521+0000","last_became_active":"2023-12-19T14:58:23.093386+0000","last_became_peered":"2023-12-19T14:58:23.093386+0000","last_unstale":"2023-12-19T14:58:23.098521+0000","last_undegraded":"2023-12-19T14:58:23.098521+0000","last_fullsized":"2023-12-19T14:58:23.098521+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:10.339485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.087471+0000","last_change":"2023-12-19T14:58:23.087471+0000","last_active":"2023-12-19T14:58:23.087471+0000","last_peered":"2023-12-19T14:58:23.087471+0000","last_clean":"2023-12-19T14:58:23.087471+0000","last_became_active":"2023-12-19T14:58:23.087154+0000","last_became_peered":"2023-12-19T14:58:23.087154+0000","last_unstale":"2023-12-19T14:58:23.087471+0000","last_undegraded":"2023-12-19T14:58:23.087471+0000","last_fullsized":"2023-12-19T14:58:23.087471+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.079696+0000","last_change":"2023-12-19T14:58:23.079696+0000","last_active":"2023-12-19T14:58:23.079696+0000","last_peered":"2023-12-19T14:58:23.079696+0000","last_clean":"2023-12-19T14:58:23.079696+0000","last_became_active":"2023-12-19T14:58:23.079426+0000","last_became_peered":"2023-12-19T14:58:23.079426+0000","last_unstale":"2023-12-19T14:58:23.079696+0000","last_undegraded":"2023-12-19T14:58:23.079696+0000","last_fullsized":"2023-12-19T14:58:23.079696+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089086+0000","last_change":"2023-12-19T14:58:23.089086+0000","last_active":"2023-12-19T14:58:23.089086+0000","last_peered":"2023-12-19T14:58:23.089086+0000","last_clean":"2023-12-19T14:58:23.089086+0000","last_became_active":"2023-12-19T14:58:23.088718+0000","last_became_peered":"2023-12-19T14:58:23.088718+0000","last_unstale":"2023-12-19T14:58:23.089086+0000","last_undegraded":"2023-12-19T14:58:23.089086+0000","last_fullsized":"2023-12-19T14:58:23.089086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:53.139512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088228+0000","last_change":"2023-12-19T14:58:23.088228+0000","last_active":"2023-12-19T14:58:23.088228+0000","last_peered":"2023-12-19T14:58:23.088228+0000","last_clean":"2023-12-19T14:58:23.088228+0000","last_became_active":"2023-12-19T14:58:23.087949+0000","last_became_peered":"2023-12-19T14:58:23.087949+0000","last_unstale":"2023-12-19T14:58:23.088228+0000","last_undegraded":"2023-12-19T14:58:23.088228+0000","last_fullsized":"2023-12-19T14:58:23.088228+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:11.558422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089554+0000","last_change":"2023-12-19T14:58:23.089554+0000","last_active":"2023-12-19T14:58:23.089554+0000","last_peered":"2023-12-19T14:58:23.089554+0000","last_clean":"2023-12-19T14:58:23.089554+0000","last_became_active":"2023-12-19T14:58:23.089254+0000","last_became_peered":"2023-12-19T14:58:23.089254+0000","last_unstale":"2023-12-19T14:58:23.089554+0000","last_undegraded":"2023-12-19T14:58:23.089554+0000","last_fullsized":"2023-12-19T14:58:23.089554+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098959+0000","last_change":"2023-12-19T14:58:23.096005+0000","last_active":"2023-12-19T14:58:23.098959+0000","last_peered":"2023-12-19T14:58:23.098959+0000","last_clean":"2023-12-19T14:58:23.098959+0000","last_became_active":"2023-12-19T14:58:23.095698+0000","last_became_peered":"2023-12-19T14:58:23.095698+0000","last_unstale":"2023-12-19T14:58:23.098959+0000","last_undegraded":"2023-12-19T14:58:23.098959+0000","last_fullsized":"2023-12-19T14:58:23.098959+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:54.610499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095461+0000","last_change":"2023-12-19T14:58:23.095461+0000","last_active":"2023-12-19T14:58:23.095461+0000","last_peered":"2023-12-19T14:58:23.095461+0000","last_clean":"2023-12-19T14:58:23.095461+0000","last_became_active":"2023-12-19T14:58:23.095127+0000","last_became_peered":"2023-12-19T14:58:23.095127+0000","last_unstale":"2023-12-19T14:58:23.095461+0000","last_undegraded":"2023-12-19T14:58:23.095461+0000","last_fullsized":"2023-12-19T14:58:23.095461+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:24.885607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.083467+0000","last_change":"2023-12-19T14:58:23.083467+0000","last_active":"2023-12-19T14:58:23.083467+0000","last_peered":"2023-12-19T14:58:23.083467+0000","last_clean":"2023-12-19T14:58:23.083467+0000","last_became_active":"2023-12-19T14:58:23.083192+0000","last_became_peered":"2023-12-19T14:58:23.083192+0000","last_unstale":"2023-12-19T14:58:23.083467+0000","last_undegraded":"2023-12-19T14:58:23.083467+0000","last_fullsized":"2023-12-19T14:58:23.083467+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:45.211673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098631+0000","last_change":"2023-12-19T14:58:23.098631+0000","last_active":"2023-12-19T14:58:23.098631+0000","last_peered":"2023-12-19T14:58:23.098631+0000","last_clean":"2023-12-19T14:58:23.098631+0000","last_became_active":"2023-12-19T14:58:23.098137+0000","last_became_peered":"2023-12-19T14:58:23.098137+0000","last_unstale":"2023-12-19T14:58:23.098631+0000","last_undegraded":"2023-12-19T14:58:23.098631+0000","last_fullsized":"2023-12-19T14:58:23.098631+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102192+0000","last_change":"2023-12-19T14:58:23.100761+0000","last_active":"2023-12-19T14:58:23.102192+0000","last_peered":"2023-12-19T14:58:23.102192+0000","last_clean":"2023-12-19T14:58:23.102192+0000","last_became_active":"2023-12-19T14:58:23.100048+0000","last_became_peered":"2023-12-19T14:58:23.100048+0000","last_unstale":"2023-12-19T14:58:23.102192+0000","last_undegraded":"2023-12-19T14:58:23.102192+0000","last_fullsized":"2023-12-19T14:58:23.102192+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100091+0000","last_change":"2023-12-19T14:58:23.100091+0000","last_active":"2023-12-19T14:58:23.100091+0000","last_peered":"2023-12-19T14:58:23.100091+0000","last_clean":"2023-12-19T14:58:23.100091+0000","last_became_active":"2023-12-19T14:58:23.099688+0000","last_became_peered":"2023-12-19T14:58:23.099688+0000","last_unstale":"2023-12-19T14:58:23.100091+0000","last_undegraded":"2023-12-19T14:58:23.100091+0000","last_fullsized":"2023-12-19T14:58:23.100091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:09.645961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085553+0000","last_change":"2023-12-19T14:58:23.083506+0000","last_active":"2023-12-19T14:58:23.085553+0000","last_peered":"2023-12-19T14:58:23.085553+0000","last_clean":"2023-12-19T14:58:23.085553+0000","last_became_active":"2023-12-19T14:58:23.083196+0000","last_became_peered":"2023-12-19T14:58:23.083196+0000","last_unstale":"2023-12-19T14:58:23.085553+0000","last_undegraded":"2023-12-19T14:58:23.085553+0000","last_fullsized":"2023-12-19T14:58:23.085553+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:12:36.914340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.082832+0000","last_change":"2023-12-19T14:58:23.082832+0000","last_active":"2023-12-19T14:58:23.082832+0000","last_peered":"2023-12-19T14:58:23.082832+0000","last_clean":"2023-12-19T14:58:23.082832+0000","last_became_active":"2023-12-19T14:58:23.082364+0000","last_became_peered":"2023-12-19T14:58:23.082364+0000","last_unstale":"2023-12-19T14:58:23.082832+0000","last_undegraded":"2023-12-19T14:58:23.082832+0000","last_fullsized":"2023-12-19T14:58:23.082832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092080+0000","last_change":"2023-12-19T14:58:23.092080+0000","last_active":"2023-12-19T14:58:23.092080+0000","last_peered":"2023-12-19T14:58:23.092080+0000","last_clean":"2023-12-19T14:58:23.092080+0000","last_became_active":"2023-12-19T14:58:23.091804+0000","last_became_peered":"2023-12-19T14:58:23.091804+0000","last_unstale":"2023-12-19T14:58:23.092080+0000","last_undegraded":"2023-12-19T14:58:23.092080+0000","last_fullsized":"2023-12-19T14:58:23.092080+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:04.224777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092257+0000","last_change":"2023-12-19T14:58:23.092257+0000","last_active":"2023-12-19T14:58:23.092257+0000","last_peered":"2023-12-19T14:58:23.092257+0000","last_clean":"2023-12-19T14:58:23.092257+0000","last_became_active":"2023-12-19T14:58:23.091966+0000","last_became_peered":"2023-12-19T14:58:23.091966+0000","last_unstale":"2023-12-19T14:58:23.092257+0000","last_undegraded":"2023-12-19T14:58:23.092257+0000","last_fullsized":"2023-12-19T14:58:23.092257+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101096+0000","last_change":"2023-12-19T14:58:23.093802+0000","last_active":"2023-12-19T14:58:23.101096+0000","last_peered":"2023-12-19T14:58:23.101096+0000","last_clean":"2023-12-19T14:58:23.101096+0000","last_became_active":"2023-12-19T14:58:23.093399+0000","last_became_peered":"2023-12-19T14:58:23.093399+0000","last_unstale":"2023-12-19T14:58:23.101096+0000","last_undegraded":"2023-12-19T14:58:23.101096+0000","last_fullsized":"2023-12-19T14:58:23.101096+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:50:13.041547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100677+0000","last_change":"2023-12-19T14:58:23.092426+0000","last_active":"2023-12-19T14:58:23.100677+0000","last_peered":"2023-12-19T14:58:23.100677+0000","last_clean":"2023-12-19T14:58:23.100677+0000","last_became_active":"2023-12-19T14:58:23.092124+0000","last_became_peered":"2023-12-19T14:58:23.092124+0000","last_unstale":"2023-12-19T14:58:23.100677+0000","last_undegraded":"2023-12-19T14:58:23.100677+0000","last_fullsized":"2023-12-19T14:58:23.100677+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:45.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090222+0000","last_change":"2023-12-19T14:58:23.084800+0000","last_active":"2023-12-19T14:58:23.090222+0000","last_peered":"2023-12-19T14:58:23.090222+0000","last_clean":"2023-12-19T14:58:23.090222+0000","last_became_active":"2023-12-19T14:58:23.084493+0000","last_became_peered":"2023-12-19T14:58:23.084493+0000","last_unstale":"2023-12-19T14:58:23.090222+0000","last_undegraded":"2023-12-19T14:58:23.090222+0000","last_fullsized":"2023-12-19T14:58:23.090222+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:44:29.077829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099966+0000","last_change":"2023-12-19T14:58:23.099966+0000","last_active":"2023-12-19T14:58:23.099966+0000","last_peered":"2023-12-19T14:58:23.099966+0000","last_clean":"2023-12-19T14:58:23.099966+0000","last_became_active":"2023-12-19T14:58:23.099148+0000","last_became_peered":"2023-12-19T14:58:23.099148+0000","last_unstale":"2023-12-19T14:58:23.099966+0000","last_undegraded":"2023-12-19T14:58:23.099966+0000","last_fullsized":"2023-12-19T14:58:23.099966+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:34.253090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085983+0000","last_change":"2023-12-19T14:58:23.085983+0000","last_active":"2023-12-19T14:58:23.085983+0000","last_peered":"2023-12-19T14:58:23.085983+0000","last_clean":"2023-12-19T14:58:23.085983+0000","last_became_active":"2023-12-19T14:58:23.085682+0000","last_became_peered":"2023-12-19T14:58:23.085682+0000","last_unstale":"2023-12-19T14:58:23.085983+0000","last_undegraded":"2023-12-19T14:58:23.085983+0000","last_fullsized":"2023-12-19T14:58:23.085983+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:37.977148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.096691+0000","last_change":"2023-12-19T14:58:23.091612+0000","last_active":"2023-12-19T14:58:23.096691+0000","last_peered":"2023-12-19T14:58:23.096691+0000","last_clean":"2023-12-19T14:58:23.096691+0000","last_became_active":"2023-12-19T14:58:23.091350+0000","last_became_peered":"2023-12-19T14:58:23.091350+0000","last_unstale":"2023-12-19T14:58:23.096691+0000","last_undegraded":"2023-12-19T14:58:23.096691+0000","last_fullsized":"2023-12-19T14:58:23.096691+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:47.718805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098967+0000","last_change":"2023-12-19T14:58:23.096801+0000","last_active":"2023-12-19T14:58:23.098967+0000","last_peered":"2023-12-19T14:58:23.098967+0000","last_clean":"2023-12-19T14:58:23.098967+0000","last_became_active":"2023-12-19T14:58:23.095926+0000","last_became_peered":"2023-12-19T14:58:23.095926+0000","last_unstale":"2023-12-19T14:58:23.098967+0000","last_undegraded":"2023-12-19T14:58:23.098967+0000","last_fullsized":"2023-12-19T14:58:23.098967+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101012+0000","last_change":"2023-12-19T14:58:23.095276+0000","last_active":"2023-12-19T14:58:23.101012+0000","last_peered":"2023-12-19T14:58:23.101012+0000","last_clean":"2023-12-19T14:58:23.101012+0000","last_became_active":"2023-12-19T14:58:23.095011+0000","last_became_peered":"2023-12-19T14:58:23.095011+0000","last_unstale":"2023-12-19T14:58:23.101012+0000","last_undegraded":"2023-12-19T14:58:23.101012+0000","last_fullsized":"2023-12-19T14:58:23.101012+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:42.139912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100684+0000","last_change":"2023-12-19T14:58:23.093069+0000","last_active":"2023-12-19T14:58:23.100684+0000","last_peered":"2023-12-19T14:58:23.100684+0000","last_clean":"2023-12-19T14:58:23.100684+0000","last_became_active":"2023-12-19T14:58:23.087629+0000","last_became_peered":"2023-12-19T14:58:23.087629+0000","last_unstale":"2023-12-19T14:58:23.100684+0000","last_undegraded":"2023-12-19T14:58:23.100684+0000","last_fullsized":"2023-12-19T14:58:23.100684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:24.885607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081811+0000","last_change":"2023-12-19T14:58:23.081811+0000","last_active":"2023-12-19T14:58:23.081811+0000","last_peered":"2023-12-19T14:58:23.081811+0000","last_clean":"2023-12-19T14:58:23.081811+0000","last_became_active":"2023-12-19T14:58:23.081367+0000","last_became_peered":"2023-12-19T14:58:23.081367+0000","last_unstale":"2023-12-19T14:58:23.081811+0000","last_undegraded":"2023-12-19T14:58:23.081811+0000","last_fullsized":"2023-12-19T14:58:23.081811+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.541358+0000","last_change":"2023-12-19T14:58:22.538805+0000","last_active":"2023-12-19T14:58:23.541358+0000","last_peered":"2023-12-19T14:58:23.541358+0000","last_clean":"2023-12-19T14:58:23.541358+0000","last_became_active":"2023-12-19T14:58:22.538505+0000","last_became_peered":"2023-12-19T14:58:22.538505+0000","last_unstale":"2023-12-19T14:58:23.541358+0000","last_undegraded":"2023-12-19T14:58:23.541358+0000","last_fullsized":"2023-12-19T14:58:23.541358+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:39.209494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092186+0000","last_change":"2023-12-19T14:58:23.092186+0000","last_active":"2023-12-19T14:58:23.092186+0000","last_peered":"2023-12-19T14:58:23.092186+0000","last_clean":"2023-12-19T14:58:23.092186+0000","last_became_active":"2023-12-19T14:58:23.091826+0000","last_became_peered":"2023-12-19T14:58:23.091826+0000","last_unstale":"2023-12-19T14:58:23.092186+0000","last_undegraded":"2023-12-19T14:58:23.092186+0000","last_fullsized":"2023-12-19T14:58:23.092186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:43:52.111926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.070511+0000","last_change":"2023-12-19T14:58:22.537786+0000","last_active":"2023-12-19T14:58:23.070511+0000","last_peered":"2023-12-19T14:58:23.070511+0000","last_clean":"2023-12-19T14:58:23.070511+0000","last_became_active":"2023-12-19T14:58:22.537458+0000","last_became_peered":"2023-12-19T14:58:22.537458+0000","last_unstale":"2023-12-19T14:58:23.070511+0000","last_undegraded":"2023-12-19T14:58:23.070511+0000","last_fullsized":"2023-12-19T14:58:23.070511+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094322+0000","last_change":"2023-12-19T14:58:23.085247+0000","last_active":"2023-12-19T14:58:23.094322+0000","last_peered":"2023-12-19T14:58:23.094322+0000","last_clean":"2023-12-19T14:58:23.094322+0000","last_became_active":"2023-12-19T14:58:23.084927+0000","last_became_peered":"2023-12-19T14:58:23.084927+0000","last_unstale":"2023-12-19T14:58:23.094322+0000","last_undegraded":"2023-12-19T14:58:23.094322+0000","last_fullsized":"2023-12-19T14:58:23.094322+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:34.523242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.544413+0000","last_change":"2023-12-19T14:58:22.540894+0000","last_active":"2023-12-19T14:58:23.544413+0000","last_peered":"2023-12-19T14:58:23.544413+0000","last_clean":"2023-12-19T14:58:23.544413+0000","last_became_active":"2023-12-19T14:58:22.539386+0000","last_became_peered":"2023-12-19T14:58:22.539386+0000","last_unstale":"2023-12-19T14:58:23.544413+0000","last_undegraded":"2023-12-19T14:58:23.544413+0000","last_fullsized":"2023-12-19T14:58:23.544413+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085487+0000","last_change":"2023-12-19T14:58:23.085487+0000","last_active":"2023-12-19T14:58:23.085487+0000","last_peered":"2023-12-19T14:58:23.085487+0000","last_clean":"2023-12-19T14:58:23.085487+0000","last_became_active":"2023-12-19T14:58:23.085086+0000","last_became_peered":"2023-12-19T14:58:23.085086+0000","last_unstale":"2023-12-19T14:58:23.085487+0000","last_undegraded":"2023-12-19T14:58:23.085487+0000","last_fullsized":"2023-12-19T14:58:23.085487+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.549557+0000","last_change":"2023-12-19T14:58:22.545939+0000","last_active":"2023-12-19T14:58:23.549557+0000","last_peered":"2023-12-19T14:58:23.549557+0000","last_clean":"2023-12-19T14:58:23.549557+0000","last_became_active":"2023-12-19T14:58:22.544686+0000","last_became_peered":"2023-12-19T14:58:22.544686+0000","last_unstale":"2023-12-19T14:58:23.549557+0000","last_undegraded":"2023-12-19T14:58:23.549557+0000","last_fullsized":"2023-12-19T14:58:23.549557+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.093562+0000","last_change":"2023-12-19T14:58:23.093562+0000","last_active":"2023-12-19T14:58:23.093562+0000","last_peered":"2023-12-19T14:58:23.093562+0000","last_clean":"2023-12-19T14:58:23.093562+0000","last_became_active":"2023-12-19T14:58:23.092692+0000","last_became_peered":"2023-12-19T14:58:23.092692+0000","last_unstale":"2023-12-19T14:58:23.093562+0000","last_undegraded":"2023-12-19T14:58:23.093562+0000","last_fullsized":"2023-12-19T14:58:23.093562+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:59.466905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.074062+0000","last_change":"2023-12-19T14:58:22.543346+0000","last_active":"2023-12-19T14:58:23.074062+0000","last_peered":"2023-12-19T14:58:23.074062+0000","last_clean":"2023-12-19T14:58:23.074062+0000","last_became_active":"2023-12-19T14:58:22.543059+0000","last_became_peered":"2023-12-19T14:58:22.543059+0000","last_unstale":"2023-12-19T14:58:23.074062+0000","last_undegraded":"2023-12-19T14:58:23.074062+0000","last_fullsized":"2023-12-19T14:58:23.074062+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:34.444674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088760+0000","last_change":"2023-12-19T14:58:23.088760+0000","last_active":"2023-12-19T14:58:23.088760+0000","last_peered":"2023-12-19T14:58:23.088760+0000","last_clean":"2023-12-19T14:58:23.088760+0000","last_became_active":"2023-12-19T14:58:23.086673+0000","last_became_peered":"2023-12-19T14:58:23.086673+0000","last_unstale":"2023-12-19T14:58:23.088760+0000","last_undegraded":"2023-12-19T14:58:23.088760+0000","last_fullsized":"2023-12-19T14:58:23.088760+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:12.499676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.074633+0000","last_change":"2023-12-19T14:58:22.554344+0000","last_active":"2023-12-19T14:58:23.074633+0000","last_peered":"2023-12-19T14:58:23.074633+0000","last_clean":"2023-12-19T14:58:23.074633+0000","last_became_active":"2023-12-19T14:58:22.553174+0000","last_became_peered":"2023-12-19T14:58:22.553174+0000","last_unstale":"2023-12-19T14:58:23.074633+0000","last_undegraded":"2023-12-19T14:58:23.074633+0000","last_fullsized":"2023-12-19T14:58:23.074633+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:33.508283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.096116+0000","last_change":"2023-12-19T14:58:23.090740+0000","last_active":"2023-12-19T14:58:23.096116+0000","last_peered":"2023-12-19T14:58:23.096116+0000","last_clean":"2023-12-19T14:58:23.096116+0000","last_became_active":"2023-12-19T14:58:23.090419+0000","last_became_peered":"2023-12-19T14:58:23.090419+0000","last_unstale":"2023-12-19T14:58:23.096116+0000","last_undegraded":"2023-12-19T14:58:23.096116+0000","last_fullsized":"2023-12-19T14:58:23.096116+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:03.766685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081928+0000","last_change":"2023-12-19T14:58:22.536509+0000","last_active":"2023-12-19T14:58:23.081928+0000","last_peered":"2023-12-19T14:58:23.081928+0000","last_clean":"2023-12-19T14:58:23.081928+0000","last_became_active":"2023-12-19T14:58:22.535101+0000","last_became_peered":"2023-12-19T14:58:22.535101+0000","last_unstale":"2023-12-19T14:58:23.081928+0000","last_undegraded":"2023-12-19T14:58:23.081928+0000","last_fullsized":"2023-12-19T14:58:23.081928+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:54.904339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090641+0000","last_change":"2023-12-19T14:58:23.090641+0000","last_active":"2023-12-19T14:58:23.090641+0000","last_peered":"2023-12-19T14:58:23.090641+0000","last_clean":"2023-12-19T14:58:23.090641+0000","last_became_active":"2023-12-19T14:58:23.090308+0000","last_became_peered":"2023-12-19T14:58:23.090308+0000","last_unstale":"2023-12-19T14:58:23.090641+0000","last_undegraded":"2023-12-19T14:58:23.090641+0000","last_fullsized":"2023-12-19T14:58:23.090641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.551546+0000","last_change":"2023-12-19T14:58:22.545995+0000","last_active":"2023-12-19T14:58:22.551546+0000","last_peered":"2023-12-19T14:58:22.551546+0000","last_clean":"2023-12-19T14:58:22.551546+0000","last_became_active":"2023-12-19T14:58:22.545730+0000","last_became_peered":"2023-12-19T14:58:22.545730+0000","last_unstale":"2023-12-19T14:58:22.551546+0000","last_undegraded":"2023-12-19T14:58:22.551546+0000","last_fullsized":"2023-12-19T14:58:22.551546+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:23.870648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098815+0000","last_change":"2023-12-19T14:58:23.098815+0000","last_active":"2023-12-19T14:58:23.098815+0000","last_peered":"2023-12-19T14:58:23.098815+0000","last_clean":"2023-12-19T14:58:23.098815+0000","last_became_active":"2023-12-19T14:58:23.098518+0000","last_became_peered":"2023-12-19T14:58:23.098518+0000","last_unstale":"2023-12-19T14:58:23.098815+0000","last_undegraded":"2023-12-19T14:58:23.098815+0000","last_fullsized":"2023-12-19T14:58:23.098815+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:17.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550732+0000","last_change":"2023-12-19T14:58:22.549758+0000","last_active":"2023-12-19T14:58:22.550732+0000","last_peered":"2023-12-19T14:58:22.550732+0000","last_clean":"2023-12-19T14:58:22.550732+0000","last_became_active":"2023-12-19T14:58:22.549408+0000","last_became_peered":"2023-12-19T14:58:22.549408+0000","last_unstale":"2023-12-19T14:58:22.550732+0000","last_undegraded":"2023-12-19T14:58:22.550732+0000","last_fullsized":"2023-12-19T14:58:22.550732+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101060+0000","last_change":"2023-12-19T14:58:23.093740+0000","last_active":"2023-12-19T14:58:23.101060+0000","last_peered":"2023-12-19T14:58:23.101060+0000","last_clean":"2023-12-19T14:58:23.101060+0000","last_became_active":"2023-12-19T14:58:23.093412+0000","last_became_peered":"2023-12-19T14:58:23.093412+0000","last_unstale":"2023-12-19T14:58:23.101060+0000","last_undegraded":"2023-12-19T14:58:23.101060+0000","last_fullsized":"2023-12-19T14:58:23.101060+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:58:59.950526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.077009+0000","last_change":"2023-12-19T14:58:22.548567+0000","last_active":"2023-12-19T14:58:23.077009+0000","last_peered":"2023-12-19T14:58:23.077009+0000","last_clean":"2023-12-19T14:58:23.077009+0000","last_became_active":"2023-12-19T14:58:22.548316+0000","last_became_peered":"2023-12-19T14:58:22.548316+0000","last_unstale":"2023-12-19T14:58:23.077009+0000","last_undegraded":"2023-12-19T14:58:23.077009+0000","last_fullsized":"2023-12-19T14:58:23.077009+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:33.271134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100894+0000","last_change":"2023-12-19T14:58:23.096683+0000","last_active":"2023-12-19T14:58:23.100894+0000","last_peered":"2023-12-19T14:58:23.100894+0000","last_clean":"2023-12-19T14:58:23.100894+0000","last_became_active":"2023-12-19T14:58:23.095608+0000","last_became_peered":"2023-12-19T14:58:23.095608+0000","last_unstale":"2023-12-19T14:58:23.100894+0000","last_undegraded":"2023-12-19T14:58:23.100894+0000","last_fullsized":"2023-12-19T14:58:23.100894+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.541457+0000","last_change":"2023-12-19T14:58:22.540462+0000","last_active":"2023-12-19T14:58:23.541457+0000","last_peered":"2023-12-19T14:58:23.541457+0000","last_clean":"2023-12-19T14:58:23.541457+0000","last_became_active":"2023-12-19T14:58:22.539701+0000","last_became_peered":"2023-12-19T14:58:22.539701+0000","last_unstale":"2023-12-19T14:58:23.541457+0000","last_undegraded":"2023-12-19T14:58:23.541457+0000","last_fullsized":"2023-12-19T14:58:23.541457+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:53.667153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099999+0000","last_change":"2023-12-19T14:58:23.090495+0000","last_active":"2023-12-19T14:58:23.099999+0000","last_peered":"2023-12-19T14:58:23.099999+0000","last_clean":"2023-12-19T14:58:23.099999+0000","last_became_active":"2023-12-19T14:58:23.088694+0000","last_became_peered":"2023-12-19T14:58:23.088694+0000","last_unstale":"2023-12-19T14:58:23.099999+0000","last_undegraded":"2023-12-19T14:58:23.099999+0000","last_fullsized":"2023-12-19T14:58:23.099999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:06.574019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.551229+0000","last_change":"2023-12-19T14:58:22.543859+0000","last_active":"2023-12-19T14:58:23.551229+0000","last_peered":"2023-12-19T14:58:23.551229+0000","last_clean":"2023-12-19T14:58:23.551229+0000","last_became_active":"2023-12-19T14:58:22.543570+0000","last_became_peered":"2023-12-19T14:58:22.543570+0000","last_unstale":"2023-12-19T14:58:23.551229+0000","last_undegraded":"2023-12-19T14:58:23.551229+0000","last_fullsized":"2023-12-19T14:58:23.551229+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:39:15.583668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095877+0000","last_change":"2023-12-19T14:58:23.095877+0000","last_active":"2023-12-19T14:58:23.095877+0000","last_peered":"2023-12-19T14:58:23.095877+0000","last_clean":"2023-12-19T14:58:23.095877+0000","last_became_active":"2023-12-19T14:58:23.091637+0000","last_became_peered":"2023-12-19T14:58:23.091637+0000","last_unstale":"2023-12-19T14:58:23.095877+0000","last_undegraded":"2023-12-19T14:58:23.095877+0000","last_fullsized":"2023-12-19T14:58:23.095877+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:15:01.581518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.548415+0000","last_change":"2023-12-19T14:58:22.549591+0000","last_active":"2023-12-19T14:58:23.548415+0000","last_peered":"2023-12-19T14:58:23.548415+0000","last_clean":"2023-12-19T14:58:23.548415+0000","last_became_active":"2023-12-19T14:58:22.549311+0000","last_became_peered":"2023-12-19T14:58:22.549311+0000","last_unstale":"2023-12-19T14:58:23.548415+0000","last_undegraded":"2023-12-19T14:58:23.548415+0000","last_fullsized":"2023-12-19T14:58:23.548415+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:42.975704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.091484+0000","last_change":"2023-12-19T14:58:23.091484+0000","last_active":"2023-12-19T14:58:23.091484+0000","last_peered":"2023-12-19T14:58:23.091484+0000","last_clean":"2023-12-19T14:58:23.091484+0000","last_became_active":"2023-12-19T14:58:23.091195+0000","last_became_peered":"2023-12-19T14:58:23.091195+0000","last_unstale":"2023-12-19T14:58:23.091484+0000","last_undegraded":"2023-12-19T14:58:23.091484+0000","last_fullsized":"2023-12-19T14:58:23.091484+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:09.645961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.087530+0000","last_change":"2023-12-19T14:58:22.548710+0000","last_active":"2023-12-19T14:58:23.087530+0000","last_peered":"2023-12-19T14:58:23.087530+0000","last_clean":"2023-12-19T14:58:23.087530+0000","last_became_active":"2023-12-19T14:58:22.548357+0000","last_became_peered":"2023-12-19T14:58:22.548357+0000","last_unstale":"2023-12-19T14:58:23.087530+0000","last_undegraded":"2023-12-19T14:58:23.087530+0000","last_fullsized":"2023-12-19T14:58:23.087530+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:21.392884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100211+0000","last_change":"2023-12-19T14:58:23.096467+0000","last_active":"2023-12-19T14:58:23.100211+0000","last_peered":"2023-12-19T14:58:23.100211+0000","last_clean":"2023-12-19T14:58:23.100211+0000","last_became_active":"2023-12-19T14:58:23.096228+0000","last_became_peered":"2023-12-19T14:58:23.096228+0000","last_unstale":"2023-12-19T14:58:23.100211+0000","last_undegraded":"2023-12-19T14:58:23.100211+0000","last_fullsized":"2023-12-19T14:58:23.100211+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.569453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086594+0000","last_change":"2023-12-19T14:58:22.547152+0000","last_active":"2023-12-19T14:58:23.086594+0000","last_peered":"2023-12-19T14:58:23.086594+0000","last_clean":"2023-12-19T14:58:23.086594+0000","last_became_active":"2023-12-19T14:58:22.545361+0000","last_became_peered":"2023-12-19T14:58:22.545361+0000","last_unstale":"2023-12-19T14:58:23.086594+0000","last_undegraded":"2023-12-19T14:58:23.086594+0000","last_fullsized":"2023-12-19T14:58:23.086594+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:39.209494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099094+0000","last_change":"2023-12-19T14:58:23.084657+0000","last_active":"2023-12-19T14:58:23.099094+0000","last_peered":"2023-12-19T14:58:23.099094+0000","last_clean":"2023-12-19T14:58:23.099094+0000","last_became_active":"2023-12-19T14:58:23.084280+0000","last_became_peered":"2023-12-19T14:58:23.084280+0000","last_unstale":"2023-12-19T14:58:23.099094+0000","last_undegraded":"2023-12-19T14:58:23.099094+0000","last_fullsized":"2023-12-19T14:58:23.099094+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:11.698607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.070720+0000","last_change":"2023-12-19T14:58:22.553836+0000","last_active":"2023-12-19T14:58:23.070720+0000","last_peered":"2023-12-19T14:58:23.070720+0000","last_clean":"2023-12-19T14:58:23.070720+0000","last_became_active":"2023-12-19T14:58:22.551441+0000","last_became_peered":"2023-12-19T14:58:22.551441+0000","last_unstale":"2023-12-19T14:58:23.070720+0000","last_undegraded":"2023-12-19T14:58:23.070720+0000","last_fullsized":"2023-12-19T14:58:23.070720+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090335+0000","last_change":"2023-12-19T14:58:23.090335+0000","last_active":"2023-12-19T14:58:23.090335+0000","last_peered":"2023-12-19T14:58:23.090335+0000","last_clean":"2023-12-19T14:58:23.090335+0000","last_became_active":"2023-12-19T14:58:23.089408+0000","last_became_peered":"2023-12-19T14:58:23.089408+0000","last_unstale":"2023-12-19T14:58:23.090335+0000","last_undegraded":"2023-12-19T14:58:23.090335+0000","last_fullsized":"2023-12-19T14:58:23.090335+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:06.593293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.547947+0000","last_change":"2023-12-19T14:58:22.552973+0000","last_active":"2023-12-19T14:58:23.547947+0000","last_peered":"2023-12-19T14:58:23.547947+0000","last_clean":"2023-12-19T14:58:23.547947+0000","last_became_active":"2023-12-19T14:58:22.552518+0000","last_became_peered":"2023-12-19T14:58:22.552518+0000","last_unstale":"2023-12-19T14:58:23.547947+0000","last_undegraded":"2023-12-19T14:58:23.547947+0000","last_fullsized":"2023-12-19T14:58:23.547947+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:24:09.668268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089775+0000","last_change":"2023-12-19T14:58:23.089775+0000","last_active":"2023-12-19T14:58:23.089775+0000","last_peered":"2023-12-19T14:58:23.089775+0000","last_clean":"2023-12-19T14:58:23.089775+0000","last_became_active":"2023-12-19T14:58:23.088713+0000","last_became_peered":"2023-12-19T14:58:23.088713+0000","last_unstale":"2023-12-19T14:58:23.089775+0000","last_undegraded":"2023-12-19T14:58:23.089775+0000","last_fullsized":"2023-12-19T14:58:23.089775+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:30.812983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.072508+0000","last_change":"2023-12-19T14:58:22.541086+0000","last_active":"2023-12-19T14:58:23.072508+0000","last_peered":"2023-12-19T14:58:23.072508+0000","last_clean":"2023-12-19T14:58:23.072508+0000","last_became_active":"2023-12-19T14:58:22.539398+0000","last_became_peered":"2023-12-19T14:58:22.539398+0000","last_unstale":"2023-12-19T14:58:23.072508+0000","last_undegraded":"2023-12-19T14:58:23.072508+0000","last_fullsized":"2023-12-19T14:58:23.072508+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099981+0000","last_change":"2023-12-19T14:58:23.090294+0000","last_active":"2023-12-19T14:58:23.099981+0000","last_peered":"2023-12-19T14:58:23.099981+0000","last_clean":"2023-12-19T14:58:23.099981+0000","last_became_active":"2023-12-19T14:58:23.088382+0000","last_became_peered":"2023-12-19T14:58:23.088382+0000","last_unstale":"2023-12-19T14:58:23.099981+0000","last_undegraded":"2023-12-19T14:58:23.099981+0000","last_fullsized":"2023-12-19T14:58:23.099981+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:36.808951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.543958+0000","last_change":"2023-12-19T14:58:22.541886+0000","last_active":"2023-12-19T14:58:22.543958+0000","last_peered":"2023-12-19T14:58:22.543958+0000","last_clean":"2023-12-19T14:58:22.543958+0000","last_became_active":"2023-12-19T14:58:22.541568+0000","last_became_peered":"2023-12-19T14:58:22.541568+0000","last_unstale":"2023-12-19T14:58:22.543958+0000","last_undegraded":"2023-12-19T14:58:22.543958+0000","last_fullsized":"2023-12-19T14:58:22.543958+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:33:47.724646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092263+0000","last_change":"2023-12-19T14:58:23.092263+0000","last_active":"2023-12-19T14:58:23.092263+0000","last_peered":"2023-12-19T14:58:23.092263+0000","last_clean":"2023-12-19T14:58:23.092263+0000","last_became_active":"2023-12-19T14:58:23.091976+0000","last_became_peered":"2023-12-19T14:58:23.091976+0000","last_unstale":"2023-12-19T14:58:23.092263+0000","last_undegraded":"2023-12-19T14:58:23.092263+0000","last_fullsized":"2023-12-19T14:58:23.092263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:37.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.548431+0000","last_change":"2023-12-19T14:58:22.550721+0000","last_active":"2023-12-19T14:58:23.548431+0000","last_peered":"2023-12-19T14:58:23.548431+0000","last_clean":"2023-12-19T14:58:23.548431+0000","last_became_active":"2023-12-19T14:58:22.550317+0000","last_became_peered":"2023-12-19T14:58:22.550317+0000","last_unstale":"2023-12-19T14:58:23.548431+0000","last_undegraded":"2023-12-19T14:58:23.548431+0000","last_fullsized":"2023-12-19T14:58:23.548431+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:25:38.261204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102159+0000","last_change":"2023-12-19T14:58:23.100318+0000","last_active":"2023-12-19T14:58:23.102159+0000","last_peered":"2023-12-19T14:58:23.102159+0000","last_clean":"2023-12-19T14:58:23.102159+0000","last_became_active":"2023-12-19T14:58:23.100042+0000","last_became_peered":"2023-12-19T14:58:23.100042+0000","last_unstale":"2023-12-19T14:58:23.102159+0000","last_undegraded":"2023-12-19T14:58:23.102159+0000","last_fullsized":"2023-12-19T14:58:23.102159+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:51.014043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.539079+0000","last_change":"2023-12-19T14:58:22.537336+0000","last_active":"2023-12-19T14:58:22.539079+0000","last_peered":"2023-12-19T14:58:22.539079+0000","last_clean":"2023-12-19T14:58:22.539079+0000","last_became_active":"2023-12-19T14:58:22.535064+0000","last_became_peered":"2023-12-19T14:58:22.535064+0000","last_unstale":"2023-12-19T14:58:22.539079+0000","last_undegraded":"2023-12-19T14:58:22.539079+0000","last_fullsized":"2023-12-19T14:58:22.539079+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:30:03.178463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098949+0000","last_change":"2023-12-19T14:58:23.094681+0000","last_active":"2023-12-19T14:58:23.098949+0000","last_peered":"2023-12-19T14:58:23.098949+0000","last_clean":"2023-12-19T14:58:23.098949+0000","last_became_active":"2023-12-19T14:58:23.093182+0000","last_became_peered":"2023-12-19T14:58:23.093182+0000","last_unstale":"2023-12-19T14:58:23.098949+0000","last_undegraded":"2023-12-19T14:58:23.098949+0000","last_fullsized":"2023-12-19T14:58:23.098949+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:59.466905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086730+0000","last_change":"2023-12-19T14:58:22.551548+0000","last_active":"2023-12-19T14:58:23.086730+0000","last_peered":"2023-12-19T14:58:23.086730+0000","last_clean":"2023-12-19T14:58:23.086730+0000","last_became_active":"2023-12-19T14:58:22.551256+0000","last_became_peered":"2023-12-19T14:58:22.551256+0000","last_unstale":"2023-12-19T14:58:23.086730+0000","last_undegraded":"2023-12-19T14:58:23.086730+0000","last_fullsized":"2023-12-19T14:58:23.086730+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094354+0000","last_change":"2023-12-19T14:58:23.085891+0000","last_active":"2023-12-19T14:58:23.094354+0000","last_peered":"2023-12-19T14:58:23.094354+0000","last_clean":"2023-12-19T14:58:23.094354+0000","last_became_active":"2023-12-19T14:58:23.085561+0000","last_became_peered":"2023-12-19T14:58:23.085561+0000","last_unstale":"2023-12-19T14:58:23.094354+0000","last_undegraded":"2023-12-19T14:58:23.094354+0000","last_fullsized":"2023-12-19T14:58:23.094354+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:25:39.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.537761+0000","last_change":"2023-12-19T14:58:22.535017+0000","last_active":"2023-12-19T14:58:22.537761+0000","last_peered":"2023-12-19T14:58:22.537761+0000","last_clean":"2023-12-19T14:58:22.537761+0000","last_became_active":"2023-12-19T14:58:22.534519+0000","last_became_peered":"2023-12-19T14:58:22.534519+0000","last_unstale":"2023-12-19T14:58:22.537761+0000","last_undegraded":"2023-12-19T14:58:22.537761+0000","last_fullsized":"2023-12-19T14:58:22.537761+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:57.722647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.080640+0000","last_change":"2023-12-19T14:58:23.080640+0000","last_active":"2023-12-19T14:58:23.080640+0000","last_peered":"2023-12-19T14:58:23.080640+0000","last_clean":"2023-12-19T14:58:23.080640+0000","last_became_active":"2023-12-19T14:58:23.080326+0000","last_became_peered":"2023-12-19T14:58:23.080326+0000","last_unstale":"2023-12-19T14:58:23.080640+0000","last_undegraded":"2023-12-19T14:58:23.080640+0000","last_fullsized":"2023-12-19T14:58:23.080640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:27:47.982461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.547877+0000","last_change":"2023-12-19T14:58:22.547978+0000","last_active":"2023-12-19T14:58:23.547877+0000","last_peered":"2023-12-19T14:58:23.547877+0000","last_clean":"2023-12-19T14:58:23.547877+0000","last_became_active":"2023-12-19T14:58:22.547702+0000","last_became_peered":"2023-12-19T14:58:22.547702+0000","last_unstale":"2023-12-19T14:58:23.547877+0000","last_undegraded":"2023-12-19T14:58:23.547877+0000","last_fullsized":"2023-12-19T14:58:23.547877+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:58:58.935568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092890+0000","last_change":"2023-12-19T14:58:23.092890+0000","last_active":"2023-12-19T14:58:23.092890+0000","last_peered":"2023-12-19T14:58:23.092890+0000","last_clean":"2023-12-19T14:58:23.092890+0000","last_became_active":"2023-12-19T14:58:23.092585+0000","last_became_peered":"2023-12-19T14:58:23.092585+0000","last_unstale":"2023-12-19T14:58:23.092890+0000","last_undegraded":"2023-12-19T14:58:23.092890+0000","last_fullsized":"2023-12-19T14:58:23.092890+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550993+0000","last_change":"2023-12-19T14:58:22.550097+0000","last_active":"2023-12-19T14:58:22.550993+0000","last_peered":"2023-12-19T14:58:22.550993+0000","last_clean":"2023-12-19T14:58:22.550993+0000","last_became_active":"2023-12-19T14:58:22.549256+0000","last_became_peered":"2023-12-19T14:58:22.549256+0000","last_unstale":"2023-12-19T14:58:22.550993+0000","last_undegraded":"2023-12-19T14:58:22.550993+0000","last_fullsized":"2023-12-19T14:58:22.550993+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092578+0000","last_change":"2023-12-19T14:58:23.092578+0000","last_active":"2023-12-19T14:58:23.092578+0000","last_peered":"2023-12-19T14:58:23.092578+0000","last_clean":"2023-12-19T14:58:23.092578+0000","last_became_active":"2023-12-19T14:58:23.092247+0000","last_became_peered":"2023-12-19T14:58:23.092247+0000","last_unstale":"2023-12-19T14:58:23.092578+0000","last_undegraded":"2023-12-19T14:58:23.092578+0000","last_fullsized":"2023-12-19T14:58:23.092578+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:54.610499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550685+0000","last_change":"2023-12-19T14:58:22.549492+0000","last_active":"2023-12-19T14:58:22.550685+0000","last_peered":"2023-12-19T14:58:22.550685+0000","last_clean":"2023-12-19T14:58:22.550685+0000","last_became_active":"2023-12-19T14:58:22.549178+0000","last_became_peered":"2023-12-19T14:58:22.549178+0000","last_unstale":"2023-12-19T14:58:22.550685+0000","last_undegraded":"2023-12-19T14:58:22.550685+0000","last_fullsized":"2023-12-19T14:58:22.550685+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102741+0000","last_change":"2023-12-19T14:58:23.101559+0000","last_active":"2023-12-19T14:58:23.102741+0000","last_peered":"2023-12-19T14:58:23.102741+0000","last_clean":"2023-12-19T14:58:23.102741+0000","last_became_active":"2023-12-19T14:58:23.101246+0000","last_became_peered":"2023-12-19T14:58:23.101246+0000","last_unstale":"2023-12-19T14:58:23.102741+0000","last_undegraded":"2023-12-19T14:58:23.102741+0000","last_fullsized":"2023-12-19T14:58:23.102741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:48:08.519586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.555138+0000","last_change":"2023-12-19T14:58:22.548601+0000","last_active":"2023-12-19T14:58:22.555138+0000","last_peered":"2023-12-19T14:58:22.555138+0000","last_clean":"2023-12-19T14:58:22.555138+0000","last_became_active":"2023-12-19T14:58:22.548329+0000","last_became_peered":"2023-12-19T14:58:22.548329+0000","last_unstale":"2023-12-19T14:58:22.555138+0000","last_undegraded":"2023-12-19T14:58:22.555138+0000","last_fullsized":"2023-12-19T14:58:22.555138+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:44.635549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094091+0000","last_change":"2023-12-19T14:58:23.094091+0000","last_active":"2023-12-19T14:58:23.094091+0000","last_peered":"2023-12-19T14:58:23.094091+0000","last_clean":"2023-12-19T14:58:23.094091+0000","last_became_active":"2023-12-19T14:58:23.093525+0000","last_became_peered":"2023-12-19T14:58:23.093525+0000","last_unstale":"2023-12-19T14:58:23.094091+0000","last_undegraded":"2023-12-19T14:58:23.094091+0000","last_fullsized":"2023-12-19T14:58:23.094091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:12:49.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.545552+0000","last_change":"2023-12-19T14:58:22.544171+0000","last_active":"2023-12-19T14:58:22.545552+0000","last_peered":"2023-12-19T14:58:22.545552+0000","last_clean":"2023-12-19T14:58:22.545552+0000","last_became_active":"2023-12-19T14:58:22.543735+0000","last_became_peered":"2023-12-19T14:58:22.543735+0000","last_unstale":"2023-12-19T14:58:22.545552+0000","last_undegraded":"2023-12-19T14:58:22.545552+0000","last_fullsized":"2023-12-19T14:58:22.545552+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:12.521483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095698+0000","last_change":"2023-12-19T14:58:23.089594+0000","last_active":"2023-12-19T14:58:23.095698+0000","last_peered":"2023-12-19T14:58:23.095698+0000","last_clean":"2023-12-19T14:58:23.095698+0000","last_became_active":"2023-12-19T14:58:23.089319+0000","last_became_peered":"2023-12-19T14:58:23.089319+0000","last_unstale":"2023-12-19T14:58:23.095698+0000","last_undegraded":"2023-12-19T14:58:23.095698+0000","last_fullsized":"2023-12-19T14:58:23.095698+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.544943+0000","last_change":"2023-12-19T14:58:22.542985+0000","last_active":"2023-12-19T14:58:22.544943+0000","last_peered":"2023-12-19T14:58:22.544943+0000","last_clean":"2023-12-19T14:58:22.544943+0000","last_became_active":"2023-12-19T14:58:22.541624+0000","last_became_peered":"2023-12-19T14:58:22.541624+0000","last_unstale":"2023-12-19T14:58:22.544943+0000","last_undegraded":"2023-12-19T14:58:22.544943+0000","last_fullsized":"2023-12-19T14:58:22.544943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:33.271134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081176+0000","last_change":"2023-12-19T14:58:23.081176+0000","last_active":"2023-12-19T14:58:23.081176+0000","last_peered":"2023-12-19T14:58:23.081176+0000","last_clean":"2023-12-19T14:58:23.081176+0000","last_became_active":"2023-12-19T14:58:23.080737+0000","last_became_peered":"2023-12-19T14:58:23.080737+0000","last_unstale":"2023-12-19T14:58:23.081176+0000","last_undegraded":"2023-12-19T14:58:23.081176+0000","last_fullsized":"2023-12-19T14:58:23.081176+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:51.014043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550877+0000","last_change":"2023-12-19T14:58:22.543457+0000","last_active":"2023-12-19T14:58:22.550877+0000","last_peered":"2023-12-19T14:58:22.550877+0000","last_clean":"2023-12-19T14:58:22.550877+0000","last_became_active":"2023-12-19T14:58:22.543155+0000","last_became_peered":"2023-12-19T14:58:22.543155+0000","last_unstale":"2023-12-19T14:58:22.550877+0000","last_undegraded":"2023-12-19T14:58:22.550877+0000","last_fullsized":"2023-12-19T14:58:22.550877+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:10.683649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.084247+0000","last_change":"2023-12-19T14:58:23.084247+0000","last_active":"2023-12-19T14:58:23.084247+0000","last_peered":"2023-12-19T14:58:23.084247+0000","last_clean":"2023-12-19T14:58:23.084247+0000","last_became_active":"2023-12-19T14:58:23.083644+0000","last_became_peered":"2023-12-19T14:58:23.083644+0000","last_unstale":"2023-12-19T14:58:23.084247+0000","last_undegraded":"2023-12-19T14:58:23.084247+0000","last_fullsized":"2023-12-19T14:58:23.084247+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:20:24.055272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.537782+0000","last_change":"2023-12-19T14:58:22.535352+0000","last_active":"2023-12-19T14:58:22.537782+0000","last_peered":"2023-12-19T14:58:22.537782+0000","last_clean":"2023-12-19T14:58:22.537782+0000","last_became_active":"2023-12-19T14:58:22.535049+0000","last_became_peered":"2023-12-19T14:58:22.535049+0000","last_unstale":"2023-12-19T14:58:22.537782+0000","last_undegraded":"2023-12-19T14:58:22.537782+0000","last_fullsized":"2023-12-19T14:58:22.537782+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:10.490471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.082637+0000","last_change":"2023-12-19T14:58:23.082637+0000","last_active":"2023-12-19T14:58:23.082637+0000","last_peered":"2023-12-19T14:58:23.082637+0000","last_clean":"2023-12-19T14:58:23.082637+0000","last_became_active":"2023-12-19T14:58:23.081999+0000","last_became_peered":"2023-12-19T14:58:23.081999+0000","last_unstale":"2023-12-19T14:58:23.082637+0000","last_undegraded":"2023-12-19T14:58:23.082637+0000","last_fullsized":"2023-12-19T14:58:23.082637+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.551741+0000","last_change":"2023-12-19T14:58:22.545204+0000","last_active":"2023-12-19T14:58:23.551741+0000","last_peered":"2023-12-19T14:58:23.551741+0000","last_clean":"2023-12-19T14:58:23.551741+0000","last_became_active":"2023-12-19T14:58:22.544771+0000","last_became_peered":"2023-12-19T14:58:22.544771+0000","last_unstale":"2023-12-19T14:58:23.551741+0000","last_undegraded":"2023-12-19T14:58:23.551741+0000","last_fullsized":"2023-12-19T14:58:23.551741+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:05.419064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'249","reported_seq":732,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.522267+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:58:22.522267+0000","last_peered":"2023-12-19T14:58:22.522267+0000","last_clean":"2023-12-19T14:58:22.522267+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:58:22.522267+0000","last_undegraded":"2023-12-19T14:58:22.522267+0000","last_fullsized":"2023-12-19T14:58:22.522267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311908,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059932,"statfs":{"total":96636764160,"available":96317370368,"internally_reserved":0,"allocated":2387968,"data_stored":1486990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0820000000000001,"15min":1.093},"min":{"1min":0.69999999999999996,"5min":0.65100000000000002,"15min":0.52300000000000002},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":3.544},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.0880000000000001,"15min":1.1319999999999999},"min":{"1min":0.69999999999999996,"5min":0.629,"15min":0.44700000000000001},"max":{"1min":1.405,"5min":1.5780000000000001,"15min":7.8239999999999998},"last":0.89200000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.109,"15min":1.1080000000000001},"min":{"1min":0.755,"5min":0.69499999999999995,"15min":0.627},"max":{"1min":1.5449999999999999,"5min":1.722,"15min":1.73},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.196,"15min":1.1859999999999999},"min":{"1min":0.60999999999999999,"5min":0.59999999999999998,"15min":0.48699999999999999},"max":{"1min":1.417,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.6140000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.264,"15min":1.2370000000000001},"min":{"1min":0.91200000000000003,"5min":0.77900000000000003,"15min":0.67900000000000005},"max":{"1min":1.488,"5min":1.583,"15min":1.6919999999999999},"last":1.494},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2789999999999999,"15min":1.28},"min":{"1min":0.83499999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.627,"5min":1.6739999999999999,"15min":1.6890000000000001},"last":1.353}]},{"osd":3,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2669999999999999,"15min":1.2869999999999999},"min":{"1min":1.0329999999999999,"5min":0.79700000000000004,"15min":0.78000000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7969999999999999},"last":2.0739999999999998},{"interface":"front","average":{"1min":1.353,"5min":1.3300000000000001,"15min":1.3160000000000001},"min":{"1min":0.99399999999999999,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.7010000000000001,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.849}]},{"osd":4,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.1859999999999999,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.66900000000000004,"15min":0.53500000000000003},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2350000000000001,"15min":1.2170000000000001},"min":{"1min":1.0209999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.7290000000000001},"last":1.8939999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.242,"15min":1.226},"min":{"1min":0.89900000000000002,"5min":0.52100000000000002,"15min":0.496},"max":{"1min":1.5860000000000001,"5min":1.73,"15min":1.8109999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3280000000000001,"15min":1.306},"min":{"1min":1.0049999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":2.105}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.409,"15min":1.391},"min":{"1min":0.99099999999999999,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.724,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":2.008},{"interface":"front","average":{"1min":1.429,"5min":1.383,"15min":1.3580000000000001},"min":{"1min":0.93799999999999994,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.837,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":2.1720000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1781760,"data_stored":896428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.139,"15min":1.1339999999999999},"min":{"1min":0.53900000000000003,"5min":0.499,"15min":0.41299999999999998},"max":{"1min":1.5309999999999999,"5min":1.611,"15min":1.9219999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1419999999999999,"15min":1.1639999999999999},"min":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.72799999999999998},"max":{"1min":1.5800000000000001,"5min":1.845,"15min":1.845},"last":0.89900000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1120000000000001,"15min":1.1830000000000001},"min":{"1min":0.78300000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.5249999999999999,"5min":2.016,"15min":4.4340000000000002},"last":1.101},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.68100000000000005},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":3.226},"last":1.6439999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2090000000000001,"15min":1.2370000000000001},"min":{"1min":0.754,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.597,"5min":1.7649999999999999,"15min":3.6280000000000001},"last":1.284},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.202,"15min":1.242},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.70799999999999996},"max":{"1min":1.573,"5min":1.802,"15min":3.1619999999999999},"last":1.5609999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.29,"15min":1.296},"min":{"1min":0.76200000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.782,"5min":1.782,"15min":1.9239999999999999},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.3009999999999999,"15min":1.3240000000000001},"min":{"1min":0.96599999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.702,"5min":1.8779999999999999,"15min":3.6899999999999999},"last":1.024}]},{"osd":4,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1639999999999999,"15min":1.2010000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.621},"max":{"1min":1.3959999999999999,"5min":2.1040000000000001,"15min":3.9359999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.262,"15min":1.2869999999999999},"min":{"1min":0.70099999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6579999999999999,"5min":1.901,"15min":3.3460000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.268,"15min":1.2869999999999999},"min":{"1min":0.91000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6599999999999999,"5min":1.746,"15min":2.0219999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.306,"15min":1.353},"min":{"1min":0.997,"5min":0.82799999999999996,"15min":0.76300000000000001},"max":{"1min":1.657,"5min":1.931,"15min":5.9610000000000003},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4039999999999999,"15min":1.411},"min":{"1min":0.86599999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8,"5min":1.847,"15min":4.2510000000000003},"last":1.347},{"interface":"front","average":{"1min":1.482,"5min":1.4259999999999999,"15min":1.4510000000000001},"min":{"1min":1.1639999999999999,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":1.784,"5min":2.0910000000000002,"15min":3.742},"last":1.5960000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575078,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303140,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068700,"statfs":{"total":96636764160,"available":96326348800,"internally_reserved":0,"allocated":1798144,"data_stored":902698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.101,"15min":1.069},"min":{"1min":0.78600000000000003,"5min":0.63700000000000001,"15min":0.40799999999999997},"max":{"1min":1.5660000000000001,"5min":1.601,"15min":1.601},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.137,"5min":1.1299999999999999,"15min":1.135},"min":{"1min":0.754,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5800000000000001,"5min":1.6539999999999999,"15min":1.7290000000000001},"last":1.2}]},{"osd":1,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.274,"15min":1.288},"min":{"1min":0.78600000000000003,"5min":0.66300000000000003,"15min":0.627},"max":{"1min":1.601,"5min":1.863,"15min":1.9650000000000001},"last":0.877},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.238,"15min":1.2350000000000001},"min":{"1min":0.81699999999999995,"5min":0.77300000000000002,"15min":0.55700000000000005},"max":{"1min":1.6259999999999999,"5min":1.677,"15min":1.677},"last":0.64400000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.96999999999999997,"5min":0.61499999999999999,"15min":0.47399999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.9319999999999999},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.1950000000000001,"15min":1.1870000000000001},"min":{"1min":0.89900000000000002,"5min":0.82099999999999995,"15min":0.54300000000000004},"max":{"1min":1.5820000000000001,"5min":1.7,"15min":1.8600000000000001},"last":1.3740000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.2649999999999999},"min":{"1min":0.91200000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":2.0600000000000001},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.26,"15min":1.2589999999999999},"min":{"1min":0.94399999999999995,"5min":0.74299999999999999,"15min":0.68999999999999995},"max":{"1min":1.5049999999999999,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.163}]},{"osd":4,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.103,"15min":1.0940000000000001},"min":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.4359999999999999,"5min":1.651,"15min":1.651},"last":0.753},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.24,"15min":1.2290000000000001},"min":{"1min":0.68200000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5800000000000001,"5min":1.7709999999999999,"15min":2.27},"last":1.3049999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.355,"15min":1.331},"min":{"1min":0.98399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.7250000000000001,"5min":1.982,"15min":1.984},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3220000000000001,"15min":1.3169999999999999},"min":{"1min":0.92000000000000004,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.6839999999999999,"5min":1.794,"15min":1.8540000000000001},"last":1.0800000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.363,"15min":1.347},"min":{"1min":0.92100000000000004,"5min":0.64500000000000002,"15min":0.58899999999999997},"max":{"1min":1.6499999999999999,"5min":1.7509999999999999,"15min":2.1720000000000002},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3919999999999999,"15min":1.375},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.68999999999999995},"max":{"1min":1.6830000000000001,"5min":2.048,"15min":2.048},"last":1.4690000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575078,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303132,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068708,"statfs":{"total":96636764160,"available":96326356992,"internally_reserved":0,"allocated":1789952,"data_stored":897410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.03,"15min":1.056},"min":{"1min":0.65600000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":1.625,"15min":1.7869999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.0900000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.524,"5min":1.7490000000000001,"15min":1.8140000000000001},"last":1.0620000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.593,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.615},{"interface":"front","average":{"1min":1.242,"5min":1.2250000000000001,"15min":1.266},"min":{"1min":0.98899999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.4430000000000001,"5min":1.677,"15min":2.7679999999999998},"last":1.5600000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1759999999999999,"15min":1.1459999999999999},"min":{"1min":0.78400000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.143,"15min":1.2},"min":{"1min":0.78200000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.385,"5min":1.7649999999999999,"15min":1.893},"last":1.3979999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.84999999999999998,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.528,"5min":1.6100000000000001,"15min":2.0190000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2490000000000001,"15min":1.2569999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7749999999999999},"last":1.2030000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.139,"15min":1.139},"min":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.57899999999999996},"max":{"1min":1.5089999999999999,"5min":1.6559999999999999,"15min":2.1800000000000002},"last":1.653},{"interface":"front","average":{"1min":1.107,"5min":1.129,"15min":1.1439999999999999},"min":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.51300000000000001},"max":{"1min":1.5920000000000001,"5min":1.6830000000000001,"15min":2.077},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.248,"15min":1.2889999999999999},"min":{"1min":0.68999999999999995,"5min":0.63900000000000001,"15min":0.55000000000000004},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.8119999999999998},"last":1.5},{"interface":"front","average":{"1min":1.383,"5min":1.3779999999999999,"15min":1.399},"min":{"1min":1.024,"5min":0.63400000000000001,"15min":0.60999999999999999},"max":{"1min":1.6040000000000001,"5min":2.1549999999999998,"15min":3.0379999999999998},"last":1.732}]},{"osd":7,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2809999999999999,"15min":1.3089999999999999},"min":{"1min":0.89600000000000002,"5min":0.69299999999999995,"15min":0.60899999999999999},"max":{"1min":1.5429999999999999,"5min":1.639,"15min":2.9910000000000001},"last":1.593},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.429,"15min":1.4279999999999999},"min":{"1min":0.97099999999999997,"5min":0.755,"15min":0.72599999999999998},"max":{"1min":1.9079999999999999,"5min":2.0329999999999999,"15min":3.008},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":904108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.96899999999999997,"15min":0.98199999999999998},"min":{"1min":0.64300000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.534,"5min":1.542,"15min":1.7270000000000001},"last":0.93300000000000005}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91400000000000003,"15min":0.93799999999999994},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2090000000000001,"5min":1.611,"15min":1.611},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.53700000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.3280000000000001,"5min":1.5780000000000001,"15min":1.758},"last":0.84499999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.117,"15min":1.123},"min":{"1min":0.64200000000000002,"5min":0.61499999999999999,"15min":0.56100000000000005},"max":{"1min":1.3460000000000001,"5min":1.778,"15min":1.778},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.77600000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6519999999999999,"5min":1.72,"15min":1.821},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3069999999999999,"15min":1.319},"min":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.6679999999999999,"5min":1.7589999999999999,"15min":1.782},"last":0.997},{"interface":"front","average":{"1min":1.204,"5min":1.2649999999999999,"15min":1.286},"min":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.708,"5min":1.708,"15min":1.7989999999999999},"last":1.0309999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3069999999999999,"15min":1.3009999999999999},"min":{"1min":0.95999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6990000000000001,"5min":1.7709999999999999,"15min":1.893},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.3540000000000001,"15min":1.3779999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.6240000000000001,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.206}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.423,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7070000000000001,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.077},{"interface":"front","average":{"1min":1.323,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.85499999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6659999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.177}]},{"osd":7,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4950000000000001,"15min":1.502},"min":{"1min":1.0129999999999999,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.7430000000000001,"5min":1.9970000000000001,"15min":2.1640000000000001},"last":1.147},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4419999999999999,"15min":1.448},"min":{"1min":1.087,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.6990000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.1319999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307224,"kb_used_data":1744,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064616,"statfs":{"total":96636764160,"available":96322166784,"internally_reserved":0,"allocated":1785856,"data_stored":901527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.93999999999999995,"15min":0.91000000000000003},"min":{"1min":0.66000000000000003,"5min":0.60199999999999998,"15min":0.39000000000000001},"max":{"1min":1.3260000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.288},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.05,"15min":1.0609999999999999},"min":{"1min":0.70299999999999996,"5min":0.57299999999999995,"15min":0.53300000000000003},"max":{"1min":1.524,"5min":2.036,"15min":2.036},"last":1.323}]},{"osd":1,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1659999999999999,"15min":1.137},"min":{"1min":0.93500000000000005,"5min":0.64300000000000002,"15min":0.56599999999999995},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0509999999999999,"15min":1.069},"min":{"1min":0.71599999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.5609999999999999,"5min":1.853,"15min":1.853},"last":1.5249999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.995,"15min":0.99399999999999999},"min":{"1min":0.80000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.623},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0760000000000001,"15min":1.0860000000000001},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.2829999999999999,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.385,"15min":1.3799999999999999},"min":{"1min":1.1339999999999999,"5min":0.747,"15min":0.58699999999999997},"max":{"1min":1.8580000000000001,"5min":2.0270000000000001,"15min":2.044},"last":1.446},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3919999999999999,"15min":1.393},"min":{"1min":1.087,"5min":0.85299999999999998,"15min":0.746},"max":{"1min":1.802,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.391}]},{"osd":5,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3600000000000001,"15min":1.341},"min":{"1min":1.131,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7769999999999999,"5min":2.0979999999999999,"15min":2.395},"last":1.589},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.331,"15min":1.331},"min":{"1min":1.073,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.5549999999999999,"5min":2.0819999999999999,"15min":2.1200000000000001},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":1.075,"5min":0.85699999999999998,"15min":0.83299999999999996},"max":{"1min":1.8100000000000001,"5min":1.9710000000000001,"15min":1.9850000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.47,"15min":1.4370000000000001},"min":{"1min":1.113,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.0390000000000001,"5min":2.2469999999999999,"15min":2.4729999999999999},"last":2.0390000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4950000000000001,"15min":1.4730000000000001},"min":{"1min":1.0509999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.7929999999999999,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.4690000000000001,"15min":1.4590000000000001},"min":{"1min":1.01,"5min":1.01,"15min":0.83599999999999997},"max":{"1min":1.994,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.994}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307220,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064620,"statfs":{"total":96636764160,"available":96322170880,"internally_reserved":0,"allocated":1781760,"data_stored":895896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88400000000000001,"15min":0.86699999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.48499999999999999},"max":{"1min":1.5580000000000001,"5min":2.238,"15min":2.238},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.96699999999999997,"15min":0.97699999999999998},"min":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.53100000000000003},"max":{"1min":1.486,"5min":1.49,"15min":1.7070000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0249999999999999,"15min":1.026},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.54600000000000004},"max":{"1min":1.9790000000000001,"5min":2.444,"15min":2.444},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.0880000000000001,"15min":1.1100000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.442},"max":{"1min":1.829,"5min":1.829,"15min":2.0230000000000001},"last":1.3979999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.216,"15min":1.228},"min":{"1min":0.82699999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.5169999999999999,"5min":1.6919999999999999,"15min":2.0720000000000001},"last":1.5429999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.085,"15min":1.0800000000000001},"min":{"1min":0.72699999999999998,"5min":0.628,"15min":0.56899999999999995},"max":{"1min":1.302,"5min":1.6890000000000001,"15min":1.7270000000000001},"last":1.272}]},{"osd":4,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.304,"15min":1.3100000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.624},"max":{"1min":2.5539999999999998,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.464},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3839999999999999,"15min":1.399},"min":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.67700000000000005},"max":{"1min":1.7190000000000001,"5min":1.8029999999999999,"15min":3.5630000000000002},"last":1.798}]},{"osd":5,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2669999999999999,"15min":1.286},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.80000000000000004},"max":{"1min":1.661,"5min":1.857,"15min":1.857},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.421,"15min":1.389},"min":{"1min":0.88,"5min":0.86399999999999999,"15min":0.71699999999999997},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.661}]},{"osd":6,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.355,"15min":1.3620000000000001},"min":{"1min":1.002,"5min":0.97899999999999998,"15min":0.53000000000000003},"max":{"1min":1.627,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4410000000000001,"15min":1.45},"min":{"1min":0.96099999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.6389999999999998,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.7709999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4419999999999999,"15min":1.4670000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.89500000000000002},"max":{"1min":1.923,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.704},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.448,"15min":1.4390000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.93799999999999994},"max":{"1min":1.863,"5min":2.0169999999999999,"15min":2.0609999999999999},"last":1.734}]}]},{"osd":0,"up_from":13,"seq":55834575077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":2371584,"data_stored":1485253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.86899999999999999,"15min":0.85299999999999998},"min":{"1min":0.64300000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.96099999999999997,"15min":0.95999999999999996},"min":{"1min":0.5,"5min":0.433,"15min":0.433},"max":{"1min":1.1970000000000001,"5min":1.577,"15min":1.8879999999999999},"last":0.87}]},{"osd":2,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.141,"15min":1.107},"min":{"1min":0.74199999999999999,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.5580000000000001,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.2090000000000001,"15min":1.2210000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7909999999999999,"5min":1.821,"15min":1.982},"last":1.071}]},{"osd":3,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.98199999999999998,"15min":1.01},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.216,"5min":1.6359999999999999,"15min":1.726},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.181},"min":{"1min":0.63300000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.714,"5min":1.829,"15min":2.0209999999999999},"last":1.177}]},{"osd":4,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.81999999999999995,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":2.153},"last":1.214},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.274,"15min":1.284},"min":{"1min":0.96899999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6759999999999999,"5min":1.7170000000000001,"15min":1.7929999999999999},"last":1.2270000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3919999999999999,"15min":1.3799999999999999},"min":{"1min":0.96699999999999997,"5min":0.879,"15min":0.71899999999999997},"max":{"1min":1.7689999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3480000000000001,"15min":1.379},"min":{"1min":1.0589999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6479999999999999,"5min":1.9339999999999999,"15min":2.2189999999999999},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3149999999999999,"15min":1.3240000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":2.0150000000000001},"last":1.337},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.452,"15min":1.446},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.859,"5min":2.0590000000000002,"15min":2.1240000000000001},"last":1.3959999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4139999999999999,"15min":1.4510000000000001},"min":{"1min":1.0129999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.823,"5min":2.0019999999999998,"15min":2.2629999999999999},"last":1.258},{"interface":"front","average":{"1min":1.357,"5min":1.3999999999999999,"15min":1.415},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.88200000000000001},"max":{"1min":1.613,"5min":1.802,"15min":2.0379999999999998},"last":1.431}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:58:27.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T14:58:28.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.024+0000 7f19efd45700 1 Processor -- start 2023-12-19T14:58:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 -- start start 2023-12-19T14:58:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f19e81502b0 0x7f19e8068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e80c5900 0x7f19e80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80c6630 con 0x7f19e8067c90 2023-12-19T14:58:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e80c6910 con 0x7f19e81502b0 2023-12-19T14:58:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19efd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80c6bf0 con 0x7f19e80c5900 2023-12-19T14:58:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19ed2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f19e81502b0 0x7f19e8068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19ed2e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f19e81502b0 0x7f19e8068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55334/0 (socket says 172.21.15.50:55334) 2023-12-19T14:58:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19edae1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f19e8067c90 0x7f19e814fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34502/0 (socket says 172.21.15.50:34502) 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19ed2e0700 1 -- 172.21.15.50:0/3571830074 learned_addr learned my addr 172.21.15.50:0/3571830074 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.028+0000 7f19ee2e2700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e80c5900 0x7f19e80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ecadf700 1 -- 172.21.15.50:0/3571830074 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627944169 0 0) 0x7f19e80c6630 con 0x7f19e8067c90 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ecadf700 1 -- 172.21.15.50:0/3571830074 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4003570 con 0x7f19e8067c90 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ed2e0700 1 -- 172.21.15.50:0/3571830074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e80c5900 msgr2=0x7f19e80c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ed2e0700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e80c5900 0x7f19e80c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ed2e0700 1 -- 172.21.15.50:0/3571830074 >> v1:172.21.15.50:6789/0 conn(0x7f19e8067c90 legacy=0x7f19e814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ed2e0700 1 -- 172.21.15.50:0/3571830074 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80c6ed0 con 0x7f19e81502b0 2023-12-19T14:58:28.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ed2e0700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e8068060 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f19d80099b0 tx=0x7f19d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610c3c625071461e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ecadf700 1 -- 172.21.15.50:0/3571830074 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800b680 con 0x7f19e81502b0 2023-12-19T14:58:28.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ecadf700 1 -- 172.21.15.50:0/3571830074 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d800b7e0 con 0x7f19e81502b0 2023-12-19T14:58:28.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19ecadf700 1 -- 172.21.15.50:0/3571830074 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800bb60 con 0x7f19e81502b0 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 -- 172.21.15.50:0/3571830074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 msgr2=0x7f19e8068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e8068060 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f19d80099b0 tx=0x7f19d8009250 comp rx=0 tx=0).stop 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 -- 172.21.15.50:0/3571830074 shutdown_connections 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e80c5900 0x7f19e80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 --2- 172.21.15.50:0/3571830074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e8068060 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 -- 172.21.15.50:0/3571830074 >> 172.21.15.50:0/3571830074 conn(0x7f19e80cd6d0 msgr2=0x7f19e806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 -- 172.21.15.50:0/3571830074 shutdown_connections 2023-12-19T14:58:28.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.032+0000 7f19efd45700 1 -- 172.21.15.50:0/3571830074 wait complete. 2023-12-19T14:58:28.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 Processor -- start 2023-12-19T14:58:28.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 -- start start 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 0x7f19e814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e80c5900 0x7f19e81691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80c7340 con 0x7f19e81502b0 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80c75b0 con 0x7f19e80c5900 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.036+0000 7f19efd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80c7890 con 0x7f19e8067c90 2023-12-19T14:58:28.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55336/0 (socket says 172.21.15.50:55336) 2023-12-19T14:58:28.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 -- 172.21.15.50:0/1791174307 learned_addr learned my addr 172.21.15.50:0/1791174307 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:28.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ed2e0700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e80c5900 0x7f19e81691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19edae1700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 0x7f19e814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 -- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 msgr2=0x7f19e814de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 0x7f19e814de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 -- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e80c5900 msgr2=0x7f19e81691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e80c5900 0x7f19e81691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 -- 172.21.15.50:0/1791174307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80094f0 con 0x7f19e81502b0 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19edae1700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 0x7f19e814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ee2e2700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009e90 tx=0x7f19e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e000cc20 con 0x7f19e81502b0 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e000f040 con 0x7f19e81502b0 2023-12-19T14:58:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0014480 con 0x7f19e81502b0 2023-12-19T14:58:28.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e80c7b70 con 0x7f19e81502b0 2023-12-19T14:58:28.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.040+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e80cbf90 con 0x7f19e81502b0 2023-12-19T14:58:28.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.044+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e8071120 con 0x7f19e81502b0 2023-12-19T14:58:28.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.044+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f19e0014830 con 0x7f19e81502b0 2023-12-19T14:58:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.044+0000 7f19ddffb700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 0x7f19cc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.044+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f19e007d660 con 0x7f19e81502b0 2023-12-19T14:58:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.044+0000 7f19edae1700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 0x7f19cc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.048+0000 7f19edae1700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 0x7f19cc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19e4007fd0 tx=0x7f19e4000bc0 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.052+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f19e0051340 con 0x7f19e81502b0 2023-12-19T14:58:28.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.224+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19e80642a0 con 0x7f19cc060670 2023-12-19T14:58:28.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.240+0000 7f19ddffb700 1 -- 172.21.15.50:0/1791174307 <== mgr.7202 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287193 (secure 0 0 0) 0x7f19e80642a0 con 0x7f19cc060670 2023-12-19T14:58:28.242 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 msgr2=0x7f19cc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 0x7f19cc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19e4007fd0 tx=0x7f19e4000bc0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 msgr2=0x7f19e80c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009e90 tx=0x7f19e000b040 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 shutdown_connections 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19e80c5900 0x7f19e81691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19cc060670 0x7f19cc062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19e81502b0 0x7f19e80c8370 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 --2- 172.21.15.50:0/1791174307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f19e8067c90 0x7f19e814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 >> 172.21.15.50:0/1791174307 conn(0x7f19e80cd6d0 msgr2=0x7f19e80cef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 shutdown_connections 2023-12-19T14:58:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.248+0000 7f19efd45700 1 -- 172.21.15.50:0/1791174307 wait complete. 2023-12-19T14:58:28.251 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T14:58:28.270 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T14:58:26.831649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2450788,"kb_used_data":15140,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752523932,"statfs":{"total":773094113280,"available":770584506368,"internally_reserved":0,"allocated":15503360,"data_stored":8370310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.329703"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088646+0000","last_change":"2023-12-19T14:58:23.088646+0000","last_active":"2023-12-19T14:58:23.088646+0000","last_peered":"2023-12-19T14:58:23.088646+0000","last_clean":"2023-12-19T14:58:23.088646+0000","last_became_active":"2023-12-19T14:58:23.087353+0000","last_became_peered":"2023-12-19T14:58:23.087353+0000","last_unstale":"2023-12-19T14:58:23.088646+0000","last_undegraded":"2023-12-19T14:58:23.088646+0000","last_fullsized":"2023-12-19T14:58:23.088646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:36.808951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102171+0000","last_change":"2023-12-19T14:58:23.100467+0000","last_active":"2023-12-19T14:58:23.102171+0000","last_peered":"2023-12-19T14:58:23.102171+0000","last_clean":"2023-12-19T14:58:23.102171+0000","last_became_active":"2023-12-19T14:58:23.099318+0000","last_became_peered":"2023-12-19T14:58:23.099318+0000","last_unstale":"2023-12-19T14:58:23.102171+0000","last_undegraded":"2023-12-19T14:58:23.102171+0000","last_fullsized":"2023-12-19T14:58:23.102171+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.097127+0000","last_change":"2023-12-19T14:58:23.093179+0000","last_active":"2023-12-19T14:58:23.097127+0000","last_peered":"2023-12-19T14:58:23.097127+0000","last_clean":"2023-12-19T14:58:23.097127+0000","last_became_active":"2023-12-19T14:58:23.092561+0000","last_became_peered":"2023-12-19T14:58:23.092561+0000","last_unstale":"2023-12-19T14:58:23.097127+0000","last_undegraded":"2023-12-19T14:58:23.097127+0000","last_fullsized":"2023-12-19T14:58:23.097127+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094247+0000","last_change":"2023-12-19T14:58:23.094247+0000","last_active":"2023-12-19T14:58:23.094247+0000","last_peered":"2023-12-19T14:58:23.094247+0000","last_clean":"2023-12-19T14:58:23.094247+0000","last_became_active":"2023-12-19T14:58:23.093176+0000","last_became_peered":"2023-12-19T14:58:23.093176+0000","last_unstale":"2023-12-19T14:58:23.094247+0000","last_undegraded":"2023-12-19T14:58:23.094247+0000","last_fullsized":"2023-12-19T14:58:23.094247+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086509+0000","last_change":"2023-12-19T14:58:23.086509+0000","last_active":"2023-12-19T14:58:23.086509+0000","last_peered":"2023-12-19T14:58:23.086509+0000","last_clean":"2023-12-19T14:58:23.086509+0000","last_became_active":"2023-12-19T14:58:23.086160+0000","last_became_peered":"2023-12-19T14:58:23.086160+0000","last_unstale":"2023-12-19T14:58:23.086509+0000","last_undegraded":"2023-12-19T14:58:23.086509+0000","last_fullsized":"2023-12-19T14:58:23.086509+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:48.754592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.097647+0000","last_change":"2023-12-19T14:58:23.097647+0000","last_active":"2023-12-19T14:58:23.097647+0000","last_peered":"2023-12-19T14:58:23.097647+0000","last_clean":"2023-12-19T14:58:23.097647+0000","last_became_active":"2023-12-19T14:58:23.097365+0000","last_became_peered":"2023-12-19T14:58:23.097365+0000","last_unstale":"2023-12-19T14:58:23.097647+0000","last_undegraded":"2023-12-19T14:58:23.097647+0000","last_fullsized":"2023-12-19T14:58:23.097647+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:55:25.092911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090829+0000","last_change":"2023-12-19T14:58:23.090829+0000","last_active":"2023-12-19T14:58:23.090829+0000","last_peered":"2023-12-19T14:58:23.090829+0000","last_clean":"2023-12-19T14:58:23.090829+0000","last_became_active":"2023-12-19T14:58:23.090557+0000","last_became_peered":"2023-12-19T14:58:23.090557+0000","last_unstale":"2023-12-19T14:58:23.090829+0000","last_undegraded":"2023-12-19T14:58:23.090829+0000","last_fullsized":"2023-12-19T14:58:23.090829+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:54:22.065431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098521+0000","last_change":"2023-12-19T14:58:23.093680+0000","last_active":"2023-12-19T14:58:23.098521+0000","last_peered":"2023-12-19T14:58:23.098521+0000","last_clean":"2023-12-19T14:58:23.098521+0000","last_became_active":"2023-12-19T14:58:23.093386+0000","last_became_peered":"2023-12-19T14:58:23.093386+0000","last_unstale":"2023-12-19T14:58:23.098521+0000","last_undegraded":"2023-12-19T14:58:23.098521+0000","last_fullsized":"2023-12-19T14:58:23.098521+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:10.339485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.087471+0000","last_change":"2023-12-19T14:58:23.087471+0000","last_active":"2023-12-19T14:58:23.087471+0000","last_peered":"2023-12-19T14:58:23.087471+0000","last_clean":"2023-12-19T14:58:23.087471+0000","last_became_active":"2023-12-19T14:58:23.087154+0000","last_became_peered":"2023-12-19T14:58:23.087154+0000","last_unstale":"2023-12-19T14:58:23.087471+0000","last_undegraded":"2023-12-19T14:58:23.087471+0000","last_fullsized":"2023-12-19T14:58:23.087471+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.079696+0000","last_change":"2023-12-19T14:58:23.079696+0000","last_active":"2023-12-19T14:58:23.079696+0000","last_peered":"2023-12-19T14:58:23.079696+0000","last_clean":"2023-12-19T14:58:23.079696+0000","last_became_active":"2023-12-19T14:58:23.079426+0000","last_became_peered":"2023-12-19T14:58:23.079426+0000","last_unstale":"2023-12-19T14:58:23.079696+0000","last_undegraded":"2023-12-19T14:58:23.079696+0000","last_fullsized":"2023-12-19T14:58:23.079696+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:25:52.126152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089086+0000","last_change":"2023-12-19T14:58:23.089086+0000","last_active":"2023-12-19T14:58:23.089086+0000","last_peered":"2023-12-19T14:58:23.089086+0000","last_clean":"2023-12-19T14:58:23.089086+0000","last_became_active":"2023-12-19T14:58:23.088718+0000","last_became_peered":"2023-12-19T14:58:23.088718+0000","last_unstale":"2023-12-19T14:58:23.089086+0000","last_undegraded":"2023-12-19T14:58:23.089086+0000","last_fullsized":"2023-12-19T14:58:23.089086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:08:53.139512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088228+0000","last_change":"2023-12-19T14:58:23.088228+0000","last_active":"2023-12-19T14:58:23.088228+0000","last_peered":"2023-12-19T14:58:23.088228+0000","last_clean":"2023-12-19T14:58:23.088228+0000","last_became_active":"2023-12-19T14:58:23.087949+0000","last_became_peered":"2023-12-19T14:58:23.087949+0000","last_unstale":"2023-12-19T14:58:23.088228+0000","last_undegraded":"2023-12-19T14:58:23.088228+0000","last_fullsized":"2023-12-19T14:58:23.088228+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:36:11.558422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089554+0000","last_change":"2023-12-19T14:58:23.089554+0000","last_active":"2023-12-19T14:58:23.089554+0000","last_peered":"2023-12-19T14:58:23.089554+0000","last_clean":"2023-12-19T14:58:23.089554+0000","last_became_active":"2023-12-19T14:58:23.089254+0000","last_became_peered":"2023-12-19T14:58:23.089254+0000","last_unstale":"2023-12-19T14:58:23.089554+0000","last_undegraded":"2023-12-19T14:58:23.089554+0000","last_fullsized":"2023-12-19T14:58:23.089554+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098959+0000","last_change":"2023-12-19T14:58:23.096005+0000","last_active":"2023-12-19T14:58:23.098959+0000","last_peered":"2023-12-19T14:58:23.098959+0000","last_clean":"2023-12-19T14:58:23.098959+0000","last_became_active":"2023-12-19T14:58:23.095698+0000","last_became_peered":"2023-12-19T14:58:23.095698+0000","last_unstale":"2023-12-19T14:58:23.098959+0000","last_undegraded":"2023-12-19T14:58:23.098959+0000","last_fullsized":"2023-12-19T14:58:23.098959+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:54.610499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095461+0000","last_change":"2023-12-19T14:58:23.095461+0000","last_active":"2023-12-19T14:58:23.095461+0000","last_peered":"2023-12-19T14:58:23.095461+0000","last_clean":"2023-12-19T14:58:23.095461+0000","last_became_active":"2023-12-19T14:58:23.095127+0000","last_became_peered":"2023-12-19T14:58:23.095127+0000","last_unstale":"2023-12-19T14:58:23.095461+0000","last_undegraded":"2023-12-19T14:58:23.095461+0000","last_fullsized":"2023-12-19T14:58:23.095461+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:24.885607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.083467+0000","last_change":"2023-12-19T14:58:23.083467+0000","last_active":"2023-12-19T14:58:23.083467+0000","last_peered":"2023-12-19T14:58:23.083467+0000","last_clean":"2023-12-19T14:58:23.083467+0000","last_became_active":"2023-12-19T14:58:23.083192+0000","last_became_peered":"2023-12-19T14:58:23.083192+0000","last_unstale":"2023-12-19T14:58:23.083467+0000","last_undegraded":"2023-12-19T14:58:23.083467+0000","last_fullsized":"2023-12-19T14:58:23.083467+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:45.211673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098631+0000","last_change":"2023-12-19T14:58:23.098631+0000","last_active":"2023-12-19T14:58:23.098631+0000","last_peered":"2023-12-19T14:58:23.098631+0000","last_clean":"2023-12-19T14:58:23.098631+0000","last_became_active":"2023-12-19T14:58:23.098137+0000","last_became_peered":"2023-12-19T14:58:23.098137+0000","last_unstale":"2023-12-19T14:58:23.098631+0000","last_undegraded":"2023-12-19T14:58:23.098631+0000","last_fullsized":"2023-12-19T14:58:23.098631+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102192+0000","last_change":"2023-12-19T14:58:23.100761+0000","last_active":"2023-12-19T14:58:23.102192+0000","last_peered":"2023-12-19T14:58:23.102192+0000","last_clean":"2023-12-19T14:58:23.102192+0000","last_became_active":"2023-12-19T14:58:23.100048+0000","last_became_peered":"2023-12-19T14:58:23.100048+0000","last_unstale":"2023-12-19T14:58:23.102192+0000","last_undegraded":"2023-12-19T14:58:23.102192+0000","last_fullsized":"2023-12-19T14:58:23.102192+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100091+0000","last_change":"2023-12-19T14:58:23.100091+0000","last_active":"2023-12-19T14:58:23.100091+0000","last_peered":"2023-12-19T14:58:23.100091+0000","last_clean":"2023-12-19T14:58:23.100091+0000","last_became_active":"2023-12-19T14:58:23.099688+0000","last_became_peered":"2023-12-19T14:58:23.099688+0000","last_unstale":"2023-12-19T14:58:23.100091+0000","last_undegraded":"2023-12-19T14:58:23.100091+0000","last_fullsized":"2023-12-19T14:58:23.100091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:09.645961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085553+0000","last_change":"2023-12-19T14:58:23.083506+0000","last_active":"2023-12-19T14:58:23.085553+0000","last_peered":"2023-12-19T14:58:23.085553+0000","last_clean":"2023-12-19T14:58:23.085553+0000","last_became_active":"2023-12-19T14:58:23.083196+0000","last_became_peered":"2023-12-19T14:58:23.083196+0000","last_unstale":"2023-12-19T14:58:23.085553+0000","last_undegraded":"2023-12-19T14:58:23.085553+0000","last_fullsized":"2023-12-19T14:58:23.085553+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:12:36.914340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.082832+0000","last_change":"2023-12-19T14:58:23.082832+0000","last_active":"2023-12-19T14:58:23.082832+0000","last_peered":"2023-12-19T14:58:23.082832+0000","last_clean":"2023-12-19T14:58:23.082832+0000","last_became_active":"2023-12-19T14:58:23.082364+0000","last_became_peered":"2023-12-19T14:58:23.082364+0000","last_unstale":"2023-12-19T14:58:23.082832+0000","last_undegraded":"2023-12-19T14:58:23.082832+0000","last_fullsized":"2023-12-19T14:58:23.082832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092080+0000","last_change":"2023-12-19T14:58:23.092080+0000","last_active":"2023-12-19T14:58:23.092080+0000","last_peered":"2023-12-19T14:58:23.092080+0000","last_clean":"2023-12-19T14:58:23.092080+0000","last_became_active":"2023-12-19T14:58:23.091804+0000","last_became_peered":"2023-12-19T14:58:23.091804+0000","last_unstale":"2023-12-19T14:58:23.092080+0000","last_undegraded":"2023-12-19T14:58:23.092080+0000","last_fullsized":"2023-12-19T14:58:23.092080+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:04.224777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092257+0000","last_change":"2023-12-19T14:58:23.092257+0000","last_active":"2023-12-19T14:58:23.092257+0000","last_peered":"2023-12-19T14:58:23.092257+0000","last_clean":"2023-12-19T14:58:23.092257+0000","last_became_active":"2023-12-19T14:58:23.091966+0000","last_became_peered":"2023-12-19T14:58:23.091966+0000","last_unstale":"2023-12-19T14:58:23.092257+0000","last_undegraded":"2023-12-19T14:58:23.092257+0000","last_fullsized":"2023-12-19T14:58:23.092257+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101096+0000","last_change":"2023-12-19T14:58:23.093802+0000","last_active":"2023-12-19T14:58:23.101096+0000","last_peered":"2023-12-19T14:58:23.101096+0000","last_clean":"2023-12-19T14:58:23.101096+0000","last_became_active":"2023-12-19T14:58:23.093399+0000","last_became_peered":"2023-12-19T14:58:23.093399+0000","last_unstale":"2023-12-19T14:58:23.101096+0000","last_undegraded":"2023-12-19T14:58:23.101096+0000","last_fullsized":"2023-12-19T14:58:23.101096+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:50:13.041547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100677+0000","last_change":"2023-12-19T14:58:23.092426+0000","last_active":"2023-12-19T14:58:23.100677+0000","last_peered":"2023-12-19T14:58:23.100677+0000","last_clean":"2023-12-19T14:58:23.100677+0000","last_became_active":"2023-12-19T14:58:23.092124+0000","last_became_peered":"2023-12-19T14:58:23.092124+0000","last_unstale":"2023-12-19T14:58:23.100677+0000","last_undegraded":"2023-12-19T14:58:23.100677+0000","last_fullsized":"2023-12-19T14:58:23.100677+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:45.650507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090222+0000","last_change":"2023-12-19T14:58:23.084800+0000","last_active":"2023-12-19T14:58:23.090222+0000","last_peered":"2023-12-19T14:58:23.090222+0000","last_clean":"2023-12-19T14:58:23.090222+0000","last_became_active":"2023-12-19T14:58:23.084493+0000","last_became_peered":"2023-12-19T14:58:23.084493+0000","last_unstale":"2023-12-19T14:58:23.090222+0000","last_undegraded":"2023-12-19T14:58:23.090222+0000","last_fullsized":"2023-12-19T14:58:23.090222+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:44:29.077829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099966+0000","last_change":"2023-12-19T14:58:23.099966+0000","last_active":"2023-12-19T14:58:23.099966+0000","last_peered":"2023-12-19T14:58:23.099966+0000","last_clean":"2023-12-19T14:58:23.099966+0000","last_became_active":"2023-12-19T14:58:23.099148+0000","last_became_peered":"2023-12-19T14:58:23.099148+0000","last_unstale":"2023-12-19T14:58:23.099966+0000","last_undegraded":"2023-12-19T14:58:23.099966+0000","last_fullsized":"2023-12-19T14:58:23.099966+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:05:34.253090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085983+0000","last_change":"2023-12-19T14:58:23.085983+0000","last_active":"2023-12-19T14:58:23.085983+0000","last_peered":"2023-12-19T14:58:23.085983+0000","last_clean":"2023-12-19T14:58:23.085983+0000","last_became_active":"2023-12-19T14:58:23.085682+0000","last_became_peered":"2023-12-19T14:58:23.085682+0000","last_unstale":"2023-12-19T14:58:23.085983+0000","last_undegraded":"2023-12-19T14:58:23.085983+0000","last_fullsized":"2023-12-19T14:58:23.085983+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:37.977148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.096691+0000","last_change":"2023-12-19T14:58:23.091612+0000","last_active":"2023-12-19T14:58:23.096691+0000","last_peered":"2023-12-19T14:58:23.096691+0000","last_clean":"2023-12-19T14:58:23.096691+0000","last_became_active":"2023-12-19T14:58:23.091350+0000","last_became_peered":"2023-12-19T14:58:23.091350+0000","last_unstale":"2023-12-19T14:58:23.096691+0000","last_undegraded":"2023-12-19T14:58:23.096691+0000","last_fullsized":"2023-12-19T14:58:23.096691+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:47.718805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098967+0000","last_change":"2023-12-19T14:58:23.096801+0000","last_active":"2023-12-19T14:58:23.098967+0000","last_peered":"2023-12-19T14:58:23.098967+0000","last_clean":"2023-12-19T14:58:23.098967+0000","last_became_active":"2023-12-19T14:58:23.095926+0000","last_became_peered":"2023-12-19T14:58:23.095926+0000","last_unstale":"2023-12-19T14:58:23.098967+0000","last_undegraded":"2023-12-19T14:58:23.098967+0000","last_fullsized":"2023-12-19T14:58:23.098967+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101012+0000","last_change":"2023-12-19T14:58:23.095276+0000","last_active":"2023-12-19T14:58:23.101012+0000","last_peered":"2023-12-19T14:58:23.101012+0000","last_clean":"2023-12-19T14:58:23.101012+0000","last_became_active":"2023-12-19T14:58:23.095011+0000","last_became_peered":"2023-12-19T14:58:23.095011+0000","last_unstale":"2023-12-19T14:58:23.101012+0000","last_undegraded":"2023-12-19T14:58:23.101012+0000","last_fullsized":"2023-12-19T14:58:23.101012+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:42.139912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100684+0000","last_change":"2023-12-19T14:58:23.093069+0000","last_active":"2023-12-19T14:58:23.100684+0000","last_peered":"2023-12-19T14:58:23.100684+0000","last_clean":"2023-12-19T14:58:23.100684+0000","last_became_active":"2023-12-19T14:58:23.087629+0000","last_became_peered":"2023-12-19T14:58:23.087629+0000","last_unstale":"2023-12-19T14:58:23.100684+0000","last_undegraded":"2023-12-19T14:58:23.100684+0000","last_fullsized":"2023-12-19T14:58:23.100684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:24.885607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081811+0000","last_change":"2023-12-19T14:58:23.081811+0000","last_active":"2023-12-19T14:58:23.081811+0000","last_peered":"2023-12-19T14:58:23.081811+0000","last_clean":"2023-12-19T14:58:23.081811+0000","last_became_active":"2023-12-19T14:58:23.081367+0000","last_became_peered":"2023-12-19T14:58:23.081367+0000","last_unstale":"2023-12-19T14:58:23.081811+0000","last_undegraded":"2023-12-19T14:58:23.081811+0000","last_fullsized":"2023-12-19T14:58:23.081811+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:54.682111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.541358+0000","last_change":"2023-12-19T14:58:22.538805+0000","last_active":"2023-12-19T14:58:23.541358+0000","last_peered":"2023-12-19T14:58:23.541358+0000","last_clean":"2023-12-19T14:58:23.541358+0000","last_became_active":"2023-12-19T14:58:22.538505+0000","last_became_peered":"2023-12-19T14:58:22.538505+0000","last_unstale":"2023-12-19T14:58:23.541358+0000","last_undegraded":"2023-12-19T14:58:23.541358+0000","last_fullsized":"2023-12-19T14:58:23.541358+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:39.209494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092186+0000","last_change":"2023-12-19T14:58:23.092186+0000","last_active":"2023-12-19T14:58:23.092186+0000","last_peered":"2023-12-19T14:58:23.092186+0000","last_clean":"2023-12-19T14:58:23.092186+0000","last_became_active":"2023-12-19T14:58:23.091826+0000","last_became_peered":"2023-12-19T14:58:23.091826+0000","last_unstale":"2023-12-19T14:58:23.092186+0000","last_undegraded":"2023-12-19T14:58:23.092186+0000","last_fullsized":"2023-12-19T14:58:23.092186+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:43:52.111926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.070511+0000","last_change":"2023-12-19T14:58:22.537786+0000","last_active":"2023-12-19T14:58:23.070511+0000","last_peered":"2023-12-19T14:58:23.070511+0000","last_clean":"2023-12-19T14:58:23.070511+0000","last_became_active":"2023-12-19T14:58:22.537458+0000","last_became_peered":"2023-12-19T14:58:22.537458+0000","last_unstale":"2023-12-19T14:58:23.070511+0000","last_undegraded":"2023-12-19T14:58:23.070511+0000","last_fullsized":"2023-12-19T14:58:23.070511+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094322+0000","last_change":"2023-12-19T14:58:23.085247+0000","last_active":"2023-12-19T14:58:23.094322+0000","last_peered":"2023-12-19T14:58:23.094322+0000","last_clean":"2023-12-19T14:58:23.094322+0000","last_became_active":"2023-12-19T14:58:23.084927+0000","last_became_peered":"2023-12-19T14:58:23.084927+0000","last_unstale":"2023-12-19T14:58:23.094322+0000","last_undegraded":"2023-12-19T14:58:23.094322+0000","last_fullsized":"2023-12-19T14:58:23.094322+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:34.523242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.544413+0000","last_change":"2023-12-19T14:58:22.540894+0000","last_active":"2023-12-19T14:58:23.544413+0000","last_peered":"2023-12-19T14:58:23.544413+0000","last_clean":"2023-12-19T14:58:23.544413+0000","last_became_active":"2023-12-19T14:58:22.539386+0000","last_became_peered":"2023-12-19T14:58:22.539386+0000","last_unstale":"2023-12-19T14:58:23.544413+0000","last_undegraded":"2023-12-19T14:58:23.544413+0000","last_fullsized":"2023-12-19T14:58:23.544413+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.085487+0000","last_change":"2023-12-19T14:58:23.085487+0000","last_active":"2023-12-19T14:58:23.085487+0000","last_peered":"2023-12-19T14:58:23.085487+0000","last_clean":"2023-12-19T14:58:23.085487+0000","last_became_active":"2023-12-19T14:58:23.085086+0000","last_became_peered":"2023-12-19T14:58:23.085086+0000","last_unstale":"2023-12-19T14:58:23.085487+0000","last_undegraded":"2023-12-19T14:58:23.085487+0000","last_fullsized":"2023-12-19T14:58:23.085487+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.549557+0000","last_change":"2023-12-19T14:58:22.545939+0000","last_active":"2023-12-19T14:58:23.549557+0000","last_peered":"2023-12-19T14:58:23.549557+0000","last_clean":"2023-12-19T14:58:23.549557+0000","last_became_active":"2023-12-19T14:58:22.544686+0000","last_became_peered":"2023-12-19T14:58:22.544686+0000","last_unstale":"2023-12-19T14:58:23.549557+0000","last_undegraded":"2023-12-19T14:58:23.549557+0000","last_fullsized":"2023-12-19T14:58:23.549557+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.093562+0000","last_change":"2023-12-19T14:58:23.093562+0000","last_active":"2023-12-19T14:58:23.093562+0000","last_peered":"2023-12-19T14:58:23.093562+0000","last_clean":"2023-12-19T14:58:23.093562+0000","last_became_active":"2023-12-19T14:58:23.092692+0000","last_became_peered":"2023-12-19T14:58:23.092692+0000","last_unstale":"2023-12-19T14:58:23.093562+0000","last_undegraded":"2023-12-19T14:58:23.093562+0000","last_fullsized":"2023-12-19T14:58:23.093562+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:59.466905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.074062+0000","last_change":"2023-12-19T14:58:22.543346+0000","last_active":"2023-12-19T14:58:23.074062+0000","last_peered":"2023-12-19T14:58:23.074062+0000","last_clean":"2023-12-19T14:58:23.074062+0000","last_became_active":"2023-12-19T14:58:22.543059+0000","last_became_peered":"2023-12-19T14:58:22.543059+0000","last_unstale":"2023-12-19T14:58:23.074062+0000","last_undegraded":"2023-12-19T14:58:23.074062+0000","last_fullsized":"2023-12-19T14:58:23.074062+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:34.444674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.088760+0000","last_change":"2023-12-19T14:58:23.088760+0000","last_active":"2023-12-19T14:58:23.088760+0000","last_peered":"2023-12-19T14:58:23.088760+0000","last_clean":"2023-12-19T14:58:23.088760+0000","last_became_active":"2023-12-19T14:58:23.086673+0000","last_became_peered":"2023-12-19T14:58:23.086673+0000","last_unstale":"2023-12-19T14:58:23.088760+0000","last_undegraded":"2023-12-19T14:58:23.088760+0000","last_fullsized":"2023-12-19T14:58:23.088760+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:12.499676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.074633+0000","last_change":"2023-12-19T14:58:22.554344+0000","last_active":"2023-12-19T14:58:23.074633+0000","last_peered":"2023-12-19T14:58:23.074633+0000","last_clean":"2023-12-19T14:58:23.074633+0000","last_became_active":"2023-12-19T14:58:22.553174+0000","last_became_peered":"2023-12-19T14:58:22.553174+0000","last_unstale":"2023-12-19T14:58:23.074633+0000","last_undegraded":"2023-12-19T14:58:23.074633+0000","last_fullsized":"2023-12-19T14:58:23.074633+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:33.508283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.096116+0000","last_change":"2023-12-19T14:58:23.090740+0000","last_active":"2023-12-19T14:58:23.096116+0000","last_peered":"2023-12-19T14:58:23.096116+0000","last_clean":"2023-12-19T14:58:23.096116+0000","last_became_active":"2023-12-19T14:58:23.090419+0000","last_became_peered":"2023-12-19T14:58:23.090419+0000","last_unstale":"2023-12-19T14:58:23.096116+0000","last_undegraded":"2023-12-19T14:58:23.096116+0000","last_fullsized":"2023-12-19T14:58:23.096116+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:03.766685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081928+0000","last_change":"2023-12-19T14:58:22.536509+0000","last_active":"2023-12-19T14:58:23.081928+0000","last_peered":"2023-12-19T14:58:23.081928+0000","last_clean":"2023-12-19T14:58:23.081928+0000","last_became_active":"2023-12-19T14:58:22.535101+0000","last_became_peered":"2023-12-19T14:58:22.535101+0000","last_unstale":"2023-12-19T14:58:23.081928+0000","last_undegraded":"2023-12-19T14:58:23.081928+0000","last_fullsized":"2023-12-19T14:58:23.081928+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:54.904339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090641+0000","last_change":"2023-12-19T14:58:23.090641+0000","last_active":"2023-12-19T14:58:23.090641+0000","last_peered":"2023-12-19T14:58:23.090641+0000","last_clean":"2023-12-19T14:58:23.090641+0000","last_became_active":"2023-12-19T14:58:23.090308+0000","last_became_peered":"2023-12-19T14:58:23.090308+0000","last_unstale":"2023-12-19T14:58:23.090641+0000","last_undegraded":"2023-12-19T14:58:23.090641+0000","last_fullsized":"2023-12-19T14:58:23.090641+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:34.286093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.551546+0000","last_change":"2023-12-19T14:58:22.545995+0000","last_active":"2023-12-19T14:58:22.551546+0000","last_peered":"2023-12-19T14:58:22.551546+0000","last_clean":"2023-12-19T14:58:22.551546+0000","last_became_active":"2023-12-19T14:58:22.545730+0000","last_became_peered":"2023-12-19T14:58:22.545730+0000","last_unstale":"2023-12-19T14:58:22.551546+0000","last_undegraded":"2023-12-19T14:58:22.551546+0000","last_fullsized":"2023-12-19T14:58:22.551546+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:16:23.870648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098815+0000","last_change":"2023-12-19T14:58:23.098815+0000","last_active":"2023-12-19T14:58:23.098815+0000","last_peered":"2023-12-19T14:58:23.098815+0000","last_clean":"2023-12-19T14:58:23.098815+0000","last_became_active":"2023-12-19T14:58:23.098518+0000","last_became_peered":"2023-12-19T14:58:23.098518+0000","last_unstale":"2023-12-19T14:58:23.098815+0000","last_undegraded":"2023-12-19T14:58:23.098815+0000","last_fullsized":"2023-12-19T14:58:23.098815+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:17.752109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550732+0000","last_change":"2023-12-19T14:58:22.549758+0000","last_active":"2023-12-19T14:58:22.550732+0000","last_peered":"2023-12-19T14:58:22.550732+0000","last_clean":"2023-12-19T14:58:22.550732+0000","last_became_active":"2023-12-19T14:58:22.549408+0000","last_became_peered":"2023-12-19T14:58:22.549408+0000","last_unstale":"2023-12-19T14:58:22.550732+0000","last_undegraded":"2023-12-19T14:58:22.550732+0000","last_fullsized":"2023-12-19T14:58:22.550732+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.101060+0000","last_change":"2023-12-19T14:58:23.093740+0000","last_active":"2023-12-19T14:58:23.101060+0000","last_peered":"2023-12-19T14:58:23.101060+0000","last_clean":"2023-12-19T14:58:23.101060+0000","last_became_active":"2023-12-19T14:58:23.093412+0000","last_became_peered":"2023-12-19T14:58:23.093412+0000","last_unstale":"2023-12-19T14:58:23.101060+0000","last_undegraded":"2023-12-19T14:58:23.101060+0000","last_fullsized":"2023-12-19T14:58:23.101060+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:58:59.950526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.077009+0000","last_change":"2023-12-19T14:58:22.548567+0000","last_active":"2023-12-19T14:58:23.077009+0000","last_peered":"2023-12-19T14:58:23.077009+0000","last_clean":"2023-12-19T14:58:23.077009+0000","last_became_active":"2023-12-19T14:58:22.548316+0000","last_became_peered":"2023-12-19T14:58:22.548316+0000","last_unstale":"2023-12-19T14:58:23.077009+0000","last_undegraded":"2023-12-19T14:58:23.077009+0000","last_fullsized":"2023-12-19T14:58:23.077009+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:33.271134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100894+0000","last_change":"2023-12-19T14:58:23.096683+0000","last_active":"2023-12-19T14:58:23.100894+0000","last_peered":"2023-12-19T14:58:23.100894+0000","last_clean":"2023-12-19T14:58:23.100894+0000","last_became_active":"2023-12-19T14:58:23.095608+0000","last_became_peered":"2023-12-19T14:58:23.095608+0000","last_unstale":"2023-12-19T14:58:23.100894+0000","last_undegraded":"2023-12-19T14:58:23.100894+0000","last_fullsized":"2023-12-19T14:58:23.100894+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:13.536441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.541457+0000","last_change":"2023-12-19T14:58:22.540462+0000","last_active":"2023-12-19T14:58:23.541457+0000","last_peered":"2023-12-19T14:58:23.541457+0000","last_clean":"2023-12-19T14:58:23.541457+0000","last_became_active":"2023-12-19T14:58:22.539701+0000","last_became_peered":"2023-12-19T14:58:22.539701+0000","last_unstale":"2023-12-19T14:58:23.541457+0000","last_undegraded":"2023-12-19T14:58:23.541457+0000","last_fullsized":"2023-12-19T14:58:23.541457+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:53.667153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099999+0000","last_change":"2023-12-19T14:58:23.090495+0000","last_active":"2023-12-19T14:58:23.099999+0000","last_peered":"2023-12-19T14:58:23.099999+0000","last_clean":"2023-12-19T14:58:23.099999+0000","last_became_active":"2023-12-19T14:58:23.088694+0000","last_became_peered":"2023-12-19T14:58:23.088694+0000","last_unstale":"2023-12-19T14:58:23.099999+0000","last_undegraded":"2023-12-19T14:58:23.099999+0000","last_fullsized":"2023-12-19T14:58:23.099999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:06.574019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.551229+0000","last_change":"2023-12-19T14:58:22.543859+0000","last_active":"2023-12-19T14:58:23.551229+0000","last_peered":"2023-12-19T14:58:23.551229+0000","last_clean":"2023-12-19T14:58:23.551229+0000","last_became_active":"2023-12-19T14:58:22.543570+0000","last_became_peered":"2023-12-19T14:58:22.543570+0000","last_unstale":"2023-12-19T14:58:23.551229+0000","last_undegraded":"2023-12-19T14:58:23.551229+0000","last_fullsized":"2023-12-19T14:58:23.551229+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:39:15.583668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095877+0000","last_change":"2023-12-19T14:58:23.095877+0000","last_active":"2023-12-19T14:58:23.095877+0000","last_peered":"2023-12-19T14:58:23.095877+0000","last_clean":"2023-12-19T14:58:23.095877+0000","last_became_active":"2023-12-19T14:58:23.091637+0000","last_became_peered":"2023-12-19T14:58:23.091637+0000","last_unstale":"2023-12-19T14:58:23.095877+0000","last_undegraded":"2023-12-19T14:58:23.095877+0000","last_fullsized":"2023-12-19T14:58:23.095877+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:15:01.581518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.548415+0000","last_change":"2023-12-19T14:58:22.549591+0000","last_active":"2023-12-19T14:58:23.548415+0000","last_peered":"2023-12-19T14:58:23.548415+0000","last_clean":"2023-12-19T14:58:23.548415+0000","last_became_active":"2023-12-19T14:58:22.549311+0000","last_became_peered":"2023-12-19T14:58:22.549311+0000","last_unstale":"2023-12-19T14:58:23.548415+0000","last_undegraded":"2023-12-19T14:58:23.548415+0000","last_fullsized":"2023-12-19T14:58:23.548415+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:42.975704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.091484+0000","last_change":"2023-12-19T14:58:23.091484+0000","last_active":"2023-12-19T14:58:23.091484+0000","last_peered":"2023-12-19T14:58:23.091484+0000","last_clean":"2023-12-19T14:58:23.091484+0000","last_became_active":"2023-12-19T14:58:23.091195+0000","last_became_peered":"2023-12-19T14:58:23.091195+0000","last_unstale":"2023-12-19T14:58:23.091484+0000","last_undegraded":"2023-12-19T14:58:23.091484+0000","last_fullsized":"2023-12-19T14:58:23.091484+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:09.645961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.087530+0000","last_change":"2023-12-19T14:58:22.548710+0000","last_active":"2023-12-19T14:58:23.087530+0000","last_peered":"2023-12-19T14:58:23.087530+0000","last_clean":"2023-12-19T14:58:23.087530+0000","last_became_active":"2023-12-19T14:58:22.548357+0000","last_became_peered":"2023-12-19T14:58:22.548357+0000","last_unstale":"2023-12-19T14:58:23.087530+0000","last_undegraded":"2023-12-19T14:58:23.087530+0000","last_fullsized":"2023-12-19T14:58:23.087530+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:21.392884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.100211+0000","last_change":"2023-12-19T14:58:23.096467+0000","last_active":"2023-12-19T14:58:23.100211+0000","last_peered":"2023-12-19T14:58:23.100211+0000","last_clean":"2023-12-19T14:58:23.100211+0000","last_became_active":"2023-12-19T14:58:23.096228+0000","last_became_peered":"2023-12-19T14:58:23.096228+0000","last_unstale":"2023-12-19T14:58:23.100211+0000","last_undegraded":"2023-12-19T14:58:23.100211+0000","last_fullsized":"2023-12-19T14:58:23.100211+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.569453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086594+0000","last_change":"2023-12-19T14:58:22.547152+0000","last_active":"2023-12-19T14:58:23.086594+0000","last_peered":"2023-12-19T14:58:23.086594+0000","last_clean":"2023-12-19T14:58:23.086594+0000","last_became_active":"2023-12-19T14:58:22.545361+0000","last_became_peered":"2023-12-19T14:58:22.545361+0000","last_unstale":"2023-12-19T14:58:23.086594+0000","last_undegraded":"2023-12-19T14:58:23.086594+0000","last_fullsized":"2023-12-19T14:58:23.086594+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:39.209494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099094+0000","last_change":"2023-12-19T14:58:23.084657+0000","last_active":"2023-12-19T14:58:23.099094+0000","last_peered":"2023-12-19T14:58:23.099094+0000","last_clean":"2023-12-19T14:58:23.099094+0000","last_became_active":"2023-12-19T14:58:23.084280+0000","last_became_peered":"2023-12-19T14:58:23.084280+0000","last_unstale":"2023-12-19T14:58:23.099094+0000","last_undegraded":"2023-12-19T14:58:23.099094+0000","last_fullsized":"2023-12-19T14:58:23.099094+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:11.698607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.070720+0000","last_change":"2023-12-19T14:58:22.553836+0000","last_active":"2023-12-19T14:58:23.070720+0000","last_peered":"2023-12-19T14:58:23.070720+0000","last_clean":"2023-12-19T14:58:23.070720+0000","last_became_active":"2023-12-19T14:58:22.551441+0000","last_became_peered":"2023-12-19T14:58:22.551441+0000","last_unstale":"2023-12-19T14:58:23.070720+0000","last_undegraded":"2023-12-19T14:58:23.070720+0000","last_fullsized":"2023-12-19T14:58:23.070720+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.090335+0000","last_change":"2023-12-19T14:58:23.090335+0000","last_active":"2023-12-19T14:58:23.090335+0000","last_peered":"2023-12-19T14:58:23.090335+0000","last_clean":"2023-12-19T14:58:23.090335+0000","last_became_active":"2023-12-19T14:58:23.089408+0000","last_became_peered":"2023-12-19T14:58:23.089408+0000","last_unstale":"2023-12-19T14:58:23.090335+0000","last_undegraded":"2023-12-19T14:58:23.090335+0000","last_fullsized":"2023-12-19T14:58:23.090335+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:06.593293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.547947+0000","last_change":"2023-12-19T14:58:22.552973+0000","last_active":"2023-12-19T14:58:23.547947+0000","last_peered":"2023-12-19T14:58:23.547947+0000","last_clean":"2023-12-19T14:58:23.547947+0000","last_became_active":"2023-12-19T14:58:22.552518+0000","last_became_peered":"2023-12-19T14:58:22.552518+0000","last_unstale":"2023-12-19T14:58:23.547947+0000","last_undegraded":"2023-12-19T14:58:23.547947+0000","last_fullsized":"2023-12-19T14:58:23.547947+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:24:09.668268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.089775+0000","last_change":"2023-12-19T14:58:23.089775+0000","last_active":"2023-12-19T14:58:23.089775+0000","last_peered":"2023-12-19T14:58:23.089775+0000","last_clean":"2023-12-19T14:58:23.089775+0000","last_became_active":"2023-12-19T14:58:23.088713+0000","last_became_peered":"2023-12-19T14:58:23.088713+0000","last_unstale":"2023-12-19T14:58:23.089775+0000","last_undegraded":"2023-12-19T14:58:23.089775+0000","last_fullsized":"2023-12-19T14:58:23.089775+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:30.812983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.072508+0000","last_change":"2023-12-19T14:58:22.541086+0000","last_active":"2023-12-19T14:58:23.072508+0000","last_peered":"2023-12-19T14:58:23.072508+0000","last_clean":"2023-12-19T14:58:23.072508+0000","last_became_active":"2023-12-19T14:58:22.539398+0000","last_became_peered":"2023-12-19T14:58:22.539398+0000","last_unstale":"2023-12-19T14:58:23.072508+0000","last_undegraded":"2023-12-19T14:58:23.072508+0000","last_fullsized":"2023-12-19T14:58:23.072508+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:31:42.150881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.099981+0000","last_change":"2023-12-19T14:58:23.090294+0000","last_active":"2023-12-19T14:58:23.099981+0000","last_peered":"2023-12-19T14:58:23.099981+0000","last_clean":"2023-12-19T14:58:23.099981+0000","last_became_active":"2023-12-19T14:58:23.088382+0000","last_became_peered":"2023-12-19T14:58:23.088382+0000","last_unstale":"2023-12-19T14:58:23.099981+0000","last_undegraded":"2023-12-19T14:58:23.099981+0000","last_fullsized":"2023-12-19T14:58:23.099981+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:36.808951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.543958+0000","last_change":"2023-12-19T14:58:22.541886+0000","last_active":"2023-12-19T14:58:22.543958+0000","last_peered":"2023-12-19T14:58:22.543958+0000","last_clean":"2023-12-19T14:58:22.543958+0000","last_became_active":"2023-12-19T14:58:22.541568+0000","last_became_peered":"2023-12-19T14:58:22.541568+0000","last_unstale":"2023-12-19T14:58:22.543958+0000","last_undegraded":"2023-12-19T14:58:22.543958+0000","last_fullsized":"2023-12-19T14:58:22.543958+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:33:47.724646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092263+0000","last_change":"2023-12-19T14:58:23.092263+0000","last_active":"2023-12-19T14:58:23.092263+0000","last_peered":"2023-12-19T14:58:23.092263+0000","last_clean":"2023-12-19T14:58:23.092263+0000","last_became_active":"2023-12-19T14:58:23.091976+0000","last_became_peered":"2023-12-19T14:58:23.091976+0000","last_unstale":"2023-12-19T14:58:23.092263+0000","last_undegraded":"2023-12-19T14:58:23.092263+0000","last_fullsized":"2023-12-19T14:58:23.092263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:37.807985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.548431+0000","last_change":"2023-12-19T14:58:22.550721+0000","last_active":"2023-12-19T14:58:23.548431+0000","last_peered":"2023-12-19T14:58:23.548431+0000","last_clean":"2023-12-19T14:58:23.548431+0000","last_became_active":"2023-12-19T14:58:22.550317+0000","last_became_peered":"2023-12-19T14:58:22.550317+0000","last_unstale":"2023-12-19T14:58:23.548431+0000","last_undegraded":"2023-12-19T14:58:23.548431+0000","last_fullsized":"2023-12-19T14:58:23.548431+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:25:38.261204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102159+0000","last_change":"2023-12-19T14:58:23.100318+0000","last_active":"2023-12-19T14:58:23.102159+0000","last_peered":"2023-12-19T14:58:23.102159+0000","last_clean":"2023-12-19T14:58:23.102159+0000","last_became_active":"2023-12-19T14:58:23.100042+0000","last_became_peered":"2023-12-19T14:58:23.100042+0000","last_unstale":"2023-12-19T14:58:23.102159+0000","last_undegraded":"2023-12-19T14:58:23.102159+0000","last_fullsized":"2023-12-19T14:58:23.102159+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:51.014043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.539079+0000","last_change":"2023-12-19T14:58:22.537336+0000","last_active":"2023-12-19T14:58:22.539079+0000","last_peered":"2023-12-19T14:58:22.539079+0000","last_clean":"2023-12-19T14:58:22.539079+0000","last_became_active":"2023-12-19T14:58:22.535064+0000","last_became_peered":"2023-12-19T14:58:22.535064+0000","last_unstale":"2023-12-19T14:58:22.539079+0000","last_undegraded":"2023-12-19T14:58:22.539079+0000","last_fullsized":"2023-12-19T14:58:22.539079+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:30:03.178463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.098949+0000","last_change":"2023-12-19T14:58:23.094681+0000","last_active":"2023-12-19T14:58:23.098949+0000","last_peered":"2023-12-19T14:58:23.098949+0000","last_clean":"2023-12-19T14:58:23.098949+0000","last_became_active":"2023-12-19T14:58:23.093182+0000","last_became_peered":"2023-12-19T14:58:23.093182+0000","last_unstale":"2023-12-19T14:58:23.098949+0000","last_undegraded":"2023-12-19T14:58:23.098949+0000","last_fullsized":"2023-12-19T14:58:23.098949+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:59.466905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.086730+0000","last_change":"2023-12-19T14:58:22.551548+0000","last_active":"2023-12-19T14:58:23.086730+0000","last_peered":"2023-12-19T14:58:23.086730+0000","last_clean":"2023-12-19T14:58:23.086730+0000","last_became_active":"2023-12-19T14:58:22.551256+0000","last_became_peered":"2023-12-19T14:58:22.551256+0000","last_unstale":"2023-12-19T14:58:23.086730+0000","last_undegraded":"2023-12-19T14:58:23.086730+0000","last_fullsized":"2023-12-19T14:58:23.086730+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094354+0000","last_change":"2023-12-19T14:58:23.085891+0000","last_active":"2023-12-19T14:58:23.094354+0000","last_peered":"2023-12-19T14:58:23.094354+0000","last_clean":"2023-12-19T14:58:23.094354+0000","last_became_active":"2023-12-19T14:58:23.085561+0000","last_became_peered":"2023-12-19T14:58:23.085561+0000","last_unstale":"2023-12-19T14:58:23.094354+0000","last_undegraded":"2023-12-19T14:58:23.094354+0000","last_fullsized":"2023-12-19T14:58:23.094354+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:25:39.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.537761+0000","last_change":"2023-12-19T14:58:22.535017+0000","last_active":"2023-12-19T14:58:22.537761+0000","last_peered":"2023-12-19T14:58:22.537761+0000","last_clean":"2023-12-19T14:58:22.537761+0000","last_became_active":"2023-12-19T14:58:22.534519+0000","last_became_peered":"2023-12-19T14:58:22.534519+0000","last_unstale":"2023-12-19T14:58:22.537761+0000","last_undegraded":"2023-12-19T14:58:22.537761+0000","last_fullsized":"2023-12-19T14:58:22.537761+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:57.722647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.080640+0000","last_change":"2023-12-19T14:58:23.080640+0000","last_active":"2023-12-19T14:58:23.080640+0000","last_peered":"2023-12-19T14:58:23.080640+0000","last_clean":"2023-12-19T14:58:23.080640+0000","last_became_active":"2023-12-19T14:58:23.080326+0000","last_became_peered":"2023-12-19T14:58:23.080326+0000","last_unstale":"2023-12-19T14:58:23.080640+0000","last_undegraded":"2023-12-19T14:58:23.080640+0000","last_fullsized":"2023-12-19T14:58:23.080640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:27:47.982461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.547877+0000","last_change":"2023-12-19T14:58:22.547978+0000","last_active":"2023-12-19T14:58:23.547877+0000","last_peered":"2023-12-19T14:58:23.547877+0000","last_clean":"2023-12-19T14:58:23.547877+0000","last_became_active":"2023-12-19T14:58:22.547702+0000","last_became_peered":"2023-12-19T14:58:22.547702+0000","last_unstale":"2023-12-19T14:58:23.547877+0000","last_undegraded":"2023-12-19T14:58:23.547877+0000","last_fullsized":"2023-12-19T14:58:23.547877+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:58:58.935568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092890+0000","last_change":"2023-12-19T14:58:23.092890+0000","last_active":"2023-12-19T14:58:23.092890+0000","last_peered":"2023-12-19T14:58:23.092890+0000","last_clean":"2023-12-19T14:58:23.092890+0000","last_became_active":"2023-12-19T14:58:23.092585+0000","last_became_peered":"2023-12-19T14:58:23.092585+0000","last_unstale":"2023-12-19T14:58:23.092890+0000","last_undegraded":"2023-12-19T14:58:23.092890+0000","last_fullsized":"2023-12-19T14:58:23.092890+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550993+0000","last_change":"2023-12-19T14:58:22.550097+0000","last_active":"2023-12-19T14:58:22.550993+0000","last_peered":"2023-12-19T14:58:22.550993+0000","last_clean":"2023-12-19T14:58:22.550993+0000","last_became_active":"2023-12-19T14:58:22.549256+0000","last_became_peered":"2023-12-19T14:58:22.549256+0000","last_unstale":"2023-12-19T14:58:22.550993+0000","last_undegraded":"2023-12-19T14:58:22.550993+0000","last_fullsized":"2023-12-19T14:58:22.550993+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:09.544430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.092578+0000","last_change":"2023-12-19T14:58:23.092578+0000","last_active":"2023-12-19T14:58:23.092578+0000","last_peered":"2023-12-19T14:58:23.092578+0000","last_clean":"2023-12-19T14:58:23.092578+0000","last_became_active":"2023-12-19T14:58:23.092247+0000","last_became_peered":"2023-12-19T14:58:23.092247+0000","last_unstale":"2023-12-19T14:58:23.092578+0000","last_undegraded":"2023-12-19T14:58:23.092578+0000","last_fullsized":"2023-12-19T14:58:23.092578+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:54.610499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550685+0000","last_change":"2023-12-19T14:58:22.549492+0000","last_active":"2023-12-19T14:58:22.550685+0000","last_peered":"2023-12-19T14:58:22.550685+0000","last_clean":"2023-12-19T14:58:22.550685+0000","last_became_active":"2023-12-19T14:58:22.549178+0000","last_became_peered":"2023-12-19T14:58:22.549178+0000","last_unstale":"2023-12-19T14:58:22.550685+0000","last_undegraded":"2023-12-19T14:58:22.550685+0000","last_fullsized":"2023-12-19T14:58:22.550685+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:05:08.036270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.102741+0000","last_change":"2023-12-19T14:58:23.101559+0000","last_active":"2023-12-19T14:58:23.102741+0000","last_peered":"2023-12-19T14:58:23.102741+0000","last_clean":"2023-12-19T14:58:23.102741+0000","last_became_active":"2023-12-19T14:58:23.101246+0000","last_became_peered":"2023-12-19T14:58:23.101246+0000","last_unstale":"2023-12-19T14:58:23.102741+0000","last_undegraded":"2023-12-19T14:58:23.102741+0000","last_fullsized":"2023-12-19T14:58:23.102741+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:48:08.519586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.555138+0000","last_change":"2023-12-19T14:58:22.548601+0000","last_active":"2023-12-19T14:58:22.555138+0000","last_peered":"2023-12-19T14:58:22.555138+0000","last_clean":"2023-12-19T14:58:22.555138+0000","last_became_active":"2023-12-19T14:58:22.548329+0000","last_became_peered":"2023-12-19T14:58:22.548329+0000","last_unstale":"2023-12-19T14:58:22.555138+0000","last_undegraded":"2023-12-19T14:58:22.555138+0000","last_fullsized":"2023-12-19T14:58:22.555138+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:44.635549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.094091+0000","last_change":"2023-12-19T14:58:23.094091+0000","last_active":"2023-12-19T14:58:23.094091+0000","last_peered":"2023-12-19T14:58:23.094091+0000","last_clean":"2023-12-19T14:58:23.094091+0000","last_became_active":"2023-12-19T14:58:23.093525+0000","last_became_peered":"2023-12-19T14:58:23.093525+0000","last_unstale":"2023-12-19T14:58:23.094091+0000","last_undegraded":"2023-12-19T14:58:23.094091+0000","last_fullsized":"2023-12-19T14:58:23.094091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:12:49.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.545552+0000","last_change":"2023-12-19T14:58:22.544171+0000","last_active":"2023-12-19T14:58:22.545552+0000","last_peered":"2023-12-19T14:58:22.545552+0000","last_clean":"2023-12-19T14:58:22.545552+0000","last_became_active":"2023-12-19T14:58:22.543735+0000","last_became_peered":"2023-12-19T14:58:22.543735+0000","last_unstale":"2023-12-19T14:58:22.545552+0000","last_undegraded":"2023-12-19T14:58:22.545552+0000","last_fullsized":"2023-12-19T14:58:22.545552+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:12.521483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.095698+0000","last_change":"2023-12-19T14:58:23.089594+0000","last_active":"2023-12-19T14:58:23.095698+0000","last_peered":"2023-12-19T14:58:23.095698+0000","last_clean":"2023-12-19T14:58:23.095698+0000","last_became_active":"2023-12-19T14:58:23.089319+0000","last_became_peered":"2023-12-19T14:58:23.089319+0000","last_unstale":"2023-12-19T14:58:23.095698+0000","last_undegraded":"2023-12-19T14:58:23.095698+0000","last_fullsized":"2023-12-19T14:58:23.095698+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:54:55.919298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.544943+0000","last_change":"2023-12-19T14:58:22.542985+0000","last_active":"2023-12-19T14:58:22.544943+0000","last_peered":"2023-12-19T14:58:22.544943+0000","last_clean":"2023-12-19T14:58:22.544943+0000","last_became_active":"2023-12-19T14:58:22.541624+0000","last_became_peered":"2023-12-19T14:58:22.541624+0000","last_unstale":"2023-12-19T14:58:22.544943+0000","last_undegraded":"2023-12-19T14:58:22.544943+0000","last_fullsized":"2023-12-19T14:58:22.544943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:33.271134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.081176+0000","last_change":"2023-12-19T14:58:23.081176+0000","last_active":"2023-12-19T14:58:23.081176+0000","last_peered":"2023-12-19T14:58:23.081176+0000","last_clean":"2023-12-19T14:58:23.081176+0000","last_became_active":"2023-12-19T14:58:23.080737+0000","last_became_peered":"2023-12-19T14:58:23.080737+0000","last_unstale":"2023-12-19T14:58:23.081176+0000","last_undegraded":"2023-12-19T14:58:23.081176+0000","last_fullsized":"2023-12-19T14:58:23.081176+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:51.014043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.550877+0000","last_change":"2023-12-19T14:58:22.543457+0000","last_active":"2023-12-19T14:58:22.550877+0000","last_peered":"2023-12-19T14:58:22.550877+0000","last_clean":"2023-12-19T14:58:22.550877+0000","last_became_active":"2023-12-19T14:58:22.543155+0000","last_became_peered":"2023-12-19T14:58:22.543155+0000","last_unstale":"2023-12-19T14:58:22.550877+0000","last_undegraded":"2023-12-19T14:58:22.550877+0000","last_fullsized":"2023-12-19T14:58:22.550877+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:10.683649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.084247+0000","last_change":"2023-12-19T14:58:23.084247+0000","last_active":"2023-12-19T14:58:23.084247+0000","last_peered":"2023-12-19T14:58:23.084247+0000","last_clean":"2023-12-19T14:58:23.084247+0000","last_became_active":"2023-12-19T14:58:23.083644+0000","last_became_peered":"2023-12-19T14:58:23.083644+0000","last_unstale":"2023-12-19T14:58:23.084247+0000","last_undegraded":"2023-12-19T14:58:23.084247+0000","last_fullsized":"2023-12-19T14:58:23.084247+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:20:24.055272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.537782+0000","last_change":"2023-12-19T14:58:22.535352+0000","last_active":"2023-12-19T14:58:22.537782+0000","last_peered":"2023-12-19T14:58:22.537782+0000","last_clean":"2023-12-19T14:58:22.537782+0000","last_became_active":"2023-12-19T14:58:22.535049+0000","last_became_peered":"2023-12-19T14:58:22.535049+0000","last_unstale":"2023-12-19T14:58:22.537782+0000","last_undegraded":"2023-12-19T14:58:22.537782+0000","last_fullsized":"2023-12-19T14:58:22.537782+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:10.490471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.082637+0000","last_change":"2023-12-19T14:58:23.082637+0000","last_active":"2023-12-19T14:58:23.082637+0000","last_peered":"2023-12-19T14:58:23.082637+0000","last_clean":"2023-12-19T14:58:23.082637+0000","last_became_active":"2023-12-19T14:58:23.081999+0000","last_became_peered":"2023-12-19T14:58:23.081999+0000","last_unstale":"2023-12-19T14:58:23.082637+0000","last_undegraded":"2023-12-19T14:58:23.082637+0000","last_fullsized":"2023-12-19T14:58:23.082637+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:22.505009+0000","last_clean_scrub_stamp":"2023-12-19T14:58:22.505009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:41:35.459633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T14:58:23.551741+0000","last_change":"2023-12-19T14:58:22.545204+0000","last_active":"2023-12-19T14:58:23.551741+0000","last_peered":"2023-12-19T14:58:23.551741+0000","last_clean":"2023-12-19T14:58:23.551741+0000","last_became_active":"2023-12-19T14:58:22.544771+0000","last_became_peered":"2023-12-19T14:58:22.544771+0000","last_unstale":"2023-12-19T14:58:23.551741+0000","last_undegraded":"2023-12-19T14:58:23.551741+0000","last_fullsized":"2023-12-19T14:58:23.551741+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:58:21.490051+0000","last_clean_scrub_stamp":"2023-12-19T14:58:21.490051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:31:05.419064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'249","reported_seq":732,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T14:58:22.522267+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T14:58:22.522267+0000","last_peered":"2023-12-19T14:58:22.522267+0000","last_clean":"2023-12-19T14:58:22.522267+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T14:58:22.522267+0000","last_undegraded":"2023-12-19T14:58:22.522267+0000","last_fullsized":"2023-12-19T14:58:22.522267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575078,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311908,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059932,"statfs":{"total":96636764160,"available":96317370368,"internally_reserved":0,"allocated":2387968,"data_stored":1486990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0820000000000001,"15min":1.093},"min":{"1min":0.69999999999999996,"5min":0.65100000000000002,"15min":0.52300000000000002},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":3.544},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.0880000000000001,"15min":1.1319999999999999},"min":{"1min":0.69999999999999996,"5min":0.629,"15min":0.44700000000000001},"max":{"1min":1.405,"5min":1.5780000000000001,"15min":7.8239999999999998},"last":0.89200000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.109,"15min":1.1080000000000001},"min":{"1min":0.755,"5min":0.69499999999999995,"15min":0.627},"max":{"1min":1.5449999999999999,"5min":1.722,"15min":1.73},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.196,"15min":1.1859999999999999},"min":{"1min":0.60999999999999999,"5min":0.59999999999999998,"15min":0.48699999999999999},"max":{"1min":1.417,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.6140000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.264,"15min":1.2370000000000001},"min":{"1min":0.91200000000000003,"5min":0.77900000000000003,"15min":0.67900000000000005},"max":{"1min":1.488,"5min":1.583,"15min":1.6919999999999999},"last":1.494},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2789999999999999,"15min":1.28},"min":{"1min":0.83499999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.627,"5min":1.6739999999999999,"15min":1.6890000000000001},"last":1.353}]},{"osd":3,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2669999999999999,"15min":1.2869999999999999},"min":{"1min":1.0329999999999999,"5min":0.79700000000000004,"15min":0.78000000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7969999999999999},"last":2.0739999999999998},{"interface":"front","average":{"1min":1.353,"5min":1.3300000000000001,"15min":1.3160000000000001},"min":{"1min":0.99399999999999999,"5min":0.84699999999999998,"15min":0.79700000000000004},"max":{"1min":1.7010000000000001,"5min":1.7729999999999999,"15min":1.9059999999999999},"last":1.849}]},{"osd":4,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.1859999999999999,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.66900000000000004,"15min":0.53500000000000003},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2350000000000001,"15min":1.2170000000000001},"min":{"1min":1.0209999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.7290000000000001},"last":1.8939999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.242,"15min":1.226},"min":{"1min":0.89900000000000002,"5min":0.52100000000000002,"15min":0.496},"max":{"1min":1.5860000000000001,"5min":1.73,"15min":1.8109999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.3280000000000001,"15min":1.306},"min":{"1min":1.0049999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":2.105}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.409,"15min":1.391},"min":{"1min":0.99099999999999999,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.724,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":2.008},{"interface":"front","average":{"1min":1.429,"5min":1.383,"15min":1.3580000000000001},"min":{"1min":0.93799999999999994,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.837,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":2.1720000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1781760,"data_stored":896428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.139,"15min":1.1339999999999999},"min":{"1min":0.53900000000000003,"5min":0.499,"15min":0.41299999999999998},"max":{"1min":1.5309999999999999,"5min":1.611,"15min":1.9219999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1419999999999999,"15min":1.1639999999999999},"min":{"1min":0.75600000000000001,"5min":0.73499999999999999,"15min":0.72799999999999998},"max":{"1min":1.5800000000000001,"5min":1.845,"15min":1.845},"last":0.89900000000000002}]},{"osd":1,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.1120000000000001,"15min":1.1830000000000001},"min":{"1min":0.78300000000000003,"5min":0.66000000000000003,"15min":0.56999999999999995},"max":{"1min":1.5249999999999999,"5min":2.016,"15min":4.4340000000000002},"last":1.101},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.68100000000000005},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":3.226},"last":1.6439999999999999}]},{"osd":2,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2090000000000001,"15min":1.2370000000000001},"min":{"1min":0.754,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.597,"5min":1.7649999999999999,"15min":3.6280000000000001},"last":1.284},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.202,"15min":1.242},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.70799999999999996},"max":{"1min":1.573,"5min":1.802,"15min":3.1619999999999999},"last":1.5609999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.29,"15min":1.296},"min":{"1min":0.76200000000000001,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.782,"5min":1.782,"15min":1.9239999999999999},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.3009999999999999,"15min":1.3240000000000001},"min":{"1min":0.96599999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.702,"5min":1.8779999999999999,"15min":3.6899999999999999},"last":1.024}]},{"osd":4,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1639999999999999,"15min":1.2010000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.621},"max":{"1min":1.3959999999999999,"5min":2.1040000000000001,"15min":3.9359999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.262,"15min":1.2869999999999999},"min":{"1min":0.70099999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6579999999999999,"5min":1.901,"15min":3.3460000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.268,"15min":1.2869999999999999},"min":{"1min":0.91000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6599999999999999,"5min":1.746,"15min":2.0219999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.306,"15min":1.353},"min":{"1min":0.997,"5min":0.82799999999999996,"15min":0.76300000000000001},"max":{"1min":1.657,"5min":1.931,"15min":5.9610000000000003},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 14:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4039999999999999,"15min":1.411},"min":{"1min":0.86599999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.8,"5min":1.847,"15min":4.2510000000000003},"last":1.347},{"interface":"front","average":{"1min":1.482,"5min":1.4259999999999999,"15min":1.4510000000000001},"min":{"1min":1.1639999999999999,"5min":0.65300000000000002,"15min":0.63100000000000001},"max":{"1min":1.784,"5min":2.0910000000000002,"15min":3.742},"last":1.5960000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575078,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303140,"kb_used_data":1756,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068700,"statfs":{"total":96636764160,"available":96326348800,"internally_reserved":0,"allocated":1798144,"data_stored":902698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.101,"15min":1.069},"min":{"1min":0.78600000000000003,"5min":0.63700000000000001,"15min":0.40799999999999997},"max":{"1min":1.5660000000000001,"5min":1.601,"15min":1.601},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.137,"5min":1.1299999999999999,"15min":1.135},"min":{"1min":0.754,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5800000000000001,"5min":1.6539999999999999,"15min":1.7290000000000001},"last":1.2}]},{"osd":1,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.274,"15min":1.288},"min":{"1min":0.78600000000000003,"5min":0.66300000000000003,"15min":0.627},"max":{"1min":1.601,"5min":1.863,"15min":1.9650000000000001},"last":0.877},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.238,"15min":1.2350000000000001},"min":{"1min":0.81699999999999995,"5min":0.77300000000000002,"15min":0.55700000000000005},"max":{"1min":1.6259999999999999,"5min":1.677,"15min":1.677},"last":0.64400000000000002}]},{"osd":2,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.1779999999999999,"15min":1.218},"min":{"1min":0.96999999999999997,"5min":0.61499999999999999,"15min":0.47399999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.9319999999999999},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.1950000000000001,"15min":1.1870000000000001},"min":{"1min":0.89900000000000002,"5min":0.82099999999999995,"15min":0.54300000000000004},"max":{"1min":1.5820000000000001,"5min":1.7,"15min":1.8600000000000001},"last":1.3740000000000001}]},{"osd":3,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.2649999999999999},"min":{"1min":0.91200000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":2.0600000000000001},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.26,"15min":1.2589999999999999},"min":{"1min":0.94399999999999995,"5min":0.74299999999999999,"15min":0.68999999999999995},"max":{"1min":1.5049999999999999,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.163}]},{"osd":4,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.103,"15min":1.0940000000000001},"min":{"1min":0.76100000000000001,"5min":0.69499999999999995,"15min":0.41799999999999998},"max":{"1min":1.4359999999999999,"5min":1.651,"15min":1.651},"last":0.753},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.24,"15min":1.2290000000000001},"min":{"1min":0.68200000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5800000000000001,"5min":1.7709999999999999,"15min":2.27},"last":1.3049999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.355,"15min":1.331},"min":{"1min":0.98399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.7250000000000001,"5min":1.982,"15min":1.984},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3220000000000001,"15min":1.3169999999999999},"min":{"1min":0.92000000000000004,"5min":0.70999999999999996,"15min":0.65200000000000002},"max":{"1min":1.6839999999999999,"5min":1.794,"15min":1.8540000000000001},"last":1.0800000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.363,"15min":1.347},"min":{"1min":0.92100000000000004,"5min":0.64500000000000002,"15min":0.58899999999999997},"max":{"1min":1.6499999999999999,"5min":1.7509999999999999,"15min":2.1720000000000002},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3919999999999999,"15min":1.375},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.68999999999999995},"max":{"1min":1.6830000000000001,"5min":2.048,"15min":2.048},"last":1.4690000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575078,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303132,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068708,"statfs":{"total":96636764160,"available":96326356992,"internally_reserved":0,"allocated":1789952,"data_stored":897410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.03,"15min":1.056},"min":{"1min":0.65600000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":1.625,"15min":1.7869999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0920000000000001,"15min":1.0900000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.524,"5min":1.7490000000000001,"15min":1.8140000000000001},"last":1.0620000000000001}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2390000000000001,"15min":1.234},"min":{"1min":0.69799999999999995,"5min":0.65800000000000003,"15min":0.623},"max":{"1min":1.593,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.615},{"interface":"front","average":{"1min":1.242,"5min":1.2250000000000001,"15min":1.266},"min":{"1min":0.98899999999999999,"5min":0.505,"15min":0.505},"max":{"1min":1.4430000000000001,"5min":1.677,"15min":2.7679999999999998},"last":1.5600000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1759999999999999,"15min":1.1459999999999999},"min":{"1min":0.78400000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.143,"15min":1.2},"min":{"1min":0.78200000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.385,"5min":1.7649999999999999,"15min":1.893},"last":1.3979999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1539999999999999,"15min":1.173},"min":{"1min":0.84999999999999998,"5min":0.78600000000000003,"15min":0.59599999999999997},"max":{"1min":1.528,"5min":1.6100000000000001,"15min":2.0190000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2490000000000001,"15min":1.2569999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7749999999999999},"last":1.2030000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.139,"15min":1.139},"min":{"1min":0.72499999999999998,"5min":0.69199999999999995,"15min":0.57899999999999996},"max":{"1min":1.5089999999999999,"5min":1.6559999999999999,"15min":2.1800000000000002},"last":1.653},{"interface":"front","average":{"1min":1.107,"5min":1.129,"15min":1.1439999999999999},"min":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.51300000000000001},"max":{"1min":1.5920000000000001,"5min":1.6830000000000001,"15min":2.077},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.248,"15min":1.2889999999999999},"min":{"1min":0.68999999999999995,"5min":0.63900000000000001,"15min":0.55000000000000004},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.8119999999999998},"last":1.5},{"interface":"front","average":{"1min":1.383,"5min":1.3779999999999999,"15min":1.399},"min":{"1min":1.024,"5min":0.63400000000000001,"15min":0.60999999999999999},"max":{"1min":1.6040000000000001,"5min":2.1549999999999998,"15min":3.0379999999999998},"last":1.732}]},{"osd":7,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2809999999999999,"15min":1.3089999999999999},"min":{"1min":0.89600000000000002,"5min":0.69299999999999995,"15min":0.60899999999999999},"max":{"1min":1.5429999999999999,"5min":1.639,"15min":2.9910000000000001},"last":1.593},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.429,"15min":1.4279999999999999},"min":{"1min":0.97099999999999997,"5min":0.755,"15min":0.72599999999999998},"max":{"1min":1.9079999999999999,"5min":2.0329999999999999,"15min":3.008},"last":1.704}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307244,"kb_used_data":1764,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064596,"statfs":{"total":96636764160,"available":96322146304,"internally_reserved":0,"allocated":1806336,"data_stored":904108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86399999999999999},"min":{"1min":0.46300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.96899999999999997,"15min":0.98199999999999998},"min":{"1min":0.64300000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.534,"5min":1.542,"15min":1.7270000000000001},"last":0.93300000000000005}]},{"osd":1,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91400000000000003,"15min":0.93799999999999994},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2090000000000001,"5min":1.611,"15min":1.611},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.53700000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.3280000000000001,"5min":1.5780000000000001,"15min":1.758},"last":0.84499999999999997}]},{"osd":2,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.117,"15min":1.123},"min":{"1min":0.64200000000000002,"5min":0.61499999999999999,"15min":0.56100000000000005},"max":{"1min":1.3460000000000001,"5min":1.778,"15min":1.778},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.77600000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6519999999999999,"5min":1.72,"15min":1.821},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3069999999999999,"15min":1.319},"min":{"1min":0.77800000000000002,"5min":0.71799999999999997,"15min":0.69899999999999995},"max":{"1min":1.6679999999999999,"5min":1.7589999999999999,"15min":1.782},"last":0.997},{"interface":"front","average":{"1min":1.204,"5min":1.2649999999999999,"15min":1.286},"min":{"1min":0.82199999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.708,"5min":1.708,"15min":1.7989999999999999},"last":1.0309999999999999}]},{"osd":5,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3069999999999999,"15min":1.3009999999999999},"min":{"1min":0.95999999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6990000000000001,"5min":1.7709999999999999,"15min":1.893},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.3540000000000001,"15min":1.3779999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.6240000000000001,"5min":2.1619999999999999,"15min":2.4990000000000001},"last":1.206}]},{"osd":6,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.423,"15min":1.4430000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7070000000000001,"5min":2.2709999999999999,"15min":2.6040000000000001},"last":1.077},{"interface":"front","average":{"1min":1.323,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.85499999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6659999999999999,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.177}]},{"osd":7,"last update":"Tue Dec 19 14:57:30 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4950000000000001,"15min":1.502},"min":{"1min":1.0129999999999999,"5min":0.93300000000000005,"15min":0.88400000000000001},"max":{"1min":1.7430000000000001,"5min":1.9970000000000001,"15min":2.1640000000000001},"last":1.147},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.4419999999999999,"15min":1.448},"min":{"1min":1.087,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.6990000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.1319999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307224,"kb_used_data":1744,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064616,"statfs":{"total":96636764160,"available":96322166784,"internally_reserved":0,"allocated":1785856,"data_stored":901527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.93999999999999995,"15min":0.91000000000000003},"min":{"1min":0.66000000000000003,"5min":0.60199999999999998,"15min":0.39000000000000001},"max":{"1min":1.3260000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.288},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.05,"15min":1.0609999999999999},"min":{"1min":0.70299999999999996,"5min":0.57299999999999995,"15min":0.53300000000000003},"max":{"1min":1.524,"5min":2.036,"15min":2.036},"last":1.323}]},{"osd":1,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1659999999999999,"15min":1.137},"min":{"1min":0.93500000000000005,"5min":0.64300000000000002,"15min":0.56599999999999995},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0509999999999999,"15min":1.069},"min":{"1min":0.71599999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.5609999999999999,"5min":1.853,"15min":1.853},"last":1.5249999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.995,"15min":0.99399999999999999},"min":{"1min":0.80000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.623},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0760000000000001,"15min":1.0860000000000001},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.55000000000000004},"max":{"1min":1.2829999999999999,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.385,"15min":1.3799999999999999},"min":{"1min":1.1339999999999999,"5min":0.747,"15min":0.58699999999999997},"max":{"1min":1.8580000000000001,"5min":2.0270000000000001,"15min":2.044},"last":1.446},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3919999999999999,"15min":1.393},"min":{"1min":1.087,"5min":0.85299999999999998,"15min":0.746},"max":{"1min":1.802,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.391}]},{"osd":5,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3600000000000001,"15min":1.341},"min":{"1min":1.131,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.7769999999999999,"5min":2.0979999999999999,"15min":2.395},"last":1.589},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.331,"15min":1.331},"min":{"1min":1.073,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.5549999999999999,"5min":2.0819999999999999,"15min":2.1200000000000001},"last":1.4339999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3520000000000001},"min":{"1min":1.075,"5min":0.85699999999999998,"15min":0.83299999999999996},"max":{"1min":1.8100000000000001,"5min":1.9710000000000001,"15min":1.9850000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.47,"15min":1.4370000000000001},"min":{"1min":1.113,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.0390000000000001,"5min":2.2469999999999999,"15min":2.4729999999999999},"last":2.0390000000000001}]},{"osd":7,"last update":"Tue Dec 19 14:58:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4950000000000001,"15min":1.4730000000000001},"min":{"1min":1.0509999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.7929999999999999,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.4690000000000001,"15min":1.4590000000000001},"min":{"1min":1.01,"5min":1.01,"15min":0.83599999999999997},"max":{"1min":1.994,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.994}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307220,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064620,"statfs":{"total":96636764160,"available":96322170880,"internally_reserved":0,"allocated":1781760,"data_stored":895896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88400000000000001,"15min":0.86699999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.48499999999999999},"max":{"1min":1.5580000000000001,"5min":2.238,"15min":2.238},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.96699999999999997,"15min":0.97699999999999998},"min":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.53100000000000003},"max":{"1min":1.486,"5min":1.49,"15min":1.7070000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0249999999999999,"15min":1.026},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.54600000000000004},"max":{"1min":1.9790000000000001,"5min":2.444,"15min":2.444},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.0880000000000001,"15min":1.1100000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.442},"max":{"1min":1.829,"5min":1.829,"15min":2.0230000000000001},"last":1.3979999999999999}]},{"osd":3,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.216,"15min":1.228},"min":{"1min":0.82699999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.5169999999999999,"5min":1.6919999999999999,"15min":2.0720000000000001},"last":1.5429999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.085,"15min":1.0800000000000001},"min":{"1min":0.72699999999999998,"5min":0.628,"15min":0.56899999999999995},"max":{"1min":1.302,"5min":1.6890000000000001,"15min":1.7270000000000001},"last":1.272}]},{"osd":4,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.304,"15min":1.3100000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.624},"max":{"1min":2.5539999999999998,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.464},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3839999999999999,"15min":1.399},"min":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.67700000000000005},"max":{"1min":1.7190000000000001,"5min":1.8029999999999999,"15min":3.5630000000000002},"last":1.798}]},{"osd":5,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2669999999999999,"15min":1.286},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.80000000000000004},"max":{"1min":1.661,"5min":1.857,"15min":1.857},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.421,"15min":1.389},"min":{"1min":0.88,"5min":0.86399999999999999,"15min":0.71699999999999997},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.661}]},{"osd":6,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.355,"15min":1.3620000000000001},"min":{"1min":1.002,"5min":0.97899999999999998,"15min":0.53000000000000003},"max":{"1min":1.627,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4410000000000001,"15min":1.45},"min":{"1min":0.96099999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.6389999999999998,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.7709999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4419999999999999,"15min":1.4670000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.89500000000000002},"max":{"1min":1.923,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":1.704},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.448,"15min":1.4390000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.93799999999999994},"max":{"1min":1.863,"5min":2.0169999999999999,"15min":2.0609999999999999},"last":1.734}]}]},{"osd":0,"up_from":13,"seq":55834575077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":2371584,"data_stored":1485253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.86899999999999999,"15min":0.85299999999999998},"min":{"1min":0.64300000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.96099999999999997,"15min":0.95999999999999996},"min":{"1min":0.5,"5min":0.433,"15min":0.433},"max":{"1min":1.1970000000000001,"5min":1.577,"15min":1.8879999999999999},"last":0.87}]},{"osd":2,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.141,"15min":1.107},"min":{"1min":0.74199999999999999,"5min":0.69299999999999995,"15min":0.55200000000000005},"max":{"1min":1.5580000000000001,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.2090000000000001,"15min":1.2210000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.7909999999999999,"5min":1.821,"15min":1.982},"last":1.071}]},{"osd":3,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.98199999999999998,"15min":1.01},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.216,"5min":1.6359999999999999,"15min":1.726},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.181},"min":{"1min":0.63300000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.714,"5min":1.829,"15min":2.0209999999999999},"last":1.177}]},{"osd":4,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.81999999999999995,"5min":0.79200000000000004,"15min":0.70199999999999996},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":2.153},"last":1.214},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.274,"15min":1.284},"min":{"1min":0.96899999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.6759999999999999,"5min":1.7170000000000001,"15min":1.7929999999999999},"last":1.2270000000000001}]},{"osd":5,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3919999999999999,"15min":1.3799999999999999},"min":{"1min":0.96699999999999997,"5min":0.879,"15min":0.71899999999999997},"max":{"1min":1.7689999999999999,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3480000000000001,"15min":1.379},"min":{"1min":1.0589999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6479999999999999,"5min":1.9339999999999999,"15min":2.2189999999999999},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3149999999999999,"15min":1.3240000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":2.0150000000000001},"last":1.337},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.452,"15min":1.446},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.859,"5min":2.0590000000000002,"15min":2.1240000000000001},"last":1.3959999999999999}]},{"osd":7,"last update":"Tue Dec 19 14:57:27 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4139999999999999,"15min":1.4510000000000001},"min":{"1min":1.0129999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.823,"5min":2.0019999999999998,"15min":2.2629999999999999},"last":1.258},{"interface":"front","average":{"1min":1.357,"5min":1.3999999999999999,"15min":1.415},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.88200000000000001},"max":{"1min":1.613,"5min":1.802,"15min":2.0379999999999998},"last":1.431}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T14:58:28.275 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T14:58:28.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T14:58:28.483 INFO:tasks.ceph.mds.b.smithi125.stderr:2023-12-19T14:58:28.478+0000 7f049bdd3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T14:58:28.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.488+0000 7fb090b85700 1 Processor -- start 2023-12-19T14:58:28.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.488+0000 7fb090b85700 1 -- start start 2023-12-19T14:58:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb090b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 0x7fb08c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb090b85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 0x7fb08c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb090b85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08c0cbdf0 con 0x7fb08c0cb970 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb090b85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08c0cc0d0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb090b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0cc3b0 con 0x7fb08c158e20 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08ad9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 0x7fb08c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 0x7fb08c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08ad9d700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 0x7fb08c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:53284/0 (socket says 172.21.15.50:53284) 2023-12-19T14:58:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08ad9d700 1 -- 172.21.15.50:0/1067595078 learned_addr learned my addr 172.21.15.50:0/1067595078 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:28.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 -- 172.21.15.50:0/1067595078 >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 msgr2=0x7fb08c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:28.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 --2- 172.21.15.50:0/1067595078 >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 0x7fb08c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 -- 172.21.15.50:0/1067595078 >> v1:172.21.15.125:6789/0 conn(0x7fb08c0cb970 legacy=0x7fb08c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 -- 172.21.15.50:0/1067595078 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c0cc690 con 0x7fb08c158e20 2023-12-19T14:58:28.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.492+0000 7fb08a59c700 1 --2- 172.21.15.50:0/1067595078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 0x7fb08c0ce5c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0098e0 tx=0x7fb07c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a08c8bc6543ce87 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb08959a700 1 -- 172.21.15.50:0/1067595078 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c00b640 con 0x7fb08c158e20 2023-12-19T14:58:28.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb08959a700 1 -- 172.21.15.50:0/1067595078 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c00b7a0 con 0x7fb08c158e20 2023-12-19T14:58:28.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb08959a700 1 -- 172.21.15.50:0/1067595078 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c00bb40 con 0x7fb08c158e20 2023-12-19T14:58:28.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 -- 172.21.15.50:0/1067595078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 msgr2=0x7fb08c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 --2- 172.21.15.50:0/1067595078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 0x7fb08c0ce5c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0098e0 tx=0x7fb07c0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 -- 172.21.15.50:0/1067595078 shutdown_connections 2023-12-19T14:58:28.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 --2- 172.21.15.50:0/1067595078 >> v2:172.21.15.125:3300/0 conn(0x7fb08c0ca2c0 0x7fb08c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 --2- 172.21.15.50:0/1067595078 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c158e20 0x7fb08c0ce5c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 -- 172.21.15.50:0/1067595078 >> 172.21.15.50:0/1067595078 conn(0x7fb08c067bb0 msgr2=0x7fb08c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 -- 172.21.15.50:0/1067595078 shutdown_connections 2023-12-19T14:58:28.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.496+0000 7fb090b85700 1 -- 172.21.15.50:0/1067595078 wait complete. 2023-12-19T14:58:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 Processor -- start 2023-12-19T14:58:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 -- start start 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb08c158e20 0x7fb08c169150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c0b20c0 0x7fb08c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c1548b0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c151f30 con 0x7fb08c158e20 2023-12-19T14:58:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb090b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c1699e0 con 0x7fb08c0b20c0 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb08a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb08a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55338/0 (socket says 172.21.15.50:55338) 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb08a59c700 1 -- 172.21.15.50:0/756751112 learned_addr learned my addr 172.21.15.50:0/756751112 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb089d9b700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb08c158e20 0x7fb08c169150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.500+0000 7fb08ad9d700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c0b20c0 0x7fb08c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 -- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c0b20c0 msgr2=0x7fb08c150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c0b20c0 0x7fb08c150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 -- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb08c158e20 msgr2=0x7fb08c169150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb08c158e20 0x7fb08c169150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 -- 172.21.15.50:0/756751112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c009420 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb08a59c700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0cc4f0 tx=0x7fb07c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c01bc40 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c01bdc0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c169cc0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c011670 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c169690 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.504+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c071120 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.508+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb07c011a20 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.508+0000 7fb07a7fc700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 0x7fb06c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.508+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb07c07dfb0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.508+0000 7fb089d9b700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 0x7fb06c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.512+0000 7fb089d9b700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 0x7fb06c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0cf0d0 tx=0x7fb080007e80 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.516+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb07c04e270 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.748+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb08c06a7a0 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.756+0000 7fb07a7fc700 1 -- 172.21.15.50:0/756751112 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb07c051890 con 0x7fb08c0ca2c0 2023-12-19T14:58:28.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.760+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 msgr2=0x7fb06c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.760+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 0x7fb06c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0cf0d0 tx=0x7fb080007e80 comp rx=0 tx=0).stop 2023-12-19T14:58:28.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.760+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 msgr2=0x7fb08c1519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.760+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0cc4f0 tx=0x7fb07c017000 comp rx=0 tx=0).stop 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 shutdown_connections 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb08c158e20 0x7fb08c169150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb06c060670 0x7fb06c062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb08c0b20c0 0x7fb08c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 --2- 172.21.15.50:0/756751112 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb08c0ca2c0 0x7fb08c1519f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 >> 172.21.15.50:0/756751112 conn(0x7fb08c067bb0 msgr2=0x7fb08c0d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 shutdown_connections 2023-12-19T14:58:28.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.764+0000 7fb090b85700 1 -- 172.21.15.50:0/756751112 wait complete. 2023-12-19T14:58:28.765 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T14:58:28.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T14:58:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.956+0000 7f7910b23700 1 Processor -- start 2023-12-19T14:58:28.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.956+0000 7f7910b23700 1 -- start start 2023-12-19T14:58:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.960+0000 7f7910b23700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f790c158d60 0x7f790c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.960+0000 7f7910b23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c06c7c0 con 0x7f790c0c5380 2023-12-19T14:58:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.960+0000 7f7910b23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c06caa0 con 0x7f790c06b7a0 2023-12-19T14:58:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.960+0000 7f7910b23700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c15ce60 con 0x7f790c158d60 2023-12-19T14:58:28.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.960+0000 7f790bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f790c0c5380 0x7f790c06cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34508/0 (socket says 172.21.15.50:34508) 2023-12-19T14:58:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790bfff700 1 -- 172.21.15.50:0/4005203888 learned_addr learned my addr 172.21.15.50:0/4005203888 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790b7fe700 1 --2- 172.21.15.50:0/4005203888 >> v2:172.21.15.125:3300/0 conn(0x7f790c158d60 0x7f790c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790affd700 1 -- 172.21.15.50:0/4005203888 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32282626 0 0) 0x7f790c06c7c0 con 0x7f790c0c5380 2023-12-19T14:58:28.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790affd700 1 -- 172.21.15.50:0/4005203888 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f8003570 con 0x7f790c0c5380 2023-12-19T14:58:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790b7fe700 1 -- 172.21.15.50:0/4005203888 >> v1:172.21.15.125:6789/0 conn(0x7f790c06b7a0 legacy=0x7f790c0c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790b7fe700 1 -- 172.21.15.50:0/4005203888 >> v1:172.21.15.50:6789/0 conn(0x7f790c0c5380 legacy=0x7f790c06cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790b7fe700 1 -- 172.21.15.50:0/4005203888 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c15d130 con 0x7f790c158d60 2023-12-19T14:58:28.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790b7fe700 1 --2- 172.21.15.50:0/4005203888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c158d60 0x7f790c15c920 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0098e0 tx=0x7f78fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8ca741023ed1b13 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790affd700 1 -- 172.21.15.50:0/4005203888 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00b640 con 0x7f790c158d60 2023-12-19T14:58:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790affd700 1 -- 172.21.15.50:0/4005203888 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc00b7a0 con 0x7f790c158d60 2023-12-19T14:58:28.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f790affd700 1 -- 172.21.15.50:0/4005203888 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00bb40 con 0x7f790c158d60 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 -- 172.21.15.50:0/4005203888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c158d60 msgr2=0x7f790c15c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 --2- 172.21.15.50:0/4005203888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c158d60 0x7f790c15c920 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0098e0 tx=0x7f78fc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 -- 172.21.15.50:0/4005203888 shutdown_connections 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 --2- 172.21.15.50:0/4005203888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c158d60 0x7f790c15c920 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 -- 172.21.15.50:0/4005203888 >> 172.21.15.50:0/4005203888 conn(0x7f790c067bb0 msgr2=0x7f790c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.964+0000 7f7910b23700 1 -- 172.21.15.50:0/4005203888 shutdown_connections 2023-12-19T14:58:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.968+0000 7f7910b23700 1 -- 172.21.15.50:0/4005203888 wait complete. 2023-12-19T14:58:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.968+0000 7f7910b23700 1 Processor -- start 2023-12-19T14:58:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.968+0000 7f7910b23700 1 -- start start 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f790c06b7a0 0x7f790c152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 0x7f790c16cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c157370 con 0x7f790c158d60 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c16af90 con 0x7f790c0c5380 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c16a980 con 0x7f790c06b7a0 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55352/0 (socket says 172.21.15.50:55352) 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 -- 172.21.15.50:0/4208643875 learned_addr learned my addr 172.21.15.50:0/4208643875 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790b7fe700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f790c06b7a0 0x7f790c152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7902ffd700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 0x7f790c16cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 -- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f790c06b7a0 msgr2=0x7f790c152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f790c06b7a0 0x7f790c152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 -- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 msgr2=0x7f790c16cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 0x7f790c16cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 -- 172.21.15.50:0/4208643875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc009420 con 0x7f790c158d60 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7902ffd700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 0x7f790c16cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f790bfff700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7904001460 tx=0x7f7904001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790400a940 con 0x7f790c158d60 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790400aaa0 con 0x7f790c158d60 2023-12-19T14:58:28.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c157090 con 0x7f790c158d60 2023-12-19T14:58:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790400ae20 con 0x7f790c158d60 2023-12-19T14:58:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.972+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c140b30 con 0x7f790c158d60 2023-12-19T14:58:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.976+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c16a7e0 con 0x7f790c158d60 2023-12-19T14:58:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.976+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7904007300 con 0x7f790c158d60 2023-12-19T14:58:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.980+0000 7f7903fff700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 0x7f78ec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.980+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f790407cf10 con 0x7f790c158d60 2023-12-19T14:58:28.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.980+0000 7f790b7fe700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 0x7f78ec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:28.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.980+0000 7f790b7fe700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 0x7f78ec062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0093f0 tx=0x7f78fc017000 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:28.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:28.984+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f790c16a7e0 con 0x7f790c158d60 2023-12-19T14:58:29.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.196+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f790c16a7e0 con 0x7f790c158d60 2023-12-19T14:58:29.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.196+0000 7f7903fff700 1 -- 172.21.15.50:0/4208643875 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f790c16a7e0 con 0x7f790c158d60 2023-12-19T14:58:29.198 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:29.198 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 msgr2=0x7f78ec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 0x7f78ec062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0093f0 tx=0x7f78fc017000 comp rx=0 tx=0).stop 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 msgr2=0x7f790c16d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f7904001460 tx=0x7f7904001ae0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 shutdown_connections 2023-12-19T14:58:29.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f790c0c5380 0x7f790c16cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f78ec060670 0x7f78ec062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f790c158d60 0x7f790c16d150 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 --2- 172.21.15.50:0/4208643875 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f790c06b7a0 0x7f790c152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 >> 172.21.15.50:0/4208643875 conn(0x7f790c067bb0 msgr2=0x7f790c0c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 shutdown_connections 2023-12-19T14:58:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.200+0000 7f7910b23700 1 -- 172.21.15.50:0/4208643875 wait complete. 2023-12-19T14:58:29.203 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T14:58:29.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T14:58:22.983915+0000', 'modified': '2023-12-19T14:58:26.556533+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8379': {'gid': 8379, 'name': 'b', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6837/528853913', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6835', 'nonce': 528853913}, {'type': 'v1', 'addr': '172.21.15.125:6837', 'nonce': 528853913}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T14:58:29.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T14:58:29.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T14:58:29.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.800+0000 7f85e2f23700 1 Processor -- start 2023-12-19T14:58:29.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.800+0000 7f85e2f23700 1 -- start start 2023-12-19T14:58:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2f23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc158e20 0x7f85dc06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2f23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85dc0ce810 0x7f85dc0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2f23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc06b730 con 0x7f85dc1591f0 2023-12-19T14:58:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2f23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc06ab20 con 0x7f85dc0ce810 2023-12-19T14:58:29.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2f23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc06ae00 con 0x7f85dc158e20 2023-12-19T14:58:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85dc0ce810 0x7f85dc0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85dc0ce810 0x7f85dc0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55366/0 (socket says 172.21.15.50:55366) 2023-12-19T14:58:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 -- 172.21.15.50:0/1188882469 learned_addr learned my addr 172.21.15.50:0/1188882469 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e1720700 1 --1- 172.21.15.50:0/1188882469 >> v1:172.21.15.125:6789/0 conn(0x7f85dc1591f0 0x7f85dc0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:34528/0 (socket says 172.21.15.50:34528) 2023-12-19T14:58:29.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e1f21700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc158e20 0x7f85dc06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 -- 172.21.15.50:0/1188882469 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc158e20 msgr2=0x7f85dc06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc158e20 0x7f85dc06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 -- 172.21.15.50:0/1188882469 >> v1:172.21.15.125:6789/0 conn(0x7f85dc1591f0 legacy=0x7f85dc0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e0f1f700 1 -- 172.21.15.50:0/1188882469 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977728228 0 0) 0x7f85dc06b730 con 0x7f85dc1591f0 2023-12-19T14:58:29.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 -- 172.21.15.50:0/1188882469 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc06b0e0 con 0x7f85dc0ce810 2023-12-19T14:58:29.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e2722700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc0ca950 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009f20 tx=0x7f85d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5643efcab35990c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e0f1f700 1 -- 172.21.15.50:0/1188882469 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400e070 con 0x7f85dc0ce810 2023-12-19T14:58:29.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e0f1f700 1 -- 172.21.15.50:0/1188882469 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d400b9d0 con 0x7f85dc0ce810 2023-12-19T14:58:29.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.804+0000 7f85e0f1f700 1 -- 172.21.15.50:0/1188882469 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d40092e0 con 0x7f85dc0ce810 2023-12-19T14:58:29.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 -- 172.21.15.50:0/1188882469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 msgr2=0x7f85dc0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc0ca950 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009f20 tx=0x7f85d4009840 comp rx=0 tx=0).stop 2023-12-19T14:58:29.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 -- 172.21.15.50:0/1188882469 shutdown_connections 2023-12-19T14:58:29.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc0ca950 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 --2- 172.21.15.50:0/1188882469 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc158e20 0x7f85dc06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 -- 172.21.15.50:0/1188882469 >> 172.21.15.50:0/1188882469 conn(0x7f85dc067bb0 msgr2=0x7f85dc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 -- 172.21.15.50:0/1188882469 shutdown_connections 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.808+0000 7f85e2f23700 1 -- 172.21.15.50:0/1188882469 wait complete. 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 Processor -- start 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- start start 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85dc158e20 0x7f85dc169600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc1591f0 0x7f85dc1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc16a5e0 con 0x7f85dc0ce810 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc1569c0 con 0x7f85dc158e20 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc16a020 con 0x7f85dc1591f0 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55378/0 (socket says 172.21.15.50:55378) 2023-12-19T14:58:29.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 -- 172.21.15.50:0/3188408445 learned_addr learned my addr 172.21.15.50:0/3188408445 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 -- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc1591f0 msgr2=0x7f85dc1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc1591f0 0x7f85dc1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 -- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85dc158e20 msgr2=0x7f85dc169600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85dc158e20 0x7f85dc169600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 -- 172.21.15.50:0/3188408445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d4009ae0 con 0x7f85dc0ce810 2023-12-19T14:58:29.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f85d800ec80 tx=0x7f85d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85d1ffb700 1 -- 172.21.15.50:0/3188408445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d800c9d0 con 0x7f85dc0ce810 2023-12-19T14:58:29.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85d1ffb700 1 -- 172.21.15.50:0/3188408445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d800cb30 con 0x7f85dc0ce810 2023-12-19T14:58:29.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc16a300 con 0x7f85dc0ce810 2023-12-19T14:58:29.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85d1ffb700 1 -- 172.21.15.50:0/3188408445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8013470 con 0x7f85dc0ce810 2023-12-19T14:58:29.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.812+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc1527e0 con 0x7f85dc0ce810 2023-12-19T14:58:29.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.816+0000 7f85d1ffb700 1 -- 172.21.15.50:0/3188408445 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f85d8013820 con 0x7f85dc0ce810 2023-12-19T14:58:29.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85d1ffb700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 0x7f85c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85d1ffb700 1 -- 172.21.15.50:0/3188408445 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f85d807cf10 con 0x7f85dc0ce810 2023-12-19T14:58:29.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85e1720700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 0x7f85c0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85e6457740 1 Processor -- start 2023-12-19T14:58:29.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85e1720700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 0x7f85c0062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0c7520 tx=0x7f85cc007b70 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.820+0000 7f85e6457740 1 -- start start 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x2309f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x2237f30 con 0x20d7790 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2233ed0 con 0x208cae0 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x21fa0b0 con 0x208c710 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55382/0 (socket says 172.21.15.50:55382) 2023-12-19T14:58:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 -- 172.21.15.50:0/2604186987 learned_addr learned my addr 172.21.15.50:0/2604186987 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e1f21700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x2309f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e1720700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 -- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 msgr2=0x230a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 -- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 msgr2=0x2309f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x2309f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 -- 172.21.15.50:0/2604186987 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x20d89e0 con 0x20d7790 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e1f21700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x2309f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85e2722700 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f85d4012000 tx=0x7f85d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf96537fee65d57 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85c6ffd700 1 -- 172.21.15.50:0/2604186987 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400bb40 con 0x20d7790 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85c6ffd700 1 -- 172.21.15.50:0/2604186987 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d401d420 con 0x20d7790 2023-12-19T14:58:29.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.824+0000 7f85c6ffd700 1 -- 172.21.15.50:0/2604186987 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d401d820 con 0x20d7790 2023-12-19T14:58:29.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- 172.21.15.50:0/2604186987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 msgr2=0x230a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f85d4012000 tx=0x7f85d4018000 comp rx=0 tx=0).stop 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- 172.21.15.50:0/2604186987 shutdown_connections 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x230a980 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 --2- 172.21.15.50:0/2604186987 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x2309f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- 172.21.15.50:0/2604186987 >> 172.21.15.50:0/2604186987 conn(0x208ceb0 msgr2=0x22c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- 172.21.15.50:0/2604186987 shutdown_connections 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- 172.21.15.50:0/2604186987 wait complete. 2023-12-19T14:58:29.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 -- start start 2023-12-19T14:58:29.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.828+0000 7f85e6457740 1 Processor -- start 2023-12-19T14:58:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x20d7790 0x22efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x21f9d50 con 0x208cae0 2023-12-19T14:58:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x21d2970 con 0x20d7790 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x2238810 con 0x208c710 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 10 client.0 _ll_register_callbacks cb 0x227d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55388/0 (socket says 172.21.15.50:55388) 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 -- 172.21.15.50:0/3641164491 learned_addr learned my addr 172.21.15.50:0/3641164491 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e2722700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x20d7790 0x22efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 -- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 msgr2=0x230e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 -- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x20d7790 msgr2=0x22efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x20d7790 0x22efcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 -- 172.21.15.50:0/3641164491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc06b0e0 con 0x208cae0 2023-12-19T14:58:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e1720700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0077b0 tx=0x7f85cc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78c03d43471a264 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc005a60 con 0x208cae0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x21d2cc0 con 0x208cae0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f85cc016c70 con 0x208cae0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85cc023e60 con 0x208cae0 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 10 client.8505 fetch_fsmap learned FSMap version 66 2023-12-19T14:58:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.832+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x2202370 con 0x208cae0 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x21d2cc0 con 0x208cae0 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 66) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f85cc01f070 con 0x208cae0 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 10 client.8505 fetch_fsmap finished waiting for FSMap version 66 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 10 client.8505 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 20 client.8505 populate_metadata read hostname 'smithi050' 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22f1250 0x23196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 4 client.8505 mds_command: new command op to 8379 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 --> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x21d63e0 con 0x22f1250 2023-12-19T14:58:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22f1250 0x23196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85e1f21700 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22f1250 0x23196b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30167b35678cb297 server_cookie=e4731daef17ade0e in_seq=0 out_seq=0 2023-12-19T14:58:29.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.836+0000 7f85bbfff700 10 client.8505 ms_handle_connect on v2:172.21.15.125:6835/528853913 2023-12-19T14:58:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85bbfff700 1 -- 172.21.15.50:0/3641164491 <== mds.0 v2:172.21.15.125:6835/528853913 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x21d63e0 con 0x22f1250 2023-12-19T14:58:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85bbfff700 10 client.8505 handle_command_reply: tid=0 2023-12-19T14:58:29.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 client.8505 shutdown 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 msgr2=0x230ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0077b0 tx=0x7f85cc011000 comp rx=0 tx=0).stop 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 shutdown_connections 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22f1250 0x23196b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x20d7790 0x22efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x208cae0 0x230ea50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 --2- 172.21.15.50:0/3641164491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x230e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85bbfff700 0 client.8505 ms_handle_reset on v2:172.21.15.125:6835/528853913 2023-12-19T14:58:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 >> 172.21.15.50:0/3641164491 conn(0x208ceb0 msgr2=0x22bef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 shutdown_connections 2023-12-19T14:58:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 1 -- 172.21.15.50:0/3641164491 wait complete. 2023-12-19T14:58:29.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.840+0000 7f85e6457740 20 client.8505 trim_cache size 0 max 16384 2023-12-19T14:58:29.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.860+0000 7f85e6457740 1 Processor -- start 2023-12-19T14:58:29.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.860+0000 7f85e6457740 1 -- start start 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 0x22c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208cae0 0x22c1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x22f00b0 con 0x20d7790 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x222fb50 con 0x208c710 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x21057c0 con 0x208cae0 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55390/0 (socket says 172.21.15.50:55390) 2023-12-19T14:58:29.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 -- 172.21.15.50:0/783290102 learned_addr learned my addr 172.21.15.50:0/783290102 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e1f21700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 0x22c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e1720700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208cae0 0x22c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 -- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208cae0 msgr2=0x22c1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208cae0 0x22c1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 -- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 msgr2=0x22c0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 0x22c0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 -- 172.21.15.50:0/783290102 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x21e8d80 con 0x20d7790 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e1f21700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 0x22c0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85e2722700 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x21e0750 tx=0x7f85d401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbde150412f680c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85c77fe700 1 -- 172.21.15.50:0/783290102 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d401dd20 con 0x20d7790 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85c77fe700 1 -- 172.21.15.50:0/783290102 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d402d040 con 0x20d7790 2023-12-19T14:58:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.864+0000 7f85c77fe700 1 -- 172.21.15.50:0/783290102 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d401d7a0 con 0x20d7790 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- 172.21.15.50:0/783290102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 msgr2=0x23096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x21e0750 tx=0x7f85d401e000 comp rx=0 tx=0).stop 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- 172.21.15.50:0/783290102 shutdown_connections 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x23096e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208cae0 0x22c1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 --2- 172.21.15.50:0/783290102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208c710 0x22c0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- 172.21.15.50:0/783290102 >> 172.21.15.50:0/783290102 conn(0x208ceb0 msgr2=0x22c01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- 172.21.15.50:0/783290102 shutdown_connections 2023-12-19T14:58:29.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- 172.21.15.50:0/783290102 wait complete. 2023-12-19T14:58:29.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 -- start start 2023-12-19T14:58:29.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.868+0000 7f85e6457740 1 Processor -- start 2023-12-19T14:58:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x2362c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x22bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x21e2c90 con 0x20d7790 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2208610 con 0x208cae0 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x2209550 con 0x208c710 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 10 client.0 _ll_register_callbacks cb 0x20589d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55392/0 (socket says 172.21.15.50:55392) 2023-12-19T14:58:29.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 -- 172.21.15.50:0/1155846193 learned_addr learned my addr 172.21.15.50:0/1155846193 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e1f21700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x2362c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e1720700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x22bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 -- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 msgr2=0x2362c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x2362c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 -- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 msgr2=0x22bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:29.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x22bf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 -- 172.21.15.50:0/1155846193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x20d89e0 con 0x20d7790 2023-12-19T14:58:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e2722700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f85d4003440 tx=0x7f85d402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b689314777e29d98 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T14:58:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4032650 con 0x20d7790 2023-12-19T14:58:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f85d40327b0 con 0x20d7790 2023-12-19T14:58:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85d401d420 con 0x20d7790 2023-12-19T14:58:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85d4032b80 con 0x20d7790 2023-12-19T14:58:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 10 client.8511 fetch_fsmap learned FSMap version 66 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.872+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x21dd130 con 0x20d7790 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d401d420 con 0x20d7790 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 66) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f85d4023030 con 0x20d7790 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 10 client.8511 fetch_fsmap finished waiting for FSMap version 66 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 10 client.8511 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 20 client.8511 populate_metadata read hostname 'smithi050' 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22c3910 0x22c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 4 client.8511 mds_command: new command op to 8379 tid=0[{"prefix": "status"}] 2023-12-19T14:58:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 --> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] -- command(tid 0: {"prefix": "status"}) v1 -- 0x2208410 con 0x22c3910 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e1f21700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22c3910 0x22c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e1f21700 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22c3910 0x22c3ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da4b29c0901b038c server_cookie=962f1b57049dd1be in_seq=0 out_seq=0 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 10 client.8511 ms_handle_connect on v2:172.21.15.125:6835/528853913 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 1 -- 172.21.15.50:0/1155846193 <== mds.0 v2:172.21.15.125:6835/528853913 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x2208410 con 0x22c3910 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 10 client.8511 handle_command_reply: tid=0 2023-12-19T14:58:29.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 client.8511 shutdown 2023-12-19T14:58:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 msgr2=0x22bfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f85d4003440 tx=0x7f85d402e000 comp rx=0 tx=0).stop 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 shutdown_connections 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:6835/528853913,v1:172.21.15.125:6837/528853913] conn(0x22c3910 0x22c3ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x208cae0 0x22bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x20d7790 0x22bfe30 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 --2- 172.21.15.50:0/1155846193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x208c710 0x2362c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85bbfff700 0 client.8511 ms_handle_reset on v2:172.21.15.125:6835/528853913 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 >> 172.21.15.50:0/1155846193 conn(0x208ceb0 msgr2=0x22c1570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 shutdown_connections 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.876+0000 7f85e6457740 1 -- 172.21.15.50:0/1155846193 wait complete. 2023-12-19T14:58:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e6457740 20 client.8511 trim_cache size 0 max 16384 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 8379, 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3489443850000002, 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 66, 2023-12-19T14:58:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 144, 2023-12-19T14:58:29.881 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 144, 2023-12-19T14:58:29.881 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.258260807999999 2023-12-19T14:58:29.881 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:58:29.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 msgr2=0x7f85c0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 0x7f85c0062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0c7520 tx=0x7f85cc007b70 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 msgr2=0x7f85dc1522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f85d800ec80 tx=0x7f85d80091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 shutdown_connections 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85dc158e20 0x7f85dc169600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85c0060670 0x7f85c0062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85dc1591f0 0x7f85dc1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 --2- 172.21.15.50:0/3188408445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85dc0ce810 0x7f85dc1522a0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 >> 172.21.15.50:0/3188408445 conn(0x7f85dc067bb0 msgr2=0x7f85dc0ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 shutdown_connections 2023-12-19T14:58:29.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:29.880+0000 7f85e2f23700 1 -- 172.21.15.50:0/3188408445 wait complete. 2023-12-19T14:58:29.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T14:58:30.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 Processor -- start 2023-12-19T14:58:30.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 -- start start 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc008140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080caac0 0x7fc0080c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0080cc000 0x7fc0080cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0080c8ab0 con 0x7fc0080cc000 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0080cbb70 con 0x7fc0080caac0 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.076+0000 7fc00d945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080caf60 con 0x7fc008140700 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080caac0 0x7fc0080c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0077fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0080cc000 0x7fc0080cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc0080caac0 0x7fc0080c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:53366/0 (socket says 172.21.15.50:53366) 2023-12-19T14:58:30.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 -- 172.21.15.50:0/2615552375 learned_addr learned my addr 172.21.15.50:0/2615552375 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc006ffd700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc008140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 -- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 msgr2=0x7fc008140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc008140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 -- 172.21.15.50:0/2615552375 >> v2:172.21.15.50:3300/0 conn(0x7fc0080cc000 msgr2=0x7fc0080cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 --2- 172.21.15.50:0/2615552375 >> v2:172.21.15.50:3300/0 conn(0x7fc0080cc000 0x7fc0080cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 -- 172.21.15.50:0/2615552375 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080cb240 con 0x7fc0080caac0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc0067fc700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c8570 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc00806cbe0 tx=0x7fbffc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7889001872a85477 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc005ffb700 1 -- 172.21.15.50:0/2615552375 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc009e70 con 0x7fc0080caac0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc005ffb700 1 -- 172.21.15.50:0/2615552375 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc00b860 con 0x7fc0080caac0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc005ffb700 1 -- 172.21.15.50:0/2615552375 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00bc00 con 0x7fc0080caac0 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 -- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 msgr2=0x7fc0080c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:30.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c8570 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc00806cbe0 tx=0x7fbffc009250 comp rx=0 tx=0).stop 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 -- 172.21.15.50:0/2615552375 shutdown_connections 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 --2- 172.21.15.50:0/2615552375 >> v2:172.21.15.50:3300/0 conn(0x7fc0080cc000 0x7fc0080cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c8570 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 --2- 172.21.15.50:0/2615552375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc008140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 -- 172.21.15.50:0/2615552375 >> 172.21.15.50:0/2615552375 conn(0x7fc0080cd9c0 msgr2=0x7fc0080c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 -- 172.21.15.50:0/2615552375 shutdown_connections 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.080+0000 7fc00d945700 1 -- 172.21.15.50:0/2615552375 wait complete. 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.084+0000 7fc00d945700 1 Processor -- start 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.084+0000 7fc00d945700 1 -- start start 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc0080b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080b0140 con 0x7fc0080cc000 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080c5540 con 0x7fc0080caac0 2023-12-19T14:58:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080c5820 con 0x7fc008140700 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55404/0 (socket says 172.21.15.50:55404) 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 -- 172.21.15.50:0/1590473891 learned_addr learned my addr 172.21.15.50:0/1590473891 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc006ffd700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0077fe700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc0080b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 -- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 msgr2=0x7fc0080b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc0080b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 -- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 msgr2=0x7fc0080c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 -- 172.21.15.50:0/1590473891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0094f0 con 0x7fc0080cc000 2023-12-19T14:58:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc0067fc700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc0080cbcb0 tx=0x7fbffc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00bbe0 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc000c90 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc011650 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.088+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0080c5b00 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.092+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0080c60c0 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.092+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc008071120 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.092+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbffc011a80 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.096+0000 7fbfeeffd700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 0x7fbff4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.096+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fbffc07df50 con 0x7fc0080cc000 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.096+0000 7fc006ffd700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 0x7fbff4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.096+0000 7fc006ffd700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 0x7fbff4062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006fd0 tx=0x7fbff8008040 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.100+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbffc04e2b0 con 0x7fc0080cc000 2023-12-19T14:58:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.312+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc0080cc660 con 0x7fc0080cc000 2023-12-19T14:58:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.312+0000 7fbfeeffd700 1 -- 172.21.15.50:0/1590473891 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbffc0518d0 con 0x7fc0080cc000 2023-12-19T14:58:30.312 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.312+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 msgr2=0x7fbff4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 0x7fbff4062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006fd0 tx=0x7fbff8008040 comp rx=0 tx=0).stop 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 msgr2=0x7fc00813eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc0080cbcb0 tx=0x7fbffc017000 comp rx=0 tx=0).stop 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 shutdown_connections 2023-12-19T14:58:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbff4060670 0x7fbff4062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc008140700 0x7fc0080b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0080cc000 0x7fc00813eb40 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 --2- 172.21.15.50:0/1590473891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0080caac0 0x7fc0080c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 >> 172.21.15.50:0/1590473891 conn(0x7fc0080cd9c0 msgr2=0x7fc0080686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 shutdown_connections 2023-12-19T14:58:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:30.316+0000 7fc00d945700 1 -- 172.21.15.50:0/1590473891 wait complete. 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T14:58:30.336 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T14:58:30.337 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:58:30.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.337 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.357 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 41240sec preferred_lft 41240sec 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T14:58:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.360 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T14:58:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T14:58:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T14:58:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T14:58:30.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.360 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T14:58:30.442 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T14:58:30.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.445 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T14:58:30.512 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T14:58:30.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T14:58:30.513 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T14:58:30.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.646 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T14:58:30.661 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T14:58:30.661 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T14:58:30.665 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:58:30.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T14:58:30.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T14:58:30.744 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T14:58:30.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:58:30.744 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:30.806 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T14:58:30.806 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T14:58:30.806 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:30.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:58:30.952 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:58:30.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:58:31.035 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:58:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:58:31.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:31.038 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:58:31.098 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T14:58:31.099 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T14:58:31.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:58:31.188 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:58:31.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:58:31.263 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:58:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:58:31.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:31.266 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:58:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T14:58:31.292+0000 7f65f2fcb380 -1 init, newargv = 0x55a5a5142e50 newargc=15 2023-12-19T14:58:31.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[54502]: starting ceph client 2023-12-19T14:58:31.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[54502]: starting fuse 2023-12-19T14:58:32.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T14:58:32.445 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T14:58:32.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T14:58:32.523 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T14:58:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T14:58:32.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:32.526 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T14:58:32.589 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T14:58:32.589 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T14:58:32.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T14:58:32.589 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T14:58:32.589 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T14:58:32.589 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T14:58:32.590 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T14:58:32.591 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T14:58:32.754 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.54502.asok 2023-12-19T14:58:32.934 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54502.asok 2023-12-19T14:58:32.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:32.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54502.asok status 2023-12-19T14:58:33.133 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T14:58:33.133 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T14:58:33.133 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T14:58:33.133 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "54502", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 8523, 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T14:58:33.134 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 8523 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2510479472 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2510479472 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.8523 192.168.0.1:0/2510479472", 2023-12-19T14:58:33.135 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/2510479472", 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 66, 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 144, 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T14:58:33.136 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T14:58:33.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:33.147 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T14:58:33.147 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:33.171 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T14:58:33.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:33.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:33.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T14:58:33.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T14:58:33.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.456+0000 7f40fae6b700 1 Processor -- start 2023-12-19T14:58:33.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 -- start start 2023-12-19T14:58:33.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 0x7f40f40ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f406a9d0 0x7f40f415af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f406c7c0 con 0x7f40f406a580 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f406cd80 con 0x7f40f406a9d0 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fae6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f406b8f0 con 0x7f40f40b1410 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f406a9d0 0x7f40f415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f40f406a9d0 0x7f40f415af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55420/0 (socket says 172.21.15.50:55420) 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 -- 172.21.15.50:0/1530462055 learned_addr learned my addr 172.21.15.50:0/1530462055 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:33.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40f9e69700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 0x7f40f40ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 -- 172.21.15.50:0/1530462055 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 msgr2=0x7f40f40ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:33.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 0x7f40f40ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 -- 172.21.15.50:0/1530462055 >> v1:172.21.15.125:6789/0 conn(0x7f40f406a580 legacy=0x7f40f415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 -- 172.21.15.50:0/1530462055 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40ac1e0 con 0x7f40f406a9d0 2023-12-19T14:58:33.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40f9e69700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 0x7f40f40ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:33.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.460+0000 7f40fa66a700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f406a9d0 0x7f40f415af70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0099b0 tx=0x7f40ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5862e9e4aa492c75 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40f8e67700 1 -- 172.21.15.50:0/1530462055 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00b860 con 0x7f40f406a9d0 2023-12-19T14:58:33.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40f8e67700 1 -- 172.21.15.50:0/1530462055 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec00b9c0 con 0x7f40f406a9d0 2023-12-19T14:58:33.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40f8e67700 1 -- 172.21.15.50:0/1530462055 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00bd60 con 0x7f40f406a9d0 2023-12-19T14:58:33.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 -- 172.21.15.50:0/1530462055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f406a9d0 msgr2=0x7f40f415af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f406a9d0 0x7f40f415af70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0099b0 tx=0x7f40ec009250 comp rx=0 tx=0).stop 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 -- 172.21.15.50:0/1530462055 shutdown_connections 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f406a9d0 0x7f40f415af70 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 --2- 172.21.15.50:0/1530462055 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f40b1410 0x7f40f40ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 -- 172.21.15.50:0/1530462055 >> 172.21.15.50:0/1530462055 conn(0x7f40f4067bb0 msgr2=0x7f40f40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 -- 172.21.15.50:0/1530462055 shutdown_connections 2023-12-19T14:58:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.464+0000 7f40fae6b700 1 -- 172.21.15.50:0/1530462055 wait complete. 2023-12-19T14:58:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 Processor -- start 2023-12-19T14:58:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 -- start start 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f406a580 0x7f40f4159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f406a9d0 0x7f40f415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f414e420 con 0x7f40f40b1410 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f415dd20 con 0x7f40f406a9d0 2023-12-19T14:58:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fae6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4159850 con 0x7f40f406a580 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40f9e69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f406a580 0x7f40f4159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55424/0 (socket says 172.21.15.50:55424) 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 -- 172.21.15.50:0/2320414715 learned_addr learned my addr 172.21.15.50:0/2320414715 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40f9668700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f406a9d0 0x7f40f415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 -- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f406a580 msgr2=0x7f40f4159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f406a580 0x7f40f4159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 -- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f406a9d0 msgr2=0x7f40f415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f406a9d0 0x7f40f415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.468+0000 7f40fa66a700 1 -- 172.21.15.50:0/2320414715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec0094f0 con 0x7f40f40b1410 2023-12-19T14:58:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40fa66a700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f40f4064040 tx=0x7f40ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00b860 con 0x7f40f40b1410 2023-12-19T14:58:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec01f070 con 0x7f40f40b1410 2023-12-19T14:58:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f4153f20 con 0x7f40f40b1410 2023-12-19T14:58:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec01a870 con 0x7f40f40b1410 2023-12-19T14:58:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f41594e0 con 0x7f40f40b1410 2023-12-19T14:58:33.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.472+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f404be40 con 0x7f40f40b1410 2023-12-19T14:58:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.476+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f40ec01aca0 con 0x7f40f40b1410 2023-12-19T14:58:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.476+0000 7f40e9ffb700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 0x7f40d8062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.476+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f40ec021070 con 0x7f40f40b1410 2023-12-19T14:58:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.476+0000 7f40f9e69700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 0x7f40d8062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.476+0000 7f40f9e69700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 0x7f40d8062bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f40f40ca860 tx=0x7f40f0009930 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.484+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f40ec04ed80 con 0x7f40f40b1410 2023-12-19T14:58:33.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.676+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f4157320 con 0x7f40d8060740 2023-12-19T14:58:33.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.680+0000 7f40e9ffb700 1 -- 172.21.15.50:0/2320414715 <== mgr.7202 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f40f4157320 con 0x7f40d8060740 2023-12-19T14:58:33.681 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T14:58:33.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T14:58:33.681 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T14:58:33.681 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T14:58:33.681 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 msgr2=0x7f40d8062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 0x7f40d8062bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f40f40ca860 tx=0x7f40f0009930 comp rx=0 tx=0).stop 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 msgr2=0x7f40f416b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f40f4064040 tx=0x7f40ec017000 comp rx=0 tx=0).stop 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 shutdown_connections 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40f406a9d0 0x7f40f415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f40d8060740 0x7f40d8062bf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f40f40b1410 0x7f40f416b290 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 --2- 172.21.15.50:0/2320414715 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f40f406a580 0x7f40f4159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 >> 172.21.15.50:0/2320414715 conn(0x7f40f4067bb0 msgr2=0x7f40f40caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 shutdown_connections 2023-12-19T14:58:33.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.684+0000 7f40fae6b700 1 -- 172.21.15.50:0/2320414715 wait complete. 2023-12-19T14:58:33.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T14:58:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.876+0000 7fe074c65700 1 Processor -- start 2023-12-19T14:58:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.876+0000 7fe074c65700 1 -- start start 2023-12-19T14:58:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe074c65700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0700cecd0 0x7fe07006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe074c65700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700cbdf0 con 0x7fe070158e20 2023-12-19T14:58:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe074c65700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700cc0d0 con 0x7fe0700cb970 2023-12-19T14:58:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe074c65700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0700cc3b0 con 0x7fe0700cecd0 2023-12-19T14:58:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0700cecd0 0x7fe07006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe0700cecd0 0x7fe07006b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55430/0 (socket says 172.21.15.50:55430) 2023-12-19T14:58:33.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 -- 172.21.15.50:0/4294865005 learned_addr learned my addr 172.21.15.50:0/4294865005 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06e59c700 1 --1- 172.21.15.50:0/4294865005 >> v1:172.21.15.50:6789/0 conn(0x7fe070158e20 0x7fe0700ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34516/0 (socket says 172.21.15.50:34516) 2023-12-19T14:58:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 -- 172.21.15.50:0/4294865005 >> v1:172.21.15.125:6789/0 conn(0x7fe0700cb970 legacy=0x7fe07006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 -- 172.21.15.50:0/4294865005 >> v1:172.21.15.50:6789/0 conn(0x7fe070158e20 legacy=0x7fe0700ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 -- 172.21.15.50:0/4294865005 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700cc690 con 0x7fe0700cecd0 2023-12-19T14:58:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06ed9d700 1 --2- 172.21.15.50:0/4294865005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0700cecd0 0x7fe07006b3d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe068009f50 tx=0x7fe068009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14e64a35e2b84d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06d59a700 1 -- 172.21.15.50:0/4294865005 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06800e070 con 0x7fe0700cecd0 2023-12-19T14:58:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.880+0000 7fe06d59a700 1 -- 172.21.15.50:0/4294865005 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06800ba70 con 0x7fe0700cecd0 2023-12-19T14:58:33.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe06d59a700 1 -- 172.21.15.50:0/4294865005 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0680092e0 con 0x7fe0700cecd0 2023-12-19T14:58:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 -- 172.21.15.50:0/4294865005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0700cecd0 msgr2=0x7fe07006b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 --2- 172.21.15.50:0/4294865005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0700cecd0 0x7fe07006b3d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe068009f50 tx=0x7fe068009840 comp rx=0 tx=0).stop 2023-12-19T14:58:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 -- 172.21.15.50:0/4294865005 shutdown_connections 2023-12-19T14:58:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 --2- 172.21.15.50:0/4294865005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0700cecd0 0x7fe07006b3d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 -- 172.21.15.50:0/4294865005 >> 172.21.15.50:0/4294865005 conn(0x7fe070067bb0 msgr2=0x7fe0700d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 -- 172.21.15.50:0/4294865005 shutdown_connections 2023-12-19T14:58:33.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 -- 172.21.15.50:0/4294865005 wait complete. 2023-12-19T14:58:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.884+0000 7fe074c65700 1 Processor -- start 2023-12-19T14:58:33.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- start start 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0700cecd0 0x7fe070153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 0x7fe07014f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070154f10 con 0x7fe070158e20 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0701551f0 con 0x7fe0701549a0 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0701554d0 con 0x7fe0700cecd0 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55436/0 (socket says 172.21.15.50:55436) 2023-12-19T14:58:33.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 -- 172.21.15.50:0/1235961566 learned_addr learned my addr 172.21.15.50:0/1235961566 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06ed9d700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 0x7fe07014f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06e59c700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0700cecd0 0x7fe070153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 -- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0700cecd0 msgr2=0x7fe070153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0700cecd0 0x7fe070153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 -- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 msgr2=0x7fe07014f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 0x7fe07014f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 -- 172.21.15.50:0/1235961566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068009ae0 con 0x7fe070158e20 2023-12-19T14:58:33.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06ed9d700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 0x7fe07014f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe06dd9b700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c590 tx=0x7fe064007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064011590 con 0x7fe070158e20 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe064011710 con 0x7fe070158e20 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064011ad0 con 0x7fe070158e20 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0701557b0 con 0x7fe070158e20 2023-12-19T14:58:33.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.888+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07014fbe0 con 0x7fe070158e20 2023-12-19T14:58:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.892+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070071120 con 0x7fe070158e20 2023-12-19T14:58:33.893 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.892+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 58) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe06401a360 con 0x7fe070158e20 2023-12-19T14:58:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.892+0000 7fe05e7fc700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 0x7fe050062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:33.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.892+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe06407ba70 con 0x7fe070158e20 2023-12-19T14:58:33.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.896+0000 7fe06e59c700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 0x7fe050062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:33.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.896+0000 7fe06e59c700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 0x7fe050062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe060007fd0 tx=0x7fe060007d70 comp rx=0 tx=0).ready entity=mgr.7202 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:33.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:33.900+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe06404bdd0 con 0x7fe070158e20 2023-12-19T14:58:34.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:34.112+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe07006bb00 con 0x7fe070158e20 2023-12-19T14:58:35.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.112+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 59) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe06404c1a0 con 0x7fe070158e20 2023-12-19T14:58:35.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.112+0000 7fe05e7fc700 1 -- 172.21.15.50:0/1235961566 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v59) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe0640777b0 con 0x7fe070158e20 2023-12-19T14:58:35.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 msgr2=0x7fe050062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 0x7fe050062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe060007fd0 tx=0x7fe060007d70 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 msgr2=0x7fe07014db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c590 tx=0x7fe064007f50 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 shutdown_connections 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0701549a0 0x7fe07014f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe050060670 0x7fe050062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe070158e20 0x7fe07014db40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 --2- 172.21.15.50:0/1235961566 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0700cecd0 0x7fe070153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 >> 172.21.15.50:0/1235961566 conn(0x7fe070067bb0 msgr2=0x7fe07006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 shutdown_connections 2023-12-19T14:58:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.116+0000 7fe074c65700 1 -- 172.21.15.50:0/1235961566 wait complete. 2023-12-19T14:58:35.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T14:58:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.300+0000 7fe3e09cc700 1 Processor -- start 2023-12-19T14:58:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.300+0000 7fe3e09cc700 1 -- start start 2023-12-19T14:58:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3e09cc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3dc158c90 0x7fe3dc0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3e09cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3e09cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc06c850 con 0x7fe3dc1595a0 2023-12-19T14:58:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3e09cc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc06b470 con 0x7fe3dc158c90 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3e09cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc06b750 con 0x7fe3dc06aeb0 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3db7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3dc158c90 0x7fe3dc0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3d3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3dbfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe3dc1595a0 0x7fe3dc15c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:34556/0 (socket says 172.21.15.50:34556) 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3dbfff700 1 -- 172.21.15.50:0/1909726888 learned_addr learned my addr 172.21.15.50:0/1909726888 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3db7fe700 1 -- 172.21.15.50:0/1909726888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 msgr2=0x7fe3dc159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3db7fe700 1 --2- 172.21.15.50:0/1909726888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3db7fe700 1 -- 172.21.15.50:0/1909726888 >> v1:172.21.15.125:6789/0 conn(0x7fe3dc1595a0 legacy=0x7fe3dc15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3db7fe700 1 -- 172.21.15.50:0/1909726888 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc06ba30 con 0x7fe3dc158c90 2023-12-19T14:58:35.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.304+0000 7fe3daffd700 1 -- 172.21.15.50:0/1909726888 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611336820 0 0) 0x7fe3dc06c850 con 0x7fe3dc1595a0 2023-12-19T14:58:35.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3db7fe700 1 --2- 172.21.15.50:0/1909726888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc0caf40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0098e0 tx=0x7fe3cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a60828d82ed2e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:35.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3daffd700 1 -- 172.21.15.50:0/1909726888 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc00b640 con 0x7fe3dc158c90 2023-12-19T14:58:35.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3daffd700 1 -- 172.21.15.50:0/1909726888 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc00b7a0 con 0x7fe3dc158c90 2023-12-19T14:58:35.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3daffd700 1 -- 172.21.15.50:0/1909726888 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc00bb40 con 0x7fe3dc158c90 2023-12-19T14:58:35.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 -- 172.21.15.50:0/1909726888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 msgr2=0x7fe3dc0caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/1909726888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc0caf40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0098e0 tx=0x7fe3cc0091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 -- 172.21.15.50:0/1909726888 shutdown_connections 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/1909726888 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/1909726888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc0caf40 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 -- 172.21.15.50:0/1909726888 >> 172.21.15.50:0/1909726888 conn(0x7fe3dc067bb0 msgr2=0x7fe3dc0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 -- 172.21.15.50:0/1909726888 shutdown_connections 2023-12-19T14:58:35.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.308+0000 7fe3e09cc700 1 -- 172.21.15.50:0/1909726888 wait complete. 2023-12-19T14:58:35.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 Processor -- start 2023-12-19T14:58:35.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 -- start start 2023-12-19T14:58:35.313 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:35.306+0000 7f2b435b1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc1595a0 0x7fe3dc0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c97e0 con 0x7fe3dc158c90 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc156360 con 0x7fe3dc1595a0 2023-12-19T14:58:35.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3e09cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0c8bd0 con 0x7fe3dc06aeb0 2023-12-19T14:58:35.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3d3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3d3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55462/0 (socket says 172.21.15.50:55462) 2023-12-19T14:58:35.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.312+0000 7fe3d3fff700 1 -- 172.21.15.50:0/477859953 learned_addr learned my addr 172.21.15.50:0/477859953 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3dbfff700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc1595a0 0x7fe3dc0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3db7fe700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 msgr2=0x7fe3dc155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc1595a0 msgr2=0x7fe3dc0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc1595a0 0x7fe3dc0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 -- 172.21.15.50:0/477859953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc009420 con 0x7fe3dc158c90 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d3fff700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ec80 tx=0x7fe3c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400c850 con 0x7fe3dc158c90 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400c9b0 con 0x7fe3dc158c90 2023-12-19T14:58:35.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc0c8eb0 con 0x7fe3dc158c90 2023-12-19T14:58:35.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400cd50 con 0x7fe3dc158c90 2023-12-19T14:58:35.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.316+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc0c9470 con 0x7fe3dc158c90 2023-12-19T14:58:35.318 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:35.316+0000 7ff1f76fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T14:58:35.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc0cf810 con 0x7fe3dc158c90 2023-12-19T14:58:35.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 59) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe3c4013680 con 0x7fe3dc158c90 2023-12-19T14:58:35.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3d37fe700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 0x7fe3bc062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe3c407ccf0 con 0x7fe3dc158c90 2023-12-19T14:58:35.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3db7fe700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 msgr2=0x7fe3bc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:35.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.320+0000 7fe3db7fe700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 0x7fe3bc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:35.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.328+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe3dc0cf810 con 0x7fe3dc158c90 2023-12-19T14:58:35.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.508+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe3dc06ad10 con 0x7fe3dc158c90 2023-12-19T14:58:35.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.520+0000 7fe3d37fe700 1 -- 172.21.15.50:0/477859953 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe3dc06ad10 con 0x7fe3dc158c90 2023-12-19T14:58:35.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3db7fe700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 msgr2=0x7fe3bc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:35.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3db7fe700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 0x7fe3bc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 msgr2=0x7fe3bc062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 0x7fe3bc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 msgr2=0x7fe3dc16a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ec80 tx=0x7fe3c40091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 shutdown_connections 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3dc1595a0 0x7fe3dc0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe3bc0606c0 0x7fe3bc062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3dc158c90 0x7fe3dc16a130 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 --2- 172.21.15.50:0/477859953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe3dc06aeb0 0x7fe3dc155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 >> 172.21.15.50:0/477859953 conn(0x7fe3dc067bb0 msgr2=0x7fe3dc0cf100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 shutdown_connections 2023-12-19T14:58:35.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.524+0000 7fe3e09cc700 1 -- 172.21.15.50:0/477859953 wait complete. 2023-12-19T14:58:35.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T14:58:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.712+0000 7f5922585700 1 Processor -- start 2023-12-19T14:58:35.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.712+0000 7f5922585700 1 -- start start 2023-12-19T14:58:35.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f591c0c6d20 0x7f591c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c06c7c0 0x7f591c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f591c15aa30 0x7f591c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c06a7a0 con 0x7f591c15aa30 2023-12-19T14:58:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0c7630 con 0x7f591c0c6d20 2023-12-19T14:58:35.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5922585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c06d4b0 con 0x7f591c06c7c0 2023-12-19T14:58:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f591c15aa30 0x7f591c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f591c15aa30 0x7f591c15ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55464/0 (socket says 172.21.15.50:55464) 2023-12-19T14:58:35.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 -- 172.21.15.50:0/2555499434 learned_addr learned my addr 172.21.15.50:0/2555499434 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f591b7fe700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c06c7c0 0x7f591c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f591bfff700 1 --2- 172.21.15.50:0/2555499434 >> v2:172.21.15.125:3300/0 conn(0x7f591c0c6d20 0x7f591c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 -- 172.21.15.50:0/2555499434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c06c7c0 msgr2=0x7f591c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c06c7c0 0x7f591c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 -- 172.21.15.50:0/2555499434 >> v2:172.21.15.125:3300/0 conn(0x7f591c0c6d20 msgr2=0x7f591c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 --2- 172.21.15.50:0/2555499434 >> v2:172.21.15.125:3300/0 conn(0x7f591c0c6d20 0x7f591c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 -- 172.21.15.50:0/2555499434 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c06d790 con 0x7f591c15aa30 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f5920b22700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c15aa30 0x7f591c15ae50 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f591c068030 tx=0x7f5914009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de8134e06f83787 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f591affd700 1 -- 172.21.15.50:0/2555499434 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400e070 con 0x7f591c15aa30 2023-12-19T14:58:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f591affd700 1 -- 172.21.15.50:0/2555499434 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400ba50 con 0x7f591c15aa30 2023-12-19T14:58:35.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.716+0000 7f591affd700 1 -- 172.21.15.50:0/2555499434 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59140092e0 con 0x7f591c15aa30 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- 172.21.15.50:0/2555499434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c15aa30 msgr2=0x7f591c15ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c15aa30 0x7f591c15ae50 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f591c068030 tx=0x7f5914009840 comp rx=0 tx=0).stop 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- 172.21.15.50:0/2555499434 shutdown_connections 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c15aa30 0x7f591c15ae50 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 --2- 172.21.15.50:0/2555499434 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c06c7c0 0x7f591c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 --2- 172.21.15.50:0/2555499434 >> v2:172.21.15.125:3300/0 conn(0x7f591c0c6d20 0x7f591c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- 172.21.15.50:0/2555499434 >> 172.21.15.50:0/2555499434 conn(0x7f591c067bb0 msgr2=0x7f591c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- 172.21.15.50:0/2555499434 shutdown_connections 2023-12-19T14:58:35.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- 172.21.15.50:0/2555499434 wait complete. 2023-12-19T14:58:35.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 Processor -- start 2023-12-19T14:58:35.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.720+0000 7f5922585700 1 -- start start 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c0c6d20 0x7f591c141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f591c15aa30 0x7f591c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0cb020 con 0x7f591c06c7c0 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c152c20 con 0x7f591c15aa30 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0ca410 con 0x7f591c0c6d20 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55468/0 (socket says 172.21.15.50:55468) 2023-12-19T14:58:35.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 -- 172.21.15.50:0/326315458 learned_addr learned my addr 172.21.15.50:0/326315458 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:35.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591b7fe700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c0c6d20 0x7f591c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 -- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c0c6d20 msgr2=0x7f591c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c0c6d20 0x7f591c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5920b22700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f591c15aa30 0x7f591c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 -- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f591c15aa30 msgr2=0x7f591c0c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f591c15aa30 0x7f591c0c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 -- 172.21.15.50:0/326315458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009ae0 con 0x7f591c06c7c0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f591bfff700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f590c00d890 tx=0x7f590c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c00b600 con 0x7f591c06c7c0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c00b760 con 0x7f591c06c7c0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c0ca6f0 con 0x7f591c06c7c0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c00bae0 con 0x7f591c06c7c0 2023-12-19T14:58:35.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.724+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0cacb0 con 0x7f591c06c7c0 2023-12-19T14:58:35.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.728+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c071120 con 0x7f591c06c7c0 2023-12-19T14:58:35.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.728+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 59) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f590c01e460 con 0x7f591c06c7c0 2023-12-19T14:58:35.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.728+0000 7f5903fff700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 0x7f5908062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:35.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.732+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f590c07d170 con 0x7f591c06c7c0 2023-12-19T14:58:35.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.732+0000 7f591b7fe700 1 -- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 msgr2=0x7f5908062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:35.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.732+0000 7f591b7fe700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 0x7f5908062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:35.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.736+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f590c04d4d0 con 0x7f591c06c7c0 2023-12-19T14:58:35.858 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:35.850+0000 7f2b435b1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:58:35.908 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:35.908+0000 7ff1f76fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T14:58:35.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.912+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f591c06da70 con 0x7f591c06c7c0 2023-12-19T14:58:35.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.916+0000 7f5903fff700 1 -- 172.21.15.50:0/326315458 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f590c050af0 con 0x7f591c06c7c0 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 msgr2=0x7f5908062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 0x7f5908062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 msgr2=0x7f591c0cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f590c00d890 tx=0x7f590c008040 comp rx=0 tx=0).stop 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 shutdown_connections 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f591c15aa30 0x7f591c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f59080606f0 0x7f5908062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f591c0c6d20 0x7f591c141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 --2- 172.21.15.50:0/326315458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f591c06c7c0 0x7f591c0cb4b0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:35.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 >> 172.21.15.50:0/326315458 conn(0x7f591c067bb0 msgr2=0x7f591c159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:35.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.920+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 shutdown_connections 2023-12-19T14:58:35.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:35.924+0000 7f5922585700 1 -- 172.21.15.50:0/326315458 wait complete. 2023-12-19T14:58:35.934 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:35.926+0000 7f2b435b1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:58:35.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T14:58:35.945 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T14:58:35.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1 1M 2023-12-19T14:58:35.995 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:35.996+0000 7ff1f76fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T14:58:36.078 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.070+0000 7f2b435b1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:36.152 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:36.152+0000 7ff1f76fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.172+0000 7f52d1dcc700 1 Processor -- start 2023-12-19T14:58:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.172+0000 7f52d1dcc700 1 -- start start 2023-12-19T14:58:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52cc0aedc0 0x7f52cc0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52cc06b3d0 con 0x7f52cc06d6b0 2023-12-19T14:58:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc0cc140 con 0x7f52cc06af50 2023-12-19T14:58:36.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc0cb3f0 con 0x7f52cc0aedc0 2023-12-19T14:58:36.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cbfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:36.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cbfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55474/0 (socket says 172.21.15.50:55474) 2023-12-19T14:58:36.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cbfff700 1 -- 172.21.15.50:0/1225756788 learned_addr learned my addr 172.21.15.50:0/1225756788 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:36.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 --2- 172.21.15.50:0/1225756788 >> v2:172.21.15.125:3300/0 conn(0x7f52cc0aedc0 0x7f52cc0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 -- 172.21.15.50:0/1225756788 >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 msgr2=0x7f52cc0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 --2- 172.21.15.50:0/1225756788 >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 -- 172.21.15.50:0/1225756788 >> v1:172.21.15.125:6789/0 conn(0x7f52cc06d6b0 legacy=0x7f52cc0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 -- 172.21.15.50:0/1225756788 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc06d220 con 0x7f52cc0aedc0 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cbfff700 1 --2- 172.21.15.50:0/1225756788 >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cb7fe700 1 -- 172.21.15.50:0/1225756788 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264628008 0 0) 0x7f52c4001490 con 0x7f52cc06d6b0 2023-12-19T14:58:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d0dca700 1 --2- 172.21.15.50:0/1225756788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc0aedc0 0x7f52cc0af190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0cc6c0 tx=0x7f52c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a66ff9cf6b0a1d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:36.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cb7fe700 1 -- 172.21.15.50:0/1225756788 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0011630 con 0x7f52cc0aedc0 2023-12-19T14:58:36.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cb7fe700 1 -- 172.21.15.50:0/1225756788 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0011790 con 0x7f52cc0aedc0 2023-12-19T14:58:36.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52cb7fe700 1 -- 172.21.15.50:0/1225756788 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0011b30 con 0x7f52cc0aedc0 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 -- 172.21.15.50:0/1225756788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc0aedc0 msgr2=0x7f52cc0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/1225756788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc0aedc0 0x7f52cc0af190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0cc6c0 tx=0x7f52c00091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 -- 172.21.15.50:0/1225756788 shutdown_connections 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/1225756788 >> v2:172.21.15.50:3300/0 conn(0x7f52cc06af50 0x7f52cc0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.176+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/1225756788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc0aedc0 0x7f52cc0af190 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- 172.21.15.50:0/1225756788 >> 172.21.15.50:0/1225756788 conn(0x7f52cc0cc760 msgr2=0x7f52cc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- 172.21.15.50:0/1225756788 shutdown_connections 2023-12-19T14:58:36.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- 172.21.15.50:0/1225756788 wait complete. 2023-12-19T14:58:36.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 Processor -- start 2023-12-19T14:58:36.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- start start 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc06d6b0 0x7f52cc0c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc0aedc0 0x7f52cc14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0c7da0 con 0x7f52cc0aedc0 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0b2b40 con 0x7f52cc06d6b0 2023-12-19T14:58:36.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d1dcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0b2de0 con 0x7f52cc06af50 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d15cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc0aedc0 0x7f52cc14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d0dca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d0dca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:36820/0 (socket says 172.21.15.50:36820) 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.180+0000 7f52d0dca700 1 -- 172.21.15.50:0/468749286 learned_addr learned my addr 172.21.15.50:0/468749286 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc06d6b0 0x7f52cc0c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:36.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc06d6b0 msgr2=0x7f52cc0c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc06d6b0 0x7f52cc0c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc0aedc0 msgr2=0x7f52cc14d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc0aedc0 0x7f52cc14d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80094f0 con 0x7f52cc06af50 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d0dca700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0cb530 tx=0x7f52c0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:36.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0011b10 con 0x7f52cc06af50 2023-12-19T14:58:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00233f0 con 0x7f52cc06af50 2023-12-19T14:58:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c0009420 con 0x7f52cc06af50 2023-12-19T14:58:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0023790 con 0x7f52cc06af50 2023-12-19T14:58:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc0b3420 con 0x7f52cc06af50 2023-12-19T14:58:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.184+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc04be40 con 0x7f52cc06af50 2023-12-19T14:58:36.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.188+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 59) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f52c0023b80 con 0x7f52cc06af50 2023-12-19T14:58:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.188+0000 7f52c8ff9700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:36.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.188+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f52c007de10 con 0x7f52cc06af50 2023-12-19T14:58:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.188+0000 7f52cbfff700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:36.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.188+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T14:58:36.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.196+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f52c0081030 con 0x7f52cc06af50 2023-12-19T14:58:36.354 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.346+0000 7f2b435b1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:58:36.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.392+0000 7f52cbfff700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:36.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.392+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T14:58:36.429 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.422+0000 7f2b435b1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:36.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.512+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f52cc06cea0 con 0x7f52b00606c0 2023-12-19T14:58:36.528 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:36.528+0000 7ff1f76fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T14:58:36.604 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:36.604+0000 7ff1f76fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:36.655 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.650+0000 7f2b435b1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:58:36.744 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.738+0000 7f2b435b1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:58:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.796+0000 7f52cbfff700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:36.796+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T14:58:36.830 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:36.822+0000 7f2b435b1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:58:36.839 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:36.840+0000 7ff1f76fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T14:58:36.932 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:36.932+0000 7ff1f76fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T14:58:37.021 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:37.020+0000 7ff1f76fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T14:58:37.240 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:37.234+0000 7f2b435b1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:58:37.409 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:37.402+0000 7f2b435b1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:58:37.436 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:37.436+0000 7ff1f76fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T14:58:37.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:37.600+0000 7f52cbfff700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:37.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:37.600+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T14:58:37.613 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:37.612+0000 7ff1f76fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T14:58:37.779 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:37.774+0000 7f2b435b1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:58:37.995 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:37.996+0000 7ff1f76fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T14:58:37.996 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:37.990+0000 7f2b435b1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:38.087 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.082+0000 7f2b435b1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:58:38.192 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.186+0000 7f2b435b1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:58:38.225 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.224+0000 7ff1f76fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T14:58:38.268 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.262+0000 7f2b435b1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:58:38.317 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.316+0000 7ff1f76fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T14:58:38.393 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.392+0000 7ff1f76fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T14:58:38.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:58:38.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:58:38.472 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.472+0000 7ff1f76fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T14:58:38.487 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.482+0000 7f2b435b1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:38.565 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.558+0000 7f2b435b1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:58:38.649 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.642+0000 7f2b435b1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:58:38.711 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.712+0000 7ff1f76fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T14:58:38.797 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.796+0000 7ff1f76fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T14:58:38.838 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.830+0000 7f2b435b1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:58:38.893 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:38.892+0000 7ff1f76fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T14:58:38.927 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:38.922+0000 7f2b435b1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:58:39.108 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:39.108+0000 7ff1f76fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T14:58:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:39.204+0000 7f52cbfff700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T14:58:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:39.204+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T14:58:39.203 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:39.204+0000 7ff1f76fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T14:58:39.669 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.662+0000 7f2b435b1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:58:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:39.684+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 60) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f52c004e520 con 0x7f52cc06af50 2023-12-19T14:58:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:39.684+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T14:58:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:39.684+0000 7f52c8ff9700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:39.861 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b06c8b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.861 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b06c8b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.861 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b06c8b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.861 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b06c8b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.861 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b06c8b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b03484700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b03484700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b03484700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b03484700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:39.862 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:58:39.854+0000 7f2b03484700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:58:40.021 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T14:58:40.020+0000 7ff1f76fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T14:58:40.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.692+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mgrmap(e 61) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f52c0058960 con 0x7f52cc06af50 2023-12-19T14:58:40.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.692+0000 7f52c8ff9700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:40.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.692+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f52cc06cea0 con 0x7f52b00606c0 2023-12-19T14:58:40.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.696+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:40.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.704+0000 7f52cbfff700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f52cc15bdf0 tx=0x7f52b8009250 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:40.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.808+0000 7f52c8ff9700 1 -- 172.21.15.50:0/468749286 <== mgr.8553 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f52cc06cea0 con 0x7f52b00606c0 2023-12-19T14:58:40.810 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1 2023-12-19T14:58:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 msgr2=0x7f52b0062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f52cc15bdf0 tx=0x7f52b8009250 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 msgr2=0x7f52cc0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0cb530 tx=0x7f52c0017000 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 shutdown_connections 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52cc06d6b0 0x7f52cc0c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f52b00606c0 0x7f52b0062b70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52cc0aedc0 0x7f52cc14d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.812+0000 7f52d1dcc700 1 --2- 172.21.15.50:0/468749286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52cc06af50 0x7f52cc0b4d00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.816+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 >> 172.21.15.50:0/468749286 conn(0x7f52cc0cc760 msgr2=0x7f52cc15af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.816+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 shutdown_connections 2023-12-19T14:58:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:40.816+0000 7f52d1dcc700 1 -- 172.21.15.50:0/468749286 wait complete. 2023-12-19T14:58:40.840 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-12-19T14:58:40.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs /snap_test_dir1 55n 2023-12-19T14:58:41.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.016+0000 7f6f9eee3700 1 Processor -- start 2023-12-19T14:58:41.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.016+0000 7f6f9eee3700 1 -- start start 2023-12-19T14:58:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9eee3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9eee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9eee3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980cbbb0 con 0x7f6f980cc040 2023-12-19T14:58:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9eee3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9815d960 con 0x7f6f980ce2d0 2023-12-19T14:58:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9eee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9806dc90 con 0x7f6f980caa10 2023-12-19T14:58:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9dee1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9dee1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60516/0 (socket says 172.21.15.50:60516) 2023-12-19T14:58:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:50672/0 (socket says 172.21.15.50:50672) 2023-12-19T14:58:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9dee1700 1 -- 172.21.15.50:0/2640396529 learned_addr learned my addr 172.21.15.50:0/2640396529 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:41.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9e6e2700 1 --1- 172.21.15.50:0/2640396529 >> v1:172.21.15.125:6789/0 conn(0x7f6f980cc040 0x7f6f98158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39098/0 (socket says 172.21.15.50:39098) 2023-12-19T14:58:41.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/2640396529 >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 msgr2=0x7f6f980ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 --2- 172.21.15.50:0/2640396529 >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/2640396529 >> v1:172.21.15.125:6789/0 conn(0x7f6f980cc040 legacy=0x7f6f98158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/2640396529 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980cafa0 con 0x7f6f980caa10 2023-12-19T14:58:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9dee1700 1 --2- 172.21.15.50:0/2640396529 >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.020+0000 7f6f9d6e0700 1 --2- 172.21.15.50:0/2640396529 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6f880098e0 tx=0x7f6f880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258f13861259e08f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9cedf700 1 -- 172.21.15.50:0/2640396529 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800b640 con 0x7f6f980caa10 2023-12-19T14:58:41.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9cedf700 1 -- 172.21.15.50:0/2640396529 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8800b7a0 con 0x7f6f980caa10 2023-12-19T14:58:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9cedf700 1 -- 172.21.15.50:0/2640396529 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800bb40 con 0x7f6f980caa10 2023-12-19T14:58:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 -- 172.21.15.50:0/2640396529 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 msgr2=0x7f6f9815d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/2640396529 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6f880098e0 tx=0x7f6f880091a0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 -- 172.21.15.50:0/2640396529 shutdown_connections 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/2640396529 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980caa10 0x7f6f9815d420 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/2640396529 >> v2:172.21.15.125:3300/0 conn(0x7f6f980ce2d0 0x7f6f980ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 -- 172.21.15.50:0/2640396529 >> 172.21.15.50:0/2640396529 conn(0x7f6f98067bb0 msgr2=0x7f6f980cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 -- 172.21.15.50:0/2640396529 shutdown_connections 2023-12-19T14:58:41.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.024+0000 7f6f9eee3700 1 -- 172.21.15.50:0/2640396529 wait complete. 2023-12-19T14:58:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 Processor -- start 2023-12-19T14:58:41.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 -- start start 2023-12-19T14:58:41.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f980caa10 0x7f6f9814c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 0x7f6f9815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f98159f70 con 0x7f6f980cc040 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f98159360 con 0x7f6f980caa10 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f98159640 con 0x7f6f980ce2d0 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52638/0 (socket says 172.21.15.50:52638) 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/67761650 learned_addr learned my addr 172.21.15.50:0/67761650 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:41.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9dee1700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f980caa10 0x7f6f9814c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9e6e2700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 0x7f6f9815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 msgr2=0x7f6f9815c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 0x7f6f9815c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f980caa10 msgr2=0x7f6f9814c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f980caa10 0x7f6f9814c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 -- 172.21.15.50:0/67761650 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f940094f0 con 0x7f6f980cc040 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9e6e2700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 0x7f6f9815c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9d6e0700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6f9806ddd0 tx=0x7f6f88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8801bcf0 con 0x7f6f980cc040 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8801be90 con 0x7f6f980cc040 2023-12-19T14:58:41.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f88009420 con 0x7f6f980cc040 2023-12-19T14:58:41.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.028+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88011670 con 0x7f6f980cc040 2023-12-19T14:58:41.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.032+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f98159c00 con 0x7f6f980cc040 2023-12-19T14:58:41.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.032+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9804be40 con 0x7f6f980cc040 2023-12-19T14:58:41.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.032+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 61) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f6f880119e0 con 0x7f6f980cc040 2023-12-19T14:58:41.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.036+0000 7f6f8dffb700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 0x7f6f7c062920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.036+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6f8807de90 con 0x7f6f980cc040 2023-12-19T14:58:41.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.036+0000 7f6f9dee1700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 0x7f6f7c062920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.036+0000 7f6f9dee1700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 0x7f6f7c062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006d20 tx=0x7f6f94009250 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.040+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f8804e390 con 0x7f6f980cc040 2023-12-19T14:58:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.232+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f980c5210 con 0x7f6f7c060470 2023-12-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.340+0000 7f6f8dffb700 1 -- 172.21.15.50:0/67761650 <== mgr.8553 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f6f980c5210 con 0x7f6f7c060470 2023-12-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:Retention added to path /snap_test_dir1 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 msgr2=0x7f6f7c062920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 0x7f6f7c062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006d20 tx=0x7f6f94009250 comp rx=0 tx=0).stop 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 msgr2=0x7f6f98158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6f9806ddd0 tx=0x7f6f88017000 comp rx=0 tx=0).stop 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 shutdown_connections 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f7c060470 0x7f6f7c062920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f980ce2d0 0x7f6f9815c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f980cc040 0x7f6f98158980 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 --2- 172.21.15.50:0/67761650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f980caa10 0x7f6f9814c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 >> 172.21.15.50:0/67761650 conn(0x7f6f98067bb0 msgr2=0x7f6f9815e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 shutdown_connections 2023-12-19T14:58:41.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.344+0000 7f6f9eee3700 1 -- 172.21.15.50:0/67761650 wait complete. 2023-12-19T14:58:41.371 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-12-19T14:58:41.372 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-12-19T14:58:41.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json /snap_test_dir1 2023-12-19T14:58:41.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.552+0000 7f8c90945700 1 Processor -- start 2023-12-19T14:58:41.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.552+0000 7f8c90945700 1 -- start start 2023-12-19T14:58:41.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c90945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c90945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 0x7f8c8c0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c90945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c0ac8e0 con 0x7f8c8c0ac480 2023-12-19T14:58:41.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c90945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c06e180 con 0x7f8c8c156e70 2023-12-19T14:58:41.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c90945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c06d7b0 con 0x7f8c8c06b060 2023-12-19T14:58:41.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c8a59c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 0x7f8c8c0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c8ad9d700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8c8c0ac480 0x7f8c8c0aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39104/0 (socket says 172.21.15.50:39104) 2023-12-19T14:58:41.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c8ad9d700 1 -- 172.21.15.50:0/454658100 learned_addr learned my addr 172.21.15.50:0/454658100 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T14:58:41.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/454658100 >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:60542/0 (socket says 172.21.15.50:60542) 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c89d9b700 1 -- 172.21.15.50:0/454658100 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699378400 0 0) 0x7f8c84001490 con 0x7f8c8c0ac480 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 -- 172.21.15.50:0/454658100 >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 msgr2=0x7f8c8c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 --2- 172.21.15.50:0/454658100 >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 -- 172.21.15.50:0/454658100 >> v1:172.21.15.125:6789/0 conn(0x7f8c8c0ac480 legacy=0x7f8c8c0aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 -- 172.21.15.50:0/454658100 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c0ae570 con 0x7f8c8c06b060 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/454658100 >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T14:58:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c83fff700 1 --2- 172.21.15.50:0/454658100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 0x7f8c8c0abf40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400dd10 tx=0x7f8c7400d630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47dc419dfb4021fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c89d9b700 1 -- 172.21.15.50:0/454658100 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c74012070 con 0x7f8c8c06b060 2023-12-19T14:58:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c89d9b700 1 -- 172.21.15.50:0/454658100 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7400de70 con 0x7f8c8c06b060 2023-12-19T14:58:41.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.556+0000 7f8c89d9b700 1 -- 172.21.15.50:0/454658100 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7400fb60 con 0x7f8c8c06b060 2023-12-19T14:58:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- 172.21.15.50:0/454658100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 msgr2=0x7f8c8c0abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 --2- 172.21.15.50:0/454658100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 0x7f8c8c0abf40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400dd10 tx=0x7f8c7400d630 comp rx=0 tx=0).stop 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- 172.21.15.50:0/454658100 shutdown_connections 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 --2- 172.21.15.50:0/454658100 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c06b060 0x7f8c8c0abf40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 --2- 172.21.15.50:0/454658100 >> v2:172.21.15.125:3300/0 conn(0x7f8c8c156e70 0x7f8c8c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- 172.21.15.50:0/454658100 >> 172.21.15.50:0/454658100 conn(0x7f8c8c067bb0 msgr2=0x7f8c8c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- 172.21.15.50:0/454658100 shutdown_connections 2023-12-19T14:58:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- 172.21.15.50:0/454658100 wait complete. 2023-12-19T14:58:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 Processor -- start 2023-12-19T14:58:41.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.560+0000 7f8c90945700 1 -- start start 2023-12-19T14:58:41.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 0x7f8c8c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c0ac480 0x7f8c8c166830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c13f240 con 0x7f8c8c156e70 2023-12-19T14:58:41.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1519a0 con 0x7f8c8c06b060 2023-12-19T14:58:41.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1672e0 con 0x7f8c8c0ac480 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52642/0 (socket says 172.21.15.50:52642) 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 -- 172.21.15.50:0/797357551 learned_addr learned my addr 172.21.15.50:0/797357551 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 0x7f8c8c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c83fff700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c0ac480 0x7f8c8c166830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 -- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c0ac480 msgr2=0x7f8c8c166830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c0ac480 0x7f8c8c166830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 -- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 msgr2=0x7f8c8c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 0x7f8c8c151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 -- 172.21.15.50:0/797357551 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c009420 con 0x7f8c8c156e70 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 0x7f8c8c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T14:58:41.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c8ad9d700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400cc90 tx=0x7f8c84001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400a960 con 0x7f8c8c156e70 2023-12-19T14:58:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8400aae0 con 0x7f8c8c156e70 2023-12-19T14:58:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7400d8d0 con 0x7f8c8c156e70 2023-12-19T14:58:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400aea0 con 0x7f8c8c156e70 2023-12-19T14:58:41.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.564+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c167670 con 0x7f8c8c156e70 2023-12-19T14:58:41.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.568+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c071120 con 0x7f8c8c156e70 2023-12-19T14:58:41.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.568+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 61) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f8c84007300 con 0x7f8c8c156e70 2023-12-19T14:58:41.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.572+0000 7f8c827fc700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 0x7f8c6c0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T14:58:41.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.572+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f8c8407ca20 con 0x7f8c8c156e70 2023-12-19T14:58:41.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.572+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 0x7f8c6c0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T14:58:41.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.572+0000 7f8c8a59c700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 0x7f8c6c0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0093f0 tx=0x7f8c7c008040 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T14:58:41.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.576+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c8404cfa0 con 0x7f8c8c156e70 2023-12-19T14:58:41.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.748+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 62) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f8c8404d370 con 0x7f8c8c156e70 2023-12-19T14:58:41.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.772+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c8c0ac8e0 con 0x7f8c6c0603f0 2023-12-19T14:58:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.788+0000 7f8c827fc700 1 -- 172.21.15.50:0/797357551 <== mgr.8553 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f8c8c0ac8e0 con 0x7f8c6c0603f0 2023-12-19T14:58:41.787 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T14:58:41.787 INFO:teuthology.orchestra.run.smithi050.stdout:{"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-12-19T14:58:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 msgr2=0x7f8c6c0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 0x7f8c6c0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0093f0 tx=0x7f8c7c008040 comp rx=0 tx=0).stop 2023-12-19T14:58:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 msgr2=0x7f8c8c0ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400cc90 tx=0x7f8c84001a50 comp rx=0 tx=0).stop 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 shutdown_connections 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8c6c0603f0 0x7f8c6c0628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8c8c156e70 0x7f8c8c0ca7e0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8c8c0ac480 0x7f8c8c166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 --2- 172.21.15.50:0/797357551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c8c06b060 0x7f8c8c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 >> 172.21.15.50:0/797357551 conn(0x7f8c8c067bb0 msgr2=0x7f8c8c0aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 shutdown_connections 2023-12-19T14:58:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T14:58:41.792+0000 7f8c90945700 1 -- 172.21.15.50:0/797357551 wait complete. 2023-12-19T14:58:41.815 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-12-19T14:58:41.815 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1M'], 'retention': [{'n': 55}]} 2023-12-19T14:59:00.362 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:59:00.353+0000 7f2af5c69700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:59:00.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:59:00.353+0000 7f2af5c69700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:59:00.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:59:00.353+0000 7f2af5c69700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:59:00.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:59:00.353+0000 7f2af5c69700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:59:00.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T14:59:00.353+0000 7f2af5c69700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T14:59:08.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:59:08.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:59:38.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T14:59:38.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:00:08.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:00:08.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:00:38.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:00:38.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:01:08.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:01:08.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:01:39.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:01:39.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:02:09.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:02:09.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:02:39.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:02:39.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:03:09.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:03:09.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:03:39.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:03:39.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:04:09.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:04:09.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:04:39.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:04:39.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:05:09.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:05:09.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:05:39.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:05:39.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:06:10.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:06:10.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:06:40.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:06:40.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:07:10.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:07:10.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:07:40.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:07:40.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:08:10.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:08:10.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:08:40.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:08:40.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:09:10.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:09:10.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:09:40.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:09:40.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:10:10.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:10:11.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:10:41.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:10:41.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:11:11.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:11:11.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:11:41.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:11:41.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:12:11.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:12:11.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:12:41.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:12:41.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:13:11.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:13:11.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:13:41.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:13:41.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:14:12.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:14:12.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:14:42.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:14:42.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:15:12.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:15:12.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:15:42.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:15:42.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:16:12.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:16:12.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:16:42.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:16:42.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:17:13.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:17:13.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:17:43.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:17:43.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:18:13.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:18:13.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:18:43.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:18:43.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:19:13.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:19:13.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:19:43.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:19:43.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:20:13.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:20:13.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:20:44.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:20:44.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:21:14.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:21:14.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:21:44.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:21:44.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:22:14.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:22:14.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:22:44.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:22:44.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:23:15.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:23:15.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:23:45.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:23:45.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:24:15.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:24:15.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:24:45.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:24:45.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:25:15.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:25:15.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:25:45.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:25:45.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:26:15.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:26:15.879 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:26:46.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:26:46.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:27:16.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:27:16.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:27:46.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:27:46.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:28:16.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:28:16.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:28:46.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:28:46.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:29:16.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:29:16.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:29:46.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:29:46.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:30:17.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:30:17.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:30:47.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:30:47.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:31:17.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:31:17.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:31:47.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:31:47.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:32:17.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:32:17.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:32:47.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:32:47.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:33:18.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:33:18.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:33:48.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:33:48.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:34:18.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:34:18.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:34:48.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:34:48.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:35:18.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:35:18.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:35:48.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:35:48.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:36:18.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:36:18.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:36:49.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:36:49.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:37:19.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:37:19.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:37:49.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:37:49.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:38:19.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:38:19.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:38:49.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:38:49.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:39:19.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:39:19.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:39:49.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:39:49.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:40:20.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:40:20.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:40:50.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:40:50.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:41:20.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:41:20.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:41:50.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:41:50.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:42:20.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:42:20.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:42:50.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:42:50.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:43:20.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:43:20.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:43:50.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:43:50.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:44:21.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:44:21.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:44:51.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:44:51.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:45:21.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:45:21.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:45:51.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:45:51.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:46:21.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:46:21.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:46:51.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:46:51.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:47:22.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:47:22.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:47:52.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:47:52.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:48:22.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:48:22.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:48:52.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:48:52.511 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:49:22.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:49:22.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:49:52.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:49:52.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:50:23.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:50:23.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:50:53.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:50:53.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:51:23.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:51:23.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:51:53.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:51:53.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:52:23.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:52:23.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:52:53.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:52:53.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:53:23.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:53:23.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:53:54.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:53:54.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:54:15.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:15.817 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T15:54:15.867 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_00_00_UTC 2023-12-19T15:54:15.867 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_01_00_UTC 2023-12-19T15:54:15.867 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_02_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_03_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_04_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_05_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_06_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_07_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_08_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_09_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_10_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_11_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_12_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_13_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_14_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_15_00_UTC 2023-12-19T15:54:15.868 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_16_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_17_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_18_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_19_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_20_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_21_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_22_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_23_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_24_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_25_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_26_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_27_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_28_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_29_00_UTC 2023-12-19T15:54:15.869 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_30_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_31_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_32_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_33_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_34_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_35_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_36_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_37_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_38_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_39_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_40_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_41_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_42_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_43_00_UTC 2023-12-19T15:54:15.870 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_44_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_45_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_46_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_47_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_48_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_49_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_50_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_51_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_52_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_53_00_UTC 2023-12-19T15:54:15.871 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_54_00_UTC 2023-12-19T15:54:15.872 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-15_00_00_UTC', 'scheduled-2023-12-19-15_01_00_UTC', 'scheduled-2023-12-19-15_02_00_UTC', 'scheduled-2023-12-19-15_03_00_UTC', 'scheduled-2023-12-19-15_04_00_UTC', 'scheduled-2023-12-19-15_05_00_UTC', 'scheduled-2023-12-19-15_06_00_UTC', 'scheduled-2023-12-19-15_07_00_UTC', 'scheduled-2023-12-19-15_08_00_UTC', 'scheduled-2023-12-19-15_09_00_UTC', 'scheduled-2023-12-19-15_10_00_UTC', 'scheduled-2023-12-19-15_11_00_UTC', 'scheduled-2023-12-19-15_12_00_UTC', 'scheduled-2023-12-19-15_13_00_UTC', 'scheduled-2023-12-19-15_14_00_UTC', 'scheduled-2023-12-19-15_15_00_UTC', 'scheduled-2023-12-19-15_16_00_UTC', 'scheduled-2023-12-19-15_17_00_UTC', 'scheduled-2023-12-19-15_18_00_UTC', 'scheduled-2023-12-19-15_19_00_UTC', 'scheduled-2023-12-19-15_20_00_UTC', 'scheduled-2023-12-19-15_21_00_UTC', 'scheduled-2023-12-19-15_22_00_UTC', 'scheduled-2023-12-19-15_23_00_UTC', 'scheduled-2023-12-19-15_24_00_UTC', 'scheduled-2023-12-19-15_25_00_UTC', 'scheduled-2023-12-19-15_26_00_UTC', 'scheduled-2023-12-19-15_27_00_UTC', 'scheduled-2023-12-19-15_28_00_UTC', 'scheduled-2023-12-19-15_29_00_UTC', 'scheduled-2023-12-19-15_30_00_UTC', 'scheduled-2023-12-19-15_31_00_UTC', 'scheduled-2023-12-19-15_32_00_UTC', 'scheduled-2023-12-19-15_33_00_UTC', 'scheduled-2023-12-19-15_34_00_UTC', 'scheduled-2023-12-19-15_35_00_UTC', 'scheduled-2023-12-19-15_36_00_UTC', 'scheduled-2023-12-19-15_37_00_UTC', 'scheduled-2023-12-19-15_38_00_UTC', 'scheduled-2023-12-19-15_39_00_UTC', 'scheduled-2023-12-19-15_40_00_UTC', 'scheduled-2023-12-19-15_41_00_UTC', 'scheduled-2023-12-19-15_42_00_UTC', 'scheduled-2023-12-19-15_43_00_UTC', 'scheduled-2023-12-19-15_44_00_UTC', 'scheduled-2023-12-19-15_45_00_UTC', 'scheduled-2023-12-19-15_46_00_UTC', 'scheduled-2023-12-19-15_47_00_UTC', 'scheduled-2023-12-19-15_48_00_UTC', 'scheduled-2023-12-19-15_49_00_UTC', 'scheduled-2023-12-19-15_50_00_UTC', 'scheduled-2023-12-19-15_51_00_UTC', 'scheduled-2023-12-19-15_52_00_UTC', 'scheduled-2023-12-19-15_53_00_UTC', 'scheduled-2023-12-19-15_54_00_UTC'] 2023-12-19T15:54:15.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1 2023-12-19T15:54:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.089+0000 7f1ddcb63700 1 Processor -- start 2023-12-19T15:54:16.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.089+0000 7f1ddcb63700 1 -- start start 2023-12-19T15:54:16.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1ddcb63700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 0x7f1dd81500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1ddcb63700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1dd8156030 0x7f1dd8159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1ddcb63700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd8156fd0 con 0x7f1dd814fcf0 2023-12-19T15:54:16.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1ddcb63700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd8156490 con 0x7f1dd8156030 2023-12-19T15:54:16.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1ddcb63700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd8156730 con 0x7f1dd80c2f60 2023-12-19T15:54:16.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1dd8156030 0x7f1dd8159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1dd8156030 0x7f1dd8159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55266/0 (socket says 172.21.15.50:55266) 2023-12-19T15:54:16.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 -- 172.21.15.50:0/152393898 learned_addr learned my addr 172.21.15.50:0/152393898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dcffff700 1 --2- 172.21.15.50:0/152393898 >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 0x7f1dd81500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 -- 172.21.15.50:0/152393898 >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 msgr2=0x7f1dd81500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 --2- 172.21.15.50:0/152393898 >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 0x7f1dd81500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 -- 172.21.15.50:0/152393898 >> v1:172.21.15.50:6789/0 conn(0x7f1dd814fcf0 legacy=0x7f1dd80ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 -- 172.21.15.50:0/152393898 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8156a10 con 0x7f1dd8156030 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dcffff700 1 --2- 172.21.15.50:0/152393898 >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 0x7f1dd81500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:16.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd7fff700 1 --2- 172.21.15.50:0/152393898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8159c80 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80c7fa0 tx=0x7f1dd0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1233715e15e2337e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd6ffd700 1 -- 172.21.15.50:0/152393898 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000e070 con 0x7f1dd8156030 2023-12-19T15:54:16.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd6ffd700 1 -- 172.21.15.50:0/152393898 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd000ba50 con 0x7f1dd8156030 2023-12-19T15:54:16.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.093+0000 7f1dd6ffd700 1 -- 172.21.15.50:0/152393898 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00092e0 con 0x7f1dd8156030 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 -- 172.21.15.50:0/152393898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 msgr2=0x7f1dd8159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/152393898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8159c80 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80c7fa0 tx=0x7f1dd0009840 comp rx=0 tx=0).stop 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 -- 172.21.15.50:0/152393898 shutdown_connections 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/152393898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8159c80 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/152393898 >> v2:172.21.15.125:3300/0 conn(0x7f1dd80c2f60 0x7f1dd81500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 -- 172.21.15.50:0/152393898 >> 172.21.15.50:0/152393898 conn(0x7f1dd8069e80 msgr2=0x7f1dd806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 -- 172.21.15.50:0/152393898 shutdown_connections 2023-12-19T15:54:16.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.097+0000 7f1ddcb63700 1 -- 172.21.15.50:0/152393898 wait complete. 2023-12-19T15:54:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 Processor -- start 2023-12-19T15:54:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 -- start start 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd80c2f60 0x7f1dd80c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd814fcf0 0x7f1dd814e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81690d0 con 0x7f1dd8156030 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80c5390 con 0x7f1dd814fcf0 2023-12-19T15:54:16.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1ddcb63700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8168dd0 con 0x7f1dd80c2f60 2023-12-19T15:54:16.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1dd7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1dd7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55282/0 (socket says 172.21.15.50:55282) 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.101+0000 7f1dd7fff700 1 -- 172.21.15.50:0/1236903001 learned_addr learned my addr 172.21.15.50:0/1236903001 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd77fe700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd80c2f60 0x7f1dd80c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dcffff700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd814fcf0 0x7f1dd814e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 -- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd80c2f60 msgr2=0x7f1dd80c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd80c2f60 0x7f1dd80c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 -- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd814fcf0 msgr2=0x7f1dd814e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd814fcf0 0x7f1dd814e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 -- 172.21.15.50:0/1236903001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0009ae0 con 0x7f1dd8156030 2023-12-19T15:54:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dd7fff700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0008950 tx=0x7f1dd0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0008d40 con 0x7f1dd8156030 2023-12-19T15:54:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0025460 con 0x7f1dd8156030 2023-12-19T15:54:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd80c68d0 con 0x7f1dd8156030 2023-12-19T15:54:16.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0008ea0 con 0x7f1dd8156030 2023-12-19T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.105+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd80c4dd0 con 0x7f1dd8156030 2023-12-19T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.109+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd8169240 con 0x7f1dd8156030 2023-12-19T15:54:16.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.109+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 64) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f1dd0003040 con 0x7f1dd8156030 2023-12-19T15:54:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.109+0000 7f1dcf7fe700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 0x7f1db8062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.109+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f1dd007e3c0 con 0x7f1dd8156030 2023-12-19T15:54:16.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.113+0000 7f1dd77fe700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 0x7f1db8062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.113+0000 7f1dd77fe700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 0x7f1db8062b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8003a80 tx=0x7f1dc8007ec0 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.117+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1dd8169240 con 0x7f1dd8156030 2023-12-19T15:54:16.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.309+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1dd8064040 con 0x7f1db80606c0 2023-12-19T15:54:16.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.329+0000 7f1dcf7fe700 1 -- 172.21.15.50:0/1236903001 <== mgr.8553 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1dd8064040 con 0x7f1db80606c0 2023-12-19T15:54:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:16.334 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:58:40", "first": "2023-12-19T14:59:00", "last": "2023-12-19T15:54:00", "last_pruned": "2023-12-19T15:54:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-12-19T15:54:16.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 msgr2=0x7f1db8062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 0x7f1db8062b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8003a80 tx=0x7f1dc8007ec0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 msgr2=0x7f1dd8067970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0008950 tx=0x7f1dd0018000 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 shutdown_connections 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dd814fcf0 0x7f1dd814e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1db80606c0 0x7f1db8062b70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dd8156030 0x7f1dd8067970 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 --2- 172.21.15.50:0/1236903001 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dd80c2f60 0x7f1dd80c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 >> 172.21.15.50:0/1236903001 conn(0x7f1dd8069e80 msgr2=0x7f1dd8154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 shutdown_connections 2023-12-19T15:54:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.333+0000 7f1ddcb63700 1 -- 172.21.15.50:0/1236903001 wait complete. 2023-12-19T15:54:16.360 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T14:58:40", "first": "2023-12-19T14:59:00", "last": "2023-12-19T15:54:00", "last_pruned": "2023-12-19T15:54:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-12-19T15:54:16.360 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1M', 'retention': {'n': 55}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T14:58:40', 'first': '2023-12-19T14:59:00', 'last': '2023-12-19T15:54:00', 'last_pruned': '2023-12-19T15:54:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-12-19T15:54:16.360 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-12-19T15:54:16.360 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-12-19T15:54:16.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1 2023-12-19T15:54:16.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 Processor -- start 2023-12-19T15:54:16.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 -- start start 2023-12-19T15:54:16.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45800cb130 0x7f45800cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f458006a6a0 0x7f458006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f458015c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458006b030 con 0x7f458006a6a0 2023-12-19T15:54:16.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800b0ac0 con 0x7f45800cb130 2023-12-19T15:54:16.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.529+0000 7f4585745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458015cbb0 con 0x7f458015c220 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f458006a6a0 0x7f458006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f458006a6a0 0x7f458006aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48608/0 (socket says 172.21.15.50:48608) 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 -- 172.21.15.50:0/3011866079 learned_addr learned my addr 172.21.15.50:0/3011866079 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457f7fe700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f458015c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457effd700 1 --2- 172.21.15.50:0/3011866079 >> v2:172.21.15.125:3300/0 conn(0x7f45800cb130 0x7f45800cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 -- 172.21.15.50:0/3011866079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 msgr2=0x7f458015c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f458015c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 -- 172.21.15.50:0/3011866079 >> v2:172.21.15.125:3300/0 conn(0x7f45800cb130 msgr2=0x7f45800cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 --2- 172.21.15.50:0/3011866079 >> v2:172.21.15.125:3300/0 conn(0x7f45800cb130 0x7f45800cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 -- 172.21.15.50:0/3011866079 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458006b210 con 0x7f458006a6a0 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457e7fc700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f458006a6a0 0x7f458006aaf0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f45800694e0 tx=0x7f4574009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a941445552d7772 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457dffb700 1 -- 172.21.15.50:0/3011866079 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574009e70 con 0x7f458006a6a0 2023-12-19T15:54:16.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457dffb700 1 -- 172.21.15.50:0/3011866079 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400b860 con 0x7f458006a6a0 2023-12-19T15:54:16.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f457dffb700 1 -- 172.21.15.50:0/3011866079 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400bc00 con 0x7f458006a6a0 2023-12-19T15:54:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 -- 172.21.15.50:0/3011866079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f458006a6a0 msgr2=0x7f458006aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f458006a6a0 0x7f458006aaf0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f45800694e0 tx=0x7f4574009250 comp rx=0 tx=0).stop 2023-12-19T15:54:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 -- 172.21.15.50:0/3011866079 shutdown_connections 2023-12-19T15:54:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f458006a6a0 0x7f458006aaf0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 --2- 172.21.15.50:0/3011866079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f458015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 --2- 172.21.15.50:0/3011866079 >> v2:172.21.15.125:3300/0 conn(0x7f45800cb130 0x7f45800cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 -- 172.21.15.50:0/3011866079 >> 172.21.15.50:0/3011866079 conn(0x7f4580067bb0 msgr2=0x7f45800cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 -- 172.21.15.50:0/3011866079 shutdown_connections 2023-12-19T15:54:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.533+0000 7f4585745700 1 -- 172.21.15.50:0/3011866079 wait complete. 2023-12-19T15:54:16.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.537+0000 7f4585745700 1 Processor -- start 2023-12-19T15:54:16.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.537+0000 7f4585745700 1 -- start start 2023-12-19T15:54:16.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458006a6a0 0x7f45800b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f4580154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458016a100 con 0x7f45800cb130 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458015bd70 con 0x7f458006a6a0 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4580169480 con 0x7f458015c220 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48614/0 (socket says 172.21.15.50:48614) 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 -- 172.21.15.50:0/2069793131 learned_addr learned my addr 172.21.15.50:0/2069793131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457effd700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458006a6a0 0x7f45800b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457f7fe700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f4580154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 -- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 msgr2=0x7f4580154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f4580154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 -- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458006a6a0 msgr2=0x7f45800b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458006a6a0 0x7f45800b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 -- 172.21.15.50:0/2069793131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740094f0 con 0x7f45800cb130 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f457e7fc700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4574006fd0 tx=0x7f4574017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400bbe0 con 0x7f45800cb130 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574000c90 con 0x7f45800cb130 2023-12-19T15:54:16.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574011630 con 0x7f45800cb130 2023-12-19T15:54:16.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4580169760 con 0x7f45800cb130 2023-12-19T15:54:16.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.541+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4580169d20 con 0x7f45800cb130 2023-12-19T15:54:16.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.545+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458016a240 con 0x7f45800cb130 2023-12-19T15:54:16.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.549+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 64) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f4574011a60 con 0x7f45800cb130 2023-12-19T15:54:16.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.549+0000 7f4566ffd700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 0x7f456c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:16.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.549+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f457407e4c0 con 0x7f45800cb130 2023-12-19T15:54:16.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.549+0000 7f457effd700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 0x7f456c062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:16.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.553+0000 7f457effd700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 0x7f456c062b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4570006fd0 tx=0x7f4570008040 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.557+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f457404e600 con 0x7f45800cb130 2023-12-19T15:54:16.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.753+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4580071120 con 0x7f456c0606c0 2023-12-19T15:54:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.885+0000 7f4566ffd700 1 -- 172.21.15.50:0/2069793131 <== mgr.8553 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f4580071120 con 0x7f456c0606c0 2023-12-19T15:54:16.891 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1 2023-12-19T15:54:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 msgr2=0x7f456c062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 0x7f456c062b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4570006fd0 tx=0x7f4570008040 comp rx=0 tx=0).stop 2023-12-19T15:54:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 msgr2=0x7f458015b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f4574006fd0 tx=0x7f4574017000 comp rx=0 tx=0).stop 2023-12-19T15:54:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 shutdown_connections 2023-12-19T15:54:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f456c0606c0 0x7f456c062b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f458015c220 0x7f4580154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45800cb130 0x7f458015b830 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 --2- 172.21.15.50:0/2069793131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458006a6a0 0x7f45800b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 >> 172.21.15.50:0/2069793131 conn(0x7f4580067bb0 msgr2=0x7f45800cdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 shutdown_connections 2023-12-19T15:54:16.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:16.889+0000 7f4585745700 1 -- 172.21.15.50:0/2069793131 wait complete. 2023-12-19T15:54:16.921 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-12-19T15:54:16.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:16.922 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_00_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_01_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_02_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_03_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_04_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_05_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_06_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_07_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_08_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_09_00_UTC 2023-12-19T15:54:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_10_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_11_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_12_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_13_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_14_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_15_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_16_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_17_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_18_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_19_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_20_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_21_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_22_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_23_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_24_00_UTC 2023-12-19T15:54:16.964 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_25_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_26_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_27_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_28_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_29_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_30_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_31_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_32_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_33_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_34_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_35_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_36_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_37_00_UTC 2023-12-19T15:54:16.965 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_38_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_39_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_40_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_41_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_42_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_43_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_44_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_45_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_46_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_47_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_48_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_49_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_50_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_51_00_UTC 2023-12-19T15:54:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_52_00_UTC 2023-12-19T15:54:16.967 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_53_00_UTC 2023-12-19T15:54:16.967 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_54_00_UTC 2023-12-19T15:54:16.967 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_00_00_UTC 2023-12-19T15:54:16.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:16.968 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_00_00_UTC) 2023-12-19T15:54:17.051 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_01_00_UTC 2023-12-19T15:54:17.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.051 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_01_00_UTC) 2023-12-19T15:54:17.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_02_00_UTC 2023-12-19T15:54:17.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.134 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_02_00_UTC) 2023-12-19T15:54:17.221 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_03_00_UTC 2023-12-19T15:54:17.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.222 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_03_00_UTC) 2023-12-19T15:54:17.312 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_04_00_UTC 2023-12-19T15:54:17.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.313 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_04_00_UTC) 2023-12-19T15:54:17.401 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_05_00_UTC 2023-12-19T15:54:17.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.401 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_05_00_UTC) 2023-12-19T15:54:17.484 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_06_00_UTC 2023-12-19T15:54:17.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.485 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_06_00_UTC) 2023-12-19T15:54:17.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_07_00_UTC 2023-12-19T15:54:17.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.573 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_07_00_UTC) 2023-12-19T15:54:17.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_08_00_UTC 2023-12-19T15:54:17.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.658 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_08_00_UTC) 2023-12-19T15:54:17.744 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_09_00_UTC 2023-12-19T15:54:17.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.744 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_09_00_UTC) 2023-12-19T15:54:17.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_10_00_UTC 2023-12-19T15:54:17.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.831 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_10_00_UTC) 2023-12-19T15:54:17.915 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_11_00_UTC 2023-12-19T15:54:17.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:17.915 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_11_00_UTC) 2023-12-19T15:54:18.005 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_12_00_UTC 2023-12-19T15:54:18.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.005 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_12_00_UTC) 2023-12-19T15:54:18.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_13_00_UTC 2023-12-19T15:54:18.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.094 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_13_00_UTC) 2023-12-19T15:54:18.174 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_14_00_UTC 2023-12-19T15:54:18.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.174 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_14_00_UTC) 2023-12-19T15:54:18.259 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_15_00_UTC 2023-12-19T15:54:18.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.259 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_15_00_UTC) 2023-12-19T15:54:18.339 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_16_00_UTC 2023-12-19T15:54:18.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.339 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_16_00_UTC) 2023-12-19T15:54:18.425 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_17_00_UTC 2023-12-19T15:54:18.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.425 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_17_00_UTC) 2023-12-19T15:54:18.508 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_18_00_UTC 2023-12-19T15:54:18.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.509 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_18_00_UTC) 2023-12-19T15:54:18.593 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_19_00_UTC 2023-12-19T15:54:18.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.593 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_19_00_UTC) 2023-12-19T15:54:18.677 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_20_00_UTC 2023-12-19T15:54:18.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.677 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_20_00_UTC) 2023-12-19T15:54:18.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_21_00_UTC 2023-12-19T15:54:18.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.763 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_21_00_UTC) 2023-12-19T15:54:18.838 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_22_00_UTC 2023-12-19T15:54:18.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.839 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_22_00_UTC) 2023-12-19T15:54:18.921 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_23_00_UTC 2023-12-19T15:54:18.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:18.921 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_23_00_UTC) 2023-12-19T15:54:19.006 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_24_00_UTC 2023-12-19T15:54:19.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.007 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_24_00_UTC) 2023-12-19T15:54:19.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_25_00_UTC 2023-12-19T15:54:19.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.087 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_25_00_UTC) 2023-12-19T15:54:19.165 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_26_00_UTC 2023-12-19T15:54:19.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.166 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_26_00_UTC) 2023-12-19T15:54:19.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_27_00_UTC 2023-12-19T15:54:19.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.246 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_27_00_UTC) 2023-12-19T15:54:19.334 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_28_00_UTC 2023-12-19T15:54:19.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.334 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_28_00_UTC) 2023-12-19T15:54:19.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_29_00_UTC 2023-12-19T15:54:19.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.415 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_29_00_UTC) 2023-12-19T15:54:19.492 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_30_00_UTC 2023-12-19T15:54:19.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.493 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_30_00_UTC) 2023-12-19T15:54:19.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_31_00_UTC 2023-12-19T15:54:19.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.582 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_31_00_UTC) 2023-12-19T15:54:19.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_32_00_UTC 2023-12-19T15:54:19.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.668 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_32_00_UTC) 2023-12-19T15:54:19.748 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_33_00_UTC 2023-12-19T15:54:19.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.748 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_33_00_UTC) 2023-12-19T15:54:19.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_34_00_UTC 2023-12-19T15:54:19.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.829 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_34_00_UTC) 2023-12-19T15:54:19.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_35_00_UTC 2023-12-19T15:54:19.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.916 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_35_00_UTC) 2023-12-19T15:54:19.996 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_36_00_UTC 2023-12-19T15:54:19.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:19.997 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_36_00_UTC) 2023-12-19T15:54:20.088 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_37_00_UTC 2023-12-19T15:54:20.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.088 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_37_00_UTC) 2023-12-19T15:54:20.165 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_38_00_UTC 2023-12-19T15:54:20.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.165 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_38_00_UTC) 2023-12-19T15:54:20.268 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_39_00_UTC 2023-12-19T15:54:20.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.268 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_39_00_UTC) 2023-12-19T15:54:20.320 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_40_00_UTC 2023-12-19T15:54:20.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.320 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_40_00_UTC) 2023-12-19T15:54:20.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_41_00_UTC 2023-12-19T15:54:20.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.353 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_41_00_UTC) 2023-12-19T15:54:20.435 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_42_00_UTC 2023-12-19T15:54:20.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.435 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_42_00_UTC) 2023-12-19T15:54:20.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_43_00_UTC 2023-12-19T15:54:20.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.519 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_43_00_UTC) 2023-12-19T15:54:20.604 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_44_00_UTC 2023-12-19T15:54:20.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.605 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_44_00_UTC) 2023-12-19T15:54:20.688 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_45_00_UTC 2023-12-19T15:54:20.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.688 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_45_00_UTC) 2023-12-19T15:54:20.770 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_46_00_UTC 2023-12-19T15:54:20.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.771 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_46_00_UTC) 2023-12-19T15:54:20.854 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_47_00_UTC 2023-12-19T15:54:20.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.855 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_47_00_UTC) 2023-12-19T15:54:20.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_48_00_UTC 2023-12-19T15:54:20.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:20.937 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_48_00_UTC) 2023-12-19T15:54:21.026 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_49_00_UTC 2023-12-19T15:54:21.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.027 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_49_00_UTC) 2023-12-19T15:54:21.130 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_50_00_UTC 2023-12-19T15:54:21.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.130 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_50_00_UTC) 2023-12-19T15:54:21.170 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_51_00_UTC 2023-12-19T15:54:21.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.171 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_51_00_UTC) 2023-12-19T15:54:21.256 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_52_00_UTC 2023-12-19T15:54:21.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.256 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_52_00_UTC) 2023-12-19T15:54:21.340 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_53_00_UTC 2023-12-19T15:54:21.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.340 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_53_00_UTC) 2023-12-19T15:54:21.421 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-15_54_00_UTC 2023-12-19T15:54:21.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.422 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-15_54_00_UTC) 2023-12-19T15:54:21.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:21.512 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T15:54:21.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T15:54:21.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.789+0000 7f1cc9acd700 1 Processor -- start 2023-12-19T15:54:21.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.789+0000 7f1cc9acd700 1 -- start start 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc9acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 0x7f1cc40a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc9acd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc40c6ab0 con 0x7f1cc415c8f0 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc9acd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc406b240 con 0x7f1cc415c520 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc9acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc406b520 con 0x7f1cc40ca950 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc2ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1cc415c8f0 0x7f1cc40ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37708/0 (socket says 172.21.15.50:37708) 2023-12-19T15:54:21.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc2ffd700 1 -- 172.21.15.50:0/3457647881 learned_addr learned my addr 172.21.15.50:0/3457647881 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:21.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc3fff700 1 --2- 172.21.15.50:0/3457647881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 0x7f1cc40a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:21.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804710912 0 0) 0x7f1cc40c6ab0 con 0x7f1cc415c8f0 2023-12-19T15:54:21.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cac003570 con 0x7f1cc415c8f0 2023-12-19T15:54:21.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677876048 0 0) 0x7f1cc406b240 con 0x7f1cc415c520 2023-12-19T15:54:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc40c6ab0 con 0x7f1cc415c520 2023-12-19T15:54:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc3fff700 1 -- 172.21.15.50:0/3457647881 >> v1:172.21.15.125:6789/0 conn(0x7f1cc415c520 legacy=0x7f1cc406a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc3fff700 1 -- 172.21.15.50:0/3457647881 >> v1:172.21.15.50:6789/0 conn(0x7f1cc415c8f0 legacy=0x7f1cc40ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:21.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.793+0000 7f1cc3fff700 1 -- 172.21.15.50:0/3457647881 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40c6d50 con 0x7f1cc40ca950 2023-12-19T15:54:21.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc3fff700 1 --2- 172.21.15.50:0/3457647881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 0x7f1cc40a8850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009e90 tx=0x7f1cbc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b478005392e9232 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:21.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc00e070 con 0x7f1cc40ca950 2023-12-19T15:54:21.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc00bc10 con 0x7f1cc40ca950 2023-12-19T15:54:21.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc27fc700 1 -- 172.21.15.50:0/3457647881 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc012430 con 0x7f1cc40ca950 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 -- 172.21.15.50:0/3457647881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 msgr2=0x7f1cc40a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/3457647881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 0x7f1cc40a8850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009e90 tx=0x7f1cbc009780 comp rx=0 tx=0).stop 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 -- 172.21.15.50:0/3457647881 shutdown_connections 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/3457647881 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc40ca950 0x7f1cc40a8850 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 -- 172.21.15.50:0/3457647881 >> 172.21.15.50:0/3457647881 conn(0x7f1cc4067bb0 msgr2=0x7f1cc40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 -- 172.21.15.50:0/3457647881 shutdown_connections 2023-12-19T15:54:21.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.797+0000 7f1cc9acd700 1 -- 172.21.15.50:0/3457647881 wait complete. 2023-12-19T15:54:21.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 Processor -- start 2023-12-19T15:54:21.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 -- start start 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc415c520 0x7f1cc40c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc415c8f0 0x7f1cc415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40c3010 con 0x7f1cc40ca950 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc4168b50 con 0x7f1cc415c520 2023-12-19T15:54:21.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc9acd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc4168e10 con 0x7f1cc415c8f0 2023-12-19T15:54:21.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48620/0 (socket says 172.21.15.50:48620) 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.801+0000 7f1cc37fe700 1 -- 172.21.15.50:0/1773604770 learned_addr learned my addr 172.21.15.50:0/1773604770 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc2ffd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc415c520 0x7f1cc40c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc3fff700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc415c8f0 0x7f1cc415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 -- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc415c8f0 msgr2=0x7f1cc415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc415c8f0 0x7f1cc415ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 -- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc415c520 msgr2=0x7f1cc40c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc415c520 0x7f1cc40c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 -- 172.21.15.50:0/1773604770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc009a20 con 0x7f1cc40ca950 2023-12-19T15:54:21.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc37fe700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400c790 tx=0x7f1cb4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4010510 con 0x7f1cc40ca950 2023-12-19T15:54:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb4010670 con 0x7f1cc40ca950 2023-12-19T15:54:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4010a10 con 0x7f1cc40ca950 2023-12-19T15:54:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc41690f0 con 0x7f1cc40ca950 2023-12-19T15:54:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.805+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc41696b0 con 0x7f1cc40ca950 2023-12-19T15:54:21.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.809+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc4071120 con 0x7f1cc40ca950 2023-12-19T15:54:21.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.809+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 64) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f1cb401b3b0 con 0x7f1cc40ca950 2023-12-19T15:54:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.809+0000 7f1cab7fe700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 0x7f1cb0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.809+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f1cb407c620 con 0x7f1cc40ca950 2023-12-19T15:54:21.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.813+0000 7f1cc2ffd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 0x7f1cb0062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:21.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.813+0000 7f1cc2ffd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 0x7f1cb0062b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1cc406c010 tx=0x7f1cb8009040 comp rx=0 tx=0).ready entity=mgr.8553 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:21.817+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1cb404c730 con 0x7f1cc40ca950 2023-12-19T15:54:22.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.033+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1cc406ded0 con 0x7f1cc40ca950 2023-12-19T15:54:22.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.085+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f1cb404cb00 con 0x7f1cc40ca950 2023-12-19T15:54:22.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.085+0000 7f1cab7fe700 1 -- 172.21.15.50:0/1773604770 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v65) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1cb4046b30 con 0x7f1cc40ca950 2023-12-19T15:54:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 msgr2=0x7f1cb0062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 0x7f1cb0062b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1cc406c010 tx=0x7f1cb8009040 comp rx=0 tx=0).stop 2023-12-19T15:54:22.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 msgr2=0x7f1cc4159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400c790 tx=0x7f1cb4007e10 comp rx=0 tx=0).stop 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 shutdown_connections 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc415c520 0x7f1cc40c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1cb00606c0 0x7f1cb0062b70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1cc415c8f0 0x7f1cc415ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 --2- 172.21.15.50:0/1773604770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cc40ca950 0x7f1cc4159ae0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.089+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 >> 172.21.15.50:0/1773604770 conn(0x7f1cc4067bb0 msgr2=0x7f1cc40c8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:22.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.093+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 shutdown_connections 2023-12-19T15:54:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.093+0000 7f1cc9acd700 1 -- 172.21.15.50:0/1773604770 wait complete. 2023-12-19T15:54:22.126 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T15:54:22.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:22.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:22.275 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T15:54:22.275 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T15:54:22.276 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T15:54:22.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:22.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[54502]: fuse finished with error 0 and tester_r 0 2023-12-19T15:54:22.304 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T15:54:22.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:22.304 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T15:54:22.377 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:22.369+0000 7fce5e8eb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:54:22.377 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:22.367+0000 7f5df3f67000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:54:22.506 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T15:54:22.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:22.506 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T15:54:22.524 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T15:54:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:54:22.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T15:54:22.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:22.525 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T15:54:22.567 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T15:54:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:54:22.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T15:54:22.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:22.569 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T15:54:22.590 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T15:54:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:54:22.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:22.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 Processor -- start 2023-12-19T15:54:22.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 -- start start 2023-12-19T15:54:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 0x7f4dcc15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0c9240 con 0x7f4dcc0acdd0 2023-12-19T15:54:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc15b060 con 0x7f4dcc0c8360 2023-12-19T15:54:22.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dd1816700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0ccb00 con 0x7f4dcc15ac00 2023-12-19T15:54:22.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.765+0000 7f4dcbfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 0x7f4dcc15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:22.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:22.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:41038/0 (socket says 172.21.15.50:41038) 2023-12-19T15:54:22.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 -- 172.21.15.50:0/833979948 learned_addr learned my addr 172.21.15.50:0/833979948 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:22.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 -- 172.21.15.50:0/833979948 >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 msgr2=0x7f4dcc15a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:22.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 --2- 172.21.15.50:0/833979948 >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 0x7f4dcc15a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 -- 172.21.15.50:0/833979948 >> v1:172.21.15.50:6789/0 conn(0x7f4dcc0acdd0 legacy=0x7f4dcc0c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 -- 172.21.15.50:0/833979948 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0cbef0 con 0x7f4dcc15ac00 2023-12-19T15:54:22.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dcbfff700 1 --2- 172.21.15.50:0/833979948 >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 0x7f4dcc15a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:22.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1015700 1 --2- 172.21.15.50:0/833979948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009f20 tx=0x7f4dc4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6b6d68ee0bb36a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:22.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dcb7fe700 1 -- 172.21.15.50:0/833979948 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400e070 con 0x7f4dcc15ac00 2023-12-19T15:54:22.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dcb7fe700 1 -- 172.21.15.50:0/833979948 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc400b9d0 con 0x7f4dcc15ac00 2023-12-19T15:54:22.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dcb7fe700 1 -- 172.21.15.50:0/833979948 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc40092e0 con 0x7f4dcc15ac00 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 -- 172.21.15.50:0/833979948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 msgr2=0x7f4dcc0ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 --2- 172.21.15.50:0/833979948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009f20 tx=0x7f4dc4009840 comp rx=0 tx=0).stop 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 -- 172.21.15.50:0/833979948 shutdown_connections 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 --2- 172.21.15.50:0/833979948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc15ac00 0x7f4dcc0ccf90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 --2- 172.21.15.50:0/833979948 >> v2:172.21.15.50:3300/0 conn(0x7f4dcc0c8360 0x7f4dcc15a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 -- 172.21.15.50:0/833979948 >> 172.21.15.50:0/833979948 conn(0x7f4dcc0cdbd0 msgr2=0x7f4dcc0cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 -- 172.21.15.50:0/833979948 shutdown_connections 2023-12-19T15:54:22.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.769+0000 7f4dd1816700 1 -- 172.21.15.50:0/833979948 wait complete. 2023-12-19T15:54:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 Processor -- start 2023-12-19T15:54:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 -- start start 2023-12-19T15:54:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dcc0acdd0 0x7f4dcc068020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc0c8360 0x7f4dcc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc067b90 con 0x7f4dcc15ac00 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc155d90 con 0x7f4dcc0acdd0 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.773+0000 7f4dd1816700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc067260 con 0x7f4dcc0c8360 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48632/0 (socket says 172.21.15.50:48632) 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 -- 172.21.15.50:0/1576816823 learned_addr learned my addr 172.21.15.50:0/1576816823 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dcbfff700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc0c8360 0x7f4dcc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd0814700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dcc0acdd0 0x7f4dcc068020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc0c8360 msgr2=0x7f4dcc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc0c8360 0x7f4dcc155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dcc0acdd0 msgr2=0x7f4dcc068020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dcc0acdd0 0x7f4dcc068020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 -- 172.21.15.50:0/1576816823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009420 con 0x7f4dcc15ac00 2023-12-19T15:54:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1015700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400b610 tx=0x7f4dc4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4008ad0 con 0x7f4dcc15ac00 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4008c30 con 0x7f4dcc15ac00 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4025460 con 0x7f4dcc15ac00 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4009ae0 con 0x7f4dcc15ac00 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.777+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc067820 con 0x7f4dcc15ac00 2023-12-19T15:54:22.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc067d40 con 0x7f4dcc15ac00 2023-12-19T15:54:22.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f4dc4003040 con 0x7f4dcc15ac00 2023-12-19T15:54:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dc8ff9700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 0x7f4db8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f4dc407e920 con 0x7f4dcc15ac00 2023-12-19T15:54:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dd0814700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 msgr2=0x7f4db8062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:22.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.781+0000 7f4dd0814700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 0x7f4db8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:22.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.789+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4dc404e920 con 0x7f4dcc15ac00 2023-12-19T15:54:22.933 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:22.927+0000 7f5df3f67000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:54:22.957 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:22.949+0000 7fce5e8eb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:54:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.985+0000 7f4dd0814700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 msgr2=0x7f4db8062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.985+0000 7f4dd0814700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 0x7f4db8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:23.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:22.997+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dcc04be40 con 0x7f4dcc15ac00 2023-12-19T15:54:23.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dc8ff9700 1 -- 172.21.15.50:0/1576816823 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f4dc4051f40 con 0x7f4dcc15ac00 2023-12-19T15:54:23.005 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:23.006 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T15:54:23.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 msgr2=0x7f4db8062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:23.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 0x7f4db8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 msgr2=0x7f4dcc068810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400b610 tx=0x7f4dc4018000 comp rx=0 tx=0).stop 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 shutdown_connections 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4db8060670 0x7f4db8062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dcc15ac00 0x7f4dcc068810 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4dcc0c8360 0x7f4dcc155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 --2- 172.21.15.50:0/1576816823 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4dcc0acdd0 0x7f4dcc068020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 >> 172.21.15.50:0/1576816823 conn(0x7f4dcc0cdbd0 msgr2=0x7f4dcc0b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 shutdown_connections 2023-12-19T15:54:23.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.001+0000 7f4dd1816700 1 -- 172.21.15.50:0/1576816823 wait complete. 2023-12-19T15:54:23.010 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.003+0000 7f5df3f67000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:54:23.010 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T15:54:23.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:23.035 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.029+0000 7fce5e8eb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:54:23.173 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.167+0000 7f5df3f67000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:23.202 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.197+0000 7fce5e8eb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:23.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 Processor -- start 2023-12-19T15:54:23.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 -- start start 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b1006b900 0x7f8b1006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1006b470 con 0x7f8b1015cc20 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1015d7b0 con 0x7f8b1006b900 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16b23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10161d10 con 0x7f8b100c6900 2023-12-19T15:54:23.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b15b21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b15b21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:41056/0 (socket says 172.21.15.50:41056) 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b15b21700 1 -- 172.21.15.50:0/3535699613 learned_addr learned my addr 172.21.15.50:0/3535699613 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b15320700 1 --2- 172.21.15.50:0/3535699613 >> v2:172.21.15.50:3300/0 conn(0x7f8b1006b900 0x7f8b1006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.197+0000 7f8b16322700 1 --1- 172.21.15.50:0/3535699613 >> v1:172.21.15.50:6789/0 conn(0x7f8b1015cc20 0x7f8b1015d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37722/0 (socket says 172.21.15.50:37722) 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b15b21700 1 -- 172.21.15.50:0/3535699613 >> v2:172.21.15.50:3300/0 conn(0x7f8b1006b900 msgr2=0x7f8b1006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:23.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b15b21700 1 --2- 172.21.15.50:0/3535699613 >> v2:172.21.15.50:3300/0 conn(0x7f8b1006b900 0x7f8b1006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b15b21700 1 -- 172.21.15.50:0/3535699613 >> v1:172.21.15.50:6789/0 conn(0x7f8b1015cc20 legacy=0x7f8b1015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b15b21700 1 -- 172.21.15.50:0/3535699613 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1006bd70 con 0x7f8b100c6900 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b15b21700 1 --2- 172.21.15.50:0/3535699613 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009f50 tx=0x7f8b0c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a980ce6fe8e1b76f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b14b1f700 1 -- 172.21.15.50:0/3535699613 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00e070 con 0x7f8b100c6900 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b14b1f700 1 -- 172.21.15.50:0/3535699613 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00ba70 con 0x7f8b100c6900 2023-12-19T15:54:23.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b14b1f700 1 -- 172.21.15.50:0/3535699613 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c0092e0 con 0x7f8b100c6900 2023-12-19T15:54:23.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 -- 172.21.15.50:0/3535699613 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 msgr2=0x7f8b100c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 --2- 172.21.15.50:0/3535699613 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009f50 tx=0x7f8b0c009840 comp rx=0 tx=0).stop 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 -- 172.21.15.50:0/3535699613 shutdown_connections 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 --2- 172.21.15.50:0/3535699613 >> v2:172.21.15.50:3300/0 conn(0x7f8b1006b900 0x7f8b1006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 --2- 172.21.15.50:0/3535699613 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b100c6cd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 -- 172.21.15.50:0/3535699613 >> 172.21.15.50:0/3535699613 conn(0x7f8b10067bb0 msgr2=0x7f8b100cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 -- 172.21.15.50:0/3535699613 shutdown_connections 2023-12-19T15:54:23.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.201+0000 7f8b16b23700 1 -- 172.21.15.50:0/3535699613 wait complete. 2023-12-19T15:54:23.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 Processor -- start 2023-12-19T15:54:23.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 -- start start 2023-12-19T15:54:23.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b10147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 0x7f8b10147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100af340 con 0x7f8b1006b900 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100b0190 con 0x7f8b1015cc20 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16b23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100b18c0 con 0x7f8b100c6900 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b15b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b15b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48656/0 (socket says 172.21.15.50:48656) 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b15b21700 1 -- 172.21.15.50:0/2096979211 learned_addr learned my addr 172.21.15.50:0/2096979211 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:23.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b16322700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 0x7f8b10147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.205+0000 7f8b15320700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b10147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 msgr2=0x7f8b10147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b10147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 msgr2=0x7f8b10147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 0x7f8b10147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 -- 172.21.15.50:0/2096979211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c009ae0 con 0x7f8b1006b900 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b16322700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 0x7f8b10147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b15b21700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00b770 tx=0x7f8b0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c008a20 con 0x7f8b1006b900 2023-12-19T15:54:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c008b80 con 0x7f8b1006b900 2023-12-19T15:54:23.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b100af580 con 0x7f8b1006b900 2023-12-19T15:54:23.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c026460 con 0x7f8b1006b900 2023-12-19T15:54:23.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b100afb40 con 0x7f8b1006b900 2023-12-19T15:54:23.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.209+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1004be40 con 0x7f8b1006b900 2023-12-19T15:54:23.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.213+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f8b0c003040 con 0x7f8b1006b900 2023-12-19T15:54:23.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.217+0000 7f8b05ffb700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 0x7f8af4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.217+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f8b0c07ed60 con 0x7f8b1006b900 2023-12-19T15:54:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.225+0000 7f8b15320700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 msgr2=0x7f8af4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.225+0000 7f8b15320700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 0x7f8af4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:23.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.225+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8b0c04ed60 con 0x7f8b1006b900 2023-12-19T15:54:23.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.429+0000 7f8b15320700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 msgr2=0x7f8af4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:23.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.429+0000 7f8b15320700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 0x7f8af4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:23.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.433+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b1006a580 con 0x7f8b1006b900 2023-12-19T15:54:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b05ffb700 1 -- 172.21.15.50:0/2096979211 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16318 (secure 0 0 0) 0x7f8b0c052380 con 0x7f8b1006b900 2023-12-19T15:54:23.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:23.442 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":153,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:23.086172+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":153,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/4275867050":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/1673033377":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2615455122":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/70137828":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2692951617":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6833/33827":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/30212990":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6832/33827":"2023-12-20T14:58:39.674128+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":18,"snaps":[{"begin":51,"length":7},{"begin":107,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:23.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 msgr2=0x7f8af4062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 0x7f8af4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 msgr2=0x7f8b100b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00b770 tx=0x7f8b0c018000 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 shutdown_connections 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b1015cc20 0x7f8b10147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8af4060670 0x7f8af4062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b100c6900 0x7f8b10147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 --2- 172.21.15.50:0/2096979211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b1006b900 0x7f8b100b1380 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.437+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 >> 172.21.15.50:0/2096979211 conn(0x7f8b10067bb0 msgr2=0x7f8b10159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.441+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 shutdown_connections 2023-12-19T15:54:23.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.441+0000 7f8b16b23700 1 -- 172.21.15.50:0/2096979211 wait complete. 2023-12-19T15:54:23.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T14:58:21.215201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T14:58:21.997104+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 153, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T15:54:23.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:54:23.477 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.471+0000 7f5df3f67000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:54:23.545 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.537+0000 7fce5e8eb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:54:23.556 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.551+0000 7f5df3f67000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:23.626 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.621+0000 7fce5e8eb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.673+0000 7f96c12f0700 1 Processor -- start 2023-12-19T15:54:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.673+0000 7f96c12f0700 1 -- start start 2023-12-19T15:54:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.673+0000 7f96c12f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0ab870 0x7f96bc0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.673+0000 7f96c12f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96bc0b08d0 0x7f96bc159a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc15a3f0 con 0x7f96bc159fb0 2023-12-19T15:54:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc0b0e40 con 0x7f96bc0b08d0 2023-12-19T15:54:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0b1120 con 0x7f96bc0ab870 2023-12-19T15:54:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96bc0b08d0 0x7f96bc159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96bc0b08d0 0x7f96bc159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48658/0 (socket says 172.21.15.50:48658) 2023-12-19T15:54:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 -- 172.21.15.50:0/23022447 learned_addr learned my addr 172.21.15.50:0/23022447 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96baffd700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0ab870 0x7f96bc0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 -- 172.21.15.50:0/23022447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0ab870 msgr2=0x7f96bc0afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0ab870 0x7f96bc0afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 -- 172.21.15.50:0/23022447 >> v1:172.21.15.50:6789/0 conn(0x7f96bc159fb0 legacy=0x7f96bc156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 -- 172.21.15.50:0/23022447 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0b1400 con 0x7f96bc0b08d0 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96ba7fc700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0b08d0 0x7f96bc159a70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988a43b5d8a01c9f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96b9ffb700 1 -- 172.21.15.50:0/23022447 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b0009ea0 con 0x7f96bc0b08d0 2023-12-19T15:54:23.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96b9ffb700 1 -- 172.21.15.50:0/23022447 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b000b860 con 0x7f96bc0b08d0 2023-12-19T15:54:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96b9ffb700 1 -- 172.21.15.50:0/23022447 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000bc00 con 0x7f96bc0b08d0 2023-12-19T15:54:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- 172.21.15.50:0/23022447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0b08d0 msgr2=0x7f96bc159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0b08d0 0x7f96bc159a70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0009250 comp rx=0 tx=0).stop 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- 172.21.15.50:0/23022447 shutdown_connections 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0b08d0 0x7f96bc159a70 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 --2- 172.21.15.50:0/23022447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0ab870 0x7f96bc0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- 172.21.15.50:0/23022447 >> 172.21.15.50:0/23022447 conn(0x7f96bc0cc580 msgr2=0x7f96bc0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- 172.21.15.50:0/23022447 shutdown_connections 2023-12-19T15:54:23.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.677+0000 7f96c12f0700 1 -- 172.21.15.50:0/23022447 wait complete. 2023-12-19T15:54:23.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 Processor -- start 2023-12-19T15:54:23.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 -- start start 2023-12-19T15:54:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0b08d0 0x7f96bc0a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 0x7f96bc0a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc1697c0 con 0x7f96bc0ab870 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0aea00 con 0x7f96bc159fb0 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96c12f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0ae160 con 0x7f96bc0b08d0 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48660/0 (socket says 172.21.15.50:48660) 2023-12-19T15:54:23.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 -- 172.21.15.50:0/3709880567 learned_addr learned my addr 172.21.15.50:0/3709880567 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96bb7fe700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 0x7f96bc0a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96ba7fc700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0b08d0 0x7f96bc0a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 -- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0b08d0 msgr2=0x7f96bc0a9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0b08d0 0x7f96bc0a9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 -- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 msgr2=0x7f96bc0a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 0x7f96bc0a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96baffd700 1 -- 172.21.15.50:0/3709880567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00094f0 con 0x7f96bc0ab870 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.681+0000 7f96bb7fe700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 0x7f96bc0a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:23.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96baffd700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00e950 tx=0x7f96ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00c850 con 0x7f96bc0ab870 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac00c9b0 con 0x7f96bc0ab870 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc0ae440 con 0x7f96bc0ab870 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac00cdb0 con 0x7f96bc0ab870 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc0add20 con 0x7f96bc0ab870 2023-12-19T15:54:23.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc071120 con 0x7f96bc0ab870 2023-12-19T15:54:23.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.685+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f96ac013680 con 0x7f96bc0ab870 2023-12-19T15:54:23.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.689+0000 7f96a2ffd700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 0x7f96a8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:23.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.689+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f96ac07cfc0 con 0x7f96bc0ab870 2023-12-19T15:54:23.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.689+0000 7f96ba7fc700 1 -- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 msgr2=0x7f96a8062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:23.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.689+0000 7f96ba7fc700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 0x7f96a8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:23.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.697+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f96ac04d0d0 con 0x7f96bc0ab870 2023-12-19T15:54:23.784 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.779+0000 7f5df3f67000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:54:23.863 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.857+0000 7fce5e8eb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:54:23.870 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.863+0000 7f5df3f67000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:54:23.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.869+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96bc15d7b0 con 0x7f96bc0ab870 2023-12-19T15:54:23.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.873+0000 7f96a2ffd700 1 -- 172.21.15.50:0/3709880567 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v153) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96ac0506f0 con 0x7f96bc0ab870 2023-12-19T15:54:23.878 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:54:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 msgr2=0x7f96a8062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 0x7f96a8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 msgr2=0x7f96bc0ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00e950 tx=0x7f96ac0091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 shutdown_connections 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96bc159fb0 0x7f96bc0a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f96a80606a0 0x7f96a8062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96bc0b08d0 0x7f96bc0a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 --2- 172.21.15.50:0/3709880567 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96bc0ab870 0x7f96bc0ad7e0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 >> 172.21.15.50:0/3709880567 conn(0x7f96bc0cc580 msgr2=0x7f96bc154f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 shutdown_connections 2023-12-19T15:54:23.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:23.877+0000 7f96c12f0700 1 -- 172.21.15.50:0/3709880567 wait complete. 2023-12-19T15:54:23.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T15:54:23.949 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:23.945+0000 7fce5e8eb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:54:23.972 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:23.967+0000 7f5df3f67000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:54:24.044 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:24.037+0000 7fce5e8eb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:54:24.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:54:24.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:54:24.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.105+0000 7f6629745700 1 Processor -- start 2023-12-19T15:54:24.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.105+0000 7f6629745700 1 -- start start 2023-12-19T15:54:24.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6629745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 0x7f66240cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6629745700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 0x7f662406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6629745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662406d3d0 con 0x7f662406d860 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6629745700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662406e1d0 con 0x7f66240c8b60 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6629745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662406c7c0 con 0x7f66240672f0 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f66227fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f662406d860 0x7f66240c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37752/0 (socket says 172.21.15.50:37752) 2023-12-19T15:54:24.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f66227fc700 1 -- 172.21.15.50:0/1330110648 learned_addr learned my addr 172.21.15.50:0/1330110648 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f66237fe700 1 --2- 172.21.15.50:0/1330110648 >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 0x7f662406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 --2- 172.21.15.50:0/1330110648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 0x7f66240cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6621ffb700 1 -- 172.21.15.50:0/1330110648 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188839420 0 0) 0x7f662406d3d0 con 0x7f662406d860 2023-12-19T15:54:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6621ffb700 1 -- 172.21.15.50:0/1330110648 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f660c003570 con 0x7f662406d860 2023-12-19T15:54:24.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 -- 172.21.15.50:0/1330110648 >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 msgr2=0x7f662406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 --2- 172.21.15.50:0/1330110648 >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 0x7f662406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 -- 172.21.15.50:0/1330110648 >> v1:172.21.15.50:6789/0 conn(0x7f662406d860 legacy=0x7f66240c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 -- 172.21.15.50:0/1330110648 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662406caa0 con 0x7f66240672f0 2023-12-19T15:54:24.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f66237fe700 1 --2- 172.21.15.50:0/1330110648 >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 0x7f662406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6622ffd700 1 --2- 172.21.15.50:0/1330110648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 0x7f66240cc170 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f66140098e0 tx=0x7f66140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbc07201dffb5c11 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6621ffb700 1 -- 172.21.15.50:0/1330110648 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614011630 con 0x7f66240672f0 2023-12-19T15:54:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6621ffb700 1 -- 172.21.15.50:0/1330110648 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614011790 con 0x7f66240672f0 2023-12-19T15:54:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.109+0000 7f6621ffb700 1 -- 172.21.15.50:0/1330110648 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614011b30 con 0x7f66240672f0 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 -- 172.21.15.50:0/1330110648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 msgr2=0x7f66240cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 --2- 172.21.15.50:0/1330110648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 0x7f66240cc170 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f66140098e0 tx=0x7f66140091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 -- 172.21.15.50:0/1330110648 shutdown_connections 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 --2- 172.21.15.50:0/1330110648 >> v2:172.21.15.125:3300/0 conn(0x7f66240c8b60 0x7f662406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 --2- 172.21.15.50:0/1330110648 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f66240672f0 0x7f66240cc170 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 -- 172.21.15.50:0/1330110648 >> 172.21.15.50:0/1330110648 conn(0x7f66240cdc00 msgr2=0x7f66240ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 -- 172.21.15.50:0/1330110648 shutdown_connections 2023-12-19T15:54:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.113+0000 7f6629745700 1 -- 172.21.15.50:0/1330110648 wait complete. 2023-12-19T15:54:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 Processor -- start 2023-12-19T15:54:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 -- start start 2023-12-19T15:54:24.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 0x7f6624151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f662406d860 0x7f66240b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240690a0 con 0x7f66240c8b60 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6624068ae0 con 0x7f66240672f0 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6629745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6624068dc0 con 0x7f662406d860 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48676/0 (socket says 172.21.15.50:48676) 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6622ffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 0x7f6624151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 -- 172.21.15.50:0/3061078784 learned_addr learned my addr 172.21.15.50:0/3061078784 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66227fc700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f662406d860 0x7f66240b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 -- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f662406d860 msgr2=0x7f66240b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f662406d860 0x7f66240b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 -- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 msgr2=0x7f6624151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 0x7f6624151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f66237fe700 1 -- 172.21.15.50:0/3061078784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c009a20 con 0x7f66240c8b60 2023-12-19T15:54:24.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.117+0000 7f6622ffd700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 0x7f6624151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:24.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f66237fe700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f661c009f20 tx=0x7f661c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00cb70 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00f040 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6614009420 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c014480 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662415a7d0 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662415a350 con 0x7f66240c8b60 2023-12-19T15:54:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.121+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f661c003020 con 0x7f66240c8b60 2023-12-19T15:54:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.125+0000 7f660affd700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 0x7f6610062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.125+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f661c07d5f0 con 0x7f66240c8b60 2023-12-19T15:54:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.125+0000 7f6622ffd700 1 -- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 msgr2=0x7f6610062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.125+0000 7f6622ffd700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 0x7f6610062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:24.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.129+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f662415a350 con 0x7f66240c8b60 2023-12-19T15:54:24.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.305+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f662415a350 con 0x7f66240c8b60 2023-12-19T15:54:24.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f660affd700 1 -- 172.21.15.50:0/3061078784 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v153) v1 ==== 104+0+11 (secure 0 0 0) 0x7f662415a350 con 0x7f66240c8b60 2023-12-19T15:54:24.313 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T15:54:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 msgr2=0x7f6610062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 0x7f6610062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 msgr2=0x7f66240b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f661c009f20 tx=0x7f661c00b040 comp rx=0 tx=0).stop 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 shutdown_connections 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6610060670 0x7f6610062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66240c8b60 0x7f66240b1980 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f662406d860 0x7f66240b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 --2- 172.21.15.50:0/3061078784 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66240672f0 0x7f6624151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.309+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 >> 172.21.15.50:0/3061078784 conn(0x7f66240cdc00 msgr2=0x7f66241538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.313+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 shutdown_connections 2023-12-19T15:54:24.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.313+0000 7f6629745700 1 -- 172.21.15.50:0/3061078784 wait complete. 2023-12-19T15:54:24.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T15:54:24.392 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:24.387+0000 7f5df3f67000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:54:24.476 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:24.469+0000 7fce5e8eb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:54:24.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.501+0000 7fdfc1785700 1 Processor -- start 2023-12-19T15:54:24.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.501+0000 7fdfc1785700 1 -- start start 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 0x7fdfbc0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 0x7fdfbc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfbc0cbdf0 con 0x7fdfbc0cb970 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfbc0cc0d0 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfc1785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc0cc3b0 con 0x7fdfbc158e20 2023-12-19T15:54:24.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48686/0 (socket says 172.21.15.50:48686) 2023-12-19T15:54:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 -- 172.21.15.50:0/2844424456 learned_addr learned my addr 172.21.15.50:0/2844424456 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfbb7fe700 1 --2- 172.21.15.50:0/2844424456 >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 0x7fdfbc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfbaffd700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 0x7fdfbc0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 -- 172.21.15.50:0/2844424456 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 msgr2=0x7fdfbc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 0x7fdfbc0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 -- 172.21.15.50:0/2844424456 >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 msgr2=0x7fdfbc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/2844424456 >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 0x7fdfbc06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 -- 172.21.15.50:0/2844424456 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc0cc690 con 0x7fdfbc0cb970 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfbaffd700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 0x7fdfbc0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfbb7fe700 1 --2- 172.21.15.50:0/2844424456 >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 0x7fdfbc06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0cfd10 tx=0x7fdfb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d8852dc4bd6576 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfb9ffb700 1 -- 172.21.15.50:0/2844424456 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0009e70 con 0x7fdfbc0cb970 2023-12-19T15:54:24.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfb9ffb700 1 -- 172.21.15.50:0/2844424456 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb000b860 con 0x7fdfbc0cb970 2023-12-19T15:54:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.505+0000 7fdfb9ffb700 1 -- 172.21.15.50:0/2844424456 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000bc00 con 0x7fdfbc0cb970 2023-12-19T15:54:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- 172.21.15.50:0/2844424456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0cb970 msgr2=0x7fdfbc0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0cfd10 tx=0x7fdfb0009250 comp rx=0 tx=0).stop 2023-12-19T15:54:24.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- 172.21.15.50:0/2844424456 shutdown_connections 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0cb970 0x7fdfbc0ceb00 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- 172.21.15.50:0/2844424456 >> v2:172.21.15.125:3300/0 conn(0x7fdfbc0ca2c0 0x7fdfbc06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- 172.21.15.50:0/2844424456 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc158e20 0x7fdfbc0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- 172.21.15.50:0/2844424456 >> 172.21.15.50:0/2844424456 conn(0x7fdfbc067bb0 msgr2=0x7fdfbc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- 172.21.15.50:0/2844424456 shutdown_connections 2023-12-19T15:54:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- 172.21.15.50:0/2844424456 wait complete. 2023-12-19T15:54:24.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 Processor -- start 2023-12-19T15:54:24.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- start start 2023-12-19T15:54:24.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 0x7fdfbc0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 0x7fdfbc0c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc0b4280 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc0b39e0 con 0x7fdfbc158e20 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfc1785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc0b3cc0 con 0x7fdfbc0b34f0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48696/0 (socket says 172.21.15.50:48696) 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.509+0000 7fdfbaffd700 1 -- 172.21.15.50:0/1101424672 learned_addr learned my addr 172.21.15.50:0/1101424672 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 0x7fdfbc0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbb7fe700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 0x7fdfbc0c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 -- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 msgr2=0x7fdfbc0c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 0x7fdfbc0c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 -- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 msgr2=0x7fdfbc0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 0x7fdfbc0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 -- 172.21.15.50:0/1101424672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb4009a20 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbb7fe700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 0x7fdfbc0c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 0x7fdfbc0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfbaffd700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00d860 tx=0x7fdfac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0125f0 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac012750 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb00094f0 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac012ad0 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfbc0c8520 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.513+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfbc158120 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.517+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fdfac014430 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.517+0000 7fdfa2ffd700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 0x7fdfa8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.517+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fdfac07d1e0 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.517+0000 7fdfba7fc700 1 -- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 msgr2=0x7fdfa8062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:24.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.517+0000 7fdfba7fc700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 0x7fdfa8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:24.531 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.525+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfbc158120 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.571 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:24.567+0000 7f5df3f67000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:54:24.654 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:24.649+0000 7fce5e8eb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:54:24.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.705+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdfbc158120 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.705+0000 7fdfa2ffd700 1 -- 172.21.15.50:0/1101424672 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v154) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdfac050d80 con 0x7fdfbc0ca2c0 2023-12-19T15:54:24.711 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 msgr2=0x7fdfa8062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 0x7fdfa8062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 msgr2=0x7fdfbc151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00d860 tx=0x7fdfac008040 comp rx=0 tx=0).stop 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 shutdown_connections 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfbc158e20 0x7fdfbc0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdfa8060670 0x7fdfa8062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfbc0b34f0 0x7fdfbc0c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 --2- 172.21.15.50:0/1101424672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfbc0ca2c0 0x7fdfbc151b80 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 >> 172.21.15.50:0/1101424672 conn(0x7fdfbc067bb0 msgr2=0x7fdfbc15bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 shutdown_connections 2023-12-19T15:54:24.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.709+0000 7fdfc1785700 1 -- 172.21.15.50:0/1101424672 wait complete. 2023-12-19T15:54:24.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 Processor -- start 2023-12-19T15:54:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 -- start start 2023-12-19T15:54:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 0x7f443c0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 0x7f443c06a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c06c7c0 con 0x7f443c15a860 2023-12-19T15:54:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c06cd80 con 0x7f443c0b1410 2023-12-19T15:54:24.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c06af10 con 0x7f443c06a580 2023-12-19T15:54:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443a59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 0x7f443c0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443a59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 0x7f443c0ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48702/0 (socket says 172.21.15.50:48702) 2023-12-19T15:54:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f4439d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 0x7f443c06a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443a59c700 1 -- 172.21.15.50:0/601880803 learned_addr learned my addr 172.21.15.50:0/601880803 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:24.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.901+0000 7f443ad9d700 1 --1- 172.21.15.50:0/601880803 >> v1:172.21.15.125:6789/0 conn(0x7f443c15a860 0x7f443c15aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33734/0 (socket says 172.21.15.50:33734) 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f4439d9b700 1 -- 172.21.15.50:0/601880803 >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 msgr2=0x7f443c0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443959a700 1 -- 172.21.15.50:0/601880803 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974306454 0 0) 0x7f443c06c7c0 con 0x7f443c15a860 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f4439d9b700 1 --2- 172.21.15.50:0/601880803 >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 0x7f443c0ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f4439d9b700 1 -- 172.21.15.50:0/601880803 >> v1:172.21.15.125:6789/0 conn(0x7f443c15a860 legacy=0x7f443c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f4439d9b700 1 -- 172.21.15.50:0/601880803 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c06b8f0 con 0x7f443c06a580 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f4439d9b700 1 --2- 172.21.15.50:0/601880803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 0x7f443c06a9d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f44300098e0 tx=0x7f44300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfa3113a96f7a1f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443959a700 1 -- 172.21.15.50:0/601880803 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000b640 con 0x7f443c06a580 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443959a700 1 -- 172.21.15.50:0/601880803 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443000b7a0 con 0x7f443c06a580 2023-12-19T15:54:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443959a700 1 -- 172.21.15.50:0/601880803 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000bb40 con 0x7f443c06a580 2023-12-19T15:54:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 -- 172.21.15.50:0/601880803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 msgr2=0x7f443c06a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 --2- 172.21.15.50:0/601880803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 0x7f443c06a9d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f44300098e0 tx=0x7f44300091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 -- 172.21.15.50:0/601880803 shutdown_connections 2023-12-19T15:54:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 --2- 172.21.15.50:0/601880803 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c06a580 0x7f443c06a9d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 --2- 172.21.15.50:0/601880803 >> v2:172.21.15.50:3300/0 conn(0x7f443c0b1410 0x7f443c0ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 -- 172.21.15.50:0/601880803 >> 172.21.15.50:0/601880803 conn(0x7f443c067bb0 msgr2=0x7f443c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:24.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 -- 172.21.15.50:0/601880803 shutdown_connections 2023-12-19T15:54:24.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.905+0000 7f443b59e700 1 -- 172.21.15.50:0/601880803 wait complete. 2023-12-19T15:54:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 Processor -- start 2023-12-19T15:54:24.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 -- start start 2023-12-19T15:54:24.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c06a580 0x7f443c157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c15a860 0x7f443c0c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0c2de0 con 0x7f443c0b1410 2023-12-19T15:54:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0c4810 con 0x7f443c06a580 2023-12-19T15:54:24.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c157860 con 0x7f443c15a860 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48718/0 (socket says 172.21.15.50:48718) 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 -- 172.21.15.50:0/2873452698 learned_addr learned my addr 172.21.15.50:0/2873452698 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443ad9d700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c15a860 0x7f443c0c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443a59c700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c06a580 0x7f443c157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c15a860 msgr2=0x7f443c0c1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c15a860 0x7f443c0c1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c06a580 msgr2=0x7f443c157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c06a580 0x7f443c157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 -- 172.21.15.50:0/2873452698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430009420 con 0x7f443c0b1410 2023-12-19T15:54:24.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f4439d9b700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f4430009f80 tx=0x7f4430017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000bb20 con 0x7f443c0b1410 2023-12-19T15:54:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4430000c90 con 0x7f443c0b1410 2023-12-19T15:54:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c15a270 con 0x7f443c0b1410 2023-12-19T15:54:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.909+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430011630 con 0x7f443c0b1410 2023-12-19T15:54:24.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.913+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c0c2b00 con 0x7f443c0b1410 2023-12-19T15:54:24.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.913+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c0c49a0 con 0x7f443c0b1410 2023-12-19T15:54:24.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.913+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f44300119a0 con 0x7f443c0b1410 2023-12-19T15:54:24.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.917+0000 7f442a7fc700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 0x7f441c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:24.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.917+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f443007e7b0 con 0x7f443c0b1410 2023-12-19T15:54:24.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.917+0000 7f443a59c700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 msgr2=0x7f441c062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:24.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.917+0000 7f443a59c700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 0x7f441c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:24.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:24.921+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f443004e940 con 0x7f443c0b1410 2023-12-19T15:54:24.951 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:24.943+0000 7f5df3f67000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:54:25.057 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.053+0000 7fce5e8eb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:54:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.121+0000 7f443a59c700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 msgr2=0x7f441c062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:25.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.121+0000 7f443a59c700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 0x7f441c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.129+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f443c04be40 con 0x7f443c0b1410 2023-12-19T15:54:25.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.133+0000 7f442a7fc700 1 -- 172.21.15.50:0/2873452698 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f4430051f60 con 0x7f443c0b1410 2023-12-19T15:54:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:25.138 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T15:54:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 msgr2=0x7f441c062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:25.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 0x7f441c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 msgr2=0x7f443c154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f4430009f80 tx=0x7f4430017000 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 shutdown_connections 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f441c0606a0 0x7f441c062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f443c15a860 0x7f443c0c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f443c0b1410 0x7f443c154280 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 --2- 172.21.15.50:0/2873452698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c06a580 0x7f443c157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 >> 172.21.15.50:0/2873452698 conn(0x7f443c067bb0 msgr2=0x7f443c0adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 shutdown_connections 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.137+0000 7f443b59e700 1 -- 172.21.15.50:0/2873452698 wait complete. 2023-12-19T15:54:25.144 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T15:54:25.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:25.181 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.175+0000 7f5df3f67000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:25.281 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.275+0000 7f5df3f67000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:54:25.292 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.285+0000 7fce5e8eb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.321+0000 7f4e5566e700 1 Processor -- start 2023-12-19T15:54:25.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.321+0000 7f4e5566e700 1 -- start start 2023-12-19T15:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e5566e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e5006b090 0x7f4e5006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e5566e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e500c7140 0x7f4e50155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e5566e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500c8200 con 0x7f4e500c6ca0 2023-12-19T15:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e5566e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5006b9a0 con 0x7f4e500c7140 2023-12-19T15:54:25.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e5566e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e500c7680 con 0x7f4e5006b090 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e500c7140 0x7f4e50155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4e500c7140 0x7f4e50155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48724/0 (socket says 172.21.15.50:48724) 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 -- 172.21.15.50:0/1663091267 learned_addr learned my addr 172.21.15.50:0/1663091267 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4effd700 1 --2- 172.21.15.50:0/1663091267 >> v2:172.21.15.125:3300/0 conn(0x7f4e5006b090 0x7f4e5006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 -- 172.21.15.50:0/1663091267 >> v2:172.21.15.125:3300/0 conn(0x7f4e5006b090 msgr2=0x7f4e5006b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 --2- 172.21.15.50:0/1663091267 >> v2:172.21.15.125:3300/0 conn(0x7f4e5006b090 0x7f4e5006b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 -- 172.21.15.50:0/1663091267 >> v1:172.21.15.50:6789/0 conn(0x7f4e500c6ca0 legacy=0x7f4e500ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 -- 172.21.15.50:0/1663091267 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e500c7960 con 0x7f4e500c7140 2023-12-19T15:54:25.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4f7fe700 1 --2- 172.21.15.50:0/1663091267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500c7140 0x7f4e50155850 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4e50068a10 tx=0x7f4e48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be149dbc7eafde27 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:25.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4e7fc700 1 -- 172.21.15.50:0/1663091267 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48009e70 con 0x7f4e500c7140 2023-12-19T15:54:25.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4e7fc700 1 -- 172.21.15.50:0/1663091267 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4800b860 con 0x7f4e500c7140 2023-12-19T15:54:25.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.325+0000 7f4e4e7fc700 1 -- 172.21.15.50:0/1663091267 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4800bc00 con 0x7f4e500c7140 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 -- 172.21.15.50:0/1663091267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500c7140 msgr2=0x7f4e50155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 --2- 172.21.15.50:0/1663091267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500c7140 0x7f4e50155850 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4e50068a10 tx=0x7f4e48009250 comp rx=0 tx=0).stop 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 -- 172.21.15.50:0/1663091267 shutdown_connections 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 --2- 172.21.15.50:0/1663091267 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e500c7140 0x7f4e50155850 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 --2- 172.21.15.50:0/1663091267 >> v2:172.21.15.125:3300/0 conn(0x7f4e5006b090 0x7f4e5006b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 -- 172.21.15.50:0/1663091267 >> 172.21.15.50:0/1663091267 conn(0x7f4e50067bb0 msgr2=0x7f4e500cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 -- 172.21.15.50:0/1663091267 shutdown_connections 2023-12-19T15:54:25.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 -- 172.21.15.50:0/1663091267 wait complete. 2023-12-19T15:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.329+0000 7f4e5566e700 1 Processor -- start 2023-12-19T15:54:25.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- start start 2023-12-19T15:54:25.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 0x7f4e50151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e500c7140 0x7f4e501557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5015c550 con 0x7f4e5006b090 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500c4350 con 0x7f4e500c7140 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e500c3740 con 0x7f4e500c6ca0 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48730/0 (socket says 172.21.15.50:48730) 2023-12-19T15:54:25.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 -- 172.21.15.50:0/613305444 learned_addr learned my addr 172.21.15.50:0/613305444 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:25.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4f7fe700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e500c7140 0x7f4e501557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 -- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 msgr2=0x7f4e50151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e47fff700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 0x7f4e50151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 0x7f4e50151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 -- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e500c7140 msgr2=0x7f4e501557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e500c7140 0x7f4e501557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 -- 172.21.15.50:0/613305444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480094f0 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e47fff700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 0x7f4e50151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e4effd700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e930 tx=0x7f4e400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000eed0 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000c850 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e500c3a20 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000cbf0 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.333+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e500c3fe0 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.337+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5004be40 con 0x7f4e5006b090 2023-12-19T15:54:25.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.337+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f4e40013470 con 0x7f4e5006b090 2023-12-19T15:54:25.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.341+0000 7f4e46ffd700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 0x7f4e30062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.341+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f4e4007cf50 con 0x7f4e5006b090 2023-12-19T15:54:25.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.341+0000 7f4e47fff700 1 -- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 msgr2=0x7f4e30062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:25.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.341+0000 7f4e47fff700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 0x7f4e30062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:25.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.345+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e4004d060 con 0x7f4e5006b090 2023-12-19T15:54:25.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.355+0000 7f5df3f67000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:54:25.389 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.385+0000 7fce5e8eb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:54:25.446 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.439+0000 7f5df3f67000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:54:25.473 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.465+0000 7fce5e8eb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:54:25.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.521+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e500c7c40 con 0x7f4e5006b090 2023-12-19T15:54:25.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.525+0000 7f4e46ffd700 1 -- 172.21.15.50:0/613305444 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f4e40050680 con 0x7f4e5006b090 2023-12-19T15:54:25.531 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:25.531 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":154,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:24.675979+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":153,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/4275867050":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/1673033377":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2615455122":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/70137828":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2692951617":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6833/33827":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/30212990":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6832/33827":"2023-12-20T14:58:39.674128+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":26,"length":32},{"begin":82,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":18,"snaps":[{"begin":3,"length":23},{"begin":59,"length":23}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:25.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 msgr2=0x7f4e30062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 0x7f4e30062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 msgr2=0x7f4e5015c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000e930 tx=0x7f4e400091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 shutdown_connections 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e500c7140 0x7f4e501557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4e30060670 0x7f4e30062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4e500c6ca0 0x7f4e50151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 --2- 172.21.15.50:0/613305444 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e5006b090 0x7f4e5015c010 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 >> 172.21.15.50:0/613305444 conn(0x7f4e50067bb0 msgr2=0x7f4e5014ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 shutdown_connections 2023-12-19T15:54:25.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.529+0000 7f4e5566e700 1 -- 172.21.15.50:0/613305444 wait complete. 2023-12-19T15:54:25.557 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T15:54:25.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:25.563 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.557+0000 7fce5e8eb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:54:25.684 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.679+0000 7f5df3f67000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:25.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 Processor -- start 2023-12-19T15:54:25.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 -- start start 2023-12-19T15:54:25.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1eb00b0110 0x7f1eb00b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1eb00ccfd0 0x7f1eb015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00ccb40 con 0x7f1eb0153290 2023-12-19T15:54:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00cd470 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb84f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00cbf30 con 0x7f1eb00b0110 2023-12-19T15:54:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb5a8b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1eb00ccfd0 0x7f1eb015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb5a8b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1eb00ccfd0 0x7f1eb015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48746/0 (socket says 172.21.15.50:48746) 2023-12-19T15:54:25.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/874303045 learned_addr learned my addr 172.21.15.50:0/874303045 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:25.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.721+0000 7f1eb628c700 1 --2- 172.21.15.50:0/874303045 >> v2:172.21.15.125:3300/0 conn(0x7f1eb00b0110 0x7f1eb00b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/874303045 >> v2:172.21.15.125:3300/0 conn(0x7f1eb00b0110 msgr2=0x7f1eb00b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb5a8b700 1 --2- 172.21.15.50:0/874303045 >> v2:172.21.15.125:3300/0 conn(0x7f1eb00b0110 0x7f1eb00b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/874303045 >> v1:172.21.15.125:6789/0 conn(0x7f1eb0153290 legacy=0x7f1eb0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/874303045 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00cc210 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb5a8b700 1 --2- 172.21.15.50:0/874303045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb00ccfd0 0x7f1eb015a170 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0064900 tx=0x7f1ea0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1aa7b877899e04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb528a700 1 -- 172.21.15.50:0/874303045 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0009e70 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb528a700 1 -- 172.21.15.50:0/874303045 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000b860 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb528a700 1 -- 172.21.15.50:0/874303045 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000bc00 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 -- 172.21.15.50:0/874303045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb00ccfd0 msgr2=0x7f1eb015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/874303045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb00ccfd0 0x7f1eb015a170 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0064900 tx=0x7f1ea0009250 comp rx=0 tx=0).stop 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 -- 172.21.15.50:0/874303045 shutdown_connections 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/874303045 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb00ccfd0 0x7f1eb015a170 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/874303045 >> v2:172.21.15.125:3300/0 conn(0x7f1eb00b0110 0x7f1eb00b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 -- 172.21.15.50:0/874303045 >> 172.21.15.50:0/874303045 conn(0x7f1eb006c7c0 msgr2=0x7f1eb006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 -- 172.21.15.50:0/874303045 shutdown_connections 2023-12-19T15:54:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.725+0000 7f1eb84f0700 1 -- 172.21.15.50:0/874303045 wait complete. 2023-12-19T15:54:25.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.729+0000 7f1eb84f0700 1 Processor -- start 2023-12-19T15:54:25.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.729+0000 7f1eb84f0700 1 -- start start 2023-12-19T15:54:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1eb00ccfd0 0x7f1eb0159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 0x7f1eb015a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb01698a0 con 0x7f1eb0153290 2023-12-19T15:54:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb015b600 con 0x7f1eb00b0110 2023-12-19T15:54:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb015a9f0 con 0x7f1eb00ccfd0 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb6a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 0x7f1eb015a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb5a8b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1eb00ccfd0 0x7f1eb0159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56434/0 (socket says 172.21.15.50:56434) 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 -- 172.21.15.50:0/3081315031 learned_addr learned my addr 172.21.15.50:0/3081315031 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1eb00ccfd0 msgr2=0x7f1eb0159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1eb00ccfd0 0x7f1eb0159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 msgr2=0x7f1eb015a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 0x7f1eb015a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 -- 172.21.15.50:0/3081315031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00094f0 con 0x7f1eb00b0110 2023-12-19T15:54:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb6a8d700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 0x7f1eb015a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb628c700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009e80 tx=0x7f1eac00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00caa0 con 0x7f1eb00b0110 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00f040 con 0x7f1eb00b0110 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00ce50 con 0x7f1eb00b0110 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb015acd0 con 0x7f1eb00b0110 2023-12-19T15:54:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.733+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb015b290 con 0x7f1eb00b0110 2023-12-19T15:54:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.737+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb0071120 con 0x7f1eb00b0110 2023-12-19T15:54:25.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.737+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f1eac014690 con 0x7f1eb00b0110 2023-12-19T15:54:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.737+0000 7f1ea67fc700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 0x7f1e94062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.737+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f1eac07d640 con 0x7f1eb00b0110 2023-12-19T15:54:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.737+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 msgr2=0x7f1e94062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:25.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.741+0000 7f1eb5a8b700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 0x7f1e94062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:25.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.749+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1eac051170 con 0x7f1eb00b0110 2023-12-19T15:54:25.763 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.755+0000 7f5df3f67000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:54:25.809 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.805+0000 7fce5e8eb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:25.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:25.851+0000 7f5df3f67000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:54:25.895 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.889+0000 7fce5e8eb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:54:25.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.945+0000 7f1eb5a8b700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 msgr2=0x7f1e94062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:25.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.945+0000 7f1eb5a8b700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 0x7f1e94062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:25.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.953+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb015c810 con 0x7f1eb00b0110 2023-12-19T15:54:25.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.957+0000 7f1ea67fc700 1 -- 172.21.15.50:0/3081315031 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f1eac050d00 con 0x7f1eb00b0110 2023-12-19T15:54:25.962 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:25.962 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.957+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 msgr2=0x7f1e94062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.957+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 0x7f1e94062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.957+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 msgr2=0x7f1eb01519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.957+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009e80 tx=0x7f1eac00b040 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 shutdown_connections 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1e94060670 0x7f1e94062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1eb0153290 0x7f1eb015a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1eb00ccfd0 0x7f1eb0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 --2- 172.21.15.50:0/3081315031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00b0110 0x7f1eb01519f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:25.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 >> 172.21.15.50:0/3081315031 conn(0x7f1eb006c7c0 msgr2=0x7f1eb00c3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 shutdown_connections 2023-12-19T15:54:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:25.961+0000 7f1eb84f0700 1 -- 172.21.15.50:0/3081315031 wait complete. 2023-12-19T15:54:25.966 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T15:54:25.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:25.990 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:25.989+0000 7fce5e8eb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:54:26.061 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.055+0000 7f5df3f67000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:54:26.151 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.143+0000 7f5df3f67000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:54:26.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 Processor -- start 2023-12-19T15:54:26.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 -- start start 2023-12-19T15:54:26.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 0x7f82a00ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82a006a580 0x7f82a006a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a006c7c0 con 0x7f82a015a860 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a006cd80 con 0x7f82a006a580 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a6a6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a006af10 con 0x7f82a00b1410 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a5a69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 0x7f82a00ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a5268700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82a006a580 0x7f82a006a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.149+0000 7f82a626a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f82a015a860 0x7f82a015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:33754/0 (socket says 172.21.15.50:33754) 2023-12-19T15:54:26.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a626a700 1 -- 172.21.15.50:0/1145093370 learned_addr learned my addr 172.21.15.50:0/1145093370 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a4a67700 1 -- 172.21.15.50:0/1145093370 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845434265 0 0) 0x7f82a006c7c0 con 0x7f82a015a860 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a4a67700 1 -- 172.21.15.50:0/1145093370 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828c003570 con 0x7f82a015a860 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5268700 1 -- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 msgr2=0x7f82a00ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5268700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 0x7f82a00ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5268700 1 -- 172.21.15.50:0/1145093370 >> v1:172.21.15.125:6789/0 conn(0x7f82a015a860 legacy=0x7f82a015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5268700 1 -- 172.21.15.50:0/1145093370 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a006b8f0 con 0x7f82a006a580 2023-12-19T15:54:26.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5a69700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 0x7f82a00ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:26.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a5268700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a006a9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f82900098e0 tx=0x7f82900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd952822c0c1779 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:26.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a4a67700 1 -- 172.21.15.50:0/1145093370 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290011630 con 0x7f82a006a580 2023-12-19T15:54:26.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a4a67700 1 -- 172.21.15.50:0/1145093370 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290011790 con 0x7f82a006a580 2023-12-19T15:54:26.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a4a67700 1 -- 172.21.15.50:0/1145093370 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8290011b30 con 0x7f82a006a580 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 -- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 msgr2=0x7f82a006a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a006a9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f82900098e0 tx=0x7f82900091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 -- 172.21.15.50:0/1145093370 shutdown_connections 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a006a9d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/1145093370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a00b1410 0x7f82a00ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.153+0000 7f82a6a6b700 1 -- 172.21.15.50:0/1145093370 >> 172.21.15.50:0/1145093370 conn(0x7f82a0067bb0 msgr2=0x7f82a00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.157+0000 7f82a6a6b700 1 -- 172.21.15.50:0/1145093370 shutdown_connections 2023-12-19T15:54:26.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.157+0000 7f82a6a6b700 1 -- 172.21.15.50:0/1145093370 wait complete. 2023-12-19T15:54:26.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.157+0000 7f82a6a6b700 1 Processor -- start 2023-12-19T15:54:26.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.157+0000 7f82a6a6b700 1 -- start start 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82a00b1410 0x7f82a0157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a015a860 0x7f82a0168d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a015c930 con 0x7f82a00b1410 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0159e00 con 0x7f82a006a580 2023-12-19T15:54:26.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a015a0e0 con 0x7f82a015a860 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5268700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82a00b1410 0x7f82a0157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56438/0 (socket says 172.21.15.50:56438) 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 -- 172.21.15.50:0/2156545568 learned_addr learned my addr 172.21.15.50:0/2156545568 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a626a700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a015a860 0x7f82a0168d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a015a860 msgr2=0x7f82a0168d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a015a860 0x7f82a0168d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82a00b1410 msgr2=0x7f82a0157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82a00b1410 0x7f82a0157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 -- 172.21.15.50:0/2156545568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c009420 con 0x7f82a006a580 2023-12-19T15:54:26.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a5a69700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f829c00ec80 tx=0x7f829c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:26.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c00c850 con 0x7f82a006a580 2023-12-19T15:54:26.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c00c9b0 con 0x7f82a006a580 2023-12-19T15:54:26.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c00cd30 con 0x7f82a006a580 2023-12-19T15:54:26.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8290009420 con 0x7f82a006a580 2023-12-19T15:54:26.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.161+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a0159af0 con 0x7f82a006a580 2023-12-19T15:54:26.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.165+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a015caa0 con 0x7f82a006a580 2023-12-19T15:54:26.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.165+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f829c013680 con 0x7f82a006a580 2023-12-19T15:54:26.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.165+0000 7f8295ffb700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 0x7f8284062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.169+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f829c07cfe0 con 0x7f82a006a580 2023-12-19T15:54:26.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.169+0000 7f82a5268700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 msgr2=0x7f8284062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:26.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.169+0000 7f82a5268700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 0x7f8284062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:26.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.177+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f829c050b80 con 0x7f82a006a580 2023-12-19T15:54:26.210 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:26.205+0000 7fce5e8eb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:54:26.311 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:26.305+0000 7fce5e8eb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:54:26.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.373+0000 7f82a5268700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 msgr2=0x7f8284062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:26.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.373+0000 7f82a5268700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 0x7f8284062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.393+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82a004be40 con 0x7f82a006a580 2023-12-19T15:54:26.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.397+0000 7f8295ffb700 1 -- 172.21.15.50:0/2156545568 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4593 (secure 0 0 0) 0x7f829c050710 con 0x7f82a006a580 2023-12-19T15:54:26.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:26.402 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T14:58:22.983915+0000","modified":"2023-12-19T14:58:26.556533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8379":{"gid":8379,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6837/528853913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":528853913},{"type":"v1","addr":"172.21.15.125:6837","nonce":528853913}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 msgr2=0x7f8284062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 0x7f8284062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 msgr2=0x7f82a00c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f829c00ec80 tx=0x7f829c0091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 shutdown_connections 2023-12-19T15:54:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f82840606a0 0x7f8284062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f82a015a860 0x7f82a0168d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82a00b1410 0x7f82a0157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 --2- 172.21.15.50:0/2156545568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82a006a580 0x7f82a00c7700 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 >> 172.21.15.50:0/2156545568 conn(0x7f82a0067bb0 msgr2=0x7f82a00ab4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 shutdown_connections 2023-12-19T15:54:26.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.401+0000 7f82a6a6b700 1 -- 172.21.15.50:0/2156545568 wait complete. 2023-12-19T15:54:26.408 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 66 2023-12-19T15:54:26.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:26.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.629+0000 7f7d78dc5700 1 Processor -- start 2023-12-19T15:54:26.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 -- start start 2023-12-19T15:54:26.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d74153290 0x7f7d741536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d740b0950 con 0x7f7d740b0110 2023-12-19T15:54:26.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d740cd510 con 0x7f7d74153290 2023-12-19T15:54:26.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d78dc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740ccb40 con 0x7f7d740b04e0 2023-12-19T15:54:26.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d72d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d74153290 0x7f7d741536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d72d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7d74153290 0x7f7d741536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34424/0 (socket says 172.21.15.50:34424) 2023-12-19T15:54:26.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d72d9d700 1 -- 172.21.15.50:0/1474284371 learned_addr learned my addr 172.21.15.50:0/1474284371 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:26.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.633+0000 7f7d6bfff700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d72d9d700 1 -- 172.21.15.50:0/1474284371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 msgr2=0x7f7d740ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d72d9d700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d72d9d700 1 -- 172.21.15.50:0/1474284371 >> v1:172.21.15.50:6789/0 conn(0x7f7d740b0110 legacy=0x7f7d7415a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d72d9d700 1 -- 172.21.15.50:0/1474284371 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d74153c10 con 0x7f7d74153290 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d72d9d700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d74153290 0x7f7d741536d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009f20 tx=0x7f7d6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6acf2002dd7b2c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:26.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d71d9b700 1 -- 172.21.15.50:0/1474284371 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00e070 con 0x7f7d74153290 2023-12-19T15:54:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d71d9b700 1 -- 172.21.15.50:0/1474284371 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c00b9d0 con 0x7f7d74153290 2023-12-19T15:54:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d71d9b700 1 -- 172.21.15.50:0/1474284371 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0092e0 con 0x7f7d74153290 2023-12-19T15:54:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 -- 172.21.15.50:0/1474284371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d74153290 msgr2=0x7f7d741536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d74153290 0x7f7d741536d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009f20 tx=0x7f7d6c009840 comp rx=0 tx=0).stop 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 -- 172.21.15.50:0/1474284371 shutdown_connections 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d74153290 0x7f7d741536d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/1474284371 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 -- 172.21.15.50:0/1474284371 >> 172.21.15.50:0/1474284371 conn(0x7f7d7406c7c0 msgr2=0x7f7d7406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 -- 172.21.15.50:0/1474284371 shutdown_connections 2023-12-19T15:54:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.637+0000 7f7d78dc5700 1 -- 172.21.15.50:0/1474284371 wait complete. 2023-12-19T15:54:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 Processor -- start 2023-12-19T15:54:26.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 -- start start 2023-12-19T15:54:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 0x7f7d740b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74169090 con 0x7f7d740b0110 2023-12-19T15:54:26.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740c7fe0 con 0x7f7d74153290 2023-12-19T15:54:26.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d78dc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7415aa30 con 0x7f7d740b04e0 2023-12-19T15:54:26.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d7259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d7259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34428/0 (socket says 172.21.15.50:34428) 2023-12-19T15:54:26.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.641+0000 7f7d7259c700 1 -- 172.21.15.50:0/2230948283 learned_addr learned my addr 172.21.15.50:0/2230948283 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d6bfff700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d72d9d700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 0x7f7d740b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 -- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 msgr2=0x7f7d740c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 -- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 msgr2=0x7f7d740b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 0x7f7d740b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 -- 172.21.15.50:0/2230948283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c009ae0 con 0x7f7d740b0110 2023-12-19T15:54:26.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d72d9d700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 0x7f7d740b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d6bfff700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d7259c700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c590 tx=0x7f7d6400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d64011590 con 0x7f7d740b0110 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d640116f0 con 0x7f7d740b0110 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d64011ab0 con 0x7f7d740b0110 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7415ad10 con 0x7f7d740b0110 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d740c7cc0 con 0x7f7d740b0110 2023-12-19T15:54:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.645+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d740af730 con 0x7f7d740b0110 2023-12-19T15:54:26.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.649+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 65) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f7d6401b3b0 con 0x7f7d740b0110 2023-12-19T15:54:26.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.653+0000 7f7d6a7fc700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 0x7f7d54062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:26.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.653+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f7d6407c680 con 0x7f7d740b0110 2023-12-19T15:54:26.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.653+0000 7f7d6bfff700 1 -- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 msgr2=0x7f7d54062af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:26.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.653+0000 7f7d6bfff700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 0x7f7d54062af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:26.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.657+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7d6404c8b0 con 0x7f7d740b0110 2023-12-19T15:54:26.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.833+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d74071120 con 0x7f7d740b0110 2023-12-19T15:54:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d6a7fc700 1 -- 172.21.15.50:0/2230948283 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16319 (secure 0 0 0) 0x7f7d64017090 con 0x7f7d740b0110 2023-12-19T15:54:26.841 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:26.841 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":154,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:24.675979+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T14:58:21.215201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T14:58:21.997104+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":153,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/4275867050":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/1673033377":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2615455122":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/70137828":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2692951617":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6833/33827":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/30212990":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6832/33827":"2023-12-20T14:58:39.674128+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":26,"length":32},{"begin":82,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":18,"snaps":[{"begin":3,"length":23},{"begin":59,"length":23}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 msgr2=0x7f7d54062af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 0x7f7d54062af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 msgr2=0x7f7d740c5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c590 tx=0x7f7d6400c950 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 shutdown_connections 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d74153290 0x7f7d740b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7d54060640 0x7f7d54062af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7d740b04e0 0x7f7d740c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 --2- 172.21.15.50:0/2230948283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d740b0110 0x7f7d740c5e00 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 >> 172.21.15.50:0/2230948283 conn(0x7f7d7406c7c0 msgr2=0x7f7d740c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 shutdown_connections 2023-12-19T15:54:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:26.837+0000 7f7d78dc5700 1 -- 172.21.15.50:0/2230948283 wait complete. 2023-12-19T15:54:26.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T15:54:26.892 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.887+0000 7f5df3f67000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:54:26.997 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbbb05700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbeb0b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbeb0b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbeb0b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbeb0b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbeb0b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbbb05700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbbb05700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.998 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbbb05700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:26.999 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:26.991+0000 7f5dbbb05700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 Processor -- start 2023-12-19T15:54:27.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 -- start start 2023-12-19T15:54:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f322c06d760 0x7f322c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c159700 0x7f322c0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c159f90 con 0x7f322c158df0 2023-12-19T15:54:27.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c15a270 con 0x7f322c06d760 2023-12-19T15:54:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0c7930 con 0x7f322c159700 2023-12-19T15:54:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322a59c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f322c158df0 0x7f322c06d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60052/0 (socket says 172.21.15.50:60052) 2023-12-19T15:54:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322a59c700 1 -- 172.21.15.50:0/4055622283 learned_addr learned my addr 172.21.15.50:0/4055622283 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:27.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f3229d9b700 1 --2- 172.21.15.50:0/4055622283 >> v2:172.21.15.125:3300/0 conn(0x7f322c06d760 0x7f322c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322ad9d700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c159700 0x7f322c0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322959a700 1 -- 172.21.15.50:0/4055622283 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331263584 0 0) 0x7f322c159f90 con 0x7f322c158df0 2023-12-19T15:54:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.021+0000 7f322959a700 1 -- 172.21.15.50:0/4055622283 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3214003570 con 0x7f322c158df0 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f3229d9b700 1 -- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c159700 msgr2=0x7f322c0c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f3229d9b700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c159700 0x7f322c0c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f3229d9b700 1 -- 172.21.15.50:0/4055622283 >> v1:172.21.15.50:6789/0 conn(0x7f322c158df0 legacy=0x7f322c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f3229d9b700 1 -- 172.21.15.50:0/4055622283 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0c7c10 con 0x7f322c06d760 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f3229d9b700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c06d760 0x7f322c1591c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32200098e0 tx=0x7f32200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48df90c6f7b6ffdf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322959a700 1 -- 172.21.15.50:0/4055622283 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220011630 con 0x7f322c06d760 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322959a700 1 -- 172.21.15.50:0/4055622283 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3220011790 con 0x7f322c06d760 2023-12-19T15:54:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322959a700 1 -- 172.21.15.50:0/4055622283 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220011b30 con 0x7f322c06d760 2023-12-19T15:54:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 -- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c06d760 msgr2=0x7f322c1591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c06d760 0x7f322c1591c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f32200098e0 tx=0x7f32200091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 -- 172.21.15.50:0/4055622283 shutdown_connections 2023-12-19T15:54:27.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c159700 0x7f322c0c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 --2- 172.21.15.50:0/4055622283 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c06d760 0x7f322c1591c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 -- 172.21.15.50:0/4055622283 >> 172.21.15.50:0/4055622283 conn(0x7f322c06c080 msgr2=0x7f322c0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 -- 172.21.15.50:0/4055622283 shutdown_connections 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.025+0000 7f322b59e700 1 -- 172.21.15.50:0/4055622283 wait complete. 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 Processor -- start 2023-12-19T15:54:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 -- start start 2023-12-19T15:54:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c158df0 0x7f322c14b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c159700 0x7f322c0aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0b2770 con 0x7f322c06d760 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0afdc0 con 0x7f322c159700 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0b0be0 con 0x7f322c158df0 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34430/0 (socket says 172.21.15.50:34430) 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 -- 172.21.15.50:0/3250198025 learned_addr learned my addr 172.21.15.50:0/3250198025 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f3229d9b700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c158df0 0x7f322c14b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322ad9d700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c159700 0x7f322c0aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 -- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c158df0 msgr2=0x7f322c14b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c158df0 0x7f322c14b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 -- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c159700 msgr2=0x7f322c0aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c159700 0x7f322c0aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 -- 172.21.15.50:0/3250198025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220009420 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.029+0000 7f322a59c700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f321c0011e0 tx=0x7f321c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c005b30 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c005c90 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c007300 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c0affd0 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c0b0590 con 0x7f322c06d760 2023-12-19T15:54:27.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.033+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c04be40 con 0x7f322c06d760 2023-12-19T15:54:27.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.037+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 66) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f321c007670 con 0x7f322c06d760 2023-12-19T15:54:27.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.037+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f321c07baa0 con 0x7f322c06d760 2023-12-19T15:54:27.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.041+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f321c04c050 con 0x7f322c06d760 2023-12-19T15:54:27.132 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:27.125+0000 7fce5e8eb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:54:27.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.249+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f322c0cd1f0 con 0x7f322c06d760 2023-12-19T15:54:27.255 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T15:54:27.249+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T15:54:27.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.261+0000 7f321a7fc700 1 -- 172.21.15.50:0/3250198025 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f321c04f670 con 0x7f322c06d760 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 msgr2=0x7f322c0af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f321c0011e0 tx=0x7f321c007df0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 shutdown_connections 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f322c159700 0x7f322c0aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f322c158df0 0x7f322c14b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 --2- 172.21.15.50:0/3250198025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f322c06d760 0x7f322c0af880 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 >> 172.21.15.50:0/3250198025 conn(0x7f322c06c080 msgr2=0x7f322c068500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 shutdown_connections 2023-12-19T15:54:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.265+0000 7f322b59e700 1 -- 172.21.15.50:0/3250198025 wait complete. 2023-12-19T15:54:27.271 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T15:54:27.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T15:54:27.332 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T15:54:27.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.457+0000 7f3fac585700 1 Processor -- start 2023-12-19T15:54:27.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.457+0000 7f3fac585700 1 -- start start 2023-12-19T15:54:27.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fac585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fac585700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3fa406c7c0 0x7f3fa406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fac585700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa406a7a0 con 0x7f3fa415aa30 2023-12-19T15:54:27.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fac585700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa40c7630 con 0x7f3fa406c7c0 2023-12-19T15:54:27.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fac585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa406d4b0 con 0x7f3fa40c6d20 2023-12-19T15:54:27.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54852/0 (socket says 172.21.15.50:54852) 2023-12-19T15:54:27.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 -- 172.21.15.50:0/608468784 learned_addr learned my addr 172.21.15.50:0/608468784 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:27.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3fa9b20700 1 --2- 172.21.15.50:0/608468784 >> v2:172.21.15.125:3300/0 conn(0x7f3fa406c7c0 0x7f3fa406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 -- 172.21.15.50:0/608468784 >> v2:172.21.15.125:3300/0 conn(0x7f3fa406c7c0 msgr2=0x7f3fa406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:27.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 --2- 172.21.15.50:0/608468784 >> v2:172.21.15.125:3300/0 conn(0x7f3fa406c7c0 0x7f3fa406cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 -- 172.21.15.50:0/608468784 >> v1:172.21.15.125:6789/0 conn(0x7f3fa415aa30 legacy=0x7f3fa415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.461+0000 7f3faa321700 1 -- 172.21.15.50:0/608468784 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa406d790 con 0x7f3fa40c6d20 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3faa321700 1 --2- 172.21.15.50:0/608468784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00099b0 tx=0x7f3fa0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5964f47600486e67 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fa931f700 1 -- 172.21.15.50:0/608468784 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0009ea0 con 0x7f3fa40c6d20 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fa931f700 1 -- 172.21.15.50:0/608468784 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa000b860 con 0x7f3fa40c6d20 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fa931f700 1 -- 172.21.15.50:0/608468784 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000bc00 con 0x7f3fa40c6d20 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 -- 172.21.15.50:0/608468784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 msgr2=0x7f3fa40c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 --2- 172.21.15.50:0/608468784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00099b0 tx=0x7f3fa0009250 comp rx=0 tx=0).stop 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 -- 172.21.15.50:0/608468784 shutdown_connections 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 --2- 172.21.15.50:0/608468784 >> v2:172.21.15.125:3300/0 conn(0x7f3fa406c7c0 0x7f3fa406cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 --2- 172.21.15.50:0/608468784 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c70f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 -- 172.21.15.50:0/608468784 >> 172.21.15.50:0/608468784 conn(0x7f3fa4067bb0 msgr2=0x7f3fa40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 -- 172.21.15.50:0/608468784 shutdown_connections 2023-12-19T15:54:27.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.465+0000 7f3fac585700 1 -- 172.21.15.50:0/608468784 wait complete. 2023-12-19T15:54:27.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 Processor -- start 2023-12-19T15:54:27.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 -- start start 2023-12-19T15:54:27.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fa415aa30 0x7f3fa40c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa40c5c50 con 0x7f3fa406c7c0 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa40b1a80 con 0x7f3fa415aa30 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fac585700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa40b0eb0 con 0x7f3fa40c6d20 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34440/0 (socket says 172.21.15.50:34440) 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 -- 172.21.15.50:0/51296636 learned_addr learned my addr 172.21.15.50:0/51296636 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3fa9b20700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faab22700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fa415aa30 0x7f3fa40c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 -- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 msgr2=0x7f3fa40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 -- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fa415aa30 msgr2=0x7f3fa40c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fa415aa30 0x7f3fa40c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.469+0000 7f3faa321700 1 -- 172.21.15.50:0/51296636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00094f0 con 0x7f3fa406c7c0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3faa321700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4069e00 tx=0x7f3fa0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0000d10 con 0x7f3fa406c7c0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa000bbe0 con 0x7f3fa406c7c0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa40c4f30 con 0x7f3fa406c7c0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0011670 con 0x7f3fa406c7c0 2023-12-19T15:54:27.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa40b1430 con 0x7f3fa406c7c0 2023-12-19T15:54:27.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.473+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa4071120 con 0x7f3fa406c7c0 2023-12-19T15:54:27.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.477+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 66) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f3fa0011a60 con 0x7f3fa406c7c0 2023-12-19T15:54:27.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.477+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3fa007dfe0 con 0x7f3fa406c7c0 2023-12-19T15:54:27.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.485+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3fa004e470 con 0x7f3fa406c7c0 2023-12-19T15:54:27.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.689+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fa406e0a0 con 0x7f3fa406c7c0 2023-12-19T15:54:27.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.913+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 67) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f3fa004df20 con 0x7f3fa406c7c0 2023-12-19T15:54:27.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.913+0000 7f3f9a7fc700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 0x7f3f880688c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:27.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.913+0000 7f3fa9b20700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 0x7f3f880688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:27.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:27.917+0000 7f3fa9b20700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 0x7f3f880688c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4159c30 tx=0x7f3f940091a0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:28.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.265+0000 7f3f9a7fc700 1 -- 172.21.15.50:0/51296636 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fa0025940 con 0x7f3fa406c7c0 2023-12-19T15:54:28.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 msgr2=0x7f3f880688c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 0x7f3f880688c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4159c30 tx=0x7f3f940091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 msgr2=0x7f3fa40c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4069e00 tx=0x7f3fa0017000 comp rx=0 tx=0).stop 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 shutdown_connections 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fa415aa30 0x7f3fa40c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3f880664e0 0x7f3f880688c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3fa40c6d20 0x7f3fa40c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 --2- 172.21.15.50:0/51296636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fa406c7c0 0x7f3fa40c5710 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 >> 172.21.15.50:0/51296636 conn(0x7f3fa4067bb0 msgr2=0x7f3fa40c65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:28.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 shutdown_connections 2023-12-19T15:54:28.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.269+0000 7f3fac585700 1 -- 172.21.15.50:0/51296636 wait complete. 2023-12-19T15:54:28.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T15:54:28.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.465+0000 7f0c9882e700 1 Processor -- start 2023-12-19T15:54:28.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.465+0000 7f0c9882e700 1 -- start start 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9882e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c940abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9882e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0c940ac480 0x7f0c940aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9882e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c940ac8e0 con 0x7f0c94156e70 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9882e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9406d7b0 con 0x7f0c940ac480 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9882e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940ae570 con 0x7f0c9406dc40 2023-12-19T15:54:28.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0c94156e70 0x7f0c9406b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60068/0 (socket says 172.21.15.50:60068) 2023-12-19T15:54:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9259c700 1 -- 172.21.15.50:0/328302602 learned_addr learned my addr 172.21.15.50:0/328302602 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c92d9d700 1 --2- 172.21.15.50:0/328302602 >> v2:172.21.15.125:3300/0 conn(0x7f0c940ac480 0x7f0c940aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/328302602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c940abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9159a700 1 -- 172.21.15.50:0/328302602 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241102838 0 0) 0x7f0c84001110 con 0x7f0c94156e70 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c9159a700 1 -- 172.21.15.50:0/328302602 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7c003570 con 0x7f0c94156e70 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c91d9b700 1 -- 172.21.15.50:0/328302602 >> v2:172.21.15.125:3300/0 conn(0x7f0c940ac480 msgr2=0x7f0c940aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/328302602 >> v2:172.21.15.125:3300/0 conn(0x7f0c940ac480 0x7f0c940aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c91d9b700 1 -- 172.21.15.50:0/328302602 >> v1:172.21.15.50:6789/0 conn(0x7f0c94156e70 legacy=0x7f0c9406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.469+0000 7f0c91d9b700 1 -- 172.21.15.50:0/328302602 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9406cba0 con 0x7f0c9406dc40 2023-12-19T15:54:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/328302602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c940abf40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0c880098e0 tx=0x7f0c880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ff273b9ad56ed8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9159a700 1 -- 172.21.15.50:0/328302602 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88011630 con 0x7f0c9406dc40 2023-12-19T15:54:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9159a700 1 -- 172.21.15.50:0/328302602 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88011790 con 0x7f0c9406dc40 2023-12-19T15:54:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9159a700 1 -- 172.21.15.50:0/328302602 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88011b30 con 0x7f0c9406dc40 2023-12-19T15:54:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 -- 172.21.15.50:0/328302602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 msgr2=0x7f0c940abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 --2- 172.21.15.50:0/328302602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c940abf40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0c880098e0 tx=0x7f0c880091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 -- 172.21.15.50:0/328302602 shutdown_connections 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 --2- 172.21.15.50:0/328302602 >> v2:172.21.15.125:3300/0 conn(0x7f0c940ac480 0x7f0c940aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 --2- 172.21.15.50:0/328302602 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c940abf40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 -- 172.21.15.50:0/328302602 >> 172.21.15.50:0/328302602 conn(0x7f0c94067bb0 msgr2=0x7f0c940cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 -- 172.21.15.50:0/328302602 shutdown_connections 2023-12-19T15:54:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.473+0000 7f0c9882e700 1 -- 172.21.15.50:0/328302602 wait complete. 2023-12-19T15:54:28.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 Processor -- start 2023-12-19T15:54:28.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 -- start start 2023-12-19T15:54:28.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c9414aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c94156e70 0x7f0c940ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9414b1e0 con 0x7f0c940ac480 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c941550c0 con 0x7f0c94156e70 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.477+0000 7f0c9882e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c941553a0 con 0x7f0c9406dc40 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c9259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c9414aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34448/0 (socket says 172.21.15.50:34448) 2023-12-19T15:54:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 -- 172.21.15.50:0/1205231953 learned_addr learned my addr 172.21.15.50:0/1205231953 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c92d9d700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c94156e70 0x7f0c940ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 -- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 msgr2=0x7f0c9414aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c9414aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 -- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c94156e70 msgr2=0x7f0c940ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c94156e70 0x7f0c940ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 -- 172.21.15.50:0/1205231953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c88009420 con 0x7f0c940ac480 2023-12-19T15:54:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c9259c700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c9414aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c91d9b700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ae6b0 tx=0x7f0c88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88011900 con 0x7f0c940ac480 2023-12-19T15:54:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88011a60 con 0x7f0c940ac480 2023-12-19T15:54:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c94155680 con 0x7f0c940ac480 2023-12-19T15:54:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88025640 con 0x7f0c940ac480 2023-12-19T15:54:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.481+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c94155c40 con 0x7f0c940ac480 2023-12-19T15:54:28.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.485+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c94155e60 con 0x7f0c940ac480 2023-12-19T15:54:28.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.485+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 67) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f0c88025a30 con 0x7f0c940ac480 2023-12-19T15:54:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.485+0000 7f0c827fc700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 0x7f0c74062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.485+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0c8807e120 con 0x7f0c940ac480 2023-12-19T15:54:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.489+0000 7f0c9259c700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 0x7f0c74062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:28.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.489+0000 7f0c9259c700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 0x7f0c74062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c84001270 tx=0x7f0c84007bf0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:28.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.493+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0c8804e4a0 con 0x7f0c940ac480 2023-12-19T15:54:28.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:28.665+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c94071120 con 0x7f0c940ac480 2023-12-19T15:54:29.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.269+0000 7f0c827fc700 1 -- 172.21.15.50:0/1205231953 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v157) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0c88051ac0 con 0x7f0c940ac480 2023-12-19T15:54:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 msgr2=0x7f0c74062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 0x7f0c74062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0c84001270 tx=0x7f0c84007bf0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 msgr2=0x7f0c94154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ae6b0 tx=0x7f0c88017000 comp rx=0 tx=0).stop 2023-12-19T15:54:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 shutdown_connections 2023-12-19T15:54:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c94156e70 0x7f0c940ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0c740603a0 0x7f0c74062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c940ac480 0x7f0c94154620 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 --2- 172.21.15.50:0/1205231953 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0c9406dc40 0x7f0c9414aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 >> 172.21.15.50:0/1205231953 conn(0x7f0c94067bb0 msgr2=0x7f0c940c95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:29.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 shutdown_connections 2023-12-19T15:54:29.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.277+0000 7f0c9882e700 1 -- 172.21.15.50:0/1205231953 wait complete. 2023-12-19T15:54:29.286 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T15:54:29.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T15:54:29.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.481+0000 7f40a226e700 1 Processor -- start 2023-12-19T15:54:29.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.481+0000 7f40a226e700 1 -- start start 2023-12-19T15:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a226e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f409c0cb500 0x7f409c15c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a226e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a226e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0b0ac0 con 0x7f409c0cb130 2023-12-19T15:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a226e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409c15cbb0 con 0x7f409c0cb500 2023-12-19T15:54:29.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a226e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409c06b210 con 0x7f409c15c760 2023-12-19T15:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f409c0cb500 0x7f409c15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f409c0cb500 0x7f409c15c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34456/0 (socket says 172.21.15.50:34456) 2023-12-19T15:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 -- 172.21.15.50:0/3425779490 learned_addr learned my addr 172.21.15.50:0/3425779490 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409bfff700 1 --1- 172.21.15.50:0/3425779490 >> v1:172.21.15.125:6789/0 conn(0x7f409c0cb130 0x7f409c06a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36836/0 (socket says 172.21.15.50:36836) 2023-12-19T15:54:29.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f40a080b700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 -- 172.21.15.50:0/3425779490 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 msgr2=0x7f409c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c0b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 -- 172.21.15.50:0/3425779490 >> v1:172.21.15.125:6789/0 conn(0x7f409c0cb130 legacy=0x7f409c06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 -- 172.21.15.50:0/3425779490 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0ca810 con 0x7f409c0cb500 2023-12-19T15:54:29.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409b7fe700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0cb500 0x7f409c15c220 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f40900099b0 tx=0x7f4090009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a063f23c01db6b0e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409affd700 1 -- 172.21.15.50:0/3425779490 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000b860 con 0x7f409c0cb500 2023-12-19T15:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409affd700 1 -- 172.21.15.50:0/3425779490 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409000b9c0 con 0x7f409c0cb500 2023-12-19T15:54:29.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.485+0000 7f409affd700 1 -- 172.21.15.50:0/3425779490 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000bd60 con 0x7f409c0cb500 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- 172.21.15.50:0/3425779490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0cb500 msgr2=0x7f409c15c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0cb500 0x7f409c15c220 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f40900099b0 tx=0x7f4090009250 comp rx=0 tx=0).stop 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- 172.21.15.50:0/3425779490 shutdown_connections 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 --2- 172.21.15.50:0/3425779490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c0cb500 0x7f409c15c220 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- 172.21.15.50:0/3425779490 >> 172.21.15.50:0/3425779490 conn(0x7f409c067bb0 msgr2=0x7f409c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- 172.21.15.50:0/3425779490 shutdown_connections 2023-12-19T15:54:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- 172.21.15.50:0/3425779490 wait complete. 2023-12-19T15:54:29.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 Processor -- start 2023-12-19T15:54:29.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.489+0000 7f40a226e700 1 -- start start 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f409c0cb130 0x7f409c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c0b4550 con 0x7f409c171a80 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c0b47f0 con 0x7f409c0cb130 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409c15d540 con 0x7f409c15c760 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34472/0 (socket says 172.21.15.50:34472) 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 -- 172.21.15.50:0/2621865922 learned_addr learned my addr 172.21.15.50:0/2621865922 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f409b7fe700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f409bfff700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f409c0cb130 0x7f409c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 -- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 msgr2=0x7f409c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c15ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 -- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f409c0cb130 msgr2=0x7f409c152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f409c0cb130 0x7f409c152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 -- 172.21.15.50:0/2621865922 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40900094f0 con 0x7f409c171a80 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a080b700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4094007f20 tx=0x7f4094009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409400eb90 con 0x7f409c171a80 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409400ed10 con 0x7f409c171a80 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c15d820 con 0x7f409c171a80 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094014670 con 0x7f409c171a80 2023-12-19T15:54:29.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.493+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c152980 con 0x7f409c171a80 2023-12-19T15:54:29.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c0b4980 con 0x7f409c171a80 2023-12-19T15:54:29.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f40940149e0 con 0x7f409c171a80 2023-12-19T15:54:29.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f4083fff700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 0x7f4088062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:29.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f409407d230 con 0x7f409c171a80 2023-12-19T15:54:29.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f409bfff700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 0x7f4088062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:29.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.501+0000 7f409bfff700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 0x7f4088062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b5ed0 tx=0x7f408c007eb0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:29.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.509+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f409404d3d0 con 0x7f409c171a80 2023-12-19T15:54:29.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:29.677+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f409c071120 con 0x7f409c171a80 2023-12-19T15:54:30.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.697+0000 7f4083fff700 1 -- 172.21.15.50:0/2621865922 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v159) v1 ==== 169+0+0 (secure 0 0 0) 0x7f40940509f0 con 0x7f409c171a80 2023-12-19T15:54:30.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.701+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 msgr2=0x7f4088062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.701+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 0x7f4088062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b5ed0 tx=0x7f408c007eb0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.701+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 msgr2=0x7f409c150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.701+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f4094007f20 tx=0x7f4094009860 comp rx=0 tx=0).stop 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 shutdown_connections 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f4088060670 0x7f4088062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f409c171a80 0x7f409c150460 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f409c15c760 0x7f409c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 --2- 172.21.15.50:0/2621865922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f409c0cb130 0x7f409c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 >> 172.21.15.50:0/2621865922 conn(0x7f409c067bb0 msgr2=0x7f409c1585e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 shutdown_connections 2023-12-19T15:54:30.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.705+0000 7f40a226e700 1 -- 172.21.15.50:0/2621865922 wait complete. 2023-12-19T15:54:30.709 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T15:54:30.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T15:54:30.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 Processor -- start 2023-12-19T15:54:30.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 -- start start 2023-12-19T15:54:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5934154780 0x7f593415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f593406baa0 0x7f5934157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593415ae30 con 0x7f59340adab0 2023-12-19T15:54:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59340ae5b0 con 0x7f593406baa0 2023-12-19T15:54:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5939a6b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59340abf70 con 0x7f5934154780 2023-12-19T15:54:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5933fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f593406baa0 0x7f5934157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5933fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f593406baa0 0x7f5934157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34484/0 (socket says 172.21.15.50:34484) 2023-12-19T15:54:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.901+0000 7f5933fff700 1 -- 172.21.15.50:0/2928684424 learned_addr learned my addr 172.21.15.50:0/2928684424 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:30.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5938a69700 1 --2- 172.21.15.50:0/2928684424 >> v2:172.21.15.125:3300/0 conn(0x7f5934154780 0x7f593415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f59337fe700 1 -- 172.21.15.50:0/2928684424 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281681946 0 0) 0x7f593415ae30 con 0x7f59340adab0 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f59337fe700 1 -- 172.21.15.50:0/2928684424 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924003570 con 0x7f59340adab0 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5933fff700 1 -- 172.21.15.50:0/2928684424 >> v2:172.21.15.125:3300/0 conn(0x7f5934154780 msgr2=0x7f593415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5933fff700 1 --2- 172.21.15.50:0/2928684424 >> v2:172.21.15.125:3300/0 conn(0x7f5934154780 0x7f593415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5933fff700 1 -- 172.21.15.50:0/2928684424 >> v1:172.21.15.125:6789/0 conn(0x7f59340adab0 legacy=0x7f59340adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5933fff700 1 -- 172.21.15.50:0/2928684424 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5934154b50 con 0x7f593406baa0 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5933fff700 1 --2- 172.21.15.50:0/2928684424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f593406baa0 0x7f5934157b60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f593406a3f0 tx=0x7f5920009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6651437b290723c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f59337fe700 1 -- 172.21.15.50:0/2928684424 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592000b640 con 0x7f593406baa0 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f59337fe700 1 -- 172.21.15.50:0/2928684424 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592000b7a0 con 0x7f593406baa0 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f59337fe700 1 -- 172.21.15.50:0/2928684424 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592000bb20 con 0x7f593406baa0 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 -- 172.21.15.50:0/2928684424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f593406baa0 msgr2=0x7f5934157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2928684424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f593406baa0 0x7f5934157b60 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f593406a3f0 tx=0x7f5920009250 comp rx=0 tx=0).stop 2023-12-19T15:54:30.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 -- 172.21.15.50:0/2928684424 shutdown_connections 2023-12-19T15:54:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2928684424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f593406baa0 0x7f5934157b60 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2928684424 >> v2:172.21.15.125:3300/0 conn(0x7f5934154780 0x7f593415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 -- 172.21.15.50:0/2928684424 >> 172.21.15.50:0/2928684424 conn(0x7f5934067b50 msgr2=0x7f59340cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 -- 172.21.15.50:0/2928684424 shutdown_connections 2023-12-19T15:54:30.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.905+0000 7f5939a6b700 1 -- 172.21.15.50:0/2928684424 wait complete. 2023-12-19T15:54:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 Processor -- start 2023-12-19T15:54:30.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 -- start start 2023-12-19T15:54:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f593406baa0 0x7f593413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934154780 0x7f59340c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59340af310 con 0x7f59340adab0 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593414f700 con 0x7f593406baa0 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5939a6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59340b1010 con 0x7f5934154780 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34494/0 (socket says 172.21.15.50:34494) 2023-12-19T15:54:30.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 -- 172.21.15.50:0/2356131877 learned_addr learned my addr 172.21.15.50:0/2356131877 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5938a69700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f593406baa0 0x7f593413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f593926a700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934154780 0x7f59340c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 -- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934154780 msgr2=0x7f59340c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934154780 0x7f59340c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 -- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f593406baa0 msgr2=0x7f593413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f593406baa0 0x7f593413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.909+0000 7f5933fff700 1 -- 172.21.15.50:0/2356131877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200094f0 con 0x7f59340adab0 2023-12-19T15:54:30.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5933fff700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f593406a0d0 tx=0x7f5920017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920000bc0 con 0x7f59340adab0 2023-12-19T15:54:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592000b910 con 0x7f59340adab0 2023-12-19T15:54:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920000d20 con 0x7f59340adab0 2023-12-19T15:54:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59340b12f0 con 0x7f59340adab0 2023-12-19T15:54:30.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59340b18b0 con 0x7f59340adab0 2023-12-19T15:54:30.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593414f8b0 con 0x7f59340adab0 2023-12-19T15:54:30.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.913+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5920023070 con 0x7f59340adab0 2023-12-19T15:54:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.917+0000 7f5930ff9700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 0x7f5918062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.917+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f592007dcb0 con 0x7f59340adab0 2023-12-19T15:54:30.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.917+0000 7f5938a69700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 0x7f5918062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:30.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.917+0000 7f5938a69700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 0x7f5918062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5928009db0 tx=0x7f59280091a0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:30.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:30.925+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f593414f8b0 con 0x7f59340adab0 2023-12-19T15:54:31.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.097+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f593414f870 con 0x7f59340adab0 2023-12-19T15:54:31.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.105+0000 7f5930ff9700 1 -- 172.21.15.50:0/2356131877 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f592004e020 con 0x7f59340adab0 2023-12-19T15:54:31.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 msgr2=0x7f5918062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 0x7f5918062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5928009db0 tx=0x7f59280091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 msgr2=0x7f59340b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f593406a0d0 tx=0x7f5920017000 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 shutdown_connections 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5918060670 0x7f5918062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5934154780 0x7f59340c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59340adab0 0x7f59340b06e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 --2- 172.21.15.50:0/2356131877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f593406baa0 0x7f593413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 >> 172.21.15.50:0/2356131877 conn(0x7f5934067b50 msgr2=0x7f593414b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 shutdown_connections 2023-12-19T15:54:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.109+0000 7f5939a6b700 1 -- 172.21.15.50:0/2356131877 wait complete. 2023-12-19T15:54:31.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T15:54:31.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.293+0000 7f52a4a8e700 1 Processor -- start 2023-12-19T15:54:31.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 -- start start 2023-12-19T15:54:31.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52a015a010 0x7f52a00c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f52a0142690 0x7f52a00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0159b80 con 0x7f52a0141e70 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00c8220 con 0x7f52a0142690 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f52a4a8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00c7f40 con 0x7f52a015a010 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f529f7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f52a0141e70 0x7f52a00c7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60082/0 (socket says 172.21.15.50:60082) 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f529f7fe700 1 -- 172.21.15.50:0/3514695089 learned_addr learned my addr 172.21.15.50:0/3514695089 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f529ffff700 1 --2- 172.21.15.50:0/3514695089 >> v2:172.21.15.50:3300/0 conn(0x7f52a0142690 0x7f52a00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.297+0000 7f5297fff700 1 --2- 172.21.15.50:0/3514695089 >> v2:172.21.15.125:3300/0 conn(0x7f52a015a010 0x7f52a00c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190147697 0 0) 0x7f52a0159b80 con 0x7f52a0141e70 2023-12-19T15:54:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c003570 con 0x7f52a0141e70 2023-12-19T15:54:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529ffff700 1 -- 172.21.15.50:0/3514695089 >> v2:172.21.15.125:3300/0 conn(0x7f52a015a010 msgr2=0x7f52a00c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429679898 0 0) 0x7f528c003570 con 0x7f52a0141e70 2023-12-19T15:54:31.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529ffff700 1 --2- 172.21.15.50:0/3514695089 >> v2:172.21.15.125:3300/0 conn(0x7f52a015a010 0x7f52a00c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529ffff700 1 -- 172.21.15.50:0/3514695089 >> v1:172.21.15.50:6789/0 conn(0x7f52a0141e70 legacy=0x7f52a00c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529ffff700 1 -- 172.21.15.50:0/3514695089 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0159530 con 0x7f52a0142690 2023-12-19T15:54:31.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529ffff700 1 --2- 172.21.15.50:0/3514695089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a00c5220 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f52a006e830 tx=0x7f5298009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78460c781ca2c64d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800e070 con 0x7f52a0142690 2023-12-19T15:54:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800b7a0 con 0x7f52a0142690 2023-12-19T15:54:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f529effd700 1 -- 172.21.15.50:0/3514695089 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290001470 con 0x7f52a0142690 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 -- 172.21.15.50:0/3514695089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 msgr2=0x7f52a00c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/3514695089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a00c5220 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f52a006e830 tx=0x7f5298009780 comp rx=0 tx=0).stop 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 -- 172.21.15.50:0/3514695089 shutdown_connections 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/3514695089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a00c5220 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/3514695089 >> v2:172.21.15.125:3300/0 conn(0x7f52a015a010 0x7f52a00c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 -- 172.21.15.50:0/3514695089 >> 172.21.15.50:0/3514695089 conn(0x7f52a0067b50 msgr2=0x7f52a00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 -- 172.21.15.50:0/3514695089 shutdown_connections 2023-12-19T15:54:31.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.301+0000 7f52a4a8e700 1 -- 172.21.15.50:0/3514695089 wait complete. 2023-12-19T15:54:31.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 Processor -- start 2023-12-19T15:54:31.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 -- start start 2023-12-19T15:54:31.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52a0141e70 0x7f52a0158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 0x7f52a00ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0138390 con 0x7f52a0142690 2023-12-19T15:54:31.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01403d0 con 0x7f52a015a010 2023-12-19T15:54:31.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f52a4a8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a013d060 con 0x7f52a0141e70 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34502/0 (socket says 172.21.15.50:34502) 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 -- 172.21.15.50:0/1978132375 learned_addr learned my addr 172.21.15.50:0/1978132375 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f529f7fe700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52a0141e70 0x7f52a0158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f529ffff700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 0x7f52a00ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 -- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52a0141e70 msgr2=0x7f52a0158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52a0141e70 0x7f52a0158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 -- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 msgr2=0x7f52a00ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 0x7f52a00ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f5297fff700 1 -- 172.21.15.50:0/1978132375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5298009a20 con 0x7f52a0142690 2023-12-19T15:54:31.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.305+0000 7f529ffff700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 0x7f52a00ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f5297fff700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f528800ec80 tx=0x7f52880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528800c850 con 0x7f52a0142690 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528800c9b0 con 0x7f52a0142690 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528800cd50 con 0x7f52a0142690 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a013d340 con 0x7f52a0142690 2023-12-19T15:54:31.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0159180 con 0x7f52a0142690 2023-12-19T15:54:31.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.309+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0138540 con 0x7f52a0142690 2023-12-19T15:54:31.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.313+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5288013680 con 0x7f52a0142690 2023-12-19T15:54:31.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.313+0000 7f52977fe700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 0x7f5280062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.313+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f528807d040 con 0x7f52a0142690 2023-12-19T15:54:31.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.313+0000 7f529f7fe700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 0x7f5280062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.317+0000 7f529f7fe700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 0x7f5280062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f52900011e0 tx=0x7f5290001a00 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.321+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f52880acbe0 con 0x7f52a0142690 2023-12-19T15:54:31.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.497+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f52a004be40 con 0x7f52a0142690 2023-12-19T15:54:31.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.505+0000 7f52977fe700 1 -- 172.21.15.50:0/1978132375 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v50) v1 ==== 115+0+0 (secure 0 0 0) 0x7f52880170b0 con 0x7f52a0142690 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 msgr2=0x7f5280062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 0x7f5280062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f52900011e0 tx=0x7f5290001a00 comp rx=0 tx=0).stop 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 msgr2=0x7f52a013c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f528800ec80 tx=0x7f52880091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 shutdown_connections 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52a015a010 0x7f52a00ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5280060670 0x7f5280062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f52a0142690 0x7f52a013c550 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 --2- 172.21.15.50:0/1978132375 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f52a0141e70 0x7f52a0158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 >> 172.21.15.50:0/1978132375 conn(0x7f52a0067b50 msgr2=0x7f52a00cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.509+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 shutdown_connections 2023-12-19T15:54:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.513+0000 7f52a4a8e700 1 -- 172.21.15.50:0/1978132375 wait complete. 2023-12-19T15:54:31.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T15:54:31.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.697+0000 7f2f8359e700 1 Processor -- start 2023-12-19T15:54:31.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.697+0000 7f2f8359e700 1 -- start start 2023-12-19T15:54:31.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f840cc040 0x7f2f84159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840cbbb0 con 0x7f2f840ce2d0 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8406dc90 con 0x7f2f840caa10 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f840cafa0 con 0x7f2f840cc040 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f81d9b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2f840caa10 0x7f2f8406a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36842/0 (socket says 172.21.15.50:36842) 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f81d9b700 1 -- 172.21.15.50:0/2004607280 learned_addr learned my addr 172.21.15.50:0/2004607280 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:31.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8259c700 1 --1- 172.21.15.50:0/2004607280 >> v1:172.21.15.50:6789/0 conn(0x7f2f840ce2d0 0x7f2f84158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60088/0 (socket says 172.21.15.50:60088) 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f82d9d700 1 --2- 172.21.15.50:0/2004607280 >> v2:172.21.15.125:3300/0 conn(0x7f2f840cc040 0x7f2f84159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358619571 0 0) 0x7f2f8406dc90 con 0x7f2f840caa10 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f6c003570 con 0x7f2f840caa10 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559573719 0 0) 0x7f2f840cbbb0 con 0x7f2f840ce2d0 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8406dc90 con 0x7f2f840ce2d0 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.701+0000 7f2f82d9d700 1 -- 172.21.15.50:0/2004607280 >> v1:172.21.15.125:6789/0 conn(0x7f2f840caa10 legacy=0x7f2f8406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.1 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433668157 0 0) 0x7f2f6c003570 con 0x7f2f840caa10 2023-12-19T15:54:31.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f82d9d700 1 -- 172.21.15.50:0/2004607280 >> v1:172.21.15.50:6789/0 conn(0x7f2f840ce2d0 legacy=0x7f2f84158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f82d9d700 1 -- 172.21.15.50:0/2004607280 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f840cb280 con 0x7f2f840cc040 2023-12-19T15:54:31.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f82d9d700 1 --2- 172.21.15.50:0/2004607280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840cc040 0x7f2f84159010 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009de0 tx=0x7f2f7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c4ecb2840c0cb8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c00e070 con 0x7f2f840cc040 2023-12-19T15:54:31.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c00b820 con 0x7f2f840cc040 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8159a700 1 -- 172.21.15.50:0/2004607280 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f78001470 con 0x7f2f840cc040 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 -- 172.21.15.50:0/2004607280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840cc040 msgr2=0x7f2f84159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 --2- 172.21.15.50:0/2004607280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840cc040 0x7f2f84159010 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009de0 tx=0x7f2f7c009780 comp rx=0 tx=0).stop 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 -- 172.21.15.50:0/2004607280 shutdown_connections 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 --2- 172.21.15.50:0/2004607280 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840cc040 0x7f2f84159010 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 -- 172.21.15.50:0/2004607280 >> 172.21.15.50:0/2004607280 conn(0x7f2f84067bb0 msgr2=0x7f2f840cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:31.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 -- 172.21.15.50:0/2004607280 shutdown_connections 2023-12-19T15:54:31.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.705+0000 7f2f8359e700 1 -- 172.21.15.50:0/2004607280 wait complete. 2023-12-19T15:54:31.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 Processor -- start 2023-12-19T15:54:31.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 -- start start 2023-12-19T15:54:31.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f840cc040 0x7f2f840c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 0x7f2f8414ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f840c8960 con 0x7f2f840caa10 2023-12-19T15:54:31.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8414f260 con 0x7f2f840ce2d0 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f840c7f90 con 0x7f2f840cc040 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34506/0 (socket says 172.21.15.50:34506) 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 -- 172.21.15.50:0/311980538 learned_addr learned my addr 172.21.15.50:0/311980538 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f81d9b700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f840cc040 0x7f2f840c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f82d9d700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 0x7f2f8414ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 -- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f840cc040 msgr2=0x7f2f840c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f840cc040 0x7f2f840c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 -- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 msgr2=0x7f2f8414ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 0x7f2f8414ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 -- 172.21.15.50:0/311980538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c009a20 con 0x7f2f840caa10 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f82d9d700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 0x7f2f8414ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.709+0000 7f2f8259c700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c5c0 tx=0x7f2f74007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74005a70 con 0x7f2f840caa10 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74005bf0 con 0x7f2f840caa10 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8415bbb0 con 0x7f2f840caa10 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f740070d0 con 0x7f2f840caa10 2023-12-19T15:54:31.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f840c7660 con 0x7f2f840caa10 2023-12-19T15:54:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8404be40 con 0x7f2f840caa10 2023-12-19T15:54:31.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.713+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2f740074c0 con 0x7f2f840caa10 2023-12-19T15:54:31.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.717+0000 7f2f727fc700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 0x7f2f64062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:31.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.717+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f2f7407bf00 con 0x7f2f840caa10 2023-12-19T15:54:31.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.717+0000 7f2f81d9b700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 0x7f2f64062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:31.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.717+0000 7f2f81d9b700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 0x7f2f64062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f84062c60 tx=0x7f2f78000bc0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:31.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.721+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f7404c310 con 0x7f2f840caa10 2023-12-19T15:54:31.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.889+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2f840cb560 con 0x7f2f840caa10 2023-12-19T15:54:31.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.897+0000 7f2f727fc700 1 -- 172.21.15.50:0/311980538 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2f7404f930 con 0x7f2f840caa10 2023-12-19T15:54:31.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 msgr2=0x7f2f64062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 0x7f2f64062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f84062c60 tx=0x7f2f78000bc0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 msgr2=0x7f2f840c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c5c0 tx=0x7f2f74007e00 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 shutdown_connections 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f840ce2d0 0x7f2f8414ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2f64060670 0x7f2f64062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f840cc040 0x7f2f840c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 --2- 172.21.15.50:0/311980538 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f840caa10 0x7f2f840c9810 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 >> 172.21.15.50:0/311980538 conn(0x7f2f84067bb0 msgr2=0x7f2f8414fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 shutdown_connections 2023-12-19T15:54:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:31.901+0000 7f2f8359e700 1 -- 172.21.15.50:0/311980538 wait complete. 2023-12-19T15:54:31.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-12-19T15:54:32.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.089+0000 7f60b926e700 1 Processor -- start 2023-12-19T15:54:32.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.089+0000 7f60b926e700 1 -- start start 2023-12-19T15:54:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b926e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4140710 0x7f60b40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b926e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60b4068800 0x7f60b40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b926e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b406e150 con 0x7f60b4140ae0 2023-12-19T15:54:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b926e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b406d540 con 0x7f60b4068800 2023-12-19T15:54:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b926e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b406d820 con 0x7f60b4140710 2023-12-19T15:54:32.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60b4068800 0x7f60b40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60b4068800 0x7f60b40ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34510/0 (socket says 172.21.15.50:34510) 2023-12-19T15:54:32.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 -- 172.21.15.50:0/1666517067 learned_addr learned my addr 172.21.15.50:0/1666517067 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:32.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b2ffd700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4140710 0x7f60b40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 -- 172.21.15.50:0/1666517067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4140710 msgr2=0x7f60b40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:32.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4140710 0x7f60b40c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 -- 172.21.15.50:0/1666517067 >> v1:172.21.15.50:6789/0 conn(0x7f60b4140ae0 legacy=0x7f60b40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 -- 172.21.15.50:0/1666517067 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b406db00 con 0x7f60b4068800 2023-12-19T15:54:32.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b37fe700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4068800 0x7f60b40ac250 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009f20 tx=0x7f60ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f476294924cc59 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b1ffb700 1 -- 172.21.15.50:0/1666517067 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00e070 con 0x7f60b4068800 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b1ffb700 1 -- 172.21.15.50:0/1666517067 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac00ba80 con 0x7f60b4068800 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.093+0000 7f60b1ffb700 1 -- 172.21.15.50:0/1666517067 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0092e0 con 0x7f60b4068800 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 -- 172.21.15.50:0/1666517067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4068800 msgr2=0x7f60b40ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4068800 0x7f60b40ac250 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009f20 tx=0x7f60ac009840 comp rx=0 tx=0).stop 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 -- 172.21.15.50:0/1666517067 shutdown_connections 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4068800 0x7f60b40ac250 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 --2- 172.21.15.50:0/1666517067 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4140710 0x7f60b40c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 -- 172.21.15.50:0/1666517067 >> 172.21.15.50:0/1666517067 conn(0x7f60b40c8850 msgr2=0x7f60b40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:32.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 -- 172.21.15.50:0/1666517067 shutdown_connections 2023-12-19T15:54:32.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 -- 172.21.15.50:0/1666517067 wait complete. 2023-12-19T15:54:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.097+0000 7f60b926e700 1 Processor -- start 2023-12-19T15:54:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 -- start start 2023-12-19T15:54:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4068800 0x7f60b4156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 0x7f60b40c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b415b860 con 0x7f60b4140ae0 2023-12-19T15:54:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b415acc0 con 0x7f60b4140710 2023-12-19T15:54:32.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b926e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b415af30 con 0x7f60b4068800 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34526/0 (socket says 172.21.15.50:34526) 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 -- 172.21.15.50:0/3064630103 learned_addr learned my addr 172.21.15.50:0/3064630103 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b2ffd700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4068800 0x7f60b4156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b27fc700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 0x7f60b40c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 -- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4068800 msgr2=0x7f60b4156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4068800 0x7f60b4156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 -- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 msgr2=0x7f60b40c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 0x7f60b40c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 -- 172.21.15.50:0/3064630103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009ae0 con 0x7f60b4140ae0 2023-12-19T15:54:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b27fc700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 0x7f60b40c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.101+0000 7f60b37fe700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a280 tx=0x7f60ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0032f0 con 0x7f60b4140ae0 2023-12-19T15:54:32.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac008d90 con 0x7f60b4140ae0 2023-12-19T15:54:32.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b415b210 con 0x7f60b4140ae0 2023-12-19T15:54:32.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac01c640 con 0x7f60b4140ae0 2023-12-19T15:54:32.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b4156e40 con 0x7f60b4140ae0 2023-12-19T15:54:32.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.105+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b415b9d0 con 0x7f60b4140ae0 2023-12-19T15:54:32.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.109+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f60ac003040 con 0x7f60b4140ae0 2023-12-19T15:54:32.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.109+0000 7f609affd700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 0x7f60a0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.109+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f60ac07e480 con 0x7f60b4140ae0 2023-12-19T15:54:32.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.109+0000 7f60b2ffd700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 0x7f60a0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.109+0000 7f60b2ffd700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 0x7f60a0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60a4006fd0 tx=0x7f60a4008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.117+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f60ac04e6e0 con 0x7f60b4140ae0 2023-12-19T15:54:32.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.349+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f60b4071120 con 0x7f60b4140ae0 2023-12-19T15:54:32.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.705+0000 7f609affd700 1 -- 172.21.15.50:0/3064630103 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2444) v1 ==== 178+0+0 (secure 0 0 0) 0x7f60ac051d00 con 0x7f60b4140ae0 2023-12-19T15:54:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 msgr2=0x7f60a0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 0x7f60a0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60a4006fd0 tx=0x7f60a4008040 comp rx=0 tx=0).stop 2023-12-19T15:54:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 msgr2=0x7f60b416a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a280 tx=0x7f60ac018000 comp rx=0 tx=0).stop 2023-12-19T15:54:32.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 shutdown_connections 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60b4140710 0x7f60b40c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f60a00606a0 0x7f60a0062b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60b4140ae0 0x7f60b416a7b0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 --2- 172.21.15.50:0/3064630103 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60b4068800 0x7f60b4156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 >> 172.21.15.50:0/3064630103 conn(0x7f60b40c8850 msgr2=0x7f60b40c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 shutdown_connections 2023-12-19T15:54:32.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.713+0000 7f60b926e700 1 -- 172.21.15.50:0/3064630103 wait complete. 2023-12-19T15:54:32.737 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-12-19T15:54:32.737 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-12-19T15:54:32.737 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T15:54:32.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:32.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.909+0000 7fa83d06e700 1 Processor -- start 2023-12-19T15:54:32.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.909+0000 7fa83d06e700 1 -- start start 2023-12-19T15:54:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83d06e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa8381556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83d06e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8380c1b90 con 0x7fa8380ae520 2023-12-19T15:54:32.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83d06e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83806a9d0 con 0x7fa8381552f0 2023-12-19T15:54:32.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83d06e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c8710 con 0x7fa83806ced0 2023-12-19T15:54:32.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83659c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa8380ae520 0x7fa83806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60104/0 (socket says 172.21.15.50:60104) 2023-12-19T15:54:32.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83659c700 1 -- 172.21.15.50:0/2002063442 learned_addr learned my addr 172.21.15.50:0/2002063442 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:32.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83759e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa8381556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009794180 0 0) 0x7fa8380c1b90 con 0x7fa8380ae520 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820003570 con 0x7fa8380ae520 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989780056 0 0) 0x7fa83806a9d0 con 0x7fa8381552f0 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8380c1b90 con 0x7fa8381552f0 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512618359 0 0) 0x7fa820003570 con 0x7fa8380ae520 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 msgr2=0x7fa8381556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa8381556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 >> v1:172.21.15.125:6789/0 conn(0x7fa8381552f0 legacy=0x7fa8380c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83806acb0 con 0x7fa8380ae520 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fa82c001470 con 0x7fa8380ae520 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 >> v1:172.21.15.50:6789/0 conn(0x7fa8380ae520 legacy=0x7fa83806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa820003c20 0x7fa820004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa820004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa820004d50 0x7fa8200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c8710 con 0x7fa820004d50 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82000bb40 con 0x7fa820003c20 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82000bd00 con 0x7fa83806ced0 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83759e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa820004d50 0x7fa8200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa836d9d700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa820003c20 0x7fa820004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.913+0000 7fa83659c700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa820004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 -- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 msgr2=0x7fa820004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa83806ced0 0x7fa820004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 -- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa820003c20 msgr2=0x7fa820004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa820003c20 0x7fa820004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 -- 172.21.15.50:0/2002063442 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa820003570 con 0x7fa820004d50 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa820004d50 0x7fa8200051c0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa83806d2d0 tx=0x7fa83000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e993ffdf40bad530 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83000ea00 con 0x7fa820004d50 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83000f040 con 0x7fa820004d50 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa835d9b700 1 -- 172.21.15.50:0/2002063442 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83000edb0 con 0x7fa820004d50 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 -- 172.21.15.50:0/2002063442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa820004d50 msgr2=0x7fa8200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 --2- 172.21.15.50:0/2002063442 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa820004d50 0x7fa8200051c0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa83806d2d0 tx=0x7fa83000b040 comp rx=0 tx=0).stop 2023-12-19T15:54:32.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83759e700 1 -- 172.21.15.50:0/2002063442 reap_dead start 2023-12-19T15:54:32.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 -- 172.21.15.50:0/2002063442 shutdown_connections 2023-12-19T15:54:32.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 -- 172.21.15.50:0/2002063442 >> 172.21.15.50:0/2002063442 conn(0x7fa8380c8850 msgr2=0x7fa8380c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:32.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 -- 172.21.15.50:0/2002063442 shutdown_connections 2023-12-19T15:54:32.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.917+0000 7fa83d06e700 1 -- 172.21.15.50:0/2002063442 wait complete. 2023-12-19T15:54:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 Processor -- start 2023-12-19T15:54:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 -- start start 2023-12-19T15:54:32.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa820004d50 0x7fa83814a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8380a9510 0x7fa838157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c3030 con 0x7fa8380a9980 2023-12-19T15:54:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c3310 con 0x7fa8380a9510 2023-12-19T15:54:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83d06e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c35f0 con 0x7fa820004d50 2023-12-19T15:54:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34542/0 (socket says 172.21.15.50:34542) 2023-12-19T15:54:32.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 -- 172.21.15.50:0/3369042594 learned_addr learned my addr 172.21.15.50:0/3369042594 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83659c700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8380a9510 0x7fa838157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa836d9d700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa820004d50 0x7fa83814a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 -- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa820004d50 msgr2=0x7fa83814a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa820004d50 0x7fa83814a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 -- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8380a9510 msgr2=0x7fa838157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8380a9510 0x7fa838157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.921+0000 7fa83759e700 1 -- 172.21.15.50:0/3369042594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380c38d0 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa83759e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fa830018000 tx=0x7fa83001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830012e70 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83001a630 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380c3bb0 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83001a9f0 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8380a8970 con 0x7fa8380a9980 2023-12-19T15:54:32.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8380c8570 con 0x7fa8380a9980 2023-12-19T15:54:32.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.925+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa830003020 con 0x7fa8380a9980 2023-12-19T15:54:32.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.929+0000 7fa81effd700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 0x7fa824062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:32.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.929+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7fa830016070 con 0x7fa8380a9980 2023-12-19T15:54:32.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.929+0000 7fa836d9d700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 0x7fa824062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.929+0000 7fa836d9d700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 0x7fa824062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa838158430 tx=0x7fa828007d30 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:32.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:32.933+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa8380c8570 con 0x7fa8380a9980 2023-12-19T15:54:33.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.105+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8380adab0 con 0x7fa8380a9980 2023-12-19T15:54:33.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.105+0000 7fa81effd700 1 -- 172.21.15.50:0/3369042594 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+13437 (secure 0 0 0) 0x7fa8380adab0 con 0x7fa8380a9980 2023-12-19T15:54:33.108 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:33.108 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":159,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:30.683731+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6837/528853913":"2023-12-20T15:54:27.255233+0000","172.21.15.125:6835/528853913":"2023-12-20T15:54:27.255233+0000","172.21.15.125:0/304294908":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/1214130067":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/2973093006":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/4101505360":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/4275867050":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/132099215":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/1673033377":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/249782187":"2023-12-20T15:54:26.896112+0000","172.21.15.125:0/2615455122":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/70137828":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/2692951617":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6833/33827":"2023-12-20T14:58:39.674128+0000","172.21.15.125:0/30212990":"2023-12-20T14:58:39.674128+0000","172.21.15.125:6832/33827":"2023-12-20T14:58:39.674128+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 msgr2=0x7fa824062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 0x7fa824062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa838158430 tx=0x7fa828007d30 comp rx=0 tx=0).stop 2023-12-19T15:54:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 msgr2=0x7fa838157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fa830018000 tx=0x7fa83001c000 comp rx=0 tx=0).stop 2023-12-19T15:54:33.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 shutdown_connections 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8380a9510 0x7fa838157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa8240606a0 0x7fa824062b50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8380a9980 0x7fa838157d50 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 --2- 172.21.15.50:0/3369042594 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa820004d50 0x7fa83814a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 >> 172.21.15.50:0/3369042594 conn(0x7fa8380c8850 msgr2=0x7fa8380b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 shutdown_connections 2023-12-19T15:54:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.109+0000 7fa83d06e700 1 -- 172.21.15.50:0/3369042594 wait complete. 2023-12-19T15:54:33.132 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T15:54:33.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:54:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.313+0000 7f194284c700 1 Processor -- start 2023-12-19T15:54:33.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.313+0000 7f194284c700 1 -- start start 2023-12-19T15:54:33.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194284c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c158c90 0x7f193c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194284c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f193c159770 0x7f193c06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194284c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c06c850 con 0x7f193c06aeb0 2023-12-19T15:54:33.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194284c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c06b470 con 0x7f193c159770 2023-12-19T15:54:33.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194284c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c06b750 con 0x7f193c158c90 2023-12-19T15:54:33.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f193c06aeb0 0x7f193c159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60108/0 (socket says 172.21.15.50:60108) 2023-12-19T15:54:33.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 -- 172.21.15.50:0/1675115655 learned_addr learned my addr 172.21.15.50:0/1675115655 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:33.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194204b700 1 --2- 172.21.15.50:0/1675115655 >> v2:172.21.15.125:3300/0 conn(0x7f193c159770 0x7f193c06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194184a700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c158c90 0x7f193c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865907028 0 0) 0x7f193c06c850 con 0x7f193c06aeb0 2023-12-19T15:54:33.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1928003570 con 0x7f193c06aeb0 2023-12-19T15:54:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2069433670 0 0) 0x7f1928003570 con 0x7f193c06aeb0 2023-12-19T15:54:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c158c90 msgr2=0x7f193c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c158c90 0x7f193c0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 >> v2:172.21.15.125:3300/0 conn(0x7f193c159770 msgr2=0x7f193c06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 --2- 172.21.15.50:0/1675115655 >> v2:172.21.15.125:3300/0 conn(0x7f193c159770 0x7f193c06cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c06ba30 con 0x7f193c06aeb0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f192c001470 con 0x7f193c06aeb0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 >> v1:172.21.15.50:6789/0 conn(0x7f193c06aeb0 legacy=0x7f193c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1928003a60 0x7f1928003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f1928004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19280051c0 0x7f192800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c06b470 con 0x7f1928004810 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c06b750 con 0x7f19280051c0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192800bfa0 con 0x7f1928003a60 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f1928004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194184a700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1928003a60 0x7f1928003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194204b700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19280051c0 0x7f192800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 -- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1928003a60 msgr2=0x7f1928003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1928003a60 0x7f1928003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 -- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19280051c0 msgr2=0x7f192800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19280051c0 0x7f192800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f1941049700 1 -- 172.21.15.50:0/1675115655 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928003570 con 0x7f1928004810 2023-12-19T15:54:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.317+0000 7f194204b700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19280051c0 0x7f192800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:33.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f1941049700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f1928004c80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f192c00c3c0 tx=0x7f192c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3d873ae6001ced server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c00cc90 con 0x7f1928004810 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c00ce30 con 0x7f1928004810 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f1940848700 1 -- 172.21.15.50:0/1675115655 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c005b70 con 0x7f1928004810 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 -- 172.21.15.50:0/1675115655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 msgr2=0x7f1928004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 --2- 172.21.15.50:0/1675115655 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f1928004c80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f192c00c3c0 tx=0x7f192c001e90 comp rx=0 tx=0).stop 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194204b700 1 -- 172.21.15.50:0/1675115655 reap_dead start 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 -- 172.21.15.50:0/1675115655 shutdown_connections 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 -- 172.21.15.50:0/1675115655 >> 172.21.15.50:0/1675115655 conn(0x7f193c067bb0 msgr2=0x7f193c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 -- 172.21.15.50:0/1675115655 shutdown_connections 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.321+0000 7f194284c700 1 -- 172.21.15.50:0/1675115655 wait complete. 2023-12-19T15:54:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 Processor -- start 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 -- start start 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c150c10 0x7f193c151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f193c153200 0x7f193c153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1696a0 con 0x7f1928004810 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1515c0 con 0x7f193c153200 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194284c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c153bb0 con 0x7f193c150c10 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34554/0 (socket says 172.21.15.50:34554) 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 -- 172.21.15.50:0/2979311310 learned_addr learned my addr 172.21.15.50:0/2979311310 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:33.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f1941049700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c150c10 0x7f193c151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194204b700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f193c153200 0x7f193c153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 -- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c150c10 msgr2=0x7f193c151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c150c10 0x7f193c151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 -- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f193c153200 msgr2=0x7f193c153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f193c153200 0x7f193c153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.325+0000 7f194184a700 1 -- 172.21.15.50:0/2979311310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c152400 con 0x7f1928004810 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f194184a700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1938011af0 tx=0x7f19380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800f990 con 0x7f1928004810 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193800faf0 con 0x7f1928004810 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c1517f0 con 0x7f1928004810 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800feb0 con 0x7f1928004810 2023-12-19T15:54:33.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c151db0 con 0x7f1928004810 2023-12-19T15:54:33.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.329+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c04be40 con 0x7f1928004810 2023-12-19T15:54:33.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.333+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1938014650 con 0x7f1928004810 2023-12-19T15:54:33.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.333+0000 7f1931ffb700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 0x7f1920062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.333+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f1938018030 con 0x7f1928004810 2023-12-19T15:54:33.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.333+0000 7f1941049700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 0x7f1920062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.333+0000 7f1941049700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 0x7f1920062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f192c00c3c0 tx=0x7f192c015000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.341+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f193804d750 con 0x7f1928004810 2023-12-19T15:54:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.517+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f193c06bd10 con 0x7f1928004810 2023-12-19T15:54:33.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.517+0000 7f1931ffb700 1 -- 172.21.15.50:0/2979311310 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v159) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1938050d70 con 0x7f1928004810 2023-12-19T15:54:33.520 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:54:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.521+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 msgr2=0x7f1920062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.521+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 0x7f1920062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f192c00c3c0 tx=0x7f192c015000 comp rx=0 tx=0).stop 2023-12-19T15:54:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.521+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 msgr2=0x7f193c168ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.521+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1938011af0 tx=0x7f19380091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 shutdown_connections 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f193c153200 0x7f193c153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1920060670 0x7f1920062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f193c150c10 0x7f193c151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 --2- 172.21.15.50:0/2979311310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1928004810 0x7f193c168ab0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 >> 172.21.15.50:0/2979311310 conn(0x7f193c067bb0 msgr2=0x7f193c0ca620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 shutdown_connections 2023-12-19T15:54:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.525+0000 7f194284c700 1 -- 172.21.15.50:0/2979311310 wait complete. 2023-12-19T15:54:33.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-12-19T15:54:33.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 Processor -- start 2023-12-19T15:54:33.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 -- start start 2023-12-19T15:54:33.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 0x7f69e415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69e40c7f00 0x7f69e40aca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e406d990 con 0x7f69e406de20 2023-12-19T15:54:33.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e415a570 con 0x7f69e40c7f00 2023-12-19T15:54:33.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.709+0000 7f69e8bcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e406cd80 con 0x7f69e4155960 2023-12-19T15:54:33.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69e40c7f00 0x7f69e40aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 0x7f69e415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69e40c7f00 0x7f69e40aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:58792/0 (socket says 172.21.15.50:58792) 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e2ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f69e406de20 0x7f69e40c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36860/0 (socket says 172.21.15.50:36860) 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e37fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 0x7f69e415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:54976/0 (socket says 172.21.15.50:54976) 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 -- 172.21.15.50:0/1912494384 learned_addr learned my addr 172.21.15.50:0/1912494384 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:33.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 -- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 msgr2=0x7f69e415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:33.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 --2- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 0x7f69e415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 -- 172.21.15.50:0/1912494384 >> v1:172.21.15.125:6789/0 conn(0x7f69e406de20 legacy=0x7f69e40c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 -- 172.21.15.50:0/1912494384 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e406d060 con 0x7f69e40c7f00 2023-12-19T15:54:33.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e3fff700 1 --2- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e40c7f00 0x7f69e40aca10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00d1a0 tx=0x7f69dc00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=843d518b1a76065e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e27fc700 1 -- 172.21.15.50:0/1912494384 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc010070 con 0x7f69e40c7f00 2023-12-19T15:54:33.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e27fc700 1 -- 172.21.15.50:0/1912494384 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc00de10 con 0x7f69e40c7f00 2023-12-19T15:54:33.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.713+0000 7f69e27fc700 1 -- 172.21.15.50:0/1912494384 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc00b320 con 0x7f69e40c7f00 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e40c7f00 msgr2=0x7f69e40aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e40c7f00 0x7f69e40aca10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00d1a0 tx=0x7f69dc00ba50 comp rx=0 tx=0).stop 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- 172.21.15.50:0/1912494384 shutdown_connections 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e40c7f00 0x7f69e40aca10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/1912494384 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e4155960 0x7f69e415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- 172.21.15.50:0/1912494384 >> 172.21.15.50:0/1912494384 conn(0x7f69e4067bb0 msgr2=0x7f69e40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- 172.21.15.50:0/1912494384 shutdown_connections 2023-12-19T15:54:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- 172.21.15.50:0/1912494384 wait complete. 2023-12-19T15:54:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 Processor -- start 2023-12-19T15:54:33.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.717+0000 7f69e8bcc700 1 -- start start 2023-12-19T15:54:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 0x7f69e414bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e4155960 0x7f69e40caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4150b00 con 0x7f69e406de20 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b1730 con 0x7f69e4155960 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e414fef0 con 0x7f69e40c7f00 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34564/0 (socket says 172.21.15.50:34564) 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 -- 172.21.15.50:0/2043750815 learned_addr learned my addr 172.21.15.50:0/2043750815 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e3fff700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e4155960 0x7f69e40caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e2ffd700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 0x7f69e414bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 -- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 msgr2=0x7f69e414bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 0x7f69e414bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 -- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e4155960 msgr2=0x7f69e40caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e4155960 0x7f69e40caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 -- 172.21.15.50:0/2043750815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc00bcf0 con 0x7f69e406de20 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e2ffd700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 0x7f69e414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:33.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e37fe700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f69d400e930 tx=0x7f69d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4013680 con 0x7f69e406de20 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d40137e0 con 0x7f69e406de20 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e41501d0 con 0x7f69e406de20 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4013b80 con 0x7f69e406de20 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.721+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4150790 con 0x7f69e406de20 2023-12-19T15:54:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.725+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e404be40 con 0x7f69e406de20 2023-12-19T15:54:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.725+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f69d400c460 con 0x7f69e406de20 2023-12-19T15:54:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.725+0000 7f69cb7fe700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 0x7f69d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.725+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f69d407cb90 con 0x7f69e406de20 2023-12-19T15:54:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.729+0000 7f69e2ffd700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 0x7f69d0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:33.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.729+0000 7f69e2ffd700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 0x7f69d0062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f69d8007fd0 tx=0x7f69d8007e10 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:33.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.733+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f69d404cf00 con 0x7f69e406de20 2023-12-19T15:54:33.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:33.969+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f69e4150de0 con 0x7f69e406de20 2023-12-19T15:54:34.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.716+0000 7f69cb7fe700 1 -- 172.21.15.50:0/2043750815 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2446) v1 ==== 177+0+0 (secure 0 0 0) 0x7f69d4050520 con 0x7f69e406de20 2023-12-19T15:54:34.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 msgr2=0x7f69d0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 0x7f69d0062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f69d8007fd0 tx=0x7f69d8007e10 comp rx=0 tx=0).stop 2023-12-19T15:54:34.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 msgr2=0x7f69e40b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f69d400e930 tx=0x7f69d40091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 shutdown_connections 2023-12-19T15:54:34.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69e4155960 0x7f69e40caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f69d0060670 0x7f69d0062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69e40c7f00 0x7f69e414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 --2- 172.21.15.50:0/2043750815 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69e406de20 0x7f69e40b11f0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 >> 172.21.15.50:0/2043750815 conn(0x7f69e4067bb0 msgr2=0x7f69e414dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.720+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 shutdown_connections 2023-12-19T15:54:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.724+0000 7f69e8bcc700 1 -- 172.21.15.50:0/2043750815 wait complete. 2023-12-19T15:54:34.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T15:54:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.920+0000 7f7b76616700 1 Processor -- start 2023-12-19T15:54:34.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.920+0000 7f7b76616700 1 -- start start 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7b70158c90 0x7f7b700caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7006c850 con 0x7f7b7006aeb0 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7006b470 con 0x7f7b70159770 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7006b750 con 0x7f7b70158c90 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7b70158c90 0x7f7b700caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7b70158c90 0x7f7b700caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34566/0 (socket says 172.21.15.50:34566) 2023-12-19T15:54:34.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 -- 172.21.15.50:0/3003817053 learned_addr learned my addr 172.21.15.50:0/3003817053 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:34.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 -- 172.21.15.50:0/3003817053 >> v1:172.21.15.125:6789/0 conn(0x7f7b70159770 legacy=0x7f7b7006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:34.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 -- 172.21.15.50:0/3003817053 >> v1:172.21.15.50:6789/0 conn(0x7f7b7006aeb0 legacy=0x7f7b70159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 -- 172.21.15.50:0/3003817053 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7006ba30 con 0x7f7b70158c90 2023-12-19T15:54:34.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b75614700 1 --2- 172.21.15.50:0/3003817053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b70158c90 0x7f7b700caf40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0099b0 tx=0x7f7b6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d540c6854a531e53 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b67fff700 1 -- 172.21.15.50:0/3003817053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00b820 con 0x7f7b70158c90 2023-12-19T15:54:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b67fff700 1 -- 172.21.15.50:0/3003817053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c00b980 con 0x7f7b70158c90 2023-12-19T15:54:34.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b67fff700 1 -- 172.21.15.50:0/3003817053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00bd20 con 0x7f7b70158c90 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- 172.21.15.50:0/3003817053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b70158c90 msgr2=0x7f7b700caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 --2- 172.21.15.50:0/3003817053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b70158c90 0x7f7b700caf40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0099b0 tx=0x7f7b6c009250 comp rx=0 tx=0).stop 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- 172.21.15.50:0/3003817053 shutdown_connections 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 --2- 172.21.15.50:0/3003817053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b70158c90 0x7f7b700caf40 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.924+0000 7f7b76616700 1 -- 172.21.15.50:0/3003817053 >> 172.21.15.50:0/3003817053 conn(0x7f7b70067bb0 msgr2=0x7f7b700cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.928+0000 7f7b76616700 1 -- 172.21.15.50:0/3003817053 shutdown_connections 2023-12-19T15:54:34.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.928+0000 7f7b76616700 1 -- 172.21.15.50:0/3003817053 wait complete. 2023-12-19T15:54:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.928+0000 7f7b76616700 1 Processor -- start 2023-12-19T15:54:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.928+0000 7f7b76616700 1 -- start start 2023-12-19T15:54:34.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:34.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 0x7f7b7014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:34.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 0x7f7b70151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70152180 con 0x7f7b7006aeb0 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7014ed00 con 0x7f7b701517c0 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7014f880 con 0x7f7b70159770 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34576/0 (socket says 172.21.15.50:34576) 2023-12-19T15:54:34.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 -- 172.21.15.50:0/3416532625 learned_addr learned my addr 172.21.15.50:0/3416532625 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75e15700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 0x7f7b70151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b74e13700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 0x7f7b7014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 -- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 msgr2=0x7f7b7014cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 0x7f7b7014cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 -- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 msgr2=0x7f7b70151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 0x7f7b70151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 -- 172.21.15.50:0/3416532625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0094f0 con 0x7f7b7006aeb0 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75e15700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 0x7f7b70151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b74e13700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 0x7f7b7014cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:34.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b75614700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7b70069b70 tx=0x7f7b6c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c011d00 con 0x7f7b7006aeb0 2023-12-19T15:54:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c020070 con 0x7f7b7006aeb0 2023-12-19T15:54:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b70150250 con 0x7f7b7006aeb0 2023-12-19T15:54:34.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c01b680 con 0x7f7b7006aeb0 2023-12-19T15:54:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.932+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7014e450 con 0x7f7b7006aeb0 2023-12-19T15:54:34.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.936+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7004be40 con 0x7f7b7006aeb0 2023-12-19T15:54:34.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.936+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7b6c022030 con 0x7f7b7006aeb0 2023-12-19T15:54:34.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.936+0000 7f7b657fa700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 0x7f7b54062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:34.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.936+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f7b6c07e9a0 con 0x7f7b7006aeb0 2023-12-19T15:54:34.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.936+0000 7f7b74e13700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 0x7f7b54062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:34.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.940+0000 7f7b74e13700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 0x7f7b54062ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7b60007fd0 tx=0x7f7b60009040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:34.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:34.944+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7b6c04ed10 con 0x7f7b7006aeb0 2023-12-19T15:54:35.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.120+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7b7004be40 con 0x7f7b7006aeb0 2023-12-19T15:54:35.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.128+0000 7f7b657fa700 1 -- 172.21.15.50:0/3416532625 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7b6c011e60 con 0x7f7b7006aeb0 2023-12-19T15:54:35.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 msgr2=0x7f7b54062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 0x7f7b54062ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7b60007fd0 tx=0x7f7b60009040 comp rx=0 tx=0).stop 2023-12-19T15:54:35.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 msgr2=0x7f7b7014fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7b70069b70 tx=0x7f7b6c016000 comp rx=0 tx=0).stop 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 shutdown_connections 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7b701517c0 0x7f7b70151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7b540606f0 0x7f7b54062ba0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7b70159770 0x7f7b7014cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 --2- 172.21.15.50:0/3416532625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b7006aeb0 0x7f7b7014fd10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 >> 172.21.15.50:0/3416532625 conn(0x7f7b70067bb0 msgr2=0x7f7b700ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 shutdown_connections 2023-12-19T15:54:35.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.132+0000 7f7b76616700 1 -- 172.21.15.50:0/3416532625 wait complete. 2023-12-19T15:54:35.158 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T15:54:35.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:35.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:35.247 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T15:54:35.247 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T15:54:35.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:35.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 Processor -- start 2023-12-19T15:54:35.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 -- start start 2023-12-19T15:54:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98300acd30 0x7f98300c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 0x7f983015a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98300cca10 con 0x7f983015acf0 2023-12-19T15:54:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983015b170 con 0x7f98300acd30 2023-12-19T15:54:35.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.416+0000 7f9836a4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98300cc0e0 con 0x7f98300c8200 2023-12-19T15:54:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98300acd30 0x7f98300c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835249700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 0x7f983015a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f983624b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f983015acf0 0x7f98300ccea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36868/0 (socket says 172.21.15.50:36868) 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f983624b700 1 -- 172.21.15.50:0/2982127742 learned_addr learned my addr 172.21.15.50:0/2982127742 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9834a48700 1 -- 172.21.15.50:0/2982127742 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29217629 0 0) 0x7f98300cca10 con 0x7f983015acf0 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9834a48700 1 -- 172.21.15.50:0/2982127742 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f981c003570 con 0x7f983015acf0 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 -- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 msgr2=0x7f983015a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 0x7f983015a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 -- 172.21.15.50:0/2982127742 >> v1:172.21.15.125:6789/0 conn(0x7f983015acf0 legacy=0x7f98300ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 -- 172.21.15.50:0/2982127742 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300cc3c0 con 0x7f98300acd30 2023-12-19T15:54:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835249700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 0x7f983015a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9835a4a700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300acd30 0x7f98300c8d80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098e0 tx=0x7f982c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d700fc8e278959b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9834a48700 1 -- 172.21.15.50:0/2982127742 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c011630 con 0x7f98300acd30 2023-12-19T15:54:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9834a48700 1 -- 172.21.15.50:0/2982127742 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c011790 con 0x7f98300acd30 2023-12-19T15:54:35.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.420+0000 7f9834a48700 1 -- 172.21.15.50:0/2982127742 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c011b30 con 0x7f98300acd30 2023-12-19T15:54:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300acd30 msgr2=0x7f98300c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300acd30 0x7f98300c8d80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098e0 tx=0x7f982c0091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- 172.21.15.50:0/2982127742 shutdown_connections 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300c8200 0x7f983015a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 --2- 172.21.15.50:0/2982127742 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300acd30 0x7f98300c8d80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- 172.21.15.50:0/2982127742 >> 172.21.15.50:0/2982127742 conn(0x7f98300cdbc0 msgr2=0x7f98300cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- 172.21.15.50:0/2982127742 shutdown_connections 2023-12-19T15:54:35.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- 172.21.15.50:0/2982127742 wait complete. 2023-12-19T15:54:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 Processor -- start 2023-12-19T15:54:35.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.424+0000 7f9836a4c700 1 -- start start 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300acd30 0x7f9830157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 0x7f98300c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830159a90 con 0x7f983015acf0 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830159790 con 0x7f98300c8200 2023-12-19T15:54:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9836a4c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98300c4d10 con 0x7f98300acd30 2023-12-19T15:54:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34586/0 (socket says 172.21.15.50:34586) 2023-12-19T15:54:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 -- 172.21.15.50:0/115622583 learned_addr learned my addr 172.21.15.50:0/115622583 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9835249700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 0x7f98300c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9835a4a700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300acd30 0x7f9830157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 -- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300acd30 msgr2=0x7f9830157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300acd30 0x7f9830157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 -- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 msgr2=0x7f98300c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 0x7f98300c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 -- 172.21.15.50:0/115622583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820009420 con 0x7f983015acf0 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9835249700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 0x7f98300c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f983624b700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f982800cc40 tx=0x7f9828001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.428+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800a830 con 0x7f983015acf0 2023-12-19T15:54:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982800a990 con 0x7f983015acf0 2023-12-19T15:54:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c009420 con 0x7f983015acf0 2023-12-19T15:54:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800ad10 con 0x7f983015acf0 2023-12-19T15:54:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98300c7a30 con 0x7f983015acf0 2023-12-19T15:54:35.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830159c00 con 0x7f983015acf0 2023-12-19T15:54:35.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.432+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9828007300 con 0x7f983015acf0 2023-12-19T15:54:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.436+0000 7f9825ffb700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 0x7f9814062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.436+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f982807cda0 con 0x7f983015acf0 2023-12-19T15:54:35.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.436+0000 7f9835a4a700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 0x7f9814062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.436+0000 7f9835a4a700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 0x7f9814062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f982c0091e0 tx=0x7f982c017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.440+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f982804d100 con 0x7f983015acf0 2023-12-19T15:54:35.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.652+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f983004be40 con 0x7f983015acf0 2023-12-19T15:54:35.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.652+0000 7f9825ffb700 1 -- 172.21.15.50:0/115622583 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f9828050720 con 0x7f983015acf0 2023-12-19T15:54:35.656 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:35.656 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1322020200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":1322020200},{"type":"v1","addr":"172.21.15.125:6837","nonce":1322020200}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8218,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/3365366543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3365366543},{"type":"v1","addr":"172.21.15.50:6833","nonce":3365366543}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/727035750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":727035750},{"type":"v1","addr":"172.21.15.125:6836","nonce":727035750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61},{"gid":8370,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/98563894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":98563894},{"type":"v1","addr":"172.21.15.50:6835","nonce":98563894}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":61}],"filesystems":[]} 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 msgr2=0x7f9814062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 0x7f9814062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f982c0091e0 tx=0x7f982c017000 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 msgr2=0x7f98301424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f982800cc40 tx=0x7f9828001a40 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 shutdown_connections 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98300c8200 0x7f98300c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f98140606a0 0x7f9814062b50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f983015acf0 0x7f98301424e0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 --2- 172.21.15.50:0/115622583 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f98300acd30 0x7f9830157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 >> 172.21.15.50:0/115622583 conn(0x7f98300cdbc0 msgr2=0x7f98301552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 shutdown_connections 2023-12-19T15:54:35.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.656+0000 7f9836a4c700 1 -- 172.21.15.50:0/115622583 wait complete. 2023-12-19T15:54:35.662 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 68 2023-12-19T15:54:35.681 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T15:54:35.681 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T15:54:35.681 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T15:54:35.681 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:54:35.681 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T15:54:35.682 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T15:54:35.682 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T15:54:35.682 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:54:35.682 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T15:54:35.682 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T15:54:35.682 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T15:54:35.682 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:54:35.682 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T15:54:35.683 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T15:54:35.683 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T15:54:35.683 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:54:35.705 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T15:54:35.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T15:54:35.706 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T15:54:35.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T15:54:35.708 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T15:54:35.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T15:54:35.709 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T15:54:35.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T15:54:35.757 INFO:tasks.ceph.mds.b:Started 2023-12-19T15:54:35.758 DEBUG:teuthology.parallel:result is None 2023-12-19T15:54:35.762 INFO:tasks.ceph.mds.a:Started 2023-12-19T15:54:35.763 DEBUG:teuthology.parallel:result is None 2023-12-19T15:54:35.763 INFO:tasks.ceph.mds.d:Started 2023-12-19T15:54:35.763 DEBUG:teuthology.parallel:result is None 2023-12-19T15:54:35.766 INFO:tasks.ceph.mds.c:Started 2023-12-19T15:54:35.766 DEBUG:teuthology.parallel:result is None 2023-12-19T15:54:35.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T15:54:35.868 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T15:54:35.869 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T15:54:35.871 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T15:54:35.872 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T15:54:35.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.940+0000 7f84996f0700 1 Processor -- start 2023-12-19T15:54:35.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.940+0000 7f84996f0700 1 -- start start 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8494140420 0x7f849415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f84941407f0 0x7f84940b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84940ae280 con 0x7f8494140420 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849406d7d0 con 0x7f84941407f0 2023-12-19T15:54:35.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84996f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849406cc50 con 0x7f849406c7c0 2023-12-19T15:54:35.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f8492ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8494140420 0x7f849415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55044/0 (socket says 172.21.15.50:55044) 2023-12-19T15:54:35.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 -- 172.21.15.50:0/1043280473 learned_addr learned my addr 172.21.15.50:0/1043280473 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:35.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84937fe700 1 --2- 172.21.15.50:0/1043280473 >> v2:172.21.15.125:3300/0 conn(0x7f84941407f0 0x7f84940b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 -- 172.21.15.50:0/1043280473 >> v2:172.21.15.125:3300/0 conn(0x7f84941407f0 msgr2=0x7f84940b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 --2- 172.21.15.50:0/1043280473 >> v2:172.21.15.125:3300/0 conn(0x7f84941407f0 0x7f84940b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 -- 172.21.15.50:0/1043280473 >> v2:172.21.15.50:3300/0 conn(0x7f8494140420 msgr2=0x7f849415b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 --2- 172.21.15.50:0/1043280473 >> v2:172.21.15.50:3300/0 conn(0x7f8494140420 0x7f849415b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 -- 172.21.15.50:0/1043280473 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849406cf30 con 0x7f849406c7c0 2023-12-19T15:54:35.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f84927fc700 1 --2- 172.21.15.50:0/1043280473 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8494140cf0 tx=0x7f8488009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcef12207365b47a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f8491ffb700 1 -- 172.21.15.50:0/1043280473 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848800b640 con 0x7f849406c7c0 2023-12-19T15:54:35.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f8491ffb700 1 -- 172.21.15.50:0/1043280473 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848800b7a0 con 0x7f849406c7c0 2023-12-19T15:54:35.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.944+0000 7f8491ffb700 1 -- 172.21.15.50:0/1043280473 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848800bb60 con 0x7f849406c7c0 2023-12-19T15:54:35.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 -- 172.21.15.50:0/1043280473 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 msgr2=0x7f84940b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 --2- 172.21.15.50:0/1043280473 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8494140cf0 tx=0x7f8488009250 comp rx=0 tx=0).stop 2023-12-19T15:54:35.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 -- 172.21.15.50:0/1043280473 shutdown_connections 2023-12-19T15:54:35.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 --2- 172.21.15.50:0/1043280473 >> v2:172.21.15.125:3300/0 conn(0x7f84941407f0 0x7f84940b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 --2- 172.21.15.50:0/1043280473 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f849406c7c0 0x7f84940b0cd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 --2- 172.21.15.50:0/1043280473 >> v2:172.21.15.50:3300/0 conn(0x7f8494140420 0x7f849415b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 -- 172.21.15.50:0/1043280473 >> 172.21.15.50:0/1043280473 conn(0x7f8494067b60 msgr2=0x7f84940ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:35.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 -- 172.21.15.50:0/1043280473 shutdown_connections 2023-12-19T15:54:35.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.948+0000 7f84996f0700 1 -- 172.21.15.50:0/1043280473 wait complete. 2023-12-19T15:54:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 Processor -- start 2023-12-19T15:54:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 -- start start 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8494140420 0x7f8494156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 0x7f8494157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84941587c0 con 0x7f849406c7c0 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494169400 con 0x7f8494140420 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84996f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8494166ae0 con 0x7f84941407f0 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34650/0 (socket says 172.21.15.50:34650) 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 -- 172.21.15.50:0/3498880341 learned_addr learned my addr 172.21.15.50:0/3498880341 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84927fc700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8494140420 0x7f8494156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84937fe700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 0x7f8494157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 -- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 msgr2=0x7f8494157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 0x7f8494157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 -- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8494140420 msgr2=0x7f8494156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8494140420 0x7f8494156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 -- 172.21.15.50:0/3498880341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84880094f0 con 0x7f849406c7c0 2023-12-19T15:54:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f84937fe700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 0x7f8494157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:35.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f8492ffd700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f848400ec80 tx=0x7f84840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400c9d0 con 0x7f849406c7c0 2023-12-19T15:54:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.952+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848400cb30 con 0x7f849406c7c0 2023-12-19T15:54:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.956+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8494157b40 con 0x7f849406c7c0 2023-12-19T15:54:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.956+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484013470 con 0x7f849406c7c0 2023-12-19T15:54:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.956+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8494158100 con 0x7f849406c7c0 2023-12-19T15:54:35.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.956+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8494071120 con 0x7f849406c7c0 2023-12-19T15:54:35.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.956+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8484013820 con 0x7f849406c7c0 2023-12-19T15:54:35.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.960+0000 7f847affd700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:35.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.960+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5917+0+0 (secure 0 0 0) 0x7f848407cdf0 con 0x7f849406c7c0 2023-12-19T15:54:35.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.960+0000 7f84927fc700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:35.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.960+0000 7f84927fc700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 0x7f8480062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8494140cf0 tx=0x7f8488017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:35.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:35.964+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f848404d160 con 0x7f849406c7c0 2023-12-19T15:54:36.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.148+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f849406d210 con 0x7f849406c7c0 2023-12-19T15:54:36.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.752+0000 7f847affd700 1 -- 172.21.15.50:0/3498880341 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v161) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8484050780 con 0x7f849406c7c0 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 msgr2=0x7f8480062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 0x7f8480062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8494140cf0 tx=0x7f8488017000 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 msgr2=0x7f84941665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f848400ec80 tx=0x7f84840091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 shutdown_connections 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8494140420 0x7f8494156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84941407f0 0x7f8494157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 --2- 172.21.15.50:0/3498880341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849406c7c0 0x7f84941665a0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 >> 172.21.15.50:0/3498880341 conn(0x7f8494067b60 msgr2=0x7f849413a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 shutdown_connections 2023-12-19T15:54:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.756+0000 7f84996f0700 1 -- 172.21.15.50:0/3498880341 wait complete. 2023-12-19T15:54:36.763 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T15:54:36.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T15:54:36.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 Processor -- start 2023-12-19T15:54:36.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 -- start start 2023-12-19T15:54:36.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2600c6900 0x7fb2600c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb26006b900 0x7fb26006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26006b470 con 0x7fb26015cc20 2023-12-19T15:54:36.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26015d790 con 0x7fb2600c6900 2023-12-19T15:54:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb26646b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb260161d10 con 0x7fb26006b900 2023-12-19T15:54:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb265469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2600c6900 0x7fb2600c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb265469700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2600c6900 0x7fb2600c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34068/0 (socket says 172.21.15.50:34068) 2023-12-19T15:54:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb265469700 1 -- 172.21.15.50:0/1820668378 learned_addr learned my addr 172.21.15.50:0/1820668378 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.944+0000 7fb264c68700 1 --2- 172.21.15.50:0/1820668378 >> v2:172.21.15.125:3300/0 conn(0x7fb26006b900 0x7fb26006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb265469700 1 -- 172.21.15.50:0/1820668378 >> v2:172.21.15.125:3300/0 conn(0x7fb26006b900 msgr2=0x7fb26006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:36.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb265469700 1 --2- 172.21.15.50:0/1820668378 >> v2:172.21.15.125:3300/0 conn(0x7fb26006b900 0x7fb26006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb265469700 1 -- 172.21.15.50:0/1820668378 >> v1:172.21.15.125:6789/0 conn(0x7fb26015cc20 legacy=0x7fb26015d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb265469700 1 -- 172.21.15.50:0/1820668378 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26006bd70 con 0x7fb2600c6900 2023-12-19T15:54:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb265469700 1 --2- 172.21.15.50:0/1820668378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2600c6900 0x7fb2600c6cd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb26006a030 tx=0x7fb25c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd00a9993903ec0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb257fff700 1 -- 172.21.15.50:0/1820668378 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c00e070 con 0x7fb2600c6900 2023-12-19T15:54:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb257fff700 1 -- 172.21.15.50:0/1820668378 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c00ba20 con 0x7fb2600c6900 2023-12-19T15:54:36.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb257fff700 1 -- 172.21.15.50:0/1820668378 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0092e0 con 0x7fb2600c6900 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 -- 172.21.15.50:0/1820668378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2600c6900 msgr2=0x7fb2600c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 --2- 172.21.15.50:0/1820668378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2600c6900 0x7fb2600c6cd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb26006a030 tx=0x7fb25c009840 comp rx=0 tx=0).stop 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 -- 172.21.15.50:0/1820668378 shutdown_connections 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 --2- 172.21.15.50:0/1820668378 >> v2:172.21.15.125:3300/0 conn(0x7fb26006b900 0x7fb26006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 --2- 172.21.15.50:0/1820668378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2600c6900 0x7fb2600c6cd0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 -- 172.21.15.50:0/1820668378 >> 172.21.15.50:0/1820668378 conn(0x7fb260067bb0 msgr2=0x7fb2600cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:36.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 -- 172.21.15.50:0/1820668378 shutdown_connections 2023-12-19T15:54:36.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.948+0000 7fb26646b700 1 -- 172.21.15.50:0/1820668378 wait complete. 2023-12-19T15:54:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.952+0000 7fb26646b700 1 Processor -- start 2023-12-19T15:54:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.952+0000 7fb26646b700 1 -- start start 2023-12-19T15:54:36.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 0x7fb2600c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 0x7fb260160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26015fcd0 con 0x7fb26006b900 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26015edf0 con 0x7fb26015cc20 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb260159720 con 0x7fb2600c6900 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34082/0 (socket says 172.21.15.50:34082) 2023-12-19T15:54:36.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 -- 172.21.15.50:0/1269102568 learned_addr learned my addr 172.21.15.50:0/1269102568 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb264c68700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 0x7fb2600c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 -- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 msgr2=0x7fb2600c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265c6a700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 0x7fb260160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 0x7fb2600c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 -- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 msgr2=0x7fb260160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 0x7fb260160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 -- 172.21.15.50:0/1269102568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c009ae0 con 0x7fb26006b900 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb264c68700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 0x7fb2600c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265c6a700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 0x7fb260160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb265469700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fb260064040 tx=0x7fb25c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c008a80 con 0x7fb26006b900 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c008c00 con 0x7fb26006b900 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26015f0c0 con 0x7fb26006b900 2023-12-19T15:54:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c026460 con 0x7fb26006b900 2023-12-19T15:54:36.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.956+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26015f680 con 0x7fb26006b900 2023-12-19T15:54:36.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.960+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2600c6f80 con 0x7fb26006b900 2023-12-19T15:54:36.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.960+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb25c003040 con 0x7fb26006b900 2023-12-19T15:54:36.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.960+0000 7fb2557fa700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 0x7fb244062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.964+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fb25c07e380 con 0x7fb26006b900 2023-12-19T15:54:36.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.964+0000 7fb264c68700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 0x7fb244062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:36.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.964+0000 7fb264c68700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 0x7fb244062b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb250006f00 tx=0x7fb250008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:36.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:36.968+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb2600c6f80 con 0x7fb26006b900 2023-12-19T15:54:37.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.200+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb2600c6f80 con 0x7fb26006b900 2023-12-19T15:54:37.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb2557fa700 1 -- 172.21.15.50:0/1269102568 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v39) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb2600c6f80 con 0x7fb26006b900 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.206 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.210 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T15:54:37.212 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T15:54:37.214 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:54:37.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T15:54:37.216 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:54:37.217 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T15:54:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:54:37.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 msgr2=0x7fb244062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 0x7fb244062b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb250006f00 tx=0x7fb250008040 comp rx=0 tx=0).stop 2023-12-19T15:54:37.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 msgr2=0x7fb2601591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fb260064040 tx=0x7fb25c018000 comp rx=0 tx=0).stop 2023-12-19T15:54:37.220 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 shutdown_connections 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb26015cc20 0x7fb260160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb2440606a0 0x7fb244062b50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2600c6900 0x7fb2600c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 --2- 172.21.15.50:0/1269102568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb26006b900 0x7fb2601591e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.204+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 >> 172.21.15.50:0/1269102568 conn(0x7fb260067bb0 msgr2=0x7fb2600cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.208+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 shutdown_connections 2023-12-19T15:54:37.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.208+0000 7fb26646b700 1 -- 172.21.15.50:0/1269102568 wait complete. 2023-12-19T15:54:37.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:37.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 Processor -- start 2023-12-19T15:54:37.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 -- start start 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f1268159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126806c9f0 con 0x7f12680ad150 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12680ca860 con 0x7f12680ca010 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f126d4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12680c8d20 con 0x7f12680ca3e0 2023-12-19T15:54:37.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.384+0000 7f1266ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1266ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34086/0 (socket says 172.21.15.50:34086) 2023-12-19T15:54:37.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1266ffd700 1 -- 172.21.15.50:0/4100664902 learned_addr learned my addr 172.21.15.50:0/4100664902 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:37.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 --2- 172.21.15.50:0/4100664902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f1268159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 -- 172.21.15.50:0/4100664902 >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 msgr2=0x7f126806a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 --2- 172.21.15.50:0/4100664902 >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 -- 172.21.15.50:0/4100664902 >> v1:172.21.15.125:6789/0 conn(0x7f12680ad150 legacy=0x7f12680c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 -- 172.21.15.50:0/4100664902 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12680c8480 con 0x7f12680ca3e0 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1266ffd700 1 --2- 172.21.15.50:0/4100664902 >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:37.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1267fff700 1 --2- 172.21.15.50:0/4100664902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f1268159c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1260009e60 tx=0x7f1260009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed1b4b058153fd8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1265ffb700 1 -- 172.21.15.50:0/4100664902 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126000e070 con 0x7f12680ca3e0 2023-12-19T15:54:37.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1265ffb700 1 -- 172.21.15.50:0/4100664902 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126000bb60 con 0x7f12680ca3e0 2023-12-19T15:54:37.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f1265ffb700 1 -- 172.21.15.50:0/4100664902 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260012430 con 0x7f12680ca3e0 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 -- 172.21.15.50:0/4100664902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 msgr2=0x7f1268159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 --2- 172.21.15.50:0/4100664902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f1268159c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1260009e60 tx=0x7f1260009780 comp rx=0 tx=0).stop 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 -- 172.21.15.50:0/4100664902 shutdown_connections 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 --2- 172.21.15.50:0/4100664902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f1268159c80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 --2- 172.21.15.50:0/4100664902 >> v2:172.21.15.50:3300/0 conn(0x7f12680ca010 0x7f126806a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 -- 172.21.15.50:0/4100664902 >> 172.21.15.50:0/4100664902 conn(0x7f1268068500 msgr2=0x7f12680688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 -- 172.21.15.50:0/4100664902 shutdown_connections 2023-12-19T15:54:37.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.388+0000 7f126d4cd700 1 -- 172.21.15.50:0/4100664902 wait complete. 2023-12-19T15:54:37.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.392+0000 7f126d4cd700 1 Processor -- start 2023-12-19T15:54:37.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.392+0000 7f126d4cd700 1 -- start start 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12680ad150 0x7f126814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f12680b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12680c4e80 con 0x7f12680ca010 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12680c5a00 con 0x7f12680ad150 2023-12-19T15:54:37.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12680c50d0 con 0x7f12680ca3e0 2023-12-19T15:54:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34092/0 (socket says 172.21.15.50:34092) 2023-12-19T15:54:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 -- 172.21.15.50:0/2625853646 learned_addr learned my addr 172.21.15.50:0/2625853646 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f1267fff700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f12680b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f1266ffd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12680ad150 0x7f126814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 -- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 msgr2=0x7f12680b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f12680b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 -- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12680ad150 msgr2=0x7f126814ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12680ad150 0x7f126814ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 -- 172.21.15.50:0/2625853646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580094f0 con 0x7f12680ca010 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f12667fc700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f125c00c5c0 tx=0x7f125c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011590 con 0x7f12680ca010 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c011710 con 0x7f12680ca010 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260009a20 con 0x7f12680ca010 2023-12-19T15:54:37.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011ad0 con 0x7f12680ca010 2023-12-19T15:54:37.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.396+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12680c5690 con 0x7f12680ca010 2023-12-19T15:54:37.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.400+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1268071120 con 0x7f12680ca010 2023-12-19T15:54:37.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.400+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f125c01b3b0 con 0x7f12680ca010 2023-12-19T15:54:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.404+0000 7f124effd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 0x7f1254062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.404+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f125c07bc50 con 0x7f12680ca010 2023-12-19T15:54:37.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.404+0000 7f1266ffd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 0x7f1254062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.404+0000 7f1266ffd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 0x7f1254062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1258006d20 tx=0x7f1258009250 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.408+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f125c04c270 con 0x7f12680ca010 2023-12-19T15:54:37.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.584+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12680c8760 con 0x7f12680ca010 2023-12-19T15:54:37.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.584+0000 7f124effd700 1 -- 172.21.15.50:0/2625853646 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+12437 (secure 0 0 0) 0x7f125c017020 con 0x7f12680ca010 2023-12-19T15:54:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:37.588 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":163,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:36.765231+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:37.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 msgr2=0x7f1254062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 0x7f1254062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1258006d20 tx=0x7f1258009250 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 msgr2=0x7f12680a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f125c00c5c0 tx=0x7f125c00c8d0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 shutdown_connections 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1254060670 0x7f1254062b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12680ca3e0 0x7f12680b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12680ca010 0x7f12680a8850 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 --2- 172.21.15.50:0/2625853646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12680ad150 0x7f126814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 >> 172.21.15.50:0/2625853646 conn(0x7f1268068500 msgr2=0x7f12680ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 shutdown_connections 2023-12-19T15:54:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.588+0000 7f126d4cd700 1 -- 172.21.15.50:0/2625853646 wait complete. 2023-12-19T15:54:37.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T15:54:37.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:54:37.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.772+0000 7fdc032ae700 1 Processor -- start 2023-12-19T15:54:37.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.772+0000 7fdc032ae700 1 -- start start 2023-12-19T15:54:37.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbfc06b1b0 0x7fdbfc06b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc0c7450 con 0x7fdbfc15d410 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc06bea0 con 0x7fdbfc0cab10 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc0c6e00 con 0x7fdbfc06b1b0 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34108/0 (socket says 172.21.15.50:34108) 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 -- 172.21.15.50:0/301593861 learned_addr learned my addr 172.21.15.50:0/301593861 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 -- 172.21.15.50:0/301593861 >> v2:172.21.15.125:3300/0 conn(0x7fdbfc06b1b0 msgr2=0x7fdbfc06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 --2- 172.21.15.50:0/301593861 >> v2:172.21.15.125:3300/0 conn(0x7fdbfc06b1b0 0x7fdbfc06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 -- 172.21.15.50:0/301593861 >> v1:172.21.15.125:6789/0 conn(0x7fdbfc15d410 legacy=0x7fdbfc0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 -- 172.21.15.50:0/301593861 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0c70e0 con 0x7fdbfc0cab10 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc00849700 1 --2- 172.21.15.50:0/301593861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc06a5a0 tx=0x7fdbec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13172f9fe212f97c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdbf3fff700 1 -- 172.21.15.50:0/301593861 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00b820 con 0x7fdbfc0cab10 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdbf3fff700 1 -- 172.21.15.50:0/301593861 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec00b980 con 0x7fdbfc0cab10 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdbf3fff700 1 -- 172.21.15.50:0/301593861 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00bd20 con 0x7fdbfc0cab10 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- 172.21.15.50:0/301593861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc0cab10 msgr2=0x7fdbfc0ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 --2- 172.21.15.50:0/301593861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc06a5a0 tx=0x7fdbec009250 comp rx=0 tx=0).stop 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- 172.21.15.50:0/301593861 shutdown_connections 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 --2- 172.21.15.50:0/301593861 >> v2:172.21.15.125:3300/0 conn(0x7fdbfc06b1b0 0x7fdbfc06b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 --2- 172.21.15.50:0/301593861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc0ab6c0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- 172.21.15.50:0/301593861 >> 172.21.15.50:0/301593861 conn(0x7fdbfc067bb0 msgr2=0x7fdbfc0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- 172.21.15.50:0/301593861 shutdown_connections 2023-12-19T15:54:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.776+0000 7fdc032ae700 1 -- 172.21.15.50:0/301593861 wait complete. 2023-12-19T15:54:37.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.780+0000 7fdc032ae700 1 Processor -- start 2023-12-19T15:54:37.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.780+0000 7fdc032ae700 1 -- start start 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdbfc06b1b0 0x7fdbfc162e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0af430 con 0x7fdbfc15d410 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1638e0 con 0x7fdbfc0cab10 2023-12-19T15:54:37.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc163bc0 con 0x7fdbfc06b1b0 2023-12-19T15:54:37.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34120/0 (socket says 172.21.15.50:34120) 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 -- 172.21.15.50:0/3677568265 learned_addr learned my addr 172.21.15.50:0/3677568265 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc00849700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0104a700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdbfc06b1b0 0x7fdbfc162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 -- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdbfc06b1b0 msgr2=0x7fdbfc162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdbfc06b1b0 0x7fdbfc162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 -- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbfc0cab10 msgr2=0x7fdbfc160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 -- 172.21.15.50:0/3677568265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0094f0 con 0x7fdbfc15d410 2023-12-19T15:54:37.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc0184b700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400bc30 tx=0x7fdbf4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40092e0 con 0x7fdbfc15d410 2023-12-19T15:54:37.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf400c630 con 0x7fdbfc15d410 2023-12-19T15:54:37.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400ca10 con 0x7fdbfc15d410 2023-12-19T15:54:37.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc163ea0 con 0x7fdbfc15d410 2023-12-19T15:54:37.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.784+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc1633d0 con 0x7fdbfc15d410 2023-12-19T15:54:37.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.788+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc0af5a0 con 0x7fdbfc15d410 2023-12-19T15:54:37.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.788+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdbf4012070 con 0x7fdbfc15d410 2023-12-19T15:54:37.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.788+0000 7fdbf17fa700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 0x7fdbe0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:37.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.788+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fdbf407c370 con 0x7fdbfc15d410 2023-12-19T15:54:37.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.788+0000 7fdc0104a700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 0x7fdbe0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:37.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.792+0000 7fdc0104a700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 0x7fdbe0062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8003a30 tx=0x7fdbf8007ea0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:37.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.796+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdbf404c990 con 0x7fdbfc15d410 2023-12-19T15:54:37.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.972+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbfc071120 con 0x7fdbfc15d410 2023-12-19T15:54:37.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdbf17fa700 1 -- 172.21.15.50:0/3677568265 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbf404ffb0 con 0x7fdbfc15d410 2023-12-19T15:54:37.977 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 msgr2=0x7fdbe0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 0x7fdbe0062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8003a30 tx=0x7fdbf8007ea0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 msgr2=0x7fdbfc161510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400bc30 tx=0x7fdbf4009840 comp rx=0 tx=0).stop 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 shutdown_connections 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbfc0cab10 0x7fdbfc160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdbe0060670 0x7fdbe0062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdbfc15d410 0x7fdbfc161510 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 --2- 172.21.15.50:0/3677568265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdbfc06b1b0 0x7fdbfc162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.976+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 >> 172.21.15.50:0/3677568265 conn(0x7fdbfc067bb0 msgr2=0x7fdbfc0c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.980+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 shutdown_connections 2023-12-19T15:54:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:37.980+0000 7fdc032ae700 1 -- 172.21.15.50:0/3677568265 wait complete. 2023-12-19T15:54:38.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T15:54:38.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.164+0000 7f0f6a5cc700 1 Processor -- start 2023-12-19T15:54:38.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.164+0000 7f0f6a5cc700 1 -- start start 2023-12-19T15:54:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f6a5cc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0f6406a540 0x7f0f6406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f6a5cc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f640ca5a0 con 0x7f0f640cc470 2023-12-19T15:54:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f6a5cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6406dac0 con 0x7f0f64158e70 2023-12-19T15:54:38.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f6a5cc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6406d220 con 0x7f0f6406a540 2023-12-19T15:54:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f69dcb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0f640cc470 0x7f0f640cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40576/0 (socket says 172.21.15.50:40576) 2023-12-19T15:54:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f69dcb700 1 -- 172.21.15.50:0/636732730 learned_addr learned my addr 172.21.15.50:0/636732730 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f68dc9700 1 --2- 172.21.15.50:0/636732730 >> v2:172.21.15.50:3300/0 conn(0x7f0f6406a540 0x7f0f6406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344848832 0 0) 0x7f0f640ca5a0 con 0x7f0f640cc470 2023-12-19T15:54:38.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50003570 con 0x7f0f640cc470 2023-12-19T15:54:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1519861246 0 0) 0x7f0f50003570 con 0x7f0f640cc470 2023-12-19T15:54:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f68dc9700 1 -- 172.21.15.50:0/636732730 >> v1:172.21.15.125:6789/0 conn(0x7f0f64158e70 legacy=0x7f0f640afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f68dc9700 1 -- 172.21.15.50:0/636732730 >> v1:172.21.15.50:6789/0 conn(0x7f0f640cc470 legacy=0x7f0f640cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.168+0000 7f0f68dc9700 1 -- 172.21.15.50:0/636732730 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6406d500 con 0x7f0f6406a540 2023-12-19T15:54:38.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f68dc9700 1 --2- 172.21.15.50:0/636732730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f6406c9d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0f540098e0 tx=0x7f0f540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28b2a4c12e2866c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c0017f0 con 0x7f0f6406a540 2023-12-19T15:54:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54011420 con 0x7f0f6406a540 2023-12-19T15:54:38.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f5bfff700 1 -- 172.21.15.50:0/636732730 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f540117c0 con 0x7f0f6406a540 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/636732730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 msgr2=0x7f0f6406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/636732730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f6406c9d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0f540098e0 tx=0x7f0f540091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/636732730 shutdown_connections 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/636732730 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f6406c9d0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/636732730 >> 172.21.15.50:0/636732730 conn(0x7f0f640670e0 msgr2=0x7f0f640674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/636732730 shutdown_connections 2023-12-19T15:54:38.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/636732730 wait complete. 2023-12-19T15:54:38.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.172+0000 7f0f6a5cc700 1 Processor -- start 2023-12-19T15:54:38.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 -- start start 2023-12-19T15:54:38.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f640cc470 0x7f0f640ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f64158e70 0x7f0f64156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640b5ae0 con 0x7f0f6406a540 2023-12-19T15:54:38.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6416ca00 con 0x7f0f640cc470 2023-12-19T15:54:38.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6416cce0 con 0x7f0f64158e70 2023-12-19T15:54:38.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34128/0 (socket says 172.21.15.50:34128) 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 -- 172.21.15.50:0/54328306 learned_addr learned my addr 172.21.15.50:0/54328306 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f68dc9700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f640cc470 0x7f0f640ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f69dcb700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f64158e70 0x7f0f64156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 -- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f64158e70 msgr2=0x7f0f64156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f64158e70 0x7f0f64156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 -- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f640cc470 msgr2=0x7f0f640ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f640cc470 0x7f0f640ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 -- 172.21.15.50:0/54328306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54009420 con 0x7f0f6406a540 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f695ca700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c5c0 tx=0x7f0f6000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60012630 con 0x7f0f6406a540 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f600127b0 con 0x7f0f6406a540 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60012b70 con 0x7f0f6406a540 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.176+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6416cfc0 con 0x7f0f6406a540 2023-12-19T15:54:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.180+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640b30b0 con 0x7f0f6406a540 2023-12-19T15:54:38.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.180+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6404be40 con 0x7f0f6406a540 2023-12-19T15:54:38.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.180+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0f6001b400 con 0x7f0f6406a540 2023-12-19T15:54:38.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.184+0000 7f0f597fa700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 0x7f0f48062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.184+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f0f6007c350 con 0x7f0f6406a540 2023-12-19T15:54:38.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.184+0000 7f0f68dc9700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 0x7f0f48062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.184+0000 7f0f68dc9700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 0x7f0f48062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0f540030c0 tx=0x7f0f54017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.188+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f6004ca10 con 0x7f0f6406a540 2023-12-19T15:54:38.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.364+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0f6406bb10 con 0x7f0f6406a540 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f597fa700 1 -- 172.21.15.50:0/54328306 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v163) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0f60050030 con 0x7f0f6406a540 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T15:54:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:38.370 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T15:54:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 msgr2=0x7f0f48062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 0x7f0f48062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0f540030c0 tx=0x7f0f54017000 comp rx=0 tx=0).stop 2023-12-19T15:54:38.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 msgr2=0x7f0f640b55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c5c0 tx=0x7f0f6000c980 comp rx=0 tx=0).stop 2023-12-19T15:54:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 shutdown_connections 2023-12-19T15:54:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f640cc470 0x7f0f640ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0f48060670 0x7f0f48062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0f64158e70 0x7f0f64156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 --2- 172.21.15.50:0/54328306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6406a540 0x7f0f640b55a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 >> 172.21.15.50:0/54328306 conn(0x7f0f640670e0 msgr2=0x7f0f64158350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 shutdown_connections 2023-12-19T15:54:38.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.368+0000 7f0f6a5cc700 1 -- 172.21.15.50:0/54328306 wait complete. 2023-12-19T15:54:38.390 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T15:54:38.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T15:54:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.560+0000 7ff4f759e700 1 Processor -- start 2023-12-19T15:54:38.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.560+0000 7ff4f759e700 1 -- start start 2023-12-19T15:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4f815a0a0 0x7ff4f815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff4f80c6cf0 0x7ff4f80c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f8153870 con 0x7ff4f80c6cf0 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f8068cc0 con 0x7ff4f815a0a0 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f806d4d0 con 0x7ff4f8068300 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f5d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff4f80c6cf0 0x7ff4f80c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35600/0 (socket says 172.21.15.50:35600) 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 -- 172.21.15.50:0/3805275764 learned_addr learned my addr 172.21.15.50:0/3805275764 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f659c700 1 --2- 172.21.15.50:0/3805275764 >> v2:172.21.15.125:3300/0 conn(0x7ff4f815a0a0 0x7ff4f815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 -- 172.21.15.50:0/3805275764 >> v2:172.21.15.125:3300/0 conn(0x7ff4f815a0a0 msgr2=0x7ff4f815a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 --2- 172.21.15.50:0/3805275764 >> v2:172.21.15.125:3300/0 conn(0x7ff4f815a0a0 0x7ff4f815a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 -- 172.21.15.50:0/3805275764 >> v2:172.21.15.50:3300/0 conn(0x7ff4f80c6cf0 msgr2=0x7ff4f80c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 --2- 172.21.15.50:0/3805275764 >> v2:172.21.15.50:3300/0 conn(0x7ff4f80c6cf0 0x7ff4f80c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 -- 172.21.15.50:0/3805275764 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8159630 con 0x7ff4f8068300 2023-12-19T15:54:38.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f6d9d700 1 --2- 172.21.15.50:0/3805275764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff4f806aad0 tx=0x7ff4f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=920762e4fdc02687 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.564+0000 7ff4f559a700 1 -- 172.21.15.50:0/3805275764 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f000e070 con 0x7ff4f8068300 2023-12-19T15:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f559a700 1 -- 172.21.15.50:0/3805275764 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f000b930 con 0x7ff4f8068300 2023-12-19T15:54:38.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f559a700 1 -- 172.21.15.50:0/3805275764 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00092e0 con 0x7ff4f8068300 2023-12-19T15:54:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- 172.21.15.50:0/3805275764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 msgr2=0x7ff4f8068780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 --2- 172.21.15.50:0/3805275764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff4f806aad0 tx=0x7ff4f0009840 comp rx=0 tx=0).stop 2023-12-19T15:54:38.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- 172.21.15.50:0/3805275764 shutdown_connections 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 --2- 172.21.15.50:0/3805275764 >> v2:172.21.15.50:3300/0 conn(0x7ff4f80c6cf0 0x7ff4f80c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 --2- 172.21.15.50:0/3805275764 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f8068300 0x7ff4f8068780 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 --2- 172.21.15.50:0/3805275764 >> v2:172.21.15.125:3300/0 conn(0x7ff4f815a0a0 0x7ff4f815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- 172.21.15.50:0/3805275764 >> 172.21.15.50:0/3805275764 conn(0x7ff4f80c8c50 msgr2=0x7ff4f80c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- 172.21.15.50:0/3805275764 shutdown_connections 2023-12-19T15:54:38.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- 172.21.15.50:0/3805275764 wait complete. 2023-12-19T15:54:38.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 Processor -- start 2023-12-19T15:54:38.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.568+0000 7ff4f759e700 1 -- start start 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f80c6cf0 0x7ff4f8150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f815a0a0 0x7ff4f80acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80b02c0 con 0x7ff4f8068300 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80af6b0 con 0x7ff4f80c6cf0 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f80af990 con 0x7ff4f815a0a0 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34134/0 (socket says 172.21.15.50:34134) 2023-12-19T15:54:38.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 -- 172.21.15.50:0/401494781 learned_addr learned my addr 172.21.15.50:0/401494781 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:38.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f6d9d700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f815a0a0 0x7ff4f80acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f5d9b700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f80c6cf0 0x7ff4f8150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 -- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f815a0a0 msgr2=0x7ff4f80acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f815a0a0 0x7ff4f80acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 -- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f80c6cf0 msgr2=0x7ff4f8150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f80c6cf0 0x7ff4f8150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 -- 172.21.15.50:0/401494781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0009ae0 con 0x7ff4f8068300 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f659c700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800ec80 tx=0x7ff4e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e800c9d0 con 0x7ff4f8068300 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e800cb30 con 0x7ff4f8068300 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8013470 con 0x7ff4f8068300 2023-12-19T15:54:38.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f80afc70 con 0x7ff4f8068300 2023-12-19T15:54:38.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.572+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f80b0750 con 0x7ff4f8068300 2023-12-19T15:54:38.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.576+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f804be40 con 0x7ff4f8068300 2023-12-19T15:54:38.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.576+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff4e8013820 con 0x7ff4f8068300 2023-12-19T15:54:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.576+0000 7ff4e67fc700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 0x7ff4d8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.576+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5229+0+0 (secure 0 0 0) 0x7ff4e807ca40 con 0x7ff4f8068300 2023-12-19T15:54:38.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.580+0000 7ff4f5d9b700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 0x7ff4d8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:38.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.580+0000 7ff4f5d9b700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 0x7ff4d8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4f816c590 tx=0x7ff4ec008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:38.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.584+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff4e807ee60 con 0x7ff4f8068300 2023-12-19T15:54:38.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:38.764+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff4f80aba30 con 0x7ff4f8068300 2023-12-19T15:54:39.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.768+0000 7ff4e67fc700 1 -- 172.21.15.50:0/401494781 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v164) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff4e804d100 con 0x7ff4f8068300 2023-12-19T15:54:39.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 msgr2=0x7ff4d8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:39.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 0x7ff4d8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4f816c590 tx=0x7ff4ec008040 comp rx=0 tx=0).stop 2023-12-19T15:54:39.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 msgr2=0x7ff4f80ab210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:39.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800ec80 tx=0x7ff4e80091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 shutdown_connections 2023-12-19T15:54:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4f80c6cf0 0x7ff4f8150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff4d8060670 0x7ff4d8062b20 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4f816c590 tx=0x7ff4ec008040 comp rx=0 tx=0).stop 2023-12-19T15:54:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff4f815a0a0 0x7ff4f80acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 --2- 172.21.15.50:0/401494781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4f8068300 0x7ff4f80ab210 secure :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800ec80 tx=0x7ff4e80091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 >> 172.21.15.50:0/401494781 conn(0x7ff4f80c8c50 msgr2=0x7ff4f8152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:39.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 shutdown_connections 2023-12-19T15:54:39.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.776+0000 7ff4f759e700 1 -- 172.21.15.50:0/401494781 wait complete. 2023-12-19T15:54:39.782 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T15:54:39.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T15:54:39.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.984+0000 7f647b1cc700 1 Processor -- start 2023-12-19T15:54:39.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.984+0000 7f647b1cc700 1 -- start start 2023-12-19T15:54:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647b1cc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64740ce2d0 0x7f64740ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647b1cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740caa10 0x7f647415d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:39.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647b1cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740cbbb0 con 0x7f64740cc040 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647b1cc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647415d960 con 0x7f64740ce2d0 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647b1cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647406dc90 con 0x7f64740caa10 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64740ce2d0 0x7f64740ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f64740ce2d0 0x7f64740ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34140/0 (socket says 172.21.15.50:34140) 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 -- 172.21.15.50:0/3224880822 learned_addr learned my addr 172.21.15.50:0/3224880822 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:39.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f64799c9700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740caa10 0x7f647415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 -- 172.21.15.50:0/3224880822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740caa10 msgr2=0x7f647415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740caa10 0x7f647415d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 -- 172.21.15.50:0/3224880822 >> v1:172.21.15.125:6789/0 conn(0x7f64740cc040 legacy=0x7f6474158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 -- 172.21.15.50:0/3224880822 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740cafa0 con 0x7f64740ce2d0 2023-12-19T15:54:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f64791c8700 1 -- 172.21.15.50:0/3224880822 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556462513 0 0) 0x7f64740cbbb0 con 0x7f64740cc040 2023-12-19T15:54:39.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f647a1ca700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740ce2d0 0x7f64740ce6a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f64700099b0 tx=0x7f6470009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c23a8d501ebe14 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f64791c8700 1 -- 172.21.15.50:0/3224880822 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470009ea0 con 0x7f64740ce2d0 2023-12-19T15:54:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f64791c8700 1 -- 172.21.15.50:0/3224880822 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647000b860 con 0x7f64740ce2d0 2023-12-19T15:54:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.988+0000 7f64791c8700 1 -- 172.21.15.50:0/3224880822 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000bc00 con 0x7f64740ce2d0 2023-12-19T15:54:39.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 -- 172.21.15.50:0/3224880822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740ce2d0 msgr2=0x7f64740ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740ce2d0 0x7f64740ce6a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f64700099b0 tx=0x7f6470009250 comp rx=0 tx=0).stop 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 -- 172.21.15.50:0/3224880822 shutdown_connections 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740caa10 0x7f647415d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 --2- 172.21.15.50:0/3224880822 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740ce2d0 0x7f64740ce6a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 -- 172.21.15.50:0/3224880822 >> 172.21.15.50:0/3224880822 conn(0x7f6474067bb0 msgr2=0x7f64740cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 -- 172.21.15.50:0/3224880822 shutdown_connections 2023-12-19T15:54:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.992+0000 7f647b1cc700 1 -- 172.21.15.50:0/3224880822 wait complete. 2023-12-19T15:54:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 Processor -- start 2023-12-19T15:54:39.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 -- start start 2023-12-19T15:54:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:39.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740cc040 0x7f647416d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740ce2d0 0x7f6474146900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647416da80 con 0x7f64740caa10 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474146e40 con 0x7f64740ce2d0 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647b1cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6474169bd0 con 0x7f64740cc040 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647a1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:40.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647a1ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34156/0 (socket says 172.21.15.50:34156) 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647a1ca700 1 -- 172.21.15.50:0/2178110342 learned_addr learned my addr 172.21.15.50:0/2178110342 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f647a9cb700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740ce2d0 0x7f6474146900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:39.996+0000 7f64799c9700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740cc040 0x7f647416d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 -- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740cc040 msgr2=0x7f647416d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740cc040 0x7f647416d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 -- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740ce2d0 msgr2=0x7f6474146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:40.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740ce2d0 0x7f6474146900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 -- 172.21.15.50:0/2178110342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700094f0 con 0x7f64740caa10 2023-12-19T15:54:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647a1ca700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f6470006fd0 tx=0x7f6470017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470000d10 con 0x7f64740caa10 2023-12-19T15:54:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647000bbe0 con 0x7f64740caa10 2023-12-19T15:54:40.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470011690 con 0x7f64740caa10 2023-12-19T15:54:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740c8ee0 con 0x7f64740caa10 2023-12-19T15:54:40.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.000+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64740c9180 con 0x7f64740caa10 2023-12-19T15:54:40.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.004+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6474155cf0 con 0x7f64740caa10 2023-12-19T15:54:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.004+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6470011a80 con 0x7f64740caa10 2023-12-19T15:54:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.004+0000 7f646a7fc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 0x7f6458062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.004+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5599+0+0 (secure 0 0 0) 0x7f647007e040 con 0x7f64740caa10 2023-12-19T15:54:40.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.008+0000 7f64799c9700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 0x7f6458062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:40.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.008+0000 7f64799c9700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 0x7f6458062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64740c9e00 tx=0x7f64640091a0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:40.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.016+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f64700adbe0 con 0x7f64740caa10 2023-12-19T15:54:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.188+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f647404be40 con 0x7f64740caa10 2023-12-19T15:54:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.784+0000 7f646a7fc700 1 -- 172.21.15.50:0/2178110342 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v165) v1 ==== 170+0+0 (secure 0 0 0) 0x7f64700150d0 con 0x7f64740caa10 2023-12-19T15:54:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 msgr2=0x7f6458062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 0x7f6458062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64740c9e00 tx=0x7f64640091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 msgr2=0x7f647416a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f6470006fd0 tx=0x7f6470017000 comp rx=0 tx=0).stop 2023-12-19T15:54:40.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 shutdown_connections 2023-12-19T15:54:40.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740ce2d0 0x7f6474146900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6458060670 0x7f6458062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f64740cc040 0x7f647416d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 --2- 172.21.15.50:0/2178110342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f64740caa10 0x7f647416a060 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:40.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.796+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 >> 172.21.15.50:0/2178110342 conn(0x7f6474067bb0 msgr2=0x7f64741555e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:40.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.800+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 shutdown_connections 2023-12-19T15:54:40.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:40.800+0000 7f647b1cc700 1 -- 172.21.15.50:0/2178110342 wait complete. 2023-12-19T15:54:40.812 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T15:54:40.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T15:54:41.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.000+0000 7f8ab58ae700 1 Processor -- start 2023-12-19T15:54:41.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.000+0000 7f8ab58ae700 1 -- start start 2023-12-19T15:54:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8ab58ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 0x7f8ab00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8ab58ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8ab58ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00cbdf0 con 0x7f8ab00cb970 2023-12-19T15:54:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8ab58ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab00cc0d0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8ab58ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab00cc3b0 con 0x7f8ab0158e20 2023-12-19T15:54:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34172/0 (socket says 172.21.15.50:34172) 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 -- 172.21.15.50:0/2793928924 learned_addr learned my addr 172.21.15.50:0/2793928924 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaeffd700 1 --2- 172.21.15.50:0/2793928924 >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 0x7f8ab00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 -- 172.21.15.50:0/2793928924 >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 msgr2=0x7f8ab00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 --2- 172.21.15.50:0/2793928924 >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 0x7f8ab00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 -- 172.21.15.50:0/2793928924 >> v1:172.21.15.125:6789/0 conn(0x7f8ab00cb970 legacy=0x7f8ab006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 -- 172.21.15.50:0/2793928924 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00cc690 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaeffd700 1 --2- 172.21.15.50:0/2793928924 >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 0x7f8ab00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aaf7fe700 1 --2- 172.21.15.50:0/2793928924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00cdc10 tx=0x7f8aa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc680328efb35e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.004+0000 7f8aadffb700 1 -- 172.21.15.50:0/2793928924 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa800e070 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8aadffb700 1 -- 172.21.15.50:0/2793928924 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa800b9d0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8aadffb700 1 -- 172.21.15.50:0/2793928924 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80092e0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 -- 172.21.15.50:0/2793928924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 msgr2=0x7f8ab00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/2793928924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00cdc10 tx=0x7f8aa8009840 comp rx=0 tx=0).stop 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 -- 172.21.15.50:0/2793928924 shutdown_connections 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/2793928924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab00ceb00 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/2793928924 >> v2:172.21.15.125:3300/0 conn(0x7f8ab0158e20 0x7f8ab00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 -- 172.21.15.50:0/2793928924 >> 172.21.15.50:0/2793928924 conn(0x7f8ab0067bb0 msgr2=0x7f8ab00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:41.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 -- 172.21.15.50:0/2793928924 shutdown_connections 2023-12-19T15:54:41.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.008+0000 7f8ab58ae700 1 -- 172.21.15.50:0/2793928924 wait complete. 2023-12-19T15:54:41.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 Processor -- start 2023-12-19T15:54:41.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 -- start start 2023-12-19T15:54:41.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8ab0158e20 0x7f8ab00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab00b34f0 0x7f8ab016ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab00b5e60 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab016c980 con 0x7f8ab00b34f0 2023-12-19T15:54:41.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.012+0000 7f8ab58ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0169880 con 0x7f8ab0158e20 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34186/0 (socket says 172.21.15.50:34186) 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 -- 172.21.15.50:0/3603186535 learned_addr learned my addr 172.21.15.50:0/3603186535 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aae7fc700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8ab0158e20 0x7f8ab00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaf7fe700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab00b34f0 0x7f8ab016ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 -- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8ab0158e20 msgr2=0x7f8ab00b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8ab0158e20 0x7f8ab00b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 -- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab00b34f0 msgr2=0x7f8ab016ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:41.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab00b34f0 0x7f8ab016ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 -- 172.21.15.50:0/3603186535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8009ae0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8aaeffd700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000c850 tx=0x7f8aa0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:41.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0011600 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa0011760 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0011b00 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab0169b60 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.016+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab016a120 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.020+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab0071120 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.020+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8aa0023480 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.024+0000 7f8a96ffd700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 0x7f8a9c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.024+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f8aa007cde0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.024+0000 7f8aae7fc700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 0x7f8a9c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:41.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.024+0000 7f8aae7fc700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 0x7f8a9c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00637d0 tx=0x7f8aa4007ea0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:41.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.028+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8aa0050b30 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.244+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8ab006a7a0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.812+0000 7f8a96ffd700 1 -- 172.21.15.50:0/3603186535 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8aa00506c0 con 0x7f8ab00ca2c0 2023-12-19T15:54:41.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 msgr2=0x7f8a9c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 0x7f8a9c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00637d0 tx=0x7f8aa4007ea0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 msgr2=0x7f8ab014fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000c850 tx=0x7f8aa0008040 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 shutdown_connections 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ab00b34f0 0x7f8ab016ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8a9c060670 0x7f8a9c062b20 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ab00637d0 tx=0x7f8aa4007ea0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8ab0158e20 0x7f8ab00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 --2- 172.21.15.50:0/3603186535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ab00ca2c0 0x7f8ab014fe60 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.816+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 >> 172.21.15.50:0/3603186535 conn(0x7f8ab0067bb0 msgr2=0x7f8ab006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:41.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.820+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 shutdown_connections 2023-12-19T15:54:41.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:41.824+0000 7f8ab58ae700 1 -- 172.21.15.50:0/3603186535 wait complete. 2023-12-19T15:54:41.830 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 19 and data pool 20 2023-12-19T15:54:41.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.012+0000 7f178facd700 1 Processor -- start 2023-12-19T15:54:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 -- start start 2023-12-19T15:54:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f17880c15b0 0x7f178815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17880aeba0 0x7f17880680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880c7190 con 0x7f178815d320 2023-12-19T15:54:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880aefd0 con 0x7f17880c15b0 2023-12-19T15:54:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178facd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880c1b50 con 0x7f17880aeba0 2023-12-19T15:54:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178d068700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f17880c15b0 0x7f178815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178e06a700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17880aeba0 0x7f17880680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178d869700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f178815d320 0x7f178806d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40582/0 (socket says 172.21.15.50:40582) 2023-12-19T15:54:42.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.016+0000 7f178d869700 1 -- 172.21.15.50:0/2214206127 learned_addr learned my addr 172.21.15.50:0/2214206127 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178e06a700 1 -- 172.21.15.50:0/2214206127 >> v2:172.21.15.50:3300/0 conn(0x7f17880c15b0 msgr2=0x7f178815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178e06a700 1 --2- 172.21.15.50:0/2214206127 >> v2:172.21.15.50:3300/0 conn(0x7f17880c15b0 0x7f178815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178e06a700 1 -- 172.21.15.50:0/2214206127 >> v1:172.21.15.50:6789/0 conn(0x7f178815d320 legacy=0x7f178806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178e06a700 1 -- 172.21.15.50:0/2214206127 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880c1e30 con 0x7f17880aeba0 2023-12-19T15:54:42.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178e06a700 1 --2- 172.21.15.50:0/2214206127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f17880680f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f178806e2b0 tx=0x7f1780009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda4d56baadec4f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178c867700 1 -- 172.21.15.50:0/2214206127 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000e070 con 0x7f17880aeba0 2023-12-19T15:54:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178c867700 1 -- 172.21.15.50:0/2214206127 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178000bb60 con 0x7f17880aeba0 2023-12-19T15:54:42.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178c867700 1 -- 172.21.15.50:0/2214206127 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780012430 con 0x7f17880aeba0 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 -- 172.21.15.50:0/2214206127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 msgr2=0x7f17880680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 --2- 172.21.15.50:0/2214206127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f17880680f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f178806e2b0 tx=0x7f1780009780 comp rx=0 tx=0).stop 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 -- 172.21.15.50:0/2214206127 shutdown_connections 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 --2- 172.21.15.50:0/2214206127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f17880680f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 --2- 172.21.15.50:0/2214206127 >> v2:172.21.15.50:3300/0 conn(0x7f17880c15b0 0x7f178815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 -- 172.21.15.50:0/2214206127 >> 172.21.15.50:0/2214206127 conn(0x7f17880c8850 msgr2=0x7f17880c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 -- 172.21.15.50:0/2214206127 shutdown_connections 2023-12-19T15:54:42.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.020+0000 7f178facd700 1 -- 172.21.15.50:0/2214206127 wait complete. 2023-12-19T15:54:42.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 Processor -- start 2023-12-19T15:54:42.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 -- start start 2023-12-19T15:54:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f1788154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f178815d320 0x7f17880c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1788160c70 con 0x7f17880c15b0 2023-12-19T15:54:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1788161ab0 con 0x7f17880aeba0 2023-12-19T15:54:42.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.024+0000 7f178facd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1788160ea0 con 0x7f178815d320 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d068700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d068700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34204/0 (socket says 172.21.15.50:34204) 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d068700 1 -- 172.21.15.50:0/3192966778 learned_addr learned my addr 172.21.15.50:0/3192966778 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f1788154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178e06a700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f178815d320 0x7f17880c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 -- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f178815d320 msgr2=0x7f17880c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f178815d320 0x7f17880c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 -- 172.21.15.50:0/3192966778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 msgr2=0x7f1788160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 -- 172.21.15.50:0/3192966778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780009a20 con 0x7f17880aeba0 2023-12-19T15:54:42.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d068700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178d869700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f1788154320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f178400c4a0 tx=0x7f1784007f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784005b30 con 0x7f17880aeba0 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1784005cb0 con 0x7f17880aeba0 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1788161180 con 0x7f17880aeba0 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784007320 con 0x7f17880aeba0 2023-12-19T15:54:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.028+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1788161740 con 0x7f17880aeba0 2023-12-19T15:54:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.032+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1788071120 con 0x7f17880aeba0 2023-12-19T15:54:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.032+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f178401b3b0 con 0x7f17880aeba0 2023-12-19T15:54:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.032+0000 7f177dffb700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 0x7f176c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.032+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f178407bf20 con 0x7f17880aeba0 2023-12-19T15:54:42.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.036+0000 7f178d068700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 0x7f176c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.036+0000 7f178d068700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 0x7f176c062bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1778009fd0 tx=0x7f17780091a0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.040+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f178404c260 con 0x7f17880aeba0 2023-12-19T15:54:42.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.216+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17880642a0 con 0x7f17880aeba0 2023-12-19T15:54:42.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.220+0000 7f177dffb700 1 -- 172.21.15.50:0/3192966778 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+15670 (secure 0 0 0) 0x7f1784017090 con 0x7f17880aeba0 2023-12-19T15:54:42.223 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:42.223 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":166,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:41.247214+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:42.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.220+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 msgr2=0x7f176c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 0x7f176c062bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1778009fd0 tx=0x7f17780091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 msgr2=0x7f1788154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f1788154320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f178400c4a0 tx=0x7f1784007f10 comp rx=0 tx=0).stop 2023-12-19T15:54:42.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 shutdown_connections 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f176c060740 0x7f176c062bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f178815d320 0x7f17880c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17880c15b0 0x7f1788160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 --2- 172.21.15.50:0/3192966778 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17880aeba0 0x7f1788154320 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 >> 172.21.15.50:0/3192966778 conn(0x7f17880c8850 msgr2=0x7f17880c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 shutdown_connections 2023-12-19T15:54:42.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.224+0000 7f178facd700 1 -- 172.21.15.50:0/3192966778 wait complete. 2023-12-19T15:54:42.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:42.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.412+0000 7f16264cd700 1 Processor -- start 2023-12-19T15:54:42.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.412+0000 7f16264cd700 1 -- start start 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f16264cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f16200cc610 0x7f16200c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f16264cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200c6630 con 0x7f1620067c90 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f16264cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200c6910 con 0x7f16201502b0 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f16264cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16200c6bf0 con 0x7f16200cc610 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f161ffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1620067c90 0x7f162014fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40588/0 (socket says 172.21.15.50:40588) 2023-12-19T15:54:42.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f161ffff700 1 -- 172.21.15.50:0/4245051202 learned_addr learned my addr 172.21.15.50:0/4245051202 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f1624a6a700 1 --2- 172.21.15.50:0/4245051202 >> v2:172.21.15.50:3300/0 conn(0x7f16200cc610 0x7f16200c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992664789 0 0) 0x7f16200c6630 con 0x7f1620067c90 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.416+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1608003570 con 0x7f1620067c90 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563404854 0 0) 0x7f16200c6910 con 0x7f16201502b0 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16200c6630 con 0x7f16201502b0 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f1624a6a700 1 -- 172.21.15.50:0/4245051202 >> v1:172.21.15.125:6789/0 conn(0x7f16201502b0 legacy=0x7f16200c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f1624a6a700 1 -- 172.21.15.50:0/4245051202 >> v1:172.21.15.50:6789/0 conn(0x7f1620067c90 legacy=0x7f162014fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 966776504 0 0) 0x7f1608003570 con 0x7f1620067c90 2023-12-19T15:54:42.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f1624a6a700 1 -- 172.21.15.50:0/4245051202 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16200c6ed0 con 0x7f16200cc610 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f1624a6a700 1 --2- 172.21.15.50:0/4245051202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200c6010 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16180098e0 tx=0x7f16180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23425ccc26abde6f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16100017f0 con 0x7f16200cc610 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618011420 con 0x7f16200cc610 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f161effd700 1 -- 172.21.15.50:0/4245051202 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16180117c0 con 0x7f16200cc610 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 -- 172.21.15.50:0/4245051202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 msgr2=0x7f16200c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 --2- 172.21.15.50:0/4245051202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200c6010 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16180098e0 tx=0x7f16180091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 -- 172.21.15.50:0/4245051202 shutdown_connections 2023-12-19T15:54:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 --2- 172.21.15.50:0/4245051202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200c6010 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 -- 172.21.15.50:0/4245051202 >> 172.21.15.50:0/4245051202 conn(0x7f16200cd6d0 msgr2=0x7f162006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 -- 172.21.15.50:0/4245051202 shutdown_connections 2023-12-19T15:54:42.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 -- 172.21.15.50:0/4245051202 wait complete. 2023-12-19T15:54:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.420+0000 7f16264cd700 1 Processor -- start 2023-12-19T15:54:42.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- start start 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1620067c90 0x7f1620153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f16201502b0 0x7f16200b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16200b0aa0 con 0x7f16200cc610 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16200b3250 con 0x7f1620067c90 2023-12-19T15:54:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16200b0cf0 con 0x7f16201502b0 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34224/0 (socket says 172.21.15.50:34224) 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 -- 172.21.15.50:0/4053468262 learned_addr learned my addr 172.21.15.50:0/4053468262 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161ffff700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1620067c90 0x7f1620153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f1624a6a700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f16201502b0 0x7f16200b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.428 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 -- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f16201502b0 msgr2=0x7f16200b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f16201502b0 0x7f16200b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 -- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1620067c90 msgr2=0x7f1620153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1620067c90 0x7f1620153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 -- 172.21.15.50:0/4053468262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618009420 con 0x7f16200cc610 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f161f7fe700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f161400c580 tx=0x7f1614007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614010510 con 0x7f16200cc610 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1614010670 con 0x7f16200cc610 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16200b0fd0 con 0x7f16200cc610 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614010a30 con 0x7f16200cc610 2023-12-19T15:54:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.424+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16200b1590 con 0x7f16200cc610 2023-12-19T15:54:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.428+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1620071120 con 0x7f16200cc610 2023-12-19T15:54:42.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.428+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f161401b400 con 0x7f16200cc610 2023-12-19T15:54:42.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.428+0000 7f1607fff700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 0x7f160c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.428+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f161407c760 con 0x7f16200cc610 2023-12-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.432+0000 7f161ffff700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 0x7f160c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.432+0000 7f161ffff700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 0x7f160c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1610005fd0 tx=0x7f1610000f40 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.436+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f161404cac0 con 0x7f16200cc610 2023-12-19T15:54:42.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.612+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16200642a0 con 0x7f16200cc610 2023-12-19T15:54:42.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.612+0000 7f1607fff700 1 -- 172.21.15.50:0/4053468262 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+15670 (secure 0 0 0) 0x7f16140500e0 con 0x7f16200cc610 2023-12-19T15:54:42.616 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:42.617 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":166,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:41.247214+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 msgr2=0x7f160c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 0x7f160c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1610005fd0 tx=0x7f1610000f40 comp rx=0 tx=0).stop 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 msgr2=0x7f16200b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f161400c580 tx=0x7f1614007e10 comp rx=0 tx=0).stop 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 shutdown_connections 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f160c060670 0x7f160c062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f16201502b0 0x7f16200b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16200cc610 0x7f16200b1d90 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 --2- 172.21.15.50:0/4053468262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1620067c90 0x7f1620153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 >> 172.21.15.50:0/4053468262 conn(0x7f16200cd6d0 msgr2=0x7f16200cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:42.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 shutdown_connections 2023-12-19T15:54:42.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.616+0000 7f16264cd700 1 -- 172.21.15.50:0/4053468262 wait complete. 2023-12-19T15:54:42.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T15:54:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.808+0000 7f95d226e700 1 Processor -- start 2023-12-19T15:54:42.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d226e700 1 -- start start 2023-12-19T15:54:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d226e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 0x7f95cc06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d226e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0cbdf0 con 0x7f95cc0cb970 2023-12-19T15:54:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d226e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc0cc0d0 con 0x7f95cc158e20 2023-12-19T15:54:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d226e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc0cc3b0 con 0x7f95cc0cecd0 2023-12-19T15:54:42.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95cb7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f95cc0cb970 0x7f95cc06acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40602/0 (socket says 172.21.15.50:40602) 2023-12-19T15:54:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95cb7fe700 1 -- 172.21.15.50:0/3847895177 learned_addr learned my addr 172.21.15.50:0/3847895177 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d080b700 1 --2- 172.21.15.50:0/3847895177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 0x7f95cc06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d080b700 1 -- 172.21.15.50:0/3847895177 >> v1:172.21.15.125:6789/0 conn(0x7f95cc158e20 legacy=0x7f95cc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d080b700 1 -- 172.21.15.50:0/3847895177 >> v1:172.21.15.50:6789/0 conn(0x7f95cc0cb970 legacy=0x7f95cc06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.812+0000 7f95d080b700 1 -- 172.21.15.50:0/3847895177 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc0cc690 con 0x7f95cc0cecd0 2023-12-19T15:54:42.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.816+0000 7f95d080b700 1 --2- 172.21.15.50:0/3847895177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 0x7f95cc06b3d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009ec0 tx=0x7f95c4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed4f788843d277b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95caffd700 1 -- 172.21.15.50:0/3847895177 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c400e070 con 0x7f95cc0cecd0 2023-12-19T15:54:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95caffd700 1 -- 172.21.15.50:0/3847895177 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c400bbf0 con 0x7f95cc0cecd0 2023-12-19T15:54:42.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95caffd700 1 -- 172.21.15.50:0/3847895177 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4012430 con 0x7f95cc0cecd0 2023-12-19T15:54:42.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 -- 172.21.15.50:0/3847895177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 msgr2=0x7f95cc06b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 --2- 172.21.15.50:0/3847895177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 0x7f95cc06b3d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009ec0 tx=0x7f95c4009780 comp rx=0 tx=0).stop 2023-12-19T15:54:42.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 -- 172.21.15.50:0/3847895177 shutdown_connections 2023-12-19T15:54:42.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 --2- 172.21.15.50:0/3847895177 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc0cecd0 0x7f95cc06b3d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 -- 172.21.15.50:0/3847895177 >> 172.21.15.50:0/3847895177 conn(0x7f95cc067bb0 msgr2=0x7f95cc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:42.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 -- 172.21.15.50:0/3847895177 shutdown_connections 2023-12-19T15:54:42.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.824+0000 7f95d226e700 1 -- 172.21.15.50:0/3847895177 wait complete. 2023-12-19T15:54:42.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.828+0000 7f95d226e700 1 Processor -- start 2023-12-19T15:54:42.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 -- start start 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 0x7f95cc0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc1557f0 0x7f95cc156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc0b5470 con 0x7f95cc158e20 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc0b4610 con 0x7f95cc0cecd0 2023-12-19T15:54:42.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d226e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc0b5e10 con 0x7f95cc1557f0 2023-12-19T15:54:42.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34236/0 (socket says 172.21.15.50:34236) 2023-12-19T15:54:42.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cb7fe700 1 -- 172.21.15.50:0/1229497197 learned_addr learned my addr 172.21.15.50:0/1229497197 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:42.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 0x7f95cc0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95d080b700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc1557f0 0x7f95cc156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 -- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc1557f0 msgr2=0x7f95cc156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc1557f0 0x7f95cc156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 -- 172.21.15.50:0/1229497197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 msgr2=0x7f95cc149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cbfff700 1 -- 172.21.15.50:0/1229497197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4009a20 con 0x7f95cc0cecd0 2023-12-19T15:54:42.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.832+0000 7f95cb7fe700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95cbfff700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 0x7f95cc0b40d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c4c0 tx=0x7f95bc007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00ce30 con 0x7f95cc0cecd0 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc011590 con 0x7f95cc0cecd0 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc0b4860 con 0x7f95cc0cecd0 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc011950 con 0x7f95cc0cecd0 2023-12-19T15:54:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc0b4e20 con 0x7f95cc0cecd0 2023-12-19T15:54:42.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.836+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc071120 con 0x7f95cc0cecd0 2023-12-19T15:54:42.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.840+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f95bc011d00 con 0x7f95cc0cecd0 2023-12-19T15:54:42.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.840+0000 7f95b3fff700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 0x7f95b8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:42.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.840+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f95bc07bdd0 con 0x7f95cc0cecd0 2023-12-19T15:54:42.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.840+0000 7f95cb7fe700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 0x7f95b8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:42.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.840+0000 7f95cb7fe700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 0x7f95b8062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f95cc14b420 tx=0x7f95c0007f40 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:42.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:42.844+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f95bc04c090 con 0x7f95cc0cecd0 2023-12-19T15:54:43.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.060+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f95cc06a7a0 con 0x7f95cc0cecd0 2023-12-19T15:54:43.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.828+0000 7f95b3fff700 1 -- 172.21.15.50:0/1229497197 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v74) v1 ==== 118+0+0 (secure 0 0 0) 0x7f95bc04f6b0 con 0x7f95cc0cecd0 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 msgr2=0x7f95b8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 0x7f95b8062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f95cc14b420 tx=0x7f95c0007f40 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 msgr2=0x7f95cc0b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 0x7f95cc0b40d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00c4c0 tx=0x7f95bc007ec0 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 shutdown_connections 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f95b8060670 0x7f95b8062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95cc1557f0 0x7f95cc156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95cc158e20 0x7f95cc149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 --2- 172.21.15.50:0/1229497197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95cc0cecd0 0x7f95cc0b40d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 >> 172.21.15.50:0/1229497197 conn(0x7f95cc067bb0 msgr2=0x7f95cc156070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:43.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 shutdown_connections 2023-12-19T15:54:43.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:43.832+0000 7f95d226e700 1 -- 172.21.15.50:0/1229497197 wait complete. 2023-12-19T15:54:43.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T15:54:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.024+0000 7fef4b0f0700 1 Processor -- start 2023-12-19T15:54:44.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.024+0000 7fef4b0f0700 1 -- start start 2023-12-19T15:54:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4b0f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef440ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4b0f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fef440ca2c0 0x7fef440ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4b0f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef440cbdf0 con 0x7fef440cb970 2023-12-19T15:54:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4b0f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef440cc0d0 con 0x7fef440ca2c0 2023-12-19T15:54:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4b0f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef440cc3b0 con 0x7fef44158e20 2023-12-19T15:54:44.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef48e8c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef440ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef43fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fef440cb970 0x7fef4406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:59916/0 (socket says 172.21.15.50:59916) 2023-12-19T15:54:44.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef43fff700 1 -- 172.21.15.50:0/477332642 learned_addr learned my addr 172.21.15.50:0/477332642 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:44.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 --2- 172.21.15.50:0/477332642 >> v2:172.21.15.125:3300/0 conn(0x7fef440ca2c0 0x7fef440ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 -- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 msgr2=0x7fef440ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 --2- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef440ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 -- 172.21.15.50:0/477332642 >> v1:172.21.15.125:6789/0 conn(0x7fef440cb970 legacy=0x7fef4406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 -- 172.21.15.50:0/477332642 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef440cc690 con 0x7fef440ca2c0 2023-12-19T15:54:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.028+0000 7fef4968d700 1 --2- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef440ca2c0 0x7fef440ceb00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009e60 tx=0x7fef3c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=338971333aa8cb8d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef437fe700 1 -- 172.21.15.50:0/477332642 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00e070 con 0x7fef440ca2c0 2023-12-19T15:54:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef437fe700 1 -- 172.21.15.50:0/477332642 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c00bb60 con 0x7fef440ca2c0 2023-12-19T15:54:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef437fe700 1 -- 172.21.15.50:0/477332642 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c012430 con 0x7fef440ca2c0 2023-12-19T15:54:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 -- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef440ca2c0 msgr2=0x7fef440ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef440ca2c0 0x7fef440ceb00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009e60 tx=0x7fef3c009780 comp rx=0 tx=0).stop 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 -- 172.21.15.50:0/477332642 shutdown_connections 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef440ca2c0 0x7fef440ceb00 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/477332642 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef440ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 -- 172.21.15.50:0/477332642 >> 172.21.15.50:0/477332642 conn(0x7fef44067bb0 msgr2=0x7fef440d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 -- 172.21.15.50:0/477332642 shutdown_connections 2023-12-19T15:54:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 -- 172.21.15.50:0/477332642 wait complete. 2023-12-19T15:54:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.032+0000 7fef4b0f0700 1 Processor -- start 2023-12-19T15:54:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 -- start start 2023-12-19T15:54:44.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef4415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 0x7fef4415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440b5380 con 0x7fef440ca2c0 2023-12-19T15:54:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440b5810 con 0x7fef4415d3e0 2023-12-19T15:54:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4b0f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef44146180 con 0x7fef44158e20 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34244/0 (socket says 172.21.15.50:34244) 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 -- 172.21.15.50:0/2665898672 learned_addr learned my addr 172.21.15.50:0/2665898672 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef43fff700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef4415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4968d700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 0x7fef4415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 -- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 msgr2=0x7fef4415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef4415cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 -- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 msgr2=0x7fef4415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 0x7fef4415e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 -- 172.21.15.50:0/2665898672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c009a20 con 0x7fef440ca2c0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef4968d700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 0x7fef4415e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef48e8c700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fef3800e960 tx=0x7fef380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.036+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38013680 con 0x7fef440ca2c0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef380137e0 con 0x7fef440ca2c0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef440b4490 con 0x7fef440ca2c0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38013b80 con 0x7fef440ca2c0 2023-12-19T15:54:44.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef440b4a50 con 0x7fef440ca2c0 2023-12-19T15:54:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef44071120 con 0x7fef440ca2c0 2023-12-19T15:54:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.040+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fef3800c490 con 0x7fef440ca2c0 2023-12-19T15:54:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.044+0000 7fef40ff9700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 0x7fef28062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.044+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fef3807ccb0 con 0x7fef440ca2c0 2023-12-19T15:54:44.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.044+0000 7fef43fff700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 0x7fef28062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:44.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.044+0000 7fef43fff700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 0x7fef28062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fef440cf3c0 tx=0x7fef30007ed0 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:44.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.048+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fef3804cf70 con 0x7fef440ca2c0 2023-12-19T15:54:44.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.260+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fef4406bd50 con 0x7fef440ca2c0 2023-12-19T15:54:44.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.836+0000 7fef40ff9700 1 -- 172.21.15.50:0/2665898672 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef38050590 con 0x7fef440ca2c0 2023-12-19T15:54:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 msgr2=0x7fef28062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 0x7fef28062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fef440cf3c0 tx=0x7fef30007ed0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 msgr2=0x7fef44145c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fef3800e960 tx=0x7fef380091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 shutdown_connections 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef4415d3e0 0x7fef4415e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fef28060670 0x7fef28062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef44158e20 0x7fef4415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 --2- 172.21.15.50:0/2665898672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef440ca2c0 0x7fef44145c40 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 >> 172.21.15.50:0/2665898672 conn(0x7fef44067bb0 msgr2=0x7fef4406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:44.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 shutdown_connections 2023-12-19T15:54:44.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:44.840+0000 7fef4b0f0700 1 -- 172.21.15.50:0/2665898672 wait complete. 2023-12-19T15:54:44.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 Processor -- start 2023-12-19T15:54:45.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 -- start start 2023-12-19T15:54:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 0x7f1a4c0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c0cbea0 con 0x7f1a4c0cc330 2023-12-19T15:54:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c0cb030 con 0x7f1a4c0cc780 2023-12-19T15:54:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a51865700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0cb2d0 con 0x7f1a4c0cea90 2023-12-19T15:54:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a4a7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1a4c0cc330 0x7f1a4c068a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40618/0 (socket says 172.21.15.50:40618) 2023-12-19T15:54:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a4a7fc700 1 -- 172.21.15.50:0/3650563213 learned_addr learned my addr 172.21.15.50:0/3650563213 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.036+0000 7f1a4affd700 1 --2- 172.21.15.50:0/3650563213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 0x7f1a4c0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473990217 0 0) 0x7f1a4c0cbea0 con 0x7f1a4c0cc330 2023-12-19T15:54:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a34003570 con 0x7f1a4c0cc330 2023-12-19T15:54:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212558684 0 0) 0x7f1a4c0cb030 con 0x7f1a4c0cc780 2023-12-19T15:54:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a4c0cbea0 con 0x7f1a4c0cc780 2023-12-19T15:54:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a4affd700 1 -- 172.21.15.50:0/3650563213 >> v1:172.21.15.125:6789/0 conn(0x7f1a4c0cc780 legacy=0x7f1a4c142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a4affd700 1 -- 172.21.15.50:0/3650563213 >> v1:172.21.15.50:6789/0 conn(0x7f1a4c0cc330 legacy=0x7f1a4c068a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a4affd700 1 -- 172.21.15.50:0/3650563213 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c067e00 con 0x7f1a4c0cea90 2023-12-19T15:54:45.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a4affd700 1 --2- 172.21.15.50:0/3650563213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 0x7f1a4c0cee60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009910 tx=0x7f1a3c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a8b3c94ec51d0c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c011630 con 0x7f1a4c0cea90 2023-12-19T15:54:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c011790 con 0x7f1a4c0cea90 2023-12-19T15:54:45.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a49ffb700 1 -- 172.21.15.50:0/3650563213 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c011b30 con 0x7f1a4c0cea90 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 -- 172.21.15.50:0/3650563213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 msgr2=0x7f1a4c0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 --2- 172.21.15.50:0/3650563213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 0x7f1a4c0cee60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009910 tx=0x7f1a3c0091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 -- 172.21.15.50:0/3650563213 shutdown_connections 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 --2- 172.21.15.50:0/3650563213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cea90 0x7f1a4c0cee60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 -- 172.21.15.50:0/3650563213 >> 172.21.15.50:0/3650563213 conn(0x7f1a4c0cd650 msgr2=0x7f1a4c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.040+0000 7f1a51865700 1 -- 172.21.15.50:0/3650563213 shutdown_connections 2023-12-19T15:54:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.044+0000 7f1a51865700 1 -- 172.21.15.50:0/3650563213 wait complete. 2023-12-19T15:54:45.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.044+0000 7f1a51865700 1 Processor -- start 2023-12-19T15:54:45.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 -- start start 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a4c0cc330 0x7f1a4c1699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cc780 0x7f1a4c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c169f10 con 0x7f1a4c0cea90 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c15d880 con 0x7f1a4c0cc330 2023-12-19T15:54:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a51865700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0b30e0 con 0x7f1a4c0cc780 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34252/0 (socket says 172.21.15.50:34252) 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 -- 172.21.15.50:0/373074176 learned_addr learned my addr 172.21.15.50:0/373074176 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4a7fc700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cc780 0x7f1a4c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4affd700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a4c0cc330 0x7f1a4c1699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 -- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cc780 msgr2=0x7f1a4c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cc780 0x7f1a4c15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 -- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a4c0cc330 msgr2=0x7f1a4c1699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a4c0cc330 0x7f1a4c1699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.048+0000 7f1a4b7fe700 1 -- 172.21.15.50:0/373074176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c009420 con 0x7f1a4c0cea90 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a4b7fe700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c940 tx=0x7f1a44007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44012630 con 0x7f1a4c0cea90 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c0b3350 con 0x7f1a4c0cea90 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a440127b0 con 0x7f1a4c0cea90 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44012b70 con 0x7f1a4c0cea90 2023-12-19T15:54:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c0b3910 con 0x7f1a4c0cea90 2023-12-19T15:54:45.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.052+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c0b3b30 con 0x7f1a4c0cea90 2023-12-19T15:54:45.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.056+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1a4401b400 con 0x7f1a4c0cea90 2023-12-19T15:54:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.056+0000 7f1a32ffd700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 0x7f1a38062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.056+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f1a4407ced0 con 0x7f1a4c0cea90 2023-12-19T15:54:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.056+0000 7f1a4affd700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 0x7f1a38062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.060+0000 7f1a4affd700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 0x7f1a38062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0091e0 tx=0x7f1a3c017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.064+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a4404d190 con 0x7f1a4c0cea90 2023-12-19T15:54:45.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.280+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a4c071120 con 0x7f1a4c0cea90 2023-12-19T15:54:45.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.280+0000 7f1a32ffd700 1 -- 172.21.15.50:0/373074176 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f1a44016090 con 0x7f1a4c0cea90 2023-12-19T15:54:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:45.285 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:54:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 msgr2=0x7f1a38062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 0x7f1a38062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0091e0 tx=0x7f1a3c017000 comp rx=0 tx=0).stop 2023-12-19T15:54:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 msgr2=0x7f1a4c0c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c940 tx=0x7f1a44007f90 comp rx=0 tx=0).stop 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 shutdown_connections 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1a38060670 0x7f1a38062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1a4c0cea90 0x7f1a4c0c7200 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1a4c0cc780 0x7f1a4c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 --2- 172.21.15.50:0/373074176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a4c0cc330 0x7f1a4c1699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 >> 172.21.15.50:0/373074176 conn(0x7f1a4c0cd650 msgr2=0x7f1a4c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 shutdown_connections 2023-12-19T15:54:45.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.284+0000 7f1a51865700 1 -- 172.21.15.50:0/373074176 wait complete. 2023-12-19T15:54:45.290 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:54:45.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:54:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.464+0000 7f1975e0c700 1 Processor -- start 2023-12-19T15:54:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.464+0000 7f1975e0c700 1 -- start start 2023-12-19T15:54:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f1975e0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f197006a580 0x7f197006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f1975e0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197006c7c0 con 0x7f197006d210 2023-12-19T15:54:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f1975e0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197006cd80 con 0x7f19700b1410 2023-12-19T15:54:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f1975e0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197006af40 con 0x7f197006a580 2023-12-19T15:54:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f197560b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f197006d210 0x7f197006d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40628/0 (socket says 172.21.15.50:40628) 2023-12-19T15:54:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f197560b700 1 -- 172.21.15.50:0/3115773507 learned_addr learned my addr 172.21.15.50:0/3115773507 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196ffff700 1 --2- 172.21.15.50:0/3115773507 >> v2:172.21.15.125:3300/0 conn(0x7f197006a580 0x7f197006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319341590 0 0) 0x7f197006c7c0 con 0x7f197006d210 2023-12-19T15:54:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1960003570 con 0x7f197006d210 2023-12-19T15:54:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661194119 0 0) 0x7f197006cd80 con 0x7f19700b1410 2023-12-19T15:54:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197006c7c0 con 0x7f19700b1410 2023-12-19T15:54:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3725624767 0 0) 0x7f1960003570 con 0x7f197006d210 2023-12-19T15:54:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 >> v2:172.21.15.125:3300/0 conn(0x7f197006a580 msgr2=0x7f197006aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 --2- 172.21.15.50:0/3115773507 >> v2:172.21.15.125:3300/0 conn(0x7f197006a580 0x7f197006aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 >> v1:172.21.15.125:6789/0 conn(0x7f19700b1410 legacy=0x7f19700ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197006b8f0 con 0x7f197006d210 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f19680017f0 con 0x7f197006d210 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.468+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 >> v1:172.21.15.50:6789/0 conn(0x7f197006d210 legacy=0x7f197006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1960003bf0 0x7f1960003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1960004810 0x7f1960004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 0x7f196000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197006af40 con 0x7f19600051c0 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196000c0b0 con 0x7f1960004810 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196000c270 con 0x7f1960003bf0 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 0x7f196000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196ffff700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1960004810 0x7f1960004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1974e0a700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1960003bf0 0x7f1960003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 -- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1960003bf0 msgr2=0x7f1960003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1960003bf0 0x7f1960003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 -- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1960004810 msgr2=0x7f1960004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1960004810 0x7f1960004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 -- 172.21.15.50:0/3115773507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960003570 con 0x7f19600051c0 2023-12-19T15:54:45.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1974e0a700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1960003bf0 0x7f1960003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 -- 172.21.15.50:0/3115773507 reap_dead start 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f197560b700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 0x7f196000ba60 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f196800a940 tx=0x7f197006ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbc7ecef960f2f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19680070d0 con 0x7f19600051c0 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1968007270 con 0x7f19600051c0 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f196f7fe700 1 -- 172.21.15.50:0/3115773507 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968007630 con 0x7f19600051c0 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 -- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 msgr2=0x7f196000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 0x7f196000ba60 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f196800a940 tx=0x7f197006ba80 comp rx=0 tx=0).stop 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 -- 172.21.15.50:0/3115773507 shutdown_connections 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 --2- 172.21.15.50:0/3115773507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19600051c0 0x7f196000ba60 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 -- 172.21.15.50:0/3115773507 >> 172.21.15.50:0/3115773507 conn(0x7f1970067bb0 msgr2=0x7f19700cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 -- 172.21.15.50:0/3115773507 shutdown_connections 2023-12-19T15:54:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.472+0000 7f1975e0c700 1 -- 172.21.15.50:0/3115773507 wait complete. 2023-12-19T15:54:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 Processor -- start 2023-12-19T15:54:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 -- start start 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19600051c0 0x7f1970159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 0x7f1970154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.476+0000 7f1975e0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197015a630 con 0x7f197015c9d0 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1975e0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19700aee80 con 0x7f19600051c0 2023-12-19T15:54:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1975e0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1970159870 con 0x7f1970157fe0 2023-12-19T15:54:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34268/0 (socket says 172.21.15.50:34268) 2023-12-19T15:54:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 -- 172.21.15.50:0/187755343 learned_addr learned my addr 172.21.15.50:0/187755343 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1974e0a700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19600051c0 0x7f1970159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f196ffff700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 0x7f1970154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 -- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 msgr2=0x7f1970154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 0x7f1970154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 -- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19600051c0 msgr2=0x7f1970159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19600051c0 0x7f1970159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 -- 172.21.15.50:0/187755343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700af9c0 con 0x7f197015c9d0 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f196ffff700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 0x7f1970154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f197560b700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f19700cc700 tx=0x7f1968013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968023070 con 0x7f197015c9d0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1968004bb0 con 0x7f197015c9d0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19700af090 con 0x7f197015c9d0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196801e430 con 0x7f197015c9d0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19700af650 con 0x7f197015c9d0 2023-12-19T15:54:45.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.480+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19700afb50 con 0x7f197015c9d0 2023-12-19T15:54:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.484+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f196801e860 con 0x7f197015c9d0 2023-12-19T15:54:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.484+0000 7f196cff9700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 0x7f1954062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.484+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f196807e560 con 0x7f197015c9d0 2023-12-19T15:54:45.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.484+0000 7f1974e0a700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 0x7f1954062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.488+0000 7f1974e0a700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 0x7f1954062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19640010e0 tx=0x7f196400c040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.492+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f196804e820 con 0x7f197015c9d0 2023-12-19T15:54:45.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.664+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f197004be40 con 0x7f197015c9d0 2023-12-19T15:54:45.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.664+0000 7f196cff9700 1 -- 172.21.15.50:0/187755343 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+15657 (secure 0 0 0) 0x7f1968051e40 con 0x7f197015c9d0 2023-12-19T15:54:45.668 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:45.668 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":167,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:42.800645+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 msgr2=0x7f1954062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 0x7f1954062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19640010e0 tx=0x7f196400c040 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 msgr2=0x7f197015ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f19700cc700 tx=0x7f1968013000 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 shutdown_connections 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f19540606a0 0x7f1954062b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f197015c9d0 0x7f197015ce40 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1970157fe0 0x7f1970154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 --2- 172.21.15.50:0/187755343 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19600051c0 0x7f1970159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 >> 172.21.15.50:0/187755343 conn(0x7f1970067bb0 msgr2=0x7f19701564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 shutdown_connections 2023-12-19T15:54:45.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.668+0000 7f1975e0c700 1 -- 172.21.15.50:0/187755343 wait complete. 2023-12-19T15:54:45.693 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T15:54:45.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T15:54:45.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 Processor -- start 2023-12-19T15:54:45.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 -- start start 2023-12-19T15:54:45.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7effc00c6900 0x7effc00c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7effc006b900 0x7effc006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc006b470 con 0x7effc015cc20 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc015d790 con 0x7effc00c6900 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc6e0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc0161d10 con 0x7effc006b900 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc5e0a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7effc00c6900 0x7effc00c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc660b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7effc015cc20 0x7effc015d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40638/0 (socket says 172.21.15.50:40638) 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc660b700 1 -- 172.21.15.50:0/3458200850 learned_addr learned my addr 172.21.15.50:0/3458200850 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.868+0000 7effc5609700 1 --2- 172.21.15.50:0/3458200850 >> v2:172.21.15.125:3300/0 conn(0x7effc006b900 0x7effc006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc5e0a700 1 -- 172.21.15.50:0/3458200850 >> v2:172.21.15.125:3300/0 conn(0x7effc006b900 msgr2=0x7effc006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc5e0a700 1 --2- 172.21.15.50:0/3458200850 >> v2:172.21.15.125:3300/0 conn(0x7effc006b900 0x7effc006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc5e0a700 1 -- 172.21.15.50:0/3458200850 >> v1:172.21.15.50:6789/0 conn(0x7effc015cc20 legacy=0x7effc015d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc5e0a700 1 -- 172.21.15.50:0/3458200850 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc006bd70 con 0x7effc00c6900 2023-12-19T15:54:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc5e0a700 1 --2- 172.21.15.50:0/3458200850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c6cd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7effc006a030 tx=0x7effbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b275def5e997a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc4e08700 1 -- 172.21.15.50:0/3458200850 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc011630 con 0x7effc00c6900 2023-12-19T15:54:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc4e08700 1 -- 172.21.15.50:0/3458200850 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc011790 con 0x7effc00c6900 2023-12-19T15:54:45.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc4e08700 1 -- 172.21.15.50:0/3458200850 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc011b30 con 0x7effc00c6900 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 -- 172.21.15.50:0/3458200850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 msgr2=0x7effc00c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 --2- 172.21.15.50:0/3458200850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c6cd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7effc006a030 tx=0x7effbc0091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 -- 172.21.15.50:0/3458200850 shutdown_connections 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 --2- 172.21.15.50:0/3458200850 >> v2:172.21.15.125:3300/0 conn(0x7effc006b900 0x7effc006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 --2- 172.21.15.50:0/3458200850 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c6cd0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 -- 172.21.15.50:0/3458200850 >> 172.21.15.50:0/3458200850 conn(0x7effc0067bb0 msgr2=0x7effc00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 -- 172.21.15.50:0/3458200850 shutdown_connections 2023-12-19T15:54:45.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.872+0000 7effc6e0c700 1 -- 172.21.15.50:0/3458200850 wait complete. 2023-12-19T15:54:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 Processor -- start 2023-12-19T15:54:45.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 -- start start 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7effc006b900 0x7effc01591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effc015cc20 0x7effc0160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc015fcd0 con 0x7effc00c6900 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc015edf0 con 0x7effc015cc20 2023-12-19T15:54:45.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc6e0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc0159720 con 0x7effc006b900 2023-12-19T15:54:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34286/0 (socket says 172.21.15.50:34286) 2023-12-19T15:54:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 -- 172.21.15.50:0/2555828786 learned_addr learned my addr 172.21.15.50:0/2555828786 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5e0a700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7effc006b900 0x7effc01591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc660b700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effc015cc20 0x7effc0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 -- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7effc006b900 msgr2=0x7effc01591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7effc006b900 0x7effc01591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 -- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effc015cc20 msgr2=0x7effc0160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effc015cc20 0x7effc0160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 -- 172.21.15.50:0/2555828786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc009420 con 0x7effc00c6900 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effc5609700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7effb000ec80 tx=0x7effb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.876+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000c850 con 0x7effc00c6900 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000c9b0 con 0x7effc00c6900 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc015f0c0 con 0x7effc00c6900 2023-12-19T15:54:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000cd50 con 0x7effc00c6900 2023-12-19T15:54:45.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc015f680 con 0x7effc00c6900 2023-12-19T15:54:45.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc00c6f80 con 0x7effc00c6900 2023-12-19T15:54:45.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.880+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7effb0013680 con 0x7effc00c6900 2023-12-19T15:54:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.884+0000 7effb5ffb700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 0x7effa4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.884+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7effb007cd80 con 0x7effc00c6900 2023-12-19T15:54:45.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.884+0000 7effc5e0a700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 0x7effa4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:45.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.884+0000 7effc5e0a700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 0x7effa4062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effbc0093f0 tx=0x7effbc017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:45.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:45.888+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7effc00c6f80 con 0x7effc00c6900 2023-12-19T15:54:46.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.056+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effc00c6f80 con 0x7effa40606a0 2023-12-19T15:54:46.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.076+0000 7effb5ffb700 1 -- 172.21.15.50:0/2555828786 <== mgr.7577 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287147 (secure 0 0 0) 0x7effc00c6f80 con 0x7effa40606a0 2023-12-19T15:54:46.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.080+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 msgr2=0x7effa4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.080+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 0x7effa4062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effbc0093f0 tx=0x7effbc017000 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.080+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 msgr2=0x7effc00c54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7effb000ec80 tx=0x7effb00091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 shutdown_connections 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effc015cc20 0x7effc0160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7effa40606a0 0x7effa4062b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effc00c6900 0x7effc00c54c0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 --2- 172.21.15.50:0/2555828786 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7effc006b900 0x7effc01591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 >> 172.21.15.50:0/2555828786 conn(0x7effc0067bb0 msgr2=0x7effc00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:46.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 shutdown_connections 2023-12-19T15:54:46.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.084+0000 7effc6e0c700 1 -- 172.21.15.50:0/2555828786 wait complete. 2023-12-19T15:54:46.087 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T15:54:46.107 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T15:54:44.923567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461216,"kb_used_data":17376,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752513504,"statfs":{"total":773094113280,"available":770573828096,"internally_reserved":0,"allocated":17793024,"data_stored":9585926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.159912"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814187+0000","last_change":"2023-12-19T15:54:41.814187+0000","last_active":"2023-12-19T15:54:41.814187+0000","last_peered":"2023-12-19T15:54:41.814187+0000","last_clean":"2023-12-19T15:54:41.814187+0000","last_became_active":"2023-12-19T15:54:41.813759+0000","last_became_peered":"2023-12-19T15:54:41.813759+0000","last_unstale":"2023-12-19T15:54:41.814187+0000","last_undegraded":"2023-12-19T15:54:41.814187+0000","last_fullsized":"2023-12-19T15:54:41.814187+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:41.691655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828060+0000","last_change":"2023-12-19T15:54:41.821043+0000","last_active":"2023-12-19T15:54:41.828060+0000","last_peered":"2023-12-19T15:54:41.828060+0000","last_clean":"2023-12-19T15:54:41.828060+0000","last_became_active":"2023-12-19T15:54:41.820644+0000","last_became_peered":"2023-12-19T15:54:41.820644+0000","last_unstale":"2023-12-19T15:54:41.828060+0000","last_undegraded":"2023-12-19T15:54:41.828060+0000","last_fullsized":"2023-12-19T15:54:41.828060+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:21.094171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812097+0000","last_change":"2023-12-19T15:54:41.812097+0000","last_active":"2023-12-19T15:54:41.812097+0000","last_peered":"2023-12-19T15:54:41.812097+0000","last_clean":"2023-12-19T15:54:41.812097+0000","last_became_active":"2023-12-19T15:54:41.811317+0000","last_became_peered":"2023-12-19T15:54:41.811317+0000","last_unstale":"2023-12-19T15:54:41.812097+0000","last_undegraded":"2023-12-19T15:54:41.812097+0000","last_fullsized":"2023-12-19T15:54:41.812097+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:37.202908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818918+0000","last_change":"2023-12-19T15:54:41.810547+0000","last_active":"2023-12-19T15:54:41.818918+0000","last_peered":"2023-12-19T15:54:41.818918+0000","last_clean":"2023-12-19T15:54:41.818918+0000","last_became_active":"2023-12-19T15:54:41.810236+0000","last_became_peered":"2023-12-19T15:54:41.810236+0000","last_unstale":"2023-12-19T15:54:41.818918+0000","last_undegraded":"2023-12-19T15:54:41.818918+0000","last_fullsized":"2023-12-19T15:54:41.818918+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:19.079075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.805538+0000","last_change":"2023-12-19T15:54:41.805538+0000","last_active":"2023-12-19T15:54:41.805538+0000","last_peered":"2023-12-19T15:54:41.805538+0000","last_clean":"2023-12-19T15:54:41.805538+0000","last_became_active":"2023-12-19T15:54:41.805262+0000","last_became_peered":"2023-12-19T15:54:41.805262+0000","last_unstale":"2023-12-19T15:54:41.805538+0000","last_undegraded":"2023-12-19T15:54:41.805538+0000","last_fullsized":"2023-12-19T15:54:41.805538+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:49.115763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.830091+0000","last_change":"2023-12-19T15:54:41.828246+0000","last_active":"2023-12-19T15:54:41.830091+0000","last_peered":"2023-12-19T15:54:41.830091+0000","last_clean":"2023-12-19T15:54:41.830091+0000","last_became_active":"2023-12-19T15:54:41.827107+0000","last_became_peered":"2023-12-19T15:54:41.827107+0000","last_unstale":"2023-12-19T15:54:41.830091+0000","last_undegraded":"2023-12-19T15:54:41.830091+0000","last_fullsized":"2023-12-19T15:54:41.830091+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:02.283783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.829020+0000","last_change":"2023-12-19T15:54:41.829020+0000","last_active":"2023-12-19T15:54:41.829020+0000","last_peered":"2023-12-19T15:54:41.829020+0000","last_clean":"2023-12-19T15:54:41.829020+0000","last_became_active":"2023-12-19T15:54:41.828574+0000","last_became_peered":"2023-12-19T15:54:41.828574+0000","last_unstale":"2023-12-19T15:54:41.829020+0000","last_undegraded":"2023-12-19T15:54:41.829020+0000","last_fullsized":"2023-12-19T15:54:41.829020+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:44.281335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.816471+0000","last_change":"2023-12-19T15:54:41.816471+0000","last_active":"2023-12-19T15:54:41.816471+0000","last_peered":"2023-12-19T15:54:41.816471+0000","last_clean":"2023-12-19T15:54:41.816471+0000","last_became_active":"2023-12-19T15:54:41.816202+0000","last_became_peered":"2023-12-19T15:54:41.816202+0000","last_unstale":"2023-12-19T15:54:41.816471+0000","last_undegraded":"2023-12-19T15:54:41.816471+0000","last_fullsized":"2023-12-19T15:54:41.816471+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:54:40.796038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828644+0000","last_change":"2023-12-19T15:54:41.826479+0000","last_active":"2023-12-19T15:54:41.828644+0000","last_peered":"2023-12-19T15:54:41.828644+0000","last_clean":"2023-12-19T15:54:41.828644+0000","last_became_active":"2023-12-19T15:54:41.826187+0000","last_became_peered":"2023-12-19T15:54:41.826187+0000","last_unstale":"2023-12-19T15:54:41.828644+0000","last_undegraded":"2023-12-19T15:54:41.828644+0000","last_fullsized":"2023-12-19T15:54:41.828644+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:32.959716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820814+0000","last_change":"2023-12-19T15:54:41.820814+0000","last_active":"2023-12-19T15:54:41.820814+0000","last_peered":"2023-12-19T15:54:41.820814+0000","last_clean":"2023-12-19T15:54:41.820814+0000","last_became_active":"2023-12-19T15:54:41.820465+0000","last_became_peered":"2023-12-19T15:54:41.820465+0000","last_unstale":"2023-12-19T15:54:41.820814+0000","last_undegraded":"2023-12-19T15:54:41.820814+0000","last_fullsized":"2023-12-19T15:54:41.820814+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:58.506059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828139+0000","last_change":"2023-12-19T15:54:41.821450+0000","last_active":"2023-12-19T15:54:41.828139+0000","last_peered":"2023-12-19T15:54:41.828139+0000","last_clean":"2023-12-19T15:54:41.828139+0000","last_became_active":"2023-12-19T15:54:41.820932+0000","last_became_peered":"2023-12-19T15:54:41.820932+0000","last_unstale":"2023-12-19T15:54:41.828139+0000","last_undegraded":"2023-12-19T15:54:41.828139+0000","last_fullsized":"2023-12-19T15:54:41.828139+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:45:47.022911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.800393+0000","last_change":"2023-12-19T15:54:41.800393+0000","last_active":"2023-12-19T15:54:41.800393+0000","last_peered":"2023-12-19T15:54:41.800393+0000","last_clean":"2023-12-19T15:54:41.800393+0000","last_became_active":"2023-12-19T15:54:41.799911+0000","last_became_peered":"2023-12-19T15:54:41.799911+0000","last_unstale":"2023-12-19T15:54:41.800393+0000","last_undegraded":"2023-12-19T15:54:41.800393+0000","last_fullsized":"2023-12-19T15:54:41.800393+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:04:34.113251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828537+0000","last_change":"2023-12-19T15:54:41.822347+0000","last_active":"2023-12-19T15:54:41.828537+0000","last_peered":"2023-12-19T15:54:41.828537+0000","last_clean":"2023-12-19T15:54:41.828537+0000","last_became_active":"2023-12-19T15:54:41.820183+0000","last_became_peered":"2023-12-19T15:54:41.820183+0000","last_unstale":"2023-12-19T15:54:41.828537+0000","last_undegraded":"2023-12-19T15:54:41.828537+0000","last_fullsized":"2023-12-19T15:54:41.828537+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:34.288102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826562+0000","last_change":"2023-12-19T15:54:41.826562+0000","last_active":"2023-12-19T15:54:41.826562+0000","last_peered":"2023-12-19T15:54:41.826562+0000","last_clean":"2023-12-19T15:54:41.826562+0000","last_became_active":"2023-12-19T15:54:41.826123+0000","last_became_peered":"2023-12-19T15:54:41.826123+0000","last_unstale":"2023-12-19T15:54:41.826562+0000","last_undegraded":"2023-12-19T15:54:41.826562+0000","last_fullsized":"2023-12-19T15:54:41.826562+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:47.924959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815990+0000","last_change":"2023-12-19T15:54:41.815990+0000","last_active":"2023-12-19T15:54:41.815990+0000","last_peered":"2023-12-19T15:54:41.815990+0000","last_clean":"2023-12-19T15:54:41.815990+0000","last_became_active":"2023-12-19T15:54:41.815702+0000","last_became_peered":"2023-12-19T15:54:41.815702+0000","last_unstale":"2023-12-19T15:54:41.815990+0000","last_undegraded":"2023-12-19T15:54:41.815990+0000","last_fullsized":"2023-12-19T15:54:41.815990+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:50:51.430483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827056+0000","last_change":"2023-12-19T15:54:41.816946+0000","last_active":"2023-12-19T15:54:41.827056+0000","last_peered":"2023-12-19T15:54:41.827056+0000","last_clean":"2023-12-19T15:54:41.827056+0000","last_became_active":"2023-12-19T15:54:41.816641+0000","last_became_peered":"2023-12-19T15:54:41.816641+0000","last_unstale":"2023-12-19T15:54:41.827056+0000","last_undegraded":"2023-12-19T15:54:41.827056+0000","last_fullsized":"2023-12-19T15:54:41.827056+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:29.211645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827061+0000","last_change":"2023-12-19T15:54:41.817145+0000","last_active":"2023-12-19T15:54:41.827061+0000","last_peered":"2023-12-19T15:54:41.827061+0000","last_clean":"2023-12-19T15:54:41.827061+0000","last_became_active":"2023-12-19T15:54:41.816767+0000","last_became_peered":"2023-12-19T15:54:41.816767+0000","last_unstale":"2023-12-19T15:54:41.827061+0000","last_undegraded":"2023-12-19T15:54:41.827061+0000","last_fullsized":"2023-12-19T15:54:41.827061+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:20.000343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.811453+0000","last_change":"2023-12-19T15:54:41.811453+0000","last_active":"2023-12-19T15:54:41.811453+0000","last_peered":"2023-12-19T15:54:41.811453+0000","last_clean":"2023-12-19T15:54:41.811453+0000","last_became_active":"2023-12-19T15:54:41.811126+0000","last_became_peered":"2023-12-19T15:54:41.811126+0000","last_unstale":"2023-12-19T15:54:41.811453+0000","last_undegraded":"2023-12-19T15:54:41.811453+0000","last_fullsized":"2023-12-19T15:54:41.811453+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:37.202908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.823610+0000","last_change":"2023-12-19T15:54:41.823610+0000","last_active":"2023-12-19T15:54:41.823610+0000","last_peered":"2023-12-19T15:54:41.823610+0000","last_clean":"2023-12-19T15:54:41.823610+0000","last_became_active":"2023-12-19T15:54:41.823193+0000","last_became_peered":"2023-12-19T15:54:41.823193+0000","last_unstale":"2023-12-19T15:54:41.823610+0000","last_undegraded":"2023-12-19T15:54:41.823610+0000","last_fullsized":"2023-12-19T15:54:41.823610+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:08.996205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827756+0000","last_change":"2023-12-19T15:54:41.827756+0000","last_active":"2023-12-19T15:54:41.827756+0000","last_peered":"2023-12-19T15:54:41.827756+0000","last_clean":"2023-12-19T15:54:41.827756+0000","last_became_active":"2023-12-19T15:54:41.824918+0000","last_became_peered":"2023-12-19T15:54:41.824918+0000","last_unstale":"2023-12-19T15:54:41.827756+0000","last_undegraded":"2023-12-19T15:54:41.827756+0000","last_fullsized":"2023-12-19T15:54:41.827756+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:03.353814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.819874+0000","last_change":"2023-12-19T15:54:41.813919+0000","last_active":"2023-12-19T15:54:41.819874+0000","last_peered":"2023-12-19T15:54:41.819874+0000","last_clean":"2023-12-19T15:54:41.819874+0000","last_became_active":"2023-12-19T15:54:41.813616+0000","last_became_peered":"2023-12-19T15:54:41.813616+0000","last_unstale":"2023-12-19T15:54:41.819874+0000","last_undegraded":"2023-12-19T15:54:41.819874+0000","last_fullsized":"2023-12-19T15:54:41.819874+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:27:09.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.813715+0000","last_change":"2023-12-19T15:54:41.813715+0000","last_active":"2023-12-19T15:54:41.813715+0000","last_peered":"2023-12-19T15:54:41.813715+0000","last_clean":"2023-12-19T15:54:41.813715+0000","last_became_active":"2023-12-19T15:54:41.810954+0000","last_became_peered":"2023-12-19T15:54:41.810954+0000","last_unstale":"2023-12-19T15:54:41.813715+0000","last_undegraded":"2023-12-19T15:54:41.813715+0000","last_fullsized":"2023-12-19T15:54:41.813715+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:13:57.376090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821174+0000","last_change":"2023-12-19T15:54:41.811358+0000","last_active":"2023-12-19T15:54:41.821174+0000","last_peered":"2023-12-19T15:54:41.821174+0000","last_clean":"2023-12-19T15:54:41.821174+0000","last_became_active":"2023-12-19T15:54:41.811092+0000","last_became_peered":"2023-12-19T15:54:41.811092+0000","last_unstale":"2023-12-19T15:54:41.821174+0000","last_undegraded":"2023-12-19T15:54:41.821174+0000","last_fullsized":"2023-12-19T15:54:41.821174+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:49:42.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826958+0000","last_change":"2023-12-19T15:54:41.815072+0000","last_active":"2023-12-19T15:54:41.826958+0000","last_peered":"2023-12-19T15:54:41.826958+0000","last_clean":"2023-12-19T15:54:41.826958+0000","last_became_active":"2023-12-19T15:54:41.814781+0000","last_became_peered":"2023-12-19T15:54:41.814781+0000","last_unstale":"2023-12-19T15:54:41.826958+0000","last_undegraded":"2023-12-19T15:54:41.826958+0000","last_fullsized":"2023-12-19T15:54:41.826958+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:07:25.795613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.804497+0000","last_change":"2023-12-19T15:54:41.804497+0000","last_active":"2023-12-19T15:54:41.804497+0000","last_peered":"2023-12-19T15:54:41.804497+0000","last_clean":"2023-12-19T15:54:41.804497+0000","last_became_active":"2023-12-19T15:54:41.804206+0000","last_became_peered":"2023-12-19T15:54:41.804206+0000","last_unstale":"2023-12-19T15:54:41.804497+0000","last_undegraded":"2023-12-19T15:54:41.804497+0000","last_fullsized":"2023-12-19T15:54:41.804497+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:40.591338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818166+0000","last_change":"2023-12-19T15:54:41.818166+0000","last_active":"2023-12-19T15:54:41.818166+0000","last_peered":"2023-12-19T15:54:41.818166+0000","last_clean":"2023-12-19T15:54:41.818166+0000","last_became_active":"2023-12-19T15:54:41.817837+0000","last_became_peered":"2023-12-19T15:54:41.817837+0000","last_unstale":"2023-12-19T15:54:41.818166+0000","last_undegraded":"2023-12-19T15:54:41.818166+0000","last_fullsized":"2023-12-19T15:54:41.818166+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:04:41.835159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827184+0000","last_change":"2023-12-19T15:54:41.827184+0000","last_active":"2023-12-19T15:54:41.827184+0000","last_peered":"2023-12-19T15:54:41.827184+0000","last_clean":"2023-12-19T15:54:41.827184+0000","last_became_active":"2023-12-19T15:54:41.826865+0000","last_became_peered":"2023-12-19T15:54:41.826865+0000","last_unstale":"2023-12-19T15:54:41.827184+0000","last_undegraded":"2023-12-19T15:54:41.827184+0000","last_fullsized":"2023-12-19T15:54:41.827184+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:30:40.870332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820198+0000","last_change":"2023-12-19T15:54:41.820198+0000","last_active":"2023-12-19T15:54:41.820198+0000","last_peered":"2023-12-19T15:54:41.820198+0000","last_clean":"2023-12-19T15:54:41.820198+0000","last_became_active":"2023-12-19T15:54:41.819870+0000","last_became_peered":"2023-12-19T15:54:41.819870+0000","last_unstale":"2023-12-19T15:54:41.820198+0000","last_undegraded":"2023-12-19T15:54:41.820198+0000","last_fullsized":"2023-12-19T15:54:41.820198+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:17:10.903420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814790+0000","last_change":"2023-12-19T15:54:41.814790+0000","last_active":"2023-12-19T15:54:41.814790+0000","last_peered":"2023-12-19T15:54:41.814790+0000","last_clean":"2023-12-19T15:54:41.814790+0000","last_became_active":"2023-12-19T15:54:41.814482+0000","last_became_peered":"2023-12-19T15:54:41.814482+0000","last_unstale":"2023-12-19T15:54:41.814790+0000","last_undegraded":"2023-12-19T15:54:41.814790+0000","last_fullsized":"2023-12-19T15:54:41.814790+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:58:36.176831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812370+0000","last_change":"2023-12-19T15:54:41.812370+0000","last_active":"2023-12-19T15:54:41.812370+0000","last_peered":"2023-12-19T15:54:41.812370+0000","last_clean":"2023-12-19T15:54:41.812370+0000","last_became_active":"2023-12-19T15:54:41.809818+0000","last_became_peered":"2023-12-19T15:54:41.809818+0000","last_unstale":"2023-12-19T15:54:41.812370+0000","last_undegraded":"2023-12-19T15:54:41.812370+0000","last_fullsized":"2023-12-19T15:54:41.812370+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:06.694443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814923+0000","last_change":"2023-12-19T15:54:41.814923+0000","last_active":"2023-12-19T15:54:41.814923+0000","last_peered":"2023-12-19T15:54:41.814923+0000","last_clean":"2023-12-19T15:54:41.814923+0000","last_became_active":"2023-12-19T15:54:41.814627+0000","last_became_peered":"2023-12-19T15:54:41.814627+0000","last_unstale":"2023-12-19T15:54:41.814923+0000","last_undegraded":"2023-12-19T15:54:41.814923+0000","last_fullsized":"2023-12-19T15:54:41.814923+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:47.615053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815877+0000","last_change":"2023-12-19T15:54:41.815877+0000","last_active":"2023-12-19T15:54:41.815877+0000","last_peered":"2023-12-19T15:54:41.815877+0000","last_clean":"2023-12-19T15:54:41.815877+0000","last_became_active":"2023-12-19T15:54:41.815580+0000","last_became_peered":"2023-12-19T15:54:41.815580+0000","last_unstale":"2023-12-19T15:54:41.815877+0000","last_undegraded":"2023-12-19T15:54:41.815877+0000","last_fullsized":"2023-12-19T15:54:41.815877+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:40.286462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812546+0000","last_change":"2023-12-19T15:54:41.812546+0000","last_active":"2023-12-19T15:54:41.812546+0000","last_peered":"2023-12-19T15:54:41.812546+0000","last_clean":"2023-12-19T15:54:41.812546+0000","last_became_active":"2023-12-19T15:54:41.812270+0000","last_became_peered":"2023-12-19T15:54:41.812270+0000","last_unstale":"2023-12-19T15:54:41.812546+0000","last_undegraded":"2023-12-19T15:54:41.812546+0000","last_fullsized":"2023-12-19T15:54:41.812546+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:12:20.404504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"166'4","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.839672+0000","last_change":"2023-12-19T15:54:40.810070+0000","last_active":"2023-12-19T15:54:41.839672+0000","last_peered":"2023-12-19T15:54:41.839672+0000","last_clean":"2023-12-19T15:54:41.839672+0000","last_became_active":"2023-12-19T15:54:40.809750+0000","last_became_peered":"2023-12-19T15:54:40.809750+0000","last_unstale":"2023-12-19T15:54:41.839672+0000","last_undegraded":"2023-12-19T15:54:41.839672+0000","last_fullsized":"2023-12-19T15:54:41.839672+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:17:11.623361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.817260+0000","last_change":"2023-12-19T15:54:41.817260+0000","last_active":"2023-12-19T15:54:41.817260+0000","last_peered":"2023-12-19T15:54:41.817260+0000","last_clean":"2023-12-19T15:54:41.817260+0000","last_became_active":"2023-12-19T15:54:41.814157+0000","last_became_peered":"2023-12-19T15:54:41.814157+0000","last_unstale":"2023-12-19T15:54:41.817260+0000","last_undegraded":"2023-12-19T15:54:41.817260+0000","last_fullsized":"2023-12-19T15:54:41.817260+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:20.089134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.798132+0000","last_change":"2023-12-19T15:54:40.807230+0000","last_active":"2023-12-19T15:54:41.798132+0000","last_peered":"2023-12-19T15:54:41.798132+0000","last_clean":"2023-12-19T15:54:41.798132+0000","last_became_active":"2023-12-19T15:54:40.806952+0000","last_became_peered":"2023-12-19T15:54:40.806952+0000","last_unstale":"2023-12-19T15:54:41.798132+0000","last_undegraded":"2023-12-19T15:54:41.798132+0000","last_fullsized":"2023-12-19T15:54:41.798132+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:17:11.623361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827656+0000","last_change":"2023-12-19T15:54:41.820811+0000","last_active":"2023-12-19T15:54:41.827656+0000","last_peered":"2023-12-19T15:54:41.827656+0000","last_clean":"2023-12-19T15:54:41.827656+0000","last_became_active":"2023-12-19T15:54:41.820109+0000","last_became_peered":"2023-12-19T15:54:41.820109+0000","last_unstale":"2023-12-19T15:54:41.827656+0000","last_undegraded":"2023-12-19T15:54:41.827656+0000","last_fullsized":"2023-12-19T15:54:41.827656+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:58.890637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837246+0000","last_change":"2023-12-19T15:54:40.814530+0000","last_active":"2023-12-19T15:54:41.837246+0000","last_peered":"2023-12-19T15:54:41.837246+0000","last_clean":"2023-12-19T15:54:41.837246+0000","last_became_active":"2023-12-19T15:54:40.814194+0000","last_became_peered":"2023-12-19T15:54:40.814194+0000","last_unstale":"2023-12-19T15:54:41.837246+0000","last_undegraded":"2023-12-19T15:54:41.837246+0000","last_fullsized":"2023-12-19T15:54:41.837246+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:33.271613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821700+0000","last_change":"2023-12-19T15:54:41.821700+0000","last_active":"2023-12-19T15:54:41.821700+0000","last_peered":"2023-12-19T15:54:41.821700+0000","last_clean":"2023-12-19T15:54:41.821700+0000","last_became_active":"2023-12-19T15:54:41.820923+0000","last_became_peered":"2023-12-19T15:54:41.820923+0000","last_unstale":"2023-12-19T15:54:41.821700+0000","last_undegraded":"2023-12-19T15:54:41.821700+0000","last_fullsized":"2023-12-19T15:54:41.821700+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:07.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837078+0000","last_change":"2023-12-19T15:54:40.810070+0000","last_active":"2023-12-19T15:54:41.837078+0000","last_peered":"2023-12-19T15:54:41.837078+0000","last_clean":"2023-12-19T15:54:41.837078+0000","last_became_active":"2023-12-19T15:54:40.809127+0000","last_became_peered":"2023-12-19T15:54:40.809127+0000","last_unstale":"2023-12-19T15:54:41.837078+0000","last_undegraded":"2023-12-19T15:54:41.837078+0000","last_fullsized":"2023-12-19T15:54:41.837078+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:01.267294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814317+0000","last_change":"2023-12-19T15:54:41.814317+0000","last_active":"2023-12-19T15:54:41.814317+0000","last_peered":"2023-12-19T15:54:41.814317+0000","last_clean":"2023-12-19T15:54:41.814317+0000","last_became_active":"2023-12-19T15:54:41.813929+0000","last_became_peered":"2023-12-19T15:54:41.813929+0000","last_unstale":"2023-12-19T15:54:41.814317+0000","last_undegraded":"2023-12-19T15:54:41.814317+0000","last_fullsized":"2023-12-19T15:54:41.814317+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:08.996205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.801995+0000","last_change":"2023-12-19T15:54:40.807968+0000","last_active":"2023-12-19T15:54:41.801995+0000","last_peered":"2023-12-19T15:54:41.801995+0000","last_clean":"2023-12-19T15:54:41.801995+0000","last_became_active":"2023-12-19T15:54:40.807640+0000","last_became_peered":"2023-12-19T15:54:40.807640+0000","last_unstale":"2023-12-19T15:54:41.801995+0000","last_undegraded":"2023-12-19T15:54:41.801995+0000","last_fullsized":"2023-12-19T15:54:41.801995+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:39.574849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814670+0000","last_change":"2023-12-19T15:54:41.814670+0000","last_active":"2023-12-19T15:54:41.814670+0000","last_peered":"2023-12-19T15:54:41.814670+0000","last_clean":"2023-12-19T15:54:41.814670+0000","last_became_active":"2023-12-19T15:54:41.814393+0000","last_became_peered":"2023-12-19T15:54:41.814393+0000","last_unstale":"2023-12-19T15:54:41.814670+0000","last_undegraded":"2023-12-19T15:54:41.814670+0000","last_fullsized":"2023-12-19T15:54:41.814670+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:44.281335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.793857+0000","last_change":"2023-12-19T15:54:40.803872+0000","last_active":"2023-12-19T15:54:41.793857+0000","last_peered":"2023-12-19T15:54:41.793857+0000","last_clean":"2023-12-19T15:54:41.793857+0000","last_became_active":"2023-12-19T15:54:40.803596+0000","last_became_peered":"2023-12-19T15:54:40.803596+0000","last_unstale":"2023-12-19T15:54:41.793857+0000","last_undegraded":"2023-12-19T15:54:41.793857+0000","last_fullsized":"2023-12-19T15:54:41.793857+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:41.314184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.822649+0000","last_change":"2023-12-19T15:54:41.822649+0000","last_active":"2023-12-19T15:54:41.822649+0000","last_peered":"2023-12-19T15:54:41.822649+0000","last_clean":"2023-12-19T15:54:41.822649+0000","last_became_active":"2023-12-19T15:54:41.822289+0000","last_became_peered":"2023-12-19T15:54:41.822289+0000","last_unstale":"2023-12-19T15:54:41.822649+0000","last_undegraded":"2023-12-19T15:54:41.822649+0000","last_fullsized":"2023-12-19T15:54:41.822649+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:10.296999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.808572+0000","last_change":"2023-12-19T15:54:40.815712+0000","last_active":"2023-12-19T15:54:41.808572+0000","last_peered":"2023-12-19T15:54:41.808572+0000","last_clean":"2023-12-19T15:54:41.808572+0000","last_became_active":"2023-12-19T15:54:40.814436+0000","last_became_peered":"2023-12-19T15:54:40.814436+0000","last_unstale":"2023-12-19T15:54:41.808572+0000","last_undegraded":"2023-12-19T15:54:41.808572+0000","last_fullsized":"2023-12-19T15:54:41.808572+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:06:09.231602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827452+0000","last_change":"2023-12-19T15:54:41.827452+0000","last_active":"2023-12-19T15:54:41.827452+0000","last_peered":"2023-12-19T15:54:41.827452+0000","last_clean":"2023-12-19T15:54:41.827452+0000","last_became_active":"2023-12-19T15:54:41.826712+0000","last_became_peered":"2023-12-19T15:54:41.826712+0000","last_unstale":"2023-12-19T15:54:41.827452+0000","last_undegraded":"2023-12-19T15:54:41.827452+0000","last_fullsized":"2023-12-19T15:54:41.827452+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:47.615053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.815241+0000","last_change":"2023-12-19T15:54:40.813957+0000","last_active":"2023-12-19T15:54:40.815241+0000","last_peered":"2023-12-19T15:54:40.815241+0000","last_clean":"2023-12-19T15:54:40.815241+0000","last_became_active":"2023-12-19T15:54:40.813441+0000","last_became_peered":"2023-12-19T15:54:40.813441+0000","last_unstale":"2023-12-19T15:54:40.815241+0000","last_undegraded":"2023-12-19T15:54:40.815241+0000","last_fullsized":"2023-12-19T15:54:40.815241+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:19.682900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821342+0000","last_change":"2023-12-19T15:54:41.821342+0000","last_active":"2023-12-19T15:54:41.821342+0000","last_peered":"2023-12-19T15:54:41.821342+0000","last_clean":"2023-12-19T15:54:41.821342+0000","last_became_active":"2023-12-19T15:54:41.820886+0000","last_became_peered":"2023-12-19T15:54:41.820886+0000","last_unstale":"2023-12-19T15:54:41.821342+0000","last_undegraded":"2023-12-19T15:54:41.821342+0000","last_fullsized":"2023-12-19T15:54:41.821342+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:13.880395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.801861+0000","last_change":"2023-12-19T15:54:40.812637+0000","last_active":"2023-12-19T15:54:41.801861+0000","last_peered":"2023-12-19T15:54:41.801861+0000","last_clean":"2023-12-19T15:54:41.801861+0000","last_became_active":"2023-12-19T15:54:40.812204+0000","last_became_peered":"2023-12-19T15:54:40.812204+0000","last_unstale":"2023-12-19T15:54:41.801861+0000","last_undegraded":"2023-12-19T15:54:41.801861+0000","last_fullsized":"2023-12-19T15:54:41.801861+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:19.515507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828573+0000","last_change":"2023-12-19T15:54:41.823134+0000","last_active":"2023-12-19T15:54:41.828573+0000","last_peered":"2023-12-19T15:54:41.828573+0000","last_clean":"2023-12-19T15:54:41.828573+0000","last_became_active":"2023-12-19T15:54:41.821787+0000","last_became_peered":"2023-12-19T15:54:41.821787+0000","last_unstale":"2023-12-19T15:54:41.828573+0000","last_undegraded":"2023-12-19T15:54:41.828573+0000","last_fullsized":"2023-12-19T15:54:41.828573+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:28.034708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.815560+0000","last_change":"2023-12-19T15:54:40.812249+0000","last_active":"2023-12-19T15:54:40.815560+0000","last_peered":"2023-12-19T15:54:40.815560+0000","last_clean":"2023-12-19T15:54:40.815560+0000","last_became_active":"2023-12-19T15:54:40.811732+0000","last_became_peered":"2023-12-19T15:54:40.811732+0000","last_unstale":"2023-12-19T15:54:40.815560+0000","last_undegraded":"2023-12-19T15:54:40.815560+0000","last_fullsized":"2023-12-19T15:54:40.815560+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:37:05.650864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821713+0000","last_change":"2023-12-19T15:54:41.821713+0000","last_active":"2023-12-19T15:54:41.821713+0000","last_peered":"2023-12-19T15:54:41.821713+0000","last_clean":"2023-12-19T15:54:41.821713+0000","last_became_active":"2023-12-19T15:54:41.813494+0000","last_became_peered":"2023-12-19T15:54:41.813494+0000","last_unstale":"2023-12-19T15:54:41.821713+0000","last_undegraded":"2023-12-19T15:54:41.821713+0000","last_fullsized":"2023-12-19T15:54:41.821713+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:10.296999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"166'3","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838197+0000","last_change":"2023-12-19T15:54:40.814994+0000","last_active":"2023-12-19T15:54:41.838197+0000","last_peered":"2023-12-19T15:54:41.838197+0000","last_clean":"2023-12-19T15:54:41.838197+0000","last_became_active":"2023-12-19T15:54:40.814677+0000","last_became_peered":"2023-12-19T15:54:40.814677+0000","last_unstale":"2023-12-19T15:54:41.838197+0000","last_undegraded":"2023-12-19T15:54:41.838197+0000","last_fullsized":"2023-12-19T15:54:41.838197+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:31.943227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814078+0000","last_change":"2023-12-19T15:54:41.814078+0000","last_active":"2023-12-19T15:54:41.814078+0000","last_peered":"2023-12-19T15:54:41.814078+0000","last_clean":"2023-12-19T15:54:41.814078+0000","last_became_active":"2023-12-19T15:54:41.813778+0000","last_became_peered":"2023-12-19T15:54:41.813778+0000","last_unstale":"2023-12-19T15:54:41.814078+0000","last_undegraded":"2023-12-19T15:54:41.814078+0000","last_fullsized":"2023-12-19T15:54:41.814078+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:20.577265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"166'6","reported_seq":15,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.839450+0000","last_change":"2023-12-19T15:54:40.811657+0000","last_active":"2023-12-19T15:54:41.839450+0000","last_peered":"2023-12-19T15:54:41.839450+0000","last_clean":"2023-12-19T15:54:41.839450+0000","last_became_active":"2023-12-19T15:54:40.811266+0000","last_became_peered":"2023-12-19T15:54:40.811266+0000","last_unstale":"2023-12-19T15:54:41.839450+0000","last_undegraded":"2023-12-19T15:54:41.839450+0000","last_fullsized":"2023-12-19T15:54:41.839450+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:24.200897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812410+0000","last_change":"2023-12-19T15:54:41.812410+0000","last_active":"2023-12-19T15:54:41.812410+0000","last_peered":"2023-12-19T15:54:41.812410+0000","last_clean":"2023-12-19T15:54:41.812410+0000","last_became_active":"2023-12-19T15:54:41.812117+0000","last_became_peered":"2023-12-19T15:54:41.812117+0000","last_unstale":"2023-12-19T15:54:41.812410+0000","last_undegraded":"2023-12-19T15:54:41.812410+0000","last_fullsized":"2023-12-19T15:54:41.812410+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:21.094171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837224+0000","last_change":"2023-12-19T15:54:40.807999+0000","last_active":"2023-12-19T15:54:41.837224+0000","last_peered":"2023-12-19T15:54:41.837224+0000","last_clean":"2023-12-19T15:54:41.837224+0000","last_became_active":"2023-12-19T15:54:40.807720+0000","last_became_peered":"2023-12-19T15:54:40.807720+0000","last_unstale":"2023-12-19T15:54:41.837224+0000","last_undegraded":"2023-12-19T15:54:41.837224+0000","last_fullsized":"2023-12-19T15:54:41.837224+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:31:32.428001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826632+0000","last_change":"2023-12-19T15:54:41.826632+0000","last_active":"2023-12-19T15:54:41.826632+0000","last_peered":"2023-12-19T15:54:41.826632+0000","last_clean":"2023-12-19T15:54:41.826632+0000","last_became_active":"2023-12-19T15:54:41.826031+0000","last_became_peered":"2023-12-19T15:54:41.826031+0000","last_unstale":"2023-12-19T15:54:41.826632+0000","last_undegraded":"2023-12-19T15:54:41.826632+0000","last_fullsized":"2023-12-19T15:54:41.826632+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:53:27.548739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.805212+0000","last_change":"2023-12-19T15:54:40.803176+0000","last_active":"2023-12-19T15:54:40.805212+0000","last_peered":"2023-12-19T15:54:40.805212+0000","last_clean":"2023-12-19T15:54:40.805212+0000","last_became_active":"2023-12-19T15:54:40.802765+0000","last_became_peered":"2023-12-19T15:54:40.802765+0000","last_unstale":"2023-12-19T15:54:40.805212+0000","last_undegraded":"2023-12-19T15:54:40.805212+0000","last_fullsized":"2023-12-19T15:54:40.805212+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:46.568208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.817735+0000","last_change":"2023-12-19T15:54:41.817735+0000","last_active":"2023-12-19T15:54:41.817735+0000","last_peered":"2023-12-19T15:54:41.817735+0000","last_clean":"2023-12-19T15:54:41.817735+0000","last_became_active":"2023-12-19T15:54:41.817421+0000","last_became_peered":"2023-12-19T15:54:41.817421+0000","last_unstale":"2023-12-19T15:54:41.817735+0000","last_undegraded":"2023-12-19T15:54:41.817735+0000","last_fullsized":"2023-12-19T15:54:41.817735+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:40.591338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"166'8","reported_seq":18,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.842677+0000","last_change":"2023-12-19T15:54:40.811396+0000","last_active":"2023-12-19T15:54:41.842677+0000","last_peered":"2023-12-19T15:54:41.842677+0000","last_clean":"2023-12-19T15:54:41.842677+0000","last_became_active":"2023-12-19T15:54:40.811037+0000","last_became_peered":"2023-12-19T15:54:40.811037+0000","last_unstale":"2023-12-19T15:54:41.842677+0000","last_undegraded":"2023-12-19T15:54:41.842677+0000","last_fullsized":"2023-12-19T15:54:41.842677+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:31.224808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812678+0000","last_change":"2023-12-19T15:54:41.812678+0000","last_active":"2023-12-19T15:54:41.812678+0000","last_peered":"2023-12-19T15:54:41.812678+0000","last_clean":"2023-12-19T15:54:41.812678+0000","last_became_active":"2023-12-19T15:54:41.812389+0000","last_became_peered":"2023-12-19T15:54:41.812389+0000","last_unstale":"2023-12-19T15:54:41.812678+0000","last_undegraded":"2023-12-19T15:54:41.812678+0000","last_fullsized":"2023-12-19T15:54:41.812678+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:03.353814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.835492+0000","last_change":"2023-12-19T15:54:40.810416+0000","last_active":"2023-12-19T15:54:41.835492+0000","last_peered":"2023-12-19T15:54:41.835492+0000","last_clean":"2023-12-19T15:54:41.835492+0000","last_became_active":"2023-12-19T15:54:40.810133+0000","last_became_peered":"2023-12-19T15:54:40.810133+0000","last_unstale":"2023-12-19T15:54:41.835492+0000","last_undegraded":"2023-12-19T15:54:41.835492+0000","last_fullsized":"2023-12-19T15:54:41.835492+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:38:18.568218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826128+0000","last_change":"2023-12-19T15:54:41.826128+0000","last_active":"2023-12-19T15:54:41.826128+0000","last_peered":"2023-12-19T15:54:41.826128+0000","last_clean":"2023-12-19T15:54:41.826128+0000","last_became_active":"2023-12-19T15:54:41.825759+0000","last_became_peered":"2023-12-19T15:54:41.825759+0000","last_unstale":"2023-12-19T15:54:41.826128+0000","last_undegraded":"2023-12-19T15:54:41.826128+0000","last_fullsized":"2023-12-19T15:54:41.826128+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:23:45.191382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838195+0000","last_change":"2023-12-19T15:54:40.807310+0000","last_active":"2023-12-19T15:54:41.838195+0000","last_peered":"2023-12-19T15:54:41.838195+0000","last_clean":"2023-12-19T15:54:41.838195+0000","last_became_active":"2023-12-19T15:54:40.806900+0000","last_became_peered":"2023-12-19T15:54:40.806900+0000","last_unstale":"2023-12-19T15:54:41.838195+0000","last_undegraded":"2023-12-19T15:54:41.838195+0000","last_fullsized":"2023-12-19T15:54:41.838195+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:41.314184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827101+0000","last_change":"2023-12-19T15:54:41.827101+0000","last_active":"2023-12-19T15:54:41.827101+0000","last_peered":"2023-12-19T15:54:41.827101+0000","last_clean":"2023-12-19T15:54:41.827101+0000","last_became_active":"2023-12-19T15:54:41.826809+0000","last_became_peered":"2023-12-19T15:54:41.826809+0000","last_unstale":"2023-12-19T15:54:41.827101+0000","last_undegraded":"2023-12-19T15:54:41.827101+0000","last_fullsized":"2023-12-19T15:54:41.827101+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:21.969475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838842+0000","last_change":"2023-12-19T15:54:40.815039+0000","last_active":"2023-12-19T15:54:41.838842+0000","last_peered":"2023-12-19T15:54:41.838842+0000","last_clean":"2023-12-19T15:54:41.838842+0000","last_became_active":"2023-12-19T15:54:40.814786+0000","last_became_peered":"2023-12-19T15:54:40.814786+0000","last_unstale":"2023-12-19T15:54:41.838842+0000","last_undegraded":"2023-12-19T15:54:41.838842+0000","last_fullsized":"2023-12-19T15:54:41.838842+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:42.555899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818244+0000","last_change":"2023-12-19T15:54:41.818244+0000","last_active":"2023-12-19T15:54:41.818244+0000","last_peered":"2023-12-19T15:54:41.818244+0000","last_clean":"2023-12-19T15:54:41.818244+0000","last_became_active":"2023-12-19T15:54:41.817105+0000","last_became_peered":"2023-12-19T15:54:41.817105+0000","last_unstale":"2023-12-19T15:54:41.818244+0000","last_undegraded":"2023-12-19T15:54:41.818244+0000","last_fullsized":"2023-12-19T15:54:41.818244+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:19.253460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.797257+0000","last_change":"2023-12-19T15:54:40.796341+0000","last_active":"2023-12-19T15:54:40.797257+0000","last_peered":"2023-12-19T15:54:40.797257+0000","last_clean":"2023-12-19T15:54:40.797257+0000","last_became_active":"2023-12-19T15:54:40.796024+0000","last_became_peered":"2023-12-19T15:54:40.796024+0000","last_unstale":"2023-12-19T15:54:40.797257+0000","last_undegraded":"2023-12-19T15:54:40.797257+0000","last_fullsized":"2023-12-19T15:54:40.797257+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:19.682900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.804896+0000","last_change":"2023-12-19T15:54:41.804896+0000","last_active":"2023-12-19T15:54:41.804896+0000","last_peered":"2023-12-19T15:54:41.804896+0000","last_clean":"2023-12-19T15:54:41.804896+0000","last_became_active":"2023-12-19T15:54:41.804604+0000","last_became_peered":"2023-12-19T15:54:41.804604+0000","last_unstale":"2023-12-19T15:54:41.804896+0000","last_undegraded":"2023-12-19T15:54:41.804896+0000","last_fullsized":"2023-12-19T15:54:41.804896+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:29.211645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837710+0000","last_change":"2023-12-19T15:54:40.806261+0000","last_active":"2023-12-19T15:54:41.837710+0000","last_peered":"2023-12-19T15:54:41.837710+0000","last_clean":"2023-12-19T15:54:41.837710+0000","last_became_active":"2023-12-19T15:54:40.805959+0000","last_became_peered":"2023-12-19T15:54:40.805959+0000","last_unstale":"2023-12-19T15:54:41.837710+0000","last_undegraded":"2023-12-19T15:54:41.837710+0000","last_fullsized":"2023-12-19T15:54:41.837710+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:48.099274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821581+0000","last_change":"2023-12-19T15:54:41.821581+0000","last_active":"2023-12-19T15:54:41.821581+0000","last_peered":"2023-12-19T15:54:41.821581+0000","last_clean":"2023-12-19T15:54:41.821581+0000","last_became_active":"2023-12-19T15:54:41.821253+0000","last_became_peered":"2023-12-19T15:54:41.821253+0000","last_unstale":"2023-12-19T15:54:41.821581+0000","last_undegraded":"2023-12-19T15:54:41.821581+0000","last_fullsized":"2023-12-19T15:54:41.821581+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:51.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.811170+0000","last_change":"2023-12-19T15:54:40.805684+0000","last_active":"2023-12-19T15:54:40.811170+0000","last_peered":"2023-12-19T15:54:40.811170+0000","last_clean":"2023-12-19T15:54:40.811170+0000","last_became_active":"2023-12-19T15:54:40.805403+0000","last_became_peered":"2023-12-19T15:54:40.805403+0000","last_unstale":"2023-12-19T15:54:40.811170+0000","last_undegraded":"2023-12-19T15:54:40.811170+0000","last_fullsized":"2023-12-19T15:54:40.811170+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:02.747028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815787+0000","last_change":"2023-12-19T15:54:41.815787+0000","last_active":"2023-12-19T15:54:41.815787+0000","last_peered":"2023-12-19T15:54:41.815787+0000","last_clean":"2023-12-19T15:54:41.815787+0000","last_became_active":"2023-12-19T15:54:41.814085+0000","last_became_peered":"2023-12-19T15:54:41.814085+0000","last_unstale":"2023-12-19T15:54:41.815787+0000","last_undegraded":"2023-12-19T15:54:41.815787+0000","last_fullsized":"2023-12-19T15:54:41.815787+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.814610+0000","last_change":"2023-12-19T15:54:40.809043+0000","last_active":"2023-12-19T15:54:40.814610+0000","last_peered":"2023-12-19T15:54:40.814610+0000","last_clean":"2023-12-19T15:54:40.814610+0000","last_became_active":"2023-12-19T15:54:40.808698+0000","last_became_peered":"2023-12-19T15:54:40.808698+0000","last_unstale":"2023-12-19T15:54:40.814610+0000","last_undegraded":"2023-12-19T15:54:40.814610+0000","last_fullsized":"2023-12-19T15:54:40.814610+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:18.983853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827822+0000","last_change":"2023-12-19T15:54:41.821030+0000","last_active":"2023-12-19T15:54:41.827822+0000","last_peered":"2023-12-19T15:54:41.827822+0000","last_clean":"2023-12-19T15:54:41.827822+0000","last_became_active":"2023-12-19T15:54:41.819703+0000","last_became_peered":"2023-12-19T15:54:41.819703+0000","last_unstale":"2023-12-19T15:54:41.827822+0000","last_undegraded":"2023-12-19T15:54:41.827822+0000","last_fullsized":"2023-12-19T15:54:41.827822+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:37:10.887215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.806770+0000","last_change":"2023-12-19T15:54:40.812154+0000","last_active":"2023-12-19T15:54:41.806770+0000","last_peered":"2023-12-19T15:54:41.806770+0000","last_clean":"2023-12-19T15:54:41.806770+0000","last_became_active":"2023-12-19T15:54:40.811878+0000","last_became_peered":"2023-12-19T15:54:40.811878+0000","last_unstale":"2023-12-19T15:54:41.806770+0000","last_undegraded":"2023-12-19T15:54:41.806770+0000","last_fullsized":"2023-12-19T15:54:41.806770+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:00:11.287374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.822811+0000","last_change":"2023-12-19T15:54:41.822811+0000","last_active":"2023-12-19T15:54:41.822811+0000","last_peered":"2023-12-19T15:54:41.822811+0000","last_clean":"2023-12-19T15:54:41.822811+0000","last_became_active":"2023-12-19T15:54:41.822519+0000","last_became_peered":"2023-12-19T15:54:41.822519+0000","last_unstale":"2023-12-19T15:54:41.822811+0000","last_undegraded":"2023-12-19T15:54:41.822811+0000","last_fullsized":"2023-12-19T15:54:41.822811+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:34.288102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838473+0000","last_change":"2023-12-19T15:54:40.808963+0000","last_active":"2023-12-19T15:54:41.838473+0000","last_peered":"2023-12-19T15:54:41.838473+0000","last_clean":"2023-12-19T15:54:41.838473+0000","last_became_active":"2023-12-19T15:54:40.808676+0000","last_became_peered":"2023-12-19T15:54:40.808676+0000","last_unstale":"2023-12-19T15:54:41.838473+0000","last_undegraded":"2023-12-19T15:54:41.838473+0000","last_fullsized":"2023-12-19T15:54:41.838473+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:58:29.350111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.808833+0000","last_change":"2023-12-19T15:54:41.808833+0000","last_active":"2023-12-19T15:54:41.808833+0000","last_peered":"2023-12-19T15:54:41.808833+0000","last_clean":"2023-12-19T15:54:41.808833+0000","last_became_active":"2023-12-19T15:54:41.808523+0000","last_became_peered":"2023-12-19T15:54:41.808523+0000","last_unstale":"2023-12-19T15:54:41.808833+0000","last_undegraded":"2023-12-19T15:54:41.808833+0000","last_fullsized":"2023-12-19T15:54:41.808833+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:49.115763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.808925+0000","last_change":"2023-12-19T15:54:40.805840+0000","last_active":"2023-12-19T15:54:40.808925+0000","last_peered":"2023-12-19T15:54:40.808925+0000","last_clean":"2023-12-19T15:54:40.808925+0000","last_became_active":"2023-12-19T15:54:40.805482+0000","last_became_peered":"2023-12-19T15:54:40.805482+0000","last_unstale":"2023-12-19T15:54:40.808925+0000","last_undegraded":"2023-12-19T15:54:40.808925+0000","last_fullsized":"2023-12-19T15:54:40.808925+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:02.337325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.830548+0000","last_change":"2023-12-19T15:54:41.830548+0000","last_active":"2023-12-19T15:54:41.830548+0000","last_peered":"2023-12-19T15:54:41.830548+0000","last_clean":"2023-12-19T15:54:41.830548+0000","last_became_active":"2023-12-19T15:54:41.830218+0000","last_became_peered":"2023-12-19T15:54:41.830218+0000","last_unstale":"2023-12-19T15:54:41.830548+0000","last_undegraded":"2023-12-19T15:54:41.830548+0000","last_fullsized":"2023-12-19T15:54:41.830548+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:12:20.404504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.812512+0000","last_change":"2023-12-19T15:54:40.808234+0000","last_active":"2023-12-19T15:54:40.812512+0000","last_peered":"2023-12-19T15:54:40.812512+0000","last_clean":"2023-12-19T15:54:40.812512+0000","last_became_active":"2023-12-19T15:54:40.807877+0000","last_became_peered":"2023-12-19T15:54:40.807877+0000","last_unstale":"2023-12-19T15:54:40.812512+0000","last_undegraded":"2023-12-19T15:54:40.812512+0000","last_fullsized":"2023-12-19T15:54:40.812512+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:09.280510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827771+0000","last_change":"2023-12-19T15:54:41.827771+0000","last_active":"2023-12-19T15:54:41.827771+0000","last_peered":"2023-12-19T15:54:41.827771+0000","last_clean":"2023-12-19T15:54:41.827771+0000","last_became_active":"2023-12-19T15:54:41.827459+0000","last_became_peered":"2023-12-19T15:54:41.827459+0000","last_unstale":"2023-12-19T15:54:41.827771+0000","last_undegraded":"2023-12-19T15:54:41.827771+0000","last_fullsized":"2023-12-19T15:54:41.827771+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:05:43.849237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.806792+0000","last_change":"2023-12-19T15:54:40.811804+0000","last_active":"2023-12-19T15:54:41.806792+0000","last_peered":"2023-12-19T15:54:41.806792+0000","last_clean":"2023-12-19T15:54:41.806792+0000","last_became_active":"2023-12-19T15:54:40.810575+0000","last_became_peered":"2023-12-19T15:54:40.810575+0000","last_unstale":"2023-12-19T15:54:41.806792+0000","last_undegraded":"2023-12-19T15:54:41.806792+0000","last_fullsized":"2023-12-19T15:54:41.806792+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:39:19.735141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821586+0000","last_change":"2023-12-19T15:54:41.810029+0000","last_active":"2023-12-19T15:54:41.821586+0000","last_peered":"2023-12-19T15:54:41.821586+0000","last_clean":"2023-12-19T15:54:41.821586+0000","last_became_active":"2023-12-19T15:54:41.809735+0000","last_became_peered":"2023-12-19T15:54:41.809735+0000","last_unstale":"2023-12-19T15:54:41.821586+0000","last_undegraded":"2023-12-19T15:54:41.821586+0000","last_fullsized":"2023-12-19T15:54:41.821586+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:47:57.492513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.800929+0000","last_change":"2023-12-19T15:54:40.810846+0000","last_active":"2023-12-19T15:54:41.800929+0000","last_peered":"2023-12-19T15:54:41.800929+0000","last_clean":"2023-12-19T15:54:41.800929+0000","last_became_active":"2023-12-19T15:54:40.810564+0000","last_became_peered":"2023-12-19T15:54:40.810564+0000","last_unstale":"2023-12-19T15:54:41.800929+0000","last_undegraded":"2023-12-19T15:54:41.800929+0000","last_fullsized":"2023-12-19T15:54:41.800929+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:59.922908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820764+0000","last_change":"2023-12-19T15:54:41.810861+0000","last_active":"2023-12-19T15:54:41.820764+0000","last_peered":"2023-12-19T15:54:41.820764+0000","last_clean":"2023-12-19T15:54:41.820764+0000","last_became_active":"2023-12-19T15:54:41.810542+0000","last_became_peered":"2023-12-19T15:54:41.810542+0000","last_unstale":"2023-12-19T15:54:41.820764+0000","last_undegraded":"2023-12-19T15:54:41.820764+0000","last_fullsized":"2023-12-19T15:54:41.820764+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:10:41.554966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.814739+0000","last_change":"2023-12-19T15:54:40.810807+0000","last_active":"2023-12-19T15:54:40.814739+0000","last_peered":"2023-12-19T15:54:40.814739+0000","last_clean":"2023-12-19T15:54:40.814739+0000","last_became_active":"2023-12-19T15:54:40.810546+0000","last_became_peered":"2023-12-19T15:54:40.810546+0000","last_unstale":"2023-12-19T15:54:40.814739+0000","last_undegraded":"2023-12-19T15:54:40.814739+0000","last_fullsized":"2023-12-19T15:54:40.814739+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:07:24.779124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828650+0000","last_change":"2023-12-19T15:54:41.824487+0000","last_active":"2023-12-19T15:54:41.828650+0000","last_peered":"2023-12-19T15:54:41.828650+0000","last_clean":"2023-12-19T15:54:41.828650+0000","last_became_active":"2023-12-19T15:54:41.824189+0000","last_became_peered":"2023-12-19T15:54:41.824189+0000","last_unstale":"2023-12-19T15:54:41.828650+0000","last_undegraded":"2023-12-19T15:54:41.828650+0000","last_fullsized":"2023-12-19T15:54:41.828650+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.809009+0000","last_change":"2023-12-19T15:54:40.807632+0000","last_active":"2023-12-19T15:54:41.809009+0000","last_peered":"2023-12-19T15:54:41.809009+0000","last_clean":"2023-12-19T15:54:41.809009+0000","last_became_active":"2023-12-19T15:54:40.807318+0000","last_became_peered":"2023-12-19T15:54:40.807318+0000","last_unstale":"2023-12-19T15:54:41.809009+0000","last_undegraded":"2023-12-19T15:54:41.809009+0000","last_fullsized":"2023-12-19T15:54:41.809009+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:43.089842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827658+0000","last_change":"2023-12-19T15:54:41.827658+0000","last_active":"2023-12-19T15:54:41.827658+0000","last_peered":"2023-12-19T15:54:41.827658+0000","last_clean":"2023-12-19T15:54:41.827658+0000","last_became_active":"2023-12-19T15:54:41.827113+0000","last_became_peered":"2023-12-19T15:54:41.827113+0000","last_unstale":"2023-12-19T15:54:41.827658+0000","last_undegraded":"2023-12-19T15:54:41.827658+0000","last_fullsized":"2023-12-19T15:54:41.827658+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:54:40.796038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.813624+0000","last_change":"2023-12-19T15:54:40.806660+0000","last_active":"2023-12-19T15:54:40.813624+0000","last_peered":"2023-12-19T15:54:40.813624+0000","last_clean":"2023-12-19T15:54:40.813624+0000","last_became_active":"2023-12-19T15:54:40.805788+0000","last_became_peered":"2023-12-19T15:54:40.805788+0000","last_unstale":"2023-12-19T15:54:40.813624+0000","last_undegraded":"2023-12-19T15:54:40.813624+0000","last_fullsized":"2023-12-19T15:54:40.813624+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:46.568208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.811977+0000","last_change":"2023-12-19T15:54:41.811977+0000","last_active":"2023-12-19T15:54:41.811977+0000","last_peered":"2023-12-19T15:54:41.811977+0000","last_clean":"2023-12-19T15:54:41.811977+0000","last_became_active":"2023-12-19T15:54:41.811504+0000","last_became_peered":"2023-12-19T15:54:41.811504+0000","last_unstale":"2023-12-19T15:54:41.811977+0000","last_undegraded":"2023-12-19T15:54:41.811977+0000","last_fullsized":"2023-12-19T15:54:41.811977+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.816475+0000","last_change":"2023-12-19T15:54:40.813601+0000","last_active":"2023-12-19T15:54:40.816475+0000","last_peered":"2023-12-19T15:54:40.816475+0000","last_clean":"2023-12-19T15:54:40.816475+0000","last_became_active":"2023-12-19T15:54:40.813282+0000","last_became_peered":"2023-12-19T15:54:40.813282+0000","last_unstale":"2023-12-19T15:54:40.816475+0000","last_undegraded":"2023-12-19T15:54:40.816475+0000","last_fullsized":"2023-12-19T15:54:40.816475+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:28.195156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"155'271","reported_seq":829,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.799674+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T15:54:40.799674+0000","last_peered":"2023-12-19T15:54:40.799674+0000","last_clean":"2023-12-19T15:54:40.799674+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T15:54:40.799674+0000","last_undegraded":"2023-12-19T15:54:40.799674+0000","last_fullsized":"2023-12-19T15:54:40.799674+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299904,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071936,"statfs":{"total":96636764160,"available":96329662464,"internally_reserved":0,"allocated":2678784,"data_stored":1642319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1379999999999999,"15min":1.135},"min":{"1min":0.68500000000000005,"5min":0.64800000000000002,"15min":0.54200000000000004},"max":{"1min":1.6160000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1779999999999999,"15min":1.1579999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.80900000000000005}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.149,"15min":1.1519999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.4330000000000001,"5min":1.7010000000000001,"15min":1.778},"last":1.202},{"interface":"front","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.542,"5min":1.698,"15min":1.698},"last":1.345}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3029999999999999,"15min":1.294},"min":{"1min":1.0369999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.669,"5min":1.6890000000000001,"15min":1.7969999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.345,"15min":1.3360000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.79300000000000004},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7889999999999999},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.875,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5960000000000001,"5min":1.6839999999999999,"15min":1.7130000000000001},"last":1.153},{"interface":"front","average":{"1min":1.276,"5min":1.355,"15min":1.367},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.5369999999999999,"5min":1.7270000000000001,"15min":1.7290000000000001},"last":1.2869999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.163,"15min":1.181},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.63200000000000001},"max":{"1min":1.5820000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.46},{"interface":"front","average":{"1min":1.212,"5min":1.2629999999999999,"15min":1.2609999999999999},"min":{"1min":0.92800000000000005,"5min":0.82099999999999995,"15min":0.63500000000000001},"max":{"1min":1.6180000000000001,"5min":1.8009999999999999,"15min":1.8899999999999999},"last":1.27}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2390000000000001,"15min":1.264},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60499999999999998},"max":{"1min":1.6519999999999999,"5min":1.758,"15min":2.0880000000000001},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.325,"15min":1.325},"min":{"1min":0.92900000000000005,"5min":0.79100000000000004,"15min":0.69999999999999996},"max":{"1min":1.7090000000000001,"5min":1.798,"15min":1.798},"last":1.5860000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4450000000000001,"15min":1.423},"min":{"1min":1.1910000000000001,"5min":0.82399999999999995,"15min":0.76000000000000001},"max":{"1min":1.778,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.70999999999999996},"max":{"1min":1.911,"5min":1.911,"15min":2.0110000000000001},"last":1.224}]}]},{"osd":6,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311604,"kb_used_data":2028,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060236,"statfs":{"total":96636764160,"available":96317681664,"internally_reserved":0,"allocated":2076672,"data_stored":1050969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.92900000000000005,"5min":0.79700000000000004,"15min":0.72799999999999998},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.081},{"interface":"front","average":{"1min":1.22,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.99399999999999999,"5min":0.78800000000000003,"15min":0.75900000000000001},"max":{"1min":1.4490000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.6970000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1910000000000001,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.52,"5min":1.617,"15min":1.6890000000000001},"last":1.272},{"interface":"front","average":{"1min":1.228,"5min":1.2549999999999999,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.5269999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.1930000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.2490000000000001,"15min":1.242},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.83699999999999997},"max":{"1min":1.611,"5min":1.629,"15min":1.6990000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.276,"15min":1.268},"min":{"1min":0.96399999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.7130000000000001},"last":1.389}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.647,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":1.917},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3600000000000001,"15min":1.3440000000000001},"min":{"1min":0.98699999999999999,"5min":0.96299999999999997,"15min":0.93300000000000005},"max":{"1min":1.6220000000000001,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.671}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2230000000000001,"15min":1.2030000000000001},"min":{"1min":0.86499999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":0.89000000000000001,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":1.8049999999999999,"5min":2.653,"15min":2.653},"last":1.9910000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.3},"min":{"1min":0.84899999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.774},{"interface":"front","average":{"1min":1.258,"5min":1.3320000000000001,"15min":1.361},"min":{"1min":0.878,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.762,"5min":1.871,"15min":1.99},"last":1.6160000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4039999999999999,"15min":1.4299999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.742,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":2.081},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":1.4490000000000001},"min":{"1min":0.97399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.905,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":1.8700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307504,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064336,"statfs":{"total":96636764160,"available":96321880064,"internally_reserved":0,"allocated":2072576,"data_stored":1050347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":2.2759999999999998},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6679999999999999,"5min":1.7909999999999999,"15min":2.7309999999999999},"last":1.4770000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.339,"15min":1.3220000000000001},"min":{"1min":1,"5min":0.86599999999999999,"15min":0.84499999999999997},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7370000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.282,"15min":1.2689999999999999},"min":{"1min":1.038,"5min":0.95699999999999996,"15min":0.82599999999999996},"max":{"1min":1.617,"5min":1.645,"15min":1.8779999999999999},"last":1.327}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6699999999999999,"5min":1.748,"15min":2.8239999999999998},"last":1.427},{"interface":"front","average":{"1min":1.234,"5min":1.2350000000000001,"15min":1.2230000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71199999999999997},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":1.871},"last":1.345}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.306},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.631,"5min":1.7350000000000001,"15min":2.4849999999999999},"last":1.359},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.2969999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6739999999999999,"5min":1.7929999999999999,"15min":1.9390000000000001},"last":1.2889999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1359999999999999,"15min":1.137},"min":{"1min":0.84299999999999997,"5min":0.69499999999999995,"15min":0.59999999999999998},"max":{"1min":1.659,"5min":1.659,"15min":2.109},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.244,"15min":1.27},"min":{"1min":0.86799999999999999,"5min":0.81799999999999995,"15min":0.56999999999999995},"max":{"1min":1.7869999999999999,"5min":1.8320000000000001,"15min":2.8999999999999999},"last":1.2030000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.333,"15min":1.3380000000000001},"min":{"1min":0.92100000000000004,"5min":0.84399999999999997,"15min":0.58899999999999997},"max":{"1min":1.8089999999999999,"5min":1.9299999999999999,"15min":2.0649999999999999},"last":1.446},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":1.1259999999999999,"5min":0.92500000000000004,"15min":0.83699999999999997},"max":{"1min":1.998,"5min":1.998,"15min":2.117},"last":1.141}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4259999999999999,"15min":1.4179999999999999},"min":{"1min":1.2310000000000001,"5min":0.95499999999999996,"15min":0.623},"max":{"1min":1.915,"5min":1.915,"15min":2.6000000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.419,"15min":1.411},"min":{"1min":1.0389999999999999,"5min":0.84599999999999997,"15min":0.68000000000000005},"max":{"1min":1.879,"5min":1.9870000000000001,"15min":2.3119999999999998},"last":1.393}]}]},{"osd":4,"up_from":13,"seq":55834575753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307500,"kb_used_data":2020,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064340,"statfs":{"total":96636764160,"available":96321884160,"internally_reserved":0,"allocated":2068480,"data_stored":1050325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1299999999999999,"15min":1.1100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.722},"last":1.014},{"interface":"front","average":{"1min":1.135,"5min":1.113,"15min":1.1399999999999999},"min":{"1min":0.76800000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.4770000000000001,"5min":1.5249999999999999,"15min":1.75},"last":1.1000000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.274,"15min":1.286},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6020000000000001,"5min":1.6819999999999999,"15min":2.2530000000000001},"last":1.133},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.69399999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.968},"last":1.349}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1850000000000001,"15min":1.1859999999999999},"min":{"1min":0.754,"5min":0.73599999999999999,"15min":0.65300000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":2.3410000000000002},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.84399999999999997,"5min":0.79900000000000004,"15min":0.57799999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1459999999999999},"last":1.2569999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.236,"15min":1.23},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.772},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.26,"15min":1.2869999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.65500000000000003},"max":{"1min":1.758,"5min":1.758,"15min":2.0800000000000001},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.204,"15min":1.1839999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.42399999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.151},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.6240000000000001,"5min":1.6579999999999999,"15min":1.998},"last":0.82099999999999995}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3140000000000001,"15min":1.306},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.60399999999999998},"max":{"1min":1.7190000000000001,"5min":1.766,"15min":2.3340000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.458,"15min":1.454},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.68100000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.429}]},{"osd":7,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3859999999999999,"15min":1.3600000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8260000000000001,"5min":1.839,"15min":2.0150000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.4890000000000001,"15min":1.47},"min":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.55600000000000005},"max":{"1min":1.8660000000000001,"5min":2.157,"15min":2.1659999999999999},"last":1.1970000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575752,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311612,"kb_used_data":2036,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060228,"statfs":{"total":96636764160,"available":96317673472,"internally_reserved":0,"allocated":2084864,"data_stored":1051931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91100000000000003},"min":{"1min":0.65600000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.28,"5min":1.5509999999999999,"15min":1.897},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0289999999999999,"15min":1.03},"min":{"1min":0.68500000000000005,"5min":0.63200000000000001,"15min":0.61099999999999999},"max":{"1min":1.375,"5min":1.4179999999999999,"15min":2.113},"last":1.117}]},{"osd":1,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98599999999999999,"15min":0.97299999999999998},"min":{"1min":0.78900000000000003,"5min":0.60999999999999999,"15min":0.57699999999999996},"max":{"1min":1.345,"5min":1.75,"15min":1.75},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1619999999999999,"15min":1.127},"min":{"1min":0.89000000000000001,"5min":0.73999999999999999,"15min":0.53900000000000003},"max":{"1min":1.6759999999999999,"5min":1.706,"15min":1.8149999999999999},"last":1.254}]},{"osd":2,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.175,"15min":1.173},"min":{"1min":0.93000000000000005,"5min":0.77900000000000003,"15min":0.74099999999999999},"max":{"1min":1.48,"5min":1.611,"15min":1.992},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2949999999999999,"15min":1.268},"min":{"1min":0.91600000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.593,"5min":1.794,"15min":1.794},"last":1.349}]},{"osd":4,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3859999999999999,"15min":1.3520000000000001},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.879},"max":{"1min":1.7130000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3560000000000001,"15min":1.325},"min":{"1min":0.93400000000000005,"5min":0.91500000000000004,"15min":0.71699999999999997},"max":{"1min":1.5680000000000001,"5min":1.8879999999999999,"15min":2.1120000000000001},"last":1.444}]},{"osd":5,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.373,"15min":1.347},"min":{"1min":1.024,"5min":1.0089999999999999,"15min":0.79200000000000004},"max":{"1min":1.5509999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.458,"15min":1.431},"min":{"1min":1.181,"5min":0.84099999999999997,"15min":0.81000000000000005},"max":{"1min":2.0609999999999999,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.4059999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5149999999999999,"15min":1.488},"min":{"1min":1.1339999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1669999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.494,"15min":1.4670000000000001},"min":{"1min":1.1910000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.948,"5min":2.141,"15min":2.1909999999999998},"last":1.2110000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.589,"15min":1.5669999999999999},"min":{"1min":1.333,"5min":1.1339999999999999,"15min":1.0169999999999999},"max":{"1min":1.863,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.552,"15min":1.518},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":0.92900000000000005},"max":{"1min":1.7490000000000001,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.3740000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575754,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311600,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060240,"statfs":{"total":96636764160,"available":96317685760,"internally_reserved":0,"allocated":2072576,"data_stored":1049902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.96299999999999997,"15min":0.95799999999999996},"min":{"1min":0.67700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2609999999999999,"5min":1.587,"15min":1.587},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.163,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.95199999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5089999999999999,"5min":1.595,"15min":1.944},"last":0.86899999999999999}]},{"osd":1,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1779999999999999,"15min":1.181},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6870000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.105,"5min":1.1299999999999999,"15min":1.1479999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69099999999999995},"max":{"1min":1.353,"5min":1.6359999999999999,"15min":1.7430000000000001},"last":1.0800000000000001}]},{"osd":3,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0169999999999999,"15min":1.0289999999999999},"min":{"1min":0.75600000000000001,"5min":0.64900000000000002,"15min":0.64000000000000001},"max":{"1min":1.474,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.006},{"interface":"front","average":{"1min":1.133,"5min":1.1379999999999999,"15min":1.129},"min":{"1min":0.82199999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.5029999999999999,"5min":1.6739999999999999,"15min":1.77},"last":0.94299999999999995}]},{"osd":4,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.387,"15min":1.413},"min":{"1min":1.038,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.649,"5min":1.7490000000000001,"15min":2.0790000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4379999999999999,"15min":1.4570000000000001},"min":{"1min":1.048,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.3410000000000002},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":1.101,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":1.8680000000000001,"5min":1.9039999999999999,"15min":2.0659999999999998},"last":1.125},{"interface":"front","average":{"1min":1.409,"5min":1.389,"15min":1.4039999999999999},"min":{"1min":1.1180000000000001,"5min":0.94499999999999995,"15min":0.92700000000000005},"max":{"1min":1.996,"5min":1.996,"15min":2.016},"last":1.349}]},{"osd":6,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.393,"15min":1.395},"min":{"1min":1.1319999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"max":{"1min":1.782,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.173},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.518,"15min":1.5089999999999999},"min":{"1min":1.196,"5min":1.073,"15min":1.073},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.242},"last":1.2649999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5369999999999999,"15min":1.5549999999999999},"min":{"1min":1.0900000000000001,"5min":1.075,"15min":1.04},"max":{"1min":1.9379999999999999,"5min":1.9750000000000001,"15min":2.2389999999999999},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5169999999999999,"15min":1.518},"min":{"1min":1.1819999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.9830000000000001,"5min":2.016,"15min":2.048},"last":1.496}]}]},{"osd":1,"up_from":13,"seq":55834575752,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311600,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060240,"statfs":{"total":96636764160,"available":96317685760,"internally_reserved":0,"allocated":2072576,"data_stored":1050395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93000000000000005,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.628,"15min":0.57699999999999996},"max":{"1min":1.4890000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.006,"15min":1.032},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.532,"5min":1.532,"15min":1.6879999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.044,"15min":1.0680000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.7989999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.121,"5min":1.175,"15min":1.1679999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.75},"max":{"1min":1.4750000000000001,"5min":1.6479999999999999,"15min":1.7549999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.278,"15min":1.2609999999999999},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.67000000000000004},"max":{"1min":1.784,"5min":1.7989999999999999,"15min":1.851},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1040000000000001,"15min":1.1020000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.1970000000000001,"5min":1.778,"15min":1.849},"last":0.89500000000000002}]},{"osd":4,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.353,"15min":1.3300000000000001},"min":{"1min":1.056,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.601,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.032},{"interface":"front","average":{"1min":1.359,"5min":1.409,"15min":1.4239999999999999},"min":{"1min":1.0860000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.6439999999999999,"5min":1.718,"15min":1.921},"last":1.1060000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.355,"15min":1.3420000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.93500000000000005},"max":{"1min":1.6279999999999999,"5min":1.7450000000000001,"15min":1.9199999999999999},"last":1.304},{"interface":"front","average":{"1min":1.413,"5min":1.431,"15min":1.427},"min":{"1min":1.0529999999999999,"5min":0.95299999999999996,"15min":0.94999999999999996},"max":{"1min":1.8540000000000001,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.95199999999999996}]},{"osd":6,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3999999999999999,"15min":1.3939999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.995},"max":{"1min":1.6040000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.349},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.52,"15min":1.504},"min":{"1min":1.109,"5min":1.109,"15min":0.88800000000000001},"max":{"1min":1.867,"5min":2.0880000000000001,"15min":2.1040000000000001},"last":1.1659999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.504,"15min":1.52},"min":{"1min":1.23,"5min":1.1799999999999999,"15min":1.107},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4830000000000001,"15min":1.48},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":1.6919999999999999,"5min":2.0659999999999998,"15min":2.0699999999999998},"last":1.224}]}]},{"osd":0,"up_from":13,"seq":55834575752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299892,"kb_used_data":2604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071948,"statfs":{"total":96636764160,"available":96329674752,"internally_reserved":0,"allocated":2666496,"data_stored":1639738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.66500000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.78000000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5649999999999999,"15min":1.895},"last":1.101}]},{"osd":2,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1259999999999999,"15min":1.1439999999999999},"min":{"1min":0.85599999999999998,"5min":0.502,"15min":0.47399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":2.5129999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.256,"15min":1.2789999999999999},"min":{"1min":0.93600000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.679,"5min":1.708,"15min":2.8250000000000002},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.038},"min":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.66200000000000003},"max":{"1min":1.5349999999999999,"5min":1.599,"15min":1.9970000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2070000000000001,"15min":1.2090000000000001},"min":{"1min":0.92200000000000004,"5min":0.621,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.794},"last":1.4359999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3919999999999999,"15min":1.351},"min":{"1min":1.0640000000000001,"5min":0.94499999999999995,"15min":0.80000000000000004},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.379},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.327,"15min":1.3169999999999999},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8280000000000001,"5min":1.9059999999999999,"15min":2.2210000000000001},"last":1.4790000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.419,"15min":1.4279999999999999},"min":{"1min":1.175,"5min":0.871,"15min":0.73099999999999998},"max":{"1min":1.879,"5min":1.901,"15min":2.4470000000000001},"last":1.754},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4670000000000001,"15min":1.4359999999999999},"min":{"1min":1.004,"5min":1.004,"15min":0.79400000000000004},"max":{"1min":1.889,"5min":1.9510000000000001,"15min":2.468},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.413,"15min":1.385},"min":{"1min":1.1830000000000001,"5min":0.90000000000000002,"15min":0.88500000000000001},"max":{"1min":1.925,"5min":1.925,"15min":2.0830000000000002},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.53,"15min":1.5},"min":{"1min":1.2110000000000001,"5min":0.92600000000000005,"15min":0.81799999999999995},"max":{"1min":1.9470000000000001,"5min":2.379,"15min":2.379},"last":1.8959999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5429999999999999,"15min":1.5049999999999999},"min":{"1min":1.0900000000000001,"5min":0.99299999999999999,"15min":0.90900000000000003},"max":{"1min":1.968,"5min":2.105,"15min":2.698},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.49,"5min":1.5089999999999999,"15min":1.482},"min":{"1min":1.022,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.8380000000000001,"5min":2.016,"15min":3.23},"last":1.948}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T15:54:46.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T15:54:46.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.324+0000 7f709b04c700 1 Processor -- start 2023-12-19T15:54:46.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.324+0000 7f709b04c700 1 -- start start 2023-12-19T15:54:46.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709b04c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70940cc470 0x7f70940cc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709b04c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70940ca5a0 con 0x7f709406a540 2023-12-19T15:54:46.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709b04c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709406dac0 con 0x7f7094158e70 2023-12-19T15:54:46.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709b04c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709406d220 con 0x7f70940cc470 2023-12-19T15:54:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a04a700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7094158e70 0x7f70940afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:59942/0 (socket says 172.21.15.50:59942) 2023-12-19T15:54:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f7099849700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f709406a540 0x7f709406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40646/0 (socket says 172.21.15.50:40646) 2023-12-19T15:54:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a04a700 1 -- 172.21.15.50:0/2763032560 learned_addr learned my addr 172.21.15.50:0/2763032560 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a84b700 1 --2- 172.21.15.50:0/2763032560 >> v2:172.21.15.125:3300/0 conn(0x7f70940cc470 0x7f70940cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192972296 0 0) 0x7f709406dac0 con 0x7f7094158e70 2023-12-19T15:54:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7080003570 con 0x7f7094158e70 2023-12-19T15:54:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209454539 0 0) 0x7f70940ca5a0 con 0x7f709406a540 2023-12-19T15:54:46.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709406dac0 con 0x7f709406a540 2023-12-19T15:54:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a84b700 1 -- 172.21.15.50:0/2763032560 >> v1:172.21.15.125:6789/0 conn(0x7f7094158e70 legacy=0x7f70940afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a84b700 1 -- 172.21.15.50:0/2763032560 >> v1:172.21.15.50:6789/0 conn(0x7f709406a540 legacy=0x7f709406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.328+0000 7f709a84b700 1 -- 172.21.15.50:0/2763032560 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709406d500 con 0x7f70940cc470 2023-12-19T15:54:46.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709a84b700 1 --2- 172.21.15.50:0/2763032560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940cc890 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f708c009e60 tx=0x7f708c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe91f570c6cd9ced server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00e070 con 0x7f70940cc470 2023-12-19T15:54:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c00bab0 con 0x7f70940cc470 2023-12-19T15:54:46.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f7099048700 1 -- 172.21.15.50:0/2763032560 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00be70 con 0x7f70940cc470 2023-12-19T15:54:46.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 -- 172.21.15.50:0/2763032560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 msgr2=0x7f70940cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 --2- 172.21.15.50:0/2763032560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940cc890 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f708c009e60 tx=0x7f708c009780 comp rx=0 tx=0).stop 2023-12-19T15:54:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 -- 172.21.15.50:0/2763032560 shutdown_connections 2023-12-19T15:54:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 --2- 172.21.15.50:0/2763032560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940cc890 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 -- 172.21.15.50:0/2763032560 >> 172.21.15.50:0/2763032560 conn(0x7f70940670e0 msgr2=0x7f70940674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 -- 172.21.15.50:0/2763032560 shutdown_connections 2023-12-19T15:54:46.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.332+0000 7f709b04c700 1 -- 172.21.15.50:0/2763032560 wait complete. 2023-12-19T15:54:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.336+0000 7f709b04c700 1 Processor -- start 2023-12-19T15:54:46.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.336+0000 7f709b04c700 1 -- start start 2023-12-19T15:54:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f709406a540 0x7f7094156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70940c6130 con 0x7f7094158e70 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70940c5520 con 0x7f70940cc470 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70940c5800 con 0x7f709406a540 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34298/0 (socket says 172.21.15.50:34298) 2023-12-19T15:54:46.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 -- 172.21.15.50:0/2578130459 learned_addr learned my addr 172.21.15.50:0/2578130459 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f7099849700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a04a700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f709406a540 0x7f7094156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 -- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f709406a540 msgr2=0x7f7094156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f709406a540 0x7f7094156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 -- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 msgr2=0x7f70940c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 -- 172.21.15.50:0/2578130459 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c009a20 con 0x7f7094158e70 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f7099849700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709a84b700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f70940cfb30 tx=0x7f708c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c026480 con 0x7f7094158e70 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c026600 con 0x7f7094158e70 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70940c5ae0 con 0x7f7094158e70 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c0269c0 con 0x7f7094158e70 2023-12-19T15:54:46.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.340+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70940c65c0 con 0x7f7094158e70 2023-12-19T15:54:46.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.344+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70940c62e0 con 0x7f7094158e70 2023-12-19T15:54:46.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.344+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f708c01c400 con 0x7f7094158e70 2023-12-19T15:54:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.344+0000 7f708a7fc700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 0x7f7078062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.344+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f708c07ef60 con 0x7f7094158e70 2023-12-19T15:54:46.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.348+0000 7f709a04a700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 0x7f7078062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.348+0000 7f709a04a700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 0x7f7078062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70900058d0 tx=0x7f7090009040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.352+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f70940c62e0 con 0x7f7094158e70 2023-12-19T15:54:46.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.520+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f709404be40 con 0x7f7078060670 2023-12-19T15:54:46.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.540+0000 7f708a7fc700 1 -- 172.21.15.50:0/2578130459 <== mgr.7577 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287147 (secure 0 0 0) 0x7f709404be40 con 0x7f7078060670 2023-12-19T15:54:46.545 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:46.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 msgr2=0x7f7078062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 0x7f7078062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70900058d0 tx=0x7f7090009040 comp rx=0 tx=0).stop 2023-12-19T15:54:46.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 msgr2=0x7f709416c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f70940cfb30 tx=0x7f708c019000 comp rx=0 tx=0).stop 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 shutdown_connections 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70940cc470 0x7f70940c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7078060670 0x7f7078062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7094158e70 0x7f709416c5f0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 --2- 172.21.15.50:0/2578130459 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f709406a540 0x7f7094156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 >> 172.21.15.50:0/2578130459 conn(0x7f70940670e0 msgr2=0x7f70940b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:46.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 shutdown_connections 2023-12-19T15:54:46.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.548+0000 7f709b04c700 1 -- 172.21.15.50:0/2578130459 wait complete. 2023-12-19T15:54:46.554 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T15:54:46.573 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T15:54:44.923567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2461216,"kb_used_data":17376,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752513504,"statfs":{"total":773094113280,"available":770573828096,"internally_reserved":0,"allocated":17793024,"data_stored":9585926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.159912"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814187+0000","last_change":"2023-12-19T15:54:41.814187+0000","last_active":"2023-12-19T15:54:41.814187+0000","last_peered":"2023-12-19T15:54:41.814187+0000","last_clean":"2023-12-19T15:54:41.814187+0000","last_became_active":"2023-12-19T15:54:41.813759+0000","last_became_peered":"2023-12-19T15:54:41.813759+0000","last_unstale":"2023-12-19T15:54:41.814187+0000","last_undegraded":"2023-12-19T15:54:41.814187+0000","last_fullsized":"2023-12-19T15:54:41.814187+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:28:41.691655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828060+0000","last_change":"2023-12-19T15:54:41.821043+0000","last_active":"2023-12-19T15:54:41.828060+0000","last_peered":"2023-12-19T15:54:41.828060+0000","last_clean":"2023-12-19T15:54:41.828060+0000","last_became_active":"2023-12-19T15:54:41.820644+0000","last_became_peered":"2023-12-19T15:54:41.820644+0000","last_unstale":"2023-12-19T15:54:41.828060+0000","last_undegraded":"2023-12-19T15:54:41.828060+0000","last_fullsized":"2023-12-19T15:54:41.828060+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:21.094171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812097+0000","last_change":"2023-12-19T15:54:41.812097+0000","last_active":"2023-12-19T15:54:41.812097+0000","last_peered":"2023-12-19T15:54:41.812097+0000","last_clean":"2023-12-19T15:54:41.812097+0000","last_became_active":"2023-12-19T15:54:41.811317+0000","last_became_peered":"2023-12-19T15:54:41.811317+0000","last_unstale":"2023-12-19T15:54:41.812097+0000","last_undegraded":"2023-12-19T15:54:41.812097+0000","last_fullsized":"2023-12-19T15:54:41.812097+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:37.202908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818918+0000","last_change":"2023-12-19T15:54:41.810547+0000","last_active":"2023-12-19T15:54:41.818918+0000","last_peered":"2023-12-19T15:54:41.818918+0000","last_clean":"2023-12-19T15:54:41.818918+0000","last_became_active":"2023-12-19T15:54:41.810236+0000","last_became_peered":"2023-12-19T15:54:41.810236+0000","last_unstale":"2023-12-19T15:54:41.818918+0000","last_undegraded":"2023-12-19T15:54:41.818918+0000","last_fullsized":"2023-12-19T15:54:41.818918+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:19.079075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.805538+0000","last_change":"2023-12-19T15:54:41.805538+0000","last_active":"2023-12-19T15:54:41.805538+0000","last_peered":"2023-12-19T15:54:41.805538+0000","last_clean":"2023-12-19T15:54:41.805538+0000","last_became_active":"2023-12-19T15:54:41.805262+0000","last_became_peered":"2023-12-19T15:54:41.805262+0000","last_unstale":"2023-12-19T15:54:41.805538+0000","last_undegraded":"2023-12-19T15:54:41.805538+0000","last_fullsized":"2023-12-19T15:54:41.805538+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:49.115763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.830091+0000","last_change":"2023-12-19T15:54:41.828246+0000","last_active":"2023-12-19T15:54:41.830091+0000","last_peered":"2023-12-19T15:54:41.830091+0000","last_clean":"2023-12-19T15:54:41.830091+0000","last_became_active":"2023-12-19T15:54:41.827107+0000","last_became_peered":"2023-12-19T15:54:41.827107+0000","last_unstale":"2023-12-19T15:54:41.830091+0000","last_undegraded":"2023-12-19T15:54:41.830091+0000","last_fullsized":"2023-12-19T15:54:41.830091+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:02.283783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.829020+0000","last_change":"2023-12-19T15:54:41.829020+0000","last_active":"2023-12-19T15:54:41.829020+0000","last_peered":"2023-12-19T15:54:41.829020+0000","last_clean":"2023-12-19T15:54:41.829020+0000","last_became_active":"2023-12-19T15:54:41.828574+0000","last_became_peered":"2023-12-19T15:54:41.828574+0000","last_unstale":"2023-12-19T15:54:41.829020+0000","last_undegraded":"2023-12-19T15:54:41.829020+0000","last_fullsized":"2023-12-19T15:54:41.829020+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:44.281335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.816471+0000","last_change":"2023-12-19T15:54:41.816471+0000","last_active":"2023-12-19T15:54:41.816471+0000","last_peered":"2023-12-19T15:54:41.816471+0000","last_clean":"2023-12-19T15:54:41.816471+0000","last_became_active":"2023-12-19T15:54:41.816202+0000","last_became_peered":"2023-12-19T15:54:41.816202+0000","last_unstale":"2023-12-19T15:54:41.816471+0000","last_undegraded":"2023-12-19T15:54:41.816471+0000","last_fullsized":"2023-12-19T15:54:41.816471+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:54:40.796038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828644+0000","last_change":"2023-12-19T15:54:41.826479+0000","last_active":"2023-12-19T15:54:41.828644+0000","last_peered":"2023-12-19T15:54:41.828644+0000","last_clean":"2023-12-19T15:54:41.828644+0000","last_became_active":"2023-12-19T15:54:41.826187+0000","last_became_peered":"2023-12-19T15:54:41.826187+0000","last_unstale":"2023-12-19T15:54:41.828644+0000","last_undegraded":"2023-12-19T15:54:41.828644+0000","last_fullsized":"2023-12-19T15:54:41.828644+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:32.959716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820814+0000","last_change":"2023-12-19T15:54:41.820814+0000","last_active":"2023-12-19T15:54:41.820814+0000","last_peered":"2023-12-19T15:54:41.820814+0000","last_clean":"2023-12-19T15:54:41.820814+0000","last_became_active":"2023-12-19T15:54:41.820465+0000","last_became_peered":"2023-12-19T15:54:41.820465+0000","last_unstale":"2023-12-19T15:54:41.820814+0000","last_undegraded":"2023-12-19T15:54:41.820814+0000","last_fullsized":"2023-12-19T15:54:41.820814+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:58.506059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828139+0000","last_change":"2023-12-19T15:54:41.821450+0000","last_active":"2023-12-19T15:54:41.828139+0000","last_peered":"2023-12-19T15:54:41.828139+0000","last_clean":"2023-12-19T15:54:41.828139+0000","last_became_active":"2023-12-19T15:54:41.820932+0000","last_became_peered":"2023-12-19T15:54:41.820932+0000","last_unstale":"2023-12-19T15:54:41.828139+0000","last_undegraded":"2023-12-19T15:54:41.828139+0000","last_fullsized":"2023-12-19T15:54:41.828139+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:45:47.022911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.800393+0000","last_change":"2023-12-19T15:54:41.800393+0000","last_active":"2023-12-19T15:54:41.800393+0000","last_peered":"2023-12-19T15:54:41.800393+0000","last_clean":"2023-12-19T15:54:41.800393+0000","last_became_active":"2023-12-19T15:54:41.799911+0000","last_became_peered":"2023-12-19T15:54:41.799911+0000","last_unstale":"2023-12-19T15:54:41.800393+0000","last_undegraded":"2023-12-19T15:54:41.800393+0000","last_fullsized":"2023-12-19T15:54:41.800393+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:04:34.113251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828537+0000","last_change":"2023-12-19T15:54:41.822347+0000","last_active":"2023-12-19T15:54:41.828537+0000","last_peered":"2023-12-19T15:54:41.828537+0000","last_clean":"2023-12-19T15:54:41.828537+0000","last_became_active":"2023-12-19T15:54:41.820183+0000","last_became_peered":"2023-12-19T15:54:41.820183+0000","last_unstale":"2023-12-19T15:54:41.828537+0000","last_undegraded":"2023-12-19T15:54:41.828537+0000","last_fullsized":"2023-12-19T15:54:41.828537+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:34.288102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826562+0000","last_change":"2023-12-19T15:54:41.826562+0000","last_active":"2023-12-19T15:54:41.826562+0000","last_peered":"2023-12-19T15:54:41.826562+0000","last_clean":"2023-12-19T15:54:41.826562+0000","last_became_active":"2023-12-19T15:54:41.826123+0000","last_became_peered":"2023-12-19T15:54:41.826123+0000","last_unstale":"2023-12-19T15:54:41.826562+0000","last_undegraded":"2023-12-19T15:54:41.826562+0000","last_fullsized":"2023-12-19T15:54:41.826562+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:47.924959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815990+0000","last_change":"2023-12-19T15:54:41.815990+0000","last_active":"2023-12-19T15:54:41.815990+0000","last_peered":"2023-12-19T15:54:41.815990+0000","last_clean":"2023-12-19T15:54:41.815990+0000","last_became_active":"2023-12-19T15:54:41.815702+0000","last_became_peered":"2023-12-19T15:54:41.815702+0000","last_unstale":"2023-12-19T15:54:41.815990+0000","last_undegraded":"2023-12-19T15:54:41.815990+0000","last_fullsized":"2023-12-19T15:54:41.815990+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:50:51.430483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827056+0000","last_change":"2023-12-19T15:54:41.816946+0000","last_active":"2023-12-19T15:54:41.827056+0000","last_peered":"2023-12-19T15:54:41.827056+0000","last_clean":"2023-12-19T15:54:41.827056+0000","last_became_active":"2023-12-19T15:54:41.816641+0000","last_became_peered":"2023-12-19T15:54:41.816641+0000","last_unstale":"2023-12-19T15:54:41.827056+0000","last_undegraded":"2023-12-19T15:54:41.827056+0000","last_fullsized":"2023-12-19T15:54:41.827056+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:29.211645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827061+0000","last_change":"2023-12-19T15:54:41.817145+0000","last_active":"2023-12-19T15:54:41.827061+0000","last_peered":"2023-12-19T15:54:41.827061+0000","last_clean":"2023-12-19T15:54:41.827061+0000","last_became_active":"2023-12-19T15:54:41.816767+0000","last_became_peered":"2023-12-19T15:54:41.816767+0000","last_unstale":"2023-12-19T15:54:41.827061+0000","last_undegraded":"2023-12-19T15:54:41.827061+0000","last_fullsized":"2023-12-19T15:54:41.827061+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:20.000343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.811453+0000","last_change":"2023-12-19T15:54:41.811453+0000","last_active":"2023-12-19T15:54:41.811453+0000","last_peered":"2023-12-19T15:54:41.811453+0000","last_clean":"2023-12-19T15:54:41.811453+0000","last_became_active":"2023-12-19T15:54:41.811126+0000","last_became_peered":"2023-12-19T15:54:41.811126+0000","last_unstale":"2023-12-19T15:54:41.811453+0000","last_undegraded":"2023-12-19T15:54:41.811453+0000","last_fullsized":"2023-12-19T15:54:41.811453+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:11:37.202908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.823610+0000","last_change":"2023-12-19T15:54:41.823610+0000","last_active":"2023-12-19T15:54:41.823610+0000","last_peered":"2023-12-19T15:54:41.823610+0000","last_clean":"2023-12-19T15:54:41.823610+0000","last_became_active":"2023-12-19T15:54:41.823193+0000","last_became_peered":"2023-12-19T15:54:41.823193+0000","last_unstale":"2023-12-19T15:54:41.823610+0000","last_undegraded":"2023-12-19T15:54:41.823610+0000","last_fullsized":"2023-12-19T15:54:41.823610+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:08.996205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827756+0000","last_change":"2023-12-19T15:54:41.827756+0000","last_active":"2023-12-19T15:54:41.827756+0000","last_peered":"2023-12-19T15:54:41.827756+0000","last_clean":"2023-12-19T15:54:41.827756+0000","last_became_active":"2023-12-19T15:54:41.824918+0000","last_became_peered":"2023-12-19T15:54:41.824918+0000","last_unstale":"2023-12-19T15:54:41.827756+0000","last_undegraded":"2023-12-19T15:54:41.827756+0000","last_fullsized":"2023-12-19T15:54:41.827756+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:03.353814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.819874+0000","last_change":"2023-12-19T15:54:41.813919+0000","last_active":"2023-12-19T15:54:41.819874+0000","last_peered":"2023-12-19T15:54:41.819874+0000","last_clean":"2023-12-19T15:54:41.819874+0000","last_became_active":"2023-12-19T15:54:41.813616+0000","last_became_peered":"2023-12-19T15:54:41.813616+0000","last_unstale":"2023-12-19T15:54:41.819874+0000","last_undegraded":"2023-12-19T15:54:41.819874+0000","last_fullsized":"2023-12-19T15:54:41.819874+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:27:09.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.813715+0000","last_change":"2023-12-19T15:54:41.813715+0000","last_active":"2023-12-19T15:54:41.813715+0000","last_peered":"2023-12-19T15:54:41.813715+0000","last_clean":"2023-12-19T15:54:41.813715+0000","last_became_active":"2023-12-19T15:54:41.810954+0000","last_became_peered":"2023-12-19T15:54:41.810954+0000","last_unstale":"2023-12-19T15:54:41.813715+0000","last_undegraded":"2023-12-19T15:54:41.813715+0000","last_fullsized":"2023-12-19T15:54:41.813715+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:13:57.376090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821174+0000","last_change":"2023-12-19T15:54:41.811358+0000","last_active":"2023-12-19T15:54:41.821174+0000","last_peered":"2023-12-19T15:54:41.821174+0000","last_clean":"2023-12-19T15:54:41.821174+0000","last_became_active":"2023-12-19T15:54:41.811092+0000","last_became_peered":"2023-12-19T15:54:41.811092+0000","last_unstale":"2023-12-19T15:54:41.821174+0000","last_undegraded":"2023-12-19T15:54:41.821174+0000","last_fullsized":"2023-12-19T15:54:41.821174+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:49:42.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826958+0000","last_change":"2023-12-19T15:54:41.815072+0000","last_active":"2023-12-19T15:54:41.826958+0000","last_peered":"2023-12-19T15:54:41.826958+0000","last_clean":"2023-12-19T15:54:41.826958+0000","last_became_active":"2023-12-19T15:54:41.814781+0000","last_became_peered":"2023-12-19T15:54:41.814781+0000","last_unstale":"2023-12-19T15:54:41.826958+0000","last_undegraded":"2023-12-19T15:54:41.826958+0000","last_fullsized":"2023-12-19T15:54:41.826958+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:07:25.795613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.804497+0000","last_change":"2023-12-19T15:54:41.804497+0000","last_active":"2023-12-19T15:54:41.804497+0000","last_peered":"2023-12-19T15:54:41.804497+0000","last_clean":"2023-12-19T15:54:41.804497+0000","last_became_active":"2023-12-19T15:54:41.804206+0000","last_became_peered":"2023-12-19T15:54:41.804206+0000","last_unstale":"2023-12-19T15:54:41.804497+0000","last_undegraded":"2023-12-19T15:54:41.804497+0000","last_fullsized":"2023-12-19T15:54:41.804497+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:40.591338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818166+0000","last_change":"2023-12-19T15:54:41.818166+0000","last_active":"2023-12-19T15:54:41.818166+0000","last_peered":"2023-12-19T15:54:41.818166+0000","last_clean":"2023-12-19T15:54:41.818166+0000","last_became_active":"2023-12-19T15:54:41.817837+0000","last_became_peered":"2023-12-19T15:54:41.817837+0000","last_unstale":"2023-12-19T15:54:41.818166+0000","last_undegraded":"2023-12-19T15:54:41.818166+0000","last_fullsized":"2023-12-19T15:54:41.818166+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:04:41.835159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827184+0000","last_change":"2023-12-19T15:54:41.827184+0000","last_active":"2023-12-19T15:54:41.827184+0000","last_peered":"2023-12-19T15:54:41.827184+0000","last_clean":"2023-12-19T15:54:41.827184+0000","last_became_active":"2023-12-19T15:54:41.826865+0000","last_became_peered":"2023-12-19T15:54:41.826865+0000","last_unstale":"2023-12-19T15:54:41.827184+0000","last_undegraded":"2023-12-19T15:54:41.827184+0000","last_fullsized":"2023-12-19T15:54:41.827184+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:30:40.870332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820198+0000","last_change":"2023-12-19T15:54:41.820198+0000","last_active":"2023-12-19T15:54:41.820198+0000","last_peered":"2023-12-19T15:54:41.820198+0000","last_clean":"2023-12-19T15:54:41.820198+0000","last_became_active":"2023-12-19T15:54:41.819870+0000","last_became_peered":"2023-12-19T15:54:41.819870+0000","last_unstale":"2023-12-19T15:54:41.820198+0000","last_undegraded":"2023-12-19T15:54:41.820198+0000","last_fullsized":"2023-12-19T15:54:41.820198+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:17:10.903420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814790+0000","last_change":"2023-12-19T15:54:41.814790+0000","last_active":"2023-12-19T15:54:41.814790+0000","last_peered":"2023-12-19T15:54:41.814790+0000","last_clean":"2023-12-19T15:54:41.814790+0000","last_became_active":"2023-12-19T15:54:41.814482+0000","last_became_peered":"2023-12-19T15:54:41.814482+0000","last_unstale":"2023-12-19T15:54:41.814790+0000","last_undegraded":"2023-12-19T15:54:41.814790+0000","last_fullsized":"2023-12-19T15:54:41.814790+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:58:36.176831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812370+0000","last_change":"2023-12-19T15:54:41.812370+0000","last_active":"2023-12-19T15:54:41.812370+0000","last_peered":"2023-12-19T15:54:41.812370+0000","last_clean":"2023-12-19T15:54:41.812370+0000","last_became_active":"2023-12-19T15:54:41.809818+0000","last_became_peered":"2023-12-19T15:54:41.809818+0000","last_unstale":"2023-12-19T15:54:41.812370+0000","last_undegraded":"2023-12-19T15:54:41.812370+0000","last_fullsized":"2023-12-19T15:54:41.812370+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:06.694443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814923+0000","last_change":"2023-12-19T15:54:41.814923+0000","last_active":"2023-12-19T15:54:41.814923+0000","last_peered":"2023-12-19T15:54:41.814923+0000","last_clean":"2023-12-19T15:54:41.814923+0000","last_became_active":"2023-12-19T15:54:41.814627+0000","last_became_peered":"2023-12-19T15:54:41.814627+0000","last_unstale":"2023-12-19T15:54:41.814923+0000","last_undegraded":"2023-12-19T15:54:41.814923+0000","last_fullsized":"2023-12-19T15:54:41.814923+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:47.615053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815877+0000","last_change":"2023-12-19T15:54:41.815877+0000","last_active":"2023-12-19T15:54:41.815877+0000","last_peered":"2023-12-19T15:54:41.815877+0000","last_clean":"2023-12-19T15:54:41.815877+0000","last_became_active":"2023-12-19T15:54:41.815580+0000","last_became_peered":"2023-12-19T15:54:41.815580+0000","last_unstale":"2023-12-19T15:54:41.815877+0000","last_undegraded":"2023-12-19T15:54:41.815877+0000","last_fullsized":"2023-12-19T15:54:41.815877+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:22:40.286462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812546+0000","last_change":"2023-12-19T15:54:41.812546+0000","last_active":"2023-12-19T15:54:41.812546+0000","last_peered":"2023-12-19T15:54:41.812546+0000","last_clean":"2023-12-19T15:54:41.812546+0000","last_became_active":"2023-12-19T15:54:41.812270+0000","last_became_peered":"2023-12-19T15:54:41.812270+0000","last_unstale":"2023-12-19T15:54:41.812546+0000","last_undegraded":"2023-12-19T15:54:41.812546+0000","last_fullsized":"2023-12-19T15:54:41.812546+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:12:20.404504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"166'4","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.839672+0000","last_change":"2023-12-19T15:54:40.810070+0000","last_active":"2023-12-19T15:54:41.839672+0000","last_peered":"2023-12-19T15:54:41.839672+0000","last_clean":"2023-12-19T15:54:41.839672+0000","last_became_active":"2023-12-19T15:54:40.809750+0000","last_became_peered":"2023-12-19T15:54:40.809750+0000","last_unstale":"2023-12-19T15:54:41.839672+0000","last_undegraded":"2023-12-19T15:54:41.839672+0000","last_fullsized":"2023-12-19T15:54:41.839672+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:17:11.623361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.817260+0000","last_change":"2023-12-19T15:54:41.817260+0000","last_active":"2023-12-19T15:54:41.817260+0000","last_peered":"2023-12-19T15:54:41.817260+0000","last_clean":"2023-12-19T15:54:41.817260+0000","last_became_active":"2023-12-19T15:54:41.814157+0000","last_became_peered":"2023-12-19T15:54:41.814157+0000","last_unstale":"2023-12-19T15:54:41.817260+0000","last_undegraded":"2023-12-19T15:54:41.817260+0000","last_fullsized":"2023-12-19T15:54:41.817260+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:20.089134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.798132+0000","last_change":"2023-12-19T15:54:40.807230+0000","last_active":"2023-12-19T15:54:41.798132+0000","last_peered":"2023-12-19T15:54:41.798132+0000","last_clean":"2023-12-19T15:54:41.798132+0000","last_became_active":"2023-12-19T15:54:40.806952+0000","last_became_peered":"2023-12-19T15:54:40.806952+0000","last_unstale":"2023-12-19T15:54:41.798132+0000","last_undegraded":"2023-12-19T15:54:41.798132+0000","last_fullsized":"2023-12-19T15:54:41.798132+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:17:11.623361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827656+0000","last_change":"2023-12-19T15:54:41.820811+0000","last_active":"2023-12-19T15:54:41.827656+0000","last_peered":"2023-12-19T15:54:41.827656+0000","last_clean":"2023-12-19T15:54:41.827656+0000","last_became_active":"2023-12-19T15:54:41.820109+0000","last_became_peered":"2023-12-19T15:54:41.820109+0000","last_unstale":"2023-12-19T15:54:41.827656+0000","last_undegraded":"2023-12-19T15:54:41.827656+0000","last_fullsized":"2023-12-19T15:54:41.827656+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:58.890637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837246+0000","last_change":"2023-12-19T15:54:40.814530+0000","last_active":"2023-12-19T15:54:41.837246+0000","last_peered":"2023-12-19T15:54:41.837246+0000","last_clean":"2023-12-19T15:54:41.837246+0000","last_became_active":"2023-12-19T15:54:40.814194+0000","last_became_peered":"2023-12-19T15:54:40.814194+0000","last_unstale":"2023-12-19T15:54:41.837246+0000","last_undegraded":"2023-12-19T15:54:41.837246+0000","last_fullsized":"2023-12-19T15:54:41.837246+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:33.271613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821700+0000","last_change":"2023-12-19T15:54:41.821700+0000","last_active":"2023-12-19T15:54:41.821700+0000","last_peered":"2023-12-19T15:54:41.821700+0000","last_clean":"2023-12-19T15:54:41.821700+0000","last_became_active":"2023-12-19T15:54:41.820923+0000","last_became_peered":"2023-12-19T15:54:41.820923+0000","last_unstale":"2023-12-19T15:54:41.821700+0000","last_undegraded":"2023-12-19T15:54:41.821700+0000","last_fullsized":"2023-12-19T15:54:41.821700+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:07.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837078+0000","last_change":"2023-12-19T15:54:40.810070+0000","last_active":"2023-12-19T15:54:41.837078+0000","last_peered":"2023-12-19T15:54:41.837078+0000","last_clean":"2023-12-19T15:54:41.837078+0000","last_became_active":"2023-12-19T15:54:40.809127+0000","last_became_peered":"2023-12-19T15:54:40.809127+0000","last_unstale":"2023-12-19T15:54:41.837078+0000","last_undegraded":"2023-12-19T15:54:41.837078+0000","last_fullsized":"2023-12-19T15:54:41.837078+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:39:01.267294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814317+0000","last_change":"2023-12-19T15:54:41.814317+0000","last_active":"2023-12-19T15:54:41.814317+0000","last_peered":"2023-12-19T15:54:41.814317+0000","last_clean":"2023-12-19T15:54:41.814317+0000","last_became_active":"2023-12-19T15:54:41.813929+0000","last_became_peered":"2023-12-19T15:54:41.813929+0000","last_unstale":"2023-12-19T15:54:41.814317+0000","last_undegraded":"2023-12-19T15:54:41.814317+0000","last_fullsized":"2023-12-19T15:54:41.814317+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:08:08.996205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.801995+0000","last_change":"2023-12-19T15:54:40.807968+0000","last_active":"2023-12-19T15:54:41.801995+0000","last_peered":"2023-12-19T15:54:41.801995+0000","last_clean":"2023-12-19T15:54:41.801995+0000","last_became_active":"2023-12-19T15:54:40.807640+0000","last_became_peered":"2023-12-19T15:54:40.807640+0000","last_unstale":"2023-12-19T15:54:41.801995+0000","last_undegraded":"2023-12-19T15:54:41.801995+0000","last_fullsized":"2023-12-19T15:54:41.801995+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:39.574849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814670+0000","last_change":"2023-12-19T15:54:41.814670+0000","last_active":"2023-12-19T15:54:41.814670+0000","last_peered":"2023-12-19T15:54:41.814670+0000","last_clean":"2023-12-19T15:54:41.814670+0000","last_became_active":"2023-12-19T15:54:41.814393+0000","last_became_peered":"2023-12-19T15:54:41.814393+0000","last_unstale":"2023-12-19T15:54:41.814670+0000","last_undegraded":"2023-12-19T15:54:41.814670+0000","last_fullsized":"2023-12-19T15:54:41.814670+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:44.281335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"166'1","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.793857+0000","last_change":"2023-12-19T15:54:40.803872+0000","last_active":"2023-12-19T15:54:41.793857+0000","last_peered":"2023-12-19T15:54:41.793857+0000","last_clean":"2023-12-19T15:54:41.793857+0000","last_became_active":"2023-12-19T15:54:40.803596+0000","last_became_peered":"2023-12-19T15:54:40.803596+0000","last_unstale":"2023-12-19T15:54:41.793857+0000","last_undegraded":"2023-12-19T15:54:41.793857+0000","last_fullsized":"2023-12-19T15:54:41.793857+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:41.314184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.822649+0000","last_change":"2023-12-19T15:54:41.822649+0000","last_active":"2023-12-19T15:54:41.822649+0000","last_peered":"2023-12-19T15:54:41.822649+0000","last_clean":"2023-12-19T15:54:41.822649+0000","last_became_active":"2023-12-19T15:54:41.822289+0000","last_became_peered":"2023-12-19T15:54:41.822289+0000","last_unstale":"2023-12-19T15:54:41.822649+0000","last_undegraded":"2023-12-19T15:54:41.822649+0000","last_fullsized":"2023-12-19T15:54:41.822649+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:10.296999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.808572+0000","last_change":"2023-12-19T15:54:40.815712+0000","last_active":"2023-12-19T15:54:41.808572+0000","last_peered":"2023-12-19T15:54:41.808572+0000","last_clean":"2023-12-19T15:54:41.808572+0000","last_became_active":"2023-12-19T15:54:40.814436+0000","last_became_peered":"2023-12-19T15:54:40.814436+0000","last_unstale":"2023-12-19T15:54:41.808572+0000","last_undegraded":"2023-12-19T15:54:41.808572+0000","last_fullsized":"2023-12-19T15:54:41.808572+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:06:09.231602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827452+0000","last_change":"2023-12-19T15:54:41.827452+0000","last_active":"2023-12-19T15:54:41.827452+0000","last_peered":"2023-12-19T15:54:41.827452+0000","last_clean":"2023-12-19T15:54:41.827452+0000","last_became_active":"2023-12-19T15:54:41.826712+0000","last_became_peered":"2023-12-19T15:54:41.826712+0000","last_unstale":"2023-12-19T15:54:41.827452+0000","last_undegraded":"2023-12-19T15:54:41.827452+0000","last_fullsized":"2023-12-19T15:54:41.827452+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:47.615053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.815241+0000","last_change":"2023-12-19T15:54:40.813957+0000","last_active":"2023-12-19T15:54:40.815241+0000","last_peered":"2023-12-19T15:54:40.815241+0000","last_clean":"2023-12-19T15:54:40.815241+0000","last_became_active":"2023-12-19T15:54:40.813441+0000","last_became_peered":"2023-12-19T15:54:40.813441+0000","last_unstale":"2023-12-19T15:54:40.815241+0000","last_undegraded":"2023-12-19T15:54:40.815241+0000","last_fullsized":"2023-12-19T15:54:40.815241+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:19.682900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821342+0000","last_change":"2023-12-19T15:54:41.821342+0000","last_active":"2023-12-19T15:54:41.821342+0000","last_peered":"2023-12-19T15:54:41.821342+0000","last_clean":"2023-12-19T15:54:41.821342+0000","last_became_active":"2023-12-19T15:54:41.820886+0000","last_became_peered":"2023-12-19T15:54:41.820886+0000","last_unstale":"2023-12-19T15:54:41.821342+0000","last_undegraded":"2023-12-19T15:54:41.821342+0000","last_fullsized":"2023-12-19T15:54:41.821342+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:01:13.880395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.801861+0000","last_change":"2023-12-19T15:54:40.812637+0000","last_active":"2023-12-19T15:54:41.801861+0000","last_peered":"2023-12-19T15:54:41.801861+0000","last_clean":"2023-12-19T15:54:41.801861+0000","last_became_active":"2023-12-19T15:54:40.812204+0000","last_became_peered":"2023-12-19T15:54:40.812204+0000","last_unstale":"2023-12-19T15:54:41.801861+0000","last_undegraded":"2023-12-19T15:54:41.801861+0000","last_fullsized":"2023-12-19T15:54:41.801861+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:19.515507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828573+0000","last_change":"2023-12-19T15:54:41.823134+0000","last_active":"2023-12-19T15:54:41.828573+0000","last_peered":"2023-12-19T15:54:41.828573+0000","last_clean":"2023-12-19T15:54:41.828573+0000","last_became_active":"2023-12-19T15:54:41.821787+0000","last_became_peered":"2023-12-19T15:54:41.821787+0000","last_unstale":"2023-12-19T15:54:41.828573+0000","last_undegraded":"2023-12-19T15:54:41.828573+0000","last_fullsized":"2023-12-19T15:54:41.828573+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:28.034708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.815560+0000","last_change":"2023-12-19T15:54:40.812249+0000","last_active":"2023-12-19T15:54:40.815560+0000","last_peered":"2023-12-19T15:54:40.815560+0000","last_clean":"2023-12-19T15:54:40.815560+0000","last_became_active":"2023-12-19T15:54:40.811732+0000","last_became_peered":"2023-12-19T15:54:40.811732+0000","last_unstale":"2023-12-19T15:54:40.815560+0000","last_undegraded":"2023-12-19T15:54:40.815560+0000","last_fullsized":"2023-12-19T15:54:40.815560+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:37:05.650864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821713+0000","last_change":"2023-12-19T15:54:41.821713+0000","last_active":"2023-12-19T15:54:41.821713+0000","last_peered":"2023-12-19T15:54:41.821713+0000","last_clean":"2023-12-19T15:54:41.821713+0000","last_became_active":"2023-12-19T15:54:41.813494+0000","last_became_peered":"2023-12-19T15:54:41.813494+0000","last_unstale":"2023-12-19T15:54:41.821713+0000","last_undegraded":"2023-12-19T15:54:41.821713+0000","last_fullsized":"2023-12-19T15:54:41.821713+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:10.296999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"166'3","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838197+0000","last_change":"2023-12-19T15:54:40.814994+0000","last_active":"2023-12-19T15:54:41.838197+0000","last_peered":"2023-12-19T15:54:41.838197+0000","last_clean":"2023-12-19T15:54:41.838197+0000","last_became_active":"2023-12-19T15:54:40.814677+0000","last_became_peered":"2023-12-19T15:54:40.814677+0000","last_unstale":"2023-12-19T15:54:41.838197+0000","last_undegraded":"2023-12-19T15:54:41.838197+0000","last_fullsized":"2023-12-19T15:54:41.838197+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:31.943227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.814078+0000","last_change":"2023-12-19T15:54:41.814078+0000","last_active":"2023-12-19T15:54:41.814078+0000","last_peered":"2023-12-19T15:54:41.814078+0000","last_clean":"2023-12-19T15:54:41.814078+0000","last_became_active":"2023-12-19T15:54:41.813778+0000","last_became_peered":"2023-12-19T15:54:41.813778+0000","last_unstale":"2023-12-19T15:54:41.814078+0000","last_undegraded":"2023-12-19T15:54:41.814078+0000","last_fullsized":"2023-12-19T15:54:41.814078+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:11:20.577265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"166'6","reported_seq":15,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.839450+0000","last_change":"2023-12-19T15:54:40.811657+0000","last_active":"2023-12-19T15:54:41.839450+0000","last_peered":"2023-12-19T15:54:41.839450+0000","last_clean":"2023-12-19T15:54:41.839450+0000","last_became_active":"2023-12-19T15:54:40.811266+0000","last_became_peered":"2023-12-19T15:54:40.811266+0000","last_unstale":"2023-12-19T15:54:41.839450+0000","last_undegraded":"2023-12-19T15:54:41.839450+0000","last_fullsized":"2023-12-19T15:54:41.839450+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:24.200897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812410+0000","last_change":"2023-12-19T15:54:41.812410+0000","last_active":"2023-12-19T15:54:41.812410+0000","last_peered":"2023-12-19T15:54:41.812410+0000","last_clean":"2023-12-19T15:54:41.812410+0000","last_became_active":"2023-12-19T15:54:41.812117+0000","last_became_peered":"2023-12-19T15:54:41.812117+0000","last_unstale":"2023-12-19T15:54:41.812410+0000","last_undegraded":"2023-12-19T15:54:41.812410+0000","last_fullsized":"2023-12-19T15:54:41.812410+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:21.094171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837224+0000","last_change":"2023-12-19T15:54:40.807999+0000","last_active":"2023-12-19T15:54:41.837224+0000","last_peered":"2023-12-19T15:54:41.837224+0000","last_clean":"2023-12-19T15:54:41.837224+0000","last_became_active":"2023-12-19T15:54:40.807720+0000","last_became_peered":"2023-12-19T15:54:40.807720+0000","last_unstale":"2023-12-19T15:54:41.837224+0000","last_undegraded":"2023-12-19T15:54:41.837224+0000","last_fullsized":"2023-12-19T15:54:41.837224+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:31:32.428001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826632+0000","last_change":"2023-12-19T15:54:41.826632+0000","last_active":"2023-12-19T15:54:41.826632+0000","last_peered":"2023-12-19T15:54:41.826632+0000","last_clean":"2023-12-19T15:54:41.826632+0000","last_became_active":"2023-12-19T15:54:41.826031+0000","last_became_peered":"2023-12-19T15:54:41.826031+0000","last_unstale":"2023-12-19T15:54:41.826632+0000","last_undegraded":"2023-12-19T15:54:41.826632+0000","last_fullsized":"2023-12-19T15:54:41.826632+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:53:27.548739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.805212+0000","last_change":"2023-12-19T15:54:40.803176+0000","last_active":"2023-12-19T15:54:40.805212+0000","last_peered":"2023-12-19T15:54:40.805212+0000","last_clean":"2023-12-19T15:54:40.805212+0000","last_became_active":"2023-12-19T15:54:40.802765+0000","last_became_peered":"2023-12-19T15:54:40.802765+0000","last_unstale":"2023-12-19T15:54:40.805212+0000","last_undegraded":"2023-12-19T15:54:40.805212+0000","last_fullsized":"2023-12-19T15:54:40.805212+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:46.568208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.817735+0000","last_change":"2023-12-19T15:54:41.817735+0000","last_active":"2023-12-19T15:54:41.817735+0000","last_peered":"2023-12-19T15:54:41.817735+0000","last_clean":"2023-12-19T15:54:41.817735+0000","last_became_active":"2023-12-19T15:54:41.817421+0000","last_became_peered":"2023-12-19T15:54:41.817421+0000","last_unstale":"2023-12-19T15:54:41.817735+0000","last_undegraded":"2023-12-19T15:54:41.817735+0000","last_fullsized":"2023-12-19T15:54:41.817735+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:26:40.591338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"166'8","reported_seq":18,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.842677+0000","last_change":"2023-12-19T15:54:40.811396+0000","last_active":"2023-12-19T15:54:41.842677+0000","last_peered":"2023-12-19T15:54:41.842677+0000","last_clean":"2023-12-19T15:54:41.842677+0000","last_became_active":"2023-12-19T15:54:40.811037+0000","last_became_peered":"2023-12-19T15:54:40.811037+0000","last_unstale":"2023-12-19T15:54:41.842677+0000","last_undegraded":"2023-12-19T15:54:41.842677+0000","last_fullsized":"2023-12-19T15:54:41.842677+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:52:31.224808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.812678+0000","last_change":"2023-12-19T15:54:41.812678+0000","last_active":"2023-12-19T15:54:41.812678+0000","last_peered":"2023-12-19T15:54:41.812678+0000","last_clean":"2023-12-19T15:54:41.812678+0000","last_became_active":"2023-12-19T15:54:41.812389+0000","last_became_peered":"2023-12-19T15:54:41.812389+0000","last_unstale":"2023-12-19T15:54:41.812678+0000","last_undegraded":"2023-12-19T15:54:41.812678+0000","last_fullsized":"2023-12-19T15:54:41.812678+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:03.353814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.835492+0000","last_change":"2023-12-19T15:54:40.810416+0000","last_active":"2023-12-19T15:54:41.835492+0000","last_peered":"2023-12-19T15:54:41.835492+0000","last_clean":"2023-12-19T15:54:41.835492+0000","last_became_active":"2023-12-19T15:54:40.810133+0000","last_became_peered":"2023-12-19T15:54:40.810133+0000","last_unstale":"2023-12-19T15:54:41.835492+0000","last_undegraded":"2023-12-19T15:54:41.835492+0000","last_fullsized":"2023-12-19T15:54:41.835492+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:38:18.568218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.826128+0000","last_change":"2023-12-19T15:54:41.826128+0000","last_active":"2023-12-19T15:54:41.826128+0000","last_peered":"2023-12-19T15:54:41.826128+0000","last_clean":"2023-12-19T15:54:41.826128+0000","last_became_active":"2023-12-19T15:54:41.825759+0000","last_became_peered":"2023-12-19T15:54:41.825759+0000","last_unstale":"2023-12-19T15:54:41.826128+0000","last_undegraded":"2023-12-19T15:54:41.826128+0000","last_fullsized":"2023-12-19T15:54:41.826128+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:23:45.191382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838195+0000","last_change":"2023-12-19T15:54:40.807310+0000","last_active":"2023-12-19T15:54:41.838195+0000","last_peered":"2023-12-19T15:54:41.838195+0000","last_clean":"2023-12-19T15:54:41.838195+0000","last_became_active":"2023-12-19T15:54:40.806900+0000","last_became_peered":"2023-12-19T15:54:40.806900+0000","last_unstale":"2023-12-19T15:54:41.838195+0000","last_undegraded":"2023-12-19T15:54:41.838195+0000","last_fullsized":"2023-12-19T15:54:41.838195+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:41.314184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827101+0000","last_change":"2023-12-19T15:54:41.827101+0000","last_active":"2023-12-19T15:54:41.827101+0000","last_peered":"2023-12-19T15:54:41.827101+0000","last_clean":"2023-12-19T15:54:41.827101+0000","last_became_active":"2023-12-19T15:54:41.826809+0000","last_became_peered":"2023-12-19T15:54:41.826809+0000","last_unstale":"2023-12-19T15:54:41.827101+0000","last_undegraded":"2023-12-19T15:54:41.827101+0000","last_fullsized":"2023-12-19T15:54:41.827101+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:17:21.969475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"166'1","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838842+0000","last_change":"2023-12-19T15:54:40.815039+0000","last_active":"2023-12-19T15:54:41.838842+0000","last_peered":"2023-12-19T15:54:41.838842+0000","last_clean":"2023-12-19T15:54:41.838842+0000","last_became_active":"2023-12-19T15:54:40.814786+0000","last_became_peered":"2023-12-19T15:54:40.814786+0000","last_unstale":"2023-12-19T15:54:41.838842+0000","last_undegraded":"2023-12-19T15:54:41.838842+0000","last_fullsized":"2023-12-19T15:54:41.838842+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:44:42.555899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.818244+0000","last_change":"2023-12-19T15:54:41.818244+0000","last_active":"2023-12-19T15:54:41.818244+0000","last_peered":"2023-12-19T15:54:41.818244+0000","last_clean":"2023-12-19T15:54:41.818244+0000","last_became_active":"2023-12-19T15:54:41.817105+0000","last_became_peered":"2023-12-19T15:54:41.817105+0000","last_unstale":"2023-12-19T15:54:41.818244+0000","last_undegraded":"2023-12-19T15:54:41.818244+0000","last_fullsized":"2023-12-19T15:54:41.818244+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:19.253460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.797257+0000","last_change":"2023-12-19T15:54:40.796341+0000","last_active":"2023-12-19T15:54:40.797257+0000","last_peered":"2023-12-19T15:54:40.797257+0000","last_clean":"2023-12-19T15:54:40.797257+0000","last_became_active":"2023-12-19T15:54:40.796024+0000","last_became_peered":"2023-12-19T15:54:40.796024+0000","last_unstale":"2023-12-19T15:54:40.797257+0000","last_undegraded":"2023-12-19T15:54:40.797257+0000","last_fullsized":"2023-12-19T15:54:40.797257+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:57:19.682900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.804896+0000","last_change":"2023-12-19T15:54:41.804896+0000","last_active":"2023-12-19T15:54:41.804896+0000","last_peered":"2023-12-19T15:54:41.804896+0000","last_clean":"2023-12-19T15:54:41.804896+0000","last_became_active":"2023-12-19T15:54:41.804604+0000","last_became_peered":"2023-12-19T15:54:41.804604+0000","last_unstale":"2023-12-19T15:54:41.804896+0000","last_undegraded":"2023-12-19T15:54:41.804896+0000","last_fullsized":"2023-12-19T15:54:41.804896+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:29.211645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.837710+0000","last_change":"2023-12-19T15:54:40.806261+0000","last_active":"2023-12-19T15:54:41.837710+0000","last_peered":"2023-12-19T15:54:41.837710+0000","last_clean":"2023-12-19T15:54:41.837710+0000","last_became_active":"2023-12-19T15:54:40.805959+0000","last_became_peered":"2023-12-19T15:54:40.805959+0000","last_unstale":"2023-12-19T15:54:41.837710+0000","last_undegraded":"2023-12-19T15:54:41.837710+0000","last_fullsized":"2023-12-19T15:54:41.837710+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:48.099274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821581+0000","last_change":"2023-12-19T15:54:41.821581+0000","last_active":"2023-12-19T15:54:41.821581+0000","last_peered":"2023-12-19T15:54:41.821581+0000","last_clean":"2023-12-19T15:54:41.821581+0000","last_became_active":"2023-12-19T15:54:41.821253+0000","last_became_peered":"2023-12-19T15:54:41.821253+0000","last_unstale":"2023-12-19T15:54:41.821581+0000","last_undegraded":"2023-12-19T15:54:41.821581+0000","last_fullsized":"2023-12-19T15:54:41.821581+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:30:51.364688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.811170+0000","last_change":"2023-12-19T15:54:40.805684+0000","last_active":"2023-12-19T15:54:40.811170+0000","last_peered":"2023-12-19T15:54:40.811170+0000","last_clean":"2023-12-19T15:54:40.811170+0000","last_became_active":"2023-12-19T15:54:40.805403+0000","last_became_peered":"2023-12-19T15:54:40.805403+0000","last_unstale":"2023-12-19T15:54:40.811170+0000","last_undegraded":"2023-12-19T15:54:40.811170+0000","last_fullsized":"2023-12-19T15:54:40.811170+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:02.747028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.815787+0000","last_change":"2023-12-19T15:54:41.815787+0000","last_active":"2023-12-19T15:54:41.815787+0000","last_peered":"2023-12-19T15:54:41.815787+0000","last_clean":"2023-12-19T15:54:41.815787+0000","last_became_active":"2023-12-19T15:54:41.814085+0000","last_became_peered":"2023-12-19T15:54:41.814085+0000","last_unstale":"2023-12-19T15:54:41.815787+0000","last_undegraded":"2023-12-19T15:54:41.815787+0000","last_fullsized":"2023-12-19T15:54:41.815787+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.814610+0000","last_change":"2023-12-19T15:54:40.809043+0000","last_active":"2023-12-19T15:54:40.814610+0000","last_peered":"2023-12-19T15:54:40.814610+0000","last_clean":"2023-12-19T15:54:40.814610+0000","last_became_active":"2023-12-19T15:54:40.808698+0000","last_became_peered":"2023-12-19T15:54:40.808698+0000","last_unstale":"2023-12-19T15:54:40.814610+0000","last_undegraded":"2023-12-19T15:54:40.814610+0000","last_fullsized":"2023-12-19T15:54:40.814610+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:18.983853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827822+0000","last_change":"2023-12-19T15:54:41.821030+0000","last_active":"2023-12-19T15:54:41.827822+0000","last_peered":"2023-12-19T15:54:41.827822+0000","last_clean":"2023-12-19T15:54:41.827822+0000","last_became_active":"2023-12-19T15:54:41.819703+0000","last_became_peered":"2023-12-19T15:54:41.819703+0000","last_unstale":"2023-12-19T15:54:41.827822+0000","last_undegraded":"2023-12-19T15:54:41.827822+0000","last_fullsized":"2023-12-19T15:54:41.827822+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:37:10.887215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.806770+0000","last_change":"2023-12-19T15:54:40.812154+0000","last_active":"2023-12-19T15:54:41.806770+0000","last_peered":"2023-12-19T15:54:41.806770+0000","last_clean":"2023-12-19T15:54:41.806770+0000","last_became_active":"2023-12-19T15:54:40.811878+0000","last_became_peered":"2023-12-19T15:54:40.811878+0000","last_unstale":"2023-12-19T15:54:41.806770+0000","last_undegraded":"2023-12-19T15:54:41.806770+0000","last_fullsized":"2023-12-19T15:54:41.806770+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:00:11.287374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.822811+0000","last_change":"2023-12-19T15:54:41.822811+0000","last_active":"2023-12-19T15:54:41.822811+0000","last_peered":"2023-12-19T15:54:41.822811+0000","last_clean":"2023-12-19T15:54:41.822811+0000","last_became_active":"2023-12-19T15:54:41.822519+0000","last_became_peered":"2023-12-19T15:54:41.822519+0000","last_unstale":"2023-12-19T15:54:41.822811+0000","last_undegraded":"2023-12-19T15:54:41.822811+0000","last_fullsized":"2023-12-19T15:54:41.822811+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:03:34.288102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"166'2","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.838473+0000","last_change":"2023-12-19T15:54:40.808963+0000","last_active":"2023-12-19T15:54:41.838473+0000","last_peered":"2023-12-19T15:54:41.838473+0000","last_clean":"2023-12-19T15:54:41.838473+0000","last_became_active":"2023-12-19T15:54:40.808676+0000","last_became_peered":"2023-12-19T15:54:40.808676+0000","last_unstale":"2023-12-19T15:54:41.838473+0000","last_undegraded":"2023-12-19T15:54:41.838473+0000","last_fullsized":"2023-12-19T15:54:41.838473+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:58:29.350111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.808833+0000","last_change":"2023-12-19T15:54:41.808833+0000","last_active":"2023-12-19T15:54:41.808833+0000","last_peered":"2023-12-19T15:54:41.808833+0000","last_clean":"2023-12-19T15:54:41.808833+0000","last_became_active":"2023-12-19T15:54:41.808523+0000","last_became_peered":"2023-12-19T15:54:41.808523+0000","last_unstale":"2023-12-19T15:54:41.808833+0000","last_undegraded":"2023-12-19T15:54:41.808833+0000","last_fullsized":"2023-12-19T15:54:41.808833+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:49.115763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.808925+0000","last_change":"2023-12-19T15:54:40.805840+0000","last_active":"2023-12-19T15:54:40.808925+0000","last_peered":"2023-12-19T15:54:40.808925+0000","last_clean":"2023-12-19T15:54:40.808925+0000","last_became_active":"2023-12-19T15:54:40.805482+0000","last_became_peered":"2023-12-19T15:54:40.805482+0000","last_unstale":"2023-12-19T15:54:40.808925+0000","last_undegraded":"2023-12-19T15:54:40.808925+0000","last_fullsized":"2023-12-19T15:54:40.808925+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:02.337325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.830548+0000","last_change":"2023-12-19T15:54:41.830548+0000","last_active":"2023-12-19T15:54:41.830548+0000","last_peered":"2023-12-19T15:54:41.830548+0000","last_clean":"2023-12-19T15:54:41.830548+0000","last_became_active":"2023-12-19T15:54:41.830218+0000","last_became_peered":"2023-12-19T15:54:41.830218+0000","last_unstale":"2023-12-19T15:54:41.830548+0000","last_undegraded":"2023-12-19T15:54:41.830548+0000","last_fullsized":"2023-12-19T15:54:41.830548+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:12:20.404504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.812512+0000","last_change":"2023-12-19T15:54:40.808234+0000","last_active":"2023-12-19T15:54:40.812512+0000","last_peered":"2023-12-19T15:54:40.812512+0000","last_clean":"2023-12-19T15:54:40.812512+0000","last_became_active":"2023-12-19T15:54:40.807877+0000","last_became_peered":"2023-12-19T15:54:40.807877+0000","last_unstale":"2023-12-19T15:54:40.812512+0000","last_undegraded":"2023-12-19T15:54:40.812512+0000","last_fullsized":"2023-12-19T15:54:40.812512+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:59:09.280510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827771+0000","last_change":"2023-12-19T15:54:41.827771+0000","last_active":"2023-12-19T15:54:41.827771+0000","last_peered":"2023-12-19T15:54:41.827771+0000","last_clean":"2023-12-19T15:54:41.827771+0000","last_became_active":"2023-12-19T15:54:41.827459+0000","last_became_peered":"2023-12-19T15:54:41.827459+0000","last_unstale":"2023-12-19T15:54:41.827771+0000","last_undegraded":"2023-12-19T15:54:41.827771+0000","last_fullsized":"2023-12-19T15:54:41.827771+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:05:43.849237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.806792+0000","last_change":"2023-12-19T15:54:40.811804+0000","last_active":"2023-12-19T15:54:41.806792+0000","last_peered":"2023-12-19T15:54:41.806792+0000","last_clean":"2023-12-19T15:54:41.806792+0000","last_became_active":"2023-12-19T15:54:40.810575+0000","last_became_peered":"2023-12-19T15:54:40.810575+0000","last_unstale":"2023-12-19T15:54:41.806792+0000","last_undegraded":"2023-12-19T15:54:41.806792+0000","last_fullsized":"2023-12-19T15:54:41.806792+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:39:19.735141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.821586+0000","last_change":"2023-12-19T15:54:41.810029+0000","last_active":"2023-12-19T15:54:41.821586+0000","last_peered":"2023-12-19T15:54:41.821586+0000","last_clean":"2023-12-19T15:54:41.821586+0000","last_became_active":"2023-12-19T15:54:41.809735+0000","last_became_peered":"2023-12-19T15:54:41.809735+0000","last_unstale":"2023-12-19T15:54:41.821586+0000","last_undegraded":"2023-12-19T15:54:41.821586+0000","last_fullsized":"2023-12-19T15:54:41.821586+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:47:57.492513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.800929+0000","last_change":"2023-12-19T15:54:40.810846+0000","last_active":"2023-12-19T15:54:41.800929+0000","last_peered":"2023-12-19T15:54:41.800929+0000","last_clean":"2023-12-19T15:54:41.800929+0000","last_became_active":"2023-12-19T15:54:40.810564+0000","last_became_peered":"2023-12-19T15:54:40.810564+0000","last_unstale":"2023-12-19T15:54:41.800929+0000","last_undegraded":"2023-12-19T15:54:41.800929+0000","last_fullsized":"2023-12-19T15:54:41.800929+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:59.922908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.820764+0000","last_change":"2023-12-19T15:54:41.810861+0000","last_active":"2023-12-19T15:54:41.820764+0000","last_peered":"2023-12-19T15:54:41.820764+0000","last_clean":"2023-12-19T15:54:41.820764+0000","last_became_active":"2023-12-19T15:54:41.810542+0000","last_became_peered":"2023-12-19T15:54:41.810542+0000","last_unstale":"2023-12-19T15:54:41.820764+0000","last_undegraded":"2023-12-19T15:54:41.820764+0000","last_fullsized":"2023-12-19T15:54:41.820764+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:10:41.554966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.814739+0000","last_change":"2023-12-19T15:54:40.810807+0000","last_active":"2023-12-19T15:54:40.814739+0000","last_peered":"2023-12-19T15:54:40.814739+0000","last_clean":"2023-12-19T15:54:40.814739+0000","last_became_active":"2023-12-19T15:54:40.810546+0000","last_became_peered":"2023-12-19T15:54:40.810546+0000","last_unstale":"2023-12-19T15:54:40.814739+0000","last_undegraded":"2023-12-19T15:54:40.814739+0000","last_fullsized":"2023-12-19T15:54:40.814739+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:07:24.779124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.828650+0000","last_change":"2023-12-19T15:54:41.824487+0000","last_active":"2023-12-19T15:54:41.828650+0000","last_peered":"2023-12-19T15:54:41.828650+0000","last_clean":"2023-12-19T15:54:41.828650+0000","last_became_active":"2023-12-19T15:54:41.824189+0000","last_became_peered":"2023-12-19T15:54:41.824189+0000","last_unstale":"2023-12-19T15:54:41.828650+0000","last_undegraded":"2023-12-19T15:54:41.828650+0000","last_fullsized":"2023-12-19T15:54:41.828650+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.809009+0000","last_change":"2023-12-19T15:54:40.807632+0000","last_active":"2023-12-19T15:54:41.809009+0000","last_peered":"2023-12-19T15:54:41.809009+0000","last_clean":"2023-12-19T15:54:41.809009+0000","last_became_active":"2023-12-19T15:54:40.807318+0000","last_became_peered":"2023-12-19T15:54:40.807318+0000","last_unstale":"2023-12-19T15:54:41.809009+0000","last_undegraded":"2023-12-19T15:54:41.809009+0000","last_fullsized":"2023-12-19T15:54:41.809009+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:43.089842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.827658+0000","last_change":"2023-12-19T15:54:41.827658+0000","last_active":"2023-12-19T15:54:41.827658+0000","last_peered":"2023-12-19T15:54:41.827658+0000","last_clean":"2023-12-19T15:54:41.827658+0000","last_became_active":"2023-12-19T15:54:41.827113+0000","last_became_peered":"2023-12-19T15:54:41.827113+0000","last_unstale":"2023-12-19T15:54:41.827658+0000","last_undegraded":"2023-12-19T15:54:41.827658+0000","last_fullsized":"2023-12-19T15:54:41.827658+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:54:40.796038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.813624+0000","last_change":"2023-12-19T15:54:40.806660+0000","last_active":"2023-12-19T15:54:40.813624+0000","last_peered":"2023-12-19T15:54:40.813624+0000","last_clean":"2023-12-19T15:54:40.813624+0000","last_became_active":"2023-12-19T15:54:40.805788+0000","last_became_peered":"2023-12-19T15:54:40.805788+0000","last_unstale":"2023-12-19T15:54:40.813624+0000","last_undegraded":"2023-12-19T15:54:40.813624+0000","last_fullsized":"2023-12-19T15:54:40.813624+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:50:46.568208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":8,"reported_epoch":166,"state":"active+clean","last_fresh":"2023-12-19T15:54:41.811977+0000","last_change":"2023-12-19T15:54:41.811977+0000","last_active":"2023-12-19T15:54:41.811977+0000","last_peered":"2023-12-19T15:54:41.811977+0000","last_clean":"2023-12-19T15:54:41.811977+0000","last_became_active":"2023-12-19T15:54:41.811504+0000","last_became_peered":"2023-12-19T15:54:41.811504+0000","last_unstale":"2023-12-19T15:54:41.811977+0000","last_undegraded":"2023-12-19T15:54:41.811977+0000","last_fullsized":"2023-12-19T15:54:41.811977+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:40.765703+0000","last_clean_scrub_stamp":"2023-12-19T15:54:40.765703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:13:25.217386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.816475+0000","last_change":"2023-12-19T15:54:40.813601+0000","last_active":"2023-12-19T15:54:40.816475+0000","last_peered":"2023-12-19T15:54:40.816475+0000","last_clean":"2023-12-19T15:54:40.816475+0000","last_became_active":"2023-12-19T15:54:40.813282+0000","last_became_peered":"2023-12-19T15:54:40.813282+0000","last_unstale":"2023-12-19T15:54:40.816475+0000","last_undegraded":"2023-12-19T15:54:40.816475+0000","last_fullsized":"2023-12-19T15:54:40.816475+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:54:39.749214+0000","last_clean_scrub_stamp":"2023-12-19T15:54:39.749214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:28.195156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"155'271","reported_seq":829,"reported_epoch":165,"state":"active+clean","last_fresh":"2023-12-19T15:54:40.799674+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T15:54:40.799674+0000","last_peered":"2023-12-19T15:54:40.799674+0000","last_clean":"2023-12-19T15:54:40.799674+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T15:54:40.799674+0000","last_undegraded":"2023-12-19T15:54:40.799674+0000","last_fullsized":"2023-12-19T15:54:40.799674+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299904,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071936,"statfs":{"total":96636764160,"available":96329662464,"internally_reserved":0,"allocated":2678784,"data_stored":1642319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1379999999999999,"15min":1.135},"min":{"1min":0.68500000000000005,"5min":0.64800000000000002,"15min":0.54200000000000004},"max":{"1min":1.6160000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1779999999999999,"15min":1.1579999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.80900000000000005}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.149,"15min":1.1519999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.4330000000000001,"5min":1.7010000000000001,"15min":1.778},"last":1.202},{"interface":"front","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.542,"5min":1.698,"15min":1.698},"last":1.345}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3029999999999999,"15min":1.294},"min":{"1min":1.0369999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.669,"5min":1.6890000000000001,"15min":1.7969999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.345,"15min":1.3360000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.79300000000000004},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7889999999999999},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.875,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5960000000000001,"5min":1.6839999999999999,"15min":1.7130000000000001},"last":1.153},{"interface":"front","average":{"1min":1.276,"5min":1.355,"15min":1.367},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.5369999999999999,"5min":1.7270000000000001,"15min":1.7290000000000001},"last":1.2869999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.163,"15min":1.181},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.63200000000000001},"max":{"1min":1.5820000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.46},{"interface":"front","average":{"1min":1.212,"5min":1.2629999999999999,"15min":1.2609999999999999},"min":{"1min":0.92800000000000005,"5min":0.82099999999999995,"15min":0.63500000000000001},"max":{"1min":1.6180000000000001,"5min":1.8009999999999999,"15min":1.8899999999999999},"last":1.27}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2390000000000001,"15min":1.264},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60499999999999998},"max":{"1min":1.6519999999999999,"5min":1.758,"15min":2.0880000000000001},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.325,"15min":1.325},"min":{"1min":0.92900000000000005,"5min":0.79100000000000004,"15min":0.69999999999999996},"max":{"1min":1.7090000000000001,"5min":1.798,"15min":1.798},"last":1.5860000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4450000000000001,"15min":1.423},"min":{"1min":1.1910000000000001,"5min":0.82399999999999995,"15min":0.76000000000000001},"max":{"1min":1.778,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.70999999999999996},"max":{"1min":1.911,"5min":1.911,"15min":2.0110000000000001},"last":1.224}]}]},{"osd":6,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311604,"kb_used_data":2028,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060236,"statfs":{"total":96636764160,"available":96317681664,"internally_reserved":0,"allocated":2076672,"data_stored":1050969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.92900000000000005,"5min":0.79700000000000004,"15min":0.72799999999999998},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.081},{"interface":"front","average":{"1min":1.22,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.99399999999999999,"5min":0.78800000000000003,"15min":0.75900000000000001},"max":{"1min":1.4490000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.6970000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1910000000000001,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.52,"5min":1.617,"15min":1.6890000000000001},"last":1.272},{"interface":"front","average":{"1min":1.228,"5min":1.2549999999999999,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.5269999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.1930000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.2490000000000001,"15min":1.242},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.83699999999999997},"max":{"1min":1.611,"5min":1.629,"15min":1.6990000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.276,"15min":1.268},"min":{"1min":0.96399999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.7130000000000001},"last":1.389}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.647,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":1.917},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3600000000000001,"15min":1.3440000000000001},"min":{"1min":0.98699999999999999,"5min":0.96299999999999997,"15min":0.93300000000000005},"max":{"1min":1.6220000000000001,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.671}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2230000000000001,"15min":1.2030000000000001},"min":{"1min":0.86499999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":0.89000000000000001,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":1.8049999999999999,"5min":2.653,"15min":2.653},"last":1.9910000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.3},"min":{"1min":0.84899999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.774},{"interface":"front","average":{"1min":1.258,"5min":1.3320000000000001,"15min":1.361},"min":{"1min":0.878,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.762,"5min":1.871,"15min":1.99},"last":1.6160000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4039999999999999,"15min":1.4299999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.742,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":2.081},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":1.4490000000000001},"min":{"1min":0.97399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.905,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":1.8700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575753,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307504,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064336,"statfs":{"total":96636764160,"available":96321880064,"internally_reserved":0,"allocated":2072576,"data_stored":1050347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":2.2759999999999998},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6679999999999999,"5min":1.7909999999999999,"15min":2.7309999999999999},"last":1.4770000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.339,"15min":1.3220000000000001},"min":{"1min":1,"5min":0.86599999999999999,"15min":0.84499999999999997},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7370000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.282,"15min":1.2689999999999999},"min":{"1min":1.038,"5min":0.95699999999999996,"15min":0.82599999999999996},"max":{"1min":1.617,"5min":1.645,"15min":1.8779999999999999},"last":1.327}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6699999999999999,"5min":1.748,"15min":2.8239999999999998},"last":1.427},{"interface":"front","average":{"1min":1.234,"5min":1.2350000000000001,"15min":1.2230000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71199999999999997},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":1.871},"last":1.345}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.306},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.631,"5min":1.7350000000000001,"15min":2.4849999999999999},"last":1.359},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.2969999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6739999999999999,"5min":1.7929999999999999,"15min":1.9390000000000001},"last":1.2889999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1359999999999999,"15min":1.137},"min":{"1min":0.84299999999999997,"5min":0.69499999999999995,"15min":0.59999999999999998},"max":{"1min":1.659,"5min":1.659,"15min":2.109},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.244,"15min":1.27},"min":{"1min":0.86799999999999999,"5min":0.81799999999999995,"15min":0.56999999999999995},"max":{"1min":1.7869999999999999,"5min":1.8320000000000001,"15min":2.8999999999999999},"last":1.2030000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.333,"15min":1.3380000000000001},"min":{"1min":0.92100000000000004,"5min":0.84399999999999997,"15min":0.58899999999999997},"max":{"1min":1.8089999999999999,"5min":1.9299999999999999,"15min":2.0649999999999999},"last":1.446},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":1.1259999999999999,"5min":0.92500000000000004,"15min":0.83699999999999997},"max":{"1min":1.998,"5min":1.998,"15min":2.117},"last":1.141}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4259999999999999,"15min":1.4179999999999999},"min":{"1min":1.2310000000000001,"5min":0.95499999999999996,"15min":0.623},"max":{"1min":1.915,"5min":1.915,"15min":2.6000000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.419,"15min":1.411},"min":{"1min":1.0389999999999999,"5min":0.84599999999999997,"15min":0.68000000000000005},"max":{"1min":1.879,"5min":1.9870000000000001,"15min":2.3119999999999998},"last":1.393}]}]},{"osd":4,"up_from":13,"seq":55834575753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307500,"kb_used_data":2020,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064340,"statfs":{"total":96636764160,"available":96321884160,"internally_reserved":0,"allocated":2068480,"data_stored":1050325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1299999999999999,"15min":1.1100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.722},"last":1.014},{"interface":"front","average":{"1min":1.135,"5min":1.113,"15min":1.1399999999999999},"min":{"1min":0.76800000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.4770000000000001,"5min":1.5249999999999999,"15min":1.75},"last":1.1000000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.274,"15min":1.286},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6020000000000001,"5min":1.6819999999999999,"15min":2.2530000000000001},"last":1.133},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.69399999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.968},"last":1.349}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1850000000000001,"15min":1.1859999999999999},"min":{"1min":0.754,"5min":0.73599999999999999,"15min":0.65300000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":2.3410000000000002},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.84399999999999997,"5min":0.79900000000000004,"15min":0.57799999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1459999999999999},"last":1.2569999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.236,"15min":1.23},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.772},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.26,"15min":1.2869999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.65500000000000003},"max":{"1min":1.758,"5min":1.758,"15min":2.0800000000000001},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.204,"15min":1.1839999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.42399999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.151},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.6240000000000001,"5min":1.6579999999999999,"15min":1.998},"last":0.82099999999999995}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3140000000000001,"15min":1.306},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.60399999999999998},"max":{"1min":1.7190000000000001,"5min":1.766,"15min":2.3340000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.458,"15min":1.454},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.68100000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.429}]},{"osd":7,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3859999999999999,"15min":1.3600000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8260000000000001,"5min":1.839,"15min":2.0150000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.4890000000000001,"15min":1.47},"min":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.55600000000000005},"max":{"1min":1.8660000000000001,"5min":2.157,"15min":2.1659999999999999},"last":1.1970000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575752,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311612,"kb_used_data":2036,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060228,"statfs":{"total":96636764160,"available":96317673472,"internally_reserved":0,"allocated":2084864,"data_stored":1051931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91100000000000003},"min":{"1min":0.65600000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.28,"5min":1.5509999999999999,"15min":1.897},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0289999999999999,"15min":1.03},"min":{"1min":0.68500000000000005,"5min":0.63200000000000001,"15min":0.61099999999999999},"max":{"1min":1.375,"5min":1.4179999999999999,"15min":2.113},"last":1.117}]},{"osd":1,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98599999999999999,"15min":0.97299999999999998},"min":{"1min":0.78900000000000003,"5min":0.60999999999999999,"15min":0.57699999999999996},"max":{"1min":1.345,"5min":1.75,"15min":1.75},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1619999999999999,"15min":1.127},"min":{"1min":0.89000000000000001,"5min":0.73999999999999999,"15min":0.53900000000000003},"max":{"1min":1.6759999999999999,"5min":1.706,"15min":1.8149999999999999},"last":1.254}]},{"osd":2,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.175,"15min":1.173},"min":{"1min":0.93000000000000005,"5min":0.77900000000000003,"15min":0.74099999999999999},"max":{"1min":1.48,"5min":1.611,"15min":1.992},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2949999999999999,"15min":1.268},"min":{"1min":0.91600000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.593,"5min":1.794,"15min":1.794},"last":1.349}]},{"osd":4,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3859999999999999,"15min":1.3520000000000001},"min":{"1min":1.056,"5min":0.94099999999999995,"15min":0.879},"max":{"1min":1.7130000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3560000000000001,"15min":1.325},"min":{"1min":0.93400000000000005,"5min":0.91500000000000004,"15min":0.71699999999999997},"max":{"1min":1.5680000000000001,"5min":1.8879999999999999,"15min":2.1120000000000001},"last":1.444}]},{"osd":5,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.373,"15min":1.347},"min":{"1min":1.024,"5min":1.0089999999999999,"15min":0.79200000000000004},"max":{"1min":1.5509999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.458,"15min":1.431},"min":{"1min":1.181,"5min":0.84099999999999997,"15min":0.81000000000000005},"max":{"1min":2.0609999999999999,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.4059999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5149999999999999,"15min":1.488},"min":{"1min":1.1339999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1669999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.494,"15min":1.4670000000000001},"min":{"1min":1.1910000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.948,"5min":2.141,"15min":2.1909999999999998},"last":1.2110000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.589,"15min":1.5669999999999999},"min":{"1min":1.333,"5min":1.1339999999999999,"15min":1.0169999999999999},"max":{"1min":1.863,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.552,"15min":1.518},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":0.92900000000000005},"max":{"1min":1.7490000000000001,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.3740000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575754,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311600,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060240,"statfs":{"total":96636764160,"available":96317685760,"internally_reserved":0,"allocated":2072576,"data_stored":1049902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.96299999999999997,"15min":0.95799999999999996},"min":{"1min":0.67700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2609999999999999,"5min":1.587,"15min":1.587},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.163,"5min":1.1639999999999999,"15min":1.1459999999999999},"min":{"1min":0.95199999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5089999999999999,"5min":1.595,"15min":1.944},"last":0.86899999999999999}]},{"osd":1,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1779999999999999,"15min":1.181},"min":{"1min":0.751,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6870000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.105,"5min":1.1299999999999999,"15min":1.1479999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69099999999999995},"max":{"1min":1.353,"5min":1.6359999999999999,"15min":1.7430000000000001},"last":1.0800000000000001}]},{"osd":3,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0169999999999999,"15min":1.0289999999999999},"min":{"1min":0.75600000000000001,"5min":0.64900000000000002,"15min":0.64000000000000001},"max":{"1min":1.474,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.006},{"interface":"front","average":{"1min":1.133,"5min":1.1379999999999999,"15min":1.129},"min":{"1min":0.82199999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.5029999999999999,"5min":1.6739999999999999,"15min":1.77},"last":0.94299999999999995}]},{"osd":4,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.387,"15min":1.413},"min":{"1min":1.038,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":1.649,"5min":1.7490000000000001,"15min":2.0790000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4379999999999999,"15min":1.4570000000000001},"min":{"1min":1.048,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.921,"5min":1.921,"15min":2.3410000000000002},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":1.101,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":1.8680000000000001,"5min":1.9039999999999999,"15min":2.0659999999999998},"last":1.125},{"interface":"front","average":{"1min":1.409,"5min":1.389,"15min":1.4039999999999999},"min":{"1min":1.1180000000000001,"5min":0.94499999999999995,"15min":0.92700000000000005},"max":{"1min":1.996,"5min":1.996,"15min":2.016},"last":1.349}]},{"osd":6,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.393,"15min":1.395},"min":{"1min":1.1319999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"max":{"1min":1.782,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.173},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.518,"15min":1.5089999999999999},"min":{"1min":1.196,"5min":1.073,"15min":1.073},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.242},"last":1.2649999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:12 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5369999999999999,"15min":1.5549999999999999},"min":{"1min":1.0900000000000001,"5min":1.075,"15min":1.04},"max":{"1min":1.9379999999999999,"5min":1.9750000000000001,"15min":2.2389999999999999},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5169999999999999,"15min":1.518},"min":{"1min":1.1819999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.9830000000000001,"5min":2.016,"15min":2.048},"last":1.496}]}]},{"osd":1,"up_from":13,"seq":55834575752,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311600,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060240,"statfs":{"total":96636764160,"available":96317685760,"internally_reserved":0,"allocated":2072576,"data_stored":1050395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93000000000000005,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.628,"15min":0.57699999999999996},"max":{"1min":1.4890000000000001,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.006,"15min":1.032},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.60199999999999998},"max":{"1min":1.532,"5min":1.532,"15min":1.6879999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.044,"15min":1.0680000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.7989999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.121,"5min":1.175,"15min":1.1679999999999999},"min":{"1min":0.86499999999999999,"5min":0.79600000000000004,"15min":0.75},"max":{"1min":1.4750000000000001,"5min":1.6479999999999999,"15min":1.7549999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.278,"15min":1.2609999999999999},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.67000000000000004},"max":{"1min":1.784,"5min":1.7989999999999999,"15min":1.851},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1040000000000001,"15min":1.1020000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.1970000000000001,"5min":1.778,"15min":1.849},"last":0.89500000000000002}]},{"osd":4,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.353,"15min":1.3300000000000001},"min":{"1min":1.056,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.601,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.032},{"interface":"front","average":{"1min":1.359,"5min":1.409,"15min":1.4239999999999999},"min":{"1min":1.0860000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.6439999999999999,"5min":1.718,"15min":1.921},"last":1.1060000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.355,"15min":1.3420000000000001},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.93500000000000005},"max":{"1min":1.6279999999999999,"5min":1.7450000000000001,"15min":1.9199999999999999},"last":1.304},{"interface":"front","average":{"1min":1.413,"5min":1.431,"15min":1.427},"min":{"1min":1.0529999999999999,"5min":0.95299999999999996,"15min":0.94999999999999996},"max":{"1min":1.8540000000000001,"5min":2.1909999999999998,"15min":2.1909999999999998},"last":0.95199999999999996}]},{"osd":6,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3999999999999999,"15min":1.3939999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.995},"max":{"1min":1.6040000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.349},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.52,"15min":1.504},"min":{"1min":1.109,"5min":1.109,"15min":0.88800000000000001},"max":{"1min":1.867,"5min":2.0880000000000001,"15min":2.1040000000000001},"last":1.1659999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.504,"15min":1.52},"min":{"1min":1.23,"5min":1.1799999999999999,"15min":1.107},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4830000000000001,"15min":1.48},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":1.6919999999999999,"5min":2.0659999999999998,"15min":2.0699999999999998},"last":1.224}]}]},{"osd":0,"up_from":13,"seq":55834575752,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299892,"kb_used_data":2604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071948,"statfs":{"total":96636764160,"available":96329674752,"internally_reserved":0,"allocated":2666496,"data_stored":1639738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.66500000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.78000000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5649999999999999,"15min":1.895},"last":1.101}]},{"osd":2,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1259999999999999,"15min":1.1439999999999999},"min":{"1min":0.85599999999999998,"5min":0.502,"15min":0.47399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":2.5129999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.256,"15min":1.2789999999999999},"min":{"1min":0.93600000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.679,"5min":1.708,"15min":2.8250000000000002},"last":1.3009999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.038},"min":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.66200000000000003},"max":{"1min":1.5349999999999999,"5min":1.599,"15min":1.9970000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2070000000000001,"15min":1.2090000000000001},"min":{"1min":0.92200000000000004,"5min":0.621,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.794},"last":1.4359999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3919999999999999,"15min":1.351},"min":{"1min":1.0640000000000001,"5min":0.94499999999999995,"15min":0.80000000000000004},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.379},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.327,"15min":1.3169999999999999},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8280000000000001,"5min":1.9059999999999999,"15min":2.2210000000000001},"last":1.4790000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.419,"15min":1.4279999999999999},"min":{"1min":1.175,"5min":0.871,"15min":0.73099999999999998},"max":{"1min":1.879,"5min":1.901,"15min":2.4470000000000001},"last":1.754},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4670000000000001,"15min":1.4359999999999999},"min":{"1min":1.004,"5min":1.004,"15min":0.79400000000000004},"max":{"1min":1.889,"5min":1.9510000000000001,"15min":2.468},"last":1.5389999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.413,"15min":1.385},"min":{"1min":1.1830000000000001,"5min":0.90000000000000002,"15min":0.88500000000000001},"max":{"1min":1.925,"5min":1.925,"15min":2.0830000000000002},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.53,"15min":1.5},"min":{"1min":1.2110000000000001,"5min":0.92600000000000005,"15min":0.81799999999999995},"max":{"1min":1.9470000000000001,"5min":2.379,"15min":2.379},"last":1.8959999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5429999999999999,"15min":1.5049999999999999},"min":{"1min":1.0900000000000001,"5min":0.99299999999999999,"15min":0.90900000000000003},"max":{"1min":1.968,"5min":2.105,"15min":2.698},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.49,"5min":1.5089999999999999,"15min":1.482},"min":{"1min":1.022,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.8380000000000001,"5min":2.016,"15min":3.23},"last":1.948}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T15:54:46.578 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T15:54:46.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T15:54:46.758 INFO:tasks.ceph.mds.d.smithi125.stderr:2023-12-19T15:54:46.754+0000 7f99871f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T15:54:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.796+0000 7f85d7345700 1 Processor -- start 2023-12-19T15:54:46.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.796+0000 7f85d7345700 1 -- start start 2023-12-19T15:54:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85d0158e20 0x7f85d00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85d00cb970 0x7f85d00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d00ca2c0 0x7f85d006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00cbdf0 con 0x7f85d00cb970 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00cc0d0 con 0x7f85d0158e20 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d7345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00cc3b0 con 0x7f85d00ca2c0 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85d0158e20 0x7f85d00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d48e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f85d00cb970 0x7f85d00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85d0158e20 0x7f85d00ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42302/0 (socket says 172.21.15.50:42302) 2023-12-19T15:54:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 -- 172.21.15.50:0/2354828489 learned_addr learned my addr 172.21.15.50:0/2354828489 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d58e2700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d00ca2c0 0x7f85d006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 -- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d00ca2c0 msgr2=0x7f85d006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d00ca2c0 0x7f85d006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 -- 172.21.15.50:0/2354828489 >> v2:172.21.15.50:3300/0 conn(0x7f85d00cb970 msgr2=0x7f85d00ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 --2- 172.21.15.50:0/2354828489 >> v2:172.21.15.50:3300/0 conn(0x7f85d00cb970 0x7f85d00ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 -- 172.21.15.50:0/2354828489 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00cc690 con 0x7f85d0158e20 2023-12-19T15:54:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85d50e1700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d0158e20 0x7f85d00ce5c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0099b0 tx=0x7f85cc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e12c434e3d4017a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85c7fff700 1 -- 172.21.15.50:0/2354828489 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00b640 con 0x7f85d0158e20 2023-12-19T15:54:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85c7fff700 1 -- 172.21.15.50:0/2354828489 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc00b7a0 con 0x7f85d0158e20 2023-12-19T15:54:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.800+0000 7f85c7fff700 1 -- 172.21.15.50:0/2354828489 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00bb40 con 0x7f85d0158e20 2023-12-19T15:54:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 -- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d0158e20 msgr2=0x7f85d00ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d0158e20 0x7f85d00ce5c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0099b0 tx=0x7f85cc009250 comp rx=0 tx=0).stop 2023-12-19T15:54:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 -- 172.21.15.50:0/2354828489 shutdown_connections 2023-12-19T15:54:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 --2- 172.21.15.50:0/2354828489 >> v2:172.21.15.50:3300/0 conn(0x7f85d00cb970 0x7f85d00ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d00ca2c0 0x7f85d006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 --2- 172.21.15.50:0/2354828489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d0158e20 0x7f85d00ce5c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 -- 172.21.15.50:0/2354828489 >> 172.21.15.50:0/2354828489 conn(0x7f85d0067bb0 msgr2=0x7f85d00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 -- 172.21.15.50:0/2354828489 shutdown_connections 2023-12-19T15:54:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.804+0000 7f85d7345700 1 -- 172.21.15.50:0/2354828489 wait complete. 2023-12-19T15:54:46.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 Processor -- start 2023-12-19T15:54:46.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 -- start start 2023-12-19T15:54:46.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 0x7f85d016ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 0x7f85d00c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d016abd0 con 0x7f85d016ad80 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d016cfa0 con 0x7f85d00ca2c0 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d7345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c80e0 con 0x7f85d0158e20 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d58e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d58e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49146/0 (socket says 172.21.15.50:49146) 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d58e2700 1 -- 172.21.15.50:0/1837496537 learned_addr learned my addr 172.21.15.50:0/1837496537 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d50e1700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 0x7f85d016ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.808+0000 7f85d48e0700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 0x7f85d00c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 -- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 msgr2=0x7f85d00c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 0x7f85d00c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 -- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 msgr2=0x7f85d016ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 0x7f85d016ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 -- 172.21.15.50:0/1837496537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc0094f0 con 0x7f85d016ad80 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d48e0700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 0x7f85d00c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d50e1700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 0x7f85d016ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d58e2700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e90 tx=0x7f85c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800cb70 con 0x7f85d016ad80 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800f040 con 0x7f85d016ad80 2023-12-19T15:54:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8014480 con 0x7f85d016ad80 2023-12-19T15:54:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d00b5220 con 0x7f85d016ad80 2023-12-19T15:54:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d016a2a0 con 0x7f85d016ad80 2023-12-19T15:54:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.812+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d00c8570 con 0x7f85d016ad80 2023-12-19T15:54:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.816+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f85c8014830 con 0x7f85d016ad80 2023-12-19T15:54:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.816+0000 7f85c57fa700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 0x7f85b4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:46.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.816+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f85c807d3d0 con 0x7f85d016ad80 2023-12-19T15:54:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.816+0000 7f85d50e1700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 0x7f85b4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:46.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.820+0000 7f85d50e1700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 0x7f85b4062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0099b0 tx=0x7f85cc017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:46.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:46.824+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f85d00c8570 con 0x7f85d016ad80 2023-12-19T15:54:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.056+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f85d0071120 con 0x7f85d016ad80 2023-12-19T15:54:47.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.064+0000 7f85c57fa700 1 -- 172.21.15.50:0/1837496537 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v40) v1 ==== 190+0+0 (secure 0 0 0) 0x7f85c8051010 con 0x7f85d016ad80 2023-12-19T15:54:47.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 msgr2=0x7f85b4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 0x7f85b4062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0099b0 tx=0x7f85cc017000 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 msgr2=0x7f85d00c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e90 tx=0x7f85c800b040 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 shutdown_connections 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f85b4060670 0x7f85b4062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f85d016ad80 0x7f85d00c6100 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f85d0158e20 0x7f85d00c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 --2- 172.21.15.50:0/1837496537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85d00ca2c0 0x7f85d016ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 >> 172.21.15.50:0/1837496537 conn(0x7f85d0067bb0 msgr2=0x7f85d006ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 shutdown_connections 2023-12-19T15:54:47.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.068+0000 7f85d7345700 1 -- 172.21.15.50:0/1837496537 wait complete. 2023-12-19T15:54:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T15:54:47.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:54:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.276+0000 7f09864f0700 1 Processor -- start 2023-12-19T15:54:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.276+0000 7f09864f0700 1 -- start start 2023-12-19T15:54:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f09864f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f09864f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0980140ae0 0x7f09800680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f09864f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098006e150 con 0x7f09800c75e0 2023-12-19T15:54:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f09864f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098006d540 con 0x7f0980140ae0 2023-12-19T15:54:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f09864f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098006d820 con 0x7f0980140710 2023-12-19T15:54:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f097ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f097ffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55186/0 (socket says 172.21.15.50:55186) 2023-12-19T15:54:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0980140ae0 0x7f09800680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f097ffff700 1 -- 172.21.15.50:0/3369285525 learned_addr learned my addr 172.21.15.50:0/3369285525 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 -- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 msgr2=0x7f09800c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 -- 172.21.15.50:0/3369285525 >> v1:172.21.15.125:6789/0 conn(0x7f09800c75e0 legacy=0x7f098006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 -- 172.21.15.50:0/3369285525 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098006db00 con 0x7f0980140ae0 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f097f7fe700 1 -- 172.21.15.50:0/3369285525 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595786677 0 0) 0x7f098006e150 con 0x7f09800c75e0 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f097ffff700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:47.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.280+0000 7f0977fff700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980140ae0 0x7f09800680f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f09680098e0 tx=0x7f09680091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a2aebc3070e5274 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:47.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f097f7fe700 1 -- 172.21.15.50:0/3369285525 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800b640 con 0x7f0980140ae0 2023-12-19T15:54:47.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f097f7fe700 1 -- 172.21.15.50:0/3369285525 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096800b7a0 con 0x7f0980140ae0 2023-12-19T15:54:47.286 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f097f7fe700 1 -- 172.21.15.50:0/3369285525 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800bb40 con 0x7f0980140ae0 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 -- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980140ae0 msgr2=0x7f09800680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980140ae0 0x7f09800680f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f09680098e0 tx=0x7f09680091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 -- 172.21.15.50:0/3369285525 shutdown_connections 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980140ae0 0x7f09800680f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 --2- 172.21.15.50:0/3369285525 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 -- 172.21.15.50:0/3369285525 >> 172.21.15.50:0/3369285525 conn(0x7f09800c8850 msgr2=0x7f09800c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 -- 172.21.15.50:0/3369285525 shutdown_connections 2023-12-19T15:54:47.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.284+0000 7f09864f0700 1 -- 172.21.15.50:0/3369285525 wait complete. 2023-12-19T15:54:47.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 Processor -- start 2023-12-19T15:54:47.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 -- start start 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09800c75e0 0x7f0980169c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800b0d00 con 0x7f0980140ae0 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800c30c0 con 0x7f09800c75e0 2023-12-19T15:54:47.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f09864f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09800b1990 con 0x7f0980140710 2023-12-19T15:54:47.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.288+0000 7f0984a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49148/0 (socket says 172.21.15.50:49148) 2023-12-19T15:54:47.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f097ffff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09800c75e0 0x7f0980169c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 -- 172.21.15.50:0/2451407863 learned_addr learned my addr 172.21.15.50:0/2451407863 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:47.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0977fff700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 -- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 msgr2=0x7f09800b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800b1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 -- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09800c75e0 msgr2=0x7f0980169c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09800c75e0 0x7f0980169c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 -- 172.21.15.50:0/2451407863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09700094f0 con 0x7f0980140ae0 2023-12-19T15:54:47.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f0984a8d700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f097800c890 tx=0x7f0978001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:47.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978012630 con 0x7f0980140ae0 2023-12-19T15:54:47.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0978012790 con 0x7f0980140ae0 2023-12-19T15:54:47.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0968009420 con 0x7f0980140ae0 2023-12-19T15:54:47.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978012b50 con 0x7f0980140ae0 2023-12-19T15:54:47.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09800b0740 con 0x7f0980140ae0 2023-12-19T15:54:47.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.292+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0980071120 con 0x7f0980140ae0 2023-12-19T15:54:47.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.296+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f097801a3a0 con 0x7f0980140ae0 2023-12-19T15:54:47.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.296+0000 7f097cff9700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 0x7f0960062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:47.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.296+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f097807ccf0 con 0x7f0980140ae0 2023-12-19T15:54:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.300+0000 7f097ffff700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 0x7f0960062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:47.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.300+0000 7f097ffff700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 0x7f0960062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0970006d20 tx=0x7f0970009250 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.300+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f097804cfb0 con 0x7f0980140ae0 2023-12-19T15:54:47.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.512+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09800c65a0 con 0x7f0980140ae0 2023-12-19T15:54:47.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.512+0000 7f097cff9700 1 -- 172.21.15.50:0/2451407863 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f09780505d0 con 0x7f0980140ae0 2023-12-19T15:54:47.514 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:54:47.514 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:54:47.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 msgr2=0x7f0960062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 0x7f0960062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0970006d20 tx=0x7f0970009250 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 msgr2=0x7f098013ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f097800c890 tx=0x7f0978001670 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 shutdown_connections 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0960060670 0x7f0960062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0980140ae0 0x7f098013ec40 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0980140710 0x7f09800b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 --2- 172.21.15.50:0/2451407863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09800c75e0 0x7f0980169c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 >> 172.21.15.50:0/2451407863 conn(0x7f09800c8850 msgr2=0x7f09800aced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 shutdown_connections 2023-12-19T15:54:47.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:47.516+0000 7f09864f0700 1 -- 172.21.15.50:0/2451407863 wait complete. 2023-12-19T15:54:47.521 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:54:47.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T15:54:41.248010+0000', 'modified': '2023-12-19T15:54:44.834638+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8701}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8701': {'gid': 8701, 'name': 'd', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.125:6836/4138116501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6834', 'nonce': 4138116501}, {'type': 'v1', 'addr': '172.21.15.125:6836', 'nonce': 4138116501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T15:54:47.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T15:54:47.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-19T15:54:48.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.112+0000 7fe5bbcf0700 1 Processor -- start 2023-12-19T15:54:48.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.112+0000 7fe5bbcf0700 1 -- start start 2023-12-19T15:54:48.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5bbcf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b4158cc0 0x7fe5b406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5bbcf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe5b40b3870 0x7fe5b40b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5bbcf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b406b0d0 con 0x7fe5b406dc00 2023-12-19T15:54:48.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5bbcf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b40b4790 con 0x7fe5b40b3870 2023-12-19T15:54:48.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5bbcf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b406d710 con 0x7fe5b4158cc0 2023-12-19T15:54:48.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5b928b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe5b406dc00 0x7fe5b406b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59164/0 (socket says 172.21.15.50:59164) 2023-12-19T15:54:48.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5b928b700 1 -- 172.21.15.50:0/2138065359 learned_addr learned my addr 172.21.15.50:0/2138065359 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:48.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/2138065359 >> v2:172.21.15.50:3300/0 conn(0x7fe5b40b3870 0x7fe5b40b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b4158cc0 0x7fe5b406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5b8a8a700 1 -- 172.21.15.50:0/2138065359 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111781298 0 0) 0x7fe5b406b0d0 con 0x7fe5b406dc00 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5b8a8a700 1 -- 172.21.15.50:0/2138065359 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a0003570 con 0x7fe5b406dc00 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5ba28d700 1 -- 172.21.15.50:0/2138065359 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b4158cc0 msgr2=0x7fe5b406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b4158cc0 0x7fe5b406d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5ba28d700 1 -- 172.21.15.50:0/2138065359 >> v1:172.21.15.50:6789/0 conn(0x7fe5b406dc00 legacy=0x7fe5b406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.116+0000 7fe5ba28d700 1 -- 172.21.15.50:0/2138065359 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b40b41c0 con 0x7fe5b40b3870 2023-12-19T15:54:48.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b40b3870 0x7fe5b40b3c80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009e60 tx=0x7fe5ac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cebb95adc20d7c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5b8a8a700 1 -- 172.21.15.50:0/2138065359 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac00e070 con 0x7fe5b40b3870 2023-12-19T15:54:48.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5b8a8a700 1 -- 172.21.15.50:0/2138065359 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ac00bab0 con 0x7fe5b40b3870 2023-12-19T15:54:48.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5b8a8a700 1 -- 172.21.15.50:0/2138065359 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac00be70 con 0x7fe5b40b3870 2023-12-19T15:54:48.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2138065359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b40b3870 msgr2=0x7fe5b40b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b40b3870 0x7fe5b40b3c80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009e60 tx=0x7fe5ac009780 comp rx=0 tx=0).stop 2023-12-19T15:54:48.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2138065359 shutdown_connections 2023-12-19T15:54:48.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b40b3870 0x7fe5b40b3c80 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2138065359 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b4158cc0 0x7fe5b406d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2138065359 >> 172.21.15.50:0/2138065359 conn(0x7fe5b4067bb0 msgr2=0x7fe5b40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2138065359 shutdown_connections 2023-12-19T15:54:48.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.120+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2138065359 wait complete. 2023-12-19T15:54:48.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 Processor -- start 2023-12-19T15:54:48.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 -- start start 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b40b3870 0x7fe5b41693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b4158cc0 0x7fe5b40aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40adcf0 con 0x7fe5b406dc00 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b41698e0 con 0x7fe5b4158cc0 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.124+0000 7fe5bbcf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40ac590 con 0x7fe5b40b3870 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49160/0 (socket says 172.21.15.50:49160) 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2564994033 learned_addr learned my addr 172.21.15.50:0/2564994033 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b928b700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b40b3870 0x7fe5b41693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b4158cc0 0x7fe5b40aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b40b3870 msgr2=0x7fe5b41693a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b40b3870 0x7fe5b41693a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b4158cc0 msgr2=0x7fe5b40aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b4158cc0 0x7fe5b40aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2564994033 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac009a20 con 0x7fe5b406dc00 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000ec80 tx=0x7fe5b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5a9ffb700 1 -- 172.21.15.50:0/2564994033 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b000c850 con 0x7fe5b406dc00 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5a9ffb700 1 -- 172.21.15.50:0/2564994033 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b000c9b0 con 0x7fe5b406dc00 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5a9ffb700 1 -- 172.21.15.50:0/2564994033 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b000cd50 con 0x7fe5b406dc00 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b40ad0e0 con 0x7fe5b406dc00 2023-12-19T15:54:48.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.128+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b40ad6a0 con 0x7fe5b406dc00 2023-12-19T15:54:48.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.132+0000 7fe5a9ffb700 1 -- 172.21.15.50:0/2564994033 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe5b0013680 con 0x7fe5b406dc00 2023-12-19T15:54:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5a9ffb700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 0x7fe598062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5a9ffb700 1 -- 172.21.15.50:0/2564994033 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe5b007cdc0 con 0x7fe5b406dc00 2023-12-19T15:54:48.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 Processor -- start 2023-12-19T15:54:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 -- start start 2023-12-19T15:54:48.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b928b700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 0x7fe598062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1648750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 0x1648c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x140d790 0x16491d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x156df30 con 0x13c2710 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1569ed0 con 0x140d790 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x15300b0 con 0x13c2ae0 2023-12-19T15:54:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b928b700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 0x7fe598062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40af930 tx=0x7fe5a4009040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 0x1648c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 0x1648c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49174/0 (socket says 172.21.15.50:49174) 2023-12-19T15:54:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b9a8c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1648750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5b928b700 1 -- 172.21.15.50:0/2142438898 learned_addr learned my addr 172.21.15.50:0/2142438898 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.136+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x140d790 0x16491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x140d790 msgr2=0x16491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x140d790 0x16491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2142438898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 msgr2=0x1648c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 0x1648c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 -- 172.21.15.50:0/2142438898 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x140e9e0 con 0x13c2ae0 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1648750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe5b007afd0 tx=0x7fe5b0050b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280b6b2a6b246629 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe596ffd700 1 -- 172.21.15.50:0/2142438898 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0066070 con 0x13c2ae0 2023-12-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe596ffd700 1 -- 172.21.15.50:0/2142438898 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b00661d0 con 0x13c2ae0 2023-12-19T15:54:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe596ffd700 1 -- 172.21.15.50:0/2142438898 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00560f0 con 0x13c2ae0 2023-12-19T15:54:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 msgr2=0x1648750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1648750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe5b007afd0 tx=0x7fe5b0050b10 comp rx=0 tx=0).stop 2023-12-19T15:54:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/2142438898 shutdown_connections 2023-12-19T15:54:48.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x140d790 0x16491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2710 0x1648c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/2142438898 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1648750 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/2142438898 >> 172.21.15.50:0/2142438898 conn(0x13c2eb0 msgr2=0x15fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/2142438898 shutdown_connections 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/2142438898 wait complete. 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 -- start start 2023-12-19T15:54:48.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.140+0000 7fe5bdfdc740 1 Processor -- start 2023-12-19T15:54:48.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1624160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 0x1607370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x152fd50 con 0x13c2ae0 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1508970 con 0x13c2710 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x156e810 con 0x140d790 2023-12-19T15:54:48.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5bdfdc740 10 client.0 _ll_register_callbacks cb 0x15b36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49184/0 (socket says 172.21.15.50:49184) 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b9a8c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1624160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 -- 172.21.15.50:0/3258215186 learned_addr learned my addr 172.21.15.50:0/3258215186 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 0x1607370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 -- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 msgr2=0x1607370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 0x1607370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 -- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 msgr2=0x1624160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1624160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 -- 172.21.15.50:0/3258215186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b40ad6a0 con 0x13c2ae0 2023-12-19T15:54:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1624160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:48.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 0x1607370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40075a0 tx=0x7fe5a4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f79ba07036d579 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.144+0000 7fe58ffff700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4014ad0 con 0x13c2ae0 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x150c3e0 con 0x13c2ae0 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe5a4023c90 con 0x13c2ae0 2023-12-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5a4023e50 con 0x13c2ae0 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 10 client.8994 fetch_fsmap learned FSMap version 75 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x1538370 con 0x13c2ae0 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x150c3e0 con 0x13c2ae0 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 75) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe5a4020070 con 0x13c2ae0 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 10 client.8994 fetch_fsmap finished waiting for FSMap version 75 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 10 client.8994 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 20 client.8994 populate_metadata read hostname 'smithi050' 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x1608100 0x1657ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 4 client.8994 mds_command: new command op to 8701 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T15:54:48.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 --> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x153e410 con 0x1608100 2023-12-19T15:54:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x1608100 0x1657ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x1608100 0x1657ae0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff869643bbf0220 server_cookie=1d79f102eb9493c1 in_seq=0 out_seq=0 2023-12-19T15:54:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.148+0000 7fe58ffff700 10 client.8994 ms_handle_connect on v2:172.21.15.125:6834/4138116501 2023-12-19T15:54:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe58ffff700 1 -- 172.21.15.50:0/3258215186 <== mds.0 v2:172.21.15.125:6834/4138116501 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x153e410 con 0x1608100 2023-12-19T15:54:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe58ffff700 10 client.8994 handle_command_reply: tid=0 2023-12-19T15:54:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 client.8994 shutdown 2023-12-19T15:54:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 msgr2=0x1606e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40075a0 tx=0x7fe5a4013000 comp rx=0 tx=0).stop 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 shutdown_connections 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x140d790 0x1607370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2ae0 0x1606e30 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x1608100 0x1657ae0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3258215186 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1624160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe58ffff700 0 client.8994 ms_handle_reset on v2:172.21.15.125:6834/4138116501 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 >> 172.21.15.50:0/3258215186 conn(0x13c2eb0 msgr2=0x15f4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 shutdown_connections 2023-12-19T15:54:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3258215186 wait complete. 2023-12-19T15:54:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.152+0000 7fe5bdfdc740 20 client.8994 trim_cache size 0 max 16384 2023-12-19T15:54:48.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.172+0000 7fe5bdfdc740 1 Processor -- start 2023-12-19T15:54:48.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.172+0000 7fe5bdfdc740 1 -- start start 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x169dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x169e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1567890 con 0x13c2eb0 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x153e610 con 0x13c2710 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x151b790 con 0x13c2ae0 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:42354/0 (socket says 172.21.15.50:42354) 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 -- 172.21.15.50:0/3017311455 learned_addr learned my addr 172.21.15.50:0/3017311455 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x169dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x169e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 -- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 msgr2=0x169e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x169e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 -- 172.21.15.50:0/3017311455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 msgr2=0x169dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x169dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 -- 172.21.15.50:0/3017311455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x151ed80 con 0x13c2710 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x169dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5b928b700 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4007c20 tx=0x7fe5a4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6eb01e9c1ec6686 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5977fe700 1 -- 172.21.15.50:0/3017311455 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4014710 con 0x13c2710 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5977fe700 1 -- 172.21.15.50:0/3017311455 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4014e90 con 0x13c2710 2023-12-19T15:54:48.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5977fe700 1 -- 172.21.15.50:0/3017311455 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4005960 con 0x13c2710 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 msgr2=0x169e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4007c20 tx=0x7fe5a4017000 comp rx=0 tx=0).stop 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3017311455 shutdown_connections 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x169e000 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x169e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/3017311455 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x169dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3017311455 >> 172.21.15.50:0/3017311455 conn(0x15f7c80 msgr2=0x1605e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3017311455 shutdown_connections 2023-12-19T15:54:48.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.176+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/3017311455 wait complete. 2023-12-19T15:54:48.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 -- start start 2023-12-19T15:54:48.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 Processor -- start 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1643f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1644490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1565b50 con 0x13c2eb0 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x143f650 con 0x13c2710 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x156d9d0 con 0x13c2ae0 2023-12-19T15:54:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5bdfdc740 10 client.0 _ll_register_callbacks cb 0x138e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T15:54:48.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5b928b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1644490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49202/0 (socket says 172.21.15.50:49202) 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 -- 172.21.15.50:0/1255005708 learned_addr learned my addr 172.21.15.50:0/1255005708 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1643f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 -- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 msgr2=0x1644490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1644490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 -- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 msgr2=0x1643f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1643f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.180+0000 7fe5ba28d700 1 -- 172.21.15.50:0/1255005708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1538370 con 0x13c2eb0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5ba28d700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fe5b406eb60 tx=0x7fe5ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2226427973784f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac02f070 con 0x13c2eb0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fe5ac01f7c0 con 0x13c2eb0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1508cc0 con 0x13c2eb0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5ac01fb90 con 0x13c2eb0 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 10 client.9000 fetch_fsmap learned FSMap version 75 2023-12-19T15:54:48.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=2+}) v3 -- 0x1513130 con 0x13c2eb0 2023-12-19T15:54:48.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1508cc0 con 0x13c2eb0 2023-12-19T15:54:48.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 75) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe5ac032020 con 0x13c2eb0 2023-12-19T15:54:48.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 10 client.9000 fetch_fsmap finished waiting for FSMap version 75 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 10 client.9000 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 20 client.9000 populate_metadata read hostname 'smithi050' 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x140d790 0x16a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 4 client.9000 mds_command: new command op to 8701 tid=0[{"prefix": "status"}] 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 --> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] -- command(tid 0: {"prefix": "status"}) v1 -- 0x141a4e0 con 0x140d790 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x140d790 0x16a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe5b9a8c700 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x140d790 0x16a9d50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e191436e899539b server_cookie=5a11e6dbb250b295 in_seq=0 out_seq=0 2023-12-19T15:54:48.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.184+0000 7fe58ffff700 10 client.9000 ms_handle_connect on v2:172.21.15.125:6834/4138116501 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe58ffff700 1 -- 172.21.15.50:0/1255005708 <== mds.0 v2:172.21.15.125:6834/4138116501 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x141a4e0 con 0x140d790 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe58ffff700 10 client.9000 handle_command_reply: tid=0 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 client.9000 shutdown 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 msgr2=0x16449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fe5b406eb60 tx=0x7fe5ac019000 comp rx=0 tx=0).stop 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 shutdown_connections 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x13c2eb0 0x16449d0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x13c2ae0 0x1644490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:6834/4138116501,v1:172.21.15.125:6836/4138116501] conn(0x140d790 0x16a9d50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 --2- 172.21.15.50:0/1255005708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x13c2710 0x1643f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 >> 172.21.15.50:0/1255005708 conn(0x15f7c80 msgr2=0x1605e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe58ffff700 0 client.9000 ms_handle_reset on v2:172.21.15.125:6834/4138116501 2023-12-19T15:54:48.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 shutdown_connections 2023-12-19T15:54:48.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 1 -- 172.21.15.50:0/1255005708 wait complete. 2023-12-19T15:54:48.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bdfdc740 20 client.9000 trim_cache size 0 max 16384 2023-12-19T15:54:48.192 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:54:48.192 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 8701, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.384467184, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 75, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 167, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 167, 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.316293558 2023-12-19T15:54:48.193 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:54:48.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 msgr2=0x7fe598062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 0x7fe598062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40af930 tx=0x7fe5a4009040 comp rx=0 tx=0).stop 2023-12-19T15:54:48.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 msgr2=0x7fe5b40b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000ec80 tx=0x7fe5b00091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 shutdown_connections 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5b4158cc0 0x7fe5b40aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5980606a0 0x7fe598062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5b40b3870 0x7fe5b41693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 --2- 172.21.15.50:0/2564994033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5b406dc00 0x7fe5b40b23f0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 >> 172.21.15.50:0/2564994033 conn(0x7fe5b4067bb0 msgr2=0x7fe5b4159090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 shutdown_connections 2023-12-19T15:54:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.188+0000 7fe5bbcf0700 1 -- 172.21.15.50:0/2564994033 wait complete. 2023-12-19T15:54:48.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T15:54:48.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.392+0000 7f6fa8ae3700 1 Processor -- start 2023-12-19T15:54:48.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.392+0000 7f6fa8ae3700 1 -- start start 2023-12-19T15:54:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa8ae3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 0x7f6fa415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa8ae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa8ae3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa406c7c0 con 0x7f6fa406b7a0 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa8ae3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa406caa0 con 0x7f6fa4158d60 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa8ae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa415ce60 con 0x7f6fa40c5380 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55246/0 (socket says 172.21.15.50:55246) 2023-12-19T15:54:48.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1920485290 learned_addr learned my addr 172.21.15.50:0/1920485290 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa37fe700 1 --2- 172.21.15.50:0/1920485290 >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 0x7f6fa415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1920485290 >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 msgr2=0x7f6fa415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 --2- 172.21.15.50:0/1920485290 >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 0x7f6fa415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1920485290 >> v1:172.21.15.125:6789/0 conn(0x7f6fa406b7a0 legacy=0x7f6fa40c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1920485290 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa415d130 con 0x7f6fa40c5380 2023-12-19T15:54:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.396+0000 7f6fa37fe700 1 --2- 172.21.15.50:0/1920485290 >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 0x7f6fa415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa3fff700 1 --2- 172.21.15.50:0/1920485290 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009f20 tx=0x7f6f9c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b427656a3c9d65 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa2ffd700 1 -- 172.21.15.50:0/1920485290 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00e070 con 0x7f6fa40c5380 2023-12-19T15:54:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa2ffd700 1 -- 172.21.15.50:0/1920485290 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00ba80 con 0x7f6fa40c5380 2023-12-19T15:54:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa2ffd700 1 -- 172.21.15.50:0/1920485290 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0092e0 con 0x7f6fa40c5380 2023-12-19T15:54:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1920485290 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 msgr2=0x7f6fa406cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1920485290 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009f20 tx=0x7f6f9c009840 comp rx=0 tx=0).stop 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1920485290 shutdown_connections 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1920485290 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa40c5380 0x7f6fa406cf30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1920485290 >> v2:172.21.15.125:3300/0 conn(0x7f6fa4158d60 0x7f6fa415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1920485290 >> 172.21.15.50:0/1920485290 conn(0x7f6fa4067bb0 msgr2=0x7f6fa40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1920485290 shutdown_connections 2023-12-19T15:54:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1920485290 wait complete. 2023-12-19T15:54:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.400+0000 7f6fa8ae3700 1 Processor -- start 2023-12-19T15:54:48.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 -- start start 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa406b7a0 0x7f6fa41570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 0x7f6fa4146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4146bf0 con 0x7f6fa40c5380 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4143c60 con 0x7f6fa406b7a0 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4155280 con 0x7f6fa4158d60 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 0x7f6fa4146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 0x7f6fa4146170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49210/0 (socket says 172.21.15.50:49210) 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55256/0 (socket says 172.21.15.50:55256) 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9bfff700 1 -- 172.21.15.50:0/1595145093 learned_addr learned my addr 172.21.15.50:0/1595145093 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:48.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa37fe700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa406b7a0 0x7f6fa41570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa406b7a0 msgr2=0x7f6fa41570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa406b7a0 0x7f6fa41570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1595145093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 msgr2=0x7f6fa4146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 0x7f6fa4146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 -- 172.21.15.50:0/1595145093 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c009ae0 con 0x7f6fa4158d60 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa3fff700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fa415c860 tx=0x7f6f9c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c009300 con 0x7f6fa4158d60 2023-12-19T15:54:48.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c0032f0 con 0x7f6fa4158d60 2023-12-19T15:54:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa4154670 con 0x7f6fa4158d60 2023-12-19T15:54:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.404+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c026460 con 0x7f6fa4158d60 2023-12-19T15:54:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.408+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa4154c30 con 0x7f6fa4158d60 2023-12-19T15:54:48.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.408+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa4155410 con 0x7f6fa4158d60 2023-12-19T15:54:48.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.408+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6f9c003040 con 0x7f6fa4158d60 2023-12-19T15:54:48.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.412+0000 7f6f9b7fe700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 0x7f6f84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.412+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6f9c07ea10 con 0x7f6fa4158d60 2023-12-19T15:54:48.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.412+0000 7f6fa37fe700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 0x7f6f84062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:48.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.412+0000 7f6fa37fe700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 0x7f6f84062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006f00 tx=0x7f6f94008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:48.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.420+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f9c04ebc0 con 0x7f6fa4158d60 2023-12-19T15:54:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.632+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6fa4154fd0 con 0x7f6fa4158d60 2023-12-19T15:54:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.636+0000 7f6f9b7fe700 1 -- 172.21.15.50:0/1595145093 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v75) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6f9c0521e0 con 0x7f6fa4158d60 2023-12-19T15:54:48.637 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T15:54:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 msgr2=0x7f6f84062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 0x7f6f84062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006f00 tx=0x7f6f94008040 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 msgr2=0x7f6fa41466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fa415c860 tx=0x7f6f9c018000 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 shutdown_connections 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f84060670 0x7f6f84062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6fa4158d60 0x7f6fa41466b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fa40c5380 0x7f6fa4146170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 --2- 172.21.15.50:0/1595145093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa406b7a0 0x7f6fa41570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 >> 172.21.15.50:0/1595145093 conn(0x7f6fa4067bb0 msgr2=0x7f6fa40c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 shutdown_connections 2023-12-19T15:54:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:48.640+0000 7f6fa8ae3700 1 -- 172.21.15.50:0/1595145093 wait complete. 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T15:54:48.663 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T15:54:48.664 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:54:48.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.664 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T15:54:48.681 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T15:54:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T15:54:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T15:54:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 37861sec preferred_lft 37861sec 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.682 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T15:54:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.684 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T15:54:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T15:54:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T15:54:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:54:48.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.685 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T15:54:48.771 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T15:54:48.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.774 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T15:54:48.834 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T15:54:48.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.835 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T15:54:48.835 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T15:54:48.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T15:54:48.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T15:54:48.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T15:54:48.836 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T15:54:48.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.970 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T15:54:48.986 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T15:54:48.986 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T15:54:48.987 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:54:48.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:48.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T15:54:49.068 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T15:54:49.068 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T15:54:49.068 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:49.127 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:54:49.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T15:54:49.127 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:49.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T15:54:49.268 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T15:54:49.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T15:54:49.350 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T15:54:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T15:54:49.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:49.354 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T15:54:49.417 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T15:54:49.417 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T15:54:49.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T15:54:49.500 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T15:54:49.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T15:54:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T15:54:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T15:54:49.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:49.579 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T15:54:49.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T15:54:49.604+0000 7f6338f1d380 -1 init, newargv = 0x55db97455c40 newargc=15 2023-12-19T15:54:49.608 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[56613]: starting ceph client 2023-12-19T15:54:49.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[56613]: starting fuse 2023-12-19T15:54:49.624 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T15:54:49.624 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T15:54:49.625 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T15:54:49.626 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T15:54:49.627 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T15:54:49.627 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:54:49.627 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T15:54:49.627 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T15:54:49.798 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.56613.asok 2023-12-19T15:54:49.983 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56613.asok 2023-12-19T15:54:49.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:49.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56613.asok status 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T15:54:50.164 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "56613", 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 9009, 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T15:54:50.165 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 9009 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1438431518 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T15:54:50.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1438431518 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.9009 192.168.0.1:0/1438431518", 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/1438431518", 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 75, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 167, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T15:54:50.167 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:54:50.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:50.179 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T15:54:50.179 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:50.198 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T15:54:50.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:50.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:50.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:50.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T15:54:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.496+0000 7fdfaca25700 1 Processor -- start 2023-12-19T15:54:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.496+0000 7fdfaca25700 1 -- start start 2023-12-19T15:54:50.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfaca25700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfa8158e50 0x7fdfa806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfaca25700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa806e160 con 0x7fdfa8159220 2023-12-19T15:54:50.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfaca25700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa806d0c0 con 0x7fdfa806ae50 2023-12-19T15:54:50.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfaca25700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa806d3a0 con 0x7fdfa8158e50 2023-12-19T15:54:50.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfa8158e50 0x7fdfa806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdfa8158e50 0x7fdfa806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49216/0 (socket says 172.21.15.50:49216) 2023-12-19T15:54:50.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 -- 172.21.15.50:0/1342275535 learned_addr learned my addr 172.21.15.50:0/1342275535 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:50.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 -- 172.21.15.50:0/1342275535 >> v1:172.21.15.125:6789/0 conn(0x7fdfa806ae50 legacy=0x7fdfa80aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 -- 172.21.15.50:0/1342275535 >> v1:172.21.15.50:6789/0 conn(0x7fdfa8159220 legacy=0x7fdfa806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.500+0000 7fdfa659c700 1 -- 172.21.15.50:0/1342275535 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa806d680 con 0x7fdfa8158e50 2023-12-19T15:54:50.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfa659c700 1 --2- 172.21.15.50:0/1342275535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8158e50 0x7fdfa806c7c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdf980099e0 tx=0x7fdf98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c677b88ea229874 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfa559a700 1 -- 172.21.15.50:0/1342275535 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800b820 con 0x7fdfa8158e50 2023-12-19T15:54:50.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfa559a700 1 -- 172.21.15.50:0/1342275535 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9800b980 con 0x7fdfa8158e50 2023-12-19T15:54:50.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfa559a700 1 -- 172.21.15.50:0/1342275535 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800bd20 con 0x7fdfa8158e50 2023-12-19T15:54:50.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 -- 172.21.15.50:0/1342275535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8158e50 msgr2=0x7fdfa806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 --2- 172.21.15.50:0/1342275535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8158e50 0x7fdfa806c7c0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdf980099e0 tx=0x7fdf98009250 comp rx=0 tx=0).stop 2023-12-19T15:54:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 -- 172.21.15.50:0/1342275535 shutdown_connections 2023-12-19T15:54:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 --2- 172.21.15.50:0/1342275535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8158e50 0x7fdfa806c7c0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 -- 172.21.15.50:0/1342275535 >> 172.21.15.50:0/1342275535 conn(0x7fdfa8067b60 msgr2=0x7fdfa80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 -- 172.21.15.50:0/1342275535 shutdown_connections 2023-12-19T15:54:50.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.504+0000 7fdfaca25700 1 -- 172.21.15.50:0/1342275535 wait complete. 2023-12-19T15:54:50.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 Processor -- start 2023-12-19T15:54:50.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 -- start start 2023-12-19T15:54:50.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfa806ae50 0x7fdfa815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa8158e50 0x7fdfa80af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa81568b0 con 0x7fdfa8159220 2023-12-19T15:54:50.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80cb5d0 con 0x7fdfa8158e50 2023-12-19T15:54:50.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfaca25700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa815a9c0 con 0x7fdfa806ae50 2023-12-19T15:54:50.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfa6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfa6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49218/0 (socket says 172.21.15.50:49218) 2023-12-19T15:54:50.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfa6d9d700 1 -- 172.21.15.50:0/3932836342 learned_addr learned my addr 172.21.15.50:0/3932836342 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfa5d9b700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa8158e50 0x7fdfa80af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.508+0000 7fdfa659c700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfa806ae50 0x7fdfa815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 -- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfa806ae50 msgr2=0x7fdfa815a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfa806ae50 0x7fdfa815a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 -- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa8158e50 msgr2=0x7fdfa80af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa8158e50 0x7fdfa80af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 -- 172.21.15.50:0/3932836342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980094f0 con 0x7fdfa8159220 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfa6d9d700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c810 tx=0x7fdfa0001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0012630 con 0x7fdfa8159220 2023-12-19T15:54:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa00127b0 con 0x7fdfa8159220 2023-12-19T15:54:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa815aca0 con 0x7fdfa8159220 2023-12-19T15:54:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0012b70 con 0x7fdfa8159220 2023-12-19T15:54:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa80c8580 con 0x7fdfa8159220 2023-12-19T15:54:50.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.512+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa815aee0 con 0x7fdfa8159220 2023-12-19T15:54:50.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.516+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdfa0003030 con 0x7fdfa8159220 2023-12-19T15:54:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.516+0000 7fdf967fc700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 0x7fdf88062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.516+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fdfa007c930 con 0x7fdfa8159220 2023-12-19T15:54:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.516+0000 7fdfa659c700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 0x7fdf88062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.520+0000 7fdfa659c700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 0x7fdf88062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf980099e0 tx=0x7fdf98017000 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.524+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfa815aee0 con 0x7fdfa8159220 2023-12-19T15:54:50.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.720+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfa81563a0 con 0x7fdf88060670 2023-12-19T15:54:50.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.724+0000 7fdf967fc700 1 -- 172.21.15.50:0/3932836342 <== mgr.7577 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdfa81563a0 con 0x7fdf88060670 2023-12-19T15:54:50.728 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T15:54:50.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:54:50.728 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T15:54:50.728 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:54:50.728 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T15:54:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 msgr2=0x7fdf88062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 0x7fdf88062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf980099e0 tx=0x7fdf98017000 comp rx=0 tx=0).stop 2023-12-19T15:54:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 msgr2=0x7fdfa80cb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c810 tx=0x7fdfa0001650 comp rx=0 tx=0).stop 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 shutdown_connections 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa8158e50 0x7fdfa80af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdf88060670 0x7fdf88062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdfa8159220 0x7fdfa80cb090 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 --2- 172.21.15.50:0/3932836342 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdfa806ae50 0x7fdfa815a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 >> 172.21.15.50:0/3932836342 conn(0x7fdfa8067b60 msgr2=0x7fdfa80ad420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 shutdown_connections 2023-12-19T15:54:50.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.728+0000 7fdfaca25700 1 -- 172.21.15.50:0/3932836342 wait complete. 2023-12-19T15:54:50.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T15:54:50.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.924+0000 7f8b2cf85700 1 Processor -- start 2023-12-19T15:54:50.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.924+0000 7f8b2cf85700 1 -- start start 2023-12-19T15:54:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.928+0000 7f8b2cf85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b28158e20 0x7f8b280ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.928+0000 7f8b2cf85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b280ca2c0 0x7f8b280ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.928+0000 7f8b2cf85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b280cbdf0 con 0x7f8b280cb970 2023-12-19T15:54:50.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.928+0000 7f8b2cf85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b280cc0d0 con 0x7f8b280ca2c0 2023-12-19T15:54:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.928+0000 7f8b2cf85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b280cc3b0 con 0x7f8b28158e20 2023-12-19T15:54:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8b280ca2c0 0x7f8b280ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b25d9b700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8b280cb970 0x7f8b2806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:45652/0 (socket says 172.21.15.50:45652) 2023-12-19T15:54:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b25d9b700 1 -- 172.21.15.50:0/928167511 learned_addr learned my addr 172.21.15.50:0/928167511 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2659c700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b28158e20 0x7f8b280ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2559a700 1 -- 172.21.15.50:0/928167511 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562220051 0 0) 0x7f8b280cbdf0 con 0x7f8b280cb970 2023-12-19T15:54:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 -- 172.21.15.50:0/928167511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b28158e20 msgr2=0x7f8b280ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b28158e20 0x7f8b280ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 -- 172.21.15.50:0/928167511 >> v1:172.21.15.125:6789/0 conn(0x7f8b280cb970 legacy=0x7f8b2806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 -- 172.21.15.50:0/928167511 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280cc690 con 0x7f8b280ca2c0 2023-12-19T15:54:50.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b26d9d700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b280ca2c0 0x7f8b280ceb00 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009e60 tx=0x7f8b20009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f710ee9b150d204 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2559a700 1 -- 172.21.15.50:0/928167511 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000e070 con 0x7f8b280ca2c0 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2559a700 1 -- 172.21.15.50:0/928167511 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2000bb60 con 0x7f8b280ca2c0 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2559a700 1 -- 172.21.15.50:0/928167511 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20012430 con 0x7f8b280ca2c0 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 -- 172.21.15.50:0/928167511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b280ca2c0 msgr2=0x7f8b280ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b280ca2c0 0x7f8b280ceb00 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009e60 tx=0x7f8b20009780 comp rx=0 tx=0).stop 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 -- 172.21.15.50:0/928167511 shutdown_connections 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b280ca2c0 0x7f8b280ceb00 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/928167511 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b28158e20 0x7f8b280ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 -- 172.21.15.50:0/928167511 >> 172.21.15.50:0/928167511 conn(0x7f8b28067bb0 msgr2=0x7f8b280d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 -- 172.21.15.50:0/928167511 shutdown_connections 2023-12-19T15:54:50.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.932+0000 7f8b2cf85700 1 -- 172.21.15.50:0/928167511 wait complete. 2023-12-19T15:54:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 Processor -- start 2023-12-19T15:54:50.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 -- start start 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 0x7f8b281527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 0x7f8b2816a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814b480 con 0x7f8b28158e20 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814d810 con 0x7f8b2814d3a0 2023-12-19T15:54:50.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2cf85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2814daf0 con 0x7f8b280ca2c0 2023-12-19T15:54:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b25d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b25d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49238/0 (socket says 172.21.15.50:49238) 2023-12-19T15:54:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b25d9b700 1 -- 172.21.15.50:0/2957617257 learned_addr learned my addr 172.21.15.50:0/2957617257 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b2659c700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 0x7f8b281527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.936+0000 7f8b26d9d700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 0x7f8b2816a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 -- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 msgr2=0x7f8b281527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 0x7f8b281527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 -- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 msgr2=0x7f8b2816a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 0x7f8b2816a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 -- 172.21.15.50:0/2957617257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20009a20 con 0x7f8b28158e20 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b26d9d700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 0x7f8b2816a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b2659c700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 0x7f8b281527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:50.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b25d9b700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c810 tx=0x7f8b1c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c005c00 con 0x7f8b28158e20 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c005d60 con 0x7f8b28158e20 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2814ddd0 con 0x7f8b28158e20 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c007340 con 0x7f8b28158e20 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b28169210 con 0x7f8b28158e20 2023-12-19T15:54:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.940+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b28071120 con 0x7f8b28158e20 2023-12-19T15:54:50.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.944+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 68) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8b1c01b400 con 0x7f8b28158e20 2023-12-19T15:54:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.944+0000 7f8b167fc700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 0x7f8b08062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:50.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.944+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8b1c07c620 con 0x7f8b28158e20 2023-12-19T15:54:50.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.944+0000 7f8b2659c700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 0x7f8b08062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:50.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.948+0000 7f8b2659c700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 0x7f8b08062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b2814f850 tx=0x7f8b18008040 comp rx=0 tx=0).ready entity=mgr.7577 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:50.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:50.952+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8b1c04c960 con 0x7f8b28158e20 2023-12-19T15:54:51.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:51.172+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f8b280b5fe0 con 0x7f8b28158e20 2023-12-19T15:54:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.004+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 69) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f8b1c04cd30 con 0x7f8b28158e20 2023-12-19T15:54:52.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.004+0000 7f8b167fc700 1 -- 172.21.15.50:0/2957617257 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) v1 ==== 92+0+0 (secure 0 0 0) 0x7f8b1c0503f0 con 0x7f8b28158e20 2023-12-19T15:54:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 msgr2=0x7f8b08062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 0x7f8b08062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b2814f850 tx=0x7f8b18008040 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 msgr2=0x7f8b2814fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00c810 tx=0x7f8b1c007dd0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 shutdown_connections 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2814d3a0 0x7f8b2816a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8b08060670 0x7f8b08062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b28158e20 0x7f8b2814fa90 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 --2- 172.21.15.50:0/2957617257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8b280ca2c0 0x7f8b281527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 >> 172.21.15.50:0/2957617257 conn(0x7f8b28067bb0 msgr2=0x7f8b2814c980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 shutdown_connections 2023-12-19T15:54:52.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.008+0000 7f8b2cf85700 1 -- 172.21.15.50:0/2957617257 wait complete. 2023-12-19T15:54:52.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T15:54:52.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.196+0000 7f5f464ae700 1 Processor -- start 2023-12-19T15:54:52.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.196+0000 7f5f464ae700 1 -- start start 2023-12-19T15:54:52.199 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:52.190+0000 7f7e490fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:54:52.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.196+0000 7f5f464ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f40158e20 0x7f5f400ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4006d5e0 con 0x7f5f400ca2c0 2023-12-19T15:54:52.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400cbdf0 con 0x7f5f400cb970 2023-12-19T15:54:52.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f400cc0d0 con 0x7f5f40158e20 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f40158e20 0x7f5f400ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f40158e20 0x7f5f400ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49256/0 (socket says 172.21.15.50:49256) 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 -- 172.21.15.50:0/27676640 learned_addr learned my addr 172.21.15.50:0/27676640 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 -- 172.21.15.50:0/27676640 >> v1:172.21.15.125:6789/0 conn(0x7f5f400cb970 legacy=0x7f5f4006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 -- 172.21.15.50:0/27676640 >> v1:172.21.15.50:6789/0 conn(0x7f5f400ca2c0 legacy=0x7f5f4006ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 -- 172.21.15.50:0/27676640 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400cc3b0 con 0x7f5f40158e20 2023-12-19T15:54:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3ffff700 1 --2- 172.21.15.50:0/27676640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f40158e20 0x7f5f400ce5c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5f300099e0 tx=0x7f5f30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca59785a46763810 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3f7fe700 1 -- 172.21.15.50:0/27676640 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3000b820 con 0x7f5f40158e20 2023-12-19T15:54:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3f7fe700 1 -- 172.21.15.50:0/27676640 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3000b980 con 0x7f5f40158e20 2023-12-19T15:54:52.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f3f7fe700 1 -- 172.21.15.50:0/27676640 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3000bd20 con 0x7f5f40158e20 2023-12-19T15:54:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- 172.21.15.50:0/27676640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f40158e20 msgr2=0x7f5f400ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 --2- 172.21.15.50:0/27676640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f40158e20 0x7f5f400ce5c0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f5f300099e0 tx=0x7f5f30009250 comp rx=0 tx=0).stop 2023-12-19T15:54:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- 172.21.15.50:0/27676640 shutdown_connections 2023-12-19T15:54:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 --2- 172.21.15.50:0/27676640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f40158e20 0x7f5f400ce5c0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- 172.21.15.50:0/27676640 >> 172.21.15.50:0/27676640 conn(0x7f5f40067bb0 msgr2=0x7f5f400d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- 172.21.15.50:0/27676640 shutdown_connections 2023-12-19T15:54:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.200+0000 7f5f464ae700 1 -- 172.21.15.50:0/27676640 wait complete. 2023-12-19T15:54:52.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.204+0000 7f5f464ae700 1 Processor -- start 2023-12-19T15:54:52.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.204+0000 7f5f464ae700 1 -- start start 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.204+0000 7f5f464ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f40158e20 0x7f5f4006c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 0x7f5f4014cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c7690 con 0x7f5f400ca2c0 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c7970 con 0x7f5f4006cea0 2023-12-19T15:54:52.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c7c50 con 0x7f5f40158e20 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49266/0 (socket says 172.21.15.50:49266) 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 -- 172.21.15.50:0/1234609436 learned_addr learned my addr 172.21.15.50:0/1234609436 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f44a4b700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 0x7f5f4014cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f37fff700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f40158e20 0x7f5f4006c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 -- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f40158e20 msgr2=0x7f5f4006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f40158e20 0x7f5f4006c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 -- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 msgr2=0x7f5f4014cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 0x7f5f4014cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 -- 172.21.15.50:0/1234609436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300094f0 con 0x7f5f400ca2c0 2023-12-19T15:54:52.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f44a4b700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 0x7f5f4014cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:52.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3ffff700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f5f300094c0 tx=0x7f5f30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:52.211 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f124c8fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:54:52.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30024440 con 0x7f5f400ca2c0 2023-12-19T15:54:52.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f300245a0 con 0x7f5f400ca2c0 2023-12-19T15:54:52.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f400c7f30 con 0x7f5f400ca2c0 2023-12-19T15:54:52.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30024940 con 0x7f5f400ca2c0 2023-12-19T15:54:52.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.208+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f400b4180 con 0x7f5f400ca2c0 2023-12-19T15:54:52.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40071120 con 0x7f5f400ca2c0 2023-12-19T15:54:52.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 69) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f5f3001a420 con 0x7f5f400ca2c0 2023-12-19T15:54:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f3cff9700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 0x7f5f20062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5f3007eb80 con 0x7f5f400ca2c0 2023-12-19T15:54:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f37fff700 1 -- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 msgr2=0x7f5f20062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:52.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.212+0000 7f5f37fff700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 0x7f5f20062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:52.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.220+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5f3004ee40 con 0x7f5f400ca2c0 2023-12-19T15:54:52.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.396+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5f400cc690 con 0x7f5f400ca2c0 2023-12-19T15:54:52.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.404+0000 7f5f3cff9700 1 -- 172.21.15.50:0/1234609436 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5f3007ee60 con 0x7f5f400ca2c0 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 msgr2=0x7f5f20062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 0x7f5f20062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 msgr2=0x7f5f400b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f5f300094c0 tx=0x7f5f30017000 comp rx=0 tx=0).stop 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 shutdown_connections 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f4006cea0 0x7f5f4014cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5f200606c0 0x7f5f20062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5f40158e20 0x7f5f4006c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 --2- 172.21.15.50:0/1234609436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f400ca2c0 0x7f5f400b3c40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 >> 172.21.15.50:0/1234609436 conn(0x7f5f40067bb0 msgr2=0x7f5f4014ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 shutdown_connections 2023-12-19T15:54:52.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.408+0000 7f5f464ae700 1 -- 172.21.15.50:0/1234609436 wait complete. 2023-12-19T15:54:52.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T15:54:52.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.588+0000 7f54bf8f0700 1 Processor -- start 2023-12-19T15:54:52.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.588+0000 7f54bf8f0700 1 -- start start 2023-12-19T15:54:52.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bf8f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 0x7f54b806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bf8f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f54b80cecd0 0x7f54b806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bf8f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b80cbdf0 con 0x7f54b8158e20 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bf8f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b80cc0d0 con 0x7f54b80cecd0 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bd68c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f54b8158e20 0x7f54b80ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59196/0 (socket says 172.21.15.50:59196) 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bd68c700 1 -- 172.21.15.50:0/3219837133 learned_addr learned my addr 172.21.15.50:0/3219837133 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b80cc3b0 con 0x7f54b80cb970 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bce8b700 1 --2- 172.21.15.50:0/3219837133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 0x7f54b806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.592+0000 7f54bde8d700 1 --2- 172.21.15.50:0/3219837133 >> v2:172.21.15.125:3300/0 conn(0x7f54b80cecd0 0x7f54b806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54affff700 1 -- 172.21.15.50:0/3219837133 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87913737 0 0) 0x7f54b80cbdf0 con 0x7f54b8158e20 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54affff700 1 -- 172.21.15.50:0/3219837133 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a4003570 con 0x7f54b8158e20 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bce8b700 1 -- 172.21.15.50:0/3219837133 >> v2:172.21.15.125:3300/0 conn(0x7f54b80cecd0 msgr2=0x7f54b806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bce8b700 1 --2- 172.21.15.50:0/3219837133 >> v2:172.21.15.125:3300/0 conn(0x7f54b80cecd0 0x7f54b806b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bce8b700 1 -- 172.21.15.50:0/3219837133 >> v1:172.21.15.50:6789/0 conn(0x7f54b8158e20 legacy=0x7f54b80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bce8b700 1 -- 172.21.15.50:0/3219837133 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b80cc690 con 0x7f54b80cb970 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bce8b700 1 --2- 172.21.15.50:0/3219837133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 0x7f54b806acc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f54a80098e0 tx=0x7f54a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d56818d9f4d22868 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54affff700 1 -- 172.21.15.50:0/3219837133 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8011630 con 0x7f54b80cb970 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54affff700 1 -- 172.21.15.50:0/3219837133 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a8011790 con 0x7f54b80cb970 2023-12-19T15:54:52.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54affff700 1 -- 172.21.15.50:0/3219837133 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8011b30 con 0x7f54b80cb970 2023-12-19T15:54:52.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 msgr2=0x7f54b806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/3219837133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 0x7f54b806acc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f54a80098e0 tx=0x7f54a80091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 shutdown_connections 2023-12-19T15:54:52.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/3219837133 >> v2:172.21.15.125:3300/0 conn(0x7f54b80cecd0 0x7f54b806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/3219837133 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cb970 0x7f54b806acc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 >> 172.21.15.50:0/3219837133 conn(0x7f54b8067bb0 msgr2=0x7f54b80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:52.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 shutdown_connections 2023-12-19T15:54:52.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.596+0000 7f54bf8f0700 1 -- 172.21.15.50:0/3219837133 wait complete. 2023-12-19T15:54:52.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 Processor -- start 2023-12-19T15:54:52.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 -- start start 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54b8158e20 0x7f54b80c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 0x7f54b80c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b80b5f00 con 0x7f54b815a900 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b80b5530 con 0x7f54b8158e20 2023-12-19T15:54:52.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bf8f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b80c9c10 con 0x7f54b80cecd0 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bde8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 0x7f54b80c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:55346/0 (socket says 172.21.15.50:55346) 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 -- 172.21.15.50:0/2113850861 learned_addr learned my addr 172.21.15.50:0/2113850861 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bde8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 0x7f54b80c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49278/0 (socket says 172.21.15.50:49278) 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bce8b700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54b8158e20 0x7f54b80c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54b8158e20 msgr2=0x7f54b80c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54b8158e20 0x7f54b80c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 msgr2=0x7f54b80c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 0x7f54b80c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 -- 172.21.15.50:0/2113850861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8009420 con 0x7f54b80cecd0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.600+0000 7f54bd68c700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54b400c7c0 tx=0x7f54b4007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4005b30 con 0x7f54b80cecd0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4005cb0 con 0x7f54b80cecd0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4007320 con 0x7f54b80cecd0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b80b4920 con 0x7f54b80cecd0 2023-12-19T15:54:52.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b80b4ee0 con 0x7f54b80cecd0 2023-12-19T15:54:52.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.604+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b8071120 con 0x7f54b80cecd0 2023-12-19T15:54:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.608+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 69) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f54b4007690 con 0x7f54b80cecd0 2023-12-19T15:54:52.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.608+0000 7f54ad7fa700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 0x7f549c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:52.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.608+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f54b407bff0 con 0x7f54b80cecd0 2023-12-19T15:54:52.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.608+0000 7f54bce8b700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 msgr2=0x7f549c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:52.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.608+0000 7f54bce8b700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 0x7f549c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:52.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.616+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54b404c310 con 0x7f54b80cecd0 2023-12-19T15:54:52.765 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:52.758+0000 7f7e490fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:54:52.800 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:52.796+0000 7f124c8fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:54:52.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.800+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f54b806bff0 con 0x7f54b80cecd0 2023-12-19T15:54:52.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.812+0000 7f54bce8b700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 msgr2=0x7f549c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:52.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.812+0000 7f54bce8b700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 0x7f549c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:52.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.816+0000 7f54ad7fa700 1 -- 172.21.15.50:0/2113850861 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7f54b404f930 con 0x7f54b80cecd0 2023-12-19T15:54:52.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.816+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 msgr2=0x7f549c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.816+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 0x7f549c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.816+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 msgr2=0x7f54b80b59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.816+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54b400c7c0 tx=0x7f54b4007e00 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 shutdown_connections 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54b8158e20 0x7f54b80c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f549c0606c0 0x7f549c062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54b815a900 0x7f54b80c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 --2- 172.21.15.50:0/2113850861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f54b80cecd0 0x7f54b80b59c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 >> 172.21.15.50:0/2113850861 conn(0x7f54b8067bb0 msgr2=0x7f54b814ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 shutdown_connections 2023-12-19T15:54:52.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:52.820+0000 7f54bf8f0700 1 -- 172.21.15.50:0/2113850861 wait complete. 2023-12-19T15:54:52.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:52.843 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T15:54:52.844 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:52.834+0000 7f7e490fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:54:52.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1h 2023-12-19T15:54:52.878 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:52.876+0000 7f124c8fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:54:52.989 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:52.982+0000 7f7e490fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:53.037 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.036+0000 7f124c8fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:53.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.048+0000 7f285afcc700 1 Processor -- start 2023-12-19T15:54:53.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.048+0000 7f285afcc700 1 -- start start 2023-12-19T15:54:53.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f285afcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 0x7f285406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:53.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f285afcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540cb3f0 con 0x7f28540aedc0 2023-12-19T15:54:53.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f285afcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285406d220 con 0x7f28540cac50 2023-12-19T15:54:53.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f285afcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540cc140 con 0x7f28540af190 2023-12-19T15:54:53.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2859fca700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f28540aedc0 0x7f285406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59200/0 (socket says 172.21.15.50:59200) 2023-12-19T15:54:53.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2859fca700 1 -- 172.21.15.50:0/2025055789 learned_addr learned my addr 172.21.15.50:0/2025055789 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f28597c9700 1 --2- 172.21.15.50:0/2025055789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 0x7f285406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176648353 0 0) 0x7f28540cb3f0 con 0x7f28540aedc0 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2840003570 con 0x7f28540aedc0 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909241161 0 0) 0x7f285406d220 con 0x7f28540cac50 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.052+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28540cb3f0 con 0x7f28540cac50 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f28597c9700 1 -- 172.21.15.50:0/2025055789 >> v1:172.21.15.125:6789/0 conn(0x7f28540cac50 legacy=0x7f28540af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f28597c9700 1 -- 172.21.15.50:0/2025055789 >> v1:172.21.15.50:6789/0 conn(0x7f28540aedc0 legacy=0x7f285406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:53.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f28597c9700 1 -- 172.21.15.50:0/2025055789 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285406c8f0 con 0x7f28540af190 2023-12-19T15:54:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f28597c9700 1 --2- 172.21.15.50:0/2025055789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 0x7f285406af50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2844009910 tx=0x7f28440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aff0616f5d5afa64 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844011630 con 0x7f28540af190 2023-12-19T15:54:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844011790 con 0x7f28540af190 2023-12-19T15:54:53.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f2858fc8700 1 -- 172.21.15.50:0/2025055789 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844011b30 con 0x7f28540af190 2023-12-19T15:54:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 -- 172.21.15.50:0/2025055789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 msgr2=0x7f285406af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 --2- 172.21.15.50:0/2025055789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 0x7f285406af50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2844009910 tx=0x7f28440091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 -- 172.21.15.50:0/2025055789 shutdown_connections 2023-12-19T15:54:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 --2- 172.21.15.50:0/2025055789 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540af190 0x7f285406af50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:53.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 -- 172.21.15.50:0/2025055789 >> 172.21.15.50:0/2025055789 conn(0x7f28540cc760 msgr2=0x7f28540ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:53.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 -- 172.21.15.50:0/2025055789 shutdown_connections 2023-12-19T15:54:53.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.056+0000 7f285afcc700 1 -- 172.21.15.50:0/2025055789 wait complete. 2023-12-19T15:54:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 Processor -- start 2023-12-19T15:54:53.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 -- start start 2023-12-19T15:54:53.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540aedc0 0x7f285415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28540af190 0x7f28540c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540b4410 con 0x7f28540cac50 2023-12-19T15:54:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.060+0000 7f285afcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2854169d60 con 0x7f28540af190 2023-12-19T15:54:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285afcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540b4660 con 0x7f28540aedc0 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f2859fca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540aedc0 0x7f285415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f28597c9700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28540af190 0x7f28540c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49290/0 (socket says 172.21.15.50:49290) 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 -- 172.21.15.50:0/3004440390 learned_addr learned my addr 172.21.15.50:0/3004440390 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540aedc0 msgr2=0x7f285415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540aedc0 0x7f285415b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28540af190 msgr2=0x7f28540c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28540af190 0x7f28540c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844009420 con 0x7f28540cac50 2023-12-19T15:54:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285a7cb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c910 tx=0x7f284c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c012630 con 0x7f28540cac50 2023-12-19T15:54:53.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c0127b0 con 0x7f28540cac50 2023-12-19T15:54:53.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c012b70 con 0x7f28540cac50 2023-12-19T15:54:53.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540b4940 con 0x7f28540cac50 2023-12-19T15:54:53.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.064+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285415b660 con 0x7f28540cac50 2023-12-19T15:54:53.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28540b4e40 con 0x7f28540cac50 2023-12-19T15:54:53.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 69) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f284c01b400 con 0x7f28540cac50 2023-12-19T15:54:53.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f2849ffb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:53.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f284c07ce60 con 0x7f28540cac50 2023-12-19T15:54:53.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f2859fca700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:53.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.068+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:54:53.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.076+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f284c04d1c0 con 0x7f28540cac50 2023-12-19T15:54:53.261 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:53.254+0000 7f7e490fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:54:53.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.272+0000 7f2859fca700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:53.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.272+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:54:53.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.304+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f285404be40 con 0x7f28380606c0 2023-12-19T15:54:53.335 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:53.330+0000 7f7e490fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:53.357 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.356+0000 7f124c8fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:54:53.433 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.428+0000 7f124c8fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:53.558 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:53.550+0000 7f7e490fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:54:53.644 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:53.638+0000 7f7e490fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:54:53.676 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.672+0000 7f124c8fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:54:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.676+0000 7f2859fca700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:53.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:53.676+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T15:54:53.734 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:53.726+0000 7f7e490fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:54:53.761 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.760+0000 7f124c8fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:54:53.852 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:53.848+0000 7f124c8fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:54:54.138 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:54.130+0000 7f7e490fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:54:54.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:54:54.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:54:54.268 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:54.264+0000 7f124c8fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:54:54.309 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:54.302+0000 7f7e490fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:54:54.447 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:54.444+0000 7f124c8fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:54:54.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:54.480+0000 7f2859fca700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:54.480+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T15:54:54.684 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:54.678+0000 7f7e490fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:54:54.838 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:54.836+0000 7f124c8fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:54:54.918 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:54.910+0000 7f7e490fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:55.017 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.010+0000 7f7e490fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:54:55.066 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.064+0000 7f124c8fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:54:55.091 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.086+0000 7f7e490fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:54:55.158 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.156+0000 7f124c8fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:54:55.177 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.166+0000 7f7e490fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:54:55.240 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.236+0000 7f124c8fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:54:55.329 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.324+0000 7f124c8fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:54:55.408 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.402+0000 7f7e490fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:55.491 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.482+0000 7f7e490fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:54:55.575 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.572+0000 7f124c8fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:54:55.585 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.578+0000 7f7e490fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:54:55.668 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.664+0000 7f124c8fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:54:55.765 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.764+0000 7f124c8fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:54:55.790 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.782+0000 7f7e490fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:54:55.881 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:55.874+0000 7f7e490fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:54:55.977 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:55.976+0000 7f124c8fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:54:56.075 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:56.072+0000 7f124c8fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:54:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:56.084+0000 7f2859fca700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:54:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:56.084+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T15:54:56.615 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.610+0000 7f7e490fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:54:56.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:56.632+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 70) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f284c080030 con 0x7f28540cac50 2023-12-19T15:54:56.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:56.632+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:54:56.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:56.632+0000 7f2849ffb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:56.825 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e08fcc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.825 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e08fcc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e08fcc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e0c7d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e08fcc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e0c7d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e0c7d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e0c7d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e0c7d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.826 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:56.818+0000 7f7e08fcc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:54:56.918 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:56.916+0000 7f124c8fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:54:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.640+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 71) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f284c0102e0 con 0x7f28540cac50 2023-12-19T15:54:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.640+0000 7f2849ffb700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.640+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f285404be40 con 0x7f28380606c0 2023-12-19T15:54:57.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.640+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.644+0000 7f2859fca700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062af0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2850007fd0 tx=0x7f2850009040 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:57.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.748+0000 7f2849ffb700 1 -- 172.21.15.50:0/3004440390 <== mgr.8878 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f285404be40 con 0x7f28380606c0 2023-12-19T15:54:57.749 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 msgr2=0x7f2838062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062af0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2850007fd0 tx=0x7f2850009040 comp rx=0 tx=0).stop 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 msgr2=0x7f28540c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c910 tx=0x7f284c0027a0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 shutdown_connections 2023-12-19T15:54:57.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28540af190 0x7f28540c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f28380606c0 0x7f2838062af0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28540cac50 0x7f28540c8290 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 --2- 172.21.15.50:0/3004440390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28540aedc0 0x7f285415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 >> 172.21.15.50:0/3004440390 conn(0x7f28540cc760 msgr2=0x7f28540afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 shutdown_connections 2023-12-19T15:54:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.752+0000 7f285afcc700 1 -- 172.21.15.50:0/3004440390 wait complete. 2023-12-19T15:54:57.779 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T15:54:57.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T15:54:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.952+0000 7fe1ffd45700 1 Processor -- start 2023-12-19T15:54:57.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.952+0000 7fe1ffd45700 1 -- start start 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1f80cb130 0x7fe1f80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 0x7fe1f806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f806b030 con 0x7fe1f815c220 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f815cbe0 con 0x7fe1f80cb130 2023-12-19T15:54:57.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1ffd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80b0ac0 con 0x7fe1f806a6a0 2023-12-19T15:54:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1fe2e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1fe2e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50680/0 (socket says 172.21.15.50:50680) 2023-12-19T15:54:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1fe2e2700 1 -- 172.21.15.50:0/552155552 learned_addr learned my addr 172.21.15.50:0/552155552 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1fdae1700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.125:3300/0 conn(0x7fe1f80cb130 0x7fe1f80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.956+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/552155552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 0x7fe1f806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 -- 172.21.15.50:0/552155552 >> v2:172.21.15.125:3300/0 conn(0x7fe1f80cb130 msgr2=0x7fe1f80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.125:3300/0 conn(0x7fe1f80cb130 0x7fe1f80cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 -- 172.21.15.50:0/552155552 >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 msgr2=0x7fe1f815c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 -- 172.21.15.50:0/552155552 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f806b210 con 0x7fe1f806a6a0 2023-12-19T15:54:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fe2e2700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/552155552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 0x7fe1f806aaf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe1f806b400 tx=0x7fe1e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc6e7ae580d6cbe8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fcadf700 1 -- 172.21.15.50:0/552155552 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011630 con 0x7fe1f806a6a0 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fcadf700 1 -- 172.21.15.50:0/552155552 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8011790 con 0x7fe1f806a6a0 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1fcadf700 1 -- 172.21.15.50:0/552155552 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011b50 con 0x7fe1f806a6a0 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 -- 172.21.15.50:0/552155552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 msgr2=0x7fe1f806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/552155552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 0x7fe1f806aaf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe1f806b400 tx=0x7fe1e80091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 -- 172.21.15.50:0/552155552 shutdown_connections 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.50:3300/0 conn(0x7fe1f815c220 0x7fe1f815c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/552155552 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f806a6a0 0x7fe1f806aaf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/552155552 >> v2:172.21.15.125:3300/0 conn(0x7fe1f80cb130 0x7fe1f80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 -- 172.21.15.50:0/552155552 >> 172.21.15.50:0/552155552 conn(0x7fe1f8067bb0 msgr2=0x7fe1f80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 -- 172.21.15.50:0/552155552 shutdown_connections 2023-12-19T15:54:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.960+0000 7fe1ffd45700 1 -- 172.21.15.50:0/552155552 wait complete. 2023-12-19T15:54:57.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 Processor -- start 2023-12-19T15:54:57.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 -- start start 2023-12-19T15:54:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 0x7fe1f815da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f815c220 0x7fe1f81595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f814e3a0 con 0x7fe1f806a6a0 2023-12-19T15:54:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f814d790 con 0x7fe1f80cb130 2023-12-19T15:54:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.964+0000 7fe1ffd45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f814da70 con 0x7fe1f815c220 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50694/0 (socket says 172.21.15.50:50694) 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 -- 172.21.15.50:0/2035529392 learned_addr learned my addr 172.21.15.50:0/2035529392 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 0x7fe1f815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fe2e2700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f815c220 0x7fe1f81595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 -- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f815c220 msgr2=0x7fe1f81595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f815c220 0x7fe1f81595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 -- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 msgr2=0x7fe1f815da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 0x7fe1f815da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 -- 172.21.15.50:0/2035529392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f0009ae0 con 0x7fe1f806a6a0 2023-12-19T15:54:57.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 0x7fe1f815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:54:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1fdae1700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400ec80 tx=0x7fe1f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f400c9d0 con 0x7fe1f806a6a0 2023-12-19T15:54:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f400cb30 con 0x7fe1f806a6a0 2023-12-19T15:54:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e8009420 con 0x7fe1f806a6a0 2023-12-19T15:54:57.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4013470 con 0x7fe1f806a6a0 2023-12-19T15:54:57.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.968+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f814e030 con 0x7fe1f806a6a0 2023-12-19T15:54:57.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.972+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8071120 con 0x7fe1f806a6a0 2023-12-19T15:54:57.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.972+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 71) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fe1f4013820 con 0x7fe1f806a6a0 2023-12-19T15:54:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.972+0000 7fe1edffb700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 0x7fe1dc062920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.976+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fe1f407cbc0 con 0x7fe1f806a6a0 2023-12-19T15:54:57.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.976+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 0x7fe1dc062920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:57.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.976+0000 7fe1fd2e0700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 0x7fe1dc062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009860 tx=0x7fe1e8014000 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:57.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:57.980+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe1f404d0c0 con 0x7fe1f806a6a0 2023-12-19T15:54:58.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.172+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1f814e570 con 0x7fe1dc060470 2023-12-19T15:54:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.284+0000 7fe1edffb700 1 -- 172.21.15.50:0/2035529392 <== mgr.8878 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe1f814e570 con 0x7fe1dc060470 2023-12-19T15:54:58.288 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path snap_test_dir1 2023-12-19T15:54:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 msgr2=0x7fe1dc062920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 0x7fe1dc062920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009860 tx=0x7fe1e8014000 comp rx=0 tx=0).stop 2023-12-19T15:54:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 msgr2=0x7fe1f815d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400ec80 tx=0x7fe1f40091a0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 shutdown_connections 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1f80cb130 0x7fe1f815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe1dc060470 0x7fe1dc062920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe1f815c220 0x7fe1f81595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 --2- 172.21.15.50:0/2035529392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe1f806a6a0 0x7fe1f815d4c0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.288+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 >> 172.21.15.50:0/2035529392 conn(0x7fe1f8067bb0 msgr2=0x7fe1f814e930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.292+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 shutdown_connections 2023-12-19T15:54:58.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.292+0000 7fe1ffd45700 1 -- 172.21.15.50:0/2035529392 wait complete. 2023-12-19T15:54:58.315 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T15:54:58.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T15:54:58.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.492+0000 7f3082923700 1 Processor -- start 2023-12-19T15:54:58.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.492+0000 7f3082923700 1 -- start start 2023-12-19T15:54:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082923700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 0x7f307c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082923700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f307c0cac50 0x7f307c0af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082923700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c06b3d0 con 0x7f307c06af50 2023-12-19T15:54:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082923700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c0cc140 con 0x7f307c0cac50 2023-12-19T15:54:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082923700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c0cb3f0 con 0x7f307c0aedc0 2023-12-19T15:54:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f307c0cac50 0x7f307c0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f307c0cac50 0x7f307c0af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50704/0 (socket says 172.21.15.50:50704) 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 -- 172.21.15.50:0/4029924216 learned_addr learned my addr 172.21.15.50:0/4029924216 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3081921700 1 --2- 172.21.15.50:0/4029924216 >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 0x7f307c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 -- 172.21.15.50:0/4029924216 >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 msgr2=0x7f307c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 --2- 172.21.15.50:0/4029924216 >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 0x7f307c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 -- 172.21.15.50:0/4029924216 >> v1:172.21.15.125:6789/0 conn(0x7f307c06af50 legacy=0x7f307c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 -- 172.21.15.50:0/4029924216 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c06d220 con 0x7f307c0cac50 2023-12-19T15:54:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3081921700 1 --2- 172.21.15.50:0/4029924216 >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 0x7f307c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f3082122700 1 --2- 172.21.15.50:0/4029924216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c0cac50 0x7f307c0af930 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f307c0cc6c0 tx=0x7f3074009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7811c1704b203f6d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f308091f700 1 -- 172.21.15.50:0/4029924216 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307400e070 con 0x7f307c0cac50 2023-12-19T15:54:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f308091f700 1 -- 172.21.15.50:0/4029924216 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307400ba10 con 0x7f307c0cac50 2023-12-19T15:54:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.496+0000 7f308091f700 1 -- 172.21.15.50:0/4029924216 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740092e0 con 0x7f307c0cac50 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- 172.21.15.50:0/4029924216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c0cac50 msgr2=0x7f307c0af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- 172.21.15.50:0/4029924216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c0cac50 0x7f307c0af930 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f307c0cc6c0 tx=0x7f3074009840 comp rx=0 tx=0).stop 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- 172.21.15.50:0/4029924216 shutdown_connections 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- 172.21.15.50:0/4029924216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c0cac50 0x7f307c0af930 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- 172.21.15.50:0/4029924216 >> v2:172.21.15.125:3300/0 conn(0x7f307c0aedc0 0x7f307c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- 172.21.15.50:0/4029924216 >> 172.21.15.50:0/4029924216 conn(0x7f307c0cc760 msgr2=0x7f307c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- 172.21.15.50:0/4029924216 shutdown_connections 2023-12-19T15:54:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- 172.21.15.50:0/4029924216 wait complete. 2023-12-19T15:54:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 Processor -- start 2023-12-19T15:54:58.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- start start 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c06af50 0x7f307c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f307c0aedc0 0x7f307c0ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f307c0cac50 0x7f307c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c16c5f0 con 0x7f307c06af50 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.500+0000 7f3082923700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c16c7d0 con 0x7f307c0aedc0 2023-12-19T15:54:58.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3082923700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c16cab0 con 0x7f307c0cac50 2023-12-19T15:54:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c06af50 0x7f307c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c06af50 0x7f307c0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50718/0 (socket says 172.21.15.50:50718) 2023-12-19T15:54:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 -- 172.21.15.50:0/827706150 learned_addr learned my addr 172.21.15.50:0/827706150 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081120700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f307c0aedc0 0x7f307c0ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3082122700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f307c0cac50 0x7f307c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 -- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f307c0cac50 msgr2=0x7f307c0ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f307c0cac50 0x7f307c0ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 -- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f307c0aedc0 msgr2=0x7f307c0ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f307c0aedc0 0x7f307c0ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 -- 172.21.15.50:0/827706150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3074009ae0 con 0x7f307c06af50 2023-12-19T15:54:58.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3081921700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f307c06af50 0x7f307c0b16b0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f307800c830 tx=0x7f3078008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078011600 con 0x7f307c06af50 2023-12-19T15:54:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078011760 con 0x7f307c06af50 2023-12-19T15:54:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078011b20 con 0x7f307c06af50 2023-12-19T15:54:58.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3082923700 1 -- 172.21.15.50:0/827706150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c16cd90 con 0x7f307c06af50 2023-12-19T15:54:58.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.504+0000 7f3082923700 1 -- 172.21.15.50:0/827706150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c16d350 con 0x7f307c06af50 2023-12-19T15:54:58.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.508+0000 7f3082923700 1 -- 172.21.15.50:0/827706150 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c06a940 con 0x7f307c06af50 2023-12-19T15:54:58.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.508+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 71) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f3078018360 con 0x7f307c06af50 2023-12-19T15:54:58.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.508+0000 7f3071ffb700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f30600603f0 0x7f30600628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:58.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.508+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f307807c6c0 con 0x7f307c06af50 2023-12-19T15:54:58.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.512+0000 7f3081120700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f30600603f0 0x7f30600628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:58.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.512+0000 7f3081120700 1 --2- 172.21.15.50:0/827706150 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f30600603f0 0x7f30600628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f307c06ba90 tx=0x7f306c009060 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:58.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.516+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f307c06a940 con 0x7f307c06af50 2023-12-19T15:54:58.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.696+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 72) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f307804d060 con 0x7f307c06af50 2023-12-19T15:54:58.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.712+0000 7f3082923700 1 -- 172.21.15.50:0/827706150 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f307c06a940 con 0x7f30600603f0 2023-12-19T15:54:58.724 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:58.718+0000 7f7e137e1700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T15:54:58.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:58.720+0000 7f3071ffb700 1 -- 172.21.15.50:0/827706150 <== mgr.8878 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f307c06a940 con 0x7f30600603f0 2023-12-19T15:54:58.727 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T15:54:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T15:54:58.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:54:58.741 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T15:54:58.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T15:54:59.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 Processor -- start 2023-12-19T15:54:59.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 -- start start 2023-12-19T15:54:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd78c06db40 0x7fd78c0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c0c9db0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c0ca950 con 0x7fd78c06db40 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd7910f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0cb2e0 con 0x7fd78c06df10 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd78ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd78c06db40 0x7fd78c0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd78ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd78c06db40 0x7fd78c0cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50730/0 (socket says 172.21.15.50:50730) 2023-12-19T15:54:59.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.000+0000 7fd78ad9d700 1 -- 172.21.15.50:0/1392464834 learned_addr learned my addr 172.21.15.50:0/1392464834 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78b59e700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78ad9d700 1 -- 172.21.15.50:0/1392464834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 msgr2=0x7fd78c159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78ad9d700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78ad9d700 1 -- 172.21.15.50:0/1392464834 >> v1:172.21.15.125:6789/0 conn(0x7fd78c0c63f0 legacy=0x7fd78c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78ad9d700 1 -- 172.21.15.50:0/1392464834 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0cb5c0 con 0x7fd78c06db40 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78b59e700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:54:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd78ad9d700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c06db40 0x7fd78c0cc3e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0099b0 tx=0x7fd77c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee999d7794126bf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:59.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd789d9b700 1 -- 172.21.15.50:0/1392464834 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00b860 con 0x7fd78c06db40 2023-12-19T15:54:59.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd789d9b700 1 -- 172.21.15.50:0/1392464834 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c00b9c0 con 0x7fd78c06db40 2023-12-19T15:54:59.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd789d9b700 1 -- 172.21.15.50:0/1392464834 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00bd60 con 0x7fd78c06db40 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 -- 172.21.15.50:0/1392464834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c06db40 msgr2=0x7fd78c0cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c06db40 0x7fd78c0cc3e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0099b0 tx=0x7fd77c009250 comp rx=0 tx=0).stop 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 -- 172.21.15.50:0/1392464834 shutdown_connections 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 --2- 172.21.15.50:0/1392464834 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c06db40 0x7fd78c0cc3e0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 -- 172.21.15.50:0/1392464834 >> 172.21.15.50:0/1392464834 conn(0x7fd78c067bb0 msgr2=0x7fd78c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 -- 172.21.15.50:0/1392464834 shutdown_connections 2023-12-19T15:54:59.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.004+0000 7fd7910f0700 1 -- 172.21.15.50:0/1392464834 wait complete. 2023-12-19T15:54:59.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 Processor -- start 2023-12-19T15:54:59.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 -- start start 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd78c06db40 0x7fd78c13f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0c4d10 con 0x7fd78c0c63f0 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0c4420 con 0x7fd78c06db40 2023-12-19T15:54:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd7910f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c0c46c0 con 0x7fd78c06df10 2023-12-19T15:54:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd78b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd78b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50744/0 (socket says 172.21.15.50:50744) 2023-12-19T15:54:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd78b59e700 1 -- 172.21.15.50:0/61445563 learned_addr learned my addr 172.21.15.50:0/61445563 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:54:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd78ad9d700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd78c06db40 0x7fd78c13f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.008+0000 7fd78a59c700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 -- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 msgr2=0x7fd78c140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c140070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 -- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd78c06db40 msgr2=0x7fd78c13f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd78c06db40 0x7fd78c13f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 -- 172.21.15.50:0/61445563 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c0094f0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd78b59e700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd78400cfc0 tx=0x7fd7840078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78400aea0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784015440 con 0x7fd78c0c63f0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c0c49a0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784015800 con 0x7fd78c0c63f0 2023-12-19T15:54:59.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c1503f0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.012+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c0c4ec0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.016+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 72) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fd78400d440 con 0x7fd78c0c63f0 2023-12-19T15:54:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.016+0000 7fd772ffd700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 0x7fd778062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:54:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.016+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fd78407dab0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.016+0000 7fd78ad9d700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 0x7fd778062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:54:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.016+0000 7fd78ad9d700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 0x7fd778062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0099b0 tx=0x7fd77c017000 comp rx=0 tx=0).ready entity=mgr.8878 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:54:59.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.020+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd78c0c4ec0 con 0x7fd78c0c63f0 2023-12-19T15:54:59.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.236+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd78c0c4e50 con 0x7fd78c0c63f0 2023-12-19T15:54:59.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.732+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fd78404de80 con 0x7fd78c0c63f0 2023-12-19T15:54:59.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.736+0000 7fd772ffd700 1 -- 172.21.15.50:0/61445563 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v73) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd784051770 con 0x7fd78c0c63f0 2023-12-19T15:54:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 msgr2=0x7fd778062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 0x7fd778062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0099b0 tx=0x7fd77c017000 comp rx=0 tx=0).stop 2023-12-19T15:54:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 msgr2=0x7fd78c1518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:54:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd78400cfc0 tx=0x7fd7840078d0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 shutdown_connections 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd778060750 0x7fd778062c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd78c0c63f0 0x7fd78c1518d0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd78c06df10 0x7fd78c140070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 --2- 172.21.15.50:0/61445563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd78c06db40 0x7fd78c13f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 >> 172.21.15.50:0/61445563 conn(0x7fd78c067bb0 msgr2=0x7fd78c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 shutdown_connections 2023-12-19T15:54:59.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:54:59.740+0000 7fd7910f0700 1 -- 172.21.15.50:0/61445563 wait complete. 2023-12-19T15:54:59.762 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T15:54:59.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:59.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:54:59.842 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T15:54:59.842 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T15:54:59.842 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T15:54:59.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T15:54:59.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[56613]: fuse finished with error 0 and tester_r 0 2023-12-19T15:54:59.879 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T15:54:59.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:54:59.879 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T15:54:59.939 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:54:59.936+0000 7f28349ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:54:59.953 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:54:59.946+0000 7f103dae7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:55:00.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T15:55:00.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:00.012 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T15:55:00.025 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T15:55:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:55:00.026 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T15:55:00.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:00.026 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T15:55:00.048 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T15:55:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:55:00.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T15:55:00.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:00.049 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T15:55:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T15:55:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T15:55:00.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:00.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 Processor -- start 2023-12-19T15:55:00.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 -- start start 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faa7806b7a0 0x7faa780c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 0x7faa7806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7806c7c0 con 0x7faa78158d60 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7806caa0 con 0x7faa7806b7a0 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7da0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa780c55a0 con 0x7faa780c51b0 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faa7806b7a0 0x7faa780c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faa7806b7a0 0x7faa780c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50774/0 (socket says 172.21.15.50:50774) 2023-12-19T15:55:00.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 -- 172.21.15.50:0/3339281154 learned_addr learned my addr 172.21.15.50:0/3339281154 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:00.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7d20b700 1 --2- 172.21.15.50:0/3339281154 >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 0x7faa7806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 -- 172.21.15.50:0/3339281154 >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 msgr2=0x7faa7806cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 --2- 172.21.15.50:0/3339281154 >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 0x7faa7806cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 -- 172.21.15.50:0/3339281154 >> v1:172.21.15.125:6789/0 conn(0x7faa78158d60 legacy=0x7faa7815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 -- 172.21.15.50:0/3339281154 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7815d130 con 0x7faa7806b7a0 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa7d20b700 1 --2- 172.21.15.50:0/3339281154 >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 0x7faa7806cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa77fff700 1 --2- 172.21.15.50:0/3339281154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa7806b7a0 0x7faa780c4c70 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7faa7806bcf0 tx=0x7faa64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bde8e3ab147583 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.264+0000 7faa777fe700 1 -- 172.21.15.50:0/3339281154 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6400b820 con 0x7faa7806b7a0 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa777fe700 1 -- 172.21.15.50:0/3339281154 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6400b980 con 0x7faa7806b7a0 2023-12-19T15:55:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa777fe700 1 -- 172.21.15.50:0/3339281154 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6400bd20 con 0x7faa7806b7a0 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 -- 172.21.15.50:0/3339281154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa7806b7a0 msgr2=0x7faa780c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 --2- 172.21.15.50:0/3339281154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa7806b7a0 0x7faa780c4c70 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7faa7806bcf0 tx=0x7faa64009250 comp rx=0 tx=0).stop 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 -- 172.21.15.50:0/3339281154 shutdown_connections 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 --2- 172.21.15.50:0/3339281154 >> v2:172.21.15.125:3300/0 conn(0x7faa780c51b0 0x7faa7806cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 --2- 172.21.15.50:0/3339281154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa7806b7a0 0x7faa780c4c70 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 -- 172.21.15.50:0/3339281154 >> 172.21.15.50:0/3339281154 conn(0x7faa78067bb0 msgr2=0x7faa780cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:00.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 -- 172.21.15.50:0/3339281154 shutdown_connections 2023-12-19T15:55:00.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 -- 172.21.15.50:0/3339281154 wait complete. 2023-12-19T15:55:00.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.268+0000 7faa7da0c700 1 Processor -- start 2023-12-19T15:55:00.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- start start 2023-12-19T15:55:00.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faa7806b7a0 0x7faa78156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa78158d60 0x7faa780acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7815adb0 con 0x7faa780c51b0 2023-12-19T15:55:00.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7815a1a0 con 0x7faa78158d60 2023-12-19T15:55:00.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7815a480 con 0x7faa7806b7a0 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50782/0 (socket says 172.21.15.50:50782) 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 -- 172.21.15.50:0/2626651238 learned_addr learned my addr 172.21.15.50:0/2626651238 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7d20b700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa78158d60 0x7faa780acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7ca0a700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faa7806b7a0 0x7faa78156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faa7806b7a0 msgr2=0x7faa78156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faa7806b7a0 0x7faa78156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa78158d60 msgr2=0x7faa780acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa78158d60 0x7faa780acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 -- 172.21.15.50:0/2626651238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa640094f0 con 0x7faa780c51b0 2023-12-19T15:55:00.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa77fff700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7faa640030c0 tx=0x7faa64016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64011e60 con 0x7faa780c51b0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64020070 con 0x7faa780c51b0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7815a760 con 0x7faa780c51b0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6401b6a0 con 0x7faa780c51b0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.272+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa78156890 con 0x7faa780c51b0 2023-12-19T15:55:00.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.276+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7804be40 con 0x7faa780c51b0 2023-12-19T15:55:00.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.276+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7faa64022030 con 0x7faa780c51b0 2023-12-19T15:55:00.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.276+0000 7faa74ff9700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 0x7faa5c062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.276+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7faa6407eda0 con 0x7faa780c51b0 2023-12-19T15:55:00.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.280+0000 7faa7ca0a700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 msgr2=0x7faa5c062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:00.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.280+0000 7faa7ca0a700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 0x7faa5c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:00.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.280+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faa640aebe0 con 0x7faa780c51b0 2023-12-19T15:55:00.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.484+0000 7faa7ca0a700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 msgr2=0x7faa5c062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:00.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.484+0000 7faa7ca0a700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 0x7faa5c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:00.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.488+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa7804be40 con 0x7faa780c51b0 2023-12-19T15:55:00.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa74ff9700 1 -- 172.21.15.50:0/2626651238 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7faa6404eda0 con 0x7faa780c51b0 2023-12-19T15:55:00.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:00.494 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 msgr2=0x7faa5c062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 0x7faa5c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 msgr2=0x7faa78157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7faa640030c0 tx=0x7faa64016000 comp rx=0 tx=0).stop 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 shutdown_connections 2023-12-19T15:55:00.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa78158d60 0x7faa780acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7faa5c060730 0x7faa5c062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa780c51b0 0x7faa78157dd0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 --2- 172.21.15.50:0/2626651238 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faa7806b7a0 0x7faa78156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 >> 172.21.15.50:0/2626651238 conn(0x7faa78067bb0 msgr2=0x7faa78153510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 shutdown_connections 2023-12-19T15:55:00.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.492+0000 7faa7da0c700 1 -- 172.21.15.50:0/2626651238 wait complete. 2023-12-19T15:55:00.498 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:55:00.505 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:00.498+0000 7f103dae7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:55:00.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:00.525 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:00.520+0000 7f28349ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:55:00.583 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:00.578+0000 7f103dae7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:55:00.601 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:00.600+0000 7f28349ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:55:00.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 Processor -- start 2023-12-19T15:55:00.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 -- start start 2023-12-19T15:55:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe6ec0b1410 0x7fe6ec0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec06c7c0 con 0x7fe6ec15a860 2023-12-19T15:55:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ec06af40 con 0x7fe6ec0b1410 2023-12-19T15:55:00.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f2363700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ec06cd80 con 0x7fe6ec06a580 2023-12-19T15:55:00.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f1361700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe6ec0b1410 0x7fe6ec0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f0b60700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f0b60700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:51220/0 (socket says 172.21.15.50:51220) 2023-12-19T15:55:00.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f0b60700 1 -- 172.21.15.50:0/379143714 learned_addr learned my addr 172.21.15.50:0/379143714 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:00.680 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.676+0000 7fe6f1b62700 1 --1- 172.21.15.50:0/379143714 >> v1:172.21.15.125:6789/0 conn(0x7fe6ec15a860 0x7fe6ec15ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:57304/0 (socket says 172.21.15.50:57304) 2023-12-19T15:55:00.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f1361700 1 -- 172.21.15.50:0/379143714 >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 msgr2=0x7fe6ec06aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:00.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f1361700 1 --2- 172.21.15.50:0/379143714 >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f1361700 1 -- 172.21.15.50:0/379143714 >> v1:172.21.15.125:6789/0 conn(0x7fe6ec15a860 legacy=0x7fe6ec15ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f1361700 1 -- 172.21.15.50:0/379143714 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec06b8f0 con 0x7fe6ec0b1410 2023-12-19T15:55:00.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f0b60700 1 --2- 172.21.15.50:0/379143714 >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:00.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f1361700 1 --2- 172.21.15.50:0/379143714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec0b1410 0x7fe6ec0ab440 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec068040 tx=0x7fe6e800d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f7b47fc8fb2440 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:00.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6e3fff700 1 -- 172.21.15.50:0/379143714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8012070 con 0x7fe6ec0b1410 2023-12-19T15:55:00.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6e3fff700 1 -- 172.21.15.50:0/379143714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e800d2e0 con 0x7fe6ec0b1410 2023-12-19T15:55:00.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6e3fff700 1 -- 172.21.15.50:0/379143714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e800f9b0 con 0x7fe6ec0b1410 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 -- 172.21.15.50:0/379143714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec0b1410 msgr2=0x7fe6ec0ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 --2- 172.21.15.50:0/379143714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec0b1410 0x7fe6ec0ab440 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec068040 tx=0x7fe6e800d630 comp rx=0 tx=0).stop 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 -- 172.21.15.50:0/379143714 shutdown_connections 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 --2- 172.21.15.50:0/379143714 >> v2:172.21.15.125:3300/0 conn(0x7fe6ec06a580 0x7fe6ec06aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 --2- 172.21.15.50:0/379143714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec0b1410 0x7fe6ec0ab440 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 -- 172.21.15.50:0/379143714 >> 172.21.15.50:0/379143714 conn(0x7fe6ec067bb0 msgr2=0x7fe6ec0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:00.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 -- 172.21.15.50:0/379143714 shutdown_connections 2023-12-19T15:55:00.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.680+0000 7fe6f2363700 1 -- 172.21.15.50:0/379143714 wait complete. 2023-12-19T15:55:00.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 Processor -- start 2023-12-19T15:55:00.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 -- start start 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6ec0b1410 0x7fe6ec0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6ec15a860 0x7fe6ec145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec0a5180 con 0x7fe6ec06a580 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec0a4570 con 0x7fe6ec15a860 2023-12-19T15:55:00.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f2363700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec0a4850 con 0x7fe6ec0b1410 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50798/0 (socket says 172.21.15.50:50798) 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 -- 172.21.15.50:0/52474017 learned_addr learned my addr 172.21.15.50:0/52474017 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1b62700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6ec15a860 0x7fe6ec145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f0b60700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6ec0b1410 0x7fe6ec0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 -- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6ec0b1410 msgr2=0x7fe6ec0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6ec0b1410 0x7fe6ec0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 -- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6ec15a860 msgr2=0x7fe6ec145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6ec15a860 0x7fe6ec145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.684+0000 7fe6f1361700 1 -- 172.21.15.50:0/52474017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc0094f0 con 0x7fe6ec06a580 2023-12-19T15:55:00.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6f1361700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800d670 tx=0x7fe6e801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:00.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e800c8c0 con 0x7fe6ec06a580 2023-12-19T15:55:00.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e800ca60 con 0x7fe6ec06a580 2023-12-19T15:55:00.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e800ce20 con 0x7fe6ec06a580 2023-12-19T15:55:00.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e800d8d0 con 0x7fe6ec06a580 2023-12-19T15:55:00.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ec0a4e10 con 0x7fe6ec06a580 2023-12-19T15:55:00.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.688+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ec04be40 con 0x7fe6ec06a580 2023-12-19T15:55:00.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.692+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fe6e8020410 con 0x7fe6ec06a580 2023-12-19T15:55:00.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.692+0000 7fe6e17fa700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 0x7fe6d0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:00.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.692+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fe6e8082440 con 0x7fe6ec06a580 2023-12-19T15:55:00.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.692+0000 7fe6f0b60700 1 -- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 msgr2=0x7fe6d0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:00.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.692+0000 7fe6f0b60700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 0x7fe6d0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:00.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.696+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe6e8052330 con 0x7fe6ec06a580 2023-12-19T15:55:00.721 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:00.714+0000 7f103dae7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:00.773 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:00.768+0000 7f28349ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:00.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.880+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6ec0ab220 con 0x7fe6ec06a580 2023-12-19T15:55:00.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.880+0000 7fe6e17fa700 1 -- 172.21.15.50:0/52474017 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16153 (secure 0 0 0) 0x7fe6e8055950 con 0x7fe6ec06a580 2023-12-19T15:55:00.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:00.884 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":168,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:56.619651+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1152230135":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/373489315":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/669887858":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/219377965":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/2706791219":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/1963925549":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6833/33827":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6832/33827":"2023-12-20T15:54:56.619577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:00.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 msgr2=0x7fe6d0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:00.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 0x7fe6d0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 msgr2=0x7fe6ec0a7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:00.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800d670 tx=0x7fe6e801c000 comp rx=0 tx=0).stop 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 shutdown_connections 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6ec15a860 0x7fe6ec145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe6d0060700 0x7fe6d0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe6ec0b1410 0x7fe6ec0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 --2- 172.21.15.50:0/52474017 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6ec06a580 0x7fe6ec0a7750 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 >> 172.21.15.50:0/52474017 conn(0x7fe6ec067bb0 msgr2=0x7fe6ec156220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 shutdown_connections 2023-12-19T15:55:00.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:00.884+0000 7fe6f2363700 1 -- 172.21.15.50:0/52474017 wait complete. 2023-12-19T15:55:00.907 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T15:54:38.835585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T15:54:40.257787+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T15:55:00.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:55:01.001 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:00.994+0000 7f103dae7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:55:01.071 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:01.068+0000 7f28349ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:55:01.076 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:01.070+0000 7f103dae7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:01.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.084+0000 7f5560b85700 1 Processor -- start 2023-12-19T15:55:01.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.084+0000 7f5560b85700 1 -- start start 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5560b85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f555c158e20 0x7f555c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5560b85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f555c0cb970 0x7f555c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5560b85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c0cbdf0 con 0x7f555c0ca2c0 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5560b85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c0cc0d0 con 0x7f555c0cb970 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5560b85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c0cc3b0 con 0x7f555c158e20 2023-12-19T15:55:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5559d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f555c0cb970 0x7f555c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f555a59c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f555c158e20 0x7f555c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5559d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f555c0cb970 0x7f555c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50806/0 (socket says 172.21.15.50:50806) 2023-12-19T15:55:01.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.088+0000 7f5559d9b700 1 -- 172.21.15.50:0/68637228 learned_addr learned my addr 172.21.15.50:0/68637228 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5559d9b700 1 -- 172.21.15.50:0/68637228 >> v2:172.21.15.125:3300/0 conn(0x7f555c158e20 msgr2=0x7f555c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5559d9b700 1 --2- 172.21.15.50:0/68637228 >> v2:172.21.15.125:3300/0 conn(0x7f555c158e20 0x7f555c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5559d9b700 1 -- 172.21.15.50:0/68637228 >> v1:172.21.15.50:6789/0 conn(0x7f555c0ca2c0 legacy=0x7f555c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5559d9b700 1 -- 172.21.15.50:0/68637228 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0cc690 con 0x7f555c0cb970 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5559d9b700 1 --2- 172.21.15.50:0/68637228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c0cb970 0x7f555c0ceb00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f55500099b0 tx=0x7f5550009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b900af00d3c46e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f555959a700 1 -- 172.21.15.50:0/68637228 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550009ea0 con 0x7f555c0cb970 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f555959a700 1 -- 172.21.15.50:0/68637228 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555000b820 con 0x7f555c0cb970 2023-12-19T15:55:01.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f555959a700 1 -- 172.21.15.50:0/68637228 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555000bbc0 con 0x7f555c0cb970 2023-12-19T15:55:01.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 -- 172.21.15.50:0/68637228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c0cb970 msgr2=0x7f555c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 --2- 172.21.15.50:0/68637228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c0cb970 0x7f555c0ceb00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f55500099b0 tx=0x7f5550009250 comp rx=0 tx=0).stop 2023-12-19T15:55:01.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 -- 172.21.15.50:0/68637228 shutdown_connections 2023-12-19T15:55:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 --2- 172.21.15.50:0/68637228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c0cb970 0x7f555c0ceb00 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 --2- 172.21.15.50:0/68637228 >> v2:172.21.15.125:3300/0 conn(0x7f555c158e20 0x7f555c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 -- 172.21.15.50:0/68637228 >> 172.21.15.50:0/68637228 conn(0x7f555c067bb0 msgr2=0x7f555c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 -- 172.21.15.50:0/68637228 shutdown_connections 2023-12-19T15:55:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.092+0000 7f5560b85700 1 -- 172.21.15.50:0/68637228 wait complete. 2023-12-19T15:55:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 Processor -- start 2023-12-19T15:55:01.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 -- start start 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f555c0ca2c0 0x7f555c169320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f555c158e20 0x7f555c0c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c0b3580 con 0x7f555c14e2e0 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c14e780 con 0x7f555c0ca2c0 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5560b85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c14ea40 con 0x7f555c158e20 2023-12-19T15:55:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50810/0 (socket says 172.21.15.50:50810) 2023-12-19T15:55:01.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 -- 172.21.15.50:0/2719365661 learned_addr learned my addr 172.21.15.50:0/2719365661 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:01.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f5559d9b700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f555c158e20 0x7f555c0c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555a59c700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f555c0ca2c0 0x7f555c169320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 -- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f555c158e20 msgr2=0x7f555c0c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f555c158e20 0x7f555c0c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 -- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f555c0ca2c0 msgr2=0x7f555c169320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f555c0ca2c0 0x7f555c169320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 -- 172.21.15.50:0/2719365661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500094f0 con 0x7f555c14e2e0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.096+0000 7f555ad9d700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f555400c8e0 tx=0x7f5554001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554012630 con 0x7f555c14e2e0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5554012790 con 0x7f555c14e2e0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554012b50 con 0x7f555c14e2e0 2023-12-19T15:55:01.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c14ed20 con 0x7f555c14e2e0 2023-12-19T15:55:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c14f2e0 con 0x7f555c14e2e0 2023-12-19T15:55:01.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c0b36f0 con 0x7f555c14e2e0 2023-12-19T15:55:01.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.100+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f555401a3a0 con 0x7f555c14e2e0 2023-12-19T15:55:01.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.104+0000 7f554a7fc700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 0x7f553c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.104+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f555407cfd0 con 0x7f555c14e2e0 2023-12-19T15:55:01.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.104+0000 7f555a59c700 1 -- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 msgr2=0x7f553c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:01.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.104+0000 7f555a59c700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 0x7f553c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:01.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.108+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5554050a60 con 0x7f555c14e2e0 2023-12-19T15:55:01.151 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:01.148+0000 7f28349ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:01.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.288+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f555c071120 con 0x7f555c14e2e0 2023-12-19T15:55:01.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.288+0000 7f554a7fc700 1 -- 172.21.15.50:0/2719365661 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55540505f0 con 0x7f555c14e2e0 2023-12-19T15:55:01.292 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 msgr2=0x7f553c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 0x7f553c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 msgr2=0x7f555c16ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f555400c8e0 tx=0x7f5554001710 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 shutdown_connections 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f553c060700 0x7f553c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f555c14e2e0 0x7f555c16ce40 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f555c158e20 0x7f555c0c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 --2- 172.21.15.50:0/2719365661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f555c0ca2c0 0x7f555c169320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 >> 172.21.15.50:0/2719365661 conn(0x7f555c067bb0 msgr2=0x7f555c14b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 shutdown_connections 2023-12-19T15:55:01.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.292+0000 7f5560b85700 1 -- 172.21.15.50:0/2719365661 wait complete. 2023-12-19T15:55:01.301 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:01.294+0000 7f103dae7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:55:01.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T15:55:01.391 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:01.388+0000 7f28349ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:55:01.392 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:01.386+0000 7f103dae7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:55:01.481 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:01.474+0000 7f103dae7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:55:01.499 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:01.496+0000 7f28349ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:55:01.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.512+0000 7f7c4f8ae700 1 Processor -- start 2023-12-19T15:55:01.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.512+0000 7f7c4f8ae700 1 -- start start 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.512+0000 7f7c4f8ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c480b1ed0 0x7f7c480684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48067790 con 0x7f7c48159790 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48068030 con 0x7f7c480b0cb0 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c480b1200 con 0x7f7c480b1ed0 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c480b1ed0 0x7f7c480684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c480b1ed0 0x7f7c480684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50824/0 (socket says 172.21.15.50:50824) 2023-12-19T15:55:01.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3155813392 learned_addr learned my addr 172.21.15.50:0/3155813392 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3155813392 >> v1:172.21.15.125:6789/0 conn(0x7f7c480b0cb0 legacy=0x7f7c48159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3155813392 >> v1:172.21.15.50:6789/0 conn(0x7f7c48159790 legacy=0x7f7c480b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3155813392 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480679e0 con 0x7f7c480b1ed0 2023-12-19T15:55:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4ce49700 1 --2- 172.21.15.50:0/3155813392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b1ed0 0x7f7c480684c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7c380099b0 tx=0x7f7c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28190e33e9836376 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c3ffff700 1 -- 172.21.15.50:0/3155813392 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800b820 con 0x7f7c480b1ed0 2023-12-19T15:55:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c3ffff700 1 -- 172.21.15.50:0/3155813392 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800b980 con 0x7f7c480b1ed0 2023-12-19T15:55:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c3ffff700 1 -- 172.21.15.50:0/3155813392 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800bd40 con 0x7f7c480b1ed0 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3155813392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b1ed0 msgr2=0x7f7c480684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3155813392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b1ed0 0x7f7c480684c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7c380099b0 tx=0x7f7c38009250 comp rx=0 tx=0).stop 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3155813392 shutdown_connections 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3155813392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b1ed0 0x7f7c480684c0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3155813392 >> 172.21.15.50:0/3155813392 conn(0x7f7c480cdef0 msgr2=0x7f7c480ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3155813392 shutdown_connections 2023-12-19T15:55:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.516+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3155813392 wait complete. 2023-12-19T15:55:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.520+0000 7f7c4f8ae700 1 Processor -- start 2023-12-19T15:55:01.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.520+0000 7f7c4f8ae700 1 -- start start 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c480b1ed0 0x7f7c48155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c48159790 0x7f7c4816cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c481561e0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48155c20 con 0x7f7c480b1ed0 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48155f00 con 0x7f7c48159790 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50830/0 (socket says 172.21.15.50:50830) 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 -- 172.21.15.50:0/3302619725 learned_addr learned my addr 172.21.15.50:0/3302619725 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4de4b700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c48159790 0x7f7c4816cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4ce49700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c480b1ed0 0x7f7c48155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c48159790 msgr2=0x7f7c4816cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c48159790 0x7f7c4816cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c480b1ed0 msgr2=0x7f7c48155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c480b1ed0 0x7f7c48155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 -- 172.21.15.50:0/3302619725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380094f0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4d64a700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c4e0 tx=0x7f7c4400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44011590 con 0x7f7c480b0cb0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c44011710 con 0x7f7c480b0cb0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c480c7580 con 0x7f7c480b0cb0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44011ad0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.524+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c481557a0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c480c7a80 con 0x7f7c480b0cb0 2023-12-19T15:55:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f7c4401b3b0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c3d7fa700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 0x7f7c2c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f7c4407c120 con 0x7f7c480b0cb0 2023-12-19T15:55:01.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 msgr2=0x7f7c2c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:01.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.528+0000 7f7c4ce49700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 0x7f7c2c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:01.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.536+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c4404c1c0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.623 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:01.616+0000 7f28349ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:55:01.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.732+0000 7f7c4ce49700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 msgr2=0x7f7c2c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:01.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.732+0000 7f7c4ce49700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 0x7f7c2c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:01.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.864+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c48071120 con 0x7f7c480b0cb0 2023-12-19T15:55:01.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.864+0000 7f7c3d7fa700 1 -- 172.21.15.50:0/3302619725 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c4404f7e0 con 0x7f7c480b0cb0 2023-12-19T15:55:01.867 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T15:55:01.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.868+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 msgr2=0x7f7c2c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:01.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.868+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 0x7f7c2c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.868+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 msgr2=0x7f7c4813b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:01.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.868+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c4e0 tx=0x7f7c4400c7f0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 shutdown_connections 2023-12-19T15:55:01.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c480b1ed0 0x7f7c48155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7c2c060700 0x7f7c2c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c48159790 0x7f7c4816cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 --2- 172.21.15.50:0/3302619725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c480b0cb0 0x7f7c4813b610 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:01.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 >> 172.21.15.50:0/3302619725 conn(0x7f7c480cdef0 msgr2=0x7f7c48158800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:01.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 shutdown_connections 2023-12-19T15:55:01.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:01.872+0000 7f7c4f8ae700 1 -- 172.21.15.50:0/3302619725 wait complete. 2023-12-19T15:55:01.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:01.882+0000 7f103dae7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:55:01.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T15:55:02.051 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.042+0000 7f103dae7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:55:02.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.064+0000 7f6f22bcc700 1 Processor -- start 2023-12-19T15:55:02.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.064+0000 7f6f22bcc700 1 -- start start 2023-12-19T15:55:02.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f22bcc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6f1c0af5b0 0x7f6f1c0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f22bcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1c0ae110 con 0x7f6f1c0ace10 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f22bcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1c06e0c0 con 0x7f6f1c0c8bf0 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f22bcc700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1c157110 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f21bca700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6f1c0ace10 0x7f6f1c0c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46622/0 (socket says 172.21.15.50:46622) 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f21bca700 1 -- 172.21.15.50:0/3424396407 learned_addr learned my addr 172.21.15.50:0/3424396407 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.068+0000 7f6f223cb700 1 --2- 172.21.15.50:0/3424396407 >> v2:172.21.15.50:3300/0 conn(0x7f6f1c0af5b0 0x7f6f1c0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f223cb700 1 -- 172.21.15.50:0/3424396407 >> v1:172.21.15.125:6789/0 conn(0x7f6f1c0c8bf0 legacy=0x7f6f1c06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f223cb700 1 -- 172.21.15.50:0/3424396407 >> v1:172.21.15.50:6789/0 conn(0x7f6f1c0ace10 legacy=0x7f6f1c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f223cb700 1 -- 172.21.15.50:0/3424396407 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c0ad7e0 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f223cb700 1 --2- 172.21.15.50:0/3424396407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c0ad1e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009e90 tx=0x7f6f14009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4b885ae89d62a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f20bc8700 1 -- 172.21.15.50:0/3424396407 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400e070 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f20bc8700 1 -- 172.21.15.50:0/3424396407 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1400bbf0 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f20bc8700 1 -- 172.21.15.50:0/3424396407 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14012430 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3424396407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0af5b0 msgr2=0x7f6f1c0ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3424396407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c0ad1e0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009e90 tx=0x7f6f14009780 comp rx=0 tx=0).stop 2023-12-19T15:55:02.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3424396407 shutdown_connections 2023-12-19T15:55:02.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3424396407 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c0ad1e0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3424396407 >> 172.21.15.50:0/3424396407 conn(0x7f6f1c0cdc00 msgr2=0x7f6f1c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:02.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3424396407 shutdown_connections 2023-12-19T15:55:02.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.072+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3424396407 wait complete. 2023-12-19T15:55:02.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 Processor -- start 2023-12-19T15:55:02.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 -- start start 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 0x7f6f1c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c16a5d0 con 0x7f6f1c0c8bf0 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c14d230 con 0x7f6f1c0af5b0 2023-12-19T15:55:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f22bcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c153d30 con 0x7f6f1c0ace10 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.076+0000 7f6f223cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 0x7f6f1c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f213c9700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f223cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50858/0 (socket says 172.21.15.50:50858) 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f223cb700 1 -- 172.21.15.50:0/3986319411 learned_addr learned my addr 172.21.15.50:0/3986319411 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 -- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f1c0af5b0 msgr2=0x7f6f1c06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 -- 172.21.15.50:0/3986319411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 msgr2=0x7f6f1c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 -- 172.21.15.50:0/3986319411 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f14009a20 con 0x7f6f1c0ace10 2023-12-19T15:55:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f223cb700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f21bca700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 0x7f6f1c154320 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c7c0 tx=0x7f6f18007eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18005b30 con 0x7f6f1c0ace10 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18005c90 con 0x7f6f1c0ace10 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18004a80 con 0x7f6f1c0ace10 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1c0675f0 con 0x7f6f1c0ace10 2023-12-19T15:55:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.080+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c16a2f0 con 0x7f6f1c0ace10 2023-12-19T15:55:02.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1c04be40 con 0x7f6f1c0ace10 2023-12-19T15:55:02.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f6f180070d0 con 0x7f6f1c0ace10 2023-12-19T15:55:02.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f11ffb700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 0x7f6f00062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6f1807c100 con 0x7f6f1c0ace10 2023-12-19T15:55:02.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f213c9700 1 -- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 msgr2=0x7f6f00062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:02.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.084+0000 7f6f213c9700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 0x7f6f00062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:02.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.092+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f18049be0 con 0x7f6f1c0ace10 2023-12-19T15:55:02.140 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:02.136+0000 7f28349ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:55:02.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.268+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6f1c0adac0 con 0x7f6f1c0ace10 2023-12-19T15:55:02.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.268+0000 7f6f11ffb700 1 -- 172.21.15.50:0/3986319411 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6f1804c100 con 0x7f6f1c0ace10 2023-12-19T15:55:02.270 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T15:55:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 msgr2=0x7f6f00062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 0x7f6f00062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 msgr2=0x7f6f1c154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 0x7f6f1c154320 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c7c0 tx=0x7f6f18007eb0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 shutdown_connections 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f1c0af5b0 0x7f6f1c06cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6f00060700 0x7f6f00062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f1c0c8bf0 0x7f6f1c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 --2- 172.21.15.50:0/3986319411 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6f1c0ace10 0x7f6f1c154320 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 >> 172.21.15.50:0/3986319411 conn(0x7f6f1c0cdc00 msgr2=0x7f6f1c14fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 shutdown_connections 2023-12-19T15:55:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.272+0000 7f6f22bcc700 1 -- 172.21.15.50:0/3986319411 wait complete. 2023-12-19T15:55:02.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:02.326 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:02.324+0000 7f28349ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:55:02.399 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.390+0000 7f103dae7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:55:02.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 Processor -- start 2023-12-19T15:55:02.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 -- start start 2023-12-19T15:55:02.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9541502b0 0x7ff954068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9540c6630 con 0x7ff9540c5900 2023-12-19T15:55:02.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9540c6910 con 0x7ff954067c90 2023-12-19T15:55:02.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff95b62e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9540c6bf0 con 0x7ff9541502b0 2023-12-19T15:55:02.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff958bc9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9541502b0 0x7ff954068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff958bc9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff9541502b0 0x7ff954068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50870/0 (socket says 172.21.15.50:50870) 2023-12-19T15:55:02.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff958bc9700 1 -- 172.21.15.50:0/1100297027 learned_addr learned my addr 172.21.15.50:0/1100297027 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:02.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.472+0000 7ff959bcb700 1 --1- 172.21.15.50:0/1100297027 >> v1:172.21.15.50:6789/0 conn(0x7ff9540c5900 0x7ff9540c5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46630/0 (socket says 172.21.15.50:46630) 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff958bc9700 1 -- 172.21.15.50:0/1100297027 >> v1:172.21.15.125:6789/0 conn(0x7ff954067c90 legacy=0x7ff95414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff958bc9700 1 -- 172.21.15.50:0/1100297027 >> v1:172.21.15.50:6789/0 conn(0x7ff9540c5900 legacy=0x7ff9540c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff958bc9700 1 -- 172.21.15.50:0/1100297027 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9540c6ed0 con 0x7ff9541502b0 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff94bfff700 1 -- 172.21.15.50:0/1100297027 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097348398 0 0) 0x7ff9540c6630 con 0x7ff9540c5900 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff958bc9700 1 --2- 172.21.15.50:0/1100297027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541502b0 0x7ff954068060 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff9440099b0 tx=0x7ff944009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3072d1df4296be1a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff94bfff700 1 -- 172.21.15.50:0/1100297027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400b860 con 0x7ff9541502b0 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff94bfff700 1 -- 172.21.15.50:0/1100297027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94400b9c0 con 0x7ff9541502b0 2023-12-19T15:55:02.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff94bfff700 1 -- 172.21.15.50:0/1100297027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400bd60 con 0x7ff9541502b0 2023-12-19T15:55:02.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 -- 172.21.15.50:0/1100297027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541502b0 msgr2=0x7ff954068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 --2- 172.21.15.50:0/1100297027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541502b0 0x7ff954068060 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff9440099b0 tx=0x7ff944009250 comp rx=0 tx=0).stop 2023-12-19T15:55:02.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 -- 172.21.15.50:0/1100297027 shutdown_connections 2023-12-19T15:55:02.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 --2- 172.21.15.50:0/1100297027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9541502b0 0x7ff954068060 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 -- 172.21.15.50:0/1100297027 >> 172.21.15.50:0/1100297027 conn(0x7ff9540cd6d0 msgr2=0x7ff95406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:02.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 -- 172.21.15.50:0/1100297027 shutdown_connections 2023-12-19T15:55:02.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.476+0000 7ff95b62e700 1 -- 172.21.15.50:0/1100297027 wait complete. 2023-12-19T15:55:02.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 Processor -- start 2023-12-19T15:55:02.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 -- start start 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 0x7ff954151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9541502b0 0x7ff95416c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9541522b0 con 0x7ff9540c5900 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff954152590 con 0x7ff954067c90 2023-12-19T15:55:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.480+0000 7ff95b62e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff954152870 con 0x7ff9541502b0 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff958bc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 0x7ff954151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff958bc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50874/0 (socket says 172.21.15.50:50874) 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff958bc9700 1 -- 172.21.15.50:0/2759331014 learned_addr learned my addr 172.21.15.50:0/2759331014 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff959bcb700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9541502b0 0x7ff95416c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9541502b0 msgr2=0x7ff95416c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9541502b0 0x7ff95416c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 msgr2=0x7ff9540c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 -- 172.21.15.50:0/2759331014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440094f0 con 0x7ff954067c90 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff958bc9700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9593ca700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 0x7ff954151910 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c590 tx=0x7ff950007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950011590 con 0x7ff954067c90 2023-12-19T15:55:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950011710 con 0x7ff954067c90 2023-12-19T15:55:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff954152b50 con 0x7ff954067c90 2023-12-19T15:55:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950011ad0 con 0x7ff954067c90 2023-12-19T15:55:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.484+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95416d490 con 0x7ff954067c90 2023-12-19T15:55:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff954071120 con 0x7ff954067c90 2023-12-19T15:55:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7ff95001b3b0 con 0x7ff954067c90 2023-12-19T15:55:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff9497fa700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 0x7ff938062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7ff95007c1b0 con 0x7ff954067c90 2023-12-19T15:55:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff958bc9700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 msgr2=0x7ff938062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.488+0000 7ff958bc9700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 0x7ff938062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:02.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.496+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff95004c1b0 con 0x7ff954067c90 2023-12-19T15:55:02.618 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.610+0000 7f103dae7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:02.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.692+0000 7ff958bc9700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 msgr2=0x7ff938062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:02.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.692+0000 7ff958bc9700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 0x7ff938062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:02.706 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.698+0000 7f103dae7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:55:02.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.708+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9540642a0 con 0x7ff954067c90 2023-12-19T15:55:02.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.708+0000 7ff9497fa700 1 -- 172.21.15.50:0/2759331014 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7ff95004f7d0 con 0x7ff954067c90 2023-12-19T15:55:02.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:02.711 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:55:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.712+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 msgr2=0x7ff938062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.712+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 0x7ff938062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.712+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 msgr2=0x7ff954151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.712+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 0x7ff954151910 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c590 tx=0x7ff950007ec0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 shutdown_connections 2023-12-19T15:55:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff938060700 0x7ff938062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9541502b0 0x7ff95416c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9540c5900 0x7ff9540c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 --2- 172.21.15.50:0/2759331014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff954067c90 0x7ff954151910 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 >> 172.21.15.50:0/2759331014 conn(0x7ff9540cd6d0 msgr2=0x7ff95406b130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 shutdown_connections 2023-12-19T15:55:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.716+0000 7ff95b62e700 1 -- 172.21.15.50:0/2759331014 wait complete. 2023-12-19T15:55:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:55:02.727 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:02.724+0000 7f28349ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:55:02.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:02.797 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.790+0000 7f103dae7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:55:02.880 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:02.874+0000 7f103dae7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:55:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 Processor -- start 2023-12-19T15:55:02.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 -- start start 2023-12-19T15:55:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6c0813f890 0x7f6c080aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c080c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0806c9f0 con 0x7f6c080c1a80 2023-12-19T15:55:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c080c7f20 con 0x7f6c0813f890 2023-12-19T15:55:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0e385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080c8190 con 0x7f6c080c2980 2023-12-19T15:55:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0c922700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6c080c1a80 0x7f6c080680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46638/0 (socket says 172.21.15.50:46638) 2023-12-19T15:55:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.944+0000 7f6c0c922700 1 -- 172.21.15.50:0/2934918257 learned_addr learned my addr 172.21.15.50:0/2934918257 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2934918257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c080c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c07fff700 1 --2- 172.21.15.50:0/2934918257 >> v2:172.21.15.125:3300/0 conn(0x7f6c0813f890 0x7f6c080aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c06ffd700 1 -- 172.21.15.50:0/2934918257 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778874949 0 0) 0x7f6c0806c9f0 con 0x7f6c080c1a80 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c06ffd700 1 -- 172.21.15.50:0/2934918257 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0003570 con 0x7f6c080c1a80 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 -- 172.21.15.50:0/2934918257 >> v2:172.21.15.125:3300/0 conn(0x7f6c0813f890 msgr2=0x7f6c080aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2934918257 >> v2:172.21.15.125:3300/0 conn(0x7f6c0813f890 0x7f6c080aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 -- 172.21.15.50:0/2934918257 >> v1:172.21.15.50:6789/0 conn(0x7f6c080c1a80 legacy=0x7f6c080680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 -- 172.21.15.50:0/2934918257 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c080c8470 con 0x7f6c080c2980 2023-12-19T15:55:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2934918257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c080c1540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098e0 tx=0x7f6bfc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409ec09f6a7653d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c06ffd700 1 -- 172.21.15.50:0/2934918257 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc011630 con 0x7f6c080c2980 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c06ffd700 1 -- 172.21.15.50:0/2934918257 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc011790 con 0x7f6c080c2980 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c06ffd700 1 -- 172.21.15.50:0/2934918257 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc011b30 con 0x7f6c080c2980 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 -- 172.21.15.50:0/2934918257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 msgr2=0x7f6c080c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2934918257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c080c1540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098e0 tx=0x7f6bfc0091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 -- 172.21.15.50:0/2934918257 shutdown_connections 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2934918257 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c080c1540 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2934918257 >> v2:172.21.15.125:3300/0 conn(0x7f6c0813f890 0x7f6c080aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 -- 172.21.15.50:0/2934918257 >> 172.21.15.50:0/2934918257 conn(0x7f6c080c8850 msgr2=0x7f6c080c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 -- 172.21.15.50:0/2934918257 shutdown_connections 2023-12-19T15:55:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.948+0000 7f6c0e385700 1 -- 172.21.15.50:0/2934918257 wait complete. 2023-12-19T15:55:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 Processor -- start 2023-12-19T15:55:02.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 -- start start 2023-12-19T15:55:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c0813e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c080c54b0 0x7f6c080b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080c66d0 con 0x7f6c0813f890 2023-12-19T15:55:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080c5ac0 con 0x7f6c080c54b0 2023-12-19T15:55:02.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.952+0000 7f6c0e385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080c5da0 con 0x7f6c080c2980 2023-12-19T15:55:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50884/0 (socket says 172.21.15.50:50884) 2023-12-19T15:55:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 -- 172.21.15.50:0/2425167173 learned_addr learned my addr 172.21.15.50:0/2425167173 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:02.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c07fff700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c0813e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c0c922700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c080c54b0 0x7f6c080b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 -- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 msgr2=0x7f6c0813e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c0813e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 -- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c080c54b0 msgr2=0x7f6c080b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c080c54b0 0x7f6c080b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:02.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 -- 172.21.15.50:0/2425167173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc009420 con 0x7f6c0813f890 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c077fe700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc013000 tx=0x7f6bfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc011900 con 0x7f6c0813f890 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc011a60 con 0x7f6c0813f890 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c080c6080 con 0x7f6c0813f890 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc025640 con 0x7f6c0813f890 2023-12-19T15:55:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.956+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c080a9470 con 0x7f6c0813f890 2023-12-19T15:55:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c08071120 con 0x7f6c0813f890 2023-12-19T15:55:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f6bfc025a30 con 0x7f6c0813f890 2023-12-19T15:55:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6beffff700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 0x7f6bf4062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6c07fff700 1 -- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 msgr2=0x7f6bf4062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f6bfc07e410 con 0x7f6c0813f890 2023-12-19T15:55:02.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.960+0000 7f6c07fff700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 0x7f6bf4062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:02.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:02.968+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6bfc04e4b0 con 0x7f6c0813f890 2023-12-19T15:55:02.981 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:02.976+0000 7f28349ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:03.077 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.076+0000 7f28349ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:55:03.097 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:03.090+0000 7f103dae7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.140+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c080642a0 con 0x7f6c0813f890 2023-12-19T15:55:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.144+0000 7f6beffff700 1 -- 172.21.15.50:0/2425167173 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16153 (secure 0 0 0) 0x7f6bfc051ad0 con 0x7f6c0813f890 2023-12-19T15:55:03.146 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:03.146 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":168,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:54:56.619651+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1152230135":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/373489315":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/669887858":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/219377965":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/2706791219":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/1963925549":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6833/33827":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6832/33827":"2023-12-20T15:54:56.619577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.144+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 msgr2=0x7f6bf4062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.144+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 0x7f6bf4062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.144+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 msgr2=0x7f6c080c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.144+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc013000 tx=0x7f6bfc017000 comp rx=0 tx=0).stop 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 shutdown_connections 2023-12-19T15:55:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c080c54b0 0x7f6c080b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6bf4060730 0x7f6bf4062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c0813f890 0x7f6c080c4f70 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 --2- 172.21.15.50:0/2425167173 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c080c2980 0x7f6c0813e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 >> 172.21.15.50:0/2425167173 conn(0x7f6c080c8850 msgr2=0x7f6c080cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 shutdown_connections 2023-12-19T15:55:03.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.148+0000 7f6c0e385700 1 -- 172.21.15.50:0/2425167173 wait complete. 2023-12-19T15:55:03.161 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.156+0000 7f28349ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:55:03.172 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T15:55:03.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:03.176 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:03.170+0000 7f103dae7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:55:03.250 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.248+0000 7f28349ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:55:03.263 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:03.254+0000 7f103dae7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:55:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 Processor -- start 2023-12-19T15:55:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 -- start start 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 0x7fba340aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fba340c2980 0x7fba340c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c1a80 0x7fba340680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3406c9f0 con 0x7fba340c2980 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba340c7f20 con 0x7fba3413f890 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba39c6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba340c8190 con 0x7fba340c1a80 2023-12-19T15:55:03.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fba340c2980 0x7fba340c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fba340c2980 0x7fba340c1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50898/0 (socket says 172.21.15.50:50898) 2023-12-19T15:55:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 -- 172.21.15.50:0/3146373692 learned_addr learned my addr 172.21.15.50:0/3146373692 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba337fe700 1 --2- 172.21.15.50:0/3146373692 >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 0x7fba340aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba33fff700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c1a80 0x7fba340680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 -- 172.21.15.50:0/3146373692 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c1a80 msgr2=0x7fba340680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c1a80 0x7fba340680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 -- 172.21.15.50:0/3146373692 >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 msgr2=0x7fba340aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 --2- 172.21.15.50:0/3146373692 >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 0x7fba340aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba32ffd700 1 -- 172.21.15.50:0/3146373692 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340c8470 con 0x7fba340c2980 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.328+0000 7fba337fe700 1 --2- 172.21.15.50:0/3146373692 >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 0x7fba340aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba32ffd700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba340c2980 0x7fba340c1540 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fba34068c90 tx=0x7fba28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad0f96d100e4109 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba327fc700 1 -- 172.21.15.50:0/3146373692 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2800b860 con 0x7fba340c2980 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba327fc700 1 -- 172.21.15.50:0/3146373692 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2800b9c0 con 0x7fba340c2980 2023-12-19T15:55:03.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba327fc700 1 -- 172.21.15.50:0/3146373692 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2800bd60 con 0x7fba340c2980 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 -- 172.21.15.50:0/3146373692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba340c2980 msgr2=0x7fba340c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba340c2980 0x7fba340c1540 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fba34068c90 tx=0x7fba28009250 comp rx=0 tx=0).stop 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 -- 172.21.15.50:0/3146373692 shutdown_connections 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba340c2980 0x7fba340c1540 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3146373692 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c1a80 0x7fba340680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3146373692 >> v2:172.21.15.125:3300/0 conn(0x7fba3413f890 0x7fba340aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 -- 172.21.15.50:0/3146373692 >> 172.21.15.50:0/3146373692 conn(0x7fba340c8850 msgr2=0x7fba340c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 -- 172.21.15.50:0/3146373692 shutdown_connections 2023-12-19T15:55:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.332+0000 7fba39c6e700 1 -- 172.21.15.50:0/3146373692 wait complete. 2023-12-19T15:55:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 Processor -- start 2023-12-19T15:55:03.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 -- start start 2023-12-19T15:55:03.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba3413f890 0x7fba34169a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 0x7fba340a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3413d6b0 con 0x7fba3413f890 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba340a67d0 con 0x7fba340c2e90 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba39c6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba34169f50 con 0x7fba340c2980 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba337fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba337fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53282/0 (socket says 172.21.15.50:53282) 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba32ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba3413f890 0x7fba34169a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba337fe700 1 -- 172.21.15.50:0/3674691837 learned_addr learned my addr 172.21.15.50:0/3674691837 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.336+0000 7fba33fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 0x7fba340a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 msgr2=0x7fba3413d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba3413f890 msgr2=0x7fba34169a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba3413f890 0x7fba34169a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 -- 172.21.15.50:0/3674691837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280094f0 con 0x7fba340c2e90 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba337fe700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba33fff700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 0x7fba340a4fe0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba3406a6a0 tx=0x7fba2c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0125f0 con 0x7fba340c2e90 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c012750 con 0x7fba340c2e90 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c012af0 con 0x7fba340c2e90 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba340a47f0 con 0x7fba340c2e90 2023-12-19T15:55:03.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.340+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba340a6460 con 0x7fba340c2e90 2023-12-19T15:55:03.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba340a6940 con 0x7fba340c2e90 2023-12-19T15:55:03.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fba2c013480 con 0x7fba340c2e90 2023-12-19T15:55:03.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba1b7fe700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 0x7fba20062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fba2c07d080 con 0x7fba340c2e90 2023-12-19T15:55:03.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba337fe700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 msgr2=0x7fba20062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:03.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.344+0000 7fba337fe700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 0x7fba20062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:03.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.352+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fba2c0acbe0 con 0x7fba340c2e90 2023-12-19T15:55:03.454 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:03.446+0000 7f103dae7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:55:03.499 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.496+0000 7f28349ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:03.542 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:03.534+0000 7f103dae7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:55:03.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.548+0000 7fba337fe700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 msgr2=0x7fba20062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:03.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.548+0000 7fba337fe700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 0x7fba20062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:03.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.564+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba34071120 con 0x7fba340c2e90 2023-12-19T15:55:03.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba1b7fe700 1 -- 172.21.15.50:0/3674691837 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fba2c04d080 con 0x7fba340c2e90 2023-12-19T15:55:03.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:03.569 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 msgr2=0x7fba20062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 0x7fba20062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 msgr2=0x7fba340a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 0x7fba340a4fe0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba3406a6a0 tx=0x7fba2c008040 comp rx=0 tx=0).stop 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 shutdown_connections 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba340c2e90 0x7fba340a4fe0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fba20060730 0x7fba20062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba3413f890 0x7fba34169a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 --2- 172.21.15.50:0/3674691837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fba340c2980 0x7fba3413d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 >> 172.21.15.50:0/3674691837 conn(0x7fba340c8850 msgr2=0x7fba3413e880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:03.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 shutdown_connections 2023-12-19T15:55:03.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.568+0000 7fba39c6e700 1 -- 172.21.15.50:0/3674691837 wait complete. 2023-12-19T15:55:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:55:03.590 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.588+0000 7f28349ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:55:03.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:03.688 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.684+0000 7f28349ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:55:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 Processor -- start 2023-12-19T15:55:03.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 -- start start 2023-12-19T15:55:03.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 0x7f26f80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f26f8068800 0x7f26f80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f806e150 con 0x7f26f8140ae0 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f806d540 con 0x7f26f8068800 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26fd0f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f806d820 con 0x7f26f8140710 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f26f8068800 0x7f26f80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f6d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 0x7f26f80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f26f8068800 0x7f26f80ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50918/0 (socket says 172.21.15.50:50918) 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 -- 172.21.15.50:0/1506577148 learned_addr learned my addr 172.21.15.50:0/1506577148 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f659c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f26f8140ae0 0x7f26f80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:57346/0 (socket says 172.21.15.50:57346) 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 -- 172.21.15.50:0/1506577148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 msgr2=0x7f26f80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 0x7f26f80c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 -- 172.21.15.50:0/1506577148 >> v1:172.21.15.125:6789/0 conn(0x7f26f8140ae0 legacy=0x7f26f80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 -- 172.21.15.50:0/1506577148 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f806db00 con 0x7f26f8068800 2023-12-19T15:55:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f6d9d700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 0x7f26f80c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f5d9b700 1 -- 172.21.15.50:0/1506577148 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287188190 0 0) 0x7f26f806e150 con 0x7f26f8140ae0 2023-12-19T15:55:03.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f759e700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8068800 0x7f26f80ac250 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009f20 tx=0x7f26f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1299e1562098d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.752+0000 7f26f5d9b700 1 -- 172.21.15.50:0/1506577148 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f000e070 con 0x7f26f8068800 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26f5d9b700 1 -- 172.21.15.50:0/1506577148 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f000b9d0 con 0x7f26f8068800 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26f5d9b700 1 -- 172.21.15.50:0/1506577148 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00092e0 con 0x7f26f8068800 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- 172.21.15.50:0/1506577148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8068800 msgr2=0x7f26f80ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8068800 0x7f26f80ac250 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009f20 tx=0x7f26f0009840 comp rx=0 tx=0).stop 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- 172.21.15.50:0/1506577148 shutdown_connections 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8068800 0x7f26f80ac250 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/1506577148 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8140710 0x7f26f80c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- 172.21.15.50:0/1506577148 >> 172.21.15.50:0/1506577148 conn(0x7f26f80c8850 msgr2=0x7f26f80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- 172.21.15.50:0/1506577148 shutdown_connections 2023-12-19T15:55:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- 172.21.15.50:0/1506577148 wait complete. 2023-12-19T15:55:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 Processor -- start 2023-12-19T15:55:03.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.756+0000 7f26fd0f0700 1 -- start start 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f8140710 0x7f26f80b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8140ae0 0x7f26f80aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8169230 con 0x7f26f8140ae0 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8169510 con 0x7f26f8140710 2023-12-19T15:55:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f81697f0 con 0x7f26f8068800 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53308/0 (socket says 172.21.15.50:53308) 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 -- 172.21.15.50:0/127556871 learned_addr learned my addr 172.21.15.50:0/127556871 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f659c700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f8140710 0x7f26f80b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f759e700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8140ae0 0x7f26f80aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f8140710 msgr2=0x7f26f80b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f8140710 0x7f26f80b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8140ae0 msgr2=0x7f26f80aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8140ae0 0x7f26f80aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 -- 172.21.15.50:0/127556871 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8009420 con 0x7f26f8068800 2023-12-19T15:55:03.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26f6d9d700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26e800ec80 tx=0x7f26e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800c920 con 0x7f26f8068800 2023-12-19T15:55:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e800ca80 con 0x7f26f8068800 2023-12-19T15:55:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800ce00 con 0x7f26f8068800 2023-12-19T15:55:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f0009ae0 con 0x7f26f8068800 2023-12-19T15:55:03.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.760+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f80a90e0 con 0x7f26f8068800 2023-12-19T15:55:03.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8071120 con 0x7f26f8068800 2023-12-19T15:55:03.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f26e8013680 con 0x7f26f8068800 2023-12-19T15:55:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26deffd700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 0x7f26e4062cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f26e807d0f0 con 0x7f26f8068800 2023-12-19T15:55:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26f659c700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 msgr2=0x7f26e4062cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:03.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.764+0000 7f26f659c700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 0x7f26e4062cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.776+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f26e804d170 con 0x7f26f8068800 2023-12-19T15:55:03.900 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.896+0000 7f28349ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:55:03.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.968+0000 7f26f659c700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 msgr2=0x7f26e4062cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.968+0000 7f26f659c700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 0x7f26e4062cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:03.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.980+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26f8169930 con 0x7f26f8068800 2023-12-19T15:55:03.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.984+0000 7f26deffd700 1 -- 172.21.15.50:0/127556871 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f26f8169930 con 0x7f26f8068800 2023-12-19T15:55:03.986 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:03.986 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:54:41.248010+0000","modified":"2023-12-19T15:54:44.834638+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8701":{"gid":8701,"name":"d","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.125:6836/4138116501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":4138116501},{"type":"v1","addr":"172.21.15.125:6836","nonce":4138116501}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 msgr2=0x7f26e4062cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 0x7f26e4062cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 msgr2=0x7f26f80c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26e800ec80 tx=0x7f26e80091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 shutdown_connections 2023-12-19T15:55:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f8140710 0x7f26f80b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f26e4060800 0x7f26e4062cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26f8140ae0 0x7f26f80aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 --2- 172.21.15.50:0/127556871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f26f8068800 0x7f26f80c5970 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 >> 172.21.15.50:0/127556871 conn(0x7f26f80c8850 msgr2=0x7f26f80c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 shutdown_connections 2023-12-19T15:55:03.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:03.988+0000 7f26fd0f0700 1 -- 172.21.15.50:0/127556871 wait complete. 2023-12-19T15:55:03.994 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 75 2023-12-19T15:55:03.996 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:03.992+0000 7f28349ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:55:04.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:04.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 Processor -- start 2023-12-19T15:55:04.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 -- start start 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44ec1502b0 0x7f44ec068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec0c5900 0x7f44ec0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0c6630 con 0x7f44ec067c90 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ec0c6910 con 0x7f44ec1502b0 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44f242e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0c6bf0 con 0x7f44ec0c5900 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44ebfff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f44ec067c90 0x7f44ec14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:57362/0 (socket says 172.21.15.50:57362) 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44ebfff700 1 -- 172.21.15.50:0/3771149246 learned_addr learned my addr 172.21.15.50:0/3771149246 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.176+0000 7f44eb7fe700 1 --2- 172.21.15.50:0/3771149246 >> v2:172.21.15.50:3300/0 conn(0x7f44ec1502b0 0x7f44ec068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f09cb700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec0c5900 0x7f44ec0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eaffd700 1 -- 172.21.15.50:0/3771149246 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803626837 0 0) 0x7f44ec0c6630 con 0x7f44ec067c90 2023-12-19T15:55:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eaffd700 1 -- 172.21.15.50:0/3771149246 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003570 con 0x7f44ec067c90 2023-12-19T15:55:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eb7fe700 1 -- 172.21.15.50:0/3771149246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec0c5900 msgr2=0x7f44ec0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eb7fe700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec0c5900 0x7f44ec0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eb7fe700 1 -- 172.21.15.50:0/3771149246 >> v1:172.21.15.125:6789/0 conn(0x7f44ec067c90 legacy=0x7f44ec14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eb7fe700 1 -- 172.21.15.50:0/3771149246 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0c6ed0 con 0x7f44ec1502b0 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eb7fe700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec068060 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f44e00098e0 tx=0x7f44e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0467d0fc662bcc1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eaffd700 1 -- 172.21.15.50:0/3771149246 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0011630 con 0x7f44ec1502b0 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eaffd700 1 -- 172.21.15.50:0/3771149246 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0011790 con 0x7f44ec1502b0 2023-12-19T15:55:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44eaffd700 1 -- 172.21.15.50:0/3771149246 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0011b30 con 0x7f44ec1502b0 2023-12-19T15:55:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 -- 172.21.15.50:0/3771149246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 msgr2=0x7f44ec068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec068060 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f44e00098e0 tx=0x7f44e00091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 -- 172.21.15.50:0/3771149246 shutdown_connections 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec0c5900 0x7f44ec0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 --2- 172.21.15.50:0/3771149246 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec068060 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.180+0000 7f44f242e700 1 -- 172.21.15.50:0/3771149246 >> 172.21.15.50:0/3771149246 conn(0x7f44ec0cd6d0 msgr2=0x7f44ec06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.184+0000 7f44f242e700 1 -- 172.21.15.50:0/3771149246 shutdown_connections 2023-12-19T15:55:04.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.184+0000 7f44f242e700 1 -- 172.21.15.50:0/3771149246 wait complete. 2023-12-19T15:55:04.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.184+0000 7f44f242e700 1 Processor -- start 2023-12-19T15:55:04.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.184+0000 7f44f242e700 1 -- start start 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec067c90 0x7f44ec0c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ec0c5900 0x7f44ec169110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0c8ea0 con 0x7f44ec1502b0 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec169650 con 0x7f44ec0c5900 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0c8290 con 0x7f44ec067c90 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44ebfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec067c90 0x7f44ec0c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50938/0 (socket says 172.21.15.50:50938) 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 -- 172.21.15.50:0/3870991016 learned_addr learned my addr 172.21.15.50:0/3870991016 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44eb7fe700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ec0c5900 0x7f44ec169110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 -- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec067c90 msgr2=0x7f44ec0c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec067c90 0x7f44ec0c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 -- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ec0c5900 msgr2=0x7f44ec169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ec0c5900 0x7f44ec169110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 -- 172.21.15.50:0/3870991016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0009420 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f09cb700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009e90 tx=0x7f44e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e400caa0 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e400f040 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec0c8570 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e400ce50 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.188+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec0c8b30 con 0x7f44ec1502b0 2023-12-19T15:55:04.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec0c9010 con 0x7f44ec1502b0 2023-12-19T15:55:04.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 73) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f44e4014690 con 0x7f44ec1502b0 2023-12-19T15:55:04.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44d3fff700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 0x7f44d8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6425+0+0 (secure 0 0 0) 0x7f44e407d820 con 0x7f44ec1502b0 2023-12-19T15:55:04.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44ebfff700 1 -- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 msgr2=0x7f44d8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:04.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.192+0000 7f44ebfff700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 0x7f44d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:04.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.200+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44e404d7b0 con 0x7f44ec1502b0 2023-12-19T15:55:04.266 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.258+0000 7f103dae7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:55:04.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.280+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 74) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f44e404db80 con 0x7f44ec1502b0 2023-12-19T15:55:04.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.280+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 msgr2=0x7f44d8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:04.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.280+0000 7f44d3fff700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 0x7f44d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f100a17d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f100a17d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f100a17d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f100a17d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f100a17d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f1006976700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.371 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f1006976700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.372 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f1006976700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.372 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f1006976700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.372 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:04.362+0000 7f1006976700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:04.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.388+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44ec071120 con 0x7f44ec1502b0 2023-12-19T15:55:04.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.388+0000 7f44d3fff700 1 -- 172.21.15.50:0/3870991016 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f44e40699a0 con 0x7f44ec1502b0 2023-12-19T15:55:04.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:04.392 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":169,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:04.270633+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:54:38.835585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T15:54:40.257787+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/2510569342":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/278018654":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2856526150":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2634924093":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2213656349":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2640199441":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/1152230135":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/373489315":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/669887858":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/810172491":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/219377965":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/2706791219":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/1963925549":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6833/33827":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6832/33827":"2023-12-20T15:54:56.619577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 msgr2=0x7f44ec1446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009e90 tx=0x7f44e400b040 comp rx=0 tx=0).stop 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 shutdown_connections 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ec0c5900 0x7f44ec169110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f44d8060700 0x7f44d8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ec1502b0 0x7f44ec1446b0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 --2- 172.21.15.50:0/3870991016 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44ec067c90 0x7f44ec0c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 >> 172.21.15.50:0/3870991016 conn(0x7f44ec0cd6d0 msgr2=0x7f44ec06d570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 shutdown_connections 2023-12-19T15:55:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.392+0000 7f44f242e700 1 -- 172.21.15.50:0/3870991016 wait complete. 2023-12-19T15:55:04.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T15:55:04.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 Processor -- start 2023-12-19T15:55:04.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 -- start start 2023-12-19T15:55:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa48158cc0 0x7ffa4806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 0x7ffa4806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa4806baa0 0x7ffa480b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4806beb0 con 0x7ffa4806baa0 2023-12-19T15:55:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4806b0d0 con 0x7ffa4806dc00 2023-12-19T15:55:04.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4e478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480b4790 con 0x7ffa48158cc0 2023-12-19T15:55:04.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa4806baa0 0x7ffa480b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa4806baa0 0x7ffa480b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50940/0 (socket says 172.21.15.50:50940) 2023-12-19T15:55:04.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 -- 172.21.15.50:0/3222965123 learned_addr learned my addr 172.21.15.50:0/3222965123 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa47fff700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa48158cc0 0x7ffa4806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa477fe700 1 --2- 172.21.15.50:0/3222965123 >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 0x7ffa4806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 -- 172.21.15.50:0/3222965123 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa48158cc0 msgr2=0x7ffa4806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa48158cc0 0x7ffa4806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 -- 172.21.15.50:0/3222965123 >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 msgr2=0x7ffa4806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/3222965123 >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 0x7ffa4806b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa4ca15700 1 -- 172.21.15.50:0/3222965123 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4806d710 con 0x7ffa4806baa0 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.580+0000 7ffa477fe700 1 --2- 172.21.15.50:0/3222965123 >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 0x7ffa4806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:04.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa4806baa0 0x7ffa480b3870 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ffa48069250 tx=0x7ffa40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a712afb2db692fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:04.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa46ffd700 1 -- 172.21.15.50:0/3222965123 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4000e070 con 0x7ffa4806baa0 2023-12-19T15:55:04.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa46ffd700 1 -- 172.21.15.50:0/3222965123 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4000ba20 con 0x7ffa4806baa0 2023-12-19T15:55:04.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa46ffd700 1 -- 172.21.15.50:0/3222965123 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa400092e0 con 0x7ffa4806baa0 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 -- 172.21.15.50:0/3222965123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa4806baa0 msgr2=0x7ffa480b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa4806baa0 0x7ffa480b3870 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ffa48069250 tx=0x7ffa40009840 comp rx=0 tx=0).stop 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 -- 172.21.15.50:0/3222965123 shutdown_connections 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa4806baa0 0x7ffa480b3870 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 --2- 172.21.15.50:0/3222965123 >> v2:172.21.15.125:3300/0 conn(0x7ffa4806dc00 0x7ffa4806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 --2- 172.21.15.50:0/3222965123 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa48158cc0 0x7ffa4806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 -- 172.21.15.50:0/3222965123 >> 172.21.15.50:0/3222965123 conn(0x7ffa48067bb0 msgr2=0x7ffa480cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 -- 172.21.15.50:0/3222965123 shutdown_connections 2023-12-19T15:55:04.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.584+0000 7ffa4e478700 1 -- 172.21.15.50:0/3222965123 wait complete. 2023-12-19T15:55:04.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.588+0000 7ffa4e478700 1 Processor -- start 2023-12-19T15:55:04.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.588+0000 7ffa4e478700 1 -- start start 2023-12-19T15:55:04.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.588+0000 7ffa4e478700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa4806baa0 0x7ffa480ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa4806dc00 0x7ffa480af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480ab780 con 0x7ffa48158cc0 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480aab70 con 0x7ffa4806baa0 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480aae50 con 0x7ffa4806dc00 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa477fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa4806dc00 0x7ffa480af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa47fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa4806baa0 0x7ffa480ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50948/0 (socket says 172.21.15.50:50948) 2023-12-19T15:55:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 -- 172.21.15.50:0/4125217151 learned_addr learned my addr 172.21.15.50:0/4125217151 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:04.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 -- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa4806dc00 msgr2=0x7ffa480af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa4806dc00 0x7ffa480af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 -- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa4806baa0 msgr2=0x7ffa480ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa4806baa0 0x7ffa480ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 -- 172.21.15.50:0/4125217151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40009ae0 con 0x7ffa48158cc0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4ca15700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806eb60 tx=0x7ffa40018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa400032f0 con 0x7ffa48158cc0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa40008db0 con 0x7ffa48158cc0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40025690 con 0x7ffa48158cc0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa480ab130 con 0x7ffa48158cc0 2023-12-19T15:55:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa480ceac0 con 0x7ffa48158cc0 2023-12-19T15:55:04.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.592+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa48071120 con 0x7ffa48158cc0 2023-12-19T15:55:04.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.596+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 74) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ffa40003040 con 0x7ffa48158cc0 2023-12-19T15:55:04.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.596+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6726+0+0 (secure 0 0 0) 0x7ffa4007dde0 con 0x7ffa48158cc0 2023-12-19T15:55:04.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.604+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffa4004e350 con 0x7ffa48158cc0 2023-12-19T15:55:04.812 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:04.808+0000 7f28349ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:55:04.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.812+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffa4806b750 con 0x7ffa48158cc0 2023-12-19T15:55:04.817 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T15:55:04.816+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T15:55:04.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.820+0000 7ffa2ffff700 1 -- 172.21.15.50:0/4125217151 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v76) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffa40051970 con 0x7ffa48158cc0 2023-12-19T15:55:04.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 msgr2=0x7ffa480afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ffa4806eb60 tx=0x7ffa40018000 comp rx=0 tx=0).stop 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 shutdown_connections 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa48158cc0 0x7ffa480afaf0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffa4806dc00 0x7ffa480af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 --2- 172.21.15.50:0/4125217151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa4806baa0 0x7ffa480ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 >> 172.21.15.50:0/4125217151 conn(0x7ffa48067bb0 msgr2=0x7ffa480b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 shutdown_connections 2023-12-19T15:55:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:04.824+0000 7ffa4e478700 1 -- 172.21.15.50:0/4125217151 wait complete. 2023-12-19T15:55:04.828 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T15:55:04.833 INFO:tasks.ceph.mds.d.smithi125.stderr: -2588> 2023-12-19T15:54:46.754+0000 7f99871f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T15:55:04.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T15:55:04.880 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T15:55:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.012+0000 7fec96e6e700 1 Processor -- start 2023-12-19T15:55:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.012+0000 7fec96e6e700 1 -- start start 2023-12-19T15:55:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec96e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec96e6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9006a7a0 con 0x7fec9006ac30 2023-12-19T15:55:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec96e6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec900c7630 con 0x7fec9006c7c0 2023-12-19T15:55:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec96e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9006d4b0 con 0x7fec900c6d20 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec94c0a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec9540b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fec9006ac30 0x7fec9006b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:46650/0 (socket says 172.21.15.50:46650) 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec9540b700 1 -- 172.21.15.50:0/2823944322 learned_addr learned my addr 172.21.15.50:0/2823944322 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec94c0a700 1 -- 172.21.15.50:0/2823944322 >> v1:172.21.15.125:6789/0 conn(0x7fec9006c7c0 legacy=0x7fec9006cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec94c0a700 1 -- 172.21.15.50:0/2823944322 >> v1:172.21.15.50:6789/0 conn(0x7fec9006ac30 legacy=0x7fec9006b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec94c0a700 1 -- 172.21.15.50:0/2823944322 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9006d790 con 0x7fec900c6d20 2023-12-19T15:55:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec94c0a700 1 --2- 172.21.15.50:0/2823944322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900c70f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fec80009910 tx=0x7fec800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beb7b11ea0bee0c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec877fe700 1 -- 172.21.15.50:0/2823944322 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8000b860 con 0x7fec900c6d20 2023-12-19T15:55:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.016+0000 7fec877fe700 1 -- 172.21.15.50:0/2823944322 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8000b9c0 con 0x7fec900c6d20 2023-12-19T15:55:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec877fe700 1 -- 172.21.15.50:0/2823944322 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8000bd60 con 0x7fec900c6d20 2023-12-19T15:55:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 -- 172.21.15.50:0/2823944322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 msgr2=0x7fec900c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 --2- 172.21.15.50:0/2823944322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900c70f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fec80009910 tx=0x7fec800091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 -- 172.21.15.50:0/2823944322 shutdown_connections 2023-12-19T15:55:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 --2- 172.21.15.50:0/2823944322 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900c70f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 -- 172.21.15.50:0/2823944322 >> 172.21.15.50:0/2823944322 conn(0x7fec90067bb0 msgr2=0x7fec900cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 -- 172.21.15.50:0/2823944322 shutdown_connections 2023-12-19T15:55:05.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.020+0000 7fec96e6e700 1 -- 172.21.15.50:0/2823944322 wait complete. 2023-12-19T15:55:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 Processor -- start 2023-12-19T15:55:05.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 -- start start 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec9006ac30 0x7fec90156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90155630 con 0x7fec9006c7c0 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90155910 con 0x7fec9006ac30 2023-12-19T15:55:05.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec96e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec90155bf0 con 0x7fec900c6d20 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec87fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec87fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50954/0 (socket says 172.21.15.50:50954) 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec87fff700 1 -- 172.21.15.50:0/4287252259 learned_addr learned my addr 172.21.15.50:0/4287252259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec94c0a700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec9006ac30 0x7fec90156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.024+0000 7fec9540b700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 -- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 msgr2=0x7fec900acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 -- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec9006ac30 msgr2=0x7fec90156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec9006ac30 0x7fec90156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 -- 172.21.15.50:0/4287252259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80009420 con 0x7fec9006c7c0 2023-12-19T15:55:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec87fff700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00c590 tx=0x7fec8c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c011590 con 0x7fec9006c7c0 2023-12-19T15:55:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c011710 con 0x7fec9006c7c0 2023-12-19T15:55:05.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90155ed0 con 0x7fec9006c7c0 2023-12-19T15:55:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c011ad0 con 0x7fec9006c7c0 2023-12-19T15:55:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90157290 con 0x7fec9006c7c0 2023-12-19T15:55:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.028+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec900aa030 con 0x7fec9006c7c0 2023-12-19T15:55:05.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.032+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 74) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fec8c01b3b0 con 0x7fec9006c7c0 2023-12-19T15:55:05.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.032+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6812+0+0 (secure 0 0 0) 0x7fec8c07bc50 con 0x7fec9006c7c0 2023-12-19T15:55:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.040+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec8c04c0c0 con 0x7fec9006c7c0 2023-12-19T15:55:05.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.244+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fec9006da70 con 0x7fec9006c7c0 2023-12-19T15:55:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.288+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 75) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fec8c053d50 con 0x7fec9006c7c0 2023-12-19T15:55:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.288+0000 7fec84ff9700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 0x7fec7c0687b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:05.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.292+0000 7fec94c0a700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 0x7fec7c0687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:05.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.292+0000 7fec94c0a700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 0x7fec7c0687b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec800093f0 tx=0x7fec80017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:05.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.824+0000 7fec84ff9700 1 -- 172.21.15.50:0/4287252259 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v77) v1 ==== 104+0+0 (secure 0 0 0) 0x7fec8c075e90 con 0x7fec9006c7c0 2023-12-19T15:55:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 msgr2=0x7fec7c0687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 0x7fec7c0687b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec800093f0 tx=0x7fec80017000 comp rx=0 tx=0).stop 2023-12-19T15:55:05.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 msgr2=0x7fec90154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00c590 tx=0x7fec8c00c8a0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 shutdown_connections 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fec7c0663d0 0x7fec7c0687b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fec900c6d20 0x7fec900acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec9006c7c0 0x7fec90154cc0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 --2- 172.21.15.50:0/4287252259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec9006ac30 0x7fec90156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 >> 172.21.15.50:0/4287252259 conn(0x7fec90067bb0 msgr2=0x7fec90159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 shutdown_connections 2023-12-19T15:55:05.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:05.828+0000 7fec96e6e700 1 -- 172.21.15.50:0/4287252259 wait complete. 2023-12-19T15:55:05.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T15:55:06.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.024+0000 7f7f676f0700 1 Processor -- start 2023-12-19T15:55:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.024+0000 7f7f676f0700 1 -- start start 2023-12-19T15:55:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f600cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7f600ad0f0 0x7f7f600cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 0x7f7f600ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600adf30 con 0x7f7f600ad0f0 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600cd6f0 con 0x7f7f600ae3c0 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f676f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600cd9d0 con 0x7f7f600acd20 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f64c8b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7f600ad0f0 0x7f7f600cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f64c8b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7f600ad0f0 0x7f7f600cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50962/0 (socket says 172.21.15.50:50962) 2023-12-19T15:55:06.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f64c8b700 1 -- 172.21.15.50:0/2675106147 learned_addr learned my addr 172.21.15.50:0/2675106147 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:06.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.028+0000 7f7f6548c700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f600cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 -- 172.21.15.50:0/2675106147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 msgr2=0x7f7f600cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:06.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f65c8d700 1 --2- 172.21.15.50:0/2675106147 >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 0x7f7f600ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f600cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 -- 172.21.15.50:0/2675106147 >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 msgr2=0x7f7f600ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 --2- 172.21.15.50:0/2675106147 >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 0x7f7f600ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 -- 172.21.15.50:0/2675106147 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f600adbc0 con 0x7f7f600ad0f0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f65c8d700 1 --2- 172.21.15.50:0/2675106147 >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 0x7f7f600ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f64c8b700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ad0f0 0x7f7f600cceb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7f600c83d0 tx=0x7f7f50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefa2f999ab00f8b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f57fff700 1 -- 172.21.15.50:0/2675106147 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000b860 con 0x7f7f600ad0f0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f57fff700 1 -- 172.21.15.50:0/2675106147 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5000b9c0 con 0x7f7f600ad0f0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f57fff700 1 -- 172.21.15.50:0/2675106147 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000bd60 con 0x7f7f600ad0f0 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 -- 172.21.15.50:0/2675106147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ad0f0 msgr2=0x7f7f600cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ad0f0 0x7f7f600cceb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7f600c83d0 tx=0x7f7f50009250 comp rx=0 tx=0).stop 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 -- 172.21.15.50:0/2675106147 shutdown_connections 2023-12-19T15:55:06.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ad0f0 0x7f7f600cceb0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 --2- 172.21.15.50:0/2675106147 >> v2:172.21.15.125:3300/0 conn(0x7f7f600ae3c0 0x7f7f600ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 --2- 172.21.15.50:0/2675106147 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f600cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 -- 172.21.15.50:0/2675106147 >> 172.21.15.50:0/2675106147 conn(0x7f7f600cdce0 msgr2=0x7f7f600ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 -- 172.21.15.50:0/2675106147 shutdown_connections 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.032+0000 7f7f676f0700 1 -- 172.21.15.50:0/2675106147 wait complete. 2023-12-19T15:55:06.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 Processor -- start 2023-12-19T15:55:06.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 -- start start 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f60159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f600ad0f0 0x7f7f60169840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60169d80 con 0x7f7f600ae3c0 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f601590a0 con 0x7f7f600ad0f0 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f676f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60159340 con 0x7f7f600acd20 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50976/0 (socket says 172.21.15.50:50976) 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 -- 172.21.15.50:0/139600068 learned_addr learned my addr 172.21.15.50:0/139600068 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f6548c700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f60159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f64c8b700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f600ad0f0 0x7f7f60169840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 -- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 msgr2=0x7f7f60159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f60159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 -- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f600ad0f0 msgr2=0x7f7f60169840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f600ad0f0 0x7f7f60169840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.036+0000 7f7f65c8d700 1 -- 172.21.15.50:0/139600068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500094f0 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f65c8d700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800cef0 tx=0x7f7f58007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5800ad20 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f58016420 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f60159620 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5800aea0 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f60157980 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.040+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f60071120 con 0x7f7f600ae3c0 2023-12-19T15:55:06.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.044+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 75) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f7f58016a70 con 0x7f7f600ae3c0 2023-12-19T15:55:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.044+0000 7f7f557fa700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 0x7f7f44062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.044+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6812+0+0 (secure 0 0 0) 0x7f7f58012070 con 0x7f7f600ae3c0 2023-12-19T15:55:06.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.044+0000 7f7f6548c700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 0x7f7f44062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:06.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.048+0000 7f7f6548c700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 0x7f7f44062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c006fd0 tx=0x7f7f5c008040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:06.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.052+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7f5804cbe0 con 0x7f7f600ae3c0 2023-12-19T15:55:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.228+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7f600cd3f0 con 0x7f7f600ae3c0 2023-12-19T15:55:06.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.832+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7f5804d940 con 0x7f7f600ae3c0 2023-12-19T15:55:06.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.844+0000 7f7f557fa700 1 -- 172.21.15.50:0/139600068 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v171) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7f5804d570 con 0x7f7f600ae3c0 2023-12-19T15:55:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.848+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 msgr2=0x7f7f44062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.848+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 0x7f7f44062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c006fd0 tx=0x7f7f5c008040 comp rx=0 tx=0).stop 2023-12-19T15:55:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.848+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 msgr2=0x7f7f6014a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:06.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.848+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800cef0 tx=0x7f7f58007860 comp rx=0 tx=0).stop 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 shutdown_connections 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f600ad0f0 0x7f7f60169840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7f440603a0 0x7f7f44062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f600ae3c0 0x7f7f6014a5f0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 --2- 172.21.15.50:0/139600068 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7f600acd20 0x7f7f60159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:06.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 >> 172.21.15.50:0/139600068 conn(0x7f7f600cdce0 msgr2=0x7f7f60155500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:06.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 shutdown_connections 2023-12-19T15:55:06.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:06.852+0000 7f7f676f0700 1 -- 172.21.15.50:0/139600068 wait complete. 2023-12-19T15:55:06.858 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T15:55:06.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T15:55:07.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.052+0000 7f0d0328e700 1 Processor -- start 2023-12-19T15:55:07.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.052+0000 7f0d0328e700 1 -- start start 2023-12-19T15:55:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d0328e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc0addc0 0x7f0cfc0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d0328e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc15b140 0x7f0cfc0b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d0328e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc06da90 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d0328e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc06a620 con 0x7f0cfc15b140 2023-12-19T15:55:07.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d0328e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc06dd30 con 0x7f0cfc0addc0 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc15b140 0x7f0cfc0b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc15b140 0x7f0cfc0b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43438/0 (socket says 172.21.15.50:43438) 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 -- 172.21.15.50:0/1981050710 learned_addr learned my addr 172.21.15.50:0/1981050710 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d01a8b700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc0addc0 0x7f0cfc0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 -- 172.21.15.50:0/1981050710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc0addc0 msgr2=0x7f0cfc0c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:07.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc0addc0 0x7f0cfc0c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 -- 172.21.15.50:0/1981050710 >> v1:172.21.15.125:6789/0 conn(0x7f0cfc0ad9f0 legacy=0x7f0cfc06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.056+0000 7f0d02a8d700 1 -- 172.21.15.50:0/1981050710 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc06a8c0 con 0x7f0cfc15b140 2023-12-19T15:55:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d02a8d700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc15b140 0x7f0cfc0b0340 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009f20 tx=0x7f0cf4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a410451dc416cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0128a700 1 -- 172.21.15.50:0/1981050710 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf400e070 con 0x7f0cfc15b140 2023-12-19T15:55:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0128a700 1 -- 172.21.15.50:0/1981050710 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf400b9d0 con 0x7f0cfc15b140 2023-12-19T15:55:07.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0128a700 1 -- 172.21.15.50:0/1981050710 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40092e0 con 0x7f0cfc15b140 2023-12-19T15:55:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 -- 172.21.15.50:0/1981050710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc15b140 msgr2=0x7f0cfc0b0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc15b140 0x7f0cfc0b0340 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009f20 tx=0x7f0cf4009840 comp rx=0 tx=0).stop 2023-12-19T15:55:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 -- 172.21.15.50:0/1981050710 shutdown_connections 2023-12-19T15:55:07.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc15b140 0x7f0cfc0b0340 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 --2- 172.21.15.50:0/1981050710 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc0addc0 0x7f0cfc0c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 -- 172.21.15.50:0/1981050710 >> 172.21.15.50:0/1981050710 conn(0x7f0cfc067bb0 msgr2=0x7f0cfc0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 -- 172.21.15.50:0/1981050710 shutdown_connections 2023-12-19T15:55:07.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.060+0000 7f0d0328e700 1 -- 172.21.15.50:0/1981050710 wait complete. 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 Processor -- start 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 -- start start 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 0x7f0cfc13e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 0x7f0cfc13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc16e560 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc16dc30 con 0x7f0cfc0addc0 2023-12-19T15:55:07.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0328e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc16df10 con 0x7f0cfc15b140 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.064+0000 7f0d0228c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43448/0 (socket says 172.21.15.50:43448) 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 -- 172.21.15.50:0/983968663 learned_addr learned my addr 172.21.15.50:0/983968663 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d02a8d700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 0x7f0cfc13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d01a8b700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 0x7f0cfc13e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 -- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 msgr2=0x7f0cfc13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 0x7f0cfc13eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 -- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 msgr2=0x7f0cfc13e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 0x7f0cfc13e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 -- 172.21.15.50:0/983968663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4009ae0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d02a8d700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 0x7f0cfc13eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:07.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d01a8b700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 0x7f0cfc13e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0228c700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c590 tx=0x7f0cf800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011590 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf80116f0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc16e1f0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011ab0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc0c7550 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.068+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc04be40 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.072+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0cf801b3b0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.072+0000 7f0cf27fc700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 0x7f0ce0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.072+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f0cf807c230 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.072+0000 7f0d01a8b700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 0x7f0ce0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.076+0000 7f0d01a8b700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 0x7f0ce0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc15ac20 tx=0x7f0cec008040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:07.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.080+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0cf804c3d0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.252+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0cfc0c8b70 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.844+0000 7f0cf27fc700 1 -- 172.21.15.50:0/983968663 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v172) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0cf80170d0 con 0x7f0cfc0ad9f0 2023-12-19T15:55:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.852+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 msgr2=0x7f0ce0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.852+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 0x7f0ce0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc15ac20 tx=0x7f0cec008040 comp rx=0 tx=0).stop 2023-12-19T15:55:07.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.852+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 msgr2=0x7f0cfc13b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.852+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c590 tx=0x7f0cf800c950 comp rx=0 tx=0).stop 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.852+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 shutdown_connections 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc0addc0 0x7f0cfc13e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce0060670 0x7f0ce0062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc15b140 0x7f0cfc13eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 --2- 172.21.15.50:0/983968663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc0ad9f0 0x7f0cfc13b460 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:07.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 >> 172.21.15.50:0/983968663 conn(0x7f0cfc067bb0 msgr2=0x7f0cfc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 shutdown_connections 2023-12-19T15:55:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:07.856+0000 7f0d0328e700 1 -- 172.21.15.50:0/983968663 wait complete. 2023-12-19T15:55:07.862 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T15:55:07.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T15:55:08.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.040+0000 7f28befcc700 1 Processor -- start 2023-12-19T15:55:08.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28befcc700 1 -- start start 2023-12-19T15:55:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28befcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 0x7f28b80ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28befcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80adc60 con 0x7f28b813f6a0 2023-12-19T15:55:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28befcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80ad050 con 0x7f28b80c7780 2023-12-19T15:55:08.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28befcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80ad330 con 0x7f28b813f2d0 2023-12-19T15:55:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28bd7c9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f28b813f6a0 0x7f28b815a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60306/0 (socket says 172.21.15.50:60306) 2023-12-19T15:55:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.044+0000 7f28bd7c9700 1 -- 172.21.15.50:0/2637198254 learned_addr learned my addr 172.21.15.50:0/2637198254 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:08.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bdfca700 1 --2- 172.21.15.50:0/2637198254 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 0x7f28b80ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431860379 0 0) 0x7f28b80ad050 con 0x7f28b80c7780 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a4003570 con 0x7f28b80c7780 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326165725 0 0) 0x7f28b80adc60 con 0x7f28b813f6a0 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b80ad050 con 0x7f28b813f6a0 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bdfca700 1 -- 172.21.15.50:0/2637198254 >> v1:172.21.15.125:6789/0 conn(0x7f28b80c7780 legacy=0x7f28b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bdfca700 1 -- 172.21.15.50:0/2637198254 >> v1:172.21.15.50:6789/0 conn(0x7f28b813f6a0 legacy=0x7f28b815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bdfca700 1 -- 172.21.15.50:0/2637198254 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b80ad610 con 0x7f28b813f2d0 2023-12-19T15:55:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bdfca700 1 --2- 172.21.15.50:0/2637198254 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 0x7f28b80ac9b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009910 tx=0x7f28b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d7fda74f3bf64bb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4011630 con 0x7f28b813f2d0 2023-12-19T15:55:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b4011790 con 0x7f28b813f2d0 2023-12-19T15:55:08.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28bcfc8700 1 -- 172.21.15.50:0/2637198254 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4011b30 con 0x7f28b813f2d0 2023-12-19T15:55:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28befcc700 1 -- 172.21.15.50:0/2637198254 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 msgr2=0x7f28b80ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.048+0000 7f28befcc700 1 --2- 172.21.15.50:0/2637198254 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 0x7f28b80ac9b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009910 tx=0x7f28b40091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- 172.21.15.50:0/2637198254 shutdown_connections 2023-12-19T15:55:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 --2- 172.21.15.50:0/2637198254 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b813f2d0 0x7f28b80ac9b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- 172.21.15.50:0/2637198254 >> 172.21.15.50:0/2637198254 conn(0x7f28b8067b50 msgr2=0x7f28b80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- 172.21.15.50:0/2637198254 shutdown_connections 2023-12-19T15:55:08.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- 172.21.15.50:0/2637198254 wait complete. 2023-12-19T15:55:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 Processor -- start 2023-12-19T15:55:08.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- start start 2023-12-19T15:55:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b80c7780 0x7f28b813d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 0x7f28b8169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80a78c0 con 0x7f28b813f2d0 2023-12-19T15:55:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b813dcd0 con 0x7f28b813f6a0 2023-12-19T15:55:08.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.052+0000 7f28befcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80a80f0 con 0x7f28b80c7780 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43454/0 (socket says 172.21.15.50:43454) 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 -- 172.21.15.50:0/3924257008 learned_addr learned my addr 172.21.15.50:0/3924257008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bdfca700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b80c7780 0x7f28b813d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28be7cb700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 0x7f28b8169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 -- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b80c7780 msgr2=0x7f28b813d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b80c7780 0x7f28b813d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 -- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 msgr2=0x7f28b8169d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 0x7f28b8169d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 -- 172.21.15.50:0/3924257008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b4009420 con 0x7f28b813f2d0 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28be7cb700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 0x7f28b8169d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28bd7c9700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c7c0 tx=0x7f28a8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8005b30 con 0x7f28b813f2d0 2023-12-19T15:55:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a8005c90 con 0x7f28b813f2d0 2023-12-19T15:55:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b80a9e40 con 0x7f28b813f2d0 2023-12-19T15:55:08.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8007320 con 0x7f28b813f2d0 2023-12-19T15:55:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b80a7d80 con 0x7f28b813f2d0 2023-12-19T15:55:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b80a82a0 con 0x7f28b813f2d0 2023-12-19T15:55:08.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.056+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f28a8007690 con 0x7f28b813f2d0 2023-12-19T15:55:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.060+0000 7f28adffb700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 0x7f289c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.060+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7f28a807bf70 con 0x7f28b813f2d0 2023-12-19T15:55:08.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.060+0000 7f28bdfca700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 0x7f289c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.060+0000 7f28bdfca700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 0x7f289c062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28b806d200 tx=0x7f28b4017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.068+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f28b80a82a0 con 0x7f28b813f2d0 2023-12-19T15:55:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.240+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f28b80a8260 con 0x7f28b813f2d0 2023-12-19T15:55:08.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.244+0000 7f28adffb700 1 -- 172.21.15.50:0/3924257008 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f28a804c350 con 0x7f28b813f2d0 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 msgr2=0x7f289c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 0x7f289c062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28b806d200 tx=0x7f28b4017000 comp rx=0 tx=0).stop 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 msgr2=0x7f28b80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c7c0 tx=0x7f28a8007e00 comp rx=0 tx=0).stop 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 shutdown_connections 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b813f6a0 0x7f28b8169d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f289c060670 0x7f289c062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28b813f2d0 0x7f28b80a5410 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 --2- 172.21.15.50:0/3924257008 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f28b80c7780 0x7f28b813d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 >> 172.21.15.50:0/3924257008 conn(0x7f28b8067b50 msgr2=0x7f28b80b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:08.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 shutdown_connections 2023-12-19T15:55:08.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.248+0000 7f28befcc700 1 -- 172.21.15.50:0/3924257008 wait complete. 2023-12-19T15:55:08.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T15:55:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.432+0000 7f8009345700 1 Processor -- start 2023-12-19T15:55:08.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.432+0000 7f8009345700 1 -- start start 2023-12-19T15:55:08.437 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8009345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f80040c6d20 0x7f80040c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8009345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f800406ac30 0x7f800406b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8009345700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800406a7a0 con 0x7f800406c7c0 2023-12-19T15:55:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8009345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040c7630 con 0x7f800406ac30 2023-12-19T15:55:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8009345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800406d4b0 con 0x7f80040c6d20 2023-12-19T15:55:08.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f800406ac30 0x7f800406b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f800406ac30 0x7f800406b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43470/0 (socket says 172.21.15.50:43470) 2023-12-19T15:55:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 -- 172.21.15.50:0/2789934754 learned_addr learned my addr 172.21.15.50:0/2789934754 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f8002ffd700 1 --2- 172.21.15.50:0/2789934754 >> v2:172.21.15.125:3300/0 conn(0x7f80040c6d20 0x7f80040c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 -- 172.21.15.50:0/2789934754 >> v2:172.21.15.125:3300/0 conn(0x7f80040c6d20 msgr2=0x7f80040c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 --2- 172.21.15.50:0/2789934754 >> v2:172.21.15.125:3300/0 conn(0x7f80040c6d20 0x7f80040c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 -- 172.21.15.50:0/2789934754 >> v1:172.21.15.125:6789/0 conn(0x7f800406c7c0 legacy=0x7f800406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 -- 172.21.15.50:0/2789934754 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800406d790 con 0x7f800406ac30 2023-12-19T15:55:08.440 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.436+0000 7f80037fe700 1 --2- 172.21.15.50:0/2789934754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f800406b000 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8004068030 tx=0x7f7ffc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a290f41aaad2d31a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8001ffb700 1 -- 172.21.15.50:0/2789934754 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00e070 con 0x7f800406ac30 2023-12-19T15:55:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8001ffb700 1 -- 172.21.15.50:0/2789934754 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc00ba20 con 0x7f800406ac30 2023-12-19T15:55:08.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8001ffb700 1 -- 172.21.15.50:0/2789934754 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0092e0 con 0x7f800406ac30 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- 172.21.15.50:0/2789934754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 msgr2=0x7f800406b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 --2- 172.21.15.50:0/2789934754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f800406b000 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8004068030 tx=0x7f7ffc009840 comp rx=0 tx=0).stop 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- 172.21.15.50:0/2789934754 shutdown_connections 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 --2- 172.21.15.50:0/2789934754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f800406b000 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 --2- 172.21.15.50:0/2789934754 >> v2:172.21.15.125:3300/0 conn(0x7f80040c6d20 0x7f80040c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- 172.21.15.50:0/2789934754 >> 172.21.15.50:0/2789934754 conn(0x7f8004067bb0 msgr2=0x7f80040cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- 172.21.15.50:0/2789934754 shutdown_connections 2023-12-19T15:55:08.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- 172.21.15.50:0/2789934754 wait complete. 2023-12-19T15:55:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 Processor -- start 2023-12-19T15:55:08.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.440+0000 7f8009345700 1 -- start start 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800406c7c0 0x7f80040abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f80040c6d20 0x7f80040ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040a9140 con 0x7f800406ac30 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040a8850 con 0x7f800406c7c0 2023-12-19T15:55:08.446 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8004148ce0 con 0x7f80040c6d20 2023-12-19T15:55:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43472/0 (socket says 172.21.15.50:43472) 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 -- 172.21.15.50:0/2931768158 learned_addr learned my addr 172.21.15.50:0/2931768158 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f80027fc700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800406c7c0 0x7f80040abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f80037fe700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f80040c6d20 0x7f80040ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 -- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f80040c6d20 msgr2=0x7f80040ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f80040c6d20 0x7f80040ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 -- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800406c7c0 msgr2=0x7f80040abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800406c7c0 0x7f80040abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 -- 172.21.15.50:0/2931768158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009ae0 con 0x7f800406ac30 2023-12-19T15:55:08.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8002ffd700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400e930 tx=0x7f7ff40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4013680 con 0x7f800406ac30 2023-12-19T15:55:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff40137e0 con 0x7f800406ac30 2023-12-19T15:55:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4013b80 con 0x7f800406ac30 2023-12-19T15:55:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80040c5a90 con 0x7f800406ac30 2023-12-19T15:55:08.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.444+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80040a8dd0 con 0x7f800406ac30 2023-12-19T15:55:08.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.448+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8004071120 con 0x7f800406ac30 2023-12-19T15:55:08.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.448+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7ff400c3e0 con 0x7f800406ac30 2023-12-19T15:55:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.452+0000 7f7feaffd700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 0x7f7ff0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.452+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7f7ff407cad0 con 0x7f800406ac30 2023-12-19T15:55:08.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.452+0000 7f80027fc700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 0x7f7ff0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.452+0000 7f80027fc700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 0x7f7ff0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8004157540 tx=0x7f7ff8007ec0 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.456+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7ff404ceb0 con 0x7f800406ac30 2023-12-19T15:55:08.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.632+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f800406da70 con 0x7f800406ac30 2023-12-19T15:55:08.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.636+0000 7f7feaffd700 1 -- 172.21.15.50:0/2931768158 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v56) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7ff40170b0 con 0x7f800406ac30 2023-12-19T15:55:08.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 msgr2=0x7f7ff0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 0x7f7ff0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8004157540 tx=0x7f7ff8007ec0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 msgr2=0x7f80041487a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400e930 tx=0x7f7ff40091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 shutdown_connections 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800406c7c0 0x7f80040abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f7ff0060670 0x7f7ff0062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f80040c6d20 0x7f80040ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 --2- 172.21.15.50:0/2931768158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f800406ac30 0x7f80041487a0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 >> 172.21.15.50:0/2931768158 conn(0x7f8004067bb0 msgr2=0x7f8004159710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 shutdown_connections 2023-12-19T15:55:08.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.640+0000 7f8009345700 1 -- 172.21.15.50:0/2931768158 wait complete. 2023-12-19T15:55:08.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T15:55:08.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.832+0000 7fe5b36cd700 1 Processor -- start 2023-12-19T15:55:08.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.836+0000 7fe5b36cd700 1 -- start start 2023-12-19T15:55:08.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe5ac0cecd0 0x7fe5ac06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac0cbdf0 con 0x7fe5ac158e20 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac0cc0d0 con 0x7fe5ac0cb970 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ac0cc3b0 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1469700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe5ac158e20 0x7fe5ac0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60320/0 (socket says 172.21.15.50:60320) 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1469700 1 -- 172.21.15.50:0/2040342742 learned_addr learned my addr 172.21.15.50:0/2040342742 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:08.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1c6a700 1 --2- 172.21.15.50:0/2040342742 >> v2:172.21.15.50:3300/0 conn(0x7fe5ac0cecd0 0x7fe5ac06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5a3fff700 1 -- 172.21.15.50:0/2040342742 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651932939 0 0) 0x7fe5ac0cbdf0 con 0x7fe5ac158e20 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5a3fff700 1 -- 172.21.15.50:0/2040342742 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe598003570 con 0x7fe5ac158e20 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1c6a700 1 -- 172.21.15.50:0/2040342742 >> v1:172.21.15.125:6789/0 conn(0x7fe5ac0cb970 legacy=0x7fe5ac06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1c6a700 1 -- 172.21.15.50:0/2040342742 >> v1:172.21.15.50:6789/0 conn(0x7fe5ac158e20 legacy=0x7fe5ac0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1c6a700 1 -- 172.21.15.50:0/2040342742 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac0cc690 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b1c6a700 1 --2- 172.21.15.50:0/2040342742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac0cecd0 0x7fe5ac06b3d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009e90 tx=0x7fe5a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51fda20a7d77c77 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5a3fff700 1 -- 172.21.15.50:0/2040342742 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a400e070 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5a3fff700 1 -- 172.21.15.50:0/2040342742 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a400bc40 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5a3fff700 1 -- 172.21.15.50:0/2040342742 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012430 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- 172.21.15.50:0/2040342742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac0cecd0 msgr2=0x7fe5ac06b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/2040342742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac0cecd0 0x7fe5ac06b3d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009e90 tx=0x7fe5a4009780 comp rx=0 tx=0).stop 2023-12-19T15:55:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- 172.21.15.50:0/2040342742 shutdown_connections 2023-12-19T15:55:08.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/2040342742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac0cecd0 0x7fe5ac06b3d0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.840+0000 7fe5b36cd700 1 -- 172.21.15.50:0/2040342742 >> 172.21.15.50:0/2040342742 conn(0x7fe5ac067bb0 msgr2=0x7fe5ac0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:08.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.844+0000 7fe5b36cd700 1 -- 172.21.15.50:0/2040342742 shutdown_connections 2023-12-19T15:55:08.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.844+0000 7fe5b36cd700 1 -- 172.21.15.50:0/2040342742 wait complete. 2023-12-19T15:55:08.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.844+0000 7fe5b36cd700 1 Processor -- start 2023-12-19T15:55:08.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.844+0000 7fe5b36cd700 1 -- start start 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5ac0cecd0 0x7fe5ac14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5ac0d0530 0x7fe5ac0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac0d09d0 con 0x7fe5ac158e20 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac0c7190 con 0x7fe5ac0d0530 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b36cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac0b5970 con 0x7fe5ac0cecd0 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43502/0 (socket says 172.21.15.50:43502) 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 -- 172.21.15.50:0/3630088305 learned_addr learned my addr 172.21.15.50:0/3630088305 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b1469700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5ac0cecd0 0x7fe5ac14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b1c6a700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5ac0d0530 0x7fe5ac0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 -- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5ac0cecd0 msgr2=0x7fe5ac14c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5ac0cecd0 0x7fe5ac14c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 -- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5ac0d0530 msgr2=0x7fe5ac0b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5ac0d0530 0x7fe5ac0b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 -- 172.21.15.50:0/3630088305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4009a20 con 0x7fe5ac158e20 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5b0c68700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c590 tx=0x7fe59c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c011590 con 0x7fe5ac158e20 2023-12-19T15:55:08.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.848+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c011710 con 0x7fe5ac158e20 2023-12-19T15:55:08.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.852+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac0b5c50 con 0x7fe5ac158e20 2023-12-19T15:55:08.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.852+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c011ad0 con 0x7fe5ac158e20 2023-12-19T15:55:08.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.852+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac0c7680 con 0x7fe5ac158e20 2023-12-19T15:55:08.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.852+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ac0c78a0 con 0x7fe5ac158e20 2023-12-19T15:55:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.852+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe59c01b3b0 con 0x7fe5ac158e20 2023-12-19T15:55:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.856+0000 7fe5a17fa700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 0x7fe590062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.856+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7fe59c07bec0 con 0x7fe5ac158e20 2023-12-19T15:55:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.856+0000 7fe5b1469700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 0x7fe590062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.856+0000 7fe5b1469700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 0x7fe590062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80058d0 tx=0x7fe5a8009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:08.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:08.860+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe59c04c200 con 0x7fe5ac158e20 2023-12-19T15:55:09.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.048+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe5ac0b5e40 con 0x7fe5ac158e20 2023-12-19T15:55:09.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5a17fa700 1 -- 172.21.15.50:0/3630088305 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe5ac0b5e40 con 0x7fe5ac158e20 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 msgr2=0x7fe590062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 0x7fe590062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80058d0 tx=0x7fe5a8009040 comp rx=0 tx=0).stop 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 msgr2=0x7fe5ac0cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c590 tx=0x7fe59c007f50 comp rx=0 tx=0).stop 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 shutdown_connections 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5ac0d0530 0x7fe5ac0b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe5900606a0 0x7fe590062b50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5ac158e20 0x7fe5ac0cfff0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.060+0000 7fe5b36cd700 1 --2- 172.21.15.50:0/3630088305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe5ac0cecd0 0x7fe5ac14c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.064+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 >> 172.21.15.50:0/3630088305 conn(0x7fe5ac067bb0 msgr2=0x7fe5ac06b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.064+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 shutdown_connections 2023-12-19T15:55:09.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.064+0000 7fe5b36cd700 1 -- 172.21.15.50:0/3630088305 wait complete. 2023-12-19T15:55:09.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-12-19T15:55:09.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.244+0000 7f0d018a5700 1 Processor -- start 2023-12-19T15:55:09.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.244+0000 7f0d018a5700 1 -- start start 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cfc158c10 0x7f0cfc158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc06de60 0x7f0cfc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06b0c0 0x7f0cfc06b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc06ac30 con 0x7f0cfc06de60 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc0ccc80 con 0x7f0cfc158c10 2023-12-19T15:55:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0d018a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc06d180 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc06de60 0x7f0cfc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0cfc06de60 0x7f0cfc0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43504/0 (socket says 172.21.15.50:43504) 2023-12-19T15:55:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 -- 172.21.15.50:0/769560637 learned_addr learned my addr 172.21.15.50:0/769560637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfaffd700 1 --2- 172.21.15.50:0/769560637 >> v2:172.21.15.125:3300/0 conn(0x7f0cfc158c10 0x7f0cfc158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfb7fe700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06b0c0 0x7f0cfc06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 -- 172.21.15.50:0/769560637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06b0c0 msgr2=0x7f0cfc06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06b0c0 0x7f0cfc06b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 -- 172.21.15.50:0/769560637 >> v2:172.21.15.125:3300/0 conn(0x7f0cfc158c10 msgr2=0x7f0cfc158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/769560637 >> v2:172.21.15.125:3300/0 conn(0x7f0cfc158c10 0x7f0cfc158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 -- 172.21.15.50:0/769560637 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc06a620 con 0x7f0cfc06de60 2023-12-19T15:55:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.248+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06de60 0x7f0cfc0cc740 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0b5ec0 tx=0x7f0cf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f379b017a8a655 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0cf9ffb700 1 -- 172.21.15.50:0/769560637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000b640 con 0x7f0cfc06de60 2023-12-19T15:55:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0cf9ffb700 1 -- 172.21.15.50:0/769560637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000b7a0 con 0x7f0cfc06de60 2023-12-19T15:55:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0cf9ffb700 1 -- 172.21.15.50:0/769560637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000bb60 con 0x7f0cfc06de60 2023-12-19T15:55:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 -- 172.21.15.50:0/769560637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06de60 msgr2=0x7f0cfc0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06de60 0x7f0cfc0cc740 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0b5ec0 tx=0x7f0cf0009250 comp rx=0 tx=0).stop 2023-12-19T15:55:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 -- 172.21.15.50:0/769560637 shutdown_connections 2023-12-19T15:55:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06de60 0x7f0cfc0cc740 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 --2- 172.21.15.50:0/769560637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06b0c0 0x7f0cfc06b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 --2- 172.21.15.50:0/769560637 >> v2:172.21.15.125:3300/0 conn(0x7f0cfc158c10 0x7f0cfc158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 -- 172.21.15.50:0/769560637 >> 172.21.15.50:0/769560637 conn(0x7f0cfc0d0920 msgr2=0x7f0cfc0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 -- 172.21.15.50:0/769560637 shutdown_connections 2023-12-19T15:55:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 -- 172.21.15.50:0/769560637 wait complete. 2023-12-19T15:55:09.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.252+0000 7f0d018a5700 1 Processor -- start 2023-12-19T15:55:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- start start 2023-12-19T15:55:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06de60 0x7f0cfc1407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc158c10 0x7f0cfc15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc156940 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc155d30 con 0x7f0cfc158c10 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc156010 con 0x7f0cfc06de60 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43516/0 (socket says 172.21.15.50:43516) 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 -- 172.21.15.50:0/3444844279 learned_addr learned my addr 172.21.15.50:0/3444844279 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06de60 0x7f0cfc1407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfb7fe700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc158c10 0x7f0cfc15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 -- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06de60 msgr2=0x7f0cfc1407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06de60 0x7f0cfc1407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 -- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc158c10 msgr2=0x7f0cfc15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc158c10 0x7f0cfc15d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 -- 172.21.15.50:0/3444844279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00094f0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0cfaffd700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00ec80 tx=0x7f0cec009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00c850 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec00c9b0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec00cd30 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc1562f0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.256+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc140d30 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.260+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc04be40 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.260+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0cec013680 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.264+0000 7f0ce2ffd700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 0x7f0ce8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.264+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7f0cec07cfd0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.264+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 0x7f0ce8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.264+0000 7f0cfa7fc700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 0x7f0ce8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0b5ec0 tx=0x7f0cf0017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:09.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.268+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0cec04d3b0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.500+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f0cfc0cc3f0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.852+0000 7f0ce2ffd700 1 -- 172.21.15.50:0/3444844279 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2477) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0cec0170f0 con 0x7f0cfc06b0c0 2023-12-19T15:55:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 msgr2=0x7f0ce8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 0x7f0ce8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0b5ec0 tx=0x7f0cf0017000 comp rx=0 tx=0).stop 2023-12-19T15:55:09.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 msgr2=0x7f0cfc156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00ec80 tx=0x7f0cec009120 comp rx=0 tx=0).stop 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 shutdown_connections 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cfc158c10 0x7f0cfc15d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f0ce8060670 0x7f0ce8062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0cfc06de60 0x7f0cfc1407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 --2- 172.21.15.50:0/3444844279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0cfc06b0c0 0x7f0cfc156dd0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 >> 172.21.15.50:0/3444844279 conn(0x7f0cfc0d0920 msgr2=0x7f0cfc153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 shutdown_connections 2023-12-19T15:55:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:09.856+0000 7f0d018a5700 1 -- 172.21.15.50:0/3444844279 wait complete. 2023-12-19T15:55:09.884 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-12-19T15:55:09.885 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-12-19T15:55:09.885 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T15:55:09.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:10.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.048+0000 7fa963e2e700 1 Processor -- start 2023-12-19T15:55:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.048+0000 7fa963e2e700 1 -- start start 2023-12-19T15:55:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa963e2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa95c0ae140 0x7fa95c0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa963e2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c06dc20 con 0x7fa95c0c1d60 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa963e2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c06d940 con 0x7fa95c0ae510 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa963e2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0c2200 con 0x7fa95c0ae140 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa9623cb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fa95c0c1d60 0x7fa95c0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60328/0 (socket says 172.21.15.50:60328) 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa9623cb700 1 -- 172.21.15.50:0/4091805421 learned_addr learned my addr 172.21.15.50:0/4091805421 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:10.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa961bca700 1 --2- 172.21.15.50:0/4091805421 >> v2:172.21.15.125:3300/0 conn(0x7fa95c0ae140 0x7fa95c0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935032549 0 0) 0x7fa95c06dc20 con 0x7fa95c0c1d60 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa948003570 con 0x7fa95c0c1d60 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093195482 0 0) 0x7fa95c06d940 con 0x7fa95c0ae510 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa95c06dc20 con 0x7fa95c0ae510 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa961bca700 1 -- 172.21.15.50:0/4091805421 >> v1:172.21.15.125:6789/0 conn(0x7fa95c0ae510 legacy=0x7fa95c06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa961bca700 1 -- 172.21.15.50:0/4091805421 >> v1:172.21.15.50:6789/0 conn(0x7fa95c0c1d60 legacy=0x7fa95c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.052+0000 7fa961bca700 1 -- 172.21.15.50:0/4091805421 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0c6e20 con 0x7fa95c0ae140 2023-12-19T15:55:10.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa961bca700 1 --2- 172.21.15.50:0/4091805421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0ae140 0x7fa95c0a8850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098e0 tx=0x7fa9580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e462974bf7ef5c5f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958011630 con 0x7fa95c0ae140 2023-12-19T15:55:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958011790 con 0x7fa95c0ae140 2023-12-19T15:55:10.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa960bc8700 1 -- 172.21.15.50:0/4091805421 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958011b30 con 0x7fa95c0ae140 2023-12-19T15:55:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 -- 172.21.15.50:0/4091805421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0ae140 msgr2=0x7fa95c0a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 --2- 172.21.15.50:0/4091805421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0ae140 0x7fa95c0a8850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098e0 tx=0x7fa9580091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 -- 172.21.15.50:0/4091805421 shutdown_connections 2023-12-19T15:55:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 --2- 172.21.15.50:0/4091805421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0ae140 0x7fa95c0a8850 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 -- 172.21.15.50:0/4091805421 >> 172.21.15.50:0/4091805421 conn(0x7fa95c0c8850 msgr2=0x7fa95c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 -- 172.21.15.50:0/4091805421 shutdown_connections 2023-12-19T15:55:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.056+0000 7fa963e2e700 1 -- 172.21.15.50:0/4091805421 wait complete. 2023-12-19T15:55:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 Processor -- start 2023-12-19T15:55:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 -- start start 2023-12-19T15:55:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 0x7fa95c0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 0x7fa95c15ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ac620 con 0x7fa95c0ae140 2023-12-19T15:55:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ac880 con 0x7fa95c0c1d60 2023-12-19T15:55:10.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.060+0000 7fa963e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0acb60 con 0x7fa95c0ae510 2023-12-19T15:55:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43526/0 (socket says 172.21.15.50:43526) 2023-12-19T15:55:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 -- 172.21.15.50:0/2539249169 learned_addr learned my addr 172.21.15.50:0/2539249169 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa9613c9700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 0x7fa95c0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa9623cb700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 0x7fa95c15ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 -- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 msgr2=0x7fa95c0ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 0x7fa95c0ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 -- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 msgr2=0x7fa95c15ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 0x7fa95c15ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 -- 172.21.15.50:0/2539249169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa958009420 con 0x7fa95c0ae140 2023-12-19T15:55:10.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa9613c9700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 0x7fa95c0ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa9623cb700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 0x7fa95c15ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa961bca700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa95c06a2b0 tx=0x7fa958017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95801f070 con 0x7fa95c0ae140 2023-12-19T15:55:10.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958000c90 con 0x7fa95c0ae140 2023-12-19T15:55:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958025420 con 0x7fa95c0ae140 2023-12-19T15:55:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c14f430 con 0x7fa95c0ae140 2023-12-19T15:55:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.064+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c14f9f0 con 0x7fa95c0ae140 2023-12-19T15:55:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.068+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c071120 con 0x7fa95c0ae140 2023-12-19T15:55:10.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.068+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa958023070 con 0x7fa95c0ae140 2023-12-19T15:55:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.068+0000 7fa951ffb700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 0x7fa940062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.072+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7fa95807deb0 con 0x7fa95c0ae140 2023-12-19T15:55:10.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.072+0000 7fa9613c9700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 0x7fa940062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.072+0000 7fa9613c9700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 0x7fa940062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007fd0 tx=0x7fa94c009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.076+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa95804e1f0 con 0x7fa95c0ae140 2023-12-19T15:55:10.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.260+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa95c0642a0 con 0x7fa95c0ae140 2023-12-19T15:55:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.260+0000 7fa951ffb700 1 -- 172.21.15.50:0/2539249169 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+13504 (secure 0 0 0) 0x7fa95802a090 con 0x7fa95c0ae140 2023-12-19T15:55:10.263 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:10.263 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":172,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:07.841982+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:6836/4138116501":"2023-12-20T15:55:04.816704+0000","172.21.15.125:0/2510569342":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/278018654":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2856526150":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2634924093":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2213656349":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/2640199441":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/1152230135":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6834/4138116501":"2023-12-20T15:55:04.816704+0000","172.21.15.125:0/373489315":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/669887858":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/810172491":"2023-12-20T15:55:04.270549+0000","172.21.15.125:0/219377965":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/2706791219":"2023-12-20T15:54:56.619577+0000","172.21.15.125:0/1963925549":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6833/33827":"2023-12-20T15:54:56.619577+0000","172.21.15.125:6832/33827":"2023-12-20T15:54:56.619577+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 msgr2=0x7fa940062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 0x7fa940062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007fd0 tx=0x7fa94c009040 comp rx=0 tx=0).stop 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 msgr2=0x7fa95c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa95c06a2b0 tx=0x7fa958017000 comp rx=0 tx=0).stop 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 shutdown_connections 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa95c0c1d60 0x7fa95c15ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fa940060670 0x7fa940062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa95c0ae510 0x7fa95c0ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 --2- 172.21.15.50:0/2539249169 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa95c0ae140 0x7fa95c15e960 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 >> 172.21.15.50:0/2539249169 conn(0x7fa95c0c8850 msgr2=0x7fa95c0a8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 shutdown_connections 2023-12-19T15:55:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.264+0000 7fa963e2e700 1 -- 172.21.15.50:0/2539249169 wait complete. 2023-12-19T15:55:10.292 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T15:55:10.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:55:10.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.468+0000 7fe05588e700 1 Processor -- start 2023-12-19T15:55:10.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.468+0000 7fe05588e700 1 -- start start 2023-12-19T15:55:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05588e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe05006aeb0 0x7fe050159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05588e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05006c850 con 0x7fe0501595a0 2023-12-19T15:55:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05588e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0500cb650 con 0x7fe050158c90 2023-12-19T15:55:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05588e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05006b470 con 0x7fe05006aeb0 2023-12-19T15:55:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05508d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe0501595a0 0x7fe05006cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60338/0 (socket says 172.21.15.50:60338) 2023-12-19T15:55:10.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe05508d700 1 -- 172.21.15.50:0/4162381601 learned_addr learned my addr 172.21.15.50:0/4162381601 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:10.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04ffff700 1 --2- 172.21.15.50:0/4162381601 >> v2:172.21.15.125:3300/0 conn(0x7fe05006aeb0 0x7fe050159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559738219 0 0) 0x7fe05006c850 con 0x7fe0501595a0 2023-12-19T15:55:10.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe040003570 con 0x7fe0501595a0 2023-12-19T15:55:10.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119774844 0 0) 0x7fe0500cb650 con 0x7fe050158c90 2023-12-19T15:55:10.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05006c850 con 0x7fe050158c90 2023-12-19T15:55:10.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 322076752 0 0) 0x7fe040003570 con 0x7fe0501595a0 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 >> v2:172.21.15.125:3300/0 conn(0x7fe05006aeb0 msgr2=0x7fe050159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 --2- 172.21.15.50:0/4162381601 >> v2:172.21.15.125:3300/0 conn(0x7fe05006aeb0 0x7fe050159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 >> v1:172.21.15.125:6789/0 conn(0x7fe050158c90 legacy=0x7fe0500caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05006b750 con 0x7fe0501595a0 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fe0480017f0 con 0x7fe0501595a0 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 >> v1:172.21.15.50:6789/0 conn(0x7fe0501595a0 legacy=0x7fe05006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe040004c50 0x7fe040005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 0x7fe040003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe040004160 0x7fe04000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05006b470 con 0x7fe0400037b0 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04000c0b0 con 0x7fe040004160 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.472+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04000c270 con 0x7fe040004c50 2023-12-19T15:55:10.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 0x7fe040003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05508d700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe040004160 0x7fe04000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05488c700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe040004c50 0x7fe040005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 -- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe040004c50 msgr2=0x7fe040005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe040004c50 0x7fe040005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 -- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe040004160 msgr2=0x7fe04000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe040004160 0x7fe04000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 -- 172.21.15.50:0/4162381601 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040003570 con 0x7fe0400037b0 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05488c700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe040004c50 0x7fe040005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05508d700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe040004160 0x7fe04000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:10.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05508d700 1 -- 172.21.15.50:0/4162381601 reap_dead start 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04ffff700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 0x7fe040003c20 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009c00 tx=0x7fe03c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdfeff61088fda0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00e710 con 0x7fe0400037b0 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c00e870 con 0x7fe0400037b0 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe04f7fe700 1 -- 172.21.15.50:0/4162381601 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00ec30 con 0x7fe0400037b0 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 -- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 msgr2=0x7fe040003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 0x7fe040003c20 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009c00 tx=0x7fe03c0091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 -- 172.21.15.50:0/4162381601 shutdown_connections 2023-12-19T15:55:10.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 --2- 172.21.15.50:0/4162381601 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0400037b0 0x7fe040003c20 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 -- 172.21.15.50:0/4162381601 >> 172.21.15.50:0/4162381601 conn(0x7fe050067bb0 msgr2=0x7fe0500cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:10.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 -- 172.21.15.50:0/4162381601 shutdown_connections 2023-12-19T15:55:10.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.476+0000 7fe05588e700 1 -- 172.21.15.50:0/4162381601 wait complete. 2023-12-19T15:55:10.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 Processor -- start 2023-12-19T15:55:10.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 -- start start 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 0x7fe0500c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe05016c4c0 0x7fe050152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05016ca30 con 0x7fe0501531d0 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05016cd10 con 0x7fe05016c4c0 2023-12-19T15:55:10.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05588e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05016cff0 con 0x7fe0400037b0 2023-12-19T15:55:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05508d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05508d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43552/0 (socket says 172.21.15.50:43552) 2023-12-19T15:55:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.480+0000 7fe05508d700 1 -- 172.21.15.50:0/3554127041 learned_addr learned my addr 172.21.15.50:0/3554127041 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05488c700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 0x7fe0500c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe04ffff700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe05016c4c0 0x7fe050152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 -- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 msgr2=0x7fe0500c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 0x7fe0500c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 -- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe05016c4c0 msgr2=0x7fe050152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe05016c4c0 0x7fe050152c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 -- 172.21.15.50:0/3554127041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05016d2d0 con 0x7fe0501531d0 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05488c700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 0x7fe0500c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05508d700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe05006e2e0 tx=0x7fe048009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04800bc50 con 0x7fe0501531d0 2023-12-19T15:55:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04800bdb0 con 0x7fe0501531d0 2023-12-19T15:55:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe050154bb0 con 0x7fe0501531d0 2023-12-19T15:55:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048004e00 con 0x7fe0501531d0 2023-12-19T15:55:10.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.484+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe050150d30 con 0x7fe0501531d0 2023-12-19T15:55:10.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.488+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05004be40 con 0x7fe0501531d0 2023-12-19T15:55:10.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.488+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe0480073a0 con 0x7fe0501531d0 2023-12-19T15:55:10.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.488+0000 7fe04cff9700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 0x7fe034062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.488+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7fe04807ca90 con 0x7fe0501531d0 2023-12-19T15:55:10.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.492+0000 7fe05488c700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 0x7fe034062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.492+0000 7fe05488c700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 0x7fe034062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe050060be0 tx=0x7fe044009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.496+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe04804abe0 con 0x7fe0501531d0 2023-12-19T15:55:10.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.664+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe05006ba30 con 0x7fe0501531d0 2023-12-19T15:55:10.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.664+0000 7fe04cff9700 1 -- 172.21.15.50:0/3554127041 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v172) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe04804cde0 con 0x7fe0501531d0 2023-12-19T15:55:10.669 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:55:10.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 msgr2=0x7fe034062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 0x7fe034062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe050060be0 tx=0x7fe044009040 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 msgr2=0x7fe0501507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe05006e2e0 tx=0x7fe048009040 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 shutdown_connections 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe05016c4c0 0x7fe050152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe034060670 0x7fe034062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0501531d0 0x7fe0501507f0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 --2- 172.21.15.50:0/3554127041 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe0400037b0 0x7fe0500c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 >> 172.21.15.50:0/3554127041 conn(0x7fe050067bb0 msgr2=0x7fe0501599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 shutdown_connections 2023-12-19T15:55:10.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.668+0000 7fe05588e700 1 -- 172.21.15.50:0/3554127041 wait complete. 2023-12-19T15:55:10.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-12-19T15:55:10.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 Processor -- start 2023-12-19T15:55:10.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 -- start start 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8438067c90 0x7f8438068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f843814fba0 0x7f8438150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84380c5900 0x7f84380c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84380c6630 con 0x7f8438067c90 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84380c6910 con 0x7f843814fba0 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843cf45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84380c6bf0 con 0x7f84380c5900 2023-12-19T15:55:10.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8438067c90 0x7f8438068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8438067c90 0x7f8438068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43562/0 (socket says 172.21.15.50:43562) 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 -- 172.21.15.50:0/562798641 learned_addr learned my addr 172.21.15.50:0/562798641 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f8436d9d700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84380c5900 0x7f84380c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f8435d9b700 1 --2- 172.21.15.50:0/562798641 >> v2:172.21.15.125:3300/0 conn(0x7f843814fba0 0x7f8438150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 -- 172.21.15.50:0/562798641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84380c5900 msgr2=0x7f84380c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84380c5900 0x7f84380c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 -- 172.21.15.50:0/562798641 >> v2:172.21.15.125:3300/0 conn(0x7f843814fba0 msgr2=0x7f8438150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 --2- 172.21.15.50:0/562798641 >> v2:172.21.15.125:3300/0 conn(0x7f843814fba0 0x7f8438150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.864+0000 7f843659c700 1 -- 172.21.15.50:0/562798641 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380c6ed0 con 0x7f8438067c90 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843659c700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438067c90 0x7f8438068060 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f84380c56b0 tx=0x7f8428009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6783169a80774a95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843559a700 1 -- 172.21.15.50:0/562798641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800b860 con 0x7f8438067c90 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843559a700 1 -- 172.21.15.50:0/562798641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842800b9c0 con 0x7f8438067c90 2023-12-19T15:55:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843559a700 1 -- 172.21.15.50:0/562798641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800bd60 con 0x7f8438067c90 2023-12-19T15:55:10.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 -- 172.21.15.50:0/562798641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438067c90 msgr2=0x7f8438068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438067c90 0x7f8438068060 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f84380c56b0 tx=0x7f8428009250 comp rx=0 tx=0).stop 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 -- 172.21.15.50:0/562798641 shutdown_connections 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f84380c5900 0x7f84380c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 --2- 172.21.15.50:0/562798641 >> v2:172.21.15.125:3300/0 conn(0x7f843814fba0 0x7f8438150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 --2- 172.21.15.50:0/562798641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8438067c90 0x7f8438068060 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 -- 172.21.15.50:0/562798641 >> 172.21.15.50:0/562798641 conn(0x7f84380cd6d0 msgr2=0x7f843806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 -- 172.21.15.50:0/562798641 shutdown_connections 2023-12-19T15:55:10.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.868+0000 7f843cf45700 1 -- 172.21.15.50:0/562798641 wait complete. 2023-12-19T15:55:10.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 Processor -- start 2023-12-19T15:55:10.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 -- start start 2023-12-19T15:55:10.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8438067c90 0x7f843814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 0x7f84381521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84380b3f80 con 0x7f84380c5900 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84380b4b00 con 0x7f8438067c90 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843cf45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843814e800 con 0x7f843814fba0 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43576/0 (socket says 172.21.15.50:43576) 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 -- 172.21.15.50:0/1949224992 learned_addr learned my addr 172.21.15.50:0/1949224992 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f843659c700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8438067c90 0x7f843814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8436d9d700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 0x7f84381521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 -- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 msgr2=0x7f84381521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 0x7f84381521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 -- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8438067c90 msgr2=0x7f843814e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8438067c90 0x7f843814e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 -- 172.21.15.50:0/1949224992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280094f0 con 0x7f84380c5900 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8436d9d700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 0x7f84381521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.872+0000 7f8435d9b700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f842c00d8c0 tx=0x7f842c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0125f0 con 0x7f84380c5900 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c012750 con 0x7f84380c5900 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c012af0 con 0x7f84380c5900 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84380b41d0 con 0x7f84380c5900 2023-12-19T15:55:10.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84380b4790 con 0x7f84380c5900 2023-12-19T15:55:10.879 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8438071120 con 0x7f84380c5900 2023-12-19T15:55:10.880 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.876+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f842c01e460 con 0x7f84380c5900 2023-12-19T15:55:10.881 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.880+0000 7f84267fc700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 0x7f8418062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.880+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7f842c07cff0 con 0x7f84380c5900 2023-12-19T15:55:10.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.880+0000 7f843659c700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 0x7f8418062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:10.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.880+0000 7f843659c700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 0x7f8418062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f84380c56b0 tx=0x7f8428017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:10.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:10.888+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f842c04d390 con 0x7f84380c5900 2023-12-19T15:55:11.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.124+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f84380642a0 con 0x7f84380c5900 2023-12-19T15:55:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.868+0000 7f84267fc700 1 -- 172.21.15.50:0/1949224992 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2479) v1 ==== 168+0+0 (secure 0 0 0) 0x7f842c0509b0 con 0x7f84380c5900 2023-12-19T15:55:11.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 msgr2=0x7f8418062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 0x7f8418062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f84380c56b0 tx=0x7f8428017000 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 msgr2=0x7f84380c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f842c00d8c0 tx=0x7f842c008040 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 shutdown_connections 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f84180606a0 0x7f8418062b50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f843814fba0 0x7f84381521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84380c5900 0x7f84380c86a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 --2- 172.21.15.50:0/1949224992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8438067c90 0x7f843814e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 >> 172.21.15.50:0/1949224992 conn(0x7f84380cd6d0 msgr2=0x7f843806a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 shutdown_connections 2023-12-19T15:55:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:11.872+0000 7f843cf45700 1 -- 172.21.15.50:0/1949224992 wait complete. 2023-12-19T15:55:11.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T15:55:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.076+0000 7fc47a0cd700 1 Processor -- start 2023-12-19T15:55:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.076+0000 7fc47a0cd700 1 -- start start 2023-12-19T15:55:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc47a0cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc47406b8c0 0x7fc4740ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc47a0cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740c7450 con 0x7fc4740cab10 2023-12-19T15:55:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc47a0cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47406bea0 con 0x7fc47415d410 2023-12-19T15:55:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc47a0cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4740c6e00 con 0x7fc47406b8c0 2023-12-19T15:55:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc473fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc47406b8c0 0x7fc4740ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc472ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc4740cab10 0x7fc47406b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60352/0 (socket says 172.21.15.50:60352) 2023-12-19T15:55:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc472ffd700 1 -- 172.21.15.50:0/1852219564 learned_addr learned my addr 172.21.15.50:0/1852219564 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc473fff700 1 -- 172.21.15.50:0/1852219564 >> v1:172.21.15.125:6789/0 conn(0x7fc47415d410 legacy=0x7fc4740ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc473fff700 1 -- 172.21.15.50:0/1852219564 >> v1:172.21.15.50:6789/0 conn(0x7fc4740cab10 legacy=0x7fc47406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc473fff700 1 -- 172.21.15.50:0/1852219564 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740c70e0 con 0x7fc47406b8c0 2023-12-19T15:55:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc473fff700 1 --2- 172.21.15.50:0/1852219564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc4740ab6c0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009e60 tx=0x7fc46c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e71505ea91a105 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.080+0000 7fc4727fc700 1 -- 172.21.15.50:0/1852219564 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00e070 con 0x7fc47406b8c0 2023-12-19T15:55:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc4727fc700 1 -- 172.21.15.50:0/1852219564 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c00bc30 con 0x7fc47406b8c0 2023-12-19T15:55:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc4727fc700 1 -- 172.21.15.50:0/1852219564 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c012430 con 0x7fc47406b8c0 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 -- 172.21.15.50:0/1852219564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 msgr2=0x7fc4740ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/1852219564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc4740ab6c0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009e60 tx=0x7fc46c009780 comp rx=0 tx=0).stop 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 -- 172.21.15.50:0/1852219564 shutdown_connections 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/1852219564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc4740ab6c0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 -- 172.21.15.50:0/1852219564 >> 172.21.15.50:0/1852219564 conn(0x7fc474067bb0 msgr2=0x7fc4740cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 -- 172.21.15.50:0/1852219564 shutdown_connections 2023-12-19T15:55:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 -- 172.21.15.50:0/1852219564 wait complete. 2023-12-19T15:55:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.084+0000 7fc47a0cd700 1 Processor -- start 2023-12-19T15:55:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 -- start start 2023-12-19T15:55:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 0x7fc47415ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc47415d410 0x7fc4740c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740c5a00 con 0x7fc47406b8c0 2023-12-19T15:55:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740c3570 con 0x7fc47415d410 2023-12-19T15:55:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc474159bd0 con 0x7fc4740cab10 2023-12-19T15:55:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43606/0 (socket says 172.21.15.50:43606) 2023-12-19T15:55:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 -- 172.21.15.50:0/2502565781 learned_addr learned my addr 172.21.15.50:0/2502565781 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc472ffd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 0x7fc47415ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc473fff700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc47415d410 0x7fc4740c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 -- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 msgr2=0x7fc47415ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 0x7fc47415ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 -- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc47415d410 msgr2=0x7fc4740c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc47415d410 0x7fc4740c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 -- 172.21.15.50:0/2502565781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c009a20 con 0x7fc47406b8c0 2023-12-19T15:55:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc472ffd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 0x7fc47415ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc4737fe700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fc46400c5c0 tx=0x7fc464007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464011590 con 0x7fc47406b8c0 2023-12-19T15:55:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc464011730 con 0x7fc47406b8c0 2023-12-19T15:55:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464011af0 con 0x7fc47406b8c0 2023-12-19T15:55:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.088+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4740c5720 con 0x7fc47406b8c0 2023-12-19T15:55:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.092+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4740c3200 con 0x7fc47406b8c0 2023-12-19T15:55:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.092+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4740c2d60 con 0x7fc47406b8c0 2023-12-19T15:55:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.092+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc464012420 con 0x7fc47406b8c0 2023-12-19T15:55:12.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.096+0000 7fc45b7fe700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 0x7fc460062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.096+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7fc46407be20 con 0x7fc47406b8c0 2023-12-19T15:55:12.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.096+0000 7fc472ffd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 0x7fc460062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.096+0000 7fc472ffd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 0x7fc460062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4740c5cc0 tx=0x7fc468009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.100+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc464049be0 con 0x7fc47406b8c0 2023-12-19T15:55:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.276+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc474071120 con 0x7fc47406b8c0 2023-12-19T15:55:12.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.280+0000 7fc45b7fe700 1 -- 172.21.15.50:0/2502565781 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc464017020 con 0x7fc47406b8c0 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 msgr2=0x7fc460062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 0x7fc460062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4740c5cc0 tx=0x7fc468009040 comp rx=0 tx=0).stop 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 msgr2=0x7fc474159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fc46400c5c0 tx=0x7fc464007f30 comp rx=0 tx=0).stop 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 shutdown_connections 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc47415d410 0x7fc4740c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc4600606a0 0x7fc460062b50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc4740cab10 0x7fc47415ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 --2- 172.21.15.50:0/2502565781 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc47406b8c0 0x7fc474159690 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 >> 172.21.15.50:0/2502565781 conn(0x7fc474067bb0 msgr2=0x7fc47415b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 shutdown_connections 2023-12-19T15:55:12.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.288+0000 7fc47a0cd700 1 -- 172.21.15.50:0/2502565781 wait complete. 2023-12-19T15:55:12.310 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T15:55:12.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:55:12.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T15:55:12.400 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T15:55:12.400 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T15:55:12.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:12.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.563+0000 7fe9e542e700 1 Processor -- start 2023-12-19T15:55:12.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.563+0000 7fe9e542e700 1 -- start start 2023-12-19T15:55:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9e542e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9e015d320 0x7fe9e015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9e542e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e00aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9e542e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00c7190 con 0x7fe9e006d690 2023-12-19T15:55:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9e542e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00c1b50 con 0x7fe9e015d320 2023-12-19T15:55:12.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9e542e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00c1e30 con 0x7fe9e00c15b0 2023-12-19T15:55:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9e015d320 0x7fe9e015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9e015d320 0x7fe9e015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:59364/0 (socket says 172.21.15.50:59364) 2023-12-19T15:55:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 -- 172.21.15.50:0/2795724526 learned_addr learned my addr 172.21.15.50:0/2795724526 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:12.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9d67fc700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e00aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 -- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 msgr2=0x7fe9e00aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e00aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 -- 172.21.15.50:0/2795724526 >> v1:172.21.15.125:6789/0 conn(0x7fe9e006d690 legacy=0x7fe9e00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 -- 172.21.15.50:0/2795724526 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00c6e20 con 0x7fe9e015d320 2023-12-19T15:55:12.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9d67fc700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e00aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.567+0000 7fe9deffd700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e015d320 0x7fe9e015d6f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00099b0 tx=0x7fe9d0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2440791778cfc6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9de7fc700 1 -- 172.21.15.50:0/2795724526 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0009ea0 con 0x7fe9e015d320 2023-12-19T15:55:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9de7fc700 1 -- 172.21.15.50:0/2795724526 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000b860 con 0x7fe9e015d320 2023-12-19T15:55:12.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9de7fc700 1 -- 172.21.15.50:0/2795724526 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000bc00 con 0x7fe9e015d320 2023-12-19T15:55:12.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 -- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e015d320 msgr2=0x7fe9e015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e015d320 0x7fe9e015d6f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00099b0 tx=0x7fe9d0009250 comp rx=0 tx=0).stop 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 -- 172.21.15.50:0/2795724526 shutdown_connections 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e00aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 --2- 172.21.15.50:0/2795724526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e015d320 0x7fe9e015d6f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 -- 172.21.15.50:0/2795724526 >> 172.21.15.50:0/2795724526 conn(0x7fe9e00c8850 msgr2=0x7fe9e00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 -- 172.21.15.50:0/2795724526 shutdown_connections 2023-12-19T15:55:12.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.571+0000 7fe9e542e700 1 -- 172.21.15.50:0/2795724526 wait complete. 2023-12-19T15:55:12.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 Processor -- start 2023-12-19T15:55:12.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 -- start start 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e016e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e015d320 0x7fe9e016d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e016e910 con 0x7fe9e015d320 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e016fdb0 con 0x7fe9e006d690 2023-12-19T15:55:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9e542e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e016d770 con 0x7fe9e00c15b0 2023-12-19T15:55:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.575+0000 7fe9df7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e015d320 0x7fe9e016d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9d67fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e016e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:59380/0 (socket says 172.21.15.50:59380) 2023-12-19T15:55:12.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 -- 172.21.15.50:0/3609347341 learned_addr learned my addr 172.21.15.50:0/3609347341 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 -- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 msgr2=0x7fe9e016e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e016e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 -- 172.21.15.50:0/3609347341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e015d320 msgr2=0x7fe9e016d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e015d320 0x7fe9e016d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 -- 172.21.15.50:0/3609347341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8009420 con 0x7fe9e006d690 2023-12-19T15:55:12.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9d67fc700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e016e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9deffd700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0006fd0 tx=0x7fe9d0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000bbe0 con 0x7fe9e006d690 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0000c90 con 0x7fe9e006d690 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d00094f0 con 0x7fe9e006d690 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0011650 con 0x7fe9e006d690 2023-12-19T15:55:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.579+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e016f480 con 0x7fe9e006d690 2023-12-19T15:55:12.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.583+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0071120 con 0x7fe9e006d690 2023-12-19T15:55:12.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.583+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe9d0011a40 con 0x7fe9e006d690 2023-12-19T15:55:12.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.583+0000 7fe9d77fe700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 0x7fe9c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:12.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.583+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7fe9d007e2d0 con 0x7fe9e006d690 2023-12-19T15:55:12.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.583+0000 7fe9d67fc700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 0x7fe9c0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:12.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.587+0000 7fe9d67fc700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 0x7fe9c0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00caae0 tx=0x7fe9c80091a0 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:12.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.591+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe9d004e610 con 0x7fe9e006d690 2023-12-19T15:55:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.799+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e00642a0 con 0x7fe9e006d690 2023-12-19T15:55:12.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9d77fe700 1 -- 172.21.15.50:0/3609347341 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fe9d0015de0 con 0x7fe9e006d690 2023-12-19T15:55:12.806 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:12.806 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":77,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3354287796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3354287796},{"type":"v1","addr":"172.21.15.125:6837","nonce":3354287796}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7751,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/3131052148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":3131052148},{"type":"v1","addr":"172.21.15.50:6834","nonce":3131052148}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":7754,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1251618303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1251618303},{"type":"v1","addr":"172.21.15.50:6835","nonce":1251618303}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70},{"gid":9073,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3362467688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3362467688},{"type":"v1","addr":"172.21.15.125:6836","nonce":3362467688}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[]} 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 msgr2=0x7fe9c0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 0x7fe9c0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00caae0 tx=0x7fe9c80091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 msgr2=0x7fe9e016eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0006fd0 tx=0x7fe9d0017000 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 shutdown_connections 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fe9c0060670 0x7fe9c0062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe9e015d320 0x7fe9e016d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe9e00c15b0 0x7fe9e016e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 --2- 172.21.15.50:0/3609347341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9e006d690 0x7fe9e016eac0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 >> 172.21.15.50:0/3609347341 conn(0x7fe9e00c8850 msgr2=0x7fe9e015c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:12.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 shutdown_connections 2023-12-19T15:55:12.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:12.803+0000 7fe9e542e700 1 -- 172.21.15.50:0/3609347341 wait complete. 2023-12-19T15:55:12.811 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 77 2023-12-19T15:55:12.828 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T15:55:12.828 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T15:55:12.828 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T15:55:12.829 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:55:12.829 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T15:55:12.829 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T15:55:12.829 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T15:55:12.829 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:55:12.829 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T15:55:12.829 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T15:55:12.830 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T15:55:12.830 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:55:12.830 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T15:55:12.830 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T15:55:12.830 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T15:55:12.830 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T15:55:12.845 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T15:55:12.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T15:55:12.852 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T15:55:12.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T15:55:12.856 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T15:55:12.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T15:55:12.857 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T15:55:12.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T15:55:12.898 INFO:tasks.ceph.mds.a:Started 2023-12-19T15:55:12.899 DEBUG:teuthology.parallel:result is None 2023-12-19T15:55:12.902 INFO:tasks.ceph.mds.c:Started 2023-12-19T15:55:12.902 DEBUG:teuthology.parallel:result is None 2023-12-19T15:55:12.909 INFO:tasks.ceph.mds.b:Started 2023-12-19T15:55:12.910 DEBUG:teuthology.parallel:result is None 2023-12-19T15:55:12.913 INFO:tasks.ceph.mds.d:Started 2023-12-19T15:55:12.914 DEBUG:teuthology.parallel:result is None 2023-12-19T15:55:12.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T15:55:13.015 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T15:55:13.022 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T15:55:13.023 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T15:55:13.025 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T15:55:13.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.079+0000 7ff820145700 1 Processor -- start 2023-12-19T15:55:13.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.079+0000 7ff820145700 1 -- start start 2023-12-19T15:55:13.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff820145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff818159220 0x7ff81806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff820145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81806e160 con 0x7ff818158e50 2023-12-19T15:55:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff820145700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81806d0c0 con 0x7ff8180a98d0 2023-12-19T15:55:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff820145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff81806ced0 con 0x7ff818159220 2023-12-19T15:55:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81dee1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff818158e50 0x7ff81806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60362/0 (socket says 172.21.15.50:60362) 2023-12-19T15:55:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81dee1700 1 -- 172.21.15.50:0/539917703 learned_addr learned my addr 172.21.15.50:0/539917703 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81d6e0700 1 --2- 172.21.15.50:0/539917703 >> v2:172.21.15.125:3300/0 conn(0x7ff818159220 0x7ff81806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81cedf700 1 -- 172.21.15.50:0/539917703 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289975567 0 0) 0x7ff81806e160 con 0x7ff818158e50 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81cedf700 1 -- 172.21.15.50:0/539917703 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff804003570 con 0x7ff818158e50 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81d6e0700 1 -- 172.21.15.50:0/539917703 >> v1:172.21.15.125:6789/0 conn(0x7ff8180a98d0 legacy=0x7ff8180aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81d6e0700 1 -- 172.21.15.50:0/539917703 >> v1:172.21.15.50:6789/0 conn(0x7ff818158e50 legacy=0x7ff81806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:13.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81d6e0700 1 -- 172.21.15.50:0/539917703 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81806d3a0 con 0x7ff818159220 2023-12-19T15:55:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81d6e0700 1 --2- 172.21.15.50:0/539917703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff818159220 0x7ff81806a740 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff8080098e0 tx=0x7ff8080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aa05de52269acef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81cedf700 1 -- 172.21.15.50:0/539917703 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800b640 con 0x7ff818159220 2023-12-19T15:55:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81cedf700 1 -- 172.21.15.50:0/539917703 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80800b7a0 con 0x7ff818159220 2023-12-19T15:55:13.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.083+0000 7ff81cedf700 1 -- 172.21.15.50:0/539917703 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800bb40 con 0x7ff818159220 2023-12-19T15:55:13.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- 172.21.15.50:0/539917703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff818159220 msgr2=0x7ff81806a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:13.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 --2- 172.21.15.50:0/539917703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff818159220 0x7ff81806a740 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff8080098e0 tx=0x7ff8080091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:13.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- 172.21.15.50:0/539917703 shutdown_connections 2023-12-19T15:55:13.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 --2- 172.21.15.50:0/539917703 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff818159220 0x7ff81806a740 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:13.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- 172.21.15.50:0/539917703 >> 172.21.15.50:0/539917703 conn(0x7ff818067b60 msgr2=0x7ff8180ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- 172.21.15.50:0/539917703 shutdown_connections 2023-12-19T15:55:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- 172.21.15.50:0/539917703 wait complete. 2023-12-19T15:55:13.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 Processor -- start 2023-12-19T15:55:13.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.087+0000 7ff820145700 1 -- start start 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 0x7ff8180af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff818159220 0x7ff8181515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8180ab500 con 0x7ff818158e50 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8180ab7e0 con 0x7ff8180a98d0 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8180abac0 con 0x7ff818159220 2023-12-19T15:55:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43650/0 (socket says 172.21.15.50:43650) 2023-12-19T15:55:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 -- 172.21.15.50:0/2530835125 learned_addr learned my addr 172.21.15.50:0/2530835125 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81e6e2700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff818159220 0x7ff8181515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81dee1700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 0x7ff8180af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 -- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff818159220 msgr2=0x7ff8181515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff818159220 0x7ff8181515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 -- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 msgr2=0x7ff8180af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 0x7ff8180af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 -- 172.21.15.50:0/2530835125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff808009420 con 0x7ff818158e50 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81dee1700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 0x7ff8180af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff81d6e0700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff808013000 tx=0x7ff808017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800bb20 con 0x7ff818158e50 2023-12-19T15:55:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff808011420 con 0x7ff818158e50 2023-12-19T15:55:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8180abda0 con 0x7ff818158e50 2023-12-19T15:55:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8080117c0 con 0x7ff818158e50 2023-12-19T15:55:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.091+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8180af9b0 con 0x7ff818158e50 2023-12-19T15:55:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.095+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8180ac2a0 con 0x7ff818158e50 2023-12-19T15:55:13.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.095+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff808011b70 con 0x7ff818158e50 2023-12-19T15:55:13.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.099+0000 7ff80dffb700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 0x7ff7fc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:13.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.099+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5960+0+0 (secure 0 0 0) 0x7ff80807e0a0 con 0x7ff818158e50 2023-12-19T15:55:13.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.099+0000 7ff81dee1700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 0x7ff7fc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:13.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.099+0000 7ff81dee1700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 0x7ff7fc062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff814007fd0 tx=0x7ff814009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:13.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.103+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff80804e3e0 con 0x7ff818158e50 2023-12-19T15:55:13.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:13.295+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff818071120 con 0x7ff818158e50 2023-12-19T15:55:14.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.315+0000 7ff80dffb700 1 -- 172.21.15.50:0/2530835125 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v177) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff808051a00 con 0x7ff818158e50 2023-12-19T15:55:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 msgr2=0x7ff7fc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 0x7ff7fc062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff814007fd0 tx=0x7ff814009040 comp rx=0 tx=0).stop 2023-12-19T15:55:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 msgr2=0x7ff81815be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff808013000 tx=0x7ff808017000 comp rx=0 tx=0).stop 2023-12-19T15:55:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 shutdown_connections 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7ff7fc060670 0x7ff7fc062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff818159220 0x7ff8181515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff818158e50 0x7ff81815be00 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 --2- 172.21.15.50:0/2530835125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8180a98d0 0x7ff8180af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 >> 172.21.15.50:0/2530835125 conn(0x7ff818067b60 msgr2=0x7ff8181549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 shutdown_connections 2023-12-19T15:55:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.323+0000 7ff820145700 1 -- 172.21.15.50:0/2530835125 wait complete. 2023-12-19T15:55:14.327 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T15:55:14.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T15:55:14.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.515+0000 7f480ce0c700 1 Processor -- start 2023-12-19T15:55:14.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.515+0000 7f480ce0c700 1 -- start start 2023-12-19T15:55:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f480ce0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 0x7f4800038a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f480ce0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800039a80 con 0x7f480003b690 2023-12-19T15:55:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f480ce0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800038f90 con 0x7f48000e2e10 2023-12-19T15:55:14.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f480ce0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000e0260 con 0x7f48000e2a40 2023-12-19T15:55:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f4807fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f480003b690 0x7f4800054e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60374/0 (socket says 172.21.15.50:60374) 2023-12-19T15:55:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f4806ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f48000e2e10 0x7f480003af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:48992/0 (socket says 172.21.15.50:48992) 2023-12-19T15:55:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f4807fff700 1 -- 172.21.15.50:0/2608978082 learned_addr learned my addr 172.21.15.50:0/2608978082 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48077fe700 1 --2- 172.21.15.50:0/2608978082 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 0x7f4800038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294893987 0 0) 0x7f4800039a80 con 0x7f480003b690 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f8003570 con 0x7f480003b690 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679709210 0 0) 0x7f4800038f90 con 0x7f48000e2e10 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800039a80 con 0x7f48000e2e10 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48077fe700 1 -- 172.21.15.50:0/2608978082 >> v1:172.21.15.125:6789/0 conn(0x7f48000e2e10 legacy=0x7f480003af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48077fe700 1 -- 172.21.15.50:0/2608978082 >> v1:172.21.15.50:6789/0 conn(0x7f480003b690 legacy=0x7f4800054e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.519+0000 7f48077fe700 1 -- 172.21.15.50:0/2608978082 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000391e0 con 0x7f48000e2a40 2023-12-19T15:55:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f48077fe700 1 --2- 172.21.15.50:0/2608978082 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 0x7f4800038a50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009910 tx=0x7f47fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca647759ea20ee1c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc011630 con 0x7f48000e2a40 2023-12-19T15:55:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc011790 con 0x7f48000e2a40 2023-12-19T15:55:14.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f48067fc700 1 -- 172.21.15.50:0/2608978082 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc011b30 con 0x7f48000e2a40 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 -- 172.21.15.50:0/2608978082 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 msgr2=0x7f4800038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 --2- 172.21.15.50:0/2608978082 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 0x7f4800038a50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009910 tx=0x7f47fc0091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 -- 172.21.15.50:0/2608978082 shutdown_connections 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 --2- 172.21.15.50:0/2608978082 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f48000e2a40 0x7f4800038a50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 -- 172.21.15.50:0/2608978082 >> 172.21.15.50:0/2608978082 conn(0x7f4800059ca0 msgr2=0x7f4800055c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 -- 172.21.15.50:0/2608978082 shutdown_connections 2023-12-19T15:55:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.523+0000 7f480ce0c700 1 -- 172.21.15.50:0/2608978082 wait complete. 2023-12-19T15:55:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 Processor -- start 2023-12-19T15:55:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 -- start start 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f480003b690 0x7f48000501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48000e2a40 0x7f48000d86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000e3280 con 0x7f48000e2e10 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000e40e0 con 0x7f48000e2a40 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.527+0000 7f480ce0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4800057b60 con 0x7f480003b690 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43654/0 (socket says 172.21.15.50:43654) 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 -- 172.21.15.50:0/391112507 learned_addr learned my addr 172.21.15.50:0/391112507 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:14.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4806ffd700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48000e2a40 0x7f48000d86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f48077fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f480003b690 0x7f48000501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 -- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f480003b690 msgr2=0x7f48000501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f480003b690 0x7f48000501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 -- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48000e2a40 msgr2=0x7f48000d86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48000e2a40 0x7f48000d86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 -- 172.21.15.50:0/391112507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc009420 con 0x7f48000e2e10 2023-12-19T15:55:14.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f4807fff700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4808060f70 tx=0x7f4808066cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480806ff70 con 0x7f48000e2e10 2023-12-19T15:55:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480808a070 con 0x7f48000e2e10 2023-12-19T15:55:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48080641a0 con 0x7f48000e2e10 2023-12-19T15:55:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48000e34d0 con 0x7f48000e2e10 2023-12-19T15:55:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.531+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48000e3a90 con 0x7f48000e2e10 2023-12-19T15:55:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48000e4270 con 0x7f48000e2e10 2023-12-19T15:55:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f480806c9d0 con 0x7f48000e2e10 2023-12-19T15:55:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f47f37fe700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 0x7f47ec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f48080f2ec0 con 0x7f48000e2e10 2023-12-19T15:55:14.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f48077fe700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 0x7f47ec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:14.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.535+0000 7f48077fe700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 0x7f47ec062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0091e0 tx=0x7f47fc017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:14.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.543+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f48080c7010 con 0x7f48000e2e10 2023-12-19T15:55:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.779+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4800005500 con 0x7f48000e2e10 2023-12-19T15:55:14.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.779+0000 7f47f37fe700 1 -- 172.21.15.50:0/391112507 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4808095020 con 0x7f48000e2e10 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:55:14.785 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T15:55:14.787 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.790 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.792 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.793 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.794 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.795 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T15:55:14.796 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.797 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.798 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T15:55:14.799 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:55:14.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.801 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T15:55:14.802 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.803 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T15:55:14.804 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T15:55:14.805 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 msgr2=0x7f47ec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 0x7f47ec062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0091e0 tx=0x7f47fc017000 comp rx=0 tx=0).stop 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 msgr2=0x7f48000de2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f4808060f70 tx=0x7f4808066cd0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 shutdown_connections 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48000e2a40 0x7f48000d86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f47ec060670 0x7f47ec062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48000e2e10 0x7f48000de2e0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 --2- 172.21.15.50:0/391112507 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f480003b690 0x7f48000501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 >> 172.21.15.50:0/391112507 conn(0x7f4800059ca0 msgr2=0x7f48000068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 shutdown_connections 2023-12-19T15:55:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.787+0000 7f480ce0c700 1 -- 172.21.15.50:0/391112507 wait complete. 2023-12-19T15:55:14.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:14.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.991+0000 7f5ebf4cd700 1 Processor -- start 2023-12-19T15:55:14.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.991+0000 7f5ebf4cd700 1 -- start start 2023-12-19T15:55:14.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebf4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 0x7f5eb815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:14.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebf4cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80c7190 con 0x7f5eb80c15b0 2023-12-19T15:55:14.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebf4cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80c1b50 con 0x7f5eb815d320 2023-12-19T15:55:14.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebf4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80c1e30 con 0x7f5eb8068800 2023-12-19T15:55:14.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebca68700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5eb80c15b0 0x7f5eb80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60384/0 (socket says 172.21.15.50:60384) 2023-12-19T15:55:14.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebca68700 1 -- 172.21.15.50:0/3033026286 learned_addr learned my addr 172.21.15.50:0/3033026286 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:14.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebda6a700 1 --2- 172.21.15.50:0/3033026286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 0x7f5eb815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613016427 0 0) 0x7f5eb80c7190 con 0x7f5eb80c15b0 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea4003570 con 0x7f5eb80c15b0 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172149406 0 0) 0x7f5eb80c1b50 con 0x7f5eb815d320 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb80c7190 con 0x7f5eb815d320 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebda6a700 1 -- 172.21.15.50:0/3033026286 >> v1:172.21.15.125:6789/0 conn(0x7f5eb815d320 legacy=0x7f5eb806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebda6a700 1 -- 172.21.15.50:0/3033026286 >> v1:172.21.15.50:6789/0 conn(0x7f5eb80c15b0 legacy=0x7f5eb80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.995+0000 7f5ebda6a700 1 -- 172.21.15.50:0/3033026286 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80c6e20 con 0x7f5eb8068800 2023-12-19T15:55:15.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebda6a700 1 --2- 172.21.15.50:0/3033026286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 0x7f5eb815d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009e90 tx=0x7f5eb0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12231c3cb599733e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000e070 con 0x7f5eb8068800 2023-12-19T15:55:15.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb000bb80 con 0x7f5eb8068800 2023-12-19T15:55:15.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5eaffff700 1 -- 172.21.15.50:0/3033026286 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0012430 con 0x7f5eb8068800 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3033026286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 msgr2=0x7f5eb815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3033026286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 0x7f5eb815d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009e90 tx=0x7f5eb0009780 comp rx=0 tx=0).stop 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3033026286 shutdown_connections 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3033026286 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb8068800 0x7f5eb815d6f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3033026286 >> 172.21.15.50:0/3033026286 conn(0x7f5eb80c8850 msgr2=0x7f5eb80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3033026286 shutdown_connections 2023-12-19T15:55:15.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:14.999+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3033026286 wait complete. 2023-12-19T15:55:15.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 Processor -- start 2023-12-19T15:55:15.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 -- start start 2023-12-19T15:55:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb8068800 0x7f5eb814af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 0x7f5eb80c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 0x7f5eb816d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80c49f0 con 0x7f5eb815d320 2023-12-19T15:55:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80c3de0 con 0x7f5eb8068800 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebf4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80c40c0 con 0x7f5eb80c15b0 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebda6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 0x7f5eb816d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebda6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 0x7f5eb816d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43668/0 (socket says 172.21.15.50:43668) 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebda6a700 1 -- 172.21.15.50:0/3227628795 learned_addr learned my addr 172.21.15.50:0/3227628795 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebca68700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 0x7f5eb80c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.003+0000 7f5ebd269700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb8068800 0x7f5eb814af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 -- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb8068800 msgr2=0x7f5eb814af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb8068800 0x7f5eb814af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 -- 172.21.15.50:0/3227628795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 msgr2=0x7f5eb816d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 0x7f5eb816d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 -- 172.21.15.50:0/3227628795 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0009a20 con 0x7f5eb80c15b0 2023-12-19T15:55:15.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebca68700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 0x7f5eb80c5cb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c580 tx=0x7f5ea8007d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8005c00 con 0x7f5eb80c15b0 2023-12-19T15:55:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea8005d60 con 0x7f5eb80c15b0 2023-12-19T15:55:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80c43a0 con 0x7f5eb80c15b0 2023-12-19T15:55:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8007320 con 0x7f5eb80c15b0 2023-12-19T15:55:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb814b470 con 0x7f5eb80c15b0 2023-12-19T15:55:15.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.007+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb80c4ba0 con 0x7f5eb80c15b0 2023-12-19T15:55:15.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.011+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5ea801b3b0 con 0x7f5eb80c15b0 2023-12-19T15:55:15.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.011+0000 7f5ead7fa700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 0x7f5e9c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.011+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f5ea807bc90 con 0x7f5eb80c15b0 2023-12-19T15:55:15.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.011+0000 7f5ebd269700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 0x7f5e9c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.015+0000 7f5ebd269700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 0x7f5e9c062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4007fd0 tx=0x7f5eb4007d50 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.019+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5eb80c4ba0 con 0x7f5eb80c15b0 2023-12-19T15:55:15.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.191+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb80c4200 con 0x7f5eb80c15b0 2023-12-19T15:55:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.195+0000 7f5ead7fa700 1 -- 172.21.15.50:0/3227628795 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+12437 (secure 0 0 0) 0x7f5eb80c4200 con 0x7f5eb80c15b0 2023-12-19T15:55:15.198 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:15.199 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":177,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:14.309717+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.195+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 msgr2=0x7f5e9c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.195+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 0x7f5e9c062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4007fd0 tx=0x7f5eb4007d50 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.195+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 msgr2=0x7f5eb80c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.195+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 0x7f5eb80c5cb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c580 tx=0x7f5ea8007d70 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 shutdown_connections 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5e9c060670 0x7f5e9c062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5eb815d320 0x7f5eb816d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5eb80c15b0 0x7f5eb80c5cb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 --2- 172.21.15.50:0/3227628795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb8068800 0x7f5eb814af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 >> 172.21.15.50:0/3227628795 conn(0x7f5eb80c8850 msgr2=0x7f5eb80ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 shutdown_connections 2023-12-19T15:55:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.199+0000 7f5ebf4cd700 1 -- 172.21.15.50:0/3227628795 wait complete. 2023-12-19T15:55:15.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T15:55:15.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T15:55:15.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.383+0000 7fac94865700 1 Processor -- start 2023-12-19T15:55:15.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.383+0000 7fac94865700 1 -- start start 2023-12-19T15:55:15.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac94865700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac9015d410 0x7fac900ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac94865700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 0x7fac9006b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac94865700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900c7450 con 0x7fac900ca400 2023-12-19T15:55:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac94865700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9006bea0 con 0x7fac9015d410 2023-12-19T15:55:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac94865700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac900c6e00 con 0x7fac9006b1b0 2023-12-19T15:55:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac9015d410 0x7fac900ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fac9015d410 0x7fac900ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43674/0 (socket says 172.21.15.50:43674) 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 -- 172.21.15.50:0/3011619253 learned_addr learned my addr 172.21.15.50:0/3011619253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8dd9b700 1 --1- 172.21.15.50:0/3011619253 >> v1:172.21.15.125:6789/0 conn(0x7fac900ca400 0x7fac900ca810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:49016/0 (socket says 172.21.15.50:49016) 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8ed9d700 1 --2- 172.21.15.50:0/3011619253 >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 0x7fac9006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 -- 172.21.15.50:0/3011619253 >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 msgr2=0x7fac9006b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 --2- 172.21.15.50:0/3011619253 >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 0x7fac9006b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8d59a700 1 -- 172.21.15.50:0/3011619253 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504621259 0 0) 0x7fac900c7450 con 0x7fac900ca400 2023-12-19T15:55:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 -- 172.21.15.50:0/3011619253 >> v1:172.21.15.125:6789/0 conn(0x7fac900ca400 legacy=0x7fac900ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8e59c700 1 -- 172.21.15.50:0/3011619253 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900c70e0 con 0x7fac9015d410 2023-12-19T15:55:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.387+0000 7fac8ed9d700 1 --2- 172.21.15.50:0/3011619253 >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 0x7fac9006b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac8e59c700 1 --2- 172.21.15.50:0/3011619253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac9015d410 0x7fac900ab6c0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fac9006a5a0 tx=0x7fac80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ad388b16042b2f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac8d59a700 1 -- 172.21.15.50:0/3011619253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8000e070 con 0x7fac9015d410 2023-12-19T15:55:15.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac8d59a700 1 -- 172.21.15.50:0/3011619253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8000ba50 con 0x7fac9015d410 2023-12-19T15:55:15.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac8d59a700 1 -- 172.21.15.50:0/3011619253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800092e0 con 0x7fac9015d410 2023-12-19T15:55:15.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 -- 172.21.15.50:0/3011619253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac9015d410 msgr2=0x7fac900ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 --2- 172.21.15.50:0/3011619253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac9015d410 0x7fac900ab6c0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fac9006a5a0 tx=0x7fac80009840 comp rx=0 tx=0).stop 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 -- 172.21.15.50:0/3011619253 shutdown_connections 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 --2- 172.21.15.50:0/3011619253 >> v2:172.21.15.125:3300/0 conn(0x7fac9006b1b0 0x7fac9006b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 --2- 172.21.15.50:0/3011619253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac9015d410 0x7fac900ab6c0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 -- 172.21.15.50:0/3011619253 >> 172.21.15.50:0/3011619253 conn(0x7fac90067bb0 msgr2=0x7fac900cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 -- 172.21.15.50:0/3011619253 shutdown_connections 2023-12-19T15:55:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.391+0000 7fac94865700 1 -- 172.21.15.50:0/3011619253 wait complete. 2023-12-19T15:55:15.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 Processor -- start 2023-12-19T15:55:15.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 -- start start 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac9006b1b0 0x7fac900b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac9015d410 0x7fac900aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac900c2d80 con 0x7fac900ca400 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac900af380 con 0x7fac9006b1b0 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac94865700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac900b0390 con 0x7fac9015d410 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43682/0 (socket says 172.21.15.50:43682) 2023-12-19T15:55:15.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 -- 172.21.15.50:0/3704440053 learned_addr learned my addr 172.21.15.50:0/3704440053 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8e59c700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac9006b1b0 0x7fac900b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8ed9d700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac9015d410 0x7fac900aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 -- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac9015d410 msgr2=0x7fac900aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac9015d410 0x7fac900aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 -- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac9006b1b0 msgr2=0x7fac900b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac9006b1b0 0x7fac900b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 -- 172.21.15.50:0/3704440053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac80009ae0 con 0x7fac900ca400 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.395+0000 7fac8dd9b700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fac840011e0 tx=0x7fac84007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84007300 con 0x7fac900ca400 2023-12-19T15:55:15.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84007460 con 0x7fac900ca400 2023-12-19T15:55:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84004830 con 0x7fac900ca400 2023-12-19T15:55:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac900b0060 con 0x7fac900ca400 2023-12-19T15:55:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac900c3270 con 0x7fac900ca400 2023-12-19T15:55:15.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac900c3490 con 0x7fac900ca400 2023-12-19T15:55:15.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.399+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fac84004be0 con 0x7fac900ca400 2023-12-19T15:55:15.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.403+0000 7fac7e7fc700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 0x7fac70062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.403+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5229+0+0 (secure 0 0 0) 0x7fac8407bd60 con 0x7fac900ca400 2023-12-19T15:55:15.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.403+0000 7fac8e59c700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 0x7fac70062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.403+0000 7fac8e59c700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 0x7fac70062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac80009ab0 tx=0x7fac80018000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.407+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac8404c380 con 0x7fac900ca400 2023-12-19T15:55:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.579+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac90071120 con 0x7fac900ca400 2023-12-19T15:55:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.579+0000 7fac7e7fc700 1 -- 172.21.15.50:0/3704440053 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac84018020 con 0x7fac900ca400 2023-12-19T15:55:15.583 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 msgr2=0x7fac70062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 0x7fac70062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac80009ab0 tx=0x7fac80018000 comp rx=0 tx=0).stop 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 msgr2=0x7fac900b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fac840011e0 tx=0x7fac84007e10 comp rx=0 tx=0).stop 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 shutdown_connections 2023-12-19T15:55:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fac70060670 0x7fac70062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac9015d410 0x7fac900aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac900ca400 0x7fac900b0820 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 --2- 172.21.15.50:0/3704440053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac9006b1b0 0x7fac900b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 >> 172.21.15.50:0/3704440053 conn(0x7fac90067bb0 msgr2=0x7fac900c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 shutdown_connections 2023-12-19T15:55:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.583+0000 7fac94865700 1 -- 172.21.15.50:0/3704440053 wait complete. 2023-12-19T15:55:15.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T15:55:15.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.771+0000 7f424880c700 1 Processor -- start 2023-12-19T15:55:15.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 -- start start 2023-12-19T15:55:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42441591f0 0x7f424406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42440ce810 0x7f42440ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424406b730 con 0x7f4244158e20 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424406ab20 con 0x7f42440ce810 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.775+0000 7f424880c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424406ae00 con 0x7f42441591f0 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f42437fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4244158e20 0x7f42440ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60396/0 (socket says 172.21.15.50:60396) 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f42437fe700 1 -- 172.21.15.50:0/1805633373 learned_addr learned my addr 172.21.15.50:0/1805633373 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:15.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 --2- 172.21.15.50:0/1805633373 >> v2:172.21.15.50:3300/0 conn(0x7f42440ce810 0x7f42440ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4242ffd700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42441591f0 0x7f424406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 -- 172.21.15.50:0/1805633373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42441591f0 msgr2=0x7f424406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:15.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42441591f0 0x7f424406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 -- 172.21.15.50:0/1805633373 >> v1:172.21.15.50:6789/0 conn(0x7f4244158e20 legacy=0x7f42440ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 -- 172.21.15.50:0/1805633373 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424406b0e0 con 0x7f42440ce810 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f4243fff700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f42440ca950 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f423c009e60 tx=0x7f423c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab17bdfa9b4effa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f42427fc700 1 -- 172.21.15.50:0/1805633373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c00e070 con 0x7f42440ce810 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f42427fc700 1 -- 172.21.15.50:0/1805633373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c00bb60 con 0x7f42440ce810 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f42427fc700 1 -- 172.21.15.50:0/1805633373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c012430 con 0x7f42440ce810 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 -- 172.21.15.50:0/1805633373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 msgr2=0x7f42440ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f42440ca950 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f423c009e60 tx=0x7f423c009780 comp rx=0 tx=0).stop 2023-12-19T15:55:15.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 -- 172.21.15.50:0/1805633373 shutdown_connections 2023-12-19T15:55:15.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f42440ca950 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 --2- 172.21.15.50:0/1805633373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42441591f0 0x7f424406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 -- 172.21.15.50:0/1805633373 >> 172.21.15.50:0/1805633373 conn(0x7f4244067bb0 msgr2=0x7f42440cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 -- 172.21.15.50:0/1805633373 shutdown_connections 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.779+0000 7f424880c700 1 -- 172.21.15.50:0/1805633373 wait complete. 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 Processor -- start 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 -- start start 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4244158e20 0x7f42441517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 0x7f42441572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4244155600 con 0x7f42440ce810 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4244151d20 con 0x7f42441591f0 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f424880c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4244154710 con 0x7f4244158e20 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f42437fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f42437fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43692/0 (socket says 172.21.15.50:43692) 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f42437fe700 1 -- 172.21.15.50:0/654882975 learned_addr learned my addr 172.21.15.50:0/654882975 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f4243fff700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 0x7f42441572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.783+0000 7f4242ffd700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4244158e20 0x7f42441517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 -- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4244158e20 msgr2=0x7f42441517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4244158e20 0x7f42441517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 -- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 msgr2=0x7f42441572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 0x7f42441572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 -- 172.21.15.50:0/654882975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c009a20 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f4243fff700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 0x7f42441572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f42437fe700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f423400c7c0 tx=0x7f4234007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234005b30 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4234005cb0 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4234004a80 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42441549f0 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4244154fb0 con 0x7f42440ce810 2023-12-19T15:55:15.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.787+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424404be40 con 0x7f42440ce810 2023-12-19T15:55:15.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.791+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f42340070f0 con 0x7f42440ce810 2023-12-19T15:55:15.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.791+0000 7f422b7fe700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 0x7f4230062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:15.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.791+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f423407bc80 con 0x7f42440ce810 2023-12-19T15:55:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.791+0000 7f4242ffd700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 0x7f4230062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:15.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.795+0000 7f4242ffd700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 0x7f4230062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4244158bc0 tx=0x7f4238008040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:15.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.799+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f423404c320 con 0x7f42440ce810 2023-12-19T15:55:15.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.975+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f424406b3c0 con 0x7f42440ce810 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.979+0000 7f422b7fe700 1 -- 172.21.15.50:0/654882975 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v177) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4234018020 con 0x7f42440ce810 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:15.981 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T15:55:15.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 msgr2=0x7f4230062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 0x7f4230062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4244158bc0 tx=0x7f4238008040 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 msgr2=0x7f4244155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f423400c7c0 tx=0x7f4234007f70 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 shutdown_connections 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42441591f0 0x7f42441572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f42300606a0 0x7f4230062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4244158e20 0x7f42441517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 --2- 172.21.15.50:0/654882975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42440ce810 0x7f4244155a90 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 >> 172.21.15.50:0/654882975 conn(0x7f4244067bb0 msgr2=0x7f4244152630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 shutdown_connections 2023-12-19T15:55:15.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:15.983+0000 7f424880c700 1 -- 172.21.15.50:0/654882975 wait complete. 2023-12-19T15:55:16.007 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T15:55:16.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T15:55:16.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.175+0000 7f21490cd700 1 Processor -- start 2023-12-19T15:55:16.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.175+0000 7f21490cd700 1 -- start start 2023-12-19T15:55:16.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f21490cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f214415c930 0x7f2144163e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f21490cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21440b0ac0 con 0x7f21440cb500 2023-12-19T15:55:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f21490cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214406b210 con 0x7f21440cb130 2023-12-19T15:55:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f21490cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21440ca810 con 0x7f214415c930 2023-12-19T15:55:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f21440cb500 0x7f214415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60402/0 (socket says 172.21.15.50:60402) 2023-12-19T15:55:16.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214259c700 1 -- 172.21.15.50:0/3386016089 learned_addr learned my addr 172.21.15.50:0/3386016089 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214359e700 1 --2- 172.21.15.50:0/3386016089 >> v2:172.21.15.125:3300/0 conn(0x7f214415c930 0x7f2144163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582697067 0 0) 0x7f21440b0ac0 con 0x7f21440cb500 2023-12-19T15:55:16.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003570 con 0x7f21440cb500 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455219022 0 0) 0x7f214406b210 con 0x7f21440cb130 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21440b0ac0 con 0x7f21440cb130 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214359e700 1 -- 172.21.15.50:0/3386016089 >> v1:172.21.15.125:6789/0 conn(0x7f21440cb130 legacy=0x7f214406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214359e700 1 -- 172.21.15.50:0/3386016089 >> v1:172.21.15.50:6789/0 conn(0x7f21440cb500 legacy=0x7f214415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.179+0000 7f214359e700 1 -- 172.21.15.50:0/3386016089 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21440b0190 con 0x7f214415c930 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f214359e700 1 --2- 172.21.15.50:0/3386016089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f2144163e10 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e60 tx=0x7f213c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=630ddcba39f6a99d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00e070 con 0x7f214415c930 2023-12-19T15:55:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c00bab0 con 0x7f214415c930 2023-12-19T15:55:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f2141d9b700 1 -- 172.21.15.50:0/3386016089 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00be70 con 0x7f214415c930 2023-12-19T15:55:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- 172.21.15.50:0/3386016089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 msgr2=0x7f2144163e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 --2- 172.21.15.50:0/3386016089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f2144163e10 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e60 tx=0x7f213c009780 comp rx=0 tx=0).stop 2023-12-19T15:55:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- 172.21.15.50:0/3386016089 shutdown_connections 2023-12-19T15:55:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 --2- 172.21.15.50:0/3386016089 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f2144163e10 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- 172.21.15.50:0/3386016089 >> 172.21.15.50:0/3386016089 conn(0x7f2144067bb0 msgr2=0x7f21440cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- 172.21.15.50:0/3386016089 shutdown_connections 2023-12-19T15:55:16.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- 172.21.15.50:0/3386016089 wait complete. 2023-12-19T15:55:16.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 Processor -- start 2023-12-19T15:55:16.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.183+0000 7f21490cd700 1 -- start start 2023-12-19T15:55:16.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 0x7f214414e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:16.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:16.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f21440d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:16.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214414e5f0 con 0x7f21440cb500 2023-12-19T15:55:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440b5030 con 0x7f214415c930 2023-12-19T15:55:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440b4140 con 0x7f21440cb130 2023-12-19T15:55:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43698/0 (socket says 172.21.15.50:43698) 2023-12-19T15:55:16.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 -- 172.21.15.50:0/2953220928 learned_addr learned my addr 172.21.15.50:0/2953220928 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214359e700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f21440d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f2142d9d700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 0x7f214414e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 -- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 msgr2=0x7f214414e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 0x7f214414e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 -- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 msgr2=0x7f21440d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f21440d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 -- 172.21.15.50:0/2953220928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c009a20 con 0x7f21440cb500 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f2142d9d700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 0x7f214414e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:16.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f214259c700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f213800c790 tx=0x7f2138007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:16.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138005b30 con 0x7f21440cb500 2023-12-19T15:55:16.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138005c90 con 0x7f21440cb500 2023-12-19T15:55:16.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440b4420 con 0x7f21440cb500 2023-12-19T15:55:16.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138007300 con 0x7f21440cb500 2023-12-19T15:55:16.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.187+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21440b49e0 con 0x7f21440cb500 2023-12-19T15:55:16.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.191+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2144071120 con 0x7f21440cb500 2023-12-19T15:55:16.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.191+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f213801b3b0 con 0x7f21440cb500 2023-12-19T15:55:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.195+0000 7f212affd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 0x7f2130062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:16.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.195+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5229+0+0 (secure 0 0 0) 0x7f213807bde0 con 0x7f21440cb500 2023-12-19T15:55:16.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.195+0000 7f2142d9d700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 0x7f2130062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:16.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.195+0000 7f2142d9d700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 0x7f2130062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2134003990 tx=0x7f2134009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:16.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.199+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f213804fe90 con 0x7f21440cb500 2023-12-19T15:55:16.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.371+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f21440cbe90 con 0x7f21440cb500 2023-12-19T15:55:16.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.915+0000 7f212affd700 1 -- 172.21.15.50:0/2953220928 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v178) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2138017020 con 0x7f21440cb500 2023-12-19T15:55:16.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 msgr2=0x7f2130062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 0x7f2130062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2134003990 tx=0x7f2134009040 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 msgr2=0x7f21440b54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f213800c790 tx=0x7f2138007ea0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 shutdown_connections 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f214415c930 0x7f21440d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f2130060670 0x7f2130062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21440cb500 0x7f21440b54c0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 --2- 172.21.15.50:0/2953220928 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f21440cb130 0x7f214414e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.919+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 >> 172.21.15.50:0/2953220928 conn(0x7f2144067bb0 msgr2=0x7f214415b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.923+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 shutdown_connections 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:16.923+0000 7f21490cd700 1 -- 172.21.15.50:0/2953220928 wait complete. 2023-12-19T15:55:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T15:55:16.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T15:55:17.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.123+0000 7fba01985700 1 Processor -- start 2023-12-19T15:55:17.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.123+0000 7fba01985700 1 -- start start 2023-12-19T15:55:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fba01985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fba01985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fba01985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0cbdf0 con 0x7fb9fc158e20 2023-12-19T15:55:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fba01985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9fc0cc0d0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fba01985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0cc3b0 con 0x7fb9fc0cb970 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fb7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58850/0 (socket says 172.21.15.50:58850) 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 -- 172.21.15.50:0/295355265 learned_addr learned my addr 172.21.15.50:0/295355265 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fb7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:48130/0 (socket says 172.21.15.50:48130) 2023-12-19T15:55:17.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9faffd700 1 --1- 172.21.15.50:0/295355265 >> v1:172.21.15.125:6789/0 conn(0x7fb9fc158e20 0x7fb9fc0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:56536/0 (socket says 172.21.15.50:56536) 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 -- 172.21.15.50:0/295355265 >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 msgr2=0x7fb9fc06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 --2- 172.21.15.50:0/295355265 >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9fa7fc700 1 -- 172.21.15.50:0/295355265 >> v1:172.21.15.125:6789/0 conn(0x7fb9fc158e20 legacy=0x7fb9fc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.127+0000 7fb9f9ffb700 1 -- 172.21.15.50:0/295355265 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959077527 0 0) 0x7fb9fc0cbdf0 con 0x7fb9fc158e20 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9fa7fc700 1 -- 172.21.15.50:0/295355265 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0cc690 con 0x7fb9fc0cb970 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9fb7fe700 1 --2- 172.21.15.50:0/295355265 >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:17.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9fa7fc700 1 --2- 172.21.15.50:0/295355265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00099b0 tx=0x7fb9f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98cb944bd7be2a3e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9f9ffb700 1 -- 172.21.15.50:0/295355265 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0009ea0 con 0x7fb9fc0cb970 2023-12-19T15:55:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9f9ffb700 1 -- 172.21.15.50:0/295355265 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f000b860 con 0x7fb9fc0cb970 2023-12-19T15:55:17.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fb9f9ffb700 1 -- 172.21.15.50:0/295355265 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f000bc00 con 0x7fb9fc0cb970 2023-12-19T15:55:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 -- 172.21.15.50:0/295355265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 msgr2=0x7fb9fc06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 --2- 172.21.15.50:0/295355265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00099b0 tx=0x7fb9f0009250 comp rx=0 tx=0).stop 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 -- 172.21.15.50:0/295355265 shutdown_connections 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 --2- 172.21.15.50:0/295355265 >> v2:172.21.15.125:3300/0 conn(0x7fb9fc0cecd0 0x7fb9fc06b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 --2- 172.21.15.50:0/295355265 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc0cb970 0x7fb9fc06acc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 -- 172.21.15.50:0/295355265 >> 172.21.15.50:0/295355265 conn(0x7fb9fc067bb0 msgr2=0x7fb9fc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 -- 172.21.15.50:0/295355265 shutdown_connections 2023-12-19T15:55:17.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.131+0000 7fba01985700 1 -- 172.21.15.50:0/295355265 wait complete. 2023-12-19T15:55:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.135+0000 7fba01985700 1 Processor -- start 2023-12-19T15:55:17.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.135+0000 7fba01985700 1 -- start start 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc158e20 0x7fb9fc14b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9fc169b40 0x7fb9fc0cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc16d180 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc16a080 con 0x7fb9fc169b40 2023-12-19T15:55:17.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc16c850 con 0x7fb9fc158e20 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35748/0 (socket says 172.21.15.50:35748) 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 -- 172.21.15.50:0/3470915980 learned_addr learned my addr 172.21.15.50:0/3470915980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9fa7fc700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc158e20 0x7fb9fc14b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9fb7fe700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9fc169b40 0x7fb9fc0cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 -- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc158e20 msgr2=0x7fb9fc14b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc158e20 0x7fb9fc14b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 -- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9fc169b40 msgr2=0x7fb9fc0cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9fc169b40 0x7fb9fc0cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 -- 172.21.15.50:0/3470915980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4009a20 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9faffd700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00c5e0 tx=0x7fb9ec001040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec005a70 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec005bd0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f00094f0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.139+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0070d0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.143+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc16ce10 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.143+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc071120 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.147+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb9ec0074c0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.147+0000 7fb9e2ffd700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 0x7fb9e8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.147+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5599+0+0 (secure 0 0 0) 0x7fb9ec07c1a0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.147+0000 7fb9fa7fc700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 0x7fb9e8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:17.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.147+0000 7fb9fa7fc700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 0x7fb9e8062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00099b0 tx=0x7fb9f0017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:17.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.151+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9ec04c650 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.335+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb9fc06a7a0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.931+0000 7fb9e2ffd700 1 -- 172.21.15.50:0/3470915980 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v179) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb9ec0170d0 con 0x7fb9fc0cecd0 2023-12-19T15:55:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 msgr2=0x7fb9e8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 0x7fb9e8062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00099b0 tx=0x7fb9f0017000 comp rx=0 tx=0).stop 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 msgr2=0x7fb9fc153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00c5e0 tx=0x7fb9ec001040 comp rx=0 tx=0).stop 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 shutdown_connections 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9fc169b40 0x7fb9fc0cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb9e80606a0 0x7fb9e8062b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9fc158e20 0x7fb9fc14b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 --2- 172.21.15.50:0/3470915980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9fc0cecd0 0x7fb9fc153390 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:17.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.939+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 >> 172.21.15.50:0/3470915980 conn(0x7fb9fc067bb0 msgr2=0x7fb9fc14cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.943+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 shutdown_connections 2023-12-19T15:55:17.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:17.943+0000 7fba01985700 1 -- 172.21.15.50:0/3470915980 wait complete. 2023-12-19T15:55:17.953 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T15:55:17.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T15:55:18.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.143+0000 7fb8b0ef0700 1 Processor -- start 2023-12-19T15:55:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.143+0000 7fb8b0ef0700 1 -- start start 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8b0ef0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb8ac06cfa0 0x7fb8ac06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8b0ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac069a20 0x7fb8ac0b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8b0ef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0ab6c0 con 0x7fb8ac158ad0 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8b0ef0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0ac240 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8b0ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0ab910 con 0x7fb8ac069a20 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb8ac06cfa0 0x7fb8ac06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aad9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb8ac158ad0 0x7fb8ac158f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47328/0 (socket says 172.21.15.50:47328) 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aad9d700 1 -- 172.21.15.50:0/2536796027 learned_addr learned my addr 172.21.15.50:0/2536796027 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8a9d9b700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac069a20 0x7fb8ac0b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 -- 172.21.15.50:0/2536796027 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac069a20 msgr2=0x7fb8ac0b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac069a20 0x7fb8ac0b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 -- 172.21.15.50:0/2536796027 >> v1:172.21.15.50:6789/0 conn(0x7fb8ac158ad0 legacy=0x7fb8ac158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 -- 172.21.15.50:0/2536796027 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0abbf0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8a959a700 1 -- 172.21.15.50:0/2536796027 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397584333 0 0) 0x7fb8ac0ab6c0 con 0x7fb8ac158ad0 2023-12-19T15:55:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8aa59c700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac06d370 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0098e0 tx=0x7fb89c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41faead378b365ea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8a959a700 1 -- 172.21.15.50:0/2536796027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00b640 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8a959a700 1 -- 172.21.15.50:0/2536796027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c00b7a0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.147+0000 7fb8a959a700 1 -- 172.21.15.50:0/2536796027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00bb40 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/2536796027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 msgr2=0x7fb8ac06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac06d370 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0098e0 tx=0x7fb89c0091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/2536796027 shutdown_connections 2023-12-19T15:55:18.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac069a20 0x7fb8ac0b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/2536796027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac06d370 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/2536796027 >> 172.21.15.50:0/2536796027 conn(0x7fb8ac0cd870 msgr2=0x7fb8ac069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/2536796027 shutdown_connections 2023-12-19T15:55:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/2536796027 wait complete. 2023-12-19T15:55:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 Processor -- start 2023-12-19T15:55:18.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.151+0000 7fb8b0ef0700 1 -- start start 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8ac069a20 0x7fb8ac1518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac158ad0 0x7fb8ac0c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0c98d0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac152800 con 0x7fb8ac069a20 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8b0ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac152ae0 con 0x7fb8ac158ad0 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35762/0 (socket says 172.21.15.50:35762) 2023-12-19T15:55:18.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 -- 172.21.15.50:0/123693443 learned_addr learned my addr 172.21.15.50:0/123693443 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8aa59c700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8ac069a20 0x7fb8ac1518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8aad9d700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac158ad0 0x7fb8ac0c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 -- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac158ad0 msgr2=0x7fb8ac0c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac158ad0 0x7fb8ac0c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 -- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8ac069a20 msgr2=0x7fb8ac1518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8ac069a20 0x7fb8ac1518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 -- 172.21.15.50:0/123693443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c009420 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.155+0000 7fb8a9d9b700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000ec80 tx=0x7fb8a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a000c850 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a000c9b0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a000cd50 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac152dc0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac152480 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.159+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac0c9a80 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.163+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb8a0013680 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.163+0000 7fb89a7fc700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 0x7fb88c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:18.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.163+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5981+0+0 (secure 0 0 0) 0x7fb8a007d300 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.163+0000 7fb8aa59c700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 0x7fb88c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:18.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.167+0000 7fb8aa59c700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 0x7fb88c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0093f0 tx=0x7fb89c017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:18.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.167+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb8a004d6b0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.379+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb8ac071120 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.967+0000 7fb89a7fc700 1 -- 172.21.15.50:0/123693443 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v79) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb8a0050cd0 con 0x7fb8ac06cfa0 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 msgr2=0x7fb88c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 0x7fb88c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0093f0 tx=0x7fb89c017000 comp rx=0 tx=0).stop 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 msgr2=0x7fb8ac151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000ec80 tx=0x7fb8a00091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 shutdown_connections 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb88c060670 0x7fb88c062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb8ac158ad0 0x7fb8ac0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8ac06cfa0 0x7fb8ac151f40 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 --2- 172.21.15.50:0/123693443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8ac069a20 0x7fb8ac1518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 >> 172.21.15.50:0/123693443 conn(0x7fb8ac0cd870 msgr2=0x7fb8ac154660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 shutdown_connections 2023-12-19T15:55:18.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:18.983+0000 7fb8b0ef0700 1 -- 172.21.15.50:0/123693443 wait complete. 2023-12-19T15:55:18.990 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 21 and data pool 22 2023-12-19T15:55:19.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:19.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.171+0000 7f41da2ae700 1 Processor -- start 2023-12-19T15:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.171+0000 7f41da2ae700 1 -- start start 2023-12-19T15:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41da2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140ae0 0x7f41d40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41da2ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41d4068630 0x7f41d40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41da2ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d406e150 con 0x7f41d4140710 2023-12-19T15:55:19.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41da2ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d4068ab0 con 0x7f41d4068630 2023-12-19T15:55:19.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41da2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d406d540 con 0x7f41d4140ae0 2023-12-19T15:55:19.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f41d4140710 0x7f41d40c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47342/0 (socket says 172.21.15.50:47342) 2023-12-19T15:55:19.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d3fff700 1 -- 172.21.15.50:0/1220027523 learned_addr learned my addr 172.21.15.50:0/1220027523 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:19.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d884b700 1 --2- 172.21.15.50:0/1220027523 >> v2:172.21.15.125:3300/0 conn(0x7f41d4068630 0x7f41d40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d37fe700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140ae0 0x7f41d40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d2ffd700 1 -- 172.21.15.50:0/1220027523 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678522046 0 0) 0x7f41d406e150 con 0x7f41d4140710 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d2ffd700 1 -- 172.21.15.50:0/1220027523 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c0003570 con 0x7f41d4140710 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d884b700 1 -- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140ae0 msgr2=0x7f41d40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d884b700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140ae0 0x7f41d40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d884b700 1 -- 172.21.15.50:0/1220027523 >> v1:172.21.15.50:6789/0 conn(0x7f41d4140710 legacy=0x7f41d40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.175+0000 7f41d884b700 1 -- 172.21.15.50:0/1220027523 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d406d820 con 0x7f41d4068630 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41d884b700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40ac250 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f41c40098e0 tx=0x7f41c40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6accd8fc324162a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41d2ffd700 1 -- 172.21.15.50:0/1220027523 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011630 con 0x7f41d4068630 2023-12-19T15:55:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41d2ffd700 1 -- 172.21.15.50:0/1220027523 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c4011790 con 0x7f41d4068630 2023-12-19T15:55:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41d2ffd700 1 -- 172.21.15.50:0/1220027523 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011b30 con 0x7f41d4068630 2023-12-19T15:55:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 -- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 msgr2=0x7f41d40ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40ac250 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f41c40098e0 tx=0x7f41c40091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 -- 172.21.15.50:0/1220027523 shutdown_connections 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40ac250 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 --2- 172.21.15.50:0/1220027523 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140ae0 0x7f41d40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 -- 172.21.15.50:0/1220027523 >> 172.21.15.50:0/1220027523 conn(0x7f41d40c8850 msgr2=0x7f41d40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 -- 172.21.15.50:0/1220027523 shutdown_connections 2023-12-19T15:55:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.179+0000 7f41da2ae700 1 -- 172.21.15.50:0/1220027523 wait complete. 2023-12-19T15:55:19.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 Processor -- start 2023-12-19T15:55:19.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 -- start start 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140710 0x7f41d416d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40b0610 con 0x7f41d4140ae0 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40b1370 con 0x7f41d4068630 2023-12-19T15:55:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41da2ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40b0050 con 0x7f41d4140710 2023-12-19T15:55:19.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41d884b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41d3fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41d884b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35766/0 (socket says 172.21.15.50:35766) 2023-12-19T15:55:19.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41d884b700 1 -- 172.21.15.50:0/3863147689 learned_addr learned my addr 172.21.15.50:0/3863147689 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:19.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.183+0000 7f41d37fe700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140710 0x7f41d416d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 -- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140710 msgr2=0x7f41d416d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140710 0x7f41d416d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 -- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 msgr2=0x7f41d40af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 -- 172.21.15.50:0/3863147689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4009420 con 0x7f41d4140ae0 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d3fff700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41d884b700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f41d406a280 tx=0x7f41c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011b10 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c40253f0 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c40257b0 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d40b0330 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d416e840 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.187+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d416e3c0 con 0x7f41d4140ae0 2023-12-19T15:55:19.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.191+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f41c4025ba0 con 0x7f41d4140ae0 2023-12-19T15:55:19.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.191+0000 7f41cbfff700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 0x7f41b4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.191+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f41c407e380 con 0x7f41d4140ae0 2023-12-19T15:55:19.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.195+0000 7f41d3fff700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 0x7f41b4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.195+0000 7f41d3fff700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 0x7f41b4062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41cc005c50 tx=0x7f41cc009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.199+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f41d416e3c0 con 0x7f41d4140ae0 2023-12-19T15:55:19.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.371+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d416e3c0 con 0x7f41d4140ae0 2023-12-19T15:55:19.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.371+0000 7f41cbfff700 1 -- 172.21.15.50:0/3863147689 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+15670 (secure 0 0 0) 0x7f41d416e3c0 con 0x7f41d4140ae0 2023-12-19T15:55:19.376 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:19.376 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":180,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:18.382421+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 msgr2=0x7f41b4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 0x7f41b4062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f41cc005c50 tx=0x7f41cc009040 comp rx=0 tx=0).stop 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 msgr2=0x7f41d416d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f41d406a280 tx=0x7f41c4017000 comp rx=0 tx=0).stop 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 shutdown_connections 2023-12-19T15:55:19.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f41b4060670 0x7f41b4062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41d4140ae0 0x7f41d416d730 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f41d4140710 0x7f41d416d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 --2- 172.21.15.50:0/3863147689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41d4068630 0x7f41d40af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 >> 172.21.15.50:0/3863147689 conn(0x7f41d40c8850 msgr2=0x7f41d40ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 shutdown_connections 2023-12-19T15:55:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.375+0000 7f41da2ae700 1 -- 172.21.15.50:0/3863147689 wait complete. 2023-12-19T15:55:19.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:19.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.559+0000 7f172ace3700 1 Processor -- start 2023-12-19T15:55:19.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.559+0000 7f172ace3700 1 -- start start 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172ace3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 0x7f172415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172ace3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f172406b7a0 0x7f17240c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172ace3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17240c5610 con 0x7f17240c51b0 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172ace3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172406c7c0 con 0x7f172406b7a0 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172ace3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172406caa0 con 0x7f1724158d60 2023-12-19T15:55:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172a4e2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f17240c51b0 0x7f172406a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47346/0 (socket says 172.21.15.50:47346) 2023-12-19T15:55:19.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f172a4e2700 1 -- 172.21.15.50:0/3253578531 learned_addr learned my addr 172.21.15.50:0/3253578531 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:19.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f17294e0700 1 --2- 172.21.15.50:0/3253578531 >> v2:172.21.15.50:3300/0 conn(0x7f172406b7a0 0x7f17240c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 --2- 172.21.15.50:0/3253578531 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 0x7f172415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 -- 172.21.15.50:0/3253578531 >> v2:172.21.15.50:3300/0 conn(0x7f172406b7a0 msgr2=0x7f17240c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 --2- 172.21.15.50:0/3253578531 >> v2:172.21.15.50:3300/0 conn(0x7f172406b7a0 0x7f17240c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 -- 172.21.15.50:0/3253578531 >> v1:172.21.15.50:6789/0 conn(0x7f17240c51b0 legacy=0x7f172406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 -- 172.21.15.50:0/3253578531 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172415ce60 con 0x7f1724158d60 2023-12-19T15:55:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.563+0000 7f1729ce1700 1 --2- 172.21.15.50:0/3253578531 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 0x7f172415c920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17200098e0 tx=0x7f17200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e01c16e32d7a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f1728cdf700 1 -- 172.21.15.50:0/3253578531 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172000b640 con 0x7f1724158d60 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f1728cdf700 1 -- 172.21.15.50:0/3253578531 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172000b7a0 con 0x7f1724158d60 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f1728cdf700 1 -- 172.21.15.50:0/3253578531 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172000bb40 con 0x7f1724158d60 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- 172.21.15.50:0/3253578531 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 msgr2=0x7f172415c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 --2- 172.21.15.50:0/3253578531 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 0x7f172415c920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17200098e0 tx=0x7f17200091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- 172.21.15.50:0/3253578531 shutdown_connections 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 --2- 172.21.15.50:0/3253578531 >> v2:172.21.15.50:3300/0 conn(0x7f172406b7a0 0x7f17240c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 --2- 172.21.15.50:0/3253578531 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1724158d60 0x7f172415c920 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- 172.21.15.50:0/3253578531 >> 172.21.15.50:0/3253578531 conn(0x7f1724067bb0 msgr2=0x7f17240cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:19.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- 172.21.15.50:0/3253578531 shutdown_connections 2023-12-19T15:55:19.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- 172.21.15.50:0/3253578531 wait complete. 2023-12-19T15:55:19.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 Processor -- start 2023-12-19T15:55:19.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.567+0000 7f172ace3700 1 -- start start 2023-12-19T15:55:19.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17240c51b0 0x7f17241575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1724158d60 0x7f1724157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172416a4b0 con 0x7f1724158d60 2023-12-19T15:55:19.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17241560f0 con 0x7f17240c51b0 2023-12-19T15:55:19.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17241557c0 con 0x7f172406b7a0 2023-12-19T15:55:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172a4e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1724158d60 0x7f1724157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:58922/0 (socket says 172.21.15.50:58922) 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 -- 172.21.15.50:0/388000083 learned_addr learned my addr 172.21.15.50:0/388000083 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f17294e0700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17240c51b0 0x7f17241575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 -- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17240c51b0 msgr2=0x7f17241575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17240c51b0 0x7f17241575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 -- 172.21.15.50:0/388000083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1724158d60 msgr2=0x7f1724157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1724158d60 0x7f1724157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 -- 172.21.15.50:0/388000083 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720009420 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1729ce1700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f17200093f0 tx=0x7f1720017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172001b9f0 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172001bb50 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1724155aa0 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720025420 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.571+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1724152020 con 0x7f172406b7a0 2023-12-19T15:55:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.575+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172404be40 con 0x7f172406b7a0 2023-12-19T15:55:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.575+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1720023070 con 0x7f172406b7a0 2023-12-19T15:55:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.575+0000 7f1719ffb700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 0x7f1708062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.579+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f172007de00 con 0x7f172406b7a0 2023-12-19T15:55:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.579+0000 7f17294e0700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 0x7f1708062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.579+0000 7f17294e0700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 0x7f1708062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1724167030 tx=0x7f17140091a0 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.587+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f172004e0c0 con 0x7f172406b7a0 2023-12-19T15:55:19.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.763+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f172415d130 con 0x7f172406b7a0 2023-12-19T15:55:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.763+0000 7f1719ffb700 1 -- 172.21.15.50:0/388000083 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+15670 (secure 0 0 0) 0x7f17200516e0 con 0x7f172406b7a0 2023-12-19T15:55:19.769 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:19.770 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":180,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:18.382421+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 msgr2=0x7f1708062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 0x7f1708062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1724167030 tx=0x7f17140091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 msgr2=0x7f17240addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f17200093f0 tx=0x7f1720017000 comp rx=0 tx=0).stop 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 shutdown_connections 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17240c51b0 0x7f17241575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f1708060670 0x7f1708062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1724158d60 0x7f1724157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 --2- 172.21.15.50:0/388000083 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f172406b7a0 0x7f17240addc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 >> 172.21.15.50:0/388000083 conn(0x7f1724067bb0 msgr2=0x7f1724153d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:19.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 shutdown_connections 2023-12-19T15:55:19.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.767+0000 7f172ace3700 1 -- 172.21.15.50:0/388000083 wait complete. 2023-12-19T15:55:19.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T15:55:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.967+0000 7f1209385700 1 Processor -- start 2023-12-19T15:55:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.967+0000 7f1209385700 1 -- start start 2023-12-19T15:55:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1209385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 0x7f120406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1209385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12040cecd0 0x7f120406b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1209385700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040cbdf0 con 0x7f1204158e20 2023-12-19T15:55:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1209385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040cc0d0 con 0x7f12040cecd0 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1209385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040cc3b0 con 0x7f12040cb970 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1202ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1204158e20 0x7f12040ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:56548/0 (socket says 172.21.15.50:56548) 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1202ffd700 1 -- 172.21.15.50:0/1217751662 learned_addr learned my addr 172.21.15.50:0/1217751662 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 --2- 172.21.15.50:0/1217751662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 0x7f120406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12037fe700 1 --2- 172.21.15.50:0/1217751662 >> v2:172.21.15.125:3300/0 conn(0x7f12040cecd0 0x7f120406b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 -- 172.21.15.50:0/1217751662 >> v2:172.21.15.125:3300/0 conn(0x7f12040cecd0 msgr2=0x7f120406b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 --2- 172.21.15.50:0/1217751662 >> v2:172.21.15.125:3300/0 conn(0x7f12040cecd0 0x7f120406b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 -- 172.21.15.50:0/1217751662 >> v1:172.21.15.125:6789/0 conn(0x7f1204158e20 legacy=0x7f12040ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 -- 172.21.15.50:0/1217751662 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040cc690 con 0x7f12040cb970 2023-12-19T15:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f12027fc700 1 --2- 172.21.15.50:0/1217751662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 0x7f120406acc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11f80098e0 tx=0x7f11f80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606e7b17e8ee4673 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1201ffb700 1 -- 172.21.15.50:0/1217751662 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f800b640 con 0x7f12040cb970 2023-12-19T15:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1201ffb700 1 -- 172.21.15.50:0/1217751662 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f800b7a0 con 0x7f12040cb970 2023-12-19T15:55:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.971+0000 7f1201ffb700 1 -- 172.21.15.50:0/1217751662 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f800bb40 con 0x7f12040cb970 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- 172.21.15.50:0/1217751662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 msgr2=0x7f120406acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 --2- 172.21.15.50:0/1217751662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 0x7f120406acc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f11f80098e0 tx=0x7f11f80091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- 172.21.15.50:0/1217751662 shutdown_connections 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 --2- 172.21.15.50:0/1217751662 >> v2:172.21.15.125:3300/0 conn(0x7f12040cecd0 0x7f120406b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 --2- 172.21.15.50:0/1217751662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040cb970 0x7f120406acc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- 172.21.15.50:0/1217751662 >> 172.21.15.50:0/1217751662 conn(0x7f1204067bb0 msgr2=0x7f12040d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- 172.21.15.50:0/1217751662 shutdown_connections 2023-12-19T15:55:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- 172.21.15.50:0/1217751662 wait complete. 2023-12-19T15:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 Processor -- start 2023-12-19T15:55:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.975+0000 7f1209385700 1 -- start start 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 0x7f12040c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 0x7f120414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 0x7f12040c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b36b0 con 0x7f1204158e20 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b3990 con 0x7f12040cecd0 2023-12-19T15:55:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b3c70 con 0x7f12040c99f0 2023-12-19T15:55:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f12027fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 0x7f120414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 0x7f12040c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f12027fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 0x7f120414f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35782/0 (socket says 172.21.15.50:35782) 2023-12-19T15:55:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f12037fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 0x7f12040c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f12027fc700 1 -- 172.21.15.50:0/2975156647 learned_addr learned my addr 172.21.15.50:0/2975156647 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 -- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 msgr2=0x7f12040c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 0x7f12040c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 -- 172.21.15.50:0/2975156647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 msgr2=0x7f120414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 0x7f120414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 -- 172.21.15.50:0/2975156647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f8009420 con 0x7f12040cecd0 2023-12-19T15:55:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f12037fe700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 0x7f12040c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1202ffd700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 0x7f12040c78a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c4a0 tx=0x7f11f4007f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400ce50 con 0x7f12040cecd0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4005b30 con 0x7f12040cecd0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f40070d0 con 0x7f12040cecd0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12040b3f50 con 0x7f12040cecd0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120414fd60 con 0x7f12040cecd0 2023-12-19T15:55:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.983+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204071120 con 0x7f12040cecd0 2023-12-19T15:55:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.983+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f11f40074c0 con 0x7f12040cecd0 2023-12-19T15:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.983+0000 7f11eaffd700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 0x7f11f0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.983+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f11f407be10 con 0x7f12040cecd0 2023-12-19T15:55:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.983+0000 7f12027fc700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 0x7f11f0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.987+0000 7f12027fc700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 0x7f11f0062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11f8003080 tx=0x7f11f8017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:19.991+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f11f407f030 con 0x7f12040cecd0 2023-12-19T15:55:20.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.203+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f120406a7a0 con 0x7f12040cecd0 2023-12-19T15:55:20.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.975+0000 7f11eaffd700 1 -- 172.21.15.50:0/2975156647 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) v1 ==== 118+0+0 (secure 0 0 0) 0x7f11f404c0d0 con 0x7f12040cecd0 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 msgr2=0x7f11f0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 0x7f11f0062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11f8003080 tx=0x7f11f8017000 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 msgr2=0x7f12040c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 0x7f12040c78a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c4a0 tx=0x7f11f4007f10 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 shutdown_connections 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f11f0060670 0x7f11f0062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f12040c99f0 0x7f12040c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1204158e20 0x7f120414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 --2- 172.21.15.50:0/2975156647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12040cecd0 0x7f12040c78a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 >> 172.21.15.50:0/2975156647 conn(0x7f1204067bb0 msgr2=0x7f120414c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 shutdown_connections 2023-12-19T15:55:20.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:20.979+0000 7f1209385700 1 -- 172.21.15.50:0/2975156647 wait complete. 2023-12-19T15:55:21.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T15:55:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.183+0000 7fb1a9625700 1 Processor -- start 2023-12-19T15:55:21.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.183+0000 7fb1a9625700 1 -- start start 2023-12-19T15:55:21.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a9625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cb970 0x7fb1a406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a9625700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1a40cecd0 0x7fb1a406b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a9625700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40cbdf0 con 0x7fb1a4158e20 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a9625700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a40cc0d0 con 0x7fb1a40cecd0 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a9625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40cc3b0 con 0x7fb1a40cb970 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a2ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb1a4158e20 0x7fb1a40ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47362/0 (socket says 172.21.15.50:47362) 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a2ffd700 1 -- 172.21.15.50:0/40514792 learned_addr learned my addr 172.21.15.50:0/40514792 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a27fc700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cb970 0x7fb1a406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/40514792 >> v2:172.21.15.125:3300/0 conn(0x7fb1a40cecd0 0x7fb1a406b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a1ffb700 1 -- 172.21.15.50:0/40514792 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082003641 0 0) 0x7fb1a40cbdf0 con 0x7fb1a4158e20 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a1ffb700 1 -- 172.21.15.50:0/40514792 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c003570 con 0x7fb1a4158e20 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 -- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cb970 msgr2=0x7fb1a406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cb970 0x7fb1a406acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 -- 172.21.15.50:0/40514792 >> v1:172.21.15.50:6789/0 conn(0x7fb1a4158e20 legacy=0x7fb1a40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 -- 172.21.15.50:0/40514792 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40cc690 con 0x7fb1a40cecd0 2023-12-19T15:55:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a40cecd0 0x7fb1a406b200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009e60 tx=0x7fb19c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd185b9abfeded06 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a1ffb700 1 -- 172.21.15.50:0/40514792 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00e070 con 0x7fb1a40cecd0 2023-12-19T15:55:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a1ffb700 1 -- 172.21.15.50:0/40514792 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c00bab0 con 0x7fb1a40cecd0 2023-12-19T15:55:21.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.187+0000 7fb1a1ffb700 1 -- 172.21.15.50:0/40514792 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00be70 con 0x7fb1a40cecd0 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 -- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a40cecd0 msgr2=0x7fb1a406b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a40cecd0 0x7fb1a406b200 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009e60 tx=0x7fb19c009780 comp rx=0 tx=0).stop 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 -- 172.21.15.50:0/40514792 shutdown_connections 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a40cecd0 0x7fb1a406b200 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 --2- 172.21.15.50:0/40514792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cb970 0x7fb1a406acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 -- 172.21.15.50:0/40514792 >> 172.21.15.50:0/40514792 conn(0x7fb1a4067bb0 msgr2=0x7fb1a40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 -- 172.21.15.50:0/40514792 shutdown_connections 2023-12-19T15:55:21.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.191+0000 7fb1a9625700 1 -- 172.21.15.50:0/40514792 wait complete. 2023-12-19T15:55:21.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 Processor -- start 2023-12-19T15:55:21.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 -- start start 2023-12-19T15:55:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 0x7fb1a40b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a4158e20 0x7fb1a414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40b4e20 con 0x7fb1a40c9d50 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40b4210 con 0x7fb1a4158e20 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a9625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40b44f0 con 0x7fb1a40cecd0 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35786/0 (socket says 172.21.15.50:35786) 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 -- 172.21.15.50:0/4191059399 learned_addr learned my addr 172.21.15.50:0/4191059399 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a2ffd700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 0x7fb1a40b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a27fc700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a4158e20 0x7fb1a414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 -- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 msgr2=0x7fb1a40b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 0x7fb1a40b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 -- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a4158e20 msgr2=0x7fb1a414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a4158e20 0x7fb1a414f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a37fe700 1 -- 172.21.15.50:0/4191059399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c009a20 con 0x7fb1a40c9d50 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.195+0000 7fb1a2ffd700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 0x7fb1a40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb1a37fe700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fb19c013000 tx=0x7fb19c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c027470 con 0x7fb1a40c9d50 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c0275f0 con 0x7fb1a40c9d50 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a40b47d0 con 0x7fb1a40c9d50 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0279b0 con 0x7fb1a40c9d50 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a40b3eb0 con 0x7fb1a40c9d50 2023-12-19T15:55:21.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.199+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a41594b0 con 0x7fb1a40c9d50 2023-12-19T15:55:21.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.203+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb19c024070 con 0x7fb1a40c9d50 2023-12-19T15:55:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.203+0000 7fb18affd700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 0x7fb190062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.203+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fb19c07f390 con 0x7fb1a40c9d50 2023-12-19T15:55:21.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.203+0000 7fb1a2ffd700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 0x7fb190062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:21.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.203+0000 7fb1a2ffd700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 0x7fb190062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1940058d0 tx=0x7fb194009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.211+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb19c04f650 con 0x7fb1a40c9d50 2023-12-19T15:55:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.419+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb1a4071120 con 0x7fb1a40c9d50 2023-12-19T15:55:21.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.987+0000 7fb18affd700 1 -- 172.21.15.50:0/4191059399 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb19c052c70 con 0x7fb1a40c9d50 2023-12-19T15:55:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 msgr2=0x7fb190062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 0x7fb190062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1940058d0 tx=0x7fb194009040 comp rx=0 tx=0).stop 2023-12-19T15:55:21.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 msgr2=0x7fb1a415ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fb19c013000 tx=0x7fb19c019000 comp rx=0 tx=0).stop 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 shutdown_connections 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1a4158e20 0x7fb1a414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fb190060670 0x7fb190062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1a40c9d50 0x7fb1a415ecb0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 --2- 172.21.15.50:0/4191059399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb1a40cecd0 0x7fb1a40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 >> 172.21.15.50:0/4191059399 conn(0x7fb1a4067bb0 msgr2=0x7fb1a40cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 shutdown_connections 2023-12-19T15:55:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:21.991+0000 7fb1a9625700 1 -- 172.21.15.50:0/4191059399 wait complete. 2023-12-19T15:55:22.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.195+0000 7f308a963700 1 Processor -- start 2023-12-19T15:55:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.195+0000 7f308a963700 1 -- start start 2023-12-19T15:55:22.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a963700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f30840c6900 0x7f30840c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a963700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308415cc20 0x7f308415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a963700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308406b470 con 0x7f308406b900 2023-12-19T15:55:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a963700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308406e110 con 0x7f30840c6900 2023-12-19T15:55:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a963700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3084161d10 con 0x7f308415cc20 2023-12-19T15:55:22.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089160700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f308406b900 0x7f308406da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47364/0 (socket says 172.21.15.50:47364) 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089160700 1 -- 172.21.15.50:0/1469949062 learned_addr learned my addr 172.21.15.50:0/1469949062 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 --2- 172.21.15.50:0/1469949062 >> v2:172.21.15.50:3300/0 conn(0x7f30840c6900 0x7f30840c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308a162700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308415cc20 0x7f308415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308895f700 1 -- 172.21.15.50:0/1469949062 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364192005 0 0) 0x7f308406b470 con 0x7f308406b900 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f308895f700 1 -- 172.21.15.50:0/1469949062 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003570 con 0x7f308406b900 2023-12-19T15:55:22.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 -- 172.21.15.50:0/1469949062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308415cc20 msgr2=0x7f308415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308415cc20 0x7f308415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 -- 172.21.15.50:0/1469949062 >> v1:172.21.15.50:6789/0 conn(0x7f308406b900 legacy=0x7f308406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 -- 172.21.15.50:0/1469949062 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308406bd70 con 0x7f30840c6900 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.199+0000 7f3089961700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f30840c6cd0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f30800098e0 tx=0x7f30800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe51696fac80a161 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308895f700 1 -- 172.21.15.50:0/1469949062 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080011630 con 0x7f30840c6900 2023-12-19T15:55:22.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308895f700 1 -- 172.21.15.50:0/1469949062 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080011790 con 0x7f30840c6900 2023-12-19T15:55:22.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308895f700 1 -- 172.21.15.50:0/1469949062 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080011b30 con 0x7f30840c6900 2023-12-19T15:55:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 -- 172.21.15.50:0/1469949062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 msgr2=0x7f30840c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f30840c6cd0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f30800098e0 tx=0x7f30800091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 -- 172.21.15.50:0/1469949062 shutdown_connections 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308415cc20 0x7f308415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 --2- 172.21.15.50:0/1469949062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f30840c6cd0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 -- 172.21.15.50:0/1469949062 >> 172.21.15.50:0/1469949062 conn(0x7f3084067bb0 msgr2=0x7f30840cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 -- 172.21.15.50:0/1469949062 shutdown_connections 2023-12-19T15:55:22.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.203+0000 7f308a963700 1 -- 172.21.15.50:0/1469949062 wait complete. 2023-12-19T15:55:22.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 Processor -- start 2023-12-19T15:55:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 -- start start 2023-12-19T15:55:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308406b900 0x7f3084150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308415cc20 0x7f30840b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084159a80 con 0x7f30840c6900 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308415dd90 con 0x7f308415cc20 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30841591e0 con 0x7f308406b900 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35818/0 (socket says 172.21.15.50:35818) 2023-12-19T15:55:22.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 -- 172.21.15.50:0/2772698460 learned_addr learned my addr 172.21.15.50:0/2772698460 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089961700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308406b900 0x7f3084150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a162700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308415cc20 0x7f30840b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 -- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308406b900 msgr2=0x7f3084150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308406b900 0x7f3084150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 -- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308415cc20 msgr2=0x7f30840b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308415cc20 0x7f30840b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 -- 172.21.15.50:0/2772698460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3080009420 con 0x7f30840c6900 2023-12-19T15:55:22.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3089160700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f30740011e0 tx=0x7f3074007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074005b30 con 0x7f30840c6900 2023-12-19T15:55:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3074005c90 con 0x7f30840c6900 2023-12-19T15:55:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30841594c0 con 0x7f30840c6900 2023-12-19T15:55:22.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.207+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30740072e0 con 0x7f30840c6900 2023-12-19T15:55:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.211+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308415e030 con 0x7f30840c6900 2023-12-19T15:55:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.211+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308404be40 con 0x7f30840c6900 2023-12-19T15:55:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.211+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3074007650 con 0x7f30840c6900 2023-12-19T15:55:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.215+0000 7f3079ffb700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 0x7f3068062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.215+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f307407bed0 con 0x7f30840c6900 2023-12-19T15:55:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.215+0000 7f3089961700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 0x7f3068062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.215+0000 7f3089961700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 0x7f3068062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30800093f0 tx=0x7f3080017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.219+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3074049be0 con 0x7f30840c6900 2023-12-19T15:55:22.433 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.431+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f308406a580 con 0x7f30840c6900 2023-12-19T15:55:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.431+0000 7f3079ffb700 1 -- 172.21.15.50:0/2772698460 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f3074017090 con 0x7f30840c6900 2023-12-19T15:55:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:22.435 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 msgr2=0x7f3068062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 0x7f3068062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30800093f0 tx=0x7f3080017000 comp rx=0 tx=0).stop 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 msgr2=0x7f3084158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f30740011e0 tx=0x7f3074007df0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 shutdown_connections 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f308415cc20 0x7f30840b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3068060670 0x7f3068062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f30840c6900 0x7f3084158940 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 --2- 172.21.15.50:0/2772698460 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f308406b900 0x7f3084150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 >> 172.21.15.50:0/2772698460 conn(0x7f3084067bb0 msgr2=0x7f30840ada10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 shutdown_connections 2023-12-19T15:55:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.435+0000 7f308a963700 1 -- 172.21.15.50:0/2772698460 wait complete. 2023-12-19T15:55:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T15:55:22.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T15:55:22.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.623+0000 7f8f1728e700 1 Processor -- start 2023-12-19T15:55:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.623+0000 7f8f1728e700 1 -- start start 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.627+0000 7f8f1728e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f1006aa00 0x7f8f1006d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.627+0000 7f8f1728e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1006c7c0 con 0x7f8f1006a580 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.627+0000 7f8f1728e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1006cd80 con 0x7f8f100b1410 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.627+0000 7f8f1728e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1006b8f0 con 0x7f8f1006aa00 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f1006a580 0x7f8f1006d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47372/0 (socket says 172.21.15.50:47372) 2023-12-19T15:55:22.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 -- 172.21.15.50:0/1915874997 learned_addr learned my addr 172.21.15.50:0/1915874997 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f16a8d700 1 --2- 172.21.15.50:0/1915874997 >> v2:172.21.15.125:3300/0 conn(0x7f8f1006aa00 0x7f8f1006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038082229 0 0) 0x7f8f1006c7c0 con 0x7f8f1006a580 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc003570 con 0x7f8f1006a580 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674538495 0 0) 0x7f8f1006cd80 con 0x7f8f100b1410 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1006c7c0 con 0x7f8f100b1410 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 144541245 0 0) 0x7f8efc003570 con 0x7f8f1006a580 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 >> v2:172.21.15.125:3300/0 conn(0x7f8f1006aa00 msgr2=0x7f8f1006d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 --2- 172.21.15.50:0/1915874997 >> v2:172.21.15.125:3300/0 conn(0x7f8f1006aa00 0x7f8f1006d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 >> v1:172.21.15.125:6789/0 conn(0x7f8f100b1410 legacy=0x7f8f100ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100ac1e0 con 0x7f8f1006a580 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f8f00001470 con 0x7f8f1006a580 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 >> v1:172.21.15.50:6789/0 conn(0x7f8f1006a580 legacy=0x7f8f1006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc003c40 0x7f8efc004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 0x7f8efc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8efc005210 0x7f8efc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1006b8f0 con 0x7f8efc004810 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc00c130 con 0x7f8efc003c40 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0106d0 con 0x7f8efc005210 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f16a8d700 1 --2- 172.21.15.50:0/1915874997 >> v2:172.21.15.125:3300/0 conn(0x7f8f1006aa00 0x7f8f1006d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 0x7f8efc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1628c700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc003c40 0x7f8efc004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 -- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8efc005210 msgr2=0x7f8efc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8efc005210 0x7f8efc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 -- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc003c40 msgr2=0x7f8efc004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc003c40 0x7f8efc004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 -- 172.21.15.50:0/1915874997 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc003570 con 0x7f8efc004810 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f16a8d700 1 -- 172.21.15.50:0/1915874997 reap_dead start 2023-12-19T15:55:22.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1628c700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc003c40 0x7f8efc004030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 0x7f8efc004c80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c210 tx=0x7f8f1006ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa1e90bed11e1cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0000cad0 con 0x7f8efc004810 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0000cc30 con 0x7f8efc004810 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.631+0000 7f8f1528a700 1 -- 172.21.15.50:0/1915874997 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00005920 con 0x7f8efc004810 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 msgr2=0x7f8efc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 0x7f8efc004c80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c210 tx=0x7f8f1006ccf0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- 172.21.15.50:0/1915874997 shutdown_connections 2023-12-19T15:55:22.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 --2- 172.21.15.50:0/1915874997 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8efc004810 0x7f8efc004c80 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- 172.21.15.50:0/1915874997 >> 172.21.15.50:0/1915874997 conn(0x7f8f10067bb0 msgr2=0x7f8f100cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- 172.21.15.50:0/1915874997 shutdown_connections 2023-12-19T15:55:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- 172.21.15.50:0/1915874997 wait complete. 2023-12-19T15:55:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 Processor -- start 2023-12-19T15:55:22.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.635+0000 7f8f1728e700 1 -- start start 2023-12-19T15:55:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc004810 0x7f8f10157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f10158a10 0x7f8f100c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10151c90 con 0x7f8f10158060 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10158e80 con 0x7f8efc004810 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1014ca50 con 0x7f8f10158a10 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35838/0 (socket says 172.21.15.50:35838) 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 -- 172.21.15.50:0/2371586595 learned_addr learned my addr 172.21.15.50:0/2371586595 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f16a8d700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f10158a10 0x7f8f100c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1628c700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc004810 0x7f8f10157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 -- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f10158a10 msgr2=0x7f8f100c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f10158a10 0x7f8f100c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 -- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc004810 msgr2=0x7f8f10157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc004810 0x7f8f10157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 -- 172.21.15.50:0/2371586595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10151010 con 0x7f8f10158060 2023-12-19T15:55:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f15a8b700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0057b0 tx=0x7f8f00015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00000fa0 con 0x7f8f10158060 2023-12-19T15:55:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00001370 con 0x7f8f10158060 2023-12-19T15:55:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f101512f0 con 0x7f8f10158060 2023-12-19T15:55:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00004a80 con 0x7f8f10158060 2023-12-19T15:55:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.639+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101518b0 con 0x7f8f10158060 2023-12-19T15:55:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.643+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f10151dd0 con 0x7f8f10158060 2023-12-19T15:55:22.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.643+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8f00019070 con 0x7f8f10158060 2023-12-19T15:55:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.643+0000 7f8f067fc700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 0x7f8ef4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.643+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8f0007d730 con 0x7f8f10158060 2023-12-19T15:55:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.643+0000 7f8f1628c700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 0x7f8ef4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.647+0000 7f8f1628c700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 0x7f8ef4062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f1004ccf0 tx=0x7f8f0c009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:22.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.651+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f10151dd0 con 0x7f8f10158060 2023-12-19T15:55:22.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.835+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f10151dd0 con 0x7f8f10158060 2023-12-19T15:55:22.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.835+0000 7f8f067fc700 1 -- 172.21.15.50:0/2371586595 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+15657 (secure 0 0 0) 0x7f8f10151dd0 con 0x7f8f10158060 2023-12-19T15:55:22.841 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:22.841 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":181,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T15:55:19.954302+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 msgr2=0x7f8ef4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 0x7f8ef4062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f1004ccf0 tx=0x7f8f0c009040 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 msgr2=0x7f8f101584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0057b0 tx=0x7f8f00015000 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 shutdown_connections 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8ef4060670 0x7f8ef4062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f10158a10 0x7f8f100c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f10158060 0x7f8f101584d0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 --2- 172.21.15.50:0/2371586595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8efc004810 0x7f8f10157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:22.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 >> 172.21.15.50:0/2371586595 conn(0x7f8f10067bb0 msgr2=0x7f8f100adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:22.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 shutdown_connections 2023-12-19T15:55:22.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:22.843+0000 7f8f1728e700 1 -- 172.21.15.50:0/2371586595 wait complete. 2023-12-19T15:55:22.870 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T15:55:22.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T15:55:23.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.039+0000 7f4701eae700 1 Processor -- start 2023-12-19T15:55:23.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.039+0000 7f4701eae700 1 -- start start 2023-12-19T15:55:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f4701eae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46fc155b30 0x7f46fc0c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f4701eae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46fc0ca3c0 0x7f46fc0af470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f4701eae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc06c860 con 0x7f46fc155f00 2023-12-19T15:55:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f4701eae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc06cb00 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f4701eae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc06cde0 con 0x7f46fc155b30 2023-12-19T15:55:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46fc0ca3c0 0x7f46fc0af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f46fc0ca3c0 0x7f46fc0af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35852/0 (socket says 172.21.15.50:35852) 2023-12-19T15:55:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 -- 172.21.15.50:0/2095126088 learned_addr learned my addr 172.21.15.50:0/2095126088 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46fbfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f46fc155f00 0x7f46fc06d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47378/0 (socket says 172.21.15.50:47378) 2023-12-19T15:55:23.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46fb7fe700 1 --2- 172.21.15.50:0/2095126088 >> v2:172.21.15.125:3300/0 conn(0x7f46fc155b30 0x7f46fc0c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 -- 172.21.15.50:0/2095126088 >> v2:172.21.15.125:3300/0 conn(0x7f46fc155b30 msgr2=0x7f46fc0c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 --2- 172.21.15.50:0/2095126088 >> v2:172.21.15.125:3300/0 conn(0x7f46fc155b30 0x7f46fc0c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 -- 172.21.15.50:0/2095126088 >> v1:172.21.15.50:6789/0 conn(0x7f46fc155f00 legacy=0x7f46fc06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 -- 172.21.15.50:0/2095126088 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc06d0c0 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46faffd700 1 --2- 172.21.15.50:0/2095126088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc0ca3c0 0x7f46fc0af470 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0af1f0 tx=0x7f46f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1fe864a92718c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46fa7fc700 1 -- 172.21.15.50:0/2095126088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0009e70 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46fa7fc700 1 -- 172.21.15.50:0/2095126088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f000b860 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.043+0000 7f46fa7fc700 1 -- 172.21.15.50:0/2095126088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f000bc00 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- 172.21.15.50:0/2095126088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc0ca3c0 msgr2=0x7f46fc0af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 --2- 172.21.15.50:0/2095126088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc0ca3c0 0x7f46fc0af470 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0af1f0 tx=0x7f46f0009250 comp rx=0 tx=0).stop 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- 172.21.15.50:0/2095126088 shutdown_connections 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 --2- 172.21.15.50:0/2095126088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc0ca3c0 0x7f46fc0af470 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 --2- 172.21.15.50:0/2095126088 >> v2:172.21.15.125:3300/0 conn(0x7f46fc155b30 0x7f46fc0c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- 172.21.15.50:0/2095126088 >> 172.21.15.50:0/2095126088 conn(0x7f46fc067410 msgr2=0x7f46fc0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- 172.21.15.50:0/2095126088 shutdown_connections 2023-12-19T15:55:23.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- 172.21.15.50:0/2095126088 wait complete. 2023-12-19T15:55:23.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 Processor -- start 2023-12-19T15:55:23.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.047+0000 7f4701eae700 1 -- start start 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46fc0ca3c0 0x7f46fc159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46fc155f00 0x7f46fc14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0abe30 con 0x7f46fc155b30 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc157320 con 0x7f46fc155f00 2023-12-19T15:55:23.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0abc20 con 0x7f46fc0ca3c0 2023-12-19T15:55:23.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35866/0 (socket says 172.21.15.50:35866) 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 -- 172.21.15.50:0/2542588488 learned_addr learned my addr 172.21.15.50:0/2542588488 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46fbfff700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46fc155f00 0x7f46fc14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46fb7fe700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46fc0ca3c0 0x7f46fc159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 -- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46fc0ca3c0 msgr2=0x7f46fc159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46fc0ca3c0 0x7f46fc159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 -- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46fc155f00 msgr2=0x7f46fc14d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46fc155f00 0x7f46fc14d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 -- 172.21.15.50:0/2542588488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00094f0 con 0x7f46fc155b30 2023-12-19T15:55:23.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46faffd700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f46fc069c10 tx=0x7f46f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f000b9d0 con 0x7f46fc155b30 2023-12-19T15:55:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f000bb30 con 0x7f46fc155b30 2023-12-19T15:55:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc0ad2c0 con 0x7f46fc155b30 2023-12-19T15:55:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0025630 con 0x7f46fc155b30 2023-12-19T15:55:23.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.051+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc157040 con 0x7f46fc155b30 2023-12-19T15:55:23.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.055+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc071120 con 0x7f46fc155b30 2023-12-19T15:55:23.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.055+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f46f0023070 con 0x7f46fc155b30 2023-12-19T15:55:23.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.059+0000 7f46e37fe700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 0x7f46e8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.059+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f46f007df80 con 0x7f46fc155b30 2023-12-19T15:55:23.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.059+0000 7f46fb7fe700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 0x7f46e8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.059+0000 7f46fb7fe700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 0x7f46e8062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009d40 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.063+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f46f004e2c0 con 0x7f46fc155b30 2023-12-19T15:55:23.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.235+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46fc06d3a0 con 0x7f46e8060670 2023-12-19T15:55:23.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.251+0000 7f46e37fe700 1 -- 172.21.15.50:0/2542588488 <== mgr.8968 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287443 (secure 0 0 0) 0x7f46fc06d3a0 con 0x7f46e8060670 2023-12-19T15:55:23.261 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:23.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 msgr2=0x7f46e8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 0x7f46e8062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009d40 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 msgr2=0x7f46fc0ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f46fc069c10 tx=0x7f46f0017000 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 shutdown_connections 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46fc155f00 0x7f46fc14d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f46e8060670 0x7f46e8062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f46fc155b30 0x7f46fc0ab6e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 --2- 172.21.15.50:0/2542588488 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f46fc0ca3c0 0x7f46fc159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 >> 172.21.15.50:0/2542588488 conn(0x7f46fc067410 msgr2=0x7f46fc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 shutdown_connections 2023-12-19T15:55:23.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.259+0000 7f4701eae700 1 -- 172.21.15.50:0/2542588488 wait complete. 2023-12-19T15:55:23.266 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T15:55:23.286 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T15:55:22.300905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466720,"kb_used_data":18784,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752508000,"statfs":{"total":773094113280,"available":770568192000,"internally_reserved":0,"allocated":19234816,"data_stored":10323438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.387839"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975815+0000","last_change":"2023-12-19T15:55:18.975815+0000","last_active":"2023-12-19T15:55:18.975815+0000","last_peered":"2023-12-19T15:55:18.975815+0000","last_clean":"2023-12-19T15:55:18.975815+0000","last_became_active":"2023-12-19T15:55:18.975523+0000","last_became_peered":"2023-12-19T15:55:18.975523+0000","last_unstale":"2023-12-19T15:55:18.975815+0000","last_undegraded":"2023-12-19T15:55:18.975815+0000","last_fullsized":"2023-12-19T15:55:18.975815+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:33.198276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987936+0000","last_change":"2023-12-19T15:55:18.973233+0000","last_active":"2023-12-19T15:55:18.987936+0000","last_peered":"2023-12-19T15:55:18.987936+0000","last_clean":"2023-12-19T15:55:18.987936+0000","last_became_active":"2023-12-19T15:55:18.972935+0000","last_became_peered":"2023-12-19T15:55:18.972935+0000","last_unstale":"2023-12-19T15:55:18.987936+0000","last_undegraded":"2023-12-19T15:55:18.987936+0000","last_fullsized":"2023-12-19T15:55:18.987936+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:57.381319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987546+0000","last_change":"2023-12-19T15:55:18.971844+0000","last_active":"2023-12-19T15:55:18.987546+0000","last_peered":"2023-12-19T15:55:18.987546+0000","last_clean":"2023-12-19T15:55:18.987546+0000","last_became_active":"2023-12-19T15:55:18.971562+0000","last_became_peered":"2023-12-19T15:55:18.971562+0000","last_unstale":"2023-12-19T15:55:18.987546+0000","last_undegraded":"2023-12-19T15:55:18.987546+0000","last_fullsized":"2023-12-19T15:55:18.987546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:14:27.846499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971253+0000","last_change":"2023-12-19T15:55:18.967476+0000","last_active":"2023-12-19T15:55:18.971253+0000","last_peered":"2023-12-19T15:55:18.971253+0000","last_clean":"2023-12-19T15:55:18.971253+0000","last_became_active":"2023-12-19T15:55:18.967139+0000","last_became_peered":"2023-12-19T15:55:18.967139+0000","last_unstale":"2023-12-19T15:55:18.971253+0000","last_undegraded":"2023-12-19T15:55:18.971253+0000","last_fullsized":"2023-12-19T15:55:18.971253+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964997+0000","last_change":"2023-12-19T15:55:18.964997+0000","last_active":"2023-12-19T15:55:18.964997+0000","last_peered":"2023-12-19T15:55:18.964997+0000","last_clean":"2023-12-19T15:55:18.964997+0000","last_became_active":"2023-12-19T15:55:18.962616+0000","last_became_peered":"2023-12-19T15:55:18.962616+0000","last_unstale":"2023-12-19T15:55:18.964997+0000","last_undegraded":"2023-12-19T15:55:18.964997+0000","last_fullsized":"2023-12-19T15:55:18.964997+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:44.409742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961745+0000","last_change":"2023-12-19T15:55:18.961745+0000","last_active":"2023-12-19T15:55:18.961745+0000","last_peered":"2023-12-19T15:55:18.961745+0000","last_clean":"2023-12-19T15:55:18.961745+0000","last_became_active":"2023-12-19T15:55:18.961414+0000","last_became_peered":"2023-12-19T15:55:18.961414+0000","last_unstale":"2023-12-19T15:55:18.961745+0000","last_undegraded":"2023-12-19T15:55:18.961745+0000","last_fullsized":"2023-12-19T15:55:18.961745+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:59:44.887035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.973289+0000","last_change":"2023-12-19T15:55:18.973289+0000","last_active":"2023-12-19T15:55:18.973289+0000","last_peered":"2023-12-19T15:55:18.973289+0000","last_clean":"2023-12-19T15:55:18.973289+0000","last_became_active":"2023-12-19T15:55:18.970736+0000","last_became_peered":"2023-12-19T15:55:18.970736+0000","last_unstale":"2023-12-19T15:55:18.973289+0000","last_undegraded":"2023-12-19T15:55:18.973289+0000","last_fullsized":"2023-12-19T15:55:18.973289+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:52.002939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978705+0000","last_change":"2023-12-19T15:55:18.967058+0000","last_active":"2023-12-19T15:55:18.978705+0000","last_peered":"2023-12-19T15:55:18.978705+0000","last_clean":"2023-12-19T15:55:18.978705+0000","last_became_active":"2023-12-19T15:55:18.966753+0000","last_became_peered":"2023-12-19T15:55:18.966753+0000","last_unstale":"2023-12-19T15:55:18.978705+0000","last_undegraded":"2023-12-19T15:55:18.978705+0000","last_fullsized":"2023-12-19T15:55:18.978705+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:24:24.052412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975035+0000","last_change":"2023-12-19T15:55:18.975035+0000","last_active":"2023-12-19T15:55:18.975035+0000","last_peered":"2023-12-19T15:55:18.975035+0000","last_clean":"2023-12-19T15:55:18.975035+0000","last_became_active":"2023-12-19T15:55:18.974740+0000","last_became_peered":"2023-12-19T15:55:18.974740+0000","last_unstale":"2023-12-19T15:55:18.975035+0000","last_undegraded":"2023-12-19T15:55:18.975035+0000","last_fullsized":"2023-12-19T15:55:18.975035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964702+0000","last_change":"2023-12-19T15:55:18.964702+0000","last_active":"2023-12-19T15:55:18.964702+0000","last_peered":"2023-12-19T15:55:18.964702+0000","last_clean":"2023-12-19T15:55:18.964702+0000","last_became_active":"2023-12-19T15:55:18.964195+0000","last_became_peered":"2023-12-19T15:55:18.964195+0000","last_unstale":"2023-12-19T15:55:18.964702+0000","last_undegraded":"2023-12-19T15:55:18.964702+0000","last_fullsized":"2023-12-19T15:55:18.964702+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971766+0000","last_change":"2023-12-19T15:55:18.971766+0000","last_active":"2023-12-19T15:55:18.971766+0000","last_peered":"2023-12-19T15:55:18.971766+0000","last_clean":"2023-12-19T15:55:18.971766+0000","last_became_active":"2023-12-19T15:55:18.971240+0000","last_became_peered":"2023-12-19T15:55:18.971240+0000","last_unstale":"2023-12-19T15:55:18.971766+0000","last_undegraded":"2023-12-19T15:55:18.971766+0000","last_fullsized":"2023-12-19T15:55:18.971766+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:41.764312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971345+0000","last_change":"2023-12-19T15:55:18.969330+0000","last_active":"2023-12-19T15:55:18.971345+0000","last_peered":"2023-12-19T15:55:18.971345+0000","last_clean":"2023-12-19T15:55:18.971345+0000","last_became_active":"2023-12-19T15:55:18.968887+0000","last_became_peered":"2023-12-19T15:55:18.968887+0000","last_unstale":"2023-12-19T15:55:18.971345+0000","last_undegraded":"2023-12-19T15:55:18.971345+0000","last_fullsized":"2023-12-19T15:55:18.971345+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:35.872104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964965+0000","last_change":"2023-12-19T15:55:18.964965+0000","last_active":"2023-12-19T15:55:18.964965+0000","last_peered":"2023-12-19T15:55:18.964965+0000","last_clean":"2023-12-19T15:55:18.964965+0000","last_became_active":"2023-12-19T15:55:18.963532+0000","last_became_peered":"2023-12-19T15:55:18.963532+0000","last_unstale":"2023-12-19T15:55:18.964965+0000","last_undegraded":"2023-12-19T15:55:18.964965+0000","last_fullsized":"2023-12-19T15:55:18.964965+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:30.795790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986219+0000","last_change":"2023-12-19T15:55:18.982714+0000","last_active":"2023-12-19T15:55:18.986219+0000","last_peered":"2023-12-19T15:55:18.986219+0000","last_clean":"2023-12-19T15:55:18.986219+0000","last_became_active":"2023-12-19T15:55:18.982437+0000","last_became_peered":"2023-12-19T15:55:18.982437+0000","last_unstale":"2023-12-19T15:55:18.986219+0000","last_undegraded":"2023-12-19T15:55:18.986219+0000","last_fullsized":"2023-12-19T15:55:18.986219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987482+0000","last_change":"2023-12-19T15:55:18.970114+0000","last_active":"2023-12-19T15:55:18.987482+0000","last_peered":"2023-12-19T15:55:18.987482+0000","last_clean":"2023-12-19T15:55:18.987482+0000","last_became_active":"2023-12-19T15:55:18.969643+0000","last_became_peered":"2023-12-19T15:55:18.969643+0000","last_unstale":"2023-12-19T15:55:18.987482+0000","last_undegraded":"2023-12-19T15:55:18.987482+0000","last_fullsized":"2023-12-19T15:55:18.987482+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.966217+0000","last_change":"2023-12-19T15:55:18.966217+0000","last_active":"2023-12-19T15:55:18.966217+0000","last_peered":"2023-12-19T15:55:18.966217+0000","last_clean":"2023-12-19T15:55:18.966217+0000","last_became_active":"2023-12-19T15:55:18.965901+0000","last_became_peered":"2023-12-19T15:55:18.965901+0000","last_unstale":"2023-12-19T15:55:18.966217+0000","last_undegraded":"2023-12-19T15:55:18.966217+0000","last_fullsized":"2023-12-19T15:55:18.966217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971331+0000","last_change":"2023-12-19T15:55:18.966751+0000","last_active":"2023-12-19T15:55:18.971331+0000","last_peered":"2023-12-19T15:55:18.971331+0000","last_clean":"2023-12-19T15:55:18.971331+0000","last_became_active":"2023-12-19T15:55:18.965992+0000","last_became_peered":"2023-12-19T15:55:18.965992+0000","last_unstale":"2023-12-19T15:55:18.971331+0000","last_undegraded":"2023-12-19T15:55:18.971331+0000","last_fullsized":"2023-12-19T15:55:18.971331+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:30.795790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.963485+0000","last_change":"2023-12-19T15:55:18.963485+0000","last_active":"2023-12-19T15:55:18.963485+0000","last_peered":"2023-12-19T15:55:18.963485+0000","last_clean":"2023-12-19T15:55:18.963485+0000","last_became_active":"2023-12-19T15:55:18.963199+0000","last_became_peered":"2023-12-19T15:55:18.963199+0000","last_unstale":"2023-12-19T15:55:18.963485+0000","last_undegraded":"2023-12-19T15:55:18.963485+0000","last_fullsized":"2023-12-19T15:55:18.963485+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:21:51.204111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979601+0000","last_change":"2023-12-19T15:55:18.979601+0000","last_active":"2023-12-19T15:55:18.979601+0000","last_peered":"2023-12-19T15:55:18.979601+0000","last_clean":"2023-12-19T15:55:18.979601+0000","last_became_active":"2023-12-19T15:55:18.978847+0000","last_became_peered":"2023-12-19T15:55:18.978847+0000","last_unstale":"2023-12-19T15:55:18.979601+0000","last_undegraded":"2023-12-19T15:55:18.979601+0000","last_fullsized":"2023-12-19T15:55:18.979601+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:56:45.625418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.974894+0000","last_change":"2023-12-19T15:55:18.974894+0000","last_active":"2023-12-19T15:55:18.974894+0000","last_peered":"2023-12-19T15:55:18.974894+0000","last_clean":"2023-12-19T15:55:18.974894+0000","last_became_active":"2023-12-19T15:55:18.974601+0000","last_became_peered":"2023-12-19T15:55:18.974601+0000","last_unstale":"2023-12-19T15:55:18.974894+0000","last_undegraded":"2023-12-19T15:55:18.974894+0000","last_fullsized":"2023-12-19T15:55:18.974894+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:33:59.338948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971195+0000","last_change":"2023-12-19T15:55:18.971195+0000","last_active":"2023-12-19T15:55:18.971195+0000","last_peered":"2023-12-19T15:55:18.971195+0000","last_clean":"2023-12-19T15:55:18.971195+0000","last_became_active":"2023-12-19T15:55:18.970757+0000","last_became_peered":"2023-12-19T15:55:18.970757+0000","last_unstale":"2023-12-19T15:55:18.971195+0000","last_undegraded":"2023-12-19T15:55:18.971195+0000","last_fullsized":"2023-12-19T15:55:18.971195+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:16.749425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965513+0000","last_change":"2023-12-19T15:55:18.965513+0000","last_active":"2023-12-19T15:55:18.965513+0000","last_peered":"2023-12-19T15:55:18.965513+0000","last_clean":"2023-12-19T15:55:18.965513+0000","last_became_active":"2023-12-19T15:55:18.965083+0000","last_became_peered":"2023-12-19T15:55:18.965083+0000","last_unstale":"2023-12-19T15:55:18.965513+0000","last_undegraded":"2023-12-19T15:55:18.965513+0000","last_fullsized":"2023-12-19T15:55:18.965513+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:20.954312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.973862+0000","last_change":"2023-12-19T15:55:18.964139+0000","last_active":"2023-12-19T15:55:18.973862+0000","last_peered":"2023-12-19T15:55:18.973862+0000","last_clean":"2023-12-19T15:55:18.973862+0000","last_became_active":"2023-12-19T15:55:18.963096+0000","last_became_peered":"2023-12-19T15:55:18.963096+0000","last_unstale":"2023-12-19T15:55:18.973862+0000","last_undegraded":"2023-12-19T15:55:18.973862+0000","last_fullsized":"2023-12-19T15:55:18.973862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:59:44.887035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965510+0000","last_change":"2023-12-19T15:55:18.965510+0000","last_active":"2023-12-19T15:55:18.965510+0000","last_peered":"2023-12-19T15:55:18.965510+0000","last_clean":"2023-12-19T15:55:18.965510+0000","last_became_active":"2023-12-19T15:55:18.963128+0000","last_became_peered":"2023-12-19T15:55:18.963128+0000","last_unstale":"2023-12-19T15:55:18.965510+0000","last_undegraded":"2023-12-19T15:55:18.965510+0000","last_fullsized":"2023-12-19T15:55:18.965510+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:38:56.734422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961827+0000","last_change":"2023-12-19T15:55:18.961827+0000","last_active":"2023-12-19T15:55:18.961827+0000","last_peered":"2023-12-19T15:55:18.961827+0000","last_clean":"2023-12-19T15:55:18.961827+0000","last_became_active":"2023-12-19T15:55:18.961550+0000","last_became_peered":"2023-12-19T15:55:18.961550+0000","last_unstale":"2023-12-19T15:55:18.961827+0000","last_undegraded":"2023-12-19T15:55:18.961827+0000","last_fullsized":"2023-12-19T15:55:18.961827+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:16.293313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964686+0000","last_change":"2023-12-19T15:55:18.964686+0000","last_active":"2023-12-19T15:55:18.964686+0000","last_peered":"2023-12-19T15:55:18.964686+0000","last_clean":"2023-12-19T15:55:18.964686+0000","last_became_active":"2023-12-19T15:55:18.964271+0000","last_became_peered":"2023-12-19T15:55:18.964271+0000","last_unstale":"2023-12-19T15:55:18.964686+0000","last_undegraded":"2023-12-19T15:55:18.964686+0000","last_fullsized":"2023-12-19T15:55:18.964686+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:34:34.746065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969473+0000","last_change":"2023-12-19T15:55:18.969473+0000","last_active":"2023-12-19T15:55:18.969473+0000","last_peered":"2023-12-19T15:55:18.969473+0000","last_clean":"2023-12-19T15:55:18.969473+0000","last_became_active":"2023-12-19T15:55:18.968173+0000","last_became_peered":"2023-12-19T15:55:18.968173+0000","last_unstale":"2023-12-19T15:55:18.969473+0000","last_undegraded":"2023-12-19T15:55:18.969473+0000","last_fullsized":"2023-12-19T15:55:18.969473+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:04.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.962359+0000","last_change":"2023-12-19T15:55:18.962359+0000","last_active":"2023-12-19T15:55:18.962359+0000","last_peered":"2023-12-19T15:55:18.962359+0000","last_clean":"2023-12-19T15:55:18.962359+0000","last_became_active":"2023-12-19T15:55:18.962035+0000","last_became_peered":"2023-12-19T15:55:18.962035+0000","last_unstale":"2023-12-19T15:55:18.962359+0000","last_undegraded":"2023-12-19T15:55:18.962359+0000","last_fullsized":"2023-12-19T15:55:18.962359+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:38.089112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978176+0000","last_change":"2023-12-19T15:55:18.971010+0000","last_active":"2023-12-19T15:55:18.978176+0000","last_peered":"2023-12-19T15:55:18.978176+0000","last_clean":"2023-12-19T15:55:18.978176+0000","last_became_active":"2023-12-19T15:55:18.970723+0000","last_became_peered":"2023-12-19T15:55:18.970723+0000","last_unstale":"2023-12-19T15:55:18.978176+0000","last_undegraded":"2023-12-19T15:55:18.978176+0000","last_fullsized":"2023-12-19T15:55:18.978176+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:48.581415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.968469+0000","last_change":"2023-12-19T15:55:18.968469+0000","last_active":"2023-12-19T15:55:18.968469+0000","last_peered":"2023-12-19T15:55:18.968469+0000","last_clean":"2023-12-19T15:55:18.968469+0000","last_became_active":"2023-12-19T15:55:18.968184+0000","last_became_peered":"2023-12-19T15:55:18.968184+0000","last_unstale":"2023-12-19T15:55:18.968469+0000","last_undegraded":"2023-12-19T15:55:18.968469+0000","last_fullsized":"2023-12-19T15:55:18.968469+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:20.998952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975867+0000","last_change":"2023-12-19T15:55:18.975867+0000","last_active":"2023-12-19T15:55:18.975867+0000","last_peered":"2023-12-19T15:55:18.975867+0000","last_clean":"2023-12-19T15:55:18.975867+0000","last_became_active":"2023-12-19T15:55:18.975538+0000","last_became_peered":"2023-12-19T15:55:18.975538+0000","last_unstale":"2023-12-19T15:55:18.975867+0000","last_undegraded":"2023-12-19T15:55:18.975867+0000","last_fullsized":"2023-12-19T15:55:18.975867+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978741+0000","last_change":"2023-12-19T15:55:18.974796+0000","last_active":"2023-12-19T15:55:18.978741+0000","last_peered":"2023-12-19T15:55:18.978741+0000","last_clean":"2023-12-19T15:55:18.978741+0000","last_became_active":"2023-12-19T15:55:18.974485+0000","last_became_peered":"2023-12-19T15:55:18.974485+0000","last_unstale":"2023-12-19T15:55:18.978741+0000","last_undegraded":"2023-12-19T15:55:18.978741+0000","last_fullsized":"2023-12-19T15:55:18.978741+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987706+0000","last_change":"2023-12-19T15:55:18.970875+0000","last_active":"2023-12-19T15:55:18.987706+0000","last_peered":"2023-12-19T15:55:18.987706+0000","last_clean":"2023-12-19T15:55:18.987706+0000","last_became_active":"2023-12-19T15:55:18.970533+0000","last_became_peered":"2023-12-19T15:55:18.970533+0000","last_unstale":"2023-12-19T15:55:18.987706+0000","last_undegraded":"2023-12-19T15:55:18.987706+0000","last_fullsized":"2023-12-19T15:55:18.987706+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:31.268476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"180'4","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.989726+0000","last_change":"2023-12-19T15:55:17.958262+0000","last_active":"2023-12-19T15:55:18.989726+0000","last_peered":"2023-12-19T15:55:18.989726+0000","last_clean":"2023-12-19T15:55:18.989726+0000","last_became_active":"2023-12-19T15:55:17.956882+0000","last_became_peered":"2023-12-19T15:55:17.956882+0000","last_unstale":"2023-12-19T15:55:18.989726+0000","last_undegraded":"2023-12-19T15:55:18.989726+0000","last_fullsized":"2023-12-19T15:55:18.989726+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:25.188298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987572+0000","last_change":"2023-12-19T15:55:18.972817+0000","last_active":"2023-12-19T15:55:18.987572+0000","last_peered":"2023-12-19T15:55:18.987572+0000","last_clean":"2023-12-19T15:55:18.987572+0000","last_became_active":"2023-12-19T15:55:18.972495+0000","last_became_peered":"2023-12-19T15:55:18.972495+0000","last_unstale":"2023-12-19T15:55:18.987572+0000","last_undegraded":"2023-12-19T15:55:18.987572+0000","last_fullsized":"2023-12-19T15:55:18.987572+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:01.844134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.958701+0000","last_change":"2023-12-19T15:55:17.967625+0000","last_active":"2023-12-19T15:55:18.958701+0000","last_peered":"2023-12-19T15:55:18.958701+0000","last_clean":"2023-12-19T15:55:18.958701+0000","last_became_active":"2023-12-19T15:55:17.966297+0000","last_became_peered":"2023-12-19T15:55:17.966297+0000","last_unstale":"2023-12-19T15:55:18.958701+0000","last_undegraded":"2023-12-19T15:55:18.958701+0000","last_fullsized":"2023-12-19T15:55:18.958701+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:54.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976195+0000","last_change":"2023-12-19T15:55:18.967410+0000","last_active":"2023-12-19T15:55:18.976195+0000","last_peered":"2023-12-19T15:55:18.976195+0000","last_clean":"2023-12-19T15:55:18.976195+0000","last_became_active":"2023-12-19T15:55:18.966398+0000","last_became_peered":"2023-12-19T15:55:18.966398+0000","last_unstale":"2023-12-19T15:55:18.976195+0000","last_undegraded":"2023-12-19T15:55:18.976195+0000","last_fullsized":"2023-12-19T15:55:18.976195+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:37.344641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986726+0000","last_change":"2023-12-19T15:55:17.967120+0000","last_active":"2023-12-19T15:55:18.986726+0000","last_peered":"2023-12-19T15:55:18.986726+0000","last_clean":"2023-12-19T15:55:18.986726+0000","last_became_active":"2023-12-19T15:55:17.966783+0000","last_became_peered":"2023-12-19T15:55:17.966783+0000","last_unstale":"2023-12-19T15:55:18.986726+0000","last_undegraded":"2023-12-19T15:55:18.986726+0000","last_fullsized":"2023-12-19T15:55:18.986726+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:56.364922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.968027+0000","last_change":"2023-12-19T15:55:18.968027+0000","last_active":"2023-12-19T15:55:18.968027+0000","last_peered":"2023-12-19T15:55:18.968027+0000","last_clean":"2023-12-19T15:55:18.968027+0000","last_became_active":"2023-12-19T15:55:18.967751+0000","last_became_peered":"2023-12-19T15:55:18.967751+0000","last_unstale":"2023-12-19T15:55:18.968027+0000","last_undegraded":"2023-12-19T15:55:18.968027+0000","last_fullsized":"2023-12-19T15:55:18.968027+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991162+0000","last_change":"2023-12-19T15:55:17.962243+0000","last_active":"2023-12-19T15:55:18.991162+0000","last_peered":"2023-12-19T15:55:18.991162+0000","last_clean":"2023-12-19T15:55:18.991162+0000","last_became_active":"2023-12-19T15:55:17.961954+0000","last_became_peered":"2023-12-19T15:55:17.961954+0000","last_unstale":"2023-12-19T15:55:18.991162+0000","last_undegraded":"2023-12-19T15:55:18.991162+0000","last_fullsized":"2023-12-19T15:55:18.991162+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:23.322810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.967189+0000","last_change":"2023-12-19T15:55:18.967189+0000","last_active":"2023-12-19T15:55:18.967189+0000","last_peered":"2023-12-19T15:55:18.967189+0000","last_clean":"2023-12-19T15:55:18.967189+0000","last_became_active":"2023-12-19T15:55:18.966908+0000","last_became_peered":"2023-12-19T15:55:18.966908+0000","last_unstale":"2023-12-19T15:55:18.967189+0000","last_undegraded":"2023-12-19T15:55:18.967189+0000","last_fullsized":"2023-12-19T15:55:18.967189+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:56:08.956732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988992+0000","last_change":"2023-12-19T15:55:17.965774+0000","last_active":"2023-12-19T15:55:18.988992+0000","last_peered":"2023-12-19T15:55:18.988992+0000","last_clean":"2023-12-19T15:55:18.988992+0000","last_became_active":"2023-12-19T15:55:17.965508+0000","last_became_peered":"2023-12-19T15:55:17.965508+0000","last_unstale":"2023-12-19T15:55:18.988992+0000","last_undegraded":"2023-12-19T15:55:18.988992+0000","last_fullsized":"2023-12-19T15:55:18.988992+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:14:26.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971636+0000","last_change":"2023-12-19T15:55:18.968597+0000","last_active":"2023-12-19T15:55:18.971636+0000","last_peered":"2023-12-19T15:55:18.971636+0000","last_clean":"2023-12-19T15:55:18.971636+0000","last_became_active":"2023-12-19T15:55:18.968064+0000","last_became_peered":"2023-12-19T15:55:18.968064+0000","last_unstale":"2023-12-19T15:55:18.971636+0000","last_undegraded":"2023-12-19T15:55:18.971636+0000","last_fullsized":"2023-12-19T15:55:18.971636+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:44.409742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.949443+0000","last_change":"2023-12-19T15:55:17.959550+0000","last_active":"2023-12-19T15:55:18.949443+0000","last_peered":"2023-12-19T15:55:18.949443+0000","last_clean":"2023-12-19T15:55:18.949443+0000","last_became_active":"2023-12-19T15:55:17.959280+0000","last_became_peered":"2023-12-19T15:55:17.959280+0000","last_unstale":"2023-12-19T15:55:18.949443+0000","last_undegraded":"2023-12-19T15:55:18.949443+0000","last_fullsized":"2023-12-19T15:55:18.949443+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:31:28.135795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.970424+0000","last_change":"2023-12-19T15:55:18.962359+0000","last_active":"2023-12-19T15:55:18.970424+0000","last_peered":"2023-12-19T15:55:18.970424+0000","last_clean":"2023-12-19T15:55:18.970424+0000","last_became_active":"2023-12-19T15:55:18.962065+0000","last_became_peered":"2023-12-19T15:55:18.962065+0000","last_unstale":"2023-12-19T15:55:18.970424+0000","last_undegraded":"2023-12-19T15:55:18.970424+0000","last_fullsized":"2023-12-19T15:55:18.970424+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:50.529014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965922+0000","last_change":"2023-12-19T15:55:17.960359+0000","last_active":"2023-12-19T15:55:17.965922+0000","last_peered":"2023-12-19T15:55:17.965922+0000","last_clean":"2023-12-19T15:55:17.965922+0000","last_became_active":"2023-12-19T15:55:17.959201+0000","last_became_peered":"2023-12-19T15:55:17.959201+0000","last_unstale":"2023-12-19T15:55:17.965922+0000","last_undegraded":"2023-12-19T15:55:17.965922+0000","last_fullsized":"2023-12-19T15:55:17.965922+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:34.296723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976061+0000","last_change":"2023-12-19T15:55:18.976061+0000","last_active":"2023-12-19T15:55:18.976061+0000","last_peered":"2023-12-19T15:55:18.976061+0000","last_clean":"2023-12-19T15:55:18.976061+0000","last_became_active":"2023-12-19T15:55:18.974488+0000","last_became_peered":"2023-12-19T15:55:18.974488+0000","last_unstale":"2023-12-19T15:55:18.976061+0000","last_undegraded":"2023-12-19T15:55:18.976061+0000","last_fullsized":"2023-12-19T15:55:18.976061+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:43:13.777553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965349+0000","last_change":"2023-12-19T15:55:17.961924+0000","last_active":"2023-12-19T15:55:17.965349+0000","last_peered":"2023-12-19T15:55:17.965349+0000","last_clean":"2023-12-19T15:55:17.965349+0000","last_became_active":"2023-12-19T15:55:17.961595+0000","last_became_peered":"2023-12-19T15:55:17.961595+0000","last_unstale":"2023-12-19T15:55:17.965349+0000","last_undegraded":"2023-12-19T15:55:17.965349+0000","last_fullsized":"2023-12-19T15:55:17.965349+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:45.186605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978363+0000","last_change":"2023-12-19T15:55:18.971112+0000","last_active":"2023-12-19T15:55:18.978363+0000","last_peered":"2023-12-19T15:55:18.978363+0000","last_clean":"2023-12-19T15:55:18.978363+0000","last_became_active":"2023-12-19T15:55:18.970820+0000","last_became_peered":"2023-12-19T15:55:18.970820+0000","last_unstale":"2023-12-19T15:55:18.978363+0000","last_undegraded":"2023-12-19T15:55:18.978363+0000","last_fullsized":"2023-12-19T15:55:18.978363+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:54.641249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.969092+0000","last_change":"2023-12-19T15:55:17.959331+0000","last_active":"2023-12-19T15:55:17.969092+0000","last_peered":"2023-12-19T15:55:17.969092+0000","last_clean":"2023-12-19T15:55:17.969092+0000","last_became_active":"2023-12-19T15:55:17.959026+0000","last_became_peered":"2023-12-19T15:55:17.959026+0000","last_unstale":"2023-12-19T15:55:17.969092+0000","last_undegraded":"2023-12-19T15:55:17.969092+0000","last_fullsized":"2023-12-19T15:55:17.969092+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:52.802694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971554+0000","last_change":"2023-12-19T15:55:18.971554+0000","last_active":"2023-12-19T15:55:18.971554+0000","last_peered":"2023-12-19T15:55:18.971554+0000","last_clean":"2023-12-19T15:55:18.971554+0000","last_became_active":"2023-12-19T15:55:18.970012+0000","last_became_peered":"2023-12-19T15:55:18.970012+0000","last_unstale":"2023-12-19T15:55:18.971554+0000","last_undegraded":"2023-12-19T15:55:18.971554+0000","last_fullsized":"2023-12-19T15:55:18.971554+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:57.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966097+0000","last_change":"2023-12-19T15:55:17.958315+0000","last_active":"2023-12-19T15:55:17.966097+0000","last_peered":"2023-12-19T15:55:17.966097+0000","last_clean":"2023-12-19T15:55:17.966097+0000","last_became_active":"2023-12-19T15:55:17.957664+0000","last_became_peered":"2023-12-19T15:55:17.957664+0000","last_unstale":"2023-12-19T15:55:17.966097+0000","last_undegraded":"2023-12-19T15:55:17.966097+0000","last_fullsized":"2023-12-19T15:55:17.966097+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:38.132121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965339+0000","last_change":"2023-12-19T15:55:18.965339+0000","last_active":"2023-12-19T15:55:18.965339+0000","last_peered":"2023-12-19T15:55:18.965339+0000","last_clean":"2023-12-19T15:55:18.965339+0000","last_became_active":"2023-12-19T15:55:18.962291+0000","last_became_peered":"2023-12-19T15:55:18.962291+0000","last_unstale":"2023-12-19T15:55:18.965339+0000","last_undegraded":"2023-12-19T15:55:18.965339+0000","last_fullsized":"2023-12-19T15:55:18.965339+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:59.607212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"180'3","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.985277+0000","last_change":"2023-12-19T15:55:17.958007+0000","last_active":"2023-12-19T15:55:18.985277+0000","last_peered":"2023-12-19T15:55:18.985277+0000","last_clean":"2023-12-19T15:55:18.985277+0000","last_became_active":"2023-12-19T15:55:17.957527+0000","last_became_peered":"2023-12-19T15:55:17.957527+0000","last_unstale":"2023-12-19T15:55:18.985277+0000","last_undegraded":"2023-12-19T15:55:18.985277+0000","last_fullsized":"2023-12-19T15:55:18.985277+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:21.324701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978725+0000","last_change":"2023-12-19T15:55:18.974154+0000","last_active":"2023-12-19T15:55:18.978725+0000","last_peered":"2023-12-19T15:55:18.978725+0000","last_clean":"2023-12-19T15:55:18.978725+0000","last_became_active":"2023-12-19T15:55:18.973848+0000","last_became_peered":"2023-12-19T15:55:18.973848+0000","last_unstale":"2023-12-19T15:55:18.978725+0000","last_undegraded":"2023-12-19T15:55:18.978725+0000","last_fullsized":"2023-12-19T15:55:18.978725+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:56:45.625418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986693+0000","last_change":"2023-12-19T15:55:17.957170+0000","last_active":"2023-12-19T15:55:18.986693+0000","last_peered":"2023-12-19T15:55:18.986693+0000","last_clean":"2023-12-19T15:55:18.986693+0000","last_became_active":"2023-12-19T15:55:17.956825+0000","last_became_peered":"2023-12-19T15:55:17.956825+0000","last_unstale":"2023-12-19T15:55:18.986693+0000","last_undegraded":"2023-12-19T15:55:18.986693+0000","last_fullsized":"2023-12-19T15:55:18.986693+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:25.052281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987456+0000","last_change":"2023-12-19T15:55:18.969923+0000","last_active":"2023-12-19T15:55:18.987456+0000","last_peered":"2023-12-19T15:55:18.987456+0000","last_clean":"2023-12-19T15:55:18.987456+0000","last_became_active":"2023-12-19T15:55:18.969258+0000","last_became_peered":"2023-12-19T15:55:18.969258+0000","last_unstale":"2023-12-19T15:55:18.987456+0000","last_undegraded":"2023-12-19T15:55:18.987456+0000","last_fullsized":"2023-12-19T15:55:18.987456+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:35.872104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961735+0000","last_change":"2023-12-19T15:55:17.961535+0000","last_active":"2023-12-19T15:55:18.961735+0000","last_peered":"2023-12-19T15:55:18.961735+0000","last_clean":"2023-12-19T15:55:18.961735+0000","last_became_active":"2023-12-19T15:55:17.961262+0000","last_became_peered":"2023-12-19T15:55:17.961262+0000","last_unstale":"2023-12-19T15:55:18.961735+0000","last_undegraded":"2023-12-19T15:55:18.961735+0000","last_fullsized":"2023-12-19T15:55:18.961735+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:33.813773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.954081+0000","last_change":"2023-12-19T15:55:18.954081+0000","last_active":"2023-12-19T15:55:18.954081+0000","last_peered":"2023-12-19T15:55:18.954081+0000","last_clean":"2023-12-19T15:55:18.954081+0000","last_became_active":"2023-12-19T15:55:18.952682+0000","last_became_peered":"2023-12-19T15:55:18.952682+0000","last_unstale":"2023-12-19T15:55:18.954081+0000","last_undegraded":"2023-12-19T15:55:18.954081+0000","last_fullsized":"2023-12-19T15:55:18.954081+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:57.381319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961220+0000","last_change":"2023-12-19T15:55:17.963702+0000","last_active":"2023-12-19T15:55:18.961220+0000","last_peered":"2023-12-19T15:55:18.961220+0000","last_clean":"2023-12-19T15:55:18.961220+0000","last_became_active":"2023-12-19T15:55:17.963429+0000","last_became_peered":"2023-12-19T15:55:17.963429+0000","last_unstale":"2023-12-19T15:55:18.961220+0000","last_undegraded":"2023-12-19T15:55:18.961220+0000","last_fullsized":"2023-12-19T15:55:18.961220+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:56.708610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988067+0000","last_change":"2023-12-19T15:55:18.973855+0000","last_active":"2023-12-19T15:55:18.988067+0000","last_peered":"2023-12-19T15:55:18.988067+0000","last_clean":"2023-12-19T15:55:18.988067+0000","last_became_active":"2023-12-19T15:55:18.973568+0000","last_became_peered":"2023-12-19T15:55:18.973568+0000","last_unstale":"2023-12-19T15:55:18.988067+0000","last_undegraded":"2023-12-19T15:55:18.988067+0000","last_fullsized":"2023-12-19T15:55:18.988067+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:13.143541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"180'8","reported_seq":17,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.989621+0000","last_change":"2023-12-19T15:55:17.957953+0000","last_active":"2023-12-19T15:55:18.989621+0000","last_peered":"2023-12-19T15:55:18.989621+0000","last_clean":"2023-12-19T15:55:18.989621+0000","last_became_active":"2023-12-19T15:55:17.957608+0000","last_became_peered":"2023-12-19T15:55:17.957608+0000","last_unstale":"2023-12-19T15:55:18.989621+0000","last_undegraded":"2023-12-19T15:55:18.989621+0000","last_fullsized":"2023-12-19T15:55:18.989621+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:06.910299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.970910+0000","last_change":"2023-12-19T15:55:18.970910+0000","last_active":"2023-12-19T15:55:18.970910+0000","last_peered":"2023-12-19T15:55:18.970910+0000","last_clean":"2023-12-19T15:55:18.970910+0000","last_became_active":"2023-12-19T15:55:18.970627+0000","last_became_peered":"2023-12-19T15:55:18.970627+0000","last_unstale":"2023-12-19T15:55:18.970910+0000","last_undegraded":"2023-12-19T15:55:18.970910+0000","last_fullsized":"2023-12-19T15:55:18.970910+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:59.642359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.944235+0000","last_change":"2023-12-19T15:55:17.957770+0000","last_active":"2023-12-19T15:55:18.944235+0000","last_peered":"2023-12-19T15:55:18.944235+0000","last_clean":"2023-12-19T15:55:18.944235+0000","last_became_active":"2023-12-19T15:55:17.957080+0000","last_became_peered":"2023-12-19T15:55:17.957080+0000","last_unstale":"2023-12-19T15:55:18.944235+0000","last_undegraded":"2023-12-19T15:55:18.944235+0000","last_fullsized":"2023-12-19T15:55:18.944235+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:16.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979782+0000","last_change":"2023-12-19T15:55:18.979782+0000","last_active":"2023-12-19T15:55:18.979782+0000","last_peered":"2023-12-19T15:55:18.979782+0000","last_clean":"2023-12-19T15:55:18.979782+0000","last_became_active":"2023-12-19T15:55:18.979297+0000","last_became_peered":"2023-12-19T15:55:18.979297+0000","last_unstale":"2023-12-19T15:55:18.979782+0000","last_undegraded":"2023-12-19T15:55:18.979782+0000","last_fullsized":"2023-12-19T15:55:18.979782+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:37.344641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987010+0000","last_change":"2023-12-19T15:55:17.964679+0000","last_active":"2023-12-19T15:55:18.987010+0000","last_peered":"2023-12-19T15:55:18.987010+0000","last_clean":"2023-12-19T15:55:18.987010+0000","last_became_active":"2023-12-19T15:55:17.964269+0000","last_became_peered":"2023-12-19T15:55:17.964269+0000","last_unstale":"2023-12-19T15:55:18.987010+0000","last_undegraded":"2023-12-19T15:55:18.987010+0000","last_fullsized":"2023-12-19T15:55:18.987010+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:34.855708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975670+0000","last_change":"2023-12-19T15:55:18.975670+0000","last_active":"2023-12-19T15:55:18.975670+0000","last_peered":"2023-12-19T15:55:18.975670+0000","last_clean":"2023-12-19T15:55:18.975670+0000","last_became_active":"2023-12-19T15:55:18.973618+0000","last_became_peered":"2023-12-19T15:55:18.973618+0000","last_unstale":"2023-12-19T15:55:18.975670+0000","last_undegraded":"2023-12-19T15:55:18.975670+0000","last_fullsized":"2023-12-19T15:55:18.975670+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:03.278922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991196+0000","last_change":"2023-12-19T15:55:17.969785+0000","last_active":"2023-12-19T15:55:18.991196+0000","last_peered":"2023-12-19T15:55:18.991196+0000","last_clean":"2023-12-19T15:55:18.991196+0000","last_became_active":"2023-12-19T15:55:17.969271+0000","last_became_peered":"2023-12-19T15:55:17.969271+0000","last_unstale":"2023-12-19T15:55:18.991196+0000","last_undegraded":"2023-12-19T15:55:18.991196+0000","last_fullsized":"2023-12-19T15:55:18.991196+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:40:22.384403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969239+0000","last_change":"2023-12-19T15:55:18.969239+0000","last_active":"2023-12-19T15:55:18.969239+0000","last_peered":"2023-12-19T15:55:18.969239+0000","last_clean":"2023-12-19T15:55:18.969239+0000","last_became_active":"2023-12-19T15:55:18.968935+0000","last_became_peered":"2023-12-19T15:55:18.968935+0000","last_unstale":"2023-12-19T15:55:18.969239+0000","last_undegraded":"2023-12-19T15:55:18.969239+0000","last_fullsized":"2023-12-19T15:55:18.969239+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:22.708126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961178+0000","last_change":"2023-12-19T15:55:17.961398+0000","last_active":"2023-12-19T15:55:18.961178+0000","last_peered":"2023-12-19T15:55:18.961178+0000","last_clean":"2023-12-19T15:55:18.961178+0000","last_became_active":"2023-12-19T15:55:17.961003+0000","last_became_peered":"2023-12-19T15:55:17.961003+0000","last_unstale":"2023-12-19T15:55:18.961178+0000","last_undegraded":"2023-12-19T15:55:18.961178+0000","last_fullsized":"2023-12-19T15:55:18.961178+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:15:43.255409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.981536+0000","last_change":"2023-12-19T15:55:18.981536+0000","last_active":"2023-12-19T15:55:18.981536+0000","last_peered":"2023-12-19T15:55:18.981536+0000","last_clean":"2023-12-19T15:55:18.981536+0000","last_became_active":"2023-12-19T15:55:18.977070+0000","last_became_peered":"2023-12-19T15:55:18.977070+0000","last_unstale":"2023-12-19T15:55:18.981536+0000","last_undegraded":"2023-12-19T15:55:18.981536+0000","last_fullsized":"2023-12-19T15:55:18.981536+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:54.641249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987651+0000","last_change":"2023-12-19T15:55:17.969021+0000","last_active":"2023-12-19T15:55:18.987651+0000","last_peered":"2023-12-19T15:55:18.987651+0000","last_clean":"2023-12-19T15:55:18.987651+0000","last_became_active":"2023-12-19T15:55:17.968704+0000","last_became_peered":"2023-12-19T15:55:17.968704+0000","last_unstale":"2023-12-19T15:55:18.987651+0000","last_undegraded":"2023-12-19T15:55:18.987651+0000","last_fullsized":"2023-12-19T15:55:18.987651+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:53.624853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971324+0000","last_change":"2023-12-19T15:55:18.959196+0000","last_active":"2023-12-19T15:55:18.971324+0000","last_peered":"2023-12-19T15:55:18.971324+0000","last_clean":"2023-12-19T15:55:18.971324+0000","last_became_active":"2023-12-19T15:55:18.958853+0000","last_became_peered":"2023-12-19T15:55:18.958853+0000","last_unstale":"2023-12-19T15:55:18.971324+0000","last_undegraded":"2023-12-19T15:55:18.971324+0000","last_fullsized":"2023-12-19T15:55:18.971324+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:01:52.197350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.955860+0000","last_change":"2023-12-19T15:55:17.953419+0000","last_active":"2023-12-19T15:55:18.955860+0000","last_peered":"2023-12-19T15:55:18.955860+0000","last_clean":"2023-12-19T15:55:18.955860+0000","last_became_active":"2023-12-19T15:55:17.953093+0000","last_became_peered":"2023-12-19T15:55:17.953093+0000","last_unstale":"2023-12-19T15:55:18.955860+0000","last_undegraded":"2023-12-19T15:55:18.955860+0000","last_fullsized":"2023-12-19T15:55:18.955860+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:49.512617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969869+0000","last_change":"2023-12-19T15:55:18.969869+0000","last_active":"2023-12-19T15:55:18.969869+0000","last_peered":"2023-12-19T15:55:18.969869+0000","last_clean":"2023-12-19T15:55:18.969869+0000","last_became_active":"2023-12-19T15:55:18.968957+0000","last_became_peered":"2023-12-19T15:55:18.968957+0000","last_unstale":"2023-12-19T15:55:18.969869+0000","last_undegraded":"2023-12-19T15:55:18.969869+0000","last_fullsized":"2023-12-19T15:55:18.969869+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.970206+0000","last_change":"2023-12-19T15:55:17.968685+0000","last_active":"2023-12-19T15:55:17.970206+0000","last_peered":"2023-12-19T15:55:17.970206+0000","last_clean":"2023-12-19T15:55:17.970206+0000","last_became_active":"2023-12-19T15:55:17.968290+0000","last_became_peered":"2023-12-19T15:55:17.968290+0000","last_unstale":"2023-12-19T15:55:17.970206+0000","last_undegraded":"2023-12-19T15:55:17.970206+0000","last_fullsized":"2023-12-19T15:55:17.970206+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:36.319744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987551+0000","last_change":"2023-12-19T15:55:18.969710+0000","last_active":"2023-12-19T15:55:18.987551+0000","last_peered":"2023-12-19T15:55:18.987551+0000","last_clean":"2023-12-19T15:55:18.987551+0000","last_became_active":"2023-12-19T15:55:18.969420+0000","last_became_peered":"2023-12-19T15:55:18.969420+0000","last_unstale":"2023-12-19T15:55:18.987551+0000","last_undegraded":"2023-12-19T15:55:18.987551+0000","last_fullsized":"2023-12-19T15:55:18.987551+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:53.819090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.953407+0000","last_change":"2023-12-19T15:55:17.957014+0000","last_active":"2023-12-19T15:55:18.953407+0000","last_peered":"2023-12-19T15:55:18.953407+0000","last_clean":"2023-12-19T15:55:18.953407+0000","last_became_active":"2023-12-19T15:55:17.956697+0000","last_became_peered":"2023-12-19T15:55:17.956697+0000","last_unstale":"2023-12-19T15:55:18.953407+0000","last_undegraded":"2023-12-19T15:55:18.953407+0000","last_fullsized":"2023-12-19T15:55:18.953407+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:12.127145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964273+0000","last_change":"2023-12-19T15:55:18.964273+0000","last_active":"2023-12-19T15:55:18.964273+0000","last_peered":"2023-12-19T15:55:18.964273+0000","last_clean":"2023-12-19T15:55:18.964273+0000","last_became_active":"2023-12-19T15:55:18.963936+0000","last_became_peered":"2023-12-19T15:55:18.963936+0000","last_unstale":"2023-12-19T15:55:18.964273+0000","last_undegraded":"2023-12-19T15:55:18.964273+0000","last_fullsized":"2023-12-19T15:55:18.964273+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:03:10.738658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986542+0000","last_change":"2023-12-19T15:55:17.967206+0000","last_active":"2023-12-19T15:55:18.986542+0000","last_peered":"2023-12-19T15:55:18.986542+0000","last_clean":"2023-12-19T15:55:18.986542+0000","last_became_active":"2023-12-19T15:55:17.966384+0000","last_became_peered":"2023-12-19T15:55:17.966384+0000","last_unstale":"2023-12-19T15:55:18.986542+0000","last_undegraded":"2023-12-19T15:55:18.986542+0000","last_fullsized":"2023-12-19T15:55:18.986542+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:59:27.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988767+0000","last_change":"2023-12-19T15:55:18.984744+0000","last_active":"2023-12-19T15:55:18.988767+0000","last_peered":"2023-12-19T15:55:18.988767+0000","last_clean":"2023-12-19T15:55:18.988767+0000","last_became_active":"2023-12-19T15:55:18.984449+0000","last_became_peered":"2023-12-19T15:55:18.984449+0000","last_unstale":"2023-12-19T15:55:18.988767+0000","last_undegraded":"2023-12-19T15:55:18.988767+0000","last_fullsized":"2023-12-19T15:55:18.988767+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:04.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966080+0000","last_change":"2023-12-19T15:55:17.958516+0000","last_active":"2023-12-19T15:55:17.966080+0000","last_peered":"2023-12-19T15:55:17.966080+0000","last_clean":"2023-12-19T15:55:17.966080+0000","last_became_active":"2023-12-19T15:55:17.958054+0000","last_became_peered":"2023-12-19T15:55:17.958054+0000","last_unstale":"2023-12-19T15:55:17.966080+0000","last_undegraded":"2023-12-19T15:55:17.966080+0000","last_fullsized":"2023-12-19T15:55:17.966080+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:48.063182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978580+0000","last_change":"2023-12-19T15:55:18.978580+0000","last_active":"2023-12-19T15:55:18.978580+0000","last_peered":"2023-12-19T15:55:18.978580+0000","last_clean":"2023-12-19T15:55:18.978580+0000","last_became_active":"2023-12-19T15:55:18.977988+0000","last_became_peered":"2023-12-19T15:55:18.977988+0000","last_unstale":"2023-12-19T15:55:18.978580+0000","last_undegraded":"2023-12-19T15:55:18.978580+0000","last_fullsized":"2023-12-19T15:55:18.978580+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:31:58.020254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.954388+0000","last_change":"2023-12-19T15:55:17.960733+0000","last_active":"2023-12-19T15:55:18.954388+0000","last_peered":"2023-12-19T15:55:18.954388+0000","last_clean":"2023-12-19T15:55:18.954388+0000","last_became_active":"2023-12-19T15:55:17.960441+0000","last_became_peered":"2023-12-19T15:55:17.960441+0000","last_unstale":"2023-12-19T15:55:18.954388+0000","last_undegraded":"2023-12-19T15:55:18.954388+0000","last_fullsized":"2023-12-19T15:55:18.954388+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:32.131618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969795+0000","last_change":"2023-12-19T15:55:18.969795+0000","last_active":"2023-12-19T15:55:18.969795+0000","last_peered":"2023-12-19T15:55:18.969795+0000","last_clean":"2023-12-19T15:55:18.969795+0000","last_became_active":"2023-12-19T15:55:18.969498+0000","last_became_peered":"2023-12-19T15:55:18.969498+0000","last_unstale":"2023-12-19T15:55:18.969795+0000","last_undegraded":"2023-12-19T15:55:18.969795+0000","last_fullsized":"2023-12-19T15:55:18.969795+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:59.642359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966471+0000","last_change":"2023-12-19T15:55:17.963651+0000","last_active":"2023-12-19T15:55:17.966471+0000","last_peered":"2023-12-19T15:55:17.966471+0000","last_clean":"2023-12-19T15:55:17.966471+0000","last_became_active":"2023-12-19T15:55:17.963387+0000","last_became_peered":"2023-12-19T15:55:17.963387+0000","last_unstale":"2023-12-19T15:55:17.966471+0000","last_undegraded":"2023-12-19T15:55:17.966471+0000","last_fullsized":"2023-12-19T15:55:17.966471+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:55.325995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.974041+0000","last_change":"2023-12-19T15:55:18.962228+0000","last_active":"2023-12-19T15:55:18.974041+0000","last_peered":"2023-12-19T15:55:18.974041+0000","last_clean":"2023-12-19T15:55:18.974041+0000","last_became_active":"2023-12-19T15:55:18.961498+0000","last_became_peered":"2023-12-19T15:55:18.961498+0000","last_unstale":"2023-12-19T15:55:18.974041+0000","last_undegraded":"2023-12-19T15:55:18.974041+0000","last_fullsized":"2023-12-19T15:55:18.974041+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965509+0000","last_change":"2023-12-19T15:55:17.963589+0000","last_active":"2023-12-19T15:55:17.965509+0000","last_peered":"2023-12-19T15:55:17.965509+0000","last_clean":"2023-12-19T15:55:17.965509+0000","last_became_active":"2023-12-19T15:55:17.963302+0000","last_became_peered":"2023-12-19T15:55:17.963302+0000","last_unstale":"2023-12-19T15:55:17.965509+0000","last_undegraded":"2023-12-19T15:55:17.965509+0000","last_fullsized":"2023-12-19T15:55:17.965509+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:39.950747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976270+0000","last_change":"2023-12-19T15:55:18.971547+0000","last_active":"2023-12-19T15:55:18.976270+0000","last_peered":"2023-12-19T15:55:18.976270+0000","last_clean":"2023-12-19T15:55:18.976270+0000","last_became_active":"2023-12-19T15:55:18.971265+0000","last_became_peered":"2023-12-19T15:55:18.971265+0000","last_unstale":"2023-12-19T15:55:18.976270+0000","last_undegraded":"2023-12-19T15:55:18.976270+0000","last_fullsized":"2023-12-19T15:55:18.976270+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:16.749425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965666+0000","last_change":"2023-12-19T15:55:17.963647+0000","last_active":"2023-12-19T15:55:17.965666+0000","last_peered":"2023-12-19T15:55:17.965666+0000","last_clean":"2023-12-19T15:55:17.965666+0000","last_became_active":"2023-12-19T15:55:17.962696+0000","last_became_peered":"2023-12-19T15:55:17.962696+0000","last_unstale":"2023-12-19T15:55:17.965666+0000","last_undegraded":"2023-12-19T15:55:17.965666+0000","last_fullsized":"2023-12-19T15:55:17.965666+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:20.414653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987877+0000","last_change":"2023-12-19T15:55:18.968133+0000","last_active":"2023-12-19T15:55:18.987877+0000","last_peered":"2023-12-19T15:55:18.987877+0000","last_clean":"2023-12-19T15:55:18.987877+0000","last_became_active":"2023-12-19T15:55:18.967747+0000","last_became_peered":"2023-12-19T15:55:18.967747+0000","last_unstale":"2023-12-19T15:55:18.987877+0000","last_undegraded":"2023-12-19T15:55:18.987877+0000","last_fullsized":"2023-12-19T15:55:18.987877+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:57.725007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.962048+0000","last_change":"2023-12-19T15:55:17.956296+0000","last_active":"2023-12-19T15:55:17.962048+0000","last_peered":"2023-12-19T15:55:17.962048+0000","last_clean":"2023-12-19T15:55:17.962048+0000","last_became_active":"2023-12-19T15:55:17.956031+0000","last_became_peered":"2023-12-19T15:55:17.956031+0000","last_unstale":"2023-12-19T15:55:17.962048+0000","last_undegraded":"2023-12-19T15:55:17.962048+0000","last_fullsized":"2023-12-19T15:55:17.962048+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:35.178870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979779+0000","last_change":"2023-12-19T15:55:18.978074+0000","last_active":"2023-12-19T15:55:18.979779+0000","last_peered":"2023-12-19T15:55:18.979779+0000","last_clean":"2023-12-19T15:55:18.979779+0000","last_became_active":"2023-12-19T15:55:18.977749+0000","last_became_peered":"2023-12-19T15:55:18.977749+0000","last_unstale":"2023-12-19T15:55:18.979779+0000","last_undegraded":"2023-12-19T15:55:18.979779+0000","last_fullsized":"2023-12-19T15:55:18.979779+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:34:22.928812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.969126+0000","last_change":"2023-12-19T15:55:17.962786+0000","last_active":"2023-12-19T15:55:17.969126+0000","last_peered":"2023-12-19T15:55:17.969126+0000","last_clean":"2023-12-19T15:55:17.969126+0000","last_became_active":"2023-12-19T15:55:17.962536+0000","last_became_peered":"2023-12-19T15:55:17.962536+0000","last_unstale":"2023-12-19T15:55:17.969126+0000","last_undegraded":"2023-12-19T15:55:17.969126+0000","last_fullsized":"2023-12-19T15:55:17.969126+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:00.827737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978305+0000","last_change":"2023-12-19T15:55:18.970551+0000","last_active":"2023-12-19T15:55:18.978305+0000","last_peered":"2023-12-19T15:55:18.978305+0000","last_clean":"2023-12-19T15:55:18.978305+0000","last_became_active":"2023-12-19T15:55:18.970118+0000","last_became_peered":"2023-12-19T15:55:18.970118+0000","last_unstale":"2023-12-19T15:55:18.978305+0000","last_undegraded":"2023-12-19T15:55:18.978305+0000","last_fullsized":"2023-12-19T15:55:18.978305+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"180'6","reported_seq":15,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991180+0000","last_change":"2023-12-19T15:55:17.966585+0000","last_active":"2023-12-19T15:55:18.991180+0000","last_peered":"2023-12-19T15:55:18.991180+0000","last_clean":"2023-12-19T15:55:18.991180+0000","last_became_active":"2023-12-19T15:55:17.966274+0000","last_became_peered":"2023-12-19T15:55:17.966274+0000","last_unstale":"2023-12-19T15:55:18.991180+0000","last_undegraded":"2023-12-19T15:55:18.991180+0000","last_fullsized":"2023-12-19T15:55:18.991180+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:36.328245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"169'293","reported_seq":917,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.951381+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T15:55:17.951381+0000","last_peered":"2023-12-19T15:55:17.951381+0000","last_clean":"2023-12-19T15:55:17.951381+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T15:55:17.951381+0000","last_undegraded":"2023-12-19T15:55:17.951381+0000","last_fullsized":"2023-12-19T15:55:17.951381+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575759,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300068,"kb_used_data":2780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071772,"statfs":{"total":96636764160,"available":96329494528,"internally_reserved":0,"allocated":2846720,"data_stored":1732904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1379999999999999,"15min":1.135},"min":{"1min":0.68500000000000005,"5min":0.64800000000000002,"15min":0.54200000000000004},"max":{"1min":1.6160000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.171},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1779999999999999,"15min":1.1579999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":1.361}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.149,"15min":1.1519999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.4330000000000001,"5min":1.7010000000000001,"15min":1.778},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.542,"5min":1.698,"15min":1.698},"last":1.512}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3029999999999999,"15min":1.294},"min":{"1min":1.0369999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.669,"5min":1.6890000000000001,"15min":1.7969999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.345,"15min":1.3360000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.79300000000000004},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7889999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.875,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5960000000000001,"5min":1.6839999999999999,"15min":1.7130000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.355,"15min":1.367},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.5369999999999999,"5min":1.7270000000000001,"15min":1.7290000000000001},"last":1.266}]},{"osd":4,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.163,"15min":1.181},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.63200000000000001},"max":{"1min":1.5820000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.206},{"interface":"front","average":{"1min":1.212,"5min":1.2629999999999999,"15min":1.2609999999999999},"min":{"1min":0.92800000000000005,"5min":0.82099999999999995,"15min":0.63500000000000001},"max":{"1min":1.6180000000000001,"5min":1.8009999999999999,"15min":1.8899999999999999},"last":1.093}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2390000000000001,"15min":1.264},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60499999999999998},"max":{"1min":1.6519999999999999,"5min":1.758,"15min":2.0880000000000001},"last":1.129},{"interface":"front","average":{"1min":1.292,"5min":1.325,"15min":1.325},"min":{"1min":0.92900000000000005,"5min":0.79100000000000004,"15min":0.69999999999999996},"max":{"1min":1.7090000000000001,"5min":1.798,"15min":1.798},"last":1.056}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4450000000000001,"15min":1.423},"min":{"1min":1.1910000000000001,"5min":0.82399999999999995,"15min":0.76000000000000001},"max":{"1min":1.778,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.70999999999999996},"max":{"1min":1.911,"5min":1.911,"15min":2.0110000000000001},"last":1.4199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575759,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1142584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.92900000000000005,"5min":0.79700000000000004,"15min":0.72799999999999998},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.387},{"interface":"front","average":{"1min":1.22,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.99399999999999999,"5min":0.78800000000000003,"15min":0.75900000000000001},"max":{"1min":1.4490000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.214}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1910000000000001,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.52,"5min":1.617,"15min":1.6890000000000001},"last":1.234},{"interface":"front","average":{"1min":1.228,"5min":1.2549999999999999,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.5269999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.4330000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.2490000000000001,"15min":1.242},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.83699999999999997},"max":{"1min":1.611,"5min":1.629,"15min":1.6990000000000001},"last":1.734},{"interface":"front","average":{"1min":1.262,"5min":1.276,"15min":1.268},"min":{"1min":0.96399999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.7130000000000001},"last":1.0349999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.647,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":1.115},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3600000000000001,"15min":1.3440000000000001},"min":{"1min":0.98699999999999999,"5min":0.96299999999999997,"15min":0.93300000000000005},"max":{"1min":1.6220000000000001,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.6559999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2230000000000001,"15min":1.2030000000000001},"min":{"1min":0.86499999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":0.89000000000000001,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":1.8049999999999999,"5min":2.653,"15min":2.653},"last":1.5229999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.3},"min":{"1min":0.84899999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.486},{"interface":"front","average":{"1min":1.258,"5min":1.3320000000000001,"15min":1.361},"min":{"1min":0.878,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.762,"5min":1.871,"15min":1.99},"last":1.1499999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4039999999999999,"15min":1.4299999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.742,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":1.4490000000000001},"min":{"1min":0.97399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.905,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":1.8069999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575759,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307676,"kb_used_data":2196,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064164,"statfs":{"total":96636764160,"available":96321703936,"internally_reserved":0,"allocated":2248704,"data_stored":1142514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":2.2759999999999998},"last":1.304},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6679999999999999,"5min":1.7909999999999999,"15min":2.7309999999999999},"last":0.91200000000000003}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.339,"15min":1.3220000000000001},"min":{"1min":1,"5min":0.86599999999999999,"15min":0.84499999999999997},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7370000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.282,"15min":1.2689999999999999},"min":{"1min":1.038,"5min":0.95699999999999996,"15min":0.82599999999999996},"max":{"1min":1.617,"5min":1.645,"15min":1.8779999999999999},"last":1.2}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6699999999999999,"5min":1.748,"15min":2.8239999999999998},"last":1.47},{"interface":"front","average":{"1min":1.234,"5min":1.2350000000000001,"15min":1.2230000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71199999999999997},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":1.871},"last":0.871}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.306},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.631,"5min":1.7350000000000001,"15min":2.4849999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.2969999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6739999999999999,"5min":1.7929999999999999,"15min":1.9390000000000001},"last":0.96699999999999997}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1359999999999999,"15min":1.137},"min":{"1min":0.84299999999999997,"5min":0.69499999999999995,"15min":0.59999999999999998},"max":{"1min":1.659,"5min":1.659,"15min":2.109},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.244,"15min":1.27},"min":{"1min":0.86799999999999999,"5min":0.81799999999999995,"15min":0.56999999999999995},"max":{"1min":1.7869999999999999,"5min":1.8320000000000001,"15min":2.8999999999999999},"last":1.1140000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.333,"15min":1.3380000000000001},"min":{"1min":0.92100000000000004,"5min":0.84399999999999997,"15min":0.58899999999999997},"max":{"1min":1.8089999999999999,"5min":1.9299999999999999,"15min":2.0649999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":1.1259999999999999,"5min":0.92500000000000004,"15min":0.83699999999999997},"max":{"1min":1.998,"5min":1.998,"15min":2.117},"last":1.3720000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4259999999999999,"15min":1.4179999999999999},"min":{"1min":1.2310000000000001,"5min":0.95499999999999996,"15min":0.623},"max":{"1min":1.915,"5min":1.915,"15min":2.6000000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.419,"15min":1.411},"min":{"1min":1.0389999999999999,"5min":0.84599999999999997,"15min":0.68000000000000005},"max":{"1min":1.879,"5min":1.9870000000000001,"15min":2.3119999999999998},"last":1.228}]}]},{"osd":4,"up_from":13,"seq":55834575759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1144050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1299999999999999,"15min":1.1100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.722},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.113,"15min":1.1399999999999999},"min":{"1min":0.76800000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.4770000000000001,"5min":1.5249999999999999,"15min":1.75},"last":1.1850000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.274,"15min":1.286},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6020000000000001,"5min":1.6819999999999999,"15min":2.2530000000000001},"last":1.232},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.69399999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.968},"last":1.5900000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1850000000000001,"15min":1.1859999999999999},"min":{"1min":0.754,"5min":0.73599999999999999,"15min":0.65300000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":2.3410000000000002},"last":1.339},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.84399999999999997,"5min":0.79900000000000004,"15min":0.57799999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1459999999999999},"last":1.2889999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.236,"15min":1.23},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.772},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.26,"15min":1.2869999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.65500000000000003},"max":{"1min":1.758,"5min":1.758,"15min":2.0800000000000001},"last":1.843}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.204,"15min":1.1839999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.42399999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.6240000000000001,"5min":1.6579999999999999,"15min":1.998},"last":1.5349999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3140000000000001,"15min":1.306},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.60399999999999998},"max":{"1min":1.7190000000000001,"5min":1.766,"15min":2.3340000000000001},"last":1.71},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.458,"15min":1.454},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.68100000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.675}]},{"osd":7,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3859999999999999,"15min":1.3600000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8260000000000001,"5min":1.839,"15min":2.0150000000000001},"last":1.617},{"interface":"front","average":{"1min":1.486,"5min":1.4890000000000001,"15min":1.47},"min":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.55600000000000005},"max":{"1min":1.8660000000000001,"5min":2.157,"15min":2.1659999999999999},"last":1.944}]}]},{"osd":3,"up_from":13,"seq":55834575758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315876,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055964,"statfs":{"total":96636764160,"available":96313307136,"internally_reserved":0,"allocated":2256896,"data_stored":1143102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94899999999999995,"15min":0.91600000000000004},"min":{"1min":0.57999999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.61099999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":2.113},"last":0.72199999999999998}]},{"osd":1,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57699999999999996},"max":{"1min":1.2250000000000001,"5min":1.5680000000000001,"15min":1.75},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1459999999999999,"15min":1.1200000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.53900000000000003},"max":{"1min":1.4950000000000001,"5min":1.706,"15min":1.8149999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1699999999999999,"15min":1.1659999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.992},"last":1.302},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.294,"15min":1.264},"min":{"1min":0.871,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.347,"15min":1.343},"min":{"1min":1.0089999999999999,"5min":0.94099999999999995,"15min":0.879},"max":{"1min":1.623,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.052},{"interface":"front","average":{"1min":1.226,"5min":1.3120000000000001,"15min":1.321},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1120000000000001},"last":1.2290000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3360000000000001,"15min":1.3440000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.79200000000000004},"max":{"1min":1.7589999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.444,"15min":1.4239999999999999},"min":{"1min":1.1200000000000001,"5min":0.84099999999999997,"15min":0.81000000000000005},"max":{"1min":1.8560000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.1799999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.508,"15min":1.482},"min":{"1min":1.1819999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.073,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4770000000000001,"15min":1.4630000000000001},"min":{"1min":0.94499999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1190000000000002,"5min":2.141,"15min":2.1909999999999998},"last":1.492}]},{"osd":7,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.554,"15min":1.5569999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.0880000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.498,"15min":1.5089999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0310000000000001,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.55}]}]},{"osd":2,"up_from":13,"seq":55834575760,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1141615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.94699999999999995,"15min":0.95499999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.446,"5min":1.587,"15min":1.587},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.1399999999999999,"15min":1.141},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.944},"last":0.82499999999999996}]},{"osd":1,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1339999999999999,"15min":1.1739999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.526,"5min":1.6060000000000001,"15min":1.6870000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.097,"15min":1.139},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4470000000000001,"5min":1.6160000000000001,"15min":1.7430000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.014,"15min":1.0249999999999999},"min":{"1min":0.65700000000000003,"5min":0.64900000000000002,"15min":0.64000000000000001},"max":{"1min":1.5009999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.999,"5min":1.105,"15min":1.123},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.3939999999999999,"5min":1.5149999999999999,"15min":1.744},"last":0.80300000000000005}]},{"osd":4,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3380000000000001,"15min":1.4019999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.825,"5min":1.825,"15min":2.0790000000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.4019999999999999,"15min":1.446},"min":{"1min":0.88300000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.778,"5min":1.921,"15min":2.3410000000000002},"last":0.98399999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3500000000000001,"15min":1.383},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0659999999999998},"last":1.274},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3700000000000001,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.875,"5min":1.996,"15min":2.016},"last":1.155}]},{"osd":6,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3660000000000001,"15min":1.391},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.651,"5min":1.782,"15min":1.9750000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4850000000000001,"15min":1.5029999999999999},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.8899999999999999,"5min":2.0489999999999999,"15min":2.242},"last":1.4039999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5009999999999999,"15min":1.5489999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.8149999999999999,"5min":1.9750000000000001,"15min":2.2389999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.478,"15min":1.508},"min":{"1min":0.871,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8560000000000001,"5min":2.016,"15min":2.016},"last":1.3859999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1142584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.89800000000000002,"15min":0.88800000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.389,"5min":1.4890000000000001,"15min":1.6579999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0029999999999999,"15min":1.024},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3320000000000001,"5min":1.532,"15min":1.6879999999999999},"last":1.256}]},{"osd":2,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.036,"15min":1.0649999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.5069999999999999,"5min":1.6000000000000001,"15min":1.7989999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1379999999999999,"15min":1.163},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.421,"5min":1.5900000000000001,"15min":1.7549999999999999},"last":1.605}]},{"osd":3,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.282,"15min":1.262},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.79,"5min":1.79,"15min":1.851},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.087,"15min":1.099},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6659999999999999,"5min":1.778,"15min":1.849},"last":0.86099999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.325,"15min":1.3260000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.992,"5min":1.992,"15min":2.0739999999999998},"last":1.514},{"interface":"front","average":{"1min":1.397,"5min":1.407,"15min":1.4239999999999999},"min":{"1min":1.1060000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.921},"last":1.4810000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.359,"15min":1.3460000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.421},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.425,"15min":1.4259999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1909999999999998},"last":1.1919999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4059999999999999,"15min":1.399},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.5},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.1040000000000001},"last":1.667}]},{"osd":7,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.494,"15min":1.5209999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.0699999999999998,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4610000000000001,"15min":1.48},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.028,"5min":2.0659999999999998,"15min":2.0699999999999998},"last":1.5549999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575758,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300084,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071756,"statfs":{"total":96636764160,"available":96329478144,"internally_reserved":0,"allocated":2863104,"data_stored":1734085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.66500000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.78000000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5649999999999999,"15min":1.895},"last":0.88400000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1259999999999999,"15min":1.1439999999999999},"min":{"1min":0.85599999999999998,"5min":0.502,"15min":0.47399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":2.5129999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.256,"15min":1.2789999999999999},"min":{"1min":0.93600000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.679,"5min":1.708,"15min":2.8250000000000002},"last":1.2190000000000001}]},{"osd":3,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.038},"min":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.66200000000000003},"max":{"1min":1.5349999999999999,"5min":1.599,"15min":1.9970000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2070000000000001,"15min":1.2090000000000001},"min":{"1min":0.92200000000000004,"5min":0.621,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.794},"last":1.246}]},{"osd":4,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3919999999999999,"15min":1.351},"min":{"1min":1.0640000000000001,"5min":0.94499999999999995,"15min":0.80000000000000004},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.321},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.327,"15min":1.3169999999999999},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8280000000000001,"5min":1.9059999999999999,"15min":2.2210000000000001},"last":1.2809999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.419,"15min":1.4279999999999999},"min":{"1min":1.175,"5min":0.871,"15min":0.73099999999999998},"max":{"1min":1.879,"5min":1.901,"15min":2.4470000000000001},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4670000000000001,"15min":1.4359999999999999},"min":{"1min":1.004,"5min":1.004,"15min":0.79400000000000004},"max":{"1min":1.889,"5min":1.9510000000000001,"15min":2.468},"last":1.381}]},{"osd":6,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.413,"15min":1.385},"min":{"1min":1.1830000000000001,"5min":0.90000000000000002,"15min":0.88500000000000001},"max":{"1min":1.925,"5min":1.925,"15min":2.0830000000000002},"last":1.484},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.53,"15min":1.5},"min":{"1min":1.2110000000000001,"5min":0.92600000000000005,"15min":0.81799999999999995},"max":{"1min":1.9470000000000001,"5min":2.379,"15min":2.379},"last":1.474}]},{"osd":7,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5429999999999999,"15min":1.5049999999999999},"min":{"1min":1.0900000000000001,"5min":0.99299999999999999,"15min":0.90900000000000003},"max":{"1min":1.968,"5min":2.105,"15min":2.698},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.5089999999999999,"15min":1.482},"min":{"1min":1.022,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.8380000000000001,"5min":2.016,"15min":3.23},"last":1.3919999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T15:55:23.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T15:55:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 Processor -- start 2023-12-19T15:55:23.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 -- start start 2023-12-19T15:55:23.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 0x7fc74c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74c0c7190 con 0x7fc74c15d320 2023-12-19T15:55:23.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74c0c1b50 con 0x7fc74c0c15b0 2023-12-19T15:55:23.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc7511c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c0c1e30 con 0x7fc74c068800 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc74ad9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc74c15d320 0x7fc74c06d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47390/0 (socket says 172.21.15.50:47390) 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc74ad9d700 1 -- 172.21.15.50:0/2922719735 learned_addr learned my addr 172.21.15.50:0/2922719735 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.491+0000 7fc74b59e700 1 --2- 172.21.15.50:0/2922719735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 0x7fc74c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336869534 0 0) 0x7fc74c0c7190 con 0x7fc74c15d320 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc738003570 con 0x7fc74c15d320 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74b59e700 1 -- 172.21.15.50:0/2922719735 >> v1:172.21.15.125:6789/0 conn(0x7fc74c0c15b0 legacy=0x7fc74c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74b59e700 1 -- 172.21.15.50:0/2922719735 >> v1:172.21.15.50:6789/0 conn(0x7fc74c15d320 legacy=0x7fc74c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163868298 0 0) 0x7fc74c0c1b50 con 0x7fc74c0c15b0 2023-12-19T15:55:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74b59e700 1 -- 172.21.15.50:0/2922719735 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c0c6e20 con 0x7fc74c068800 2023-12-19T15:55:23.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74b59e700 1 --2- 172.21.15.50:0/2922719735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 0x7fc74c15d6f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc744009e90 tx=0x7fc744009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303de11f1c16a4be server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74400e070 con 0x7fc74c068800 2023-12-19T15:55:23.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74400bc30 con 0x7fc74c068800 2023-12-19T15:55:23.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc74a59c700 1 -- 172.21.15.50:0/2922719735 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744012430 con 0x7fc74c068800 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 -- 172.21.15.50:0/2922719735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 msgr2=0x7fc74c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 --2- 172.21.15.50:0/2922719735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 0x7fc74c15d6f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc744009e90 tx=0x7fc744009780 comp rx=0 tx=0).stop 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 -- 172.21.15.50:0/2922719735 shutdown_connections 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 --2- 172.21.15.50:0/2922719735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c068800 0x7fc74c15d6f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 -- 172.21.15.50:0/2922719735 >> 172.21.15.50:0/2922719735 conn(0x7fc74c0c8850 msgr2=0x7fc74c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:23.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 -- 172.21.15.50:0/2922719735 shutdown_connections 2023-12-19T15:55:23.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.495+0000 7fc7511c5700 1 -- 172.21.15.50:0/2922719735 wait complete. 2023-12-19T15:55:23.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 Processor -- start 2023-12-19T15:55:23.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 -- start start 2023-12-19T15:55:23.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc74c068800 0x7fc74c154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c0c15b0 0x7fc74c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c159600 con 0x7fc74c15d320 2023-12-19T15:55:23.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c154c70 con 0x7fc74c068800 2023-12-19T15:55:23.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc7511c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c0af970 con 0x7fc74c0c15b0 2023-12-19T15:55:23.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35868/0 (socket says 172.21.15.50:35868) 2023-12-19T15:55:23.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 -- 172.21.15.50:0/3734041535 learned_addr learned my addr 172.21.15.50:0/3734041535 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:23.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74ad9d700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc74c068800 0x7fc74c154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc743fff700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c0c15b0 0x7fc74c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 -- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c0c15b0 msgr2=0x7fc74c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c0c15b0 0x7fc74c0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 -- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc74c068800 msgr2=0x7fc74c154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc74c068800 0x7fc74c154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.499+0000 7fc74b59e700 1 -- 172.21.15.50:0/3734041535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc744009a20 con 0x7fc74c15d320 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc74b59e700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc74c06c260 tx=0x7fc744019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74401b420 con 0x7fc74c15d320 2023-12-19T15:55:23.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74401b580 con 0x7fc74c15d320 2023-12-19T15:55:23.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c0afc50 con 0x7fc74c15d320 2023-12-19T15:55:23.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74401b940 con 0x7fc74c15d320 2023-12-19T15:55:23.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c0b0210 con 0x7fc74c15d320 2023-12-19T15:55:23.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c071120 con 0x7fc74c15d320 2023-12-19T15:55:23.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.503+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc74401bd30 con 0x7fc74c15d320 2023-12-19T15:55:23.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.507+0000 7fc742ffd700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 0x7fc72c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.507+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fc74407eb40 con 0x7fc74c15d320 2023-12-19T15:55:23.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.507+0000 7fc74ad9d700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 0x7fc72c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.507+0000 7fc74ad9d700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 0x7fc72c062bf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007fd0 tx=0x7fc73c009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.515+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc74404ed70 con 0x7fc74c15d320 2023-12-19T15:55:23.687 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.683+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc74c0642a0 con 0x7fc72c060740 2023-12-19T15:55:23.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.699+0000 7fc742ffd700 1 -- 172.21.15.50:0/3734041535 <== mgr.8968 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287443 (secure 0 0 0) 0x7fc74c0642a0 con 0x7fc72c060740 2023-12-19T15:55:23.710 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 msgr2=0x7fc72c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 0x7fc72c062bf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007fd0 tx=0x7fc73c009040 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 msgr2=0x7fc74c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc74c06c260 tx=0x7fc744019000 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 shutdown_connections 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fc72c060740 0x7fc72c062bf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc74c15d320 0x7fc74c0cae00 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc74c0c15b0 0x7fc74c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 --2- 172.21.15.50:0/3734041535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc74c068800 0x7fc74c154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 >> 172.21.15.50:0/3734041535 conn(0x7fc74c0c8850 msgr2=0x7fc74c1575c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 shutdown_connections 2023-12-19T15:55:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.707+0000 7fc7511c5700 1 -- 172.21.15.50:0/3734041535 wait complete. 2023-12-19T15:55:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T15:55:23.732 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T15:55:22.300905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466720,"kb_used_data":18784,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752508000,"statfs":{"total":773094113280,"available":770568192000,"internally_reserved":0,"allocated":19234816,"data_stored":10323438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.387839"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975815+0000","last_change":"2023-12-19T15:55:18.975815+0000","last_active":"2023-12-19T15:55:18.975815+0000","last_peered":"2023-12-19T15:55:18.975815+0000","last_clean":"2023-12-19T15:55:18.975815+0000","last_became_active":"2023-12-19T15:55:18.975523+0000","last_became_peered":"2023-12-19T15:55:18.975523+0000","last_unstale":"2023-12-19T15:55:18.975815+0000","last_undegraded":"2023-12-19T15:55:18.975815+0000","last_fullsized":"2023-12-19T15:55:18.975815+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:33.198276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987936+0000","last_change":"2023-12-19T15:55:18.973233+0000","last_active":"2023-12-19T15:55:18.987936+0000","last_peered":"2023-12-19T15:55:18.987936+0000","last_clean":"2023-12-19T15:55:18.987936+0000","last_became_active":"2023-12-19T15:55:18.972935+0000","last_became_peered":"2023-12-19T15:55:18.972935+0000","last_unstale":"2023-12-19T15:55:18.987936+0000","last_undegraded":"2023-12-19T15:55:18.987936+0000","last_fullsized":"2023-12-19T15:55:18.987936+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:57.381319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987546+0000","last_change":"2023-12-19T15:55:18.971844+0000","last_active":"2023-12-19T15:55:18.987546+0000","last_peered":"2023-12-19T15:55:18.987546+0000","last_clean":"2023-12-19T15:55:18.987546+0000","last_became_active":"2023-12-19T15:55:18.971562+0000","last_became_peered":"2023-12-19T15:55:18.971562+0000","last_unstale":"2023-12-19T15:55:18.987546+0000","last_undegraded":"2023-12-19T15:55:18.987546+0000","last_fullsized":"2023-12-19T15:55:18.987546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:14:27.846499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971253+0000","last_change":"2023-12-19T15:55:18.967476+0000","last_active":"2023-12-19T15:55:18.971253+0000","last_peered":"2023-12-19T15:55:18.971253+0000","last_clean":"2023-12-19T15:55:18.971253+0000","last_became_active":"2023-12-19T15:55:18.967139+0000","last_became_peered":"2023-12-19T15:55:18.967139+0000","last_unstale":"2023-12-19T15:55:18.971253+0000","last_undegraded":"2023-12-19T15:55:18.971253+0000","last_fullsized":"2023-12-19T15:55:18.971253+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964997+0000","last_change":"2023-12-19T15:55:18.964997+0000","last_active":"2023-12-19T15:55:18.964997+0000","last_peered":"2023-12-19T15:55:18.964997+0000","last_clean":"2023-12-19T15:55:18.964997+0000","last_became_active":"2023-12-19T15:55:18.962616+0000","last_became_peered":"2023-12-19T15:55:18.962616+0000","last_unstale":"2023-12-19T15:55:18.964997+0000","last_undegraded":"2023-12-19T15:55:18.964997+0000","last_fullsized":"2023-12-19T15:55:18.964997+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:44.409742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961745+0000","last_change":"2023-12-19T15:55:18.961745+0000","last_active":"2023-12-19T15:55:18.961745+0000","last_peered":"2023-12-19T15:55:18.961745+0000","last_clean":"2023-12-19T15:55:18.961745+0000","last_became_active":"2023-12-19T15:55:18.961414+0000","last_became_peered":"2023-12-19T15:55:18.961414+0000","last_unstale":"2023-12-19T15:55:18.961745+0000","last_undegraded":"2023-12-19T15:55:18.961745+0000","last_fullsized":"2023-12-19T15:55:18.961745+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:59:44.887035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.973289+0000","last_change":"2023-12-19T15:55:18.973289+0000","last_active":"2023-12-19T15:55:18.973289+0000","last_peered":"2023-12-19T15:55:18.973289+0000","last_clean":"2023-12-19T15:55:18.973289+0000","last_became_active":"2023-12-19T15:55:18.970736+0000","last_became_peered":"2023-12-19T15:55:18.970736+0000","last_unstale":"2023-12-19T15:55:18.973289+0000","last_undegraded":"2023-12-19T15:55:18.973289+0000","last_fullsized":"2023-12-19T15:55:18.973289+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:00:52.002939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978705+0000","last_change":"2023-12-19T15:55:18.967058+0000","last_active":"2023-12-19T15:55:18.978705+0000","last_peered":"2023-12-19T15:55:18.978705+0000","last_clean":"2023-12-19T15:55:18.978705+0000","last_became_active":"2023-12-19T15:55:18.966753+0000","last_became_peered":"2023-12-19T15:55:18.966753+0000","last_unstale":"2023-12-19T15:55:18.978705+0000","last_undegraded":"2023-12-19T15:55:18.978705+0000","last_fullsized":"2023-12-19T15:55:18.978705+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:24:24.052412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975035+0000","last_change":"2023-12-19T15:55:18.975035+0000","last_active":"2023-12-19T15:55:18.975035+0000","last_peered":"2023-12-19T15:55:18.975035+0000","last_clean":"2023-12-19T15:55:18.975035+0000","last_became_active":"2023-12-19T15:55:18.974740+0000","last_became_peered":"2023-12-19T15:55:18.974740+0000","last_unstale":"2023-12-19T15:55:18.975035+0000","last_undegraded":"2023-12-19T15:55:18.975035+0000","last_fullsized":"2023-12-19T15:55:18.975035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964702+0000","last_change":"2023-12-19T15:55:18.964702+0000","last_active":"2023-12-19T15:55:18.964702+0000","last_peered":"2023-12-19T15:55:18.964702+0000","last_clean":"2023-12-19T15:55:18.964702+0000","last_became_active":"2023-12-19T15:55:18.964195+0000","last_became_peered":"2023-12-19T15:55:18.964195+0000","last_unstale":"2023-12-19T15:55:18.964702+0000","last_undegraded":"2023-12-19T15:55:18.964702+0000","last_fullsized":"2023-12-19T15:55:18.964702+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971766+0000","last_change":"2023-12-19T15:55:18.971766+0000","last_active":"2023-12-19T15:55:18.971766+0000","last_peered":"2023-12-19T15:55:18.971766+0000","last_clean":"2023-12-19T15:55:18.971766+0000","last_became_active":"2023-12-19T15:55:18.971240+0000","last_became_peered":"2023-12-19T15:55:18.971240+0000","last_unstale":"2023-12-19T15:55:18.971766+0000","last_undegraded":"2023-12-19T15:55:18.971766+0000","last_fullsized":"2023-12-19T15:55:18.971766+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:41.764312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971345+0000","last_change":"2023-12-19T15:55:18.969330+0000","last_active":"2023-12-19T15:55:18.971345+0000","last_peered":"2023-12-19T15:55:18.971345+0000","last_clean":"2023-12-19T15:55:18.971345+0000","last_became_active":"2023-12-19T15:55:18.968887+0000","last_became_peered":"2023-12-19T15:55:18.968887+0000","last_unstale":"2023-12-19T15:55:18.971345+0000","last_undegraded":"2023-12-19T15:55:18.971345+0000","last_fullsized":"2023-12-19T15:55:18.971345+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:35.872104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964965+0000","last_change":"2023-12-19T15:55:18.964965+0000","last_active":"2023-12-19T15:55:18.964965+0000","last_peered":"2023-12-19T15:55:18.964965+0000","last_clean":"2023-12-19T15:55:18.964965+0000","last_became_active":"2023-12-19T15:55:18.963532+0000","last_became_peered":"2023-12-19T15:55:18.963532+0000","last_unstale":"2023-12-19T15:55:18.964965+0000","last_undegraded":"2023-12-19T15:55:18.964965+0000","last_fullsized":"2023-12-19T15:55:18.964965+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:30.795790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986219+0000","last_change":"2023-12-19T15:55:18.982714+0000","last_active":"2023-12-19T15:55:18.986219+0000","last_peered":"2023-12-19T15:55:18.986219+0000","last_clean":"2023-12-19T15:55:18.986219+0000","last_became_active":"2023-12-19T15:55:18.982437+0000","last_became_peered":"2023-12-19T15:55:18.982437+0000","last_unstale":"2023-12-19T15:55:18.986219+0000","last_undegraded":"2023-12-19T15:55:18.986219+0000","last_fullsized":"2023-12-19T15:55:18.986219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987482+0000","last_change":"2023-12-19T15:55:18.970114+0000","last_active":"2023-12-19T15:55:18.987482+0000","last_peered":"2023-12-19T15:55:18.987482+0000","last_clean":"2023-12-19T15:55:18.987482+0000","last_became_active":"2023-12-19T15:55:18.969643+0000","last_became_peered":"2023-12-19T15:55:18.969643+0000","last_unstale":"2023-12-19T15:55:18.987482+0000","last_undegraded":"2023-12-19T15:55:18.987482+0000","last_fullsized":"2023-12-19T15:55:18.987482+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:56.008506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.966217+0000","last_change":"2023-12-19T15:55:18.966217+0000","last_active":"2023-12-19T15:55:18.966217+0000","last_peered":"2023-12-19T15:55:18.966217+0000","last_clean":"2023-12-19T15:55:18.966217+0000","last_became_active":"2023-12-19T15:55:18.965901+0000","last_became_peered":"2023-12-19T15:55:18.965901+0000","last_unstale":"2023-12-19T15:55:18.966217+0000","last_undegraded":"2023-12-19T15:55:18.966217+0000","last_fullsized":"2023-12-19T15:55:18.966217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971331+0000","last_change":"2023-12-19T15:55:18.966751+0000","last_active":"2023-12-19T15:55:18.971331+0000","last_peered":"2023-12-19T15:55:18.971331+0000","last_clean":"2023-12-19T15:55:18.971331+0000","last_became_active":"2023-12-19T15:55:18.965992+0000","last_became_peered":"2023-12-19T15:55:18.965992+0000","last_unstale":"2023-12-19T15:55:18.971331+0000","last_undegraded":"2023-12-19T15:55:18.971331+0000","last_fullsized":"2023-12-19T15:55:18.971331+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:03:30.795790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.963485+0000","last_change":"2023-12-19T15:55:18.963485+0000","last_active":"2023-12-19T15:55:18.963485+0000","last_peered":"2023-12-19T15:55:18.963485+0000","last_clean":"2023-12-19T15:55:18.963485+0000","last_became_active":"2023-12-19T15:55:18.963199+0000","last_became_peered":"2023-12-19T15:55:18.963199+0000","last_unstale":"2023-12-19T15:55:18.963485+0000","last_undegraded":"2023-12-19T15:55:18.963485+0000","last_fullsized":"2023-12-19T15:55:18.963485+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:21:51.204111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979601+0000","last_change":"2023-12-19T15:55:18.979601+0000","last_active":"2023-12-19T15:55:18.979601+0000","last_peered":"2023-12-19T15:55:18.979601+0000","last_clean":"2023-12-19T15:55:18.979601+0000","last_became_active":"2023-12-19T15:55:18.978847+0000","last_became_peered":"2023-12-19T15:55:18.978847+0000","last_unstale":"2023-12-19T15:55:18.979601+0000","last_undegraded":"2023-12-19T15:55:18.979601+0000","last_fullsized":"2023-12-19T15:55:18.979601+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:56:45.625418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.974894+0000","last_change":"2023-12-19T15:55:18.974894+0000","last_active":"2023-12-19T15:55:18.974894+0000","last_peered":"2023-12-19T15:55:18.974894+0000","last_clean":"2023-12-19T15:55:18.974894+0000","last_became_active":"2023-12-19T15:55:18.974601+0000","last_became_peered":"2023-12-19T15:55:18.974601+0000","last_unstale":"2023-12-19T15:55:18.974894+0000","last_undegraded":"2023-12-19T15:55:18.974894+0000","last_fullsized":"2023-12-19T15:55:18.974894+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:33:59.338948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971195+0000","last_change":"2023-12-19T15:55:18.971195+0000","last_active":"2023-12-19T15:55:18.971195+0000","last_peered":"2023-12-19T15:55:18.971195+0000","last_clean":"2023-12-19T15:55:18.971195+0000","last_became_active":"2023-12-19T15:55:18.970757+0000","last_became_peered":"2023-12-19T15:55:18.970757+0000","last_unstale":"2023-12-19T15:55:18.971195+0000","last_undegraded":"2023-12-19T15:55:18.971195+0000","last_fullsized":"2023-12-19T15:55:18.971195+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:16.749425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965513+0000","last_change":"2023-12-19T15:55:18.965513+0000","last_active":"2023-12-19T15:55:18.965513+0000","last_peered":"2023-12-19T15:55:18.965513+0000","last_clean":"2023-12-19T15:55:18.965513+0000","last_became_active":"2023-12-19T15:55:18.965083+0000","last_became_peered":"2023-12-19T15:55:18.965083+0000","last_unstale":"2023-12-19T15:55:18.965513+0000","last_undegraded":"2023-12-19T15:55:18.965513+0000","last_fullsized":"2023-12-19T15:55:18.965513+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:35:20.954312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.973862+0000","last_change":"2023-12-19T15:55:18.964139+0000","last_active":"2023-12-19T15:55:18.973862+0000","last_peered":"2023-12-19T15:55:18.973862+0000","last_clean":"2023-12-19T15:55:18.973862+0000","last_became_active":"2023-12-19T15:55:18.963096+0000","last_became_peered":"2023-12-19T15:55:18.963096+0000","last_unstale":"2023-12-19T15:55:18.973862+0000","last_undegraded":"2023-12-19T15:55:18.973862+0000","last_fullsized":"2023-12-19T15:55:18.973862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:59:44.887035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965510+0000","last_change":"2023-12-19T15:55:18.965510+0000","last_active":"2023-12-19T15:55:18.965510+0000","last_peered":"2023-12-19T15:55:18.965510+0000","last_clean":"2023-12-19T15:55:18.965510+0000","last_became_active":"2023-12-19T15:55:18.963128+0000","last_became_peered":"2023-12-19T15:55:18.963128+0000","last_unstale":"2023-12-19T15:55:18.965510+0000","last_undegraded":"2023-12-19T15:55:18.965510+0000","last_fullsized":"2023-12-19T15:55:18.965510+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:38:56.734422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961827+0000","last_change":"2023-12-19T15:55:18.961827+0000","last_active":"2023-12-19T15:55:18.961827+0000","last_peered":"2023-12-19T15:55:18.961827+0000","last_clean":"2023-12-19T15:55:18.961827+0000","last_became_active":"2023-12-19T15:55:18.961550+0000","last_became_peered":"2023-12-19T15:55:18.961550+0000","last_unstale":"2023-12-19T15:55:18.961827+0000","last_undegraded":"2023-12-19T15:55:18.961827+0000","last_fullsized":"2023-12-19T15:55:18.961827+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:16.293313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964686+0000","last_change":"2023-12-19T15:55:18.964686+0000","last_active":"2023-12-19T15:55:18.964686+0000","last_peered":"2023-12-19T15:55:18.964686+0000","last_clean":"2023-12-19T15:55:18.964686+0000","last_became_active":"2023-12-19T15:55:18.964271+0000","last_became_peered":"2023-12-19T15:55:18.964271+0000","last_unstale":"2023-12-19T15:55:18.964686+0000","last_undegraded":"2023-12-19T15:55:18.964686+0000","last_fullsized":"2023-12-19T15:55:18.964686+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:34:34.746065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969473+0000","last_change":"2023-12-19T15:55:18.969473+0000","last_active":"2023-12-19T15:55:18.969473+0000","last_peered":"2023-12-19T15:55:18.969473+0000","last_clean":"2023-12-19T15:55:18.969473+0000","last_became_active":"2023-12-19T15:55:18.968173+0000","last_became_peered":"2023-12-19T15:55:18.968173+0000","last_unstale":"2023-12-19T15:55:18.969473+0000","last_undegraded":"2023-12-19T15:55:18.969473+0000","last_fullsized":"2023-12-19T15:55:18.969473+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:04.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.962359+0000","last_change":"2023-12-19T15:55:18.962359+0000","last_active":"2023-12-19T15:55:18.962359+0000","last_peered":"2023-12-19T15:55:18.962359+0000","last_clean":"2023-12-19T15:55:18.962359+0000","last_became_active":"2023-12-19T15:55:18.962035+0000","last_became_peered":"2023-12-19T15:55:18.962035+0000","last_unstale":"2023-12-19T15:55:18.962359+0000","last_undegraded":"2023-12-19T15:55:18.962359+0000","last_fullsized":"2023-12-19T15:55:18.962359+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:38.089112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978176+0000","last_change":"2023-12-19T15:55:18.971010+0000","last_active":"2023-12-19T15:55:18.978176+0000","last_peered":"2023-12-19T15:55:18.978176+0000","last_clean":"2023-12-19T15:55:18.978176+0000","last_became_active":"2023-12-19T15:55:18.970723+0000","last_became_peered":"2023-12-19T15:55:18.970723+0000","last_unstale":"2023-12-19T15:55:18.978176+0000","last_undegraded":"2023-12-19T15:55:18.978176+0000","last_fullsized":"2023-12-19T15:55:18.978176+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:48.581415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.968469+0000","last_change":"2023-12-19T15:55:18.968469+0000","last_active":"2023-12-19T15:55:18.968469+0000","last_peered":"2023-12-19T15:55:18.968469+0000","last_clean":"2023-12-19T15:55:18.968469+0000","last_became_active":"2023-12-19T15:55:18.968184+0000","last_became_peered":"2023-12-19T15:55:18.968184+0000","last_unstale":"2023-12-19T15:55:18.968469+0000","last_undegraded":"2023-12-19T15:55:18.968469+0000","last_fullsized":"2023-12-19T15:55:18.968469+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:20.998952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975867+0000","last_change":"2023-12-19T15:55:18.975867+0000","last_active":"2023-12-19T15:55:18.975867+0000","last_peered":"2023-12-19T15:55:18.975867+0000","last_clean":"2023-12-19T15:55:18.975867+0000","last_became_active":"2023-12-19T15:55:18.975538+0000","last_became_peered":"2023-12-19T15:55:18.975538+0000","last_unstale":"2023-12-19T15:55:18.975867+0000","last_undegraded":"2023-12-19T15:55:18.975867+0000","last_fullsized":"2023-12-19T15:55:18.975867+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978741+0000","last_change":"2023-12-19T15:55:18.974796+0000","last_active":"2023-12-19T15:55:18.978741+0000","last_peered":"2023-12-19T15:55:18.978741+0000","last_clean":"2023-12-19T15:55:18.978741+0000","last_became_active":"2023-12-19T15:55:18.974485+0000","last_became_peered":"2023-12-19T15:55:18.974485+0000","last_unstale":"2023-12-19T15:55:18.978741+0000","last_undegraded":"2023-12-19T15:55:18.978741+0000","last_fullsized":"2023-12-19T15:55:18.978741+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987706+0000","last_change":"2023-12-19T15:55:18.970875+0000","last_active":"2023-12-19T15:55:18.987706+0000","last_peered":"2023-12-19T15:55:18.987706+0000","last_clean":"2023-12-19T15:55:18.987706+0000","last_became_active":"2023-12-19T15:55:18.970533+0000","last_became_peered":"2023-12-19T15:55:18.970533+0000","last_unstale":"2023-12-19T15:55:18.987706+0000","last_undegraded":"2023-12-19T15:55:18.987706+0000","last_fullsized":"2023-12-19T15:55:18.987706+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:31.268476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"180'4","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.989726+0000","last_change":"2023-12-19T15:55:17.958262+0000","last_active":"2023-12-19T15:55:18.989726+0000","last_peered":"2023-12-19T15:55:18.989726+0000","last_clean":"2023-12-19T15:55:18.989726+0000","last_became_active":"2023-12-19T15:55:17.956882+0000","last_became_peered":"2023-12-19T15:55:17.956882+0000","last_unstale":"2023-12-19T15:55:18.989726+0000","last_undegraded":"2023-12-19T15:55:18.989726+0000","last_fullsized":"2023-12-19T15:55:18.989726+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:25.188298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987572+0000","last_change":"2023-12-19T15:55:18.972817+0000","last_active":"2023-12-19T15:55:18.987572+0000","last_peered":"2023-12-19T15:55:18.987572+0000","last_clean":"2023-12-19T15:55:18.987572+0000","last_became_active":"2023-12-19T15:55:18.972495+0000","last_became_peered":"2023-12-19T15:55:18.972495+0000","last_unstale":"2023-12-19T15:55:18.987572+0000","last_undegraded":"2023-12-19T15:55:18.987572+0000","last_fullsized":"2023-12-19T15:55:18.987572+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:01.844134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.958701+0000","last_change":"2023-12-19T15:55:17.967625+0000","last_active":"2023-12-19T15:55:18.958701+0000","last_peered":"2023-12-19T15:55:18.958701+0000","last_clean":"2023-12-19T15:55:18.958701+0000","last_became_active":"2023-12-19T15:55:17.966297+0000","last_became_peered":"2023-12-19T15:55:17.966297+0000","last_unstale":"2023-12-19T15:55:18.958701+0000","last_undegraded":"2023-12-19T15:55:18.958701+0000","last_fullsized":"2023-12-19T15:55:18.958701+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:19:54.992110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976195+0000","last_change":"2023-12-19T15:55:18.967410+0000","last_active":"2023-12-19T15:55:18.976195+0000","last_peered":"2023-12-19T15:55:18.976195+0000","last_clean":"2023-12-19T15:55:18.976195+0000","last_became_active":"2023-12-19T15:55:18.966398+0000","last_became_peered":"2023-12-19T15:55:18.966398+0000","last_unstale":"2023-12-19T15:55:18.976195+0000","last_undegraded":"2023-12-19T15:55:18.976195+0000","last_fullsized":"2023-12-19T15:55:18.976195+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:37.344641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986726+0000","last_change":"2023-12-19T15:55:17.967120+0000","last_active":"2023-12-19T15:55:18.986726+0000","last_peered":"2023-12-19T15:55:18.986726+0000","last_clean":"2023-12-19T15:55:18.986726+0000","last_became_active":"2023-12-19T15:55:17.966783+0000","last_became_peered":"2023-12-19T15:55:17.966783+0000","last_unstale":"2023-12-19T15:55:18.986726+0000","last_undegraded":"2023-12-19T15:55:18.986726+0000","last_fullsized":"2023-12-19T15:55:18.986726+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:56.364922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.968027+0000","last_change":"2023-12-19T15:55:18.968027+0000","last_active":"2023-12-19T15:55:18.968027+0000","last_peered":"2023-12-19T15:55:18.968027+0000","last_clean":"2023-12-19T15:55:18.968027+0000","last_became_active":"2023-12-19T15:55:18.967751+0000","last_became_peered":"2023-12-19T15:55:18.967751+0000","last_unstale":"2023-12-19T15:55:18.968027+0000","last_undegraded":"2023-12-19T15:55:18.968027+0000","last_fullsized":"2023-12-19T15:55:18.968027+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:37.336141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991162+0000","last_change":"2023-12-19T15:55:17.962243+0000","last_active":"2023-12-19T15:55:18.991162+0000","last_peered":"2023-12-19T15:55:18.991162+0000","last_clean":"2023-12-19T15:55:18.991162+0000","last_became_active":"2023-12-19T15:55:17.961954+0000","last_became_peered":"2023-12-19T15:55:17.961954+0000","last_unstale":"2023-12-19T15:55:18.991162+0000","last_undegraded":"2023-12-19T15:55:18.991162+0000","last_fullsized":"2023-12-19T15:55:18.991162+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:23.322810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.967189+0000","last_change":"2023-12-19T15:55:18.967189+0000","last_active":"2023-12-19T15:55:18.967189+0000","last_peered":"2023-12-19T15:55:18.967189+0000","last_clean":"2023-12-19T15:55:18.967189+0000","last_became_active":"2023-12-19T15:55:18.966908+0000","last_became_peered":"2023-12-19T15:55:18.966908+0000","last_unstale":"2023-12-19T15:55:18.967189+0000","last_undegraded":"2023-12-19T15:55:18.967189+0000","last_fullsized":"2023-12-19T15:55:18.967189+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:56:08.956732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988992+0000","last_change":"2023-12-19T15:55:17.965774+0000","last_active":"2023-12-19T15:55:18.988992+0000","last_peered":"2023-12-19T15:55:18.988992+0000","last_clean":"2023-12-19T15:55:18.988992+0000","last_became_active":"2023-12-19T15:55:17.965508+0000","last_became_peered":"2023-12-19T15:55:17.965508+0000","last_unstale":"2023-12-19T15:55:18.988992+0000","last_undegraded":"2023-12-19T15:55:18.988992+0000","last_fullsized":"2023-12-19T15:55:18.988992+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:14:26.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971636+0000","last_change":"2023-12-19T15:55:18.968597+0000","last_active":"2023-12-19T15:55:18.971636+0000","last_peered":"2023-12-19T15:55:18.971636+0000","last_clean":"2023-12-19T15:55:18.971636+0000","last_became_active":"2023-12-19T15:55:18.968064+0000","last_became_peered":"2023-12-19T15:55:18.968064+0000","last_unstale":"2023-12-19T15:55:18.971636+0000","last_undegraded":"2023-12-19T15:55:18.971636+0000","last_fullsized":"2023-12-19T15:55:18.971636+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:44.409742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.949443+0000","last_change":"2023-12-19T15:55:17.959550+0000","last_active":"2023-12-19T15:55:18.949443+0000","last_peered":"2023-12-19T15:55:18.949443+0000","last_clean":"2023-12-19T15:55:18.949443+0000","last_became_active":"2023-12-19T15:55:17.959280+0000","last_became_peered":"2023-12-19T15:55:17.959280+0000","last_unstale":"2023-12-19T15:55:18.949443+0000","last_undegraded":"2023-12-19T15:55:18.949443+0000","last_fullsized":"2023-12-19T15:55:18.949443+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:31:28.135795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.970424+0000","last_change":"2023-12-19T15:55:18.962359+0000","last_active":"2023-12-19T15:55:18.970424+0000","last_peered":"2023-12-19T15:55:18.970424+0000","last_clean":"2023-12-19T15:55:18.970424+0000","last_became_active":"2023-12-19T15:55:18.962065+0000","last_became_peered":"2023-12-19T15:55:18.962065+0000","last_unstale":"2023-12-19T15:55:18.970424+0000","last_undegraded":"2023-12-19T15:55:18.970424+0000","last_fullsized":"2023-12-19T15:55:18.970424+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:50.529014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965922+0000","last_change":"2023-12-19T15:55:17.960359+0000","last_active":"2023-12-19T15:55:17.965922+0000","last_peered":"2023-12-19T15:55:17.965922+0000","last_clean":"2023-12-19T15:55:17.965922+0000","last_became_active":"2023-12-19T15:55:17.959201+0000","last_became_peered":"2023-12-19T15:55:17.959201+0000","last_unstale":"2023-12-19T15:55:17.965922+0000","last_undegraded":"2023-12-19T15:55:17.965922+0000","last_fullsized":"2023-12-19T15:55:17.965922+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:16:34.296723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976061+0000","last_change":"2023-12-19T15:55:18.976061+0000","last_active":"2023-12-19T15:55:18.976061+0000","last_peered":"2023-12-19T15:55:18.976061+0000","last_clean":"2023-12-19T15:55:18.976061+0000","last_became_active":"2023-12-19T15:55:18.974488+0000","last_became_peered":"2023-12-19T15:55:18.974488+0000","last_unstale":"2023-12-19T15:55:18.976061+0000","last_undegraded":"2023-12-19T15:55:18.976061+0000","last_fullsized":"2023-12-19T15:55:18.976061+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:43:13.777553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965349+0000","last_change":"2023-12-19T15:55:17.961924+0000","last_active":"2023-12-19T15:55:17.965349+0000","last_peered":"2023-12-19T15:55:17.965349+0000","last_clean":"2023-12-19T15:55:17.965349+0000","last_became_active":"2023-12-19T15:55:17.961595+0000","last_became_peered":"2023-12-19T15:55:17.961595+0000","last_unstale":"2023-12-19T15:55:17.965349+0000","last_undegraded":"2023-12-19T15:55:17.965349+0000","last_fullsized":"2023-12-19T15:55:17.965349+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:45.186605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978363+0000","last_change":"2023-12-19T15:55:18.971112+0000","last_active":"2023-12-19T15:55:18.978363+0000","last_peered":"2023-12-19T15:55:18.978363+0000","last_clean":"2023-12-19T15:55:18.978363+0000","last_became_active":"2023-12-19T15:55:18.970820+0000","last_became_peered":"2023-12-19T15:55:18.970820+0000","last_unstale":"2023-12-19T15:55:18.978363+0000","last_undegraded":"2023-12-19T15:55:18.978363+0000","last_fullsized":"2023-12-19T15:55:18.978363+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:54.641249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.969092+0000","last_change":"2023-12-19T15:55:17.959331+0000","last_active":"2023-12-19T15:55:17.969092+0000","last_peered":"2023-12-19T15:55:17.969092+0000","last_clean":"2023-12-19T15:55:17.969092+0000","last_became_active":"2023-12-19T15:55:17.959026+0000","last_became_peered":"2023-12-19T15:55:17.959026+0000","last_unstale":"2023-12-19T15:55:17.969092+0000","last_undegraded":"2023-12-19T15:55:17.969092+0000","last_fullsized":"2023-12-19T15:55:17.969092+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:52.802694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971554+0000","last_change":"2023-12-19T15:55:18.971554+0000","last_active":"2023-12-19T15:55:18.971554+0000","last_peered":"2023-12-19T15:55:18.971554+0000","last_clean":"2023-12-19T15:55:18.971554+0000","last_became_active":"2023-12-19T15:55:18.970012+0000","last_became_peered":"2023-12-19T15:55:18.970012+0000","last_unstale":"2023-12-19T15:55:18.971554+0000","last_undegraded":"2023-12-19T15:55:18.971554+0000","last_fullsized":"2023-12-19T15:55:18.971554+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:57.681711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966097+0000","last_change":"2023-12-19T15:55:17.958315+0000","last_active":"2023-12-19T15:55:17.966097+0000","last_peered":"2023-12-19T15:55:17.966097+0000","last_clean":"2023-12-19T15:55:17.966097+0000","last_became_active":"2023-12-19T15:55:17.957664+0000","last_became_peered":"2023-12-19T15:55:17.957664+0000","last_unstale":"2023-12-19T15:55:17.966097+0000","last_undegraded":"2023-12-19T15:55:17.966097+0000","last_fullsized":"2023-12-19T15:55:17.966097+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:10:38.132121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.965339+0000","last_change":"2023-12-19T15:55:18.965339+0000","last_active":"2023-12-19T15:55:18.965339+0000","last_peered":"2023-12-19T15:55:18.965339+0000","last_clean":"2023-12-19T15:55:18.965339+0000","last_became_active":"2023-12-19T15:55:18.962291+0000","last_became_peered":"2023-12-19T15:55:18.962291+0000","last_unstale":"2023-12-19T15:55:18.965339+0000","last_undegraded":"2023-12-19T15:55:18.965339+0000","last_fullsized":"2023-12-19T15:55:18.965339+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:11:59.607212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"180'3","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.985277+0000","last_change":"2023-12-19T15:55:17.958007+0000","last_active":"2023-12-19T15:55:18.985277+0000","last_peered":"2023-12-19T15:55:18.985277+0000","last_clean":"2023-12-19T15:55:18.985277+0000","last_became_active":"2023-12-19T15:55:17.957527+0000","last_became_peered":"2023-12-19T15:55:17.957527+0000","last_unstale":"2023-12-19T15:55:18.985277+0000","last_undegraded":"2023-12-19T15:55:18.985277+0000","last_fullsized":"2023-12-19T15:55:18.985277+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:21.324701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978725+0000","last_change":"2023-12-19T15:55:18.974154+0000","last_active":"2023-12-19T15:55:18.978725+0000","last_peered":"2023-12-19T15:55:18.978725+0000","last_clean":"2023-12-19T15:55:18.978725+0000","last_became_active":"2023-12-19T15:55:18.973848+0000","last_became_peered":"2023-12-19T15:55:18.973848+0000","last_unstale":"2023-12-19T15:55:18.978725+0000","last_undegraded":"2023-12-19T15:55:18.978725+0000","last_fullsized":"2023-12-19T15:55:18.978725+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:56:45.625418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986693+0000","last_change":"2023-12-19T15:55:17.957170+0000","last_active":"2023-12-19T15:55:18.986693+0000","last_peered":"2023-12-19T15:55:18.986693+0000","last_clean":"2023-12-19T15:55:18.986693+0000","last_became_active":"2023-12-19T15:55:17.956825+0000","last_became_peered":"2023-12-19T15:55:17.956825+0000","last_unstale":"2023-12-19T15:55:18.986693+0000","last_undegraded":"2023-12-19T15:55:18.986693+0000","last_fullsized":"2023-12-19T15:55:18.986693+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:20:25.052281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987456+0000","last_change":"2023-12-19T15:55:18.969923+0000","last_active":"2023-12-19T15:55:18.987456+0000","last_peered":"2023-12-19T15:55:18.987456+0000","last_clean":"2023-12-19T15:55:18.987456+0000","last_became_active":"2023-12-19T15:55:18.969258+0000","last_became_peered":"2023-12-19T15:55:18.969258+0000","last_unstale":"2023-12-19T15:55:18.987456+0000","last_undegraded":"2023-12-19T15:55:18.987456+0000","last_fullsized":"2023-12-19T15:55:18.987456+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:35.872104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961735+0000","last_change":"2023-12-19T15:55:17.961535+0000","last_active":"2023-12-19T15:55:18.961735+0000","last_peered":"2023-12-19T15:55:18.961735+0000","last_clean":"2023-12-19T15:55:18.961735+0000","last_became_active":"2023-12-19T15:55:17.961262+0000","last_became_peered":"2023-12-19T15:55:17.961262+0000","last_unstale":"2023-12-19T15:55:18.961735+0000","last_undegraded":"2023-12-19T15:55:18.961735+0000","last_fullsized":"2023-12-19T15:55:18.961735+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:19:33.813773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.954081+0000","last_change":"2023-12-19T15:55:18.954081+0000","last_active":"2023-12-19T15:55:18.954081+0000","last_peered":"2023-12-19T15:55:18.954081+0000","last_clean":"2023-12-19T15:55:18.954081+0000","last_became_active":"2023-12-19T15:55:18.952682+0000","last_became_peered":"2023-12-19T15:55:18.952682+0000","last_unstale":"2023-12-19T15:55:18.954081+0000","last_undegraded":"2023-12-19T15:55:18.954081+0000","last_fullsized":"2023-12-19T15:55:18.954081+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:55:57.381319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961220+0000","last_change":"2023-12-19T15:55:17.963702+0000","last_active":"2023-12-19T15:55:18.961220+0000","last_peered":"2023-12-19T15:55:18.961220+0000","last_clean":"2023-12-19T15:55:18.961220+0000","last_became_active":"2023-12-19T15:55:17.963429+0000","last_became_peered":"2023-12-19T15:55:17.963429+0000","last_unstale":"2023-12-19T15:55:18.961220+0000","last_undegraded":"2023-12-19T15:55:18.961220+0000","last_fullsized":"2023-12-19T15:55:18.961220+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:56.708610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988067+0000","last_change":"2023-12-19T15:55:18.973855+0000","last_active":"2023-12-19T15:55:18.988067+0000","last_peered":"2023-12-19T15:55:18.988067+0000","last_clean":"2023-12-19T15:55:18.988067+0000","last_became_active":"2023-12-19T15:55:18.973568+0000","last_became_peered":"2023-12-19T15:55:18.973568+0000","last_unstale":"2023-12-19T15:55:18.988067+0000","last_undegraded":"2023-12-19T15:55:18.988067+0000","last_fullsized":"2023-12-19T15:55:18.988067+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:13.143541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"180'8","reported_seq":17,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.989621+0000","last_change":"2023-12-19T15:55:17.957953+0000","last_active":"2023-12-19T15:55:18.989621+0000","last_peered":"2023-12-19T15:55:18.989621+0000","last_clean":"2023-12-19T15:55:18.989621+0000","last_became_active":"2023-12-19T15:55:17.957608+0000","last_became_peered":"2023-12-19T15:55:17.957608+0000","last_unstale":"2023-12-19T15:55:18.989621+0000","last_undegraded":"2023-12-19T15:55:18.989621+0000","last_fullsized":"2023-12-19T15:55:18.989621+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:28:06.910299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.970910+0000","last_change":"2023-12-19T15:55:18.970910+0000","last_active":"2023-12-19T15:55:18.970910+0000","last_peered":"2023-12-19T15:55:18.970910+0000","last_clean":"2023-12-19T15:55:18.970910+0000","last_became_active":"2023-12-19T15:55:18.970627+0000","last_became_peered":"2023-12-19T15:55:18.970627+0000","last_unstale":"2023-12-19T15:55:18.970910+0000","last_undegraded":"2023-12-19T15:55:18.970910+0000","last_fullsized":"2023-12-19T15:55:18.970910+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:59.642359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"180'1","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.944235+0000","last_change":"2023-12-19T15:55:17.957770+0000","last_active":"2023-12-19T15:55:18.944235+0000","last_peered":"2023-12-19T15:55:18.944235+0000","last_clean":"2023-12-19T15:55:18.944235+0000","last_became_active":"2023-12-19T15:55:17.957080+0000","last_became_peered":"2023-12-19T15:55:17.957080+0000","last_unstale":"2023-12-19T15:55:18.944235+0000","last_undegraded":"2023-12-19T15:55:18.944235+0000","last_fullsized":"2023-12-19T15:55:18.944235+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:16.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979782+0000","last_change":"2023-12-19T15:55:18.979782+0000","last_active":"2023-12-19T15:55:18.979782+0000","last_peered":"2023-12-19T15:55:18.979782+0000","last_clean":"2023-12-19T15:55:18.979782+0000","last_became_active":"2023-12-19T15:55:18.979297+0000","last_became_peered":"2023-12-19T15:55:18.979297+0000","last_unstale":"2023-12-19T15:55:18.979782+0000","last_undegraded":"2023-12-19T15:55:18.979782+0000","last_fullsized":"2023-12-19T15:55:18.979782+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:37.344641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987010+0000","last_change":"2023-12-19T15:55:17.964679+0000","last_active":"2023-12-19T15:55:18.987010+0000","last_peered":"2023-12-19T15:55:18.987010+0000","last_clean":"2023-12-19T15:55:18.987010+0000","last_became_active":"2023-12-19T15:55:17.964269+0000","last_became_peered":"2023-12-19T15:55:17.964269+0000","last_unstale":"2023-12-19T15:55:18.987010+0000","last_undegraded":"2023-12-19T15:55:18.987010+0000","last_fullsized":"2023-12-19T15:55:18.987010+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:34.855708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.975670+0000","last_change":"2023-12-19T15:55:18.975670+0000","last_active":"2023-12-19T15:55:18.975670+0000","last_peered":"2023-12-19T15:55:18.975670+0000","last_clean":"2023-12-19T15:55:18.975670+0000","last_became_active":"2023-12-19T15:55:18.973618+0000","last_became_peered":"2023-12-19T15:55:18.973618+0000","last_unstale":"2023-12-19T15:55:18.975670+0000","last_undegraded":"2023-12-19T15:55:18.975670+0000","last_fullsized":"2023-12-19T15:55:18.975670+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:03.278922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991196+0000","last_change":"2023-12-19T15:55:17.969785+0000","last_active":"2023-12-19T15:55:18.991196+0000","last_peered":"2023-12-19T15:55:18.991196+0000","last_clean":"2023-12-19T15:55:18.991196+0000","last_became_active":"2023-12-19T15:55:17.969271+0000","last_became_peered":"2023-12-19T15:55:17.969271+0000","last_unstale":"2023-12-19T15:55:18.991196+0000","last_undegraded":"2023-12-19T15:55:18.991196+0000","last_fullsized":"2023-12-19T15:55:18.991196+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:40:22.384403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969239+0000","last_change":"2023-12-19T15:55:18.969239+0000","last_active":"2023-12-19T15:55:18.969239+0000","last_peered":"2023-12-19T15:55:18.969239+0000","last_clean":"2023-12-19T15:55:18.969239+0000","last_became_active":"2023-12-19T15:55:18.968935+0000","last_became_peered":"2023-12-19T15:55:18.968935+0000","last_unstale":"2023-12-19T15:55:18.969239+0000","last_undegraded":"2023-12-19T15:55:18.969239+0000","last_fullsized":"2023-12-19T15:55:18.969239+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:32:22.708126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.961178+0000","last_change":"2023-12-19T15:55:17.961398+0000","last_active":"2023-12-19T15:55:18.961178+0000","last_peered":"2023-12-19T15:55:18.961178+0000","last_clean":"2023-12-19T15:55:18.961178+0000","last_became_active":"2023-12-19T15:55:17.961003+0000","last_became_peered":"2023-12-19T15:55:17.961003+0000","last_unstale":"2023-12-19T15:55:18.961178+0000","last_undegraded":"2023-12-19T15:55:18.961178+0000","last_fullsized":"2023-12-19T15:55:18.961178+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:15:43.255409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.981536+0000","last_change":"2023-12-19T15:55:18.981536+0000","last_active":"2023-12-19T15:55:18.981536+0000","last_peered":"2023-12-19T15:55:18.981536+0000","last_clean":"2023-12-19T15:55:18.981536+0000","last_became_active":"2023-12-19T15:55:18.977070+0000","last_became_peered":"2023-12-19T15:55:18.977070+0000","last_unstale":"2023-12-19T15:55:18.981536+0000","last_undegraded":"2023-12-19T15:55:18.981536+0000","last_fullsized":"2023-12-19T15:55:18.981536+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:54.641249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"180'2","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987651+0000","last_change":"2023-12-19T15:55:17.969021+0000","last_active":"2023-12-19T15:55:18.987651+0000","last_peered":"2023-12-19T15:55:18.987651+0000","last_clean":"2023-12-19T15:55:18.987651+0000","last_became_active":"2023-12-19T15:55:17.968704+0000","last_became_peered":"2023-12-19T15:55:17.968704+0000","last_unstale":"2023-12-19T15:55:18.987651+0000","last_undegraded":"2023-12-19T15:55:18.987651+0000","last_fullsized":"2023-12-19T15:55:18.987651+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:57:53.624853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.971324+0000","last_change":"2023-12-19T15:55:18.959196+0000","last_active":"2023-12-19T15:55:18.971324+0000","last_peered":"2023-12-19T15:55:18.971324+0000","last_clean":"2023-12-19T15:55:18.971324+0000","last_became_active":"2023-12-19T15:55:18.958853+0000","last_became_peered":"2023-12-19T15:55:18.958853+0000","last_unstale":"2023-12-19T15:55:18.971324+0000","last_undegraded":"2023-12-19T15:55:18.971324+0000","last_fullsized":"2023-12-19T15:55:18.971324+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:01:52.197350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.955860+0000","last_change":"2023-12-19T15:55:17.953419+0000","last_active":"2023-12-19T15:55:18.955860+0000","last_peered":"2023-12-19T15:55:18.955860+0000","last_clean":"2023-12-19T15:55:18.955860+0000","last_became_active":"2023-12-19T15:55:17.953093+0000","last_became_peered":"2023-12-19T15:55:17.953093+0000","last_unstale":"2023-12-19T15:55:18.955860+0000","last_undegraded":"2023-12-19T15:55:18.955860+0000","last_fullsized":"2023-12-19T15:55:18.955860+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:49.512617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969869+0000","last_change":"2023-12-19T15:55:18.969869+0000","last_active":"2023-12-19T15:55:18.969869+0000","last_peered":"2023-12-19T15:55:18.969869+0000","last_clean":"2023-12-19T15:55:18.969869+0000","last_became_active":"2023-12-19T15:55:18.968957+0000","last_became_peered":"2023-12-19T15:55:18.968957+0000","last_unstale":"2023-12-19T15:55:18.969869+0000","last_undegraded":"2023-12-19T15:55:18.969869+0000","last_fullsized":"2023-12-19T15:55:18.969869+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.970206+0000","last_change":"2023-12-19T15:55:17.968685+0000","last_active":"2023-12-19T15:55:17.970206+0000","last_peered":"2023-12-19T15:55:17.970206+0000","last_clean":"2023-12-19T15:55:17.970206+0000","last_became_active":"2023-12-19T15:55:17.968290+0000","last_became_peered":"2023-12-19T15:55:17.968290+0000","last_unstale":"2023-12-19T15:55:17.970206+0000","last_undegraded":"2023-12-19T15:55:17.970206+0000","last_fullsized":"2023-12-19T15:55:17.970206+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:36.319744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987551+0000","last_change":"2023-12-19T15:55:18.969710+0000","last_active":"2023-12-19T15:55:18.987551+0000","last_peered":"2023-12-19T15:55:18.987551+0000","last_clean":"2023-12-19T15:55:18.987551+0000","last_became_active":"2023-12-19T15:55:18.969420+0000","last_became_peered":"2023-12-19T15:55:18.969420+0000","last_unstale":"2023-12-19T15:55:18.987551+0000","last_undegraded":"2023-12-19T15:55:18.987551+0000","last_fullsized":"2023-12-19T15:55:18.987551+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:53.819090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.953407+0000","last_change":"2023-12-19T15:55:17.957014+0000","last_active":"2023-12-19T15:55:18.953407+0000","last_peered":"2023-12-19T15:55:18.953407+0000","last_clean":"2023-12-19T15:55:18.953407+0000","last_became_active":"2023-12-19T15:55:17.956697+0000","last_became_peered":"2023-12-19T15:55:17.956697+0000","last_unstale":"2023-12-19T15:55:18.953407+0000","last_undegraded":"2023-12-19T15:55:18.953407+0000","last_fullsized":"2023-12-19T15:55:18.953407+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:48:12.127145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.964273+0000","last_change":"2023-12-19T15:55:18.964273+0000","last_active":"2023-12-19T15:55:18.964273+0000","last_peered":"2023-12-19T15:55:18.964273+0000","last_clean":"2023-12-19T15:55:18.964273+0000","last_became_active":"2023-12-19T15:55:18.963936+0000","last_became_peered":"2023-12-19T15:55:18.963936+0000","last_unstale":"2023-12-19T15:55:18.964273+0000","last_undegraded":"2023-12-19T15:55:18.964273+0000","last_fullsized":"2023-12-19T15:55:18.964273+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:03:10.738658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.986542+0000","last_change":"2023-12-19T15:55:17.967206+0000","last_active":"2023-12-19T15:55:18.986542+0000","last_peered":"2023-12-19T15:55:18.986542+0000","last_clean":"2023-12-19T15:55:18.986542+0000","last_became_active":"2023-12-19T15:55:17.966384+0000","last_became_peered":"2023-12-19T15:55:17.966384+0000","last_unstale":"2023-12-19T15:55:18.986542+0000","last_undegraded":"2023-12-19T15:55:18.986542+0000","last_fullsized":"2023-12-19T15:55:18.986542+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:59:27.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.988767+0000","last_change":"2023-12-19T15:55:18.984744+0000","last_active":"2023-12-19T15:55:18.988767+0000","last_peered":"2023-12-19T15:55:18.988767+0000","last_clean":"2023-12-19T15:55:18.988767+0000","last_became_active":"2023-12-19T15:55:18.984449+0000","last_became_peered":"2023-12-19T15:55:18.984449+0000","last_unstale":"2023-12-19T15:55:18.988767+0000","last_undegraded":"2023-12-19T15:55:18.988767+0000","last_fullsized":"2023-12-19T15:55:18.988767+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:47:04.314869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966080+0000","last_change":"2023-12-19T15:55:17.958516+0000","last_active":"2023-12-19T15:55:17.966080+0000","last_peered":"2023-12-19T15:55:17.966080+0000","last_clean":"2023-12-19T15:55:17.966080+0000","last_became_active":"2023-12-19T15:55:17.958054+0000","last_became_peered":"2023-12-19T15:55:17.958054+0000","last_unstale":"2023-12-19T15:55:17.966080+0000","last_undegraded":"2023-12-19T15:55:17.966080+0000","last_fullsized":"2023-12-19T15:55:17.966080+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:29:48.063182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978580+0000","last_change":"2023-12-19T15:55:18.978580+0000","last_active":"2023-12-19T15:55:18.978580+0000","last_peered":"2023-12-19T15:55:18.978580+0000","last_clean":"2023-12-19T15:55:18.978580+0000","last_became_active":"2023-12-19T15:55:18.977988+0000","last_became_peered":"2023-12-19T15:55:18.977988+0000","last_unstale":"2023-12-19T15:55:18.978580+0000","last_undegraded":"2023-12-19T15:55:18.978580+0000","last_fullsized":"2023-12-19T15:55:18.978580+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:31:58.020254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.954388+0000","last_change":"2023-12-19T15:55:17.960733+0000","last_active":"2023-12-19T15:55:18.954388+0000","last_peered":"2023-12-19T15:55:18.954388+0000","last_clean":"2023-12-19T15:55:18.954388+0000","last_became_active":"2023-12-19T15:55:17.960441+0000","last_became_peered":"2023-12-19T15:55:17.960441+0000","last_unstale":"2023-12-19T15:55:18.954388+0000","last_undegraded":"2023-12-19T15:55:18.954388+0000","last_fullsized":"2023-12-19T15:55:18.954388+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:32.131618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":8,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.969795+0000","last_change":"2023-12-19T15:55:18.969795+0000","last_active":"2023-12-19T15:55:18.969795+0000","last_peered":"2023-12-19T15:55:18.969795+0000","last_clean":"2023-12-19T15:55:18.969795+0000","last_became_active":"2023-12-19T15:55:18.969498+0000","last_became_peered":"2023-12-19T15:55:18.969498+0000","last_unstale":"2023-12-19T15:55:18.969795+0000","last_undegraded":"2023-12-19T15:55:18.969795+0000","last_fullsized":"2023-12-19T15:55:18.969795+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:47:59.642359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.966471+0000","last_change":"2023-12-19T15:55:17.963651+0000","last_active":"2023-12-19T15:55:17.966471+0000","last_peered":"2023-12-19T15:55:17.966471+0000","last_clean":"2023-12-19T15:55:17.966471+0000","last_became_active":"2023-12-19T15:55:17.963387+0000","last_became_peered":"2023-12-19T15:55:17.963387+0000","last_unstale":"2023-12-19T15:55:17.966471+0000","last_undegraded":"2023-12-19T15:55:17.966471+0000","last_fullsized":"2023-12-19T15:55:17.966471+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:55.325995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.974041+0000","last_change":"2023-12-19T15:55:18.962228+0000","last_active":"2023-12-19T15:55:18.974041+0000","last_peered":"2023-12-19T15:55:18.974041+0000","last_clean":"2023-12-19T15:55:18.974041+0000","last_became_active":"2023-12-19T15:55:18.961498+0000","last_became_peered":"2023-12-19T15:55:18.961498+0000","last_unstale":"2023-12-19T15:55:18.974041+0000","last_undegraded":"2023-12-19T15:55:18.974041+0000","last_fullsized":"2023-12-19T15:55:18.974041+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:28:29.191381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965509+0000","last_change":"2023-12-19T15:55:17.963589+0000","last_active":"2023-12-19T15:55:17.965509+0000","last_peered":"2023-12-19T15:55:17.965509+0000","last_clean":"2023-12-19T15:55:17.965509+0000","last_became_active":"2023-12-19T15:55:17.963302+0000","last_became_peered":"2023-12-19T15:55:17.963302+0000","last_unstale":"2023-12-19T15:55:17.965509+0000","last_undegraded":"2023-12-19T15:55:17.965509+0000","last_fullsized":"2023-12-19T15:55:17.965509+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:39.950747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.976270+0000","last_change":"2023-12-19T15:55:18.971547+0000","last_active":"2023-12-19T15:55:18.976270+0000","last_peered":"2023-12-19T15:55:18.976270+0000","last_clean":"2023-12-19T15:55:18.976270+0000","last_became_active":"2023-12-19T15:55:18.971265+0000","last_became_peered":"2023-12-19T15:55:18.971265+0000","last_unstale":"2023-12-19T15:55:18.976270+0000","last_undegraded":"2023-12-19T15:55:18.976270+0000","last_fullsized":"2023-12-19T15:55:18.976270+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:16.749425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.965666+0000","last_change":"2023-12-19T15:55:17.963647+0000","last_active":"2023-12-19T15:55:17.965666+0000","last_peered":"2023-12-19T15:55:17.965666+0000","last_clean":"2023-12-19T15:55:17.965666+0000","last_became_active":"2023-12-19T15:55:17.962696+0000","last_became_peered":"2023-12-19T15:55:17.962696+0000","last_unstale":"2023-12-19T15:55:17.965666+0000","last_undegraded":"2023-12-19T15:55:17.965666+0000","last_fullsized":"2023-12-19T15:55:17.965666+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:54:20.414653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.987877+0000","last_change":"2023-12-19T15:55:18.968133+0000","last_active":"2023-12-19T15:55:18.987877+0000","last_peered":"2023-12-19T15:55:18.987877+0000","last_clean":"2023-12-19T15:55:18.987877+0000","last_became_active":"2023-12-19T15:55:18.967747+0000","last_became_peered":"2023-12-19T15:55:18.967747+0000","last_unstale":"2023-12-19T15:55:18.987877+0000","last_undegraded":"2023-12-19T15:55:18.987877+0000","last_fullsized":"2023-12-19T15:55:18.987877+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:31:57.725007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.962048+0000","last_change":"2023-12-19T15:55:17.956296+0000","last_active":"2023-12-19T15:55:17.962048+0000","last_peered":"2023-12-19T15:55:17.962048+0000","last_clean":"2023-12-19T15:55:17.962048+0000","last_became_active":"2023-12-19T15:55:17.956031+0000","last_became_peered":"2023-12-19T15:55:17.956031+0000","last_unstale":"2023-12-19T15:55:17.962048+0000","last_undegraded":"2023-12-19T15:55:17.962048+0000","last_fullsized":"2023-12-19T15:55:17.962048+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:41:35.178870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.979779+0000","last_change":"2023-12-19T15:55:18.978074+0000","last_active":"2023-12-19T15:55:18.979779+0000","last_peered":"2023-12-19T15:55:18.979779+0000","last_clean":"2023-12-19T15:55:18.979779+0000","last_became_active":"2023-12-19T15:55:18.977749+0000","last_became_peered":"2023-12-19T15:55:18.977749+0000","last_unstale":"2023-12-19T15:55:18.979779+0000","last_undegraded":"2023-12-19T15:55:18.979779+0000","last_fullsized":"2023-12-19T15:55:18.979779+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:34:22.928812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.969126+0000","last_change":"2023-12-19T15:55:17.962786+0000","last_active":"2023-12-19T15:55:17.969126+0000","last_peered":"2023-12-19T15:55:17.969126+0000","last_clean":"2023-12-19T15:55:17.969126+0000","last_became_active":"2023-12-19T15:55:17.962536+0000","last_became_peered":"2023-12-19T15:55:17.962536+0000","last_unstale":"2023-12-19T15:55:17.969126+0000","last_undegraded":"2023-12-19T15:55:17.969126+0000","last_fullsized":"2023-12-19T15:55:17.969126+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:57:00.827737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.978305+0000","last_change":"2023-12-19T15:55:18.970551+0000","last_active":"2023-12-19T15:55:18.978305+0000","last_peered":"2023-12-19T15:55:18.978305+0000","last_clean":"2023-12-19T15:55:18.978305+0000","last_became_active":"2023-12-19T15:55:18.970118+0000","last_became_peered":"2023-12-19T15:55:18.970118+0000","last_unstale":"2023-12-19T15:55:18.978305+0000","last_undegraded":"2023-12-19T15:55:18.978305+0000","last_fullsized":"2023-12-19T15:55:18.978305+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:17.915418+0000","last_clean_scrub_stamp":"2023-12-19T15:55:17.915418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:14:39.401673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"180'6","reported_seq":15,"reported_epoch":180,"state":"active+clean","last_fresh":"2023-12-19T15:55:18.991180+0000","last_change":"2023-12-19T15:55:17.966585+0000","last_active":"2023-12-19T15:55:18.991180+0000","last_peered":"2023-12-19T15:55:18.991180+0000","last_clean":"2023-12-19T15:55:18.991180+0000","last_became_active":"2023-12-19T15:55:17.966274+0000","last_became_peered":"2023-12-19T15:55:17.966274+0000","last_unstale":"2023-12-19T15:55:18.991180+0000","last_undegraded":"2023-12-19T15:55:18.991180+0000","last_fullsized":"2023-12-19T15:55:18.991180+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T15:55:16.899021+0000","last_clean_scrub_stamp":"2023-12-19T15:55:16.899021+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:53:36.328245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"169'293","reported_seq":917,"reported_epoch":179,"state":"active+clean","last_fresh":"2023-12-19T15:55:17.951381+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T15:55:17.951381+0000","last_peered":"2023-12-19T15:55:17.951381+0000","last_clean":"2023-12-19T15:55:17.951381+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T15:55:17.951381+0000","last_undegraded":"2023-12-19T15:55:17.951381+0000","last_fullsized":"2023-12-19T15:55:17.951381+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575759,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300068,"kb_used_data":2780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071772,"statfs":{"total":96636764160,"available":96329494528,"internally_reserved":0,"allocated":2846720,"data_stored":1732904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1379999999999999,"15min":1.135},"min":{"1min":0.68500000000000005,"5min":0.64800000000000002,"15min":0.54200000000000004},"max":{"1min":1.6160000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.171},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1779999999999999,"15min":1.1579999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":1.361}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.149,"15min":1.1519999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.4330000000000001,"5min":1.7010000000000001,"15min":1.778},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.77600000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.542,"5min":1.698,"15min":1.698},"last":1.512}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3029999999999999,"15min":1.294},"min":{"1min":1.0369999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.669,"5min":1.6890000000000001,"15min":1.7969999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.345,"15min":1.3360000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.79300000000000004},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7889999999999999},"last":0.90500000000000003}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.875,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5960000000000001,"5min":1.6839999999999999,"15min":1.7130000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.355,"15min":1.367},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.5369999999999999,"5min":1.7270000000000001,"15min":1.7290000000000001},"last":1.266}]},{"osd":4,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.163,"15min":1.181},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.63200000000000001},"max":{"1min":1.5820000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.206},{"interface":"front","average":{"1min":1.212,"5min":1.2629999999999999,"15min":1.2609999999999999},"min":{"1min":0.92800000000000005,"5min":0.82099999999999995,"15min":0.63500000000000001},"max":{"1min":1.6180000000000001,"5min":1.8009999999999999,"15min":1.8899999999999999},"last":1.093}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2390000000000001,"15min":1.264},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60499999999999998},"max":{"1min":1.6519999999999999,"5min":1.758,"15min":2.0880000000000001},"last":1.129},{"interface":"front","average":{"1min":1.292,"5min":1.325,"15min":1.325},"min":{"1min":0.92900000000000005,"5min":0.79100000000000004,"15min":0.69999999999999996},"max":{"1min":1.7090000000000001,"5min":1.798,"15min":1.798},"last":1.056}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4450000000000001,"15min":1.423},"min":{"1min":1.1910000000000001,"5min":0.82399999999999995,"15min":0.76000000000000001},"max":{"1min":1.778,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3859999999999999,"15min":1.395},"min":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.70999999999999996},"max":{"1min":1.911,"5min":1.911,"15min":2.0110000000000001},"last":1.4199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575759,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1142584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1599999999999999,"15min":1.155},"min":{"1min":0.92900000000000005,"5min":0.79700000000000004,"15min":0.72799999999999998},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.387},{"interface":"front","average":{"1min":1.22,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.99399999999999999,"5min":0.78800000000000003,"15min":0.75900000000000001},"max":{"1min":1.4490000000000001,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.214}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1910000000000001,"15min":1.1879999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.52,"5min":1.617,"15min":1.6890000000000001},"last":1.234},{"interface":"front","average":{"1min":1.228,"5min":1.2549999999999999,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.5269999999999999,"5min":2.7480000000000002,"15min":2.7480000000000002},"last":1.4330000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.2490000000000001,"15min":1.242},"min":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.83699999999999997},"max":{"1min":1.611,"5min":1.629,"15min":1.6990000000000001},"last":1.734},{"interface":"front","average":{"1min":1.262,"5min":1.276,"15min":1.268},"min":{"1min":0.96399999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.7130000000000001},"last":1.0349999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.647,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":1.115},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3600000000000001,"15min":1.3440000000000001},"min":{"1min":0.98699999999999999,"5min":0.96299999999999997,"15min":0.93300000000000005},"max":{"1min":1.6220000000000001,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.6559999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2230000000000001,"15min":1.2030000000000001},"min":{"1min":0.86499999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.294,"15min":1.3109999999999999},"min":{"1min":0.89000000000000001,"5min":0.79500000000000004,"15min":0.73199999999999998},"max":{"1min":1.8049999999999999,"5min":2.653,"15min":2.653},"last":1.5229999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.3},"min":{"1min":0.84899999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8160000000000001,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.486},{"interface":"front","average":{"1min":1.258,"5min":1.3320000000000001,"15min":1.361},"min":{"1min":0.878,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.762,"5min":1.871,"15min":1.99},"last":1.1499999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.4039999999999999,"15min":1.4299999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.742,"5min":2.8340000000000001,"15min":2.8340000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":1.4490000000000001},"min":{"1min":0.97399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.905,"5min":2.4889999999999999,"15min":2.4889999999999999},"last":1.8069999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575759,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307676,"kb_used_data":2196,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064164,"statfs":{"total":96636764160,"available":96321703936,"internally_reserved":0,"allocated":2248704,"data_stored":1142514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":2.2759999999999998},"last":1.304},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1870000000000001,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6679999999999999,"5min":1.7909999999999999,"15min":2.7309999999999999},"last":0.91200000000000003}]},{"osd":1,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.339,"15min":1.3220000000000001},"min":{"1min":1,"5min":0.86599999999999999,"15min":0.84499999999999997},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7370000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.282,"15min":1.2689999999999999},"min":{"1min":1.038,"5min":0.95699999999999996,"15min":0.82599999999999996},"max":{"1min":1.617,"5min":1.645,"15min":1.8779999999999999},"last":1.2}]},{"osd":2,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.6699999999999999,"5min":1.748,"15min":2.8239999999999998},"last":1.47},{"interface":"front","average":{"1min":1.234,"5min":1.2350000000000001,"15min":1.2230000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.71199999999999997},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":1.871},"last":0.871}]},{"osd":3,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.306},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.631,"5min":1.7350000000000001,"15min":2.4849999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.2969999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6739999999999999,"5min":1.7929999999999999,"15min":1.9390000000000001},"last":0.96699999999999997}]},{"osd":4,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1359999999999999,"15min":1.137},"min":{"1min":0.84299999999999997,"5min":0.69499999999999995,"15min":0.59999999999999998},"max":{"1min":1.659,"5min":1.659,"15min":2.109},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.244,"15min":1.27},"min":{"1min":0.86799999999999999,"5min":0.81799999999999995,"15min":0.56999999999999995},"max":{"1min":1.7869999999999999,"5min":1.8320000000000001,"15min":2.8999999999999999},"last":1.1140000000000001}]},{"osd":6,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.333,"15min":1.3380000000000001},"min":{"1min":0.92100000000000004,"5min":0.84399999999999997,"15min":0.58899999999999997},"max":{"1min":1.8089999999999999,"5min":1.9299999999999999,"15min":2.0649999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":1.1259999999999999,"5min":0.92500000000000004,"15min":0.83699999999999997},"max":{"1min":1.998,"5min":1.998,"15min":2.117},"last":1.3720000000000001}]},{"osd":7,"last update":"Tue Dec 19 15:54:31 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4259999999999999,"15min":1.4179999999999999},"min":{"1min":1.2310000000000001,"5min":0.95499999999999996,"15min":0.623},"max":{"1min":1.915,"5min":1.915,"15min":2.6000000000000001},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.419,"15min":1.411},"min":{"1min":1.0389999999999999,"5min":0.84599999999999997,"15min":0.68000000000000005},"max":{"1min":1.879,"5min":1.9870000000000001,"15min":2.3119999999999998},"last":1.228}]}]},{"osd":4,"up_from":13,"seq":55834575759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307688,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064152,"statfs":{"total":96636764160,"available":96321691648,"internally_reserved":0,"allocated":2260992,"data_stored":1144050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1299999999999999,"15min":1.1100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.7050000000000001,"5min":1.712,"15min":1.722},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.113,"15min":1.1399999999999999},"min":{"1min":0.76800000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.4770000000000001,"5min":1.5249999999999999,"15min":1.75},"last":1.1850000000000001}]},{"osd":1,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.274,"15min":1.286},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6020000000000001,"5min":1.6819999999999999,"15min":2.2530000000000001},"last":1.232},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3200000000000001,"15min":1.3280000000000001},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.69399999999999995},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.968},"last":1.5900000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1850000000000001,"15min":1.1859999999999999},"min":{"1min":0.754,"5min":0.73599999999999999,"15min":0.65300000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":2.3410000000000002},"last":1.339},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.272,"15min":1.2669999999999999},"min":{"1min":0.84399999999999997,"5min":0.79900000000000004,"15min":0.57799999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1459999999999999},"last":1.2889999999999999}]},{"osd":3,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.236,"15min":1.23},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.772},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.26,"15min":1.2869999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.65500000000000003},"max":{"1min":1.758,"5min":1.758,"15min":2.0800000000000001},"last":1.843}]},{"osd":5,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.204,"15min":1.1839999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.42399999999999999},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.6240000000000001,"5min":1.6579999999999999,"15min":1.998},"last":1.5349999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3140000000000001,"15min":1.306},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.60399999999999998},"max":{"1min":1.7190000000000001,"5min":1.766,"15min":2.3340000000000001},"last":1.71},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.458,"15min":1.454},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.68100000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.675}]},{"osd":7,"last update":"Tue Dec 19 15:54:29 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3859999999999999,"15min":1.3600000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8260000000000001,"5min":1.839,"15min":2.0150000000000001},"last":1.617},{"interface":"front","average":{"1min":1.486,"5min":1.4890000000000001,"15min":1.47},"min":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.55600000000000005},"max":{"1min":1.8660000000000001,"5min":2.157,"15min":2.1659999999999999},"last":1.944}]}]},{"osd":3,"up_from":13,"seq":55834575758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315876,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055964,"statfs":{"total":96636764160,"available":96313307136,"internally_reserved":0,"allocated":2256896,"data_stored":1143102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94899999999999995,"15min":0.91600000000000004},"min":{"1min":0.57999999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.61099999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":2.113},"last":0.72199999999999998}]},{"osd":1,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57699999999999996},"max":{"1min":1.2250000000000001,"5min":1.5680000000000001,"15min":1.75},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1459999999999999,"15min":1.1200000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.53900000000000003},"max":{"1min":1.4950000000000001,"5min":1.706,"15min":1.8149999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1699999999999999,"15min":1.1659999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.645,"5min":1.645,"15min":1.992},"last":1.302},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.294,"15min":1.264},"min":{"1min":0.871,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.347,"15min":1.343},"min":{"1min":1.0089999999999999,"5min":0.94099999999999995,"15min":0.879},"max":{"1min":1.623,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.052},{"interface":"front","average":{"1min":1.226,"5min":1.3120000000000001,"15min":1.321},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1120000000000001},"last":1.2290000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3360000000000001,"15min":1.3440000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.79200000000000004},"max":{"1min":1.7589999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.444,"15min":1.4239999999999999},"min":{"1min":1.1200000000000001,"5min":0.84099999999999997,"15min":0.81000000000000005},"max":{"1min":1.8560000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.1799999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.508,"15min":1.482},"min":{"1min":1.1819999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.073,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4770000000000001,"15min":1.4630000000000001},"min":{"1min":0.94499999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1190000000000002,"5min":2.141,"15min":2.1909999999999998},"last":1.492}]},{"osd":7,"last update":"Tue Dec 19 15:55:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.554,"15min":1.5569999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.0880000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.498,"15min":1.5089999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0310000000000001,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":1.55}]}]},{"osd":2,"up_from":13,"seq":55834575760,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1141615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.94699999999999995,"15min":0.95499999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.446,"5min":1.587,"15min":1.587},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.1399999999999999,"15min":1.141},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.944},"last":0.82499999999999996}]},{"osd":1,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1339999999999999,"15min":1.1739999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.526,"5min":1.6060000000000001,"15min":1.6870000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.097,"15min":1.139},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4470000000000001,"5min":1.6160000000000001,"15min":1.7430000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.014,"15min":1.0249999999999999},"min":{"1min":0.65700000000000003,"5min":0.64900000000000002,"15min":0.64000000000000001},"max":{"1min":1.5009999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.999,"5min":1.105,"15min":1.123},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.3939999999999999,"5min":1.5149999999999999,"15min":1.744},"last":0.80300000000000005}]},{"osd":4,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3380000000000001,"15min":1.4019999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.825,"5min":1.825,"15min":2.0790000000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.4019999999999999,"15min":1.446},"min":{"1min":0.88300000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.778,"5min":1.921,"15min":2.3410000000000002},"last":0.98399999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3500000000000001,"15min":1.383},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0659999999999998},"last":1.274},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3700000000000001,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.875,"5min":1.996,"15min":2.016},"last":1.155}]},{"osd":6,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3660000000000001,"15min":1.391},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.651,"5min":1.782,"15min":1.9750000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4850000000000001,"15min":1.5029999999999999},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":1.8899999999999999,"5min":2.0489999999999999,"15min":2.242},"last":1.4039999999999999}]},{"osd":7,"last update":"Tue Dec 19 15:55:14 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5009999999999999,"15min":1.5489999999999999},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.8149999999999999,"5min":1.9750000000000001,"15min":2.2389999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.478,"15min":1.508},"min":{"1min":0.871,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8560000000000001,"5min":2.016,"15min":2.016},"last":1.3859999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311776,"kb_used_data":2200,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060064,"statfs":{"total":96636764160,"available":96317505536,"internally_reserved":0,"allocated":2252800,"data_stored":1142584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.89800000000000002,"15min":0.88800000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.389,"5min":1.4890000000000001,"15min":1.6579999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0029999999999999,"15min":1.024},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3320000000000001,"5min":1.532,"15min":1.6879999999999999},"last":1.256}]},{"osd":2,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.036,"15min":1.0649999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.5069999999999999,"5min":1.6000000000000001,"15min":1.7989999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1379999999999999,"15min":1.163},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.421,"5min":1.5900000000000001,"15min":1.7549999999999999},"last":1.605}]},{"osd":3,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.282,"15min":1.262},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.79,"5min":1.79,"15min":1.851},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.087,"15min":1.099},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.6659999999999999,"5min":1.778,"15min":1.849},"last":0.86099999999999999}]},{"osd":4,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.325,"15min":1.3260000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.992,"5min":1.992,"15min":2.0739999999999998},"last":1.514},{"interface":"front","average":{"1min":1.397,"5min":1.407,"15min":1.4239999999999999},"min":{"1min":1.1060000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.921},"last":1.4810000000000001}]},{"osd":5,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.359,"15min":1.3460000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":1.421},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.425,"15min":1.4259999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.1909999999999998},"last":1.1919999999999999}]},{"osd":6,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4059999999999999,"15min":1.399},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.5},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.1040000000000001},"last":1.667}]},{"osd":7,"last update":"Tue Dec 19 15:55:01 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.494,"15min":1.5209999999999999},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.0699999999999998,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4610000000000001,"15min":1.48},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.028,"5min":2.0659999999999998,"15min":2.0699999999999998},"last":1.5549999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575758,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300084,"kb_used_data":2796,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071756,"statfs":{"total":96636764160,"available":96329478144,"internally_reserved":0,"allocated":2863104,"data_stored":1734085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.66500000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.78000000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3819999999999999,"5min":1.5649999999999999,"15min":1.895},"last":0.88400000000000001}]},{"osd":2,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.1259999999999999,"15min":1.1439999999999999},"min":{"1min":0.85599999999999998,"5min":0.502,"15min":0.47399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":2.5129999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.256,"15min":1.2789999999999999},"min":{"1min":0.93600000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.679,"5min":1.708,"15min":2.8250000000000002},"last":1.2190000000000001}]},{"osd":3,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.038},"min":{"1min":0.79100000000000004,"5min":0.70199999999999996,"15min":0.66200000000000003},"max":{"1min":1.5349999999999999,"5min":1.599,"15min":1.9970000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2070000000000001,"15min":1.2090000000000001},"min":{"1min":0.92200000000000004,"5min":0.621,"15min":0.59399999999999997},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.794},"last":1.246}]},{"osd":4,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3919999999999999,"15min":1.351},"min":{"1min":1.0640000000000001,"5min":0.94499999999999995,"15min":0.80000000000000004},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.321},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.327,"15min":1.3169999999999999},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8280000000000001,"5min":1.9059999999999999,"15min":2.2210000000000001},"last":1.2809999999999999}]},{"osd":5,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.419,"15min":1.4279999999999999},"min":{"1min":1.175,"5min":0.871,"15min":0.73099999999999998},"max":{"1min":1.879,"5min":1.901,"15min":2.4470000000000001},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4670000000000001,"15min":1.4359999999999999},"min":{"1min":1.004,"5min":1.004,"15min":0.79400000000000004},"max":{"1min":1.889,"5min":1.9510000000000001,"15min":2.468},"last":1.381}]},{"osd":6,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.413,"15min":1.385},"min":{"1min":1.1830000000000001,"5min":0.90000000000000002,"15min":0.88500000000000001},"max":{"1min":1.925,"5min":1.925,"15min":2.0830000000000002},"last":1.484},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.53,"15min":1.5},"min":{"1min":1.2110000000000001,"5min":0.92600000000000005,"15min":0.81799999999999995},"max":{"1min":1.9470000000000001,"5min":2.379,"15min":2.379},"last":1.474}]},{"osd":7,"last update":"Tue Dec 19 15:54:21 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5429999999999999,"15min":1.5049999999999999},"min":{"1min":1.0900000000000001,"5min":0.99299999999999999,"15min":0.90900000000000003},"max":{"1min":1.968,"5min":2.105,"15min":2.698},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.5089999999999999,"15min":1.482},"min":{"1min":1.022,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":1.8380000000000001,"5min":2.016,"15min":3.23},"last":1.3919999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T15:55:23.738 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T15:55:23.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T15:55:23.900 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T15:55:23.895+0000 7fe5ae31b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T15:55:23.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.943+0000 7f6c7ea6e700 1 Processor -- start 2023-12-19T15:55:23.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.943+0000 7f6c7ea6e700 1 -- start start 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7ea6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c780c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7ea6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6c7806ac30 0x7f6c7806b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7ea6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7806a7a0 con 0x7f6c7806c7c0 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7ea6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c780c7630 con 0x7f6c7806ac30 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7ea6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7806d4b0 con 0x7f6c780c6d20 2023-12-19T15:55:23.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7d00b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6c7806ac30 0x7f6c7806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7d00b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6c7806ac30 0x7f6c7806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35882/0 (socket says 172.21.15.50:35882) 2023-12-19T15:55:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7d00b700 1 -- 172.21.15.50:0/3356713625 learned_addr learned my addr 172.21.15.50:0/3356713625 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.947+0000 7f6c7c80a700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c780c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7d00b700 1 -- 172.21.15.50:0/3356713625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 msgr2=0x7f6c780c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:23.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7d00b700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c780c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7d00b700 1 -- 172.21.15.50:0/3356713625 >> v1:172.21.15.125:6789/0 conn(0x7f6c7806c7c0 legacy=0x7f6c7806cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7d00b700 1 -- 172.21.15.50:0/3356713625 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7806d790 con 0x7f6c7806ac30 2023-12-19T15:55:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7d00b700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c7806b030 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009f20 tx=0x7f6c70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca0b2e7a200de7e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c777fe700 1 -- 172.21.15.50:0/3356713625 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7000e070 con 0x7f6c7806ac30 2023-12-19T15:55:23.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c777fe700 1 -- 172.21.15.50:0/3356713625 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7000ba80 con 0x7f6c7806ac30 2023-12-19T15:55:23.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c777fe700 1 -- 172.21.15.50:0/3356713625 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c700092e0 con 0x7f6c7806ac30 2023-12-19T15:55:23.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/3356713625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 msgr2=0x7f6c7806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c7806b030 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009f20 tx=0x7f6c70009840 comp rx=0 tx=0).stop 2023-12-19T15:55:23.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/3356713625 shutdown_connections 2023-12-19T15:55:23.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c7806b030 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/3356713625 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c780c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/3356713625 >> 172.21.15.50:0/3356713625 conn(0x7f6c78067bb0 msgr2=0x7f6c780cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:23.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/3356713625 shutdown_connections 2023-12-19T15:55:23.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.951+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/3356713625 wait complete. 2023-12-19T15:55:23.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 Processor -- start 2023-12-19T15:55:23.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- start start 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 0x7f6c780c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c78148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7816d890 con 0x7f6c7806ac30 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7816cc80 con 0x7f6c7806c7c0 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7816cf60 con 0x7f6c780c6d20 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35888/0 (socket says 172.21.15.50:35888) 2023-12-19T15:55:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 -- 172.21.15.50:0/1719067337 learned_addr learned my addr 172.21.15.50:0/1719067337 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:23.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7d00b700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c78148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c77fff700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 0x7f6c780c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 -- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 msgr2=0x7f6c78148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c78148440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 -- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 msgr2=0x7f6c780c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 0x7f6c780c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 -- 172.21.15.50:0/1719067337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c70009ae0 con 0x7f6c7806ac30 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7d00b700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c78148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c77fff700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 0x7f6c780c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7c80a700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00d860 tx=0x7f6c6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c009610 con 0x7f6c7806ac30 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c009770 con 0x7f6c7806ac30 2023-12-19T15:55:23.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c009b30 con 0x7f6c7806ac30 2023-12-19T15:55:23.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7816d240 con 0x7f6c7806ac30 2023-12-19T15:55:23.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.955+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c780aa9b0 con 0x7f6c7806ac30 2023-12-19T15:55:23.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.959+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c78071120 con 0x7f6c7806ac30 2023-12-19T15:55:23.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.959+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6c6c01e460 con 0x7f6c7806ac30 2023-12-19T15:55:23.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.959+0000 7f6c74ff9700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 0x7f6c5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:23.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.959+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f6c6c07d0e0 con 0x7f6c7806ac30 2023-12-19T15:55:23.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.963+0000 7f6c77fff700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 0x7f6c5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:23.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.963+0000 7f6c77fff700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 0x7f6c5c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c781670a0 tx=0x7f6c64009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:23.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:23.967+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c6c04d440 con 0x7f6c7806ac30 2023-12-19T15:55:24.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.199+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6c7806da70 con 0x7f6c7806ac30 2023-12-19T15:55:24.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.207+0000 7f6c74ff9700 1 -- 172.21.15.50:0/1719067337 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6c6c050a60 con 0x7f6c7806ac30 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 msgr2=0x7f6c5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 0x7f6c5c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c781670a0 tx=0x7f6c64009040 comp rx=0 tx=0).stop 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 msgr2=0x7f6c78159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00d860 tx=0x7f6c6c008040 comp rx=0 tx=0).stop 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 shutdown_connections 2023-12-19T15:55:24.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6c7806c7c0 0x7f6c780c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f6c5c060670 0x7f6c5c062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6c780c6d20 0x7f6c78148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 --2- 172.21.15.50:0/1719067337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6c7806ac30 0x7f6c78159770 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 >> 172.21.15.50:0/1719067337 conn(0x7f6c78067bb0 msgr2=0x7f6c781579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 shutdown_connections 2023-12-19T15:55:24.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.211+0000 7f6c7ea6e700 1 -- 172.21.15.50:0/1719067337 wait complete. 2023-12-19T15:55:24.216 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T15:55:24.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T15:55:24.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.395+0000 7f38ac385700 1 Processor -- start 2023-12-19T15:55:24.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.395+0000 7f38ac385700 1 -- start start 2023-12-19T15:55:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38ac385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f38a40ae510 0x7f38a40a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38ac385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38a40c1d60 0x7f38a40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38ac385700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a406dc20 con 0x7f38a40ae140 2023-12-19T15:55:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38ac385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a40c2200 con 0x7f38a40ae510 2023-12-19T15:55:24.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38ac385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a406d940 con 0x7f38a40c1d60 2023-12-19T15:55:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38aa121700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f38a40ae140 0x7f38a406d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47396/0 (socket says 172.21.15.50:47396) 2023-12-19T15:55:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38aa121700 1 -- 172.21.15.50:0/1739327579 learned_addr learned my addr 172.21.15.50:0/1739327579 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a9920700 1 --2- 172.21.15.50:0/1739327579 >> v2:172.21.15.50:3300/0 conn(0x7f38a40ae510 0x7f38a40a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38aa922700 1 --2- 172.21.15.50:0/1739327579 >> v2:172.21.15.125:3300/0 conn(0x7f38a40c1d60 0x7f38a40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a9920700 1 -- 172.21.15.50:0/1739327579 >> v2:172.21.15.125:3300/0 conn(0x7f38a40c1d60 msgr2=0x7f38a40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a9920700 1 --2- 172.21.15.50:0/1739327579 >> v2:172.21.15.125:3300/0 conn(0x7f38a40c1d60 0x7f38a40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a9920700 1 -- 172.21.15.50:0/1739327579 >> v1:172.21.15.50:6789/0 conn(0x7f38a40ae140 legacy=0x7f38a406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a9920700 1 -- 172.21.15.50:0/1739327579 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40c6e20 con 0x7f38a40ae510 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.399+0000 7f38a911f700 1 -- 172.21.15.50:0/1739327579 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943590419 0 0) 0x7f38a406dc20 con 0x7f38a40ae140 2023-12-19T15:55:24.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38a9920700 1 --2- 172.21.15.50:0/1739327579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae510 0x7f38a40a8850 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f38a40c23b0 tx=0x7f38940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95de5c409b7e116a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:24.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38a911f700 1 -- 172.21.15.50:0/1739327579 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389400b6b0 con 0x7f38a40ae510 2023-12-19T15:55:24.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38a911f700 1 -- 172.21.15.50:0/1739327579 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389400b810 con 0x7f38a40ae510 2023-12-19T15:55:24.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38a911f700 1 -- 172.21.15.50:0/1739327579 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389400bbb0 con 0x7f38a40ae510 2023-12-19T15:55:24.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 -- 172.21.15.50:0/1739327579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae510 msgr2=0x7f38a40a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 --2- 172.21.15.50:0/1739327579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae510 0x7f38a40a8850 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f38a40c23b0 tx=0x7f38940091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 -- 172.21.15.50:0/1739327579 shutdown_connections 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 --2- 172.21.15.50:0/1739327579 >> v2:172.21.15.125:3300/0 conn(0x7f38a40c1d60 0x7f38a40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 --2- 172.21.15.50:0/1739327579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae510 0x7f38a40a8850 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 -- 172.21.15.50:0/1739327579 >> 172.21.15.50:0/1739327579 conn(0x7f38a40c8850 msgr2=0x7f38a40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 -- 172.21.15.50:0/1739327579 shutdown_connections 2023-12-19T15:55:24.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 -- 172.21.15.50:0/1739327579 wait complete. 2023-12-19T15:55:24.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:55:24.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.403+0000 7f38ac385700 1 Processor -- start 2023-12-19T15:55:24.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:55:24.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- start start 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38a40ae510 0x7f38a4154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38a40c1d60 0x7f38a40670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a41557b0 con 0x7f38a40ae140 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40afad0 con 0x7f38a40c1d60 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40c61a0 con 0x7f38a40ae510 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35902/0 (socket says 172.21.15.50:35902) 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 -- 172.21.15.50:0/904828422 learned_addr learned my addr 172.21.15.50:0/904828422 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa922700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38a40c1d60 0x7f38a40670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38a9920700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38a40ae510 0x7f38a4154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 -- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38a40ae510 msgr2=0x7f38a4154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38a40ae510 0x7f38a4154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 -- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38a40c1d60 msgr2=0x7f38a40670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38a40c1d60 0x7f38a40670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 -- 172.21.15.50:0/904828422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894009420 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38aa121700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c810 tx=0x7f38a0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0005b30 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0005c90 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0007300 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a415acb0 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.407+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a415a660 con 0x7f38a40ae140 2023-12-19T15:55:24.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.411+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a4071120 con 0x7f38a40ae140 2023-12-19T15:55:24.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.411+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f38a001b3b0 con 0x7f38a40ae140 2023-12-19T15:55:24.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.411+0000 7f389a7fc700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 0x7f3888062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:24.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.415+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f38a007bf30 con 0x7f38a40ae140 2023-12-19T15:55:24.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.415+0000 7f38a9920700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 0x7f3888062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:24.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.415+0000 7f38a9920700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 0x7f3888062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38a40c23b0 tx=0x7f3894017000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:24.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.419+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38a004c290 con 0x7f38a40ae140 2023-12-19T15:55:24.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.635+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38a40642a0 con 0x7f38a40ae140 2023-12-19T15:55:24.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.635+0000 7f389a7fc700 1 -- 172.21.15.50:0/904828422 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f38a0017090 con 0x7f38a40ae140 2023-12-19T15:55:24.640 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T15:55:24.641 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T15:55:24.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 msgr2=0x7f3888062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 0x7f3888062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f38a40c23b0 tx=0x7f3894017000 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 msgr2=0x7f38a40af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c810 tx=0x7f38a0007df0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 shutdown_connections 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38a40c1d60 0x7f38a40670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3888060670 0x7f3888062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38a40ae510 0x7f38a4154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 --2- 172.21.15.50:0/904828422 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38a40ae140 0x7f38a40af590 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 >> 172.21.15.50:0/904828422 conn(0x7f38a40c8850 msgr2=0x7f38a4157120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 shutdown_connections 2023-12-19T15:55:24.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:24.639+0000 7f38ac385700 1 -- 172.21.15.50:0/904828422 wait complete. 2023-12-19T15:55:24.646 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T15:55:25.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T15:55:18.383203+0000', 'modified': '2023-12-19T15:55:21.981660+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9333}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9333': {'gid': 9333, 'name': 'c', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/1475106130', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 1475106130}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 1475106130}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T15:55:25.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T15:55:25.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T15:55:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.227+0000 7fbaee945700 1 Processor -- start 2023-12-19T15:55:25.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.227+0000 7fbaee945700 1 -- start start 2023-12-19T15:55:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbaee945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbaee945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae806b0d0 con 0x7fbae806dc00 2023-12-19T15:55:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbaee945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80b4790 con 0x7fbae80b3870 2023-12-19T15:55:25.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbaee945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae806d710 con 0x7fbae8158cc0 2023-12-19T15:55:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbadf7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbae806dc00 0x7fbae806b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47412/0 (socket says 172.21.15.50:47412) 2023-12-19T15:55:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbadf7fe700 1 -- 172.21.15.50:0/182094894 learned_addr learned my addr 172.21.15.50:0/182094894 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:25.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae7fff700 1 --2- 172.21.15.50:0/182094894 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829059133 0 0) 0x7fbae806b0d0 con 0x7fbae806dc00 2023-12-19T15:55:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad4003570 con 0x7fbae806dc00 2023-12-19T15:55:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675151672 0 0) 0x7fbae80b4790 con 0x7fbae80b3870 2023-12-19T15:55:25.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae7fff700 1 -- 172.21.15.50:0/182094894 >> v1:172.21.15.125:6789/0 conn(0x7fbae80b3870 legacy=0x7fbae80b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae7fff700 1 -- 172.21.15.50:0/182094894 >> v1:172.21.15.50:6789/0 conn(0x7fbae806dc00 legacy=0x7fbae806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.231+0000 7fbae7fff700 1 -- 172.21.15.50:0/182094894 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae806a7e0 con 0x7fbae8158cc0 2023-12-19T15:55:25.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbae7fff700 1 --2- 172.21.15.50:0/182094894 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae806d1d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009e90 tx=0x7fbad8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dbd90e3d9f25721 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad800e070 con 0x7fbae8158cc0 2023-12-19T15:55:25.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad800bc30 con 0x7fbae8158cc0 2023-12-19T15:55:25.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbae77fe700 1 -- 172.21.15.50:0/182094894 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8012430 con 0x7fbae8158cc0 2023-12-19T15:55:25.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 -- 172.21.15.50:0/182094894 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 msgr2=0x7fbae806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 --2- 172.21.15.50:0/182094894 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae806d1d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009e90 tx=0x7fbad8009780 comp rx=0 tx=0).stop 2023-12-19T15:55:25.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 -- 172.21.15.50:0/182094894 shutdown_connections 2023-12-19T15:55:25.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 --2- 172.21.15.50:0/182094894 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae806d1d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 -- 172.21.15.50:0/182094894 >> 172.21.15.50:0/182094894 conn(0x7fbae8067bb0 msgr2=0x7fbae80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 -- 172.21.15.50:0/182094894 shutdown_connections 2023-12-19T15:55:25.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.235+0000 7fbaee945700 1 -- 172.21.15.50:0/182094894 wait complete. 2023-12-19T15:55:25.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 Processor -- start 2023-12-19T15:55:25.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 -- start start 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae806dc00 0x7fbae8169260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae80ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae81697a0 con 0x7fbae80b3870 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80adc20 con 0x7fbae806dc00 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaee945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80acb80 con 0x7fbae8158cc0 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35918/0 (socket says 172.21.15.50:35918) 2023-12-19T15:55:25.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 -- 172.21.15.50:0/586077961 learned_addr learned my addr 172.21.15.50:0/586077961 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbae7fff700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae806dc00 0x7fbae8169260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaecee2700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae80ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 -- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 msgr2=0x7fbae80ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae80ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 -- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae806dc00 msgr2=0x7fbae8169260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae806dc00 0x7fbae8169260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 -- 172.21.15.50:0/586077961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009a20 con 0x7fbae80b3870 2023-12-19T15:55:25.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbaecee2700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae80ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:25.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbad00011e0 tx=0x7fbad0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.239+0000 7fbae4ff9700 1 -- 172.21.15.50:0/586077961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0005b30 con 0x7fbae80b3870 2023-12-19T15:55:25.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.243+0000 7fbae4ff9700 1 -- 172.21.15.50:0/586077961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad0005c90 con 0x7fbae80b3870 2023-12-19T15:55:25.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.243+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae80ace60 con 0x7fbae80b3870 2023-12-19T15:55:25.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.243+0000 7fbae4ff9700 1 -- 172.21.15.50:0/586077961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00072e0 con 0x7fbae80b3870 2023-12-19T15:55:25.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.243+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae80ad420 con 0x7fbae80b3870 2023-12-19T15:55:25.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbae4ff9700 1 -- 172.21.15.50:0/586077961 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbad0007650 con 0x7fbae80b3870 2023-12-19T15:55:25.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbae4ff9700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 0x7fbac8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbae4ff9700 1 -- 172.21.15.50:0/586077961 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fbad007bfd0 con 0x7fbae80b3870 2023-12-19T15:55:25.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbae7fff700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 0x7fbac8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbaf0c1b740 1 Processor -- start 2023-12-19T15:55:25.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.247+0000 7fbaf0c1b740 1 -- start start 2023-12-19T15:55:25.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbae7fff700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 0x7fbac8062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbad80099f0 tx=0x7fbad8019000 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25eff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25f0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x251df30 con 0x23bd790 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x2519ed0 con 0x2372ae0 2023-12-19T15:55:25.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x24e00b0 con 0x2372710 2023-12-19T15:55:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35920/0 (socket says 172.21.15.50:35920) 2023-12-19T15:55:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 -- 172.21.15.50:0/1229374644 learned_addr learned my addr 172.21.15.50:0/1229374644 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbae7fff700 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 -- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 msgr2=0x25f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 -- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 msgr2=0x25eff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25eff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 -- 172.21.15.50:0/1229374644 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23be9e0 con 0x23bd790 2023-12-19T15:55:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbae0001770 tx=0x7fbae0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3022a73da59984f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbadd7fa700 1 -- 172.21.15.50:0/1229374644 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000e600 con 0x23bd790 2023-12-19T15:55:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbadd7fa700 1 -- 172.21.15.50:0/1229374644 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae000e760 con 0x23bd790 2023-12-19T15:55:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.251+0000 7fbadd7fa700 1 -- 172.21.15.50:0/1229374644 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000eae0 con 0x23bd790 2023-12-19T15:55:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1229374644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 msgr2=0x25f0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbae0001770 tx=0x7fbae0009040 comp rx=0 tx=0).stop 2023-12-19T15:55:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1229374644 shutdown_connections 2023-12-19T15:55:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25f0980 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1229374644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1229374644 >> 172.21.15.50:0/1229374644 conn(0x2372eb0 msgr2=0x25ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1229374644 shutdown_connections 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1229374644 wait complete. 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- start start 2023-12-19T15:55:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 Processor -- start 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372710 0x25f4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 0x25f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x24dfd50 con 0x23bd790 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x24b8970 con 0x2372710 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x251a770 con 0x2372ae0 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.255+0000 7fbaf0c1b740 10 client.0 _ll_register_callbacks cb 0x25636a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T15:55:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35932/0 (socket says 172.21.15.50:35932) 2023-12-19T15:55:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 -- 172.21.15.50:0/3005510242 learned_addr learned my addr 172.21.15.50:0/3005510242 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372710 0x25f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 0x25f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 -- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 msgr2=0x25f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 0x25f4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 -- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372710 msgr2=0x25f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372710 0x25f4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 -- 172.21.15.50:0/3005510242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae000f040 con 0x23bd790 2023-12-19T15:55:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 0x25f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbae0001480 tx=0x7fbae0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97f0a4880ffd41a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae000eae0 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fbae0014480 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x24b8cc0 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae0014850 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 10 client.9465 fetch_fsmap learned FSMap version 82 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x24e8370 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x24b8cc0 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 82) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fbae00149b0 con 0x23bd790 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 10 client.9465 fetch_fsmap finished waiting for FSMap version 82 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 10 client.9465 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 20 client.9465 populate_metadata read hostname 'smithi050' 2023-12-19T15:55:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x25d7370 0x25ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 4 client.9465 mds_command: new command op to 9333 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T15:55:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.259+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 --> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x24bc3e0 con 0x25d7370 2023-12-19T15:55:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x25d7370 0x25ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x25d7370 0x25ff6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456cff7b583f185d server_cookie=e1601af4d396a17d in_seq=0 out_seq=0 2023-12-19T15:55:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbac67fc700 10 client.9465 ms_handle_connect on v2:172.21.15.50:6834/1475106130 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbac67fc700 1 -- 172.21.15.50:0/3005510242 <== mds.0 v2:172.21.15.50:6834/1475106130 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x24bc3e0 con 0x25d7370 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbac67fc700 10 client.9465 handle_command_reply: tid=0 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 client.9465 shutdown 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 msgr2=0x25d5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbae0001480 tx=0x7fbae0017000 comp rx=0 tx=0).stop 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 shutdown_connections 2023-12-19T15:55:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x25d7370 0x25ff6b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d5cb0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372ae0 0x25f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbac67fc700 0 client.9465 ms_handle_reset on v2:172.21.15.50:6834/1475106130 2023-12-19T15:55:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3005510242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372710 0x25f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 >> 172.21.15.50:0/3005510242 conn(0x2372eb0 msgr2=0x25a4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 shutdown_connections 2023-12-19T15:55:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3005510242 wait complete. 2023-12-19T15:55:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.263+0000 7fbaf0c1b740 20 client.9465 trim_cache size 0 max 16384 2023-12-19T15:55:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.283+0000 7fbaf0c1b740 1 Processor -- start 2023-12-19T15:55:25.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.283+0000 7fbaf0c1b740 1 -- start start 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.283+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x26454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.283+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x2645a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x23eb7c0 con 0x23bd790 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x251eef0 con 0x2372ae0 2023-12-19T15:55:25.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x251d9d0 con 0x2372710 2023-12-19T15:55:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbadf7fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x2645a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35938/0 (socket says 172.21.15.50:35938) 2023-12-19T15:55:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbae7fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x26454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 -- 172.21.15.50:0/1454269770 learned_addr learned my addr 172.21.15.50:0/1454269770 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 -- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 msgr2=0x26454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x26454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 -- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 msgr2=0x2645a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x2645a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 -- 172.21.15.50:0/1454269770 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24ced80 con 0x23bd790 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x2645a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:25.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaecee2700 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000ec0 tx=0x7fbae0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db221d0dccd722e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaddffb700 1 -- 172.21.15.50:0/1454269770 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0014480 con 0x23bd790 2023-12-19T15:55:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaddffb700 1 -- 172.21.15.50:0/1454269770 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae000e3f0 con 0x23bd790 2023-12-19T15:55:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaddffb700 1 -- 172.21.15.50:0/1454269770 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0014850 con 0x23bd790 2023-12-19T15:55:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1454269770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 msgr2=0x2645f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000ec0 tx=0x7fbae0021000 comp rx=0 tx=0).stop 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1454269770 shutdown_connections 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x2645a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x2645f70 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/1454269770 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x26454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.287+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1454269770 >> 172.21.15.50:0/1454269770 conn(0x2372eb0 msgr2=0x25d5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1454269770 shutdown_connections 2023-12-19T15:55:25.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/1454269770 wait complete. 2023-12-19T15:55:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- start start 2023-12-19T15:55:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 Processor -- start 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x24ee610 con 0x23bd790 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x23bdea0 con 0x2372ae0 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x23ef650 con 0x2372710 2023-12-19T15:55:25.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaf0c1b740 10 client.0 _ll_register_callbacks cb 0x233e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35940/0 (socket says 172.21.15.50:35940) 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 -- 172.21.15.50:0/3405296903 learned_addr learned my addr 172.21.15.50:0/3405296903 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbadf7fe700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 -- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 msgr2=0x25d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25d0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 -- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 msgr2=0x25d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25d1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.291+0000 7fbaecee2700 1 -- 172.21.15.50:0/3405296903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae80ad420 con 0x23bd790 2023-12-19T15:55:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaecee2700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000f60 tx=0x7fbae0032040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f25f735ceb72f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0014480 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fbae0014850 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae000e760 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae0007df0 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 10 client.9471 fetch_fsmap learned FSMap version 82 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x24e8370 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae000e760 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 82) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fbae0039460 con 0x23bd790 2023-12-19T15:55:25.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 10 client.9471 fetch_fsmap finished waiting for FSMap version 82 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 10 client.9471 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 20 client.9471 populate_metadata read hostname 'smithi050' 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x264f770 0x2651b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 4 client.9471 mds_command: new command op to 9333 tid=0[{"prefix": "status"}] 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 --> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] -- command(tid 0: {"prefix": "status"}) v1 -- 0x24ee410 con 0x264f770 2023-12-19T15:55:25.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x264f770 0x2651b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 10 client.9471 ms_handle_connect on v2:172.21.15.50:6834/1475106130 2023-12-19T15:55:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbae7fff700 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x264f770 0x2651b50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=127b54d2f40ed29a server_cookie=7003012cea7a1926 in_seq=0 out_seq=0 2023-12-19T15:55:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 1 -- 172.21.15.50:0/3405296903 <== mds.0 v2:172.21.15.50:6834/1475106130 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x24ee410 con 0x264f770 2023-12-19T15:55:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 10 client.9471 handle_command_reply: tid=0 2023-12-19T15:55:25.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 client.9471 shutdown 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 msgr2=0x25d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000f60 tx=0x7fbae0032040 comp rx=0 tx=0).stop 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 shutdown_connections 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:6834/1475106130,v1:172.21.15.50:6835/1475106130] conn(0x264f770 0x2651b50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x2372ae0 0x25d1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x23bd790 0x25d1570 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 --2- 172.21.15.50:0/3405296903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x2372710 0x25d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 >> 172.21.15.50:0/3405296903 conn(0x2372eb0 msgr2=0x25a4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.295+0000 7fbac67fc700 0 client.9471 ms_handle_reset on v2:172.21.15.50:6834/1475106130 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 shutdown_connections 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaf0c1b740 1 -- 172.21.15.50:0/3405296903 wait complete. 2023-12-19T15:55:25.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaf0c1b740 20 client.9471 trim_cache size 0 max 16384 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 9333, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.344444642, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 82, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 181, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 181, 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.270023784999999 2023-12-19T15:55:25.303 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:55:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 msgr2=0x7fbac8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 0x7fbac8062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbad80099f0 tx=0x7fbad8019000 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 msgr2=0x7fbae80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fbad00011e0 tx=0x7fbad0007df0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 shutdown_connections 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbac8060670 0x7fbac8062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbae8158cc0 0x7fbae80ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbae80b3870 0x7fbae80b2160 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 --2- 172.21.15.50:0/586077961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbae806dc00 0x7fbae8169260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 >> 172.21.15.50:0/586077961 conn(0x7fbae8067bb0 msgr2=0x7fbae80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 shutdown_connections 2023-12-19T15:55:25.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.299+0000 7fbaee945700 1 -- 172.21.15.50:0/586077961 wait complete. 2023-12-19T15:55:25.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T15:55:25.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.487+0000 7f35aae6e700 1 Processor -- start 2023-12-19T15:55:25.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.487+0000 7f35aae6e700 1 -- start start 2023-12-19T15:55:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35aae6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35a4067c90 0x7f35a4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35aae6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35aae6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40c6630 con 0x7f35a414fba0 2023-12-19T15:55:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35aae6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40c6910 con 0x7f35a4067c90 2023-12-19T15:55:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35aae6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40c6bf0 con 0x7f35a40c5900 2023-12-19T15:55:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35a8c0a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35a4067c90 0x7f35a4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35a8c0a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35a4067c90 0x7f35a4068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35950/0 (socket says 172.21.15.50:35950) 2023-12-19T15:55:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35a8c0a700 1 -- 172.21.15.50:0/2130859225 learned_addr learned my addr 172.21.15.50:0/2130859225 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35a3fff700 1 --1- 172.21.15.50:0/2130859225 >> v1:172.21.15.50:6789/0 conn(0x7f35a414fba0 0x7f35a4150000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47424/0 (socket says 172.21.15.50:47424) 2023-12-19T15:55:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.491+0000 7f35a940b700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a8c0a700 1 -- 172.21.15.50:0/2130859225 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 msgr2=0x7f35a40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a8c0a700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a8c0a700 1 -- 172.21.15.50:0/2130859225 >> v1:172.21.15.50:6789/0 conn(0x7f35a414fba0 legacy=0x7f35a4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a8c0a700 1 -- 172.21.15.50:0/2130859225 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40c6ed0 con 0x7f35a4067c90 2023-12-19T15:55:25.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a940b700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a8c0a700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a4068060 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f35980099b0 tx=0x7f3598009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bbfef319d1dc84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a37fe700 1 -- 172.21.15.50:0/2130859225 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598009ea0 con 0x7f35a4067c90 2023-12-19T15:55:25.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a37fe700 1 -- 172.21.15.50:0/2130859225 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800b860 con 0x7f35a4067c90 2023-12-19T15:55:25.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35a37fe700 1 -- 172.21.15.50:0/2130859225 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800bc00 con 0x7f35a4067c90 2023-12-19T15:55:25.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 -- 172.21.15.50:0/2130859225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 msgr2=0x7f35a4068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a4068060 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f35980099b0 tx=0x7f3598009250 comp rx=0 tx=0).stop 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 -- 172.21.15.50:0/2130859225 shutdown_connections 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 --2- 172.21.15.50:0/2130859225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a4068060 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 -- 172.21.15.50:0/2130859225 >> 172.21.15.50:0/2130859225 conn(0x7f35a40cd6d0 msgr2=0x7f35a406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 -- 172.21.15.50:0/2130859225 shutdown_connections 2023-12-19T15:55:25.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.495+0000 7f35aae6e700 1 -- 172.21.15.50:0/2130859225 wait complete. 2023-12-19T15:55:25.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.499+0000 7f35aae6e700 1 Processor -- start 2023-12-19T15:55:25.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.499+0000 7f35aae6e700 1 -- start start 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35a414fba0 0x7f35a40c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a41697f0 con 0x7f35a4067c90 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a416a030 con 0x7f35a414fba0 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40ca6e0 con 0x7f35a40c5900 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35962/0 (socket says 172.21.15.50:35962) 2023-12-19T15:55:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 -- 172.21.15.50:0/4228710010 learned_addr learned my addr 172.21.15.50:0/4228710010 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a3fff700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a940b700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35a414fba0 0x7f35a40c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 -- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 msgr2=0x7f35a40ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 -- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35a414fba0 msgr2=0x7f35a40c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35a414fba0 0x7f35a40c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 -- 172.21.15.50:0/4228710010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c009a20 con 0x7f35a4067c90 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a8c0a700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3598006fd0 tx=0x7f3598017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598000d10 con 0x7f35a4067c90 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800bbe0 con 0x7f35a4067c90 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598011650 con 0x7f35a4067c90 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35980094f0 con 0x7f35a4067c90 2023-12-19T15:55:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.503+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a41699e0 con 0x7f35a4067c90 2023-12-19T15:55:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.507+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a4071120 con 0x7f35a4067c90 2023-12-19T15:55:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.507+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3598011a80 con 0x7f35a4067c90 2023-12-19T15:55:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.511+0000 7f35a0ff9700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 0x7f3588062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.511+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f359807df60 con 0x7f35a4067c90 2023-12-19T15:55:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.511+0000 7f35a3fff700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 0x7f3588062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:25.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.511+0000 7f35a3fff700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 0x7f3588062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f35a416a970 tx=0x7f3590007eb0 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:25.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.515+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f359804e360 con 0x7f35a4067c90 2023-12-19T15:55:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35a40642a0 con 0x7f35a4067c90 2023-12-19T15:55:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35a0ff9700 1 -- 172.21.15.50:0/4228710010 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3598028060 con 0x7f35a4067c90 2023-12-19T15:55:25.734 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T15:55:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 msgr2=0x7f3588062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 0x7f3588062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f35a416a970 tx=0x7f3590007eb0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 msgr2=0x7f35a40c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.731+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3598006fd0 tx=0x7f3598017000 comp rx=0 tx=0).stop 2023-12-19T15:55:25.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 shutdown_connections 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35a414fba0 0x7f35a40c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f3588060670 0x7f3588062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35a40c5900 0x7f35a40ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 --2- 172.21.15.50:0/4228710010 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35a4067c90 0x7f35a40c72a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 >> 172.21.15.50:0/4228710010 conn(0x7f35a40cd6d0 msgr2=0x7f35a40b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 shutdown_connections 2023-12-19T15:55:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:25.735+0000 7f35aae6e700 1 -- 172.21.15.50:0/4228710010 wait complete. 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T15:55:25.757 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:55:25.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:25.758 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T15:55:25.774 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 37824sec preferred_lft 37824sec 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T15:55:25.775 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T15:55:25.776 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T15:55:25.777 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T15:55:25.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:25.779 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T15:55:25.859 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T15:55:25.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:25.862 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T15:55:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T15:55:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T15:55:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T15:55:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T15:55:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T15:55:25.932 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T15:55:25.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T15:55:25.932 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T15:55:26.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:26.080 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T15:55:26.099 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T15:55:26.100 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T15:55:26.100 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:55:26.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T15:55:26.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T15:55:26.178 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T15:55:26.178 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T15:55:26.178 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:26.240 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T15:55:26.240 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T15:55:26.240 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:26.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T15:55:26.390 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T15:55:26.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T15:55:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T15:55:26.476 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T15:55:26.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:26.476 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T15:55:26.534 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T15:55:26.534 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T15:55:26.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T15:55:26.617 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T15:55:26.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T15:55:26.695 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T15:55:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T15:55:26.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:26.699 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T15:55:26.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T15:55:26.727+0000 7f453edcd380 -1 init, newargv = 0x5592207e0270 newargc=15 2023-12-19T15:55:26.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[58286]: starting ceph client 2023-12-19T15:55:26.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[58286]: starting fuse 2023-12-19T15:55:26.748 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T15:55:26.748 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T15:55:26.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T15:55:26.749 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T15:55:26.750 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T15:55:26.916 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.58286.asok 2023-12-19T15:55:27.102 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58286.asok 2023-12-19T15:55:27.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:27.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58286.asok status 2023-12-19T15:55:27.293 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T15:55:27.293 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T15:55:27.293 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "58286", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 9486, 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T15:55:27.294 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 9486 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3170185718 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3170185718 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.9486 192.168.0.1:0/3170185718", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/3170185718", 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T15:55:27.295 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 82, 2023-12-19T15:55:27.296 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 181, 2023-12-19T15:55:27.296 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T15:55:27.296 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T15:55:27.296 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T15:55:27.296 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T15:55:27.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:27.308 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T15:55:27.308 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:27.331 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T15:55:27.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:27.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:27.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T15:55:27.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T15:55:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.623+0000 7fd9af478700 1 Processor -- start 2023-12-19T15:55:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.623+0000 7fd9af478700 1 -- start start 2023-12-19T15:55:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9af478700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd9a815d050 0x7fd9a806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9af478700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a806ba50 con 0x7fd9a8160e60 2023-12-19T15:55:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9af478700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a80af600 con 0x7fd9a80afef0 2023-12-19T15:55:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9af478700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a806bee0 con 0x7fd9a815d050 2023-12-19T15:55:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd9a815d050 0x7fd9a806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd9a815d050 0x7fd9a806a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43928/0 (socket says 172.21.15.50:43928) 2023-12-19T15:55:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 -- 172.21.15.50:0/2354919077 learned_addr learned my addr 172.21.15.50:0/2354919077 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 -- 172.21.15.50:0/2354919077 >> v1:172.21.15.125:6789/0 conn(0x7fd9a80afef0 legacy=0x7fd9a815d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 -- 172.21.15.50:0/2354919077 >> v1:172.21.15.50:6789/0 conn(0x7fd9a8160e60 legacy=0x7fd9a80af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 -- 172.21.15.50:0/2354919077 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a806ae40 con 0x7fd9a815d050 2023-12-19T15:55:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd9ad214700 1 --2- 172.21.15.50:0/2354919077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a815d050 0x7fd9a806a740 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40099b0 tx=0x7fd9a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0c4523f8efc294 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd99ffff700 1 -- 172.21.15.50:0/2354919077 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400b860 con 0x7fd9a815d050 2023-12-19T15:55:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd99ffff700 1 -- 172.21.15.50:0/2354919077 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a400b9c0 con 0x7fd9a815d050 2023-12-19T15:55:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.627+0000 7fd99ffff700 1 -- 172.21.15.50:0/2354919077 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400bd60 con 0x7fd9a815d050 2023-12-19T15:55:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- 172.21.15.50:0/2354919077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a815d050 msgr2=0x7fd9a806a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 --2- 172.21.15.50:0/2354919077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a815d050 0x7fd9a806a740 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40099b0 tx=0x7fd9a4009250 comp rx=0 tx=0).stop 2023-12-19T15:55:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- 172.21.15.50:0/2354919077 shutdown_connections 2023-12-19T15:55:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 --2- 172.21.15.50:0/2354919077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a815d050 0x7fd9a806a740 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- 172.21.15.50:0/2354919077 >> 172.21.15.50:0/2354919077 conn(0x7fd9a8067b60 msgr2=0x7fd9a80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- 172.21.15.50:0/2354919077 shutdown_connections 2023-12-19T15:55:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- 172.21.15.50:0/2354919077 wait complete. 2023-12-19T15:55:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 Processor -- start 2023-12-19T15:55:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.631+0000 7fd9af478700 1 -- start start 2023-12-19T15:55:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 0x7fd9a80c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 0x7fd9a806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c1980 con 0x7fd9a80afef0 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c0d70 con 0x7fd9a815d050 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80c1050 con 0x7fd9a8160e60 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43938/0 (socket says 172.21.15.50:43938) 2023-12-19T15:55:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 -- 172.21.15.50:0/1257857259 learned_addr learned my addr 172.21.15.50:0/1257857259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9aca13700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 0x7fd9a80c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ada15700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 0x7fd9a806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 -- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 msgr2=0x7fd9a806ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 0x7fd9a806ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 -- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 msgr2=0x7fd9a80c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 0x7fd9a80c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 -- 172.21.15.50:0/1257857259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40094f0 con 0x7fd9a80afef0 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ada15700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 0x7fd9a806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9aca13700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 0x7fd9a80c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9ad214700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80af790 tx=0x7fd9a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4011ea0 con 0x7fd9a80afef0 2023-12-19T15:55:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a401a630 con 0x7fd9a80afef0 2023-12-19T15:55:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a401a9b0 con 0x7fd9a80afef0 2023-12-19T15:55:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.635+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a80c1330 con 0x7fd9a80afef0 2023-12-19T15:55:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.639+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a81481e0 con 0x7fd9a80afef0 2023-12-19T15:55:27.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.639+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a8071120 con 0x7fd9a80afef0 2023-12-19T15:55:27.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.639+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd9a402d440 con 0x7fd9a80afef0 2023-12-19T15:55:27.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.643+0000 7fd99d7fa700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 0x7fd98c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.643+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fd9a407e620 con 0x7fd9a80afef0 2023-12-19T15:55:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.643+0000 7fd9aca13700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 0x7fd98c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:27.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.643+0000 7fd9aca13700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 0x7fd98c062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd998007fd0 tx=0x7fd998009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.647+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd9a404e980 con 0x7fd9a80afef0 2023-12-19T15:55:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.839+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9a80c1ac0 con 0x7fd98c0606a0 2023-12-19T15:55:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.843+0000 7fd99d7fa700 1 -- 172.21.15.50:0/1257857259 <== mgr.8968 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd998001c80 con 0x7fd98c0606a0 2023-12-19T15:55:27.846 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T15:55:27.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T15:55:27.846 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T15:55:27.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T15:55:27.846 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T15:55:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 msgr2=0x7fd98c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 0x7fd98c062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd998007fd0 tx=0x7fd998009040 comp rx=0 tx=0).stop 2023-12-19T15:55:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 msgr2=0x7fd9a8147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:27.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80af790 tx=0x7fd9a4017000 comp rx=0 tx=0).stop 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 shutdown_connections 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a815d050 0x7fd9a80c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fd98c0606a0 0x7fd98c062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd9a8160e60 0x7fd9a806ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 --2- 172.21.15.50:0/1257857259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9a80afef0 0x7fd9a8147ca0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 >> 172.21.15.50:0/1257857259 conn(0x7fd9a8067b60 msgr2=0x7fd9a815eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 shutdown_connections 2023-12-19T15:55:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:27.847+0000 7fd9af478700 1 -- 172.21.15.50:0/1257857259 wait complete. 2023-12-19T15:55:27.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T15:55:28.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 Processor -- start 2023-12-19T15:55:28.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 -- start start 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 0x7f8e2006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200cbdf0 con 0x7f8e200cb970 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200cc0d0 con 0x7f8e20158e20 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.039+0000 7f8e254cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200cc3b0 con 0x7f8e200cecd0 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1e7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8e200cb970 0x7f8e2006acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56754/0 (socket says 172.21.15.50:56754) 2023-12-19T15:55:28.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1e7fc700 1 -- 172.21.15.50:0/2657825549 learned_addr learned my addr 172.21.15.50:0/2657825549 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1f7fe700 1 --2- 172.21.15.50:0/2657825549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 0x7f8e2006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1dffb700 1 -- 172.21.15.50:0/2657825549 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168548266 0 0) 0x7f8e200cbdf0 con 0x7f8e200cb970 2023-12-19T15:55:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1dffb700 1 -- 172.21.15.50:0/2657825549 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e08003570 con 0x7f8e200cb970 2023-12-19T15:55:28.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1f7fe700 1 -- 172.21.15.50:0/2657825549 >> v1:172.21.15.125:6789/0 conn(0x7f8e20158e20 legacy=0x7f8e200ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1f7fe700 1 -- 172.21.15.50:0/2657825549 >> v1:172.21.15.50:6789/0 conn(0x7f8e200cb970 legacy=0x7f8e2006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1f7fe700 1 -- 172.21.15.50:0/2657825549 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200cc690 con 0x7f8e200cecd0 2023-12-19T15:55:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1f7fe700 1 --2- 172.21.15.50:0/2657825549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 0x7f8e2006b3d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009ec0 tx=0x7f8e18009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce3f62740de2cdd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:28.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1dffb700 1 -- 172.21.15.50:0/2657825549 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1800e070 con 0x7f8e200cecd0 2023-12-19T15:55:28.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1dffb700 1 -- 172.21.15.50:0/2657825549 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1800bbf0 con 0x7f8e200cecd0 2023-12-19T15:55:28.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e1dffb700 1 -- 172.21.15.50:0/2657825549 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18012430 con 0x7f8e200cecd0 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e254cd700 1 -- 172.21.15.50:0/2657825549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 msgr2=0x7f8e2006b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.043+0000 7f8e254cd700 1 --2- 172.21.15.50:0/2657825549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 0x7f8e2006b3d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009ec0 tx=0x7f8e18009780 comp rx=0 tx=0).stop 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 -- 172.21.15.50:0/2657825549 shutdown_connections 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 --2- 172.21.15.50:0/2657825549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e200cecd0 0x7f8e2006b3d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 -- 172.21.15.50:0/2657825549 >> 172.21.15.50:0/2657825549 conn(0x7f8e20067bb0 msgr2=0x7f8e200d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 -- 172.21.15.50:0/2657825549 shutdown_connections 2023-12-19T15:55:28.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 -- 172.21.15.50:0/2657825549 wait complete. 2023-12-19T15:55:28.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 Processor -- start 2023-12-19T15:55:28.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.047+0000 7f8e254cd700 1 -- start start 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e20158e20 0x7f8e2016e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e20154bc0 0x7f8e2014c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2014cf30 con 0x7f8e200cecd0 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200b1060 con 0x7f8e20158e20 2023-12-19T15:55:28.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e201550c0 con 0x7f8e20154bc0 2023-12-19T15:55:28.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:28.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43942/0 (socket says 172.21.15.50:43942) 2023-12-19T15:55:28.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 -- 172.21.15.50:0/1528711076 learned_addr learned my addr 172.21.15.50:0/1528711076 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:28.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1f7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e20154bc0 0x7f8e2014c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:28.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1e7fc700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e20158e20 0x7f8e2016e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 -- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e20154bc0 msgr2=0x7f8e2014c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e20154bc0 0x7f8e2014c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 -- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e20158e20 msgr2=0x7f8e2016e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e20158e20 0x7f8e2016e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 -- 172.21.15.50:0/1528711076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18009a20 con 0x7f8e200cecd0 2023-12-19T15:55:28.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e1effd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000c4e0 tx=0x7f8e1000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:28.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011590 con 0x7f8e200cecd0 2023-12-19T15:55:28.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10011710 con 0x7f8e200cecd0 2023-12-19T15:55:28.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e200b1320 con 0x7f8e200cecd0 2023-12-19T15:55:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.051+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011ad0 con 0x7f8e200cecd0 2023-12-19T15:55:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.055+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e200b18e0 con 0x7f8e200cecd0 2023-12-19T15:55:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.055+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e200b1b30 con 0x7f8e200cecd0 2023-12-19T15:55:28.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.055+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 76) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8e1001b3b0 con 0x7f8e200cecd0 2023-12-19T15:55:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.059+0000 7f8e06ffd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 0x7f8e0c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.059+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f8e1007bec0 con 0x7f8e200cecd0 2023-12-19T15:55:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.059+0000 7f8e1e7fc700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 0x7f8e0c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.059+0000 7f8e1e7fc700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 0x7f8e0c062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e200ca290 tx=0x7f8e14009040 comp rx=0 tx=0).ready entity=mgr.8968 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:28.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.067+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e1004c180 con 0x7f8e200cecd0 2023-12-19T15:55:28.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:28.275+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f8e20071120 con 0x7f8e200cecd0 2023-12-19T15:55:29.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.247+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 77) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f8e10080070 con 0x7f8e200cecd0 2023-12-19T15:55:29.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.247+0000 7f8e06ffd700 1 -- 172.21.15.50:0/1528711076 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v77) v1 ==== 92+0+0 (secure 0 0 0) 0x7f8e1004fc10 con 0x7f8e200cecd0 2023-12-19T15:55:29.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 msgr2=0x7f8e0c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 0x7f8e0c062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e200ca290 tx=0x7f8e14009040 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 msgr2=0x7f8e20150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000c4e0 tx=0x7f8e1000c7f0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 shutdown_connections 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e20158e20 0x7f8e2016e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f8e0c0606a0 0x7f8e0c062b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8e20154bc0 0x7f8e2014c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 --2- 172.21.15.50:0/1528711076 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e200cecd0 0x7f8e20150930 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 >> 172.21.15.50:0/1528711076 conn(0x7f8e20067bb0 msgr2=0x7f8e200cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 shutdown_connections 2023-12-19T15:55:29.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.251+0000 7f8e254cd700 1 -- 172.21.15.50:0/1528711076 wait complete. 2023-12-19T15:55:29.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T15:55:29.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.439+0000 7f38be723700 1 Processor -- start 2023-12-19T15:55:29.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.439+0000 7f38be723700 1 -- start start 2023-12-19T15:55:29.447 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:29.438+0000 7f3f334ea000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38be723700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f38b806aeb0 0x7f38b8159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38be723700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b806c850 con 0x7f38b81595a0 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38be723700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80cb650 con 0x7f38b8158c90 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38be723700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b806b470 con 0x7f38b806aeb0 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bdf22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f38b81595a0 0x7f38b806cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56756/0 (socket says 172.21.15.50:56756) 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bdf22700 1 -- 172.21.15.50:0/1195449632 learned_addr learned my addr 172.21.15.50:0/1195449632 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bcf20700 1 --2- 172.21.15.50:0/1195449632 >> v2:172.21.15.125:3300/0 conn(0x7f38b806aeb0 0x7f38b8159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38affff700 1 -- 172.21.15.50:0/1195449632 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793636885 0 0) 0x7f38b806c850 con 0x7f38b81595a0 2023-12-19T15:55:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38affff700 1 -- 172.21.15.50:0/1195449632 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a4003570 con 0x7f38b81595a0 2023-12-19T15:55:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bcf20700 1 -- 172.21.15.50:0/1195449632 >> v1:172.21.15.125:6789/0 conn(0x7f38b8158c90 legacy=0x7f38b80caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bcf20700 1 -- 172.21.15.50:0/1195449632 >> v1:172.21.15.50:6789/0 conn(0x7f38b81595a0 legacy=0x7f38b806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bcf20700 1 -- 172.21.15.50:0/1195449632 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b806b750 con 0x7f38b806aeb0 2023-12-19T15:55:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38bcf20700 1 --2- 172.21.15.50:0/1195449632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b806aeb0 0x7f38b8159060 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f38a80098e0 tx=0x7f38a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4aafa94b7a97495 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:29.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38affff700 1 -- 172.21.15.50:0/1195449632 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800b640 con 0x7f38b806aeb0 2023-12-19T15:55:29.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38affff700 1 -- 172.21.15.50:0/1195449632 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a800b7a0 con 0x7f38b806aeb0 2023-12-19T15:55:29.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.443+0000 7f38affff700 1 -- 172.21.15.50:0/1195449632 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800bb40 con 0x7f38b806aeb0 2023-12-19T15:55:29.450 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:29.447+0000 7fdf94ffe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- 172.21.15.50:0/1195449632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b806aeb0 msgr2=0x7f38b8159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 --2- 172.21.15.50:0/1195449632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b806aeb0 0x7f38b8159060 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f38a80098e0 tx=0x7f38a80091a0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- 172.21.15.50:0/1195449632 shutdown_connections 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 --2- 172.21.15.50:0/1195449632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b806aeb0 0x7f38b8159060 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- 172.21.15.50:0/1195449632 >> 172.21.15.50:0/1195449632 conn(0x7f38b8067bb0 msgr2=0x7f38b80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- 172.21.15.50:0/1195449632 shutdown_connections 2023-12-19T15:55:29.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- 172.21.15.50:0/1195449632 wait complete. 2023-12-19T15:55:29.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 Processor -- start 2023-12-19T15:55:29.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.447+0000 7f38be723700 1 -- start start 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 0x7f38b80af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 0x7f38b81532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c4ce0 con 0x7f38b8158c90 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c9990 con 0x7f38b81595a0 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c4ec0 con 0x7f38b806aeb0 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43980/0 (socket says 172.21.15.50:43980) 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 -- 172.21.15.50:0/3863839951 learned_addr learned my addr 172.21.15.50:0/3863839951 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bdf22700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 0x7f38b81532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bd721700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 0x7f38b80af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 -- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 msgr2=0x7f38b80af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 0x7f38b80af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 -- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 msgr2=0x7f38b81532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 0x7f38b81532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 -- 172.21.15.50:0/3863839951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8009420 con 0x7f38b8158c90 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bd721700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 0x7f38b80af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bdf22700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 0x7f38b81532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38bcf20700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f38b8069b70 tx=0x7f38a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800bb20 con 0x7f38b8158c90 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a8011420 con 0x7f38b8158c90 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a80117a0 con 0x7f38b8158c90 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b80c51a0 con 0x7f38b8158c90 2023-12-19T15:55:29.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.451+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b8158390 con 0x7f38b8158c90 2023-12-19T15:55:29.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.455+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b804be40 con 0x7f38b8158c90 2023-12-19T15:55:29.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.455+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 77) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f38a8011b50 con 0x7f38b8158c90 2023-12-19T15:55:29.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.455+0000 7f38ad7fa700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 0x7f389c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.455+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f38a807e250 con 0x7f38b8158c90 2023-12-19T15:55:29.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.459+0000 7f38bd721700 1 -- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 msgr2=0x7f389c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:29.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.459+0000 7f38bd721700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 0x7f389c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:29.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.463+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38a804e5b0 con 0x7f38b8158c90 2023-12-19T15:55:29.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.643+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f38b806ba30 con 0x7f38b8158c90 2023-12-19T15:55:29.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.647+0000 7f38ad7fa700 1 -- 172.21.15.50:0/3863839951 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f38a8051bd0 con 0x7f38b8158c90 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 msgr2=0x7f389c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 0x7f389c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 msgr2=0x7f38b8154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f38b8069b70 tx=0x7f38a8017000 comp rx=0 tx=0).stop 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 shutdown_connections 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b81595a0 0x7f38b81532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f389c0606c0 0x7f389c062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38b8158c90 0x7f38b8154bf0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 --2- 172.21.15.50:0/3863839951 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f38b806aeb0 0x7f38b80af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 >> 172.21.15.50:0/3863839951 conn(0x7f38b8067bb0 msgr2=0x7f38b80aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 shutdown_connections 2023-12-19T15:55:29.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.651+0000 7f38be723700 1 -- 172.21.15.50:0/3863839951 wait complete. 2023-12-19T15:55:29.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T15:55:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 Processor -- start 2023-12-19T15:55:29.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 -- start start 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b8406dcf0 0x7f5b84159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406c850 con 0x7f5b84158c90 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b840cb650 con 0x7f5b8406aeb0 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b88984700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8406b470 con 0x7f5b8406dcf0 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b82d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5b84158c90 0x7f5b840caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56758/0 (socket says 172.21.15.50:56758) 2023-12-19T15:55:29.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b82d9d700 1 -- 172.21.15.50:0/2154473738 learned_addr learned my addr 172.21.15.50:0/2154473738 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T15:55:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b8359e700 1 --2- 172.21.15.50:0/2154473738 >> v2:172.21.15.125:3300/0 conn(0x7f5b8406dcf0 0x7f5b84159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b81d9b700 1 -- 172.21.15.50:0/2154473738 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387187529 0 0) 0x7f5b8406c850 con 0x7f5b84158c90 2023-12-19T15:55:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b81d9b700 1 -- 172.21.15.50:0/2154473738 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b6c003570 con 0x7f5b84158c90 2023-12-19T15:55:29.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b8359e700 1 -- 172.21.15.50:0/2154473738 >> v1:172.21.15.125:6789/0 conn(0x7f5b8406aeb0 legacy=0x7f5b84159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b8359e700 1 -- 172.21.15.50:0/2154473738 >> v1:172.21.15.50:6789/0 conn(0x7f5b84158c90 legacy=0x7f5b840caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.843+0000 7f5b8359e700 1 -- 172.21.15.50:0/2154473738 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8406b750 con 0x7f5b8406dcf0 2023-12-19T15:55:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b8359e700 1 --2- 172.21.15.50:0/2154473738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b8406dcf0 0x7f5b84159770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e60 tx=0x7f5b7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1abe04def79d60c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b81d9b700 1 -- 172.21.15.50:0/2154473738 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00e070 con 0x7f5b8406dcf0 2023-12-19T15:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b81d9b700 1 -- 172.21.15.50:0/2154473738 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c00bb60 con 0x7f5b8406dcf0 2023-12-19T15:55:29.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b81d9b700 1 -- 172.21.15.50:0/2154473738 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c012430 con 0x7f5b8406dcf0 2023-12-19T15:55:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 -- 172.21.15.50:0/2154473738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b8406dcf0 msgr2=0x7f5b84159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 --2- 172.21.15.50:0/2154473738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b8406dcf0 0x7f5b84159770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e60 tx=0x7f5b7c009780 comp rx=0 tx=0).stop 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 -- 172.21.15.50:0/2154473738 shutdown_connections 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 --2- 172.21.15.50:0/2154473738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b8406dcf0 0x7f5b84159770 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 -- 172.21.15.50:0/2154473738 >> 172.21.15.50:0/2154473738 conn(0x7f5b84067bb0 msgr2=0x7f5b840cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 -- 172.21.15.50:0/2154473738 shutdown_connections 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.847+0000 7f5b88984700 1 -- 172.21.15.50:0/2154473738 wait complete. 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 Processor -- start 2023-12-19T15:55:29.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- start start 2023-12-19T15:55:29.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5b8406aeb0 0x7f5b8415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b84158c90 0x7f5b840ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8415d2b0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b841693a0 con 0x7f5b84158c90 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8415c730 con 0x7f5b8406aeb0 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43984/0 (socket says 172.21.15.50:43984) 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 -- 172.21.15.50:0/2737453771 learned_addr learned my addr 172.21.15.50:0/2737453771 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b82d9d700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5b8406aeb0 0x7f5b8415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8359e700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b84158c90 0x7f5b840ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 -- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5b8406aeb0 msgr2=0x7f5b8415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5b8406aeb0 0x7f5b8415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 -- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b84158c90 msgr2=0x7f5b840ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:29.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b84158c90 0x7f5b840ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 -- 172.21.15.50:0/2737453771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c009a20 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b8259c700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c590 tx=0x7f5b7800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011590 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b780116f0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78011ab0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8415ca10 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.851+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8415d050 con 0x7f5b8406dcf0 2023-12-19T15:55:29.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8415c4d0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 77) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f5b7801b3b0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b6affd700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 0x7f5b70062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7f5b7807bfd0 con 0x7f5b8406dcf0 2023-12-19T15:55:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b82d9d700 1 -- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 msgr2=0x7f5b70062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.855+0000 7f5b82d9d700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 0x7f5b70062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:29.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:29.863+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5b7804c330 con 0x7f5b8406dcf0 2023-12-19T15:55:29.991 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:29.986+0000 7f3f334ea000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:55:30.025 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.019+0000 7fdf94ffe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T15:55:30.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.039+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5b84169580 con 0x7f5b8406dcf0 2023-12-19T15:55:30.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.051+0000 7f5b6affd700 1 -- 172.21.15.50:0/2737453771 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5b780170c0 con 0x7f5b8406dcf0 2023-12-19T15:55:30.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 msgr2=0x7f5b70062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:30.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 0x7f5b70062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 msgr2=0x7f5b84155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:30.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c590 tx=0x7f5b7800c950 comp rx=0 tx=0).stop 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 shutdown_connections 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b84158c90 0x7f5b840ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f5b700606c0 0x7f5b70062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b8406dcf0 0x7f5b84155070 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 --2- 172.21.15.50:0/2737453771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5b8406aeb0 0x7f5b8415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 >> 172.21.15.50:0/2737453771 conn(0x7f5b84067bb0 msgr2=0x7f5b840b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 shutdown_connections 2023-12-19T15:55:30.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.055+0000 7f5b88984700 1 -- 172.21.15.50:0/2737453771 wait complete. 2023-12-19T15:55:30.069 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.062+0000 7f3f334ea000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:55:30.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:30.078 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T15:55:30.102 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.099+0000 7fdf94ffe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T15:55:30.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T15:55:30.105 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-12-19T15:55:30.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T15:55:30.214 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.206+0000 7f3f334ea000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:30.253 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.247+0000 7fdf94ffe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.383+0000 7fdeb9ae3700 1 Processor -- start 2023-12-19T15:55:30.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.383+0000 7fdeb9ae3700 1 -- start start 2023-12-19T15:55:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.383+0000 7fdeb9ae3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdeb4158ed0 0x7fdeb40ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.383+0000 7fdeb9ae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.383+0000 7fdeb9ae3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40c85d0 con 0x7fdeb406cd00 2023-12-19T15:55:30.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb40ae290 con 0x7fdeb4158ed0 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb40ad680 con 0x7fdeb40acef0 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdeb4158ed0 0x7fdeb40ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdeb4158ed0 0x7fdeb40ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43990/0 (socket says 172.21.15.50:43990) 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/3617681862 learned_addr learned my addr 172.21.15.50:0/3617681862 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb3fff700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/3617681862 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 msgr2=0x7fdeb406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/3617681862 >> v1:172.21.15.125:6789/0 conn(0x7fdeb406cd00 legacy=0x7fdeb40c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/3617681862 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb40ad960 con 0x7fdeb4158ed0 2023-12-19T15:55:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb3fff700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T15:55:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40ae720 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fdea800bc50 tx=0x7fdea800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1978fe7847352b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb37fe700 1 -- 172.21.15.50:0/3617681862 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800be60 con 0x7fdeb4158ed0 2023-12-19T15:55:30.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb37fe700 1 -- 172.21.15.50:0/3617681862 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea800de00 con 0x7fdeb4158ed0 2023-12-19T15:55:30.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb37fe700 1 -- 172.21.15.50:0/3617681862 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80196b0 con 0x7fdeb4158ed0 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/3617681862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 msgr2=0x7fdeb40ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40ae720 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fdea800bc50 tx=0x7fdea800b4f0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/3617681862 shutdown_connections 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/3617681862 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40ae720 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/3617681862 >> 172.21.15.50:0/3617681862 conn(0x7fdeb40c8c00 msgr2=0x7fdeb40c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/3617681862 shutdown_connections 2023-12-19T15:55:30.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.387+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/3617681862 wait complete. 2023-12-19T15:55:30.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.391+0000 7fdeb9ae3700 1 Processor -- start 2023-12-19T15:55:30.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.391+0000 7fdeb9ae3700 1 -- start start 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 0x7fdeb4151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb4169910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb41523a0 con 0x7fdeb4158ed0 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb416ad60 con 0x7fdeb406cd00 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4169e50 con 0x7fdeb40acef0 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:30.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44000/0 (socket says 172.21.15.50:44000) 2023-12-19T15:55:30.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 -- 172.21.15.50:0/1274344277 learned_addr learned my addr 172.21.15.50:0/1274344277 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:30.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb8ae1700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 0x7fdeb4151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:30.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb3fff700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb4169910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 msgr2=0x7fdeb4169910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb4169910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 msgr2=0x7fdeb4151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 0x7fdeb4151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea800b790 con 0x7fdeb4158ed0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 0x7fdeb4151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb92e2700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00c8e0 tx=0x7fdeac001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac012630 con 0x7fdeb4158ed0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac012790 con 0x7fdeb4158ed0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb416a150 con 0x7fdeb4158ed0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac012b50 con 0x7fdeb4158ed0 2023-12-19T15:55:30.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.395+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb416a710 con 0x7fdeb4158ed0 2023-12-19T15:55:30.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb416aed0 con 0x7fdeb4158ed0 2023-12-19T15:55:30.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 77) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fdeac01a3a0 con 0x7fdeb4158ed0 2023-12-19T15:55:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb0ff9700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6081+0+0 (secure 0 0 0) 0x7fdeac07cd00 con 0x7fdeb4158ed0 2023-12-19T15:55:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.399+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T15:55:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.407+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdeac050a50 con 0x7fdeb4158ed0 2023-12-19T15:55:30.496 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.490+0000 7f3f334ea000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:55:30.530 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.527+0000 7fdf94ffe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T15:55:30.571 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.562+0000 7f3f334ea000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:30.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.595+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeb404be40 con 0x7fde980606c0 2023-12-19T15:55:30.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.603+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:30.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:30.603+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T15:55:30.606 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.603+0000 7fdf94ffe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:30.797 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.790+0000 7f3f334ea000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:55:30.833 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.827+0000 7fdf94ffe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T15:55:30.888 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.882+0000 7f3f334ea000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:55:30.920 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.915+0000 7fdf94ffe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T15:55:30.976 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:30.970+0000 7f3f334ea000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:55:31.005 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:30.999+0000 7fdf94ffe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T15:55:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:31.007+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:31.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:31.007+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T15:55:31.394 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:31.386+0000 7f3f334ea000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:55:31.402 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:31.399+0000 7fdf94ffe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T15:55:31.569 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:31.563+0000 7fdf94ffe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:55:31.570 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:31.562+0000 7f3f334ea000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T15:55:31.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:31.811+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:31.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:31.811+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T15:55:31.924 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:31.919+0000 7fdf94ffe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:55:31.948 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:31.942+0000 7f3f334ea000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T15:55:32.156 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.151+0000 7fdf94ffe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:32.184 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.178+0000 7f3f334ea000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T15:55:32.249 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.247+0000 7fdf94ffe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:55:32.282 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.274+0000 7f3f334ea000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T15:55:32.326 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.323+0000 7fdf94ffe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:55:32.366 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.358+0000 7f3f334ea000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T15:55:32.412 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.407+0000 7fdf94ffe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:55:32.452 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.446+0000 7f3f334ea000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T15:55:32.634 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.631+0000 7fdf94ffe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:32.687 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.682+0000 7f3f334ea000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T15:55:32.714 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.711+0000 7fdf94ffe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:55:32.772 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.766+0000 7f3f334ea000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T15:55:32.806 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:32.803+0000 7fdf94ffe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:55:32.865 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:32.858+0000 7f3f334ea000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T15:55:33.007 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:33.003+0000 7fdf94ffe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:55:33.077 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:33.070+0000 7f3f334ea000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T15:55:33.097 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:33.091+0000 7fdf94ffe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:55:33.171 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:33.162+0000 7f3f334ea000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T15:55:33.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:33.415+0000 7fdeb8ae1700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/33827 2023-12-19T15:55:33.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:33.415+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T15:55:33.907 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T15:55:33.903+0000 7fdf94ffe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:55:33.935 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:33.926+0000 7f3f334ea000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T15:55:33.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:33.947+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 78) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fdeac04d390 con 0x7fdeb4158ed0 2023-12-19T15:55:33.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:33.947+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T15:55:33.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:33.947+0000 7fdeb0ff9700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:34.146 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef6b4a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.146 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef6b4a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.146 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef6b4a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.146 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef6b4a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.146 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef6b4a700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef3343700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef3343700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef3343700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef3343700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.147 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:55:34.138+0000 7f3ef3343700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:55:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:34.983+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 79) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fdeac002230 con 0x7fdeb4158ed0 2023-12-19T15:55:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:34.983+0000 7fdeb0ff9700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:34.983+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeb404be40 con 0x7fde980606c0 2023-12-19T15:55:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:34.987+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:34.987+0000 7fdeb8ae1700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdea800bc50 tx=0x7fdea801b000 comp rx=0 tx=0).ready entity=mgr.8255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:35.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.087+0000 7fdeb0ff9700 1 -- 172.21.15.50:0/1274344277 <== mgr.8255 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdeb404be40 con 0x7fde980606c0 2023-12-19T15:55:35.089 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-12-19T15:55:35.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 msgr2=0x7fde98062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdea800bc50 tx=0x7fdea801b000 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 msgr2=0x7fdeb40c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00c8e0 tx=0x7fdeac001710 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 shutdown_connections 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fde980606c0 0x7fde98062b70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdeb4158ed0 0x7fdeb40c74b0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdeb40acef0 0x7fdeb4169910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 --2- 172.21.15.50:0/1274344277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeb406cd00 0x7fdeb4151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 >> 172.21.15.50:0/1274344277 conn(0x7fdeb40c8c00 msgr2=0x7fdeb406e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 shutdown_connections 2023-12-19T15:55:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.091+0000 7fdeb9ae3700 1 -- 172.21.15.50:0/1274344277 wait complete. 2023-12-19T15:55:35.120 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-12-19T15:55:35.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T15:55:35.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 Processor -- start 2023-12-19T15:55:35.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 -- start start 2023-12-19T15:55:35.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdad00cac50 0x7fdad00af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad00cb3f0 con 0x7fdad006af50 2023-12-19T15:55:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad00cc140 con 0x7fdad00cac50 2023-12-19T15:55:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad006d220 con 0x7fdad00aedc0 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdad00cac50 0x7fdad00af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdad00cac50 0x7fdad00af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44004/0 (socket says 172.21.15.50:44004) 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 -- 172.21.15.50:0/1893688137 learned_addr learned my addr 172.21.15.50:0/1893688137 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5121700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 -- 172.21.15.50:0/1893688137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 msgr2=0x7fdad00af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 -- 172.21.15.50:0/1893688137 >> v1:172.21.15.50:6789/0 conn(0x7fdad006af50 legacy=0x7fdad006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5922700 1 -- 172.21.15.50:0/1893688137 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad006b3a0 con 0x7fdad00cac50 2023-12-19T15:55:35.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.331+0000 7fdad5121700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad5922700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad00cac50 0x7fdad00af930 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009f20 tx=0x7fdac8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abf1ed9a809ed1c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdac7fff700 1 -- 172.21.15.50:0/1893688137 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac800e070 con 0x7fdad00cac50 2023-12-19T15:55:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdac7fff700 1 -- 172.21.15.50:0/1893688137 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac800ba80 con 0x7fdad00cac50 2023-12-19T15:55:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdac7fff700 1 -- 172.21.15.50:0/1893688137 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac80092e0 con 0x7fdad00cac50 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 -- 172.21.15.50:0/1893688137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad00cac50 msgr2=0x7fdad00af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad00cac50 0x7fdad00af930 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009f20 tx=0x7fdac8009840 comp rx=0 tx=0).stop 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 -- 172.21.15.50:0/1893688137 shutdown_connections 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad00cac50 0x7fdad00af930 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 --2- 172.21.15.50:0/1893688137 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 -- 172.21.15.50:0/1893688137 >> 172.21.15.50:0/1893688137 conn(0x7fdad00cc760 msgr2=0x7fdad00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 -- 172.21.15.50:0/1893688137 shutdown_connections 2023-12-19T15:55:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.335+0000 7fdad6123700 1 -- 172.21.15.50:0/1893688137 wait complete. 2023-12-19T15:55:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 Processor -- start 2023-12-19T15:55:35.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 -- start start 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad00cac50 0x7fdad00cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad00c7d80 con 0x7fdad006af50 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad00c9a40 con 0x7fdad00cac50 2023-12-19T15:55:35.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad6123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad00c9070 con 0x7fdad00aedc0 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:44018/0 (socket says 172.21.15.50:44018) 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 -- 172.21.15.50:0/2398246637 learned_addr learned my addr 172.21.15.50:0/2398246637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5922700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad00cac50 0x7fdad00cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad4920700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 -- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 msgr2=0x7fdad00c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 -- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad00cac50 msgr2=0x7fdad00cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad00cac50 0x7fdad00cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 -- 172.21.15.50:0/2398246637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8009ae0 con 0x7fdad006af50 2023-12-19T15:55:35.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad4920700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.339+0000 7fdad5121700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c860 tx=0x7fdacc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc011600 con 0x7fdad006af50 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc011760 con 0x7fdad006af50 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc011b00 con 0x7fdad006af50 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad00c7aa0 con 0x7fdad006af50 2023-12-19T15:55:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad00c8d00 con 0x7fdad006af50 2023-12-19T15:55:35.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad004be40 con 0x7fdad006af50 2023-12-19T15:55:35.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.343+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 79) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fdacc00a3e0 con 0x7fdad006af50 2023-12-19T15:55:35.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.347+0000 7fdac57fa700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 0x7fdab40628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T15:55:35.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.347+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fdacc07c930 con 0x7fdad006af50 2023-12-19T15:55:35.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.347+0000 7fdad4920700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 0x7fdab40628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T15:55:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.347+0000 7fdad4920700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 0x7fdab40628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdad00ca970 tx=0x7fdac0009040 comp rx=0 tx=0).ready entity=mgr.8255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T15:55:35.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.351+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdacc04ce30 con 0x7fdad006af50 2023-12-19T15:55:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.547+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdad00c91b0 con 0x7fdab40603f0 2023-12-19T15:55:35.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.631+0000 7fdac57fa700 1 -- 172.21.15.50:0/2398246637 <== mgr.8255 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fdac000d660 con 0x7fdab40603f0 2023-12-19T15:55:35.634 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-12-19T15:55:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 msgr2=0x7fdab40628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 0x7fdab40628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdad00ca970 tx=0x7fdac0009040 comp rx=0 tx=0).stop 2023-12-19T15:55:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 msgr2=0x7fdad0158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T15:55:35.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c860 tx=0x7fdacc008040 comp rx=0 tx=0).stop 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 shutdown_connections 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad00cac50 0x7fdad00cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fdab40603f0 0x7fdab40628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad00aedc0 0x7fdad00c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 --2- 172.21.15.50:0/2398246637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad006af50 0x7fdad0158d20 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T15:55:35.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 >> 172.21.15.50:0/2398246637 conn(0x7fdad00cc760 msgr2=0x7fdad00afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T15:55:35.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 shutdown_connections 2023-12-19T15:55:35.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T15:55:35.635+0000 7fdad6123700 1 -- 172.21.15.50:0/2398246637 wait complete. 2023-12-19T15:55:35.660 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-12-19T15:55:54.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:55:54.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:56:00.654 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:56:00.645+0000 7f3ee5b28700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:56:00.654 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:56:00.645+0000 7f3ee5b28700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:56:00.654 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:56:00.645+0000 7f3ee5b28700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:56:00.654 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:56:00.645+0000 7f3ee5b28700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:56:00.654 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T15:56:00.645+0000 7f3ee5b28700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T15:56:24.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:56:24.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:56:54.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:56:54.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:57:25.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:57:25.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:57:55.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:57:55.175 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:58:25.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:58:25.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:58:55.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:58:55.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:59:25.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:59:25.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:59:55.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T15:59:55.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:00:25.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:00:25.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:00:56.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:00:56.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:01:26.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:01:26.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:01:56.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:01:56.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:02:26.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:02:26.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:02:56.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:02:56.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:03:26.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:03:26.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:03:56.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:03:56.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:04:27.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:04:27.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:04:57.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:04:57.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:05:27.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:05:27.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:05:57.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:05:57.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:06:15.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:06:15.662 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_56_00_UTC 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_57_00_UTC 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_58_00_UTC 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_59_01_UTC 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_00_00_UTC 2023-12-19T16:06:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_01_00_UTC 2023-12-19T16:06:15.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_02_00_UTC 2023-12-19T16:06:15.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_03_00_UTC 2023-12-19T16:06:15.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_04_00_UTC 2023-12-19T16:06:15.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_05_00_UTC 2023-12-19T16:06:15.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_06_00_UTC 2023-12-19T16:06:15.700 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-15_56_00_UTC', 'scheduled-2023-12-19-15_57_00_UTC', 'scheduled-2023-12-19-15_58_00_UTC', 'scheduled-2023-12-19-15_59_01_UTC', 'scheduled-2023-12-19-16_00_00_UTC', 'scheduled-2023-12-19-16_01_00_UTC', 'scheduled-2023-12-19-16_02_00_UTC', 'scheduled-2023-12-19-16_03_00_UTC', 'scheduled-2023-12-19-16_04_00_UTC', 'scheduled-2023-12-19-16_05_00_UTC', 'scheduled-2023-12-19-16_06_00_UTC'] 2023-12-19T16:06:15.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-12-19T16:06:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 Processor -- start 2023-12-19T16:06:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 -- start start 2023-12-19T16:06:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbfc4068670 0x7fbfc41580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbfc406ac90 0x7fbfc415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc40af280 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40acbd0 con 0x7fbfc4068670 2023-12-19T16:06:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.906+0000 7fbfc88e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40ae950 con 0x7fbfc406ac90 2023-12-19T16:06:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfbbfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbfc40ac1f0 0x7fbfc406a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49186/0 (socket says 172.21.15.50:49186) 2023-12-19T16:06:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfbbfff700 1 -- 172.21.15.50:0/2400392960 learned_addr learned my addr 172.21.15.50:0/2400392960 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:06:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/2400392960 >> v2:172.21.15.50:3300/0 conn(0x7fbfc4068670 0x7fbfc41580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc3fff700 1 --2- 172.21.15.50:0/2400392960 >> v2:172.21.15.125:3300/0 conn(0x7fbfc406ac90 0x7fbfc415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc2ffd700 1 -- 172.21.15.50:0/2400392960 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190151223 0 0) 0x7fbfc40af280 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc2ffd700 1 -- 172.21.15.50:0/2400392960 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb0003570 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 -- 172.21.15.50:0/2400392960 >> v2:172.21.15.125:3300/0 conn(0x7fbfc406ac90 msgr2=0x7fbfc415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/2400392960 >> v2:172.21.15.125:3300/0 conn(0x7fbfc406ac90 0x7fbfc415f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 -- 172.21.15.50:0/2400392960 >> v1:172.21.15.50:6789/0 conn(0x7fbfc40ac1f0 legacy=0x7fbfc406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 -- 172.21.15.50:0/2400392960 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40aec30 con 0x7fbfc4068670 2023-12-19T16:06:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/2400392960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc4068670 0x7fbfc41580c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aba70 tx=0x7fbfb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c08e0f4a5b635 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc2ffd700 1 -- 172.21.15.50:0/2400392960 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4011630 con 0x7fbfc4068670 2023-12-19T16:06:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc2ffd700 1 -- 172.21.15.50:0/2400392960 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb40117b0 con 0x7fbfc4068670 2023-12-19T16:06:15.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.910+0000 7fbfc2ffd700 1 -- 172.21.15.50:0/2400392960 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4011b70 con 0x7fbfc4068670 2023-12-19T16:06:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- 172.21.15.50:0/2400392960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc4068670 msgr2=0x7fbfc41580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/2400392960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc4068670 0x7fbfc41580c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aba70 tx=0x7fbfb40091a0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- 172.21.15.50:0/2400392960 shutdown_connections 2023-12-19T16:06:15.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/2400392960 >> v2:172.21.15.125:3300/0 conn(0x7fbfc406ac90 0x7fbfc415f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/2400392960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc4068670 0x7fbfc41580c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- 172.21.15.50:0/2400392960 >> 172.21.15.50:0/2400392960 conn(0x7fbfc4067bb0 msgr2=0x7fbfc40c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:06:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- 172.21.15.50:0/2400392960 shutdown_connections 2023-12-19T16:06:15.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- 172.21.15.50:0/2400392960 wait complete. 2023-12-19T16:06:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 Processor -- start 2023-12-19T16:06:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 -- start start 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfc4068670 0x7fbfc40b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.914+0000 7fbfc88e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfc406ac90 0x7fbfc4143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40afbd0 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40afeb0 con 0x7fbfc406ac90 2023-12-19T16:06:15.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40b0190 con 0x7fbfc4068670 2023-12-19T16:06:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58398/0 (socket says 172.21.15.50:58398) 2023-12-19T16:06:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 -- 172.21.15.50:0/862982571 learned_addr learned my addr 172.21.15.50:0/862982571 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:06:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfc4068670 0x7fbfc40b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfbbfff700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfc406ac90 0x7fbfc4143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 -- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfc4068670 msgr2=0x7fbfc40b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfc4068670 0x7fbfc40b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 -- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfc406ac90 msgr2=0x7fbfc4143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfc406ac90 0x7fbfc4143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 -- 172.21.15.50:0/862982571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb4009420 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc3fff700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009ef0 tx=0x7fbfbc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc00cb70 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc00f040 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc40b0470 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc014480 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.918+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc40b18e0 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.922+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc404be40 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.922+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 82) v1 ==== 80092+0+0 (secure 0 0 0) 0x7fbfbc014830 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.922+0000 7fbfbb7fe700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 0x7fbfa4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.926+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6541+0+0 (secure 0 0 0) 0x7fbfbc07dab0 con 0x7fbfc40ac1f0 2023-12-19T16:06:15.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.926+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 0x7fbfa4062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:15.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.926+0000 7fbfc37fe700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 0x7fbfa4062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aba70 tx=0x7fbfb4017000 comp rx=0 tx=0).ready entity=mgr.8255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:15.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:15.930+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbfbc04daa0 con 0x7fbfc40ac1f0 2023-12-19T16:06:16.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.126+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 --> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfc40ad930 con 0x7fbfa40606c0 2023-12-19T16:06:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.142+0000 7fbfbb7fe700 1 -- 172.21.15.50:0/862982571 <== mgr.8255 v2:172.21.15.125:6832/33827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fbfc40ad930 con 0x7fbfa40606c0 2023-12-19T16:06:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:06:16.149 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T15:55:34", "first": "2023-12-19T15:56:00", "last": "2023-12-19T16:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 msgr2=0x7fbfa4062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 0x7fbfa4062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aba70 tx=0x7fbfb4017000 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 msgr2=0x7fbfc414d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009ef0 tx=0x7fbfbc00b040 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 shutdown_connections 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfc406ac90 0x7fbfc4143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7fbfa40606c0 0x7fbfa4062b70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfc40ac1f0 0x7fbfc414d140 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 --2- 172.21.15.50:0/862982571 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbfc4068670 0x7fbfc40b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 >> 172.21.15.50:0/862982571 conn(0x7fbfc4067bb0 msgr2=0x7fbfc40c93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 shutdown_connections 2023-12-19T16:06:16.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.146+0000 7fbfc88e3700 1 -- 172.21.15.50:0/862982571 wait complete. 2023-12-19T16:06:16.175 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T15:55:34", "first": "2023-12-19T15:56:00", "last": "2023-12-19T16:06:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-12-19T16:06:16.176 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T15:55:34', 'first': '2023-12-19T15:56:00', 'last': '2023-12-19T16:06:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-12-19T16:06:16.176 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T16:06:16.176 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T16:06:16.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-19T16:06:16.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 Processor -- start 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 -- start start 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75bc0acd30 0x7f75bc0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75bc0c8200 0x7f75bc15a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc15b170 con 0x7f75bc0c8200 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc0cca10 con 0x7f75bc0acd30 2023-12-19T16:06:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.362+0000 7f75c1443700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0cc0e0 con 0x7f75bc15acf0 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75bc0c8200 0x7f75bc15a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75bbfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75bc0acd30 0x7f75bc0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75bc0c8200 0x7f75bc15a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58404/0 (socket says 172.21.15.50:58404) 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 -- 172.21.15.50:0/3116823471 learned_addr learned my addr 172.21.15.50:0/3116823471 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c0c42700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 -- 172.21.15.50:0/3116823471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 msgr2=0x7f75bc0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 -- 172.21.15.50:0/3116823471 >> v2:172.21.15.125:3300/0 conn(0x7f75bc0acd30 msgr2=0x7f75bc0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3116823471 >> v2:172.21.15.125:3300/0 conn(0x7f75bc0acd30 0x7f75bc0c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 -- 172.21.15.50:0/3116823471 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0cc3c0 con 0x7f75bc0c8200 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0c8200 0x7f75bc15a7b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f75bc070a70 tx=0x7f75a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721a4f63723c9a8c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75bb7fe700 1 -- 172.21.15.50:0/3116823471 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400b6d0 con 0x7f75bc0c8200 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75bb7fe700 1 -- 172.21.15.50:0/3116823471 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400b830 con 0x7f75bc0c8200 2023-12-19T16:06:16.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75bb7fe700 1 -- 172.21.15.50:0/3116823471 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400bbf0 con 0x7f75bc0c8200 2023-12-19T16:06:16.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 -- 172.21.15.50:0/3116823471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0c8200 msgr2=0x7f75bc15a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0c8200 0x7f75bc15a7b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f75bc070a70 tx=0x7f75a4009250 comp rx=0 tx=0).stop 2023-12-19T16:06:16.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 -- 172.21.15.50:0/3116823471 shutdown_connections 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0c8200 0x7f75bc15a7b0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 --2- 172.21.15.50:0/3116823471 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 --2- 172.21.15.50:0/3116823471 >> v2:172.21.15.125:3300/0 conn(0x7f75bc0acd30 0x7f75bc0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 -- 172.21.15.50:0/3116823471 >> 172.21.15.50:0/3116823471 conn(0x7f75bc0cdbc0 msgr2=0x7f75bc0cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 -- 172.21.15.50:0/3116823471 shutdown_connections 2023-12-19T16:06:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.366+0000 7f75c1443700 1 -- 172.21.15.50:0/3116823471 wait complete. 2023-12-19T16:06:16.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.370+0000 7f75c1443700 1 Processor -- start 2023-12-19T16:06:16.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.370+0000 7f75c1443700 1 -- start start 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 0x7f75bc0c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc169ff0 con 0x7f75bc0acd30 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc16ceb0 con 0x7f75bc0c8200 2023-12-19T16:06:16.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc1587d0 con 0x7f75bc15acf0 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58420/0 (socket says 172.21.15.50:58420) 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 -- 172.21.15.50:0/3117120395 learned_addr learned my addr 172.21.15.50:0/3117120395 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c0c42700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 0x7f75bc0c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 -- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 msgr2=0x7f75bc0c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 -- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 msgr2=0x7f75bc0c3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 0x7f75bc0c3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 -- 172.21.15.50:0/3117120395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40094f0 con 0x7f75bc0acd30 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 0x7f75bc0c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75bbfff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00ecb0 tx=0x7f75ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00c9d0 con 0x7f75bc0acd30 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00cb30 con 0x7f75bc0acd30 2023-12-19T16:06:16.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc169d10 con 0x7f75bc0acd30 2023-12-19T16:06:16.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00ced0 con 0x7f75bc0acd30 2023-12-19T16:06:16.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.374+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc16cad0 con 0x7f75bc0acd30 2023-12-19T16:06:16.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.378+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc04be40 con 0x7f75bc0acd30 2023-12-19T16:06:16.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.378+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 82) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f75ac013680 con 0x7f75bc0acd30 2023-12-19T16:06:16.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.382+0000 7f75b8ff9700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 0x7f759c062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:06:16.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.382+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6541+0+0 (secure 0 0 0) 0x7f75ac07d080 con 0x7f75bc0acd30 2023-12-19T16:06:16.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.382+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 0x7f759c062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:06:16.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.382+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 0x7f759c062ba0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75bc070a70 tx=0x7f75a4017000 comp rx=0 tx=0).ready entity=mgr.8255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:06:16.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.390+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f75ac0acbe0 con 0x7f75bc0acd30 2023-12-19T16:06:16.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.602+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f75bc0cb860 con 0x7f75bc0acd30 2023-12-19T16:06:16.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.606+0000 7f75b8ff9700 1 -- 172.21.15.50:0/3117120395 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v82) v1 ==== 74+0+154349 (secure 0 0 0) 0x7f75ac050c10 con 0x7f75bc0acd30 2023-12-19T16:06:16.613 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:06:16.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 msgr2=0x7f759c062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 0x7f759c062ba0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75bc070a70 tx=0x7f75a4017000 comp rx=0 tx=0).stop 2023-12-19T16:06:16.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 msgr2=0x7f75bc158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00ecb0 tx=0x7f75ac0091a0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 shutdown_connections 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75bc0c8200 0x7f75bc0c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:6832/33827,v1:172.21.15.125:6833/33827] conn(0x7f759c0606f0 0x7f759c062ba0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75bc15acf0 0x7f75bc0c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 --2- 172.21.15.50:0/3117120395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75bc0acd30 0x7f75bc158f80 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:06:16.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 >> 172.21.15.50:0/3117120395 conn(0x7f75bc0cdbc0 msgr2=0x7f75bc067130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:06:16.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 shutdown_connections 2023-12-19T16:06:16.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:06:16.610+0000 7f75c1443700 1 -- 172.21.15.50:0/3117120395 wait complete. 2023-12-19T16:06:16.635 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"active_gid":8255,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":33827},{"type":"v1","addr":"172.21.15.125:6833","nonce":33827}]},"active_addr":"172.21.15.125:6833/33827","active_change":"2023-12-19T15:55:33.938914+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":9328,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":182,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":3295762643}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":2491435626}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":1719917206}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":2545221666}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":3093853131}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":4089126800}]}]} 2023-12-19T16:06:16.639 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-12-19T16:06:16.639 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-19T16:06:16.639 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-12-19T16:06:16.639 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-19T16:06:16.639 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:06:16.700 INFO:tasks.ceph.mgr.x:Stopped 2023-12-19T16:06:16.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-19T16:06:16.704 INFO:tasks.ceph.mgr.x:Started 2023-12-19T16:06:16.927 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:16.917+0000 7f6e89d2c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:06:17.473 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:17.465+0000 7f6e89d2c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:06:17.542 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:17.533+0000 7f6e89d2c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:06:17.674 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:17.665+0000 7f6e89d2c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:06:17.921 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:17.913+0000 7f6e89d2c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:06:17.989 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:17.981+0000 7f6e89d2c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:06:18.190 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:18.181+0000 7f6e89d2c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:06:18.270 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:18.261+0000 7f6e89d2c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:06:18.353 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:18.345+0000 7f6e89d2c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:06:18.730 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:18.721+0000 7f6e89d2c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:06:18.892 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:18.881+0000 7f6e89d2c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:06:19.241 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.233+0000 7f6e89d2c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:06:19.457 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.449+0000 7f6e89d2c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:06:19.548 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.537+0000 7f6e89d2c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:06:19.623 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.613+0000 7f6e89d2c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:06:19.700 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.689+0000 7f6e89d2c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:06:19.917 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.905+0000 7f6e89d2c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:06:19.997 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:19.989+0000 7f6e89d2c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:06:20.084 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:20.073+0000 7f6e89d2c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:06:20.278 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:20.269+0000 7f6e89d2c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:06:20.363 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:20.353+0000 7f6e89d2c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:06:21.083 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.073+0000 7f6e89d2c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:06:21.344 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.333+0000 7f6e49bfc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.333+0000 7f6e49bfc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.333+0000 7f6e49bfc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.333+0000 7f6e49bfc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.333+0000 7f6e49bfc700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.337+0000 7f6e4cc02700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.346 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.337+0000 7f6e4cc02700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.346 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.337+0000 7f6e4cc02700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.346 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.337+0000 7f6e4cc02700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:21.346 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:06:21.337+0000 7f6e4cc02700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:06:27.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:06:27.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:06:57.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:06:57.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:07:00.339 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:07:00.333+0000 7f6e54c12700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:07:00.340 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:07:00.333+0000 7f6e54c12700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:07:00.340 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:07:00.333+0000 7f6e54c12700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:07:00.340 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:07:00.333+0000 7f6e54c12700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:07:00.340 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:07:00.333+0000 7f6e54c12700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:07:27.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:07:27.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:07:57.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:07:57.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:08:28.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:08:28.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:08:58.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:08:58.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:09:28.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:09:28.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:09:58.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:09:58.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:10:28.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:10:28.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:10:58.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:10:58.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:11:16.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T16:11:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.879+0000 7f4831723700 1 Processor -- start 2023-12-19T16:11:16.884 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.879+0000 7f4831723700 1 -- start start 2023-12-19T16:11:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4831723700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 0x7f482c06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4831723700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f482c0caca0 0x7f482c06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4831723700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c15d720 con 0x7f482c06df10 2023-12-19T16:11:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4831723700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c0ca290 con 0x7f482c0caca0 2023-12-19T16:11:16.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4831723700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0ca4a0 con 0x7f482c158da0 2023-12-19T16:11:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f48237fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f482c06df10 0x7f482c06a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55448/0 (socket says 172.21.15.50:55448) 2023-12-19T16:11:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f48237fe700 1 -- 172.21.15.50:0/2790868528 learned_addr learned my addr 172.21.15.50:0/2790868528 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f482bfff700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 0x7f482c06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4830f22700 1 --2- 172.21.15.50:0/2790868528 >> v2:172.21.15.125:3300/0 conn(0x7f482c0caca0 0x7f482c06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f482b7fe700 1 -- 172.21.15.50:0/2790868528 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803745412 0 0) 0x7f482c15d720 con 0x7f482c06df10 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f482b7fe700 1 -- 172.21.15.50:0/2790868528 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4818003570 con 0x7f482c06df10 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4830f22700 1 -- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 msgr2=0x7f482c06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4830f22700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 0x7f482c06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4830f22700 1 -- 172.21.15.50:0/2790868528 >> v1:172.21.15.50:6789/0 conn(0x7f482c06df10 legacy=0x7f482c06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f4830f22700 1 -- 172.21.15.50:0/2790868528 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c0ca780 con 0x7f482c0caca0 2023-12-19T16:11:16.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.883+0000 7f482bfff700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 0x7f482c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4830f22700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c0caca0 0x7f482c06ebb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e60 tx=0x7f4824009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bfcf4322814dcdf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f482b7fe700 1 -- 172.21.15.50:0/2790868528 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482400e070 con 0x7f482c0caca0 2023-12-19T16:11:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f482b7fe700 1 -- 172.21.15.50:0/2790868528 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482400bab0 con 0x7f482c0caca0 2023-12-19T16:11:16.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f482b7fe700 1 -- 172.21.15.50:0/2790868528 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482400be70 con 0x7f482c0caca0 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 -- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c0caca0 msgr2=0x7f482c06ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c0caca0 0x7f482c06ebb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e60 tx=0x7f4824009780 comp rx=0 tx=0).stop 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 -- 172.21.15.50:0/2790868528 shutdown_connections 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c0caca0 0x7f482c06ebb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 --2- 172.21.15.50:0/2790868528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c158da0 0x7f482c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 -- 172.21.15.50:0/2790868528 >> 172.21.15.50:0/2790868528 conn(0x7f482c0cc760 msgr2=0x7f482c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 -- 172.21.15.50:0/2790868528 shutdown_connections 2023-12-19T16:11:16.892 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.887+0000 7f4831723700 1 -- 172.21.15.50:0/2790868528 wait complete. 2023-12-19T16:11:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 Processor -- start 2023-12-19T16:11:16.895 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 -- start start 2023-12-19T16:11:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c06df10 0x7f482c0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c158da0 0x7f482c0c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0b0750 con 0x7f482c0caca0 2023-12-19T16:11:16.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0af860 con 0x7f482c158da0 2023-12-19T16:11:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.891+0000 7f4831723700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0afb40 con 0x7f482c06df10 2023-12-19T16:11:16.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f482bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c06df10 0x7f482c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36552/0 (socket says 172.21.15.50:36552) 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 -- 172.21.15.50:0/507137390 learned_addr learned my addr 172.21.15.50:0/507137390 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4830f22700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c158da0 0x7f482c0c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 -- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c06df10 msgr2=0x7f482c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c06df10 0x7f482c0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 -- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c158da0 msgr2=0x7f482c0c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c158da0 0x7f482c0c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 -- 172.21.15.50:0/507137390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c009420 con 0x7f482c0caca0 2023-12-19T16:11:16.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f48237fe700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f481400c7c0 tx=0x7f4814007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:16.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814005cd0 con 0x7f482c0caca0 2023-12-19T16:11:16.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814005e30 con 0x7f482c0caca0 2023-12-19T16:11:16.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814007320 con 0x7f482c0caca0 2023-12-19T16:11:16.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4824009a20 con 0x7f482c0caca0 2023-12-19T16:11:16.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c0b0100 con 0x7f482c0caca0 2023-12-19T16:11:16.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.895+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c0b08c0 con 0x7f482c0caca0 2023-12-19T16:11:16.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.899+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 87) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f481401b3b0 con 0x7f482c0caca0 2023-12-19T16:11:16.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.899+0000 7f4828ff9700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 0x7f480c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:16.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.899+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f481407c520 con 0x7f482c0caca0 2023-12-19T16:11:16.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.899+0000 7f482bfff700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 0x7f480c062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:16.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.899+0000 7f482bfff700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 0x7f480c062b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f482c1581f0 tx=0x7f481c0091a0 comp rx=0 tx=0).ready entity=mgr.9418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:16.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:16.907+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f481404c520 con 0x7f482c0caca0 2023-12-19T16:11:17.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.095+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f482c0c7ca0 con 0x7f480c0606c0 2023-12-19T16:11:17.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.199+0000 7f4828ff9700 1 -- 172.21.15.50:0/507137390 <== mgr.9418 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f482c0c7ca0 con 0x7f480c0606c0 2023-12-19T16:11:17.202 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-12-19T16:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 msgr2=0x7f480c062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 0x7f480c062b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f482c1581f0 tx=0x7f481c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 msgr2=0x7f482c0c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f481400c7c0 tx=0x7f4814007e00 comp rx=0 tx=0).stop 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 shutdown_connections 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f480c0606c0 0x7f480c062b70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f482c158da0 0x7f482c0c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f482c0caca0 0x7f482c0c57e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 --2- 172.21.15.50:0/507137390 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f482c06df10 0x7f482c0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 >> 172.21.15.50:0/507137390 conn(0x7f482c0cc760 msgr2=0x7f482c06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 shutdown_connections 2023-12-19T16:11:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.203+0000 7f4831723700 1 -- 172.21.15.50:0/507137390 wait complete. 2023-12-19T16:11:17.231 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-12-19T16:11:17.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:17.231 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T16:11:17.264 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_56_00_UTC 2023-12-19T16:11:17.264 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_57_00_UTC 2023-12-19T16:11:17.264 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_58_00_UTC 2023-12-19T16:11:17.264 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_59_01_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_00_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_01_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_02_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_03_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_04_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_05_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_06_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_07_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_08_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_09_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_10_00_UTC 2023-12-19T16:11:17.265 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_11_00_UTC 2023-12-19T16:11:17.266 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-15_56_00_UTC', 'scheduled-2023-12-19-15_57_00_UTC', 'scheduled-2023-12-19-15_58_00_UTC', 'scheduled-2023-12-19-15_59_01_UTC', 'scheduled-2023-12-19-16_00_00_UTC', 'scheduled-2023-12-19-16_01_00_UTC', 'scheduled-2023-12-19-16_02_00_UTC', 'scheduled-2023-12-19-16_03_00_UTC', 'scheduled-2023-12-19-16_04_00_UTC', 'scheduled-2023-12-19-16_05_00_UTC', 'scheduled-2023-12-19-16_06_00_UTC', 'scheduled-2023-12-19-16_07_00_UTC', 'scheduled-2023-12-19-16_08_00_UTC', 'scheduled-2023-12-19-16_09_00_UTC', 'scheduled-2023-12-19-16_10_00_UTC', 'scheduled-2023-12-19-16_11_00_UTC'] 2023-12-19T16:11:17.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-12-19T16:11:17.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.483+0000 7ff34ce6b700 1 Processor -- start 2023-12-19T16:11:17.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.483+0000 7ff34ce6b700 1 -- start start 2023-12-19T16:11:17.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff34ce6b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff34806ed80 0x7ff34806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff34ce6b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34806b420 con 0x7ff34806df10 2023-12-19T16:11:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff34ce6b700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34815d720 con 0x7ff348158da0 2023-12-19T16:11:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff34ce6b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3480ca290 con 0x7ff34806ed80 2023-12-19T16:11:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff347fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff34806df10 0x7ff34806ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55462/0 (socket says 172.21.15.50:55462) 2023-12-19T16:11:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff347fff700 1 -- 172.21.15.50:0/1334473377 learned_addr learned my addr 172.21.15.50:0/1334473377 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:17.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.487+0000 7ff346ffd700 1 --2- 172.21.15.50:0/1334473377 >> v2:172.21.15.125:3300/0 conn(0x7ff34806ed80 0x7ff34806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062482860 0 0) 0x7ff34806b420 con 0x7ff34806df10 2023-12-19T16:11:17.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff330003570 con 0x7ff34806df10 2023-12-19T16:11:17.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff346ffd700 1 -- 172.21.15.50:0/1334473377 >> v1:172.21.15.125:6789/0 conn(0x7ff348158da0 legacy=0x7ff34806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600012091 0 0) 0x7ff34815d720 con 0x7ff348158da0 2023-12-19T16:11:17.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff346ffd700 1 -- 172.21.15.50:0/1334473377 >> v1:172.21.15.50:6789/0 conn(0x7ff34806df10 legacy=0x7ff34806ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff346ffd700 1 -- 172.21.15.50:0/1334473377 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480ca4a0 con 0x7ff34806ed80 2023-12-19T16:11:17.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff346ffd700 1 --2- 172.21.15.50:0/1334473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff34806ed80 0x7ff34806a7d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0098e0 tx=0x7ff33c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54483f31553c44c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c00b640 con 0x7ff34806ed80 2023-12-19T16:11:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c00b7a0 con 0x7ff34806ed80 2023-12-19T16:11:17.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff3467fc700 1 -- 172.21.15.50:0/1334473377 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c00bb40 con 0x7ff34806ed80 2023-12-19T16:11:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1334473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff34806ed80 msgr2=0x7ff34806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1334473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff34806ed80 0x7ff34806a7d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0098e0 tx=0x7ff33c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1334473377 shutdown_connections 2023-12-19T16:11:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1334473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff34806ed80 0x7ff34806a7d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1334473377 >> 172.21.15.50:0/1334473377 conn(0x7ff3480cc760 msgr2=0x7ff3480ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.491+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1334473377 shutdown_connections 2023-12-19T16:11:17.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.495+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1334473377 wait complete. 2023-12-19T16:11:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.495+0000 7ff34ce6b700 1 Processor -- start 2023-12-19T16:11:17.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.495+0000 7ff34ce6b700 1 -- start start 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff34806df10 0x7ff3480c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff348158da0 0x7ff3480b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c6d00 con 0x7ff34806ed80 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c7b60 con 0x7ff348158da0 2023-12-19T16:11:17.503 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c5740 con 0x7ff34806df10 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36554/0 (socket says 172.21.15.50:36554) 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 -- 172.21.15.50:0/1454105200 learned_addr learned my addr 172.21.15.50:0/1454105200 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff347fff700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff348158da0 0x7ff3480b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff3477fe700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff34806df10 0x7ff3480c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 -- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff34806df10 msgr2=0x7ff3480c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff34806df10 0x7ff3480c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 -- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff348158da0 msgr2=0x7ff3480b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff348158da0 0x7ff3480b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 -- 172.21.15.50:0/1454105200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c009420 con 0x7ff34806ed80 2023-12-19T16:11:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff346ffd700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009f80 tx=0x7ff33c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c01f070 con 0x7ff34806ed80 2023-12-19T16:11:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c00b910 con 0x7ff34806ed80 2023-12-19T16:11:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3480c6f50 con 0x7ff34806ed80 2023-12-19T16:11:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c011420 con 0x7ff34806ed80 2023-12-19T16:11:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.499+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3480c7510 con 0x7ff34806ed80 2023-12-19T16:11:17.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.503+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34804be40 con 0x7ff34806ed80 2023-12-19T16:11:17.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.503+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 87) v1 ==== 80092+0+0 (secure 0 0 0) 0x7ff33c0117d0 con 0x7ff34806ed80 2023-12-19T16:11:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.507+0000 7ff32f7fe700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 0x7ff334062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.507+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7ff33c07e320 con 0x7ff34806ed80 2023-12-19T16:11:17.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.507+0000 7ff3477fe700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 0x7ff334062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.507+0000 7ff3477fe700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 0x7ff334062ba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff338003820 tx=0x7ff338009040 comp rx=0 tx=0).ready entity=mgr.9418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.515+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff33c04e320 con 0x7ff34806ed80 2023-12-19T16:11:17.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.707+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff34806f400 con 0x7ff3340606f0 2023-12-19T16:11:17.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.723+0000 7ff32f7fe700 1 -- 172.21.15.50:0/1454105200 <== mgr.9418 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7ff338003cf0 con 0x7ff3340606f0 2023-12-19T16:11:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:17.728 INFO:teuthology.orchestra.run.smithi050.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T15:55:34", "first": "2023-12-19T15:56:00", "last": "2023-12-19T16:11:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-12-19T16:11:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 msgr2=0x7ff334062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 0x7ff334062ba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff338003820 tx=0x7ff338009040 comp rx=0 tx=0).stop 2023-12-19T16:11:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 msgr2=0x7ff3480b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009f80 tx=0x7ff33c017000 comp rx=0 tx=0).stop 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 shutdown_connections 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff3340606f0 0x7ff334062ba0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff348158da0 0x7ff3480b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff34806ed80 0x7ff3480b2d60 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 --2- 172.21.15.50:0/1454105200 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff34806df10 0x7ff3480c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 >> 172.21.15.50:0/1454105200 conn(0x7ff3480cc760 msgr2=0x7ff348155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 shutdown_connections 2023-12-19T16:11:17.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.727+0000 7ff34ce6b700 1 -- 172.21.15.50:0/1454105200 wait complete. 2023-12-19T16:11:17.752 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T15:55:34", "first": "2023-12-19T15:56:00", "last": "2023-12-19T16:11:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-12-19T16:11:17.752 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T15:55:34', 'first': '2023-12-19T15:56:00', 'last': '2023-12-19T16:11:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-12-19T16:11:17.752 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-12-19T16:11:17.752 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-12-19T16:11:17.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T16:11:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.923+0000 7f10e18cd700 1 Processor -- start 2023-12-19T16:11:17.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.923+0000 7f10e18cd700 1 -- start start 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.923+0000 7f10e18cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.923+0000 7f10e18cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f10dc0ae270 0x7f10dc0c8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10e18cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0c7130 con 0x7f10dc06e0f0 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10e18cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc156140 con 0x7f10dc0ae270 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10e18cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0c7410 con 0x7f10dc06e4c0 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10daffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f10dc06e0f0 0x7f10dc0c68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55468/0 (socket says 172.21.15.50:55468) 2023-12-19T16:11:17.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10daffd700 1 -- 172.21.15.50:0/520224839 learned_addr learned my addr 172.21.15.50:0/520224839 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10da7fc700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 --2- 172.21.15.50:0/520224839 >> v2:172.21.15.125:3300/0 conn(0x7f10dc0ae270 0x7f10dc0c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10d9ffb700 1 -- 172.21.15.50:0/520224839 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267434684 0 0) 0x7f10dc0c7130 con 0x7f10dc06e0f0 2023-12-19T16:11:17.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10d9ffb700 1 -- 172.21.15.50:0/520224839 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c4003570 con 0x7f10dc06e0f0 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 -- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 msgr2=0x7f10dc0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 -- 172.21.15.50:0/520224839 >> v1:172.21.15.50:6789/0 conn(0x7f10dc06e0f0 legacy=0x7f10dc0c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 -- 172.21.15.50:0/520224839 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0c76f0 con 0x7f10dc0ae270 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10db7fe700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c8720 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009e60 tx=0x7f10d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9496939c0111e30d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10d9ffb700 1 -- 172.21.15.50:0/520224839 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400e070 con 0x7f10dc0ae270 2023-12-19T16:11:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10d9ffb700 1 -- 172.21.15.50:0/520224839 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d400bab0 con 0x7f10dc0ae270 2023-12-19T16:11:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10d9ffb700 1 -- 172.21.15.50:0/520224839 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400be70 con 0x7f10dc0ae270 2023-12-19T16:11:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10e18cd700 1 -- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 msgr2=0x7f10dc0c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.927+0000 7f10e18cd700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c8720 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009e60 tx=0x7f10d4009780 comp rx=0 tx=0).stop 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 -- 172.21.15.50:0/520224839 shutdown_connections 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c8720 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 --2- 172.21.15.50:0/520224839 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 -- 172.21.15.50:0/520224839 >> 172.21.15.50:0/520224839 conn(0x7f10dc0c8db0 msgr2=0x7f10dc0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 -- 172.21.15.50:0/520224839 shutdown_connections 2023-12-19T16:11:17.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 -- 172.21.15.50:0/520224839 wait complete. 2023-12-19T16:11:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 Processor -- start 2023-12-19T16:11:17.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.931+0000 7f10e18cd700 1 -- start start 2023-12-19T16:11:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc156280 con 0x7f10dc06e0f0 2023-12-19T16:11:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc156b20 con 0x7f10dc0ae270 2023-12-19T16:11:17.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc155d10 con 0x7f10dc06e4c0 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36558/0 (socket says 172.21.15.50:36558) 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 -- 172.21.15.50:0/662343400 learned_addr learned my addr 172.21.15.50:0/662343400 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10da7fc700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10db7fe700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 -- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 msgr2=0x7f10dc0c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 -- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 msgr2=0x7f10dc0c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 -- 172.21.15.50:0/662343400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d4009a20 con 0x7f10dc06e0f0 2023-12-19T16:11:17.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10db7fe700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10daffd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c7f0 tx=0x7f10cc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc005b30 con 0x7f10dc06e0f0 2023-12-19T16:11:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc005cb0 con 0x7f10dc06e0f0 2023-12-19T16:11:17.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc007320 con 0x7f10dc06e0f0 2023-12-19T16:11:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc0c4c70 con 0x7f10dc06e0f0 2023-12-19T16:11:17.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.935+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc1567b0 con 0x7f10dc06e0f0 2023-12-19T16:11:17.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.939+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc071120 con 0x7f10dc06e0f0 2023-12-19T16:11:17.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.939+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 87) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f10cc01b3b0 con 0x7f10dc06e0f0 2023-12-19T16:11:17.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.939+0000 7f10c2ffd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 0x7f10c8062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.943+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f10cc07c3d0 con 0x7f10dc06e0f0 2023-12-19T16:11:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.943+0000 7f10da7fc700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 0x7f10c8062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:17.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.943+0000 7f10da7fc700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 0x7f10c8062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0c27b0 tx=0x7f10d00091a0 comp rx=0 tx=0).ready entity=mgr.9418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:17.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:17.947+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10cc04c3d0 con 0x7f10dc06e0f0 2023-12-19T16:11:18.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.143+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f10dc156c60 con 0x7f10c80606c0 2023-12-19T16:11:18.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.259+0000 7f10c2ffd700 1 -- 172.21.15.50:0/662343400 <== mgr.9418 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f10d0009ad0 con 0x7f10c80606c0 2023-12-19T16:11:18.265 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.263+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 msgr2=0x7f10c8062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 0x7f10c8062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0c27b0 tx=0x7f10d00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 msgr2=0x7f10dc1557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c7f0 tx=0x7f10cc007e00 comp rx=0 tx=0).stop 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 shutdown_connections 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f10c80606c0 0x7f10c8062b70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:18.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10dc0ae270 0x7f10dc0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f10dc06e4c0 0x7f10dc0c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 --2- 172.21.15.50:0/662343400 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10dc06e0f0 0x7f10dc1557d0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 >> 172.21.15.50:0/662343400 conn(0x7f10dc0c8db0 msgr2=0x7f10dc15ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 shutdown_connections 2023-12-19T16:11:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:18.267+0000 7f10e18cd700 1 -- 172.21.15.50:0/662343400 wait complete. 2023-12-19T16:11:18.293 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-12-19T16:11:18.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.294 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T16:11:18.327 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_56_00_UTC 2023-12-19T16:11:18.327 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_57_00_UTC 2023-12-19T16:11:18.327 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_58_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-15_59_01_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_00_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_01_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_02_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_03_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_04_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_05_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_06_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_07_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_08_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_09_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_10_00_UTC 2023-12-19T16:11:18.328 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_11_00_UTC 2023-12-19T16:11:18.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_56_00_UTC 2023-12-19T16:11:18.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.330 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_56_00_UTC) 2023-12-19T16:11:18.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_57_00_UTC 2023-12-19T16:11:18.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.414 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_57_00_UTC) 2023-12-19T16:11:18.498 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_58_00_UTC 2023-12-19T16:11:18.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.498 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_58_00_UTC) 2023-12-19T16:11:18.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_59_01_UTC 2023-12-19T16:11:18.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.583 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-15_59_01_UTC) 2023-12-19T16:11:18.670 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_00_00_UTC 2023-12-19T16:11:18.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.671 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_00_00_UTC) 2023-12-19T16:11:18.751 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_01_00_UTC 2023-12-19T16:11:18.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.751 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_01_00_UTC) 2023-12-19T16:11:18.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_02_00_UTC 2023-12-19T16:11:18.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.832 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_02_00_UTC) 2023-12-19T16:11:18.912 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_03_00_UTC 2023-12-19T16:11:18.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.912 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_03_00_UTC) 2023-12-19T16:11:18.993 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_04_00_UTC 2023-12-19T16:11:18.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:18.994 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_04_00_UTC) 2023-12-19T16:11:19.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_05_00_UTC 2023-12-19T16:11:19.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.082 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_05_00_UTC) 2023-12-19T16:11:19.166 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_06_00_UTC 2023-12-19T16:11:19.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.166 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_06_00_UTC) 2023-12-19T16:11:19.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_07_00_UTC 2023-12-19T16:11:19.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.252 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_07_00_UTC) 2023-12-19T16:11:19.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_08_00_UTC 2023-12-19T16:11:19.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.330 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_08_00_UTC) 2023-12-19T16:11:19.413 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_09_00_UTC 2023-12-19T16:11:19.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.413 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_09_00_UTC) 2023-12-19T16:11:19.491 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_10_00_UTC 2023-12-19T16:11:19.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.492 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_10_00_UTC) 2023-12-19T16:11:19.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_11_00_UTC 2023-12-19T16:11:19.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.573 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-16_11_00_UTC) 2023-12-19T16:11:19.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:19.654 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-12-19T16:11:19.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T16:11:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.955+0000 7f070359e700 1 Processor -- start 2023-12-19T16:11:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.955+0000 7f070359e700 1 -- start start 2023-12-19T16:11:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.959+0000 7f070359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0704142680 0x7f07040c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.959+0000 7f070359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f07040c8f40 0x7f070406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.959+0000 7f070359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040cc7e0 con 0x7f0704142a50 2023-12-19T16:11:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.959+0000 7f070359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07040cbbf0 con 0x7f0704142680 2023-12-19T16:11:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.959+0000 7f070359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040cbeb0 con 0x7f07040c8f40 2023-12-19T16:11:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0702d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0704142a50 0x7f07040ca7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55482/0 (socket says 172.21.15.50:55482) 2023-12-19T16:11:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0702d9d700 1 -- 172.21.15.50:0/1112647638 learned_addr learned my addr 172.21.15.50:0/1112647638 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 --2- 172.21.15.50:0/1112647638 >> v2:172.21.15.125:3300/0 conn(0x7f0704142680 0x7f07040c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f06fbfff700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f07040c8f40 0x7f070406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 -- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f07040c8f40 msgr2=0x7f070406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f07040c8f40 0x7f070406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 -- 172.21.15.50:0/1112647638 >> v1:172.21.15.50:6789/0 conn(0x7f0704142a50 legacy=0x7f07040ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 -- 172.21.15.50:0/1112647638 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040cc190 con 0x7f0704142680 2023-12-19T16:11:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0701d9b700 1 -- 172.21.15.50:0/1112647638 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428332381 0 0) 0x7f07040cc7e0 con 0x7f0704142a50 2023-12-19T16:11:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070259c700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0704142680 0x7f07040c8a00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f06f40098e0 tx=0x7f06f40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00954af490b5d0e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0701d9b700 1 -- 172.21.15.50:0/1112647638 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f400b640 con 0x7f0704142680 2023-12-19T16:11:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0701d9b700 1 -- 172.21.15.50:0/1112647638 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f400b7a0 con 0x7f0704142680 2023-12-19T16:11:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f0701d9b700 1 -- 172.21.15.50:0/1112647638 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f400bb40 con 0x7f0704142680 2023-12-19T16:11:19.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 -- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0704142680 msgr2=0x7f07040c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:19.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0704142680 0x7f07040c8a00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f06f40098e0 tx=0x7f06f40091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 -- 172.21.15.50:0/1112647638 shutdown_connections 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f07040c8f40 0x7f070406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 --2- 172.21.15.50:0/1112647638 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0704142680 0x7f07040c8a00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 -- 172.21.15.50:0/1112647638 >> 172.21.15.50:0/1112647638 conn(0x7f07040ccb30 msgr2=0x7f07040ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 -- 172.21.15.50:0/1112647638 shutdown_connections 2023-12-19T16:11:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.963+0000 7f070359e700 1 -- 172.21.15.50:0/1112647638 wait complete. 2023-12-19T16:11:19.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 Processor -- start 2023-12-19T16:11:19.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 -- start start 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 0x7f07040c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0704142680 0x7f0704169920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070413c0b0 con 0x7f0704142a50 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070416a4c0 con 0x7f07040c8f40 2023-12-19T16:11:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f070359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c5650 con 0x7f0704142680 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.967+0000 7f0702d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36560/0 (socket says 172.21.15.50:36560) 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 -- 172.21.15.50:0/3101996063 learned_addr learned my addr 172.21.15.50:0/3101996063 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f070259c700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 0x7f07040c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f06fbfff700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0704142680 0x7f0704169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 -- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0704142680 msgr2=0x7f0704169920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0704142680 0x7f0704169920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 -- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 msgr2=0x7f07040c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 0x7f07040c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 -- 172.21.15.50:0/3101996063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4009420 con 0x7f0704142a50 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f070259c700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 0x7f07040c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f0702d9d700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00cbf0 tx=0x7f06fc0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc00a8b0 con 0x7f0704142a50 2023-12-19T16:11:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc00aa10 con 0x7f0704142a50 2023-12-19T16:11:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc00add0 con 0x7f0704142a50 2023-12-19T16:11:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07040c5930 con 0x7f0704142a50 2023-12-19T16:11:19.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07040c5ef0 con 0x7f0704142a50 2023-12-19T16:11:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.971+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07040c6110 con 0x7f0704142a50 2023-12-19T16:11:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.975+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 87) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f06fc007300 con 0x7f0704142a50 2023-12-19T16:11:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.975+0000 7f06fa7fc700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 0x7f06e4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.975+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f06fc07d0e0 con 0x7f0704142a50 2023-12-19T16:11:19.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.975+0000 7f070259c700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 0x7f06e4062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.979+0000 7f070259c700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 0x7f06e4062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f06f40093f0 tx=0x7f06f4017000 comp rx=0 tx=0).ready entity=mgr.9418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:19.983+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f06fc04d160 con 0x7f0704142a50 2023-12-19T16:11:20.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:20.191+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f070404be40 con 0x7f0704142a50 2023-12-19T16:11:21.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.199+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f06fc056080 con 0x7f0704142a50 2023-12-19T16:11:21.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.203+0000 7f06fa7fc700 1 -- 172.21.15.50:0/3101996063 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v88) v1 ==== 93+0+0 (secure 0 0 0) 0x7f06fc050e20 con 0x7f0704142a50 2023-12-19T16:11:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 msgr2=0x7f06e4062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 0x7f06e4062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f06f40093f0 tx=0x7f06f4017000 comp rx=0 tx=0).stop 2023-12-19T16:11:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 msgr2=0x7f07040c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00cbf0 tx=0x7f06fc0019b0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 shutdown_connections 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f06e40606c0 0x7f06e4062b70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0704142a50 0x7f07040c45f0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0704142680 0x7f0704169920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 --2- 172.21.15.50:0/3101996063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07040c8f40 0x7f07040c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 >> 172.21.15.50:0/3101996063 conn(0x7f07040ccb30 msgr2=0x7f07040aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 shutdown_connections 2023-12-19T16:11:21.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.207+0000 7f070359e700 1 -- 172.21.15.50:0/3101996063 wait complete. 2023-12-19T16:11:21.238 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T16:11:21.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:11:21.279 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:11:21.324 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T16:11:21.324 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T16:11:21.325 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T16:11:21.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:21.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[58286]: fuse finished with error 0 and tester_r 0 2023-12-19T16:11:21.360 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:11:21.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:21.360 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T16:11:21.405 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:21.396+0000 7f089b76b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:11:21.417 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:21.411+0000 7fb2dfbce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:11:21.547 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:11:21.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:21.548 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T16:11:21.569 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T16:11:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:11:21.570 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:11:21.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:21.570 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T16:11:21.610 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T16:11:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:11:21.612 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:11:21.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:21.612 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T16:11:21.632 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T16:11:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:11:21.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:21.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.815+0000 7ff189985700 1 Processor -- start 2023-12-19T16:11:21.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.815+0000 7ff189985700 1 -- start start 2023-12-19T16:11:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff189985700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff1841502b0 0x7ff184068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff189985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff189985700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840c6630 con 0x7ff184067c90 2023-12-19T16:11:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff189985700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840c6910 con 0x7ff1841502b0 2023-12-19T16:11:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff189985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840c6bf0 con 0x7ff1840c5900 2023-12-19T16:11:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff1841502b0 0x7ff184068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff182ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff184067c90 0x7ff18414fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:43958/0 (socket says 172.21.15.50:43958) 2023-12-19T16:11:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff182ffd700 1 -- 172.21.15.50:0/1314181872 learned_addr learned my addr 172.21.15.50:0/1314181872 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:21.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1837fe700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 -- 172.21.15.50:0/1314181872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 msgr2=0x7ff1840c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 -- 172.21.15.50:0/1314181872 >> v1:172.21.15.125:6789/0 conn(0x7ff184067c90 legacy=0x7ff18414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 -- 172.21.15.50:0/1314181872 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840c6ed0 con 0x7ff1841502b0 2023-12-19T16:11:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff181ffb700 1 -- 172.21.15.50:0/1314181872 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270244307 0 0) 0x7ff1840c6630 con 0x7ff184067c90 2023-12-19T16:11:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff1827fc700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841502b0 0x7ff184068060 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff1780098e0 tx=0x7ff1780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d80edcd811d470 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.819+0000 7ff181ffb700 1 -- 172.21.15.50:0/1314181872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800b640 con 0x7ff1841502b0 2023-12-19T16:11:21.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff181ffb700 1 -- 172.21.15.50:0/1314181872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17800b7a0 con 0x7ff1841502b0 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff181ffb700 1 -- 172.21.15.50:0/1314181872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800bb40 con 0x7ff1841502b0 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- 172.21.15.50:0/1314181872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841502b0 msgr2=0x7ff184068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841502b0 0x7ff184068060 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff1780098e0 tx=0x7ff1780091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- 172.21.15.50:0/1314181872 shutdown_connections 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 --2- 172.21.15.50:0/1314181872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1841502b0 0x7ff184068060 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- 172.21.15.50:0/1314181872 >> 172.21.15.50:0/1314181872 conn(0x7ff1840cd6d0 msgr2=0x7ff18406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- 172.21.15.50:0/1314181872 shutdown_connections 2023-12-19T16:11:21.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- 172.21.15.50:0/1314181872 wait complete. 2023-12-19T16:11:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 Processor -- start 2023-12-19T16:11:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.823+0000 7ff189985700 1 -- start start 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 0x7ff1840c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 0x7ff1840c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841698a0 con 0x7ff184067c90 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff184151e60 con 0x7ff1841502b0 2023-12-19T16:11:21.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840ca230 con 0x7ff1840c5900 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff182ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 0x7ff1840c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 0x7ff1840c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1837fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 0x7ff1840c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:52428/0 (socket says 172.21.15.50:52428) 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff182ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 0x7ff1840c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36582/0 (socket says 172.21.15.50:36582) 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:47944/0 (socket says 172.21.15.50:47944) 2023-12-19T16:11:21.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1837fe700 1 -- 172.21.15.50:0/1081845079 learned_addr learned my addr 172.21.15.50:0/1081845079 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 msgr2=0x7ff1840c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 0x7ff1840c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 msgr2=0x7ff1840c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 0x7ff1840c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 -- 172.21.15.50:0/1081845079 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178009420 con 0x7ff1840c5900 2023-12-19T16:11:21.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff1827fc700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff178016000 tx=0x7ff178017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800bb20 con 0x7ff1840c5900 2023-12-19T16:11:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178000c90 con 0x7ff1840c5900 2023-12-19T16:11:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1840ca510 con 0x7ff1840c5900 2023-12-19T16:11:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178011630 con 0x7ff1840c5900 2023-12-19T16:11:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.827+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff184151af0 con 0x7ff1840c5900 2023-12-19T16:11:21.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18406b920 con 0x7ff1840c5900 2023-12-19T16:11:21.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7ff1780119e0 con 0x7ff1840c5900 2023-12-19T16:11:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff16affd700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 0x7ff170062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7ff17807e0d0 con 0x7ff1840c5900 2023-12-19T16:11:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff182ffd700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 msgr2=0x7ff170062bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:21.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.831+0000 7ff182ffd700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 0x7ff170062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:21.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:21.843+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff18406b920 con 0x7ff1840c5900 2023-12-19T16:11:21.949 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:21.940+0000 7f089b76b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:11:21.991 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:21.987+0000 7fb2dfbce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:11:22.030 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.024+0000 7f089b76b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:11:22.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.035+0000 7ff182ffd700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 msgr2=0x7ff170062bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:22.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.035+0000 7ff182ffd700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 0x7ff170062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:22.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.051+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff18406b920 con 0x7ff1840c5900 2023-12-19T16:11:22.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.051+0000 7ff16affd700 1 -- 172.21.15.50:0/1081845079 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7ff18406b920 con 0x7ff1840c5900 2023-12-19T16:11:22.057 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:22.057 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T16:11:22.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 msgr2=0x7ff170062bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:22.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 0x7ff170062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 msgr2=0x7ff1840c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff178016000 tx=0x7ff178017000 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 shutdown_connections 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff170060740 0x7ff170062bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1841502b0 0x7ff1840c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff1840c5900 0x7ff1840c77e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 --2- 172.21.15.50:0/1081845079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff184067c90 0x7ff1840c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 >> 172.21.15.50:0/1081845079 conn(0x7ff1840cd6d0 msgr2=0x7ff1840b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 shutdown_connections 2023-12-19T16:11:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.055+0000 7ff189985700 1 -- 172.21.15.50:0/1081845079 wait complete. 2023-12-19T16:11:22.062 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T16:11:22.070 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.067+0000 7fb2dfbce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:11:22.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:22.179 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.172+0000 7f089b76b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:22.231 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.227+0000 7fb2dfbce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.239+0000 7f45b90cd700 1 Processor -- start 2023-12-19T16:11:22.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 -- start start 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45b4067c90 0x7f45b4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45b414fba0 0x7f45b4150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40c6630 con 0x7f45b40c5900 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40c6910 con 0x7f45b4067c90 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b90cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40c6bf0 con 0x7f45b414fba0 2023-12-19T16:11:22.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b2d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45b4067c90 0x7f45b4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b2d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f45b4067c90 0x7f45b4068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36594/0 (socket says 172.21.15.50:36594) 2023-12-19T16:11:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.243+0000 7f45b2d9d700 1 -- 172.21.15.50:0/2888423395 learned_addr learned my addr 172.21.15.50:0/2888423395 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:22.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b259c700 1 --2- 172.21.15.50:0/2888423395 >> v2:172.21.15.125:3300/0 conn(0x7f45b414fba0 0x7f45b4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b2d9d700 1 -- 172.21.15.50:0/2888423395 >> v2:172.21.15.125:3300/0 conn(0x7f45b414fba0 msgr2=0x7f45b4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b2d9d700 1 --2- 172.21.15.50:0/2888423395 >> v2:172.21.15.125:3300/0 conn(0x7f45b414fba0 0x7f45b4150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b2d9d700 1 -- 172.21.15.50:0/2888423395 >> v1:172.21.15.50:6789/0 conn(0x7f45b40c5900 legacy=0x7f45b40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b2d9d700 1 -- 172.21.15.50:0/2888423395 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40c6ed0 con 0x7f45b4067c90 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b1d9b700 1 -- 172.21.15.50:0/2888423395 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951094436 0 0) 0x7f45b40c6630 con 0x7f45b40c5900 2023-12-19T16:11:22.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b2d9d700 1 --2- 172.21.15.50:0/2888423395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b4067c90 0x7f45b4068060 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f45b40c56b0 tx=0x7f45a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2459f4ba8a67260 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b1d9b700 1 -- 172.21.15.50:0/2888423395 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4009e70 con 0x7f45b4067c90 2023-12-19T16:11:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b1d9b700 1 -- 172.21.15.50:0/2888423395 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a400b860 con 0x7f45b4067c90 2023-12-19T16:11:22.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b1d9b700 1 -- 172.21.15.50:0/2888423395 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a400bc00 con 0x7f45b4067c90 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 -- 172.21.15.50:0/2888423395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b4067c90 msgr2=0x7f45b4068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 --2- 172.21.15.50:0/2888423395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b4067c90 0x7f45b4068060 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f45b40c56b0 tx=0x7f45a4009250 comp rx=0 tx=0).stop 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 -- 172.21.15.50:0/2888423395 shutdown_connections 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 --2- 172.21.15.50:0/2888423395 >> v2:172.21.15.125:3300/0 conn(0x7f45b414fba0 0x7f45b4150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 --2- 172.21.15.50:0/2888423395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b4067c90 0x7f45b4068060 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 -- 172.21.15.50:0/2888423395 >> 172.21.15.50:0/2888423395 conn(0x7f45b40cd6d0 msgr2=0x7f45b406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 -- 172.21.15.50:0/2888423395 shutdown_connections 2023-12-19T16:11:22.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.247+0000 7f45b90cd700 1 -- 172.21.15.50:0/2888423395 wait complete. 2023-12-19T16:11:22.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.251+0000 7f45b90cd700 1 Processor -- start 2023-12-19T16:11:22.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.251+0000 7f45b90cd700 1 -- start start 2023-12-19T16:11:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45b4067c90 0x7f45b414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 0x7f45b40b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40c9370 con 0x7f45b40c5900 2023-12-19T16:11:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40c8d60 con 0x7f45b414fba0 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40b3dd0 con 0x7f45b4067c90 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36602/0 (socket says 172.21.15.50:36602) 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 -- 172.21.15.50:0/1509655635 learned_addr learned my addr 172.21.15.50:0/1509655635 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b2d9d700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45b4067c90 0x7f45b414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b359e700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 0x7f45b40b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 -- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45b4067c90 msgr2=0x7f45b414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45b4067c90 0x7f45b414c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 -- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 msgr2=0x7f45b40b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 0x7f45b40b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 -- 172.21.15.50:0/1509655635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40094f0 con 0x7f45b40c5900 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b359e700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 0x7f45b40b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b259c700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f45a800e930 tx=0x7f45a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8013680 con 0x7f45b40c5900 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a80137e0 con 0x7f45b40c5900 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b40c9000 con 0x7f45b40c5900 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8013b80 con 0x7f45b40c5900 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b40b40b0 con 0x7f45b40c5900 2023-12-19T16:11:22.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.255+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b40c94e0 con 0x7f45b40c5900 2023-12-19T16:11:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.259+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f45a800c460 con 0x7f45b40c5900 2023-12-19T16:11:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.259+0000 7f459affd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 0x7f45a0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.259+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f45a807cf60 con 0x7f45b40c5900 2023-12-19T16:11:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.259+0000 7f45b2d9d700 1 -- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 msgr2=0x7f45a0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.259+0000 7f45b2d9d700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 0x7f45a0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:22.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.271+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f45a804cf60 con 0x7f45b40c5900 2023-12-19T16:11:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.443+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45b4071120 con 0x7f45b40c5900 2023-12-19T16:11:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.443+0000 7f459affd700 1 -- 172.21.15.50:0/1509655635 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16727 (secure 0 0 0) 0x7f45a8050580 con 0x7f45b40c5900 2023-12-19T16:11:22.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:22.450 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":185,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:06:21.086411+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.10","mappings":[{"from":4,"to":0}]},{"pgid":"22.23","mappings":[{"from":3,"to":2}]},{"pgid":"22.28","mappings":[{"from":3,"to":0}]},{"pgid":"22.33","mappings":[{"from":3,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3295762643":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1719917206":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4089126800":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/2491435626":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4259555793":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/1927379651":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/2545221666":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1055917704":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3978276206":"2023-12-20T15:55:33.938573+0000","172.21.15.125:6833/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3093853131":"2023-12-20T16:06:21.086343+0000","172.21.15.125:6832/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3363816454":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/932161281":"2023-12-20T15:55:33.938573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 msgr2=0x7f45a0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 0x7f45a0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 msgr2=0x7f45b40c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f45a800e930 tx=0x7f45a80091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 shutdown_connections 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f45a0060670 0x7f45a0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45b414fba0 0x7f45b40b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45b40c5900 0x7f45b40c9800 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 --2- 172.21.15.50:0/1509655635 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f45b4067c90 0x7f45b414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 >> 172.21.15.50:0/1509655635 conn(0x7f45b40cd6d0 msgr2=0x7f45b414ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 shutdown_connections 2023-12-19T16:11:22.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.447+0000 7f45b90cd700 1 -- 172.21.15.50:0/1509655635 wait complete. 2023-12-19T16:11:22.453 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.444+0000 7f089b76b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:11:22.475 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T15:55:16.440898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T15:55:17.404856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T16:11:22.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:11:22.519 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.515+0000 7fb2dfbce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:11:22.529 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.524+0000 7f089b76b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:22.596 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.591+0000 7fb2dfbce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:22.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.639+0000 7f42a56e3700 1 Processor -- start 2023-12-19T16:11:22.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.639+0000 7f42a56e3700 1 -- start start 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.639+0000 7f42a56e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a0158c90 0x7f42a00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a56e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42a006aeb0 0x7f42a0159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a56e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42a01595a0 0x7f42a0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a56e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a006c850 con 0x7f42a01595a0 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a56e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a006b470 con 0x7f42a006aeb0 2023-12-19T16:11:22.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a56e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a006b750 con 0x7f42a0158c90 2023-12-19T16:11:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42a01595a0 0x7f42a0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f42a01595a0 0x7f42a0159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36604/0 (socket says 172.21.15.50:36604) 2023-12-19T16:11:22.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 -- 172.21.15.50:0/3012492291 learned_addr learned my addr 172.21.15.50:0/3012492291 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f429ffff700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a0158c90 0x7f42a00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f4297fff700 1 --2- 172.21.15.50:0/3012492291 >> v2:172.21.15.125:3300/0 conn(0x7f42a006aeb0 0x7f42a0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 -- 172.21.15.50:0/3012492291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a0158c90 msgr2=0x7f42a00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a0158c90 0x7f42a00caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 -- 172.21.15.50:0/3012492291 >> v2:172.21.15.125:3300/0 conn(0x7f42a006aeb0 msgr2=0x7f42a0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 --2- 172.21.15.50:0/3012492291 >> v2:172.21.15.125:3300/0 conn(0x7f42a006aeb0 0x7f42a0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 -- 172.21.15.50:0/3012492291 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a006ba30 con 0x7f42a01595a0 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f42a4ee2700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a01595a0 0x7f42a0159a20 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f42a006a000 tx=0x7f4298009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d83c55776143850 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f429f7fe700 1 -- 172.21.15.50:0/3012492291 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429800e070 con 0x7f42a01595a0 2023-12-19T16:11:22.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f429f7fe700 1 -- 172.21.15.50:0/3012492291 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429800b970 con 0x7f42a01595a0 2023-12-19T16:11:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.643+0000 7f429f7fe700 1 -- 172.21.15.50:0/3012492291 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980092e0 con 0x7f42a01595a0 2023-12-19T16:11:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- 172.21.15.50:0/3012492291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a01595a0 msgr2=0x7f42a0159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a01595a0 0x7f42a0159a20 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f42a006a000 tx=0x7f4298009840 comp rx=0 tx=0).stop 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- 172.21.15.50:0/3012492291 shutdown_connections 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a01595a0 0x7f42a0159a20 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3012492291 >> v2:172.21.15.125:3300/0 conn(0x7f42a006aeb0 0x7f42a0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3012492291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a0158c90 0x7f42a00caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- 172.21.15.50:0/3012492291 >> 172.21.15.50:0/3012492291 conn(0x7f42a0067bb0 msgr2=0x7f42a00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- 172.21.15.50:0/3012492291 shutdown_connections 2023-12-19T16:11:22.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- 172.21.15.50:0/3012492291 wait complete. 2023-12-19T16:11:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 Processor -- start 2023-12-19T16:11:22.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.647+0000 7f42a56e3700 1 -- start start 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a006aeb0 0x7f42a0169af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42a01595a0 0x7f42a0156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a01580d0 con 0x7f42a0158c90 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a01574c0 con 0x7f42a01595a0 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a01577a0 con 0x7f42a006aeb0 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36620/0 (socket says 172.21.15.50:36620) 2023-12-19T16:11:22.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 -- 172.21.15.50:0/3950189350 learned_addr learned my addr 172.21.15.50:0/3950189350 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f429ffff700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a006aeb0 0x7f42a0169af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a4ee2700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42a01595a0 0x7f42a0156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 -- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a006aeb0 msgr2=0x7f42a0169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a006aeb0 0x7f42a0169af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 -- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42a01595a0 msgr2=0x7f42a0156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42a01595a0 0x7f42a0156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 -- 172.21.15.50:0/3950189350 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298009ae0 con 0x7f42a0158c90 2023-12-19T16:11:22.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f4297fff700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f428800ec80 tx=0x7f42880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428800c9d0 con 0x7f42a0158c90 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428800cb30 con 0x7f42a0158c90 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288013470 con 0x7f42a0158c90 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a0157a80 con 0x7f42a0158c90 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.651+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a016a030 con 0x7f42a0158c90 2023-12-19T16:11:22.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a004be40 con 0x7f42a0158c90 2023-12-19T16:11:22.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f4288013820 con 0x7f42a0158c90 2023-12-19T16:11:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f429cff9700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 0x7f4284062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7f428807d170 con 0x7f42a0158c90 2023-12-19T16:11:22.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f429ffff700 1 -- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 msgr2=0x7f4284062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:22.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.655+0000 7f429ffff700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 0x7f4284062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:22.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.663+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f428804d170 con 0x7f42a0158c90 2023-12-19T16:11:22.754 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.748+0000 7f089b76b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:11:22.827 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.823+0000 7fb2dfbce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:11:22.843 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.836+0000 7f089b76b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:11:22.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.839+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42a0157f00 con 0x7f42a0158c90 2023-12-19T16:11:22.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.839+0000 7f429cff9700 1 -- 172.21.15.50:0/3950189350 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v185) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4288050790 con 0x7f42a0158c90 2023-12-19T16:11:22.845 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 msgr2=0x7f4284062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 0x7f4284062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 msgr2=0x7f42a0156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f428800ec80 tx=0x7f42880091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 shutdown_connections 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4284060670 0x7f4284062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42a01595a0 0x7f42a0156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42a0158c90 0x7f42a0156970 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 --2- 172.21.15.50:0/3950189350 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f42a006aeb0 0x7f42a0169af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 >> 172.21.15.50:0/3950189350 conn(0x7f42a0067bb0 msgr2=0x7f42a01599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 shutdown_connections 2023-12-19T16:11:22.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:22.843+0000 7f42a56e3700 1 -- 172.21.15.50:0/3950189350 wait complete. 2023-12-19T16:11:22.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T16:11:22.920 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:22.911+0000 7fb2dfbce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:11:22.935 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:22.928+0000 7f089b76b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:11:23.022 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:23.019+0000 7fb2dfbce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:11:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.079+0000 7fc8936f0700 1 Processor -- start 2023-12-19T16:11:23.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.079+0000 7fc8936f0700 1 -- start start 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc8936f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc88c15d410 0x7fc88c0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc8936f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc88c06b1b0 0x7fc88c06b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc8936f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c0c7450 con 0x7fc88c0ca400 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc8936f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c06bea0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc8936f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0c6e00 con 0x7fc88c15d410 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc88c06b1b0 0x7fc88c06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc88c06b1b0 0x7fc88c06b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36624/0 (socket says 172.21.15.50:36624) 2023-12-19T16:11:23.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 -- 172.21.15.50:0/1547927467 learned_addr learned my addr 172.21.15.50:0/1547927467 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc89148c700 1 --2- 172.21.15.50:0/1547927467 >> v2:172.21.15.125:3300/0 conn(0x7fc88c15d410 0x7fc88c0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 -- 172.21.15.50:0/1547927467 >> v2:172.21.15.125:3300/0 conn(0x7fc88c15d410 msgr2=0x7fc88c0ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 --2- 172.21.15.50:0/1547927467 >> v2:172.21.15.125:3300/0 conn(0x7fc88c15d410 0x7fc88c0ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 -- 172.21.15.50:0/1547927467 >> v1:172.21.15.125:6789/0 conn(0x7fc88c0ca400 legacy=0x7fc88c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 -- 172.21.15.50:0/1547927467 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c0c70e0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc891c8d700 1 --2- 172.21.15.50:0/1547927467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c06b1b0 0x7fc88c06b650 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc88c068030 tx=0x7fc884009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c8ddbed38bf302 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc883fff700 1 -- 172.21.15.50:0/1547927467 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88400e070 con 0x7fc88c06b1b0 2023-12-19T16:11:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc883fff700 1 -- 172.21.15.50:0/1547927467 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88400ba20 con 0x7fc88c06b1b0 2023-12-19T16:11:23.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.083+0000 7fc883fff700 1 -- 172.21.15.50:0/1547927467 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8840092e0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- 172.21.15.50:0/1547927467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c06b1b0 msgr2=0x7fc88c06b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1547927467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c06b1b0 0x7fc88c06b650 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc88c068030 tx=0x7fc884009840 comp rx=0 tx=0).stop 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- 172.21.15.50:0/1547927467 shutdown_connections 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1547927467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c06b1b0 0x7fc88c06b650 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1547927467 >> v2:172.21.15.125:3300/0 conn(0x7fc88c15d410 0x7fc88c0ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- 172.21.15.50:0/1547927467 >> 172.21.15.50:0/1547927467 conn(0x7fc88c067bb0 msgr2=0x7fc88c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- 172.21.15.50:0/1547927467 shutdown_connections 2023-12-19T16:11:23.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- 172.21.15.50:0/1547927467 wait complete. 2023-12-19T16:11:23.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 Processor -- start 2023-12-19T16:11:23.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.087+0000 7fc8936f0700 1 -- start start 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 0x7fc88c0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc88c0ca400 0x7fc88c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 0x7fc88c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c4580 con 0x7fc88c15d410 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c160fd0 con 0x7fc88c0ca400 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c161b50 con 0x7fc88c06b1b0 2023-12-19T16:11:23.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc891c8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 0x7fc88c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc891c8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 0x7fc88c0c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36636/0 (socket says 172.21.15.50:36636) 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 0x7fc88c0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc891c8d700 1 -- 172.21.15.50:0/1891983447 learned_addr learned my addr 172.21.15.50:0/1891983447 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc890c8b700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc88c0ca400 0x7fc88c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc88c0ca400 msgr2=0x7fc88c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc88c0ca400 0x7fc88c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 msgr2=0x7fc88c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 0x7fc88c0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 -- 172.21.15.50:0/1891983447 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884009ae0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc89148c700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 0x7fc88c0ad6e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc88800ea50 tx=0x7fc8880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88800ee90 con 0x7fc88c06b1b0 2023-12-19T16:11:23.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88800c7f0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c161220 con 0x7fc88c06b1b0 2023-12-19T16:11:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88800cb90 con 0x7fc88c06b1b0 2023-12-19T16:11:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.091+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c1617e0 con 0x7fc88c06b1b0 2023-12-19T16:11:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.095+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c071120 con 0x7fc88c06b1b0 2023-12-19T16:11:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.095+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fc888013470 con 0x7fc88c06b1b0 2023-12-19T16:11:23.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.095+0000 7fc8817fa700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 0x7fc870062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.095+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6799+0+0 (secure 0 0 0) 0x7fc88807d310 con 0x7fc88c06b1b0 2023-12-19T16:11:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.099+0000 7fc890c8b700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 msgr2=0x7fc870062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.099+0000 7fc890c8b700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 0x7fc870062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:23.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.107+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc88804d310 con 0x7fc88c06b1b0 2023-12-19T16:11:23.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.303+0000 7fc890c8b700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 msgr2=0x7fc870062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:23.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.303+0000 7fc890c8b700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 0x7fc870062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:23.382 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:23.376+0000 7f089b76b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:11:23.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.391+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc88c064270 con 0x7fc88c06b1b0 2023-12-19T16:11:23.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.391+0000 7fc8817fa700 1 -- 172.21.15.50:0/1891983447 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v186) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc888050930 con 0x7fc88c06b1b0 2023-12-19T16:11:23.398 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T16:11:23.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 msgr2=0x7fc870062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 0x7fc870062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 msgr2=0x7fc88c0ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 0x7fc88c0ad6e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc88800ea50 tx=0x7fc8880091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 shutdown_connections 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc870060670 0x7fc870062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc88c0ca400 0x7fc88c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc88c15d410 0x7fc88c0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 --2- 172.21.15.50:0/1891983447 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc88c06b1b0 0x7fc88c0ad6e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 >> 172.21.15.50:0/1891983447 conn(0x7fc88c067bb0 msgr2=0x7fc88c0c1840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:23.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 shutdown_connections 2023-12-19T16:11:23.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.399+0000 7fc8936f0700 1 -- 172.21.15.50:0/1891983447 wait complete. 2023-12-19T16:11:23.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T16:11:23.507 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:23.503+0000 7fb2dfbce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:11:23.557 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:23.548+0000 7f089b76b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:11:23.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.583+0000 7ff851945700 1 Processor -- start 2023-12-19T16:11:23.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.583+0000 7ff851945700 1 -- start start 2023-12-19T16:11:23.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 0x7ff84c06a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0ac620 con 0x7ff84c156a10 2023-12-19T16:11:23.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0ac060 con 0x7ff84c0abaf0 2023-12-19T16:11:23.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0ac340 con 0x7ff84c156de0 2023-12-19T16:11:23.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84affd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff84c156a10 0x7ff84c0cc360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55498/0 (socket says 172.21.15.50:55498) 2023-12-19T16:11:23.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84affd700 1 -- 172.21.15.50:0/1598502370 learned_addr learned my addr 172.21.15.50:0/1598502370 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:23.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84a7fc700 1 --2- 172.21.15.50:0/1598502370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 0x7ff84c06a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43947130 0 0) 0x7ff84c0ac620 con 0x7ff84c156a10 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff834003570 con 0x7ff84c156a10 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522099712 0 0) 0x7ff84c0ac060 con 0x7ff84c0abaf0 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff84c0ac620 con 0x7ff84c0abaf0 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84a7fc700 1 -- 172.21.15.50:0/1598502370 >> v1:172.21.15.125:6789/0 conn(0x7ff84c0abaf0 legacy=0x7ff84c06d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84a7fc700 1 -- 172.21.15.50:0/1598502370 >> v1:172.21.15.50:6789/0 conn(0x7ff84c156a10 legacy=0x7ff84c0cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84a7fc700 1 -- 172.21.15.50:0/1598502370 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c06c860 con 0x7ff84c156de0 2023-12-19T16:11:23.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff84a7fc700 1 --2- 172.21.15.50:0/1598502370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 0x7ff84c06a580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff840009910 tx=0x7ff8400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3fab778125eda9d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840011630 con 0x7ff84c156de0 2023-12-19T16:11:23.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840011790 con 0x7ff84c156de0 2023-12-19T16:11:23.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff849ffb700 1 -- 172.21.15.50:0/1598502370 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840011b10 con 0x7ff84c156de0 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 -- 172.21.15.50:0/1598502370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 msgr2=0x7ff84c06a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.587+0000 7ff851945700 1 --2- 172.21.15.50:0/1598502370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 0x7ff84c06a580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff840009910 tx=0x7ff8400091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 -- 172.21.15.50:0/1598502370 shutdown_connections 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 --2- 172.21.15.50:0/1598502370 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c156de0 0x7ff84c06a580 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 -- 172.21.15.50:0/1598502370 >> 172.21.15.50:0/1598502370 conn(0x7ff84c067bb0 msgr2=0x7ff84c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 -- 172.21.15.50:0/1598502370 shutdown_connections 2023-12-19T16:11:23.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 -- 172.21.15.50:0/1598502370 wait complete. 2023-12-19T16:11:23.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 Processor -- start 2023-12-19T16:11:23.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.591+0000 7ff851945700 1 -- start start 2023-12-19T16:11:23.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 0x7ff84c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 0x7ff84c0c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff84c0c9d50 0x7ff84c0c1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0c1e90 con 0x7ff84c156a10 2023-12-19T16:11:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0c2170 con 0x7ff84c0c9d50 2023-12-19T16:11:23.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff851945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0c2450 con 0x7ff84c0abaf0 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 0x7ff84c0c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 0x7ff84c0c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36644/0 (socket says 172.21.15.50:36644) 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 0x7ff84c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84a7fc700 1 -- 172.21.15.50:0/1197799579 learned_addr learned my addr 172.21.15.50:0/1197799579 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84b7fe700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff84c0c9d50 0x7ff84c0c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 -- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff84c0c9d50 msgr2=0x7ff84c0c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff84c0c9d50 0x7ff84c0c1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 -- 172.21.15.50:0/1197799579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 msgr2=0x7ff84c0c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 0x7ff84c0c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 -- 172.21.15.50:0/1197799579 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840009420 con 0x7ff84c0abaf0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff84affd700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 0x7ff84c0c9810 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00c580 tx=0x7ff83c007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.595+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c005b30 con 0x7ff84c0abaf0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c005c90 con 0x7ff84c0abaf0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c007300 con 0x7ff84c0abaf0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c0c2730 con 0x7ff84c0abaf0 2023-12-19T16:11:23.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0c1ba0 con 0x7ff84c0abaf0 2023-12-19T16:11:23.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c04be40 con 0x7ff84c0abaf0 2023-12-19T16:11:23.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.599+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7ff83c01b3b0 con 0x7ff84c0abaf0 2023-12-19T16:11:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.603+0000 7ff832ffd700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 0x7ff838062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.603+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7ff83c07c280 con 0x7ff84c0abaf0 2023-12-19T16:11:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.603+0000 7ff84a7fc700 1 -- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 msgr2=0x7ff838062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:23.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.603+0000 7ff84a7fc700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 0x7ff838062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:23.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.611+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff83c04c240 con 0x7ff84c0abaf0 2023-12-19T16:11:23.687 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:23.683+0000 7fb2dfbce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:11:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.791+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff84c0c2590 con 0x7ff84c0abaf0 2023-12-19T16:11:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff832ffd700 1 -- 172.21.15.50:0/1197799579 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v186) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff84c0c2590 con 0x7ff84c0abaf0 2023-12-19T16:11:23.798 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T16:11:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 msgr2=0x7ff838062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 0x7ff838062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 msgr2=0x7ff84c0c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 0x7ff84c0c9810 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00c580 tx=0x7ff83c007d90 comp rx=0 tx=0).stop 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 shutdown_connections 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff838060670 0x7ff838062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff84c0c9d50 0x7ff84c0c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff84c156a10 0x7ff84c0c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 --2- 172.21.15.50:0/1197799579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff84c0abaf0 0x7ff84c0c9810 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 >> 172.21.15.50:0/1197799579 conn(0x7ff84c067bb0 msgr2=0x7ff84c06d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 shutdown_connections 2023-12-19T16:11:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.795+0000 7ff851945700 1 -- 172.21.15.50:0/1197799579 wait complete. 2023-12-19T16:11:23.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:23.954 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:23.948+0000 7f089b76b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:11:23.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 Processor -- start 2023-12-19T16:11:23.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 -- start start 2023-12-19T16:11:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 0x7faf1806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf180ce810 0x7faf180ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1806b730 con 0x7faf181591f0 2023-12-19T16:11:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1806ab20 con 0x7faf18158e20 2023-12-19T16:11:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1c877700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1806ae00 con 0x7faf180ce810 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf16d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 0x7faf1806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.978+0000 7faf1759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf180ce810 0x7faf180ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1659c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faf181591f0 0x7faf180ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:43974/0 (socket says 172.21.15.50:43974) 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1659c700 1 -- 172.21.15.50:0/164277220 learned_addr learned my addr 172.21.15.50:0/164277220 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1759e700 1 -- 172.21.15.50:0/164277220 >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 msgr2=0x7faf1806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1759e700 1 --2- 172.21.15.50:0/164277220 >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 0x7faf1806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1759e700 1 -- 172.21.15.50:0/164277220 >> v1:172.21.15.125:6789/0 conn(0x7faf181591f0 legacy=0x7faf180ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1759e700 1 -- 172.21.15.50:0/164277220 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1806b0e0 con 0x7faf180ce810 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf16d9d700 1 --2- 172.21.15.50:0/164277220 >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 0x7faf1806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1759e700 1 --2- 172.21.15.50:0/164277220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf180ce810 0x7faf180ca950 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7faf1806dc00 tx=0x7faf10009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e845e683e92dee8c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf15d9b700 1 -- 172.21.15.50:0/164277220 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000e070 con 0x7faf180ce810 2023-12-19T16:11:23.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf15d9b700 1 -- 172.21.15.50:0/164277220 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000bb60 con 0x7faf180ce810 2023-12-19T16:11:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf15d9b700 1 -- 172.21.15.50:0/164277220 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10012430 con 0x7faf180ce810 2023-12-19T16:11:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 -- 172.21.15.50:0/164277220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf180ce810 msgr2=0x7faf180ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 --2- 172.21.15.50:0/164277220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf180ce810 0x7faf180ca950 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7faf1806dc00 tx=0x7faf10009780 comp rx=0 tx=0).stop 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 -- 172.21.15.50:0/164277220 shutdown_connections 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 --2- 172.21.15.50:0/164277220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf180ce810 0x7faf180ca950 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 --2- 172.21.15.50:0/164277220 >> v2:172.21.15.50:3300/0 conn(0x7faf18158e20 0x7faf1806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 -- 172.21.15.50:0/164277220 >> 172.21.15.50:0/164277220 conn(0x7faf18067bb0 msgr2=0x7faf180cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 -- 172.21.15.50:0/164277220 shutdown_connections 2023-12-19T16:11:23.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.982+0000 7faf1c877700 1 -- 172.21.15.50:0/164277220 wait complete. 2023-12-19T16:11:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.986+0000 7faf1c877700 1 Processor -- start 2023-12-19T16:11:23.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.986+0000 7faf1c877700 1 -- start start 2023-12-19T16:11:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf180ce810 0x7faf1815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf181591f0 0x7faf18169160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1815a6f0 con 0x7faf18158e20 2023-12-19T16:11:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1815bad0 con 0x7faf181591f0 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf180c7430 con 0x7faf180ce810 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36652/0 (socket says 172.21.15.50:36652) 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 -- 172.21.15.50:0/2606556436 learned_addr learned my addr 172.21.15.50:0/2606556436 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf16d9d700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf180ce810 0x7faf1815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1759e700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf181591f0 0x7faf18169160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:23.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf180ce810 msgr2=0x7faf1815bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf180ce810 0x7faf1815bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf181591f0 msgr2=0x7faf18169160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf181591f0 0x7faf18169160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 -- 172.21.15.50:0/2606556436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08009420 con 0x7faf18158e20 2023-12-19T16:11:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1659c700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c560 tx=0x7faf0c001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:23.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c005b30 con 0x7faf18158e20 2023-12-19T16:11:23.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c005c90 con 0x7faf18158e20 2023-12-19T16:11:23.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf10009a20 con 0x7faf18158e20 2023-12-19T16:11:23.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c007320 con 0x7faf18158e20 2023-12-19T16:11:23.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.990+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf180c79f0 con 0x7faf18158e20 2023-12-19T16:11:23.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1815a860 con 0x7faf18158e20 2023-12-19T16:11:23.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7faf0c01b3b0 con 0x7faf18158e20 2023-12-19T16:11:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faefeffd700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 0x7faf04062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7faf0c07c130 con 0x7faf18158e20 2023-12-19T16:11:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faf16d9d700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 msgr2=0x7faf04062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:24.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:23.994+0000 7faf16d9d700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 0x7faf04062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.002+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faf0c04c190 con 0x7faf18158e20 2023-12-19T16:11:24.112 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.106+0000 7fb2dfbce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:11:24.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.198+0000 7faf16d9d700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 msgr2=0x7faf04062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:24.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.198+0000 7faf16d9d700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 0x7faf04062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:24.229 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.220+0000 7f089b76b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.246+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf1804be40 con 0x7faf18158e20 2023-12-19T16:11:24.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.250+0000 7faefeffd700 1 -- 172.21.15.50:0/2606556436 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7faf0c04f7b0 con 0x7faf18158e20 2023-12-19T16:11:24.254 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:24.254 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 msgr2=0x7faf04062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 0x7faf04062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 msgr2=0x7faf18153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c560 tx=0x7faf0c001020 comp rx=0 tx=0).stop 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 shutdown_connections 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf040606a0 0x7faf04062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf181591f0 0x7faf18169160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf18158e20 0x7faf18153990 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 --2- 172.21.15.50:0/2606556436 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf180ce810 0x7faf1815bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 >> 172.21.15.50:0/2606556436 conn(0x7faf18067bb0 msgr2=0x7faf1815eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 shutdown_connections 2023-12-19T16:11:24.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.254+0000 7faf1c877700 1 -- 172.21.15.50:0/2606556436 wait complete. 2023-12-19T16:11:24.261 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T16:11:24.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:24.339 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.332+0000 7f089b76b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:11:24.372 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.366+0000 7fb2dfbce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:24.420 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.412+0000 7f089b76b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:11:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.434+0000 7f2e99e6e700 1 Processor -- start 2023-12-19T16:11:24.439 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.434+0000 7f2e99e6e700 1 -- start start 2023-12-19T16:11:24.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.434+0000 7f2e99e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e9413f890 0x7f2e940aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.434+0000 7f2e99e6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e940680f0 0x7f2e94068570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.434+0000 7f2e99e6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9406c9f0 con 0x7f2e940c2980 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e94068ab0 con 0x7f2e940680f0 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940c7f20 con 0x7f2e9413f890 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e940680f0 0x7f2e94068570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e940680f0 0x7f2e94068570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36668/0 (socket says 172.21.15.50:36668) 2023-12-19T16:11:24.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 -- 172.21.15.50:0/8946496 learned_addr learned my addr 172.21.15.50:0/8946496 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:24.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e937fe700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e9413f890 0x7f2e940aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 -- 172.21.15.50:0/8946496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e9413f890 msgr2=0x7f2e940aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:24.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e9413f890 0x7f2e940aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 -- 172.21.15.50:0/8946496 >> v1:172.21.15.125:6789/0 conn(0x7f2e940c2980 legacy=0x7f2e940c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 -- 172.21.15.50:0/8946496 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940c1c50 con 0x7f2e940680f0 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e93fff700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e940680f0 0x7f2e94068570 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0099b0 tx=0x7f2e8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51c7dfeeeb9b645 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e927fc700 1 -- 172.21.15.50:0/8946496 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00b860 con 0x7f2e940680f0 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e927fc700 1 -- 172.21.15.50:0/8946496 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c00b9c0 con 0x7f2e940680f0 2023-12-19T16:11:24.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e927fc700 1 -- 172.21.15.50:0/8946496 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00bd60 con 0x7f2e940680f0 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- 172.21.15.50:0/8946496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e940680f0 msgr2=0x7f2e94068570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e940680f0 0x7f2e94068570 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0099b0 tx=0x7f2e8c009250 comp rx=0 tx=0).stop 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- 172.21.15.50:0/8946496 shutdown_connections 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e940680f0 0x7f2e94068570 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/8946496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e9413f890 0x7f2e940aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- 172.21.15.50:0/8946496 >> 172.21.15.50:0/8946496 conn(0x7f2e940c8850 msgr2=0x7f2e940c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- 172.21.15.50:0/8946496 shutdown_connections 2023-12-19T16:11:24.445 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.438+0000 7f2e99e6e700 1 -- 172.21.15.50:0/8946496 wait complete. 2023-12-19T16:11:24.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 Processor -- start 2023-12-19T16:11:24.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 -- start start 2023-12-19T16:11:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 0x7f2e9415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 0x7f2e94169070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9415e7a0 con 0x7f2e9413f890 2023-12-19T16:11:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940af710 con 0x7f2e940680f0 2023-12-19T16:11:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.442+0000 7f2e99e6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9415e1e0 con 0x7f2e940c2980 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36678/0 (socket says 172.21.15.50:36678) 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 -- 172.21.15.50:0/3084953368 learned_addr learned my addr 172.21.15.50:0/3084953368 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e937fe700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 0x7f2e9415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e92ffd700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 0x7f2e94169070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 -- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 msgr2=0x7f2e94169070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 0x7f2e94169070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 -- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 msgr2=0x7f2e9415d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 0x7f2e9415d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 -- 172.21.15.50:0/3084953368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0094f0 con 0x7f2e9413f890 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e92ffd700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 0x7f2e94169070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e937fe700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 0x7f2e9415d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e93fff700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009290 tx=0x7f2e8c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:24.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c010680 con 0x7f2e9413f890 2023-12-19T16:11:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8c010800 con 0x7f2e9413f890 2023-12-19T16:11:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c010bc0 con 0x7f2e9413f890 2023-12-19T16:11:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9415e4c0 con 0x7f2e9413f890 2023-12-19T16:11:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.446+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9415de80 con 0x7f2e9413f890 2023-12-19T16:11:24.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e940af200 con 0x7f2e9413f890 2023-12-19T16:11:24.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f2e8c02c440 con 0x7f2e9413f890 2023-12-19T16:11:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e7b7fe700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 0x7f2e80062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f2e8c014070 con 0x7f2e9413f890 2023-12-19T16:11:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e937fe700 1 -- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 msgr2=0x7f2e80062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:24.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.450+0000 7f2e937fe700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 0x7f2e80062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:24.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.458+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e940af200 con 0x7f2e9413f890 2023-12-19T16:11:24.494 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.490+0000 7fb2dfbce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:11:24.504 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.496+0000 7f089b76b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:11:24.590 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.582+0000 7fb2dfbce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:11:24.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.646+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e940af200 con 0x7f2e9413f890 2023-12-19T16:11:24.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.646+0000 7f2e7b7fe700 1 -- 172.21.15.50:0/3084953368 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16842 (secure 0 0 0) 0x7f2e940af200 con 0x7f2e9413f890 2023-12-19T16:11:24.653 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:24.653 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":186,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:23.199397+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":186,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.10","mappings":[{"from":4,"to":0}]},{"pgid":"22.23","mappings":[{"from":3,"to":2}]},{"pgid":"22.28","mappings":[{"from":3,"to":0}]},{"pgid":"22.33","mappings":[{"from":3,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3295762643":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1719917206":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4089126800":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/2491435626":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4259555793":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/1927379651":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/2545221666":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1055917704":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3978276206":"2023-12-20T15:55:33.938573+0000","172.21.15.125:6833/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3093853131":"2023-12-20T16:06:21.086343+0000","172.21.15.125:6832/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3363816454":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/932161281":"2023-12-20T15:55:33.938573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 msgr2=0x7f2e80062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 0x7f2e80062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 msgr2=0x7f2e941695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009290 tx=0x7f2e8c019000 comp rx=0 tx=0).stop 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 shutdown_connections 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2e80060670 0x7f2e80062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e9413f890 0x7f2e941695b0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2e940c2980 0x7f2e94169070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 --2- 172.21.15.50:0/3084953368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940680f0 0x7f2e9415d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 >> 172.21.15.50:0/3084953368 conn(0x7f2e940c8850 msgr2=0x7f2e9415a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 shutdown_connections 2023-12-19T16:11:24.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.650+0000 7f2e99e6e700 1 -- 172.21.15.50:0/3084953368 wait complete. 2023-12-19T16:11:24.677 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.670+0000 7fb2dfbce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:11:24.679 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T16:11:24.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:24.754 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.748+0000 7f089b76b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:24.833 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.828+0000 7f089b76b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:11:24.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.834+0000 7f794d84c700 1 Processor -- start 2023-12-19T16:11:24.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.834+0000 7f794d84c700 1 -- start start 2023-12-19T16:11:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d84c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f794806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d84c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794806b2c0 con 0x7f794806df20 2023-12-19T16:11:24.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d84c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794806a6b0 con 0x7f79480cad10 2023-12-19T16:11:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d84c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79480cb550 con 0x7f79480cb0e0 2023-12-19T16:11:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d04b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f794806df20 0x7f794815bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55510/0 (socket says 172.21.15.50:55510) 2023-12-19T16:11:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f794d04b700 1 -- 172.21.15.50:0/2046354205 learned_addr learned my addr 172.21.15.50:0/2046354205 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:24.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f7947fff700 1 --2- 172.21.15.50:0/2046354205 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f794806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f7947fff700 1 -- 172.21.15.50:0/2046354205 >> v1:172.21.15.125:6789/0 conn(0x7f79480cad10 legacy=0x7f794806b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f7947fff700 1 -- 172.21.15.50:0/2046354205 >> v1:172.21.15.50:6789/0 conn(0x7f794806df20 legacy=0x7f794815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f7947fff700 1 -- 172.21.15.50:0/2046354205 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794806a990 con 0x7f79480cb0e0 2023-12-19T16:11:24.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.838+0000 7f79477fe700 1 -- 172.21.15.50:0/2046354205 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300947963 0 0) 0x7f794806a6b0 con 0x7f79480cad10 2023-12-19T16:11:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f7947fff700 1 --2- 172.21.15.50:0/2046354205 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f794806d9e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7934009940 tx=0x7f79340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d26540baa2f3c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f79477fe700 1 -- 172.21.15.50:0/2046354205 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934009ee0 con 0x7f79480cb0e0 2023-12-19T16:11:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f79477fe700 1 -- 172.21.15.50:0/2046354205 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793400b760 con 0x7f79480cb0e0 2023-12-19T16:11:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f79477fe700 1 -- 172.21.15.50:0/2046354205 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793400bb00 con 0x7f79480cb0e0 2023-12-19T16:11:24.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 -- 172.21.15.50:0/2046354205 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 msgr2=0x7f794806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 --2- 172.21.15.50:0/2046354205 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f794806d9e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7934009940 tx=0x7f79340091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 -- 172.21.15.50:0/2046354205 shutdown_connections 2023-12-19T16:11:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 --2- 172.21.15.50:0/2046354205 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f794806d9e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 -- 172.21.15.50:0/2046354205 >> 172.21.15.50:0/2046354205 conn(0x7f7948067bb0 msgr2=0x7f79480cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:24.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 -- 172.21.15.50:0/2046354205 shutdown_connections 2023-12-19T16:11:24.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.842+0000 7f794d84c700 1 -- 172.21.15.50:0/2046354205 wait complete. 2023-12-19T16:11:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 Processor -- start 2023-12-19T16:11:24.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 -- start start 2023-12-19T16:11:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f794806df20 0x7f794814a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f79480b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b3be0 con 0x7f79480cad10 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480b3600 con 0x7f794806df20 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d84c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794814af30 con 0x7f79480cb0e0 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36680/0 (socket says 172.21.15.50:36680) 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 -- 172.21.15.50:0/3470676029 learned_addr learned my addr 172.21.15.50:0/3470676029 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794d04b700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f79480b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f794c84a700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f794806df20 0x7f794814a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:24.854 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 msgr2=0x7f79480b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f79480b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f794806df20 msgr2=0x7f794814a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f794806df20 0x7f794814a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 -- 172.21.15.50:0/3470676029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934009420 con 0x7f79480cad10 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7947fff700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f793400bef0 tx=0x7f7934016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.846+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793400b8d0 con 0x7f79480cad10 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7934020070 con 0x7f79480cad10 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794814fd70 con 0x7f79480cad10 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793401b450 con 0x7f79480cad10 2023-12-19T16:11:24.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7948150050 con 0x7f79480cad10 2023-12-19T16:11:24.856 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79480b3d90 con 0x7f79480cad10 2023-12-19T16:11:24.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.850+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f7934022030 con 0x7f79480cad10 2023-12-19T16:11:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.854+0000 7f7944ff9700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 0x7f792c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.854+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f793407ec00 con 0x7f79480cad10 2023-12-19T16:11:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.854+0000 7f794c84a700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 msgr2=0x7f792c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.854+0000 7f794c84a700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 0x7f792c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:24.858+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f79480b3d90 con 0x7f79480cad10 2023-12-19T16:11:24.927 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:24.920+0000 7f089b76b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:11:24.933 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:24.926+0000 7fb2dfbce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:25.022 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:25.018+0000 7fb2dfbce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.058+0000 7f794c84a700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 msgr2=0x7f792c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:25.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.058+0000 7f794c84a700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 0x7f792c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:25.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.066+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79480b3d90 con 0x7f79480cad10 2023-12-19T16:11:25.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.066+0000 7f7944ff9700 1 -- 172.21.15.50:0/3470676029 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f79480b3d90 con 0x7f79480cad10 2023-12-19T16:11:25.072 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:25.072 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T16:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 msgr2=0x7f792c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 0x7f792c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 msgr2=0x7f79480b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f793400bef0 tx=0x7f7934016000 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 shutdown_connections 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f792c060670 0x7f792c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79480cb0e0 0x7f79480b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79480cad10 0x7f79480b05e0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 --2- 172.21.15.50:0/3470676029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f794806df20 0x7f794814a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 >> 172.21.15.50:0/3470676029 conn(0x7f7948067bb0 msgr2=0x7f794815d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 shutdown_connections 2023-12-19T16:11:25.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.070+0000 7f794d84c700 1 -- 172.21.15.50:0/3470676029 wait complete. 2023-12-19T16:11:25.078 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T16:11:25.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:25.119 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:25.114+0000 7fb2dfbce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:11:25.134 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:25.128+0000 7f089b76b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:11:25.226 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:25.220+0000 7f089b76b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:11:25.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.250+0000 7fed44185700 1 Processor -- start 2023-12-19T16:11:25.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.250+0000 7fed44185700 1 -- start start 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.250+0000 7fed44185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fed3c1502b0 0x7fed3c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c0c6630 con 0x7fed3c067c90 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c0c6910 con 0x7fed3c1502b0 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0c6bf0 con 0x7fed3c0c5900 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fed3c1502b0 0x7fed3c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fed3c1502b0 0x7fed3c068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36692/0 (socket says 172.21.15.50:36692) 2023-12-19T16:11:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 -- 172.21.15.50:0/639014576 learned_addr learned my addr 172.21.15.50:0/639014576 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed42722700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 -- 172.21.15.50:0/639014576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 msgr2=0x7fed3c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 -- 172.21.15.50:0/639014576 >> v1:172.21.15.125:6789/0 conn(0x7fed3c067c90 legacy=0x7fed3c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 -- 172.21.15.50:0/639014576 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c0c6ed0 con 0x7fed3c1502b0 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed41720700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c1502b0 0x7fed3c068060 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0099b0 tx=0x7fed2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0d93cf1a905545 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed40f1f700 1 -- 172.21.15.50:0/639014576 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00b860 con 0x7fed3c1502b0 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed40f1f700 1 -- 172.21.15.50:0/639014576 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c00b9c0 con 0x7fed3c1502b0 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed40f1f700 1 -- 172.21.15.50:0/639014576 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00bd60 con 0x7fed3c1502b0 2023-12-19T16:11:25.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- 172.21.15.50:0/639014576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c1502b0 msgr2=0x7fed3c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c1502b0 0x7fed3c068060 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0099b0 tx=0x7fed2c009250 comp rx=0 tx=0).stop 2023-12-19T16:11:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- 172.21.15.50:0/639014576 shutdown_connections 2023-12-19T16:11:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 --2- 172.21.15.50:0/639014576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c1502b0 0x7fed3c068060 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- 172.21.15.50:0/639014576 >> 172.21.15.50:0/639014576 conn(0x7fed3c0cd6d0 msgr2=0x7fed3c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- 172.21.15.50:0/639014576 shutdown_connections 2023-12-19T16:11:25.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.254+0000 7fed44185700 1 -- 172.21.15.50:0/639014576 wait complete. 2023-12-19T16:11:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 Processor -- start 2023-12-19T16:11:25.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 -- start start 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 0x7fed3c16c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0b5920 con 0x7fed3c067c90 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0b5490 con 0x7fed3c1502b0 2023-12-19T16:11:25.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed44185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c0b4880 con 0x7fed3c0c5900 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed41f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed41f21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36698/0 (socket says 172.21.15.50:36698) 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed41f21700 1 -- 172.21.15.50:0/160184904 learned_addr learned my addr 172.21.15.50:0/160184904 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed42722700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 0x7fed3c16c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.258+0000 7fed41720700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 msgr2=0x7fed3c0c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 msgr2=0x7fed3c16c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 0x7fed3c16c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 -- 172.21.15.50:0/160184904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c0094f0 con 0x7fed3c067c90 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41720700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed42722700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 0x7fed3c16c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed41f21700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fed3800c590 tx=0x7fed3800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38011590 con 0x7fed3c067c90 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed380116f0 con 0x7fed3c067c90 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c0b4b60 con 0x7fed3c067c90 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38011ab0 con 0x7fed3c067c90 2023-12-19T16:11:25.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c0b5120 con 0x7fed3c067c90 2023-12-19T16:11:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.262+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c071120 con 0x7fed3c067c90 2023-12-19T16:11:25.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.266+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fed3801b3b0 con 0x7fed3c067c90 2023-12-19T16:11:25.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.266+0000 7fed31ffb700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 0x7fed20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.266+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fed3807c350 con 0x7fed3c067c90 2023-12-19T16:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.266+0000 7fed41720700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 msgr2=0x7fed20062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:25.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.266+0000 7fed41720700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 0x7fed20062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:25.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.274+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fed3804c310 con 0x7fed3c067c90 2023-12-19T16:11:25.344 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:25.338+0000 7fb2dfbce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:11:25.440 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:25.434+0000 7fb2dfbce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:11:25.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.470+0000 7fed41720700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 msgr2=0x7fed20062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:25.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.470+0000 7fed41720700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 0x7fed20062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.482+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed3c0642a0 con 0x7fed3c067c90 2023-12-19T16:11:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.482+0000 7fed31ffb700 1 -- 172.21.15.50:0/160184904 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fed3804f930 con 0x7fed3c067c90 2023-12-19T16:11:25.489 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:25.489 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T15:55:18.383203+0000","modified":"2023-12-19T15:55:21.981660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9333":{"gid":9333,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/1475106130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":1475106130},{"type":"v1","addr":"172.21.15.50:6835","nonce":1475106130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T16:11:25.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.486+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 msgr2=0x7fed20062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:25.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.486+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 0x7fed20062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.486+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 msgr2=0x7fed3c0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.486+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fed3800c590 tx=0x7fed3800c950 comp rx=0 tx=0).stop 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.486+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 shutdown_connections 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fed20060670 0x7fed20062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed3c1502b0 0x7fed3c16c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fed3c0c5900 0x7fed3c0c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 --2- 172.21.15.50:0/160184904 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed3c067c90 0x7fed3c0b3f80 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 >> 172.21.15.50:0/160184904 conn(0x7fed3c0cd6d0 msgr2=0x7fed3c0cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 shutdown_connections 2023-12-19T16:11:25.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.490+0000 7fed44185700 1 -- 172.21.15.50:0/160184904 wait complete. 2023-12-19T16:11:25.496 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 82 2023-12-19T16:11:25.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:25.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.722+0000 7fad432cd700 1 Processor -- start 2023-12-19T16:11:25.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.722+0000 7fad432cd700 1 -- start start 2023-12-19T16:11:25.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.722+0000 7fad432cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad3c158f70 0x7fad3c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c159340 0x7fad3c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c068c70 con 0x7fad3c069100 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3c068060 con 0x7fad3c158f70 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c068340 con 0x7fad3c159340 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad40868700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fad3c069100 0x7fad3c0cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55514/0 (socket says 172.21.15.50:55514) 2023-12-19T16:11:25.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad40868700 1 -- 172.21.15.50:0/3191085861 learned_addr learned my addr 172.21.15.50:0/3191085861 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 --2- 172.21.15.50:0/3191085861 >> v2:172.21.15.125:3300/0 conn(0x7fad3c158f70 0x7fad3c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad4186a700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c159340 0x7fad3c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad33fff700 1 -- 172.21.15.50:0/3191085861 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493572956 0 0) 0x7fad3c068c70 con 0x7fad3c069100 2023-12-19T16:11:25.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad33fff700 1 -- 172.21.15.50:0/3191085861 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad28003570 con 0x7fad3c069100 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 -- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c159340 msgr2=0x7fad3c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c159340 0x7fad3c0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 -- 172.21.15.50:0/3191085861 >> v1:172.21.15.50:6789/0 conn(0x7fad3c069100 legacy=0x7fad3c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 -- 172.21.15.50:0/3191085861 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c068620 con 0x7fad3c158f70 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad41069700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c158f70 0x7fad3c067980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fad380098e0 tx=0x7fad380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e980189c0cf054 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:25.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad33fff700 1 -- 172.21.15.50:0/3191085861 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38011630 con 0x7fad3c158f70 2023-12-19T16:11:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad33fff700 1 -- 172.21.15.50:0/3191085861 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38011790 con 0x7fad3c158f70 2023-12-19T16:11:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad33fff700 1 -- 172.21.15.50:0/3191085861 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38011b30 con 0x7fad3c158f70 2023-12-19T16:11:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 -- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c158f70 msgr2=0x7fad3c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.726+0000 7fad432cd700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c158f70 0x7fad3c067980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fad380098e0 tx=0x7fad380091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- 172.21.15.50:0/3191085861 shutdown_connections 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c159340 0x7fad3c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 --2- 172.21.15.50:0/3191085861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c158f70 0x7fad3c067980 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- 172.21.15.50:0/3191085861 >> 172.21.15.50:0/3191085861 conn(0x7fad3c0ccc40 msgr2=0x7fad3c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- 172.21.15.50:0/3191085861 shutdown_connections 2023-12-19T16:11:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- 172.21.15.50:0/3191085861 wait complete. 2023-12-19T16:11:25.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 Processor -- start 2023-12-19T16:11:25.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- start start 2023-12-19T16:11:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c158f70 0x7fad3c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c159340 0x7fad3c0c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c15e440 con 0x7fad3c069100 2023-12-19T16:11:25.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c16a740 con 0x7fad3c159340 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad432cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c15d830 con 0x7fad3c158f70 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad41069700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad41069700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36706/0 (socket says 172.21.15.50:36706) 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.730+0000 7fad41069700 1 -- 172.21.15.50:0/1902893961 learned_addr learned my addr 172.21.15.50:0/1902893961 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad40868700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c158f70 0x7fad3c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad4186a700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c159340 0x7fad3c0c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 -- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c158f70 msgr2=0x7fad3c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c158f70 0x7fad3c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 -- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c159340 msgr2=0x7fad3c0c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c159340 0x7fad3c0c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 -- 172.21.15.50:0/1902893961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38009420 con 0x7fad3c069100 2023-12-19T16:11:25.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad41069700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fad3c06b980 tx=0x7fad38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38011b10 con 0x7fad3c069100 2023-12-19T16:11:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad380253f0 con 0x7fad3c069100 2023-12-19T16:11:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c15db10 con 0x7fad3c069100 2023-12-19T16:11:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38025790 con 0x7fad3c069100 2023-12-19T16:11:25.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c15e0d0 con 0x7fad3c069100 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c071120 con 0x7fad3c069100 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.734+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 88) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fad38025b80 con 0x7fad3c069100 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.738+0000 7fad317fa700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 0x7fad20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.738+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fad3807e680 con 0x7fad3c069100 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.738+0000 7fad40868700 1 -- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 msgr2=0x7fad20062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:25.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.738+0000 7fad40868700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 0x7fad20062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:25.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.742+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad3804e6e0 con 0x7fad3c069100 2023-12-19T16:11:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.914+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad3c068900 con 0x7fad3c069100 2023-12-19T16:11:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.918+0000 7fad317fa700 1 -- 172.21.15.50:0/1902893961 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16842 (secure 0 0 0) 0x7fad38051d00 con 0x7fad3c069100 2023-12-19T16:11:25.924 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:25.924 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":186,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:23.199397+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T15:55:16.440898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T15:55:17.404856+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":186,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.10","mappings":[{"from":4,"to":0}]},{"pgid":"22.23","mappings":[{"from":3,"to":2}]},{"pgid":"22.28","mappings":[{"from":3,"to":0}]},{"pgid":"22.33","mappings":[{"from":3,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3295762643":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1719917206":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4089126800":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/2491435626":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4259555793":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/1927379651":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/2545221666":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1055917704":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3978276206":"2023-12-20T15:55:33.938573+0000","172.21.15.125:6833/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3093853131":"2023-12-20T16:06:21.086343+0000","172.21.15.125:6832/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3363816454":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/932161281":"2023-12-20T15:55:33.938573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:25.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 msgr2=0x7fad20062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:25.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 0x7fad20062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 msgr2=0x7fad3c14dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:25.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fad3c06b980 tx=0x7fad38017000 comp rx=0 tx=0).stop 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 shutdown_connections 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fad20060670 0x7fad20062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad3c159340 0x7fad3c0c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fad3c158f70 0x7fad3c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 --2- 172.21.15.50:0/1902893961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad3c069100 0x7fad3c14dcc0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 >> 172.21.15.50:0/1902893961 conn(0x7fad3c0ccc40 msgr2=0x7fad3c15ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 shutdown_connections 2023-12-19T16:11:25.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:25.922+0000 7fad432cd700 1 -- 172.21.15.50:0/1902893961 wait complete. 2023-12-19T16:11:25.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T16:11:25.987 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:25.980+0000 7f089b76b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:11:26.098 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08623ab700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.098 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08623ab700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08623ab700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08623ab700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08623ab700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08663b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08663b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08663b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08663b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.099 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:26.092+0000 7f08663b3700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:26.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.118+0000 7fa47759e700 1 Processor -- start 2023-12-19T16:11:26.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.118+0000 7fa47759e700 1 -- start start 2023-12-19T16:11:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa478158e20 0x7fa47806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4781591f0 0x7fa4780ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47806b730 con 0x7fa47815ca80 2023-12-19T16:11:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47806ab20 con 0x7fa4781591f0 2023-12-19T16:11:26.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47806ae00 con 0x7fa478158e20 2023-12-19T16:11:26.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4781591f0 0x7fa4780ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4781591f0 0x7fa4780ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36708/0 (socket says 172.21.15.50:36708) 2023-12-19T16:11:26.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 -- 172.21.15.50:0/1016781172 learned_addr learned my addr 172.21.15.50:0/1016781172 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:26.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47659c700 1 --2- 172.21.15.50:0/1016781172 >> v2:172.21.15.125:3300/0 conn(0x7fa478158e20 0x7fa47806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 -- 172.21.15.50:0/1016781172 >> v2:172.21.15.125:3300/0 conn(0x7fa478158e20 msgr2=0x7fa47806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 --2- 172.21.15.50:0/1016781172 >> v2:172.21.15.125:3300/0 conn(0x7fa478158e20 0x7fa47806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 -- 172.21.15.50:0/1016781172 >> v1:172.21.15.125:6789/0 conn(0x7fa47815ca80 legacy=0x7fa47806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47559a700 1 -- 172.21.15.50:0/1016781172 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040527489 0 0) 0x7fa47806b730 con 0x7fa47815ca80 2023-12-19T16:11:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 -- 172.21.15.50:0/1016781172 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47806b0e0 con 0x7fa4781591f0 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa475d9b700 1 --2- 172.21.15.50:0/1016781172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4781591f0 0x7fa4780ce100 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fa47806e510 tx=0x7fa46c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3441f830dd3d987 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47559a700 1 -- 172.21.15.50:0/1016781172 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c009e70 con 0x7fa4781591f0 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47559a700 1 -- 172.21.15.50:0/1016781172 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c00b860 con 0x7fa4781591f0 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47559a700 1 -- 172.21.15.50:0/1016781172 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00bc00 con 0x7fa4781591f0 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- 172.21.15.50:0/1016781172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4781591f0 msgr2=0x7fa4780ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 --2- 172.21.15.50:0/1016781172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4781591f0 0x7fa4780ce100 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fa47806e510 tx=0x7fa46c009250 comp rx=0 tx=0).stop 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- 172.21.15.50:0/1016781172 shutdown_connections 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 --2- 172.21.15.50:0/1016781172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa4781591f0 0x7fa4780ce100 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 --2- 172.21.15.50:0/1016781172 >> v2:172.21.15.125:3300/0 conn(0x7fa478158e20 0x7fa47806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.122+0000 7fa47759e700 1 -- 172.21.15.50:0/1016781172 >> 172.21.15.50:0/1016781172 conn(0x7fa478067bb0 msgr2=0x7fa4780cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- 172.21.15.50:0/1016781172 shutdown_connections 2023-12-19T16:11:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- 172.21.15.50:0/1016781172 wait complete. 2023-12-19T16:11:26.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 Processor -- start 2023-12-19T16:11:26.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- start start 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 0x7fa4780cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa47815ca80 0x7fa4780ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781404a0 con 0x7fa478158e20 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4780ca070 con 0x7fa4781591f0 2023-12-19T16:11:26.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47813fb70 con 0x7fa47815ca80 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36718/0 (socket says 172.21.15.50:36718) 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.126+0000 7fa47659c700 1 -- 172.21.15.50:0/1570831232 learned_addr learned my addr 172.21.15.50:0/1570831232 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa476d9d700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa47815ca80 0x7fa4780ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa475d9b700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 0x7fa4780cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 -- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa47815ca80 msgr2=0x7fa4780ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa47815ca80 0x7fa4780ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 -- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 msgr2=0x7fa4780cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 0x7fa4780cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 -- 172.21.15.50:0/1570831232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0094f0 con 0x7fa478158e20 2023-12-19T16:11:26.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa475d9b700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 0x7fa4780cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47659c700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ec80 tx=0x7fa4680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46800c850 con 0x7fa478158e20 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47813fe50 con 0x7fa478158e20 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46800c9b0 con 0x7fa478158e20 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46800cd50 con 0x7fa478158e20 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47813f670 con 0x7fa478158e20 2023-12-19T16:11:26.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.130+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47804be40 con 0x7fa478158e20 2023-12-19T16:11:26.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.134+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 89) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fa468013680 con 0x7fa478158e20 2023-12-19T16:11:26.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.134+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 7171+0+0 (secure 0 0 0) 0x7fa46807ccd0 con 0x7fa478158e20 2023-12-19T16:11:26.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.138+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa46804cfd0 con 0x7fa478158e20 2023-12-19T16:11:26.256 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:26.250+0000 7fb2dfbce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:11:26.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.346+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa47806b3c0 con 0x7fa478158e20 2023-12-19T16:11:26.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T16:11:26.346+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T16:11:26.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.358+0000 7fa4667fc700 1 -- 172.21.15.50:0/1570831232 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa4680505f0 con 0x7fa478158e20 2023-12-19T16:11:26.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 msgr2=0x7fa47813f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ec80 tx=0x7fa4680091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 shutdown_connections 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4781591f0 0x7fa4780cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa47815ca80 0x7fa4780ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 --2- 172.21.15.50:0/1570831232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa478158e20 0x7fa47813f130 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 >> 172.21.15.50:0/1570831232 conn(0x7fa478067bb0 msgr2=0x7fa4780cc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 shutdown_connections 2023-12-19T16:11:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.362+0000 7fa47759e700 1 -- 172.21.15.50:0/1570831232 wait complete. 2023-12-19T16:11:26.368 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T16:11:26.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T16:11:26.427 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T16:11:26.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 Processor -- start 2023-12-19T16:11:26.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 -- start start 2023-12-19T16:11:26.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 0x7f05e0159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e006c850 con 0x7f05e0158c90 2023-12-19T16:11:26.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e00cb650 con 0x7f05e006aeb0 2023-12-19T16:11:26.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05e480c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e006b470 con 0x7f05e006dcf0 2023-12-19T16:11:26.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05df7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f05e0158c90 0x7f05e00caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36362/0 (socket says 172.21.15.50:36362) 2023-12-19T16:11:26.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05df7fe700 1 -- 172.21.15.50:0/3574614136 learned_addr learned my addr 172.21.15.50:0/3574614136 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:26.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05dffff700 1 --2- 172.21.15.50:0/3574614136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 0x7f05e0159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532381673 0 0) 0x7f05e006c850 con 0x7f05e0158c90 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.538+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05c0003570 con 0x7f05e0158c90 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522199245 0 0) 0x7f05e00cb650 con 0x7f05e006aeb0 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05dffff700 1 -- 172.21.15.50:0/3574614136 >> v1:172.21.15.125:6789/0 conn(0x7f05e006aeb0 legacy=0x7f05e0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05dffff700 1 -- 172.21.15.50:0/3574614136 >> v1:172.21.15.50:6789/0 conn(0x7f05e0158c90 legacy=0x7f05e00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05dffff700 1 -- 172.21.15.50:0/3574614136 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e006b750 con 0x7f05e006dcf0 2023-12-19T16:11:26.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05dffff700 1 --2- 172.21.15.50:0/3574614136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 0x7f05e0159770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009e90 tx=0x7f05d8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03ced715f211d66 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:26.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d800e070 con 0x7f05e006dcf0 2023-12-19T16:11:26.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d800bc30 con 0x7f05e006dcf0 2023-12-19T16:11:26.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05deffd700 1 -- 172.21.15.50:0/3574614136 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8012430 con 0x7f05e006dcf0 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 -- 172.21.15.50:0/3574614136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 msgr2=0x7f05e0159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 --2- 172.21.15.50:0/3574614136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 0x7f05e0159770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009e90 tx=0x7f05d8009780 comp rx=0 tx=0).stop 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 -- 172.21.15.50:0/3574614136 shutdown_connections 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 --2- 172.21.15.50:0/3574614136 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e006dcf0 0x7f05e0159770 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 -- 172.21.15.50:0/3574614136 >> 172.21.15.50:0/3574614136 conn(0x7f05e0067bb0 msgr2=0x7f05e00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 -- 172.21.15.50:0/3574614136 shutdown_connections 2023-12-19T16:11:26.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 -- 172.21.15.50:0/3574614136 wait complete. 2023-12-19T16:11:26.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.542+0000 7f05e480c700 1 Processor -- start 2023-12-19T16:11:26.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- start start 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05e006aeb0 0x7f05e00b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e0158c90 0x7f05e014da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e00b5bf0 con 0x7f05e006dcf0 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e014bdd0 con 0x7f05e006aeb0 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e014b1c0 con 0x7f05e0158c90 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51346/0 (socket says 172.21.15.50:51346) 2023-12-19T16:11:26.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 -- 172.21.15.50:0/3419796481 learned_addr learned my addr 172.21.15.50:0/3419796481 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 -- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e0158c90 msgr2=0x7f05e014da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e0158c90 0x7f05e014da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 -- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05e006aeb0 msgr2=0x7f05e00b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05e006aeb0 0x7f05e00b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 -- 172.21.15.50:0/3419796481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8009a20 con 0x7f05e006dcf0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d6ffd700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c7e0 tx=0x7f05c8007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8011590 con 0x7f05e006dcf0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c80116f0 con 0x7f05e006dcf0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8011ab0 con 0x7f05e006dcf0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e014b4a0 con 0x7f05e006dcf0 2023-12-19T16:11:26.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.546+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e014ba60 con 0x7f05e006dcf0 2023-12-19T16:11:26.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.550+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e004be40 con 0x7f05e006dcf0 2023-12-19T16:11:26.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.550+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 89) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f05c801b3b0 con 0x7f05e006dcf0 2023-12-19T16:11:26.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.550+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7257+0+0 (secure 0 0 0) 0x7f05c807bff0 con 0x7f05e006dcf0 2023-12-19T16:11:26.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.558+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f05c80abbe0 con 0x7f05e006dcf0 2023-12-19T16:11:26.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:26.766+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f05e006ba30 con 0x7f05e006dcf0 2023-12-19T16:11:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.006+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 90) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f05c804bd70 con 0x7f05e006dcf0 2023-12-19T16:11:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.006+0000 7f05d7fff700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 0x7f05c4068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.010+0000 7f05df7fe700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 0x7f05c4068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.014+0000 7f05df7fe700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 0x7f05c4068cb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05d00058d0 tx=0x7f05d00079d0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:27.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.358+0000 7f05d7fff700 1 -- 172.21.15.50:0/3419796481 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7f05c8075260 con 0x7f05e006dcf0 2023-12-19T16:11:27.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 msgr2=0x7f05c4068cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 0x7f05c4068cb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05d00058d0 tx=0x7f05d00079d0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 msgr2=0x7f05e014a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c7e0 tx=0x7f05c8007da0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 shutdown_connections 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f05c40668d0 0x7f05c4068cb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f05e0158c90 0x7f05e014da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05e006dcf0 0x7f05e014a790 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 --2- 172.21.15.50:0/3419796481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05e006aeb0 0x7f05e00b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 >> 172.21.15.50:0/3419796481 conn(0x7f05e0067bb0 msgr2=0x7f05e014c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:27.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 shutdown_connections 2023-12-19T16:11:27.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.362+0000 7f05e480c700 1 -- 172.21.15.50:0/3419796481 wait complete. 2023-12-19T16:11:27.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T16:11:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 Processor -- start 2023-12-19T16:11:27.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 -- start start 2023-12-19T16:11:27.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb500 0x7fbdbc15c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbdbc15c760 0x7fbdbc0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc0b0ac0 con 0x7fbdbc0cb130 2023-12-19T16:11:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdbc15cbe0 con 0x7fbdbc15c760 2023-12-19T16:11:27.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc406e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc06b210 con 0x7fbdbc0cb500 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc1e0a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbdbc0cb130 0x7fbdbc06a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36378/0 (socket says 172.21.15.50:36378) 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc1e0a700 1 -- 172.21.15.50:0/2423076046 learned_addr learned my addr 172.21.15.50:0/2423076046 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc260b700 1 --2- 172.21.15.50:0/2423076046 >> v2:172.21.15.125:3300/0 conn(0x7fbdbc15c760 0x7fbdbc0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc1609700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb500 0x7fbdbc15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc0e08700 1 -- 172.21.15.50:0/2423076046 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487641831 0 0) 0x7fbdbc0b0ac0 con 0x7fbdbc0cb130 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc0e08700 1 -- 172.21.15.50:0/2423076046 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda8003570 con 0x7fbdbc0cb130 2023-12-19T16:11:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc260b700 1 -- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb500 msgr2=0x7fbdbc15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc260b700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb500 0x7fbdbc15c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc260b700 1 -- 172.21.15.50:0/2423076046 >> v1:172.21.15.50:6789/0 conn(0x7fbdbc0cb130 legacy=0x7fbdbc06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.618+0000 7fbdc260b700 1 -- 172.21.15.50:0/2423076046 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0ca810 con 0x7fbdbc15c760 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc260b700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0b0f50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4009e60 tx=0x7fbdb4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec796bc47d427900 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc0e08700 1 -- 172.21.15.50:0/2423076046 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb400e070 con 0x7fbdbc15c760 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc0e08700 1 -- 172.21.15.50:0/2423076046 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb400bb60 con 0x7fbdbc15c760 2023-12-19T16:11:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc0e08700 1 -- 172.21.15.50:0/2423076046 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4012430 con 0x7fbdbc15c760 2023-12-19T16:11:27.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 -- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 msgr2=0x7fbdbc0b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0b0f50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4009e60 tx=0x7fbdb4009780 comp rx=0 tx=0).stop 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 -- 172.21.15.50:0/2423076046 shutdown_connections 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0b0f50 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 --2- 172.21.15.50:0/2423076046 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb500 0x7fbdbc15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 -- 172.21.15.50:0/2423076046 >> 172.21.15.50:0/2423076046 conn(0x7fbdbc067bb0 msgr2=0x7fbdbc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 -- 172.21.15.50:0/2423076046 shutdown_connections 2023-12-19T16:11:27.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.622+0000 7fbdc406e700 1 -- 172.21.15.50:0/2423076046 wait complete. 2023-12-19T16:11:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 Processor -- start 2023-12-19T16:11:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 -- start start 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb130 0x7fbdbc0891d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0874b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc15b3b0 con 0x7fbdbc0cb500 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc08a630 con 0x7fbdbc15c760 2023-12-19T16:11:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc406e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc08aac0 con 0x7fbdbc0cb130 2023-12-19T16:11:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51354/0 (socket says 172.21.15.50:51354) 2023-12-19T16:11:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 -- 172.21.15.50:0/3999522180 learned_addr learned my addr 172.21.15.50:0/3999522180 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1e0a700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb130 0x7fbdbc0891d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc260b700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0874b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 -- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb130 msgr2=0x7fbdbc0891d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb130 0x7fbdbc0891d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 -- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 msgr2=0x7fbdbc0874b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0874b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.626+0000 7fbdc1609700 1 -- 172.21.15.50:0/3999522180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb4009a20 con 0x7fbdbc0cb500 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdc1609700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d890 tx=0x7fbdac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0125f0 con 0x7fbdbc0cb500 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac012750 con 0x7fbdbc0cb500 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac012af0 con 0x7fbdbc0cb500 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc089a20 con 0x7fbdbc0cb500 2023-12-19T16:11:27.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc089fe0 con 0x7fbdbc0cb500 2023-12-19T16:11:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc0ce080 con 0x7fbdbc0cb500 2023-12-19T16:11:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.630+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 90) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fbdac01e460 con 0x7fbdbc0cb500 2023-12-19T16:11:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.634+0000 7fbdb1ffb700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 0x7fbda0062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.634+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 7257+0+0 (secure 0 0 0) 0x7fbdac07d0f0 con 0x7fbdbc0cb500 2023-12-19T16:11:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.634+0000 7fbdc1e0a700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 0x7fbda0062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:27.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.634+0000 7fbdc1e0a700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 0x7fbda0062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:27.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.638+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbdbc0ce080 con 0x7fbdbc0cb500 2023-12-19T16:11:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:27.814+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbdbc0ce080 con 0x7fbdbc0cb500 2023-12-19T16:11:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.366+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbdac04d680 con 0x7fbdbc0cb500 2023-12-19T16:11:28.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.378+0000 7fbdb1ffb700 1 -- 172.21.15.50:0/3999522180 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v189) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbdac050d40 con 0x7fbdbc0cb500 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 msgr2=0x7fbda0062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 0x7fbda0062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).stop 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 msgr2=0x7fbdbc086f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d890 tx=0x7fbdac008040 comp rx=0 tx=0).stop 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 shutdown_connections 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda00603a0 0x7fbda0062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdbc15c760 0x7fbdbc0874b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdbc0cb500 0x7fbdbc086f70 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 --2- 172.21.15.50:0/3999522180 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdbc0cb130 0x7fbdbc0891d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 >> 172.21.15.50:0/3999522180 conn(0x7fbdbc067bb0 msgr2=0x7fbdbc15cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 shutdown_connections 2023-12-19T16:11:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.382+0000 7fbdc406e700 1 -- 172.21.15.50:0/3999522180 wait complete. 2023-12-19T16:11:28.391 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T16:11:28.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T16:11:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 Processor -- start 2023-12-19T16:11:28.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 -- start start 2023-12-19T16:11:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8f80af190 0x7fe8f806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f80cb3f0 con 0x7fe8f80aedc0 2023-12-19T16:11:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f806d220 con 0x7fe8f80cac50 2023-12-19T16:11:28.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.582+0000 7fe8fd48e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f80cc140 con 0x7fe8f80af190 2023-12-19T16:11:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f7fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe8f80aedc0 0x7fe8f806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36394/0 (socket says 172.21.15.50:36394) 2023-12-19T16:11:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f7fff700 1 -- 172.21.15.50:0/1412981589 learned_addr learned my addr 172.21.15.50:0/1412981589 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:28.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f77fe700 1 --2- 172.21.15.50:0/1412981589 >> v2:172.21.15.125:3300/0 conn(0x7fe8f80af190 0x7fe8f806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644659039 0 0) 0x7fe8f80cb3f0 con 0x7fe8f80aedc0 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e0003570 con 0x7fe8f80aedc0 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238705620 0 0) 0x7fe8f806d220 con 0x7fe8f80cac50 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f80cb3f0 con 0x7fe8f80cac50 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391322869 0 0) 0x7fe8e0003570 con 0x7fe8f80aedc0 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f77fe700 1 -- 172.21.15.50:0/1412981589 >> v1:172.21.15.125:6789/0 conn(0x7fe8f80cac50 legacy=0x7fe8f80af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f77fe700 1 -- 172.21.15.50:0/1412981589 >> v1:172.21.15.50:6789/0 conn(0x7fe8f80aedc0 legacy=0x7fe8f806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f77fe700 1 -- 172.21.15.50:0/1412981589 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f806c8f0 con 0x7fe8f80af190 2023-12-19T16:11:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f77fe700 1 --2- 172.21.15.50:0/1412981589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f806af50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0098e0 tx=0x7fe8ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d08412efde77e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8001470 con 0x7fe8f80af190 2023-12-19T16:11:28.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec011420 con 0x7fe8f80af190 2023-12-19T16:11:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8f6ffd700 1 -- 172.21.15.50:0/1412981589 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0117c0 con 0x7fe8f80af190 2023-12-19T16:11:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8fd48e700 1 -- 172.21.15.50:0/1412981589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 msgr2=0x7fe8f806af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/1412981589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f806af50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0098e0 tx=0x7fe8ec0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8fd48e700 1 -- 172.21.15.50:0/1412981589 shutdown_connections 2023-12-19T16:11:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.586+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/1412981589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f806af50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.590+0000 7fe8fd48e700 1 -- 172.21.15.50:0/1412981589 >> 172.21.15.50:0/1412981589 conn(0x7fe8f80cc760 msgr2=0x7fe8f80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.590+0000 7fe8fd48e700 1 -- 172.21.15.50:0/1412981589 shutdown_connections 2023-12-19T16:11:28.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.590+0000 7fe8fd48e700 1 -- 172.21.15.50:0/1412981589 wait complete. 2023-12-19T16:11:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.590+0000 7fe8fd48e700 1 Processor -- start 2023-12-19T16:11:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.590+0000 7fe8fd48e700 1 -- start start 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f80c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 0x7fe8f80c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80af610 con 0x7fe8f80aedc0 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80c45a0 con 0x7fe8f80af190 2023-12-19T16:11:28.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80c3a00 con 0x7fe8f80cac50 2023-12-19T16:11:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f77fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f80c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:28.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51364/0 (socket says 172.21.15.50:51364) 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 -- 172.21.15.50:0/2124131549 learned_addr learned my addr 172.21.15.50:0/2124131549 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fcc8d700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 0x7fe8f80c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 -- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 msgr2=0x7fe8f80c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 0x7fe8f80c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 -- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 msgr2=0x7fe8f80c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f80c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 -- 172.21.15.50:0/2124131549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009420 con 0x7fe8f80aedc0 2023-12-19T16:11:28.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fcc8d700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 0x7fe8f80c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8f7fff700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cba0 tx=0x7fe8e8001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8001d20 con 0x7fe8f80aedc0 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e8007360 con 0x7fe8f80aedc0 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8004830 con 0x7fe8f80aedc0 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f80c3cd0 con 0x7fe8f80aedc0 2023-12-19T16:11:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.594+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f80c4f70 con 0x7fe8f80aedc0 2023-12-19T16:11:28.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.598+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8070400 con 0x7fe8f80aedc0 2023-12-19T16:11:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.598+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe8e8004ba0 con 0x7fe8f80aedc0 2023-12-19T16:11:28.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.598+0000 7fe8dffff700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 0x7fe8e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.602+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fe8e807c110 con 0x7fe8f80aedc0 2023-12-19T16:11:28.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.602+0000 7fe8f77fe700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 0x7fe8e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:28.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.602+0000 7fe8f77fe700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 0x7fe8e4062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0093f0 tx=0x7fe8ec016000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:28.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.606+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe8e804c170 con 0x7fe8f80aedc0 2023-12-19T16:11:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:28.786+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe8f804be40 con 0x7fe8f80aedc0 2023-12-19T16:11:28.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:11:28.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:11:29.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.386+0000 7fe8dffff700 1 -- 172.21.15.50:0/2124131549 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v190) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe8e804f790 con 0x7fe8f80aedc0 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.390+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 msgr2=0x7fe8e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.390+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 0x7fe8e4062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0093f0 tx=0x7fe8ec016000 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.390+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 msgr2=0x7fe8f80c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.390+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cba0 tx=0x7fe8e8001ab0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 shutdown_connections 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe8e4060670 0x7fe8e4062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8f80af190 0x7fe8f80c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe8f80cac50 0x7fe8f80c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 --2- 172.21.15.50:0/2124131549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe8f80aedc0 0x7fe8f80c4a30 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 >> 172.21.15.50:0/2124131549 conn(0x7fe8f80cc760 msgr2=0x7fe8f815bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 shutdown_connections 2023-12-19T16:11:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.394+0000 7fe8fd48e700 1 -- 172.21.15.50:0/2124131549 wait complete. 2023-12-19T16:11:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T16:11:29.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T16:11:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.586+0000 7f75b9323700 1 Processor -- start 2023-12-19T16:11:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.586+0000 7f75b9323700 1 -- start start 2023-12-19T16:11:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b9323700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f75b4158c90 0x7f75b40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b9323700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75b4159770 0x7f75b406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b9323700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b406c850 con 0x7f75b406aeb0 2023-12-19T16:11:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b9323700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b406b470 con 0x7f75b4158c90 2023-12-19T16:11:29.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b9323700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b406b750 con 0x7f75b4159770 2023-12-19T16:11:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75ab7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75b406aeb0 0x7f75b4159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36406/0 (socket says 172.21.15.50:36406) 2023-12-19T16:11:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75ab7fe700 1 -- 172.21.15.50:0/1373365327 learned_addr learned my addr 172.21.15.50:0/1373365327 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 --2- 172.21.15.50:0/1373365327 >> v2:172.21.15.50:3300/0 conn(0x7f75b4158c90 0x7f75b40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b8b22700 1 --2- 172.21.15.50:0/1373365327 >> v2:172.21.15.125:3300/0 conn(0x7f75b4159770 0x7f75b406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b37fe700 1 -- 172.21.15.50:0/1373365327 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495587199 0 0) 0x7f75b406c850 con 0x7f75b406aeb0 2023-12-19T16:11:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b37fe700 1 -- 172.21.15.50:0/1373365327 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0003570 con 0x7f75b406aeb0 2023-12-19T16:11:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 -- 172.21.15.50:0/1373365327 >> v2:172.21.15.125:3300/0 conn(0x7f75b4159770 msgr2=0x7f75b406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 --2- 172.21.15.50:0/1373365327 >> v2:172.21.15.125:3300/0 conn(0x7f75b4159770 0x7f75b406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 -- 172.21.15.50:0/1373365327 >> v1:172.21.15.50:6789/0 conn(0x7f75b406aeb0 legacy=0x7f75b4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 -- 172.21.15.50:0/1373365327 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b406ba30 con 0x7f75b4158c90 2023-12-19T16:11:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b3fff700 1 --2- 172.21.15.50:0/1373365327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4158c90 0x7f75b40caf40 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f75b40cc7f0 tx=0x7f75a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95de50829578727e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b37fe700 1 -- 172.21.15.50:0/1373365327 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400e070 con 0x7f75b4158c90 2023-12-19T16:11:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b37fe700 1 -- 172.21.15.50:0/1373365327 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400bab0 con 0x7f75b4158c90 2023-12-19T16:11:29.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.590+0000 7f75b37fe700 1 -- 172.21.15.50:0/1373365327 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400be70 con 0x7f75b4158c90 2023-12-19T16:11:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- 172.21.15.50:0/1373365327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4158c90 msgr2=0x7f75b40caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- 172.21.15.50:0/1373365327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4158c90 0x7f75b40caf40 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f75b40cc7f0 tx=0x7f75a4009780 comp rx=0 tx=0).stop 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- 172.21.15.50:0/1373365327 shutdown_connections 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- 172.21.15.50:0/1373365327 >> v2:172.21.15.125:3300/0 conn(0x7f75b4159770 0x7f75b406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- 172.21.15.50:0/1373365327 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4158c90 0x7f75b40caf40 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- 172.21.15.50:0/1373365327 >> 172.21.15.50:0/1373365327 conn(0x7f75b4067bb0 msgr2=0x7f75b40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- 172.21.15.50:0/1373365327 shutdown_connections 2023-12-19T16:11:29.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- 172.21.15.50:0/1373365327 wait complete. 2023-12-19T16:11:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 Processor -- start 2023-12-19T16:11:29.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- start start 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75b406aeb0 0x7f75b4152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75b4158c90 0x7f75b40b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4152b90 con 0x7f75b4159770 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4155980 con 0x7f75b4158c90 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.594+0000 7f75b9323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b408ac70 con 0x7f75b406aeb0 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51382/0 (socket says 172.21.15.50:51382) 2023-12-19T16:11:29.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 -- 172.21.15.50:0/3515233187 learned_addr learned my addr 172.21.15.50:0/3515233187 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75ab7fe700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75b4158c90 0x7f75b40b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75b406aeb0 0x7f75b4152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 -- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75b406aeb0 msgr2=0x7f75b4152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75b406aeb0 0x7f75b4152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 -- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75b4158c90 msgr2=0x7f75b40b1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75b4158c90 0x7f75b40b1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 -- 172.21.15.50:0/3515233187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a4009a20 con 0x7f75b4159770 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b8b22700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00e900 tx=0x7f75ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00eea0 con 0x7f75b4159770 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00c7f0 con 0x7f75b4159770 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b40892b0 con 0x7f75b4159770 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00cbb0 con 0x7f75b4159770 2023-12-19T16:11:29.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b4154f20 con 0x7f75b4159770 2023-12-19T16:11:29.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.598+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b404be40 con 0x7f75b4159770 2023-12-19T16:11:29.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.602+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f75ac013470 con 0x7f75b4159770 2023-12-19T16:11:29.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.602+0000 7f75b0ff9700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 0x7f7594062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.602+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f75ac07cd00 con 0x7f75b4159770 2023-12-19T16:11:29.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.602+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 0x7f7594062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:29.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.602+0000 7f75b3fff700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 0x7f7594062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75a40097c0 tx=0x7f75a4019000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:29.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.610+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f75ac04cef0 con 0x7f75b4159770 2023-12-19T16:11:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.782+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f75b406bd10 con 0x7f75b4159770 2023-12-19T16:11:29.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.786+0000 7f75b0ff9700 1 -- 172.21.15.50:0/3515233187 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f75ac050510 con 0x7f75b4159770 2023-12-19T16:11:29.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 msgr2=0x7f7594062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 0x7f7594062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75a40097c0 tx=0x7f75a4019000 comp rx=0 tx=0).stop 2023-12-19T16:11:29.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 msgr2=0x7f75b40b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:29.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00e900 tx=0x7f75ac0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 shutdown_connections 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7594060670 0x7f7594062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75b4158c90 0x7f75b40b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75b4159770 0x7f75b40b1ec0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 --2- 172.21.15.50:0/3515233187 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f75b406aeb0 0x7f75b4152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.790+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 >> 172.21.15.50:0/3515233187 conn(0x7f75b4067bb0 msgr2=0x7f75b40afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.794+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 shutdown_connections 2023-12-19T16:11:29.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.794+0000 7f75b9323700 1 -- 172.21.15.50:0/3515233187 wait complete. 2023-12-19T16:11:29.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T16:11:29.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.994+0000 7f2047fff700 1 Processor -- start 2023-12-19T16:11:29.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.994+0000 7f2047fff700 1 -- start start 2023-12-19T16:11:30.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.994+0000 7f2047fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048156d00 0x7f204806b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.994+0000 7f2047fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f20480abfe0 0x7f204806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2047fff700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204806c7c0 con 0x7f2048156930 2023-12-19T16:11:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2047fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204806d060 con 0x7f20480abfe0 2023-12-19T16:11:30.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2047fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204806bcd0 con 0x7f2048156d00 2023-12-19T16:11:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f20467fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048156d00 0x7f204806b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f20467fc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048156d00 0x7f204806b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:57640/0 (socket says 172.21.15.50:57640) 2023-12-19T16:11:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f20467fc700 1 -- 172.21.15.50:0/126316952 learned_addr learned my addr 172.21.15.50:0/126316952 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:30.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f20477fe700 1 --2- 172.21.15.50:0/126316952 >> v2:172.21.15.50:3300/0 conn(0x7f20480abfe0 0x7f204806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322746871 0 0) 0x7f204806c7c0 con 0x7f2048156930 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030003570 con 0x7f2048156930 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669926069 0 0) 0x7f2030003570 con 0x7f2048156930 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048156d00 msgr2=0x7f204806b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048156d00 0x7f204806b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 >> v2:172.21.15.50:3300/0 conn(0x7f20480abfe0 msgr2=0x7f204806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 --2- 172.21.15.50:0/126316952 >> v2:172.21.15.50:3300/0 conn(0x7f20480abfe0 0x7f204806a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204806ca10 con 0x7f2048156930 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f2038001470 con 0x7f2048156930 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 >> v1:172.21.15.50:6789/0 conn(0x7f2048156930 legacy=0x7f20480ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2030003ae0 0x7f2030003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2030004810 0x7f2030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20300051c0 0x7f2030014270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204806d060 con 0x7f20300051c0 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204806bcd0 con 0x7f2030004810 2023-12-19T16:11:30.006 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:29.998+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2030014990 con 0x7f2030003ae0 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f2046ffd700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2030003ae0 0x7f2030003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20300051c0 0x7f2030014270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20467fc700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2030004810 0x7f2030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 -- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2030003ae0 msgr2=0x7f2030003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2030003ae0 0x7f2030003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 -- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2030004810 msgr2=0x7f2030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2030004810 0x7f2030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 -- 172.21.15.50:0/126316952 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204806ca10 con 0x7f20300051c0 2023-12-19T16:11:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f20477fe700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20300051c0 0x7f2030014270 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f204000d290 tx=0x7f204000f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe67afa018dbbcfd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204000bbc0 con 0x7f20300051c0 2023-12-19T16:11:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204000bd20 con 0x7f20300051c0 2023-12-19T16:11:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.002+0000 7f2045ffb700 1 -- 172.21.15.50:0/126316952 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040015630 con 0x7f20300051c0 2023-12-19T16:11:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 -- 172.21.15.50:0/126316952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20300051c0 msgr2=0x7f2030014270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 --2- 172.21.15.50:0/126316952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20300051c0 0x7f2030014270 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f204000d290 tx=0x7f204000f040 comp rx=0 tx=0).stop 2023-12-19T16:11:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f20477fe700 1 -- 172.21.15.50:0/126316952 reap_dead start 2023-12-19T16:11:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 -- 172.21.15.50:0/126316952 shutdown_connections 2023-12-19T16:11:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 -- 172.21.15.50:0/126316952 >> 172.21.15.50:0/126316952 conn(0x7f2048067bb0 msgr2=0x7f20480cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 -- 172.21.15.50:0/126316952 shutdown_connections 2023-12-19T16:11:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.006+0000 7f2047fff700 1 -- 172.21.15.50:0/126316952 wait complete. 2023-12-19T16:11:30.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 Processor -- start 2023-12-19T16:11:30.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- start start 2023-12-19T16:11:30.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20300051c0 0x7f2048167e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048159550 0x7f20481599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204814e890 con 0x7f2048164f00 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204814ae30 con 0x7f20300051c0 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2048159f00 con 0x7f2048159550 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51394/0 (socket says 172.21.15.50:51394) 2023-12-19T16:11:30.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 -- 172.21.15.50:0/3036985659 learned_addr learned my addr 172.21.15.50:0/3036985659 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20467fc700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048159550 0x7f20481599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 -- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048159550 msgr2=0x7f20481599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048159550 0x7f20481599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 -- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20300051c0 msgr2=0x7f2048167e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20300051c0 0x7f2048167e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 -- 172.21.15.50:0/3036985659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2048157610 con 0x7f2048164f00 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f20477fe700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2040014000 tx=0x7f2040018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204001f620 con 0x7f2048164f00 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204000be90 con 0x7f2048164f00 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2048166100 con 0x7f2048164f00 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204001f9f0 con 0x7f2048164f00 2023-12-19T16:11:30.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.010+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20481663d0 con 0x7f2048164f00 2023-12-19T16:11:30.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204804be40 con 0x7f2048164f00 2023-12-19T16:11:30.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f204001c070 con 0x7f2048164f00 2023-12-19T16:11:30.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f202effd700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 0x7f2034062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f204007dd50 con 0x7f2048164f00 2023-12-19T16:11:30.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f2046ffd700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 0x7f2034062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.014+0000 7f2046ffd700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 0x7f2034062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2038007940 tx=0x7f2038007830 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.018+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2040081030 con 0x7f2048164f00 2023-12-19T16:11:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.194+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f20481577a0 con 0x7f2048164f00 2023-12-19T16:11:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.202+0000 7f202effd700 1 -- 172.21.15.50:0/3036985659 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v62) v1 ==== 115+0+0 (secure 0 0 0) 0x7f204004ded0 con 0x7f2048164f00 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 msgr2=0x7f2034062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 0x7f2034062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2038007940 tx=0x7f2038007830 comp rx=0 tx=0).stop 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 msgr2=0x7f2048165350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2040014000 tx=0x7f2040018000 comp rx=0 tx=0).stop 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 shutdown_connections 2023-12-19T16:11:30.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f20340606a0 0x7f2034062b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2048164f00 0x7f2048165350 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2048159550 0x7f20481599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 --2- 172.21.15.50:0/3036985659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20300051c0 0x7f2048167e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 >> 172.21.15.50:0/3036985659 conn(0x7f2048067bb0 msgr2=0x7f204806d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 shutdown_connections 2023-12-19T16:11:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.206+0000 7f2047fff700 1 -- 172.21.15.50:0/3036985659 wait complete. 2023-12-19T16:11:30.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T16:11:30.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.394+0000 7f49e4985700 1 Processor -- start 2023-12-19T16:11:30.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.394+0000 7f49e4985700 1 -- start start 2023-12-19T16:11:30.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49e4985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f49e0158e20 0x7f49e00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49e4985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 0x7f49e00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49e4985700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e00cbdf0 con 0x7f49e00cb970 2023-12-19T16:11:30.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49e4985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e00cc0d0 con 0x7f49e0158e20 2023-12-19T16:11:30.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49e4985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00cc3b0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ddd9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f49e00cb970 0x7f49e006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36422/0 (socket says 172.21.15.50:36422) 2023-12-19T16:11:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ddd9b700 1 -- 172.21.15.50:0/2864752310 learned_addr learned my addr 172.21.15.50:0/2864752310 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49de59c700 1 --2- 172.21.15.50:0/2864752310 >> v2:172.21.15.125:3300/0 conn(0x7f49e0158e20 0x7f49e00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 --2- 172.21.15.50:0/2864752310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 0x7f49e00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49dd59a700 1 -- 172.21.15.50:0/2864752310 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634514142 0 0) 0x7f49e00cbdf0 con 0x7f49e00cb970 2023-12-19T16:11:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 -- 172.21.15.50:0/2864752310 >> v2:172.21.15.125:3300/0 conn(0x7f49e0158e20 msgr2=0x7f49e00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 --2- 172.21.15.50:0/2864752310 >> v2:172.21.15.125:3300/0 conn(0x7f49e0158e20 0x7f49e00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 -- 172.21.15.50:0/2864752310 >> v1:172.21.15.50:6789/0 conn(0x7f49e00cb970 legacy=0x7f49e006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 -- 172.21.15.50:0/2864752310 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e00cc690 con 0x7f49e00ca2c0 2023-12-19T16:11:30.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49ded9d700 1 --2- 172.21.15.50:0/2864752310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 0x7f49e00ceb00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e90 tx=0x7f49d8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=332ba1ab9e5835dc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49dd59a700 1 -- 172.21.15.50:0/2864752310 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d800e070 con 0x7f49e00ca2c0 2023-12-19T16:11:30.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49dd59a700 1 -- 172.21.15.50:0/2864752310 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d800bbf0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.398+0000 7f49dd59a700 1 -- 172.21.15.50:0/2864752310 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8012430 con 0x7f49e00ca2c0 2023-12-19T16:11:30.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- 172.21.15.50:0/2864752310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 msgr2=0x7f49e00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 --2- 172.21.15.50:0/2864752310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 0x7f49e00ceb00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e90 tx=0x7f49d8009780 comp rx=0 tx=0).stop 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- 172.21.15.50:0/2864752310 shutdown_connections 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 --2- 172.21.15.50:0/2864752310 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e00ca2c0 0x7f49e00ceb00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 --2- 172.21.15.50:0/2864752310 >> v2:172.21.15.125:3300/0 conn(0x7f49e0158e20 0x7f49e00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- 172.21.15.50:0/2864752310 >> 172.21.15.50:0/2864752310 conn(0x7f49e0067bb0 msgr2=0x7f49e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- 172.21.15.50:0/2864752310 shutdown_connections 2023-12-19T16:11:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- 172.21.15.50:0/2864752310 wait complete. 2023-12-19T16:11:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 Processor -- start 2023-12-19T16:11:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.402+0000 7f49e4985700 1 -- start start 2023-12-19T16:11:30.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 0x7f49e00b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e01569b0 0x7f49e00b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0154c50 con 0x7f49e00ca2c0 2023-12-19T16:11:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0154f30 con 0x7f49e0158e20 2023-12-19T16:11:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0155210 con 0x7f49e01569b0 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51398/0 (socket says 172.21.15.50:51398) 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 -- 172.21.15.50:0/1445006913 learned_addr learned my addr 172.21.15.50:0/1445006913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ddd9b700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 0x7f49e00b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ded9d700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e01569b0 0x7f49e00b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 -- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e01569b0 msgr2=0x7f49e00b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e01569b0 0x7f49e00b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 -- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 msgr2=0x7f49e00b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 0x7f49e00b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 -- 172.21.15.50:0/1445006913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8009a20 con 0x7f49e00ca2c0 2023-12-19T16:11:30.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ddd9b700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 0x7f49e00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:30.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49de59c700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f49d000e930 tx=0x7f49d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0013680 con 0x7f49e00ca2c0 2023-12-19T16:11:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d00137e0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0013ba0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e01554f0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.406+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e01548d0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.410+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e0071120 con 0x7f49e00ca2c0 2023-12-19T16:11:30.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.410+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f49d000c3e0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.414+0000 7f49ce7fc700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 0x7f49c0062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.414+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f49d007cb40 con 0x7f49e00ca2c0 2023-12-19T16:11:30.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.414+0000 7f49ddd9b700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 0x7f49c0062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.414+0000 7f49ddd9b700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 0x7f49c0062bf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49e006b470 tx=0x7f49d4007d30 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.427 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.422+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f49d004ce50 con 0x7f49e00ca2c0 2023-12-19T16:11:30.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.598+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f49e006a7a0 con 0x7f49e00ca2c0 2023-12-19T16:11:30.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.602+0000 7f49ce7fc700 1 -- 172.21.15.50:0/1445006913 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7f49d0050470 con 0x7f49e00ca2c0 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 msgr2=0x7f49c0062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 0x7f49c0062bf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49e006b470 tx=0x7f49d4007d30 comp rx=0 tx=0).stop 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 msgr2=0x7f49e0156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f49d000e930 tx=0x7f49d00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 shutdown_connections 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f49c0060740 0x7f49c0062bf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49e0158e20 0x7f49e00b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f49e01569b0 0x7f49e00b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 --2- 172.21.15.50:0/1445006913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f49e00ca2c0 0x7f49e0156470 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 >> 172.21.15.50:0/1445006913 conn(0x7f49e0067bb0 msgr2=0x7f49e00ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:30.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 shutdown_connections 2023-12-19T16:11:30.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.606+0000 7f49e4985700 1 -- 172.21.15.50:0/1445006913 wait complete. 2023-12-19T16:11:30.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-12-19T16:11:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.794+0000 7ff26308e700 1 Processor -- start 2023-12-19T16:11:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.794+0000 7ff26308e700 1 -- start start 2023-12-19T16:11:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26308e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 0x7ff25c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26308e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 0x7ff25c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26308e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c06b730 con 0x7ff25c158e20 2023-12-19T16:11:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26308e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c06ab20 con 0x7ff25c1591f0 2023-12-19T16:11:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26308e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c06ae00 con 0x7ff25c0ce810 2023-12-19T16:11:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 0x7ff25c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26188b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 0x7ff25c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26208c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff25c158e20 0x7ff25c0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:43198/0 (socket says 172.21.15.50:43198) 2023-12-19T16:11:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26208c700 1 -- 172.21.15.50:0/1552576161 learned_addr learned my addr 172.21.15.50:0/1552576161 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26108a700 1 -- 172.21.15.50:0/1552576161 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878584779 0 0) 0x7ff25c06b730 con 0x7ff25c158e20 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26108a700 1 -- 172.21.15.50:0/1552576161 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248003570 con 0x7ff25c158e20 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 -- 172.21.15.50:0/1552576161 >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 msgr2=0x7ff25c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 --2- 172.21.15.50:0/1552576161 >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 0x7ff25c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 -- 172.21.15.50:0/1552576161 >> v1:172.21.15.125:6789/0 conn(0x7ff25c158e20 legacy=0x7ff25c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 -- 172.21.15.50:0/1552576161 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c06b0e0 con 0x7ff25c0ce810 2023-12-19T16:11:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26188b700 1 --2- 172.21.15.50:0/1552576161 >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 0x7ff25c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26288d700 1 --2- 172.21.15.50:0/1552576161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 0x7ff25c0ca950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff254009e60 tx=0x7ff254009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c03ac397a319382 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26108a700 1 -- 172.21.15.50:0/1552576161 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400e070 con 0x7ff25c0ce810 2023-12-19T16:11:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26108a700 1 -- 172.21.15.50:0/1552576161 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400bab0 con 0x7ff25c0ce810 2023-12-19T16:11:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.798+0000 7ff26108a700 1 -- 172.21.15.50:0/1552576161 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400be70 con 0x7ff25c0ce810 2023-12-19T16:11:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- 172.21.15.50:0/1552576161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 msgr2=0x7ff25c0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 --2- 172.21.15.50:0/1552576161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 0x7ff25c0ca950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff254009e60 tx=0x7ff254009780 comp rx=0 tx=0).stop 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- 172.21.15.50:0/1552576161 shutdown_connections 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 --2- 172.21.15.50:0/1552576161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c0ce810 0x7ff25c0ca950 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 --2- 172.21.15.50:0/1552576161 >> v2:172.21.15.50:3300/0 conn(0x7ff25c1591f0 0x7ff25c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- 172.21.15.50:0/1552576161 >> 172.21.15.50:0/1552576161 conn(0x7ff25c067bb0 msgr2=0x7ff25c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- 172.21.15.50:0/1552576161 shutdown_connections 2023-12-19T16:11:30.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- 172.21.15.50:0/1552576161 wait complete. 2023-12-19T16:11:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 Processor -- start 2023-12-19T16:11:30.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.802+0000 7ff26308e700 1 -- start start 2023-12-19T16:11:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 0x7ff25c169da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff25c1591f0 0x7ff25c0c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0c9fa0 con 0x7ff25c0ce810 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c169910 con 0x7ff25c1591f0 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c151fc0 con 0x7ff25c158e20 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26208c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 0x7ff25c169da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26208c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51416/0 (socket says 172.21.15.50:51416) 2023-12-19T16:11:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26208c700 1 -- 172.21.15.50:0/4233458729 learned_addr learned my addr 172.21.15.50:0/4233458729 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26288d700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff25c1591f0 0x7ff25c0c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 -- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff25c1591f0 msgr2=0x7ff25c0c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff25c1591f0 0x7ff25c0c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 -- 172.21.15.50:0/4233458729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 msgr2=0x7ff25c151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 -- 172.21.15.50:0/4233458729 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c009420 con 0x7ff25c158e20 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26208c700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26188b700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 0x7ff25c169da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ea30 tx=0x7ff24c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00ee70 con 0x7ff25c158e20 2023-12-19T16:11:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c00c850 con 0x7ff25c158e20 2023-12-19T16:11:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00cbf0 con 0x7ff25c158e20 2023-12-19T16:11:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.806+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff254009a20 con 0x7ff25c158e20 2023-12-19T16:11:30.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.810+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0c9cc0 con 0x7ff25c158e20 2023-12-19T16:11:30.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.810+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c06a8e0 con 0x7ff25c158e20 2023-12-19T16:11:30.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.810+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff24c013470 con 0x7ff25c158e20 2023-12-19T16:11:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.814+0000 7ff2527fc700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 0x7ff240062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:30.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.814+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7ff24c07cd70 con 0x7ff25c158e20 2023-12-19T16:11:30.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.814+0000 7ff26208c700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 0x7ff240062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.814+0000 7ff26208c700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 0x7ff240062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff258001ed0 tx=0x7ff258009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:30.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:30.822+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff25c06a8e0 con 0x7ff25c158e20 2023-12-19T16:11:31.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.050+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7ff25c06a8e0 con 0x7ff25c158e20 2023-12-19T16:11:31.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.394+0000 7ff2527fc700 1 -- 172.21.15.50:0/4233458729 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2987) v1 ==== 165+0+0 (secure 0 0 0) 0x7ff25c06a8e0 con 0x7ff25c158e20 2023-12-19T16:11:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 msgr2=0x7ff240062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 0x7ff240062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff258001ed0 tx=0x7ff258009040 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 msgr2=0x7ff25c169da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 0x7ff25c169da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ea30 tx=0x7ff24c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 shutdown_connections 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff240060670 0x7ff240062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff25c1591f0 0x7ff25c0c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff25c158e20 0x7ff25c169da0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 --2- 172.21.15.50:0/4233458729 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff25c0ce810 0x7ff25c151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 >> 172.21.15.50:0/4233458729 conn(0x7ff25c067bb0 msgr2=0x7ff25c152400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 shutdown_connections 2023-12-19T16:11:31.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.398+0000 7ff26308e700 1 -- 172.21.15.50:0/4233458729 wait complete. 2023-12-19T16:11:31.427 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-12-19T16:11:31.427 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-12-19T16:11:31.428 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T16:11:31.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.582+0000 7f632e216700 1 Processor -- start 2023-12-19T16:11:31.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.582+0000 7f632e216700 1 -- start start 2023-12-19T16:11:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632e216700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63280c6900 0x7f63280c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632e216700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632806b900 0x7f632806da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632e216700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632806b470 con 0x7f632815cc20 2023-12-19T16:11:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632e216700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632815d7b0 con 0x7f63280c6900 2023-12-19T16:11:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632e216700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6328161d10 con 0x7f632806b900 2023-12-19T16:11:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632da15700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f632815cc20 0x7f632815d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36434/0 (socket says 172.21.15.50:36434) 2023-12-19T16:11:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632da15700 1 -- 172.21.15.50:0/1410838127 learned_addr learned my addr 172.21.15.50:0/1410838127 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 --2- 172.21.15.50:0/1410838127 >> v2:172.21.15.125:3300/0 conn(0x7f63280c6900 0x7f63280c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632ca13700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632806b900 0x7f632806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f631ffff700 1 -- 172.21.15.50:0/1410838127 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518667180 0 0) 0x7f632806b470 con 0x7f632815cc20 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f631ffff700 1 -- 172.21.15.50:0/1410838127 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314003570 con 0x7f632815cc20 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 -- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632806b900 msgr2=0x7f632806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632806b900 0x7f632806da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 -- 172.21.15.50:0/1410838127 >> v1:172.21.15.50:6789/0 conn(0x7f632815cc20 legacy=0x7f632815d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 -- 172.21.15.50:0/1410838127 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632806bd70 con 0x7f63280c6900 2023-12-19T16:11:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f632d214700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63280c6900 0x7f63280c6cd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19abd8e5dbaac078 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.586+0000 7f631ffff700 1 -- 172.21.15.50:0/1410838127 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324011630 con 0x7f63280c6900 2023-12-19T16:11:31.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f631ffff700 1 -- 172.21.15.50:0/1410838127 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324011790 con 0x7f63280c6900 2023-12-19T16:11:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f631ffff700 1 -- 172.21.15.50:0/1410838127 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324011b30 con 0x7f63280c6900 2023-12-19T16:11:31.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63280c6900 msgr2=0x7f63280c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63280c6900 0x7f63280c6cd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- 172.21.15.50:0/1410838127 shutdown_connections 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632806b900 0x7f632806da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 --2- 172.21.15.50:0/1410838127 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63280c6900 0x7f63280c6cd0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- 172.21.15.50:0/1410838127 >> 172.21.15.50:0/1410838127 conn(0x7f6328067bb0 msgr2=0x7f63280cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- 172.21.15.50:0/1410838127 shutdown_connections 2023-12-19T16:11:31.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- 172.21.15.50:0/1410838127 wait complete. 2023-12-19T16:11:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 Processor -- start 2023-12-19T16:11:31.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.590+0000 7f632e216700 1 -- start start 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f632806b900 0x7f63281702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632815cc20 0x7f632815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328147940 con 0x7f63280c6900 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632815e1b0 con 0x7f632806b900 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632815ded0 con 0x7f632815cc20 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51432/0 (socket says 172.21.15.50:51432) 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 -- 172.21.15.50:0/1630471463 learned_addr learned my addr 172.21.15.50:0/1630471463 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:31.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632d214700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f632806b900 0x7f63281702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632da15700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632815cc20 0x7f632815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 -- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632815cc20 msgr2=0x7f632815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632815cc20 0x7f632815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 -- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f632806b900 msgr2=0x7f63281702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f632806b900 0x7f63281702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 -- 172.21.15.50:0/1630471463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009420 con 0x7f63280c6900 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632ca13700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f631800ecb0 tx=0x7f63180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631800c920 con 0x7f63280c6900 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632815eec0 con 0x7f63280c6900 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631800ca80 con 0x7f63280c6900 2023-12-19T16:11:31.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631800ce20 con 0x7f63280c6900 2023-12-19T16:11:31.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.594+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632815f480 con 0x7f63280c6900 2023-12-19T16:11:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.598+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632804be40 con 0x7f63280c6900 2023-12-19T16:11:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.598+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6318013680 con 0x7f63280c6900 2023-12-19T16:11:31.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.598+0000 7f631d7fa700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 0x7f630c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.598+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f631807cf50 con 0x7f63280c6900 2023-12-19T16:11:31.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.602+0000 7f632d214700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 0x7f630c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.602+0000 7f632d214700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 0x7f630c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f63240091e0 tx=0x7f6324017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:31.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.606+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f631804d140 con 0x7f63280c6900 2023-12-19T16:11:31.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.778+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f632806a580 con 0x7f63280c6900 2023-12-19T16:11:31.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.778+0000 7f631d7fa700 1 -- 172.21.15.50:0/1630471463 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+13987 (secure 0 0 0) 0x7f6318050760 con 0x7f63280c6900 2023-12-19T16:11:31.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:31.783 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":190,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:29.378252+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/1475106130":"2023-12-20T16:11:26.351040+0000","172.21.15.50:6834/1475106130":"2023-12-20T16:11:26.351040+0000","172.21.15.125:0/892642910":"2023-12-20T16:11:25.988095+0000","172.21.15.125:6832/44063":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/1052086557":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/3295762643":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/3119103899":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/1719917206":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4089126800":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/2491435626":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/4259555793":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/362161206":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/1927379651":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/1547310500":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/2545221666":"2023-12-20T16:06:21.086343+0000","172.21.15.125:0/1055917704":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3978276206":"2023-12-20T15:55:33.938573+0000","172.21.15.125:6833/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/3093853131":"2023-12-20T16:06:21.086343+0000","172.21.15.125:6832/33827":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/4123021093":"2023-12-20T16:11:25.988095+0000","172.21.15.125:6833/44063":"2023-12-20T16:11:25.988095+0000","172.21.15.125:0/3363816454":"2023-12-20T15:55:33.938573+0000","172.21.15.125:0/932161281":"2023-12-20T15:55:33.938573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.778+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 msgr2=0x7f630c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.778+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 0x7f630c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f63240091e0 tx=0x7f6324017000 comp rx=0 tx=0).stop 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 msgr2=0x7f6328147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f631800ecb0 tx=0x7f63180091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 shutdown_connections 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f630c060670 0x7f630c062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f632815cc20 0x7f632815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63280c6900 0x7f6328147bd0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 --2- 172.21.15.50:0/1630471463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f632806b900 0x7f63281702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 >> 172.21.15.50:0/1630471463 conn(0x7f6328067bb0 msgr2=0x7f6328159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 shutdown_connections 2023-12-19T16:11:31.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.782+0000 7f632e216700 1 -- 172.21.15.50:0/1630471463 wait complete. 2023-12-19T16:11:31.806 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T16:11:31.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:11:31.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.974+0000 7f8a7e745700 1 Processor -- start 2023-12-19T16:11:31.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.974+0000 7f8a7e745700 1 -- start start 2023-12-19T16:11:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a7e745700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a780c6d20 0x7f8a780c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a7e745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7806a7a0 con 0x7f8a7806ac30 2023-12-19T16:11:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a7e745700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a780c7630 con 0x7f8a7806c7c0 2023-12-19T16:11:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a7e745700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7806d4b0 con 0x7f8a780c6d20 2023-12-19T16:11:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a780c6d20 0x7f8a780c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a780c6d20 0x7f8a780c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51448/0 (socket says 172.21.15.50:51448) 2023-12-19T16:11:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 -- 172.21.15.50:0/889781525 learned_addr learned my addr 172.21.15.50:0/889781525 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:31.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a7cce2700 1 --1- 172.21.15.50:0/889781525 >> v1:172.21.15.50:6789/0 conn(0x7f8a7806ac30 0x7f8a7806b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36450/0 (socket says 172.21.15.50:36450) 2023-12-19T16:11:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 -- 172.21.15.50:0/889781525 >> v1:172.21.15.125:6789/0 conn(0x7f8a7806c7c0 legacy=0x7f8a7806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a777fe700 1 -- 172.21.15.50:0/889781525 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261090068 0 0) 0x7f8a780c7630 con 0x7f8a7806c7c0 2023-12-19T16:11:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 -- 172.21.15.50:0/889781525 >> v1:172.21.15.50:6789/0 conn(0x7f8a7806ac30 legacy=0x7f8a7806b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 -- 172.21.15.50:0/889781525 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7806d790 con 0x7f8a780c6d20 2023-12-19T16:11:31.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.978+0000 7f8a77fff700 1 --2- 172.21.15.50:0/889781525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a780c6d20 0x7f8a780c70f0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8a680099b0 tx=0x7f8a68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbd90c279e0b610 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a777fe700 1 -- 172.21.15.50:0/889781525 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800b760 con 0x7f8a780c6d20 2023-12-19T16:11:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a777fe700 1 -- 172.21.15.50:0/889781525 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800b8c0 con 0x7f8a780c6d20 2023-12-19T16:11:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a777fe700 1 -- 172.21.15.50:0/889781525 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800bc60 con 0x7f8a780c6d20 2023-12-19T16:11:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 -- 172.21.15.50:0/889781525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a780c6d20 msgr2=0x7f8a780c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 --2- 172.21.15.50:0/889781525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a780c6d20 0x7f8a780c70f0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8a680099b0 tx=0x7f8a68009250 comp rx=0 tx=0).stop 2023-12-19T16:11:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 -- 172.21.15.50:0/889781525 shutdown_connections 2023-12-19T16:11:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 --2- 172.21.15.50:0/889781525 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a780c6d20 0x7f8a780c70f0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 -- 172.21.15.50:0/889781525 >> 172.21.15.50:0/889781525 conn(0x7f8a78067bb0 msgr2=0x7f8a780cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 -- 172.21.15.50:0/889781525 shutdown_connections 2023-12-19T16:11:31.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.982+0000 7f8a7e745700 1 -- 172.21.15.50:0/889781525 wait complete. 2023-12-19T16:11:31.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 Processor -- start 2023-12-19T16:11:31.990 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 -- start start 2023-12-19T16:11:31.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7806ac30 0x7f8a78153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a780c6d20 0x7f8a780b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78153fc0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780afa40 con 0x7f8a780c6d20 2023-12-19T16:11:31.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7e745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7816a070 con 0x7f8a7806ac30 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a6f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a6f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51458/0 (socket says 172.21.15.50:51458) 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a6f7fe700 1 -- 172.21.15.50:0/1174100568 learned_addr learned my addr 172.21.15.50:0/1174100568 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a77fff700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7806ac30 0x7f8a78153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.986+0000 7f8a7cce2700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a780c6d20 0x7f8a780b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 -- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7806ac30 msgr2=0x7f8a78153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7806ac30 0x7f8a78153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 -- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a780c6d20 msgr2=0x7f8a780b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a780c6d20 0x7f8a780b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 -- 172.21.15.50:0/1174100568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680094f0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a6f7fe700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c7e0 tx=0x7f8a60007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60011590 con 0x7f8a7806c7c0 2023-12-19T16:11:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a600116f0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60011ab0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a780b18b0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a78169a20 con 0x7f8a7806c7c0 2023-12-19T16:11:31.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.990+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a78071120 con 0x7f8a7806c7c0 2023-12-19T16:11:31.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.994+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8a6001b3b0 con 0x7f8a7806c7c0 2023-12-19T16:11:31.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.994+0000 7f8a74ff9700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 0x7f8a58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:31.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.994+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f8a6007c210 con 0x7f8a7806c7c0 2023-12-19T16:11:32.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.994+0000 7f8a77fff700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 0x7f8a58062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.994+0000 7f8a77fff700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 0x7f8a58062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a680099b0 tx=0x7f8a68016000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:32.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:31.998+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a6004c400 con 0x7f8a7806c7c0 2023-12-19T16:11:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.170+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a7816a1b0 con 0x7f8a7806c7c0 2023-12-19T16:11:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.170+0000 7f8a74ff9700 1 -- 172.21.15.50:0/1174100568 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v190) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a600170a0 con 0x7f8a7806c7c0 2023-12-19T16:11:32.176 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:11:32.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 msgr2=0x7f8a58062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 0x7f8a58062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8a680099b0 tx=0x7f8a68016000 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 msgr2=0x7f8a780b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c7e0 tx=0x7f8a60007da0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 shutdown_connections 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8a58060670 0x7f8a58062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a780c6d20 0x7f8a780b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a7806c7c0 0x7f8a780b0e30 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 --2- 172.21.15.50:0/1174100568 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8a7806ac30 0x7f8a78153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 >> 172.21.15.50:0/1174100568 conn(0x7f8a78067bb0 msgr2=0x7f8a780c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 shutdown_connections 2023-12-19T16:11:32.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.174+0000 7f8a7e745700 1 -- 172.21.15.50:0/1174100568 wait complete. 2023-12-19T16:11:32.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-12-19T16:11:32.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.366+0000 7f7128ccd700 1 Processor -- start 2023-12-19T16:11:32.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.366+0000 7f7128ccd700 1 -- start start 2023-12-19T16:11:32.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f7124161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:32.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712406e150 con 0x7f7124140710 2023-12-19T16:11:32.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712406d540 con 0x7f7124140ae0 2023-12-19T16:11:32.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712406d820 con 0x7f7124068800 2023-12-19T16:11:32.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7124140710 0x7f71240c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36464/0 (socket says 172.21.15.50:36464) 2023-12-19T16:11:32.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712259c700 1 -- 172.21.15.50:0/405120593 learned_addr learned my addr 172.21.15.50:0/405120593 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:32.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7122d9d700 1 --2- 172.21.15.50:0/405120593 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f7124161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7122d9d700 1 -- 172.21.15.50:0/405120593 >> v1:172.21.15.125:6789/0 conn(0x7f7124140ae0 legacy=0x7f71240680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712159a700 1 -- 172.21.15.50:0/405120593 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729113422 0 0) 0x7f712406e150 con 0x7f7124140710 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7122d9d700 1 -- 172.21.15.50:0/405120593 >> v1:172.21.15.50:6789/0 conn(0x7f7124140710 legacy=0x7f71240c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7122d9d700 1 -- 172.21.15.50:0/405120593 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712406db00 con 0x7f7124068800 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7122d9d700 1 --2- 172.21.15.50:0/405120593 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f7124161cd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f711c009e90 tx=0x7f711c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db17457e3d4d434a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712159a700 1 -- 172.21.15.50:0/405120593 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c00e070 con 0x7f7124068800 2023-12-19T16:11:32.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712159a700 1 -- 172.21.15.50:0/405120593 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c00bc30 con 0x7f7124068800 2023-12-19T16:11:32.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f712159a700 1 -- 172.21.15.50:0/405120593 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c012430 con 0x7f7124068800 2023-12-19T16:11:32.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 -- 172.21.15.50:0/405120593 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 msgr2=0x7f7124161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.370+0000 7f7128ccd700 1 --2- 172.21.15.50:0/405120593 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f7124161cd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f711c009e90 tx=0x7f711c009780 comp rx=0 tx=0).stop 2023-12-19T16:11:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- 172.21.15.50:0/405120593 shutdown_connections 2023-12-19T16:11:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 --2- 172.21.15.50:0/405120593 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f7124161cd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- 172.21.15.50:0/405120593 >> 172.21.15.50:0/405120593 conn(0x7f71240c8850 msgr2=0x7f71240c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- 172.21.15.50:0/405120593 shutdown_connections 2023-12-19T16:11:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- 172.21.15.50:0/405120593 wait complete. 2023-12-19T16:11:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 Processor -- start 2023-12-19T16:11:32.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- start start 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f71241687c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7124140710 0x7f712416c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712413e320 con 0x7f7124140ae0 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71240c23e0 con 0x7f7124140710 2023-12-19T16:11:32.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.374+0000 7f7128ccd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71240a8f20 con 0x7f7124068800 2023-12-19T16:11:32.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51464/0 (socket says 172.21.15.50:51464) 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 -- 172.21.15.50:0/4106784750 learned_addr learned my addr 172.21.15.50:0/4106784750 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f712259c700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f71241687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7121d9b700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7124140710 0x7f712416c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 -- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 msgr2=0x7f71241687c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f71241687c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 -- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7124140710 msgr2=0x7f712416c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7124140710 0x7f712416c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:32.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 -- 172.21.15.50:0/4106784750 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c009a20 con 0x7f7124140ae0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7122d9d700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f71240ae960 tx=0x7f711c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c012e50 con 0x7f7124140ae0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c01b630 con 0x7f7124140ae0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71240a9200 con 0x7f7124140ae0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c01b9f0 con 0x7f7124140ae0 2023-12-19T16:11:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.378+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71240a97c0 con 0x7f7124140ae0 2023-12-19T16:11:32.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.382+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71240a99b0 con 0x7f7124140ae0 2023-12-19T16:11:32.387 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.382+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f711c01c400 con 0x7f7124140ae0 2023-12-19T16:11:32.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.382+0000 7f71127fc700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 0x7f7104062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:32.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.382+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f711c07eb30 con 0x7f7124140ae0 2023-12-19T16:11:32.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.382+0000 7f712259c700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 0x7f7104062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.386+0000 7f712259c700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 0x7f7104062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7114007fd0 tx=0x7f7114007d10 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:32.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.390+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f711c04ec10 con 0x7f7124140ae0 2023-12-19T16:11:32.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:32.622+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f7124071120 con 0x7f7124140ae0 2023-12-19T16:11:33.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.410+0000 7f71127fc700 1 -- 172.21.15.50:0/4106784750 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v2989) v1 ==== 166+0+0 (secure 0 0 0) 0x7f711c052230 con 0x7f7124140ae0 2023-12-19T16:11:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 msgr2=0x7f7104062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 0x7f7104062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7114007fd0 tx=0x7f7114007d10 comp rx=0 tx=0).stop 2023-12-19T16:11:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 msgr2=0x7f712416c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f71240ae960 tx=0x7f711c019000 comp rx=0 tx=0).stop 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 shutdown_connections 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7104060670 0x7f7104062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7124140710 0x7f712416c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7124140ae0 0x7f712416c9d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 --2- 172.21.15.50:0/4106784750 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7124068800 0x7f71241687c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 >> 172.21.15.50:0/4106784750 conn(0x7f71240c8850 msgr2=0x7f71240c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 shutdown_connections 2023-12-19T16:11:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.414+0000 7f7128ccd700 1 -- 172.21.15.50:0/4106784750 wait complete. 2023-12-19T16:11:33.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T16:11:33.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.610+0000 7f7335ef0700 1 Processor -- start 2023-12-19T16:11:33.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.610+0000 7f7335ef0700 1 -- start start 2023-12-19T16:11:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f7335ef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73300cecd0 0x7f733006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f7335ef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300cbdf0 con 0x7f73300cb970 2023-12-19T16:11:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f7335ef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300cc0d0 con 0x7f7330158e20 2023-12-19T16:11:33.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f7335ef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300cc3b0 con 0x7f73300cecd0 2023-12-19T16:11:33.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732effd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f73300cb970 0x7f733006acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36478/0 (socket says 172.21.15.50:36478) 2023-12-19T16:11:33.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732effd700 1 -- 172.21.15.50:0/1563142491 learned_addr learned my addr 172.21.15.50:0/1563142491 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:33.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732ffff700 1 --2- 172.21.15.50:0/1563142491 >> v2:172.21.15.125:3300/0 conn(0x7f73300cecd0 0x7f733006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132625225 0 0) 0x7f73300cbdf0 con 0x7f73300cb970 2023-12-19T16:11:33.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318003570 con 0x7f73300cb970 2023-12-19T16:11:33.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912456573 0 0) 0x7f7318003570 con 0x7f73300cb970 2023-12-19T16:11:33.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 >> v2:172.21.15.125:3300/0 conn(0x7f73300cecd0 msgr2=0x7f733006b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 --2- 172.21.15.50:0/1563142491 >> v2:172.21.15.125:3300/0 conn(0x7f73300cecd0 0x7f733006b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 >> v1:172.21.15.125:6789/0 conn(0x7f7330158e20 legacy=0x7f73300ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300cc690 con 0x7f73300cb970 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f7324001470 con 0x7f73300cb970 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 >> v1:172.21.15.50:6789/0 conn(0x7f73300cb970 legacy=0x7f733006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7318003a60 0x7f7318003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f7318004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73180051c0 0x7f731800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300cc3b0 con 0x7f7318004810 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731800c160 con 0x7f7318003a60 2023-12-19T16:11:33.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731800c320 con 0x7f73180051c0 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.614+0000 7f732effd700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f7318004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732ffff700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73180051c0 0x7f731800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732f7fe700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7318003a60 0x7f7318003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 -- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73180051c0 msgr2=0x7f731800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f73180051c0 0x7f731800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 -- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7318003a60 msgr2=0x7f7318003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7318003a60 0x7f7318003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 -- 172.21.15.50:0/1563142491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318003570 con 0x7f7318004810 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732f7fe700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7318003a60 0x7f7318003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:33.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732ffff700 1 -- 172.21.15.50:0/1563142491 reap_dead start 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732effd700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f7318004c80 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f732400c390 tx=0x7f7324001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a809d24b9d085b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732400cbe0 con 0x7f7318004810 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732400cd40 con 0x7f7318004810 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f732e7fc700 1 -- 172.21.15.50:0/1563142491 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324005b70 con 0x7f7318004810 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 -- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 msgr2=0x7f7318004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f7318004c80 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f732400c390 tx=0x7f7324001e50 comp rx=0 tx=0).stop 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 -- 172.21.15.50:0/1563142491 shutdown_connections 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 --2- 172.21.15.50:0/1563142491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f7318004c80 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 -- 172.21.15.50:0/1563142491 >> 172.21.15.50:0/1563142491 conn(0x7f7330067bb0 msgr2=0x7f73300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 -- 172.21.15.50:0/1563142491 shutdown_connections 2023-12-19T16:11:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.618+0000 7f7335ef0700 1 -- 172.21.15.50:0/1563142491 wait complete. 2023-12-19T16:11:33.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 Processor -- start 2023-12-19T16:11:33.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 -- start start 2023-12-19T16:11:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7330169830 0x7f733016c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f733016ce50 0x7f733016d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733016d7c0 con 0x7f7318004810 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733016a4e0 con 0x7f7330169830 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f7335ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733016ad80 con 0x7f733016ce50 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.622+0000 7f732f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51484/0 (socket says 172.21.15.50:51484) 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 -- 172.21.15.50:0/2083762057 learned_addr learned my addr 172.21.15.50:0/2083762057 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732effd700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7330169830 0x7f733016c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732ffff700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f733016ce50 0x7f733016d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 -- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f733016ce50 msgr2=0x7f733016d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f733016ce50 0x7f733016d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 -- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7330169830 msgr2=0x7f733016c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7330169830 0x7f733016c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 -- 172.21.15.50:0/2083762057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733015ada0 con 0x7f7318004810 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f732f7fe700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f732000d830 tx=0x7f7320009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320006820 con 0x7f7318004810 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7320006980 con 0x7f7318004810 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733016a730 con 0x7f7318004810 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320006d40 con 0x7f7318004810 2023-12-19T16:11:33.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7330169c80 con 0x7f7318004810 2023-12-19T16:11:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.626+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7330071120 con 0x7f7318004810 2023-12-19T16:11:33.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.630+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7320007240 con 0x7f7318004810 2023-12-19T16:11:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.630+0000 7f73177fe700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 0x7f731c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.630+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f7320001e40 con 0x7f7318004810 2023-12-19T16:11:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.634+0000 7f732effd700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 0x7f731c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:33.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.634+0000 7f732effd700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 0x7f731c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7324001e90 tx=0x7f7324015000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:33.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.638+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f732004c950 con 0x7f7318004810 2023-12-19T16:11:33.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.814+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f733006a7a0 con 0x7f7318004810 2023-12-19T16:11:33.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f73177fe700 1 -- 172.21.15.50:0/2083762057 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f732004ff70 con 0x7f7318004810 2023-12-19T16:11:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 msgr2=0x7f731c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 0x7f731c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7324001e90 tx=0x7f7324015000 comp rx=0 tx=0).stop 2023-12-19T16:11:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 msgr2=0x7f73301692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f732000d830 tx=0x7f7320009040 comp rx=0 tx=0).stop 2023-12-19T16:11:33.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 shutdown_connections 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f731c060670 0x7f731c062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7330169830 0x7f733016c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f733016ce50 0x7f733016d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 --2- 172.21.15.50:0/2083762057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7318004810 0x7f73301692f0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 >> 172.21.15.50:0/2083762057 conn(0x7f7330067bb0 msgr2=0x7f73300d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 shutdown_connections 2023-12-19T16:11:33.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:33.826+0000 7f7335ef0700 1 -- 172.21.15.50:0/2083762057 wait complete. 2023-12-19T16:11:33.859 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T16:11:33.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:11:33.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:11:33.947 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T16:11:33.947 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T16:11:33.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 Processor -- start 2023-12-19T16:11:34.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 -- start start 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7400140710 0x7f74000c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7400140ae0 0x7f74000680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74000c75e0 0x7f7400068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740006e150 con 0x7f7400140710 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740006d540 con 0x7f7400140ae0 2023-12-19T16:11:34.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.106+0000 7f7406345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740006d820 con 0x7f74000c75e0 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7400140710 0x7f74000c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7400140710 0x7f74000c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51494/0 (socket says 172.21.15.50:51494) 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 -- 172.21.15.50:0/212718393 learned_addr learned my addr 172.21.15.50:0/212718393 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73f77fe700 1 --2- 172.21.15.50:0/212718393 >> v2:172.21.15.125:3300/0 conn(0x7f7400140ae0 0x7f74000680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f74048e2700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74000c75e0 0x7f7400068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 -- 172.21.15.50:0/212718393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74000c75e0 msgr2=0x7f7400068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74000c75e0 0x7f7400068630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 -- 172.21.15.50:0/212718393 >> v2:172.21.15.125:3300/0 conn(0x7f7400140ae0 msgr2=0x7f74000680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 --2- 172.21.15.50:0/212718393 >> v2:172.21.15.125:3300/0 conn(0x7f7400140ae0 0x7f74000680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 -- 172.21.15.50:0/212718393 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740006db00 con 0x7f7400140710 2023-12-19T16:11:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73fffff700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140710 0x7f74000c70a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f74000c9960 tx=0x7f73f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da31e5fc1b2020cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73ff7fe700 1 -- 172.21.15.50:0/212718393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f000e070 con 0x7f7400140710 2023-12-19T16:11:34.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73ff7fe700 1 -- 172.21.15.50:0/212718393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f000ba50 con 0x7f7400140710 2023-12-19T16:11:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f73ff7fe700 1 -- 172.21.15.50:0/212718393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f00092e0 con 0x7f7400140710 2023-12-19T16:11:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f7406345700 1 -- 172.21.15.50:0/212718393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140710 msgr2=0x7f74000c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.110+0000 7f7406345700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140710 0x7f74000c70a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f74000c9960 tx=0x7f73f0009840 comp rx=0 tx=0).stop 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 -- 172.21.15.50:0/212718393 shutdown_connections 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74000c75e0 0x7f7400068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 --2- 172.21.15.50:0/212718393 >> v2:172.21.15.125:3300/0 conn(0x7f7400140ae0 0x7f74000680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 --2- 172.21.15.50:0/212718393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140710 0x7f74000c70a0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 -- 172.21.15.50:0/212718393 >> 172.21.15.50:0/212718393 conn(0x7f74000c8850 msgr2=0x7f74000c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:34.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 -- 172.21.15.50:0/212718393 shutdown_connections 2023-12-19T16:11:34.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 -- 172.21.15.50:0/212718393 wait complete. 2023-12-19T16:11:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 Processor -- start 2023-12-19T16:11:34.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 -- start start 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74000c75e0 0x7f7400159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.114+0000 7f7406345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7400140710 0x7f74000c5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740015a550 con 0x7f7400140ae0 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000c31e0 con 0x7f74000c75e0 2023-12-19T16:11:34.122 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7400159cb0 con 0x7f7400140710 2023-12-19T16:11:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51500/0 (socket says 172.21.15.50:51500) 2023-12-19T16:11:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 -- 172.21.15.50:0/2269257814 learned_addr learned my addr 172.21.15.50:0/2269257814 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f73fffff700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74000c75e0 0x7f7400159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f73f77fe700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7400140710 0x7f74000c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 -- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7400140710 msgr2=0x7f74000c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7400140710 0x7f74000c5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 -- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74000c75e0 msgr2=0x7f7400159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74000c75e0 0x7f7400159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 -- 172.21.15.50:0/2269257814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0009ae0 con 0x7f7400140ae0 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f74048e2700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f73f800d860 tx=0x7f73f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80125f0 con 0x7f7400140ae0 2023-12-19T16:11:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8012790 con 0x7f7400140ae0 2023-12-19T16:11:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8012b50 con 0x7f7400140ae0 2023-12-19T16:11:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7400159f90 con 0x7f7400140ae0 2023-12-19T16:11:34.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.118+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74000aed40 con 0x7f7400140ae0 2023-12-19T16:11:34.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.122+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74000aeba0 con 0x7f7400140ae0 2023-12-19T16:11:34.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.122+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f73f8013480 con 0x7f7400140ae0 2023-12-19T16:11:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.122+0000 7f73fcff9700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 0x7f73e0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.122+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f73f807ced0 con 0x7f7400140ae0 2023-12-19T16:11:34.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.122+0000 7f73fffff700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 0x7f73e0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.126+0000 7f73fffff700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 0x7f73e0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009880 tx=0x7f73f0018000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.130+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f74000aeba0 con 0x7f7400140ae0 2023-12-19T16:11:34.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.338+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74000aeba0 con 0x7f7400140ae0 2023-12-19T16:11:34.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.338+0000 7f73fcff9700 1 -- 172.21.15.50:0/2269257814 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f74000aeba0 con 0x7f7400140ae0 2023-12-19T16:11:34.345 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:34.345 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8117,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/288935748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":288935748},{"type":"v1","addr":"172.21.15.125:6836","nonce":288935748}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9169,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029795137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029795137},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029795137}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/1883663818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1883663818},{"type":"v1","addr":"172.21.15.50:6833","nonce":1883663818}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":78},{"gid":9789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/619214050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":619214050},{"type":"v1","addr":"172.21.15.50:6835","nonce":619214050}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[]} 2023-12-19T16:11:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 msgr2=0x7f73e0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 0x7f73e0062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009880 tx=0x7f73f0018000 comp rx=0 tx=0).stop 2023-12-19T16:11:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 msgr2=0x7f740016ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f73f800d860 tx=0x7f73f8008040 comp rx=0 tx=0).stop 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 shutdown_connections 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f73e0060670 0x7f73e0062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7400140ae0 0x7f740016ddf0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7400140710 0x7f74000c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 --2- 172.21.15.50:0/2269257814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74000c75e0 0x7f7400159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 >> 172.21.15.50:0/2269257814 conn(0x7f74000c8850 msgr2=0x7f74000c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 shutdown_connections 2023-12-19T16:11:34.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.342+0000 7f7406345700 1 -- 172.21.15.50:0/2269257814 wait complete. 2023-12-19T16:11:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 84 2023-12-19T16:11:34.373 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T16:11:34.373 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T16:11:34.374 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T16:11:34.374 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:11:34.374 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T16:11:34.374 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T16:11:34.374 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T16:11:34.374 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:11:34.375 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T16:11:34.375 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T16:11:34.375 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T16:11:34.375 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:11:34.375 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T16:11:34.375 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T16:11:34.375 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T16:11:34.376 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:11:34.397 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T16:11:34.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T16:11:34.399 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T16:11:34.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T16:11:34.399 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T16:11:34.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T16:11:34.400 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T16:11:34.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T16:11:34.448 INFO:tasks.ceph.mds.a:Started 2023-12-19T16:11:34.448 DEBUG:teuthology.parallel:result is None 2023-12-19T16:11:34.450 INFO:tasks.ceph.mds.b:Started 2023-12-19T16:11:34.450 DEBUG:teuthology.parallel:result is None 2023-12-19T16:11:34.454 INFO:tasks.ceph.mds.c:Started 2023-12-19T16:11:34.454 DEBUG:teuthology.parallel:result is None 2023-12-19T16:11:34.455 INFO:tasks.ceph.mds.d:Started 2023-12-19T16:11:34.455 DEBUG:teuthology.parallel:result is None 2023-12-19T16:11:34.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T16:11:34.558 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T16:11:34.570 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T16:11:34.571 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T16:11:34.574 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T16:11:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.610+0000 7f01fffff700 1 Processor -- start 2023-12-19T16:11:34.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.610+0000 7f01fffff700 1 -- start start 2023-12-19T16:11:34.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 0x7f02000ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fffff700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020006b730 con 0x7f020015ca80 2023-12-19T16:11:34.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fffff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020006ab20 con 0x7f02001591f0 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fffff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020006ae00 con 0x7f0200158e20 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01feffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01fe7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 0x7f02000ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01ff7fe700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f020015ca80 0x7f020006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:43232/0 (socket says 172.21.15.50:43232) 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01feffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:57776/0 (socket says 172.21.15.50:57776) 2023-12-19T16:11:34.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.614+0000 7f01ff7fe700 1 -- 172.21.15.50:0/1166994879 learned_addr learned my addr 172.21.15.50:0/1166994879 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:34.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01feffd700 1 -- 172.21.15.50:0/1166994879 >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 msgr2=0x7f02000ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:34.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01feffd700 1 --2- 172.21.15.50:0/1166994879 >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 0x7f02000ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01feffd700 1 -- 172.21.15.50:0/1166994879 >> v1:172.21.15.125:6789/0 conn(0x7f020015ca80 legacy=0x7f020006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01feffd700 1 -- 172.21.15.50:0/1166994879 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020006b0e0 con 0x7f0200158e20 2023-12-19T16:11:34.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fdffb700 1 -- 172.21.15.50:0/1166994879 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812725644 0 0) 0x7f020006b730 con 0x7f020015ca80 2023-12-19T16:11:34.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fe7fc700 1 --2- 172.21.15.50:0/1166994879 >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 0x7f02000ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:34.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01feffd700 1 --2- 172.21.15.50:0/1166994879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f01f00098e0 tx=0x7f01f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd0fdab5c3a7109 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fdffb700 1 -- 172.21.15.50:0/1166994879 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000b640 con 0x7f0200158e20 2023-12-19T16:11:34.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fdffb700 1 -- 172.21.15.50:0/1166994879 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f000b7a0 con 0x7f0200158e20 2023-12-19T16:11:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fdffb700 1 -- 172.21.15.50:0/1166994879 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000bb40 con 0x7f0200158e20 2023-12-19T16:11:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 -- 172.21.15.50:0/1166994879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 msgr2=0x7f020006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 --2- 172.21.15.50:0/1166994879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f01f00098e0 tx=0x7f01f00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 -- 172.21.15.50:0/1166994879 shutdown_connections 2023-12-19T16:11:34.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 --2- 172.21.15.50:0/1166994879 >> v2:172.21.15.50:3300/0 conn(0x7f02001591f0 0x7f02000ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 --2- 172.21.15.50:0/1166994879 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0200158e20 0x7f020006a540 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 -- 172.21.15.50:0/1166994879 >> 172.21.15.50:0/1166994879 conn(0x7f0200067bb0 msgr2=0x7f02000cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:34.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 -- 172.21.15.50:0/1166994879 shutdown_connections 2023-12-19T16:11:34.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.618+0000 7f01fffff700 1 -- 172.21.15.50:0/1166994879 wait complete. 2023-12-19T16:11:34.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 Processor -- start 2023-12-19T16:11:34.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 -- start start 2023-12-19T16:11:34.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200158e20 0x7f02000b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02001591f0 0x7f02001595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020015ab50 con 0x7f020015ca80 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200159f40 con 0x7f0200158e20 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fffff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020015a220 con 0x7f02001591f0 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01ff7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51536/0 (socket says 172.21.15.50:51536) 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01ff7fe700 1 -- 172.21.15.50:0/204316980 learned_addr learned my addr 172.21.15.50:0/204316980 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01feffd700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200158e20 0x7f02000b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.622+0000 7f01fe7fc700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02001591f0 0x7f02001595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 -- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02001591f0 msgr2=0x7f02001595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02001591f0 0x7f02001595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 -- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200158e20 msgr2=0x7f02000b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200158e20 0x7f02000b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 -- 172.21.15.50:0/204316980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4009420 con 0x7f020015ca80 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01ff7fe700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f01f800cbf0 tx=0x7f01f80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800a8b0 con 0x7f020015ca80 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f800aa30 con 0x7f020015ca80 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800adf0 con 0x7f020015ca80 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f0009420 con 0x7f020015ca80 2023-12-19T16:11:34.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020015a7e0 con 0x7f020015ca80 2023-12-19T16:11:34.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.626+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020015ad00 con 0x7f020015ca80 2023-12-19T16:11:34.635 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.630+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f01f8007300 con 0x7f020015ca80 2023-12-19T16:11:34.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.630+0000 7f01e6ffd700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 0x7f01ec062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:34.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.630+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f01f807cdd0 con 0x7f020015ca80 2023-12-19T16:11:34.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.630+0000 7f01feffd700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 0x7f01ec062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:34.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.634+0000 7f01feffd700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 0x7f01ec062b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01f00093f0 tx=0x7f01f0017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:34.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.638+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f020015ad00 con 0x7f020015ca80 2023-12-19T16:11:34.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:34.822+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f020004be40 con 0x7f020015ca80 2023-12-19T16:11:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.858+0000 7f01e6ffd700 1 -- 172.21.15.50:0/204316980 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v195) v1 ==== 97+0+0 (secure 0 0 0) 0x7f01f804d040 con 0x7f020015ca80 2023-12-19T16:11:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 msgr2=0x7f01ec062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 0x7f01ec062b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01f00093f0 tx=0x7f01f0017000 comp rx=0 tx=0).stop 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 msgr2=0x7f02000c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f01f800cbf0 tx=0x7f01f80019b0 comp rx=0 tx=0).stop 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 shutdown_connections 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f01ec0606a0 0x7f01ec062b50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f020015ca80 0x7f02000c7880 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f02001591f0 0x7f02001595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 --2- 172.21.15.50:0/204316980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200158e20 0x7f02000b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 >> 172.21.15.50:0/204316980 conn(0x7f0200067bb0 msgr2=0x7f02000c4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 shutdown_connections 2023-12-19T16:11:35.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:35.862+0000 7f01fffff700 1 -- 172.21.15.50:0/204316980 wait complete. 2023-12-19T16:11:35.868 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T16:11:35.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T16:11:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.046+0000 7ff9f359e700 1 Processor -- start 2023-12-19T16:11:36.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.050+0000 7ff9f359e700 1 -- start start 2023-12-19T16:11:36.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.050+0000 7ff9f359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9f40cad10 0x7ff9f40cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f406b2c0 con 0x7ff9f415bd00 2023-12-19T16:11:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f406e0f0 con 0x7ff9f406b750 2023-12-19T16:11:36.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f406a6b0 con 0x7ff9f40cad10 2023-12-19T16:11:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f2d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff9f415bd00 0x7ff9f415c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36496/0 (socket says 172.21.15.50:36496) 2023-12-19T16:11:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f2d9d700 1 -- 172.21.15.50:0/1976471917 learned_addr learned my addr 172.21.15.50:0/1976471917 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:36.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f259c700 1 --2- 172.21.15.50:0/1976471917 >> v2:172.21.15.125:3300/0 conn(0x7ff9f40cad10 0x7ff9f40cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f159a700 1 -- 172.21.15.50:0/1976471917 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238198170 0 0) 0x7ff9f406b2c0 con 0x7ff9f415bd00 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f159a700 1 -- 172.21.15.50:0/1976471917 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc003570 con 0x7ff9f415bd00 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f259c700 1 -- 172.21.15.50:0/1976471917 >> v1:172.21.15.125:6789/0 conn(0x7ff9f406b750 legacy=0x7ff9f406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f259c700 1 -- 172.21.15.50:0/1976471917 >> v1:172.21.15.50:6789/0 conn(0x7ff9f415bd00 legacy=0x7ff9f415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f259c700 1 -- 172.21.15.50:0/1976471917 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f406a990 con 0x7ff9f40cad10 2023-12-19T16:11:36.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f259c700 1 --2- 172.21.15.50:0/1976471917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f40cad10 0x7ff9f40cb0e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40098e0 tx=0x7ff9e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f196dde20aa65d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f159a700 1 -- 172.21.15.50:0/1976471917 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400b640 con 0x7ff9f40cad10 2023-12-19T16:11:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f159a700 1 -- 172.21.15.50:0/1976471917 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e400b7a0 con 0x7ff9f40cad10 2023-12-19T16:11:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f159a700 1 -- 172.21.15.50:0/1976471917 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400bb40 con 0x7ff9f40cad10 2023-12-19T16:11:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- 172.21.15.50:0/1976471917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f40cad10 msgr2=0x7ff9f40cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 --2- 172.21.15.50:0/1976471917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f40cad10 0x7ff9f40cb0e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40098e0 tx=0x7ff9e40091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- 172.21.15.50:0/1976471917 shutdown_connections 2023-12-19T16:11:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 --2- 172.21.15.50:0/1976471917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f40cad10 0x7ff9f40cb0e0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.054+0000 7ff9f359e700 1 -- 172.21.15.50:0/1976471917 >> 172.21.15.50:0/1976471917 conn(0x7ff9f4067bb0 msgr2=0x7ff9f40cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.058+0000 7ff9f359e700 1 -- 172.21.15.50:0/1976471917 shutdown_connections 2023-12-19T16:11:36.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.058+0000 7ff9f359e700 1 -- 172.21.15.50:0/1976471917 wait complete. 2023-12-19T16:11:36.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.058+0000 7ff9f359e700 1 Processor -- start 2023-12-19T16:11:36.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.058+0000 7ff9f359e700 1 -- start start 2023-12-19T16:11:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f406b750 0x7ff9f40c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9f415bd00 0x7ff9f416bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f415a7c0 con 0x7ff9f40cad10 2023-12-19T16:11:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4159bb0 con 0x7ff9f406b750 2023-12-19T16:11:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4159e90 con 0x7ff9f415bd00 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51538/0 (socket says 172.21.15.50:51538) 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 -- 172.21.15.50:0/210811493 learned_addr learned my addr 172.21.15.50:0/210811493 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f259c700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f406b750 0x7ff9f40c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f2d9d700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9f415bd00 0x7ff9f416bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 -- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9f415bd00 msgr2=0x7ff9f416bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9f415bd00 0x7ff9f416bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 -- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f406b750 msgr2=0x7ff9f40c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f406b750 0x7ff9f40c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 -- 172.21.15.50:0/210811493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4009420 con 0x7ff9f40cad10 2023-12-19T16:11:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f1d9b700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800c4e0 tx=0x7ff9e800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8011590 con 0x7ff9f40cad10 2023-12-19T16:11:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e8011710 con 0x7ff9f40cad10 2023-12-19T16:11:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f415a170 con 0x7ff9f40cad10 2023-12-19T16:11:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8011ad0 con 0x7ff9f40cad10 2023-12-19T16:11:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.062+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f41598b0 con 0x7ff9f40cad10 2023-12-19T16:11:36.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.066+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f404be40 con 0x7ff9f40cad10 2023-12-19T16:11:36.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.066+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff9e801b3b0 con 0x7ff9f40cad10 2023-12-19T16:11:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.070+0000 7ff9e27fc700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 0x7ff9d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.070+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5257+0+0 (secure 0 0 0) 0x7ff9e807bb70 con 0x7ff9f40cad10 2023-12-19T16:11:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.070+0000 7ff9f259c700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 0x7ff9d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.070+0000 7ff9f259c700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 0x7ff9d4062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40093f0 tx=0x7ff9e4017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.074+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff9e804c170 con 0x7ff9f40cad10 2023-12-19T16:11:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.306+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff9f415c750 con 0x7ff9f40cad10 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9e27fc700 1 -- 172.21.15.50:0/210811493 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff9e804f790 con 0x7ff9f40cad10 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:11:36.314 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T16:11:36.315 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.317 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.319 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T16:11:36.320 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:11:36.321 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.322 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:11:36.324 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:36.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T16:11:36.326 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 msgr2=0x7ff9d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 0x7ff9d4062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9e40093f0 tx=0x7ff9e4017000 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 msgr2=0x7ff9f4159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800c4e0 tx=0x7ff9e800c7f0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 shutdown_connections 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff9d4060670 0x7ff9d4062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff9f415bd00 0x7ff9f416bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff9f40cad10 0x7ff9f4159370 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 --2- 172.21.15.50:0/210811493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9f406b750 0x7ff9f40c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 >> 172.21.15.50:0/210811493 conn(0x7ff9f4067bb0 msgr2=0x7ff9f40b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 shutdown_connections 2023-12-19T16:11:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.310+0000 7ff9f359e700 1 -- 172.21.15.50:0/210811493 wait complete. 2023-12-19T16:11:36.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:36.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.502+0000 7f2c6406e700 1 Processor -- start 2023-12-19T16:11:36.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 -- start start 2023-12-19T16:11:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 0x7f2c5c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2c5c159340 0x7f2c5c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5c068c70 con 0x7f2c5c158f70 2023-12-19T16:11:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5c068060 con 0x7f2c5c159340 2023-12-19T16:11:36.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.506+0000 7f2c6406e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c068340 con 0x7f2c5c069100 2023-12-19T16:11:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2c5c159340 0x7f2c5c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c61609700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 0x7f2c5c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2c5c159340 0x7f2c5c0cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35020/0 (socket says 172.21.15.50:35020) 2023-12-19T16:11:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 -- 172.21.15.50:0/1383634125 learned_addr learned my addr 172.21.15.50:0/1383634125 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:36.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c61e0a700 1 --1- 172.21.15.50:0/1383634125 >> v1:172.21.15.125:6789/0 conn(0x7f2c5c158f70 0x7f2c5c0cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:51958/0 (socket says 172.21.15.50:51958) 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 -- 172.21.15.50:0/1383634125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 msgr2=0x7f2c5c067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 0x7f2c5c067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 -- 172.21.15.50:0/1383634125 >> v1:172.21.15.125:6789/0 conn(0x7f2c5c158f70 legacy=0x7f2c5c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 -- 172.21.15.50:0/1383634125 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c068620 con 0x7f2c5c159340 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c61609700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 0x7f2c5c067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:36.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6260b700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c159340 0x7f2c5c0cb3e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009f20 tx=0x7f2c54009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2195f6d1731ce8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c60e08700 1 -- 172.21.15.50:0/1383634125 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5400e070 con 0x7f2c5c159340 2023-12-19T16:11:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c60e08700 1 -- 172.21.15.50:0/1383634125 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5400b9d0 con 0x7f2c5c159340 2023-12-19T16:11:36.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c60e08700 1 -- 172.21.15.50:0/1383634125 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c540092e0 con 0x7f2c5c159340 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 -- 172.21.15.50:0/1383634125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c159340 msgr2=0x7f2c5c0cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c159340 0x7f2c5c0cb3e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009f20 tx=0x7f2c54009840 comp rx=0 tx=0).stop 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 -- 172.21.15.50:0/1383634125 shutdown_connections 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c159340 0x7f2c5c0cb3e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 --2- 172.21.15.50:0/1383634125 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c069100 0x7f2c5c067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.510+0000 7f2c6406e700 1 -- 172.21.15.50:0/1383634125 >> 172.21.15.50:0/1383634125 conn(0x7f2c5c0ccc40 msgr2=0x7f2c5c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- 172.21.15.50:0/1383634125 shutdown_connections 2023-12-19T16:11:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- 172.21.15.50:0/1383634125 wait complete. 2023-12-19T16:11:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 Processor -- start 2023-12-19T16:11:36.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- start start 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 0x7f2c5c169470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c159340 0x7f2c5c150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c151a70 con 0x7f2c5c158f70 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c0b06d0 con 0x7f2c5c069100 2023-12-19T16:11:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c6406e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c16d9b0 con 0x7f2c5c159340 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.514+0000 7f2c61609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35024/0 (socket says 172.21.15.50:35024) 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 -- 172.21.15.50:0/616405004 learned_addr learned my addr 172.21.15.50:0/616405004 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c6260b700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c159340 0x7f2c5c150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61e0a700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 0x7f2c5c169470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 -- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c159340 msgr2=0x7f2c5c150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c159340 0x7f2c5c150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 -- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 msgr2=0x7f2c5c169470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 0x7f2c5c169470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 -- 172.21.15.50:0/616405004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c009420 con 0x7f2c5c158f70 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61e0a700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 0x7f2c5c169470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c61609700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00ecb0 tx=0x7f2c4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00c950 con 0x7f2c5c158f70 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c00cab0 con 0x7f2c5c158f70 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00ce30 con 0x7f2c5c158f70 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c54009ae0 con 0x7f2c5c158f70 2023-12-19T16:11:36.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.518+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5c169de0 con 0x7f2c5c158f70 2023-12-19T16:11:36.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.522+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5c0b0860 con 0x7f2c5c158f70 2023-12-19T16:11:36.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.522+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2c4c013680 con 0x7f2c5c158f70 2023-12-19T16:11:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.522+0000 7f2c51ffb700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 0x7f2c40062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.522+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5257+0+0 (secure 0 0 0) 0x7f2c4c07cba0 con 0x7f2c5c158f70 2023-12-19T16:11:36.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.522+0000 7f2c61e0a700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 0x7f2c40062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.526+0000 7f2c61e0a700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 0x7f2c40062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2c58007fd0 tx=0x7f2c58007e30 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.530+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c4c04d1a0 con 0x7f2c5c158f70 2023-12-19T16:11:36.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.698+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c5c071120 con 0x7f2c5c158f70 2023-12-19T16:11:36.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.698+0000 7f2c51ffb700 1 -- 172.21.15.50:0/616405004 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+12483 (secure 0 0 0) 0x7f2c4c0507c0 con 0x7f2c5c158f70 2023-12-19T16:11:36.705 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:36.705 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":195,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:35.850532+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 msgr2=0x7f2c40062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 0x7f2c40062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2c58007fd0 tx=0x7f2c58007e30 comp rx=0 tx=0).stop 2023-12-19T16:11:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 msgr2=0x7f2c5c16d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00ecb0 tx=0x7f2c4c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 shutdown_connections 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2c40060670 0x7f2c40062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2c5c159340 0x7f2c5c150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c5c158f70 0x7f2c5c16d470 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 --2- 172.21.15.50:0/616405004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c5c069100 0x7f2c5c169470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 >> 172.21.15.50:0/616405004 conn(0x7f2c5c0ccc40 msgr2=0x7f2c5c153590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 shutdown_connections 2023-12-19T16:11:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.702+0000 7f2c6406e700 1 -- 172.21.15.50:0/616405004 wait complete. 2023-12-19T16:11:36.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T16:11:36.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:11:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 Processor -- start 2023-12-19T16:11:36.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 -- start start 2023-12-19T16:11:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0394158e20 0x7f03940ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039406d5e0 con 0x7f03940cb970 2023-12-19T16:11:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03940cbdf0 con 0x7f03940ca2c0 2023-12-19T16:11:36.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f039b5c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03940cc0d0 con 0x7f0394158e20 2023-12-19T16:11:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0398b60700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f03940cb970 0x7f039406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50438/0 (socket says 172.21.15.50:50438) 2023-12-19T16:11:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0398b60700 1 -- 172.21.15.50:0/3221897728 learned_addr learned my addr 172.21.15.50:0/3221897728 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:36.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0399361700 1 --2- 172.21.15.50:0/3221897728 >> v2:172.21.15.125:3300/0 conn(0x7f0394158e20 0x7f03940ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0399361700 1 -- 172.21.15.50:0/3221897728 >> v1:172.21.15.125:6789/0 conn(0x7f03940ca2c0 legacy=0x7f039406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f038bfff700 1 -- 172.21.15.50:0/3221897728 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498133307 0 0) 0x7f039406d5e0 con 0x7f03940cb970 2023-12-19T16:11:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0399361700 1 -- 172.21.15.50:0/3221897728 >> v1:172.21.15.50:6789/0 conn(0x7f03940cb970 legacy=0x7f039406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0399361700 1 -- 172.21.15.50:0/3221897728 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03940cc3b0 con 0x7f0394158e20 2023-12-19T16:11:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.894+0000 7f0399361700 1 --2- 172.21.15.50:0/3221897728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0394158e20 0x7f03940ce5c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0390009910 tx=0x7f03900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c26fad0559ef95d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f038bfff700 1 -- 172.21.15.50:0/3221897728 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039000b820 con 0x7f0394158e20 2023-12-19T16:11:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f038bfff700 1 -- 172.21.15.50:0/3221897728 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039000b980 con 0x7f0394158e20 2023-12-19T16:11:36.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f038bfff700 1 -- 172.21.15.50:0/3221897728 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039000bd20 con 0x7f0394158e20 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 -- 172.21.15.50:0/3221897728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0394158e20 msgr2=0x7f03940ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3221897728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0394158e20 0x7f03940ce5c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0390009910 tx=0x7f03900091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 -- 172.21.15.50:0/3221897728 shutdown_connections 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3221897728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0394158e20 0x7f03940ce5c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 -- 172.21.15.50:0/3221897728 >> 172.21.15.50:0/3221897728 conn(0x7f0394067bb0 msgr2=0x7f03940d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 -- 172.21.15.50:0/3221897728 shutdown_connections 2023-12-19T16:11:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.898+0000 7f039b5c5700 1 -- 172.21.15.50:0/3221897728 wait complete. 2023-12-19T16:11:36.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 Processor -- start 2023-12-19T16:11:36.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 -- start start 2023-12-19T16:11:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0394158e20 0x7f03940b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03940b5680 0x7f03940c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03940b3ab0 con 0x7f0394158e20 2023-12-19T16:11:36.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03940b3d90 con 0x7f03940ca2c0 2023-12-19T16:11:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f039b5c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03940b4070 con 0x7f03940b5680 2023-12-19T16:11:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0398b60700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0394158e20 0x7f03940b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:40388/0 (socket says 172.21.15.50:40388) 2023-12-19T16:11:36.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 -- 172.21.15.50:0/3388878579 learned_addr learned my addr 172.21.15.50:0/3388878579 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399b62700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03940b5680 0x7f03940c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 -- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03940b5680 msgr2=0x7f03940c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03940b5680 0x7f03940c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 -- 172.21.15.50:0/3388878579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0394158e20 msgr2=0x7f03940b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0394158e20 0x7f03940b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.902+0000 7f0399361700 1 -- 172.21.15.50:0/3388878579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0390009420 con 0x7f03940ca2c0 2023-12-19T16:11:36.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f0399361700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f03900093f0 tx=0x7f0390017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039001f070 con 0x7f03940ca2c0 2023-12-19T16:11:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039000baf0 con 0x7f03940ca2c0 2023-12-19T16:11:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390025420 con 0x7f03940ca2c0 2023-12-19T16:11:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03940b4350 con 0x7f03940ca2c0 2023-12-19T16:11:36.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03940b5cf0 con 0x7f03940ca2c0 2023-12-19T16:11:36.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.906+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03940b4830 con 0x7f03940ca2c0 2023-12-19T16:11:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.910+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0390023070 con 0x7f03940ca2c0 2023-12-19T16:11:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.910+0000 7f03897fa700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 0x7f0378062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:36.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.910+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5257+0+0 (secure 0 0 0) 0x7f039007dcf0 con 0x7f03940ca2c0 2023-12-19T16:11:36.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.910+0000 7f0398b60700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 0x7f0378062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.910+0000 7f0398b60700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 0x7f0378062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03940cecd0 tx=0x7f0384007d10 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:36.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:36.918+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0390082070 con 0x7f03940ca2c0 2023-12-19T16:11:37.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.098+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03940cc690 con 0x7f03940ca2c0 2023-12-19T16:11:37.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.098+0000 7f03897fa700 1 -- 172.21.15.50:0/3388878579 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) v1 ==== 93+0+10 (secure 0 0 0) 0x7f039002a0a0 con 0x7f03940ca2c0 2023-12-19T16:11:37.105 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:11:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 msgr2=0x7f0378062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 0x7f0378062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03940cecd0 tx=0x7f0384007d10 comp rx=0 tx=0).stop 2023-12-19T16:11:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 msgr2=0x7f03940b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f03900093f0 tx=0x7f0390017000 comp rx=0 tx=0).stop 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 shutdown_connections 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0378060670 0x7f0378062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03940b5680 0x7f03940c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0394158e20 0x7f03940b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 --2- 172.21.15.50:0/3388878579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03940ca2c0 0x7f03940b5140 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 >> 172.21.15.50:0/3388878579 conn(0x7f0394067bb0 msgr2=0x7f039414eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 shutdown_connections 2023-12-19T16:11:37.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.102+0000 7f039b5c5700 1 -- 172.21.15.50:0/3388878579 wait complete. 2023-12-19T16:11:37.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T16:11:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.294+0000 7f6075745700 1 Processor -- start 2023-12-19T16:11:37.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.294+0000 7f6075745700 1 -- start start 2023-12-19T16:11:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f6070158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f607006de60 0x7f60700cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607006ac30 con 0x7f607006b0c0 2023-12-19T16:11:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60700b4590 con 0x7f607006de60 2023-12-19T16:11:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60700ccc80 con 0x7f6070158c10 2023-12-19T16:11:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f607006de60 0x7f60700cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f607006de60 0x7f60700cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35052/0 (socket says 172.21.15.50:35052) 2023-12-19T16:11:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 -- 172.21.15.50:0/3375267610 learned_addr learned my addr 172.21.15.50:0/3375267610 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:37.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606effd700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f6070158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 -- 172.21.15.50:0/3375267610 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 msgr2=0x7f6070158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f6070158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 -- 172.21.15.50:0/3375267610 >> v1:172.21.15.50:6789/0 conn(0x7f607006b0c0 legacy=0x7f607006b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 -- 172.21.15.50:0/3375267610 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607006d180 con 0x7f607006de60 2023-12-19T16:11:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606e7fc700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006de60 0x7f60700cc740 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4fd2c29bf029e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606dffb700 1 -- 172.21.15.50:0/3375267610 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606400b860 con 0x7f607006de60 2023-12-19T16:11:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606dffb700 1 -- 172.21.15.50:0/3375267610 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606400b9c0 con 0x7f607006de60 2023-12-19T16:11:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f606dffb700 1 -- 172.21.15.50:0/3375267610 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606400bd60 con 0x7f607006de60 2023-12-19T16:11:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 -- 172.21.15.50:0/3375267610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006de60 msgr2=0x7f60700cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.298+0000 7f6075745700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006de60 0x7f60700cc740 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064009250 comp rx=0 tx=0).stop 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 -- 172.21.15.50:0/3375267610 shutdown_connections 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006de60 0x7f60700cc740 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 --2- 172.21.15.50:0/3375267610 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f6070158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 -- 172.21.15.50:0/3375267610 >> 172.21.15.50:0/3375267610 conn(0x7f60700d0920 msgr2=0x7f60700b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 -- 172.21.15.50:0/3375267610 shutdown_connections 2023-12-19T16:11:37.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.302+0000 7f6075745700 1 -- 172.21.15.50:0/3375267610 wait complete. 2023-12-19T16:11:37.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 Processor -- start 2023-12-19T16:11:37.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 -- start start 2023-12-19T16:11:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 0x7f6070159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f607014d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607014e770 con 0x7f607006b0c0 2023-12-19T16:11:37.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607014d660 con 0x7f607006de60 2023-12-19T16:11:37.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f6075745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607014d8c0 con 0x7f6070158c10 2023-12-19T16:11:37.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35060/0 (socket says 172.21.15.50:35060) 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 -- 172.21.15.50:0/4039944792 learned_addr learned my addr 172.21.15.50:0/4039944792 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606e7fc700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 0x7f6070159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606f7fe700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f607014d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 -- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 msgr2=0x7f607014d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f607014d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 -- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 msgr2=0x7f6070159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 0x7f6070159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 -- 172.21.15.50:0/4039944792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640094f0 con 0x7f607006b0c0 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606e7fc700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 0x7f6070159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606f7fe700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f607014d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.306+0000 7f606effd700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f606000d840 tx=0x7f6060008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060014620 con 0x7f607006b0c0 2023-12-19T16:11:37.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060014780 con 0x7f607006b0c0 2023-12-19T16:11:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607014dba0 con 0x7f607006b0c0 2023-12-19T16:11:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060014b20 con 0x7f607006b0c0 2023-12-19T16:11:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607014e160 con 0x7f607006b0c0 2023-12-19T16:11:37.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607004be40 con 0x7f607006b0c0 2023-12-19T16:11:37.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.310+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f606001d400 con 0x7f607006b0c0 2023-12-19T16:11:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.314+0000 7f6056ffd700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 0x7f605c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.314+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5257+0+0 (secure 0 0 0) 0x7f606007cf20 con 0x7f607006b0c0 2023-12-19T16:11:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.314+0000 7f606e7fc700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 0x7f605c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.314+0000 7f606e7fc700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 0x7f605c062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.318+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f606004d5c0 con 0x7f607006b0c0 2023-12-19T16:11:37.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.498+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f607014c330 con 0x7f607006b0c0 2023-12-19T16:11:37.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.502+0000 7f6056ffd700 1 -- 172.21.15.50:0/4039944792 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v195) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6060050be0 con 0x7f607006b0c0 2023-12-19T16:11:37.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:37.507 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 msgr2=0x7f605c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 0x7f605c062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064017000 comp rx=0 tx=0).stop 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 msgr2=0x7f607013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f606000d840 tx=0x7f6060008040 comp rx=0 tx=0).stop 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 shutdown_connections 2023-12-19T16:11:37.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f605c0606a0 0x7f605c062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f607006de60 0x7f6070159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6070158c10 0x7f607014d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 --2- 172.21.15.50:0/4039944792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f607006b0c0 0x7f607013c9c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 >> 172.21.15.50:0/4039944792 conn(0x7f60700d0920 msgr2=0x7f607015d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 shutdown_connections 2023-12-19T16:11:37.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.506+0000 7f6075745700 1 -- 172.21.15.50:0/4039944792 wait complete. 2023-12-19T16:11:37.531 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T16:11:37.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T16:11:37.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.698+0000 7f127a8ae700 1 Processor -- start 2023-12-19T16:11:37.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.698+0000 7f127a8ae700 1 -- start start 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f127a8ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12740cb130 0x7f12740cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f127a8ae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f127406a6a0 0x7f127406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f127a8ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12740b0ac0 con 0x7f127415c220 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f127a8ae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127406b030 con 0x7f127406a6a0 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f127a8ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127406b210 con 0x7f12740cb130 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f127406a6a0 0x7f127406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f127406a6a0 0x7f127406aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35076/0 (socket says 172.21.15.50:35076) 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 -- 172.21.15.50:0/1591998421 learned_addr learned my addr 172.21.15.50:0/1591998421 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:37.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f1278e4b700 1 --1- 172.21.15.50:0/1591998421 >> v1:172.21.15.50:6789/0 conn(0x7f127415c220 0x7f127406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50460/0 (socket says 172.21.15.50:50460) 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f1273fff700 1 --2- 172.21.15.50:0/1591998421 >> v2:172.21.15.125:3300/0 conn(0x7f12740cb130 0x7f12740cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 -- 172.21.15.50:0/1591998421 >> v2:172.21.15.125:3300/0 conn(0x7f12740cb130 msgr2=0x7f12740cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 --2- 172.21.15.50:0/1591998421 >> v2:172.21.15.125:3300/0 conn(0x7f12740cb130 0x7f12740cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 -- 172.21.15.50:0/1591998421 >> v1:172.21.15.50:6789/0 conn(0x7f127415c220 legacy=0x7f127406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 -- 172.21.15.50:0/1591998421 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12740ca810 con 0x7f127406a6a0 2023-12-19T16:11:37.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.702+0000 7f12737fe700 1 --2- 172.21.15.50:0/1591998421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f127406a6a0 0x7f127406aaf0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f12740694e0 tx=0x7f1268009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527810b1c74144a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f1272ffd700 1 -- 172.21.15.50:0/1591998421 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268009e70 con 0x7f127406a6a0 2023-12-19T16:11:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f1272ffd700 1 -- 172.21.15.50:0/1591998421 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126800b860 con 0x7f127406a6a0 2023-12-19T16:11:37.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f1272ffd700 1 -- 172.21.15.50:0/1591998421 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126800bc00 con 0x7f127406a6a0 2023-12-19T16:11:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 -- 172.21.15.50:0/1591998421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f127406a6a0 msgr2=0x7f127406aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 --2- 172.21.15.50:0/1591998421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f127406a6a0 0x7f127406aaf0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f12740694e0 tx=0x7f1268009250 comp rx=0 tx=0).stop 2023-12-19T16:11:37.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 -- 172.21.15.50:0/1591998421 shutdown_connections 2023-12-19T16:11:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 --2- 172.21.15.50:0/1591998421 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f127406a6a0 0x7f127406aaf0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 --2- 172.21.15.50:0/1591998421 >> v2:172.21.15.125:3300/0 conn(0x7f12740cb130 0x7f12740cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 -- 172.21.15.50:0/1591998421 >> 172.21.15.50:0/1591998421 conn(0x7f1274067bb0 msgr2=0x7f12740cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 -- 172.21.15.50:0/1591998421 shutdown_connections 2023-12-19T16:11:37.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.706+0000 7f127a8ae700 1 -- 172.21.15.50:0/1591998421 wait complete. 2023-12-19T16:11:37.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 Processor -- start 2023-12-19T16:11:37.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 -- start start 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f127406a6a0 0x7f1274157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f127415c220 0x7f12740b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c87f0 con 0x7f12740cb130 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c9880 con 0x7f127415c220 2023-12-19T16:11:37.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f127a8ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127416dcb0 con 0x7f127406a6a0 2023-12-19T16:11:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35084/0 (socket says 172.21.15.50:35084) 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 -- 172.21.15.50:0/2170719348 learned_addr learned my addr 172.21.15.50:0/2170719348 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f1278e4b700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f127415c220 0x7f12740b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f1273fff700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f127406a6a0 0x7f1274157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 -- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f127406a6a0 msgr2=0x7f1274157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f127406a6a0 0x7f1274157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 -- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f127415c220 msgr2=0x7f12740b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f127415c220 0x7f12740b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:37.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.710+0000 7f12737fe700 1 -- 172.21.15.50:0/2170719348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12680094f0 con 0x7f12740cb130 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f12737fe700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1268006fd0 tx=0x7f1268017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126800bbe0 con 0x7f12740cb130 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268011420 con 0x7f12740cb130 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680117e0 con 0x7f12740cb130 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127416d9d0 con 0x7f12740cb130 2023-12-19T16:11:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1274169bd0 con 0x7f12740cb130 2023-12-19T16:11:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.714+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1274169750 con 0x7f12740cb130 2023-12-19T16:11:37.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.718+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1268011c10 con 0x7f12740cb130 2023-12-19T16:11:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.718+0000 7f125bfff700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 0x7f1260062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.718+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5257+0+0 (secure 0 0 0) 0x7f126807df60 con 0x7f12740cb130 2023-12-19T16:11:37.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.718+0000 7f1273fff700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 0x7f1260062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:37.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.722+0000 7f1273fff700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 0x7f1260062b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12740a9300 tx=0x7f12640091a0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:37.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.726+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1274169750 con 0x7f12740cb130 2023-12-19T16:11:37.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:37.898+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1274169750 con 0x7f12740cb130 2023-12-19T16:11:38.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.450+0000 7f125bfff700 1 -- 172.21.15.50:0/2170719348 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v196) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1274169750 con 0x7f12740cb130 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 msgr2=0x7f1260062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 0x7f1260062b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12740a9300 tx=0x7f12640091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 msgr2=0x7f12740b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1268006fd0 tx=0x7f1268017000 comp rx=0 tx=0).stop 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 shutdown_connections 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f12600606a0 0x7f1260062b50 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12740a9300 tx=0x7f12640091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f127415c220 0x7f12740b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f12740cb130 0x7f12740b4f90 secure :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f1268006fd0 tx=0x7f1268017000 comp rx=0 tx=0).stop 2023-12-19T16:11:38.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 --2- 172.21.15.50:0/2170719348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f127406a6a0 0x7f1274157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.458+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 >> 172.21.15.50:0/2170719348 conn(0x7f1274067bb0 msgr2=0x7f127415b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:38.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.462+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 shutdown_connections 2023-12-19T16:11:38.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.462+0000 7f127a8ae700 1 -- 172.21.15.50:0/2170719348 wait complete. 2023-12-19T16:11:38.468 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T16:11:38.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T16:11:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.658+0000 7fc66eb63700 1 Processor -- start 2023-12-19T16:11:38.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.658+0000 7fc66eb63700 1 -- start start 2023-12-19T16:11:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66eb63700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc66815d140 0x7fc66815d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66eb63700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 0x7fc66806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66eb63700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680c7820 con 0x7fc66806d750 2023-12-19T16:11:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66eb63700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66806db80 con 0x7fc66815d140 2023-12-19T16:11:38.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66eb63700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6680ac5c0 con 0x7fc6680ac170 2023-12-19T16:11:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc66815d140 0x7fc66815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66d360700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 0x7fc66806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc66815d140 0x7fc66815d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35096/0 (socket says 172.21.15.50:35096) 2023-12-19T16:11:38.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 -- 172.21.15.50:0/641995260 learned_addr learned my addr 172.21.15.50:0/641995260 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 -- 172.21.15.50:0/641995260 >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 msgr2=0x7fc66806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 --2- 172.21.15.50:0/641995260 >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 0x7fc66806d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 -- 172.21.15.50:0/641995260 >> v1:172.21.15.125:6789/0 conn(0x7fc66806d750 legacy=0x7fc66806ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 -- 172.21.15.50:0/641995260 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680c6c10 con 0x7fc66815d140 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66cb5f700 1 -- 172.21.15.50:0/641995260 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969501364 0 0) 0x7fc6680c7820 con 0x7fc66806d750 2023-12-19T16:11:38.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66d360700 1 --2- 172.21.15.50:0/641995260 >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 0x7fc66806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66db61700 1 --2- 172.21.15.50:0/641995260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc66815d510 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc668160f00 tx=0x7fc664009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df989bfab9f1fd66 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.662+0000 7fc66cb5f700 1 -- 172.21.15.50:0/641995260 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664009e70 con 0x7fc66815d140 2023-12-19T16:11:38.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66cb5f700 1 -- 172.21.15.50:0/641995260 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400b860 con 0x7fc66815d140 2023-12-19T16:11:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66cb5f700 1 -- 172.21.15.50:0/641995260 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66400bc00 con 0x7fc66815d140 2023-12-19T16:11:38.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 -- 172.21.15.50:0/641995260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 msgr2=0x7fc66815d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 --2- 172.21.15.50:0/641995260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc66815d510 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc668160f00 tx=0x7fc664009250 comp rx=0 tx=0).stop 2023-12-19T16:11:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 -- 172.21.15.50:0/641995260 shutdown_connections 2023-12-19T16:11:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 --2- 172.21.15.50:0/641995260 >> v2:172.21.15.125:3300/0 conn(0x7fc6680ac170 0x7fc66806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 --2- 172.21.15.50:0/641995260 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc66815d510 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 -- 172.21.15.50:0/641995260 >> 172.21.15.50:0/641995260 conn(0x7fc6680c8cd0 msgr2=0x7fc6680c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 -- 172.21.15.50:0/641995260 shutdown_connections 2023-12-19T16:11:38.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.666+0000 7fc66eb63700 1 -- 172.21.15.50:0/641995260 wait complete. 2023-12-19T16:11:38.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 Processor -- start 2023-12-19T16:11:38.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 -- start start 2023-12-19T16:11:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc66806d750 0x7fc668160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 0x7fc66815aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668160950 con 0x7fc66815d140 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680afad0 con 0x7fc6680ac170 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c1c20 con 0x7fc66806d750 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35098/0 (socket says 172.21.15.50:35098) 2023-12-19T16:11:38.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 -- 172.21.15.50:0/3653964091 learned_addr learned my addr 172.21.15.50:0/3653964091 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66db61700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc66806d750 0x7fc668160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66d360700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 0x7fc66815aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 -- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc66806d750 msgr2=0x7fc668160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc66806d750 0x7fc668160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 -- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 msgr2=0x7fc66815aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 0x7fc66815aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 -- 172.21.15.50:0/3653964091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009420 con 0x7fc66815d140 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66d360700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 0x7fc66815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66e362700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc66000c8e0 tx=0x7fc6600016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660012630 con 0x7fc66815d140 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6600127b0 con 0x7fc66815d140 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6640094f0 con 0x7fc66815d140 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.670+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660012b70 con 0x7fc66815d140 2023-12-19T16:11:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.674+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6680af760 con 0x7fc66815d140 2023-12-19T16:11:38.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.674+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6680c2100 con 0x7fc66815d140 2023-12-19T16:11:38.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.674+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc660003030 con 0x7fc66815d140 2023-12-19T16:11:38.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.678+0000 7fc65dffb700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 0x7fc64c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.678+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5627+0+0 (secure 0 0 0) 0x7fc66007ca20 con 0x7fc66815d140 2023-12-19T16:11:38.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.678+0000 7fc66db61700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 0x7fc64c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:38.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.678+0000 7fc66db61700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 0x7fc64c062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc668160f00 tx=0x7fc664017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:38.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.682+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc66004ceb0 con 0x7fc66815d140 2023-12-19T16:11:38.867 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:38.862+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc66804be40 con 0x7fc66815d140 2023-12-19T16:11:39.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.462+0000 7fc65dffb700 1 -- 172.21.15.50:0/3653964091 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v197) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc6600504d0 con 0x7fc66815d140 2023-12-19T16:11:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 msgr2=0x7fc64c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 0x7fc64c062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc668160f00 tx=0x7fc664017000 comp rx=0 tx=0).stop 2023-12-19T16:11:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 msgr2=0x7fc6680c0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc66000c8e0 tx=0x7fc6600016c0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 shutdown_connections 2023-12-19T16:11:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc64c060670 0x7fc64c062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6680ac170 0x7fc66815aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc66815d140 0x7fc6680c0d40 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.466+0000 7fc66eb63700 1 --2- 172.21.15.50:0/3653964091 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc66806d750 0x7fc668160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.470+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 >> 172.21.15.50:0/3653964091 conn(0x7fc6680c8cd0 msgr2=0x7fc66815b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.470+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 shutdown_connections 2023-12-19T16:11:39.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.474+0000 7fc66eb63700 1 -- 172.21.15.50:0/3653964091 wait complete. 2023-12-19T16:11:39.489 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T16:11:39.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T16:11:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 Processor -- start 2023-12-19T16:11:39.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 -- start start 2023-12-19T16:11:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33f40c6d20 0x7f33f40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f406a7a0 con 0x7f33f406c7c0 2023-12-19T16:11:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f40c7630 con 0x7f33f40c6d20 2023-12-19T16:11:39.681 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.674+0000 7f33f8f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f406d4b0 con 0x7f33f406ac30 2023-12-19T16:11:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33f40c6d20 0x7f33f40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.682 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33f40c6d20 0x7f33f40c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:40438/0 (socket says 172.21.15.50:40438) 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 -- 172.21.15.50:0/2151734505 learned_addr learned my addr 172.21.15.50:0/2151734505 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f2d9d700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 -- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 msgr2=0x7f33f406b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f406b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 -- 172.21.15.50:0/2151734505 >> v1:172.21.15.125:6789/0 conn(0x7f33f406c7c0 legacy=0x7f33f406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 -- 172.21.15.50:0/2151734505 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f406d790 con 0x7f33f40c6d20 2023-12-19T16:11:39.683 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f259c700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40c70f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f33e40099b0 tx=0x7f33e4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20539cfca1813af2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f159a700 1 -- 172.21.15.50:0/2151734505 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4009ea0 con 0x7f33f40c6d20 2023-12-19T16:11:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f159a700 1 -- 172.21.15.50:0/2151734505 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e400b860 con 0x7f33f40c6d20 2023-12-19T16:11:39.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f159a700 1 -- 172.21.15.50:0/2151734505 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e400bc00 con 0x7f33f40c6d20 2023-12-19T16:11:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f8f45700 1 -- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 msgr2=0x7f33f40c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.678+0000 7f33f8f45700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40c70f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f33e40099b0 tx=0x7f33e4009250 comp rx=0 tx=0).stop 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 -- 172.21.15.50:0/2151734505 shutdown_connections 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f406b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 --2- 172.21.15.50:0/2151734505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40c70f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 -- 172.21.15.50:0/2151734505 >> 172.21.15.50:0/2151734505 conn(0x7f33f4067bb0 msgr2=0x7f33f40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 -- 172.21.15.50:0/2151734505 shutdown_connections 2023-12-19T16:11:39.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 -- 172.21.15.50:0/2151734505 wait complete. 2023-12-19T16:11:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 Processor -- start 2023-12-19T16:11:39.688 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.682+0000 7f33f8f45700 1 -- start start 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f4152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40ab340 con 0x7f33f406c7c0 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40c62f0 con 0x7f33f40c6d20 2023-12-19T16:11:39.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4167b50 con 0x7f33f406ac30 2023-12-19T16:11:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35114/0 (socket says 172.21.15.50:35114) 2023-12-19T16:11:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 -- 172.21.15.50:0/595976193 learned_addr learned my addr 172.21.15.50:0/595976193 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f259c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f4152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f2d9d700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 -- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 msgr2=0x7f33f4152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f4152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 -- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 msgr2=0x7f33f40b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:39.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 -- 172.21.15.50:0/595976193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e40094f0 con 0x7f33f406c7c0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f1d9b700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c7f0 tx=0x7f33e8007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8011590 con 0x7f33f406c7c0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e8011710 con 0x7f33f406c7c0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f40aadc0 con 0x7f33f406c7c0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8011ad0 con 0x7f33f406c7c0 2023-12-19T16:11:39.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.686+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f41677e0 con 0x7f33f406c7c0 2023-12-19T16:11:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.690+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f4071120 con 0x7f33f406c7c0 2023-12-19T16:11:39.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.690+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f33e801c460 con 0x7f33f406c7c0 2023-12-19T16:11:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.694+0000 7f33e27fc700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 0x7f33d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:39.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.694+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6009+0+0 (secure 0 0 0) 0x7f33e807d1c0 con 0x7f33f406c7c0 2023-12-19T16:11:39.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.694+0000 7f33f259c700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 0x7f33d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:39.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.694+0000 7f33f259c700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 0x7f33d4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f33e40099b0 tx=0x7f33e4017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:39.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.698+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f33e804d550 con 0x7f33f406c7c0 2023-12-19T16:11:39.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:39.910+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f33f4153690 con 0x7f33f406c7c0 2023-12-19T16:11:40.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.470+0000 7f33e27fc700 1 -- 172.21.15.50:0/595976193 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v86) v1 ==== 174+0+0 (secure 0 0 0) 0x7f33e8050b70 con 0x7f33f406c7c0 2023-12-19T16:11:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.474+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 msgr2=0x7f33d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.474+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 0x7f33d4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f33e40099b0 tx=0x7f33e4017000 comp rx=0 tx=0).stop 2023-12-19T16:11:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.474+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 msgr2=0x7f33f4167fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.474+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c7f0 tx=0x7f33e8007f10 comp rx=0 tx=0).stop 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 shutdown_connections 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f33d4060670 0x7f33d4062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f40c6d20 0x7f33f40b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f33f406c7c0 0x7f33f4167fe0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 --2- 172.21.15.50:0/595976193 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f33f406ac30 0x7f33f4152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 >> 172.21.15.50:0/595976193 conn(0x7f33f4067bb0 msgr2=0x7f33f40c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:40.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 shutdown_connections 2023-12-19T16:11:40.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.478+0000 7f33f8f45700 1 -- 172.21.15.50:0/595976193 wait complete. 2023-12-19T16:11:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 23 and data pool 24 2023-12-19T16:11:40.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:40.685 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.678+0000 7f633cce3700 1 Processor -- start 2023-12-19T16:11:40.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 -- start start 2023-12-19T16:11:40.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 0x7f63380b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63380b13b0 0x7f633806d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63380c9c20 con 0x7f6338155990 2023-12-19T16:11:40.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63380ae7e0 con 0x7f63380b13b0 2023-12-19T16:11:40.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f633cce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63380cbc50 con 0x7f63380c97d0 2023-12-19T16:11:40.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.682+0000 7f63377fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6338155990 0x7f63380b0670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50462/0 (socket says 172.21.15.50:50462) 2023-12-19T16:11:40.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f63377fe700 1 -- 172.21.15.50:0/2782083241 learned_addr learned my addr 172.21.15.50:0/2782083241 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:40.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 --2- 172.21.15.50:0/2782083241 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 0x7f63380b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6337fff700 1 --2- 172.21.15.50:0/2782083241 >> v2:172.21.15.125:3300/0 conn(0x7f63380b13b0 0x7f633806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6336ffd700 1 -- 172.21.15.50:0/2782083241 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980604891 0 0) 0x7f63380c9c20 con 0x7f6338155990 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6336ffd700 1 -- 172.21.15.50:0/2782083241 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324003570 con 0x7f6338155990 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 -- 172.21.15.50:0/2782083241 >> v2:172.21.15.125:3300/0 conn(0x7f63380b13b0 msgr2=0x7f633806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 --2- 172.21.15.50:0/2782083241 >> v2:172.21.15.125:3300/0 conn(0x7f63380b13b0 0x7f633806d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 -- 172.21.15.50:0/2782083241 >> v1:172.21.15.50:6789/0 conn(0x7f6338155990 legacy=0x7f63380b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.691 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 -- 172.21.15.50:0/2782083241 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633806cf00 con 0x7f63380c97d0 2023-12-19T16:11:40.692 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f632ffff700 1 --2- 172.21.15.50:0/2782083241 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 0x7f63380b0e70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63200098e0 tx=0x7f63200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755924be09939989 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:40.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6336ffd700 1 -- 172.21.15.50:0/2782083241 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320011630 con 0x7f63380c97d0 2023-12-19T16:11:40.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6336ffd700 1 -- 172.21.15.50:0/2782083241 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320011790 con 0x7f63380c97d0 2023-12-19T16:11:40.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.686+0000 7f6336ffd700 1 -- 172.21.15.50:0/2782083241 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320011b30 con 0x7f63380c97d0 2023-12-19T16:11:40.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- 172.21.15.50:0/2782083241 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 msgr2=0x7f63380b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- 172.21.15.50:0/2782083241 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 0x7f63380b0e70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63200098e0 tx=0x7f63200091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- 172.21.15.50:0/2782083241 shutdown_connections 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- 172.21.15.50:0/2782083241 >> v2:172.21.15.125:3300/0 conn(0x7f63380b13b0 0x7f633806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- 172.21.15.50:0/2782083241 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63380c97d0 0x7f63380b0e70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- 172.21.15.50:0/2782083241 >> 172.21.15.50:0/2782083241 conn(0x7f6338067bb0 msgr2=0x7f63380add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- 172.21.15.50:0/2782083241 shutdown_connections 2023-12-19T16:11:40.695 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- 172.21.15.50:0/2782083241 wait complete. 2023-12-19T16:11:40.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 Processor -- start 2023-12-19T16:11:40.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- start start 2023-12-19T16:11:40.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63380b13b0 0x7f633814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6338155990 0x7f63381586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.690+0000 7f633cce3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633816d220 con 0x7f63380c97d0 2023-12-19T16:11:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f633cce3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633816c610 con 0x7f63380b13b0 2023-12-19T16:11:40.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f633cce3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633816c8f0 con 0x7f6338155990 2023-12-19T16:11:40.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35130/0 (socket says 172.21.15.50:35130) 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 -- 172.21.15.50:0/810779727 learned_addr learned my addr 172.21.15.50:0/810779727 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f63377fe700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63380b13b0 0x7f633814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f6337fff700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6338155990 0x7f63381586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 -- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6338155990 msgr2=0x7f63381586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6338155990 0x7f63381586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 -- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63380b13b0 msgr2=0x7f633814e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63380b13b0 0x7f633814e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 -- 172.21.15.50:0/810779727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6320009420 con 0x7f63380c97d0 2023-12-19T16:11:40.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632ffff700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f63380c9590 tx=0x7f6320017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320011900 con 0x7f63380c97d0 2023-12-19T16:11:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320011a60 con 0x7f63380c97d0 2023-12-19T16:11:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320023620 con 0x7f63380c97d0 2023-12-19T16:11:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633816cbd0 con 0x7f63380c97d0 2023-12-19T16:11:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.694+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633814ec30 con 0x7f63380c97d0 2023-12-19T16:11:40.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.698+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633816d3d0 con 0x7f63380c97d0 2023-12-19T16:11:40.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.698+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f632002c3b0 con 0x7f63380c97d0 2023-12-19T16:11:40.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.698+0000 7f632f7fe700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 0x7f6318062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:40.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.702+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f632007deb0 con 0x7f63380c97d0 2023-12-19T16:11:40.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.702+0000 7f63377fe700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 0x7f6318062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:40.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.702+0000 7f63377fe700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 0x7f6318062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6328007fd0 tx=0x7f6328009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:40.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.706+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f633816d3d0 con 0x7f63380c97d0 2023-12-19T16:11:40.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.890+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f633804be40 con 0x7f63380c97d0 2023-12-19T16:11:40.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.894+0000 7f632f7fe700 1 -- 172.21.15.50:0/810779727 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+15716 (secure 0 0 0) 0x7f632004e1e0 con 0x7f63380c97d0 2023-12-19T16:11:40.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:40.899 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":198,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:39.914828+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 msgr2=0x7f6318062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 0x7f6318062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6328007fd0 tx=0x7f6328009040 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 msgr2=0x7f6338143aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f63380c9590 tx=0x7f6320017000 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 shutdown_connections 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6318060670 0x7f6318062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6338155990 0x7f63381586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63380c97d0 0x7f6338143aa0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 --2- 172.21.15.50:0/810779727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63380b13b0 0x7f633814e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 >> 172.21.15.50:0/810779727 conn(0x7f6338067bb0 msgr2=0x7f63380adbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 shutdown_connections 2023-12-19T16:11:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:40.898+0000 7f633cce3700 1 -- 172.21.15.50:0/810779727 wait complete. 2023-12-19T16:11:40.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.094+0000 7f03d759e700 1 Processor -- start 2023-12-19T16:11:41.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.094+0000 7f03d759e700 1 -- start start 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.094+0000 7f03d759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 0x7f03d80cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.094+0000 7f03d759e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03d806b750 0x7f03d806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d759e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d806b2c0 con 0x7f03d806df20 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d806a6b0 con 0x7f03d80cad10 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d759e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d806a990 con 0x7f03d806b750 2023-12-19T16:11:41.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03d806b750 0x7f03d806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 0x7f03d80cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d6d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f03d806df20 0x7f03d815bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50464/0 (socket says 172.21.15.50:50464) 2023-12-19T16:11:41.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d6d9d700 1 -- 172.21.15.50:0/3532964656 learned_addr learned my addr 172.21.15.50:0/3532964656 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 -- 172.21.15.50:0/3532964656 >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 msgr2=0x7f03d80cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 --2- 172.21.15.50:0/3532964656 >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 0x7f03d80cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 -- 172.21.15.50:0/3532964656 >> v1:172.21.15.50:6789/0 conn(0x7f03d806df20 legacy=0x7f03d815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 -- 172.21.15.50:0/3532964656 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d806ac70 con 0x7f03d806b750 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d559a700 1 -- 172.21.15.50:0/3532964656 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802858049 0 0) 0x7f03d806b2c0 con 0x7f03d806df20 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d659c700 1 --2- 172.21.15.50:0/3532964656 >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 0x7f03d80cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:41.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d5d9b700 1 --2- 172.21.15.50:0/3532964656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806b750 0x7f03d806d9e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f03d80696f0 tx=0x7f03cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9333784be39e69b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d559a700 1 -- 172.21.15.50:0/3532964656 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc011630 con 0x7f03d806b750 2023-12-19T16:11:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d559a700 1 -- 172.21.15.50:0/3532964656 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc011790 con 0x7f03d806b750 2023-12-19T16:11:41.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.098+0000 7f03d559a700 1 -- 172.21.15.50:0/3532964656 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc011b30 con 0x7f03d806b750 2023-12-19T16:11:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- 172.21.15.50:0/3532964656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806b750 msgr2=0x7f03d806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 --2- 172.21.15.50:0/3532964656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806b750 0x7f03d806d9e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f03d80696f0 tx=0x7f03cc0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- 172.21.15.50:0/3532964656 shutdown_connections 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 --2- 172.21.15.50:0/3532964656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806b750 0x7f03d806d9e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 --2- 172.21.15.50:0/3532964656 >> v2:172.21.15.50:3300/0 conn(0x7f03d80cad10 0x7f03d80cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- 172.21.15.50:0/3532964656 >> 172.21.15.50:0/3532964656 conn(0x7f03d8067bb0 msgr2=0x7f03d80cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- 172.21.15.50:0/3532964656 shutdown_connections 2023-12-19T16:11:41.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- 172.21.15.50:0/3532964656 wait complete. 2023-12-19T16:11:41.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 Processor -- start 2023-12-19T16:11:41.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.102+0000 7f03d759e700 1 -- start start 2023-12-19T16:11:41.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d806b750 0x7f03d8159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806df20 0x7f03d815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815b240 con 0x7f03d80cad10 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815cf70 con 0x7f03d806df20 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815ac80 con 0x7f03d806b750 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35144/0 (socket says 172.21.15.50:35144) 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 -- 172.21.15.50:0/1118593774 learned_addr learned my addr 172.21.15.50:0/1118593774 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d659c700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d806b750 0x7f03d8159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d5d9b700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806df20 0x7f03d815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 -- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d806b750 msgr2=0x7f03d8159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d806b750 0x7f03d8159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 -- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806df20 msgr2=0x7f03d815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806df20 0x7f03d815a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 -- 172.21.15.50:0/1118593774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8009420 con 0x7f03d80cad10 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d6d9d700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f03d000ccf0 tx=0x7f03d0001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000a870 con 0x7f03d80cad10 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d000a9d0 con 0x7f03d80cad10 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc009420 con 0x7f03d80cad10 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000ad70 con 0x7f03d80cad10 2023-12-19T16:11:41.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.106+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d815cc00 con 0x7f03d80cad10 2023-12-19T16:11:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.110+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d815d100 con 0x7f03d80cad10 2023-12-19T16:11:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.110+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f03d0007300 con 0x7f03d80cad10 2023-12-19T16:11:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.114+0000 7f03c67fc700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 0x7f03b8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.114+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f03d007ce50 con 0x7f03d80cad10 2023-12-19T16:11:41.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.114+0000 7f03d659c700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 0x7f03b8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.114+0000 7f03d659c700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 0x7f03b8062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f03d8157a70 tx=0x7f03c80091a0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.118+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f03d004d180 con 0x7f03d80cad10 2023-12-19T16:11:41.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.298+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d804be40 con 0x7f03d80cad10 2023-12-19T16:11:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03c67fc700 1 -- 172.21.15.50:0/1118593774 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+15703 (secure 0 0 0) 0x7f03d00507a0 con 0x7f03d80cad10 2023-12-19T16:11:41.307 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:41.307 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":199,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:41.202942+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 msgr2=0x7f03b8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 0x7f03b8062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f03d8157a70 tx=0x7f03c80091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 msgr2=0x7f03d8168c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f03d000ccf0 tx=0x7f03d0001a20 comp rx=0 tx=0).stop 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 shutdown_connections 2023-12-19T16:11:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03b8060670 0x7f03b8062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d806df20 0x7f03d815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d80cad10 0x7f03d8168c70 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 --2- 172.21.15.50:0/1118593774 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d806b750 0x7f03d8159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 >> 172.21.15.50:0/1118593774 conn(0x7f03d8067bb0 msgr2=0x7f03d80b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 shutdown_connections 2023-12-19T16:11:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.302+0000 7f03d759e700 1 -- 172.21.15.50:0/1118593774 wait complete. 2023-12-19T16:11:41.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T16:11:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.498+0000 7f8d69323700 1 Processor -- start 2023-12-19T16:11:41.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.498+0000 7f8d69323700 1 -- start start 2023-12-19T16:11:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d69323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d64157fc0 0x7f8d64158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d69323700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d6406d050 0x7f8d6406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d69323700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640c7820 con 0x7f8d641547a0 2023-12-19T16:11:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d69323700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640ade40 con 0x7f8d6406d050 2023-12-19T16:11:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d69323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6406da30 con 0x7f8d64157fc0 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d63fff700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8d641547a0 0x7f8d640ad730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:51992/0 (socket says 172.21.15.50:51992) 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d63fff700 1 -- 172.21.15.50:0/2648313095 learned_addr learned my addr 172.21.15.50:0/2648313095 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d68b22700 1 --2- 172.21.15.50:0/2648313095 >> v2:172.21.15.125:3300/0 conn(0x7f8d6406d050 0x7f8d6406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.502+0000 7f8d5b7fe700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d64157fc0 0x7f8d64158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d637fe700 1 -- 172.21.15.50:0/2648313095 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260074395 0 0) 0x7f8d640c7820 con 0x7f8d641547a0 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d637fe700 1 -- 172.21.15.50:0/2648313095 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50003570 con 0x7f8d641547a0 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d68b22700 1 -- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d64157fc0 msgr2=0x7f8d64158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d68b22700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d64157fc0 0x7f8d64158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d68b22700 1 -- 172.21.15.50:0/2648313095 >> v1:172.21.15.125:6789/0 conn(0x7f8d641547a0 legacy=0x7f8d640ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d68b22700 1 -- 172.21.15.50:0/2648313095 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640c7210 con 0x7f8d6406d050 2023-12-19T16:11:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d68b22700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d6406d050 0x7f8d6406d4f0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0098e0 tx=0x7f8d5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b458f8dfa9151549 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d637fe700 1 -- 172.21.15.50:0/2648313095 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c011630 con 0x7f8d6406d050 2023-12-19T16:11:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d637fe700 1 -- 172.21.15.50:0/2648313095 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c011790 con 0x7f8d6406d050 2023-12-19T16:11:41.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d637fe700 1 -- 172.21.15.50:0/2648313095 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c011b30 con 0x7f8d6406d050 2023-12-19T16:11:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 -- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d6406d050 msgr2=0x7f8d6406d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d6406d050 0x7f8d6406d4f0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0098e0 tx=0x7f8d5c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 -- 172.21.15.50:0/2648313095 shutdown_connections 2023-12-19T16:11:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d6406d050 0x7f8d6406d4f0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 --2- 172.21.15.50:0/2648313095 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d64157fc0 0x7f8d64158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 -- 172.21.15.50:0/2648313095 >> 172.21.15.50:0/2648313095 conn(0x7f8d640ccb30 msgr2=0x7f8d640ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:41.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 -- 172.21.15.50:0/2648313095 shutdown_connections 2023-12-19T16:11:41.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.506+0000 7f8d69323700 1 -- 172.21.15.50:0/2648313095 wait complete. 2023-12-19T16:11:41.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 Processor -- start 2023-12-19T16:11:41.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 -- start start 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d6406d050 0x7f8d6413e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d64157fc0 0x7f8d640b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64157b60 con 0x7f8d641547a0 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64152030 con 0x7f8d64157fc0 2023-12-19T16:11:41.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d69323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64152310 con 0x7f8d6406d050 2023-12-19T16:11:41.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d5b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d5b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35158/0 (socket says 172.21.15.50:35158) 2023-12-19T16:11:41.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d5b7fe700 1 -- 172.21.15.50:0/4004514589 learned_addr learned my addr 172.21.15.50:0/4004514589 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:41.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d68b22700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d64157fc0 0x7f8d640b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.510+0000 7f8d63fff700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d6406d050 0x7f8d6413e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 -- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d6406d050 msgr2=0x7f8d6413e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d6406d050 0x7f8d6413e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 -- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d64157fc0 msgr2=0x7f8d640b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d64157fc0 0x7f8d640b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:41.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 -- 172.21.15.50:0/4004514589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c009420 con 0x7f8d641547a0 2023-12-19T16:11:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d5b7fe700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00e930 tx=0x7f8d4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c013680 con 0x7f8d641547a0 2023-12-19T16:11:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c0137e0 con 0x7f8d641547a0 2023-12-19T16:11:41.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c013b80 con 0x7f8d641547a0 2023-12-19T16:11:41.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d641525f0 con 0x7f8d641547a0 2023-12-19T16:11:41.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d64151ce0 con 0x7f8d641547a0 2023-12-19T16:11:41.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.514+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6404be40 con 0x7f8d641547a0 2023-12-19T16:11:41.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.518+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8d4c00c3e0 con 0x7f8d641547a0 2023-12-19T16:11:41.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.518+0000 7f8d60ff9700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 0x7f8d44062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:41.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.518+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f8d4c07cb90 con 0x7f8d641547a0 2023-12-19T16:11:41.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.522+0000 7f8d63fff700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 0x7f8d44062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:41.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.522+0000 7f8d63fff700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 0x7f8d44062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005c50 tx=0x7f8d54009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:41.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.526+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d4c04ce40 con 0x7f8d641547a0 2023-12-19T16:11:41.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:41.734+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8d640ae030 con 0x7f8d641547a0 2023-12-19T16:11:42.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.498+0000 7f8d60ff9700 1 -- 172.21.15.50:0/4004514589 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8d4c0170b0 con 0x7f8d641547a0 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 msgr2=0x7f8d44062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 0x7f8d44062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005c50 tx=0x7f8d54009040 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 msgr2=0x7f8d64157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00e930 tx=0x7f8d4c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 shutdown_connections 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8d44060670 0x7f8d44062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d64157fc0 0x7f8d640b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d641547a0 0x7f8d64157200 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 --2- 172.21.15.50:0/4004514589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8d6406d050 0x7f8d6413e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 >> 172.21.15.50:0/4004514589 conn(0x7f8d640ccb30 msgr2=0x7f8d640b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 shutdown_connections 2023-12-19T16:11:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.502+0000 7f8d69323700 1 -- 172.21.15.50:0/4004514589 wait complete. 2023-12-19T16:11:42.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T16:11:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 Processor -- start 2023-12-19T16:11:42.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 -- start start 2023-12-19T16:11:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac7c0c6d20 0x7fac7c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c06a7a0 con 0x7fac7c15aa30 2023-12-19T16:11:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7c0c7630 con 0x7fac7c0c6d20 2023-12-19T16:11:42.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac827c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c06d4b0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac7bfff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac7c0c6d20 0x7fac7c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.694+0000 7fac73fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:35980/0 (socket says 172.21.15.50:35980) 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 -- 172.21.15.50:0/3621494285 learned_addr learned my addr 172.21.15.50:0/3621494285 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac80d62700 1 --1- 172.21.15.50:0/3621494285 >> v1:172.21.15.125:6789/0 conn(0x7fac7c15aa30 0x7fac7c15ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:52000/0 (socket says 172.21.15.50:52000) 2023-12-19T16:11:42.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 -- 172.21.15.50:0/3621494285 >> v2:172.21.15.125:3300/0 conn(0x7fac7c0c6d20 msgr2=0x7fac7c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 --2- 172.21.15.50:0/3621494285 >> v2:172.21.15.125:3300/0 conn(0x7fac7c0c6d20 0x7fac7c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 -- 172.21.15.50:0/3621494285 >> v1:172.21.15.125:6789/0 conn(0x7fac7c15aa30 legacy=0x7fac7c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 -- 172.21.15.50:0/3621494285 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c06d790 con 0x7fac7c06c7c0 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac73fff700 1 --2- 172.21.15.50:0/3621494285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fac640099b0 tx=0x7fac64009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45715d3915b45fe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac7b7fe700 1 -- 172.21.15.50:0/3621494285 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac64009ea0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac7b7fe700 1 -- 172.21.15.50:0/3621494285 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6400b860 con 0x7fac7c06c7c0 2023-12-19T16:11:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac7b7fe700 1 -- 172.21.15.50:0/3621494285 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6400bc00 con 0x7fac7c06c7c0 2023-12-19T16:11:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 -- 172.21.15.50:0/3621494285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 msgr2=0x7fac7c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 --2- 172.21.15.50:0/3621494285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fac640099b0 tx=0x7fac64009250 comp rx=0 tx=0).stop 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 -- 172.21.15.50:0/3621494285 shutdown_connections 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 --2- 172.21.15.50:0/3621494285 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c06c7c0 0x7fac7c06cc60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 --2- 172.21.15.50:0/3621494285 >> v2:172.21.15.125:3300/0 conn(0x7fac7c0c6d20 0x7fac7c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 -- 172.21.15.50:0/3621494285 >> 172.21.15.50:0/3621494285 conn(0x7fac7c067bb0 msgr2=0x7fac7c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 -- 172.21.15.50:0/3621494285 shutdown_connections 2023-12-19T16:11:42.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.698+0000 7fac827c5700 1 -- 172.21.15.50:0/3621494285 wait complete. 2023-12-19T16:11:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.702+0000 7fac827c5700 1 Processor -- start 2023-12-19T16:11:42.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.702+0000 7fac827c5700 1 -- start start 2023-12-19T16:11:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c0c6d20 0x7fac7c0a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac7c15aa30 0x7fac7c0a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0ade30 con 0x7fac7c06c7c0 2023-12-19T16:11:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0a8850 con 0x7fac7c15aa30 2023-12-19T16:11:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0a8e10 con 0x7fac7c0c6d20 2023-12-19T16:11:42.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35170/0 (socket says 172.21.15.50:35170) 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 -- 172.21.15.50:0/3289731202 learned_addr learned my addr 172.21.15.50:0/3289731202 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac73fff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c0c6d20 0x7fac7c0a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac80d62700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac7c15aa30 0x7fac7c0a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 -- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c0c6d20 msgr2=0x7fac7c0a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c0c6d20 0x7fac7c0a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 -- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac7c15aa30 msgr2=0x7fac7c0a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac7c15aa30 0x7fac7c0a97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:42.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 -- 172.21.15.50:0/3289731202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac640094f0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac7bfff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00ecb0 tx=0x7fac6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00c850 con 0x7fac7c06c7c0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6c00c9b0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c0aca40 con 0x7fac7c06c7c0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00cd70 con 0x7fac7c06c7c0 2023-12-19T16:11:42.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.706+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c0a8aa0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.710+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c071120 con 0x7fac7c06c7c0 2023-12-19T16:11:42.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.710+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fac6c013680 con 0x7fac7c06c7c0 2023-12-19T16:11:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.710+0000 7fac78ff9700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:42.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.710+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fac6c07ccc0 con 0x7fac7c06c7c0 2023-12-19T16:11:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.714+0000 7fac73fff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:42.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.714+0000 7fac73fff700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 0x7fac5c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac640099b0 tx=0x7fac64017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:42.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.718+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac6c083080 con 0x7fac7c06c7c0 2023-12-19T16:11:42.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:42.930+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fac7c0ae6f0 con 0x7fac7c06c7c0 2023-12-19T16:11:43.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac78ff9700 1 -- 172.21.15.50:0/3289731202 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) v1 ==== 122+0+0 (secure 0 0 0) 0x7fac6c04cf70 con 0x7fac7c06c7c0 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 msgr2=0x7fac5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 0x7fac5c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac640099b0 tx=0x7fac64017000 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 msgr2=0x7fac7c1537d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00ecb0 tx=0x7fac6c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 shutdown_connections 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac7c15aa30 0x7fac7c0a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fac7c0c6d20 0x7fac7c0a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 --2- 172.21.15.50:0/3289731202 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c06c7c0 0x7fac7c1537d0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.522+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 >> 172.21.15.50:0/3289731202 conn(0x7fac7c067bb0 msgr2=0x7fac7c1592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.526+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 shutdown_connections 2023-12-19T16:11:43.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.526+0000 7fac827c5700 1 -- 172.21.15.50:0/3289731202 wait complete. 2023-12-19T16:11:43.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:43.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.714+0000 7fbdc359e700 1 Processor -- start 2023-12-19T16:11:43.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc359e700 1 -- start start 2023-12-19T16:11:43.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdc406a540 0x7fbdc406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc40ca5a0 con 0x7fbdc40cc470 2023-12-19T16:11:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc359e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc406dac0 con 0x7fbdc4158e70 2023-12-19T16:11:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc406d220 con 0x7fbdc406a540 2023-12-19T16:11:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc2d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbdc40cc470 0x7fbdc40cc8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50470/0 (socket says 172.21.15.50:50470) 2023-12-19T16:11:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc2d9d700 1 -- 172.21.15.50:0/752000829 learned_addr learned my addr 172.21.15.50:0/752000829 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc1d9b700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdc406a540 0x7fbdc406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989689588 0 0) 0x7fbdc40ca5a0 con 0x7fbdc40cc470 2023-12-19T16:11:43.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdac003570 con 0x7fbdc40cc470 2023-12-19T16:11:43.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656271225 0 0) 0x7fbdc406dac0 con 0x7fbdc4158e70 2023-12-19T16:11:43.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc40ca5a0 con 0x7fbdc4158e70 2023-12-19T16:11:43.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.718+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2177837744 0 0) 0x7fbdac003570 con 0x7fbdc40cc470 2023-12-19T16:11:43.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdc406a540 msgr2=0x7fbdc406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:43.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdc406a540 0x7fbdc406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 >> v1:172.21.15.125:6789/0 conn(0x7fbdc4158e70 legacy=0x7fbdc40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc406d500 con 0x7fbdc40cc470 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7fbdbc0017f0 con 0x7fbdc40cc470 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 >> v1:172.21.15.50:6789/0 conn(0x7fbdc40cc470 legacy=0x7fbdc40cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdac003c20 0x7fbdac004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac004810 0x7fbdac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdac0051c0 0x7fbdac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc406d220 con 0x7fbdac0051c0 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac00c1b0 con 0x7fbdac003c20 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac00c370 con 0x7fbdac004810 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdac0051c0 0x7fbdac00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc1d9b700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac004810 0x7fbdac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc259c700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdac003c20 0x7fbdac004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 -- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac004810 msgr2=0x7fbdac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac004810 0x7fbdac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 -- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdac003c20 msgr2=0x7fbdac004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdac003c20 0x7fbdac004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 -- 172.21.15.50:0/752000829 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac003570 con 0x7fbdac0051c0 2023-12-19T16:11:43.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc259c700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdac003c20 0x7fbdac004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:43.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdac0051c0 0x7fbdac00ba60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00a8f0 tx=0x7fbdbc002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e273f148cd863e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc0070d0 con 0x7fbdac0051c0 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc007250 con 0x7fbdac0051c0 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc159a700 1 -- 172.21.15.50:0/752000829 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc007610 con 0x7fbdac0051c0 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc359e700 1 -- 172.21.15.50:0/752000829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdac0051c0 msgr2=0x7fbdac00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc359e700 1 --2- 172.21.15.50:0/752000829 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdac0051c0 0x7fbdac00ba60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00a8f0 tx=0x7fbdbc002190 comp rx=0 tx=0).stop 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc2d9d700 1 -- 172.21.15.50:0/752000829 reap_dead start 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc359e700 1 -- 172.21.15.50:0/752000829 shutdown_connections 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.722+0000 7fbdc359e700 1 -- 172.21.15.50:0/752000829 >> 172.21.15.50:0/752000829 conn(0x7fbdc40670e0 msgr2=0x7fbdc40674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- 172.21.15.50:0/752000829 shutdown_connections 2023-12-19T16:11:43.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- 172.21.15.50:0/752000829 wait complete. 2023-12-19T16:11:43.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 Processor -- start 2023-12-19T16:11:43.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- start start 2023-12-19T16:11:43.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac0051c0 0x7fbdc41565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc40b3f80 0x7fbdc415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc415ae30 con 0x7fbdc40c9300 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc415a500 con 0x7fbdc40b3f80 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.726+0000 7fbdc359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc415a7e0 con 0x7fbdac0051c0 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35192/0 (socket says 172.21.15.50:35192) 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 -- 172.21.15.50:0/1904880684 learned_addr learned my addr 172.21.15.50:0/1904880684 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc259c700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac0051c0 0x7fbdc41565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc2d9d700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc40b3f80 0x7fbdc415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 -- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac0051c0 msgr2=0x7fbdc41565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac0051c0 0x7fbdc41565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 -- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc40b3f80 msgr2=0x7fbdc415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc40b3f80 0x7fbdc415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 -- 172.21.15.50:0/1904880684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc415aac0 con 0x7fbdc40c9300 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc1d9b700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80119a0 tx=0x7fbdb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800f830 con 0x7fbdc40c9300 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb800f990 con 0x7fbdc40c9300 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc4156b10 con 0x7fbdc40c9300 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800fd30 con 0x7fbdc40c9300 2023-12-19T16:11:43.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc416cea0 con 0x7fbdc40c9300 2023-12-19T16:11:43.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.730+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc40b4350 con 0x7fbdc40c9300 2023-12-19T16:11:43.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.734+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbdb8014650 con 0x7fbdc40c9300 2023-12-19T16:11:43.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.734+0000 7fbdb27fc700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 0x7fbda4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:43.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.734+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fbdb8018030 con 0x7fbdc40c9300 2023-12-19T16:11:43.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.734+0000 7fbdc259c700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 0x7fbda4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:43.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.734+0000 7fbdc259c700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 0x7fbda4062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007fd0 tx=0x7fbdb4009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:43.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.742+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbdb804d680 con 0x7fbdc40c9300 2023-12-19T16:11:43.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.950+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdc404be40 con 0x7fbdc40c9300 2023-12-19T16:11:43.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.954+0000 7fbdb27fc700 1 -- 172.21.15.50:0/1904880684 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fbdb8050ca0 con 0x7fbdc40c9300 2023-12-19T16:11:43.958 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:43.959 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:11:43.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 msgr2=0x7fbda4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 0x7fbda4062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007fd0 tx=0x7fbdb4009040 comp rx=0 tx=0).stop 2023-12-19T16:11:43.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 msgr2=0x7fbdc40b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80119a0 tx=0x7fbdb80091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 shutdown_connections 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbda4060670 0x7fbda4062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbdc40b3f80 0x7fbdc415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbdc40c9300 0x7fbdc40b3a40 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 --2- 172.21.15.50:0/1904880684 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbdac0051c0 0x7fbdc41565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 >> 172.21.15.50:0/1904880684 conn(0x7fbdc40670e0 msgr2=0x7fbdc41584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 shutdown_connections 2023-12-19T16:11:43.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:43.958+0000 7fbdc359e700 1 -- 172.21.15.50:0/1904880684 wait complete. 2023-12-19T16:11:43.965 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:11:43.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:11:44.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.142+0000 7f03ba24c700 1 Processor -- start 2023-12-19T16:11:44.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 -- start start 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03b4158d60 0x7f03b415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b406b7a0 0x7f03b40c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03b40c51b0 0x7f03b406a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40c5610 con 0x7f03b4158d60 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b406c7c0 con 0x7f03b40c51b0 2023-12-19T16:11:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03ba24c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b406caa0 con 0x7f03b406b7a0 2023-12-19T16:11:44.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b924a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03b4158d60 0x7f03b415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b924a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03b4158d60 0x7f03b415c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35200/0 (socket says 172.21.15.50:35200) 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b924a700 1 -- 172.21.15.50:0/3337854154 learned_addr learned my addr 172.21.15.50:0/3337854154 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b9a4b700 1 --2- 172.21.15.50:0/3337854154 >> v2:172.21.15.125:3300/0 conn(0x7f03b40c51b0 0x7f03b406a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b8a49700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b406b7a0 0x7f03b40c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b924a700 1 -- 172.21.15.50:0/3337854154 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b406b7a0 msgr2=0x7f03b40c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.146+0000 7f03b924a700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b406b7a0 0x7f03b40c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03b924a700 1 -- 172.21.15.50:0/3337854154 >> v2:172.21.15.125:3300/0 conn(0x7f03b40c51b0 msgr2=0x7f03b406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03b924a700 1 --2- 172.21.15.50:0/3337854154 >> v2:172.21.15.125:3300/0 conn(0x7f03b40c51b0 0x7f03b406a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03b924a700 1 -- 172.21.15.50:0/3337854154 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b415ce60 con 0x7f03b4158d60 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03b924a700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b4158d60 0x7f03b415c920 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c5b50 tx=0x7f03b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d664a6b57a8f292 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03abfff700 1 -- 172.21.15.50:0/3337854154 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0009e70 con 0x7f03b4158d60 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03abfff700 1 -- 172.21.15.50:0/3337854154 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b000b860 con 0x7f03b4158d60 2023-12-19T16:11:44.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03abfff700 1 -- 172.21.15.50:0/3337854154 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000bc20 con 0x7f03b4158d60 2023-12-19T16:11:44.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 -- 172.21.15.50:0/3337854154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b4158d60 msgr2=0x7f03b415c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b4158d60 0x7f03b415c920 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c5b50 tx=0x7f03b0009250 comp rx=0 tx=0).stop 2023-12-19T16:11:44.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 -- 172.21.15.50:0/3337854154 shutdown_connections 2023-12-19T16:11:44.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 --2- 172.21.15.50:0/3337854154 >> v2:172.21.15.125:3300/0 conn(0x7f03b40c51b0 0x7f03b406a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b406b7a0 0x7f03b40c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 --2- 172.21.15.50:0/3337854154 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b4158d60 0x7f03b415c920 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 -- 172.21.15.50:0/3337854154 >> 172.21.15.50:0/3337854154 conn(0x7f03b4067bb0 msgr2=0x7f03b40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:44.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 -- 172.21.15.50:0/3337854154 shutdown_connections 2023-12-19T16:11:44.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.150+0000 7f03ba24c700 1 -- 172.21.15.50:0/3337854154 wait complete. 2023-12-19T16:11:44.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 Processor -- start 2023-12-19T16:11:44.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 -- start start 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03b406b7a0 0x7f03b4169c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b4158d60 0x7f03b40cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b415c1e0 con 0x7f03b40c51b0 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b416a190 con 0x7f03b406b7a0 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03ba24c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b415b660 con 0x7f03b4158d60 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35216/0 (socket says 172.21.15.50:35216) 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 -- 172.21.15.50:0/871741329 learned_addr learned my addr 172.21.15.50:0/871741329 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b9a4b700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b4158d60 0x7f03b40cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b924a700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03b406b7a0 0x7f03b4169c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 -- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b4158d60 msgr2=0x7f03b40cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b4158d60 0x7f03b40cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 -- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03b406b7a0 msgr2=0x7f03b4169c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03b406b7a0 0x7f03b4169c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.154+0000 7f03b8a49700 1 -- 172.21.15.50:0/871741329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b00094f0 con 0x7f03b40c51b0 2023-12-19T16:11:44.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03b8a49700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d890 tx=0x7f03a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a400b600 con 0x7f03b40c51b0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a400b760 con 0x7f03b40c51b0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a400bae0 con 0x7f03b40c51b0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b415b940 con 0x7f03b40c51b0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b415bf00 con 0x7f03b40c51b0 2023-12-19T16:11:44.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b404be40 con 0x7f03b40c51b0 2023-12-19T16:11:44.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.158+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f03a401e460 con 0x7f03b40c51b0 2023-12-19T16:11:44.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.162+0000 7f03a97fa700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 0x7f0398062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.162+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f03a407d1e0 con 0x7f03b40c51b0 2023-12-19T16:11:44.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.162+0000 7f03b924a700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 0x7f0398062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.162+0000 7f03b924a700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 0x7f0398062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c5b50 tx=0x7f03b0017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.170+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f03a404d530 con 0x7f03b40c51b0 2023-12-19T16:11:44.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.338+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b415d130 con 0x7f03b40c51b0 2023-12-19T16:11:44.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.342+0000 7f03a97fa700 1 -- 172.21.15.50:0/871741329 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+15703 (secure 0 0 0) 0x7f03a4050b50 con 0x7f03b40c51b0 2023-12-19T16:11:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:44.348 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":199,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:11:41.202942+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:11:44.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 msgr2=0x7f0398062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 0x7f0398062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f03b40c5b50 tx=0x7f03b0017000 comp rx=0 tx=0).stop 2023-12-19T16:11:44.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 msgr2=0x7f03b4151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d890 tx=0x7f03a4008040 comp rx=0 tx=0).stop 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 shutdown_connections 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0398060670 0x7f0398062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03b4158d60 0x7f03b40cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03b40c51b0 0x7f03b4151690 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 --2- 172.21.15.50:0/871741329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03b406b7a0 0x7f03b4169c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 >> 172.21.15.50:0/871741329 conn(0x7f03b4067bb0 msgr2=0x7f03b41527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 shutdown_connections 2023-12-19T16:11:44.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.346+0000 7f03ba24c700 1 -- 172.21.15.50:0/871741329 wait complete. 2023-12-19T16:11:44.373 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T16:11:44.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:11:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.534+0000 7f15a1523700 1 Processor -- start 2023-12-19T16:11:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.534+0000 7f15a1523700 1 -- start start 2023-12-19T16:11:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a1523700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a1523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 0x7f15940549d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a1523700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1594054540 con 0x7f1594051410 2023-12-19T16:11:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a1523700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1594054210 con 0x7f15940dc830 2023-12-19T16:11:44.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a1523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940090d0 con 0x7f15940e3250 2023-12-19T16:11:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f159bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f159bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35226/0 (socket says 172.21.15.50:35226) 2023-12-19T16:11:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f159bfff700 1 -- 172.21.15.50:0/3379326519 learned_addr learned my addr 172.21.15.50:0/3379326519 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:44.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 --2- 172.21.15.50:0/3379326519 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 0x7f15940549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 -- 172.21.15.50:0/3379326519 >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 msgr2=0x7f1594008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 --2- 172.21.15.50:0/3379326519 >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 -- 172.21.15.50:0/3379326519 >> v1:172.21.15.50:6789/0 conn(0x7f1594051410 legacy=0x7f15940e2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 -- 172.21.15.50:0/3379326519 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940093b0 con 0x7f15940e3250 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f159bfff700 1 --2- 172.21.15.50:0/3379326519 >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.538+0000 7f15a0d22700 1 --2- 172.21.15.50:0/3379326519 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 0x7f15940549d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f159c071030 tx=0x7f159c066cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ffa709b660481e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f159affd700 1 -- 172.21.15.50:0/3379326519 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c04be40 con 0x7f15940e3250 2023-12-19T16:11:44.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f159affd700 1 -- 172.21.15.50:0/3379326519 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c0008d0 con 0x7f15940e3250 2023-12-19T16:11:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f159affd700 1 -- 172.21.15.50:0/3379326519 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c070460 con 0x7f15940e3250 2023-12-19T16:11:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 -- 172.21.15.50:0/3379326519 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 msgr2=0x7f15940549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 --2- 172.21.15.50:0/3379326519 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 0x7f15940549d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f159c071030 tx=0x7f159c066cd0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 -- 172.21.15.50:0/3379326519 shutdown_connections 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 --2- 172.21.15.50:0/3379326519 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f15940e3250 0x7f15940549d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 --2- 172.21.15.50:0/3379326519 >> v2:172.21.15.50:3300/0 conn(0x7f15940dc830 0x7f1594008780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 -- 172.21.15.50:0/3379326519 >> 172.21.15.50:0/3379326519 conn(0x7f1594052980 msgr2=0x7f1594052d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 -- 172.21.15.50:0/3379326519 shutdown_connections 2023-12-19T16:11:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.542+0000 7f15a1523700 1 -- 172.21.15.50:0/3379326519 wait complete. 2023-12-19T16:11:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 Processor -- start 2023-12-19T16:11:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 -- start start 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15940dc830 0x7f15940dba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15940e3250 0x7f159403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15940def00 con 0x7f15940e3250 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159404be40 con 0x7f15940dc830 2023-12-19T16:11:44.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a1523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940394b0 con 0x7f1594051410 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f159bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f15a0d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15940e3250 0x7f159403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:36042/0 (socket says 172.21.15.50:36042) 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 -- 172.21.15.50:0/1761963192 learned_addr learned my addr 172.21.15.50:0/1761963192 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.546+0000 7f159b7fe700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15940dc830 0x7f15940dba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 -- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15940dc830 msgr2=0x7f15940dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15940dc830 0x7f15940dba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 -- 172.21.15.50:0/1761963192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15940e3250 msgr2=0x7f159403b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15940e3250 0x7f159403b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 -- 172.21.15.50:0/1761963192 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590009480 con 0x7f1594051410 2023-12-19T16:11:44.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f159bfff700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f159000ecb0 tx=0x7f1590009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590005a00 con 0x7f1594051410 2023-12-19T16:11:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1590005b60 con 0x7f1594051410 2023-12-19T16:11:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15940dec20 con 0x7f1594051410 2023-12-19T16:11:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159000c850 con 0x7f1594051410 2023-12-19T16:11:44.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159404bad0 con 0x7f1594051410 2023-12-19T16:11:44.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.550+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1594005950 con 0x7f1594051410 2023-12-19T16:11:44.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.554+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f159000cc00 con 0x7f1594051410 2023-12-19T16:11:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.554+0000 7f1587fff700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 0x7f1580062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.554+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f159007cfe0 con 0x7f1594051410 2023-12-19T16:11:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.554+0000 7f159b7fe700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 0x7f1580062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:44.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.554+0000 7f159b7fe700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 0x7f1580062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f159404cf10 tx=0x7f1588007ec0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:44.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.562+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f159004d290 con 0x7f1594051410 2023-12-19T16:11:44.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.738+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f159404bf80 con 0x7f1580060670 2023-12-19T16:11:44.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.758+0000 7f1587fff700 1 -- 172.21.15.50:0/1761963192 <== mgr.9678 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286654 (secure 0 0 0) 0x7f1588014450 con 0x7f1580060670 2023-12-19T16:11:44.766 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 msgr2=0x7f1580062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 0x7f1580062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f159404cf10 tx=0x7f1588007ec0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 msgr2=0x7f159404c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f159000ecb0 tx=0x7f1590009220 comp rx=0 tx=0).stop 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 shutdown_connections 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1580060670 0x7f1580062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15940dc830 0x7f15940dba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15940e3250 0x7f159403b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 --2- 172.21.15.50:0/1761963192 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1594051410 0x7f159404c2d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:44.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 >> 172.21.15.50:0/1761963192 conn(0x7f1594052980 msgr2=0x7f1594055250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:44.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 shutdown_connections 2023-12-19T16:11:44.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.766+0000 7f15a1523700 1 -- 172.21.15.50:0/1761963192 wait complete. 2023-12-19T16:11:44.773 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:11:44.793 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T16:11:44.019926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460344,"kb_used_data":20600,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752514376,"statfs":{"total":773094113280,"available":770574721024,"internally_reserved":0,"allocated":21094400,"data_stored":11328486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.574872"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243354+0000","last_change":"2023-12-19T16:11:40.237609+0000","last_active":"2023-12-19T16:11:40.243354+0000","last_peered":"2023-12-19T16:11:40.243354+0000","last_clean":"2023-12-19T16:11:40.243354+0000","last_became_active":"2023-12-19T16:11:40.233513+0000","last_became_peered":"2023-12-19T16:11:40.233513+0000","last_unstale":"2023-12-19T16:11:40.243354+0000","last_undegraded":"2023-12-19T16:11:40.243354+0000","last_fullsized":"2023-12-19T16:11:40.243354+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:13.538074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.236119+0000","last_change":"2023-12-19T16:11:40.236119+0000","last_active":"2023-12-19T16:11:40.236119+0000","last_peered":"2023-12-19T16:11:40.236119+0000","last_clean":"2023-12-19T16:11:40.236119+0000","last_became_active":"2023-12-19T16:11:40.235845+0000","last_became_peered":"2023-12-19T16:11:40.235845+0000","last_unstale":"2023-12-19T16:11:40.236119+0000","last_undegraded":"2023-12-19T16:11:40.236119+0000","last_fullsized":"2023-12-19T16:11:40.236119+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:10.177568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243472+0000","last_change":"2023-12-19T16:11:40.239764+0000","last_active":"2023-12-19T16:11:40.243472+0000","last_peered":"2023-12-19T16:11:40.243472+0000","last_clean":"2023-12-19T16:11:40.243472+0000","last_became_active":"2023-12-19T16:11:40.239436+0000","last_became_peered":"2023-12-19T16:11:40.239436+0000","last_unstale":"2023-12-19T16:11:40.243472+0000","last_undegraded":"2023-12-19T16:11:40.243472+0000","last_fullsized":"2023-12-19T16:11:40.243472+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:02:07.204644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243545+0000","last_change":"2023-12-19T16:11:40.241007+0000","last_active":"2023-12-19T16:11:40.243545+0000","last_peered":"2023-12-19T16:11:40.243545+0000","last_clean":"2023-12-19T16:11:40.243545+0000","last_became_active":"2023-12-19T16:11:40.239049+0000","last_became_peered":"2023-12-19T16:11:40.239049+0000","last_unstale":"2023-12-19T16:11:40.243545+0000","last_undegraded":"2023-12-19T16:11:40.243545+0000","last_fullsized":"2023-12-19T16:11:40.243545+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238158+0000","last_change":"2023-12-19T16:11:40.238158+0000","last_active":"2023-12-19T16:11:40.238158+0000","last_peered":"2023-12-19T16:11:40.238158+0000","last_clean":"2023-12-19T16:11:40.238158+0000","last_became_active":"2023-12-19T16:11:40.233495+0000","last_became_peered":"2023-12-19T16:11:40.233495+0000","last_unstale":"2023-12-19T16:11:40.238158+0000","last_undegraded":"2023-12-19T16:11:40.238158+0000","last_fullsized":"2023-12-19T16:11:40.238158+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:43.978826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240234+0000","last_change":"2023-12-19T16:11:40.240234+0000","last_active":"2023-12-19T16:11:40.240234+0000","last_peered":"2023-12-19T16:11:40.240234+0000","last_clean":"2023-12-19T16:11:40.240234+0000","last_became_active":"2023-12-19T16:11:40.239654+0000","last_became_peered":"2023-12-19T16:11:40.239654+0000","last_unstale":"2023-12-19T16:11:40.240234+0000","last_undegraded":"2023-12-19T16:11:40.240234+0000","last_fullsized":"2023-12-19T16:11:40.240234+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238540+0000","last_change":"2023-12-19T16:11:40.232578+0000","last_active":"2023-12-19T16:11:40.238540+0000","last_peered":"2023-12-19T16:11:40.238540+0000","last_clean":"2023-12-19T16:11:40.238540+0000","last_became_active":"2023-12-19T16:11:40.232277+0000","last_became_peered":"2023-12-19T16:11:40.232277+0000","last_unstale":"2023-12-19T16:11:40.238540+0000","last_undegraded":"2023-12-19T16:11:40.238540+0000","last_fullsized":"2023-12-19T16:11:40.238540+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:39.270097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237990+0000","last_change":"2023-12-19T16:11:40.228738+0000","last_active":"2023-12-19T16:11:40.237990+0000","last_peered":"2023-12-19T16:11:40.237990+0000","last_clean":"2023-12-19T16:11:40.237990+0000","last_became_active":"2023-12-19T16:11:40.228429+0000","last_became_peered":"2023-12-19T16:11:40.228429+0000","last_unstale":"2023-12-19T16:11:40.237990+0000","last_undegraded":"2023-12-19T16:11:40.237990+0000","last_fullsized":"2023-12-19T16:11:40.237990+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:10.177568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237905+0000","last_change":"2023-12-19T16:11:40.226675+0000","last_active":"2023-12-19T16:11:40.237905+0000","last_peered":"2023-12-19T16:11:40.237905+0000","last_clean":"2023-12-19T16:11:40.237905+0000","last_became_active":"2023-12-19T16:11:40.226211+0000","last_became_peered":"2023-12-19T16:11:40.226211+0000","last_unstale":"2023-12-19T16:11:40.237905+0000","last_undegraded":"2023-12-19T16:11:40.237905+0000","last_fullsized":"2023-12-19T16:11:40.237905+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:02:07.204644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238747+0000","last_change":"2023-12-19T16:11:40.229195+0000","last_active":"2023-12-19T16:11:40.238747+0000","last_peered":"2023-12-19T16:11:40.238747+0000","last_clean":"2023-12-19T16:11:40.238747+0000","last_became_active":"2023-12-19T16:11:40.228564+0000","last_became_peered":"2023-12-19T16:11:40.228564+0000","last_unstale":"2023-12-19T16:11:40.238747+0000","last_undegraded":"2023-12-19T16:11:40.238747+0000","last_fullsized":"2023-12-19T16:11:40.238747+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:56.472818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.229730+0000","last_change":"2023-12-19T16:11:40.229730+0000","last_active":"2023-12-19T16:11:40.229730+0000","last_peered":"2023-12-19T16:11:40.229730+0000","last_clean":"2023-12-19T16:11:40.229730+0000","last_became_active":"2023-12-19T16:11:40.229438+0000","last_became_peered":"2023-12-19T16:11:40.229438+0000","last_unstale":"2023-12-19T16:11:40.229730+0000","last_undegraded":"2023-12-19T16:11:40.229730+0000","last_fullsized":"2023-12-19T16:11:40.229730+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:57:58.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240305+0000","last_change":"2023-12-19T16:11:40.240305+0000","last_active":"2023-12-19T16:11:40.240305+0000","last_peered":"2023-12-19T16:11:40.240305+0000","last_clean":"2023-12-19T16:11:40.240305+0000","last_became_active":"2023-12-19T16:11:40.240016+0000","last_became_peered":"2023-12-19T16:11:40.240016+0000","last_unstale":"2023-12-19T16:11:40.240305+0000","last_undegraded":"2023-12-19T16:11:40.240305+0000","last_fullsized":"2023-12-19T16:11:40.240305+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:15.986396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240650+0000","last_change":"2023-12-19T16:11:40.232463+0000","last_active":"2023-12-19T16:11:40.240650+0000","last_peered":"2023-12-19T16:11:40.240650+0000","last_clean":"2023-12-19T16:11:40.240650+0000","last_became_active":"2023-12-19T16:11:40.231321+0000","last_became_peered":"2023-12-19T16:11:40.231321+0000","last_unstale":"2023-12-19T16:11:40.240650+0000","last_undegraded":"2023-12-19T16:11:40.240650+0000","last_fullsized":"2023-12-19T16:11:40.240650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:51.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240456+0000","last_change":"2023-12-19T16:11:40.240456+0000","last_active":"2023-12-19T16:11:40.240456+0000","last_peered":"2023-12-19T16:11:40.240456+0000","last_clean":"2023-12-19T16:11:40.240456+0000","last_became_active":"2023-12-19T16:11:40.240109+0000","last_became_peered":"2023-12-19T16:11:40.240109+0000","last_unstale":"2023-12-19T16:11:40.240456+0000","last_undegraded":"2023-12-19T16:11:40.240456+0000","last_fullsized":"2023-12-19T16:11:40.240456+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:43:20.946037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237031+0000","last_change":"2023-12-19T16:11:40.237031+0000","last_active":"2023-12-19T16:11:40.237031+0000","last_peered":"2023-12-19T16:11:40.237031+0000","last_clean":"2023-12-19T16:11:40.237031+0000","last_became_active":"2023-12-19T16:11:40.236726+0000","last_became_peered":"2023-12-19T16:11:40.236726+0000","last_unstale":"2023-12-19T16:11:40.237031+0000","last_undegraded":"2023-12-19T16:11:40.237031+0000","last_fullsized":"2023-12-19T16:11:40.237031+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:33.279979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238690+0000","last_change":"2023-12-19T16:11:40.232022+0000","last_active":"2023-12-19T16:11:40.238690+0000","last_peered":"2023-12-19T16:11:40.238690+0000","last_clean":"2023-12-19T16:11:40.238690+0000","last_became_active":"2023-12-19T16:11:40.231203+0000","last_became_peered":"2023-12-19T16:11:40.231203+0000","last_unstale":"2023-12-19T16:11:40.238690+0000","last_undegraded":"2023-12-19T16:11:40.238690+0000","last_fullsized":"2023-12-19T16:11:40.238690+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239099+0000","last_change":"2023-12-19T16:11:40.235320+0000","last_active":"2023-12-19T16:11:40.239099+0000","last_peered":"2023-12-19T16:11:40.239099+0000","last_clean":"2023-12-19T16:11:40.239099+0000","last_became_active":"2023-12-19T16:11:40.235026+0000","last_became_peered":"2023-12-19T16:11:40.235026+0000","last_unstale":"2023-12-19T16:11:40.239099+0000","last_undegraded":"2023-12-19T16:11:40.239099+0000","last_fullsized":"2023-12-19T16:11:40.239099+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.222539+0000","last_change":"2023-12-19T16:11:40.222539+0000","last_active":"2023-12-19T16:11:40.222539+0000","last_peered":"2023-12-19T16:11:40.222539+0000","last_clean":"2023-12-19T16:11:40.222539+0000","last_became_active":"2023-12-19T16:11:40.222244+0000","last_became_peered":"2023-12-19T16:11:40.222244+0000","last_unstale":"2023-12-19T16:11:40.222539+0000","last_undegraded":"2023-12-19T16:11:40.222539+0000","last_fullsized":"2023-12-19T16:11:40.222539+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:05:54.627131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235050+0000","last_change":"2023-12-19T16:11:40.235050+0000","last_active":"2023-12-19T16:11:40.235050+0000","last_peered":"2023-12-19T16:11:40.235050+0000","last_clean":"2023-12-19T16:11:40.235050+0000","last_became_active":"2023-12-19T16:11:40.234733+0000","last_became_peered":"2023-12-19T16:11:40.234733+0000","last_unstale":"2023-12-19T16:11:40.235050+0000","last_undegraded":"2023-12-19T16:11:40.235050+0000","last_fullsized":"2023-12-19T16:11:40.235050+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:06:15.097643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231747+0000","last_change":"2023-12-19T16:11:40.231747+0000","last_active":"2023-12-19T16:11:40.231747+0000","last_peered":"2023-12-19T16:11:40.231747+0000","last_clean":"2023-12-19T16:11:40.231747+0000","last_became_active":"2023-12-19T16:11:40.231437+0000","last_became_peered":"2023-12-19T16:11:40.231437+0000","last_unstale":"2023-12-19T16:11:40.231747+0000","last_undegraded":"2023-12-19T16:11:40.231747+0000","last_fullsized":"2023-12-19T16:11:40.231747+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:14:03.883989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.241761+0000","last_change":"2023-12-19T16:11:40.235333+0000","last_active":"2023-12-19T16:11:40.241761+0000","last_peered":"2023-12-19T16:11:40.241761+0000","last_clean":"2023-12-19T16:11:40.241761+0000","last_became_active":"2023-12-19T16:11:40.235027+0000","last_became_peered":"2023-12-19T16:11:40.235027+0000","last_unstale":"2023-12-19T16:11:40.241761+0000","last_undegraded":"2023-12-19T16:11:40.241761+0000","last_fullsized":"2023-12-19T16:11:40.241761+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:22.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235640+0000","last_change":"2023-12-19T16:11:40.235640+0000","last_active":"2023-12-19T16:11:40.235640+0000","last_peered":"2023-12-19T16:11:40.235640+0000","last_clean":"2023-12-19T16:11:40.235640+0000","last_became_active":"2023-12-19T16:11:40.235335+0000","last_became_peered":"2023-12-19T16:11:40.235335+0000","last_unstale":"2023-12-19T16:11:40.235640+0000","last_undegraded":"2023-12-19T16:11:40.235640+0000","last_fullsized":"2023-12-19T16:11:40.235640+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:51.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.233236+0000","last_change":"2023-12-19T16:11:40.233236+0000","last_active":"2023-12-19T16:11:40.233236+0000","last_peered":"2023-12-19T16:11:40.233236+0000","last_clean":"2023-12-19T16:11:40.233236+0000","last_became_active":"2023-12-19T16:11:40.232816+0000","last_became_peered":"2023-12-19T16:11:40.232816+0000","last_unstale":"2023-12-19T16:11:40.233236+0000","last_undegraded":"2023-12-19T16:11:40.233236+0000","last_fullsized":"2023-12-19T16:11:40.233236+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:22.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238202+0000","last_change":"2023-12-19T16:11:40.229992+0000","last_active":"2023-12-19T16:11:40.238202+0000","last_peered":"2023-12-19T16:11:40.238202+0000","last_clean":"2023-12-19T16:11:40.238202+0000","last_became_active":"2023-12-19T16:11:40.229513+0000","last_became_peered":"2023-12-19T16:11:40.229513+0000","last_unstale":"2023-12-19T16:11:40.238202+0000","last_undegraded":"2023-12-19T16:11:40.238202+0000","last_fullsized":"2023-12-19T16:11:40.238202+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.241020+0000","last_change":"2023-12-19T16:11:40.234939+0000","last_active":"2023-12-19T16:11:40.241020+0000","last_peered":"2023-12-19T16:11:40.241020+0000","last_clean":"2023-12-19T16:11:40.241020+0000","last_became_active":"2023-12-19T16:11:40.234661+0000","last_became_peered":"2023-12-19T16:11:40.234661+0000","last_unstale":"2023-12-19T16:11:40.241020+0000","last_undegraded":"2023-12-19T16:11:40.241020+0000","last_fullsized":"2023-12-19T16:11:40.241020+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:22.168120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234741+0000","last_change":"2023-12-19T16:11:40.234741+0000","last_active":"2023-12-19T16:11:40.234741+0000","last_peered":"2023-12-19T16:11:40.234741+0000","last_clean":"2023-12-19T16:11:40.234741+0000","last_became_active":"2023-12-19T16:11:40.234380+0000","last_became_peered":"2023-12-19T16:11:40.234380+0000","last_unstale":"2023-12-19T16:11:40.234741+0000","last_undegraded":"2023-12-19T16:11:40.234741+0000","last_fullsized":"2023-12-19T16:11:40.234741+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:12:30.002649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.228674+0000","last_change":"2023-12-19T16:11:40.228674+0000","last_active":"2023-12-19T16:11:40.228674+0000","last_peered":"2023-12-19T16:11:40.228674+0000","last_clean":"2023-12-19T16:11:40.228674+0000","last_became_active":"2023-12-19T16:11:40.228349+0000","last_became_peered":"2023-12-19T16:11:40.228349+0000","last_unstale":"2023-12-19T16:11:40.228674+0000","last_undegraded":"2023-12-19T16:11:40.228674+0000","last_fullsized":"2023-12-19T16:11:40.228674+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:59.452795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243162+0000","last_change":"2023-12-19T16:11:40.237619+0000","last_active":"2023-12-19T16:11:40.243162+0000","last_peered":"2023-12-19T16:11:40.243162+0000","last_clean":"2023-12-19T16:11:40.243162+0000","last_became_active":"2023-12-19T16:11:40.237303+0000","last_became_peered":"2023-12-19T16:11:40.237303+0000","last_unstale":"2023-12-19T16:11:40.243162+0000","last_undegraded":"2023-12-19T16:11:40.243162+0000","last_fullsized":"2023-12-19T16:11:40.243162+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:43:20.946037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.223525+0000","last_change":"2023-12-19T16:11:40.223525+0000","last_active":"2023-12-19T16:11:40.223525+0000","last_peered":"2023-12-19T16:11:40.223525+0000","last_clean":"2023-12-19T16:11:40.223525+0000","last_became_active":"2023-12-19T16:11:40.223177+0000","last_became_peered":"2023-12-19T16:11:40.223177+0000","last_unstale":"2023-12-19T16:11:40.223525+0000","last_undegraded":"2023-12-19T16:11:40.223525+0000","last_fullsized":"2023-12-19T16:11:40.223525+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:56.472818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.236827+0000","last_change":"2023-12-19T16:11:40.236827+0000","last_active":"2023-12-19T16:11:40.236827+0000","last_peered":"2023-12-19T16:11:40.236827+0000","last_clean":"2023-12-19T16:11:40.236827+0000","last_became_active":"2023-12-19T16:11:40.236119+0000","last_became_peered":"2023-12-19T16:11:40.236119+0000","last_unstale":"2023-12-19T16:11:40.236827+0000","last_undegraded":"2023-12-19T16:11:40.236827+0000","last_fullsized":"2023-12-19T16:11:40.236827+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232773+0000","last_change":"2023-12-19T16:11:40.232773+0000","last_active":"2023-12-19T16:11:40.232773+0000","last_peered":"2023-12-19T16:11:40.232773+0000","last_clean":"2023-12-19T16:11:40.232773+0000","last_became_active":"2023-12-19T16:11:40.232344+0000","last_became_peered":"2023-12-19T16:11:40.232344+0000","last_unstale":"2023-12-19T16:11:40.232773+0000","last_undegraded":"2023-12-19T16:11:40.232773+0000","last_fullsized":"2023-12-19T16:11:40.232773+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:49.544222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.224820+0000","last_change":"2023-12-19T16:11:40.224820+0000","last_active":"2023-12-19T16:11:40.224820+0000","last_peered":"2023-12-19T16:11:40.224820+0000","last_clean":"2023-12-19T16:11:40.224820+0000","last_became_active":"2023-12-19T16:11:40.224516+0000","last_became_peered":"2023-12-19T16:11:40.224516+0000","last_unstale":"2023-12-19T16:11:40.224820+0000","last_undegraded":"2023-12-19T16:11:40.224820+0000","last_fullsized":"2023-12-19T16:11:40.224820+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:26.302439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239202+0000","last_change":"2023-12-19T16:11:40.239202+0000","last_active":"2023-12-19T16:11:40.239202+0000","last_peered":"2023-12-19T16:11:40.239202+0000","last_clean":"2023-12-19T16:11:40.239202+0000","last_became_active":"2023-12-19T16:11:40.238922+0000","last_became_peered":"2023-12-19T16:11:40.238922+0000","last_unstale":"2023-12-19T16:11:40.239202+0000","last_undegraded":"2023-12-19T16:11:40.239202+0000","last_fullsized":"2023-12-19T16:11:40.239202+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:51.074591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"198'4","reported_seq":14,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487567+0000","last_change":"2023-12-19T16:11:39.486904+0000","last_active":"2023-12-19T16:11:40.487567+0000","last_peered":"2023-12-19T16:11:40.487567+0000","last_clean":"2023-12-19T16:11:40.487567+0000","last_became_active":"2023-12-19T16:11:39.486470+0000","last_became_peered":"2023-12-19T16:11:39.486470+0000","last_unstale":"2023-12-19T16:11:40.487567+0000","last_undegraded":"2023-12-19T16:11:40.487567+0000","last_fullsized":"2023-12-19T16:11:40.487567+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:01:46.149503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238268+0000","last_change":"2023-12-19T16:11:40.229804+0000","last_active":"2023-12-19T16:11:40.238268+0000","last_peered":"2023-12-19T16:11:40.238268+0000","last_clean":"2023-12-19T16:11:40.238268+0000","last_became_active":"2023-12-19T16:11:40.227247+0000","last_became_peered":"2023-12-19T16:11:40.227247+0000","last_unstale":"2023-12-19T16:11:40.238268+0000","last_undegraded":"2023-12-19T16:11:40.238268+0000","last_fullsized":"2023-12-19T16:11:40.238268+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:45:29.365616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227063+0000","last_change":"2023-12-19T16:11:39.504070+0000","last_active":"2023-12-19T16:11:40.227063+0000","last_peered":"2023-12-19T16:11:40.227063+0000","last_clean":"2023-12-19T16:11:40.227063+0000","last_became_active":"2023-12-19T16:11:39.503756+0000","last_became_peered":"2023-12-19T16:11:39.503756+0000","last_unstale":"2023-12-19T16:11:40.227063+0000","last_undegraded":"2023-12-19T16:11:40.227063+0000","last_fullsized":"2023-12-19T16:11:40.227063+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:56.677127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234367+0000","last_change":"2023-12-19T16:11:40.234367+0000","last_active":"2023-12-19T16:11:40.234367+0000","last_peered":"2023-12-19T16:11:40.234367+0000","last_clean":"2023-12-19T16:11:40.234367+0000","last_became_active":"2023-12-19T16:11:40.233920+0000","last_became_peered":"2023-12-19T16:11:40.233920+0000","last_unstale":"2023-12-19T16:11:40.234367+0000","last_undegraded":"2023-12-19T16:11:40.234367+0000","last_fullsized":"2023-12-19T16:11:40.234367+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487036+0000","last_change":"2023-12-19T16:11:39.488578+0000","last_active":"2023-12-19T16:11:40.487036+0000","last_peered":"2023-12-19T16:11:40.487036+0000","last_clean":"2023-12-19T16:11:40.487036+0000","last_became_active":"2023-12-19T16:11:39.487481+0000","last_became_peered":"2023-12-19T16:11:39.487481+0000","last_unstale":"2023-12-19T16:11:40.487036+0000","last_undegraded":"2023-12-19T16:11:40.487036+0000","last_fullsized":"2023-12-19T16:11:40.487036+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:45:28.351088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238073+0000","last_change":"2023-12-19T16:11:40.238073+0000","last_active":"2023-12-19T16:11:40.238073+0000","last_peered":"2023-12-19T16:11:40.238073+0000","last_clean":"2023-12-19T16:11:40.238073+0000","last_became_active":"2023-12-19T16:11:40.237804+0000","last_became_peered":"2023-12-19T16:11:40.237804+0000","last_unstale":"2023-12-19T16:11:40.238073+0000","last_undegraded":"2023-12-19T16:11:40.238073+0000","last_fullsized":"2023-12-19T16:11:40.238073+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:36.484146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.490051+0000","last_change":"2023-12-19T16:11:39.486815+0000","last_active":"2023-12-19T16:11:40.490051+0000","last_peered":"2023-12-19T16:11:40.490051+0000","last_clean":"2023-12-19T16:11:40.490051+0000","last_became_active":"2023-12-19T16:11:39.486146+0000","last_became_peered":"2023-12-19T16:11:39.486146+0000","last_unstale":"2023-12-19T16:11:40.490051+0000","last_undegraded":"2023-12-19T16:11:40.490051+0000","last_fullsized":"2023-12-19T16:11:40.490051+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:22:58.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239165+0000","last_change":"2023-12-19T16:11:40.233273+0000","last_active":"2023-12-19T16:11:40.239165+0000","last_peered":"2023-12-19T16:11:40.239165+0000","last_clean":"2023-12-19T16:11:40.239165+0000","last_became_active":"2023-12-19T16:11:40.232947+0000","last_became_peered":"2023-12-19T16:11:40.232947+0000","last_unstale":"2023-12-19T16:11:40.239165+0000","last_undegraded":"2023-12-19T16:11:40.239165+0000","last_fullsized":"2023-12-19T16:11:40.239165+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"198'1","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.491376+0000","last_change":"2023-12-19T16:11:39.505963+0000","last_active":"2023-12-19T16:11:40.491376+0000","last_peered":"2023-12-19T16:11:40.491376+0000","last_clean":"2023-12-19T16:11:40.491376+0000","last_became_active":"2023-12-19T16:11:39.505662+0000","last_became_peered":"2023-12-19T16:11:39.505662+0000","last_unstale":"2023-12-19T16:11:40.491376+0000","last_undegraded":"2023-12-19T16:11:40.491376+0000","last_fullsized":"2023-12-19T16:11:40.491376+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:11.151376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.221287+0000","last_change":"2023-12-19T16:11:40.221287+0000","last_active":"2023-12-19T16:11:40.221287+0000","last_peered":"2023-12-19T16:11:40.221287+0000","last_clean":"2023-12-19T16:11:40.221287+0000","last_became_active":"2023-12-19T16:11:40.220735+0000","last_became_peered":"2023-12-19T16:11:40.220735+0000","last_unstale":"2023-12-19T16:11:40.221287+0000","last_undegraded":"2023-12-19T16:11:40.221287+0000","last_fullsized":"2023-12-19T16:11:40.221287+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:37:19.405072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226701+0000","last_change":"2023-12-19T16:11:39.501153+0000","last_active":"2023-12-19T16:11:40.226701+0000","last_peered":"2023-12-19T16:11:40.226701+0000","last_clean":"2023-12-19T16:11:40.226701+0000","last_became_active":"2023-12-19T16:11:39.500853+0000","last_became_peered":"2023-12-19T16:11:39.500853+0000","last_unstale":"2023-12-19T16:11:40.226701+0000","last_undegraded":"2023-12-19T16:11:40.226701+0000","last_fullsized":"2023-12-19T16:11:40.226701+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:50.060063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.228115+0000","last_change":"2023-12-19T16:11:40.228115+0000","last_active":"2023-12-19T16:11:40.228115+0000","last_peered":"2023-12-19T16:11:40.228115+0000","last_clean":"2023-12-19T16:11:40.228115+0000","last_became_active":"2023-12-19T16:11:40.227810+0000","last_became_peered":"2023-12-19T16:11:40.227810+0000","last_unstale":"2023-12-19T16:11:40.228115+0000","last_undegraded":"2023-12-19T16:11:40.228115+0000","last_fullsized":"2023-12-19T16:11:40.228115+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:33.288888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.499192+0000","last_change":"2023-12-19T16:11:39.496611+0000","last_active":"2023-12-19T16:11:39.499192+0000","last_peered":"2023-12-19T16:11:39.499192+0000","last_clean":"2023-12-19T16:11:39.499192+0000","last_became_active":"2023-12-19T16:11:39.496306+0000","last_became_peered":"2023-12-19T16:11:39.496306+0000","last_unstale":"2023-12-19T16:11:39.499192+0000","last_undegraded":"2023-12-19T16:11:39.499192+0000","last_fullsized":"2023-12-19T16:11:39.499192+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:52.265135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.244868+0000","last_change":"2023-12-19T16:11:40.241420+0000","last_active":"2023-12-19T16:11:40.244868+0000","last_peered":"2023-12-19T16:11:40.244868+0000","last_clean":"2023-12-19T16:11:40.244868+0000","last_became_active":"2023-12-19T16:11:40.237233+0000","last_became_peered":"2023-12-19T16:11:40.237233+0000","last_unstale":"2023-12-19T16:11:40.244868+0000","last_undegraded":"2023-12-19T16:11:40.244868+0000","last_fullsized":"2023-12-19T16:11:40.244868+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:39.270097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.210249+0000","last_change":"2023-12-19T16:11:39.484982+0000","last_active":"2023-12-19T16:11:40.210249+0000","last_peered":"2023-12-19T16:11:40.210249+0000","last_clean":"2023-12-19T16:11:40.210249+0000","last_became_active":"2023-12-19T16:11:39.484606+0000","last_became_peered":"2023-12-19T16:11:39.484606+0000","last_unstale":"2023-12-19T16:11:40.210249+0000","last_undegraded":"2023-12-19T16:11:40.210249+0000","last_fullsized":"2023-12-19T16:11:40.210249+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:42.384599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.229740+0000","last_change":"2023-12-19T16:11:40.229740+0000","last_active":"2023-12-19T16:11:40.229740+0000","last_peered":"2023-12-19T16:11:40.229740+0000","last_clean":"2023-12-19T16:11:40.229740+0000","last_became_active":"2023-12-19T16:11:40.229350+0000","last_became_peered":"2023-12-19T16:11:40.229350+0000","last_unstale":"2023-12-19T16:11:40.229740+0000","last_undegraded":"2023-12-19T16:11:40.229740+0000","last_fullsized":"2023-12-19T16:11:40.229740+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:37:55.501533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.215242+0000","last_change":"2023-12-19T16:11:39.500513+0000","last_active":"2023-12-19T16:11:40.215242+0000","last_peered":"2023-12-19T16:11:40.215242+0000","last_clean":"2023-12-19T16:11:40.215242+0000","last_became_active":"2023-12-19T16:11:39.499306+0000","last_became_peered":"2023-12-19T16:11:39.499306+0000","last_unstale":"2023-12-19T16:11:40.215242+0000","last_undegraded":"2023-12-19T16:11:40.215242+0000","last_fullsized":"2023-12-19T16:11:40.215242+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:35.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226537+0000","last_change":"2023-12-19T16:11:40.224815+0000","last_active":"2023-12-19T16:11:40.226537+0000","last_peered":"2023-12-19T16:11:40.226537+0000","last_clean":"2023-12-19T16:11:40.226537+0000","last_became_active":"2023-12-19T16:11:40.224239+0000","last_became_peered":"2023-12-19T16:11:40.224239+0000","last_unstale":"2023-12-19T16:11:40.226537+0000","last_undegraded":"2023-12-19T16:11:40.226537+0000","last_fullsized":"2023-12-19T16:11:40.226537+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.255854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"198'3","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487003+0000","last_change":"2023-12-19T16:11:39.482928+0000","last_active":"2023-12-19T16:11:40.487003+0000","last_peered":"2023-12-19T16:11:40.487003+0000","last_clean":"2023-12-19T16:11:40.487003+0000","last_became_active":"2023-12-19T16:11:39.482627+0000","last_became_peered":"2023-12-19T16:11:39.482627+0000","last_unstale":"2023-12-19T16:11:40.487003+0000","last_undegraded":"2023-12-19T16:11:40.487003+0000","last_fullsized":"2023-12-19T16:11:40.487003+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:22:10.107299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239822+0000","last_change":"2023-12-19T16:11:40.239822+0000","last_active":"2023-12-19T16:11:40.239822+0000","last_peered":"2023-12-19T16:11:40.239822+0000","last_clean":"2023-12-19T16:11:40.239822+0000","last_became_active":"2023-12-19T16:11:40.239468+0000","last_became_peered":"2023-12-19T16:11:40.239468+0000","last_unstale":"2023-12-19T16:11:40.239822+0000","last_undegraded":"2023-12-19T16:11:40.239822+0000","last_fullsized":"2023-12-19T16:11:40.239822+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:41.730382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"198'6","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.494348+0000","last_change":"2023-12-19T16:11:39.495200+0000","last_active":"2023-12-19T16:11:40.494348+0000","last_peered":"2023-12-19T16:11:40.494348+0000","last_clean":"2023-12-19T16:11:40.494348+0000","last_became_active":"2023-12-19T16:11:39.491795+0000","last_became_peered":"2023-12-19T16:11:39.491795+0000","last_unstale":"2023-12-19T16:11:40.494348+0000","last_undegraded":"2023-12-19T16:11:40.494348+0000","last_fullsized":"2023-12-19T16:11:40.494348+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:09.163040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232023+0000","last_change":"2023-12-19T16:11:40.232023+0000","last_active":"2023-12-19T16:11:40.232023+0000","last_peered":"2023-12-19T16:11:40.232023+0000","last_clean":"2023-12-19T16:11:40.232023+0000","last_became_active":"2023-12-19T16:11:40.231728+0000","last_became_peered":"2023-12-19T16:11:40.231728+0000","last_unstale":"2023-12-19T16:11:40.232023+0000","last_undegraded":"2023-12-19T16:11:40.232023+0000","last_fullsized":"2023-12-19T16:11:40.232023+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.490780+0000","last_change":"2023-12-19T16:11:39.503180+0000","last_active":"2023-12-19T16:11:40.490780+0000","last_peered":"2023-12-19T16:11:40.490780+0000","last_clean":"2023-12-19T16:11:40.490780+0000","last_became_active":"2023-12-19T16:11:39.502791+0000","last_became_peered":"2023-12-19T16:11:39.502791+0000","last_unstale":"2023-12-19T16:11:40.490780+0000","last_undegraded":"2023-12-19T16:11:40.490780+0000","last_fullsized":"2023-12-19T16:11:40.490780+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:51.241326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239207+0000","last_change":"2023-12-19T16:11:40.236817+0000","last_active":"2023-12-19T16:11:40.239207+0000","last_peered":"2023-12-19T16:11:40.239207+0000","last_clean":"2023-12-19T16:11:40.239207+0000","last_became_active":"2023-12-19T16:11:40.236185+0000","last_became_peered":"2023-12-19T16:11:40.236185+0000","last_unstale":"2023-12-19T16:11:40.239207+0000","last_undegraded":"2023-12-19T16:11:40.239207+0000","last_fullsized":"2023-12-19T16:11:40.239207+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:15.986396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.213162+0000","last_change":"2023-12-19T16:11:39.488250+0000","last_active":"2023-12-19T16:11:40.213162+0000","last_peered":"2023-12-19T16:11:40.213162+0000","last_clean":"2023-12-19T16:11:40.213162+0000","last_became_active":"2023-12-19T16:11:39.487386+0000","last_became_peered":"2023-12-19T16:11:39.487386+0000","last_unstale":"2023-12-19T16:11:40.213162+0000","last_undegraded":"2023-12-19T16:11:40.213162+0000","last_fullsized":"2023-12-19T16:11:40.213162+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:18.470316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232452+0000","last_change":"2023-12-19T16:11:40.232452+0000","last_active":"2023-12-19T16:11:40.232452+0000","last_peered":"2023-12-19T16:11:40.232452+0000","last_clean":"2023-12-19T16:11:40.232452+0000","last_became_active":"2023-12-19T16:11:40.232149+0000","last_became_peered":"2023-12-19T16:11:40.232149+0000","last_unstale":"2023-12-19T16:11:40.232452+0000","last_undegraded":"2023-12-19T16:11:40.232452+0000","last_fullsized":"2023-12-19T16:11:40.232452+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.499495+0000","last_change":"2023-12-19T16:11:39.497671+0000","last_active":"2023-12-19T16:11:39.499495+0000","last_peered":"2023-12-19T16:11:39.499495+0000","last_clean":"2023-12-19T16:11:39.499495+0000","last_became_active":"2023-12-19T16:11:39.497473+0000","last_became_peered":"2023-12-19T16:11:39.497473+0000","last_unstale":"2023-12-19T16:11:39.499495+0000","last_undegraded":"2023-12-19T16:11:39.499495+0000","last_fullsized":"2023-12-19T16:11:39.499495+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:22.997804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232477+0000","last_change":"2023-12-19T16:11:40.232477+0000","last_active":"2023-12-19T16:11:40.232477+0000","last_peered":"2023-12-19T16:11:40.232477+0000","last_clean":"2023-12-19T16:11:40.232477+0000","last_became_active":"2023-12-19T16:11:40.229817+0000","last_became_peered":"2023-12-19T16:11:40.229817+0000","last_unstale":"2023-12-19T16:11:40.232477+0000","last_undegraded":"2023-12-19T16:11:40.232477+0000","last_fullsized":"2023-12-19T16:11:40.232477+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:23:22.461430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"198'8","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.492316+0000","last_change":"2023-12-19T16:11:39.487468+0000","last_active":"2023-12-19T16:11:40.492316+0000","last_peered":"2023-12-19T16:11:40.492316+0000","last_clean":"2023-12-19T16:11:40.492316+0000","last_became_active":"2023-12-19T16:11:39.487176+0000","last_became_peered":"2023-12-19T16:11:39.487176+0000","last_unstale":"2023-12-19T16:11:40.492316+0000","last_undegraded":"2023-12-19T16:11:40.492316+0000","last_fullsized":"2023-12-19T16:11:40.492316+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:46.979659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234502+0000","last_change":"2023-12-19T16:11:40.225688+0000","last_active":"2023-12-19T16:11:40.234502+0000","last_peered":"2023-12-19T16:11:40.234502+0000","last_clean":"2023-12-19T16:11:40.234502+0000","last_became_active":"2023-12-19T16:11:40.225297+0000","last_became_peered":"2023-12-19T16:11:40.225297+0000","last_unstale":"2023-12-19T16:11:40.234502+0000","last_undegraded":"2023-12-19T16:11:40.234502+0000","last_fullsized":"2023-12-19T16:11:40.234502+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:13.538074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"198'1","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.489975+0000","last_change":"2023-12-19T16:11:39.489356+0000","last_active":"2023-12-19T16:11:40.489975+0000","last_peered":"2023-12-19T16:11:40.489975+0000","last_clean":"2023-12-19T16:11:40.489975+0000","last_became_active":"2023-12-19T16:11:39.489037+0000","last_became_peered":"2023-12-19T16:11:39.489037+0000","last_unstale":"2023-12-19T16:11:40.489975+0000","last_undegraded":"2023-12-19T16:11:40.489975+0000","last_fullsized":"2023-12-19T16:11:40.489975+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:54.697560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.230088+0000","last_change":"2023-12-19T16:11:40.230088+0000","last_active":"2023-12-19T16:11:40.230088+0000","last_peered":"2023-12-19T16:11:40.230088+0000","last_clean":"2023-12-19T16:11:40.230088+0000","last_became_active":"2023-12-19T16:11:40.229743+0000","last_became_peered":"2023-12-19T16:11:40.229743+0000","last_unstale":"2023-12-19T16:11:40.230088+0000","last_undegraded":"2023-12-19T16:11:40.230088+0000","last_fullsized":"2023-12-19T16:11:40.230088+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:45:44.649317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"198'2","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.491547+0000","last_change":"2023-12-19T16:11:39.496632+0000","last_active":"2023-12-19T16:11:40.491547+0000","last_peered":"2023-12-19T16:11:40.491547+0000","last_clean":"2023-12-19T16:11:40.491547+0000","last_became_active":"2023-12-19T16:11:39.492350+0000","last_became_peered":"2023-12-19T16:11:39.492350+0000","last_unstale":"2023-12-19T16:11:40.491547+0000","last_undegraded":"2023-12-19T16:11:40.491547+0000","last_fullsized":"2023-12-19T16:11:40.491547+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:05:15.663991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.233934+0000","last_change":"2023-12-19T16:11:40.233934+0000","last_active":"2023-12-19T16:11:40.233934+0000","last_peered":"2023-12-19T16:11:40.233934+0000","last_clean":"2023-12-19T16:11:40.233934+0000","last_became_active":"2023-12-19T16:11:40.233640+0000","last_became_peered":"2023-12-19T16:11:40.233640+0000","last_unstale":"2023-12-19T16:11:40.233934+0000","last_undegraded":"2023-12-19T16:11:40.233934+0000","last_fullsized":"2023-12-19T16:11:40.233934+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:29:18.069075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.225968+0000","last_change":"2023-12-19T16:11:39.501864+0000","last_active":"2023-12-19T16:11:40.225968+0000","last_peered":"2023-12-19T16:11:40.225968+0000","last_clean":"2023-12-19T16:11:40.225968+0000","last_became_active":"2023-12-19T16:11:39.501593+0000","last_became_peered":"2023-12-19T16:11:39.501593+0000","last_unstale":"2023-12-19T16:11:40.225968+0000","last_undegraded":"2023-12-19T16:11:40.225968+0000","last_fullsized":"2023-12-19T16:11:40.225968+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:58.438267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235640+0000","last_change":"2023-12-19T16:11:40.235640+0000","last_active":"2023-12-19T16:11:40.235640+0000","last_peered":"2023-12-19T16:11:40.235640+0000","last_clean":"2023-12-19T16:11:40.235640+0000","last_became_active":"2023-12-19T16:11:40.235368+0000","last_became_peered":"2023-12-19T16:11:40.235368+0000","last_unstale":"2023-12-19T16:11:40.235640+0000","last_undegraded":"2023-12-19T16:11:40.235640+0000","last_fullsized":"2023-12-19T16:11:40.235640+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:45.649577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.210063+0000","last_change":"2023-12-19T16:11:39.503117+0000","last_active":"2023-12-19T16:11:40.210063+0000","last_peered":"2023-12-19T16:11:40.210063+0000","last_clean":"2023-12-19T16:11:40.210063+0000","last_became_active":"2023-12-19T16:11:39.502847+0000","last_became_peered":"2023-12-19T16:11:39.502847+0000","last_unstale":"2023-12-19T16:11:40.210063+0000","last_undegraded":"2023-12-19T16:11:40.210063+0000","last_fullsized":"2023-12-19T16:11:40.210063+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:01:36.672177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243501+0000","last_change":"2023-12-19T16:11:40.239771+0000","last_active":"2023-12-19T16:11:40.243501+0000","last_peered":"2023-12-19T16:11:40.243501+0000","last_clean":"2023-12-19T16:11:40.243501+0000","last_became_active":"2023-12-19T16:11:40.236979+0000","last_became_peered":"2023-12-19T16:11:40.236979+0000","last_unstale":"2023-12-19T16:11:40.243501+0000","last_undegraded":"2023-12-19T16:11:40.243501+0000","last_fullsized":"2023-12-19T16:11:40.243501+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:22.199558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.489779+0000","last_change":"2023-12-19T16:11:39.501556+0000","last_active":"2023-12-19T16:11:40.489779+0000","last_peered":"2023-12-19T16:11:40.489779+0000","last_clean":"2023-12-19T16:11:40.489779+0000","last_became_active":"2023-12-19T16:11:39.501100+0000","last_became_peered":"2023-12-19T16:11:39.501100+0000","last_unstale":"2023-12-19T16:11:40.489779+0000","last_undegraded":"2023-12-19T16:11:40.489779+0000","last_fullsized":"2023-12-19T16:11:40.489779+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:50.279745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235578+0000","last_change":"2023-12-19T16:11:40.235578+0000","last_active":"2023-12-19T16:11:40.235578+0000","last_peered":"2023-12-19T16:11:40.235578+0000","last_clean":"2023-12-19T16:11:40.235578+0000","last_became_active":"2023-12-19T16:11:40.231078+0000","last_became_peered":"2023-12-19T16:11:40.231078+0000","last_unstale":"2023-12-19T16:11:40.235578+0000","last_undegraded":"2023-12-19T16:11:40.235578+0000","last_fullsized":"2023-12-19T16:11:40.235578+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:29:07.800630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.486774+0000","last_change":"2023-12-19T16:11:39.485097+0000","last_active":"2023-12-19T16:11:39.486774+0000","last_peered":"2023-12-19T16:11:39.486774+0000","last_clean":"2023-12-19T16:11:39.486774+0000","last_became_active":"2023-12-19T16:11:39.484747+0000","last_became_peered":"2023-12-19T16:11:39.484747+0000","last_unstale":"2023-12-19T16:11:39.486774+0000","last_undegraded":"2023-12-19T16:11:39.486774+0000","last_fullsized":"2023-12-19T16:11:39.486774+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:56.677127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238270+0000","last_change":"2023-12-19T16:11:40.238270+0000","last_active":"2023-12-19T16:11:40.238270+0000","last_peered":"2023-12-19T16:11:40.238270+0000","last_clean":"2023-12-19T16:11:40.238270+0000","last_became_active":"2023-12-19T16:11:40.237980+0000","last_became_peered":"2023-12-19T16:11:40.237980+0000","last_unstale":"2023-12-19T16:11:40.238270+0000","last_undegraded":"2023-12-19T16:11:40.238270+0000","last_fullsized":"2023-12-19T16:11:40.238270+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:05.523201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.493261+0000","last_change":"2023-12-19T16:11:39.485864+0000","last_active":"2023-12-19T16:11:39.493261+0000","last_peered":"2023-12-19T16:11:39.493261+0000","last_clean":"2023-12-19T16:11:39.493261+0000","last_became_active":"2023-12-19T16:11:39.484925+0000","last_became_peered":"2023-12-19T16:11:39.484925+0000","last_unstale":"2023-12-19T16:11:39.493261+0000","last_undegraded":"2023-12-19T16:11:39.493261+0000","last_fullsized":"2023-12-19T16:11:39.493261+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:20:52.583832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227374+0000","last_change":"2023-12-19T16:11:40.227374+0000","last_active":"2023-12-19T16:11:40.227374+0000","last_peered":"2023-12-19T16:11:40.227374+0000","last_clean":"2023-12-19T16:11:40.227374+0000","last_became_active":"2023-12-19T16:11:40.227067+0000","last_became_peered":"2023-12-19T16:11:40.227067+0000","last_unstale":"2023-12-19T16:11:40.227374+0000","last_undegraded":"2023-12-19T16:11:40.227374+0000","last_fullsized":"2023-12-19T16:11:40.227374+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.482572+0000","last_change":"2023-12-19T16:11:39.480826+0000","last_active":"2023-12-19T16:11:39.482572+0000","last_peered":"2023-12-19T16:11:39.482572+0000","last_clean":"2023-12-19T16:11:39.482572+0000","last_became_active":"2023-12-19T16:11:39.480515+0000","last_became_peered":"2023-12-19T16:11:39.480515+0000","last_unstale":"2023-12-19T16:11:39.482572+0000","last_undegraded":"2023-12-19T16:11:39.482572+0000","last_fullsized":"2023-12-19T16:11:39.482572+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:26:45.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226192+0000","last_change":"2023-12-19T16:11:40.226192+0000","last_active":"2023-12-19T16:11:40.226192+0000","last_peered":"2023-12-19T16:11:40.226192+0000","last_clean":"2023-12-19T16:11:40.226192+0000","last_became_active":"2023-12-19T16:11:40.225916+0000","last_became_peered":"2023-12-19T16:11:40.225916+0000","last_unstale":"2023-12-19T16:11:40.226192+0000","last_undegraded":"2023-12-19T16:11:40.226192+0000","last_fullsized":"2023-12-19T16:11:40.226192+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:06:14.919875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.488382+0000","last_change":"2023-12-19T16:11:39.502305+0000","last_active":"2023-12-19T16:11:40.488382+0000","last_peered":"2023-12-19T16:11:40.488382+0000","last_clean":"2023-12-19T16:11:40.488382+0000","last_became_active":"2023-12-19T16:11:39.501012+0000","last_became_peered":"2023-12-19T16:11:39.501012+0000","last_unstale":"2023-12-19T16:11:40.488382+0000","last_undegraded":"2023-12-19T16:11:40.488382+0000","last_fullsized":"2023-12-19T16:11:40.488382+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:35.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238505+0000","last_change":"2023-12-19T16:11:40.227386+0000","last_active":"2023-12-19T16:11:40.238505+0000","last_peered":"2023-12-19T16:11:40.238505+0000","last_clean":"2023-12-19T16:11:40.238505+0000","last_became_active":"2023-12-19T16:11:40.227102+0000","last_became_peered":"2023-12-19T16:11:40.227102+0000","last_unstale":"2023-12-19T16:11:40.238505+0000","last_undegraded":"2023-12-19T16:11:40.238505+0000","last_fullsized":"2023-12-19T16:11:40.238505+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214858+0000","last_change":"2023-12-19T16:11:39.491592+0000","last_active":"2023-12-19T16:11:40.214858+0000","last_peered":"2023-12-19T16:11:40.214858+0000","last_clean":"2023-12-19T16:11:40.214858+0000","last_became_active":"2023-12-19T16:11:39.491283+0000","last_became_peered":"2023-12-19T16:11:39.491283+0000","last_unstale":"2023-12-19T16:11:40.214858+0000","last_undegraded":"2023-12-19T16:11:40.214858+0000","last_fullsized":"2023-12-19T16:11:40.214858+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:55.458290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231796+0000","last_change":"2023-12-19T16:11:40.231796+0000","last_active":"2023-12-19T16:11:40.231796+0000","last_peered":"2023-12-19T16:11:40.231796+0000","last_clean":"2023-12-19T16:11:40.231796+0000","last_became_active":"2023-12-19T16:11:40.231479+0000","last_became_peered":"2023-12-19T16:11:40.231479+0000","last_unstale":"2023-12-19T16:11:40.231796+0000","last_undegraded":"2023-12-19T16:11:40.231796+0000","last_fullsized":"2023-12-19T16:11:40.231796+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:34.846952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214162+0000","last_change":"2023-12-19T16:11:39.489719+0000","last_active":"2023-12-19T16:11:40.214162+0000","last_peered":"2023-12-19T16:11:40.214162+0000","last_clean":"2023-12-19T16:11:40.214162+0000","last_became_active":"2023-12-19T16:11:39.489390+0000","last_became_peered":"2023-12-19T16:11:39.489390+0000","last_unstale":"2023-12-19T16:11:40.214162+0000","last_undegraded":"2023-12-19T16:11:40.214162+0000","last_fullsized":"2023-12-19T16:11:40.214162+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:38.255569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234535+0000","last_change":"2023-12-19T16:11:40.223671+0000","last_active":"2023-12-19T16:11:40.234535+0000","last_peered":"2023-12-19T16:11:40.234535+0000","last_clean":"2023-12-19T16:11:40.234535+0000","last_became_active":"2023-12-19T16:11:40.223296+0000","last_became_peered":"2023-12-19T16:11:40.223296+0000","last_unstale":"2023-12-19T16:11:40.234535+0000","last_undegraded":"2023-12-19T16:11:40.234535+0000","last_fullsized":"2023-12-19T16:11:40.234535+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.493234+0000","last_change":"2023-12-19T16:11:39.487377+0000","last_active":"2023-12-19T16:11:39.493234+0000","last_peered":"2023-12-19T16:11:39.493234+0000","last_clean":"2023-12-19T16:11:39.493234+0000","last_became_active":"2023-12-19T16:11:39.486537+0000","last_became_peered":"2023-12-19T16:11:39.486537+0000","last_unstale":"2023-12-19T16:11:39.493234+0000","last_undegraded":"2023-12-19T16:11:39.493234+0000","last_fullsized":"2023-12-19T16:11:39.493234+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:09.886098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239192+0000","last_change":"2023-12-19T16:11:40.234079+0000","last_active":"2023-12-19T16:11:40.239192+0000","last_peered":"2023-12-19T16:11:40.239192+0000","last_clean":"2023-12-19T16:11:40.239192+0000","last_became_active":"2023-12-19T16:11:40.233787+0000","last_became_peered":"2023-12-19T16:11:40.233787+0000","last_unstale":"2023-12-19T16:11:40.239192+0000","last_undegraded":"2023-12-19T16:11:40.239192+0000","last_fullsized":"2023-12-19T16:11:40.239192+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.216810+0000","last_change":"2023-12-19T16:11:39.500737+0000","last_active":"2023-12-19T16:11:40.216810+0000","last_peered":"2023-12-19T16:11:40.216810+0000","last_clean":"2023-12-19T16:11:40.216810+0000","last_became_active":"2023-12-19T16:11:39.500363+0000","last_became_peered":"2023-12-19T16:11:39.500363+0000","last_unstale":"2023-12-19T16:11:40.216810+0000","last_undegraded":"2023-12-19T16:11:40.216810+0000","last_fullsized":"2023-12-19T16:11:40.216810+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:26:45.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227676+0000","last_change":"2023-12-19T16:11:40.227676+0000","last_active":"2023-12-19T16:11:40.227676+0000","last_peered":"2023-12-19T16:11:40.227676+0000","last_clean":"2023-12-19T16:11:40.227676+0000","last_became_active":"2023-12-19T16:11:40.227326+0000","last_became_peered":"2023-12-19T16:11:40.227326+0000","last_unstale":"2023-12-19T16:11:40.227676+0000","last_undegraded":"2023-12-19T16:11:40.227676+0000","last_fullsized":"2023-12-19T16:11:40.227676+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:22:15.025200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.213325+0000","last_change":"2023-12-19T16:11:39.492749+0000","last_active":"2023-12-19T16:11:40.213325+0000","last_peered":"2023-12-19T16:11:40.213325+0000","last_clean":"2023-12-19T16:11:40.213325+0000","last_became_active":"2023-12-19T16:11:39.491630+0000","last_became_peered":"2023-12-19T16:11:39.491630+0000","last_unstale":"2023-12-19T16:11:40.213325+0000","last_undegraded":"2023-12-19T16:11:40.213325+0000","last_fullsized":"2023-12-19T16:11:40.213325+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:12.523546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231901+0000","last_change":"2023-12-19T16:11:40.228970+0000","last_active":"2023-12-19T16:11:40.231901+0000","last_peered":"2023-12-19T16:11:40.231901+0000","last_clean":"2023-12-19T16:11:40.231901+0000","last_became_active":"2023-12-19T16:11:40.226741+0000","last_became_peered":"2023-12-19T16:11:40.226741+0000","last_unstale":"2023-12-19T16:11:40.231901+0000","last_undegraded":"2023-12-19T16:11:40.231901+0000","last_fullsized":"2023-12-19T16:11:40.231901+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.489961+0000","last_change":"2023-12-19T16:11:39.488851+0000","last_active":"2023-12-19T16:11:39.489961+0000","last_peered":"2023-12-19T16:11:39.489961+0000","last_clean":"2023-12-19T16:11:39.489961+0000","last_became_active":"2023-12-19T16:11:39.488152+0000","last_became_peered":"2023-12-19T16:11:39.488152+0000","last_unstale":"2023-12-19T16:11:39.489961+0000","last_undegraded":"2023-12-19T16:11:39.489961+0000","last_fullsized":"2023-12-19T16:11:39.489961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:05:58.957511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232170+0000","last_change":"2023-12-19T16:11:40.232170+0000","last_active":"2023-12-19T16:11:40.232170+0000","last_peered":"2023-12-19T16:11:40.232170+0000","last_clean":"2023-12-19T16:11:40.232170+0000","last_became_active":"2023-12-19T16:11:40.231785+0000","last_became_peered":"2023-12-19T16:11:40.231785+0000","last_unstale":"2023-12-19T16:11:40.232170+0000","last_undegraded":"2023-12-19T16:11:40.232170+0000","last_fullsized":"2023-12-19T16:11:40.232170+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:08.796681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.506644+0000","last_change":"2023-12-19T16:11:39.502487+0000","last_active":"2023-12-19T16:11:39.506644+0000","last_peered":"2023-12-19T16:11:39.506644+0000","last_clean":"2023-12-19T16:11:39.506644+0000","last_became_active":"2023-12-19T16:11:39.501428+0000","last_became_peered":"2023-12-19T16:11:39.501428+0000","last_unstale":"2023-12-19T16:11:39.506644+0000","last_undegraded":"2023-12-19T16:11:39.506644+0000","last_fullsized":"2023-12-19T16:11:39.506644+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:41.733403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231288+0000","last_change":"2023-12-19T16:11:40.231288+0000","last_active":"2023-12-19T16:11:40.231288+0000","last_peered":"2023-12-19T16:11:40.231288+0000","last_clean":"2023-12-19T16:11:40.231288+0000","last_became_active":"2023-12-19T16:11:40.230985+0000","last_became_peered":"2023-12-19T16:11:40.230985+0000","last_unstale":"2023-12-19T16:11:40.231288+0000","last_undegraded":"2023-12-19T16:11:40.231288+0000","last_fullsized":"2023-12-19T16:11:40.231288+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:42.747931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214135+0000","last_change":"2023-12-19T16:11:39.494555+0000","last_active":"2023-12-19T16:11:40.214135+0000","last_peered":"2023-12-19T16:11:40.214135+0000","last_clean":"2023-12-19T16:11:40.214135+0000","last_became_active":"2023-12-19T16:11:39.491704+0000","last_became_peered":"2023-12-19T16:11:39.491704+0000","last_unstale":"2023-12-19T16:11:40.214135+0000","last_undegraded":"2023-12-19T16:11:40.214135+0000","last_fullsized":"2023-12-19T16:11:40.214135+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:14.971868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"187'326","reported_seq":1049,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.471013+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:11:39.471013+0000","last_peered":"2023-12-19T16:11:39.471013+0000","last_clean":"2023-12-19T16:11:39.471013+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:11:39.471013+0000","last_undegraded":"2023-12-19T16:11:39.471013+0000","last_fullsized":"2023-12-19T16:11:39.471013+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304384,"kb_used_data":3000,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067456,"statfs":{"total":96636764160,"available":96325074944,"internally_reserved":0,"allocated":3072000,"data_stored":1856840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.113,"15min":1.119},"min":{"1min":0.84299999999999997,"5min":0.70899999999999996,"15min":0.56399999999999995},"max":{"1min":1.661,"5min":1.661,"15min":1.6919999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.875,"5min":0.80500000000000005,"15min":0.496},"max":{"1min":1.597,"5min":1.597,"15min":1.9379999999999999},"last":0.95899999999999996}]},{"osd":1,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1479999999999999,"15min":1.139},"min":{"1min":0.85399999999999998,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":1.45,"5min":1.623,"15min":1.903},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.24,"15min":1.2410000000000001},"min":{"1min":1.0580000000000001,"5min":0.82699999999999996,"15min":0.73299999999999998},"max":{"1min":1.5489999999999999,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.45}]},{"osd":2,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.272,"15min":1.298},"min":{"1min":0.98299999999999998,"5min":0.89300000000000002,"15min":0.63300000000000001},"max":{"1min":1.5860000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.3100000000000001,"15min":1.3240000000000001},"min":{"1min":1.095,"5min":0.995,"15min":0.68999999999999995},"max":{"1min":1.631,"5min":1.7010000000000001,"15min":1.7709999999999999},"last":1.373}]},{"osd":3,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.319,"15min":1.3380000000000001},"min":{"1min":1.133,"5min":0.94099999999999995,"15min":0.52500000000000002},"max":{"1min":1.623,"5min":1.736,"15min":1.839},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3560000000000001,"15min":1.3700000000000001},"min":{"1min":1.224,"5min":0.84299999999999997,"15min":0.60699999999999998},"max":{"1min":1.603,"5min":1.726,"15min":1.9630000000000001},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.90900000000000003,"5min":0.74199999999999999,"15min":0.55600000000000005},"max":{"1min":1.528,"5min":1.77,"15min":1.849},"last":1.327},{"interface":"front","average":{"1min":1.23,"5min":1.1990000000000001,"15min":1.232},"min":{"1min":0.82099999999999995,"5min":0.58699999999999997,"15min":0.47699999999999998},"max":{"1min":1.5780000000000001,"5min":1.651,"15min":1.784},"last":1.113}]},{"osd":5,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.94399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.679,"5min":1.821,"15min":1.825},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.343,"15min":1.337},"min":{"1min":1.048,"5min":0.84599999999999997,"15min":0.45000000000000001},"max":{"1min":1.6499999999999999,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.1950000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4299999999999999,"15min":1.4330000000000001},"min":{"1min":1.1839999999999999,"5min":0.97699999999999998,"15min":0.54200000000000004},"max":{"1min":1.714,"5min":1.994,"15min":1.994},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.3660000000000001,"15min":1.3919999999999999},"min":{"1min":1.0580000000000001,"5min":0.746,"15min":0.63100000000000001},"max":{"1min":1.7330000000000001,"5min":2.0760000000000001,"15min":2.1059999999999999},"last":1.528}]}]},{"osd":6,"up_from":13,"seq":55834575953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311996,"kb_used_data":2420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059844,"statfs":{"total":96636764160,"available":96317280256,"internally_reserved":0,"allocated":2478080,"data_stored":1266472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1739999999999999,"15min":1.175},"min":{"1min":0.80300000000000005,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.5669999999999999,"5min":1.573,"15min":1.7150000000000001},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.97599999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.74,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.2170000000000001,"15min":1.1950000000000001},"min":{"1min":0.94799999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.5049999999999999,"5min":1.6559999999999999,"15min":1.7010000000000001},"last":1.121},{"interface":"front","average":{"1min":1.353,"5min":1.3140000000000001,"15min":1.2849999999999999},"min":{"1min":1.0549999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.806,"5min":1.827,"15min":1.8300000000000001},"last":1.3660000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.23,"15min":1.236},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.81100000000000005},"max":{"1min":1.4790000000000001,"5min":1.6180000000000001,"15min":1.8080000000000001},"last":1.474},{"interface":"front","average":{"1min":1.351,"5min":1.296,"15min":1.2929999999999999},"min":{"1min":1.044,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.673,"5min":1.732,"15min":1.756},"last":1.2110000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.359,"15min":1.3400000000000001},"min":{"1min":0.96499999999999997,"5min":0.91400000000000003,"15min":0.84099999999999997},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.94399999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.671}]},{"osd":4,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1950000000000001,"15min":1.1990000000000001},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.579,"5min":1.6910000000000001,"15min":1.774},"last":1.302},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.274,"15min":1.3140000000000001},"min":{"1min":0.82499999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":1.2729999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2749999999999999,"15min":1.3100000000000001},"min":{"1min":0.92900000000000005,"5min":0.75600000000000001,"15min":0.68000000000000005},"max":{"1min":1.788,"5min":1.788,"15min":1.923},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":1.03,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6839999999999999,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4159999999999999,"15min":1.4430000000000001},"min":{"1min":0.92900000000000005,"5min":0.92400000000000004,"15min":0.73799999999999999},"max":{"1min":1.839,"5min":1.8720000000000001,"15min":2.169},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.417,"15min":1.464},"min":{"1min":1.163,"5min":0.88800000000000001,"15min":0.77200000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.861}]}]},{"osd":5,"up_from":13,"seq":55834575953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307908,"kb_used_data":2428,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063932,"statfs":{"total":96636764160,"available":96321466368,"internally_reserved":0,"allocated":2486272,"data_stored":1266596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.131,"15min":1.113},"min":{"1min":0.86499999999999999,"5min":0.73099999999999998,"15min":0.66900000000000004},"max":{"1min":1.55,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.2290000000000001,"15min":1.2150000000000001},"min":{"1min":0.94199999999999995,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.907},"last":1.3220000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3300000000000001,"15min":1.325},"min":{"1min":0.82399999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6759999999999999,"5min":1.7350000000000001,"15min":1.7490000000000001},"last":1.218},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.2909999999999999,"15min":1.29},"min":{"1min":1.1220000000000001,"5min":0.91100000000000003,"15min":0.85999999999999999},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.2769999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.284,"15min":1.278},"min":{"1min":1.0569999999999999,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.5600000000000001,"5min":1.7629999999999999,"15min":2.0249999999999999},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2609999999999999,"15min":1.236},"min":{"1min":0.90800000000000003,"5min":0.88100000000000001,"15min":0.84499999999999997},"max":{"1min":1.7769999999999999,"5min":1.831,"15min":1.831},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.298,"15min":1.3120000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.643,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.393},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.353,"15min":1.321},"min":{"1min":0.995,"5min":0.97599999999999998,"15min":0.85799999999999998},"max":{"1min":1.734,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1479999999999999,"15min":1.1339999999999999},"min":{"1min":0.77800000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.452,"5min":1.8,"15min":1.8},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.294,"5min":1.3120000000000001,"15min":1.2809999999999999},"min":{"1min":0.94699999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6160000000000001,"5min":1.8640000000000001,"15min":1.869},"last":1.46}]},{"osd":6,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.91300000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.843,"5min":1.968,"15min":1.968},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4490000000000001,"15min":1.3979999999999999},"min":{"1min":1.133,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6479999999999999,"5min":1.7929999999999999,"15min":1.804},"last":1.5940000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4510000000000001,"15min":1.4159999999999999},"min":{"1min":0.92600000000000005,"5min":0.82299999999999995,"15min":0.81999999999999995},"max":{"1min":1.6870000000000001,"5min":1.9299999999999999,"15min":2.0790000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4570000000000001,"15min":1.423},"min":{"1min":1.167,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.7849999999999999,"5min":1.9770000000000001,"15min":1.99},"last":1.415}]}]},{"osd":4,"up_from":13,"seq":55834575953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":2494464,"data_stored":1267208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0780000000000001,"15min":1.111},"min":{"1min":0.96299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.47,"5min":1.5429999999999999,"15min":1.625},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.159,"15min":1.1559999999999999},"min":{"1min":0.86399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3939999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":1.248}]},{"osd":1,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.3160000000000001,"15min":1.3140000000000001},"min":{"1min":0.98599999999999999,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.716,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.026},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3109999999999999,"15min":1.319},"min":{"1min":1.071,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.575,"5min":1.6659999999999999,"15min":2.024},"last":1.4159999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.204,"15min":1.2290000000000001},"min":{"1min":0.90900000000000003,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.5629999999999999,"5min":1.617,"15min":1.73},"last":1.123},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.248,"15min":1.25},"min":{"1min":0.90000000000000002,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.7030000000000001,"5min":1.764,"15min":1.764},"last":1.569}]},{"osd":3,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.222,"15min":1.238},"min":{"1min":1.002,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.4399999999999999,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.804},{"interface":"front","average":{"1min":1.296,"5min":1.3260000000000001,"15min":1.3109999999999999},"min":{"1min":0.94899999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6579999999999999,"5min":1.825,"15min":1.825},"last":1.708}]},{"osd":5,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.171,"15min":1.157},"min":{"1min":0.81699999999999995,"5min":0.71799999999999997,"15min":0.53700000000000003},"max":{"1min":1.6259999999999999,"5min":1.8260000000000001,"15min":1.855},"last":1.744},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.1930000000000001},"min":{"1min":0.82599999999999996,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.5780000000000001,"5min":1.726,"15min":1.8200000000000001},"last":1.343}]},{"osd":6,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.367,"15min":1.3140000000000001},"min":{"1min":1.155,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.784,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.367},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1060000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7210000000000001,"5min":1.877,"15min":1.982},"last":1.97}]},{"osd":7,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4039999999999999,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.73799999999999999,"15min":0.61299999999999999},"max":{"1min":1.7829999999999999,"5min":1.8500000000000001,"15min":1.9450000000000001},"last":1.667},{"interface":"front","average":{"1min":1.528,"5min":1.5149999999999999,"15min":1.488},"min":{"1min":1.252,"5min":0.79800000000000004,"15min":0.60199999999999998},"max":{"1min":1.887,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.607}]}]},{"osd":3,"up_from":13,"seq":55834575952,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299740,"kb_used_data":2452,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072100,"statfs":{"total":96636764160,"available":96329830400,"internally_reserved":0,"allocated":2510848,"data_stored":1275276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.90100000000000002},"min":{"1min":0.63200000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.633,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.002,"15min":1.018},"min":{"1min":0.72999999999999998,"5min":0.57999999999999996,"15min":0.55600000000000005},"max":{"1min":1.379,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.80300000000000005}]},{"osd":1,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.94499999999999995,"15min":0.96599999999999997},"min":{"1min":0.73199999999999998,"5min":0.621,"15min":0.44400000000000001},"max":{"1min":1.2150000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1439999999999999,"15min":1.115},"min":{"1min":0.98399999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.724,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.024}]},{"osd":2,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1659999999999999,"15min":1.1679999999999999},"min":{"1min":0.83599999999999997,"5min":0.82299999999999995,"15min":0.63700000000000001},"max":{"1min":1.393,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.258,"15min":1.26},"min":{"1min":0.95399999999999996,"5min":0.76100000000000001,"15min":0.66600000000000004},"max":{"1min":1.6339999999999999,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.2150000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3360000000000001,"15min":1.3320000000000001},"min":{"1min":1.0629999999999999,"5min":0.82099999999999995,"15min":0.72699999999999998},"max":{"1min":1.5249999999999999,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":1.718},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.302,"15min":1.3120000000000001},"min":{"1min":1.121,"5min":0.72199999999999998,"15min":0.65200000000000002},"max":{"1min":1.5940000000000001,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.4450000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3400000000000001,"15min":1.353},"min":{"1min":0.98799999999999999,"5min":0.93799999999999994,"15min":0.60899999999999999},"max":{"1min":1.569,"5min":2.177,"15min":2.1909999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.431,"15min":1.4239999999999999},"min":{"1min":1.171,"5min":0.97299999999999998,"15min":0.82199999999999995},"max":{"1min":1.6639999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.8480000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.492,"15min":1.504},"min":{"1min":1.341,"5min":0.90200000000000002,"15min":0.79100000000000004},"max":{"1min":1.748,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4450000000000001,"15min":1.4570000000000001},"min":{"1min":1.1339999999999999,"5min":1.0329999999999999,"15min":0.80500000000000005},"max":{"1min":1.859,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":2.0270000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":1.3280000000000001,"5min":1.119,"15min":0.93100000000000005},"max":{"1min":1.97,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.887},{"interface":"front","average":{"1min":1.452,"5min":1.4790000000000001,"15min":1.502},"min":{"1min":1.1739999999999999,"5min":1.069,"15min":0.75},"max":{"1min":1.732,"5min":2.27,"15min":2.2709999999999999},"last":1.5249999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311992,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059848,"statfs":{"total":96636764160,"available":96317284352,"internally_reserved":0,"allocated":2473984,"data_stored":1265461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":0.96399999999999997},"min":{"1min":0.69299999999999995,"5min":0.61499999999999999,"15min":0.54600000000000004},"max":{"1min":1.1699999999999999,"5min":1.508,"15min":1.581},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1830000000000001,"15min":1.1499999999999999},"min":{"1min":0.879,"5min":0.81000000000000005,"15min":0.66200000000000003},"max":{"1min":1.321,"5min":1.796,"15min":1.796},"last":1.119}]},{"osd":1,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.198,"15min":1.1910000000000001},"min":{"1min":0.84699999999999998,"5min":0.83599999999999997,"15min":0.78900000000000003},"max":{"1min":1.4179999999999999,"5min":1.6279999999999999,"15min":1.7989999999999999},"last":1.228},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.163,"15min":1.127},"min":{"1min":0.754,"5min":0.754,"15min":0.621},"max":{"1min":1.3420000000000001,"5min":1.696,"15min":1.696},"last":1.258}]},{"osd":3,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.048,"15min":1.038},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4770000000000001,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1579999999999999,"15min":1.1479999999999999},"min":{"1min":0.82299999999999995,"5min":0.77000000000000002,"15min":0.59799999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.8340000000000001},"last":0.87}]},{"osd":4,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":1.022,"5min":0.95199999999999996,"15min":0.71299999999999997},"max":{"1min":1.625,"5min":1.786,"15min":1.986},"last":1.524},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4570000000000001,"15min":1.446},"min":{"1min":1.05,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.641,"5min":1.988,"15min":2.0070000000000001},"last":1.345}]},{"osd":5,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.4359999999999999,"15min":1.417},"min":{"1min":1.0880000000000001,"5min":1.044,"15min":0.84999999999999998},"max":{"1min":1.732,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4399999999999999,"15min":1.4019999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.6439999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.4610000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4139999999999999,"15min":1.397},"min":{"1min":1.0900000000000001,"5min":1.0680000000000001,"15min":0.95799999999999996},"max":{"1min":1.881,"5min":2.077,"15min":2.081},"last":1.377},{"interface":"front","average":{"1min":1.49,"5min":1.534,"15min":1.5209999999999999},"min":{"1min":1.1599999999999999,"5min":1.071,"15min":1.034},"max":{"1min":1.883,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5649999999999999,"15min":1.544},"min":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":0.89600000000000002},"max":{"1min":1.7529999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.5580000000000001,"15min":1.532},"min":{"1min":1.073,"5min":1.073,"15min":0.90300000000000002},"max":{"1min":1.948,"5min":2.1659999999999999,"15min":2.4830000000000001},"last":1.4890000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575952,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312020,"kb_used_data":2444,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059820,"statfs":{"total":96636764160,"available":96317255680,"internally_reserved":0,"allocated":2502656,"data_stored":1274700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.6559999999999999,"15min":1.6579999999999999},"last":1.351},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.042,"15min":1.042},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4730000000000001,"5min":1.6020000000000001,"15min":1.9119999999999999},"last":1.46}]},{"osd":2,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.081,"15min":1.073},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.325,"5min":1.5449999999999999,"15min":1.6830000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.115,"15min":1.165},"min":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4710000000000001,"5min":1.6930000000000001,"15min":1.7949999999999999},"last":0.73999999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.3029999999999999,"15min":1.2949999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5369999999999999,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.262},{"interface":"front","average":{"1min":1.038,"5min":1.1060000000000001,"15min":1.1100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.7310000000000001,"5min":1.7829999999999999,"15min":1.8520000000000001},"last":0.80000000000000004}]},{"osd":4,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3,"15min":1.3340000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.552,"5min":1.754,"15min":2.1459999999999999},"last":1.903},{"interface":"front","average":{"1min":1.274,"5min":1.393,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":2.0110000000000001},"last":1.577}]},{"osd":5,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.335,"15min":1.3480000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.82099999999999995},"max":{"1min":1.72,"5min":1.833,"15min":1.9650000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.393,"15min":1.45},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.6180000000000001,"5min":1.8999999999999999,"15min":2.323},"last":2.0579999999999998}]},{"osd":6,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.393,"15min":1.4159999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.6479999999999999,"5min":1.972,"15min":2.2160000000000002},"last":1.167},{"interface":"front","average":{"1min":1.365,"5min":1.502,"15min":1.526},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":0.95599999999999996},"max":{"1min":1.671,"5min":1.861,"15min":2.3109999999999999},"last":2.1629999999999998}]},{"osd":7,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.51,"15min":1.5409999999999999},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":2.0489999999999999,"5min":2.2010000000000001,"15min":2.427},"last":1.661},{"interface":"front","average":{"1min":1.335,"5min":1.486,"15min":1.502},"min":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":0.97799999999999998},"max":{"1min":1.952,"5min":2.0670000000000002,"15min":2.3170000000000002},"last":1.633}]}]},{"osd":0,"up_from":13,"seq":55834575952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300292,"kb_used_data":3004,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071548,"statfs":{"total":96636764160,"available":96329265152,"internally_reserved":0,"allocated":3076096,"data_stored":1855933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.71299999999999997,"5min":0.629,"15min":0.439},"max":{"1min":1.29,"5min":1.4530000000000001,"15min":1.6419999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0089999999999999},"min":{"1min":0.77800000000000002,"5min":0.75900000000000001,"15min":0.52200000000000002},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.093}]},{"osd":2,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1830000000000001,"15min":1.145},"min":{"1min":0.83299999999999996,"5min":0.68300000000000005,"15min":0.57099999999999995},"max":{"1min":1.6359999999999999,"5min":1.7410000000000001,"15min":1.8400000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.29,"15min":1.278},"min":{"1min":0.99199999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.655,"5min":1.8440000000000001,"15min":1.8500000000000001},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.081,"15min":1.0640000000000001},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.65300000000000002},"max":{"1min":1.601,"5min":1.601,"15min":1.6359999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.2809999999999999,"15min":1.2609999999999999},"min":{"1min":1.012,"5min":0.88500000000000001,"15min":0.55300000000000005},"max":{"1min":1.601,"5min":1.752,"15min":1.968},"last":0.89300000000000002}]},{"osd":4,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":1.175,"5min":1.0600000000000001,"15min":0.69499999999999995},"max":{"1min":1.885,"5min":1.885,"15min":2.0209999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.379,"15min":1.3360000000000001},"min":{"1min":1.0860000000000001,"5min":1.034,"15min":0.71999999999999997},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.488,"15min":1.448},"min":{"1min":1.2689999999999999,"5min":1.1499999999999999,"15min":0.90300000000000002},"max":{"1min":1.7969999999999999,"5min":2.0249999999999999,"15min":2.0779999999999998},"last":1.133},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.48,"15min":1.462},"min":{"1min":1.2969999999999999,"5min":1.079,"15min":0.98699999999999999},"max":{"1min":1.853,"5min":1.901,"15min":2.0539999999999998},"last":1.4410000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.4690000000000001,"15min":1.429},"min":{"1min":1.1699999999999999,"5min":1.0780000000000001,"15min":0.95299999999999996},"max":{"1min":1.8779999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.5640000000000001,"15min":1.5229999999999999},"min":{"1min":1.278,"5min":1.1559999999999999,"15min":0.93899999999999995},"max":{"1min":1.9159999999999999,"5min":2.0390000000000001,"15min":2.1669999999999998},"last":1.056}]},{"osd":7,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.579,"15min":1.538},"min":{"1min":1.194,"5min":1.0469999999999999,"15min":0.97899999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.532},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.54,"15min":1.516},"min":{"1min":1.248,"5min":1.119,"15min":1.008},"max":{"1min":2.0219999999999998,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.3540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:11:44.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:11:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.998+0000 7f3396f45700 1 Processor -- start 2023-12-19T16:11:45.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:44.998+0000 7f3396f45700 1 -- start start 2023-12-19T16:11:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3396f45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33900cc3b0 0x7f33900cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3396f45700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33900cd5a0 con 0x7f3390156c90 2023-12-19T16:11:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3396f45700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33900cc990 con 0x7f3390157060 2023-12-19T16:11:45.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3396f45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3390157470 con 0x7f33900cc3b0 2023-12-19T16:11:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3394ce1700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3390156c90 0x7f33900c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50486/0 (socket says 172.21.15.50:50486) 2023-12-19T16:11:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f3394ce1700 1 -- 172.21.15.50:0/3231353740 learned_addr learned my addr 172.21.15.50:0/3231353740 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338ffff700 1 --2- 172.21.15.50:0/3231353740 >> v2:172.21.15.125:3300/0 conn(0x7f33900cc3b0 0x7f33900cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338f7fe700 1 -- 172.21.15.50:0/3231353740 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599139354 0 0) 0x7f33900cd5a0 con 0x7f3390156c90 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338f7fe700 1 -- 172.21.15.50:0/3231353740 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380003570 con 0x7f3390156c90 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338ffff700 1 -- 172.21.15.50:0/3231353740 >> v1:172.21.15.125:6789/0 conn(0x7f3390157060 legacy=0x7f339015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338ffff700 1 -- 172.21.15.50:0/3231353740 >> v1:172.21.15.50:6789/0 conn(0x7f3390156c90 legacy=0x7f33900c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.002+0000 7f338ffff700 1 -- 172.21.15.50:0/3231353740 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33900ccc70 con 0x7f33900cc3b0 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f338ffff700 1 --2- 172.21.15.50:0/3231353740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f33900cda30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f337c0098e0 tx=0x7f337c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f54b39fc6682439 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f338f7fe700 1 -- 172.21.15.50:0/3231353740 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00b640 con 0x7f33900cc3b0 2023-12-19T16:11:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f338f7fe700 1 -- 172.21.15.50:0/3231353740 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c00b7a0 con 0x7f33900cc3b0 2023-12-19T16:11:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f338f7fe700 1 -- 172.21.15.50:0/3231353740 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00bb40 con 0x7f33900cc3b0 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 -- 172.21.15.50:0/3231353740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 msgr2=0x7f33900cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 --2- 172.21.15.50:0/3231353740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f33900cda30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f337c0098e0 tx=0x7f337c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 -- 172.21.15.50:0/3231353740 shutdown_connections 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 --2- 172.21.15.50:0/3231353740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f33900cda30 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 -- 172.21.15.50:0/3231353740 >> 172.21.15.50:0/3231353740 conn(0x7f339006c7c0 msgr2=0x7f339006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 -- 172.21.15.50:0/3231353740 shutdown_connections 2023-12-19T16:11:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.006+0000 7f3396f45700 1 -- 172.21.15.50:0/3231353740 wait complete. 2023-12-19T16:11:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 Processor -- start 2023-12-19T16:11:45.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- start start 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f3390159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 0x7f33900670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339015a470 con 0x7f3390157060 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339015a1d0 con 0x7f33900cc3b0 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3390157750 con 0x7f3390156c90 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35242/0 (socket says 172.21.15.50:35242) 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 -- 172.21.15.50:0/655398480 learned_addr learned my addr 172.21.15.50:0/655398480 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f338ffff700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 0x7f33900670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3394ce1700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f3390159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 -- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 msgr2=0x7f33900670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 0x7f33900670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 -- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 msgr2=0x7f3390159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f3390159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 -- 172.21.15.50:0/655398480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c009420 con 0x7f3390157060 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f338ffff700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 0x7f33900670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3394ce1700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f3390159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f33954e2700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f338800c840 tx=0x7f3388001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388012630 con 0x7f3390157060 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33880127b0 con 0x7f3390157060 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3390157a30 con 0x7f3390157060 2023-12-19T16:11:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388012b70 con 0x7f3390157060 2023-12-19T16:11:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.010+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3390157ff0 con 0x7f3390157060 2023-12-19T16:11:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.014+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3390158210 con 0x7f3390157060 2023-12-19T16:11:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.014+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3388003030 con 0x7f3390157060 2023-12-19T16:11:45.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.014+0000 7f338cff9700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 0x7f3374062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.014+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f338807cb10 con 0x7f3390157060 2023-12-19T16:11:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.018+0000 7f3394ce1700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 0x7f3374062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.018+0000 7f3394ce1700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 0x7f3374062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3384007fd0 tx=0x7f3384009040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.018+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f338804cdc0 con 0x7f3390157060 2023-12-19T16:11:45.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.186+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3390071120 con 0x7f3374060670 2023-12-19T16:11:45.212 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.206+0000 7f338cff9700 1 -- 172.21.15.50:0/655398480 <== mgr.9678 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286654 (secure 0 0 0) 0x7f3390071120 con 0x7f3374060670 2023-12-19T16:11:45.216 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 msgr2=0x7f3374062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 0x7f3374062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3384007fd0 tx=0x7f3384009040 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 msgr2=0x7f3390067620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f338800c840 tx=0x7f3388001670 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 shutdown_connections 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3374060670 0x7f3374062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3390157060 0x7f3390067620 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3390156c90 0x7f33900670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 --2- 172.21.15.50:0/655398480 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33900cc3b0 0x7f3390159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.214+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 >> 172.21.15.50:0/655398480 conn(0x7f339006c7c0 msgr2=0x7f339006cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.218+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 shutdown_connections 2023-12-19T16:11:45.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.218+0000 7f3396f45700 1 -- 172.21.15.50:0/655398480 wait complete. 2023-12-19T16:11:45.224 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:11:45.243 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T16:11:44.019926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460344,"kb_used_data":20600,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752514376,"statfs":{"total":773094113280,"available":770574721024,"internally_reserved":0,"allocated":21094400,"data_stored":11328486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.574872"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243354+0000","last_change":"2023-12-19T16:11:40.237609+0000","last_active":"2023-12-19T16:11:40.243354+0000","last_peered":"2023-12-19T16:11:40.243354+0000","last_clean":"2023-12-19T16:11:40.243354+0000","last_became_active":"2023-12-19T16:11:40.233513+0000","last_became_peered":"2023-12-19T16:11:40.233513+0000","last_unstale":"2023-12-19T16:11:40.243354+0000","last_undegraded":"2023-12-19T16:11:40.243354+0000","last_fullsized":"2023-12-19T16:11:40.243354+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:13.538074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.236119+0000","last_change":"2023-12-19T16:11:40.236119+0000","last_active":"2023-12-19T16:11:40.236119+0000","last_peered":"2023-12-19T16:11:40.236119+0000","last_clean":"2023-12-19T16:11:40.236119+0000","last_became_active":"2023-12-19T16:11:40.235845+0000","last_became_peered":"2023-12-19T16:11:40.235845+0000","last_unstale":"2023-12-19T16:11:40.236119+0000","last_undegraded":"2023-12-19T16:11:40.236119+0000","last_fullsized":"2023-12-19T16:11:40.236119+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:10.177568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243472+0000","last_change":"2023-12-19T16:11:40.239764+0000","last_active":"2023-12-19T16:11:40.243472+0000","last_peered":"2023-12-19T16:11:40.243472+0000","last_clean":"2023-12-19T16:11:40.243472+0000","last_became_active":"2023-12-19T16:11:40.239436+0000","last_became_peered":"2023-12-19T16:11:40.239436+0000","last_unstale":"2023-12-19T16:11:40.243472+0000","last_undegraded":"2023-12-19T16:11:40.243472+0000","last_fullsized":"2023-12-19T16:11:40.243472+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:02:07.204644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243545+0000","last_change":"2023-12-19T16:11:40.241007+0000","last_active":"2023-12-19T16:11:40.243545+0000","last_peered":"2023-12-19T16:11:40.243545+0000","last_clean":"2023-12-19T16:11:40.243545+0000","last_became_active":"2023-12-19T16:11:40.239049+0000","last_became_peered":"2023-12-19T16:11:40.239049+0000","last_unstale":"2023-12-19T16:11:40.243545+0000","last_undegraded":"2023-12-19T16:11:40.243545+0000","last_fullsized":"2023-12-19T16:11:40.243545+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238158+0000","last_change":"2023-12-19T16:11:40.238158+0000","last_active":"2023-12-19T16:11:40.238158+0000","last_peered":"2023-12-19T16:11:40.238158+0000","last_clean":"2023-12-19T16:11:40.238158+0000","last_became_active":"2023-12-19T16:11:40.233495+0000","last_became_peered":"2023-12-19T16:11:40.233495+0000","last_unstale":"2023-12-19T16:11:40.238158+0000","last_undegraded":"2023-12-19T16:11:40.238158+0000","last_fullsized":"2023-12-19T16:11:40.238158+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:52:43.978826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240234+0000","last_change":"2023-12-19T16:11:40.240234+0000","last_active":"2023-12-19T16:11:40.240234+0000","last_peered":"2023-12-19T16:11:40.240234+0000","last_clean":"2023-12-19T16:11:40.240234+0000","last_became_active":"2023-12-19T16:11:40.239654+0000","last_became_peered":"2023-12-19T16:11:40.239654+0000","last_unstale":"2023-12-19T16:11:40.240234+0000","last_undegraded":"2023-12-19T16:11:40.240234+0000","last_fullsized":"2023-12-19T16:11:40.240234+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238540+0000","last_change":"2023-12-19T16:11:40.232578+0000","last_active":"2023-12-19T16:11:40.238540+0000","last_peered":"2023-12-19T16:11:40.238540+0000","last_clean":"2023-12-19T16:11:40.238540+0000","last_became_active":"2023-12-19T16:11:40.232277+0000","last_became_peered":"2023-12-19T16:11:40.232277+0000","last_unstale":"2023-12-19T16:11:40.238540+0000","last_undegraded":"2023-12-19T16:11:40.238540+0000","last_fullsized":"2023-12-19T16:11:40.238540+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:39.270097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237990+0000","last_change":"2023-12-19T16:11:40.228738+0000","last_active":"2023-12-19T16:11:40.237990+0000","last_peered":"2023-12-19T16:11:40.237990+0000","last_clean":"2023-12-19T16:11:40.237990+0000","last_became_active":"2023-12-19T16:11:40.228429+0000","last_became_peered":"2023-12-19T16:11:40.228429+0000","last_unstale":"2023-12-19T16:11:40.237990+0000","last_undegraded":"2023-12-19T16:11:40.237990+0000","last_fullsized":"2023-12-19T16:11:40.237990+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:10.177568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237905+0000","last_change":"2023-12-19T16:11:40.226675+0000","last_active":"2023-12-19T16:11:40.237905+0000","last_peered":"2023-12-19T16:11:40.237905+0000","last_clean":"2023-12-19T16:11:40.237905+0000","last_became_active":"2023-12-19T16:11:40.226211+0000","last_became_peered":"2023-12-19T16:11:40.226211+0000","last_unstale":"2023-12-19T16:11:40.237905+0000","last_undegraded":"2023-12-19T16:11:40.237905+0000","last_fullsized":"2023-12-19T16:11:40.237905+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:02:07.204644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238747+0000","last_change":"2023-12-19T16:11:40.229195+0000","last_active":"2023-12-19T16:11:40.238747+0000","last_peered":"2023-12-19T16:11:40.238747+0000","last_clean":"2023-12-19T16:11:40.238747+0000","last_became_active":"2023-12-19T16:11:40.228564+0000","last_became_peered":"2023-12-19T16:11:40.228564+0000","last_unstale":"2023-12-19T16:11:40.238747+0000","last_undegraded":"2023-12-19T16:11:40.238747+0000","last_fullsized":"2023-12-19T16:11:40.238747+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:56.472818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.229730+0000","last_change":"2023-12-19T16:11:40.229730+0000","last_active":"2023-12-19T16:11:40.229730+0000","last_peered":"2023-12-19T16:11:40.229730+0000","last_clean":"2023-12-19T16:11:40.229730+0000","last_became_active":"2023-12-19T16:11:40.229438+0000","last_became_peered":"2023-12-19T16:11:40.229438+0000","last_unstale":"2023-12-19T16:11:40.229730+0000","last_undegraded":"2023-12-19T16:11:40.229730+0000","last_fullsized":"2023-12-19T16:11:40.229730+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:57:58.106053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240305+0000","last_change":"2023-12-19T16:11:40.240305+0000","last_active":"2023-12-19T16:11:40.240305+0000","last_peered":"2023-12-19T16:11:40.240305+0000","last_clean":"2023-12-19T16:11:40.240305+0000","last_became_active":"2023-12-19T16:11:40.240016+0000","last_became_peered":"2023-12-19T16:11:40.240016+0000","last_unstale":"2023-12-19T16:11:40.240305+0000","last_undegraded":"2023-12-19T16:11:40.240305+0000","last_fullsized":"2023-12-19T16:11:40.240305+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:15.986396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240650+0000","last_change":"2023-12-19T16:11:40.232463+0000","last_active":"2023-12-19T16:11:40.240650+0000","last_peered":"2023-12-19T16:11:40.240650+0000","last_clean":"2023-12-19T16:11:40.240650+0000","last_became_active":"2023-12-19T16:11:40.231321+0000","last_became_peered":"2023-12-19T16:11:40.231321+0000","last_unstale":"2023-12-19T16:11:40.240650+0000","last_undegraded":"2023-12-19T16:11:40.240650+0000","last_fullsized":"2023-12-19T16:11:40.240650+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:51.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.240456+0000","last_change":"2023-12-19T16:11:40.240456+0000","last_active":"2023-12-19T16:11:40.240456+0000","last_peered":"2023-12-19T16:11:40.240456+0000","last_clean":"2023-12-19T16:11:40.240456+0000","last_became_active":"2023-12-19T16:11:40.240109+0000","last_became_peered":"2023-12-19T16:11:40.240109+0000","last_unstale":"2023-12-19T16:11:40.240456+0000","last_undegraded":"2023-12-19T16:11:40.240456+0000","last_fullsized":"2023-12-19T16:11:40.240456+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:43:20.946037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.237031+0000","last_change":"2023-12-19T16:11:40.237031+0000","last_active":"2023-12-19T16:11:40.237031+0000","last_peered":"2023-12-19T16:11:40.237031+0000","last_clean":"2023-12-19T16:11:40.237031+0000","last_became_active":"2023-12-19T16:11:40.236726+0000","last_became_peered":"2023-12-19T16:11:40.236726+0000","last_unstale":"2023-12-19T16:11:40.237031+0000","last_undegraded":"2023-12-19T16:11:40.237031+0000","last_fullsized":"2023-12-19T16:11:40.237031+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:33.279979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238690+0000","last_change":"2023-12-19T16:11:40.232022+0000","last_active":"2023-12-19T16:11:40.238690+0000","last_peered":"2023-12-19T16:11:40.238690+0000","last_clean":"2023-12-19T16:11:40.238690+0000","last_became_active":"2023-12-19T16:11:40.231203+0000","last_became_peered":"2023-12-19T16:11:40.231203+0000","last_unstale":"2023-12-19T16:11:40.238690+0000","last_undegraded":"2023-12-19T16:11:40.238690+0000","last_fullsized":"2023-12-19T16:11:40.238690+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239099+0000","last_change":"2023-12-19T16:11:40.235320+0000","last_active":"2023-12-19T16:11:40.239099+0000","last_peered":"2023-12-19T16:11:40.239099+0000","last_clean":"2023-12-19T16:11:40.239099+0000","last_became_active":"2023-12-19T16:11:40.235026+0000","last_became_peered":"2023-12-19T16:11:40.235026+0000","last_unstale":"2023-12-19T16:11:40.239099+0000","last_undegraded":"2023-12-19T16:11:40.239099+0000","last_fullsized":"2023-12-19T16:11:40.239099+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.222539+0000","last_change":"2023-12-19T16:11:40.222539+0000","last_active":"2023-12-19T16:11:40.222539+0000","last_peered":"2023-12-19T16:11:40.222539+0000","last_clean":"2023-12-19T16:11:40.222539+0000","last_became_active":"2023-12-19T16:11:40.222244+0000","last_became_peered":"2023-12-19T16:11:40.222244+0000","last_unstale":"2023-12-19T16:11:40.222539+0000","last_undegraded":"2023-12-19T16:11:40.222539+0000","last_fullsized":"2023-12-19T16:11:40.222539+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:05:54.627131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235050+0000","last_change":"2023-12-19T16:11:40.235050+0000","last_active":"2023-12-19T16:11:40.235050+0000","last_peered":"2023-12-19T16:11:40.235050+0000","last_clean":"2023-12-19T16:11:40.235050+0000","last_became_active":"2023-12-19T16:11:40.234733+0000","last_became_peered":"2023-12-19T16:11:40.234733+0000","last_unstale":"2023-12-19T16:11:40.235050+0000","last_undegraded":"2023-12-19T16:11:40.235050+0000","last_fullsized":"2023-12-19T16:11:40.235050+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:06:15.097643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231747+0000","last_change":"2023-12-19T16:11:40.231747+0000","last_active":"2023-12-19T16:11:40.231747+0000","last_peered":"2023-12-19T16:11:40.231747+0000","last_clean":"2023-12-19T16:11:40.231747+0000","last_became_active":"2023-12-19T16:11:40.231437+0000","last_became_peered":"2023-12-19T16:11:40.231437+0000","last_unstale":"2023-12-19T16:11:40.231747+0000","last_undegraded":"2023-12-19T16:11:40.231747+0000","last_fullsized":"2023-12-19T16:11:40.231747+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:14:03.883989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.241761+0000","last_change":"2023-12-19T16:11:40.235333+0000","last_active":"2023-12-19T16:11:40.241761+0000","last_peered":"2023-12-19T16:11:40.241761+0000","last_clean":"2023-12-19T16:11:40.241761+0000","last_became_active":"2023-12-19T16:11:40.235027+0000","last_became_peered":"2023-12-19T16:11:40.235027+0000","last_unstale":"2023-12-19T16:11:40.241761+0000","last_undegraded":"2023-12-19T16:11:40.241761+0000","last_fullsized":"2023-12-19T16:11:40.241761+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:22.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235640+0000","last_change":"2023-12-19T16:11:40.235640+0000","last_active":"2023-12-19T16:11:40.235640+0000","last_peered":"2023-12-19T16:11:40.235640+0000","last_clean":"2023-12-19T16:11:40.235640+0000","last_became_active":"2023-12-19T16:11:40.235335+0000","last_became_peered":"2023-12-19T16:11:40.235335+0000","last_unstale":"2023-12-19T16:11:40.235640+0000","last_undegraded":"2023-12-19T16:11:40.235640+0000","last_fullsized":"2023-12-19T16:11:40.235640+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:51.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.233236+0000","last_change":"2023-12-19T16:11:40.233236+0000","last_active":"2023-12-19T16:11:40.233236+0000","last_peered":"2023-12-19T16:11:40.233236+0000","last_clean":"2023-12-19T16:11:40.233236+0000","last_became_active":"2023-12-19T16:11:40.232816+0000","last_became_peered":"2023-12-19T16:11:40.232816+0000","last_unstale":"2023-12-19T16:11:40.233236+0000","last_undegraded":"2023-12-19T16:11:40.233236+0000","last_fullsized":"2023-12-19T16:11:40.233236+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:22.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238202+0000","last_change":"2023-12-19T16:11:40.229992+0000","last_active":"2023-12-19T16:11:40.238202+0000","last_peered":"2023-12-19T16:11:40.238202+0000","last_clean":"2023-12-19T16:11:40.238202+0000","last_became_active":"2023-12-19T16:11:40.229513+0000","last_became_peered":"2023-12-19T16:11:40.229513+0000","last_unstale":"2023-12-19T16:11:40.238202+0000","last_undegraded":"2023-12-19T16:11:40.238202+0000","last_fullsized":"2023-12-19T16:11:40.238202+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.241020+0000","last_change":"2023-12-19T16:11:40.234939+0000","last_active":"2023-12-19T16:11:40.241020+0000","last_peered":"2023-12-19T16:11:40.241020+0000","last_clean":"2023-12-19T16:11:40.241020+0000","last_became_active":"2023-12-19T16:11:40.234661+0000","last_became_peered":"2023-12-19T16:11:40.234661+0000","last_unstale":"2023-12-19T16:11:40.241020+0000","last_undegraded":"2023-12-19T16:11:40.241020+0000","last_fullsized":"2023-12-19T16:11:40.241020+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:22.168120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234741+0000","last_change":"2023-12-19T16:11:40.234741+0000","last_active":"2023-12-19T16:11:40.234741+0000","last_peered":"2023-12-19T16:11:40.234741+0000","last_clean":"2023-12-19T16:11:40.234741+0000","last_became_active":"2023-12-19T16:11:40.234380+0000","last_became_peered":"2023-12-19T16:11:40.234380+0000","last_unstale":"2023-12-19T16:11:40.234741+0000","last_undegraded":"2023-12-19T16:11:40.234741+0000","last_fullsized":"2023-12-19T16:11:40.234741+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:12:30.002649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.228674+0000","last_change":"2023-12-19T16:11:40.228674+0000","last_active":"2023-12-19T16:11:40.228674+0000","last_peered":"2023-12-19T16:11:40.228674+0000","last_clean":"2023-12-19T16:11:40.228674+0000","last_became_active":"2023-12-19T16:11:40.228349+0000","last_became_peered":"2023-12-19T16:11:40.228349+0000","last_unstale":"2023-12-19T16:11:40.228674+0000","last_undegraded":"2023-12-19T16:11:40.228674+0000","last_fullsized":"2023-12-19T16:11:40.228674+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:59.452795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243162+0000","last_change":"2023-12-19T16:11:40.237619+0000","last_active":"2023-12-19T16:11:40.243162+0000","last_peered":"2023-12-19T16:11:40.243162+0000","last_clean":"2023-12-19T16:11:40.243162+0000","last_became_active":"2023-12-19T16:11:40.237303+0000","last_became_peered":"2023-12-19T16:11:40.237303+0000","last_unstale":"2023-12-19T16:11:40.243162+0000","last_undegraded":"2023-12-19T16:11:40.243162+0000","last_fullsized":"2023-12-19T16:11:40.243162+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:43:20.946037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.223525+0000","last_change":"2023-12-19T16:11:40.223525+0000","last_active":"2023-12-19T16:11:40.223525+0000","last_peered":"2023-12-19T16:11:40.223525+0000","last_clean":"2023-12-19T16:11:40.223525+0000","last_became_active":"2023-12-19T16:11:40.223177+0000","last_became_peered":"2023-12-19T16:11:40.223177+0000","last_unstale":"2023-12-19T16:11:40.223525+0000","last_undegraded":"2023-12-19T16:11:40.223525+0000","last_fullsized":"2023-12-19T16:11:40.223525+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:56.472818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.236827+0000","last_change":"2023-12-19T16:11:40.236827+0000","last_active":"2023-12-19T16:11:40.236827+0000","last_peered":"2023-12-19T16:11:40.236827+0000","last_clean":"2023-12-19T16:11:40.236827+0000","last_became_active":"2023-12-19T16:11:40.236119+0000","last_became_peered":"2023-12-19T16:11:40.236119+0000","last_unstale":"2023-12-19T16:11:40.236827+0000","last_undegraded":"2023-12-19T16:11:40.236827+0000","last_fullsized":"2023-12-19T16:11:40.236827+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232773+0000","last_change":"2023-12-19T16:11:40.232773+0000","last_active":"2023-12-19T16:11:40.232773+0000","last_peered":"2023-12-19T16:11:40.232773+0000","last_clean":"2023-12-19T16:11:40.232773+0000","last_became_active":"2023-12-19T16:11:40.232344+0000","last_became_peered":"2023-12-19T16:11:40.232344+0000","last_unstale":"2023-12-19T16:11:40.232773+0000","last_undegraded":"2023-12-19T16:11:40.232773+0000","last_fullsized":"2023-12-19T16:11:40.232773+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:49.544222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.224820+0000","last_change":"2023-12-19T16:11:40.224820+0000","last_active":"2023-12-19T16:11:40.224820+0000","last_peered":"2023-12-19T16:11:40.224820+0000","last_clean":"2023-12-19T16:11:40.224820+0000","last_became_active":"2023-12-19T16:11:40.224516+0000","last_became_peered":"2023-12-19T16:11:40.224516+0000","last_unstale":"2023-12-19T16:11:40.224820+0000","last_undegraded":"2023-12-19T16:11:40.224820+0000","last_fullsized":"2023-12-19T16:11:40.224820+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:26.302439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239202+0000","last_change":"2023-12-19T16:11:40.239202+0000","last_active":"2023-12-19T16:11:40.239202+0000","last_peered":"2023-12-19T16:11:40.239202+0000","last_clean":"2023-12-19T16:11:40.239202+0000","last_became_active":"2023-12-19T16:11:40.238922+0000","last_became_peered":"2023-12-19T16:11:40.238922+0000","last_unstale":"2023-12-19T16:11:40.239202+0000","last_undegraded":"2023-12-19T16:11:40.239202+0000","last_fullsized":"2023-12-19T16:11:40.239202+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:51.074591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"198'4","reported_seq":14,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487567+0000","last_change":"2023-12-19T16:11:39.486904+0000","last_active":"2023-12-19T16:11:40.487567+0000","last_peered":"2023-12-19T16:11:40.487567+0000","last_clean":"2023-12-19T16:11:40.487567+0000","last_became_active":"2023-12-19T16:11:39.486470+0000","last_became_peered":"2023-12-19T16:11:39.486470+0000","last_unstale":"2023-12-19T16:11:40.487567+0000","last_undegraded":"2023-12-19T16:11:40.487567+0000","last_fullsized":"2023-12-19T16:11:40.487567+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:01:46.149503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238268+0000","last_change":"2023-12-19T16:11:40.229804+0000","last_active":"2023-12-19T16:11:40.238268+0000","last_peered":"2023-12-19T16:11:40.238268+0000","last_clean":"2023-12-19T16:11:40.238268+0000","last_became_active":"2023-12-19T16:11:40.227247+0000","last_became_peered":"2023-12-19T16:11:40.227247+0000","last_unstale":"2023-12-19T16:11:40.238268+0000","last_undegraded":"2023-12-19T16:11:40.238268+0000","last_fullsized":"2023-12-19T16:11:40.238268+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:45:29.365616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227063+0000","last_change":"2023-12-19T16:11:39.504070+0000","last_active":"2023-12-19T16:11:40.227063+0000","last_peered":"2023-12-19T16:11:40.227063+0000","last_clean":"2023-12-19T16:11:40.227063+0000","last_became_active":"2023-12-19T16:11:39.503756+0000","last_became_peered":"2023-12-19T16:11:39.503756+0000","last_unstale":"2023-12-19T16:11:40.227063+0000","last_undegraded":"2023-12-19T16:11:40.227063+0000","last_fullsized":"2023-12-19T16:11:40.227063+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:56.677127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234367+0000","last_change":"2023-12-19T16:11:40.234367+0000","last_active":"2023-12-19T16:11:40.234367+0000","last_peered":"2023-12-19T16:11:40.234367+0000","last_clean":"2023-12-19T16:11:40.234367+0000","last_became_active":"2023-12-19T16:11:40.233920+0000","last_became_peered":"2023-12-19T16:11:40.233920+0000","last_unstale":"2023-12-19T16:11:40.234367+0000","last_undegraded":"2023-12-19T16:11:40.234367+0000","last_fullsized":"2023-12-19T16:11:40.234367+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487036+0000","last_change":"2023-12-19T16:11:39.488578+0000","last_active":"2023-12-19T16:11:40.487036+0000","last_peered":"2023-12-19T16:11:40.487036+0000","last_clean":"2023-12-19T16:11:40.487036+0000","last_became_active":"2023-12-19T16:11:39.487481+0000","last_became_peered":"2023-12-19T16:11:39.487481+0000","last_unstale":"2023-12-19T16:11:40.487036+0000","last_undegraded":"2023-12-19T16:11:40.487036+0000","last_fullsized":"2023-12-19T16:11:40.487036+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:45:28.351088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238073+0000","last_change":"2023-12-19T16:11:40.238073+0000","last_active":"2023-12-19T16:11:40.238073+0000","last_peered":"2023-12-19T16:11:40.238073+0000","last_clean":"2023-12-19T16:11:40.238073+0000","last_became_active":"2023-12-19T16:11:40.237804+0000","last_became_peered":"2023-12-19T16:11:40.237804+0000","last_unstale":"2023-12-19T16:11:40.238073+0000","last_undegraded":"2023-12-19T16:11:40.238073+0000","last_fullsized":"2023-12-19T16:11:40.238073+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:36.484146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.490051+0000","last_change":"2023-12-19T16:11:39.486815+0000","last_active":"2023-12-19T16:11:40.490051+0000","last_peered":"2023-12-19T16:11:40.490051+0000","last_clean":"2023-12-19T16:11:40.490051+0000","last_became_active":"2023-12-19T16:11:39.486146+0000","last_became_peered":"2023-12-19T16:11:39.486146+0000","last_unstale":"2023-12-19T16:11:40.490051+0000","last_undegraded":"2023-12-19T16:11:40.490051+0000","last_fullsized":"2023-12-19T16:11:40.490051+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:22:58.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239165+0000","last_change":"2023-12-19T16:11:40.233273+0000","last_active":"2023-12-19T16:11:40.239165+0000","last_peered":"2023-12-19T16:11:40.239165+0000","last_clean":"2023-12-19T16:11:40.239165+0000","last_became_active":"2023-12-19T16:11:40.232947+0000","last_became_peered":"2023-12-19T16:11:40.232947+0000","last_unstale":"2023-12-19T16:11:40.239165+0000","last_undegraded":"2023-12-19T16:11:40.239165+0000","last_fullsized":"2023-12-19T16:11:40.239165+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"198'1","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.491376+0000","last_change":"2023-12-19T16:11:39.505963+0000","last_active":"2023-12-19T16:11:40.491376+0000","last_peered":"2023-12-19T16:11:40.491376+0000","last_clean":"2023-12-19T16:11:40.491376+0000","last_became_active":"2023-12-19T16:11:39.505662+0000","last_became_peered":"2023-12-19T16:11:39.505662+0000","last_unstale":"2023-12-19T16:11:40.491376+0000","last_undegraded":"2023-12-19T16:11:40.491376+0000","last_fullsized":"2023-12-19T16:11:40.491376+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:11.151376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.221287+0000","last_change":"2023-12-19T16:11:40.221287+0000","last_active":"2023-12-19T16:11:40.221287+0000","last_peered":"2023-12-19T16:11:40.221287+0000","last_clean":"2023-12-19T16:11:40.221287+0000","last_became_active":"2023-12-19T16:11:40.220735+0000","last_became_peered":"2023-12-19T16:11:40.220735+0000","last_unstale":"2023-12-19T16:11:40.221287+0000","last_undegraded":"2023-12-19T16:11:40.221287+0000","last_fullsized":"2023-12-19T16:11:40.221287+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:37:19.405072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226701+0000","last_change":"2023-12-19T16:11:39.501153+0000","last_active":"2023-12-19T16:11:40.226701+0000","last_peered":"2023-12-19T16:11:40.226701+0000","last_clean":"2023-12-19T16:11:40.226701+0000","last_became_active":"2023-12-19T16:11:39.500853+0000","last_became_peered":"2023-12-19T16:11:39.500853+0000","last_unstale":"2023-12-19T16:11:40.226701+0000","last_undegraded":"2023-12-19T16:11:40.226701+0000","last_fullsized":"2023-12-19T16:11:40.226701+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:36:50.060063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.228115+0000","last_change":"2023-12-19T16:11:40.228115+0000","last_active":"2023-12-19T16:11:40.228115+0000","last_peered":"2023-12-19T16:11:40.228115+0000","last_clean":"2023-12-19T16:11:40.228115+0000","last_became_active":"2023-12-19T16:11:40.227810+0000","last_became_peered":"2023-12-19T16:11:40.227810+0000","last_unstale":"2023-12-19T16:11:40.228115+0000","last_undegraded":"2023-12-19T16:11:40.228115+0000","last_fullsized":"2023-12-19T16:11:40.228115+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:39:33.288888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.499192+0000","last_change":"2023-12-19T16:11:39.496611+0000","last_active":"2023-12-19T16:11:39.499192+0000","last_peered":"2023-12-19T16:11:39.499192+0000","last_clean":"2023-12-19T16:11:39.499192+0000","last_became_active":"2023-12-19T16:11:39.496306+0000","last_became_peered":"2023-12-19T16:11:39.496306+0000","last_unstale":"2023-12-19T16:11:39.499192+0000","last_undegraded":"2023-12-19T16:11:39.499192+0000","last_fullsized":"2023-12-19T16:11:39.499192+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:58:52.265135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.244868+0000","last_change":"2023-12-19T16:11:40.241420+0000","last_active":"2023-12-19T16:11:40.244868+0000","last_peered":"2023-12-19T16:11:40.244868+0000","last_clean":"2023-12-19T16:11:40.244868+0000","last_became_active":"2023-12-19T16:11:40.237233+0000","last_became_peered":"2023-12-19T16:11:40.237233+0000","last_unstale":"2023-12-19T16:11:40.244868+0000","last_undegraded":"2023-12-19T16:11:40.244868+0000","last_fullsized":"2023-12-19T16:11:40.244868+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:39.270097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.210249+0000","last_change":"2023-12-19T16:11:39.484982+0000","last_active":"2023-12-19T16:11:40.210249+0000","last_peered":"2023-12-19T16:11:40.210249+0000","last_clean":"2023-12-19T16:11:40.210249+0000","last_became_active":"2023-12-19T16:11:39.484606+0000","last_became_peered":"2023-12-19T16:11:39.484606+0000","last_unstale":"2023-12-19T16:11:40.210249+0000","last_undegraded":"2023-12-19T16:11:40.210249+0000","last_fullsized":"2023-12-19T16:11:40.210249+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:18:42.384599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.229740+0000","last_change":"2023-12-19T16:11:40.229740+0000","last_active":"2023-12-19T16:11:40.229740+0000","last_peered":"2023-12-19T16:11:40.229740+0000","last_clean":"2023-12-19T16:11:40.229740+0000","last_became_active":"2023-12-19T16:11:40.229350+0000","last_became_peered":"2023-12-19T16:11:40.229350+0000","last_unstale":"2023-12-19T16:11:40.229740+0000","last_undegraded":"2023-12-19T16:11:40.229740+0000","last_fullsized":"2023-12-19T16:11:40.229740+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:37:55.501533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.215242+0000","last_change":"2023-12-19T16:11:39.500513+0000","last_active":"2023-12-19T16:11:40.215242+0000","last_peered":"2023-12-19T16:11:40.215242+0000","last_clean":"2023-12-19T16:11:40.215242+0000","last_became_active":"2023-12-19T16:11:39.499306+0000","last_became_peered":"2023-12-19T16:11:39.499306+0000","last_unstale":"2023-12-19T16:11:40.215242+0000","last_undegraded":"2023-12-19T16:11:40.215242+0000","last_fullsized":"2023-12-19T16:11:40.215242+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:35.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226537+0000","last_change":"2023-12-19T16:11:40.224815+0000","last_active":"2023-12-19T16:11:40.226537+0000","last_peered":"2023-12-19T16:11:40.226537+0000","last_clean":"2023-12-19T16:11:40.226537+0000","last_became_active":"2023-12-19T16:11:40.224239+0000","last_became_peered":"2023-12-19T16:11:40.224239+0000","last_unstale":"2023-12-19T16:11:40.226537+0000","last_undegraded":"2023-12-19T16:11:40.226537+0000","last_fullsized":"2023-12-19T16:11:40.226537+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:52.255854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"198'3","reported_seq":13,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.487003+0000","last_change":"2023-12-19T16:11:39.482928+0000","last_active":"2023-12-19T16:11:40.487003+0000","last_peered":"2023-12-19T16:11:40.487003+0000","last_clean":"2023-12-19T16:11:40.487003+0000","last_became_active":"2023-12-19T16:11:39.482627+0000","last_became_peered":"2023-12-19T16:11:39.482627+0000","last_unstale":"2023-12-19T16:11:40.487003+0000","last_undegraded":"2023-12-19T16:11:40.487003+0000","last_fullsized":"2023-12-19T16:11:40.487003+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:22:10.107299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239822+0000","last_change":"2023-12-19T16:11:40.239822+0000","last_active":"2023-12-19T16:11:40.239822+0000","last_peered":"2023-12-19T16:11:40.239822+0000","last_clean":"2023-12-19T16:11:40.239822+0000","last_became_active":"2023-12-19T16:11:40.239468+0000","last_became_peered":"2023-12-19T16:11:40.239468+0000","last_unstale":"2023-12-19T16:11:40.239822+0000","last_undegraded":"2023-12-19T16:11:40.239822+0000","last_fullsized":"2023-12-19T16:11:40.239822+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:41.730382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"198'6","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.494348+0000","last_change":"2023-12-19T16:11:39.495200+0000","last_active":"2023-12-19T16:11:40.494348+0000","last_peered":"2023-12-19T16:11:40.494348+0000","last_clean":"2023-12-19T16:11:40.494348+0000","last_became_active":"2023-12-19T16:11:39.491795+0000","last_became_peered":"2023-12-19T16:11:39.491795+0000","last_unstale":"2023-12-19T16:11:40.494348+0000","last_undegraded":"2023-12-19T16:11:40.494348+0000","last_fullsized":"2023-12-19T16:11:40.494348+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:06:09.163040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232023+0000","last_change":"2023-12-19T16:11:40.232023+0000","last_active":"2023-12-19T16:11:40.232023+0000","last_peered":"2023-12-19T16:11:40.232023+0000","last_clean":"2023-12-19T16:11:40.232023+0000","last_became_active":"2023-12-19T16:11:40.231728+0000","last_became_peered":"2023-12-19T16:11:40.231728+0000","last_unstale":"2023-12-19T16:11:40.232023+0000","last_undegraded":"2023-12-19T16:11:40.232023+0000","last_fullsized":"2023-12-19T16:11:40.232023+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.490780+0000","last_change":"2023-12-19T16:11:39.503180+0000","last_active":"2023-12-19T16:11:40.490780+0000","last_peered":"2023-12-19T16:11:40.490780+0000","last_clean":"2023-12-19T16:11:40.490780+0000","last_became_active":"2023-12-19T16:11:39.502791+0000","last_became_peered":"2023-12-19T16:11:39.502791+0000","last_unstale":"2023-12-19T16:11:40.490780+0000","last_undegraded":"2023-12-19T16:11:40.490780+0000","last_fullsized":"2023-12-19T16:11:40.490780+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:15:51.241326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239207+0000","last_change":"2023-12-19T16:11:40.236817+0000","last_active":"2023-12-19T16:11:40.239207+0000","last_peered":"2023-12-19T16:11:40.239207+0000","last_clean":"2023-12-19T16:11:40.239207+0000","last_became_active":"2023-12-19T16:11:40.236185+0000","last_became_peered":"2023-12-19T16:11:40.236185+0000","last_unstale":"2023-12-19T16:11:40.239207+0000","last_undegraded":"2023-12-19T16:11:40.239207+0000","last_fullsized":"2023-12-19T16:11:40.239207+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:15.986396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.213162+0000","last_change":"2023-12-19T16:11:39.488250+0000","last_active":"2023-12-19T16:11:40.213162+0000","last_peered":"2023-12-19T16:11:40.213162+0000","last_clean":"2023-12-19T16:11:40.213162+0000","last_became_active":"2023-12-19T16:11:39.487386+0000","last_became_peered":"2023-12-19T16:11:39.487386+0000","last_unstale":"2023-12-19T16:11:40.213162+0000","last_undegraded":"2023-12-19T16:11:40.213162+0000","last_fullsized":"2023-12-19T16:11:40.213162+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:18.470316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232452+0000","last_change":"2023-12-19T16:11:40.232452+0000","last_active":"2023-12-19T16:11:40.232452+0000","last_peered":"2023-12-19T16:11:40.232452+0000","last_clean":"2023-12-19T16:11:40.232452+0000","last_became_active":"2023-12-19T16:11:40.232149+0000","last_became_peered":"2023-12-19T16:11:40.232149+0000","last_unstale":"2023-12-19T16:11:40.232452+0000","last_undegraded":"2023-12-19T16:11:40.232452+0000","last_fullsized":"2023-12-19T16:11:40.232452+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.499495+0000","last_change":"2023-12-19T16:11:39.497671+0000","last_active":"2023-12-19T16:11:39.499495+0000","last_peered":"2023-12-19T16:11:39.499495+0000","last_clean":"2023-12-19T16:11:39.499495+0000","last_became_active":"2023-12-19T16:11:39.497473+0000","last_became_peered":"2023-12-19T16:11:39.497473+0000","last_unstale":"2023-12-19T16:11:39.499495+0000","last_undegraded":"2023-12-19T16:11:39.499495+0000","last_fullsized":"2023-12-19T16:11:39.499495+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:06:22.997804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232477+0000","last_change":"2023-12-19T16:11:40.232477+0000","last_active":"2023-12-19T16:11:40.232477+0000","last_peered":"2023-12-19T16:11:40.232477+0000","last_clean":"2023-12-19T16:11:40.232477+0000","last_became_active":"2023-12-19T16:11:40.229817+0000","last_became_peered":"2023-12-19T16:11:40.229817+0000","last_unstale":"2023-12-19T16:11:40.232477+0000","last_undegraded":"2023-12-19T16:11:40.232477+0000","last_fullsized":"2023-12-19T16:11:40.232477+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:23:22.461430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"198'8","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.492316+0000","last_change":"2023-12-19T16:11:39.487468+0000","last_active":"2023-12-19T16:11:40.492316+0000","last_peered":"2023-12-19T16:11:40.492316+0000","last_clean":"2023-12-19T16:11:40.492316+0000","last_became_active":"2023-12-19T16:11:39.487176+0000","last_became_peered":"2023-12-19T16:11:39.487176+0000","last_unstale":"2023-12-19T16:11:40.492316+0000","last_undegraded":"2023-12-19T16:11:40.492316+0000","last_fullsized":"2023-12-19T16:11:40.492316+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:24:46.979659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234502+0000","last_change":"2023-12-19T16:11:40.225688+0000","last_active":"2023-12-19T16:11:40.234502+0000","last_peered":"2023-12-19T16:11:40.234502+0000","last_clean":"2023-12-19T16:11:40.234502+0000","last_became_active":"2023-12-19T16:11:40.225297+0000","last_became_peered":"2023-12-19T16:11:40.225297+0000","last_unstale":"2023-12-19T16:11:40.234502+0000","last_undegraded":"2023-12-19T16:11:40.234502+0000","last_fullsized":"2023-12-19T16:11:40.234502+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:13.538074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"198'1","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.489975+0000","last_change":"2023-12-19T16:11:39.489356+0000","last_active":"2023-12-19T16:11:40.489975+0000","last_peered":"2023-12-19T16:11:40.489975+0000","last_clean":"2023-12-19T16:11:40.489975+0000","last_became_active":"2023-12-19T16:11:39.489037+0000","last_became_peered":"2023-12-19T16:11:39.489037+0000","last_unstale":"2023-12-19T16:11:40.489975+0000","last_undegraded":"2023-12-19T16:11:40.489975+0000","last_fullsized":"2023-12-19T16:11:40.489975+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:54.697560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.230088+0000","last_change":"2023-12-19T16:11:40.230088+0000","last_active":"2023-12-19T16:11:40.230088+0000","last_peered":"2023-12-19T16:11:40.230088+0000","last_clean":"2023-12-19T16:11:40.230088+0000","last_became_active":"2023-12-19T16:11:40.229743+0000","last_became_peered":"2023-12-19T16:11:40.229743+0000","last_unstale":"2023-12-19T16:11:40.230088+0000","last_undegraded":"2023-12-19T16:11:40.230088+0000","last_fullsized":"2023-12-19T16:11:40.230088+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:45:44.649317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"198'2","reported_seq":12,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.491547+0000","last_change":"2023-12-19T16:11:39.496632+0000","last_active":"2023-12-19T16:11:40.491547+0000","last_peered":"2023-12-19T16:11:40.491547+0000","last_clean":"2023-12-19T16:11:40.491547+0000","last_became_active":"2023-12-19T16:11:39.492350+0000","last_became_peered":"2023-12-19T16:11:39.492350+0000","last_unstale":"2023-12-19T16:11:40.491547+0000","last_undegraded":"2023-12-19T16:11:40.491547+0000","last_fullsized":"2023-12-19T16:11:40.491547+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:05:15.663991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.233934+0000","last_change":"2023-12-19T16:11:40.233934+0000","last_active":"2023-12-19T16:11:40.233934+0000","last_peered":"2023-12-19T16:11:40.233934+0000","last_clean":"2023-12-19T16:11:40.233934+0000","last_became_active":"2023-12-19T16:11:40.233640+0000","last_became_peered":"2023-12-19T16:11:40.233640+0000","last_unstale":"2023-12-19T16:11:40.233934+0000","last_undegraded":"2023-12-19T16:11:40.233934+0000","last_fullsized":"2023-12-19T16:11:40.233934+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:29:18.069075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"198'1","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.225968+0000","last_change":"2023-12-19T16:11:39.501864+0000","last_active":"2023-12-19T16:11:40.225968+0000","last_peered":"2023-12-19T16:11:40.225968+0000","last_clean":"2023-12-19T16:11:40.225968+0000","last_became_active":"2023-12-19T16:11:39.501593+0000","last_became_peered":"2023-12-19T16:11:39.501593+0000","last_unstale":"2023-12-19T16:11:40.225968+0000","last_undegraded":"2023-12-19T16:11:40.225968+0000","last_fullsized":"2023-12-19T16:11:40.225968+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:58.438267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235640+0000","last_change":"2023-12-19T16:11:40.235640+0000","last_active":"2023-12-19T16:11:40.235640+0000","last_peered":"2023-12-19T16:11:40.235640+0000","last_clean":"2023-12-19T16:11:40.235640+0000","last_became_active":"2023-12-19T16:11:40.235368+0000","last_became_peered":"2023-12-19T16:11:40.235368+0000","last_unstale":"2023-12-19T16:11:40.235640+0000","last_undegraded":"2023-12-19T16:11:40.235640+0000","last_fullsized":"2023-12-19T16:11:40.235640+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:45.649577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.210063+0000","last_change":"2023-12-19T16:11:39.503117+0000","last_active":"2023-12-19T16:11:40.210063+0000","last_peered":"2023-12-19T16:11:40.210063+0000","last_clean":"2023-12-19T16:11:40.210063+0000","last_became_active":"2023-12-19T16:11:39.502847+0000","last_became_peered":"2023-12-19T16:11:39.502847+0000","last_unstale":"2023-12-19T16:11:40.210063+0000","last_undegraded":"2023-12-19T16:11:40.210063+0000","last_fullsized":"2023-12-19T16:11:40.210063+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:01:36.672177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.243501+0000","last_change":"2023-12-19T16:11:40.239771+0000","last_active":"2023-12-19T16:11:40.243501+0000","last_peered":"2023-12-19T16:11:40.243501+0000","last_clean":"2023-12-19T16:11:40.243501+0000","last_became_active":"2023-12-19T16:11:40.236979+0000","last_became_peered":"2023-12-19T16:11:40.236979+0000","last_unstale":"2023-12-19T16:11:40.243501+0000","last_undegraded":"2023-12-19T16:11:40.243501+0000","last_fullsized":"2023-12-19T16:11:40.243501+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:12:22.199558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.489779+0000","last_change":"2023-12-19T16:11:39.501556+0000","last_active":"2023-12-19T16:11:40.489779+0000","last_peered":"2023-12-19T16:11:40.489779+0000","last_clean":"2023-12-19T16:11:40.489779+0000","last_became_active":"2023-12-19T16:11:39.501100+0000","last_became_peered":"2023-12-19T16:11:39.501100+0000","last_unstale":"2023-12-19T16:11:40.489779+0000","last_undegraded":"2023-12-19T16:11:40.489779+0000","last_fullsized":"2023-12-19T16:11:40.489779+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:51:50.279745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.235578+0000","last_change":"2023-12-19T16:11:40.235578+0000","last_active":"2023-12-19T16:11:40.235578+0000","last_peered":"2023-12-19T16:11:40.235578+0000","last_clean":"2023-12-19T16:11:40.235578+0000","last_became_active":"2023-12-19T16:11:40.231078+0000","last_became_peered":"2023-12-19T16:11:40.231078+0000","last_unstale":"2023-12-19T16:11:40.235578+0000","last_undegraded":"2023-12-19T16:11:40.235578+0000","last_fullsized":"2023-12-19T16:11:40.235578+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:29:07.800630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.486774+0000","last_change":"2023-12-19T16:11:39.485097+0000","last_active":"2023-12-19T16:11:39.486774+0000","last_peered":"2023-12-19T16:11:39.486774+0000","last_clean":"2023-12-19T16:11:39.486774+0000","last_became_active":"2023-12-19T16:11:39.484747+0000","last_became_peered":"2023-12-19T16:11:39.484747+0000","last_unstale":"2023-12-19T16:11:39.486774+0000","last_undegraded":"2023-12-19T16:11:39.486774+0000","last_fullsized":"2023-12-19T16:11:39.486774+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:56.677127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238270+0000","last_change":"2023-12-19T16:11:40.238270+0000","last_active":"2023-12-19T16:11:40.238270+0000","last_peered":"2023-12-19T16:11:40.238270+0000","last_clean":"2023-12-19T16:11:40.238270+0000","last_became_active":"2023-12-19T16:11:40.237980+0000","last_became_peered":"2023-12-19T16:11:40.237980+0000","last_unstale":"2023-12-19T16:11:40.238270+0000","last_undegraded":"2023-12-19T16:11:40.238270+0000","last_fullsized":"2023-12-19T16:11:40.238270+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:05.523201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.493261+0000","last_change":"2023-12-19T16:11:39.485864+0000","last_active":"2023-12-19T16:11:39.493261+0000","last_peered":"2023-12-19T16:11:39.493261+0000","last_clean":"2023-12-19T16:11:39.493261+0000","last_became_active":"2023-12-19T16:11:39.484925+0000","last_became_peered":"2023-12-19T16:11:39.484925+0000","last_unstale":"2023-12-19T16:11:39.493261+0000","last_undegraded":"2023-12-19T16:11:39.493261+0000","last_fullsized":"2023-12-19T16:11:39.493261+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:20:52.583832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227374+0000","last_change":"2023-12-19T16:11:40.227374+0000","last_active":"2023-12-19T16:11:40.227374+0000","last_peered":"2023-12-19T16:11:40.227374+0000","last_clean":"2023-12-19T16:11:40.227374+0000","last_became_active":"2023-12-19T16:11:40.227067+0000","last_became_peered":"2023-12-19T16:11:40.227067+0000","last_unstale":"2023-12-19T16:11:40.227374+0000","last_undegraded":"2023-12-19T16:11:40.227374+0000","last_fullsized":"2023-12-19T16:11:40.227374+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:55.712088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.482572+0000","last_change":"2023-12-19T16:11:39.480826+0000","last_active":"2023-12-19T16:11:39.482572+0000","last_peered":"2023-12-19T16:11:39.482572+0000","last_clean":"2023-12-19T16:11:39.482572+0000","last_became_active":"2023-12-19T16:11:39.480515+0000","last_became_peered":"2023-12-19T16:11:39.480515+0000","last_unstale":"2023-12-19T16:11:39.482572+0000","last_undegraded":"2023-12-19T16:11:39.482572+0000","last_fullsized":"2023-12-19T16:11:39.482572+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:26:45.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.226192+0000","last_change":"2023-12-19T16:11:40.226192+0000","last_active":"2023-12-19T16:11:40.226192+0000","last_peered":"2023-12-19T16:11:40.226192+0000","last_clean":"2023-12-19T16:11:40.226192+0000","last_became_active":"2023-12-19T16:11:40.225916+0000","last_became_peered":"2023-12-19T16:11:40.225916+0000","last_unstale":"2023-12-19T16:11:40.226192+0000","last_undegraded":"2023-12-19T16:11:40.226192+0000","last_fullsized":"2023-12-19T16:11:40.226192+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:06:14.919875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"198'2","reported_seq":11,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.488382+0000","last_change":"2023-12-19T16:11:39.502305+0000","last_active":"2023-12-19T16:11:40.488382+0000","last_peered":"2023-12-19T16:11:40.488382+0000","last_clean":"2023-12-19T16:11:40.488382+0000","last_became_active":"2023-12-19T16:11:39.501012+0000","last_became_peered":"2023-12-19T16:11:39.501012+0000","last_unstale":"2023-12-19T16:11:40.488382+0000","last_undegraded":"2023-12-19T16:11:40.488382+0000","last_fullsized":"2023-12-19T16:11:40.488382+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:39:35.469618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.238505+0000","last_change":"2023-12-19T16:11:40.227386+0000","last_active":"2023-12-19T16:11:40.238505+0000","last_peered":"2023-12-19T16:11:40.238505+0000","last_clean":"2023-12-19T16:11:40.238505+0000","last_became_active":"2023-12-19T16:11:40.227102+0000","last_became_peered":"2023-12-19T16:11:40.227102+0000","last_unstale":"2023-12-19T16:11:40.238505+0000","last_undegraded":"2023-12-19T16:11:40.238505+0000","last_fullsized":"2023-12-19T16:11:40.238505+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:46:56.057099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214858+0000","last_change":"2023-12-19T16:11:39.491592+0000","last_active":"2023-12-19T16:11:40.214858+0000","last_peered":"2023-12-19T16:11:40.214858+0000","last_clean":"2023-12-19T16:11:40.214858+0000","last_became_active":"2023-12-19T16:11:39.491283+0000","last_became_peered":"2023-12-19T16:11:39.491283+0000","last_unstale":"2023-12-19T16:11:40.214858+0000","last_undegraded":"2023-12-19T16:11:40.214858+0000","last_fullsized":"2023-12-19T16:11:40.214858+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:55.458290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231796+0000","last_change":"2023-12-19T16:11:40.231796+0000","last_active":"2023-12-19T16:11:40.231796+0000","last_peered":"2023-12-19T16:11:40.231796+0000","last_clean":"2023-12-19T16:11:40.231796+0000","last_became_active":"2023-12-19T16:11:40.231479+0000","last_became_peered":"2023-12-19T16:11:40.231479+0000","last_unstale":"2023-12-19T16:11:40.231796+0000","last_undegraded":"2023-12-19T16:11:40.231796+0000","last_fullsized":"2023-12-19T16:11:40.231796+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:34.846952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214162+0000","last_change":"2023-12-19T16:11:39.489719+0000","last_active":"2023-12-19T16:11:40.214162+0000","last_peered":"2023-12-19T16:11:40.214162+0000","last_clean":"2023-12-19T16:11:40.214162+0000","last_became_active":"2023-12-19T16:11:39.489390+0000","last_became_peered":"2023-12-19T16:11:39.489390+0000","last_unstale":"2023-12-19T16:11:40.214162+0000","last_undegraded":"2023-12-19T16:11:40.214162+0000","last_fullsized":"2023-12-19T16:11:40.214162+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:11:38.255569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.234535+0000","last_change":"2023-12-19T16:11:40.223671+0000","last_active":"2023-12-19T16:11:40.234535+0000","last_peered":"2023-12-19T16:11:40.234535+0000","last_clean":"2023-12-19T16:11:40.234535+0000","last_became_active":"2023-12-19T16:11:40.223296+0000","last_became_peered":"2023-12-19T16:11:40.223296+0000","last_unstale":"2023-12-19T16:11:40.234535+0000","last_undegraded":"2023-12-19T16:11:40.234535+0000","last_fullsized":"2023-12-19T16:11:40.234535+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:48:46.744838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.493234+0000","last_change":"2023-12-19T16:11:39.487377+0000","last_active":"2023-12-19T16:11:39.493234+0000","last_peered":"2023-12-19T16:11:39.493234+0000","last_clean":"2023-12-19T16:11:39.493234+0000","last_became_active":"2023-12-19T16:11:39.486537+0000","last_became_peered":"2023-12-19T16:11:39.486537+0000","last_unstale":"2023-12-19T16:11:39.493234+0000","last_undegraded":"2023-12-19T16:11:39.493234+0000","last_fullsized":"2023-12-19T16:11:39.493234+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:17:09.886098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.239192+0000","last_change":"2023-12-19T16:11:40.234079+0000","last_active":"2023-12-19T16:11:40.239192+0000","last_peered":"2023-12-19T16:11:40.239192+0000","last_clean":"2023-12-19T16:11:40.239192+0000","last_became_active":"2023-12-19T16:11:40.233787+0000","last_became_peered":"2023-12-19T16:11:40.233787+0000","last_unstale":"2023-12-19T16:11:40.239192+0000","last_undegraded":"2023-12-19T16:11:40.239192+0000","last_fullsized":"2023-12-19T16:11:40.239192+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:40:19.484844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.216810+0000","last_change":"2023-12-19T16:11:39.500737+0000","last_active":"2023-12-19T16:11:40.216810+0000","last_peered":"2023-12-19T16:11:40.216810+0000","last_clean":"2023-12-19T16:11:40.216810+0000","last_became_active":"2023-12-19T16:11:39.500363+0000","last_became_peered":"2023-12-19T16:11:39.500363+0000","last_unstale":"2023-12-19T16:11:40.216810+0000","last_undegraded":"2023-12-19T16:11:40.216810+0000","last_fullsized":"2023-12-19T16:11:40.216810+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:26:45.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.227676+0000","last_change":"2023-12-19T16:11:40.227676+0000","last_active":"2023-12-19T16:11:40.227676+0000","last_peered":"2023-12-19T16:11:40.227676+0000","last_clean":"2023-12-19T16:11:40.227676+0000","last_became_active":"2023-12-19T16:11:40.227326+0000","last_became_peered":"2023-12-19T16:11:40.227326+0000","last_unstale":"2023-12-19T16:11:40.227676+0000","last_undegraded":"2023-12-19T16:11:40.227676+0000","last_fullsized":"2023-12-19T16:11:40.227676+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:22:15.025200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.213325+0000","last_change":"2023-12-19T16:11:39.492749+0000","last_active":"2023-12-19T16:11:40.213325+0000","last_peered":"2023-12-19T16:11:40.213325+0000","last_clean":"2023-12-19T16:11:40.213325+0000","last_became_active":"2023-12-19T16:11:39.491630+0000","last_became_peered":"2023-12-19T16:11:39.491630+0000","last_unstale":"2023-12-19T16:11:40.213325+0000","last_undegraded":"2023-12-19T16:11:40.213325+0000","last_fullsized":"2023-12-19T16:11:40.213325+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:48:12.523546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231901+0000","last_change":"2023-12-19T16:11:40.228970+0000","last_active":"2023-12-19T16:11:40.231901+0000","last_peered":"2023-12-19T16:11:40.231901+0000","last_clean":"2023-12-19T16:11:40.231901+0000","last_became_active":"2023-12-19T16:11:40.226741+0000","last_became_peered":"2023-12-19T16:11:40.226741+0000","last_unstale":"2023-12-19T16:11:40.231901+0000","last_undegraded":"2023-12-19T16:11:40.231901+0000","last_fullsized":"2023-12-19T16:11:40.231901+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:04:50.343935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.489961+0000","last_change":"2023-12-19T16:11:39.488851+0000","last_active":"2023-12-19T16:11:39.489961+0000","last_peered":"2023-12-19T16:11:39.489961+0000","last_clean":"2023-12-19T16:11:39.489961+0000","last_became_active":"2023-12-19T16:11:39.488152+0000","last_became_peered":"2023-12-19T16:11:39.488152+0000","last_unstale":"2023-12-19T16:11:39.489961+0000","last_undegraded":"2023-12-19T16:11:39.489961+0000","last_fullsized":"2023-12-19T16:11:39.489961+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:05:58.957511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.232170+0000","last_change":"2023-12-19T16:11:40.232170+0000","last_active":"2023-12-19T16:11:40.232170+0000","last_peered":"2023-12-19T16:11:40.232170+0000","last_clean":"2023-12-19T16:11:40.232170+0000","last_became_active":"2023-12-19T16:11:40.231785+0000","last_became_peered":"2023-12-19T16:11:40.231785+0000","last_unstale":"2023-12-19T16:11:40.232170+0000","last_undegraded":"2023-12-19T16:11:40.232170+0000","last_fullsized":"2023-12-19T16:11:40.232170+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:32:08.796681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.506644+0000","last_change":"2023-12-19T16:11:39.502487+0000","last_active":"2023-12-19T16:11:39.506644+0000","last_peered":"2023-12-19T16:11:39.506644+0000","last_clean":"2023-12-19T16:11:39.506644+0000","last_became_active":"2023-12-19T16:11:39.501428+0000","last_became_peered":"2023-12-19T16:11:39.501428+0000","last_unstale":"2023-12-19T16:11:39.506644+0000","last_undegraded":"2023-12-19T16:11:39.506644+0000","last_fullsized":"2023-12-19T16:11:39.506644+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:41.733403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":8,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.231288+0000","last_change":"2023-12-19T16:11:40.231288+0000","last_active":"2023-12-19T16:11:40.231288+0000","last_peered":"2023-12-19T16:11:40.231288+0000","last_clean":"2023-12-19T16:11:40.231288+0000","last_became_active":"2023-12-19T16:11:40.230985+0000","last_became_peered":"2023-12-19T16:11:40.230985+0000","last_unstale":"2023-12-19T16:11:40.231288+0000","last_undegraded":"2023-12-19T16:11:40.231288+0000","last_fullsized":"2023-12-19T16:11:40.231288+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":197,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:39.447845+0000","last_clean_scrub_stamp":"2023-12-19T16:11:39.447845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:38:42.747931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-12-19T16:11:40.214135+0000","last_change":"2023-12-19T16:11:39.494555+0000","last_active":"2023-12-19T16:11:40.214135+0000","last_peered":"2023-12-19T16:11:40.214135+0000","last_clean":"2023-12-19T16:11:40.214135+0000","last_became_active":"2023-12-19T16:11:39.491704+0000","last_became_peered":"2023-12-19T16:11:39.491704+0000","last_unstale":"2023-12-19T16:11:40.214135+0000","last_undegraded":"2023-12-19T16:11:40.214135+0000","last_fullsized":"2023-12-19T16:11:40.214135+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":196,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:11:38.433317+0000","last_clean_scrub_stamp":"2023-12-19T16:11:38.433317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:33:14.971868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"187'326","reported_seq":1049,"reported_epoch":197,"state":"active+clean","last_fresh":"2023-12-19T16:11:39.471013+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:11:39.471013+0000","last_peered":"2023-12-19T16:11:39.471013+0000","last_clean":"2023-12-19T16:11:39.471013+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:11:39.471013+0000","last_undegraded":"2023-12-19T16:11:39.471013+0000","last_fullsized":"2023-12-19T16:11:39.471013+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304384,"kb_used_data":3000,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067456,"statfs":{"total":96636764160,"available":96325074944,"internally_reserved":0,"allocated":3072000,"data_stored":1856840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.113,"15min":1.119},"min":{"1min":0.84299999999999997,"5min":0.70899999999999996,"15min":0.56399999999999995},"max":{"1min":1.661,"5min":1.661,"15min":1.6919999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.875,"5min":0.80500000000000005,"15min":0.496},"max":{"1min":1.597,"5min":1.597,"15min":1.9379999999999999},"last":0.95899999999999996}]},{"osd":1,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1479999999999999,"15min":1.139},"min":{"1min":0.85399999999999998,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":1.45,"5min":1.623,"15min":1.903},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.24,"15min":1.2410000000000001},"min":{"1min":1.0580000000000001,"5min":0.82699999999999996,"15min":0.73299999999999998},"max":{"1min":1.5489999999999999,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.45}]},{"osd":2,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.272,"15min":1.298},"min":{"1min":0.98299999999999998,"5min":0.89300000000000002,"15min":0.63300000000000001},"max":{"1min":1.5860000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.3100000000000001,"15min":1.3240000000000001},"min":{"1min":1.095,"5min":0.995,"15min":0.68999999999999995},"max":{"1min":1.631,"5min":1.7010000000000001,"15min":1.7709999999999999},"last":1.373}]},{"osd":3,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.319,"15min":1.3380000000000001},"min":{"1min":1.133,"5min":0.94099999999999995,"15min":0.52500000000000002},"max":{"1min":1.623,"5min":1.736,"15min":1.839},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3560000000000001,"15min":1.3700000000000001},"min":{"1min":1.224,"5min":0.84299999999999997,"15min":0.60699999999999998},"max":{"1min":1.603,"5min":1.726,"15min":1.9630000000000001},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.90900000000000003,"5min":0.74199999999999999,"15min":0.55600000000000005},"max":{"1min":1.528,"5min":1.77,"15min":1.849},"last":1.327},{"interface":"front","average":{"1min":1.23,"5min":1.1990000000000001,"15min":1.232},"min":{"1min":0.82099999999999995,"5min":0.58699999999999997,"15min":0.47699999999999998},"max":{"1min":1.5780000000000001,"5min":1.651,"15min":1.784},"last":1.113}]},{"osd":5,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.94399999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.679,"5min":1.821,"15min":1.825},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.343,"15min":1.337},"min":{"1min":1.048,"5min":0.84599999999999997,"15min":0.45000000000000001},"max":{"1min":1.6499999999999999,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.1950000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:11:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4299999999999999,"15min":1.4330000000000001},"min":{"1min":1.1839999999999999,"5min":0.97699999999999998,"15min":0.54200000000000004},"max":{"1min":1.714,"5min":1.994,"15min":1.994},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.3660000000000001,"15min":1.3919999999999999},"min":{"1min":1.0580000000000001,"5min":0.746,"15min":0.63100000000000001},"max":{"1min":1.7330000000000001,"5min":2.0760000000000001,"15min":2.1059999999999999},"last":1.528}]}]},{"osd":6,"up_from":13,"seq":55834575953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311996,"kb_used_data":2420,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059844,"statfs":{"total":96636764160,"available":96317280256,"internally_reserved":0,"allocated":2478080,"data_stored":1266472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1739999999999999,"15min":1.175},"min":{"1min":0.80300000000000005,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.5669999999999999,"5min":1.573,"15min":1.7150000000000001},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.97599999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.74,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.2170000000000001,"15min":1.1950000000000001},"min":{"1min":0.94799999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.5049999999999999,"5min":1.6559999999999999,"15min":1.7010000000000001},"last":1.121},{"interface":"front","average":{"1min":1.353,"5min":1.3140000000000001,"15min":1.2849999999999999},"min":{"1min":1.0549999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.806,"5min":1.827,"15min":1.8300000000000001},"last":1.3660000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.23,"15min":1.236},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.81100000000000005},"max":{"1min":1.4790000000000001,"5min":1.6180000000000001,"15min":1.8080000000000001},"last":1.474},{"interface":"front","average":{"1min":1.351,"5min":1.296,"15min":1.2929999999999999},"min":{"1min":1.044,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.673,"5min":1.732,"15min":1.756},"last":1.2110000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.359,"15min":1.3400000000000001},"min":{"1min":0.96499999999999997,"5min":0.91400000000000003,"15min":0.84099999999999997},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.2849999999999999,"15min":1.3340000000000001},"min":{"1min":0.94399999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.671}]},{"osd":4,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1950000000000001,"15min":1.1990000000000001},"min":{"1min":0.85099999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.579,"5min":1.6910000000000001,"15min":1.774},"last":1.302},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.274,"15min":1.3140000000000001},"min":{"1min":0.82499999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":1.2729999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2749999999999999,"15min":1.3100000000000001},"min":{"1min":0.92900000000000005,"5min":0.75600000000000001,"15min":0.68000000000000005},"max":{"1min":1.788,"5min":1.788,"15min":1.923},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":1.03,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.6839999999999999,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 16:10:57 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4159999999999999,"15min":1.4430000000000001},"min":{"1min":0.92900000000000005,"5min":0.92400000000000004,"15min":0.73799999999999999},"max":{"1min":1.839,"5min":1.8720000000000001,"15min":2.169},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.417,"15min":1.464},"min":{"1min":1.163,"5min":0.88800000000000001,"15min":0.77200000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.861}]}]},{"osd":5,"up_from":13,"seq":55834575953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307908,"kb_used_data":2428,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063932,"statfs":{"total":96636764160,"available":96321466368,"internally_reserved":0,"allocated":2486272,"data_stored":1266596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.131,"15min":1.113},"min":{"1min":0.86499999999999999,"5min":0.73099999999999998,"15min":0.66900000000000004},"max":{"1min":1.55,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.2290000000000001,"15min":1.2150000000000001},"min":{"1min":0.94199999999999995,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.907},"last":1.3220000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3300000000000001,"15min":1.325},"min":{"1min":0.82399999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6759999999999999,"5min":1.7350000000000001,"15min":1.7490000000000001},"last":1.218},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.2909999999999999,"15min":1.29},"min":{"1min":1.1220000000000001,"5min":0.91100000000000003,"15min":0.85999999999999999},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.2769999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.284,"15min":1.278},"min":{"1min":1.0569999999999999,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.5600000000000001,"5min":1.7629999999999999,"15min":2.0249999999999999},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2609999999999999,"15min":1.236},"min":{"1min":0.90800000000000003,"5min":0.88100000000000001,"15min":0.84499999999999997},"max":{"1min":1.7769999999999999,"5min":1.831,"15min":1.831},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.298,"15min":1.3120000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.643,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.393},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.353,"15min":1.321},"min":{"1min":0.995,"5min":0.97599999999999998,"15min":0.85799999999999998},"max":{"1min":1.734,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1479999999999999,"15min":1.1339999999999999},"min":{"1min":0.77800000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.452,"5min":1.8,"15min":1.8},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.294,"5min":1.3120000000000001,"15min":1.2809999999999999},"min":{"1min":0.94699999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6160000000000001,"5min":1.8640000000000001,"15min":1.869},"last":1.46}]},{"osd":6,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.91300000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.843,"5min":1.968,"15min":1.968},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4490000000000001,"15min":1.3979999999999999},"min":{"1min":1.133,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6479999999999999,"5min":1.7929999999999999,"15min":1.804},"last":1.5940000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:53 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4510000000000001,"15min":1.4159999999999999},"min":{"1min":0.92600000000000005,"5min":0.82299999999999995,"15min":0.81999999999999995},"max":{"1min":1.6870000000000001,"5min":1.9299999999999999,"15min":2.0790000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4570000000000001,"15min":1.423},"min":{"1min":1.167,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.7849999999999999,"5min":1.9770000000000001,"15min":1.99},"last":1.415}]}]},{"osd":4,"up_from":13,"seq":55834575953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312012,"kb_used_data":2436,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059828,"statfs":{"total":96636764160,"available":96317263872,"internally_reserved":0,"allocated":2494464,"data_stored":1267208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0780000000000001,"15min":1.111},"min":{"1min":0.96299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.47,"5min":1.5429999999999999,"15min":1.625},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.159,"15min":1.1559999999999999},"min":{"1min":0.86399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3939999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":1.248}]},{"osd":1,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.3160000000000001,"15min":1.3140000000000001},"min":{"1min":0.98599999999999999,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.716,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.026},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3109999999999999,"15min":1.319},"min":{"1min":1.071,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.575,"5min":1.6659999999999999,"15min":2.024},"last":1.4159999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.204,"15min":1.2290000000000001},"min":{"1min":0.90900000000000003,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.5629999999999999,"5min":1.617,"15min":1.73},"last":1.123},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.248,"15min":1.25},"min":{"1min":0.90000000000000002,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.7030000000000001,"5min":1.764,"15min":1.764},"last":1.569}]},{"osd":3,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.222,"15min":1.238},"min":{"1min":1.002,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.4399999999999999,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.804},{"interface":"front","average":{"1min":1.296,"5min":1.3260000000000001,"15min":1.3109999999999999},"min":{"1min":0.94899999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6579999999999999,"5min":1.825,"15min":1.825},"last":1.708}]},{"osd":5,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.171,"15min":1.157},"min":{"1min":0.81699999999999995,"5min":0.71799999999999997,"15min":0.53700000000000003},"max":{"1min":1.6259999999999999,"5min":1.8260000000000001,"15min":1.855},"last":1.744},{"interface":"front","average":{"1min":1.157,"5min":1.2190000000000001,"15min":1.1930000000000001},"min":{"1min":0.82599999999999996,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.5780000000000001,"5min":1.726,"15min":1.8200000000000001},"last":1.343}]},{"osd":6,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.367,"15min":1.3140000000000001},"min":{"1min":1.155,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.784,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.367},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4550000000000001,"15min":1.4339999999999999},"min":{"1min":1.1060000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7210000000000001,"5min":1.877,"15min":1.982},"last":1.97}]},{"osd":7,"last update":"Tue Dec 19 16:11:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4039999999999999,"15min":1.3500000000000001},"min":{"1min":0.99399999999999999,"5min":0.73799999999999999,"15min":0.61299999999999999},"max":{"1min":1.7829999999999999,"5min":1.8500000000000001,"15min":1.9450000000000001},"last":1.667},{"interface":"front","average":{"1min":1.528,"5min":1.5149999999999999,"15min":1.488},"min":{"1min":1.252,"5min":0.79800000000000004,"15min":0.60199999999999998},"max":{"1min":1.887,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.607}]}]},{"osd":3,"up_from":13,"seq":55834575952,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299740,"kb_used_data":2452,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072100,"statfs":{"total":96636764160,"available":96329830400,"internally_reserved":0,"allocated":2510848,"data_stored":1275276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.90100000000000002},"min":{"1min":0.63200000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.633,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.002,"15min":1.018},"min":{"1min":0.72999999999999998,"5min":0.57999999999999996,"15min":0.55600000000000005},"max":{"1min":1.379,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.80300000000000005}]},{"osd":1,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.94499999999999995,"15min":0.96599999999999997},"min":{"1min":0.73199999999999998,"5min":0.621,"15min":0.44400000000000001},"max":{"1min":1.2150000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1439999999999999,"15min":1.115},"min":{"1min":0.98399999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.724,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.024}]},{"osd":2,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1659999999999999,"15min":1.1679999999999999},"min":{"1min":0.83599999999999997,"5min":0.82299999999999995,"15min":0.63700000000000001},"max":{"1min":1.393,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.258,"15min":1.26},"min":{"1min":0.95399999999999996,"5min":0.76100000000000001,"15min":0.66600000000000004},"max":{"1min":1.6339999999999999,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.2150000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3360000000000001,"15min":1.3320000000000001},"min":{"1min":1.0629999999999999,"5min":0.82099999999999995,"15min":0.72699999999999998},"max":{"1min":1.5249999999999999,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":1.718},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.302,"15min":1.3120000000000001},"min":{"1min":1.121,"5min":0.72199999999999998,"15min":0.65200000000000002},"max":{"1min":1.5940000000000001,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.4450000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3400000000000001,"15min":1.353},"min":{"1min":0.98799999999999999,"5min":0.93799999999999994,"15min":0.60899999999999999},"max":{"1min":1.569,"5min":2.177,"15min":2.1909999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.431,"15min":1.4239999999999999},"min":{"1min":1.171,"5min":0.97299999999999998,"15min":0.82199999999999995},"max":{"1min":1.6639999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.8480000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.492,"15min":1.504},"min":{"1min":1.341,"5min":0.90200000000000002,"15min":0.79100000000000004},"max":{"1min":1.748,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4450000000000001,"15min":1.4570000000000001},"min":{"1min":1.1339999999999999,"5min":1.0329999999999999,"15min":0.80500000000000005},"max":{"1min":1.859,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":2.0270000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:52 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":1.3280000000000001,"5min":1.119,"15min":0.93100000000000005},"max":{"1min":1.97,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.887},{"interface":"front","average":{"1min":1.452,"5min":1.4790000000000001,"15min":1.502},"min":{"1min":1.1739999999999999,"5min":1.069,"15min":0.75},"max":{"1min":1.732,"5min":2.27,"15min":2.2709999999999999},"last":1.5249999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311992,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059848,"statfs":{"total":96636764160,"available":96317284352,"internally_reserved":0,"allocated":2473984,"data_stored":1265461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":0.96399999999999997},"min":{"1min":0.69299999999999995,"5min":0.61499999999999999,"15min":0.54600000000000004},"max":{"1min":1.1699999999999999,"5min":1.508,"15min":1.581},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1830000000000001,"15min":1.1499999999999999},"min":{"1min":0.879,"5min":0.81000000000000005,"15min":0.66200000000000003},"max":{"1min":1.321,"5min":1.796,"15min":1.796},"last":1.119}]},{"osd":1,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.198,"15min":1.1910000000000001},"min":{"1min":0.84699999999999998,"5min":0.83599999999999997,"15min":0.78900000000000003},"max":{"1min":1.4179999999999999,"5min":1.6279999999999999,"15min":1.7989999999999999},"last":1.228},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.163,"15min":1.127},"min":{"1min":0.754,"5min":0.754,"15min":0.621},"max":{"1min":1.3420000000000001,"5min":1.696,"15min":1.696},"last":1.258}]},{"osd":3,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.048,"15min":1.038},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4770000000000001,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1579999999999999,"15min":1.1479999999999999},"min":{"1min":0.82299999999999995,"5min":0.77000000000000002,"15min":0.59799999999999998},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.8340000000000001},"last":0.87}]},{"osd":4,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4239999999999999,"15min":1.4350000000000001},"min":{"1min":1.022,"5min":0.95199999999999996,"15min":0.71299999999999997},"max":{"1min":1.625,"5min":1.786,"15min":1.986},"last":1.524},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4570000000000001,"15min":1.446},"min":{"1min":1.05,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.641,"5min":1.988,"15min":2.0070000000000001},"last":1.345}]},{"osd":5,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.4359999999999999,"15min":1.417},"min":{"1min":1.0880000000000001,"5min":1.044,"15min":0.84999999999999998},"max":{"1min":1.732,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4399999999999999,"15min":1.4019999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.6439999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.4610000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4139999999999999,"15min":1.397},"min":{"1min":1.0900000000000001,"5min":1.0680000000000001,"15min":0.95799999999999996},"max":{"1min":1.881,"5min":2.077,"15min":2.081},"last":1.377},{"interface":"front","average":{"1min":1.49,"5min":1.534,"15min":1.5209999999999999},"min":{"1min":1.1599999999999999,"5min":1.071,"15min":1.034},"max":{"1min":1.883,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.5649999999999999,"15min":1.544},"min":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":0.89600000000000002},"max":{"1min":1.7529999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.5580000000000001,"15min":1.532},"min":{"1min":1.073,"5min":1.073,"15min":0.90300000000000002},"max":{"1min":1.948,"5min":2.1659999999999999,"15min":2.4830000000000001},"last":1.4890000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575952,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312020,"kb_used_data":2444,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059820,"statfs":{"total":96636764160,"available":96317255680,"internally_reserved":0,"allocated":2502656,"data_stored":1274700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.6559999999999999,"15min":1.6579999999999999},"last":1.351},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.042,"15min":1.042},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4730000000000001,"5min":1.6020000000000001,"15min":1.9119999999999999},"last":1.46}]},{"osd":2,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.081,"15min":1.073},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.325,"5min":1.5449999999999999,"15min":1.6830000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.115,"15min":1.165},"min":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.4710000000000001,"5min":1.6930000000000001,"15min":1.7949999999999999},"last":0.73999999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.3029999999999999,"15min":1.2949999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5369999999999999,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.262},{"interface":"front","average":{"1min":1.038,"5min":1.1060000000000001,"15min":1.1100000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.7310000000000001,"5min":1.7829999999999999,"15min":1.8520000000000001},"last":0.80000000000000004}]},{"osd":4,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3,"15min":1.3340000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.552,"5min":1.754,"15min":2.1459999999999999},"last":1.903},{"interface":"front","average":{"1min":1.274,"5min":1.393,"15min":1.4299999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":2.0110000000000001},"last":1.577}]},{"osd":5,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.335,"15min":1.3480000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.82099999999999995},"max":{"1min":1.72,"5min":1.833,"15min":1.9650000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.393,"15min":1.45},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.6180000000000001,"5min":1.8999999999999999,"15min":2.323},"last":2.0579999999999998}]},{"osd":6,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.393,"15min":1.4159999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.6479999999999999,"5min":1.972,"15min":2.2160000000000002},"last":1.167},{"interface":"front","average":{"1min":1.365,"5min":1.502,"15min":1.526},"min":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":0.95599999999999996},"max":{"1min":1.671,"5min":1.861,"15min":2.3109999999999999},"last":2.1629999999999998}]},{"osd":7,"last update":"Tue Dec 19 16:11:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.51,"15min":1.5409999999999999},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":2.0489999999999999,"5min":2.2010000000000001,"15min":2.427},"last":1.661},{"interface":"front","average":{"1min":1.335,"5min":1.486,"15min":1.502},"min":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":0.97799999999999998},"max":{"1min":1.952,"5min":2.0670000000000002,"15min":2.3170000000000002},"last":1.633}]}]},{"osd":0,"up_from":13,"seq":55834575952,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300292,"kb_used_data":3004,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071548,"statfs":{"total":96636764160,"available":96329265152,"internally_reserved":0,"allocated":3076096,"data_stored":1855933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.71299999999999997,"5min":0.629,"15min":0.439},"max":{"1min":1.29,"5min":1.4530000000000001,"15min":1.6419999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0089999999999999},"min":{"1min":0.77800000000000002,"5min":0.75900000000000001,"15min":0.52200000000000002},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.093}]},{"osd":2,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1830000000000001,"15min":1.145},"min":{"1min":0.83299999999999996,"5min":0.68300000000000005,"15min":0.57099999999999995},"max":{"1min":1.6359999999999999,"5min":1.7410000000000001,"15min":1.8400000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.29,"15min":1.278},"min":{"1min":0.99199999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.655,"5min":1.8440000000000001,"15min":1.8500000000000001},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.081,"15min":1.0640000000000001},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.65300000000000002},"max":{"1min":1.601,"5min":1.601,"15min":1.6359999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.2809999999999999,"15min":1.2609999999999999},"min":{"1min":1.012,"5min":0.88500000000000001,"15min":0.55300000000000005},"max":{"1min":1.601,"5min":1.752,"15min":1.968},"last":0.89300000000000002}]},{"osd":4,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":1.175,"5min":1.0600000000000001,"15min":0.69499999999999995},"max":{"1min":1.885,"5min":1.885,"15min":2.0209999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.379,"15min":1.3360000000000001},"min":{"1min":1.0860000000000001,"5min":1.034,"15min":0.71999999999999997},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.488,"15min":1.448},"min":{"1min":1.2689999999999999,"5min":1.1499999999999999,"15min":0.90300000000000002},"max":{"1min":1.7969999999999999,"5min":2.0249999999999999,"15min":2.0779999999999998},"last":1.133},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.48,"15min":1.462},"min":{"1min":1.2969999999999999,"5min":1.079,"15min":0.98699999999999999},"max":{"1min":1.853,"5min":1.901,"15min":2.0539999999999998},"last":1.4410000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.4690000000000001,"15min":1.429},"min":{"1min":1.1699999999999999,"5min":1.0780000000000001,"15min":0.95299999999999996},"max":{"1min":1.8779999999999999,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.5640000000000001,"15min":1.5229999999999999},"min":{"1min":1.278,"5min":1.1559999999999999,"15min":0.93899999999999995},"max":{"1min":1.9159999999999999,"5min":2.0390000000000001,"15min":2.1669999999999998},"last":1.056}]},{"osd":7,"last update":"Tue Dec 19 16:10:47 2023","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.579,"15min":1.538},"min":{"1min":1.194,"5min":1.0469999999999999,"15min":0.97899999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.532},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.54,"15min":1.516},"min":{"1min":1.248,"5min":1.119,"15min":1.008},"max":{"1min":2.0219999999999998,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.3540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:11:45.248 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T16:11:45.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T16:11:45.440 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T16:11:45.434+0000 7fe953fa2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T16:11:45.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.458+0000 7f91212e3700 1 Processor -- start 2023-12-19T16:11:45.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.458+0000 7f91212e3700 1 -- start start 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f911c0ce2d0 0x7f911c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0caa10 0x7f911c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f911c0cc040 0x7f911c158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911c15d960 con 0x7f911c0ce2d0 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911c0cbbb0 con 0x7f911c0cc040 2023-12-19T16:11:45.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f91212e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c06dc90 con 0x7f911c0caa10 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f911c0ce2d0 0x7f911c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f911c0ce2d0 0x7f911c0ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35246/0 (socket says 172.21.15.50:35246) 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 -- 172.21.15.50:0/3504659368 learned_addr learned my addr 172.21.15.50:0/3504659368 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911b7fe700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0caa10 0x7f911c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f9120ae2700 1 --2- 172.21.15.50:0/3504659368 >> v2:172.21.15.125:3300/0 conn(0x7f911c0cc040 0x7f911c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 -- 172.21.15.50:0/3504659368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0caa10 msgr2=0x7f911c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0caa10 0x7f911c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 -- 172.21.15.50:0/3504659368 >> v2:172.21.15.125:3300/0 conn(0x7f911c0cc040 msgr2=0x7f911c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 --2- 172.21.15.50:0/3504659368 >> v2:172.21.15.125:3300/0 conn(0x7f911c0cc040 0x7f911c158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.462+0000 7f911bfff700 1 -- 172.21.15.50:0/3504659368 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c0cafa0 con 0x7f911c0ce2d0 2023-12-19T16:11:45.471 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f911bfff700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0ce2d0 0x7f911c0ce6a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f911c0b2ab0 tx=0x7f910c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5620bf83a93da3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f911affd700 1 -- 172.21.15.50:0/3504659368 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c00b860 con 0x7f911c0ce2d0 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f911affd700 1 -- 172.21.15.50:0/3504659368 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c00b9c0 con 0x7f911c0ce2d0 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f911affd700 1 -- 172.21.15.50:0/3504659368 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c00bd60 con 0x7f911c0ce2d0 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 -- 172.21.15.50:0/3504659368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0ce2d0 msgr2=0x7f911c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0ce2d0 0x7f911c0ce6a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f911c0b2ab0 tx=0x7f910c009250 comp rx=0 tx=0).stop 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 -- 172.21.15.50:0/3504659368 shutdown_connections 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 --2- 172.21.15.50:0/3504659368 >> v2:172.21.15.125:3300/0 conn(0x7f911c0cc040 0x7f911c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0caa10 0x7f911c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 --2- 172.21.15.50:0/3504659368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0ce2d0 0x7f911c0ce6a0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 -- 172.21.15.50:0/3504659368 >> 172.21.15.50:0/3504659368 conn(0x7f911c067bb0 msgr2=0x7f911c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 -- 172.21.15.50:0/3504659368 shutdown_connections 2023-12-19T16:11:45.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.466+0000 7f91212e3700 1 -- 172.21.15.50:0/3504659368 wait complete. 2023-12-19T16:11:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 Processor -- start 2023-12-19T16:11:45.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 -- start start 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 0x7f911c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 0x7f911c15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c15c180 con 0x7f911c0caa10 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c15b570 con 0x7f911c0cc040 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f91212e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c15b850 con 0x7f911c0ce2d0 2023-12-19T16:11:45.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f911bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f911bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35254/0 (socket says 172.21.15.50:35254) 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f9120ae2700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 0x7f911c15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.470+0000 7f911bfff700 1 -- 172.21.15.50:0/3500018204 learned_addr learned my addr 172.21.15.50:0/3500018204 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f911b7fe700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 0x7f911c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 -- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 msgr2=0x7f911c15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 0x7f911c15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 -- 172.21.15.50:0/3500018204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 msgr2=0x7f911c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 -- 172.21.15.50:0/3500018204 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0094f0 con 0x7f911c0ce2d0 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f911b7fe700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 0x7f911c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f911bfff700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:45.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9120ae2700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 0x7f911c15af80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9114007ed0 tx=0x7f9114009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114018070 con 0x7f911c0ce2d0 2023-12-19T16:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911400ec40 con 0x7f911c0ce2d0 2023-12-19T16:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c15bb30 con 0x7f911c0ce2d0 2023-12-19T16:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114014420 con 0x7f911c0ce2d0 2023-12-19T16:11:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c16c6f0 con 0x7f911c0ce2d0 2023-12-19T16:11:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.474+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c04be40 con 0x7f911c0ce2d0 2023-12-19T16:11:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.478+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9114003020 con 0x7f911c0ce2d0 2023-12-19T16:11:45.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.478+0000 7f9103fff700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 0x7f9108062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.482+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f911407cca0 con 0x7f911c0ce2d0 2023-12-19T16:11:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.482+0000 7f911bfff700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 0x7f9108062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.482+0000 7f911bfff700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 0x7f9108062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f910c00c010 tx=0x7f910c017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.486+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f911404cf50 con 0x7f911c0ce2d0 2023-12-19T16:11:45.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.714+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f911c0cb280 con 0x7f911c0ce2d0 2023-12-19T16:11:45.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.730+0000 7f9103fff700 1 -- 172.21.15.50:0/3500018204 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9114050570 con 0x7f911c0ce2d0 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 msgr2=0x7f9108062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 0x7f9108062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f910c00c010 tx=0x7f910c017000 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 msgr2=0x7f911c15af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 0x7f911c15af80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9114007ed0 tx=0x7f9114009850 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 shutdown_connections 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f91080606a0 0x7f9108062b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f911c0cc040 0x7f911c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f911c0ce2d0 0x7f911c15af80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 --2- 172.21.15.50:0/3500018204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f911c0caa10 0x7f911c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 >> 172.21.15.50:0/3500018204 conn(0x7f911c067bb0 msgr2=0x7f911c155620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 shutdown_connections 2023-12-19T16:11:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.734+0000 7f91212e3700 1 -- 172.21.15.50:0/3500018204 wait complete. 2023-12-19T16:11:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T16:11:45.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:11:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.918+0000 7f263b59e700 1 Processor -- start 2023-12-19T16:11:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.918+0000 7f263b59e700 1 -- start start 2023-12-19T16:11:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263b59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f263c06aa00 0x7f263c06d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263b59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06c7c0 con 0x7f263c06a580 2023-12-19T16:11:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263b59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06cd80 con 0x7f263c0b1410 2023-12-19T16:11:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263b59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c06b8f0 con 0x7f263c06aa00 2023-12-19T16:11:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2631d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f263c06a580 0x7f263c06d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50494/0 (socket says 172.21.15.50:50494) 2023-12-19T16:11:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2631d9b700 1 -- 172.21.15.50:0/2197164004 learned_addr learned my addr 172.21.15.50:0/2197164004 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263ad9d700 1 --2- 172.21.15.50:0/2197164004 >> v2:172.21.15.50:3300/0 conn(0x7f263c06aa00 0x7f263c06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972117967 0 0) 0x7f263c06c7c0 con 0x7f263c06a580 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628003570 con 0x7f263c06a580 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602408605 0 0) 0x7f263c06cd80 con 0x7f263c0b1410 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c06c7c0 con 0x7f263c0b1410 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263ad9d700 1 -- 172.21.15.50:0/2197164004 >> v1:172.21.15.125:6789/0 conn(0x7f263c0b1410 legacy=0x7f263c0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263ad9d700 1 -- 172.21.15.50:0/2197164004 >> v1:172.21.15.50:6789/0 conn(0x7f263c06a580 legacy=0x7f263c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.922+0000 7f263ad9d700 1 -- 172.21.15.50:0/2197164004 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0ac1e0 con 0x7f263c06aa00 2023-12-19T16:11:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263ad9d700 1 --2- 172.21.15.50:0/2197164004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c06d920 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f2634009e60 tx=0x7f2634009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b04a4f32fe3868 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400e070 con 0x7f263c06aa00 2023-12-19T16:11:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263400bab0 con 0x7f263c06aa00 2023-12-19T16:11:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f2639d9b700 1 -- 172.21.15.50:0/2197164004 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400be70 con 0x7f263c06aa00 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 -- 172.21.15.50:0/2197164004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 msgr2=0x7f263c06d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 --2- 172.21.15.50:0/2197164004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c06d920 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f2634009e60 tx=0x7f2634009780 comp rx=0 tx=0).stop 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 -- 172.21.15.50:0/2197164004 shutdown_connections 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 --2- 172.21.15.50:0/2197164004 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c06d920 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 -- 172.21.15.50:0/2197164004 >> 172.21.15.50:0/2197164004 conn(0x7f263c067bb0 msgr2=0x7f263c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:45.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 -- 172.21.15.50:0/2197164004 shutdown_connections 2023-12-19T16:11:45.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.926+0000 7f263b59e700 1 -- 172.21.15.50:0/2197164004 wait complete. 2023-12-19T16:11:45.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 Processor -- start 2023-12-19T16:11:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 -- start start 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f263c06a580 0x7f263c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 0x7f263c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c151080 con 0x7f263c06aa00 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c169050 con 0x7f263c06a580 2023-12-19T16:11:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c15d110 con 0x7f263c0b1410 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f2631d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f2631d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35280/0 (socket says 172.21.15.50:35280) 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f2631d9b700 1 -- 172.21.15.50:0/3788007105 learned_addr learned my addr 172.21.15.50:0/3788007105 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263a59c700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f263c06a580 0x7f263c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.930+0000 7f263ad9d700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 0x7f263c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 -- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 msgr2=0x7f263c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 0x7f263c15b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 -- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f263c06a580 msgr2=0x7f263c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f263c06a580 0x7f263c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 -- 172.21.15.50:0/3788007105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634009a20 con 0x7f263c06aa00 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f263ad9d700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 0x7f263c15b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2631d9b700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f262400c790 tx=0x7f2624007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624005b30 con 0x7f263c06aa00 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2624005c90 con 0x7f263c06aa00 2023-12-19T16:11:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c15d3f0 con 0x7f263c06aa00 2023-12-19T16:11:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624007300 con 0x7f263c06aa00 2023-12-19T16:11:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c15cdc0 con 0x7f263c06aa00 2023-12-19T16:11:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.934+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c1691f0 con 0x7f263c06aa00 2023-12-19T16:11:45.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.938+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f262401b3b0 con 0x7f263c06aa00 2023-12-19T16:11:45.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.938+0000 7f2632ffd700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 0x7f261c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.938+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f262407bfe0 con 0x7f263c06aa00 2023-12-19T16:11:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.938+0000 7f263a59c700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 0x7f261c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.942+0000 7f263a59c700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 0x7f261c062b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f262c007fd0 tx=0x7f262c007d50 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:45.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:45.946+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f263c1691f0 con 0x7f263c06aa00 2023-12-19T16:11:46.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.150+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f263c169190 con 0x7f263c06aa00 2023-12-19T16:11:46.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f2632ffd700 1 -- 172.21.15.50:0/3788007105 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f262404fd80 con 0x7f263c06aa00 2023-12-19T16:11:46.158 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:46.158 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 msgr2=0x7f261c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 0x7f261c062b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f262c007fd0 tx=0x7f262c007d50 comp rx=0 tx=0).stop 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 msgr2=0x7f263c150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f262400c790 tx=0x7f2624007ea0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 shutdown_connections 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f261c0606a0 0x7f261c062b50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f263c0b1410 0x7f263c15b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f263c06aa00 0x7f263c150b40 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 --2- 172.21.15.50:0/3788007105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f263c06a580 0x7f263c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 >> 172.21.15.50:0/3788007105 conn(0x7f263c067bb0 msgr2=0x7f263c0c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 shutdown_connections 2023-12-19T16:11:46.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.154+0000 7f263b59e700 1 -- 172.21.15.50:0/3788007105 wait complete. 2023-12-19T16:11:46.164 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:11:46.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T16:11:39.915607+0000', 'modified': '2023-12-19T16:11:43.508152+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9894}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9894': {'gid': 9894, 'name': 'c', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/3963498493', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 3963498493}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 3963498493}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T16:11:46.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T16:11:46.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T16:11:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.750+0000 7fb6b8545700 1 Processor -- start 2023-12-19T16:11:46.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.750+0000 7fb6b8545700 1 -- start start 2023-12-19T16:11:46.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b8545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 0x7fb6b015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b8545700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b8545700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b006d7d0 con 0x7fb6b01407f0 2023-12-19T16:11:46.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b8545700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b00ae280 con 0x7fb6b006c7c0 2023-12-19T16:11:46.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b8545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00cb8f0 con 0x7fb6b0140420 2023-12-19T16:11:46.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b5ae0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b6ae2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb6b01407f0 0x7fb6b006a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56048/0 (socket says 172.21.15.50:56048) 2023-12-19T16:11:46.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b5ae0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52860/0 (socket says 172.21.15.50:52860) 2023-12-19T16:11:46.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.754+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/871683618 learned_addr learned my addr 172.21.15.50:0/871683618 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:46.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 0x7fb6b015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/871683618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 msgr2=0x7fb6b015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 0x7fb6b015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/871683618 >> v1:172.21.15.50:6789/0 conn(0x7fb6b01407f0 legacy=0x7fb6b006a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/871683618 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b006cc50 con 0x7fb6b006c7c0 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 0x7fb6b015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b52df700 1 -- 172.21.15.50:0/871683618 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550921901 0 0) 0x7fb6b006d7d0 con 0x7fb6b01407f0 2023-12-19T16:11:46.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00098e0 tx=0x7fb6a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ae52fe10debade server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b52df700 1 -- 172.21.15.50:0/871683618 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000b640 con 0x7fb6b006c7c0 2023-12-19T16:11:46.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b52df700 1 -- 172.21.15.50:0/871683618 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a000b7a0 con 0x7fb6b006c7c0 2023-12-19T16:11:46.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b52df700 1 -- 172.21.15.50:0/871683618 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000bb40 con 0x7fb6b006c7c0 2023-12-19T16:11:46.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 -- 172.21.15.50:0/871683618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b006c7c0 msgr2=0x7fb6b00b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00098e0 tx=0x7fb6a00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 -- 172.21.15.50:0/871683618 shutdown_connections 2023-12-19T16:11:46.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b006c7c0 0x7fb6b00b0cd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 --2- 172.21.15.50:0/871683618 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b0140420 0x7fb6b015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 -- 172.21.15.50:0/871683618 >> 172.21.15.50:0/871683618 conn(0x7fb6b0067b60 msgr2=0x7fb6b00ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 -- 172.21.15.50:0/871683618 shutdown_connections 2023-12-19T16:11:46.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.758+0000 7fb6b8545700 1 -- 172.21.15.50:0/871683618 wait complete. 2023-12-19T16:11:46.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.762+0000 7fb6b8545700 1 Processor -- start 2023-12-19T16:11:46.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.762+0000 7fb6b8545700 1 -- start start 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b006c7c0 0x7fb6b01365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6b01407f0 0x7fb6b0139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b0136b00 con 0x7fb6b0140420 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00c5010 con 0x7fb6b01407f0 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00c7c40 con 0x7fb6b006c7c0 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52862/0 (socket says 172.21.15.50:52862) 2023-12-19T16:11:46.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2996745662 learned_addr learned my addr 172.21.15.50:0/2996745662 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6b01407f0 0x7fb6b0139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b006c7c0 0x7fb6b01365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b006c7c0 msgr2=0x7fb6b01365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b006c7c0 0x7fb6b01365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6b01407f0 msgr2=0x7fb6b0139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6b01407f0 0x7fb6b0139070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2996745662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac009420 con 0x7fb6b0140420 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0069d50 tx=0x7fb6a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6a67fc700 1 -- 172.21.15.50:0/2996745662 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000bb20 con 0x7fb6b0140420 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6a67fc700 1 -- 172.21.15.50:0/2996745662 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a0011420 con 0x7fb6b0140420 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a0009420 con 0x7fb6b0140420 2023-12-19T16:11:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6a67fc700 1 -- 172.21.15.50:0/2996745662 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00117c0 con 0x7fb6b0140420 2023-12-19T16:11:46.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.766+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b00c5540 con 0x7fb6b0140420 2023-12-19T16:11:46.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.770+0000 7fb6a67fc700 1 -- 172.21.15.50:0/2996745662 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb6a0011b30 con 0x7fb6b0140420 2023-12-19T16:11:46.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.774+0000 7fb6a67fc700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 0x7fb694062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.774+0000 7fb6a67fc700 1 -- 172.21.15.50:0/2996745662 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fb6a007e130 con 0x7fb6b0140420 2023-12-19T16:11:46.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.774+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 0x7fb694062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 0x7fb694062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0093f0 tx=0x7fb6ac008040 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 Processor -- start 2023-12-19T16:11:46.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 -- start start 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ee750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1713f30 con 0x1568710 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x170fed0 con 0x1568ae0 2023-12-19T16:11:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x16d60b0 con 0x15b3790 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52872/0 (socket says 172.21.15.50:52872) 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2627905146 learned_addr learned my addr 172.21.15.50:0/2627905146 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 msgr2=0x17ef1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17ef1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 msgr2=0x17ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 -- 172.21.15.50:0/2627905146 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x15b49e0 con 0x1568710 2023-12-19T16:11:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17ef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:46.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:11:46.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000b770 tx=0x7fb6a00092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7175c9d5925654d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6937fe700 1 -- 172.21.15.50:0/2627905146 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0059080 con 0x1568710 2023-12-19T16:11:46.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6937fe700 1 -- 172.21.15.50:0/2627905146 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a00501e0 con 0x1568710 2023-12-19T16:11:46.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.778+0000 7fb6937fe700 1 -- 172.21.15.50:0/2627905146 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0052340 con 0x1568710 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- 172.21.15.50:0/2627905146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 msgr2=0x17eec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000b770 tx=0x7fb6a00092a0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- 172.21.15.50:0/2627905146 shutdown_connections 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17ef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17eec90 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- 172.21.15.50:0/2627905146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- 172.21.15.50:0/2627905146 >> 172.21.15.50:0/2627905146 conn(0x1568eb0 msgr2=0x17a2930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- 172.21.15.50:0/2627905146 shutdown_connections 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- 172.21.15.50:0/2627905146 wait complete. 2023-12-19T16:11:46.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- start start 2023-12-19T16:11:46.788 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 Processor -- start 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568710 0x17ca160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x16d5d50 con 0x15b3790 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x16ae970 con 0x1568ae0 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1712590 con 0x1568710 2023-12-19T16:11:46.789 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6ba820740 10 client.0 _ll_register_callbacks cb 0x17596a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6b6ae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6b6ae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52876/0 (socket says 172.21.15.50:52876) 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.782+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/64650414 learned_addr learned my addr 172.21.15.50:0/64650414 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568710 0x17ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568710 msgr2=0x17ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568710 0x17ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 msgr2=0x17ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/64650414 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b00c5540 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8007570 tx=0x7fb6a8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fd27cb708d6c37 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800d070 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fb6a8006400 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x16aecc0 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6a80017d0 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 10 client.10020 fetch_fsmap learned FSMap version 90 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x16de370 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x16aecc0 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 90) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fb6a801c020 con 0x15b3790 2023-12-19T16:11:46.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 10 client.10020 fetch_fsmap finished waiting for FSMap version 90 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 10 client.10020 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 20 client.10020 populate_metadata read hostname 'smithi050' 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x17ae100 0x17fdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 4 client.10020 mds_command: new command op to 9894 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 --> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x16b23e0 con 0x17ae100 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.786+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x17ae100 0x17fdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb690ff9700 10 client.10020 ms_handle_connect on v2:172.21.15.50:6834/3963498493 2023-12-19T16:11:46.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x17ae100 0x17fdae0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42e149df9dee044e server_cookie=d866106847e9a62b in_seq=0 out_seq=0 2023-12-19T16:11:46.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb690ff9700 1 -- 172.21.15.50:0/64650414 <== mds.0 v2:172.21.15.50:6834/3963498493 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x16b23e0 con 0x17ae100 2023-12-19T16:11:46.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb690ff9700 10 client.10020 handle_command_reply: tid=0 2023-12-19T16:11:46.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 client.10020 shutdown 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 msgr2=0x17ad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8007570 tx=0x7fb6a8009040 comp rx=0 tx=0).stop 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 shutdown_connections 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x17ae100 0x17fdae0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ad370 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 --2- 172.21.15.50:0/64650414 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568710 0x17ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb690ff9700 0 client.10020 ms_handle_reset on v2:172.21.15.50:6834/3963498493 2023-12-19T16:11:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 >> 172.21.15.50:0/64650414 conn(0x1568eb0 msgr2=0x179af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 shutdown_connections 2023-12-19T16:11:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 1 -- 172.21.15.50:0/64650414 wait complete. 2023-12-19T16:11:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.790+0000 7fb6ba820740 20 client.10020 trim_cache size 0 max 16384 2023-12-19T16:11:46.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 Processor -- start 2023-12-19T16:11:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 -- start start 2023-12-19T16:11:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x170bb50 con 0x1568710 2023-12-19T16:11:46.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1710770 con 0x1568ae0 2023-12-19T16:11:46.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.814+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x16bee00 con 0x15b3790 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52886/0 (socket says 172.21.15.50:52886) 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 -- 172.21.15.50:0/3303600679 learned_addr learned my addr 172.21.15.50:0/3303600679 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 -- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 msgr2=0x17a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 -- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 msgr2=0x17ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 -- 172.21.15.50:0/3303600679 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b005eb20 con 0x1568710 2023-12-19T16:11:46.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b5ae0700 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:46.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac015690 tx=0x7fb6ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd99c2e91d6538e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb693fff700 1 -- 172.21.15.50:0/3303600679 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac027040 con 0x1568710 2023-12-19T16:11:46.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb693fff700 1 -- 172.21.15.50:0/3303600679 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac011a50 con 0x1568710 2023-12-19T16:11:46.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb693fff700 1 -- 172.21.15.50:0/3303600679 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac02b410 con 0x1568710 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 -- 172.21.15.50:0/3303600679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 msgr2=0x17abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac015690 tx=0x7fb6ac018000 comp rx=0 tx=0).stop 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 -- 172.21.15.50:0/3303600679 shutdown_connections 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568ae0 0x17ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x15b3790 0x17a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 --2- 172.21.15.50:0/3303600679 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x1568710 0x17abe60 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 -- 172.21.15.50:0/3303600679 >> 172.21.15.50:0/3303600679 conn(0x1568eb0 msgr2=0x17acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 -- 172.21.15.50:0/3303600679 shutdown_connections 2023-12-19T16:11:46.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.818+0000 7fb6ba820740 1 -- 172.21.15.50:0/3303600679 wait complete. 2023-12-19T16:11:46.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 -- start start 2023-12-19T16:11:46.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 Processor -- start 2023-12-19T16:11:46.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568710 0x1847d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 0x1848260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x16d9400 con 0x15b3790 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x17139d0 con 0x1568710 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x16e3f40 con 0x1568ae0 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.822+0000 7fb6ba820740 10 client.0 _ll_register_callbacks cb 0x15349d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 0x1848260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52888/0 (socket says 172.21.15.50:52888) 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b5ae0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 0x1848260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44186/0 (socket says 172.21.15.50:44186) 2023-12-19T16:11:46.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/365691307 learned_addr learned my addr 172.21.15.50:0/365691307 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:46.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568710 0x1847d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 msgr2=0x1848260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 0x1848260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568710 msgr2=0x1847d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568710 0x1847d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 -- 172.21.15.50:0/365691307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x15b49e0 con 0x15b3790 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b6ae2700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006ef0 tx=0x7fb6a801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3c0ce9c1f9e885 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80017d0 con 0x15b3790 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fb6a8004bb0 con 0x15b3790 2023-12-19T16:11:46.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6a00501e0 con 0x15b3790 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6a8001930 con 0x15b3790 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 10 client.10026 fetch_fsmap learned FSMap version 90 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x7fb6ac015440 con 0x15b3790 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a00501e0 con 0x15b3790 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 90) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fb6a80075a0 con 0x15b3790 2023-12-19T16:11:46.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 10 client.10026 fetch_fsmap finished waiting for FSMap version 90 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 10 client.10026 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 20 client.10026 populate_metadata read hostname 'smithi050' 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x184ebb0 0x1850f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 4 client.10026 mds_command: new command op to 9894 tid=0[{"prefix": "status"}] 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 --> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] -- command(tid 0: {"prefix": "status"}) v1 -- 0x16e4410 con 0x184ebb0 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.826+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x184ebb0 0x1850f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb690ff9700 10 client.10026 ms_handle_connect on v2:172.21.15.50:6834/3963498493 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b62e1700 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x184ebb0 0x1850f90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=511575baeb7412d3 server_cookie=33571146e5030b7c in_seq=0 out_seq=0 2023-12-19T16:11:46.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb690ff9700 1 -- 172.21.15.50:0/365691307 <== mds.0 v2:172.21.15.50:6834/3963498493 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x16e4410 con 0x184ebb0 2023-12-19T16:11:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb690ff9700 10 client.10026 handle_command_reply: tid=0 2023-12-19T16:11:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 client.10026 shutdown 2023-12-19T16:11:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 msgr2=0x17ea200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006ef0 tx=0x7fb6a801f000 comp rx=0 tx=0).stop 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 shutdown_connections 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:6834/3963498493,v1:172.21.15.50:6835/3963498493] conn(0x184ebb0 0x1850f90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x15b3790 0x17ea200 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x1568ae0 0x1848260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 --2- 172.21.15.50:0/365691307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1568710 0x1847d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 >> 172.21.15.50:0/365691307 conn(0x1568eb0 msgr2=0x179b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb690ff9700 0 client.10026 ms_handle_reset on v2:172.21.15.50:6834/3963498493 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 shutdown_connections 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 1 -- 172.21.15.50:0/365691307 wait complete. 2023-12-19T16:11:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6ba820740 20 client.10026 trim_cache size 0 max 16384 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 9894, 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3646456760000003, 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 90, 2023-12-19T16:11:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 199, 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 199, 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.25472757 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 msgr2=0x7fb694062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 0x7fb694062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0093f0 tx=0x7fb6ac008040 comp rx=0 tx=0).stop 2023-12-19T16:11:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 msgr2=0x7fb6b00c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0069d50 tx=0x7fb6a0017000 comp rx=0 tx=0).stop 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 shutdown_connections 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb694060670 0x7fb694062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb6b01407f0 0x7fb6b0139070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6b0140420 0x7fb6b00c7260 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 --2- 172.21.15.50:0/2996745662 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb6b006c7c0 0x7fb6b01365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.830+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 >> 172.21.15.50:0/2996745662 conn(0x7fb6b0067b60 msgr2=0x7fb6b00b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.834+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 shutdown_connections 2023-12-19T16:11:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:46.834+0000 7fb6b8545700 1 -- 172.21.15.50:0/2996745662 wait complete. 2023-12-19T16:11:46.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T16:11:47.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.022+0000 7fabd2123700 1 Processor -- start 2023-12-19T16:11:47.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.022+0000 7fabd2123700 1 -- start start 2023-12-19T16:11:47.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd2123700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fabcc1591f0 0x7fabcc06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd2123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 0x7fabcc0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd2123700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc06b730 con 0x7fabcc158e20 2023-12-19T16:11:47.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd2123700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabcc06ab20 con 0x7fabcc1591f0 2023-12-19T16:11:47.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd2123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc06ae00 con 0x7fabcc0ce810 2023-12-19T16:11:47.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd0920700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fabcc1591f0 0x7fabcc06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1922700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 0x7fabcc0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1121700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fabcc158e20 0x7fabcc0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:42662/0 (socket says 172.21.15.50:42662) 2023-12-19T16:11:47.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1121700 1 -- 172.21.15.50:0/1772264334 learned_addr learned my addr 172.21.15.50:0/1772264334 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabc3fff700 1 -- 172.21.15.50:0/1772264334 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184216198 0 0) 0x7fabcc06b730 con 0x7fabcc158e20 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabc3fff700 1 -- 172.21.15.50:0/1772264334 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003570 con 0x7fabcc158e20 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1922700 1 -- 172.21.15.50:0/1772264334 >> v2:172.21.15.50:3300/0 conn(0x7fabcc1591f0 msgr2=0x7fabcc06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1922700 1 --2- 172.21.15.50:0/1772264334 >> v2:172.21.15.50:3300/0 conn(0x7fabcc1591f0 0x7fabcc06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1922700 1 -- 172.21.15.50:0/1772264334 >> v1:172.21.15.125:6789/0 conn(0x7fabcc158e20 legacy=0x7fabcc0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.026+0000 7fabd1922700 1 -- 172.21.15.50:0/1772264334 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc06b0e0 con 0x7fabcc0ce810 2023-12-19T16:11:47.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd1922700 1 --2- 172.21.15.50:0/1772264334 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 0x7fabcc0ca950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009e60 tx=0x7fabc4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585a664f4c7da2b5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabc3fff700 1 -- 172.21.15.50:0/1772264334 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400e070 con 0x7fabcc0ce810 2023-12-19T16:11:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabc3fff700 1 -- 172.21.15.50:0/1772264334 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400bab0 con 0x7fabcc0ce810 2023-12-19T16:11:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabc3fff700 1 -- 172.21.15.50:0/1772264334 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400be70 con 0x7fabcc0ce810 2023-12-19T16:11:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 -- 172.21.15.50:0/1772264334 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 msgr2=0x7fabcc0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 --2- 172.21.15.50:0/1772264334 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 0x7fabcc0ca950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009e60 tx=0x7fabc4009780 comp rx=0 tx=0).stop 2023-12-19T16:11:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 -- 172.21.15.50:0/1772264334 shutdown_connections 2023-12-19T16:11:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 --2- 172.21.15.50:0/1772264334 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc0ce810 0x7fabcc0ca950 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 --2- 172.21.15.50:0/1772264334 >> v2:172.21.15.50:3300/0 conn(0x7fabcc1591f0 0x7fabcc06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 -- 172.21.15.50:0/1772264334 >> 172.21.15.50:0/1772264334 conn(0x7fabcc067bb0 msgr2=0x7fabcc0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:47.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 -- 172.21.15.50:0/1772264334 shutdown_connections 2023-12-19T16:11:47.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.030+0000 7fabd2123700 1 -- 172.21.15.50:0/1772264334 wait complete. 2023-12-19T16:11:47.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 Processor -- start 2023-12-19T16:11:47.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 -- start start 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabcc158e20 0x7fabcc0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc1591f0 0x7fabcc168d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc15a690 con 0x7fabcc0ce810 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc159a80 con 0x7fabcc158e20 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.034+0000 7fabd2123700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc159d60 con 0x7fabcc1591f0 2023-12-19T16:11:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52906/0 (socket says 172.21.15.50:52906) 2023-12-19T16:11:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 -- 172.21.15.50:0/1182796981 learned_addr learned my addr 172.21.15.50:0/1182796981 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:47.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd0920700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabcc158e20 0x7fabcc0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1922700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc1591f0 0x7fabcc168d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 -- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc1591f0 msgr2=0x7fabcc168d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc1591f0 0x7fabcc168d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 -- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabcc158e20 msgr2=0x7fabcc0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabcc158e20 0x7fabcc0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 -- 172.21.15.50:0/1182796981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4009a20 con 0x7fabcc0ce810 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd1121700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fabc800c7c0 tx=0x7fabc8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8005cd0 con 0x7fabcc0ce810 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8005e30 con 0x7fabcc0ce810 2023-12-19T16:11:47.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8007320 con 0x7fabcc0ce810 2023-12-19T16:11:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc15a040 con 0x7fabcc0ce810 2023-12-19T16:11:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc0c6fc0 con 0x7fabcc0ce810 2023-12-19T16:11:47.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.038+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc15a4c0 con 0x7fabcc0ce810 2023-12-19T16:11:47.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.042+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fabc801b3b0 con 0x7fabcc0ce810 2023-12-19T16:11:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.042+0000 7fabc17fa700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 0x7fabb0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.042+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fabc807c200 con 0x7fabcc0ce810 2023-12-19T16:11:47.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.042+0000 7fabd0920700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 0x7fabb0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:47.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.046+0000 7fabd0920700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 0x7fabb0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fabcc158860 tx=0x7fabbc0091a0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:47.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.050+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fabc804c4b0 con 0x7fabcc0ce810 2023-12-19T16:11:47.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.262+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fabcc04be40 con 0x7fabcc0ce810 2023-12-19T16:11:47.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.262+0000 7fabc17fa700 1 -- 172.21.15.50:0/1182796981 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) v1 ==== 53+0+73 (secure 0 0 0) 0x7fabc8017060 con 0x7fabcc0ce810 2023-12-19T16:11:47.267 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T16:11:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 msgr2=0x7fabb0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 0x7fabb0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fabcc158860 tx=0x7fabbc0091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 msgr2=0x7fabcc151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:47.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fabc800c7c0 tx=0x7fabc8007e00 comp rx=0 tx=0).stop 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 shutdown_connections 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fabb0060670 0x7fabb0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabcc158e20 0x7fabcc0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fabcc1591f0 0x7fabcc168d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 --2- 172.21.15.50:0/1182796981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabcc0ce810 0x7fabcc151ed0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 >> 172.21.15.50:0/1182796981 conn(0x7fabcc067bb0 msgr2=0x7fabcc0c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 shutdown_connections 2023-12-19T16:11:47.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:47.266+0000 7fabd2123700 1 -- 172.21.15.50:0/1182796981 wait complete. 2023-12-19T16:11:47.291 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T16:11:47.292 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:11:47.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.292 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T16:11:47.315 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T16:11:47.315 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T16:11:47.315 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T16:11:47.315 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T16:11:47.316 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 36843sec preferred_lft 36843sec 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T16:11:47.317 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.318 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:11:47.319 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T16:11:47.319 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T16:11:47.319 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:11:47.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.320 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T16:11:47.407 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T16:11:47.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.410 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T16:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T16:11:47.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T16:11:47.478 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T16:11:47.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.612 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T16:11:47.627 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T16:11:47.627 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T16:11:47.628 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:11:47.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:11:47.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T16:11:47.713 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T16:11:47.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:11:47.713 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:47.772 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:11:47.772 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:11:47.772 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:47.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:11:47.918 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:11:47.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:11:47.997 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:11:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:11:48.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:48.002 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:11:48.069 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T16:11:48.069 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T16:11:48.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:11:48.159 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:11:48.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:11:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:11:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:11:48.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:48.241 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:11:48.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T16:11:48.254+0000 7f32ac129380 -1 init, newargv = 0x55982d910a30 newargc=15 2023-12-19T16:11:48.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[60197]: starting ceph client 2023-12-19T16:11:48.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[60197]: starting fuse 2023-12-19T16:11:48.278 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T16:11:48.279 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T16:11:48.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T16:11:48.279 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T16:11:48.279 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T16:11:48.280 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T16:11:48.281 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T16:11:48.446 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.60197.asok 2023-12-19T16:11:48.628 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60197.asok 2023-12-19T16:11:48.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:48.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60197.asok status 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T16:11:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "60197", 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 10038, 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 10038 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3697811792 2023-12-19T16:11:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3697811792 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.10038 192.168.0.1:0/3697811792", 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/3697811792", 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 90, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 199, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T16:11:48.815 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:11:48.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:48.827 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T16:11:48.827 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:48.848 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T16:11:48.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:48.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:48.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:11:48.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T16:11:49.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 Processor -- start 2023-12-19T16:11:49.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 -- start start 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3b0140420 0x7fa3b015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b00b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa3b01407f0 0x7fa3b00b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b00ae280 con 0x7fa3b01407f0 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b006d7d0 con 0x7fa3b0140420 2023-12-19T16:11:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3b5985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b006cc50 con 0x7fa3b006c7c0 2023-12-19T16:11:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3af7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa3b01407f0 0x7fa3b00b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3af7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa3b01407f0 0x7fa3b00b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52916/0 (socket says 172.21.15.50:52916) 2023-12-19T16:11:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3aeffd700 1 --2- 172.21.15.50:0/841660333 >> v2:172.21.15.125:3300/0 conn(0x7fa3b0140420 0x7fa3b015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3af7fe700 1 -- 172.21.15.50:0/841660333 learned_addr learned my addr 172.21.15.50:0/841660333 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3a7fff700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b00b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3af7fe700 1 -- 172.21.15.50:0/841660333 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 msgr2=0x7fa3b00b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.142+0000 7fa3af7fe700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b00b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3af7fe700 1 -- 172.21.15.50:0/841660333 >> v2:172.21.15.125:3300/0 conn(0x7fa3b0140420 msgr2=0x7fa3b015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3af7fe700 1 --2- 172.21.15.50:0/841660333 >> v2:172.21.15.125:3300/0 conn(0x7fa3b0140420 0x7fa3b015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3af7fe700 1 -- 172.21.15.50:0/841660333 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b006cf30 con 0x7fa3b01407f0 2023-12-19T16:11:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3af7fe700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b01407f0 0x7fa3b00b1210 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fa3b006a340 tx=0x7fa3a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330e03bdedf80697 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3ae7fc700 1 -- 172.21.15.50:0/841660333 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800e070 con 0x7fa3b01407f0 2023-12-19T16:11:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3ae7fc700 1 -- 172.21.15.50:0/841660333 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a800ba20 con 0x7fa3b01407f0 2023-12-19T16:11:49.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3ae7fc700 1 -- 172.21.15.50:0/841660333 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80092e0 con 0x7fa3b01407f0 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 -- 172.21.15.50:0/841660333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b01407f0 msgr2=0x7fa3b00b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b01407f0 0x7fa3b00b1210 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fa3b006a340 tx=0x7fa3a8009840 comp rx=0 tx=0).stop 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 -- 172.21.15.50:0/841660333 shutdown_connections 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b01407f0 0x7fa3b00b1210 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 --2- 172.21.15.50:0/841660333 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b00b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 --2- 172.21.15.50:0/841660333 >> v2:172.21.15.125:3300/0 conn(0x7fa3b0140420 0x7fa3b015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 -- 172.21.15.50:0/841660333 >> 172.21.15.50:0/841660333 conn(0x7fa3b0067b60 msgr2=0x7fa3b00ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:49.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 -- 172.21.15.50:0/841660333 shutdown_connections 2023-12-19T16:11:49.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.146+0000 7fa3b5985700 1 -- 172.21.15.50:0/841660333 wait complete. 2023-12-19T16:11:49.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 Processor -- start 2023-12-19T16:11:49.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 -- start start 2023-12-19T16:11:49.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3b01407f0 0x7fa3b00c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00af840 con 0x7fa3b0140420 2023-12-19T16:11:49.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00afb20 con 0x7fa3b01407f0 2023-12-19T16:11:49.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3b5985700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00afe00 con 0x7fa3b006c7c0 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52926/0 (socket says 172.21.15.50:52926) 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 -- 172.21.15.50:0/1682736871 learned_addr learned my addr 172.21.15.50:0/1682736871 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3aeffd700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3af7fe700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3b01407f0 0x7fa3b00c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 -- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 msgr2=0x7fa3b013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 -- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3b01407f0 msgr2=0x7fa3b00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3b01407f0 0x7fa3b00c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 -- 172.21.15.50:0/1682736871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8009ae0 con 0x7fa3b0140420 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a7fff700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa39800d890 tx=0x7fa398008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.150+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3980125f0 con 0x7fa3b0140420 2023-12-19T16:11:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398012750 con 0x7fa3b0140420 2023-12-19T16:11:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398012af0 con 0x7fa3b0140420 2023-12-19T16:11:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b00b00e0 con 0x7fa3b0140420 2023-12-19T16:11:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b00af550 con 0x7fa3b0140420 2023-12-19T16:11:49.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b0071120 con 0x7fa3b0140420 2023-12-19T16:11:49.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.154+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa39801e460 con 0x7fa3b0140420 2023-12-19T16:11:49.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.158+0000 7fa3a6ffd700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 0x7fa390062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.158+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7fa39807d0f0 con 0x7fa3b0140420 2023-12-19T16:11:49.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.158+0000 7fa3aeffd700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 0x7fa390062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.158+0000 7fa3aeffd700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 0x7fa390062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009db0 tx=0x7fa3a00091a0 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.166+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa39804d3b0 con 0x7fa3b0140420 2023-12-19T16:11:49.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.362+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3b006d210 con 0x7fa390060670 2023-12-19T16:11:49.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.366+0000 7fa3a6ffd700 1 -- 172.21.15.50:0/1682736871 <== mgr.9678 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa3b006d210 con 0x7fa390060670 2023-12-19T16:11:49.372 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T16:11:49.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:11:49.373 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T16:11:49.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:11:49.373 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T16:11:49.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 msgr2=0x7fa390062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 0x7fa390062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009db0 tx=0x7fa3a00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 msgr2=0x7fa3b00c4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa39800d890 tx=0x7fa398008040 comp rx=0 tx=0).stop 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 shutdown_connections 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa390060670 0x7fa390062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3b01407f0 0x7fa3b00c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa3b0140420 0x7fa3b00c4fd0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 --2- 172.21.15.50:0/1682736871 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa3b006c7c0 0x7fa3b013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 >> 172.21.15.50:0/1682736871 conn(0x7fa3b0067b60 msgr2=0x7fa3b013dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:49.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 shutdown_connections 2023-12-19T16:11:49.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.370+0000 7fa3b5985700 1 -- 172.21.15.50:0/1682736871 wait complete. 2023-12-19T16:11:49.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T16:11:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 Processor -- start 2023-12-19T16:11:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 -- start start 2023-12-19T16:11:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3df015d410 0x7f3df00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df00ca400 0x7f3df00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00c7450 con 0x7f3df006b1b0 2023-12-19T16:11:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df00cad50 con 0x7f3df015d410 2023-12-19T16:11:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.558+0000 7f3df8185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df006bea0 con 0x7f3df00ca400 2023-12-19T16:11:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3df015d410 0x7f3df00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3df015d410 0x7f3df00ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34950/0 (socket says 172.21.15.50:34950) 2023-12-19T16:11:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 -- 172.21.15.50:0/2601263887 learned_addr learned my addr 172.21.15.50:0/2601263887 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5720700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df00ca400 0x7f3df00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 -- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df00ca400 msgr2=0x7f3df00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df00ca400 0x7f3df00ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 -- 172.21.15.50:0/2601263887 >> v1:172.21.15.125:6789/0 conn(0x7f3df006b1b0 legacy=0x7f3df006b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 -- 172.21.15.50:0/2601263887 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00c6e00 con 0x7f3df015d410 2023-12-19T16:11:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df5f21700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df00ab6c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0099b0 tx=0x7f3dec009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e512984abb72b53 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df4f1f700 1 -- 172.21.15.50:0/2601263887 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec009ea0 con 0x7f3df015d410 2023-12-19T16:11:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df4f1f700 1 -- 172.21.15.50:0/2601263887 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec00b860 con 0x7f3df015d410 2023-12-19T16:11:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df4f1f700 1 -- 172.21.15.50:0/2601263887 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00bc00 con 0x7f3df015d410 2023-12-19T16:11:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 -- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 msgr2=0x7f3df00ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df00ab6c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0099b0 tx=0x7f3dec009250 comp rx=0 tx=0).stop 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 -- 172.21.15.50:0/2601263887 shutdown_connections 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df00ca400 0x7f3df00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 --2- 172.21.15.50:0/2601263887 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df00ab6c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.562+0000 7f3df8185700 1 -- 172.21.15.50:0/2601263887 >> 172.21.15.50:0/2601263887 conn(0x7f3df0067bb0 msgr2=0x7f3df00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- 172.21.15.50:0/2601263887 shutdown_connections 2023-12-19T16:11:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- 172.21.15.50:0/2601263887 wait complete. 2023-12-19T16:11:49.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 Processor -- start 2023-12-19T16:11:49.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- start start 2023-12-19T16:11:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df006b1b0 0x7f3df00afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df016e500 con 0x7f3df00ca400 2023-12-19T16:11:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00aecd0 con 0x7f3df015d410 2023-12-19T16:11:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.566+0000 7f3df8185700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00aefb0 con 0x7f3df006b1b0 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df5720700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df5f21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df006b1b0 0x7f3df00afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34964/0 (socket says 172.21.15.50:34964) 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df5720700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52932/0 (socket says 172.21.15.50:52932) 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 -- 172.21.15.50:0/1873604317 learned_addr learned my addr 172.21.15.50:0/1873604317 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 -- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df006b1b0 msgr2=0x7f3df00afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df006b1b0 0x7f3df00afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 -- 172.21.15.50:0/1873604317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 msgr2=0x7f3df016dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 -- 172.21.15.50:0/1873604317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec0094f0 con 0x7f3df015d410 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df5720700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df6722700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c840 tx=0x7f3de8001670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8012630 con 0x7f3df015d410 2023-12-19T16:11:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80127b0 con 0x7f3df015d410 2023-12-19T16:11:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df00af290 con 0x7f3df015d410 2023-12-19T16:11:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8012b70 con 0x7f3df015d410 2023-12-19T16:11:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.570+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df00af850 con 0x7f3df015d410 2023-12-19T16:11:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.574+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df00afa70 con 0x7f3df015d410 2023-12-19T16:11:49.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.574+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 91) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3de801a3a0 con 0x7f3df015d410 2023-12-19T16:11:49.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.574+0000 7f3de5ffb700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 0x7f3dd4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:49.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.574+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f3de807cc90 con 0x7f3df015d410 2023-12-19T16:11:49.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.578+0000 7f3df5f21700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 0x7f3dd4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:49.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.578+0000 7f3df5f21700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 0x7f3dd4062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0099b0 tx=0x7f3dec017000 comp rx=0 tx=0).ready entity=mgr.9678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.582+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3de804cf40 con 0x7f3df015d410 2023-12-19T16:11:49.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:49.794+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3df0071120 con 0x7f3df015d410 2023-12-19T16:11:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.246+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 92) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f3de804d310 con 0x7f3df015d410 2023-12-19T16:11:50.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3de5ffb700 1 -- 172.21.15.50:0/1873604317 <== mon.1 v2:172.21.15.125:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v92) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3de80483b0 con 0x7f3df015d410 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 msgr2=0x7f3dd4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 0x7f3dd4062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0099b0 tx=0x7f3dec017000 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 msgr2=0x7f3df01599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c840 tx=0x7f3de8001670 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 shutdown_connections 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3dd4060670 0x7f3dd4062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3df015d410 0x7f3df01599b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3df00ca400 0x7f3df016dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 --2- 172.21.15.50:0/1873604317 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3df006b1b0 0x7f3df00afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 >> 172.21.15.50:0/1873604317 conn(0x7f3df0067bb0 msgr2=0x7f3df006a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:50.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 shutdown_connections 2023-12-19T16:11:50.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.250+0000 7f3df8185700 1 -- 172.21.15.50:0/1873604317 wait complete. 2023-12-19T16:11:50.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T16:11:50.439 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:50.431+0000 7f540d549000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:11:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.446+0000 7f63174a3700 1 Processor -- start 2023-12-19T16:11:50.451 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.446+0000 7f63174a3700 1 -- start start 2023-12-19T16:11:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6310158b00 0x7f6310158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f631006bb50 0x7f631015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631006b6c0 con 0x7f631006a4f0 2023-12-19T16:11:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631015d880 con 0x7f631006bb50 2023-12-19T16:11:50.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631006aab0 con 0x7f6310158b00 2023-12-19T16:11:50.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6310158b00 0x7f6310158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6310158b00 0x7f6310158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:34978/0 (socket says 172.21.15.50:34978) 2023-12-19T16:11:50.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 -- 172.21.15.50:0/2640011999 learned_addr learned my addr 172.21.15.50:0/2640011999 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:50.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f6315ca0700 1 --2- 172.21.15.50:0/2640011999 >> v2:172.21.15.50:3300/0 conn(0x7f631006bb50 0x7f631015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f6316ca2700 1 --1- 172.21.15.50:0/2640011999 >> v1:172.21.15.50:6789/0 conn(0x7f631006a4f0 0x7f63100cab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56074/0 (socket says 172.21.15.50:56074) 2023-12-19T16:11:50.455 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f59522ef000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:11:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 -- 172.21.15.50:0/2640011999 >> v2:172.21.15.50:3300/0 conn(0x7f631006bb50 msgr2=0x7f631015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 --2- 172.21.15.50:0/2640011999 >> v2:172.21.15.50:3300/0 conn(0x7f631006bb50 0x7f631015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 -- 172.21.15.50:0/2640011999 >> v1:172.21.15.50:6789/0 conn(0x7f631006a4f0 legacy=0x7f63100cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 -- 172.21.15.50:0/2640011999 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631006ad90 con 0x7f6310158b00 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f631549f700 1 -- 172.21.15.50:0/2640011999 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297721270 0 0) 0x7f631006b6c0 con 0x7f631006a4f0 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63164a1700 1 --2- 172.21.15.50:0/2640011999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6310158b00 0x7f6310158ed0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f631006a2e0 tx=0x7f630c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ab76ae854baa03c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f631549f700 1 -- 172.21.15.50:0/2640011999 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c009e70 con 0x7f6310158b00 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f631549f700 1 -- 172.21.15.50:0/2640011999 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c00b860 con 0x7f6310158b00 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f631549f700 1 -- 172.21.15.50:0/2640011999 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c00bc00 con 0x7f6310158b00 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 -- 172.21.15.50:0/2640011999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6310158b00 msgr2=0x7f6310158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.450+0000 7f63174a3700 1 --2- 172.21.15.50:0/2640011999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6310158b00 0x7f6310158ed0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f631006a2e0 tx=0x7f630c009250 comp rx=0 tx=0).stop 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 -- 172.21.15.50:0/2640011999 shutdown_connections 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 --2- 172.21.15.50:0/2640011999 >> v2:172.21.15.50:3300/0 conn(0x7f631006bb50 0x7f631015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 --2- 172.21.15.50:0/2640011999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6310158b00 0x7f6310158ed0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 -- 172.21.15.50:0/2640011999 >> 172.21.15.50:0/2640011999 conn(0x7f6310067bb0 msgr2=0x7f63100cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 -- 172.21.15.50:0/2640011999 shutdown_connections 2023-12-19T16:11:50.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 -- 172.21.15.50:0/2640011999 wait complete. 2023-12-19T16:11:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 Processor -- start 2023-12-19T16:11:50.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.454+0000 7f63174a3700 1 -- start start 2023-12-19T16:11:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 0x7f63100c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f631006bb50 0x7f631015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631015f220 con 0x7f6310158b00 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6310154b40 con 0x7f631006a4f0 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631015e5a0 con 0x7f631006bb50 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52968/0 (socket says 172.21.15.50:52968) 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 -- 172.21.15.50:0/1126729329 learned_addr learned my addr 172.21.15.50:0/1126729329 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63164a1700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 0x7f63100c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6315ca0700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f631006bb50 0x7f631015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 -- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f631006bb50 msgr2=0x7f631015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f631006bb50 0x7f631015d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 -- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 msgr2=0x7f63100c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 0x7f63100c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 -- 172.21.15.50:0/1126729329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c0094f0 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63164a1700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 0x7f63100c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f6316ca2700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f630800cd50 tx=0x7f6308001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308005df0 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630800a960 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631015e880 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630800ad20 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631015ee40 con 0x7f6310158b00 2023-12-19T16:11:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.458+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631004be40 con 0x7f6310158b00 2023-12-19T16:11:50.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.462+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 92) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f6308007300 con 0x7f6310158b00 2023-12-19T16:11:50.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.462+0000 7f63067fc700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 0x7f62f4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.462+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f630807cda0 con 0x7f6310158b00 2023-12-19T16:11:50.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.462+0000 7f63164a1700 1 -- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 msgr2=0x7f62f4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:50.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.462+0000 7f63164a1700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 0x7f62f4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:50.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.470+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f630804d0d0 con 0x7f6310158b00 2023-12-19T16:11:50.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.642+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f631006b070 con 0x7f6310158b00 2023-12-19T16:11:50.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.650+0000 7f63067fc700 1 -- 172.21.15.50:0/1126729329 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f63080506f0 con 0x7f6310158b00 2023-12-19T16:11:50.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 msgr2=0x7f62f4062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:50.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 0x7f62f4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 msgr2=0x7f631015df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f630800cd50 tx=0x7f6308001b10 comp rx=0 tx=0).stop 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 shutdown_connections 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f62f40606c0 0x7f62f4062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6310158b00 0x7f631015df00 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f631006bb50 0x7f631015d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 --2- 172.21.15.50:0/1126729329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f631006a4f0 0x7f63100c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 >> 172.21.15.50:0/1126729329 conn(0x7f6310067bb0 msgr2=0x7f63100cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 shutdown_connections 2023-12-19T16:11:50.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.654+0000 7f63174a3700 1 -- 172.21.15.50:0/1126729329 wait complete. 2023-12-19T16:11:50.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T16:11:50.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.838+0000 7f9ca6f85700 1 Processor -- start 2023-12-19T16:11:50.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.838+0000 7f9ca6f85700 1 -- start start 2023-12-19T16:11:50.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca6f85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ca0158e20 0x7f9ca00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca6f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca6f85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca00cbdf0 con 0x7f9ca00cb970 2023-12-19T16:11:50.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca6f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00cc0d0 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca6f85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00cc3b0 con 0x7f9ca0158e20 2023-12-19T16:11:50.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca5522700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca5522700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52982/0 (socket says 172.21.15.50:52982) 2023-12-19T16:11:50.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca5522700 1 -- 172.21.15.50:0/2482097137 learned_addr learned my addr 172.21.15.50:0/2482097137 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:50.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9ca4d21700 1 --2- 172.21.15.50:0/2482097137 >> v2:172.21.15.125:3300/0 conn(0x7f9ca0158e20 0x7f9ca00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.842+0000 7f9c9f7fe700 1 -- 172.21.15.50:0/2482097137 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508481686 0 0) 0x7f9ca00cbdf0 con 0x7f9ca00cb970 2023-12-19T16:11:50.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca5522700 1 -- 172.21.15.50:0/2482097137 >> v2:172.21.15.125:3300/0 conn(0x7f9ca0158e20 msgr2=0x7f9ca00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:50.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca5522700 1 --2- 172.21.15.50:0/2482097137 >> v2:172.21.15.125:3300/0 conn(0x7f9ca0158e20 0x7f9ca00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca5522700 1 -- 172.21.15.50:0/2482097137 >> v1:172.21.15.125:6789/0 conn(0x7f9ca00cb970 legacy=0x7f9ca006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca5522700 1 -- 172.21.15.50:0/2482097137 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca00cc690 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca5522700 1 --2- 172.21.15.50:0/2482097137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009f20 tx=0x7f9c98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d89a4add935ee7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:50.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9c9f7fe700 1 -- 172.21.15.50:0/2482097137 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800e070 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9c9f7fe700 1 -- 172.21.15.50:0/2482097137 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9800b990 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9c9f7fe700 1 -- 172.21.15.50:0/2482097137 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c980092e0 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 -- 172.21.15.50:0/2482097137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca00ca2c0 msgr2=0x7f9ca00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/2482097137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009f20 tx=0x7f9c98009840 comp rx=0 tx=0).stop 2023-12-19T16:11:50.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 -- 172.21.15.50:0/2482097137 shutdown_connections 2023-12-19T16:11:50.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/2482097137 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca00ca2c0 0x7f9ca00ceb00 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/2482097137 >> v2:172.21.15.125:3300/0 conn(0x7f9ca0158e20 0x7f9ca00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 -- 172.21.15.50:0/2482097137 >> 172.21.15.50:0/2482097137 conn(0x7f9ca0067bb0 msgr2=0x7f9ca00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:50.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 -- 172.21.15.50:0/2482097137 shutdown_connections 2023-12-19T16:11:50.853 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.846+0000 7f9ca6f85700 1 -- 172.21.15.50:0/2482097137 wait complete. 2023-12-19T16:11:50.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 Processor -- start 2023-12-19T16:11:50.855 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 -- start start 2023-12-19T16:11:50.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 0x7f9ca00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca0158e20 0x7f9ca00b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b41a0 con 0x7f9ca0158e20 2023-12-19T16:11:50.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b4b70 con 0x7f9ca014b970 2023-12-19T16:11:50.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.850+0000 7f9ca6f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b3590 con 0x7f9ca00ca2c0 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca4d21700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 0x7f9ca00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9c9ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca0158e20 0x7f9ca00b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:35004/0 (socket says 172.21.15.50:35004) 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 -- 172.21.15.50:0/3555027245 learned_addr learned my addr 172.21.15.50:0/3555027245 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 -- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 msgr2=0x7f9ca00b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 0x7f9ca00b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 -- 172.21.15.50:0/3555027245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca0158e20 msgr2=0x7f9ca00b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca0158e20 0x7f9ca00b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 -- 172.21.15.50:0/3555027245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c98009ae0 con 0x7f9ca014b970 2023-12-19T16:11:50.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca4d21700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 0x7f9ca00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:11:50.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca5522700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0069a80 tx=0x7f9c98018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:50.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98008ad0 con 0x7f9ca014b970 2023-12-19T16:11:50.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98008c50 con 0x7f9ca014b970 2023-12-19T16:11:50.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca00b3870 con 0x7f9ca014b970 2023-12-19T16:11:50.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98026460 con 0x7f9ca014b970 2023-12-19T16:11:50.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca00b3e30 con 0x7f9ca014b970 2023-12-19T16:11:50.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.854+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca00b4350 con 0x7f9ca014b970 2023-12-19T16:11:50.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.858+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 92) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f9c98003040 con 0x7f9ca014b970 2023-12-19T16:11:50.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.858+0000 7f9c9cff9700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 0x7f9c84062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:50.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.858+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f9c9807e9a0 con 0x7f9ca014b970 2023-12-19T16:11:50.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.858+0000 7f9ca4d21700 1 -- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 msgr2=0x7f9c84062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:50.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.858+0000 7f9ca4d21700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 0x7f9c84062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:50.872 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:50.866+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9ca00b4350 con 0x7f9ca014b970 2023-12-19T16:11:50.989 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:50.979+0000 7f540d549000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:11:51.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.046+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9ca0071120 con 0x7f9ca014b970 2023-12-19T16:11:51.052 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.046+0000 7f59522ef000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:11:51.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9c9cff9700 1 -- 172.21.15.50:0/3555027245 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9c9804ebc0 con 0x7f9ca014b970 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 msgr2=0x7f9c84062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 0x7f9c84062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 msgr2=0x7f9ca00c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0069a80 tx=0x7f9c98018000 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 shutdown_connections 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9c84060790 0x7f9c84062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ca014b970 0x7f9ca00c9070 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ca0158e20 0x7f9ca00b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 --2- 172.21.15.50:0/3555027245 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9ca00ca2c0 0x7f9ca00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.058+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 >> 172.21.15.50:0/3555027245 conn(0x7f9ca0067bb0 msgr2=0x7f9ca006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:51.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.062+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 shutdown_connections 2023-12-19T16:11:51.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.062+0000 7f9ca6f85700 1 -- 172.21.15.50:0/3555027245 wait complete. 2023-12-19T16:11:51.067 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.059+0000 7f540d549000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:11:51.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:51.087 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T16:11:51.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T16:11:51.134 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.126+0000 7f59522ef000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:11:51.202 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.195+0000 7f540d549000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 Processor -- start 2023-12-19T16:11:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 -- start start 2023-12-19T16:11:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f08800cb130 0x7f08800cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08800b0ac0 con 0x7f088006a6a0 2023-12-19T16:11:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088006b210 con 0x7f088015c930 2023-12-19T16:11:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0887145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08800ca810 con 0x7f08800cb130 2023-12-19T16:11:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f087ffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f088006a6a0 0x7f088015c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56078/0 (socket says 172.21.15.50:56078) 2023-12-19T16:11:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f087ffff700 1 -- 172.21.15.50:0/808886197 learned_addr learned my addr 172.21.15.50:0/808886197 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:11:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0884ee1700 1 --2- 172.21.15.50:0/808886197 >> v2:172.21.15.125:3300/0 conn(0x7f08800cb130 0x7f08800cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584955320 0 0) 0x7f08800b0ac0 con 0x7f088006a6a0 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0870003570 con 0x7f088006a6a0 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250354615 0 0) 0x7f088006b210 con 0x7f088015c930 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0884ee1700 1 -- 172.21.15.50:0/808886197 >> v1:172.21.15.125:6789/0 conn(0x7f088015c930 legacy=0x7f08800b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.294+0000 7f0884ee1700 1 -- 172.21.15.50:0/808886197 >> v1:172.21.15.50:6789/0 conn(0x7f088006a6a0 legacy=0x7f088015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0884ee1700 1 -- 172.21.15.50:0/808886197 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800b0190 con 0x7f08800cb130 2023-12-19T16:11:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0884ee1700 1 --2- 172.21.15.50:0/808886197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08800cb130 0x7f08800cb500 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f08740098e0 tx=0x7f08740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e297ea3b3cf161 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400b640 con 0x7f08800cb130 2023-12-19T16:11:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087400b7a0 con 0x7f08800cb130 2023-12-19T16:11:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f087f7fe700 1 -- 172.21.15.50:0/808886197 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400bb40 con 0x7f08800cb130 2023-12-19T16:11:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 -- 172.21.15.50:0/808886197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08800cb130 msgr2=0x7f08800cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 --2- 172.21.15.50:0/808886197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08800cb130 0x7f08800cb500 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f08740098e0 tx=0x7f08740091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 -- 172.21.15.50:0/808886197 shutdown_connections 2023-12-19T16:11:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 --2- 172.21.15.50:0/808886197 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08800cb130 0x7f08800cb500 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 -- 172.21.15.50:0/808886197 >> 172.21.15.50:0/808886197 conn(0x7f0880067bb0 msgr2=0x7f08800cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:51.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 -- 172.21.15.50:0/808886197 shutdown_connections 2023-12-19T16:11:51.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.298+0000 7f0887145700 1 -- 172.21.15.50:0/808886197 wait complete. 2023-12-19T16:11:51.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 Processor -- start 2023-12-19T16:11:51.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 -- start start 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08800cb130 0x7f0880153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f088015c930 0x7f08801550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08801537d0 con 0x7f08800cb130 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880152950 con 0x7f088015c930 2023-12-19T16:11:51.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0887145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800cea60 con 0x7f088006a6a0 2023-12-19T16:11:51.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0884ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:51.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f087ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08800cb130 0x7f0880153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:51.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f08856e2700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f088015c930 0x7f08801550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:51.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0884ee1700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44278/0 (socket says 172.21.15.50:44278) 2023-12-19T16:11:51.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.302+0000 7f0884ee1700 1 -- 172.21.15.50:0/507845856 learned_addr learned my addr 172.21.15.50:0/507845856 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f088015c930 msgr2=0x7f08801550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f088015c930 0x7f08801550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08800cb130 msgr2=0x7f0880153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08800cb130 0x7f0880153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874009420 con 0x7f088006a6a0 2023-12-19T16:11:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0884ee1700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f088006f300 tx=0x7f0874017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:51.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087401f070 con 0x7f088006a6a0 2023-12-19T16:11:51.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0874002f70 con 0x7f088006a6a0 2023-12-19T16:11:51.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400b910 con 0x7f088006a6a0 2023-12-19T16:11:51.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0880152ba0 con 0x7f088006a6a0 2023-12-19T16:11:51.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800ce6f0 con 0x7f088006a6a0 2023-12-19T16:11:51.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.306+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880071120 con 0x7f088006a6a0 2023-12-19T16:11:51.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.310+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 92) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f0874023070 con 0x7f088006a6a0 2023-12-19T16:11:51.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.310+0000 7f087cff9700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:51.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.310+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6109+0+0 (secure 0 0 0) 0x7f087407df30 con 0x7f088006a6a0 2023-12-19T16:11:51.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.310+0000 7f087ffff700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:51.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.310+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:11:51.326 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.322+0000 7f59522ef000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:51.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.322+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f087404e1e0 con 0x7f088006a6a0 2023-12-19T16:11:51.459 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.451+0000 7f540d549000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:11:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.514+0000 7f087ffff700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:51.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.514+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:11:51.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.518+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f08800ceba0 con 0x7f08640606c0 2023-12-19T16:11:51.527 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.519+0000 7f540d549000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:51.613 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.606+0000 7f59522ef000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:11:51.692 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.686+0000 7f59522ef000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:51.734 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.727+0000 7f540d549000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:11:51.815 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.807+0000 7f540d549000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:11:51.900 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:51.891+0000 7f540d549000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:11:51.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.918+0000 7f087ffff700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:51.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:51.918+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T16:11:51.932 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:51.926+0000 7f59522ef000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:11:52.025 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:52.018+0000 7f59522ef000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:11:52.118 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:52.110+0000 7f59522ef000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:11:52.272 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:52.263+0000 7f540d549000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:11:52.429 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:52.423+0000 7f540d549000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:11:52.534 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:52.530+0000 7f59522ef000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:11:52.709 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:52.702+0000 7f59522ef000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:11:52.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:52.722+0000 7f087ffff700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:52.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:52.722+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T16:11:52.770 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:52.763+0000 7f540d549000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:11:52.989 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:52.979+0000 7f540d549000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:53.078 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.071+0000 7f540d549000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:11:53.088 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.082+0000 7f59522ef000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:11:53.150 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.143+0000 7f540d549000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:11:53.228 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.219+0000 7f540d549000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:11:53.320 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.314+0000 7f59522ef000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:11:53.415 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.410+0000 7f59522ef000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:11:53.461 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.451+0000 7f540d549000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:53.494 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.486+0000 7f59522ef000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:11:53.546 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.539+0000 7f540d549000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:11:53.576 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.570+0000 7f59522ef000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:11:53.636 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.627+0000 7f540d549000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:11:53.798 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.790+0000 7f59522ef000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:11:53.839 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.831+0000 7f540d549000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:11:53.877 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.870+0000 7f59522ef000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:11:53.928 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:53.919+0000 7f540d549000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:11:53.975 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:53.970+0000 7f59522ef000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:11:54.174 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:54.170+0000 7f59522ef000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:11:54.264 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:54.258+0000 7f59522ef000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:11:54.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:54.326+0000 7f087ffff700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:11:54.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:54.326+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T16:11:54.655 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.647+0000 7f540d549000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:11:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:54.666+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mgrmap(e 93) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f08740570f0 con 0x7f088006a6a0 2023-12-19T16:11:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:54.670+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:11:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:54.670+0000 7f087cff9700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:54.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53d0c19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53d0c19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53d0c19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53d0c19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53d0c19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53cdc13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53cdc13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53cdc13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53cdc13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:54.860 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:11:54.851+0000 7f53cdc13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:11:55.077 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:11:55.070+0000 7f59522ef000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.678+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mon.2 v2:172.21.15.125:3301/0 8 ==== mgrmap(e 94) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f0874027d40 con 0x7f088006a6a0 2023-12-19T16:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.678+0000 7f087cff9700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.678+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0864068c30 con 0x7f08640606c0 2023-12-19T16:11:55.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.686+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.690+0000 7f087ffff700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f088016caf0 tx=0x7f086c009040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:55.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.806+0000 7f087cff9700 1 -- 172.21.15.50:0/507845856 <== mgr.9895 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0864068c30 con 0x7f08640606c0 2023-12-19T16:11:55.813 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 msgr2=0x7f0864062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f088016caf0 tx=0x7f086c009040 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 msgr2=0x7f08800d0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f088006f300 tx=0x7f0874017000 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 shutdown_connections 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f08640606c0 0x7f0864062af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f088015c930 0x7f08801550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08800cb130 0x7f0880153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 --2- 172.21.15.50:0/507845856 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f088006a6a0 0x7f08800d0660 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 >> 172.21.15.50:0/507845856 conn(0x7f0880067bb0 msgr2=0x7f0880152050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:55.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.810+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 shutdown_connections 2023-12-19T16:11:55.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:55.814+0000 7f0887145700 1 -- 172.21.15.50:0/507845856 wait complete. 2023-12-19T16:11:55.841 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T16:11:55.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-12-19T16:11:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.018+0000 7f6afc145700 1 Processor -- start 2023-12-19T16:11:56.024 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.018+0000 7f6afc145700 1 -- start start 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6af4140990 0x7f6af406d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 0x7f6af415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6af415b0b0 0x7f6af40cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af415b530 con 0x7f6af4140990 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af406e170 con 0x7f6af415b0b0 2023-12-19T16:11:56.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afc145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40b1db0 con 0x7f6af406d6c0 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6af9ee1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6af4140990 0x7f6af406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6af96e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 0x7f6af415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6af415b0b0 0x7f6af40cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6af96e0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 0x7f6af415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:44280/0 (socket says 172.21.15.50:44280) 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6af96e0700 1 -- 172.21.15.50:0/28857431 learned_addr learned my addr 172.21.15.50:0/28857431 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 -- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 msgr2=0x7f6af415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 --2- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 0x7f6af415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 -- 172.21.15.50:0/28857431 >> v2:172.21.15.50:3300/0 conn(0x7f6af4140990 msgr2=0x7f6af406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 --2- 172.21.15.50:0/28857431 >> v2:172.21.15.50:3300/0 conn(0x7f6af4140990 0x7f6af406d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 -- 172.21.15.50:0/28857431 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af406dbb0 con 0x7f6af415b0b0 2023-12-19T16:11:56.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6afa6e2700 1 --2- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af415b0b0 0x7f6af40cc540 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6af40cef30 tx=0x7f6aec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c9f15f8ae3c2a95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.022+0000 7f6af8edf700 1 -- 172.21.15.50:0/28857431 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00e070 con 0x7f6af415b0b0 2023-12-19T16:11:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6af8edf700 1 -- 172.21.15.50:0/28857431 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec00bab0 con 0x7f6af415b0b0 2023-12-19T16:11:56.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6af8edf700 1 -- 172.21.15.50:0/28857431 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00be70 con 0x7f6af415b0b0 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 -- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af415b0b0 msgr2=0x7f6af40cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 --2- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af415b0b0 0x7f6af40cc540 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6af40cef30 tx=0x7f6aec009780 comp rx=0 tx=0).stop 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 -- 172.21.15.50:0/28857431 shutdown_connections 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 --2- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af415b0b0 0x7f6af40cc540 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 --2- 172.21.15.50:0/28857431 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af406d6c0 0x7f6af415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 --2- 172.21.15.50:0/28857431 >> v2:172.21.15.50:3300/0 conn(0x7f6af4140990 0x7f6af406d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 -- 172.21.15.50:0/28857431 >> 172.21.15.50:0/28857431 conn(0x7f6af40cdef0 msgr2=0x7f6af40ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:56.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 -- 172.21.15.50:0/28857431 shutdown_connections 2023-12-19T16:11:56.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.026+0000 7f6afc145700 1 -- 172.21.15.50:0/28857431 wait complete. 2023-12-19T16:11:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 Processor -- start 2023-12-19T16:11:56.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 -- start start 2023-12-19T16:11:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af4140990 0x7f6af40ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af415b0b0 0x7f6af40ac560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40ab380 con 0x7f6af406d6c0 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40aa770 con 0x7f6af4140990 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6afc145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40aaa50 con 0x7f6af415b0b0 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53016/0 (socket says 172.21.15.50:53016) 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 -- 172.21.15.50:0/3427553279 learned_addr learned my addr 172.21.15.50:0/3427553279 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af96e0700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af4140990 0x7f6af40ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 -- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af415b0b0 msgr2=0x7f6af40ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af415b0b0 0x7f6af40ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 -- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af4140990 msgr2=0x7f6af40ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af4140990 0x7f6af40ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.030+0000 7f6af9ee1700 1 -- 172.21.15.50:0/3427553279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec009a20 con 0x7f6af406d6c0 2023-12-19T16:11:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6af9ee1700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f6af000ec80 tx=0x7f6af00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000c9d0 con 0x7f6af406d6c0 2023-12-19T16:11:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000cb30 con 0x7f6af406d6c0 2023-12-19T16:11:56.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0013470 con 0x7f6af406d6c0 2023-12-19T16:11:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af40aad30 con 0x7f6af406d6c0 2023-12-19T16:11:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af416c3f0 con 0x7f6af406d6c0 2023-12-19T16:11:56.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.034+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af406b3f0 con 0x7f6af406d6c0 2023-12-19T16:11:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.038+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 94) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f6af0013820 con 0x7f6af406d6c0 2023-12-19T16:11:56.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.038+0000 7f6ae9ffb700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 0x7f6ad80628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.038+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6af007cc20 con 0x7f6af406d6c0 2023-12-19T16:11:56.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.038+0000 7f6af96e0700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 0x7f6ad80628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.042+0000 7f6af96e0700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 0x7f6ad80628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6af413e890 tx=0x7f6ae4008040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.046+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6af406b3f0 con 0x7f6af406d6c0 2023-12-19T16:11:56.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.246+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6af40ab4c0 con 0x7f6ad80603f0 2023-12-19T16:11:56.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.370+0000 7f6ae9ffb700 1 -- 172.21.15.50:0/3427553279 <== mgr.9895 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f6ae4009a50 con 0x7f6ad80603f0 2023-12-19T16:11:56.375 INFO:teuthology.orchestra.run.smithi050.stdout:Retention added to path snap_test_dir1 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 msgr2=0x7f6ad80628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 0x7f6ad80628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6af413e890 tx=0x7f6ae4008040 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 msgr2=0x7f6af416beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f6af000ec80 tx=0x7f6af00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 shutdown_connections 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6ad80603f0 0x7f6ad80628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6af4140990 0x7f6af40ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6af415b0b0 0x7f6af40ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 --2- 172.21.15.50:0/3427553279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af406d6c0 0x7f6af416beb0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 >> 172.21.15.50:0/3427553279 conn(0x7f6af40cdef0 msgr2=0x7f6af40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 shutdown_connections 2023-12-19T16:11:56.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.374+0000 7f6afc145700 1 -- 172.21.15.50:0/3427553279 wait complete. 2023-12-19T16:11:56.401 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-12-19T16:11:56.401 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-16_12_00 in ~4s... 2023-12-19T16:11:56.401 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-12-19T16:11:56.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T16:11:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.570+0000 7f63d486b700 1 Processor -- start 2023-12-19T16:11:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.570+0000 7f63d486b700 1 -- start start 2023-12-19T16:11:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63d006b6f0 0x7f63d006bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63d00b2160 0x7f63d0159d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00cd3d0 con 0x7f63d00b2160 2023-12-19T16:11:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d01598d0 con 0x7f63d006b6f0 2023-12-19T16:11:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.574+0000 7f63d486b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00b2590 con 0x7f63d006a580 2023-12-19T16:11:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63d00b2160 0x7f63d0159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63cf7fe700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63d006b6f0 0x7f63d006bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63cffff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f63d00b2160 0x7f63d0159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51842/0 (socket says 172.21.15.50:51842) 2023-12-19T16:11:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 -- 172.21.15.50:0/967436860 learned_addr learned my addr 172.21.15.50:0/967436860 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 -- 172.21.15.50:0/967436860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 msgr2=0x7f63d006a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 --2- 172.21.15.50:0/967436860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d006a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 -- 172.21.15.50:0/967436860 >> v2:172.21.15.125:3300/0 conn(0x7f63d006b6f0 msgr2=0x7f63d006bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:11:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 --2- 172.21.15.50:0/967436860 >> v2:172.21.15.125:3300/0 conn(0x7f63d006b6f0 0x7f63d006bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 -- 172.21.15.50:0/967436860 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d01592a0 con 0x7f63d00b2160 2023-12-19T16:11:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ceffd700 1 --2- 172.21.15.50:0/967436860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d0159d60 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f63d0159490 tx=0x7f63c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd71119cdef059e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ce7fc700 1 -- 172.21.15.50:0/967436860 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4011630 con 0x7f63d00b2160 2023-12-19T16:11:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ce7fc700 1 -- 172.21.15.50:0/967436860 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4011790 con 0x7f63d00b2160 2023-12-19T16:11:56.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63ce7fc700 1 -- 172.21.15.50:0/967436860 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4011b50 con 0x7f63d00b2160 2023-12-19T16:11:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 -- 172.21.15.50:0/967436860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 msgr2=0x7f63d0159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 --2- 172.21.15.50:0/967436860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d0159d60 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f63d0159490 tx=0x7f63c4009250 comp rx=0 tx=0).stop 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 -- 172.21.15.50:0/967436860 shutdown_connections 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 --2- 172.21.15.50:0/967436860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d0159d60 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 --2- 172.21.15.50:0/967436860 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d006a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 --2- 172.21.15.50:0/967436860 >> v2:172.21.15.125:3300/0 conn(0x7f63d006b6f0 0x7f63d006bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.578+0000 7f63d486b700 1 -- 172.21.15.50:0/967436860 >> 172.21.15.50:0/967436860 conn(0x7f63d0067bb0 msgr2=0x7f63d00cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- 172.21.15.50:0/967436860 shutdown_connections 2023-12-19T16:11:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- 172.21.15.50:0/967436860 wait complete. 2023-12-19T16:11:56.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 Processor -- start 2023-12-19T16:11:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- start start 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d014ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63d006b6f0 0x7f63d00c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00afbc0 con 0x7f63d00b2160 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00ca1d0 con 0x7f63d006b6f0 2023-12-19T16:11:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.582+0000 7f63d486b700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d014f1b0 con 0x7f63d006a580 2023-12-19T16:11:56.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cf7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d014ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51846/0 (socket says 172.21.15.50:51846) 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 -- 172.21.15.50:0/3849236903 learned_addr learned my addr 172.21.15.50:0/3849236903 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63ceffd700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63d006b6f0 0x7f63d00c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 -- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 msgr2=0x7f63d014ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d014ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 -- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63d006b6f0 msgr2=0x7f63d00c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63d006b6f0 0x7f63d00c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 -- 172.21.15.50:0/3849236903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40094f0 con 0x7f63d00b2160 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63cffff700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009df0 tx=0x7f63c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800caa0 con 0x7f63d00b2160 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c800f040 con 0x7f63d00b2160 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d00c82c0 con 0x7f63d00b2160 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800ce70 con 0x7f63d00b2160 2023-12-19T16:11:56.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.586+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d00c9b80 con 0x7f63d00b2160 2023-12-19T16:11:56.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.590+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d00ca360 con 0x7f63d00b2160 2023-12-19T16:11:56.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.590+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 94) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f63c8014690 con 0x7f63d00b2160 2023-12-19T16:11:56.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.590+0000 7f63b77fe700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 0x7f63bc0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:11:56.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.590+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f63c807cf50 con 0x7f63d00b2160 2023-12-19T16:11:56.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.590+0000 7f63cf7fe700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 0x7f63bc0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:11:56.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.594+0000 7f63cf7fe700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 0x7f63bc0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d20 tx=0x7f63c00091a0 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:11:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.598+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f63c8050f90 con 0x7f63d00b2160 2023-12-19T16:11:56.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.726+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 95) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f63c804d8d0 con 0x7f63d00b2160 2023-12-19T16:11:56.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.798+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63d004be40 con 0x7f63bc0603f0 2023-12-19T16:11:56.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.814+0000 7f63b77fe700 1 -- 172.21.15.50:0/3849236903 <== mgr.9895 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f63d004be40 con 0x7f63bc0603f0 2023-12-19T16:11:56.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:11:56.820 INFO:teuthology.orchestra.run.smithi050.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 msgr2=0x7f63bc0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 0x7f63bc0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d20 tx=0x7f63c00091a0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 msgr2=0x7f63d00c7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009df0 tx=0x7f63c800b040 comp rx=0 tx=0).stop 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 shutdown_connections 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f63bc0603f0 0x7f63bc0628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63d006b6f0 0x7f63d00c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d00b2160 0x7f63d00c7c40 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 --2- 172.21.15.50:0/3849236903 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f63d006a580 0x7f63d014ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:11:56.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 >> 172.21.15.50:0/3849236903 conn(0x7f63d0067bb0 msgr2=0x7f63d00cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:11:56.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 shutdown_connections 2023-12-19T16:11:56.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:11:56.818+0000 7f63d486b700 1 -- 172.21.15.50:0/3849236903 wait complete. 2023-12-19T16:11:56.850 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T16:11:56.850 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-12-19T16:11:56.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:56.850 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:11:56.884 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T16:11:56.885 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:11:56.885 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:11:57.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:57.886 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:11:57.920 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T16:11:57.921 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:11:57.921 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:11:58.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:58.922 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:11:58.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T16:11:58.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:11:58.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:11:59.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:11:59.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:11:59.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:11:59.958 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:11:59.995 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T16:11:59.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:11:59.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:00.397 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:12:00.387+0000 7f53bfbf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:12:00.397 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:12:00.387+0000 7f53bfbf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:12:00.397 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:12:00.387+0000 7f53bfbf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:12:00.397 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:12:00.387+0000 7f53bfbf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:12:00.397 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:12:00.387+0000 7f53bfbf7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:12:00.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:00.996 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:01.031 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:01.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:01.033 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:01.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:01.033 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-16_12_00_UTC'] 2023-12-19T16:12:01.033 DEBUG:tasks.cephfs.test_snap_schedules:exec=1703002316.4013453, now = 1703002321.033434, timo = 4 2023-12-19T16:12:02.036 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-12-19-16_12_00 in ~64s... 2023-12-19T16:12:02.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:02.036 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:02.071 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:02.072 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:02.072 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:02.072 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:03.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:03.074 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:03.101 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:03.103 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:03.103 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:03.103 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:04.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:04.104 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:04.137 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:04.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:04.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:04.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:05.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:05.140 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:05.175 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:05.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:05.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:05.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:06.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:06.178 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:06.211 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:06.212 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:06.212 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:06.213 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:07.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:07.214 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:07.247 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:07.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:07.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:07.249 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:08.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:08.250 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:08.282 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:08.284 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:08.284 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:08.284 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:09.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:09.284 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:09.313 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:09.314 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:09.314 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:09.314 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:10.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:10.316 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:10.348 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:10.349 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:10.349 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:10.350 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:11.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:11.350 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:11.383 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:11.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:11.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:11.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:12.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:12.387 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:12.419 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:12.421 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:12.421 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:12.421 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:13.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:13.421 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:13.454 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:13.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:13.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:13.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:14.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:14.457 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:14.491 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:14.492 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:14.493 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:14.493 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:15.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:15.494 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:15.527 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:15.528 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:15.528 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:15.529 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:16.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:16.529 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:16.561 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:16.562 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:16.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:16.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:17.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:17.563 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:17.594 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:17.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:17.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:17.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:18.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:18.598 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:18.629 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:18.631 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:18.631 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:18.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:19.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:19.632 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:19.666 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:19.666 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:19.666 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:19.666 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:20.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:20.668 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:20.699 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:20.700 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:20.700 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:20.700 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:21.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:21.704 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:21.740 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:21.742 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:21.742 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:21.742 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:22.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:22.743 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:22.777 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:22.778 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:22.778 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:22.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:23.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:23.780 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:23.815 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:23.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:23.817 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:23.817 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:24.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:24.819 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:24.853 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:24.855 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:24.855 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:24.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:25.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:25.857 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:25.890 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:25.891 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:25.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:25.892 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:26.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:26.892 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:26.926 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:26.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:26.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:26.928 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:27.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:27.929 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:27.962 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:27.963 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:27.964 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:27.964 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:28.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:28.965 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:28.998 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:28.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:28.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:28.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:29.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:12:29.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:12:30.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:30.000 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:30.031 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:30.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:30.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:30.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:31.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:31.034 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:31.069 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:31.071 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:31.071 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:31.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:32.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:32.072 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:32.106 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:32.107 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:32.108 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:32.108 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:33.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:33.109 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:33.141 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:33.142 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:33.142 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:33.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:34.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:34.143 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:34.178 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:34.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:34.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:34.180 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:35.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:35.181 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:35.216 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:35.217 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:35.218 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:35.218 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:36.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:36.219 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:36.248 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:36.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:36.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:36.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:37.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:37.254 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:37.288 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:37.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:37.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:37.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:38.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:38.290 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:38.322 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:38.324 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:38.324 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:38.324 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:39.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:39.325 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:39.356 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:39.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:39.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:39.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:40.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:40.359 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:40.393 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:40.395 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:40.395 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:40.395 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:41.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:41.395 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:41.428 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:41.429 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:41.430 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:41.430 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:42.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:42.431 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:42.464 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:42.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:42.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:42.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:43.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:43.467 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:43.498 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:43.499 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:43.499 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:43.499 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:44.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:44.500 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:44.530 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:44.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:44.531 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:44.531 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:45.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:45.533 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:45.565 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:45.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:45.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:45.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:46.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:46.568 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:46.602 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:46.604 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:46.604 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:46.604 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:47.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:47.605 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:47.636 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:47.637 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:47.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:47.638 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:48.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:48.639 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:48.671 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:48.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:48.673 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:48.673 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:49.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:49.674 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:49.706 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:49.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:49.708 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:49.708 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:50.709 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:50.709 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:50.742 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:50.744 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:50.744 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:50.744 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:51.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:51.745 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:51.778 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:51.780 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:51.780 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:51.780 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:52.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:52.781 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:52.814 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:52.816 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:52.816 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:52.816 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:53.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:53.816 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:53.851 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:53.853 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:53.853 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:53.853 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:54.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:54.854 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:54.884 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:54.885 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:54.886 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:54.886 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:55.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:55.886 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:55.921 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:55.923 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:55.923 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:55.924 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:56.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:56.925 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:56.961 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:56.963 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:56.963 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:56.963 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:57.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:57.965 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:57.996 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:57.998 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:57.998 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:57.998 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:58.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:12:58.999 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:12:59.033 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:12:59.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:12:59.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:12:59.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:12:59.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:12:59.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:13:00.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:00.035 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:13:00.068 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_12_00_UTC 2023-12-19T16:13:00.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:13:00.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T16:13:00.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T16:13:01.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:01.072 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:13:01.107 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_13_00_UTC 2023-12-19T16:13:01.109 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-16_13_00_UTC'} 2023-12-19T16:13:01.109 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-16_13_00_UTC'} 2023-12-19T16:13:01.109 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-12-19-16_12_00_UTC'} 2023-12-19T16:13:01.109 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-12-19-16_12_00_UTC'] 2023-12-19T16:13:01.109 DEBUG:tasks.cephfs.test_snap_schedules:exec=1703002316.4013453, now = 1703002381.1096256, timo = 64 2023-12-19T16:13:02.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T16:13:02.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 Processor -- start 2023-12-19T16:13:02.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 -- start start 2023-12-19T16:13:02.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf48161da0 0x7fbf4806a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf4806abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480c6560 con 0x7fbf4815d350 2023-12-19T16:13:02.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf480c70e0 con 0x7fbf48161da0 2023-12-19T16:13:02.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf4d745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480cc900 con 0x7fbf4815d720 2023-12-19T16:13:02.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf46ffd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbf4815d350 0x7fbf480cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55924/0 (socket says 172.21.15.50:55924) 2023-12-19T16:13:02.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf46ffd700 1 -- 172.21.15.50:0/2015908607 learned_addr learned my addr 172.21.15.50:0/2015908607 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:02.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf3e7fc700 1 --2- 172.21.15.50:0/2015908607 >> v2:172.21.15.125:3300/0 conn(0x7fbf48161da0 0x7fbf4806a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf477fe700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf4806abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.288+0000 7fbf467fc700 1 -- 172.21.15.50:0/2015908607 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370757379 0 0) 0x7fbf480c6560 con 0x7fbf4815d350 2023-12-19T16:13:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf467fc700 1 -- 172.21.15.50:0/2015908607 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34003570 con 0x7fbf4815d350 2023-12-19T16:13:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf3e7fc700 1 -- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 msgr2=0x7fbf4806abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf3e7fc700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf4806abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf3e7fc700 1 -- 172.21.15.50:0/2015908607 >> v1:172.21.15.50:6789/0 conn(0x7fbf4815d350 legacy=0x7fbf480cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf3e7fc700 1 -- 172.21.15.50:0/2015908607 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480c67b0 con 0x7fbf48161da0 2023-12-19T16:13:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf3e7fc700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf48161da0 0x7fbf4806a6a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009e60 tx=0x7fbf30009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e6c3a982d501fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf467fc700 1 -- 172.21.15.50:0/2015908607 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000e070 con 0x7fbf48161da0 2023-12-19T16:13:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf467fc700 1 -- 172.21.15.50:0/2015908607 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3000bab0 con 0x7fbf48161da0 2023-12-19T16:13:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf467fc700 1 -- 172.21.15.50:0/2015908607 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000be70 con 0x7fbf48161da0 2023-12-19T16:13:02.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 -- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf48161da0 msgr2=0x7fbf4806a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf48161da0 0x7fbf4806a6a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009e60 tx=0x7fbf30009780 comp rx=0 tx=0).stop 2023-12-19T16:13:02.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 -- 172.21.15.50:0/2015908607 shutdown_connections 2023-12-19T16:13:02.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf4806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 --2- 172.21.15.50:0/2015908607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf48161da0 0x7fbf4806a6a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 -- 172.21.15.50:0/2015908607 >> 172.21.15.50:0/2015908607 conn(0x7fbf48067bb0 msgr2=0x7fbf480cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 -- 172.21.15.50:0/2015908607 shutdown_connections 2023-12-19T16:13:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.292+0000 7fbf4d745700 1 -- 172.21.15.50:0/2015908607 wait complete. 2023-12-19T16:13:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 Processor -- start 2023-12-19T16:13:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 -- start start 2023-12-19T16:13:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf4815d350 0x7fbf4815a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf48170f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4815b090 con 0x7fbf48161da0 2023-12-19T16:13:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4815b370 con 0x7fbf4815d350 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf4d745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4815b650 con 0x7fbf4815d720 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53012/0 (socket says 172.21.15.50:53012) 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 -- 172.21.15.50:0/4288020130 learned_addr learned my addr 172.21.15.50:0/4288020130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf3e7fc700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf48170f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf46ffd700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf4815d350 0x7fbf4815a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 -- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 msgr2=0x7fbf48170f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf48170f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 -- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf4815d350 msgr2=0x7fbf4815a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf4815d350 0x7fbf4815a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 -- 172.21.15.50:0/4288020130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30009a20 con 0x7fbf48161da0 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.296+0000 7fbf477fe700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000ea50 tx=0x7fbf400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000c920 con 0x7fbf48161da0 2023-12-19T16:13:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000caa0 con 0x7fbf48161da0 2023-12-19T16:13:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4815b930 con 0x7fbf48161da0 2023-12-19T16:13:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000ce60 con 0x7fbf48161da0 2023-12-19T16:13:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4815ac30 con 0x7fbf48161da0 2023-12-19T16:13:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4815be30 con 0x7fbf48161da0 2023-12-19T16:13:02.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.300+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 96) v1 ==== 80194+0+0 (secure 0 0 0) 0x7fbf40013680 con 0x7fbf48161da0 2023-12-19T16:13:02.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.304+0000 7fbf3f7fe700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 0x7fbf28062cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:02.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.304+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fbf4007d2e0 con 0x7fbf48161da0 2023-12-19T16:13:02.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.304+0000 7fbf46ffd700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 0x7fbf28062cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:02.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.304+0000 7fbf46ffd700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 0x7fbf28062cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf380058d0 tx=0x7fbf38009040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:02.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.308+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf4004d2b0 con 0x7fbf48161da0 2023-12-19T16:13:02.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.500+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf4815ae80 con 0x7fbf28060810 2023-12-19T16:13:02.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf3f7fe700 1 -- 172.21.15.50:0/4288020130 <== mgr.9895 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbf4815ae80 con 0x7fbf28060810 2023-12-19T16:13:02.634 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path snap_test_dir1 2023-12-19T16:13:02.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 msgr2=0x7fbf28062cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 0x7fbf28062cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf380058d0 tx=0x7fbf38009040 comp rx=0 tx=0).stop 2023-12-19T16:13:02.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 msgr2=0x7fbf48143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:02.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000ea50 tx=0x7fbf400091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 shutdown_connections 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbf28060810 0x7fbf28062cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf48161da0 0x7fbf48143b10 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf4815d720 0x7fbf48170f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.632+0000 7fbf4d745700 1 --2- 172.21.15.50:0/4288020130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf4815d350 0x7fbf4815a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.636+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 >> 172.21.15.50:0/4288020130 conn(0x7fbf48067bb0 msgr2=0x7fbf48156820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.636+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 shutdown_connections 2023-12-19T16:13:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:02.636+0000 7fbf4d745700 1 -- 172.21.15.50:0/4288020130 wait complete. 2023-12-19T16:13:02.662 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T16:13:02.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:02.662 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:13:02.700 INFO:teuthology.orchestra.run.smithi050.stdout:scheduled-2023-12-19-16_13_00_UTC 2023-12-19T16:13:02.702 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-16_13_00_UTC 2023-12-19T16:13:02.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:02.703 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-16_13_00_UTC) 2023-12-19T16:13:02.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:02.789 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T16:13:02.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T16:13:03.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.084+0000 7f71d562e700 1 Processor -- start 2023-12-19T16:13:03.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.084+0000 7f71d562e700 1 -- start start 2023-12-19T16:13:03.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71d562e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f71d00cecd0 0x7f71d006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:03.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71d562e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00cbdf0 con 0x7f71d00cb970 2023-12-19T16:13:03.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71d562e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00cc0d0 con 0x7f71d0158e20 2023-12-19T16:13:03.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71d562e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00cc3b0 con 0x7f71d00cecd0 2023-12-19T16:13:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71ce7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f71d00cb970 0x7f71d006acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55938/0 (socket says 172.21.15.50:55938) 2023-12-19T16:13:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71ce7fc700 1 -- 172.21.15.50:0/1742854336 learned_addr learned my addr 172.21.15.50:0/1742854336 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:03.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cf7fe700 1 --2- 172.21.15.50:0/1742854336 >> v2:172.21.15.50:3300/0 conn(0x7f71d00cecd0 0x7f71d006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804894527 0 0) 0x7f71d00cbdf0 con 0x7f71d00cb970 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b8003570 con 0x7f71d00cb970 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595531522 0 0) 0x7f71d00cc0d0 con 0x7f71d0158e20 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d00cbdf0 con 0x7f71d0158e20 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cf7fe700 1 -- 172.21.15.50:0/1742854336 >> v1:172.21.15.125:6789/0 conn(0x7f71d0158e20 legacy=0x7f71d00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cf7fe700 1 -- 172.21.15.50:0/1742854336 >> v1:172.21.15.50:6789/0 conn(0x7f71d00cb970 legacy=0x7f71d006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:03.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cf7fe700 1 -- 172.21.15.50:0/1742854336 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00cc690 con 0x7f71d00cecd0 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.088+0000 7f71cf7fe700 1 --2- 172.21.15.50:0/1742854336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d00cecd0 0x7f71d006b3d0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009e90 tx=0x7f71c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc527d614615b34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c800e070 con 0x7f71d00cecd0 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c800bb90 con 0x7f71d00cecd0 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71cdffb700 1 -- 172.21.15.50:0/1742854336 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8012430 con 0x7f71d00cecd0 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 -- 172.21.15.50:0/1742854336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d00cecd0 msgr2=0x7f71d006b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:03.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 --2- 172.21.15.50:0/1742854336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d00cecd0 0x7f71d006b3d0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009e90 tx=0x7f71c8009780 comp rx=0 tx=0).stop 2023-12-19T16:13:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 -- 172.21.15.50:0/1742854336 shutdown_connections 2023-12-19T16:13:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 --2- 172.21.15.50:0/1742854336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d00cecd0 0x7f71d006b3d0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 -- 172.21.15.50:0/1742854336 >> 172.21.15.50:0/1742854336 conn(0x7f71d0067bb0 msgr2=0x7f71d00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:03.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 -- 172.21.15.50:0/1742854336 shutdown_connections 2023-12-19T16:13:03.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.092+0000 7f71d562e700 1 -- 172.21.15.50:0/1742854336 wait complete. 2023-12-19T16:13:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 Processor -- start 2023-12-19T16:13:03.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 -- start start 2023-12-19T16:13:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 0x7f71d00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:03.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 0x7f71d016c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d016cf80 con 0x7f71d016cae0 2023-12-19T16:13:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d016a170 con 0x7f71d0158e20 2023-12-19T16:13:03.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71d562e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d014e6a0 con 0x7f71d00cecd0 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53018/0 (socket says 172.21.15.50:53018) 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 -- 172.21.15.50:0/3395343561 learned_addr learned my addr 172.21.15.50:0/3395343561 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71ce7fc700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 0x7f71d016c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71ceffd700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 0x7f71d00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 -- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 msgr2=0x7f71d00b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 0x7f71d00b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 -- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 msgr2=0x7f71d016c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:03.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 0x7f71d016c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 -- 172.21.15.50:0/3395343561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8009a20 con 0x7f71d016cae0 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71ce7fc700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 0x7f71d016c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71ceffd700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 0x7f71d00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71cf7fe700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f71d0069a80 tx=0x7f71c8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.096+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8012e00 con 0x7f71d016cae0 2023-12-19T16:13:03.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c801c610 con 0x7f71d016cae0 2023-12-19T16:13:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d014e370 con 0x7f71d016cae0 2023-12-19T16:13:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c801c9d0 con 0x7f71d016cae0 2023-12-19T16:13:03.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d00b5be0 con 0x7f71d016cae0 2023-12-19T16:13:03.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d00ca120 con 0x7f71d016cae0 2023-12-19T16:13:03.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.100+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 96) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f71c801cd40 con 0x7f71d016cae0 2023-12-19T16:13:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.104+0000 7f71b6ffd700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 0x7f71bc062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:03.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.104+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f71c807efe0 con 0x7f71d016cae0 2023-12-19T16:13:03.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.104+0000 7f71ceffd700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 0x7f71bc062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:03.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.104+0000 7f71ceffd700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 0x7f71bc062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71c0007fd0 tx=0x7f71c0009040 comp rx=0 tx=0).ready entity=mgr.9895 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:03.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.108+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f71d00ca120 con 0x7f71d016cae0 2023-12-19T16:13:03.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:03.316+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f71d00ca120 con 0x7f71d016cae0 2023-12-19T16:13:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.300+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f71c804f2b0 con 0x7f71d016cae0 2023-12-19T16:13:04.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71b6ffd700 1 -- 172.21.15.50:0/3395343561 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v97) v1 ==== 93+0+0 (secure 0 0 0) 0x7f71c8052970 con 0x7f71d016cae0 2023-12-19T16:13:04.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 msgr2=0x7f71bc062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 0x7f71bc062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71c0007fd0 tx=0x7f71c0009040 comp rx=0 tx=0).stop 2023-12-19T16:13:04.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 msgr2=0x7f71d013b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f71d0069a80 tx=0x7f71c8019000 comp rx=0 tx=0).stop 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 shutdown_connections 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f71bc0607e0 0x7f71bc062c90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71d0158e20 0x7f71d016c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71d016cae0 0x7f71d013b1e0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 --2- 172.21.15.50:0/3395343561 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f71d00cecd0 0x7f71d00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 >> 172.21.15.50:0/3395343561 conn(0x7f71d0067bb0 msgr2=0x7f71d00c6310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 shutdown_connections 2023-12-19T16:13:04.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.304+0000 7f71d562e700 1 -- 172.21.15.50:0/3395343561 wait complete. 2023-12-19T16:13:04.331 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T16:13:04.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:04.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:04.412 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T16:13:04.413 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T16:13:04.413 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T16:13:04.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:04.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[60197]: fuse finished with error 0 and tester_r 0 2023-12-19T16:13:04.453 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:13:04.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:04.453 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T16:13:04.505 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:04.500+0000 7fabd6767000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:04.519 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:04.510+0000 7fb2e8308000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:04.644 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:13:04.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:04.644 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T16:13:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T16:13:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:04.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:13:04.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:04.667 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T16:13:04.706 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T16:13:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:04.707 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:13:04.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:04.708 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T16:13:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T16:13:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:04.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:04.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 Processor -- start 2023-12-19T16:13:04.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 -- start start 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f681c140710 0x7f681c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 0x7f681c0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c06e150 con 0x7f681c140ae0 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c06d540 con 0x7f681c140710 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6822ccd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c06d820 con 0x7f681c068800 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f681c140710 0x7f681c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f681c140710 0x7f681c0c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53036/0 (socket says 172.21.15.50:53036) 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 -- 172.21.15.50:0/535771142 learned_addr learned my addr 172.21.15.50:0/535771142 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:04.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f682126a700 1 --2- 172.21.15.50:0/535771142 >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 0x7f681c0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 -- 172.21.15.50:0/535771142 >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 msgr2=0x7f681c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 --2- 172.21.15.50:0/535771142 >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 0x7f681c0ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 -- 172.21.15.50:0/535771142 >> v1:172.21.15.125:6789/0 conn(0x7f681c140ae0 legacy=0x7f681c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 -- 172.21.15.50:0/535771142 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c06db00 con 0x7f681c140710 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f682126a700 1 --2- 172.21.15.50:0/535771142 >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 0x7f681c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:04.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.916+0000 7f6820a69700 1 --2- 172.21.15.50:0/535771142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c0c70a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f681c06bfc0 tx=0x7f6818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0138088206eb5c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:04.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f68137fe700 1 -- 172.21.15.50:0/535771142 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818009e70 con 0x7f681c140710 2023-12-19T16:13:04.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f68137fe700 1 -- 172.21.15.50:0/535771142 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681800b860 con 0x7f681c140710 2023-12-19T16:13:04.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f68137fe700 1 -- 172.21.15.50:0/535771142 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681800bc00 con 0x7f681c140710 2023-12-19T16:13:04.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 -- 172.21.15.50:0/535771142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 msgr2=0x7f681c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 --2- 172.21.15.50:0/535771142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c0c70a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f681c06bfc0 tx=0x7f6818009250 comp rx=0 tx=0).stop 2023-12-19T16:13:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 -- 172.21.15.50:0/535771142 shutdown_connections 2023-12-19T16:13:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 --2- 172.21.15.50:0/535771142 >> v2:172.21.15.125:3300/0 conn(0x7f681c068800 0x7f681c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 --2- 172.21.15.50:0/535771142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c0c70a0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 -- 172.21.15.50:0/535771142 >> 172.21.15.50:0/535771142 conn(0x7f681c0c8850 msgr2=0x7f681c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 -- 172.21.15.50:0/535771142 shutdown_connections 2023-12-19T16:13:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.920+0000 7f6822ccd700 1 -- 172.21.15.50:0/535771142 wait complete. 2023-12-19T16:13:04.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 Processor -- start 2023-12-19T16:13:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 -- start start 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 0x7f681c0c2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 0x7f681c0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c1c90 con 0x7f681c140710 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c15d5e0 con 0x7f681c140ae0 2023-12-19T16:13:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6822ccd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c13e160 con 0x7f681c068800 2023-12-19T16:13:04.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6813fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6820a69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 0x7f681c0c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6820a69700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 0x7f681c0c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39472/0 (socket says 172.21.15.50:39472) 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f6820a69700 1 -- 172.21.15.50:0/1755906496 learned_addr learned my addr 172.21.15.50:0/1755906496 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 0x7f681c0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 msgr2=0x7f681c0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 0x7f681c0c2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 msgr2=0x7f681c15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c15d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.924+0000 7f682126a700 1 -- 172.21.15.50:0/1755906496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814009a20 con 0x7f681c140ae0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f682126a700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 0x7f681c0c5810 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f681c140f50 tx=0x7f681400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400caa0 con 0x7f681c140ae0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681400f040 con 0x7f681c140ae0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400ce70 con 0x7f681c140ae0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68180094f0 con 0x7f681c140ae0 2023-12-19T16:13:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c0c1360 con 0x7f681c140ae0 2023-12-19T16:13:04.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.928+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c0c1e40 con 0x7f681c140ae0 2023-12-19T16:13:04.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.932+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f6814014690 con 0x7f681c140ae0 2023-12-19T16:13:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.932+0000 7f6810ff9700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 0x7f6808062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.932+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f681407dac0 con 0x7f681c140ae0 2023-12-19T16:13:04.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.932+0000 7f6820a69700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 msgr2=0x7f6808062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:04.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.932+0000 7f6820a69700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 0x7f6808062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:04.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:04.940+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f681c0c1e40 con 0x7f681c140ae0 2023-12-19T16:13:05.070 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.062+0000 7fb2e8308000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:05.106 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.100+0000 7fabd6767000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:05.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.136+0000 7f6820a69700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 msgr2=0x7f6808062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:05.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.136+0000 7f6820a69700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 0x7f6808062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:05.142 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.134+0000 7fb2e8308000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:05.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.160+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681c0c1e00 con 0x7f681c140ae0 2023-12-19T16:13:05.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.160+0000 7f6810ff9700 1 -- 172.21.15.50:0/1755906496 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f6814018090 con 0x7f681c140ae0 2023-12-19T16:13:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:05.163 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:13:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 msgr2=0x7f6808062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 0x7f6808062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 msgr2=0x7f681c0c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 0x7f681c0c5810 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f681c140f50 tx=0x7f681400b040 comp rx=0 tx=0).stop 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 shutdown_connections 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68080607c0 0x7f6808062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f681c140ae0 0x7f681c0c5810 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f681c140710 0x7f681c15d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 --2- 172.21.15.50:0/1755906496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f681c068800 0x7f681c0c2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 >> 172.21.15.50:0/1755906496 conn(0x7f681c0c8850 msgr2=0x7f681c067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 shutdown_connections 2023-12-19T16:13:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.164+0000 7f6822ccd700 1 -- 172.21.15.50:0/1755906496 wait complete. 2023-12-19T16:13:05.168 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:13:05.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:05.186 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.184+0000 7fabd6767000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:05.291 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.282+0000 7fb2e8308000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 Processor -- start 2023-12-19T16:13:05.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 -- start start 2023-12-19T16:13:05.348 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fabd6767000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2840cc9b0 0x7fb2840ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840afc60 0x7fb28415bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28406cfb0 con 0x7fb28415c3e0 2023-12-19T16:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840cb660 con 0x7fb2840cc9b0 2023-12-19T16:13:05.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2898cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28415c840 con 0x7fb2840afc60 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb282ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb2840cc9b0 0x7fb2840ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.344+0000 7fb2827fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840afc60 0x7fb28415bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840afc60 0x7fb28415bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:45102/0 (socket says 172.21.15.50:45102) 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 -- 172.21.15.50:0/1089706451 learned_addr learned my addr 172.21.15.50:0/1089706451 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 -- 172.21.15.50:0/1089706451 >> v2:172.21.15.50:3300/0 conn(0x7fb2840cc9b0 msgr2=0x7fb2840ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:05.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 --2- 172.21.15.50:0/1089706451 >> v2:172.21.15.50:3300/0 conn(0x7fb2840cc9b0 0x7fb2840ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 -- 172.21.15.50:0/1089706451 >> v1:172.21.15.50:6789/0 conn(0x7fb28415c3e0 legacy=0x7fb2840ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 -- 172.21.15.50:0/1089706451 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840cb380 con 0x7fb2840afc60 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2827fc700 1 --2- 172.21.15.50:0/1089706451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840afc60 0x7fb28415bea0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb28406bec0 tx=0x7fb278009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619cb0da65cc5962 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb281ffb700 1 -- 172.21.15.50:0/1089706451 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278009e70 con 0x7fb2840afc60 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb281ffb700 1 -- 172.21.15.50:0/1089706451 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27800b860 con 0x7fb2840afc60 2023-12-19T16:13:05.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb281ffb700 1 -- 172.21.15.50:0/1089706451 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27800bc00 con 0x7fb2840afc60 2023-12-19T16:13:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 -- 172.21.15.50:0/1089706451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840afc60 msgr2=0x7fb28415bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 --2- 172.21.15.50:0/1089706451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840afc60 0x7fb28415bea0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb28406bec0 tx=0x7fb278009250 comp rx=0 tx=0).stop 2023-12-19T16:13:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 -- 172.21.15.50:0/1089706451 shutdown_connections 2023-12-19T16:13:05.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 --2- 172.21.15.50:0/1089706451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840afc60 0x7fb28415bea0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 --2- 172.21.15.50:0/1089706451 >> v2:172.21.15.50:3300/0 conn(0x7fb2840cc9b0 0x7fb2840ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 -- 172.21.15.50:0/1089706451 >> 172.21.15.50:0/1089706451 conn(0x7fb2840670e0 msgr2=0x7fb2840674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 -- 172.21.15.50:0/1089706451 shutdown_connections 2023-12-19T16:13:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.348+0000 7fb2898cd700 1 -- 172.21.15.50:0/1089706451 wait complete. 2023-12-19T16:13:05.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 Processor -- start 2023-12-19T16:13:05.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 -- start start 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2840afc60 0x7fb28414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 0x7fb2840c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb284150290 con 0x7fb2840cc9b0 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840b2c70 con 0x7fb28415c3e0 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2898cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840c51e0 con 0x7fb2840afc60 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53054/0 (socket says 172.21.15.50:53054) 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 -- 172.21.15.50:0/3113438900 learned_addr learned my addr 172.21.15.50:0/3113438900 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:05.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb282ffd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2840afc60 0x7fb28414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2837fe700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 0x7fb2840c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 -- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2840afc60 msgr2=0x7fb28414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2840afc60 0x7fb28414fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 -- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 msgr2=0x7fb2840c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 0x7fb2840c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2827fc700 1 -- 172.21.15.50:0/3113438900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780094f0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.352+0000 7fb2837fe700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 0x7fb2840c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb2827fc700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb28415c980 tx=0x7fb278017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278011420 con 0x7fb2840cc9b0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2780115a0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278011960 con 0x7fb2840cc9b0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2840c54c0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2840b31a0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.356+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2840b33c0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.360+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fb2780253f0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.360+0000 7fb26affd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 0x7fb270062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.360+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fb27807e8a0 con 0x7fb2840cc9b0 2023-12-19T16:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.360+0000 7fb282ffd700 1 -- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 msgr2=0x7fb270062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:05.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.360+0000 7fb282ffd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 0x7fb270062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:05.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.368+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb27804e840 con 0x7fb2840cc9b0 2023-12-19T16:13:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.548+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28404be40 con 0x7fb2840cc9b0 2023-12-19T16:13:05.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.552+0000 7fb26affd700 1 -- 172.21.15.50:0/3113438900 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+16311 (secure 0 0 0) 0x7fb278051e60 con 0x7fb2840cc9b0 2023-12-19T16:13:05.556 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:05.556 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":201,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:05.296456+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1157332980":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/625495530":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/4264135625":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/915278715":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/780783709":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/3568404314":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6833/44063":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6832/44063":"2023-12-20T16:11:54.658801+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 msgr2=0x7fb270062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 0x7fb270062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 msgr2=0x7fb2840b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fb28415c980 tx=0x7fb278017000 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 shutdown_connections 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb2700607c0 0x7fb270062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb28415c3e0 0x7fb2840c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2840cc9b0 0x7fb2840b4540 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 --2- 172.21.15.50:0/3113438900 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb2840afc60 0x7fb28414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 >> 172.21.15.50:0/3113438900 conn(0x7fb2840670e0 msgr2=0x7fb2840b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 shutdown_connections 2023-12-19T16:13:05.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.556+0000 7fb2898cd700 1 -- 172.21.15.50:0/3113438900 wait complete. 2023-12-19T16:13:05.560 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.550+0000 7fb2e8308000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:05.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T16:11:37.969076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '198', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T16:11:38.930456+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 201, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T16:13:05.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:13:05.650 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.644+0000 7fabd6767000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:05.652 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.642+0000 7fb2e8308000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:05.728 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.724+0000 7fabd6767000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:05.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.740+0000 7fd8a64ae700 1 Processor -- start 2023-12-19T16:13:05.744 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.740+0000 7fd8a64ae700 1 -- start start 2023-12-19T16:13:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.740+0000 7fd8a64ae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8a0158e20 0x7fd8a00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.740+0000 7fd8a64ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.740+0000 7fd8a64ae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a00cbdf0 con 0x7fd8a00cb970 2023-12-19T16:13:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a00cc0d0 con 0x7fd8a0158e20 2023-12-19T16:13:05.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00cc3b0 con 0x7fd8a00ca2c0 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89f7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fd8a00cb970 0x7fd8a006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55960/0 (socket says 172.21.15.50:55960) 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89f7fe700 1 -- 172.21.15.50:0/4162940050 learned_addr learned my addr 172.21.15.50:0/4162940050 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 --2- 172.21.15.50:0/4162940050 >> v2:172.21.15.125:3300/0 conn(0x7fd8a0158e20 0x7fd8a00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a4a4b700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89effd700 1 -- 172.21.15.50:0/4162940050 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47532821 0 0) 0x7fd8a00cbdf0 con 0x7fd8a00cb970 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89effd700 1 -- 172.21.15.50:0/4162940050 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd888003570 con 0x7fd8a00cb970 2023-12-19T16:13:05.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 -- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 msgr2=0x7fd8a00ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a00ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 -- 172.21.15.50:0/4162940050 >> v1:172.21.15.50:6789/0 conn(0x7fd8a00cb970 legacy=0x7fd8a006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 -- 172.21.15.50:0/4162940050 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00cc690 con 0x7fd8a0158e20 2023-12-19T16:13:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89ffff700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a00ce5c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd890009910 tx=0x7fd8900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ad70a390c8eed8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:05.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89effd700 1 -- 172.21.15.50:0/4162940050 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890011630 con 0x7fd8a0158e20 2023-12-19T16:13:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89effd700 1 -- 172.21.15.50:0/4162940050 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd890011790 con 0x7fd8a0158e20 2023-12-19T16:13:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd89effd700 1 -- 172.21.15.50:0/4162940050 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890011b30 con 0x7fd8a0158e20 2023-12-19T16:13:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 msgr2=0x7fd8a00ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a00ce5c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd890009910 tx=0x7fd8900091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- 172.21.15.50:0/4162940050 shutdown_connections 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a00ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/4162940050 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a00ce5c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- 172.21.15.50:0/4162940050 >> 172.21.15.50:0/4162940050 conn(0x7fd8a0067bb0 msgr2=0x7fd8a00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- 172.21.15.50:0/4162940050 shutdown_connections 2023-12-19T16:13:05.750 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.744+0000 7fd8a64ae700 1 -- 172.21.15.50:0/4162940050 wait complete. 2023-12-19T16:13:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 Processor -- start 2023-12-19T16:13:05.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 -- start start 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a014eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a014f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a014fe70 con 0x7fd8a014f9d0 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c8bf0 con 0x7fd8a0158e20 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.748+0000 7fd8a64ae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00c9080 con 0x7fd8a00ca2c0 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53066/0 (socket says 172.21.15.50:53066) 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 -- 172.21.15.50:0/2589921373 learned_addr learned my addr 172.21.15.50:0/2589921373 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd89f7fe700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a014f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.754 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd89ffff700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a014eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:05.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 -- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 msgr2=0x7fd8a014eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a014eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 -- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 msgr2=0x7fd8a014f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a014f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 -- 172.21.15.50:0/2589921373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890009420 con 0x7fd8a014f9d0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a4a4b700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd898009e90 tx=0x7fd89800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89800cb70 con 0x7fd8a014f9d0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89800f040 con 0x7fd8a014f9d0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898014480 con 0x7fd8a014f9d0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a00c7fe0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a00c85a0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.752+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a0069cc0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.756+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fd8980147f0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.756+0000 7fd887fff700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 0x7fd88c062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.756+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fd89807dc30 con 0x7fd8a014f9d0 2023-12-19T16:13:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.756+0000 7fd89ffff700 1 -- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 msgr2=0x7fd88c062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:05.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.756+0000 7fd89ffff700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 0x7fd88c062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.764+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8a0069cc0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.859 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.850+0000 7fb2e8308000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:05.942 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:05.934+0000 7fb2e8308000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:05.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.944+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8a0069cc0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd887fff700 1 -- 172.21.15.50:0/2589921373 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8a0069cc0 con 0x7fd8a014f9d0 2023-12-19T16:13:05.950 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:13:05.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 msgr2=0x7fd88c062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 0x7fd88c062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 msgr2=0x7fd8a00c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd898009e90 tx=0x7fd89800b040 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.948+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 shutdown_connections 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd88c0607c0 0x7fd88c062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0158e20 0x7fd8a014f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8a014f9d0 0x7fd8a00c7270 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 --2- 172.21.15.50:0/2589921373 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd8a00ca2c0 0x7fd8a014eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:05.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 >> 172.21.15.50:0/2589921373 conn(0x7fd8a0067bb0 msgr2=0x7fd8a0156210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 shutdown_connections 2023-12-19T16:13:05.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:05.952+0000 7fd8a64ae700 1 -- 172.21.15.50:0/2589921373 wait complete. 2023-12-19T16:13:05.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T16:13:05.987 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:05.984+0000 7fabd6767000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:06.030 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:06.022+0000 7fb2e8308000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:06.082 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:06.080+0000 7fabd6767000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.160+0000 7f68f700c700 1 Processor -- start 2023-12-19T16:13:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.160+0000 7f68f700c700 1 -- start start 2023-12-19T16:13:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 0x7f68f0159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f68f01595a0 0x7f68f0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f006c850 con 0x7f68f01595a0 2023-12-19T16:13:06.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f006b470 con 0x7f68f006aeb0 2023-12-19T16:13:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f700c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f006b750 con 0x7f68f0158c90 2023-12-19T16:13:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f680b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f68f01595a0 0x7f68f0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:39520/0 (socket says 172.21.15.50:39520) 2023-12-19T16:13:06.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 -- 172.21.15.50:0/1094143651 learned_addr learned my addr 172.21.15.50:0/1094143651 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f5809700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 0x7f68f0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 -- 172.21.15.50:0/1094143651 >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 msgr2=0x7f68f0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 0x7f68f0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 -- 172.21.15.50:0/1094143651 >> v2:172.21.15.50:3300/0 conn(0x7f68f01595a0 msgr2=0x7f68f0159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.50:3300/0 conn(0x7f68f01595a0 0x7f68f0159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 -- 172.21.15.50:0/1094143651 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f006ba30 con 0x7f68f0158c90 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f5809700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 0x7f68f0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.164+0000 7f68f600a700 1 --2- 172.21.15.50:0/1094143651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68f006a000 tx=0x7f68ec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a140d28048770b98 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f5008700 1 -- 172.21.15.50:0/1094143651 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec009e70 con 0x7f68f0158c90 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f5008700 1 -- 172.21.15.50:0/1094143651 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec00b860 con 0x7f68f0158c90 2023-12-19T16:13:06.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f5008700 1 -- 172.21.15.50:0/1094143651 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec00bc00 con 0x7f68f0158c90 2023-12-19T16:13:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- 172.21.15.50:0/1094143651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 msgr2=0x7f68f00caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 --2- 172.21.15.50:0/1094143651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f68f006a000 tx=0x7f68ec009250 comp rx=0 tx=0).stop 2023-12-19T16:13:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- 172.21.15.50:0/1094143651 shutdown_connections 2023-12-19T16:13:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.50:3300/0 conn(0x7f68f01595a0 0x7f68f0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 --2- 172.21.15.50:0/1094143651 >> v2:172.21.15.125:3300/0 conn(0x7f68f006aeb0 0x7f68f0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 --2- 172.21.15.50:0/1094143651 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00caf40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- 172.21.15.50:0/1094143651 >> 172.21.15.50:0/1094143651 conn(0x7f68f0067bb0 msgr2=0x7f68f00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- 172.21.15.50:0/1094143651 shutdown_connections 2023-12-19T16:13:06.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- 172.21.15.50:0/1094143651 wait complete. 2023-12-19T16:13:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 Processor -- start 2023-12-19T16:13:06.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.168+0000 7f68f700c700 1 -- start start 2023-12-19T16:13:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68f006aeb0 0x7f68f0152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.174 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00c93c0 con 0x7f68f01595a0 2023-12-19T16:13:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01537f0 con 0x7f68f006aeb0 2023-12-19T16:13:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0153070 con 0x7f68f0158c90 2023-12-19T16:13:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53086/0 (socket says 172.21.15.50:53086) 2023-12-19T16:13:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 -- 172.21.15.50:0/1400734303 learned_addr learned my addr 172.21.15.50:0/1400734303 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f5809700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f600a700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68f006aeb0 0x7f68f0152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 -- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 msgr2=0x7f68f00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 -- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68f006aeb0 msgr2=0x7f68f0152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68f006aeb0 0x7f68f0152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 -- 172.21.15.50:0/1400734303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0094f0 con 0x7f68f01595a0 2023-12-19T16:13:06.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f680b700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009e20 tx=0x7f68e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e800caa0 con 0x7f68f01595a0 2023-12-19T16:13:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e800f040 con 0x7f68f01595a0 2023-12-19T16:13:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e800ce70 con 0x7f68f01595a0 2023-12-19T16:13:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f0152c80 con 0x7f68f01595a0 2023-12-19T16:13:06.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.172+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f00c9890 con 0x7f68f01595a0 2023-12-19T16:13:06.178 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:06.176+0000 7fabd6767000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:06.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.176+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f00c9ab0 con 0x7f68f01595a0 2023-12-19T16:13:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.176+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f68e8014690 con 0x7f68f01595a0 2023-12-19T16:13:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.180+0000 7f68e67fc700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 0x7f68d4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.180+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f68e807d860 con 0x7f68f01595a0 2023-12-19T16:13:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.180+0000 7f68f600a700 1 -- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 msgr2=0x7f68d4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:06.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.180+0000 7f68f600a700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 0x7f68d4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:06.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.188+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f68e8051230 con 0x7f68f01595a0 2023-12-19T16:13:06.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.364+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68f004be40 con 0x7f68f01595a0 2023-12-19T16:13:06.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.364+0000 7f68e67fc700 1 -- 172.21.15.50:0/1400734303 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v201) v1 ==== 104+0+11 (secure 0 0 0) 0x7f68e8050dc0 con 0x7f68f01595a0 2023-12-19T16:13:06.369 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T16:13:06.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 msgr2=0x7f68d4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 0x7f68d4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 msgr2=0x7f68f00c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009e20 tx=0x7f68e800b040 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 shutdown_connections 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f68d4060790 0x7f68d4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68f01595a0 0x7f68f00c6d80 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f68f0158c90 0x7f68f00c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 --2- 172.21.15.50:0/1400734303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f68f006aeb0 0x7f68f0152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 >> 172.21.15.50:0/1400734303 conn(0x7f68f0067bb0 msgr2=0x7f68f00b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 shutdown_connections 2023-12-19T16:13:06.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.368+0000 7f68f700c700 1 -- 172.21.15.50:0/1400734303 wait complete. 2023-12-19T16:13:06.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T16:13:06.424 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:06.414+0000 7fb2e8308000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:06.585 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:06.578+0000 7fb2e8308000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 Processor -- start 2023-12-19T16:13:06.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 -- start start 2023-12-19T16:13:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff406df00 0x7f2ff4156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff406d1d0 con 0x7f2ff4156b10 2023-12-19T16:13:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff406da70 con 0x7f2ff415b110 2023-12-19T16:13:06.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff8aae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff40cda60 con 0x7f2ff406df00 2023-12-19T16:13:06.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff406df00 0x7f2ff4156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ff406df00 0x7f2ff4156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56794/0 (socket says 172.21.15.50:56794) 2023-12-19T16:13:06.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/3695901762 learned_addr learned my addr 172.21.15.50:0/3695901762 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff259c700 1 --1- 172.21.15.50:0/3695901762 >> v1:172.21.15.50:6789/0 conn(0x7f2ff4156b10 0x7f2ff415aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50380/0 (socket says 172.21.15.50:50380) 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff2d9d700 1 --1- 172.21.15.50:0/3695901762 >> v1:172.21.15.125:6789/0 conn(0x7f2ff415b110 0x7f2ff40cc3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36956/0 (socket says 172.21.15.50:36956) 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/3695901762 >> v1:172.21.15.125:6789/0 conn(0x7f2ff415b110 legacy=0x7f2ff40cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/3695901762 >> v1:172.21.15.50:6789/0 conn(0x7f2ff4156b10 legacy=0x7f2ff415aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.596+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/3695901762 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff406d420 con 0x7f2ff406df00 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff1d9b700 1 --2- 172.21.15.50:0/3695901762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff406df00 0x7f2ff4156ee0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80099b0 tx=0x7f2fe8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ff50dd0a2ea6109 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff159a700 1 -- 172.21.15.50:0/3695901762 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800b860 con 0x7f2ff406df00 2023-12-19T16:13:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff159a700 1 -- 172.21.15.50:0/3695901762 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe800b9c0 con 0x7f2ff406df00 2023-12-19T16:13:06.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff159a700 1 -- 172.21.15.50:0/3695901762 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800bd80 con 0x7f2ff406df00 2023-12-19T16:13:06.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 -- 172.21.15.50:0/3695901762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff406df00 msgr2=0x7f2ff4156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/3695901762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff406df00 0x7f2ff4156ee0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80099b0 tx=0x7f2fe8009250 comp rx=0 tx=0).stop 2023-12-19T16:13:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 -- 172.21.15.50:0/3695901762 shutdown_connections 2023-12-19T16:13:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/3695901762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff406df00 0x7f2ff4156ee0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 -- 172.21.15.50:0/3695901762 >> 172.21.15.50:0/3695901762 conn(0x7f2ff40692f0 msgr2=0x7f2ff40af990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 -- 172.21.15.50:0/3695901762 shutdown_connections 2023-12-19T16:13:06.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.600+0000 7f2ff8aae700 1 -- 172.21.15.50:0/3695901762 wait complete. 2023-12-19T16:13:06.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 Processor -- start 2023-12-19T16:13:06.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- start start 2023-12-19T16:13:06.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ff406df00 0x7f2ff40670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff415b110 0x7f2ff40b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4068160 con 0x7f2ff4156b10 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff414ebf0 con 0x7f2ff415b110 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4067620 con 0x7f2ff406df00 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55952/0 (socket says 172.21.15.50:55952) 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/1309830056 learned_addr learned my addr 172.21.15.50:0/1309830056 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff2d9d700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff415b110 0x7f2ff40b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff259c700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ff406df00 0x7f2ff40670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ff406df00 msgr2=0x7f2ff40670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ff406df00 0x7f2ff40670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff415b110 msgr2=0x7f2ff40b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff415b110 0x7f2ff40b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 -- 172.21.15.50:0/1309830056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe80094f0 con 0x7f2ff4156b10 2023-12-19T16:13:06.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff1d9b700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe801f070 con 0x7f2ff4156b10 2023-12-19T16:13:06.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe8006a90 con 0x7f2ff4156b10 2023-12-19T16:13:06.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff40678c0 con 0x7f2ff4156b10 2023-12-19T16:13:06.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8011e70 con 0x7f2ff4156b10 2023-12-19T16:13:06.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.604+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff4067e80 con 0x7f2ff4156b10 2023-12-19T16:13:06.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.608+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4071120 con 0x7f2ff4156b10 2023-12-19T16:13:06.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.608+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f2fe8023070 con 0x7f2ff4156b10 2023-12-19T16:13:06.614 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:06.612+0000 7fabd6767000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:06.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.612+0000 7f2fe27fc700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 0x7f2fd4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.612+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f2fe807e390 con 0x7f2ff4156b10 2023-12-19T16:13:06.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.612+0000 7f2ff259c700 1 -- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 msgr2=0x7f2fd4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:06.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.612+0000 7f2ff259c700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 0x7f2fd4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:06.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.620+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2fe804e330 con 0x7f2ff4156b10 2023-12-19T16:13:06.790 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ff40b12b0 con 0x7f2ff4156b10 2023-12-19T16:13:06.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2fe27fc700 1 -- 172.21.15.50:0/1309830056 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v201) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2fe8015020 con 0x7f2ff4156b10 2023-12-19T16:13:06.791 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 msgr2=0x7f2fd4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 0x7f2fd4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 msgr2=0x7f2ff4140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8017000 comp rx=0 tx=0).stop 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 shutdown_connections 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2fd4060790 0x7f2fd4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ff415b110 0x7f2ff40b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ff4156b10 0x7f2ff4140810 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 --2- 172.21.15.50:0/1309830056 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ff406df00 0x7f2ff40670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 >> 172.21.15.50:0/1309830056 conn(0x7f2ff40692f0 msgr2=0x7f2ff4155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:06.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.788+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 shutdown_connections 2023-12-19T16:13:06.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.792+0000 7f2ff8aae700 1 -- 172.21.15.50:0/1309830056 wait complete. 2023-12-19T16:13:06.805 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:06.800+0000 7fabd6767000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:06.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:06.942 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:06.934+0000 7fb2e8308000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.960+0000 7f5acd945700 1 Processor -- start 2023-12-19T16:13:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.960+0000 7f5acd945700 1 -- start start 2023-12-19T16:13:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ac8158a70 0x7f5ac806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac8158e40 0x7f5ac806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac806e100 con 0x7f5ac806dd10 2023-12-19T16:13:06.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac806a9d0 con 0x7f5ac8158e40 2023-12-19T16:13:06.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac806acb0 con 0x7f5ac8158a70 2023-12-19T16:13:06.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac8158e40 0x7f5ac806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ac8158e40 0x7f5ac806d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55966/0 (socket says 172.21.15.50:55966) 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1821778406 learned_addr learned my addr 172.21.15.50:0/1821778406 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac6ffd700 1 --2- 172.21.15.50:0/1821778406 >> v2:172.21.15.125:3300/0 conn(0x7f5ac8158a70 0x7f5ac806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1821778406 >> v2:172.21.15.125:3300/0 conn(0x7f5ac8158a70 msgr2=0x7f5ac806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 --2- 172.21.15.50:0/1821778406 >> v2:172.21.15.125:3300/0 conn(0x7f5ac8158a70 0x7f5ac806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1821778406 >> v1:172.21.15.125:6789/0 conn(0x7f5ac806dd10 legacy=0x7f5ac80c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1821778406 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac806af90 con 0x7f5ac8158e40 2023-12-19T16:13:06.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac67fc700 1 --2- 172.21.15.50:0/1821778406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac8158e40 0x7f5ac806d7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8068350 tx=0x7f5abc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440c8f7d190fbd88 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac5ffb700 1 -- 172.21.15.50:0/1821778406 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc009e70 con 0x7f5ac8158e40 2023-12-19T16:13:06.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac5ffb700 1 -- 172.21.15.50:0/1821778406 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc00b860 con 0x7f5ac8158e40 2023-12-19T16:13:06.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5ac5ffb700 1 -- 172.21.15.50:0/1821778406 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc00bc00 con 0x7f5ac8158e40 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 -- 172.21.15.50:0/1821778406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac8158e40 msgr2=0x7f5ac806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.964+0000 7f5acd945700 1 --2- 172.21.15.50:0/1821778406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac8158e40 0x7f5ac806d7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8068350 tx=0x7f5abc009250 comp rx=0 tx=0).stop 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 -- 172.21.15.50:0/1821778406 shutdown_connections 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 --2- 172.21.15.50:0/1821778406 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac8158e40 0x7f5ac806d7d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 --2- 172.21.15.50:0/1821778406 >> v2:172.21.15.125:3300/0 conn(0x7f5ac8158a70 0x7f5ac806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 -- 172.21.15.50:0/1821778406 >> 172.21.15.50:0/1821778406 conn(0x7f5ac8067bb0 msgr2=0x7f5ac80cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:06.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 -- 172.21.15.50:0/1821778406 shutdown_connections 2023-12-19T16:13:06.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 -- 172.21.15.50:0/1821778406 wait complete. 2023-12-19T16:13:06.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 Processor -- start 2023-12-19T16:13:06.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.968+0000 7f5acd945700 1 -- start start 2023-12-19T16:13:06.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac8158a70 0x7f5ac80c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 0x7f5ac80c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac815ae50 con 0x7f5ac806dd10 2023-12-19T16:13:06.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80c1c40 con 0x7f5ac8158e40 2023-12-19T16:13:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac814aad0 con 0x7f5ac8158a70 2023-12-19T16:13:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55970/0 (socket says 172.21.15.50:55970) 2023-12-19T16:13:06.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 -- 172.21.15.50:0/1455948253 learned_addr learned my addr 172.21.15.50:0/1455948253 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac77fe700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 0x7f5ac80c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac67fc700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac8158a70 0x7f5ac80c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac8158a70 msgr2=0x7f5ac80c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac8158a70 0x7f5ac80c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 msgr2=0x7f5ac80c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 0x7f5ac80c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 -- 172.21.15.50:0/1455948253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc0094f0 con 0x7f5ac806dd10 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac77fe700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 0x7f5ac80c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5ac6ffd700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec80 tx=0x7f5ab80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800c9d0 con 0x7f5ac806dd10 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab800cb30 con 0x7f5ac806dd10 2023-12-19T16:13:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8013470 con 0x7f5ac806dd10 2023-12-19T16:13:06.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac815a520 con 0x7f5ac806dd10 2023-12-19T16:13:06.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.972+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac815aae0 con 0x7f5ac806dd10 2023-12-19T16:13:06.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac804be40 con 0x7f5ac806dd10 2023-12-19T16:13:06.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f5ab8013820 con 0x7f5ac806dd10 2023-12-19T16:13:06.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5aaeffd700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 0x7f5ab4062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:06.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f5ab807d1e0 con 0x7f5ac806dd10 2023-12-19T16:13:06.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 msgr2=0x7f5ab4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:06.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.976+0000 7f5ac67fc700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 0x7f5ab4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:06.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:06.984+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ab804d180 con 0x7f5ac806dd10 2023-12-19T16:13:07.160 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.150+0000 7fb2e8308000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:07.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.180+0000 7f5ac67fc700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 msgr2=0x7f5ab4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:07.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.180+0000 7f5ac67fc700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 0x7f5ab4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:07.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.192+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ac806b270 con 0x7f5ac806dd10 2023-12-19T16:13:07.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.192+0000 7f5aaeffd700 1 -- 172.21.15.50:0/1455948253 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f5ab80507a0 con 0x7f5ac806dd10 2023-12-19T16:13:07.196 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:07.196 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:13:07.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 msgr2=0x7f5ab4062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:07.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 0x7f5ab4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 msgr2=0x7f5ac814a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec80 tx=0x7f5ab80091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 shutdown_connections 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ab40607c0 0x7f5ab4062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ac8158e40 0x7f5ac80c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ac8158a70 0x7f5ac80c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 --2- 172.21.15.50:0/1455948253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ac806dd10 0x7f5ac814a590 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 >> 172.21.15.50:0/1455948253 conn(0x7f5ac8067bb0 msgr2=0x7f5ac80c2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 shutdown_connections 2023-12-19T16:13:07.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.196+0000 7f5acd945700 1 -- 172.21.15.50:0/1455948253 wait complete. 2023-12-19T16:13:07.204 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:13:07.208 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.204+0000 7fabd6767000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:07.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:07.254 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.246+0000 7fb2e8308000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:07.336 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.326+0000 7fb2e8308000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:07.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.380+0000 7f69b088e700 1 Processor -- start 2023-12-19T16:13:07.386 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.380+0000 7f69b088e700 1 -- start start 2023-12-19T16:13:07.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69b088e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69ac158e20 0x7f69ac06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69b088e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac0ce810 0x7f69ac0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69b088e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac06b730 con 0x7f69ac1591f0 2023-12-19T16:13:07.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69b088e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac06ab20 con 0x7f69ac158e20 2023-12-19T16:13:07.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69b088e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac06ae00 con 0x7f69ac0ce810 2023-12-19T16:13:07.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69ab7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69ac158e20 0x7f69ac06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69ab7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f69ac158e20 0x7f69ac06a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55978/0 (socket says 172.21.15.50:55978) 2023-12-19T16:13:07.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69ab7fe700 1 -- 172.21.15.50:0/3289433735 learned_addr learned my addr 172.21.15.50:0/3289433735 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:07.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69aaffd700 1 --1- 172.21.15.50:0/3289433735 >> v1:172.21.15.125:6789/0 conn(0x7f69ac1591f0 0x7f69ac0ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:36970/0 (socket says 172.21.15.50:36970) 2023-12-19T16:13:07.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.384+0000 7f69abfff700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac0ce810 0x7f69ac0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69ab7fe700 1 -- 172.21.15.50:0/3289433735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac0ce810 msgr2=0x7f69ac0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:07.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69ab7fe700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac0ce810 0x7f69ac0ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69ab7fe700 1 -- 172.21.15.50:0/3289433735 >> v1:172.21.15.125:6789/0 conn(0x7f69ac1591f0 legacy=0x7f69ac0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69ab7fe700 1 -- 172.21.15.50:0/3289433735 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac06b0e0 con 0x7f69ac158e20 2023-12-19T16:13:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69ab7fe700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac158e20 0x7f69ac06a540 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f699c0099b0 tx=0x7f699c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6697ff116fa068d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69aa7fc700 1 -- 172.21.15.50:0/3289433735 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00b860 con 0x7f69ac158e20 2023-12-19T16:13:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69aa7fc700 1 -- 172.21.15.50:0/3289433735 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c00b9c0 con 0x7f69ac158e20 2023-12-19T16:13:07.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69aa7fc700 1 -- 172.21.15.50:0/3289433735 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c00bd60 con 0x7f69ac158e20 2023-12-19T16:13:07.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 -- 172.21.15.50:0/3289433735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac158e20 msgr2=0x7f69ac06a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac158e20 0x7f69ac06a540 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f699c0099b0 tx=0x7f699c009250 comp rx=0 tx=0).stop 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 -- 172.21.15.50:0/3289433735 shutdown_connections 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac0ce810 0x7f69ac0ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 --2- 172.21.15.50:0/3289433735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac158e20 0x7f69ac06a540 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 -- 172.21.15.50:0/3289433735 >> 172.21.15.50:0/3289433735 conn(0x7f69ac067bb0 msgr2=0x7f69ac0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 -- 172.21.15.50:0/3289433735 shutdown_connections 2023-12-19T16:13:07.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.388+0000 7f69b088e700 1 -- 172.21.15.50:0/3289433735 wait complete. 2023-12-19T16:13:07.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 Processor -- start 2023-12-19T16:13:07.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 -- start start 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 0x7f69ac149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ac16ce20 0x7f69ac16d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac169ec0 con 0x7f69ac0ce810 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac16d8e0 con 0x7f69ac16ce20 2023-12-19T16:13:07.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69b088e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac16dbc0 con 0x7f69ac1591f0 2023-12-19T16:13:07.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69ab7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69ab7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55988/0 (socket says 172.21.15.50:55988) 2023-12-19T16:13:07.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69ab7fe700 1 -- 172.21.15.50:0/2328304773 learned_addr learned my addr 172.21.15.50:0/2328304773 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:07.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69abfff700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ac16ce20 0x7f69ac16d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.392+0000 7f69aaffd700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 0x7f69ac149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 -- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 msgr2=0x7f69ac149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 0x7f69ac149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 -- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ac16ce20 msgr2=0x7f69ac16d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ac16ce20 0x7f69ac16d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 -- 172.21.15.50:0/2328304773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c0094f0 con 0x7f69ac0ce810 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69aaffd700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 0x7f69ac149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69ab7fe700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f69ac06e810 tx=0x7f699c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c011e20 con 0x7f69ac0ce810 2023-12-19T16:13:07.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c024440 con 0x7f69ac0ce810 2023-12-19T16:13:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c024800 con 0x7f69ac0ce810 2023-12-19T16:13:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac169b50 con 0x7f69ac0ce810 2023-12-19T16:13:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac1584f0 con 0x7f69ac0ce810 2023-12-19T16:13:07.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.396+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac169960 con 0x7f69ac0ce810 2023-12-19T16:13:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.400+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f699c01a420 con 0x7f69ac0ce810 2023-12-19T16:13:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.400+0000 7f69937fe700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 0x7f6998062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.400+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f699c07eb70 con 0x7f69ac0ce810 2023-12-19T16:13:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.400+0000 7f69aaffd700 1 -- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 msgr2=0x7f6998062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:07.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.400+0000 7f69aaffd700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 0x7f6998062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:07.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.408+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f699c04eb10 con 0x7f69ac0ce810 2023-12-19T16:13:07.419 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.410+0000 7fb2e8308000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:07.456 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.452+0000 7fabd6767000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:07.561 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.556+0000 7fabd6767000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:07.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.592+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69ac04be40 con 0x7f69ac0ce810 2023-12-19T16:13:07.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.592+0000 7f69937fe700 1 -- 172.21.15.50:0/2328304773 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+16311 (secure 0 0 0) 0x7f699c015090 con 0x7f69ac0ce810 2023-12-19T16:13:07.597 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:07.597 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":201,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:05.296456+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1157332980":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/625495530":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/4264135625":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/915278715":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/780783709":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/3568404314":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6833/44063":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6832/44063":"2023-12-20T16:11:54.658801+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:07.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 msgr2=0x7f6998062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 0x7f6998062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 msgr2=0x7f69ac16a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f69ac06e810 tx=0x7f699c017000 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 shutdown_connections 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f69980607c0 0x7f6998062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69ac16ce20 0x7f69ac16d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f69ac1591f0 0x7f69ac149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 --2- 172.21.15.50:0/2328304773 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f69ac0ce810 0x7f69ac16a350 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 >> 172.21.15.50:0/2328304773 conn(0x7f69ac067bb0 msgr2=0x7f69ac0ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 shutdown_connections 2023-12-19T16:13:07.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.596+0000 7f69b088e700 1 -- 172.21.15.50:0/2328304773 wait complete. 2023-12-19T16:13:07.622 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T16:13:07.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:07.642 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.634+0000 7fb2e8308000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:07.668 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.664+0000 7fabd6767000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:07.719 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.710+0000 7fb2e8308000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:07.757 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.752+0000 7fabd6767000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:07.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.792+0000 7f75537c5700 1 Processor -- start 2023-12-19T16:13:07.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 -- start start 2023-12-19T16:13:07.799 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f754c140710 0x7f754c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f754c0c75e0 0x7f754c068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c06e150 con 0x7f754c140710 2023-12-19T16:13:07.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c06d540 con 0x7f754c0c75e0 2023-12-19T16:13:07.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f75537c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c06d820 con 0x7f754c140ae0 2023-12-19T16:13:07.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f754c140710 0x7f754c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7550d60700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f754c140710 0x7f754c0c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55996/0 (socket says 172.21.15.50:55996) 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7550d60700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:53658/0 (socket says 172.21.15.50:53658) 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551d62700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f754c0c75e0 0x7f754c068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 -- 172.21.15.50:0/505028135 learned_addr learned my addr 172.21.15.50:0/505028135 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 -- 172.21.15.50:0/505028135 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 msgr2=0x7f754c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 --2- 172.21.15.50:0/505028135 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 -- 172.21.15.50:0/505028135 >> v2:172.21.15.125:3300/0 conn(0x7f754c0c75e0 msgr2=0x7f754c068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:07.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 --2- 172.21.15.50:0/505028135 >> v2:172.21.15.125:3300/0 conn(0x7f754c0c75e0 0x7f754c068630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 -- 172.21.15.50:0/505028135 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c06db00 con 0x7f754c140710 2023-12-19T16:13:07.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.796+0000 7f7551561700 1 --2- 172.21.15.50:0/505028135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c140710 0x7f754c0c70a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f754c0ae930 tx=0x7f7548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3356dea24c9d9c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:07.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f7543fff700 1 -- 172.21.15.50:0/505028135 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800b640 con 0x7f754c140710 2023-12-19T16:13:07.804 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f7543fff700 1 -- 172.21.15.50:0/505028135 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800b7a0 con 0x7f754c140710 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f7543fff700 1 -- 172.21.15.50:0/505028135 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800bb60 con 0x7f754c140710 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 -- 172.21.15.50:0/505028135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c140710 msgr2=0x7f754c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 --2- 172.21.15.50:0/505028135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c140710 0x7f754c0c70a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f754c0ae930 tx=0x7f7548009250 comp rx=0 tx=0).stop 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 -- 172.21.15.50:0/505028135 shutdown_connections 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 --2- 172.21.15.50:0/505028135 >> v2:172.21.15.125:3300/0 conn(0x7f754c0c75e0 0x7f754c068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 --2- 172.21.15.50:0/505028135 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 --2- 172.21.15.50:0/505028135 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c140710 0x7f754c0c70a0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 -- 172.21.15.50:0/505028135 >> 172.21.15.50:0/505028135 conn(0x7f754c0c8850 msgr2=0x7f754c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 -- 172.21.15.50:0/505028135 shutdown_connections 2023-12-19T16:13:07.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.800+0000 7f75537c5700 1 -- 172.21.15.50:0/505028135 wait complete. 2023-12-19T16:13:07.806 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.798+0000 7fb2e8308000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:07.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 Processor -- start 2023-12-19T16:13:07.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 -- start start 2023-12-19T16:13:07.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c140710 0x7f754c0c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1352a0 con 0x7f754c0c75e0 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c159450 con 0x7f754c140710 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f75537c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c132260 con 0x7f754c140ae0 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56004/0 (socket says 172.21.15.50:56004) 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 -- 172.21.15.50:0/4002445319 learned_addr learned my addr 172.21.15.50:0/4002445319 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551d62700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7550d60700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c140710 0x7f754c0c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:07.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 msgr2=0x7f754c0a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c140710 msgr2=0x7f754c0c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c140710 0x7f754c0c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 -- 172.21.15.50:0/4002445319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480094f0 con 0x7f754c0c75e0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.804+0000 7f7551561700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f754c06a280 tx=0x7f7548017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548000c90 con 0x7f754c0c75e0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800b910 con 0x7f754c0c75e0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c158840 con 0x7f754c0c75e0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548026430 con 0x7f754c0c75e0 2023-12-19T16:13:07.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c158e00 con 0x7f754c0c75e0 2023-12-19T16:13:07.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c071120 con 0x7f754c0c75e0 2023-12-19T16:13:07.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.808+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f7548023030 con 0x7f754c0c75e0 2023-12-19T16:13:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.812+0000 7f75417fa700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 0x7f7530062d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.812+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f754807e870 con 0x7f754c0c75e0 2023-12-19T16:13:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.812+0000 7f7550d60700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 msgr2=0x7f7530062d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:07.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.812+0000 7f7550d60700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 0x7f7530062d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:07.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:07.816+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f754804e910 con 0x7f754c0c75e0 2023-12-19T16:13:07.999 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:07.996+0000 7fabd6767000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:07.999 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:07.990+0000 7fb2e8308000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.016+0000 7f7550d60700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 msgr2=0x7f7530062d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:08.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.016+0000 7f7550d60700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 0x7f7530062d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:08.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.032+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754c0642a0 con 0x7f754c0c75e0 2023-12-19T16:13:08.037 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.032+0000 7f75417fa700 1 -- 172.21.15.50:0/4002445319 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f7548051f30 con 0x7f754c0c75e0 2023-12-19T16:13:08.037 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:08.037 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 msgr2=0x7f7530062d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 0x7f7530062d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 msgr2=0x7f754c134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f754c06a280 tx=0x7f7548017000 comp rx=0 tx=0).stop 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 shutdown_connections 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7530060860 0x7f7530062d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c140710 0x7f754c0c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f754c140ae0 0x7f754c0a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 --2- 172.21.15.50:0/4002445319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f754c0c75e0 0x7f754c134d60 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 >> 172.21.15.50:0/4002445319 conn(0x7f754c0c8850 msgr2=0x7f754c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 shutdown_connections 2023-12-19T16:13:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.036+0000 7f75537c5700 1 -- 172.21.15.50:0/4002445319 wait complete. 2023-12-19T16:13:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:13:08.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:08.088 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:08.084+0000 7fabd6767000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:08.089 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.078+0000 7fb2e8308000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:08.187 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:08.184+0000 7fabd6767000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:08.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 Processor -- start 2023-12-19T16:13:08.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 -- start start 2023-12-19T16:13:08.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18a0158ee0 0x7f18a0067af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f18a006d760 0x7f18a00b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a00688d0 con 0x7f18a01592b0 2023-12-19T16:13:08.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a00b1510 con 0x7f18a006d760 2023-12-19T16:13:08.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f18a4af0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a006a180 con 0x7f18a0158ee0 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f189dd9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f18a01592b0 0x7f18a006d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50390/0 (socket says 172.21.15.50:50390) 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.300+0000 7f189dd9b700 1 -- 172.21.15.50:0/2142861803 learned_addr learned my addr 172.21.15.50:0/2142861803 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 --2- 172.21.15.50:0/2142861803 >> v2:172.21.15.50:3300/0 conn(0x7f18a006d760 0x7f18a00b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189e59c700 1 --2- 172.21.15.50:0/2142861803 >> v2:172.21.15.125:3300/0 conn(0x7f18a0158ee0 0x7f18a0067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189d59a700 1 -- 172.21.15.50:0/2142861803 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581052529 0 0) 0x7f18a00688d0 con 0x7f18a01592b0 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189d59a700 1 -- 172.21.15.50:0/2142861803 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1888003570 con 0x7f18a01592b0 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 -- 172.21.15.50:0/2142861803 >> v2:172.21.15.125:3300/0 conn(0x7f18a0158ee0 msgr2=0x7f18a0067af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:08.307 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 --2- 172.21.15.50:0/2142861803 >> v2:172.21.15.125:3300/0 conn(0x7f18a0158ee0 0x7f18a0067af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 -- 172.21.15.50:0/2142861803 >> v1:172.21.15.50:6789/0 conn(0x7f18a01592b0 legacy=0x7f18a006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 -- 172.21.15.50:0/2142861803 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00685f0 con 0x7f18a006d760 2023-12-19T16:13:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189ed9d700 1 --2- 172.21.15.50:0/2142861803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00b19a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f18a00cdfd0 tx=0x7f1898009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9980882f2876ce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189d59a700 1 -- 172.21.15.50:0/2142861803 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800e070 con 0x7f18a006d760 2023-12-19T16:13:08.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189d59a700 1 -- 172.21.15.50:0/2142861803 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189800bab0 con 0x7f18a006d760 2023-12-19T16:13:08.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f189d59a700 1 -- 172.21.15.50:0/2142861803 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800be70 con 0x7f18a006d760 2023-12-19T16:13:08.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 -- 172.21.15.50:0/2142861803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 msgr2=0x7f18a00b19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 --2- 172.21.15.50:0/2142861803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00b19a0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f18a00cdfd0 tx=0x7f1898009780 comp rx=0 tx=0).stop 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 -- 172.21.15.50:0/2142861803 shutdown_connections 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 --2- 172.21.15.50:0/2142861803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00b19a0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 --2- 172.21.15.50:0/2142861803 >> v2:172.21.15.125:3300/0 conn(0x7f18a0158ee0 0x7f18a0067af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 -- 172.21.15.50:0/2142861803 >> 172.21.15.50:0/2142861803 conn(0x7f18a00cd990 msgr2=0x7f18a00ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.304+0000 7f18a4af0700 1 -- 172.21.15.50:0/2142861803 shutdown_connections 2023-12-19T16:13:08.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.308+0000 7f18a4af0700 1 -- 172.21.15.50:0/2142861803 wait complete. 2023-12-19T16:13:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.308+0000 7f18a4af0700 1 Processor -- start 2023-12-19T16:13:08.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.308+0000 7f18a4af0700 1 -- start start 2023-12-19T16:13:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18a0158ee0 0x7f18a014c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18a01592b0 0x7f18a0151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0067f20 con 0x7f18a006d760 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00c98b0 con 0x7f18a0158ee0 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00c9b90 con 0x7f18a01592b0 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56028/0 (socket says 172.21.15.50:56028) 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 -- 172.21.15.50:0/1639789053 learned_addr learned my addr 172.21.15.50:0/1639789053 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189dd9b700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18a0158ee0 0x7f18a014c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189ed9d700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18a01592b0 0x7f18a0151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18a01592b0 msgr2=0x7f18a0151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18a01592b0 0x7f18a0151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18a0158ee0 msgr2=0x7f18a014c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18a0158ee0 0x7f18a014c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 -- 172.21.15.50:0/1639789053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898009a20 con 0x7f18a006d760 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f189e59c700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f189000ec80 tx=0x7f18900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189000c850 con 0x7f18a006d760 2023-12-19T16:13:08.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189000c9b0 con 0x7f18a006d760 2023-12-19T16:13:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a00c9e70 con 0x7f18a006d760 2023-12-19T16:13:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189000cd50 con 0x7f18a006d760 2023-12-19T16:13:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.312+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a00ca200 con 0x7f18a006d760 2023-12-19T16:13:08.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.316+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0071120 con 0x7f18a006d760 2023-12-19T16:13:08.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.320+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f1890013680 con 0x7f18a006d760 2023-12-19T16:13:08.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.320+0000 7f188e7fc700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 0x7f1880062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.320+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7f189007d8c0 con 0x7f18a006d760 2023-12-19T16:13:08.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.320+0000 7f189dd9b700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 msgr2=0x7f1880062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:08.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.320+0000 7f189dd9b700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 0x7f1880062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:08.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.324+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f189004d8e0 con 0x7f18a006d760 2023-12-19T16:13:08.403 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:08.400+0000 7fabd6767000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:08.495 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:08.492+0000 7fabd6767000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:08.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.524+0000 7f189dd9b700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 msgr2=0x7f1880062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:08.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.524+0000 7f189dd9b700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 0x7f1880062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:08.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.524+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18a00b0ec0 con 0x7f18a006d760 2023-12-19T16:13:08.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f188e7fc700 1 -- 172.21.15.50:0/1639789053 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f1890050f00 con 0x7f18a006d760 2023-12-19T16:13:08.530 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:08.530 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":90,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:11:39.915607+0000","modified":"2023-12-19T16:11:43.508152+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/3963498493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3963498493},{"type":"v1","addr":"172.21.15.50:6835","nonce":3963498493}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T16:13:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 msgr2=0x7f1880062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 0x7f1880062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 msgr2=0x7f18a00679e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f189000ec80 tx=0x7f18900091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 shutdown_connections 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1880060790 0x7f1880062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18a0158ee0 0x7f18a014c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f18a01592b0 0x7f18a0151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 --2- 172.21.15.50:0/1639789053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18a006d760 0x7f18a00679e0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 >> 172.21.15.50:0/1639789053 conn(0x7f18a00cd990 msgr2=0x7f18a014d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 shutdown_connections 2023-12-19T16:13:08.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.528+0000 7f18a4af0700 1 -- 172.21.15.50:0/1639789053 wait complete. 2023-12-19T16:13:08.535 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 90 2023-12-19T16:13:08.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:08.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.716+0000 7fb9da625700 1 Processor -- start 2023-12-19T16:13:08.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9da625700 1 -- start start 2023-12-19T16:13:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9da625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 0x7fb9d4159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9da625700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40c22e0 con 0x7fb9d4159c10 2023-12-19T16:13:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9da625700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40cc8e0 con 0x7fb9d413fb40 2023-12-19T16:13:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9da625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40cc6d0 con 0x7fb9d415a690 2023-12-19T16:13:08.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9d3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb9d4159c10 0x7fb9d413f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50402/0 (socket says 172.21.15.50:50402) 2023-12-19T16:13:08.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.720+0000 7fb9d3fff700 1 -- 172.21.15.50:0/2520786244 learned_addr learned my addr 172.21.15.50:0/2520786244 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9cb7fe700 1 --2- 172.21.15.50:0/2520786244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 0x7fb9d4159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9d37fe700 1 -- 172.21.15.50:0/2520786244 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794378647 0 0) 0x7fb9d40c22e0 con 0x7fb9d4159c10 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9d37fe700 1 -- 172.21.15.50:0/2520786244 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c0003570 con 0x7fb9d4159c10 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9cb7fe700 1 -- 172.21.15.50:0/2520786244 >> v1:172.21.15.125:6789/0 conn(0x7fb9d413fb40 legacy=0x7fb9d40c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9cb7fe700 1 -- 172.21.15.50:0/2520786244 >> v1:172.21.15.50:6789/0 conn(0x7fb9d4159c10 legacy=0x7fb9d413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9cb7fe700 1 -- 172.21.15.50:0/2520786244 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40ccbc0 con 0x7fb9d415a690 2023-12-19T16:13:08.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9cb7fe700 1 --2- 172.21.15.50:0/2520786244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 0x7fb9d4159fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009910 tx=0x7fb9bc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90566a6bf17266d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:08.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9d37fe700 1 -- 172.21.15.50:0/2520786244 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00b680 con 0x7fb9d415a690 2023-12-19T16:13:08.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9d37fe700 1 -- 172.21.15.50:0/2520786244 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc00b7e0 con 0x7fb9d415a690 2023-12-19T16:13:08.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9d37fe700 1 -- 172.21.15.50:0/2520786244 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00bba0 con 0x7fb9d415a690 2023-12-19T16:13:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 -- 172.21.15.50:0/2520786244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 msgr2=0x7fb9d4159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 --2- 172.21.15.50:0/2520786244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 0x7fb9d4159fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009910 tx=0x7fb9bc0091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 -- 172.21.15.50:0/2520786244 shutdown_connections 2023-12-19T16:13:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 --2- 172.21.15.50:0/2520786244 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d415a690 0x7fb9d4159fe0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 -- 172.21.15.50:0/2520786244 >> 172.21.15.50:0/2520786244 conn(0x7fb9d40cd9c0 msgr2=0x7fb9d40c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 -- 172.21.15.50:0/2520786244 shutdown_connections 2023-12-19T16:13:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.724+0000 7fb9da625700 1 -- 172.21.15.50:0/2520786244 wait complete. 2023-12-19T16:13:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 Processor -- start 2023-12-19T16:13:08.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 -- start start 2023-12-19T16:13:08.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d413fb40 0x7fb9d40b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 0x7fb9d40c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40b2040 con 0x7fb9d4159c10 2023-12-19T16:13:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d413b0e0 con 0x7fb9d415a690 2023-12-19T16:13:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9da625700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40b0ec0 con 0x7fb9d413fb40 2023-12-19T16:13:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9cb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d3fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d413fb40 0x7fb9d40b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9cb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56036/0 (socket says 172.21.15.50:56036) 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9cb7fe700 1 -- 172.21.15.50:0/1703363399 learned_addr learned my addr 172.21.15.50:0/1703363399 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 0x7fb9d40c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 -- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d413fb40 msgr2=0x7fb9d40b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d413fb40 0x7fb9d40b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 -- 172.21.15.50:0/1703363399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 msgr2=0x7fb9d413aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9d8bc2700 1 -- 172.21.15.50:0/1703363399 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc009420 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.728+0000 7fb9cb7fe700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9d8bc2700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 0x7fb9d40c4960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c860 tx=0x7fb9cc001690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc012630 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc012790 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d40c4ea0 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc012b50 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d40b0360 con 0x7fb9d415a690 2023-12-19T16:13:08.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4071120 con 0x7fb9d415a690 2023-12-19T16:13:08.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.732+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 97) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fb9cc011380 con 0x7fb9d415a690 2023-12-19T16:13:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.736+0000 7fb9d0ff9700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 0x7fb9b4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.736+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6509+0+0 (secure 0 0 0) 0x7fb9cc07cec0 con 0x7fb9d415a690 2023-12-19T16:13:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.736+0000 7fb9d3fff700 1 -- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 msgr2=0x7fb9b4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:08.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.736+0000 7fb9d3fff700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 0x7fb9b4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.744+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9cc04cf00 con 0x7fb9d415a690 2023-12-19T16:13:08.824 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.814+0000 7fb2e8308000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.840+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mgrmap(e 98) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fb9cc04d2d0 con 0x7fb9d415a690 2023-12-19T16:13:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.840+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 msgr2=0x7fb9b4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:08.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.840+0000 7fb9d0ff9700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 0x7fb9b4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b0263700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b0263700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b0263700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b0263700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b0263700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b3269700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b3269700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b3269700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b3269700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.937 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:08.926+0000 7fb2b3269700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.936+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9d40c1c90 con 0x7fb9d415a690 2023-12-19T16:13:08.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9d0ff9700 1 -- 172.21.15.50:0/1703363399 <== mon.1 v2:172.21.15.125:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16770 (secure 0 0 0) 0x7fb9cc0690f0 con 0x7fb9d415a690 2023-12-19T16:13:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:08.942 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":202,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:08.827701+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:11:37.969076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"198","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T16:11:38.930456+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/1389374138":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/3068537263":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/886062417":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/3653663279":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/1065219367":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/1157332980":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/625495530":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/1746967300":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/4264135625":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/915278715":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/4113080782":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/780783709":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/3568404314":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6833/44063":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/2646692858":"2023-12-20T16:13:08.827635+0000","172.21.15.125:6832/44063":"2023-12-20T16:11:54.658801+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 msgr2=0x7fb9d40c4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 0x7fb9d40c4960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c860 tx=0x7fb9cc001690 comp rx=0 tx=0).stop 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 shutdown_connections 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fb9b4060790 0x7fb9b4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9d415a690 0x7fb9d40c4960 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb9d4159c10 0x7fb9d413aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 --2- 172.21.15.50:0/1703363399 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fb9d413fb40 0x7fb9d40b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:08.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 >> 172.21.15.50:0/1703363399 conn(0x7fb9d40cd9c0 msgr2=0x7fb9d40abc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:08.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 shutdown_connections 2023-12-19T16:13:08.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:08.940+0000 7fb9da625700 1 -- 172.21.15.50:0/1703363399 wait complete. 2023-12-19T16:13:08.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T16:13:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 Processor -- start 2023-12-19T16:13:09.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 -- start start 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab7006dc00 0x7fab7006b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7006b0d0 con 0x7fab70158cc0 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700b4790 con 0x7fab7006baa0 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab768cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7006bec0 con 0x7fab7006dc00 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab6f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab7006dc00 0x7fab7006b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab6f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fab7006dc00 0x7fab7006b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56038/0 (socket says 172.21.15.50:56038) 2023-12-19T16:13:09.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.136+0000 7fab6f7fe700 1 -- 172.21.15.50:0/1762455932 learned_addr learned my addr 172.21.15.50:0/1762455932 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6f7fe700 1 -- 172.21.15.50:0/1762455932 >> v1:172.21.15.125:6789/0 conn(0x7fab7006baa0 legacy=0x7fab700b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6f7fe700 1 -- 172.21.15.50:0/1762455932 >> v1:172.21.15.50:6789/0 conn(0x7fab70158cc0 legacy=0x7fab7006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6f7fe700 1 -- 172.21.15.50:0/1762455932 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7006a7e0 con 0x7fab7006dc00 2023-12-19T16:13:09.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6f7fe700 1 --2- 172.21.15.50:0/1762455932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab7006b560 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fab640099b0 tx=0x7fab64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b24df5316f15aab server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6effd700 1 -- 172.21.15.50:0/1762455932 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400b860 con 0x7fab7006dc00 2023-12-19T16:13:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6effd700 1 -- 172.21.15.50:0/1762455932 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6400b9c0 con 0x7fab7006dc00 2023-12-19T16:13:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab6effd700 1 -- 172.21.15.50:0/1762455932 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400bd60 con 0x7fab7006dc00 2023-12-19T16:13:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 -- 172.21.15.50:0/1762455932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 msgr2=0x7fab7006b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 --2- 172.21.15.50:0/1762455932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab7006b560 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fab640099b0 tx=0x7fab64009250 comp rx=0 tx=0).stop 2023-12-19T16:13:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 -- 172.21.15.50:0/1762455932 shutdown_connections 2023-12-19T16:13:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 --2- 172.21.15.50:0/1762455932 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab7006b560 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 -- 172.21.15.50:0/1762455932 >> 172.21.15.50:0/1762455932 conn(0x7fab70067bb0 msgr2=0x7fab700cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 -- 172.21.15.50:0/1762455932 shutdown_connections 2023-12-19T16:13:09.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 -- 172.21.15.50:0/1762455932 wait complete. 2023-12-19T16:13:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.140+0000 7fab768cd700 1 Processor -- start 2023-12-19T16:13:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- start start 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 0x7fab70152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 0x7fab70150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7016c710 con 0x7fab7006dc00 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7016c9f0 con 0x7fab70158cc0 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab701699b0 con 0x7fab7006baa0 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56054/0 (socket says 172.21.15.50:56054) 2023-12-19T16:13:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 -- 172.21.15.50:0/3811247924 learned_addr learned my addr 172.21.15.50:0/3811247924 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 -- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 msgr2=0x7fab70152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6ffff700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 0x7fab70152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 0x7fab70152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab74e6a700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 0x7fab70150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 -- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 msgr2=0x7fab70150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 0x7fab70150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 -- 172.21.15.50:0/3811247924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab640094f0 con 0x7fab7006dc00 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6ffff700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 0x7fab70152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab74e6a700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 0x7fab70150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab6f7fe700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fab64006fd0 tx=0x7fab64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64011e60 con 0x7fab7006dc00 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6401a630 con 0x7fab7006dc00 2023-12-19T16:13:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6401a9d0 con 0x7fab7006dc00 2023-12-19T16:13:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab70169c90 con 0x7fab7006dc00 2023-12-19T16:13:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.144+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70169610 con 0x7fab7006dc00 2023-12-19T16:13:09.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.148+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7016cb80 con 0x7fab7006dc00 2023-12-19T16:13:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.148+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 98) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fab6402d440 con 0x7fab7006dc00 2023-12-19T16:13:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.148+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6825+0+0 (secure 0 0 0) 0x7fab6407e2e0 con 0x7fab7006dc00 2023-12-19T16:13:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.156+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fab6404e740 con 0x7fab7006dc00 2023-12-19T16:13:09.297 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:09.292+0000 7fabd6767000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:09.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.364+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fab70071120 con 0x7fab7006dc00 2023-12-19T16:13:09.369 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T16:13:09.364+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T16:13:09.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.376+0000 7fab57fff700 1 -- 172.21.15.50:0/3811247924 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v91) v1 ==== 161+0+0 (secure 0 0 0) 0x7fab64051d60 con 0x7fab7006dc00 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 msgr2=0x7fab701690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fab64006fd0 tx=0x7fab64017000 comp rx=0 tx=0).stop 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 shutdown_connections 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab70158cc0 0x7fab70150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab7006dc00 0x7fab701690d0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 --2- 172.21.15.50:0/3811247924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fab7006baa0 0x7fab70152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 >> 172.21.15.50:0/3811247924 conn(0x7fab70067bb0 msgr2=0x7fab70151290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 shutdown_connections 2023-12-19T16:13:09.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.380+0000 7fab768cd700 1 -- 172.21.15.50:0/3811247924 wait complete. 2023-12-19T16:13:09.386 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T16:13:09.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T16:13:09.447 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T16:13:09.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 Processor -- start 2023-12-19T16:13:09.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 -- start start 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20100c6d20 0x7f20100c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f201006c7c0 0x7f201006cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f201015aa30 0x7f201015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201006a7a0 con 0x7f201015aa30 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100c7630 con 0x7f201006c7c0 2023-12-19T16:13:09.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.564+0000 7f20148f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201006d4b0 con 0x7f20100c6d20 2023-12-19T16:13:09.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f201015aa30 0x7f201015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200dd9b700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f201006c7c0 0x7f201006cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f201015aa30 0x7f201015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56092/0 (socket says 172.21.15.50:56092) 2023-12-19T16:13:09.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 -- 172.21.15.50:0/3382437589 learned_addr learned my addr 172.21.15.50:0/3382437589 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:09.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200e59c700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20100c6d20 0x7f20100c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 -- 172.21.15.50:0/3382437589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20100c6d20 msgr2=0x7f20100c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20100c6d20 0x7f20100c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 -- 172.21.15.50:0/3382437589 >> v2:172.21.15.125:3300/0 conn(0x7f201006c7c0 msgr2=0x7f201006cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 --2- 172.21.15.50:0/3382437589 >> v2:172.21.15.125:3300/0 conn(0x7f201006c7c0 0x7f201006cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 -- 172.21.15.50:0/3382437589 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201006d790 con 0x7f201015aa30 2023-12-19T16:13:09.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200ed9d700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201015aa30 0x7f201015ae30 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f2010068060 tx=0x7f2008009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894ee3240f0b6ccb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:09.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200d59a700 1 -- 172.21.15.50:0/3382437589 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200800e070 con 0x7f201015aa30 2023-12-19T16:13:09.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200d59a700 1 -- 172.21.15.50:0/3382437589 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200800b970 con 0x7f201015aa30 2023-12-19T16:13:09.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f200d59a700 1 -- 172.21.15.50:0/3382437589 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080092e0 con 0x7f201015aa30 2023-12-19T16:13:09.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 -- 172.21.15.50:0/3382437589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201015aa30 msgr2=0x7f201015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201015aa30 0x7f201015ae30 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f2010068060 tx=0x7f2008009840 comp rx=0 tx=0).stop 2023-12-19T16:13:09.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 -- 172.21.15.50:0/3382437589 shutdown_connections 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f201015aa30 0x7f201015ae30 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 --2- 172.21.15.50:0/3382437589 >> v2:172.21.15.125:3300/0 conn(0x7f201006c7c0 0x7f201006cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 --2- 172.21.15.50:0/3382437589 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f20100c6d20 0x7f20100c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 -- 172.21.15.50:0/3382437589 >> 172.21.15.50:0/3382437589 conn(0x7f2010067bb0 msgr2=0x7f20100cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 -- 172.21.15.50:0/3382437589 shutdown_connections 2023-12-19T16:13:09.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.568+0000 7f20148f0700 1 -- 172.21.15.50:0/3382437589 wait complete. 2023-12-19T16:13:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 Processor -- start 2023-12-19T16:13:09.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 -- start start 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201006c7c0 0x7f20100a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201015aa30 0x7f20100ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100ca840 con 0x7f20100c6d20 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100cab20 con 0x7f201015aa30 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f20148f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100cae00 con 0x7f201006c7c0 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f200dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f200dd9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56108/0 (socket says 172.21.15.50:56108) 2023-12-19T16:13:09.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f200dd9b700 1 -- 172.21.15.50:0/372958878 learned_addr learned my addr 172.21.15.50:0/372958878 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f200ed9d700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201015aa30 0x7f20100ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.572+0000 7f200e59c700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201006c7c0 0x7f20100a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 -- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201006c7c0 msgr2=0x7f20100a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201006c7c0 0x7f20100a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 -- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201015aa30 msgr2=0x7f20100ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201015aa30 0x7f20100ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 -- 172.21.15.50:0/372958878 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008009ae0 con 0x7f20100c6d20 2023-12-19T16:13:09.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f200dd9b700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f200400ec80 tx=0x7f20040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400c850 con 0x7f20100c6d20 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200400c9b0 con 0x7f20100c6d20 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100cb0e0 con 0x7f20100c6d20 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400cd50 con 0x7f20100c6d20 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100c30e0 con 0x7f20100c6d20 2023-12-19T16:13:09.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.576+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2010071120 con 0x7f20100c6d20 2023-12-19T16:13:09.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.580+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 98) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f2004013680 con 0x7f20100c6d20 2023-12-19T16:13:09.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.580+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6911+0+0 (secure 0 0 0) 0x7f200407caa0 con 0x7f20100c6d20 2023-12-19T16:13:09.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.588+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f200404ceb0 con 0x7f20100c6d20 2023-12-19T16:13:09.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.800+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f201006da70 con 0x7f20100c6d20 2023-12-19T16:13:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.844+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 99) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f200404c960 con 0x7f20100c6d20 2023-12-19T16:13:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.844+0000 7f1ffe7fc700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 0x7f1ff0068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:09.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.844+0000 7f200e59c700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 0x7f1ff0068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:09.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:09.848+0000 7f200e59c700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 0x7f1ff0068980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2000009db0 tx=0x7f20000091a0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:10.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.380+0000 7f1ffe7fc700 1 -- 172.21.15.50:0/372958878 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7f20040688b0 con 0x7f20100c6d20 2023-12-19T16:13:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 msgr2=0x7f1ff0068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 0x7f1ff0068980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2000009db0 tx=0x7f20000091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 msgr2=0x7f20100c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f200400ec80 tx=0x7f20040091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 shutdown_connections 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1ff00665a0 0x7f1ff0068980 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f201015aa30 0x7f20100ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20100c6d20 0x7f20100c9ad0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 --2- 172.21.15.50:0/372958878 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f201006c7c0 0x7f20100a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 >> 172.21.15.50:0/372958878 conn(0x7f2010067bb0 msgr2=0x7f2010156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 shutdown_connections 2023-12-19T16:13:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.384+0000 7f20148f0700 1 -- 172.21.15.50:0/372958878 wait complete. 2023-12-19T16:13:10.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T16:13:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.632+0000 7efc1a4cd700 1 Processor -- start 2023-12-19T16:13:10.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.632+0000 7efc1a4cd700 1 -- start start 2023-12-19T16:13:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc1a4cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efc140cb500 0x7efc1415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc1a4cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140b0ac0 con 0x7efc1415c760 2023-12-19T16:13:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc1a4cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1406b210 con 0x7efc140cb130 2023-12-19T16:13:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc1a4cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1415cbe0 con 0x7efc140cb500 2023-12-19T16:13:10.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc0bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7efc140cb500 0x7efc1415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc18a6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efc1415c760 0x7efc140b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50424/0 (socket says 172.21.15.50:50424) 2023-12-19T16:13:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.636+0000 7efc18a6a700 1 -- 172.21.15.50:0/2871687394 learned_addr learned my addr 172.21.15.50:0/2871687394 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:10.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731989974 0 0) 0x7efc1406b210 con 0x7efc140cb130 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc00003570 con 0x7efc140cb130 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc0bfff700 1 -- 172.21.15.50:0/2871687394 >> v1:172.21.15.125:6789/0 conn(0x7efc140cb130 legacy=0x7efc1406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc0bfff700 1 -- 172.21.15.50:0/2871687394 >> v1:172.21.15.50:6789/0 conn(0x7efc1415c760 legacy=0x7efc140b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986843627 0 0) 0x7efc140b0ac0 con 0x7efc1415c760 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc0bfff700 1 -- 172.21.15.50:0/2871687394 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140ca810 con 0x7efc140cb500 2023-12-19T16:13:10.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc0bfff700 1 --2- 172.21.15.50:0/2871687394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc1415c220 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0098e0 tx=0x7efbfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceca33e83f061daa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00b640 con 0x7efc140cb500 2023-12-19T16:13:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc00b7a0 con 0x7efc140cb500 2023-12-19T16:13:10.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc137fe700 1 -- 172.21.15.50:0/2871687394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00bb40 con 0x7efc140cb500 2023-12-19T16:13:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2871687394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 msgr2=0x7efc1415c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2871687394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc1415c220 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0098e0 tx=0x7efbfc0091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2871687394 shutdown_connections 2023-12-19T16:13:10.645 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2871687394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc1415c220 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2871687394 >> 172.21.15.50:0/2871687394 conn(0x7efc14067bb0 msgr2=0x7efc140cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2871687394 shutdown_connections 2023-12-19T16:13:10.646 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.640+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2871687394 wait complete. 2023-12-19T16:13:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 Processor -- start 2023-12-19T16:13:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 -- start start 2023-12-19T16:13:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 0x7efc1415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 0x7efc140c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:10.648 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14147120 con 0x7efc140cb500 2023-12-19T16:13:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1415f0c0 con 0x7efc140cb130 2023-12-19T16:13:10.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc1a4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1415e4b0 con 0x7efc1415c760 2023-12-19T16:13:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc0bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc0bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56122/0 (socket says 172.21.15.50:56122) 2023-12-19T16:13:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc0bfff700 1 -- 172.21.15.50:0/2213480357 learned_addr learned my addr 172.21.15.50:0/2213480357 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:10.650 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc18a6a700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 0x7efc140c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.644+0000 7efc13fff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 0x7efc1415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 -- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 msgr2=0x7efc140c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 0x7efc140c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 -- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 msgr2=0x7efc1415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 0x7efc1415f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 -- 172.21.15.50:0/2213480357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc009420 con 0x7efc140cb500 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc18a6a700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 0x7efc140c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc13fff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 0x7efc1415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:10.651 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc0bfff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7efc1406f300 tx=0x7efbfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc01f070 con 0x7efc140cb500 2023-12-19T16:13:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc000c90 con 0x7efc140cb500 2023-12-19T16:13:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1415e790 con 0x7efc140cb500 2023-12-19T16:13:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00bb40 con 0x7efc140cb500 2023-12-19T16:13:10.652 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1415ed50 con 0x7efc140cb500 2023-12-19T16:13:10.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc14071120 con 0x7efc140cb500 2023-12-19T16:13:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.648+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 99) v1 ==== 79799+0+0 (secure 0 0 0) 0x7efbfc023070 con 0x7efc140cb500 2023-12-19T16:13:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.652+0000 7efc10ff9700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 0x7efbf4062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:10.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.652+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 6911+0+0 (secure 0 0 0) 0x7efbfc07def0 con 0x7efc140cb500 2023-12-19T16:13:10.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.652+0000 7efc13fff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 0x7efbf4062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.656+0000 7efc13fff700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 0x7efbf4062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc04000c00 tx=0x7efc04009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:10.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.660+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efbfc04e210 con 0x7efc140cb500 2023-12-19T16:13:10.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:10.844+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc140cbe90 con 0x7efc140cb500 2023-12-19T16:13:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.388+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7efbfc04e5e0 con 0x7efc140cb500 2023-12-19T16:13:11.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.860+0000 7efc10ff9700 1 -- 172.21.15.50:0/2213480357 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v204) v1 ==== 181+0+0 (secure 0 0 0) 0x7efbfc051ca0 con 0x7efc140cb500 2023-12-19T16:13:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.864+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 msgr2=0x7efbf4062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.864+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 0x7efbf4062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc04000c00 tx=0x7efc04009040 comp rx=0 tx=0).stop 2023-12-19T16:13:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.864+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 msgr2=0x7efc14146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.864+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7efc1406f300 tx=0x7efbfc017000 comp rx=0 tx=0).stop 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 shutdown_connections 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7efbf40603a0 0x7efbf4062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7efc1415c760 0x7efc140c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc140cb500 0x7efc14146be0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 --2- 172.21.15.50:0/2213480357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc140cb130 0x7efc1415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 >> 172.21.15.50:0/2213480357 conn(0x7efc14067bb0 msgr2=0x7efc140cc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 shutdown_connections 2023-12-19T16:13:11.873 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:11.868+0000 7efc1a4cd700 1 -- 172.21.15.50:0/2213480357 wait complete. 2023-12-19T16:13:11.876 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T16:13:11.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T16:13:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.068+0000 7fcb99f63700 1 Processor -- start 2023-12-19T16:13:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb99f63700 1 -- start start 2023-12-19T16:13:12.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb99f63700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb940b0040 0x7fcb9406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb99f63700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb940cb3f0 con 0x7fcb940aedc0 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb99f63700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9406d220 con 0x7fcb940af190 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb99f63700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb940cc140 con 0x7fcb940b0040 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.072+0000 7fcb98f61700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcb940aedc0 0x7fcb9406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50426/0 (socket says 172.21.15.50:50426) 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb98f61700 1 -- 172.21.15.50:0/3516846211 learned_addr learned my addr 172.21.15.50:0/3516846211 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:12.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99762700 1 --2- 172.21.15.50:0/3516846211 >> v2:172.21.15.50:3300/0 conn(0x7fcb940b0040 0x7fcb9406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99762700 1 -- 172.21.15.50:0/3516846211 >> v1:172.21.15.125:6789/0 conn(0x7fcb940af190 legacy=0x7fcb940af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99762700 1 -- 172.21.15.50:0/3516846211 >> v1:172.21.15.50:6789/0 conn(0x7fcb940aedc0 legacy=0x7fcb9406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99762700 1 -- 172.21.15.50:0/3516846211 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9406c8f0 con 0x7fcb940b0040 2023-12-19T16:13:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99762700 1 --2- 172.21.15.50:0/3516846211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940b0040 0x7fcb9406af50 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009e60 tx=0x7fcb8c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82c19364dac0b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb937fe700 1 -- 172.21.15.50:0/3516846211 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00e070 con 0x7fcb940b0040 2023-12-19T16:13:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb937fe700 1 -- 172.21.15.50:0/3516846211 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c00bc30 con 0x7fcb940b0040 2023-12-19T16:13:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb937fe700 1 -- 172.21.15.50:0/3516846211 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c012430 con 0x7fcb940b0040 2023-12-19T16:13:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 -- 172.21.15.50:0/3516846211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940b0040 msgr2=0x7fcb9406af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 --2- 172.21.15.50:0/3516846211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940b0040 0x7fcb9406af50 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009e60 tx=0x7fcb8c009780 comp rx=0 tx=0).stop 2023-12-19T16:13:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 -- 172.21.15.50:0/3516846211 shutdown_connections 2023-12-19T16:13:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 --2- 172.21.15.50:0/3516846211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940b0040 0x7fcb9406af50 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 -- 172.21.15.50:0/3516846211 >> 172.21.15.50:0/3516846211 conn(0x7fcb940cc760 msgr2=0x7fcb940ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 -- 172.21.15.50:0/3516846211 shutdown_connections 2023-12-19T16:13:12.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.076+0000 7fcb99f63700 1 -- 172.21.15.50:0/3516846211 wait complete. 2023-12-19T16:13:12.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 Processor -- start 2023-12-19T16:13:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 -- start start 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb940aedc0 0x7fcb940b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 0x7fcb9414c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940af610 con 0x7fcb940af190 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940b4650 con 0x7fcb940b0040 2023-12-19T16:13:12.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99f63700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940b4930 con 0x7fcb940aedc0 2023-12-19T16:13:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb93fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 0x7fcb9414c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb98f61700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb940aedc0 0x7fcb940b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99762700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99762700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:56944/0 (socket says 172.21.15.50:56944) 2023-12-19T16:13:12.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.080+0000 7fcb99762700 1 -- 172.21.15.50:0/2755976695 learned_addr learned my addr 172.21.15.50:0/2755976695 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 -- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb940aedc0 msgr2=0x7fcb940b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb940aedc0 0x7fcb940b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 -- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 msgr2=0x7fcb940cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 -- 172.21.15.50:0/2755976695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c009a20 con 0x7fcb940af190 2023-12-19T16:13:12.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb99762700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb93fff700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 0x7fcb9414c000 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c590 tx=0x7fcb80007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80011590 con 0x7fcb940af190 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb80011710 con 0x7fcb940af190 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb940b4c10 con 0x7fcb940af190 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80011ad0 con 0x7fcb940af190 2023-12-19T16:13:12.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb940b51d0 con 0x7fcb940af190 2023-12-19T16:13:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.084+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb940b53f0 con 0x7fcb940af190 2023-12-19T16:13:12.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.088+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcb8001b3b0 con 0x7fcb940af190 2023-12-19T16:13:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.088+0000 7fcb90ff9700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 0x7fcb78062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:12.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.088+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fcb8007bf00 con 0x7fcb940af190 2023-12-19T16:13:12.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.088+0000 7fcb98f61700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 0x7fcb78062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:12.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.088+0000 7fcb98f61700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 0x7fcb78062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb88007130 tx=0x7fcb88007eb0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:12.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.096+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcb8004c0e0 con 0x7fcb940af190 2023-12-19T16:13:12.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.268+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb9404be40 con 0x7fcb940af190 2023-12-19T16:13:12.868 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.864+0000 7fcb90ff9700 1 -- 172.21.15.50:0/2755976695 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v205) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcb8004f700 con 0x7fcb940af190 2023-12-19T16:13:12.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 msgr2=0x7fcb78062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 0x7fcb78062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb88007130 tx=0x7fcb88007eb0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 msgr2=0x7fcb9414c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:12.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 0x7fcb9414c000 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c590 tx=0x7fcb80007e80 comp rx=0 tx=0).stop 2023-12-19T16:13:12.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 shutdown_connections 2023-12-19T16:13:12.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb78060670 0x7fcb78062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb940b0040 0x7fcb940cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb940af190 0x7fcb9414c000 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 --2- 172.21.15.50:0/2755976695 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb940aedc0 0x7fcb940b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 >> 172.21.15.50:0/2755976695 conn(0x7fcb940cc760 msgr2=0x7fcb9406db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.872+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 shutdown_connections 2023-12-19T16:13:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:12.876+0000 7fcb99f63700 1 -- 172.21.15.50:0/2755976695 wait complete. 2023-12-19T16:13:12.885 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T16:13:12.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T16:13:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 Processor -- start 2023-12-19T16:13:13.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 -- start start 2023-12-19T16:13:13.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d8806c7c0 0x7f3d8806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8806a7a0 con 0x7f3d880c6d20 2023-12-19T16:13:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d880c7630 con 0x7f3d8806c7c0 2023-12-19T16:13:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.088+0000 7f3d8f785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8806d4b0 con 0x7f3d8815aa30 2023-12-19T16:13:13.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8d521700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8cd20700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d8806c7c0 0x7f3d8806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8d521700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56148/0 (socket says 172.21.15.50:56148) 2023-12-19T16:13:13.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8d521700 1 -- 172.21.15.50:0/352736864 learned_addr learned my addr 172.21.15.50:0/352736864 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 -- 172.21.15.50:0/352736864 >> v2:172.21.15.125:3300/0 conn(0x7f3d8806c7c0 msgr2=0x7f3d8806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 --2- 172.21.15.50:0/352736864 >> v2:172.21.15.125:3300/0 conn(0x7f3d8806c7c0 0x7f3d8806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 -- 172.21.15.50:0/352736864 >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 msgr2=0x7f3d880c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 --2- 172.21.15.50:0/352736864 >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 -- 172.21.15.50:0/352736864 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8806d790 con 0x7f3d8815aa30 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8d521700 1 --2- 172.21.15.50:0/352736864 >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:13.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8dd22700 1 --2- 172.21.15.50:0/352736864 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815ae50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c86f0 tx=0x7f3d80009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98386134da3358f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d7ffff700 1 -- 172.21.15.50:0/352736864 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8000e070 con 0x7f3d8815aa30 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d7ffff700 1 -- 172.21.15.50:0/352736864 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8000bab0 con 0x7f3d8815aa30 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d7ffff700 1 -- 172.21.15.50:0/352736864 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8000be70 con 0x7f3d8815aa30 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 -- 172.21.15.50:0/352736864 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 msgr2=0x7f3d8815ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 --2- 172.21.15.50:0/352736864 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815ae50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c86f0 tx=0x7f3d80009780 comp rx=0 tx=0).stop 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 -- 172.21.15.50:0/352736864 shutdown_connections 2023-12-19T16:13:13.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 --2- 172.21.15.50:0/352736864 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815ae50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 --2- 172.21.15.50:0/352736864 >> v2:172.21.15.125:3300/0 conn(0x7f3d8806c7c0 0x7f3d8806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 --2- 172.21.15.50:0/352736864 >> v2:172.21.15.50:3300/0 conn(0x7f3d880c6d20 0x7f3d880c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.092+0000 7f3d8f785700 1 -- 172.21.15.50:0/352736864 >> 172.21.15.50:0/352736864 conn(0x7f3d88067bb0 msgr2=0x7f3d880cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.096+0000 7f3d8f785700 1 -- 172.21.15.50:0/352736864 shutdown_connections 2023-12-19T16:13:13.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.096+0000 7f3d8f785700 1 -- 172.21.15.50:0/352736864 wait complete. 2023-12-19T16:13:13.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.096+0000 7f3d8f785700 1 Processor -- start 2023-12-19T16:13:13.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 -- start start 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8806c7c0 0x7f3d88168f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c98a0 con 0x7f3d880c6d20 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c9b80 con 0x7f3d8806c7c0 2023-12-19T16:13:13.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8f785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d881697d0 con 0x7f3d8815aa30 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56150/0 (socket says 172.21.15.50:56150) 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 -- 172.21.15.50:0/2709211634 learned_addr learned my addr 172.21.15.50:0/2709211634 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8d521700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8806c7c0 0x7f3d88168f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8dd22700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 -- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 msgr2=0x7f3d8815cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 -- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8806c7c0 msgr2=0x7f3d88168f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8806c7c0 0x7f3d88168f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.100+0000 7f3d8cd20700 1 -- 172.21.15.50:0/2709211634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840094f0 con 0x7f3d880c6d20 2023-12-19T16:13:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d8cd20700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800ecb0 tx=0x7f3d780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800c850 con 0x7f3d880c6d20 2023-12-19T16:13:13.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7800c9b0 con 0x7f3d880c6d20 2023-12-19T16:13:13.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d80009a20 con 0x7f3d880c6d20 2023-12-19T16:13:13.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800cd50 con 0x7f3d880c6d20 2023-12-19T16:13:13.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d88169d90 con 0x7f3d880c6d20 2023-12-19T16:13:13.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d88071120 con 0x7f3d880c6d20 2023-12-19T16:13:13.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.104+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3d78013680 con 0x7f3d880c6d20 2023-12-19T16:13:13.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.108+0000 7f3d7d7fa700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 0x7f3d6c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.108+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7f3d7807cc90 con 0x7f3d880c6d20 2023-12-19T16:13:13.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.108+0000 7f3d8d521700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 0x7f3d6c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.112+0000 7f3d8d521700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 0x7f3d6c062bf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006d20 tx=0x7f3d84009250 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.116+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3d7804cff0 con 0x7f3d880c6d20 2023-12-19T16:13:13.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.292+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3d8806da70 con 0x7f3d880c6d20 2023-12-19T16:13:13.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.300+0000 7f3d7d7fa700 1 -- 172.21.15.50:0/2709211634 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3d78050610 con 0x7f3d880c6d20 2023-12-19T16:13:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 msgr2=0x7f3d6c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 0x7f3d6c062bf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006d20 tx=0x7f3d84009250 comp rx=0 tx=0).stop 2023-12-19T16:13:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 msgr2=0x7f3d88153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.308 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800ecb0 tx=0x7f3d780091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 shutdown_connections 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3d6c060740 0x7f3d6c062bf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3d8815aa30 0x7f3d8815cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3d880c6d20 0x7f3d88153b80 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 --2- 172.21.15.50:0/2709211634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d8806c7c0 0x7f3d88168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 >> 172.21.15.50:0/2709211634 conn(0x7f3d88067bb0 msgr2=0x7f3d88157990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 shutdown_connections 2023-12-19T16:13:13.309 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.304+0000 7f3d8f785700 1 -- 172.21.15.50:0/2709211634 wait complete. 2023-12-19T16:13:13.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T16:13:13.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.508+0000 7ff84e2cd700 1 Processor -- start 2023-12-19T16:13:13.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.508+0000 7ff84e2cd700 1 -- start start 2023-12-19T16:13:13.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff84e2cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff84815d410 0x7ff8480ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff84e2cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480c7450 con 0x7ff8480ca400 2023-12-19T16:13:13.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff84e2cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84806bea0 con 0x7ff84806b1b0 2023-12-19T16:13:13.515 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff84e2cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8480c6e00 con 0x7ff84815d410 2023-12-19T16:13:13.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff8477fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff8480ca400 0x7ff8480ca810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50430/0 (socket says 172.21.15.50:50430) 2023-12-19T16:13:13.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff8477fe700 1 -- 172.21.15.50:0/3070640956 learned_addr learned my addr 172.21.15.50:0/3070640956 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:13.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff847fff700 1 --2- 172.21.15.50:0/3070640956 >> v2:172.21.15.125:3300/0 conn(0x7ff84815d410 0x7ff8480ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.516 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509108630 0 0) 0x7ff8480c7450 con 0x7ff8480ca400 2023-12-19T16:13:13.517 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff834003570 con 0x7ff8480ca400 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089947466 0 0) 0x7ff84806bea0 con 0x7ff84806b1b0 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8480c7450 con 0x7ff84806b1b0 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045398805 0 0) 0x7ff834003570 con 0x7ff8480ca400 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 >> v2:172.21.15.125:3300/0 conn(0x7ff84815d410 msgr2=0x7ff8480ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 --2- 172.21.15.50:0/3070640956 >> v2:172.21.15.125:3300/0 conn(0x7ff84815d410 0x7ff8480ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 >> v1:172.21.15.125:6789/0 conn(0x7ff84806b1b0 legacy=0x7ff84806b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.518 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480c70e0 con 0x7ff8480ca400 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7ff838001470 con 0x7ff8480ca400 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.512+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 >> v1:172.21.15.50:6789/0 conn(0x7ff8480ca400 legacy=0x7ff8480ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff834003bf0 0x7ff834003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480c6e00 con 0x7ff8340051c0 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83400c160 con 0x7ff834004810 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83400c320 con 0x7ff834003bf0 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff8477fe700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff847fff700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff834003bf0 0x7ff834003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 -- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff834003bf0 msgr2=0x7ff834003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff834003bf0 0x7ff834003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 -- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff834004810 msgr2=0x7ff834004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff834004810 0x7ff834004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 -- 172.21.15.50:0/3070640956 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834003570 con 0x7ff8340051c0 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 -- 172.21.15.50:0/3070640956 reap_dead start 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84c86a700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff8300019e0 tx=0x7ff830007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d88bc1021c33ea1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83000c5d0 con 0x7ff8340051c0 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83000c730 con 0x7ff8340051c0 2023-12-19T16:13:13.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff846ffd700 1 -- 172.21.15.50:0/3070640956 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83000cad0 con 0x7ff8340051c0 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 -- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 msgr2=0x7ff83400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7ff8300019e0 tx=0x7ff830007d30 comp rx=0 tx=0).stop 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 -- 172.21.15.50:0/3070640956 shutdown_connections 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/3070640956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8340051c0 0x7ff83400ba60 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 -- 172.21.15.50:0/3070640956 >> 172.21.15.50:0/3070640956 conn(0x7ff848067bb0 msgr2=0x7ff8480cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.516+0000 7ff84e2cd700 1 -- 172.21.15.50:0/3070640956 shutdown_connections 2023-12-19T16:13:13.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.520+0000 7ff84e2cd700 1 -- 172.21.15.50:0/3070640956 wait complete. 2023-12-19T16:13:13.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.520+0000 7ff84e2cd700 1 Processor -- start 2023-12-19T16:13:13.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.520+0000 7ff84e2cd700 1 -- start start 2023-12-19T16:13:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 0x7ff848158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 0x7ff8480c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff848148530 con 0x7ff8481480c0 2023-12-19T16:13:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480b0a70 con 0x7ff8480c57f0 2023-12-19T16:13:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480b0330 con 0x7ff8340051c0 2023-12-19T16:13:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56168/0 (socket says 172.21.15.50:56168) 2023-12-19T16:13:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 -- 172.21.15.50:0/1319430266 learned_addr learned my addr 172.21.15.50:0/1319430266 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff847fff700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 0x7ff848158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 -- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 msgr2=0x7ff848158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84c86a700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 0x7ff8480c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 0x7ff848158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 -- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 msgr2=0x7ff8480c0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 0x7ff8480c0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 -- 172.21.15.50:0/1319430266 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480afd70 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff847fff700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 0x7ff848158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84c86a700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 0x7ff8480c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff8477fe700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ff8380073c0 tx=0x7ff838001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff838018030 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff838007830 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8480b0050 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800fae0 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.524+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8480c5c60 con 0x7ff8481480c0 2023-12-19T16:13:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.528+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8480b08a0 con 0x7ff8481480c0 2023-12-19T16:13:13.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.528+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff83801b070 con 0x7ff8481480c0 2023-12-19T16:13:13.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.528+0000 7ff83ffff700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 0x7ff828062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.528+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7ff838019070 con 0x7ff8481480c0 2023-12-19T16:13:13.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.532+0000 7ff847fff700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 0x7ff828062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.532+0000 7ff847fff700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 0x7ff828062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8400094d0 tx=0x7ff84000b040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.540 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.536+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff83804cc10 con 0x7ff8481480c0 2023-12-19T16:13:13.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.720+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ff8480b0c00 con 0x7ff8481480c0 2023-12-19T16:13:13.730 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.728+0000 7ff83ffff700 1 -- 172.21.15.50:0/1319430266 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v68) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff838050230 con 0x7ff8481480c0 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 msgr2=0x7ff828062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 0x7ff828062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8400094d0 tx=0x7ff84000b040 comp rx=0 tx=0).stop 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 msgr2=0x7ff848148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7ff8380073c0 tx=0x7ff838001df0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 shutdown_connections 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff828060670 0x7ff828062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8480c57f0 0x7ff8480c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8481480c0 0x7ff848148770 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 --2- 172.21.15.50:0/1319430266 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff8340051c0 0x7ff848158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 >> 172.21.15.50:0/1319430266 conn(0x7ff848067bb0 msgr2=0x7ff84815c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 shutdown_connections 2023-12-19T16:13:13.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.732+0000 7ff84e2cd700 1 -- 172.21.15.50:0/1319430266 wait complete. 2023-12-19T16:13:13.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T16:13:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.916+0000 7fe328078700 1 Processor -- start 2023-12-19T16:13:13.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.916+0000 7fe328078700 1 -- start start 2023-12-19T16:13:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe328078700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32015abd0 0x7fe32015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe328078700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3200b2470 0x7fe3200cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe328078700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32006d270 con 0x7fe32015b4e0 2023-12-19T16:13:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe328078700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32015b960 con 0x7fe3200b2470 2023-12-19T16:13:13.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe328078700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32006d550 con 0x7fe32015abd0 2023-12-19T16:13:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3200b2470 0x7fe3200cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe3200b2470 0x7fe3200cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56172/0 (socket says 172.21.15.50:56172) 2023-12-19T16:13:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 -- 172.21.15.50:0/940048804 learned_addr learned my addr 172.21.15.50:0/940048804 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:13.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe325e14700 1 --2- 172.21.15.50:0/940048804 >> v2:172.21.15.125:3300/0 conn(0x7fe32015abd0 0x7fe32015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 -- 172.21.15.50:0/940048804 >> v2:172.21.15.125:3300/0 conn(0x7fe32015abd0 msgr2=0x7fe32015afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 --2- 172.21.15.50:0/940048804 >> v2:172.21.15.125:3300/0 conn(0x7fe32015abd0 0x7fe32015afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 -- 172.21.15.50:0/940048804 >> v1:172.21.15.125:6789/0 conn(0x7fe32015b4e0 legacy=0x7fe3201567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:13.925 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 -- 172.21.15.50:0/940048804 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32006d830 con 0x7fe3200b2470 2023-12-19T16:13:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.920+0000 7fe326615700 1 --2- 172.21.15.50:0/940048804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200cc3e0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe32006be00 tx=0x7fe318009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c121b5081364c2e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe324e12700 1 -- 172.21.15.50:0/940048804 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31800e070 con 0x7fe3200b2470 2023-12-19T16:13:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe324e12700 1 -- 172.21.15.50:0/940048804 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31800ba20 con 0x7fe3200b2470 2023-12-19T16:13:13.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe324e12700 1 -- 172.21.15.50:0/940048804 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3180092e0 con 0x7fe3200b2470 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- 172.21.15.50:0/940048804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 msgr2=0x7fe3200cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 --2- 172.21.15.50:0/940048804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200cc3e0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fe32006be00 tx=0x7fe318009840 comp rx=0 tx=0).stop 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- 172.21.15.50:0/940048804 shutdown_connections 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 --2- 172.21.15.50:0/940048804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200cc3e0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 --2- 172.21.15.50:0/940048804 >> v2:172.21.15.125:3300/0 conn(0x7fe32015abd0 0x7fe32015afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- 172.21.15.50:0/940048804 >> 172.21.15.50:0/940048804 conn(0x7fe3200cdef0 msgr2=0x7fe3200ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- 172.21.15.50:0/940048804 shutdown_connections 2023-12-19T16:13:13.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- 172.21.15.50:0/940048804 wait complete. 2023-12-19T16:13:13.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 Processor -- start 2023-12-19T16:13:13.929 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.924+0000 7fe328078700 1 -- start start 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe32015abd0 0x7fe3200ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 0x7fe3200687c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe320169780 con 0x7fe3200b2470 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe320169a40 con 0x7fe32015b4e0 2023-12-19T16:13:13.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe320169d20 con 0x7fe32015abd0 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe325e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 0x7fe3200687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe325e14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200ad7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56182/0 (socket says 172.21.15.50:56182) 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe325e14700 1 -- 172.21.15.50:0/3966404728 learned_addr learned my addr 172.21.15.50:0/3966404728 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe325613700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe32015abd0 0x7fe3200ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 -- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe32015abd0 msgr2=0x7fe3200ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe32015abd0 0x7fe3200ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 -- 172.21.15.50:0/3966404728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 msgr2=0x7fe3200ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 -- 172.21.15.50:0/3966404728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009ae0 con 0x7fe32015b4e0 2023-12-19T16:13:13.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe326615700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 0x7fe3200687c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe318015000 tx=0x7fe318018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318008a80 con 0x7fe32015b4e0 2023-12-19T16:13:13.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318008c00 con 0x7fe32015b4e0 2023-12-19T16:13:13.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318026460 con 0x7fe32015b4e0 2023-12-19T16:13:13.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.928+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32016a000 con 0x7fe32015b4e0 2023-12-19T16:13:13.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.932+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32016a5c0 con 0x7fe32015b4e0 2023-12-19T16:13:13.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.932+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200670e0 con 0x7fe32015b4e0 2023-12-19T16:13:13.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.932+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe318003040 con 0x7fe32015b4e0 2023-12-19T16:13:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.936+0000 7fe315ffb700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 0x7fe304062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:13.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.936+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7fe31807e440 con 0x7fe32015b4e0 2023-12-19T16:13:13.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.936+0000 7fe325e14700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 0x7fe304062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:13.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.936+0000 7fe325e14700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 0x7fe304062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009eb0 tx=0x7fe31c0091a0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:13.944+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe3200670e0 con 0x7fe32015b4e0 2023-12-19T16:13:14.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.124+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe3200670e0 con 0x7fe32015b4e0 2023-12-19T16:13:14.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.132+0000 7fe315ffb700 1 -- 172.21.15.50:0/3966404728 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe3200670e0 con 0x7fe32015b4e0 2023-12-19T16:13:14.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 msgr2=0x7fe304062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 0x7fe304062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009eb0 tx=0x7fe31c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 msgr2=0x7fe3200687c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 0x7fe3200687c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe318015000 tx=0x7fe318018000 comp rx=0 tx=0).stop 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 shutdown_connections 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe304060670 0x7fe304062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32015b4e0 0x7fe3200687c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe32015abd0 0x7fe3200ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 --2- 172.21.15.50:0/3966404728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe3200b2470 0x7fe3200ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 >> 172.21.15.50:0/3966404728 conn(0x7fe3200cdef0 msgr2=0x7fe32014c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.136+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 shutdown_connections 2023-12-19T16:13:14.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.140+0000 7fe328078700 1 -- 172.21.15.50:0/3966404728 wait complete. 2023-12-19T16:13:14.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-12-19T16:13:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 Processor -- start 2023-12-19T16:13:14.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 -- start start 2023-12-19T16:13:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf280c15b0 0x7faf2815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf280c7190 con 0x7faf2815d320 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280aefd0 con 0x7faf280c15b0 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.316+0000 7faf2cc2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280c1b50 con 0x7faf280aeba0 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf280c15b0 0x7faf2815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf2659c700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faf2815d320 0x7faf2806d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:37048/0 (socket says 172.21.15.50:37048) 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf2659c700 1 -- 172.21.15.50:0/1214823656 learned_addr learned my addr 172.21.15.50:0/1214823656 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:14.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf26d9d700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf25d9b700 1 -- 172.21.15.50:0/1214823656 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233566813 0 0) 0x7faf280c7190 con 0x7faf2815d320 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 -- 172.21.15.50:0/1214823656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 msgr2=0x7faf280680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 -- 172.21.15.50:0/1214823656 >> v1:172.21.15.125:6789/0 conn(0x7faf2815d320 legacy=0x7faf2806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 -- 172.21.15.50:0/1214823656 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280c1e30 con 0x7faf280c15b0 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf26d9d700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf1ffff700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815d6f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7faf100098e0 tx=0x7faf100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56833d972a75b3a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:14.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf25d9b700 1 -- 172.21.15.50:0/1214823656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000b640 con 0x7faf280c15b0 2023-12-19T16:13:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf25d9b700 1 -- 172.21.15.50:0/1214823656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000b7a0 con 0x7faf280c15b0 2023-12-19T16:13:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf25d9b700 1 -- 172.21.15.50:0/1214823656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000bb40 con 0x7faf280c15b0 2023-12-19T16:13:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1214823656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 msgr2=0x7faf2815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.320+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815d6f0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7faf100098e0 tx=0x7faf100091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1214823656 shutdown_connections 2023-12-19T16:13:14.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1214823656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815d6f0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1214823656 >> 172.21.15.50:0/1214823656 conn(0x7faf280c8850 msgr2=0x7faf280c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1214823656 shutdown_connections 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1214823656 wait complete. 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 Processor -- start 2023-12-19T16:13:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.324+0000 7faf2cc2e700 1 -- start start 2023-12-19T16:13:14.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 0x7faf2815f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2815e480 con 0x7faf280c15b0 2023-12-19T16:13:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf28151a40 con 0x7faf2815d320 2023-12-19T16:13:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2815d870 con 0x7faf280aeba0 2023-12-19T16:13:14.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56198/0 (socket says 172.21.15.50:56198) 2023-12-19T16:13:14.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 -- 172.21.15.50:0/1005873348 learned_addr learned my addr 172.21.15.50:0/1005873348 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:14.332 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2659c700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf26d9d700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 0x7faf2815f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 -- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 msgr2=0x7faf280c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 -- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 msgr2=0x7faf2815f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 0x7faf2815f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 -- 172.21.15.50:0/1005873348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20009a20 con 0x7faf280c15b0 2023-12-19T16:13:14.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf26d9d700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 0x7faf2815f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:14.334 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1ffff700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7faf2806c260 tx=0x7faf10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10000d10 con 0x7faf280c15b0 2023-12-19T16:13:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000bb20 con 0x7faf280c15b0 2023-12-19T16:13:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf10009420 con 0x7faf280c15b0 2023-12-19T16:13:14.335 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10011650 con 0x7faf280c15b0 2023-12-19T16:13:14.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.328+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2815de30 con 0x7faf280c15b0 2023-12-19T16:13:14.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.332+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2815e5f0 con 0x7faf280c15b0 2023-12-19T16:13:14.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.332+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7faf10011a00 con 0x7faf280c15b0 2023-12-19T16:13:14.337 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.336+0000 7faf1e7fc700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 0x7faf08062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.336+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7faf1007e0c0 con 0x7faf280c15b0 2023-12-19T16:13:14.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.336+0000 7faf2659c700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 0x7faf08062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:14.339 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.336+0000 7faf2659c700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 0x7faf08062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf18007160 tx=0x7faf18007f50 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:14.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.340+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faf1004e3a0 con 0x7faf280c15b0 2023-12-19T16:13:14.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.572+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7faf28071120 con 0x7faf280c15b0 2023-12-19T16:13:14.878 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.872+0000 7faf1e7fc700 1 -- 172.21.15.50:0/1005873348 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3053) v1 ==== 163+0+0 (secure 0 0 0) 0x7faf10011cb0 con 0x7faf280c15b0 2023-12-19T16:13:14.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 msgr2=0x7faf08062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 0x7faf08062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf18007160 tx=0x7faf18007f50 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 msgr2=0x7faf2815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7faf2806c260 tx=0x7faf10017000 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 shutdown_connections 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faf08060670 0x7faf08062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf2815d320 0x7faf2815f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf280c15b0 0x7faf2815e910 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 --2- 172.21.15.50:0/1005873348 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf280aeba0 0x7faf280c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 >> 172.21.15.50:0/1005873348 conn(0x7faf280c8850 msgr2=0x7faf280c9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 shutdown_connections 2023-12-19T16:13:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:14.880+0000 7faf2cc2e700 1 -- 172.21.15.50:0/1005873348 wait complete. 2023-12-19T16:13:14.906 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-12-19T16:13:14.907 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-12-19T16:13:14.907 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-12-19T16:13:14.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.088+0000 7faebcdcc700 1 Processor -- start 2023-12-19T16:13:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.088+0000 7faebcdcc700 1 -- start start 2023-12-19T16:13:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faebcdcc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faeb806a580 0x7faeb806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faebcdcc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb806c7c0 con 0x7faeb80b1410 2023-12-19T16:13:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faebcdcc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb806cd80 con 0x7faeb806d210 2023-12-19T16:13:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faebcdcc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb806af40 con 0x7faeb806a580 2023-12-19T16:13:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb77fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7faeb80b1410 0x7faeb80ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50432/0 (socket says 172.21.15.50:50432) 2023-12-19T16:13:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb77fe700 1 -- 172.21.15.50:0/1600012261 learned_addr learned my addr 172.21.15.50:0/1600012261 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeaeffd700 1 --2- 172.21.15.50:0/1600012261 >> v2:172.21.15.125:3300/0 conn(0x7faeb806a580 0x7faeb806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580277119 0 0) 0x7faeb806c7c0 con 0x7faeb80b1410 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea4003570 con 0x7faeb80b1410 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231042449 0 0) 0x7faeb806cd80 con 0x7faeb806d210 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb806c7c0 con 0x7faeb806d210 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeaeffd700 1 -- 172.21.15.50:0/1600012261 >> v1:172.21.15.125:6789/0 conn(0x7faeb806d210 legacy=0x7faeb806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeaeffd700 1 -- 172.21.15.50:0/1600012261 >> v1:172.21.15.50:6789/0 conn(0x7faeb80b1410 legacy=0x7faeb80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.092+0000 7faeaeffd700 1 -- 172.21.15.50:0/1600012261 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb806b8f0 con 0x7faeb806a580 2023-12-19T16:13:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faeaeffd700 1 --2- 172.21.15.50:0/1600012261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806a580 0x7faeb806aa00 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9405c5d8bdeb9f1b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0011630 con 0x7faeb806a580 2023-12-19T16:13:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea0011790 con 0x7faeb806a580 2023-12-19T16:13:15.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faeb6ffd700 1 -- 172.21.15.50:0/1600012261 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0011b30 con 0x7faeb806a580 2023-12-19T16:13:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 -- 172.21.15.50:0/1600012261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806a580 msgr2=0x7faeb806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 --2- 172.21.15.50:0/1600012261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806a580 0x7faeb806aa00 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7faea00098e0 tx=0x7faea00091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 -- 172.21.15.50:0/1600012261 shutdown_connections 2023-12-19T16:13:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 --2- 172.21.15.50:0/1600012261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806a580 0x7faeb806aa00 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 -- 172.21.15.50:0/1600012261 >> 172.21.15.50:0/1600012261 conn(0x7faeb8067bb0 msgr2=0x7faeb80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 -- 172.21.15.50:0/1600012261 shutdown_connections 2023-12-19T16:13:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.096+0000 7faebcdcc700 1 -- 172.21.15.50:0/1600012261 wait complete. 2023-12-19T16:13:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 Processor -- start 2023-12-19T16:13:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 -- start start 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 0x7faeb8149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 0x7faeb815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb815bfa0 con 0x7faeb80b1410 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb815c3c0 con 0x7faeb806d210 2023-12-19T16:13:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faebcdcc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb815a400 con 0x7faeb806a580 2023-12-19T16:13:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56206/0 (socket says 172.21.15.50:56206) 2023-12-19T16:13:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 -- 172.21.15.50:0/881400502 learned_addr learned my addr 172.21.15.50:0/881400502 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb77fe700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 0x7faeb8149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeaeffd700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 0x7faeb815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 -- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 msgr2=0x7faeb8149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 0x7faeb8149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 -- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 msgr2=0x7faeb815a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 0x7faeb815a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb7fff700 1 -- 172.21.15.50:0/881400502 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0009420 con 0x7faeb80b1410 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeaeffd700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 0x7faeb815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.100+0000 7faeb77fe700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 0x7faeb8149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faeb7fff700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c940 tx=0x7faeb00027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000a7a0 con 0x7faeb80b1410 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb000a920 con 0x7faeb80b1410 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb815b320 con 0x7faeb80b1410 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000ace0 con 0x7faeb80b1410 2023-12-19T16:13:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb815b8e0 con 0x7faeb80b1410 2023-12-19T16:13:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.104+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb804be40 con 0x7faeb80b1410 2023-12-19T16:13:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.108+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7faeb0012630 con 0x7faeb80b1410 2023-12-19T16:13:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.108+0000 7faeaffff700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 0x7fae98062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.108+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7faeb007cfc0 con 0x7faeb80b1410 2023-12-19T16:13:15.113 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.108+0000 7faeb77fe700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 0x7fae98062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.112+0000 7faeb77fe700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 0x7fae98062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faeb80cb100 tx=0x7faea8009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.116+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faeb004d2a0 con 0x7faeb80b1410 2023-12-19T16:13:15.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.296+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeb80b0e50 con 0x7faeb80b1410 2023-12-19T16:13:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faeaffff700 1 -- 172.21.15.50:0/881400502 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+13657 (secure 0 0 0) 0x7faeb0016090 con 0x7faeb80b1410 2023-12-19T16:13:15.302 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:15.302 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":205,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:12.859654+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6835/3963498493":"2023-12-20T16:13:09.368159+0000","172.21.15.50:6834/3963498493":"2023-12-20T16:13:09.368159+0000","172.21.15.125:0/1389374138":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/3068537263":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/886062417":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/3653663279":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/1065219367":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/1157332980":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/625495530":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/1746967300":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/4264135625":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/915278715":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/4113080782":"2023-12-20T16:13:08.827635+0000","172.21.15.125:0/780783709":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/3568404314":"2023-12-20T16:11:54.658801+0000","172.21.15.125:6833/44063":"2023-12-20T16:11:54.658801+0000","172.21.15.125:0/2646692858":"2023-12-20T16:13:08.827635+0000","172.21.15.125:6832/44063":"2023-12-20T16:11:54.658801+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 msgr2=0x7fae98062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 0x7fae98062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faeb80cb100 tx=0x7faea8009040 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 msgr2=0x7faeb815acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c940 tx=0x7faeb00027a0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 shutdown_connections 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fae98060670 0x7fae98062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faeb806d210 0x7faeb815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faeb80b1410 0x7faeb815acf0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 --2- 172.21.15.50:0/881400502 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faeb806a580 0x7faeb8149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 >> 172.21.15.50:0/881400502 conn(0x7faeb8067bb0 msgr2=0x7faeb8156280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 shutdown_connections 2023-12-19T16:13:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.300+0000 7faebcdcc700 1 -- 172.21.15.50:0/881400502 wait complete. 2023-12-19T16:13:15.325 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T16:13:15.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:13:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.488+0000 7fcd98c8e700 1 Processor -- start 2023-12-19T16:13:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.488+0000 7fcd98c8e700 1 -- start start 2023-12-19T16:13:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd98c8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd94158c90 0x7fcd940caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd98c8e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 0x7fcd9406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd98c8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9406c850 con 0x7fcd9406aeb0 2023-12-19T16:13:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd98c8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9406b470 con 0x7fcd94158c90 2023-12-19T16:13:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd98c8e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9406b750 con 0x7fcd94159770 2023-12-19T16:13:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd94158c90 0x7fcd940caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd94158c90 0x7fcd940caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56212/0 (socket says 172.21.15.50:56212) 2023-12-19T16:13:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 -- 172.21.15.50:0/1927484142 learned_addr learned my addr 172.21.15.50:0/1927484142 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd93fff700 1 --2- 172.21.15.50:0/1927484142 >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 0x7fcd9406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 -- 172.21.15.50:0/1927484142 >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 msgr2=0x7fcd9406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 --2- 172.21.15.50:0/1927484142 >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 0x7fcd9406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 -- 172.21.15.50:0/1927484142 >> v1:172.21.15.125:6789/0 conn(0x7fcd9406aeb0 legacy=0x7fcd94159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd937fe700 1 -- 172.21.15.50:0/1927484142 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9406ba30 con 0x7fcd94158c90 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.492+0000 7fcd93fff700 1 --2- 172.21.15.50:0/1927484142 >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 0x7fcd9406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd937fe700 1 --2- 172.21.15.50:0/1927484142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd94158c90 0x7fcd940caf40 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fcd940cc7f0 tx=0x7fcd84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3425070e0550ecd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd927fc700 1 -- 172.21.15.50:0/1927484142 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400b820 con 0x7fcd94158c90 2023-12-19T16:13:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd927fc700 1 -- 172.21.15.50:0/1927484142 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8400b980 con 0x7fcd94158c90 2023-12-19T16:13:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd927fc700 1 -- 172.21.15.50:0/1927484142 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400bd20 con 0x7fcd94158c90 2023-12-19T16:13:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 -- 172.21.15.50:0/1927484142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd94158c90 msgr2=0x7fcd940caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/1927484142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd94158c90 0x7fcd940caf40 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fcd940cc7f0 tx=0x7fcd84009250 comp rx=0 tx=0).stop 2023-12-19T16:13:15.499 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 -- 172.21.15.50:0/1927484142 shutdown_connections 2023-12-19T16:13:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/1927484142 >> v2:172.21.15.125:3300/0 conn(0x7fcd94159770 0x7fcd9406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/1927484142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd94158c90 0x7fcd940caf40 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 -- 172.21.15.50:0/1927484142 >> 172.21.15.50:0/1927484142 conn(0x7fcd94067bb0 msgr2=0x7fcd940cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 -- 172.21.15.50:0/1927484142 shutdown_connections 2023-12-19T16:13:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.496+0000 7fcd98c8e700 1 -- 172.21.15.50:0/1927484142 wait complete. 2023-12-19T16:13:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 Processor -- start 2023-12-19T16:13:15.502 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 -- start start 2023-12-19T16:13:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd94158c90 0x7fcd940c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd94159770 0x7fcd940c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940c5790 con 0x7fcd9406aeb0 2023-12-19T16:13:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940c4e60 con 0x7fcd94159770 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd98c8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940c5140 con 0x7fcd94158c90 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56224/0 (socket says 172.21.15.50:56224) 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd937fe700 1 -- 172.21.15.50:0/3405208893 learned_addr learned my addr 172.21.15.50:0/3405208893 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd93fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd94159770 0x7fcd940c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.505 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.500+0000 7fcd92ffd700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd94158c90 0x7fcd940c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 -- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd94158c90 msgr2=0x7fcd940c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd94158c90 0x7fcd940c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 -- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd94159770 msgr2=0x7fcd940c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd94159770 0x7fcd940c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 -- 172.21.15.50:0/3405208893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840094f0 con 0x7fcd9406aeb0 2023-12-19T16:13:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd937fe700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fcd840030c0 tx=0x7fcd84016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84011d00 con 0x7fcd9406aeb0 2023-12-19T16:13:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84020070 con 0x7fcd9406aeb0 2023-12-19T16:13:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd940c5420 con 0x7fcd9406aeb0 2023-12-19T16:13:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8401b6a0 con 0x7fcd9406aeb0 2023-12-19T16:13:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940c5c20 con 0x7fcd9406aeb0 2023-12-19T16:13:15.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.504+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404be40 con 0x7fcd9406aeb0 2023-12-19T16:13:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.508+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcd84022030 con 0x7fcd9406aeb0 2023-12-19T16:13:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.508+0000 7fcd7b7fe700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 0x7fcd80062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.508+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7fcd8407e970 con 0x7fcd9406aeb0 2023-12-19T16:13:15.512 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.508+0000 7fcd92ffd700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 0x7fcd80062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.512+0000 7fcd92ffd700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 0x7fcd80062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ce0e0 tx=0x7fcd88007ea0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.516+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcd8404ec50 con 0x7fcd9406aeb0 2023-12-19T16:13:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.692+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd9404be40 con 0x7fcd9406aeb0 2023-12-19T16:13:15.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.692+0000 7fcd7b7fe700 1 -- 172.21.15.50:0/3405208893 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd84011e60 con 0x7fcd9406aeb0 2023-12-19T16:13:15.698 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:13:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 msgr2=0x7fcd80062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 0x7fcd80062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ce0e0 tx=0x7fcd88007ea0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 msgr2=0x7fcd94156370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fcd840030c0 tx=0x7fcd84016000 comp rx=0 tx=0).stop 2023-12-19T16:13:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 shutdown_connections 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcd80060670 0x7fcd80062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd94159770 0x7fcd940c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcd94158c90 0x7fcd940c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 --2- 172.21.15.50:0/3405208893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd9406aeb0 0x7fcd94156370 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 >> 172.21.15.50:0/3405208893 conn(0x7fcd94067bb0 msgr2=0x7fcd940cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 shutdown_connections 2023-12-19T16:13:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.700+0000 7fcd98c8e700 1 -- 172.21.15.50:0/3405208893 wait complete. 2023-12-19T16:13:15.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-12-19T16:13:15.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.892+0000 7f65f7fff700 1 Processor -- start 2023-12-19T16:13:15.896 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.892+0000 7f65f7fff700 1 -- start start 2023-12-19T16:13:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8158e20 0x7f65f806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f81591f0 0x7f65f80ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 0x7f65f80ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f806b730 con 0x7f65f8158e20 2023-12-19T16:13:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f806ab20 con 0x7f65f815ca80 2023-12-19T16:13:15.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f7fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f806ae00 con 0x7f65f81591f0 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8158e20 0x7f65f806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65f8158e20 0x7f65f806a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56240/0 (socket says 172.21.15.50:56240) 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 -- 172.21.15.50:0/969625913 learned_addr learned my addr 172.21.15.50:0/969625913 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f67fc700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f81591f0 0x7f65f80ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f77fe700 1 --2- 172.21.15.50:0/969625913 >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 0x7f65f80ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 -- 172.21.15.50:0/969625913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f81591f0 msgr2=0x7f65f80ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f81591f0 0x7f65f80ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 -- 172.21.15.50:0/969625913 >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 msgr2=0x7f65f80ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/969625913 >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 0x7f65f80ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 -- 172.21.15.50:0/969625913 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f806b0e0 con 0x7f65f8158e20 2023-12-19T16:13:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f77fe700 1 --2- 172.21.15.50:0/969625913 >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 0x7f65f80ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.896+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f806a540 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f65f80b2ab0 tx=0x7f65e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771eecd6c2243d95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f5ffb700 1 -- 172.21.15.50:0/969625913 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8009e70 con 0x7f65f8158e20 2023-12-19T16:13:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f5ffb700 1 -- 172.21.15.50:0/969625913 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e800b860 con 0x7f65f8158e20 2023-12-19T16:13:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f5ffb700 1 -- 172.21.15.50:0/969625913 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e800bc00 con 0x7f65f8158e20 2023-12-19T16:13:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 -- 172.21.15.50:0/969625913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 msgr2=0x7f65f806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f806a540 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f65f80b2ab0 tx=0x7f65e8009250 comp rx=0 tx=0).stop 2023-12-19T16:13:15.903 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 -- 172.21.15.50:0/969625913 shutdown_connections 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 --2- 172.21.15.50:0/969625913 >> v2:172.21.15.125:3300/0 conn(0x7f65f815ca80 0x7f65f80ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f81591f0 0x7f65f80ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 --2- 172.21.15.50:0/969625913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f806a540 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 -- 172.21.15.50:0/969625913 >> 172.21.15.50:0/969625913 conn(0x7f65f8067bb0 msgr2=0x7f65f80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 -- 172.21.15.50:0/969625913 shutdown_connections 2023-12-19T16:13:15.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.900+0000 7f65f7fff700 1 -- 172.21.15.50:0/969625913 wait complete. 2023-12-19T16:13:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 Processor -- start 2023-12-19T16:13:15.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 -- start start 2023-12-19T16:13:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65f81591f0 0x7f65f816a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 0x7f65f815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f815a6c0 con 0x7f65f8158e20 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f814a400 con 0x7f65f81591f0 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f7fff700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f8159e20 con 0x7f65f815ca80 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56252/0 (socket says 172.21.15.50:56252) 2023-12-19T16:13:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 -- 172.21.15.50:0/4058330948 learned_addr learned my addr 172.21.15.50:0/4058330948 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f77fe700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 0x7f65f815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 -- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 msgr2=0x7f65f815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 0x7f65f815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 -- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65f81591f0 msgr2=0x7f65f816a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65f81591f0 0x7f65f816a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f6ffd700 1 -- 172.21.15.50:0/4058330948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e80094f0 con 0x7f65f8158e20 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.904+0000 7f65f77fe700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 0x7f65f815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65f6ffd700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009290 tx=0x7f65e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e800bbe0 con 0x7f65f8158e20 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e801f070 con 0x7f65f8158e20 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e8026690 con 0x7f65f8158e20 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f815a100 con 0x7f65f8158e20 2023-12-19T16:13:15.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f81595f0 con 0x7f65f8158e20 2023-12-19T16:13:15.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f804be40 con 0x7f65f8158e20 2023-12-19T16:13:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.908+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f65e801a430 con 0x7f65f8158e20 2023-12-19T16:13:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.912+0000 7f65deffd700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 0x7f65e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:15.914 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.912+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7f65e8021070 con 0x7f65f8158e20 2023-12-19T16:13:15.915 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.912+0000 7f65f67fc700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 0x7f65e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.912+0000 7f65f67fc700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 0x7f65e4062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f80cc710 tx=0x7f65ec0091a0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:15.916+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f65e804ec10 con 0x7f65f8158e20 2023-12-19T16:13:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.156+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f65f814a1d0 con 0x7f65f8158e20 2023-12-19T16:13:16.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.892+0000 7f65deffd700 1 -- 172.21.15.50:0/4058330948 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3055) v1 ==== 158+0+0 (secure 0 0 0) 0x7f65e80150e0 con 0x7f65f8158e20 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 msgr2=0x7f65e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 0x7f65e4062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f65f80cc710 tx=0x7f65ec0091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 msgr2=0x7f65f8149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009290 tx=0x7f65e8017000 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 shutdown_connections 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f65e4060670 0x7f65e4062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65f81591f0 0x7f65f816a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65f815ca80 0x7f65f815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 --2- 172.21.15.50:0/4058330948 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65f8158e20 0x7f65f8149a60 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 >> 172.21.15.50:0/4058330948 conn(0x7f65f8067bb0 msgr2=0x7f65f80c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:16.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 shutdown_connections 2023-12-19T16:13:16.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:16.896+0000 7f65f7fff700 1 -- 172.21.15.50:0/4058330948 wait complete. 2023-12-19T16:13:16.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T16:13:17.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 Processor -- start 2023-12-19T16:13:17.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 -- start start 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6028158e20 0x7f60280ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60280cb970 0x7f60280ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60280cbdf0 con 0x7f60280ca2c0 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60280cc0d0 con 0x7f60280cb970 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602eecd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60280cc3b0 con 0x7f6028158e20 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60280cb970 0x7f60280ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f60280cb970 0x7f60280ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46948/0 (socket says 172.21.15.50:46948) 2023-12-19T16:13:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 -- 172.21.15.50:0/682091377 learned_addr learned my addr 172.21.15.50:0/682091377 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:17.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602d46a700 1 --1- 172.21.15.50:0/682091377 >> v1:172.21.15.50:6789/0 conn(0x7f60280ca2c0 0x7f602806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57082/0 (socket says 172.21.15.50:57082) 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f602cc69700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6028158e20 0x7f60280ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 -- 172.21.15.50:0/682091377 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6028158e20 msgr2=0x7f60280ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6028158e20 0x7f60280ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 -- 172.21.15.50:0/682091377 >> v1:172.21.15.50:6789/0 conn(0x7f60280ca2c0 legacy=0x7f602806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 -- 172.21.15.50:0/682091377 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280cc690 con 0x7f60280cb970 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.088+0000 7f6027fff700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280cb970 0x7f60280ceb00 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f60140099b0 tx=0x7f6014009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e0da58aabd2286 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f60277fe700 1 -- 172.21.15.50:0/682091377 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400b860 con 0x7f60280cb970 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f60277fe700 1 -- 172.21.15.50:0/682091377 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601400b9c0 con 0x7f60280cb970 2023-12-19T16:13:17.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f60277fe700 1 -- 172.21.15.50:0/682091377 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400bd60 con 0x7f60280cb970 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- 172.21.15.50:0/682091377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280cb970 msgr2=0x7f60280ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280cb970 0x7f60280ceb00 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f60140099b0 tx=0x7f6014009250 comp rx=0 tx=0).stop 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- 172.21.15.50:0/682091377 shutdown_connections 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280cb970 0x7f60280ceb00 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 --2- 172.21.15.50:0/682091377 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6028158e20 0x7f60280ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- 172.21.15.50:0/682091377 >> 172.21.15.50:0/682091377 conn(0x7f6028067bb0 msgr2=0x7f60280d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- 172.21.15.50:0/682091377 shutdown_connections 2023-12-19T16:13:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- 172.21.15.50:0/682091377 wait complete. 2023-12-19T16:13:17.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 Processor -- start 2023-12-19T16:13:17.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.092+0000 7f602eecd700 1 -- start start 2023-12-19T16:13:17.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 0x7f602814c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60280c9870 0x7f6028154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602814e900 con 0x7f60280ca2c0 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602814dcf0 con 0x7f6028158e20 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602814dfd0 con 0x7f60280c9870 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46952/0 (socket says 172.21.15.50:46952) 2023-12-19T16:13:17.100 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 -- 172.21.15.50:0/3997905217 learned_addr learned my addr 172.21.15.50:0/3997905217 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:17.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f6027fff700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 0x7f602814c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.101 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602d46a700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60280c9870 0x7f6028154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 -- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60280c9870 msgr2=0x7f6028154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60280c9870 0x7f6028154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 -- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 msgr2=0x7f602814c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 0x7f602814c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 -- 172.21.15.50:0/3997905217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60140094f0 con 0x7f60280ca2c0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f6027fff700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 0x7f602814c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602cc69700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f601c00d860 tx=0x7f601c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c009610 con 0x7f60280ca2c0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c009770 con 0x7f60280ca2c0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602814e2b0 con 0x7f60280ca2c0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.096+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c009b30 con 0x7f60280ca2c0 2023-12-19T16:13:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.100+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602814d8e0 con 0x7f60280ca2c0 2023-12-19T16:13:17.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.100+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6028071120 con 0x7f60280ca2c0 2023-12-19T16:13:17.104 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.100+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f601c01e460 con 0x7f60280ca2c0 2023-12-19T16:13:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.104+0000 7f6024ff9700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 0x7f600c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.104+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7f601c07d160 con 0x7f60280ca2c0 2023-12-19T16:13:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.104+0000 7f6027fff700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 0x7f600c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.104+0000 7f6027fff700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 0x7f600c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f60140099b0 tx=0x7f6014017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.108+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f601c04d440 con 0x7f60280ca2c0 2023-12-19T16:13:17.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.284+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f602806a7a0 con 0x7f60280ca2c0 2023-12-19T16:13:17.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.292+0000 7f6024ff9700 1 -- 172.21.15.50:0/3997905217 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f601c050a60 con 0x7f60280ca2c0 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 msgr2=0x7f600c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 0x7f600c062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f60140099b0 tx=0x7f6014017000 comp rx=0 tx=0).stop 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 msgr2=0x7f602814d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f601c00d860 tx=0x7f601c008040 comp rx=0 tx=0).stop 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 shutdown_connections 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f600c060670 0x7f600c062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6028158e20 0x7f602814c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f60280c9870 0x7f6028154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 --2- 172.21.15.50:0/3997905217 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f60280ca2c0 0x7f602814d3a0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 >> 172.21.15.50:0/3997905217 conn(0x7f6028067bb0 msgr2=0x7f6028156260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 shutdown_connections 2023-12-19T16:13:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.296+0000 7f602eecd700 1 -- 172.21.15.50:0/3997905217 wait complete. 2023-12-19T16:13:17.320 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T16:13:17.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:17.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:17.409 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T16:13:17.409 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T16:13:17.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:17.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 Processor -- start 2023-12-19T16:13:17.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 -- start start 2023-12-19T16:13:17.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1800140710 0x7f18000c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180006e150 con 0x7f1800140ae0 2023-12-19T16:13:17.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180006d540 con 0x7f1800068800 2023-12-19T16:13:17.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f180722e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180006d820 con 0x7f1800140710 2023-12-19T16:13:17.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.580+0000 7f18057cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f18057cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46958/0 (socket says 172.21.15.50:46958) 2023-12-19T16:13:17.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f18057cb700 1 -- 172.21.15.50:0/3011426608 learned_addr learned my addr 172.21.15.50:0/3011426608 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:17.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 --2- 172.21.15.50:0/3011426608 >> v2:172.21.15.125:3300/0 conn(0x7f1800140710 0x7f18000c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 -- 172.21.15.50:0/3011426608 >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 msgr2=0x7f18000ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:17.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 --2- 172.21.15.50:0/3011426608 >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 -- 172.21.15.50:0/3011426608 >> v1:172.21.15.50:6789/0 conn(0x7f1800140ae0 legacy=0x7f18000680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 -- 172.21.15.50:0/3011426608 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180006db00 con 0x7f1800140710 2023-12-19T16:13:17.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f18057cb700 1 --2- 172.21.15.50:0/3011426608 >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:17.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f1804fca700 1 --2- 172.21.15.50:0/3011426608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000c70a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1800068c30 tx=0x7f17f40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d40e71c83069e777 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f17ff7fe700 1 -- 172.21.15.50:0/3011426608 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4011630 con 0x7f1800140710 2023-12-19T16:13:17.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f17ff7fe700 1 -- 172.21.15.50:0/3011426608 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f4011790 con 0x7f1800140710 2023-12-19T16:13:17.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f17ff7fe700 1 -- 172.21.15.50:0/3011426608 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4011b30 con 0x7f1800140710 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 -- 172.21.15.50:0/3011426608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 msgr2=0x7f18000c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 --2- 172.21.15.50:0/3011426608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000c70a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1800068c30 tx=0x7f17f40091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 -- 172.21.15.50:0/3011426608 shutdown_connections 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 --2- 172.21.15.50:0/3011426608 >> v2:172.21.15.50:3300/0 conn(0x7f1800068800 0x7f18000ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 --2- 172.21.15.50:0/3011426608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000c70a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 -- 172.21.15.50:0/3011426608 >> 172.21.15.50:0/3011426608 conn(0x7f18000c8850 msgr2=0x7f18000c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 -- 172.21.15.50:0/3011426608 shutdown_connections 2023-12-19T16:13:17.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.584+0000 7f180722e700 1 -- 172.21.15.50:0/3011426608 wait complete. 2023-12-19T16:13:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.588+0000 7f180722e700 1 Processor -- start 2023-12-19T16:13:17.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.588+0000 7f180722e700 1 -- start start 2023-12-19T16:13:17.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1800140ae0 0x7f18000a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180013e470 con 0x7f1800068800 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180013d860 con 0x7f1800140710 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180013db40 con 0x7f1800140ae0 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46966/0 (socket says 172.21.15.50:46966) 2023-12-19T16:13:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 -- 172.21.15.50:0/1773953605 learned_addr learned my addr 172.21.15.50:0/1773953605 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f18057cb700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1800140ae0 0x7f18000a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f17fffff700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 -- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1800140ae0 msgr2=0x7f18000a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1800140ae0 0x7f18000a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 -- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 msgr2=0x7f18000a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000a9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 -- 172.21.15.50:0/1773953605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f8009ae0 con 0x7f1800068800 2023-12-19T16:13:17.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f1804fca700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f180006d960 tx=0x7f17f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4011b10 con 0x7f1800068800 2023-12-19T16:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f40253f0 con 0x7f1800068800 2023-12-19T16:13:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f4009420 con 0x7f1800068800 2023-12-19T16:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4025790 con 0x7f1800068800 2023-12-19T16:13:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.592+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180013e100 con 0x7f1800068800 2023-12-19T16:13:17.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.596+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18000b0bb0 con 0x7f1800068800 2023-12-19T16:13:17.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.596+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f17f4025b80 con 0x7f1800068800 2023-12-19T16:13:17.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.600+0000 7f17fcff9700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 0x7f17e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:17.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.600+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7f17f407e410 con 0x7f1800068800 2023-12-19T16:13:17.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.600+0000 7f17fffff700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 0x7f17e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:17.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.600+0000 7f17fffff700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 0x7f17e4062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1800157970 tx=0x7f17ec009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:17.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.608+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18000b0bb0 con 0x7f1800068800 2023-12-19T16:13:17.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.812+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18000b0bb0 con 0x7f1800068800 2023-12-19T16:13:17.817 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.812+0000 7f17fcff9700 1 -- 172.21.15.50:0/1773953605 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+3533 (secure 0 0 0) 0x7f17f4015de0 con 0x7f1800068800 2023-12-19T16:13:17.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:17.817 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":92,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1224616302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1224616302},{"type":"v1","addr":"172.21.15.125:6836","nonce":1224616302}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3029091751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":3029091751},{"type":"v1","addr":"172.21.15.125:6837","nonce":3029091751}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":9876,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/718687286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":718687286},{"type":"v1","addr":"172.21.15.50:6833","nonce":718687286}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85},{"gid":10284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/3079022061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":3079022061},{"type":"v1","addr":"172.21.15.50:6835","nonce":3079022061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[]} 2023-12-19T16:13:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 msgr2=0x7f17e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 0x7f17e4062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1800157970 tx=0x7f17ec009040 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 msgr2=0x7f180013cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f180006d960 tx=0x7f17f4017000 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 shutdown_connections 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f17e4060670 0x7f17e4062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1800140710 0x7f18000a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1800140ae0 0x7f18000a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 --2- 172.21.15.50:0/1773953605 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1800068800 0x7f180013cf80 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 >> 172.21.15.50:0/1773953605 conn(0x7f18000c8850 msgr2=0x7f18000c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 shutdown_connections 2023-12-19T16:13:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:17.816+0000 7f180722e700 1 -- 172.21.15.50:0/1773953605 wait complete. 2023-12-19T16:13:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 92 2023-12-19T16:13:17.842 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T16:13:17.842 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T16:13:17.842 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T16:13:17.843 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:13:17.843 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T16:13:17.843 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T16:13:17.843 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T16:13:17.844 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:13:17.844 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T16:13:17.844 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T16:13:17.844 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T16:13:17.844 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:13:17.845 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T16:13:17.845 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T16:13:17.845 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T16:13:17.845 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:13:17.863 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T16:13:17.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T16:13:17.864 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T16:13:17.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T16:13:17.866 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T16:13:17.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T16:13:17.867 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T16:13:17.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T16:13:17.914 INFO:tasks.ceph.mds.b:Started 2023-12-19T16:13:17.914 DEBUG:teuthology.parallel:result is None 2023-12-19T16:13:17.916 INFO:tasks.ceph.mds.c:Started 2023-12-19T16:13:17.916 DEBUG:teuthology.parallel:result is None 2023-12-19T16:13:17.919 INFO:tasks.ceph.mds.d:Started 2023-12-19T16:13:17.919 INFO:tasks.ceph.mds.a:Started 2023-12-19T16:13:17.920 DEBUG:teuthology.parallel:result is None 2023-12-19T16:13:17.920 DEBUG:teuthology.parallel:result is None 2023-12-19T16:13:17.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T16:13:18.029 INFO:tasks.ceph.mds.a.smithi050.stdout:starting mds.a at 2023-12-19T16:13:18.029 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T16:13:18.031 INFO:tasks.ceph.mds.b.smithi125.stdout:starting mds.b at 2023-12-19T16:13:18.032 INFO:tasks.ceph.mds.d.smithi125.stdout:starting mds.d at 2023-12-19T16:13:18.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.076+0000 7fc0e7945700 1 Processor -- start 2023-12-19T16:13:18.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.076+0000 7fc0e7945700 1 -- start start 2023-12-19T16:13:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e7945700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0e0158cc0 0x7fc0e006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e7945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e00b3870 0x7fc0e00b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e7945700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e006b0d0 con 0x7fc0e006dc00 2023-12-19T16:13:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e7945700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e00b4790 con 0x7fc0e0158cc0 2023-12-19T16:13:18.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e7945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e006d710 con 0x7fc0e00b3870 2023-12-19T16:13:18.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0e0158cc0 0x7fc0e006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc0e0158cc0 0x7fc0e006d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47004/0 (socket says 172.21.15.50:47004) 2023-12-19T16:13:18.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 -- 172.21.15.50:0/23052236 learned_addr learned my addr 172.21.15.50:0/23052236 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:18.083 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e5ee2700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e00b3870 0x7fc0e00b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 -- 172.21.15.50:0/23052236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e00b3870 msgr2=0x7fc0e00b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e00b3870 0x7fc0e00b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 -- 172.21.15.50:0/23052236 >> v1:172.21.15.125:6789/0 conn(0x7fc0e006dc00 legacy=0x7fc0e006b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 -- 172.21.15.50:0/23052236 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e006a7e0 con 0x7fc0e0158cc0 2023-12-19T16:13:18.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0e56e1700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0158cc0 0x7fc0e006d1d0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0099b0 tx=0x7fc0dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37446855fb2ae6d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:18.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0d7fff700 1 -- 172.21.15.50:0/23052236 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc009ea0 con 0x7fc0e0158cc0 2023-12-19T16:13:18.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0d7fff700 1 -- 172.21.15.50:0/23052236 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00b860 con 0x7fc0e0158cc0 2023-12-19T16:13:18.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.080+0000 7fc0d7fff700 1 -- 172.21.15.50:0/23052236 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00bc00 con 0x7fc0e0158cc0 2023-12-19T16:13:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- 172.21.15.50:0/23052236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0158cc0 msgr2=0x7fc0e006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0158cc0 0x7fc0e006d1d0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0099b0 tx=0x7fc0dc009250 comp rx=0 tx=0).stop 2023-12-19T16:13:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- 172.21.15.50:0/23052236 shutdown_connections 2023-12-19T16:13:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e00b3870 0x7fc0e00b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 --2- 172.21.15.50:0/23052236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e0158cc0 0x7fc0e006d1d0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- 172.21.15.50:0/23052236 >> 172.21.15.50:0/23052236 conn(0x7fc0e0067bb0 msgr2=0x7fc0e00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:18.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- 172.21.15.50:0/23052236 shutdown_connections 2023-12-19T16:13:18.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- 172.21.15.50:0/23052236 wait complete. 2023-12-19T16:13:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 Processor -- start 2023-12-19T16:13:18.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.084+0000 7fc0e7945700 1 -- start start 2023-12-19T16:13:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0e00b3870 0x7fc0e01545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e0158cc0 0x7fc0e01570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0168e50 con 0x7fc0e006dc00 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0151b20 con 0x7fc0e00b3870 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0151e00 con 0x7fc0e0158cc0 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47016/0 (socket says 172.21.15.50:47016) 2023-12-19T16:13:18.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 -- 172.21.15.50:0/3784340741 learned_addr learned my addr 172.21.15.50:0/3784340741 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e5ee2700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e0158cc0 0x7fc0e01570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e4ee0700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0e00b3870 0x7fc0e01545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 -- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e0158cc0 msgr2=0x7fc0e01570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e0158cc0 0x7fc0e01570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 -- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0e00b3870 msgr2=0x7fc0e01545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0e00b3870 0x7fc0e01545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 -- 172.21.15.50:0/3784340741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0094f0 con 0x7fc0e006dc00 2023-12-19T16:13:18.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e56e1700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fc0e006eb60 tx=0x7fc0dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc000d10 con 0x7fc0e006dc00 2023-12-19T16:13:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00bbe0 con 0x7fc0e006dc00 2023-12-19T16:13:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc011670 con 0x7fc0e006dc00 2023-12-19T16:13:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e0156aa0 con 0x7fc0e006dc00 2023-12-19T16:13:18.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.088+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e0155f20 con 0x7fc0e006dc00 2023-12-19T16:13:18.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.092+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e0071120 con 0x7fc0e006dc00 2023-12-19T16:13:18.096 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.092+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc0dc011aa0 con 0x7fc0e006dc00 2023-12-19T16:13:18.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.092+0000 7fc0d57fa700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 0x7fc0c4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:18.097 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.092+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6059+0+0 (secure 0 0 0) 0x7fc0dc07dfa0 con 0x7fc0e006dc00 2023-12-19T16:13:18.098 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.096+0000 7fc0e4ee0700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 0x7fc0c4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:18.099 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.096+0000 7fc0e4ee0700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 0x7fc0c4062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00aba80 tx=0x7fc0d0007ec0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:18.103 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.100+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc0dc04e300 con 0x7fc0e006dc00 2023-12-19T16:13:18.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.288+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc0e00008d0 con 0x7fc0e006dc00 2023-12-19T16:13:18.923 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.920+0000 7fc0d57fa700 1 -- 172.21.15.50:0/3784340741 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v207) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc0dc051920 con 0x7fc0e006dc00 2023-12-19T16:13:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 msgr2=0x7fc0c4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 0x7fc0c4062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00aba80 tx=0x7fc0d0007ec0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 msgr2=0x7fc0e0151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:18.927 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fc0e006eb60 tx=0x7fc0dc017000 comp rx=0 tx=0).stop 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 shutdown_connections 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc0c4060670 0x7fc0c4062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc0e00b3870 0x7fc0e01545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc0e0158cc0 0x7fc0e01570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 --2- 172.21.15.50:0/3784340741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc0e006dc00 0x7fc0e0151330 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 >> 172.21.15.50:0/3784340741 conn(0x7fc0e0067bb0 msgr2=0x7fc0e00aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 shutdown_connections 2023-12-19T16:13:18.928 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:18.924+0000 7fc0e7945700 1 -- 172.21.15.50:0/3784340741 wait complete. 2023-12-19T16:13:18.930 INFO:teuthology.orchestra.run.smithi050.stderr: removed all blocklist entries 2023-12-19T16:13:18.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T16:13:19.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.116+0000 7f422b59e700 1 Processor -- start 2023-12-19T16:13:19.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.116+0000 7f422b59e700 1 -- start start 2023-12-19T16:13:19.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422b59e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f422c06b7a0 0x7f422c0c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422b59e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c0c5610 con 0x7f422c0c51b0 2023-12-19T16:13:19.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422b59e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c06c7c0 con 0x7f422c06b7a0 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c06caa0 con 0x7f422c158d60 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f422c06b7a0 0x7f422c0c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f422c06b7a0 0x7f422c0c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47026/0 (socket says 172.21.15.50:47026) 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 -- 172.21.15.50:0/3244477617 learned_addr learned my addr 172.21.15.50:0/3244477617 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f422a59c700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 -- 172.21.15.50:0/3244477617 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 msgr2=0x7f422c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 -- 172.21.15.50:0/3244477617 >> v1:172.21.15.125:6789/0 conn(0x7f422c0c51b0 legacy=0x7f422c06a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 -- 172.21.15.50:0/3244477617 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c15ce60 con 0x7f422c06b7a0 2023-12-19T16:13:19.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4223fff700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c0c4c70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f42140099b0 tx=0x7f4214009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e188a2b81cdfca9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4229d9b700 1 -- 172.21.15.50:0/3244477617 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421400b860 con 0x7f422c06b7a0 2023-12-19T16:13:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4229d9b700 1 -- 172.21.15.50:0/3244477617 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421400b9c0 con 0x7f422c06b7a0 2023-12-19T16:13:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.120+0000 7f4229d9b700 1 -- 172.21.15.50:0/3244477617 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421400bd80 con 0x7f422c06b7a0 2023-12-19T16:13:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 -- 172.21.15.50:0/3244477617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 msgr2=0x7f422c0c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c0c4c70 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f42140099b0 tx=0x7f4214009250 comp rx=0 tx=0).stop 2023-12-19T16:13:19.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 -- 172.21.15.50:0/3244477617 shutdown_connections 2023-12-19T16:13:19.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c0c4c70 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 --2- 172.21.15.50:0/3244477617 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 -- 172.21.15.50:0/3244477617 >> 172.21.15.50:0/3244477617 conn(0x7f422c067bb0 msgr2=0x7f422c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:19.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 -- 172.21.15.50:0/3244477617 shutdown_connections 2023-12-19T16:13:19.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.124+0000 7f422b59e700 1 -- 172.21.15.50:0/3244477617 wait complete. 2023-12-19T16:13:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 Processor -- start 2023-12-19T16:13:19.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 -- start start 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f422c0c51b0 0x7f422c152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c155200 con 0x7f422c06b7a0 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c156880 con 0x7f422c0c51b0 2023-12-19T16:13:19.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422b59e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c16d5d0 con 0x7f422c158d60 2023-12-19T16:13:19.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47032/0 (socket says 172.21.15.50:47032) 2023-12-19T16:13:19.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 -- 172.21.15.50:0/3370226709 learned_addr learned my addr 172.21.15.50:0/3370226709 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f4223fff700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f422c0c51b0 0x7f422c152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422ad9d700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 -- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 msgr2=0x7f422c157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 -- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f422c0c51b0 msgr2=0x7f422c152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f422c0c51b0 0x7f422c152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422a59c700 1 -- 172.21.15.50:0/3370226709 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140094f0 con 0x7f422c06b7a0 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.128+0000 7f422ad9d700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:19.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f422a59c700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d860 tx=0x7f421c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0125f0 con 0x7f422c06b7a0 2023-12-19T16:13:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c012750 con 0x7f422c06b7a0 2023-12-19T16:13:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c012b10 con 0x7f422c06b7a0 2023-12-19T16:13:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c152ab0 con 0x7f422c06b7a0 2023-12-19T16:13:19.135 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c155f50 con 0x7f422c06b7a0 2023-12-19T16:13:19.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.132+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c04be40 con 0x7f422c06b7a0 2023-12-19T16:13:19.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.136+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f421c01e460 con 0x7f422c06b7a0 2023-12-19T16:13:19.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.136+0000 7f42227fc700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 0x7f420c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.136+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5285+0+0 (secure 0 0 0) 0x7f421c07cdc0 con 0x7f422c06b7a0 2023-12-19T16:13:19.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.136+0000 7f4223fff700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 0x7f420c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.140+0000 7f4223fff700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 0x7f420c062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42140099b0 tx=0x7f4214017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.144+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f421c04d3a0 con 0x7f422c06b7a0 2023-12-19T16:13:19.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.376+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f422c15d130 con 0x7f422c06b7a0 2023-12-19T16:13:19.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.376+0000 7f42227fc700 1 -- 172.21.15.50:0/3370226709 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f421c0509c0 con 0x7f422c06b7a0 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_dump": [ 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.a", 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlfiF4LBAAXuqcAgNrFc/oDYQNJ6o7hQ==", 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:13:19.380 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.b", 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFl5cvOAhAAYkx+bcpfYmxcQVS75aJJiw==", 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.c", 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlBkabNhAAC0REmTnSnpa9go+wPUNITg==", 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.382 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mds.d", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlJQliDRAAQzaxZE34FBn9+sg+ZDU9RQ==", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.0", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDSqoFl/GAWARAARgr/xHBrgSj1F2vCcrlTNQ==", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.383 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.1", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDVqoFlQpxyDhAAItCAH5YyKkz9Aoum2OQZnA==", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.2", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDYqoFlDvPYGhAArhcKhOoCTrPpG9lpUz5VQQ==", 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.3", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDbqoFlQw7YJhAAia8xQRRIjjZhmslpQper0g==", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.4", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDiqoFlzRFiAxAAw5uVGNe5REFBLNK586E0Eg==", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.5", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDlqoFlBJmBEBAAESIqltSIRootyLL05shYdQ==", 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.6", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDoqoFls53wHBAA+YeUm2GnQZICd/M62R4ueg==", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "osd.7", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDrqoFlzOw8KhAAbtbYiHuWybqvp5WgQ/3pWg==", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow profile osd", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile osd", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.0", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlumf6ExAAdEw3KhO3InjEdoMdxVuszA==", 2023-12-19T16:13:19.388 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow r", 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.1", 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlBVqCHxAAtuMXJ/ryiLZF2DU5MIV7Uw==", 2023-12-19T16:13:19.389 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.2", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlkqeqGRAAP2Pafyu9kNUFV7lLg1+SEA==", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:13:19.390 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.3", 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDQqoFlcB6cJBAAt3UsEOWd+CSzvK3vX6javA==", 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow", 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow r", 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow rw", 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow rwx" 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.391 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "client.admin", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlui0SBRAAuqrD6TErwy7tTFraf6tevw==", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": "allow *", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow *", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.x", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFlcZ6rJhAApGgJT2+cAnTSpd1xGHTwvA==", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:13:19.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "entity": "mgr.y", 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "AQDPqoFl3vbvIBAAKXc63iWve0y36th+cmM/Bw==", 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "caps": { 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds": "allow *", 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": "allow profile mgr", 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "allow *" 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2023-12-19T16:13:19.393 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:13:19.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 msgr2=0x7f420c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 0x7f420c062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42140099b0 tx=0x7f4214017000 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 msgr2=0x7f422c16d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d860 tx=0x7f421c008040 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 shutdown_connections 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f420c060670 0x7f420c062b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f422c0c51b0 0x7f422c152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f422c158d60 0x7f422c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 --2- 172.21.15.50:0/3370226709 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f422c06b7a0 0x7f422c16d090 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 >> 172.21.15.50:0/3370226709 conn(0x7f422c067bb0 msgr2=0x7f422c153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 shutdown_connections 2023-12-19T16:13:19.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.380+0000 7f422b59e700 1 -- 172.21.15.50:0/3370226709 wait complete. 2023-12-19T16:13:19.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:19.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 Processor -- start 2023-12-19T16:13:19.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 -- start start 2023-12-19T16:13:19.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f23ac06a540 0x7f23ac06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0ca5a0 con 0x7f23ac0cc470 2023-12-19T16:13:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac06dac0 con 0x7f23ac158e70 2023-12-19T16:13:19.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.572+0000 7f23b18e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac06d220 con 0x7f23ac06a540 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b10e2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f23ac0cc470 0x7f23ac0cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57100/0 (socket says 172.21.15.50:57100) 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b10e2700 1 -- 172.21.15.50:0/3820297302 learned_addr learned my addr 172.21.15.50:0/3820297302 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3820297302 >> v2:172.21.15.50:3300/0 conn(0x7f23ac06a540 0x7f23ac06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835121471 0 0) 0x7f23ac0ca5a0 con 0x7f23ac0cc470 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394003570 con 0x7f23ac0cc470 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220104482 0 0) 0x7f23ac06dac0 con 0x7f23ac158e70 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac0ca5a0 con 0x7f23ac158e70 2023-12-19T16:13:19.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a3fff700 1 -- 172.21.15.50:0/3820297302 >> v1:172.21.15.125:6789/0 conn(0x7f23ac158e70 legacy=0x7f23ac0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271992762 0 0) 0x7f2394003570 con 0x7f23ac0cc470 2023-12-19T16:13:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a3fff700 1 -- 172.21.15.50:0/3820297302 >> v1:172.21.15.50:6789/0 conn(0x7f23ac0cc470 legacy=0x7f23ac0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a3fff700 1 -- 172.21.15.50:0/3820297302 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac06d500 con 0x7f23ac06a540 2023-12-19T16:13:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3820297302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac06c9d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f239c0098e0 tx=0x7f239c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16184ba4d1bba503 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a40017f0 con 0x7f23ac06a540 2023-12-19T16:13:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c011420 con 0x7f23ac06a540 2023-12-19T16:13:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23a37fe700 1 -- 172.21.15.50:0/3820297302 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c0117c0 con 0x7f23ac06a540 2023-12-19T16:13:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 -- 172.21.15.50:0/3820297302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 msgr2=0x7f23ac06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3820297302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac06c9d0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f239c0098e0 tx=0x7f239c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 -- 172.21.15.50:0/3820297302 shutdown_connections 2023-12-19T16:13:19.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3820297302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac06c9d0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 -- 172.21.15.50:0/3820297302 >> 172.21.15.50:0/3820297302 conn(0x7f23ac0670e0 msgr2=0x7f23ac0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:19.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 -- 172.21.15.50:0/3820297302 shutdown_connections 2023-12-19T16:13:19.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.576+0000 7f23b18e3700 1 -- 172.21.15.50:0/3820297302 wait complete. 2023-12-19T16:13:19.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 Processor -- start 2023-12-19T16:13:19.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 -- start start 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 0x7f23ac0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23ac158e70 0x7f23ac0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac15c240 con 0x7f23ac06a540 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b4ee0 con 0x7f23ac158e70 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b18e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b51c0 con 0x7f23ac0cc470 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.580+0000 7f23b08e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47050/0 (socket says 172.21.15.50:47050) 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 -- 172.21.15.50:0/3728256818 learned_addr learned my addr 172.21.15.50:0/3728256818 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 0x7f23ac0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b10e2700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23ac158e70 0x7f23ac0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 -- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 msgr2=0x7f23ac0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 0x7f23ac0b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 -- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23ac158e70 msgr2=0x7f23ac0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23ac158e70 0x7f23ac0ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 -- 172.21.15.50:0/3728256818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c009420 con 0x7f23ac06a540 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 0x7f23ac0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b08e1700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c580 tx=0x7f23a8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8010510 con 0x7f23ac06a540 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8010690 con 0x7f23ac06a540 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8010a50 con 0x7f23ac06a540 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac0b54a0 con 0x7f23ac06a540 2023-12-19T16:13:19.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.584+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac1547c0 con 0x7f23ac06a540 2023-12-19T16:13:19.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.588+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac0b5920 con 0x7f23ac06a540 2023-12-19T16:13:19.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.588+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f23a801b400 con 0x7f23ac06a540 2023-12-19T16:13:19.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.588+0000 7f23a0ff9700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 0x7f2398062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.588+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5285+0+0 (secure 0 0 0) 0x7f23a807c460 con 0x7f23ac06a540 2023-12-19T16:13:19.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.588+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 0x7f2398062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.592+0000 7f23a3fff700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 0x7f2398062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f239c0093f0 tx=0x7f239c016000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.596+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23a804ca40 con 0x7f23ac06a540 2023-12-19T16:13:19.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.772+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23ac04be40 con 0x7f23ac06a540 2023-12-19T16:13:19.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.772+0000 7f23a0ff9700 1 -- 172.21.15.50:0/3728256818 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+12528 (secure 0 0 0) 0x7f23a8050060 con 0x7f23ac06a540 2023-12-19T16:13:19.778 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:19.778 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":209,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:18.931999+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 msgr2=0x7f2398062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 0x7f2398062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f239c0093f0 tx=0x7f239c016000 comp rx=0 tx=0).stop 2023-12-19T16:13:19.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 msgr2=0x7f23ac0b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c580 tx=0x7f23a8007e10 comp rx=0 tx=0).stop 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 shutdown_connections 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2398060670 0x7f2398062b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23ac158e70 0x7f23ac0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f23ac0cc470 0x7f23ac0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 --2- 172.21.15.50:0/3728256818 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23ac06a540 0x7f23ac0b4370 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 >> 172.21.15.50:0/3728256818 conn(0x7f23ac0670e0 msgr2=0x7f23ac0b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 shutdown_connections 2023-12-19T16:13:19.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.776+0000 7f23b18e3700 1 -- 172.21.15.50:0/3728256818 wait complete. 2023-12-19T16:13:19.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T16:13:19.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:13:19.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.968+0000 7f00f7385700 1 Processor -- start 2023-12-19T16:13:19.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.972+0000 7f00f7385700 1 -- start start 2023-12-19T16:13:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.972+0000 7f00f7385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f01557f0 0x7f00f006b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.972+0000 7f00f7385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00f006b5d0 0x7f00f00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.972+0000 7f00f7385700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f00aff50 con 0x7f00f0155420 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.972+0000 7f00f7385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f00c95b0 con 0x7f00f006b5d0 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f7385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f006ba50 con 0x7f00f01557f0 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00f006b5d0 0x7f00f00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f00f006b5d0 0x7f00f00b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47060/0 (socket says 172.21.15.50:47060) 2023-12-19T16:13:19.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 -- 172.21.15.50:0/1837138536 learned_addr learned my addr 172.21.15.50:0/1837138536 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f4920700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f01557f0 0x7f00f006b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 -- 172.21.15.50:0/1837138536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f01557f0 msgr2=0x7f00f006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f01557f0 0x7f00f006b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 -- 172.21.15.50:0/1837138536 >> v1:172.21.15.125:6789/0 conn(0x7f00f0155420 legacy=0x7f00f006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00e7fff700 1 -- 172.21.15.50:0/1837138536 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116013664 0 0) 0x7f00f00aff50 con 0x7f00f0155420 2023-12-19T16:13:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 -- 172.21.15.50:0/1837138536 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f00af380 con 0x7f00f006b5d0 2023-12-19T16:13:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f5922700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f006b5d0 0x7f00f00b03e0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c0d0 tx=0x7f00e800a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f3f6b32dc6a9d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00e7fff700 1 -- 172.21.15.50:0/1837138536 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e800f070 con 0x7f00f006b5d0 2023-12-19T16:13:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00e7fff700 1 -- 172.21.15.50:0/1837138536 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e800a2e0 con 0x7f00f006b5d0 2023-12-19T16:13:19.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00e7fff700 1 -- 172.21.15.50:0/1837138536 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e800cd80 con 0x7f00f006b5d0 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f7385700 1 -- 172.21.15.50:0/1837138536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f006b5d0 msgr2=0x7f00f00b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.976+0000 7f00f7385700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f006b5d0 0x7f00f00b03e0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c0d0 tx=0x7f00e800a950 comp rx=0 tx=0).stop 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- 172.21.15.50:0/1837138536 shutdown_connections 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f006b5d0 0x7f00f00b03e0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 --2- 172.21.15.50:0/1837138536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f01557f0 0x7f00f006b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- 172.21.15.50:0/1837138536 >> 172.21.15.50:0/1837138536 conn(0x7f00f00cc580 msgr2=0x7f00f00cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- 172.21.15.50:0/1837138536 shutdown_connections 2023-12-19T16:13:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- 172.21.15.50:0/1837138536 wait complete. 2023-12-19T16:13:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 Processor -- start 2023-12-19T16:13:19.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- start start 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006b5d0 0x7f00f00ab8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f01557f0 0x7f00f00a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0150e80 con 0x7f00f0155420 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00add60 con 0x7f00f01557f0 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f7385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00ad150 con 0x7f00f006b5d0 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.980+0000 7f00f4920700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47068/0 (socket says 172.21.15.50:47068) 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 -- 172.21.15.50:0/1932694501 learned_addr learned my addr 172.21.15.50:0/1932694501 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f5922700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f01557f0 0x7f00f00a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f5121700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006b5d0 0x7f00f00ab8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 -- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006b5d0 msgr2=0x7f00f00ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006b5d0 0x7f00f00ab8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 -- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f01557f0 msgr2=0x7f00f00a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f01557f0 0x7f00f00a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 -- 172.21.15.50:0/1932694501 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e800abf0 con 0x7f00f0155420 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f4920700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e930 tx=0x7f00e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000eed0 con 0x7f00f0155420 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e000c850 con 0x7f00f0155420 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000cbf0 con 0x7f00f0155420 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f00ad430 con 0x7f00f0155420 2023-12-19T16:13:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f00ada70 con 0x7f00f0155420 2023-12-19T16:13:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.984+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f00acec0 con 0x7f00f0155420 2023-12-19T16:13:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.988+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f00e0013470 con 0x7f00f0155420 2023-12-19T16:13:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.988+0000 7f00e57fa700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 0x7f00d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.988+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5285+0+0 (secure 0 0 0) 0x7f00e007ca30 con 0x7f00f0155420 2023-12-19T16:13:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.988+0000 7f00f5121700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 0x7f00d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.992+0000 7f00f5121700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 0x7f00d4062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0011e0 tx=0x7f00ec009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:19.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:19.996+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00e004d010 con 0x7f00f0155420 2023-12-19T16:13:20.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.168+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00f0071120 con 0x7f00f0155420 2023-12-19T16:13:20.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.168+0000 7f00e57fa700 1 -- 172.21.15.50:0/1932694501 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7f00e0050630 con 0x7f00f0155420 2023-12-19T16:13:20.172 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:13:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 msgr2=0x7f00d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 0x7f00d4062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0011e0 tx=0x7f00ec009040 comp rx=0 tx=0).stop 2023-12-19T16:13:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 msgr2=0x7f00f016bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e930 tx=0x7f00e00091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 shutdown_connections 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f00d4060670 0x7f00d4062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00f01557f0 0x7f00f00a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00f0155420 0x7f00f016bd10 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 --2- 172.21.15.50:0/1932694501 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f00f006b5d0 0x7f00f00ab8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 >> 172.21.15.50:0/1932694501 conn(0x7f00f00cc580 msgr2=0x7f00f006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 shutdown_connections 2023-12-19T16:13:20.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.172+0000 7f00f7385700 1 -- 172.21.15.50:0/1932694501 wait complete. 2023-12-19T16:13:20.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T16:13:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 Processor -- start 2023-12-19T16:13:20.403 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 -- start start 2023-12-19T16:13:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6f81502b0 0x7ff6f8068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 0x7ff6f80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80c6630 con 0x7ff6f8067c90 2023-12-19T16:13:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80c6910 con 0x7ff6f80c5900 2023-12-19T16:13:20.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.400+0000 7ff6fd065700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80c6bf0 con 0x7ff6f81502b0 2023-12-19T16:13:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 0x7ff6f80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6f81502b0 0x7ff6f8068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6f81502b0 0x7ff6f8068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:54786/0 (socket says 172.21.15.50:54786) 2023-12-19T16:13:20.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 -- 172.21.15.50:0/2107221840 learned_addr learned my addr 172.21.15.50:0/2107221840 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 -- 172.21.15.50:0/2107221840 >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 msgr2=0x7ff6f80c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 --2- 172.21.15.50:0/2107221840 >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 0x7ff6f80c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 -- 172.21.15.50:0/2107221840 >> v1:172.21.15.125:6789/0 conn(0x7ff6f8067c90 legacy=0x7ff6f814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 -- 172.21.15.50:0/2107221840 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80c6ed0 con 0x7ff6f81502b0 2023-12-19T16:13:20.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f759e700 1 --2- 172.21.15.50:0/2107221840 >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 0x7ff6f80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f659c700 1 --2- 172.21.15.50:0/2107221840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f81502b0 0x7ff6f8068060 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80c56b0 tx=0x7ff6ec009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd196744a947ff5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f5d9b700 1 -- 172.21.15.50:0/2107221840 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec009e70 con 0x7ff6f81502b0 2023-12-19T16:13:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f5d9b700 1 -- 172.21.15.50:0/2107221840 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec00b860 con 0x7ff6f81502b0 2023-12-19T16:13:20.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6f5d9b700 1 -- 172.21.15.50:0/2107221840 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec00bc00 con 0x7ff6f81502b0 2023-12-19T16:13:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 -- 172.21.15.50:0/2107221840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f81502b0 msgr2=0x7ff6f8068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2107221840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f81502b0 0x7ff6f8068060 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80c56b0 tx=0x7ff6ec009250 comp rx=0 tx=0).stop 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 -- 172.21.15.50:0/2107221840 shutdown_connections 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2107221840 >> v2:172.21.15.50:3300/0 conn(0x7ff6f80c5900 0x7ff6f80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2107221840 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f81502b0 0x7ff6f8068060 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 -- 172.21.15.50:0/2107221840 >> 172.21.15.50:0/2107221840 conn(0x7ff6f80cd6d0 msgr2=0x7ff6f806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.404+0000 7ff6fd065700 1 -- 172.21.15.50:0/2107221840 shutdown_connections 2023-12-19T16:13:20.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.408+0000 7ff6fd065700 1 -- 172.21.15.50:0/2107221840 wait complete. 2023-12-19T16:13:20.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.408+0000 7ff6fd065700 1 Processor -- start 2023-12-19T16:13:20.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.408+0000 7ff6fd065700 1 -- start start 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 0x7ff6f80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6f80c5900 0x7ff6f80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8152240 con 0x7ff6f81502b0 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8151630 con 0x7ff6f8067c90 2023-12-19T16:13:20.415 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8151910 con 0x7ff6f80c5900 2023-12-19T16:13:20.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47076/0 (socket says 172.21.15.50:47076) 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 -- 172.21.15.50:0/2228985404 learned_addr learned my addr 172.21.15.50:0/2228985404 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f6d9d700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 0x7ff6f80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f659c700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6f80c5900 0x7ff6f80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 -- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6f80c5900 msgr2=0x7ff6f80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6f80c5900 0x7ff6f80b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 -- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 msgr2=0x7ff6f80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 0x7ff6f80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 -- 172.21.15.50:0/2228985404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009a20 con 0x7ff6f81502b0 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f6d9d700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 0x7ff6f80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6f759e700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009fc0 tx=0x7ff6f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000cd30 con 0x7ff6f81502b0 2023-12-19T16:13:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f000f040 con 0x7ff6f81502b0 2023-12-19T16:13:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec0094f0 con 0x7ff6f81502b0 2023-12-19T16:13:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f00146d0 con 0x7ff6f81502b0 2023-12-19T16:13:20.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.412+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f8151ed0 con 0x7ff6f81502b0 2023-12-19T16:13:20.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.416+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8071120 con 0x7ff6f81502b0 2023-12-19T16:13:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.416+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff6f0003020 con 0x7ff6f81502b0 2023-12-19T16:13:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.420+0000 7ff6deffd700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 0x7ff6e4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.420+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5285+0+0 (secure 0 0 0) 0x7ff6f007d330 con 0x7ff6f81502b0 2023-12-19T16:13:20.424 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.420+0000 7ff6f6d9d700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 0x7ff6e4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.424+0000 7ff6f6d9d700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 0x7ff6e4062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8007fd0 tx=0x7ff6e8009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.428+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff6f004d800 con 0x7ff6f81502b0 2023-12-19T16:13:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.604+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff6f8152380 con 0x7ff6f81502b0 2023-12-19T16:13:20.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.604+0000 7ff6deffd700 1 -- 172.21.15.50:0/2228985404 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v209) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff6f8152380 con 0x7ff6f81502b0 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout: "poolnum": 1, 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout: "poolname": ".mgr" 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:20.609 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T16:13:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.608+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 msgr2=0x7ff6e4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.608+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 0x7ff6e4062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8007fd0 tx=0x7ff6e8009040 comp rx=0 tx=0).stop 2023-12-19T16:13:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.608+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 msgr2=0x7ff6f80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.608+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009fc0 tx=0x7ff6f000b040 comp rx=0 tx=0).stop 2023-12-19T16:13:20.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 shutdown_connections 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ff6e40606a0 0x7ff6e4062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6f81502b0 0x7ff6f80b5050 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ff6f80c5900 0x7ff6f80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 --2- 172.21.15.50:0/2228985404 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6f8067c90 0x7ff6f80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 >> 172.21.15.50:0/2228985404 conn(0x7ff6f80cd6d0 msgr2=0x7ff6f80ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 shutdown_connections 2023-12-19T16:13:20.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.612+0000 7ff6fd065700 1 -- 172.21.15.50:0/2228985404 wait complete. 2023-12-19T16:13:20.633 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T16:13:20.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T16:13:20.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.804+0000 7fcfc1e65700 1 Processor -- start 2023-12-19T16:13:20.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.804+0000 7fcfc1e65700 1 -- start start 2023-12-19T16:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfbc15c930 0x7fcfbc163e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfbc0b0ac0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfbc06b210 con 0x7fcfbc0cb130 2023-12-19T16:13:20.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfbc0ca810 con 0x7fcfbc15c930 2023-12-19T16:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfbc15c930 0x7fcfbc163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcfbc15c930 0x7fcfbc163e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47080/0 (socket says 172.21.15.50:47080) 2023-12-19T16:13:20.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 -- 172.21.15.50:0/1397842892 learned_addr learned my addr 172.21.15.50:0/1397842892 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 -- 172.21.15.50:0/1397842892 >> v1:172.21.15.125:6789/0 conn(0x7fcfbc0cb130 legacy=0x7fcfbc06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 -- 172.21.15.50:0/1397842892 >> v1:172.21.15.50:6789/0 conn(0x7fcfbc0cb500 legacy=0x7fcfbc15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfba7fc700 1 -- 172.21.15.50:0/1397842892 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320924775 0 0) 0x7fcfbc0b0ac0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 -- 172.21.15.50:0/1397842892 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc0b0190 con 0x7fcfbc15c930 2023-12-19T16:13:20.812 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfbbfff700 1 --2- 172.21.15.50:0/1397842892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc15c930 0x7fcfbc163e10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009f20 tx=0x7fcfb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1302fa6004ffadb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfba7fc700 1 -- 172.21.15.50:0/1397842892 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400e070 con 0x7fcfbc15c930 2023-12-19T16:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfba7fc700 1 -- 172.21.15.50:0/1397842892 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb400ba80 con 0x7fcfbc15c930 2023-12-19T16:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfba7fc700 1 -- 172.21.15.50:0/1397842892 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40092e0 con 0x7fcfbc15c930 2023-12-19T16:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 -- 172.21.15.50:0/1397842892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc15c930 msgr2=0x7fcfbc163e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.808+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/1397842892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc15c930 0x7fcfbc163e10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009f20 tx=0x7fcfb4009840 comp rx=0 tx=0).stop 2023-12-19T16:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 -- 172.21.15.50:0/1397842892 shutdown_connections 2023-12-19T16:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/1397842892 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc15c930 0x7fcfbc163e10 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 -- 172.21.15.50:0/1397842892 >> 172.21.15.50:0/1397842892 conn(0x7fcfbc067bb0 msgr2=0x7fcfbc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 -- 172.21.15.50:0/1397842892 shutdown_connections 2023-12-19T16:13:20.814 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 -- 172.21.15.50:0/1397842892 wait complete. 2023-12-19T16:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 Processor -- start 2023-12-19T16:13:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.812+0000 7fcfc1e65700 1 -- start start 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 0x7fcfbc1436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfbc15c930 0x7fcfbc0c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc157490 con 0x7fcfbc0cb500 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc145840 con 0x7fcfbc15c930 2023-12-19T16:13:20.818 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc144c30 con 0x7fcfbc0cb130 2023-12-19T16:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47094/0 (socket says 172.21.15.50:47094) 2023-12-19T16:13:20.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 -- 172.21.15.50:0/2596785606 learned_addr learned my addr 172.21.15.50:0/2596785606 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbbfff700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfbc15c930 0x7fcfbc0c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbb7fe700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 0x7fcfbc1436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 -- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 msgr2=0x7fcfbc1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 0x7fcfbc1436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 -- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfbc15c930 msgr2=0x7fcfbc0c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfbc15c930 0x7fcfbc0c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 -- 172.21.15.50:0/2596785606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4009ae0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbb7fe700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 0x7fcfbc1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfbaffd700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c7e0 tx=0x7fcfb0007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011590 con 0x7fcfbc0cb500 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0011710 con 0x7fcfbc0cb500 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc144f10 con 0x7fcfbc0cb500 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011ad0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.816+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc1454d0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.820+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc1459b0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.824 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.820+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcfb001b3b0 con 0x7fcfbc0cb500 2023-12-19T16:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.820+0000 7fcfa37fe700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 0x7fcfa8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.824+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5285+0+0 (secure 0 0 0) 0x7fcfb007bd90 con 0x7fcfbc0cb500 2023-12-19T16:13:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.824+0000 7fcfbb7fe700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 0x7fcfa8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.824+0000 7fcfbb7fe700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 0x7fcfa8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfac007fd0 tx=0x7fcfac009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:20.828+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcfb004c370 con 0x7fcfbc0cb500 2023-12-19T16:13:21.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.000+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcfbc071120 con 0x7fcfbc0cb500 2023-12-19T16:13:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.932+0000 7fcfa37fe700 1 -- 172.21.15.50:0/2596785606 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v210) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcfb0017020 con 0x7fcfbc0cb500 2023-12-19T16:13:21.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.936+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 msgr2=0x7fcfa8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:21.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.936+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 0x7fcfa8062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfac007fd0 tx=0x7fcfac009040 comp rx=0 tx=0).stop 2023-12-19T16:13:21.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.936+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 msgr2=0x7fcfbc0c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:21.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.936+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c7e0 tx=0x7fcfb0007da0 comp rx=0 tx=0).stop 2023-12-19T16:13:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 shutdown_connections 2023-12-19T16:13:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcfa8060670 0x7fcfa8062b20 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfac007fd0 tx=0x7fcfac009040 comp rx=0 tx=0).stop 2023-12-19T16:13:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfbc15c930 0x7fcfbc0c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfbc0cb500 0x7fcfbc0c4dc0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 --2- 172.21.15.50:0/2596785606 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcfbc0cb130 0x7fcfbc1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 >> 172.21.15.50:0/2596785606 conn(0x7fcfbc067bb0 msgr2=0x7fcfbc0b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 shutdown_connections 2023-12-19T16:13:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:21.940+0000 7fcfc1e65700 1 -- 172.21.15.50:0/2596785606 wait complete. 2023-12-19T16:13:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' created 2023-12-19T16:13:21.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T16:13:22.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.139+0000 7f7790385700 1 Processor -- start 2023-12-19T16:13:22.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.139+0000 7f7790385700 1 -- start start 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f7790385700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77880cb970 0x7f778806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f7790385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77880cecd0 0x7f778806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f7790385700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77880cbdf0 con 0x7f7788158e20 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f7790385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77880cc0d0 con 0x7f77880cecd0 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f7790385700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77880cc3b0 con 0x7f77880cb970 2023-12-19T16:13:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77880cecd0 0x7f778806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f77880cecd0 0x7f778806b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47104/0 (socket says 172.21.15.50:47104) 2023-12-19T16:13:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 -- 172.21.15.50:0/251134338 learned_addr learned my addr 172.21.15.50:0/251134338 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778d920700 1 --2- 172.21.15.50:0/251134338 >> v2:172.21.15.125:3300/0 conn(0x7f77880cb970 0x7f778806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 -- 172.21.15.50:0/251134338 >> v2:172.21.15.125:3300/0 conn(0x7f77880cb970 msgr2=0x7f778806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 --2- 172.21.15.50:0/251134338 >> v2:172.21.15.125:3300/0 conn(0x7f77880cb970 0x7f778806acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 -- 172.21.15.50:0/251134338 >> v1:172.21.15.125:6789/0 conn(0x7f7788158e20 legacy=0x7f77880ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 -- 172.21.15.50:0/251134338 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880cc690 con 0x7f77880cecd0 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778e922700 1 --2- 172.21.15.50:0/251134338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77880cecd0 0x7f778806b200 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f7780009f20 tx=0x7f7780009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0684272383e160 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778d11f700 1 -- 172.21.15.50:0/251134338 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778000e070 con 0x7f77880cecd0 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778d11f700 1 -- 172.21.15.50:0/251134338 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778000ba00 con 0x7f77880cecd0 2023-12-19T16:13:22.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.143+0000 7f778d11f700 1 -- 172.21.15.50:0/251134338 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77800092e0 con 0x7f77880cecd0 2023-12-19T16:13:22.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 -- 172.21.15.50:0/251134338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77880cecd0 msgr2=0x7f778806b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:22.150 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 --2- 172.21.15.50:0/251134338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77880cecd0 0x7f778806b200 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f7780009f20 tx=0x7f7780009840 comp rx=0 tx=0).stop 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 -- 172.21.15.50:0/251134338 shutdown_connections 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 --2- 172.21.15.50:0/251134338 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77880cecd0 0x7f778806b200 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 --2- 172.21.15.50:0/251134338 >> v2:172.21.15.125:3300/0 conn(0x7f77880cb970 0x7f778806acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 -- 172.21.15.50:0/251134338 >> 172.21.15.50:0/251134338 conn(0x7f7788067bb0 msgr2=0x7f77880d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 -- 172.21.15.50:0/251134338 shutdown_connections 2023-12-19T16:13:22.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 -- 172.21.15.50:0/251134338 wait complete. 2023-12-19T16:13:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.147+0000 7f7790385700 1 Processor -- start 2023-12-19T16:13:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 -- start start 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 0x7f778814bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7788158e20 0x7f778814ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778814f9c0 con 0x7f778814e330 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778814c1f0 con 0x7f77880cecd0 2023-12-19T16:13:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f7790385700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778814ead0 con 0x7f7788158e20 2023-12-19T16:13:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47120/0 (socket says 172.21.15.50:47120) 2023-12-19T16:13:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 -- 172.21.15.50:0/3389327229 learned_addr learned my addr 172.21.15.50:0/3389327229 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e121700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 0x7f778814bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778d920700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7788158e20 0x7f778814ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 -- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7788158e20 msgr2=0x7f778814ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7788158e20 0x7f778814ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 -- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 msgr2=0x7f778814bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 0x7f778814bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 -- 172.21.15.50:0/3389327229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7780009ae0 con 0x7f778814e330 2023-12-19T16:13:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e121700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 0x7f778814bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f778e922700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7780003000 tx=0x7f7780017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.151+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77800092e0 con 0x7f778814e330 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.155+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7780012db0 con 0x7f778814e330 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.155+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778001d670 con 0x7f778814e330 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.155+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778814edb0 con 0x7f778814e330 2023-12-19T16:13:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.155+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778814f370 con 0x7f778814e330 2023-12-19T16:13:22.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.155+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778806b7b0 con 0x7f778814e330 2023-12-19T16:13:22.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.159+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f778001da20 con 0x7f778814e330 2023-12-19T16:13:22.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.159+0000 7f777e7fc700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 0x7f776c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:22.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.159+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5655+0+0 (secure 0 0 0) 0x7f778007e4e0 con 0x7f778814e330 2023-12-19T16:13:22.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.159+0000 7f778e121700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 0x7f776c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:22.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.159+0000 7f778e121700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 0x7f776c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.163+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f778004e8c0 con 0x7f778814e330 2023-12-19T16:13:22.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.339+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f778806b7b0 con 0x7f778814e330 2023-12-19T16:13:22.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.947+0000 7f777e7fc700 1 -- 172.21.15.50:0/3389327229 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v211) v1 ==== 170+0+0 (secure 0 0 0) 0x7f778806b7b0 con 0x7f778814e330 2023-12-19T16:13:22.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.951+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 msgr2=0x7f776c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:22.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.951+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 0x7f776c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).stop 2023-12-19T16:13:22.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.951+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 msgr2=0x7f77880cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:22.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.951+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7780003000 tx=0x7f7780017000 comp rx=0 tx=0).stop 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 shutdown_connections 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f776c0606a0 0x7f776c062b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f778814e330 0x7f77880cde50 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7788158e20 0x7f778814ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 --2- 172.21.15.50:0/3389327229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77880cecd0 0x7f778814bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:22.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.955+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 >> 172.21.15.50:0/3389327229 conn(0x7f7788067bb0 msgr2=0x7f778814ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:22.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.959+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 shutdown_connections 2023-12-19T16:13:22.965 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:22.959+0000 7f7790385700 1 -- 172.21.15.50:0/3389327229 wait complete. 2023-12-19T16:13:22.970 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' created 2023-12-19T16:13:22.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T16:13:23.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.159+0000 7f6b9a323700 1 Processor -- start 2023-12-19T16:13:23.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.159+0000 7f6b9a323700 1 -- start start 2023-12-19T16:13:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b9a323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 0x7f6b94159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b9a323700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9406c850 con 0x7f6b941595a0 2023-12-19T16:13:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b9a323700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940cb650 con 0x7f6b94158c90 2023-12-19T16:13:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b9a323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9406b470 con 0x7f6b9406aeb0 2023-12-19T16:13:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b99b22700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6b941595a0 0x7f6b9406cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57114/0 (socket says 172.21.15.50:57114) 2023-12-19T16:13:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b99b22700 1 -- 172.21.15.50:0/2827476307 learned_addr learned my addr 172.21.15.50:0/2827476307 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:23.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b98b20700 1 --2- 172.21.15.50:0/2827476307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 0x7f6b94159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674440049 0 0) 0x7f6b940cb650 con 0x7f6b94158c90 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b80003570 con 0x7f6b94158c90 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530309026 0 0) 0x7f6b9406c850 con 0x7f6b941595a0 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b940cb650 con 0x7f6b941595a0 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b98b20700 1 -- 172.21.15.50:0/2827476307 >> v1:172.21.15.125:6789/0 conn(0x7f6b94158c90 legacy=0x7f6b940caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b98b20700 1 -- 172.21.15.50:0/2827476307 >> v1:172.21.15.50:6789/0 conn(0x7f6b941595a0 legacy=0x7f6b9406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b98b20700 1 -- 172.21.15.50:0/2827476307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9406b750 con 0x7f6b9406aeb0 2023-12-19T16:13:23.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b98b20700 1 --2- 172.21.15.50:0/2827476307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 0x7f6b94159060 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009910 tx=0x7f6b840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3893bb2af37bd8c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:23.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.163+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84011630 con 0x7f6b9406aeb0 2023-12-19T16:13:23.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b84011790 con 0x7f6b9406aeb0 2023-12-19T16:13:23.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b8bfff700 1 -- 172.21.15.50:0/2827476307 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84011b30 con 0x7f6b9406aeb0 2023-12-19T16:13:23.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 -- 172.21.15.50:0/2827476307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 msgr2=0x7f6b94159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 --2- 172.21.15.50:0/2827476307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 0x7f6b94159060 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009910 tx=0x7f6b840091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 -- 172.21.15.50:0/2827476307 shutdown_connections 2023-12-19T16:13:23.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 --2- 172.21.15.50:0/2827476307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b9406aeb0 0x7f6b94159060 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 -- 172.21.15.50:0/2827476307 >> 172.21.15.50:0/2827476307 conn(0x7f6b94067bb0 msgr2=0x7f6b940cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:23.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 -- 172.21.15.50:0/2827476307 shutdown_connections 2023-12-19T16:13:23.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 -- 172.21.15.50:0/2827476307 wait complete. 2023-12-19T16:13:23.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.167+0000 7f6b9a323700 1 Processor -- start 2023-12-19T16:13:23.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 -- start start 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b9406aeb0 0x7f6b941536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b941595a0 0x7f6b940c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b94153c30 con 0x7f6b94158c90 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b940c9b80 con 0x7f6b9406aeb0 2023-12-19T16:13:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b9a323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b940c5180 con 0x7f6b941595a0 2023-12-19T16:13:23.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:23.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47124/0 (socket says 172.21.15.50:47124) 2023-12-19T16:13:23.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 -- 172.21.15.50:0/1894711554 learned_addr learned my addr 172.21.15.50:0/1894711554 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:23.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b99321700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b9406aeb0 0x7f6b941536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:23.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b99b22700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b941595a0 0x7f6b940c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:23.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 -- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b941595a0 msgr2=0x7f6b940c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b941595a0 0x7f6b940c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 -- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b9406aeb0 msgr2=0x7f6b941536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b9406aeb0 0x7f6b941536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 -- 172.21.15.50:0/1894711554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b84009420 con 0x7f6b94158c90 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.171+0000 7f6b98b20700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009f00 tx=0x7f6b84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8401f070 con 0x7f6b94158c90 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b84002f70 con 0x7f6b94158c90 2023-12-19T16:13:23.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b940c95c0 con 0x7f6b94158c90 2023-12-19T16:13:23.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84011900 con 0x7f6b94158c90 2023-12-19T16:13:23.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b940c4e10 con 0x7f6b94158c90 2023-12-19T16:13:23.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.175+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b940c5310 con 0x7f6b94158c90 2023-12-19T16:13:23.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.179+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6b84023070 con 0x7f6b94158c90 2023-12-19T16:13:23.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.179+0000 7f6b897fa700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 0x7f6b78062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:23.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.179+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6037+0+0 (secure 0 0 0) 0x7f6b8407e0f0 con 0x7f6b94158c90 2023-12-19T16:13:23.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.179+0000 7f6b99321700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 0x7f6b78062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:23.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.183+0000 7f6b99321700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 0x7f6b78062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b90003a50 tx=0x7f6b90007d40 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:23.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.183+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6b8404e500 con 0x7f6b94158c90 2023-12-19T16:13:23.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.395+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6b9404be40 con 0x7f6b94158c90 2023-12-19T16:13:23.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.963+0000 7f6b897fa700 1 -- 172.21.15.50:0/1894711554 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v95) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6b840150c0 con 0x7f6b94158c90 2023-12-19T16:13:23.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 msgr2=0x7f6b78062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 0x7f6b78062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b90003a50 tx=0x7f6b90007d40 comp rx=0 tx=0).stop 2023-12-19T16:13:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 msgr2=0x7f6b940c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009f00 tx=0x7f6b84017000 comp rx=0 tx=0).stop 2023-12-19T16:13:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 shutdown_connections 2023-12-19T16:13:23.976 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6b78060670 0x7f6b78062b20 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b90003a50 tx=0x7f6b90007d40 comp rx=0 tx=0).stop 2023-12-19T16:13:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f6b941595a0 0x7f6b940c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b94158c90 0x7f6b940c8a00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 --2- 172.21.15.50:0/1894711554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b9406aeb0 0x7f6b941536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:23.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.971+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 >> 172.21.15.50:0/1894711554 conn(0x7f6b94067bb0 msgr2=0x7f6b940cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:23.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.979+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 shutdown_connections 2023-12-19T16:13:23.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:23.979+0000 7f6b9a323700 1 -- 172.21.15.50:0/1894711554 wait complete. 2023-12-19T16:13:23.993 INFO:teuthology.orchestra.run.smithi050.stderr:new fs with metadata pool 25 and data pool 26 2023-12-19T16:13:24.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:24.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.171+0000 7fe28b42e700 1 Processor -- start 2023-12-19T16:13:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.171+0000 7fe28b42e700 1 -- start start 2023-12-19T16:13:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe28b42e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 0x7fe28406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe28b42e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2840c2d30 0x7fe2841605e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe28b42e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840c92c0 con 0x7fe2840cbdb0 2023-12-19T16:13:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe28b42e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2840c8450 con 0x7fe2840c2d30 2023-12-19T16:13:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe28b42e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2840c86b0 con 0x7fe28406d1f0 2023-12-19T16:13:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2840c2d30 0x7fe2841605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2891ca700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 0x7fe28406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe2840c2d30 0x7fe2841605e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47132/0 (socket says 172.21.15.50:47132) 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2891ca700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 0x7fe28406d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:54858/0 (socket says 172.21.15.50:54858) 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 -- 172.21.15.50:0/2323535008 learned_addr learned my addr 172.21.15.50:0/2323535008 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 -- 172.21.15.50:0/2323535008 >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 msgr2=0x7fe28406d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 --2- 172.21.15.50:0/2323535008 >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 0x7fe28406d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.181 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 -- 172.21.15.50:0/2323535008 >> v1:172.21.15.125:6789/0 conn(0x7fe2840cbdb0 legacy=0x7fe2840c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.175+0000 7fe2899cb700 1 -- 172.21.15.50:0/2323535008 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840c8990 con 0x7fe2840c2d30 2023-12-19T16:13:24.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe2899cb700 1 --2- 172.21.15.50:0/2323535008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2840c2d30 0x7fe2841605e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe28406b7b0 tx=0x7fe27c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a272222f76bde server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe27bfff700 1 -- 172.21.15.50:0/2323535008 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00e070 con 0x7fe2840c2d30 2023-12-19T16:13:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe27bfff700 1 -- 172.21.15.50:0/2323535008 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c00ba20 con 0x7fe2840c2d30 2023-12-19T16:13:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe27bfff700 1 -- 172.21.15.50:0/2323535008 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c0092e0 con 0x7fe2840c2d30 2023-12-19T16:13:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 -- 172.21.15.50:0/2323535008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2840c2d30 msgr2=0x7fe2841605e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 --2- 172.21.15.50:0/2323535008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2840c2d30 0x7fe2841605e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe28406b7b0 tx=0x7fe27c009840 comp rx=0 tx=0).stop 2023-12-19T16:13:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 -- 172.21.15.50:0/2323535008 shutdown_connections 2023-12-19T16:13:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 --2- 172.21.15.50:0/2323535008 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe2840c2d30 0x7fe2841605e0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 --2- 172.21.15.50:0/2323535008 >> v2:172.21.15.125:3300/0 conn(0x7fe28406d1f0 0x7fe28406d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 -- 172.21.15.50:0/2323535008 >> 172.21.15.50:0/2323535008 conn(0x7fe2840cd9c0 msgr2=0x7fe2840c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 -- 172.21.15.50:0/2323535008 shutdown_connections 2023-12-19T16:13:24.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.179+0000 7fe28b42e700 1 -- 172.21.15.50:0/2323535008 wait complete. 2023-12-19T16:13:24.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 Processor -- start 2023-12-19T16:13:24.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 -- start start 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2840c2d30 0x7fe2840a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2840cbdb0 0x7fe2841502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2840a6100 con 0x7fe28406d1f0 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2840a54f0 con 0x7fe2840c2d30 2023-12-19T16:13:24.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.183+0000 7fe28b42e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2840a57d0 con 0x7fe2840cbdb0 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47138/0 (socket says 172.21.15.50:47138) 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 -- 172.21.15.50:0/1804033292 learned_addr learned my addr 172.21.15.50:0/1804033292 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2889c9700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2840c2d30 0x7fe2840a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2899cb700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2840cbdb0 0x7fe2841502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 -- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2840cbdb0 msgr2=0x7fe2841502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2840cbdb0 0x7fe2841502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 -- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2840c2d30 msgr2=0x7fe2840a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2840c2d30 0x7fe2840a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 -- 172.21.15.50:0/1804033292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c009ae0 con 0x7fe28406d1f0 2023-12-19T16:13:24.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2891ca700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fe28000ec80 tx=0x7fe2800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28000c9d0 con 0x7fe28406d1f0 2023-12-19T16:13:24.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28000cb30 con 0x7fe28406d1f0 2023-12-19T16:13:24.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280013470 con 0x7fe28406d1f0 2023-12-19T16:13:24.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2840a5ab0 con 0x7fe28406d1f0 2023-12-19T16:13:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2840c60f0 con 0x7fe28406d1f0 2023-12-19T16:13:24.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.187+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe284071120 con 0x7fe28406d1f0 2023-12-19T16:13:24.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.191+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe280013820 con 0x7fe28406d1f0 2023-12-19T16:13:24.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.191+0000 7fe2797fa700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 0x7fe268062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.191+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fe28007ce70 con 0x7fe28406d1f0 2023-12-19T16:13:24.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.195+0000 7fe2889c9700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 0x7fe268062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.195+0000 7fe2889c9700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 0x7fe268062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe284170a80 tx=0x7fe274009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.199+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe28004d100 con 0x7fe28406d1f0 2023-12-19T16:13:24.377 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.371+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2840c8c70 con 0x7fe28406d1f0 2023-12-19T16:13:24.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.375+0000 7fe2797fa700 1 -- 172.21.15.50:0/1804033292 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+15761 (secure 0 0 0) 0x7fe280050720 con 0x7fe28406d1f0 2023-12-19T16:13:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":212,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:23.400509+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:24.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 msgr2=0x7fe268062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 0x7fe268062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe284170a80 tx=0x7fe274009040 comp rx=0 tx=0).stop 2023-12-19T16:13:24.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 msgr2=0x7fe2840c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fe28000ec80 tx=0x7fe2800091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 shutdown_connections 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe268060670 0x7fe268062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2840c2d30 0x7fe2840a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe2840cbdb0 0x7fe2841502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 --2- 172.21.15.50:0/1804033292 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe28406d1f0 0x7fe2840c6a80 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 >> 172.21.15.50:0/1804033292 conn(0x7fe2840cd9c0 msgr2=0x7fe284067430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 shutdown_connections 2023-12-19T16:13:24.383 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.379+0000 7fe28b42e700 1 -- 172.21.15.50:0/1804033292 wait complete. 2023-12-19T16:13:24.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:24.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.563+0000 7f65292e3700 1 Processor -- start 2023-12-19T16:13:24.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.563+0000 7f65292e3700 1 -- start start 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65292e3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 0x7f652406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65292e3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65241598d0 0x7f65240c8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65292e3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6524159cd0 con 0x7f652406d590 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65292e3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f652406da10 con 0x7f65241598d0 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65292e3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6524159f90 con 0x7f6524158df0 2023-12-19T16:13:24.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65241598d0 0x7f65240c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f65241598d0 0x7f65240c8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47150/0 (socket says 172.21.15.50:47150) 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 -- 172.21.15.50:0/3400073258 learned_addr learned my addr 172.21.15.50:0/3400073258 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f65237fe700 1 --1- 172.21.15.50:0/3400073258 >> v1:172.21.15.50:6789/0 conn(0x7f652406d590 0x7f65241591c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57128/0 (socket says 172.21.15.50:57128) 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6523fff700 1 --2- 172.21.15.50:0/3400073258 >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 0x7f652406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 -- 172.21.15.50:0/3400073258 >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 msgr2=0x7f652406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 --2- 172.21.15.50:0/3400073258 >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 0x7f652406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 -- 172.21.15.50:0/3400073258 >> v1:172.21.15.50:6789/0 conn(0x7f652406d590 legacy=0x7f65241591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6528ae2700 1 -- 172.21.15.50:0/3400073258 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652415a270 con 0x7f65241598d0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.567+0000 7f6523fff700 1 --2- 172.21.15.50:0/3400073258 >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 0x7f652406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f6528ae2700 1 --2- 172.21.15.50:0/3400073258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65241598d0 0x7f65240c8130 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f652406c5a0 tx=0x7f651c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a845238e0534c54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f6522ffd700 1 -- 172.21.15.50:0/3400073258 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00e070 con 0x7f65241598d0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f6522ffd700 1 -- 172.21.15.50:0/3400073258 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c00ba50 con 0x7f65241598d0 2023-12-19T16:13:24.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f6522ffd700 1 -- 172.21.15.50:0/3400073258 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c0092e0 con 0x7f65241598d0 2023-12-19T16:13:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 -- 172.21.15.50:0/3400073258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65241598d0 msgr2=0x7f65240c8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 --2- 172.21.15.50:0/3400073258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65241598d0 0x7f65240c8130 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f652406c5a0 tx=0x7f651c009840 comp rx=0 tx=0).stop 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 -- 172.21.15.50:0/3400073258 shutdown_connections 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 --2- 172.21.15.50:0/3400073258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f65241598d0 0x7f65240c8130 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 --2- 172.21.15.50:0/3400073258 >> v2:172.21.15.125:3300/0 conn(0x7f6524158df0 0x7f652406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 -- 172.21.15.50:0/3400073258 >> 172.21.15.50:0/3400073258 conn(0x7f652406c080 msgr2=0x7f65240ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 -- 172.21.15.50:0/3400073258 shutdown_connections 2023-12-19T16:13:24.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.571+0000 7f65292e3700 1 -- 172.21.15.50:0/3400073258 wait complete. 2023-12-19T16:13:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 Processor -- start 2023-12-19T16:13:24.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 -- start start 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 0x7f6524169a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65241598d0 0x7f65240c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652415b5b0 con 0x7f652406d590 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652415c410 con 0x7f6524158df0 2023-12-19T16:13:24.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f65292e3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65240c5020 con 0x7f65241598d0 2023-12-19T16:13:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f6523fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f6523fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47158/0 (socket says 172.21.15.50:47158) 2023-12-19T16:13:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.575+0000 7f6523fff700 1 -- 172.21.15.50:0/821656500 learned_addr learned my addr 172.21.15.50:0/821656500 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6528ae2700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65241598d0 0x7f65240c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f65237fe700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 0x7f6524169a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 -- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65241598d0 msgr2=0x7f65240c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65241598d0 0x7f65240c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 -- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 msgr2=0x7f6524169a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 0x7f6524169a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 -- 172.21.15.50:0/821656500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c009ae0 con 0x7f652406d590 2023-12-19T16:13:24.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f65237fe700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 0x7f6524169a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f6523fff700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f651400c830 tx=0x7f6514008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514011600 con 0x7f652406d590 2023-12-19T16:13:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6514011760 con 0x7f652406d590 2023-12-19T16:13:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652415b800 con 0x7f652406d590 2023-12-19T16:13:24.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514011b20 con 0x7f652406d590 2023-12-19T16:13:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652415bdc0 con 0x7f652406d590 2023-12-19T16:13:24.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.579+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652404be40 con 0x7f652406d590 2023-12-19T16:13:24.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.583+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f65140193b0 con 0x7f652406d590 2023-12-19T16:13:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.583+0000 7f650bfff700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 0x7f6510062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.583+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f651407cbd0 con 0x7f652406d590 2023-12-19T16:13:24.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.583+0000 7f65237fe700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 0x7f6510062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.587+0000 7f65237fe700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 0x7f6510062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6518007fd0 tx=0x7f6518007e10 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.591+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f651404cdf0 con 0x7f652406d590 2023-12-19T16:13:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.759+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f652415d8c0 con 0x7f652406d590 2023-12-19T16:13:24.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f650bfff700 1 -- 172.21.15.50:0/821656500 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+15761 (secure 0 0 0) 0x7f6514015090 con 0x7f652406d590 2023-12-19T16:13:24.767 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:24.767 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":212,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:23.400509+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 msgr2=0x7f6510062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 0x7f6510062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6518007fd0 tx=0x7f6518007e10 comp rx=0 tx=0).stop 2023-12-19T16:13:24.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 msgr2=0x7f6524155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f651400c830 tx=0x7f6514008040 comp rx=0 tx=0).stop 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 shutdown_connections 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f6510060670 0x7f6510062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6524158df0 0x7f6524169a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f65241598d0 0x7f65240c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 --2- 172.21.15.50:0/821656500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f652406d590 0x7f6524155ff0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 >> 172.21.15.50:0/821656500 conn(0x7f652406c080 msgr2=0x7f65240ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 shutdown_connections 2023-12-19T16:13:24.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.763+0000 7f65292e3700 1 -- 172.21.15.50:0/821656500 wait complete. 2023-12-19T16:13:24.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T16:13:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.963+0000 7f9d2560c700 1 Processor -- start 2023-12-19T16:13:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.963+0000 7f9d2560c700 1 -- start start 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d2560c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9d2013f6a0 0x7f9d200ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d2560c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200adc60 con 0x7f9d200c7780 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d2560c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200ad050 con 0x7f9d2013f2d0 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d2560c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d200ad330 con 0x7f9d2013f6a0 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1f7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9d2013f6a0 0x7f9d200ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d24e0b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9d200c7780 0x7f9d2006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57130/0 (socket says 172.21.15.50:57130) 2023-12-19T16:13:24.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d24e0b700 1 -- 172.21.15.50:0/1479761802 learned_addr learned my addr 172.21.15.50:0/1479761802 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1f7fe700 1 -- 172.21.15.50:0/1479761802 >> v1:172.21.15.125:6789/0 conn(0x7f9d2013f2d0 legacy=0x7f9d2015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1f7fe700 1 -- 172.21.15.50:0/1479761802 >> v1:172.21.15.50:6789/0 conn(0x7f9d200c7780 legacy=0x7f9d2006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1f7fe700 1 -- 172.21.15.50:0/1479761802 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200ad610 con 0x7f9d2013f6a0 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1effd700 1 -- 172.21.15.50:0/1479761802 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522139624 0 0) 0x7f9d200adc60 con 0x7f9d200c7780 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1f7fe700 1 --2- 172.21.15.50:0/1479761802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200ac9b0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9d140098e0 tx=0x7f9d140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54de7e16e2d7954d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1effd700 1 -- 172.21.15.50:0/1479761802 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14009e80 con 0x7f9d2013f6a0 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1effd700 1 -- 172.21.15.50:0/1479761802 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1400b820 con 0x7f9d2013f6a0 2023-12-19T16:13:24.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.967+0000 7f9d1effd700 1 -- 172.21.15.50:0/1479761802 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400bba0 con 0x7f9d2013f6a0 2023-12-19T16:13:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 -- 172.21.15.50:0/1479761802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 msgr2=0x7f9d200ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 --2- 172.21.15.50:0/1479761802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200ac9b0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9d140098e0 tx=0x7f9d140091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 -- 172.21.15.50:0/1479761802 shutdown_connections 2023-12-19T16:13:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 --2- 172.21.15.50:0/1479761802 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200ac9b0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 -- 172.21.15.50:0/1479761802 >> 172.21.15.50:0/1479761802 conn(0x7f9d20067b50 msgr2=0x7f9d200cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 -- 172.21.15.50:0/1479761802 shutdown_connections 2023-12-19T16:13:24.975 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.971+0000 7f9d2560c700 1 -- 172.21.15.50:0/1479761802 wait complete. 2023-12-19T16:13:24.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 Processor -- start 2023-12-19T16:13:24.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 -- start start 2023-12-19T16:13:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d200c7780 0x7f9d2015ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 0x7f9d200c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2016a070 con 0x7f9d2013f6a0 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d20169d90 con 0x7f9d200c7780 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d2560c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2015a300 con 0x7f9d2013f2d0 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47176/0 (socket says 172.21.15.50:47176) 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 -- 172.21.15.50:0/3518611222 learned_addr learned my addr 172.21.15.50:0/3518611222 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:24.981 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d1ffff700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d200c7780 0x7f9d2015ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d1f7fe700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 0x7f9d200c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 -- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 msgr2=0x7f9d200c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 0x7f9d200c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 -- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d200c7780 msgr2=0x7f9d2015ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d200c7780 0x7f9d2015ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d24e0b700 1 -- 172.21.15.50:0/3518611222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14009420 con 0x7f9d2013f6a0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.975+0000 7f9d1f7fe700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 0x7f9d200c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d24e0b700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800cbf0 tx=0x7f9d180019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800a910 con 0x7f9d2013f6a0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1800aa70 con 0x7f9d2013f6a0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800ae30 con 0x7f9d2013f6a0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2015a5e0 con 0x7f9d2013f6a0 2023-12-19T16:13:24.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2015b320 con 0x7f9d2013f6a0 2023-12-19T16:13:24.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2015a160 con 0x7f9d2013f6a0 2023-12-19T16:13:24.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.979+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9d18007300 con 0x7f9d2013f6a0 2023-12-19T16:13:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.983+0000 7f9d07fff700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 0x7f9d0c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:24.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.983+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f9d1807ced0 con 0x7f9d2013f6a0 2023-12-19T16:13:24.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.983+0000 7f9d1ffff700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 0x7f9d0c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:24.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.983+0000 7f9d1ffff700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 0x7f9d0c062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d200c9e30 tx=0x7f9d10007ea0 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:24.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:24.987+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d2015a160 con 0x7f9d2013f6a0 2023-12-19T16:13:25.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.195+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9d2015a160 con 0x7f9d2013f6a0 2023-12-19T16:13:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.975+0000 7f9d07fff700 1 -- 172.21.15.50:0/3518611222 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v98) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9d2015a160 con 0x7f9d2013f6a0 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 msgr2=0x7f9d0c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 0x7f9d0c062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d200c9e30 tx=0x7f9d10007ea0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 msgr2=0x7f9d200aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800cbf0 tx=0x7f9d180019b0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 shutdown_connections 2023-12-19T16:13:25.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9d0c060670 0x7f9d0c062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d2013f6a0 0x7f9d200aa6d0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9d2013f2d0 0x7f9d200c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 --2- 172.21.15.50:0/3518611222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d200c7780 0x7f9d2015ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 >> 172.21.15.50:0/3518611222 conn(0x7f9d20067b50 msgr2=0x7f9d200b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 shutdown_connections 2023-12-19T16:13:25.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:25.983+0000 7f9d2560c700 1 -- 172.21.15.50:0/3518611222 wait complete. 2023-12-19T16:13:26.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T16:13:26.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.183+0000 7f5710d23700 1 Processor -- start 2023-12-19T16:13:26.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.183+0000 7f5710d23700 1 -- start start 2023-12-19T16:13:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f570c0ce2d0 0x7f570c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f570c0caa10 0x7f570c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 0x7f570c158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c15d960 con 0x7f570c0ce2d0 2023-12-19T16:13:26.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c0cbbb0 con 0x7f570c0caa10 2023-12-19T16:13:26.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f5710d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c06dc90 con 0x7f570c0cc040 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f570c0ce2d0 0x7f570c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f570c0ce2d0 0x7f570c0ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47186/0 (socket says 172.21.15.50:47186) 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 -- 172.21.15.50:0/2487259426 learned_addr learned my addr 172.21.15.50:0/2487259426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570bfff700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 0x7f570c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570affd700 1 --2- 172.21.15.50:0/2487259426 >> v2:172.21.15.125:3300/0 conn(0x7f570c0caa10 0x7f570c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 -- 172.21.15.50:0/2487259426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 msgr2=0x7f570c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 0x7f570c158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 -- 172.21.15.50:0/2487259426 >> v2:172.21.15.125:3300/0 conn(0x7f570c0caa10 msgr2=0x7f570c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 --2- 172.21.15.50:0/2487259426 >> v2:172.21.15.125:3300/0 conn(0x7f570c0caa10 0x7f570c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570b7fe700 1 -- 172.21.15.50:0/2487259426 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0cafa0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.187+0000 7f570bfff700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 0x7f570c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:26.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f570b7fe700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0ce6a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f570c0696f0 tx=0x7f56fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c94b4011b1a94f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:26.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f570a7fc700 1 -- 172.21.15.50:0/2487259426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc00b860 con 0x7f570c0ce2d0 2023-12-19T16:13:26.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f570a7fc700 1 -- 172.21.15.50:0/2487259426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc00b9c0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f570a7fc700 1 -- 172.21.15.50:0/2487259426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc00bd60 con 0x7f570c0ce2d0 2023-12-19T16:13:26.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 -- 172.21.15.50:0/2487259426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 msgr2=0x7f570c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:26.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0ce6a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f570c0696f0 tx=0x7f56fc009250 comp rx=0 tx=0).stop 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 -- 172.21.15.50:0/2487259426 shutdown_connections 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0cc040 0x7f570c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 --2- 172.21.15.50:0/2487259426 >> v2:172.21.15.125:3300/0 conn(0x7f570c0caa10 0x7f570c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 --2- 172.21.15.50:0/2487259426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0ce6a0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 -- 172.21.15.50:0/2487259426 >> 172.21.15.50:0/2487259426 conn(0x7f570c067bb0 msgr2=0x7f570c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 -- 172.21.15.50:0/2487259426 shutdown_connections 2023-12-19T16:13:26.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.191+0000 7f5710d23700 1 -- 172.21.15.50:0/2487259426 wait complete. 2023-12-19T16:13:26.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 Processor -- start 2023-12-19T16:13:26.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 -- start start 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0caa10 0x7f570c0b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570c0cc040 0x7f570c142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0b4870 con 0x7f570c0ce2d0 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0b4b10 con 0x7f570c0cc040 2023-12-19T16:13:26.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f5710d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0b4df0 con 0x7f570c0caa10 2023-12-19T16:13:26.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47202/0 (socket says 172.21.15.50:47202) 2023-12-19T16:13:26.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 -- 172.21.15.50:0/3551304155 learned_addr learned my addr 172.21.15.50:0/3551304155 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:26.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570b7fe700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0caa10 0x7f570c0b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.201 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570affd700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570c0cc040 0x7f570c142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 -- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0caa10 msgr2=0x7f570c0b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0caa10 0x7f570c0b58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 -- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570c0cc040 msgr2=0x7f570c142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570c0cc040 0x7f570c142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.195+0000 7f570bfff700 1 -- 172.21.15.50:0/3551304155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0094f0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f570bfff700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f570400d180 tx=0x7f5704009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570400dcc0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570400f040 con 0x7f570c0ce2d0 2023-12-19T16:13:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c0b50d0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704015690 con 0x7f570c0ce2d0 2023-12-19T16:13:26.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c1431b0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.199+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c04be40 con 0x7f570c0ce2d0 2023-12-19T16:13:26.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.203+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f570401e3b0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.203+0000 7f56f37fe700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 0x7f56f8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:26.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.203+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f570407d090 con 0x7f570c0ce2d0 2023-12-19T16:13:26.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.203+0000 7f570b7fe700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 0x7f56f8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:26.210 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.207+0000 7f570b7fe700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 0x7f56f8062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f570c0696f0 tx=0x7f56fc017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:26.214 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.211+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f570404d2b0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.419+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f570c0cb280 con 0x7f570c0ce2d0 2023-12-19T16:13:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.987+0000 7f56f37fe700 1 -- 172.21.15.50:0/3551304155 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v99) v1 ==== 122+0+0 (secure 0 0 0) 0x7f57040508d0 con 0x7f570c0ce2d0 2023-12-19T16:13:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 msgr2=0x7f56f8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 0x7f56f8062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f570c0696f0 tx=0x7f56fc017000 comp rx=0 tx=0).stop 2023-12-19T16:13:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 msgr2=0x7f570c0c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f570400d180 tx=0x7f5704009a50 comp rx=0 tx=0).stop 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 shutdown_connections 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f56f8060670 0x7f56f8062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f570c0cc040 0x7f570c142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f570c0ce2d0 0x7f570c0c8c30 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 --2- 172.21.15.50:0/3551304155 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f570c0caa10 0x7f570c0b58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 >> 172.21.15.50:0/3551304155 conn(0x7f570c067bb0 msgr2=0x7f570c0cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 shutdown_connections 2023-12-19T16:13:26.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:26.991+0000 7f5710d23700 1 -- 172.21.15.50:0/3551304155 wait complete. 2023-12-19T16:13:27.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:27.193 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.187+0000 7fd559cae700 1 Processor -- start 2023-12-19T16:13:27.194 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.187+0000 7fd559cae700 1 -- start start 2023-12-19T16:13:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd559cae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 0x7fd5540c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd559cae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd554140ae0 0x7fd5540680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd559cae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55406e150 con 0x7fd5540c75e0 2023-12-19T16:13:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd559cae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55406d540 con 0x7fd554140ae0 2023-12-19T16:13:27.195 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd559cae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55406d820 con 0x7fd554140710 2023-12-19T16:13:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd554140ae0 0x7fd5540680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd554140ae0 0x7fd5540680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:52480/0 (socket says 172.21.15.50:52480) 2023-12-19T16:13:27.196 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 -- 172.21.15.50:0/4139514223 learned_addr learned my addr 172.21.15.50:0/4139514223 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd5537fe700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 0x7fd5540c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.197 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 -- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 msgr2=0x7fd5540c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 0x7fd5540c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 -- 172.21.15.50:0/4139514223 >> v1:172.21.15.125:6789/0 conn(0x7fd5540c75e0 legacy=0x7fd55406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd54affd700 1 -- 172.21.15.50:0/4139514223 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55406db00 con 0x7fd554140ae0 2023-12-19T16:13:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.191+0000 7fd5537fe700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 0x7fd5540c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:27.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd54affd700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140ae0 0x7fd5540680f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0099b0 tx=0x7fd53c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=897f670cf091c957 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd552ffd700 1 -- 172.21.15.50:0/4139514223 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c009ea0 con 0x7fd554140ae0 2023-12-19T16:13:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd552ffd700 1 -- 172.21.15.50:0/4139514223 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c00b860 con 0x7fd554140ae0 2023-12-19T16:13:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd552ffd700 1 -- 172.21.15.50:0/4139514223 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00bc00 con 0x7fd554140ae0 2023-12-19T16:13:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 -- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140ae0 msgr2=0x7fd5540680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140ae0 0x7fd5540680f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0099b0 tx=0x7fd53c009250 comp rx=0 tx=0).stop 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 -- 172.21.15.50:0/4139514223 shutdown_connections 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140ae0 0x7fd5540680f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 --2- 172.21.15.50:0/4139514223 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd554140710 0x7fd5540c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 -- 172.21.15.50:0/4139514223 >> 172.21.15.50:0/4139514223 conn(0x7fd5540c8850 msgr2=0x7fd5540c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 -- 172.21.15.50:0/4139514223 shutdown_connections 2023-12-19T16:13:27.200 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.195+0000 7fd559cae700 1 -- 172.21.15.50:0/4139514223 wait complete. 2023-12-19T16:13:27.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 Processor -- start 2023-12-19T16:13:27.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 -- start start 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 0x7fd5541713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 0x7fd554159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd554170920 con 0x7fd554140ae0 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55415a310 con 0x7fd554140710 2023-12-19T16:13:27.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd559cae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55415a5f0 con 0x7fd5540c75e0 2023-12-19T16:13:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd553fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd54affd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 0x7fd554159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd553fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60634/0 (socket says 172.21.15.50:60634) 2023-12-19T16:13:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd5537fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 0x7fd5541713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.199+0000 7fd553fff700 1 -- 172.21.15.50:0/1909497536 learned_addr learned my addr 172.21.15.50:0/1909497536 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 -- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 msgr2=0x7fd554159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 0x7fd554159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 -- 172.21.15.50:0/1909497536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 msgr2=0x7fd554157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 -- 172.21.15.50:0/1909497536 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544009420 con 0x7fd5540c75e0 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd54affd700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 0x7fd554159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd553fff700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:27.207 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd5537fe700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 0x7fd5541713e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd54400ec80 tx=0x7fd5440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400c850 con 0x7fd5540c75e0 2023-12-19T16:13:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400c9b0 con 0x7fd5540c75e0 2023-12-19T16:13:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c0094f0 con 0x7fd5540c75e0 2023-12-19T16:13:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400cd70 con 0x7fd5540c75e0 2023-12-19T16:13:27.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd554170ec0 con 0x7fd5540c75e0 2023-12-19T16:13:27.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.203+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5541710b0 con 0x7fd5540c75e0 2023-12-19T16:13:27.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.207+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd544013680 con 0x7fd5540c75e0 2023-12-19T16:13:27.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.207+0000 7fd54bfff700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 0x7fd534062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.207+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fd54407cde0 con 0x7fd5540c75e0 2023-12-19T16:13:27.213 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.211+0000 7fd54affd700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 0x7fd534062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.215 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.211+0000 7fd54affd700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 0x7fd534062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0099b0 tx=0x7fd53c017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.221 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.215+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd54404d070 con 0x7fd5540c75e0 2023-12-19T16:13:27.431 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.427+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd554071120 con 0x7fd5540c75e0 2023-12-19T16:13:27.432 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.427+0000 7fd54bfff700 1 -- 172.21.15.50:0/1909497536 <== mon.2 v2:172.21.15.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fd544050690 con 0x7fd5540c75e0 2023-12-19T16:13:27.433 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:27.433 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:27.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 msgr2=0x7fd534062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 0x7fd534062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0099b0 tx=0x7fd53c017000 comp rx=0 tx=0).stop 2023-12-19T16:13:27.435 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 msgr2=0x7fd5541713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 0x7fd5541713e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd54400ec80 tx=0x7fd5440091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 shutdown_connections 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fd534060670 0x7fd534062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd554140710 0x7fd554159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd554140ae0 0x7fd554157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 --2- 172.21.15.50:0/1909497536 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fd5540c75e0 0x7fd5541713e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 >> 172.21.15.50:0/1909497536 conn(0x7fd5540c8850 msgr2=0x7fd5540c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 shutdown_connections 2023-12-19T16:13:27.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.431+0000 7fd559cae700 1 -- 172.21.15.50:0/1909497536 wait complete. 2023-12-19T16:13:27.438 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:27.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:27.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 Processor -- start 2023-12-19T16:13:27.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 -- start start 2023-12-19T16:13:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2938156c90 0x7f29380cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2938157060 0x7f293815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29380cd5a0 con 0x7f29380cda30 2023-12-19T16:13:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2938157470 con 0x7f2938156c90 2023-12-19T16:13:27.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293f345700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29380cc990 con 0x7f2938157060 2023-12-19T16:13:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293d0e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2938156c90 0x7f29380cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293d0e1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2938156c90 0x7f29380cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60650/0 (socket says 172.21.15.50:60650) 2023-12-19T16:13:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293c8e0700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f29380cda30 0x7f29380c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41222/0 (socket says 172.21.15.50:41222) 2023-12-19T16:13:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293d0e1700 1 -- 172.21.15.50:0/2125455078 learned_addr learned my addr 172.21.15.50:0/2125455078 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:27.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.623+0000 7f293d8e2700 1 --2- 172.21.15.50:0/2125455078 >> v2:172.21.15.125:3300/0 conn(0x7f2938157060 0x7f293815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293d0e1700 1 -- 172.21.15.50:0/2125455078 >> v2:172.21.15.125:3300/0 conn(0x7f2938157060 msgr2=0x7f293815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293d0e1700 1 --2- 172.21.15.50:0/2125455078 >> v2:172.21.15.125:3300/0 conn(0x7f2938157060 0x7f293815ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293d0e1700 1 -- 172.21.15.50:0/2125455078 >> v1:172.21.15.50:6789/0 conn(0x7f29380cda30 legacy=0x7f29380c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293d0e1700 1 -- 172.21.15.50:0/2125455078 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380ccc70 con 0x7f2938156c90 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293d0e1700 1 --2- 172.21.15.50:0/2125455078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f29380cc3b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f29380871b0 tx=0x7f2934009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9c1443ed7e0e46 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f292ffff700 1 -- 172.21.15.50:0/2125455078 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934009e70 con 0x7f2938156c90 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f292ffff700 1 -- 172.21.15.50:0/2125455078 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400b860 con 0x7f2938156c90 2023-12-19T16:13:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f292ffff700 1 -- 172.21.15.50:0/2125455078 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400bc00 con 0x7f2938156c90 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 -- 172.21.15.50:0/2125455078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 msgr2=0x7f29380cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 --2- 172.21.15.50:0/2125455078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f29380cc3b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f29380871b0 tx=0x7f2934009250 comp rx=0 tx=0).stop 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 -- 172.21.15.50:0/2125455078 shutdown_connections 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 --2- 172.21.15.50:0/2125455078 >> v2:172.21.15.125:3300/0 conn(0x7f2938157060 0x7f293815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 --2- 172.21.15.50:0/2125455078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f29380cc3b0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 -- 172.21.15.50:0/2125455078 >> 172.21.15.50:0/2125455078 conn(0x7f293806c7c0 msgr2=0x7f293806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 -- 172.21.15.50:0/2125455078 shutdown_connections 2023-12-19T16:13:27.633 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.627+0000 7f293f345700 1 -- 172.21.15.50:0/2125455078 wait complete. 2023-12-19T16:13:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 Processor -- start 2023-12-19T16:13:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 -- start start 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29380cda30 0x7f29381575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2938157060 0x7f29381453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29381550a0 con 0x7f2938156c90 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29381587a0 con 0x7f29380cda30 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293f345700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2938145920 con 0x7f2938157060 2023-12-19T16:13:27.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60662/0 (socket says 172.21.15.50:60662) 2023-12-19T16:13:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 -- 172.21.15.50:0/1576300728 learned_addr learned my addr 172.21.15.50:0/1576300728 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293d0e1700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29380cda30 0x7f29381575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293d8e2700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2938157060 0x7f29381453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 -- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2938157060 msgr2=0x7f29381453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2938157060 0x7f29381453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 -- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29380cda30 msgr2=0x7f29381575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29380cda30 0x7f29381575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 -- 172.21.15.50:0/1576300728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340094f0 con 0x7f2938156c90 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.631+0000 7f293c8e0700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f292800c7c0 tx=0x7f2928007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928005a70 con 0x7f2938156c90 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928005bd0 con 0x7f2938156c90 2023-12-19T16:13:27.638 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2938157b90 con 0x7f2938156c90 2023-12-19T16:13:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280070d0 con 0x7f2938156c90 2023-12-19T16:13:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2938158150 con 0x7f2938156c90 2023-12-19T16:13:27.639 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2938071120 con 0x7f2938156c90 2023-12-19T16:13:27.641 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.635+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2928007480 con 0x7f2938156c90 2023-12-19T16:13:27.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.639+0000 7f292d7fa700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 0x7f291c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:27.642 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.639+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f292807bf40 con 0x7f2938156c90 2023-12-19T16:13:27.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.639+0000 7f293d0e1700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 0x7f291c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:27.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.639+0000 7f293d0e1700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 0x7f291c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29380871b0 tx=0x7f2934017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:27.649 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.643+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f292804c1d0 con 0x7f2938156c90 2023-12-19T16:13:27.819 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.815+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29380ccf50 con 0x7f2938156c90 2023-12-19T16:13:27.821 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.815+0000 7f292d7fa700 1 -- 172.21.15.50:0/1576300728 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+15748 (secure 0 0 0) 0x7f2928053e60 con 0x7f2938156c90 2023-12-19T16:13:27.822 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:27.822 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":213,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:24.959854+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 msgr2=0x7f291c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 0x7f291c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29380871b0 tx=0x7f2934017000 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 msgr2=0x7f2938158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f292800c7c0 tx=0x7f2928007f70 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 shutdown_connections 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f291c060670 0x7f291c062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2938157060 0x7f29381453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2938156c90 0x7f2938158c30 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 --2- 172.21.15.50:0/1576300728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29380cda30 0x7f29381575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 >> 172.21.15.50:0/1576300728 conn(0x7f293806c7c0 msgr2=0x7f2938155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 shutdown_connections 2023-12-19T16:13:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:27.819+0000 7f293f345700 1 -- 172.21.15.50:0/1576300728 wait complete. 2023-12-19T16:13:27.844 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T16:13:27.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:13:28.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.011+0000 7f9098f23700 1 Processor -- start 2023-12-19T16:13:28.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.011+0000 7f9098f23700 1 -- start start 2023-12-19T16:13:28.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9098f23700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f909406a580 0x7f909406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9098f23700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 0x7f909406d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9098f23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909406c7c0 con 0x7f90940b1410 2023-12-19T16:13:28.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9098f23700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909406af40 con 0x7f909406a580 2023-12-19T16:13:28.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9098f23700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909406cd80 con 0x7f909406d210 2023-12-19T16:13:28.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f909406a580 0x7f909406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f909406a580 0x7f909406aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60674/0 (socket says 172.21.15.50:60674) 2023-12-19T16:13:28.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 -- 172.21.15.50:0/3048100287 learned_addr learned my addr 172.21.15.50:0/3048100287 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:28.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9093fff700 1 --2- 172.21.15.50:0/3048100287 >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 0x7f909406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 -- 172.21.15.50:0/3048100287 >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 msgr2=0x7f909406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 --2- 172.21.15.50:0/3048100287 >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 0x7f909406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 -- 172.21.15.50:0/3048100287 >> v1:172.21.15.50:6789/0 conn(0x7f90940b1410 legacy=0x7f90940ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 -- 172.21.15.50:0/3048100287 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909406b8f0 con 0x7f909406a580 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9093fff700 1 --2- 172.21.15.50:0/3048100287 >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 0x7f909406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:28.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f9092ffd700 1 --2- 172.21.15.50:0/3048100287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f909406aa00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f909415b900 tx=0x7f9088009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b65164b25a37f44 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f90927fc700 1 -- 172.21.15.50:0/3048100287 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800b860 con 0x7f909406a580 2023-12-19T16:13:28.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.015+0000 7f90927fc700 1 -- 172.21.15.50:0/3048100287 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908800b9c0 con 0x7f909406a580 2023-12-19T16:13:28.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f90927fc700 1 -- 172.21.15.50:0/3048100287 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800bd60 con 0x7f909406a580 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 -- 172.21.15.50:0/3048100287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 msgr2=0x7f909406aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 --2- 172.21.15.50:0/3048100287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f909406aa00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f909415b900 tx=0x7f9088009250 comp rx=0 tx=0).stop 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 -- 172.21.15.50:0/3048100287 shutdown_connections 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 --2- 172.21.15.50:0/3048100287 >> v2:172.21.15.125:3300/0 conn(0x7f909406d210 0x7f909406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 --2- 172.21.15.50:0/3048100287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f909406aa00 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 -- 172.21.15.50:0/3048100287 >> 172.21.15.50:0/3048100287 conn(0x7f9094067bb0 msgr2=0x7f90940cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 -- 172.21.15.50:0/3048100287 shutdown_connections 2023-12-19T16:13:28.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 -- 172.21.15.50:0/3048100287 wait complete. 2023-12-19T16:13:28.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.019+0000 7f9098f23700 1 Processor -- start 2023-12-19T16:13:28.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- start start 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f909406d210 0x7f90940c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 0x7f909415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909415bdf0 con 0x7f909406a580 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940c7ce0 con 0x7f90940b1410 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940c70d0 con 0x7f909406d210 2023-12-19T16:13:28.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60690/0 (socket says 172.21.15.50:60690) 2023-12-19T16:13:28.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 -- 172.21.15.50:0/577075196 learned_addr learned my addr 172.21.15.50:0/577075196 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9092ffd700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f909406d210 0x7f90940c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9093fff700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 0x7f909415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 -- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f909406d210 msgr2=0x7f90940c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f909406d210 0x7f90940c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 -- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 msgr2=0x7f909415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 0x7f909415c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 -- 172.21.15.50:0/577075196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880094f0 con 0x7f909406a580 2023-12-19T16:13:28.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9093fff700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 0x7f909415c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f90937fe700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f908400c510 tx=0x7f908400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084011590 con 0x7f909406a580 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084011710 con 0x7f909406a580 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90940c73b0 con 0x7f909406a580 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084011ad0 con 0x7f909406a580 2023-12-19T16:13:28.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.023+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90941548b0 con 0x7f909406a580 2023-12-19T16:13:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.027+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940c75a0 con 0x7f909406a580 2023-12-19T16:13:28.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.027+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f908401b3b0 con 0x7f909406a580 2023-12-19T16:13:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.031+0000 7f907b7fe700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 0x7f9080062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.031+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f908407bf90 con 0x7f909406a580 2023-12-19T16:13:28.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.031+0000 7f9092ffd700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 0x7f9080062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.031+0000 7f9092ffd700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 0x7f9080062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f909415b900 tx=0x7f9088017000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.039+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f908404c220 con 0x7f909406a580 2023-12-19T16:13:28.211 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.207+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90940c7e20 con 0x7f9080060670 2023-12-19T16:13:28.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.227+0000 7f907b7fe700 1 -- 172.21.15.50:0/577075196 <== mgr.10012 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286655 (secure 0 0 0) 0x7f908800bb30 con 0x7f9080060670 2023-12-19T16:13:28.247 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 msgr2=0x7f9080062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 0x7f9080062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f909415b900 tx=0x7f9088017000 comp rx=0 tx=0).stop 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 msgr2=0x7f9094154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f908400c510 tx=0x7f908400c820 comp rx=0 tx=0).stop 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 shutdown_connections 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9080060670 0x7f9080062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90940b1410 0x7f909415c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f909406d210 0x7f90940c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 --2- 172.21.15.50:0/577075196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f909406a580 0x7f9094154370 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 >> 172.21.15.50:0/577075196 conn(0x7f9094067bb0 msgr2=0x7f90940c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:28.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 shutdown_connections 2023-12-19T16:13:28.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.235+0000 7f9098f23700 1 -- 172.21.15.50:0/577075196 wait complete. 2023-12-19T16:13:28.250 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:13:28.263 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T16:13:26.856972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2465808,"kb_used_data":21968,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752508912,"statfs":{"total":773094113280,"available":770569125888,"internally_reserved":0,"allocated":22495232,"data_stored":12054812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.925038"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970036+0000","last_change":"2023-12-19T16:13:23.970036+0000","last_active":"2023-12-19T16:13:23.970036+0000","last_peered":"2023-12-19T16:13:23.970036+0000","last_clean":"2023-12-19T16:13:23.970036+0000","last_became_active":"2023-12-19T16:13:23.968273+0000","last_became_peered":"2023-12-19T16:13:23.968273+0000","last_unstale":"2023-12-19T16:13:23.970036+0000","last_undegraded":"2023-12-19T16:13:23.970036+0000","last_fullsized":"2023-12-19T16:13:23.970036+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:03.353242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981922+0000","last_change":"2023-12-19T16:13:23.981922+0000","last_active":"2023-12-19T16:13:23.981922+0000","last_peered":"2023-12-19T16:13:23.981922+0000","last_clean":"2023-12-19T16:13:23.981922+0000","last_became_active":"2023-12-19T16:13:23.981536+0000","last_became_peered":"2023-12-19T16:13:23.981536+0000","last_unstale":"2023-12-19T16:13:23.981922+0000","last_undegraded":"2023-12-19T16:13:23.981922+0000","last_fullsized":"2023-12-19T16:13:23.981922+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:16.148355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.962573+0000","last_change":"2023-12-19T16:13:23.962573+0000","last_active":"2023-12-19T16:13:23.962573+0000","last_peered":"2023-12-19T16:13:23.962573+0000","last_clean":"2023-12-19T16:13:23.962573+0000","last_became_active":"2023-12-19T16:13:23.962201+0000","last_became_peered":"2023-12-19T16:13:23.962201+0000","last_unstale":"2023-12-19T16:13:23.962573+0000","last_undegraded":"2023-12-19T16:13:23.962573+0000","last_fullsized":"2023-12-19T16:13:23.962573+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:24.937995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973637+0000","last_change":"2023-12-19T16:13:23.973637+0000","last_active":"2023-12-19T16:13:23.973637+0000","last_peered":"2023-12-19T16:13:23.973637+0000","last_clean":"2023-12-19T16:13:23.973637+0000","last_became_active":"2023-12-19T16:13:23.973331+0000","last_became_peered":"2023-12-19T16:13:23.973331+0000","last_unstale":"2023-12-19T16:13:23.973637+0000","last_undegraded":"2023-12-19T16:13:23.973637+0000","last_fullsized":"2023-12-19T16:13:23.973637+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:36.715851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.979869+0000","last_change":"2023-12-19T16:13:23.979869+0000","last_active":"2023-12-19T16:13:23.979869+0000","last_peered":"2023-12-19T16:13:23.979869+0000","last_clean":"2023-12-19T16:13:23.979869+0000","last_became_active":"2023-12-19T16:13:23.978150+0000","last_became_peered":"2023-12-19T16:13:23.978150+0000","last_unstale":"2023-12-19T16:13:23.979869+0000","last_undegraded":"2023-12-19T16:13:23.979869+0000","last_fullsized":"2023-12-19T16:13:23.979869+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981696+0000","last_change":"2023-12-19T16:13:23.981696+0000","last_active":"2023-12-19T16:13:23.981696+0000","last_peered":"2023-12-19T16:13:23.981696+0000","last_clean":"2023-12-19T16:13:23.981696+0000","last_became_active":"2023-12-19T16:13:23.974418+0000","last_became_peered":"2023-12-19T16:13:23.974418+0000","last_unstale":"2023-12-19T16:13:23.981696+0000","last_undegraded":"2023-12-19T16:13:23.981696+0000","last_fullsized":"2023-12-19T16:13:23.981696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:27:42.410785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974635+0000","last_change":"2023-12-19T16:13:23.974635+0000","last_active":"2023-12-19T16:13:23.974635+0000","last_peered":"2023-12-19T16:13:23.974635+0000","last_clean":"2023-12-19T16:13:23.974635+0000","last_became_active":"2023-12-19T16:13:23.974292+0000","last_became_peered":"2023-12-19T16:13:23.974292+0000","last_unstale":"2023-12-19T16:13:23.974635+0000","last_undegraded":"2023-12-19T16:13:23.974635+0000","last_fullsized":"2023-12-19T16:13:23.974635+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:40:20.044406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970147+0000","last_change":"2023-12-19T16:13:23.970147+0000","last_active":"2023-12-19T16:13:23.970147+0000","last_peered":"2023-12-19T16:13:23.970147+0000","last_clean":"2023-12-19T16:13:23.970147+0000","last_became_active":"2023-12-19T16:13:23.969859+0000","last_became_peered":"2023-12-19T16:13:23.969859+0000","last_unstale":"2023-12-19T16:13:23.970147+0000","last_undegraded":"2023-12-19T16:13:23.970147+0000","last_fullsized":"2023-12-19T16:13:23.970147+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976891+0000","last_change":"2023-12-19T16:13:23.976891+0000","last_active":"2023-12-19T16:13:23.976891+0000","last_peered":"2023-12-19T16:13:23.976891+0000","last_clean":"2023-12-19T16:13:23.976891+0000","last_became_active":"2023-12-19T16:13:23.976182+0000","last_became_peered":"2023-12-19T16:13:23.976182+0000","last_unstale":"2023-12-19T16:13:23.976891+0000","last_undegraded":"2023-12-19T16:13:23.976891+0000","last_fullsized":"2023-12-19T16:13:23.976891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:24.937467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.985668+0000","last_change":"2023-12-19T16:13:23.985668+0000","last_active":"2023-12-19T16:13:23.985668+0000","last_peered":"2023-12-19T16:13:23.985668+0000","last_clean":"2023-12-19T16:13:23.985668+0000","last_became_active":"2023-12-19T16:13:23.985248+0000","last_became_peered":"2023-12-19T16:13:23.985248+0000","last_unstale":"2023-12-19T16:13:23.985668+0000","last_undegraded":"2023-12-19T16:13:23.985668+0000","last_fullsized":"2023-12-19T16:13:23.985668+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:40:10.806959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970992+0000","last_change":"2023-12-19T16:13:23.970992+0000","last_active":"2023-12-19T16:13:23.970992+0000","last_peered":"2023-12-19T16:13:23.970992+0000","last_clean":"2023-12-19T16:13:23.970992+0000","last_became_active":"2023-12-19T16:13:23.970667+0000","last_became_peered":"2023-12-19T16:13:23.970667+0000","last_unstale":"2023-12-19T16:13:23.970992+0000","last_undegraded":"2023-12-19T16:13:23.970992+0000","last_fullsized":"2023-12-19T16:13:23.970992+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973646+0000","last_change":"2023-12-19T16:13:23.973646+0000","last_active":"2023-12-19T16:13:23.973646+0000","last_peered":"2023-12-19T16:13:23.973646+0000","last_clean":"2023-12-19T16:13:23.973646+0000","last_became_active":"2023-12-19T16:13:23.973354+0000","last_became_peered":"2023-12-19T16:13:23.973354+0000","last_unstale":"2023-12-19T16:13:23.973646+0000","last_undegraded":"2023-12-19T16:13:23.973646+0000","last_fullsized":"2023-12-19T16:13:23.973646+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968470+0000","last_change":"2023-12-19T16:13:23.968470+0000","last_active":"2023-12-19T16:13:23.968470+0000","last_peered":"2023-12-19T16:13:23.968470+0000","last_clean":"2023-12-19T16:13:23.968470+0000","last_became_active":"2023-12-19T16:13:23.968194+0000","last_became_peered":"2023-12-19T16:13:23.968194+0000","last_unstale":"2023-12-19T16:13:23.968470+0000","last_undegraded":"2023-12-19T16:13:23.968470+0000","last_fullsized":"2023-12-19T16:13:23.968470+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.960911+0000","last_change":"2023-12-19T16:13:23.960911+0000","last_active":"2023-12-19T16:13:23.960911+0000","last_peered":"2023-12-19T16:13:23.960911+0000","last_clean":"2023-12-19T16:13:23.960911+0000","last_became_active":"2023-12-19T16:13:23.960608+0000","last_became_peered":"2023-12-19T16:13:23.960608+0000","last_unstale":"2023-12-19T16:13:23.960911+0000","last_undegraded":"2023-12-19T16:13:23.960911+0000","last_fullsized":"2023-12-19T16:13:23.960911+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:59.167915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975515+0000","last_change":"2023-12-19T16:13:23.975515+0000","last_active":"2023-12-19T16:13:23.975515+0000","last_peered":"2023-12-19T16:13:23.975515+0000","last_clean":"2023-12-19T16:13:23.975515+0000","last_became_active":"2023-12-19T16:13:23.975138+0000","last_became_peered":"2023-12-19T16:13:23.975138+0000","last_unstale":"2023-12-19T16:13:23.975515+0000","last_undegraded":"2023-12-19T16:13:23.975515+0000","last_fullsized":"2023-12-19T16:13:23.975515+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973912+0000","last_change":"2023-12-19T16:13:23.973912+0000","last_active":"2023-12-19T16:13:23.973912+0000","last_peered":"2023-12-19T16:13:23.973912+0000","last_clean":"2023-12-19T16:13:23.973912+0000","last_became_active":"2023-12-19T16:13:23.972177+0000","last_became_peered":"2023-12-19T16:13:23.972177+0000","last_unstale":"2023-12-19T16:13:23.973912+0000","last_undegraded":"2023-12-19T16:13:23.973912+0000","last_fullsized":"2023-12-19T16:13:23.973912+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:40:49.211205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975376+0000","last_change":"2023-12-19T16:13:23.975376+0000","last_active":"2023-12-19T16:13:23.975376+0000","last_peered":"2023-12-19T16:13:23.975376+0000","last_clean":"2023-12-19T16:13:23.975376+0000","last_became_active":"2023-12-19T16:13:23.974864+0000","last_became_peered":"2023-12-19T16:13:23.974864+0000","last_unstale":"2023-12-19T16:13:23.975376+0000","last_undegraded":"2023-12-19T16:13:23.975376+0000","last_fullsized":"2023-12-19T16:13:23.975376+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:04.026807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968085+0000","last_change":"2023-12-19T16:13:23.968085+0000","last_active":"2023-12-19T16:13:23.968085+0000","last_peered":"2023-12-19T16:13:23.968085+0000","last_clean":"2023-12-19T16:13:23.968085+0000","last_became_active":"2023-12-19T16:13:23.967731+0000","last_became_peered":"2023-12-19T16:13:23.967731+0000","last_unstale":"2023-12-19T16:13:23.968085+0000","last_undegraded":"2023-12-19T16:13:23.968085+0000","last_fullsized":"2023-12-19T16:13:23.968085+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966377+0000","last_change":"2023-12-19T16:13:23.966377+0000","last_active":"2023-12-19T16:13:23.966377+0000","last_peered":"2023-12-19T16:13:23.966377+0000","last_clean":"2023-12-19T16:13:23.966377+0000","last_became_active":"2023-12-19T16:13:23.966057+0000","last_became_peered":"2023-12-19T16:13:23.966057+0000","last_unstale":"2023-12-19T16:13:23.966377+0000","last_undegraded":"2023-12-19T16:13:23.966377+0000","last_fullsized":"2023-12-19T16:13:23.966377+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:51.122504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976050+0000","last_change":"2023-12-19T16:13:23.976050+0000","last_active":"2023-12-19T16:13:23.976050+0000","last_peered":"2023-12-19T16:13:23.976050+0000","last_clean":"2023-12-19T16:13:23.976050+0000","last_became_active":"2023-12-19T16:13:23.975540+0000","last_became_peered":"2023-12-19T16:13:23.975540+0000","last_unstale":"2023-12-19T16:13:23.976050+0000","last_undegraded":"2023-12-19T16:13:23.976050+0000","last_fullsized":"2023-12-19T16:13:23.976050+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:50:51.076791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977066+0000","last_change":"2023-12-19T16:13:23.977066+0000","last_active":"2023-12-19T16:13:23.977066+0000","last_peered":"2023-12-19T16:13:23.977066+0000","last_clean":"2023-12-19T16:13:23.977066+0000","last_became_active":"2023-12-19T16:13:23.976715+0000","last_became_peered":"2023-12-19T16:13:23.976715+0000","last_unstale":"2023-12-19T16:13:23.977066+0000","last_undegraded":"2023-12-19T16:13:23.977066+0000","last_fullsized":"2023-12-19T16:13:23.977066+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:35:54.061285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981074+0000","last_change":"2023-12-19T16:13:23.981074+0000","last_active":"2023-12-19T16:13:23.981074+0000","last_peered":"2023-12-19T16:13:23.981074+0000","last_clean":"2023-12-19T16:13:23.981074+0000","last_became_active":"2023-12-19T16:13:23.978102+0000","last_became_peered":"2023-12-19T16:13:23.978102+0000","last_unstale":"2023-12-19T16:13:23.981074+0000","last_undegraded":"2023-12-19T16:13:23.981074+0000","last_fullsized":"2023-12-19T16:13:23.981074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:52:01.357308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974849+0000","last_change":"2023-12-19T16:13:23.974849+0000","last_active":"2023-12-19T16:13:23.974849+0000","last_peered":"2023-12-19T16:13:23.974849+0000","last_clean":"2023-12-19T16:13:23.974849+0000","last_became_active":"2023-12-19T16:13:23.973941+0000","last_became_peered":"2023-12-19T16:13:23.973941+0000","last_unstale":"2023-12-19T16:13:23.974849+0000","last_undegraded":"2023-12-19T16:13:23.974849+0000","last_fullsized":"2023-12-19T16:13:23.974849+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:53:46.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.979783+0000","last_change":"2023-12-19T16:13:23.979783+0000","last_active":"2023-12-19T16:13:23.979783+0000","last_peered":"2023-12-19T16:13:23.979783+0000","last_clean":"2023-12-19T16:13:23.979783+0000","last_became_active":"2023-12-19T16:13:23.979498+0000","last_became_peered":"2023-12-19T16:13:23.979498+0000","last_unstale":"2023-12-19T16:13:23.979783+0000","last_undegraded":"2023-12-19T16:13:23.979783+0000","last_fullsized":"2023-12-19T16:13:23.979783+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:31.025793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972134+0000","last_change":"2023-12-19T16:13:23.972134+0000","last_active":"2023-12-19T16:13:23.972134+0000","last_peered":"2023-12-19T16:13:23.972134+0000","last_clean":"2023-12-19T16:13:23.972134+0000","last_became_active":"2023-12-19T16:13:23.971794+0000","last_became_peered":"2023-12-19T16:13:23.971794+0000","last_unstale":"2023-12-19T16:13:23.972134+0000","last_undegraded":"2023-12-19T16:13:23.972134+0000","last_fullsized":"2023-12-19T16:13:23.972134+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:40.239434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973230+0000","last_change":"2023-12-19T16:13:23.973230+0000","last_active":"2023-12-19T16:13:23.973230+0000","last_peered":"2023-12-19T16:13:23.973230+0000","last_clean":"2023-12-19T16:13:23.973230+0000","last_became_active":"2023-12-19T16:13:23.972890+0000","last_became_peered":"2023-12-19T16:13:23.972890+0000","last_unstale":"2023-12-19T16:13:23.973230+0000","last_undegraded":"2023-12-19T16:13:23.973230+0000","last_fullsized":"2023-12-19T16:13:23.973230+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972074+0000","last_change":"2023-12-19T16:13:23.972074+0000","last_active":"2023-12-19T16:13:23.972074+0000","last_peered":"2023-12-19T16:13:23.972074+0000","last_clean":"2023-12-19T16:13:23.972074+0000","last_became_active":"2023-12-19T16:13:23.971730+0000","last_became_peered":"2023-12-19T16:13:23.971730+0000","last_unstale":"2023-12-19T16:13:23.972074+0000","last_undegraded":"2023-12-19T16:13:23.972074+0000","last_fullsized":"2023-12-19T16:13:23.972074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973455+0000","last_change":"2023-12-19T16:13:23.973455+0000","last_active":"2023-12-19T16:13:23.973455+0000","last_peered":"2023-12-19T16:13:23.973455+0000","last_clean":"2023-12-19T16:13:23.973455+0000","last_became_active":"2023-12-19T16:13:23.973172+0000","last_became_peered":"2023-12-19T16:13:23.973172+0000","last_unstale":"2023-12-19T16:13:23.973455+0000","last_undegraded":"2023-12-19T16:13:23.973455+0000","last_fullsized":"2023-12-19T16:13:23.973455+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:08:12.544870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974712+0000","last_change":"2023-12-19T16:13:23.974712+0000","last_active":"2023-12-19T16:13:23.974712+0000","last_peered":"2023-12-19T16:13:23.974712+0000","last_clean":"2023-12-19T16:13:23.974712+0000","last_became_active":"2023-12-19T16:13:23.974313+0000","last_became_peered":"2023-12-19T16:13:23.974313+0000","last_unstale":"2023-12-19T16:13:23.974712+0000","last_undegraded":"2023-12-19T16:13:23.974712+0000","last_fullsized":"2023-12-19T16:13:23.974712+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:27:42.410785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975220+0000","last_change":"2023-12-19T16:13:23.975220+0000","last_active":"2023-12-19T16:13:23.975220+0000","last_peered":"2023-12-19T16:13:23.975220+0000","last_clean":"2023-12-19T16:13:23.975220+0000","last_became_active":"2023-12-19T16:13:23.974852+0000","last_became_peered":"2023-12-19T16:13:23.974852+0000","last_unstale":"2023-12-19T16:13:23.975220+0000","last_undegraded":"2023-12-19T16:13:23.975220+0000","last_fullsized":"2023-12-19T16:13:23.975220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:50:51.076791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970695+0000","last_change":"2023-12-19T16:13:23.970695+0000","last_active":"2023-12-19T16:13:23.970695+0000","last_peered":"2023-12-19T16:13:23.970695+0000","last_clean":"2023-12-19T16:13:23.970695+0000","last_became_active":"2023-12-19T16:13:23.970298+0000","last_became_peered":"2023-12-19T16:13:23.970298+0000","last_unstale":"2023-12-19T16:13:23.970695+0000","last_undegraded":"2023-12-19T16:13:23.970695+0000","last_fullsized":"2023-12-19T16:13:23.970695+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:40:57.360787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.971241+0000","last_change":"2023-12-19T16:13:23.971241+0000","last_active":"2023-12-19T16:13:23.971241+0000","last_peered":"2023-12-19T16:13:23.971241+0000","last_clean":"2023-12-19T16:13:23.971241+0000","last_became_active":"2023-12-19T16:13:23.970830+0000","last_became_peered":"2023-12-19T16:13:23.970830+0000","last_unstale":"2023-12-19T16:13:23.971241+0000","last_undegraded":"2023-12-19T16:13:23.971241+0000","last_fullsized":"2023-12-19T16:13:23.971241+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:35.816448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970418+0000","last_change":"2023-12-19T16:13:23.970418+0000","last_active":"2023-12-19T16:13:23.970418+0000","last_peered":"2023-12-19T16:13:23.970418+0000","last_clean":"2023-12-19T16:13:23.970418+0000","last_became_active":"2023-12-19T16:13:23.970017+0000","last_became_peered":"2023-12-19T16:13:23.970017+0000","last_unstale":"2023-12-19T16:13:23.970418+0000","last_undegraded":"2023-12-19T16:13:23.970418+0000","last_fullsized":"2023-12-19T16:13:23.970418+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"212'4","reported_seq":13,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.993588+0000","last_change":"2023-12-19T16:13:22.985592+0000","last_active":"2023-12-19T16:13:23.993588+0000","last_peered":"2023-12-19T16:13:23.993588+0000","last_clean":"2023-12-19T16:13:23.993588+0000","last_became_active":"2023-12-19T16:13:22.985029+0000","last_became_peered":"2023-12-19T16:13:22.985029+0000","last_unstale":"2023-12-19T16:13:23.993588+0000","last_undegraded":"2023-12-19T16:13:23.993588+0000","last_fullsized":"2023-12-19T16:13:23.993588+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:27:10.353292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.965087+0000","last_change":"2023-12-19T16:13:23.965087+0000","last_active":"2023-12-19T16:13:23.965087+0000","last_peered":"2023-12-19T16:13:23.965087+0000","last_clean":"2023-12-19T16:13:23.965087+0000","last_became_active":"2023-12-19T16:13:23.963517+0000","last_became_peered":"2023-12-19T16:13:23.963517+0000","last_unstale":"2023-12-19T16:13:23.965087+0000","last_undegraded":"2023-12-19T16:13:23.965087+0000","last_fullsized":"2023-12-19T16:13:23.965087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:53.856584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966263+0000","last_change":"2023-12-19T16:13:22.971877+0000","last_active":"2023-12-19T16:13:23.966263+0000","last_peered":"2023-12-19T16:13:23.966263+0000","last_clean":"2023-12-19T16:13:23.966263+0000","last_became_active":"2023-12-19T16:13:22.971379+0000","last_became_peered":"2023-12-19T16:13:22.971379+0000","last_unstale":"2023-12-19T16:13:23.966263+0000","last_undegraded":"2023-12-19T16:13:23.966263+0000","last_fullsized":"2023-12-19T16:13:23.966263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:15.133576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970699+0000","last_change":"2023-12-19T16:13:23.970699+0000","last_active":"2023-12-19T16:13:23.970699+0000","last_peered":"2023-12-19T16:13:23.970699+0000","last_clean":"2023-12-19T16:13:23.970699+0000","last_became_active":"2023-12-19T16:13:23.970380+0000","last_became_peered":"2023-12-19T16:13:23.970380+0000","last_unstale":"2023-12-19T16:13:23.970699+0000","last_undegraded":"2023-12-19T16:13:23.970699+0000","last_fullsized":"2023-12-19T16:13:23.970699+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:40:49.211205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.989216+0000","last_change":"2023-12-19T16:13:22.968588+0000","last_active":"2023-12-19T16:13:23.989216+0000","last_peered":"2023-12-19T16:13:23.989216+0000","last_clean":"2023-12-19T16:13:23.989216+0000","last_became_active":"2023-12-19T16:13:22.968279+0000","last_became_peered":"2023-12-19T16:13:22.968279+0000","last_unstale":"2023-12-19T16:13:23.989216+0000","last_undegraded":"2023-12-19T16:13:23.989216+0000","last_fullsized":"2023-12-19T16:13:23.989216+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:23.922688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.969754+0000","last_change":"2023-12-19T16:13:23.969754+0000","last_active":"2023-12-19T16:13:23.969754+0000","last_peered":"2023-12-19T16:13:23.969754+0000","last_clean":"2023-12-19T16:13:23.969754+0000","last_became_active":"2023-12-19T16:13:23.969457+0000","last_became_peered":"2023-12-19T16:13:23.969457+0000","last_unstale":"2023-12-19T16:13:23.969754+0000","last_undegraded":"2023-12-19T16:13:23.969754+0000","last_fullsized":"2023-12-19T16:13:23.969754+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:34:28.820521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.986316+0000","last_change":"2023-12-19T16:13:22.975829+0000","last_active":"2023-12-19T16:13:23.986316+0000","last_peered":"2023-12-19T16:13:23.986316+0000","last_clean":"2023-12-19T16:13:23.986316+0000","last_became_active":"2023-12-19T16:13:22.975534+0000","last_became_peered":"2023-12-19T16:13:22.975534+0000","last_unstale":"2023-12-19T16:13:23.986316+0000","last_undegraded":"2023-12-19T16:13:23.986316+0000","last_fullsized":"2023-12-19T16:13:23.986316+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:38.021765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975561+0000","last_change":"2023-12-19T16:13:23.975561+0000","last_active":"2023-12-19T16:13:23.975561+0000","last_peered":"2023-12-19T16:13:23.975561+0000","last_clean":"2023-12-19T16:13:23.975561+0000","last_became_active":"2023-12-19T16:13:23.972636+0000","last_became_peered":"2023-12-19T16:13:23.972636+0000","last_unstale":"2023-12-19T16:13:23.975561+0000","last_undegraded":"2023-12-19T16:13:23.975561+0000","last_fullsized":"2023-12-19T16:13:23.975561+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.957587+0000","last_change":"2023-12-19T16:13:22.969848+0000","last_active":"2023-12-19T16:13:23.957587+0000","last_peered":"2023-12-19T16:13:23.957587+0000","last_clean":"2023-12-19T16:13:23.957587+0000","last_became_active":"2023-12-19T16:13:22.969550+0000","last_became_peered":"2023-12-19T16:13:22.969550+0000","last_unstale":"2023-12-19T16:13:23.957587+0000","last_undegraded":"2023-12-19T16:13:23.957587+0000","last_fullsized":"2023-12-19T16:13:23.957587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:30.011014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968982+0000","last_change":"2023-12-19T16:13:23.968982+0000","last_active":"2023-12-19T16:13:23.968982+0000","last_peered":"2023-12-19T16:13:23.968982+0000","last_clean":"2023-12-19T16:13:23.968982+0000","last_became_active":"2023-12-19T16:13:23.967337+0000","last_became_peered":"2023-12-19T16:13:23.967337+0000","last_unstale":"2023-12-19T16:13:23.968982+0000","last_undegraded":"2023-12-19T16:13:23.968982+0000","last_fullsized":"2023-12-19T16:13:23.968982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:51.122504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.950587+0000","last_change":"2023-12-19T16:13:22.978661+0000","last_active":"2023-12-19T16:13:23.950587+0000","last_peered":"2023-12-19T16:13:23.950587+0000","last_clean":"2023-12-19T16:13:23.950587+0000","last_became_active":"2023-12-19T16:13:22.978363+0000","last_became_peered":"2023-12-19T16:13:22.978363+0000","last_unstale":"2023-12-19T16:13:23.950587+0000","last_undegraded":"2023-12-19T16:13:23.950587+0000","last_fullsized":"2023-12-19T16:13:23.950587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:16.553027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974682+0000","last_change":"2023-12-19T16:13:23.974682+0000","last_active":"2023-12-19T16:13:23.974682+0000","last_peered":"2023-12-19T16:13:23.974682+0000","last_clean":"2023-12-19T16:13:23.974682+0000","last_became_active":"2023-12-19T16:13:23.972382+0000","last_became_peered":"2023-12-19T16:13:23.972382+0000","last_unstale":"2023-12-19T16:13:23.974682+0000","last_undegraded":"2023-12-19T16:13:23.974682+0000","last_fullsized":"2023-12-19T16:13:23.974682+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.975997+0000","last_change":"2023-12-19T16:13:22.974491+0000","last_active":"2023-12-19T16:13:22.975997+0000","last_peered":"2023-12-19T16:13:22.975997+0000","last_clean":"2023-12-19T16:13:22.975997+0000","last_became_active":"2023-12-19T16:13:22.974174+0000","last_became_peered":"2023-12-19T16:13:22.974174+0000","last_unstale":"2023-12-19T16:13:22.975997+0000","last_undegraded":"2023-12-19T16:13:22.975997+0000","last_fullsized":"2023-12-19T16:13:22.975997+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:34:27.805742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974528+0000","last_change":"2023-12-19T16:13:23.974528+0000","last_active":"2023-12-19T16:13:23.974528+0000","last_peered":"2023-12-19T16:13:23.974528+0000","last_clean":"2023-12-19T16:13:23.974528+0000","last_became_active":"2023-12-19T16:13:23.974253+0000","last_became_peered":"2023-12-19T16:13:23.974253+0000","last_unstale":"2023-12-19T16:13:23.974528+0000","last_undegraded":"2023-12-19T16:13:23.974528+0000","last_fullsized":"2023-12-19T16:13:23.974528+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.983031+0000","last_change":"2023-12-19T16:13:22.977080+0000","last_active":"2023-12-19T16:13:22.983031+0000","last_peered":"2023-12-19T16:13:22.983031+0000","last_clean":"2023-12-19T16:13:22.983031+0000","last_became_active":"2023-12-19T16:13:22.976810+0000","last_became_peered":"2023-12-19T16:13:22.976810+0000","last_unstale":"2023-12-19T16:13:22.983031+0000","last_undegraded":"2023-12-19T16:13:22.983031+0000","last_fullsized":"2023-12-19T16:13:22.983031+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:50.107725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976284+0000","last_change":"2023-12-19T16:13:23.976284+0000","last_active":"2023-12-19T16:13:23.976284+0000","last_peered":"2023-12-19T16:13:23.976284+0000","last_clean":"2023-12-19T16:13:23.976284+0000","last_became_active":"2023-12-19T16:13:23.976005+0000","last_became_peered":"2023-12-19T16:13:23.976005+0000","last_unstale":"2023-12-19T16:13:23.976284+0000","last_undegraded":"2023-12-19T16:13:23.976284+0000","last_fullsized":"2023-12-19T16:13:23.976284+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:42:05.874463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981143+0000","last_change":"2023-12-19T16:13:22.977002+0000","last_active":"2023-12-19T16:13:22.981143+0000","last_peered":"2023-12-19T16:13:22.981143+0000","last_clean":"2023-12-19T16:13:22.981143+0000","last_became_active":"2023-12-19T16:13:22.976711+0000","last_became_peered":"2023-12-19T16:13:22.976711+0000","last_unstale":"2023-12-19T16:13:22.981143+0000","last_undegraded":"2023-12-19T16:13:22.981143+0000","last_fullsized":"2023-12-19T16:13:22.981143+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:53.278721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972786+0000","last_change":"2023-12-19T16:13:23.972786+0000","last_active":"2023-12-19T16:13:23.972786+0000","last_peered":"2023-12-19T16:13:23.972786+0000","last_clean":"2023-12-19T16:13:23.972786+0000","last_became_active":"2023-12-19T16:13:23.972371+0000","last_became_peered":"2023-12-19T16:13:23.972371+0000","last_unstale":"2023-12-19T16:13:23.972786+0000","last_undegraded":"2023-12-19T16:13:23.972786+0000","last_fullsized":"2023-12-19T16:13:23.972786+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"212'3","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.992763+0000","last_change":"2023-12-19T16:13:22.982586+0000","last_active":"2023-12-19T16:13:23.992763+0000","last_peered":"2023-12-19T16:13:23.992763+0000","last_clean":"2023-12-19T16:13:23.992763+0000","last_became_active":"2023-12-19T16:13:22.982288+0000","last_became_peered":"2023-12-19T16:13:22.982288+0000","last_unstale":"2023-12-19T16:13:23.992763+0000","last_undegraded":"2023-12-19T16:13:23.992763+0000","last_fullsized":"2023-12-19T16:13:23.992763+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:18:38.074071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.982962+0000","last_change":"2023-12-19T16:13:23.982962+0000","last_active":"2023-12-19T16:13:23.982962+0000","last_peered":"2023-12-19T16:13:23.982962+0000","last_clean":"2023-12-19T16:13:23.982962+0000","last_became_active":"2023-12-19T16:13:23.980078+0000","last_became_peered":"2023-12-19T16:13:23.980078+0000","last_unstale":"2023-12-19T16:13:23.982962+0000","last_undegraded":"2023-12-19T16:13:23.982962+0000","last_fullsized":"2023-12-19T16:13:23.982962+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:17.856062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"212'6","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.990016+0000","last_change":"2023-12-19T16:13:22.971936+0000","last_active":"2023-12-19T16:13:23.990016+0000","last_peered":"2023-12-19T16:13:23.990016+0000","last_clean":"2023-12-19T16:13:23.990016+0000","last_became_active":"2023-12-19T16:13:22.971262+0000","last_became_peered":"2023-12-19T16:13:22.971262+0000","last_unstale":"2023-12-19T16:13:23.990016+0000","last_undegraded":"2023-12-19T16:13:23.990016+0000","last_fullsized":"2023-12-19T16:13:23.990016+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:42:45.282864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972842+0000","last_change":"2023-12-19T16:13:23.972842+0000","last_active":"2023-12-19T16:13:23.972842+0000","last_peered":"2023-12-19T16:13:23.972842+0000","last_clean":"2023-12-19T16:13:23.972842+0000","last_became_active":"2023-12-19T16:13:23.972572+0000","last_became_peered":"2023-12-19T16:13:23.972572+0000","last_unstale":"2023-12-19T16:13:23.972842+0000","last_undegraded":"2023-12-19T16:13:23.972842+0000","last_fullsized":"2023-12-19T16:13:23.972842+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:35:17.640963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991660+0000","last_change":"2023-12-19T16:13:22.980227+0000","last_active":"2023-12-19T16:13:23.991660+0000","last_peered":"2023-12-19T16:13:23.991660+0000","last_clean":"2023-12-19T16:13:23.991660+0000","last_became_active":"2023-12-19T16:13:22.979897+0000","last_became_peered":"2023-12-19T16:13:22.979897+0000","last_unstale":"2023-12-19T16:13:23.991660+0000","last_undegraded":"2023-12-19T16:13:23.991660+0000","last_fullsized":"2023-12-19T16:13:23.991660+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:15.967328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977431+0000","last_change":"2023-12-19T16:13:23.977431+0000","last_active":"2023-12-19T16:13:23.977431+0000","last_peered":"2023-12-19T16:13:23.977431+0000","last_clean":"2023-12-19T16:13:23.977431+0000","last_became_active":"2023-12-19T16:13:23.972608+0000","last_became_peered":"2023-12-19T16:13:23.972608+0000","last_unstale":"2023-12-19T16:13:23.977431+0000","last_undegraded":"2023-12-19T16:13:23.977431+0000","last_fullsized":"2023-12-19T16:13:23.977431+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986563+0000","last_change":"2023-12-19T16:13:22.982663+0000","last_active":"2023-12-19T16:13:22.986563+0000","last_peered":"2023-12-19T16:13:22.986563+0000","last_clean":"2023-12-19T16:13:22.986563+0000","last_became_active":"2023-12-19T16:13:22.982391+0000","last_became_peered":"2023-12-19T16:13:22.982391+0000","last_unstale":"2023-12-19T16:13:22.986563+0000","last_undegraded":"2023-12-19T16:13:22.986563+0000","last_fullsized":"2023-12-19T16:13:22.986563+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:49.526251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.964024+0000","last_change":"2023-12-19T16:13:23.964024+0000","last_active":"2023-12-19T16:13:23.964024+0000","last_peered":"2023-12-19T16:13:23.964024+0000","last_clean":"2023-12-19T16:13:23.964024+0000","last_became_active":"2023-12-19T16:13:23.963649+0000","last_became_peered":"2023-12-19T16:13:23.963649+0000","last_unstale":"2023-12-19T16:13:23.964024+0000","last_undegraded":"2023-12-19T16:13:23.964024+0000","last_fullsized":"2023-12-19T16:13:23.964024+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:03.353242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.987287+0000","last_change":"2023-12-19T16:13:22.981232+0000","last_active":"2023-12-19T16:13:22.987287+0000","last_peered":"2023-12-19T16:13:22.987287+0000","last_clean":"2023-12-19T16:13:22.987287+0000","last_became_active":"2023-12-19T16:13:22.980929+0000","last_became_peered":"2023-12-19T16:13:22.980929+0000","last_unstale":"2023-12-19T16:13:22.987287+0000","last_undegraded":"2023-12-19T16:13:22.987287+0000","last_fullsized":"2023-12-19T16:13:22.987287+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:14.707493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976276+0000","last_change":"2023-12-19T16:13:23.976276+0000","last_active":"2023-12-19T16:13:23.976276+0000","last_peered":"2023-12-19T16:13:23.976276+0000","last_clean":"2023-12-19T16:13:23.976276+0000","last_became_active":"2023-12-19T16:13:23.973860+0000","last_became_peered":"2023-12-19T16:13:23.973860+0000","last_unstale":"2023-12-19T16:13:23.976276+0000","last_undegraded":"2023-12-19T16:13:23.976276+0000","last_fullsized":"2023-12-19T16:13:23.976276+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:09.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"212'8","reported_seq":17,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991327+0000","last_change":"2023-12-19T16:13:22.980604+0000","last_active":"2023-12-19T16:13:23.991327+0000","last_peered":"2023-12-19T16:13:23.991327+0000","last_clean":"2023-12-19T16:13:23.991327+0000","last_became_active":"2023-12-19T16:13:22.979693+0000","last_became_peered":"2023-12-19T16:13:22.979693+0000","last_unstale":"2023-12-19T16:13:23.991327+0000","last_undegraded":"2023-12-19T16:13:23.991327+0000","last_fullsized":"2023-12-19T16:13:23.991327+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:53:20.656471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970531+0000","last_change":"2023-12-19T16:13:23.970531+0000","last_active":"2023-12-19T16:13:23.970531+0000","last_peered":"2023-12-19T16:13:23.970531+0000","last_clean":"2023-12-19T16:13:23.970531+0000","last_became_active":"2023-12-19T16:13:23.970218+0000","last_became_peered":"2023-12-19T16:13:23.970218+0000","last_unstale":"2023-12-19T16:13:23.970531+0000","last_undegraded":"2023-12-19T16:13:23.970531+0000","last_fullsized":"2023-12-19T16:13:23.970531+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:44:38.534471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.961675+0000","last_change":"2023-12-19T16:13:22.973700+0000","last_active":"2023-12-19T16:13:23.961675+0000","last_peered":"2023-12-19T16:13:23.961675+0000","last_clean":"2023-12-19T16:13:23.961675+0000","last_became_active":"2023-12-19T16:13:22.973412+0000","last_became_peered":"2023-12-19T16:13:22.973412+0000","last_unstale":"2023-12-19T16:13:23.961675+0000","last_undegraded":"2023-12-19T16:13:23.961675+0000","last_fullsized":"2023-12-19T16:13:23.961675+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:29.812164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.985046+0000","last_change":"2023-12-19T16:13:23.985046+0000","last_active":"2023-12-19T16:13:23.985046+0000","last_peered":"2023-12-19T16:13:23.985046+0000","last_clean":"2023-12-19T16:13:23.985046+0000","last_became_active":"2023-12-19T16:13:23.984698+0000","last_became_peered":"2023-12-19T16:13:23.984698+0000","last_unstale":"2023-12-19T16:13:23.985046+0000","last_undegraded":"2023-12-19T16:13:23.985046+0000","last_fullsized":"2023-12-19T16:13:23.985046+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:36.715851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.992008+0000","last_change":"2023-12-19T16:13:22.982473+0000","last_active":"2023-12-19T16:13:23.992008+0000","last_peered":"2023-12-19T16:13:23.992008+0000","last_clean":"2023-12-19T16:13:23.992008+0000","last_became_active":"2023-12-19T16:13:22.982170+0000","last_became_peered":"2023-12-19T16:13:22.982170+0000","last_unstale":"2023-12-19T16:13:23.992008+0000","last_undegraded":"2023-12-19T16:13:23.992008+0000","last_fullsized":"2023-12-19T16:13:23.992008+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:29.812164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966316+0000","last_change":"2023-12-19T16:13:23.966316+0000","last_active":"2023-12-19T16:13:23.966316+0000","last_peered":"2023-12-19T16:13:23.966316+0000","last_clean":"2023-12-19T16:13:23.966316+0000","last_became_active":"2023-12-19T16:13:23.966025+0000","last_became_peered":"2023-12-19T16:13:23.966025+0000","last_unstale":"2023-12-19T16:13:23.966316+0000","last_undegraded":"2023-12-19T16:13:23.966316+0000","last_fullsized":"2023-12-19T16:13:23.966316+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:04.026807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.951594+0000","last_change":"2023-12-19T16:13:22.980710+0000","last_active":"2023-12-19T16:13:23.951594+0000","last_peered":"2023-12-19T16:13:23.951594+0000","last_clean":"2023-12-19T16:13:23.951594+0000","last_became_active":"2023-12-19T16:13:22.980054+0000","last_became_peered":"2023-12-19T16:13:22.980054+0000","last_unstale":"2023-12-19T16:13:23.951594+0000","last_undegraded":"2023-12-19T16:13:23.951594+0000","last_fullsized":"2023-12-19T16:13:23.951594+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:10:49.814105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977111+0000","last_change":"2023-12-19T16:13:23.977111+0000","last_active":"2023-12-19T16:13:23.977111+0000","last_peered":"2023-12-19T16:13:23.977111+0000","last_clean":"2023-12-19T16:13:23.977111+0000","last_became_active":"2023-12-19T16:13:23.976577+0000","last_became_peered":"2023-12-19T16:13:23.976577+0000","last_unstale":"2023-12-19T16:13:23.977111+0000","last_undegraded":"2023-12-19T16:13:23.977111+0000","last_fullsized":"2023-12-19T16:13:23.977111+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.972767+0000","last_change":"2023-12-19T16:13:22.969403+0000","last_active":"2023-12-19T16:13:22.972767+0000","last_peered":"2023-12-19T16:13:22.972767+0000","last_clean":"2023-12-19T16:13:22.972767+0000","last_became_active":"2023-12-19T16:13:22.969060+0000","last_became_peered":"2023-12-19T16:13:22.969060+0000","last_unstale":"2023-12-19T16:13:22.972767+0000","last_undegraded":"2023-12-19T16:13:22.972767+0000","last_fullsized":"2023-12-19T16:13:22.972767+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:16.553027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.967488+0000","last_change":"2023-12-19T16:13:23.967488+0000","last_active":"2023-12-19T16:13:23.967488+0000","last_peered":"2023-12-19T16:13:23.967488+0000","last_clean":"2023-12-19T16:13:23.967488+0000","last_became_active":"2023-12-19T16:13:23.967173+0000","last_became_peered":"2023-12-19T16:13:23.967173+0000","last_unstale":"2023-12-19T16:13:23.967488+0000","last_undegraded":"2023-12-19T16:13:23.967488+0000","last_fullsized":"2023-12-19T16:13:23.967488+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.986952+0000","last_change":"2023-12-19T16:13:22.978603+0000","last_active":"2023-12-19T16:13:23.986952+0000","last_peered":"2023-12-19T16:13:23.986952+0000","last_clean":"2023-12-19T16:13:23.986952+0000","last_became_active":"2023-12-19T16:13:22.978271+0000","last_became_peered":"2023-12-19T16:13:22.978271+0000","last_unstale":"2023-12-19T16:13:23.986952+0000","last_undegraded":"2023-12-19T16:13:23.986952+0000","last_fullsized":"2023-12-19T16:13:23.986952+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:23.923216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968520+0000","last_change":"2023-12-19T16:13:23.968520+0000","last_active":"2023-12-19T16:13:23.968520+0000","last_peered":"2023-12-19T16:13:23.968520+0000","last_clean":"2023-12-19T16:13:23.968520+0000","last_became_active":"2023-12-19T16:13:23.968173+0000","last_became_peered":"2023-12-19T16:13:23.968173+0000","last_unstale":"2023-12-19T16:13:23.968520+0000","last_undegraded":"2023-12-19T16:13:23.968520+0000","last_fullsized":"2023-12-19T16:13:23.968520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:35.816448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.987970+0000","last_change":"2023-12-19T16:13:22.986081+0000","last_active":"2023-12-19T16:13:22.987970+0000","last_peered":"2023-12-19T16:13:22.987970+0000","last_clean":"2023-12-19T16:13:22.987970+0000","last_became_active":"2023-12-19T16:13:22.985780+0000","last_became_peered":"2023-12-19T16:13:22.985780+0000","last_unstale":"2023-12-19T16:13:22.987970+0000","last_undegraded":"2023-12-19T16:13:22.987970+0000","last_fullsized":"2023-12-19T16:13:22.987970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:19:19.690490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974372+0000","last_change":"2023-12-19T16:13:23.974372+0000","last_active":"2023-12-19T16:13:23.974372+0000","last_peered":"2023-12-19T16:13:23.974372+0000","last_clean":"2023-12-19T16:13:23.974372+0000","last_became_active":"2023-12-19T16:13:23.974093+0000","last_became_peered":"2023-12-19T16:13:23.974093+0000","last_unstale":"2023-12-19T16:13:23.974372+0000","last_undegraded":"2023-12-19T16:13:23.974372+0000","last_fullsized":"2023-12-19T16:13:23.974372+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986709+0000","last_change":"2023-12-19T16:13:22.977533+0000","last_active":"2023-12-19T16:13:22.986709+0000","last_peered":"2023-12-19T16:13:22.986709+0000","last_clean":"2023-12-19T16:13:22.986709+0000","last_became_active":"2023-12-19T16:13:22.977218+0000","last_became_peered":"2023-12-19T16:13:22.977218+0000","last_unstale":"2023-12-19T16:13:22.986709+0000","last_undegraded":"2023-12-19T16:13:22.986709+0000","last_fullsized":"2023-12-19T16:13:22.986709+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:20.030474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.964250+0000","last_change":"2023-12-19T16:13:23.964250+0000","last_active":"2023-12-19T16:13:23.964250+0000","last_peered":"2023-12-19T16:13:23.964250+0000","last_clean":"2023-12-19T16:13:23.964250+0000","last_became_active":"2023-12-19T16:13:23.961324+0000","last_became_peered":"2023-12-19T16:13:23.961324+0000","last_unstale":"2023-12-19T16:13:23.964250+0000","last_undegraded":"2023-12-19T16:13:23.964250+0000","last_fullsized":"2023-12-19T16:13:23.964250+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:39.572048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.970724+0000","last_change":"2023-12-19T16:13:22.969473+0000","last_active":"2023-12-19T16:13:22.970724+0000","last_peered":"2023-12-19T16:13:22.970724+0000","last_clean":"2023-12-19T16:13:22.970724+0000","last_became_active":"2023-12-19T16:13:22.969128+0000","last_became_peered":"2023-12-19T16:13:22.969128+0000","last_unstale":"2023-12-19T16:13:22.970724+0000","last_undegraded":"2023-12-19T16:13:22.970724+0000","last_fullsized":"2023-12-19T16:13:22.970724+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:17:49.588817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975141+0000","last_change":"2023-12-19T16:13:23.975141+0000","last_active":"2023-12-19T16:13:23.975141+0000","last_peered":"2023-12-19T16:13:23.975141+0000","last_clean":"2023-12-19T16:13:23.975141+0000","last_became_active":"2023-12-19T16:13:23.974801+0000","last_became_peered":"2023-12-19T16:13:23.974801+0000","last_unstale":"2023-12-19T16:13:23.975141+0000","last_undegraded":"2023-12-19T16:13:23.975141+0000","last_fullsized":"2023-12-19T16:13:23.975141+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:37:09.093365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991585+0000","last_change":"2023-12-19T16:13:22.980663+0000","last_active":"2023-12-19T16:13:23.991585+0000","last_peered":"2023-12-19T16:13:23.991585+0000","last_clean":"2023-12-19T16:13:23.991585+0000","last_became_active":"2023-12-19T16:13:22.980227+0000","last_became_peered":"2023-12-19T16:13:22.980227+0000","last_unstale":"2023-12-19T16:13:23.991585+0000","last_undegraded":"2023-12-19T16:13:23.991585+0000","last_fullsized":"2023-12-19T16:13:23.991585+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:40:18.559957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970393+0000","last_change":"2023-12-19T16:13:23.970393+0000","last_active":"2023-12-19T16:13:23.970393+0000","last_peered":"2023-12-19T16:13:23.970393+0000","last_clean":"2023-12-19T16:13:23.970393+0000","last_became_active":"2023-12-19T16:13:23.969809+0000","last_became_peered":"2023-12-19T16:13:23.969809+0000","last_unstale":"2023-12-19T16:13:23.970393+0000","last_undegraded":"2023-12-19T16:13:23.970393+0000","last_fullsized":"2023-12-19T16:13:23.970393+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:12:46.205371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986901+0000","last_change":"2023-12-19T16:13:22.980021+0000","last_active":"2023-12-19T16:13:22.986901+0000","last_peered":"2023-12-19T16:13:22.986901+0000","last_clean":"2023-12-19T16:13:22.986901+0000","last_became_active":"2023-12-19T16:13:22.979727+0000","last_became_peered":"2023-12-19T16:13:22.979727+0000","last_unstale":"2023-12-19T16:13:22.986901+0000","last_undegraded":"2023-12-19T16:13:22.986901+0000","last_fullsized":"2023-12-19T16:13:22.986901+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:19:50.395415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.978349+0000","last_change":"2023-12-19T16:13:23.978349+0000","last_active":"2023-12-19T16:13:23.978349+0000","last_peered":"2023-12-19T16:13:23.978349+0000","last_clean":"2023-12-19T16:13:23.978349+0000","last_became_active":"2023-12-19T16:13:23.975361+0000","last_became_peered":"2023-12-19T16:13:23.975361+0000","last_unstale":"2023-12-19T16:13:23.978349+0000","last_undegraded":"2023-12-19T16:13:23.978349+0000","last_fullsized":"2023-12-19T16:13:23.978349+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981097+0000","last_change":"2023-12-19T16:13:22.972652+0000","last_active":"2023-12-19T16:13:22.981097+0000","last_peered":"2023-12-19T16:13:22.981097+0000","last_clean":"2023-12-19T16:13:22.981097+0000","last_became_active":"2023-12-19T16:13:22.971701+0000","last_became_peered":"2023-12-19T16:13:22.971701+0000","last_unstale":"2023-12-19T16:13:22.981097+0000","last_undegraded":"2023-12-19T16:13:22.981097+0000","last_fullsized":"2023-12-19T16:13:22.981097+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:52:00.342528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974813+0000","last_change":"2023-12-19T16:13:23.974813+0000","last_active":"2023-12-19T16:13:23.974813+0000","last_peered":"2023-12-19T16:13:23.974813+0000","last_clean":"2023-12-19T16:13:23.974813+0000","last_became_active":"2023-12-19T16:13:23.974504+0000","last_became_peered":"2023-12-19T16:13:23.974504+0000","last_unstale":"2023-12-19T16:13:23.974813+0000","last_undegraded":"2023-12-19T16:13:23.974813+0000","last_fullsized":"2023-12-19T16:13:23.974813+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:38:49.842475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.977917+0000","last_change":"2023-12-19T16:13:22.969654+0000","last_active":"2023-12-19T16:13:22.977917+0000","last_peered":"2023-12-19T16:13:22.977917+0000","last_clean":"2023-12-19T16:13:22.977917+0000","last_became_active":"2023-12-19T16:13:22.968326+0000","last_became_peered":"2023-12-19T16:13:22.968326+0000","last_unstale":"2023-12-19T16:13:22.977917+0000","last_undegraded":"2023-12-19T16:13:22.977917+0000","last_fullsized":"2023-12-19T16:13:22.977917+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:16.841283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.967299+0000","last_change":"2023-12-19T16:13:23.967299+0000","last_active":"2023-12-19T16:13:23.967299+0000","last_peered":"2023-12-19T16:13:23.967299+0000","last_clean":"2023-12-19T16:13:23.967299+0000","last_became_active":"2023-12-19T16:13:23.965278+0000","last_became_peered":"2023-12-19T16:13:23.965278+0000","last_unstale":"2023-12-19T16:13:23.967299+0000","last_undegraded":"2023-12-19T16:13:23.967299+0000","last_fullsized":"2023-12-19T16:13:23.967299+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:54.293501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.975200+0000","last_change":"2023-12-19T16:13:22.973975+0000","last_active":"2023-12-19T16:13:22.975200+0000","last_peered":"2023-12-19T16:13:22.975200+0000","last_clean":"2023-12-19T16:13:22.975200+0000","last_became_active":"2023-12-19T16:13:22.973655+0000","last_became_peered":"2023-12-19T16:13:22.973655+0000","last_unstale":"2023-12-19T16:13:22.975200+0000","last_undegraded":"2023-12-19T16:13:22.975200+0000","last_fullsized":"2023-12-19T16:13:22.975200+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:42:45.282864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970698+0000","last_change":"2023-12-19T16:13:23.970698+0000","last_active":"2023-12-19T16:13:23.970698+0000","last_peered":"2023-12-19T16:13:23.970698+0000","last_clean":"2023-12-19T16:13:23.970698+0000","last_became_active":"2023-12-19T16:13:23.970371+0000","last_became_peered":"2023-12-19T16:13:23.970371+0000","last_unstale":"2023-12-19T16:13:23.970698+0000","last_undegraded":"2023-12-19T16:13:23.970698+0000","last_fullsized":"2023-12-19T16:13:23.970698+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.971704+0000","last_change":"2023-12-19T16:13:22.969865+0000","last_active":"2023-12-19T16:13:22.971704+0000","last_peered":"2023-12-19T16:13:22.971704+0000","last_clean":"2023-12-19T16:13:22.971704+0000","last_became_active":"2023-12-19T16:13:22.969507+0000","last_became_peered":"2023-12-19T16:13:22.969507+0000","last_unstale":"2023-12-19T16:13:22.971704+0000","last_undegraded":"2023-12-19T16:13:22.971704+0000","last_fullsized":"2023-12-19T16:13:22.971704+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:38.021765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.982269+0000","last_change":"2023-12-19T16:13:23.982269+0000","last_active":"2023-12-19T16:13:23.982269+0000","last_peered":"2023-12-19T16:13:23.982269+0000","last_clean":"2023-12-19T16:13:23.982269+0000","last_became_active":"2023-12-19T16:13:23.971527+0000","last_became_peered":"2023-12-19T16:13:23.971527+0000","last_unstale":"2023-12-19T16:13:23.982269+0000","last_undegraded":"2023-12-19T16:13:23.982269+0000","last_fullsized":"2023-12-19T16:13:23.982269+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986766+0000","last_change":"2023-12-19T16:13:22.977822+0000","last_active":"2023-12-19T16:13:22.986766+0000","last_peered":"2023-12-19T16:13:22.986766+0000","last_clean":"2023-12-19T16:13:22.986766+0000","last_became_active":"2023-12-19T16:13:22.977507+0000","last_became_peered":"2023-12-19T16:13:22.977507+0000","last_unstale":"2023-12-19T16:13:22.986766+0000","last_undegraded":"2023-12-19T16:13:22.986766+0000","last_fullsized":"2023-12-19T16:13:22.986766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:08:24.790765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974327+0000","last_change":"2023-12-19T16:13:23.974327+0000","last_active":"2023-12-19T16:13:23.974327+0000","last_peered":"2023-12-19T16:13:23.974327+0000","last_clean":"2023-12-19T16:13:23.974327+0000","last_became_active":"2023-12-19T16:13:23.974000+0000","last_became_peered":"2023-12-19T16:13:23.974000+0000","last_unstale":"2023-12-19T16:13:23.974327+0000","last_undegraded":"2023-12-19T16:13:23.974327+0000","last_fullsized":"2023-12-19T16:13:23.974327+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:09.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.976527+0000","last_change":"2023-12-19T16:13:22.974512+0000","last_active":"2023-12-19T16:13:22.976527+0000","last_peered":"2023-12-19T16:13:22.976527+0000","last_clean":"2023-12-19T16:13:22.976527+0000","last_became_active":"2023-12-19T16:13:22.974207+0000","last_became_peered":"2023-12-19T16:13:22.974207+0000","last_unstale":"2023-12-19T16:13:22.976527+0000","last_undegraded":"2023-12-19T16:13:22.976527+0000","last_fullsized":"2023-12-19T16:13:22.976527+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:18:46.942157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.978238+0000","last_change":"2023-12-19T16:13:23.978238+0000","last_active":"2023-12-19T16:13:23.978238+0000","last_peered":"2023-12-19T16:13:23.978238+0000","last_clean":"2023-12-19T16:13:23.978238+0000","last_became_active":"2023-12-19T16:13:23.977509+0000","last_became_peered":"2023-12-19T16:13:23.977509+0000","last_unstale":"2023-12-19T16:13:23.978238+0000","last_undegraded":"2023-12-19T16:13:23.978238+0000","last_fullsized":"2023-12-19T16:13:23.978238+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:16.982107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981753+0000","last_change":"2023-12-19T16:13:22.975500+0000","last_active":"2023-12-19T16:13:22.981753+0000","last_peered":"2023-12-19T16:13:22.981753+0000","last_clean":"2023-12-19T16:13:22.981753+0000","last_became_active":"2023-12-19T16:13:22.975228+0000","last_became_peered":"2023-12-19T16:13:22.975228+0000","last_unstale":"2023-12-19T16:13:22.981753+0000","last_undegraded":"2023-12-19T16:13:22.981753+0000","last_fullsized":"2023-12-19T16:13:22.981753+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:38:45.917623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"202'348","reported_seq":1138,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.959701+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:22.959701+0000","last_peered":"2023-12-19T16:13:22.959701+0000","last_clean":"2023-12-19T16:13:22.959701+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:22.959701+0000","last_undegraded":"2023-12-19T16:13:22.959701+0000","last_fullsized":"2023-12-19T16:13:22.959701+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304560,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067280,"statfs":{"total":96636764160,"available":96324894720,"internally_reserved":0,"allocated":3252224,"data_stored":1949500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.052},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":0.98799999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":1.081},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.4670000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":1.2529999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.339}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.3029999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":1.2},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.6739999999999999}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.732},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.2270000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575972,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312176,"kb_used_data":2600,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059664,"statfs":{"total":96636764160,"available":96317095936,"internally_reserved":0,"allocated":2662400,"data_stored":1359202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.163,"15min":1.179},"min":{"1min":0.877,"5min":0.69999999999999996,"15min":0.60099999999999998},"max":{"1min":1.488,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.224,"15min":1.2390000000000001},"min":{"1min":0.89700000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.597,"5min":1.74,"15min":1.7909999999999999},"last":1.1739999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.76500000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5509999999999999,"5min":1.643,"15min":1.7010000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2609999999999999,"15min":1.274},"min":{"1min":0.89400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.8080000000000001,"15min":1.827},"last":1.075}]},{"osd":2,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.202,"15min":1.23},"min":{"1min":0.878,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.504,"5min":1.651,"15min":1.8080000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.28,"15min":1.288},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.599,"5min":1.732,"15min":1.732},"last":1.266}]},{"osd":3,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.042,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5840000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.3280000000000001},"min":{"1min":0.80200000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0309999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.212,"15min":1.2050000000000001},"min":{"1min":1.0289999999999999,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.4710000000000001,"5min":1.708,"15min":1.774},"last":1.361},{"interface":"front","average":{"1min":1.375,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":1.02,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.8140000000000001,"5min":1.966,"15min":1.966},"last":1.3400000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.296,"15min":1.3129999999999999},"min":{"1min":0.88200000000000001,"5min":0.83299999999999996,"15min":0.68000000000000005},"max":{"1min":1.7509999999999999,"5min":1.788,"15min":1.923},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.3620000000000001,"15min":1.3720000000000001},"min":{"1min":1.1519999999999999,"5min":0.94099999999999995,"15min":0.68899999999999995},"max":{"1min":1.593,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.1279999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4219999999999999,"15min":1.4350000000000001},"min":{"1min":1.101,"5min":0.92900000000000005,"15min":0.73799999999999999},"max":{"1min":1.706,"5min":1.8720000000000001,"15min":2.169},"last":1.238},{"interface":"front","average":{"1min":1.478,"5min":1.4430000000000001,"15min":1.46},"min":{"1min":0.97499999999999998,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.855,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.111}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308092,"kb_used_data":2612,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063748,"statfs":{"total":96636764160,"available":96321277952,"internally_reserved":0,"allocated":2674688,"data_stored":1360690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.101,"15min":1.1060000000000001},"min":{"1min":0.622,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.337,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.746,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.5600000000000001,"5min":1.6619999999999999,"15min":1.907},"last":0.58699999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2689999999999999,"15min":1.3100000000000001},"min":{"1min":0.86099999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6559999999999999,"5min":1.7350000000000001,"15min":1.7490000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2509999999999999,"15min":1.276},"min":{"1min":0.88400000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.552,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.92800000000000005}]},{"osd":2,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2769999999999999,"15min":1.2749999999999999},"min":{"1min":0.92900000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.641,"5min":1.9259999999999999,"15min":2.0249999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.92200000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.6639999999999999,"5min":1.7769999999999999,"15min":1.831},"last":0.88700000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2809999999999999,"15min":1.306},"min":{"1min":1.0049999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.575,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.284,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":1.024,"5min":0.89300000000000002,"15min":0.85799999999999998},"max":{"1min":1.585,"5min":1.734,"15min":2.0449999999999999},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.73999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.8},"last":1.01},{"interface":"front","average":{"1min":1.306,"5min":1.2969999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7869999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.83499999999999996}]},{"osd":6,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.385,"15min":1.3779999999999999},"min":{"1min":1.208,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.8680000000000001,"15min":1.968},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.3999999999999999,"15min":1.401},"min":{"1min":0.93999999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.724,"5min":1.7929999999999999,"15min":1.794},"last":1.0980000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4419999999999999,"15min":1.4179999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.71,"5min":2.395,"15min":2.395},"last":1.139},{"interface":"front","average":{"1min":1.429,"5min":1.411,"15min":1.4179999999999999},"min":{"1min":1.018,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8240000000000001,"5min":1.9770000000000001,"15min":1.99},"last":1.169}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1359132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.355},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.4139999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":1.012}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.244},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":1.2090000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":1.157},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":0.81699999999999995}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.484}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.391},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4490000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299892,"kb_used_data":2604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071948,"statfs":{"total":96636764160,"available":96329674752,"internally_reserved":0,"allocated":2666496,"data_stored":1359292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.90100000000000002,"15min":0.90800000000000003},"min":{"1min":0.66700000000000004,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.5660000000000001,"5min":1.633,"15min":1.7729999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.97599999999999998,"15min":1.0089999999999999},"min":{"1min":0.59599999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.367,"5min":1.5600000000000001,"15min":1.9370000000000001},"last":0.67200000000000004}]},{"osd":1,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.94099999999999995,"15min":0.96599999999999997},"min":{"1min":0.69499999999999995,"5min":0.67200000000000004,"15min":0.53900000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":2.0350000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.1459999999999999,"15min":1.125},"min":{"1min":0.75900000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5760000000000001,"5min":1.724,"15min":2.2269999999999999},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.151,"15min":1.169},"min":{"1min":0.89500000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6020000000000001,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.84599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7370000000000001,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.302,"15min":1.333},"min":{"1min":1.0820000000000001,"5min":0.754,"15min":0.754},"max":{"1min":1.6080000000000001,"5min":1.7270000000000001,"15min":2.5099999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.3200000000000001,"15min":1.3180000000000001},"min":{"1min":1.052,"5min":0.96199999999999997,"15min":0.72199999999999998},"max":{"1min":1.788,"5min":1.788,"15min":2.3450000000000002},"last":1.3660000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.367,"15min":1.3640000000000001},"min":{"1min":1.117,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.133},{"interface":"front","average":{"1min":1.5,"5min":1.4470000000000001,"15min":1.4379999999999999},"min":{"1min":1.105,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.508,"15min":1.516},"min":{"1min":1.194,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.04},{"interface":"front","average":{"1min":1.516,"5min":1.4419999999999999,"15min":1.4670000000000001},"min":{"1min":1.054,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9970000000000001,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.5409999999999999,"15min":1.5600000000000001},"min":{"1min":1.234,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":0.87},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.494,"15min":1.512},"min":{"1min":1.248,"5min":1.018,"15min":1.018},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.244}]}]},{"osd":2,"up_from":13,"seq":55834575973,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1358675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.95499999999999996,"15min":0.96499999999999997},"min":{"1min":0.73899999999999999,"5min":0.61899999999999999,"15min":0.54600000000000004},"max":{"1min":1.2070000000000001,"5min":1.4510000000000001,"15min":1.581},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.157,"15min":1.151},"min":{"1min":0.874,"5min":0.79900000000000004,"15min":0.66200000000000003},"max":{"1min":1.339,"5min":1.629,"15min":1.796},"last":0.97099999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.194,"15min":1.1870000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7090000000000001,"5min":2.327,"15min":2.327},"last":1.274},{"interface":"front","average":{"1min":1.234,"5min":1.169,"15min":1.143},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.621},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0289999999999999,"15min":1.0409999999999999},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.284,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.141,"15min":1.149},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5169999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.66500000000000004}]},{"osd":4,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4550000000000001,"15min":1.444},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.6839999999999999,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4670000000000001,"15min":1.4510000000000001},"min":{"1min":1.002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6839999999999999,"5min":2.218,"15min":2.218},"last":1.377}]},{"osd":5,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.4399999999999999,"15min":1.425},"min":{"1min":1.1000000000000001,"5min":1.0660000000000001,"15min":0.91600000000000004},"max":{"1min":1.901,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.431,"15min":1.411},"min":{"1min":0.96099999999999997,"5min":0.879,"15min":0.879},"max":{"1min":1.841,"5min":2.0139999999999998,"15min":2.4340000000000002},"last":1.234}]},{"osd":6,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4159999999999999,"15min":1.407},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5669999999999999,"15min":1.5389999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.071},"max":{"1min":1.8029999999999999,"5min":2.609,"15min":2.609},"last":1.351}]},{"osd":7,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5820000000000001,"15min":1.5609999999999999},"min":{"1min":1.2350000000000001,"5min":1.1479999999999999,"15min":0.89600000000000002},"max":{"1min":1.8440000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.323},{"interface":"front","average":{"1min":1.643,"5min":1.5669999999999999,"15min":1.548},"min":{"1min":1.2729999999999999,"5min":1.028,"15min":1.028},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.1899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1359132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.90800000000000003},"min":{"1min":0.55800000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.351,"5min":1.6559999999999999,"15min":1.6579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0129999999999999,"15min":1.0369999999999999},"min":{"1min":0.77600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.46,"5min":1.548,"15min":1.9119999999999999},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.05,"15min":1.069},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.149,"5min":1.1240000000000001,"15min":1.161},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.63900000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2809999999999999,"15min":1.2869999999999999},"min":{"1min":0.88800000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7330000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.107,"15min":1.1060000000000001},"min":{"1min":0.72899999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4099999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.93500000000000005}]},{"osd":4,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.306,"15min":1.331},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.903,"5min":1.903,"15min":2.1459999999999999},"last":1.214},{"interface":"front","average":{"1min":1.423,"5min":1.397,"15min":1.427},"min":{"1min":1.1659999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.0110000000000001},"last":1.014}]},{"osd":5,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.325,"15min":1.345},"min":{"1min":1.0129999999999999,"5min":0.86399999999999999,"15min":0.82099999999999995},"max":{"1min":1.5109999999999999,"5min":1.833,"15min":1.9159999999999999},"last":1.167},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4430000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.323},"last":1.0700000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3799999999999999,"15min":1.4039999999999999},"min":{"1min":1.109,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5920000000000001,"5min":1.972,"15min":2.1139999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4950000000000001,"15min":1.518},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2400000000000002},"last":1.327}]},{"osd":7,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.52,"15min":1.5349999999999999},"min":{"1min":1.0900000000000001,"5min":1.026,"15min":1.026},"max":{"1min":1.905,"5min":2.2010000000000001,"15min":2.2229999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.468,"15min":1.496},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.7150000000000001,"5min":2.0670000000000002,"15min":2.3170000000000002},"last":1.383}]}]},{"osd":0,"up_from":13,"seq":55834575971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304572,"kb_used_data":3188,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067268,"statfs":{"total":96636764160,"available":96324882432,"internally_reserved":0,"allocated":3264512,"data_stored":1949189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90600000000000003,"15min":0.91200000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.439},"max":{"1min":1.3500000000000001,"5min":1.4530000000000001,"15min":1.6419999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0049999999999999,"15min":1.012},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.165,"15min":1.145},"min":{"1min":0.72299999999999998,"5min":0.68300000000000005,"15min":0.57099999999999995},"max":{"1min":1.536,"5min":1.748,"15min":1.8400000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.244,"15min":1.272},"min":{"1min":0.75800000000000001,"5min":0.66200000000000003,"15min":0.54300000000000004},"max":{"1min":1.633,"5min":1.655,"15min":1.8500000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.036,"15min":1.046},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1919999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.873},{"interface":"front","average":{"1min":1.175,"5min":1.2490000000000001,"15min":1.2549999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.55300000000000005},"max":{"1min":1.609,"5min":1.974,"15min":1.974},"last":1.385}]},{"osd":4,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3859999999999999,"15min":1.3899999999999999},"min":{"1min":1.01,"5min":1.01,"15min":0.69499999999999995},"max":{"1min":1.6699999999999999,"5min":1.885,"15min":2.0209999999999999},"last":1},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.367,"15min":1.3360000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.71999999999999997},"max":{"1min":1.6599999999999999,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.103}]},{"osd":5,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4510000000000001,"15min":1.4390000000000001},"min":{"1min":1.103,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.8839999999999999,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.548},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4530000000000001,"15min":1.4590000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.919,"5min":1.919,"15min":1.9810000000000001},"last":1.238}]},{"osd":6,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.4119999999999999,"15min":1.417},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.577,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.254},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.5329999999999999,"15min":1.516},"min":{"1min":1.1180000000000001,"5min":1.056,"15min":0.93899999999999995},"max":{"1min":1.998,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.6459999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.54,"15min":1.5309999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.776,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.21},{"interface":"front","average":{"1min":1.363,"5min":1.5,"15min":1.5049999999999999},"min":{"1min":1.113,"5min":1.1080000000000001,"15min":1.008},"max":{"1min":1.734,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.298}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:13:28.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:13:28.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.467+0000 7fbec7f85700 1 Processor -- start 2023-12-19T16:13:28.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 -- start start 2023-12-19T16:13:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec015d320 0x7fbec015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbec00aeba0 0x7fbec00680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00c7190 con 0x7fbec00c15b0 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00c1b50 con 0x7fbec00aeba0 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec7f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00c1e30 con 0x7fbec015d320 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec5520700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbec00c15b0 0x7fbec006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41236/0 (socket says 172.21.15.50:41236) 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec5520700 1 -- 172.21.15.50:0/3030962517 learned_addr learned my addr 172.21.15.50:0/3030962517 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec6522700 1 --2- 172.21.15.50:0/3030962517 >> v2:172.21.15.50:3300/0 conn(0x7fbec00aeba0 0x7fbec00680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.471+0000 7fbec5d21700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec015d320 0x7fbec015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec6522700 1 -- 172.21.15.50:0/3030962517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec015d320 msgr2=0x7fbec015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec6522700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec015d320 0x7fbec015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec6522700 1 -- 172.21.15.50:0/3030962517 >> v1:172.21.15.50:6789/0 conn(0x7fbec00c15b0 legacy=0x7fbec006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec6522700 1 -- 172.21.15.50:0/3030962517 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00c6e20 con 0x7fbec00aeba0 2023-12-19T16:13:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec6522700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00aeba0 0x7fbec00680f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009e60 tx=0x7fbeb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9ad5be5e25ba6b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec4d1f700 1 -- 172.21.15.50:0/3030962517 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb800e070 con 0x7fbec00aeba0 2023-12-19T16:13:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec4d1f700 1 -- 172.21.15.50:0/3030962517 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800bc30 con 0x7fbec00aeba0 2023-12-19T16:13:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec4d1f700 1 -- 172.21.15.50:0/3030962517 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8012430 con 0x7fbec00aeba0 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 -- 172.21.15.50:0/3030962517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00aeba0 msgr2=0x7fbec00680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00aeba0 0x7fbec00680f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009e60 tx=0x7fbeb8009780 comp rx=0 tx=0).stop 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 -- 172.21.15.50:0/3030962517 shutdown_connections 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00aeba0 0x7fbec00680f0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 --2- 172.21.15.50:0/3030962517 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec015d320 0x7fbec015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 -- 172.21.15.50:0/3030962517 >> 172.21.15.50:0/3030962517 conn(0x7fbec00c8850 msgr2=0x7fbec00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:28.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 -- 172.21.15.50:0/3030962517 shutdown_connections 2023-12-19T16:13:28.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.475+0000 7fbec7f85700 1 -- 172.21.15.50:0/3030962517 wait complete. 2023-12-19T16:13:28.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 Processor -- start 2023-12-19T16:13:28.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 -- start start 2023-12-19T16:13:28.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec00aeba0 0x7fbec016e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 0x7fbec016ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec016f3c0 con 0x7fbec00c15b0 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec016e7b0 con 0x7fbec015d320 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.479+0000 7fbec7f85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec016ea90 con 0x7fbec00aeba0 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60702/0 (socket says 172.21.15.50:60702) 2023-12-19T16:13:28.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 -- 172.21.15.50:0/4047521450 learned_addr learned my addr 172.21.15.50:0/4047521450 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec6522700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 0x7fbec016ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5d21700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec00aeba0 0x7fbec016e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 -- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec00aeba0 msgr2=0x7fbec016e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec00aeba0 0x7fbec016e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 -- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 msgr2=0x7fbec016ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 0x7fbec016ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 -- 172.21.15.50:0/4047521450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb8009a20 con 0x7fbec00c15b0 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec6522700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 0x7fbec016ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:28.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec5520700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c7c0 tx=0x7fbeb0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0005b30 con 0x7fbec00c15b0 2023-12-19T16:13:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0005c90 con 0x7fbec00c15b0 2023-12-19T16:13:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0004a80 con 0x7fbec00c15b0 2023-12-19T16:13:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec016ed70 con 0x7fbec00c15b0 2023-12-19T16:13:28.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.483+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec0162da0 con 0x7fbec00c15b0 2023-12-19T16:13:28.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.487+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec016f530 con 0x7fbec00c15b0 2023-12-19T16:13:28.491 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.487+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbeb00070d0 con 0x7fbec00c15b0 2023-12-19T16:13:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.487+0000 7fbeb5ffb700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 0x7fbea4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.487+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fbeb007bec0 con 0x7fbec00c15b0 2023-12-19T16:13:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.491+0000 7fbec5d21700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 0x7fbea4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.491+0000 7fbec5d21700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 0x7fbea4062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006fd0 tx=0x7fbebc008040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.498 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.495+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbeb0049be0 con 0x7fbec00c15b0 2023-12-19T16:13:28.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.659+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbec0071120 con 0x7fbea4060670 2023-12-19T16:13:28.679 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.675+0000 7fbeb5ffb700 1 -- 172.21.15.50:0/4047521450 <== mgr.10012 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286655 (secure 0 0 0) 0x7fbec0071120 con 0x7fbea4060670 2023-12-19T16:13:28.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 msgr2=0x7fbea4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 0x7fbea4062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006fd0 tx=0x7fbebc008040 comp rx=0 tx=0).stop 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 msgr2=0x7fbec0162860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c7c0 tx=0x7fbeb0007f70 comp rx=0 tx=0).stop 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 shutdown_connections 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbea4060670 0x7fbea4062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.689 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbec015d320 0x7fbec016ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbec00c15b0 0x7fbec0162860 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 --2- 172.21.15.50:0/4047521450 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbec00aeba0 0x7fbec016e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 >> 172.21.15.50:0/4047521450 conn(0x7fbec00c8850 msgr2=0x7fbec006a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 shutdown_connections 2023-12-19T16:13:28.690 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.683+0000 7fbec7f85700 1 -- 172.21.15.50:0/4047521450 wait complete. 2023-12-19T16:13:28.691 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:13:28.709 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T16:13:26.856972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2465808,"kb_used_data":21968,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752508912,"statfs":{"total":773094113280,"available":770569125888,"internally_reserved":0,"allocated":22495232,"data_stored":12054812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.925038"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970036+0000","last_change":"2023-12-19T16:13:23.970036+0000","last_active":"2023-12-19T16:13:23.970036+0000","last_peered":"2023-12-19T16:13:23.970036+0000","last_clean":"2023-12-19T16:13:23.970036+0000","last_became_active":"2023-12-19T16:13:23.968273+0000","last_became_peered":"2023-12-19T16:13:23.968273+0000","last_unstale":"2023-12-19T16:13:23.970036+0000","last_undegraded":"2023-12-19T16:13:23.970036+0000","last_fullsized":"2023-12-19T16:13:23.970036+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:03.353242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981922+0000","last_change":"2023-12-19T16:13:23.981922+0000","last_active":"2023-12-19T16:13:23.981922+0000","last_peered":"2023-12-19T16:13:23.981922+0000","last_clean":"2023-12-19T16:13:23.981922+0000","last_became_active":"2023-12-19T16:13:23.981536+0000","last_became_peered":"2023-12-19T16:13:23.981536+0000","last_unstale":"2023-12-19T16:13:23.981922+0000","last_undegraded":"2023-12-19T16:13:23.981922+0000","last_fullsized":"2023-12-19T16:13:23.981922+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:16.148355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.962573+0000","last_change":"2023-12-19T16:13:23.962573+0000","last_active":"2023-12-19T16:13:23.962573+0000","last_peered":"2023-12-19T16:13:23.962573+0000","last_clean":"2023-12-19T16:13:23.962573+0000","last_became_active":"2023-12-19T16:13:23.962201+0000","last_became_peered":"2023-12-19T16:13:23.962201+0000","last_unstale":"2023-12-19T16:13:23.962573+0000","last_undegraded":"2023-12-19T16:13:23.962573+0000","last_fullsized":"2023-12-19T16:13:23.962573+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:24.937995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973637+0000","last_change":"2023-12-19T16:13:23.973637+0000","last_active":"2023-12-19T16:13:23.973637+0000","last_peered":"2023-12-19T16:13:23.973637+0000","last_clean":"2023-12-19T16:13:23.973637+0000","last_became_active":"2023-12-19T16:13:23.973331+0000","last_became_peered":"2023-12-19T16:13:23.973331+0000","last_unstale":"2023-12-19T16:13:23.973637+0000","last_undegraded":"2023-12-19T16:13:23.973637+0000","last_fullsized":"2023-12-19T16:13:23.973637+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:36.715851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.979869+0000","last_change":"2023-12-19T16:13:23.979869+0000","last_active":"2023-12-19T16:13:23.979869+0000","last_peered":"2023-12-19T16:13:23.979869+0000","last_clean":"2023-12-19T16:13:23.979869+0000","last_became_active":"2023-12-19T16:13:23.978150+0000","last_became_peered":"2023-12-19T16:13:23.978150+0000","last_unstale":"2023-12-19T16:13:23.979869+0000","last_undegraded":"2023-12-19T16:13:23.979869+0000","last_fullsized":"2023-12-19T16:13:23.979869+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981696+0000","last_change":"2023-12-19T16:13:23.981696+0000","last_active":"2023-12-19T16:13:23.981696+0000","last_peered":"2023-12-19T16:13:23.981696+0000","last_clean":"2023-12-19T16:13:23.981696+0000","last_became_active":"2023-12-19T16:13:23.974418+0000","last_became_peered":"2023-12-19T16:13:23.974418+0000","last_unstale":"2023-12-19T16:13:23.981696+0000","last_undegraded":"2023-12-19T16:13:23.981696+0000","last_fullsized":"2023-12-19T16:13:23.981696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:27:42.410785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974635+0000","last_change":"2023-12-19T16:13:23.974635+0000","last_active":"2023-12-19T16:13:23.974635+0000","last_peered":"2023-12-19T16:13:23.974635+0000","last_clean":"2023-12-19T16:13:23.974635+0000","last_became_active":"2023-12-19T16:13:23.974292+0000","last_became_peered":"2023-12-19T16:13:23.974292+0000","last_unstale":"2023-12-19T16:13:23.974635+0000","last_undegraded":"2023-12-19T16:13:23.974635+0000","last_fullsized":"2023-12-19T16:13:23.974635+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:40:20.044406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970147+0000","last_change":"2023-12-19T16:13:23.970147+0000","last_active":"2023-12-19T16:13:23.970147+0000","last_peered":"2023-12-19T16:13:23.970147+0000","last_clean":"2023-12-19T16:13:23.970147+0000","last_became_active":"2023-12-19T16:13:23.969859+0000","last_became_peered":"2023-12-19T16:13:23.969859+0000","last_unstale":"2023-12-19T16:13:23.970147+0000","last_undegraded":"2023-12-19T16:13:23.970147+0000","last_fullsized":"2023-12-19T16:13:23.970147+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976891+0000","last_change":"2023-12-19T16:13:23.976891+0000","last_active":"2023-12-19T16:13:23.976891+0000","last_peered":"2023-12-19T16:13:23.976891+0000","last_clean":"2023-12-19T16:13:23.976891+0000","last_became_active":"2023-12-19T16:13:23.976182+0000","last_became_peered":"2023-12-19T16:13:23.976182+0000","last_unstale":"2023-12-19T16:13:23.976891+0000","last_undegraded":"2023-12-19T16:13:23.976891+0000","last_fullsized":"2023-12-19T16:13:23.976891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:24.937467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.985668+0000","last_change":"2023-12-19T16:13:23.985668+0000","last_active":"2023-12-19T16:13:23.985668+0000","last_peered":"2023-12-19T16:13:23.985668+0000","last_clean":"2023-12-19T16:13:23.985668+0000","last_became_active":"2023-12-19T16:13:23.985248+0000","last_became_peered":"2023-12-19T16:13:23.985248+0000","last_unstale":"2023-12-19T16:13:23.985668+0000","last_undegraded":"2023-12-19T16:13:23.985668+0000","last_fullsized":"2023-12-19T16:13:23.985668+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:40:10.806959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970992+0000","last_change":"2023-12-19T16:13:23.970992+0000","last_active":"2023-12-19T16:13:23.970992+0000","last_peered":"2023-12-19T16:13:23.970992+0000","last_clean":"2023-12-19T16:13:23.970992+0000","last_became_active":"2023-12-19T16:13:23.970667+0000","last_became_peered":"2023-12-19T16:13:23.970667+0000","last_unstale":"2023-12-19T16:13:23.970992+0000","last_undegraded":"2023-12-19T16:13:23.970992+0000","last_fullsized":"2023-12-19T16:13:23.970992+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973646+0000","last_change":"2023-12-19T16:13:23.973646+0000","last_active":"2023-12-19T16:13:23.973646+0000","last_peered":"2023-12-19T16:13:23.973646+0000","last_clean":"2023-12-19T16:13:23.973646+0000","last_became_active":"2023-12-19T16:13:23.973354+0000","last_became_peered":"2023-12-19T16:13:23.973354+0000","last_unstale":"2023-12-19T16:13:23.973646+0000","last_undegraded":"2023-12-19T16:13:23.973646+0000","last_fullsized":"2023-12-19T16:13:23.973646+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968470+0000","last_change":"2023-12-19T16:13:23.968470+0000","last_active":"2023-12-19T16:13:23.968470+0000","last_peered":"2023-12-19T16:13:23.968470+0000","last_clean":"2023-12-19T16:13:23.968470+0000","last_became_active":"2023-12-19T16:13:23.968194+0000","last_became_peered":"2023-12-19T16:13:23.968194+0000","last_unstale":"2023-12-19T16:13:23.968470+0000","last_undegraded":"2023-12-19T16:13:23.968470+0000","last_fullsized":"2023-12-19T16:13:23.968470+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.960911+0000","last_change":"2023-12-19T16:13:23.960911+0000","last_active":"2023-12-19T16:13:23.960911+0000","last_peered":"2023-12-19T16:13:23.960911+0000","last_clean":"2023-12-19T16:13:23.960911+0000","last_became_active":"2023-12-19T16:13:23.960608+0000","last_became_peered":"2023-12-19T16:13:23.960608+0000","last_unstale":"2023-12-19T16:13:23.960911+0000","last_undegraded":"2023-12-19T16:13:23.960911+0000","last_fullsized":"2023-12-19T16:13:23.960911+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:59.167915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975515+0000","last_change":"2023-12-19T16:13:23.975515+0000","last_active":"2023-12-19T16:13:23.975515+0000","last_peered":"2023-12-19T16:13:23.975515+0000","last_clean":"2023-12-19T16:13:23.975515+0000","last_became_active":"2023-12-19T16:13:23.975138+0000","last_became_peered":"2023-12-19T16:13:23.975138+0000","last_unstale":"2023-12-19T16:13:23.975515+0000","last_undegraded":"2023-12-19T16:13:23.975515+0000","last_fullsized":"2023-12-19T16:13:23.975515+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973912+0000","last_change":"2023-12-19T16:13:23.973912+0000","last_active":"2023-12-19T16:13:23.973912+0000","last_peered":"2023-12-19T16:13:23.973912+0000","last_clean":"2023-12-19T16:13:23.973912+0000","last_became_active":"2023-12-19T16:13:23.972177+0000","last_became_peered":"2023-12-19T16:13:23.972177+0000","last_unstale":"2023-12-19T16:13:23.973912+0000","last_undegraded":"2023-12-19T16:13:23.973912+0000","last_fullsized":"2023-12-19T16:13:23.973912+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:40:49.211205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975376+0000","last_change":"2023-12-19T16:13:23.975376+0000","last_active":"2023-12-19T16:13:23.975376+0000","last_peered":"2023-12-19T16:13:23.975376+0000","last_clean":"2023-12-19T16:13:23.975376+0000","last_became_active":"2023-12-19T16:13:23.974864+0000","last_became_peered":"2023-12-19T16:13:23.974864+0000","last_unstale":"2023-12-19T16:13:23.975376+0000","last_undegraded":"2023-12-19T16:13:23.975376+0000","last_fullsized":"2023-12-19T16:13:23.975376+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:04.026807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968085+0000","last_change":"2023-12-19T16:13:23.968085+0000","last_active":"2023-12-19T16:13:23.968085+0000","last_peered":"2023-12-19T16:13:23.968085+0000","last_clean":"2023-12-19T16:13:23.968085+0000","last_became_active":"2023-12-19T16:13:23.967731+0000","last_became_peered":"2023-12-19T16:13:23.967731+0000","last_unstale":"2023-12-19T16:13:23.968085+0000","last_undegraded":"2023-12-19T16:13:23.968085+0000","last_fullsized":"2023-12-19T16:13:23.968085+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966377+0000","last_change":"2023-12-19T16:13:23.966377+0000","last_active":"2023-12-19T16:13:23.966377+0000","last_peered":"2023-12-19T16:13:23.966377+0000","last_clean":"2023-12-19T16:13:23.966377+0000","last_became_active":"2023-12-19T16:13:23.966057+0000","last_became_peered":"2023-12-19T16:13:23.966057+0000","last_unstale":"2023-12-19T16:13:23.966377+0000","last_undegraded":"2023-12-19T16:13:23.966377+0000","last_fullsized":"2023-12-19T16:13:23.966377+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:51.122504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976050+0000","last_change":"2023-12-19T16:13:23.976050+0000","last_active":"2023-12-19T16:13:23.976050+0000","last_peered":"2023-12-19T16:13:23.976050+0000","last_clean":"2023-12-19T16:13:23.976050+0000","last_became_active":"2023-12-19T16:13:23.975540+0000","last_became_peered":"2023-12-19T16:13:23.975540+0000","last_unstale":"2023-12-19T16:13:23.976050+0000","last_undegraded":"2023-12-19T16:13:23.976050+0000","last_fullsized":"2023-12-19T16:13:23.976050+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:50:51.076791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977066+0000","last_change":"2023-12-19T16:13:23.977066+0000","last_active":"2023-12-19T16:13:23.977066+0000","last_peered":"2023-12-19T16:13:23.977066+0000","last_clean":"2023-12-19T16:13:23.977066+0000","last_became_active":"2023-12-19T16:13:23.976715+0000","last_became_peered":"2023-12-19T16:13:23.976715+0000","last_unstale":"2023-12-19T16:13:23.977066+0000","last_undegraded":"2023-12-19T16:13:23.977066+0000","last_fullsized":"2023-12-19T16:13:23.977066+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:35:54.061285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.981074+0000","last_change":"2023-12-19T16:13:23.981074+0000","last_active":"2023-12-19T16:13:23.981074+0000","last_peered":"2023-12-19T16:13:23.981074+0000","last_clean":"2023-12-19T16:13:23.981074+0000","last_became_active":"2023-12-19T16:13:23.978102+0000","last_became_peered":"2023-12-19T16:13:23.978102+0000","last_unstale":"2023-12-19T16:13:23.981074+0000","last_undegraded":"2023-12-19T16:13:23.981074+0000","last_fullsized":"2023-12-19T16:13:23.981074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:52:01.357308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974849+0000","last_change":"2023-12-19T16:13:23.974849+0000","last_active":"2023-12-19T16:13:23.974849+0000","last_peered":"2023-12-19T16:13:23.974849+0000","last_clean":"2023-12-19T16:13:23.974849+0000","last_became_active":"2023-12-19T16:13:23.973941+0000","last_became_peered":"2023-12-19T16:13:23.973941+0000","last_unstale":"2023-12-19T16:13:23.974849+0000","last_undegraded":"2023-12-19T16:13:23.974849+0000","last_fullsized":"2023-12-19T16:13:23.974849+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:53:46.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.979783+0000","last_change":"2023-12-19T16:13:23.979783+0000","last_active":"2023-12-19T16:13:23.979783+0000","last_peered":"2023-12-19T16:13:23.979783+0000","last_clean":"2023-12-19T16:13:23.979783+0000","last_became_active":"2023-12-19T16:13:23.979498+0000","last_became_peered":"2023-12-19T16:13:23.979498+0000","last_unstale":"2023-12-19T16:13:23.979783+0000","last_undegraded":"2023-12-19T16:13:23.979783+0000","last_fullsized":"2023-12-19T16:13:23.979783+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:31.025793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972134+0000","last_change":"2023-12-19T16:13:23.972134+0000","last_active":"2023-12-19T16:13:23.972134+0000","last_peered":"2023-12-19T16:13:23.972134+0000","last_clean":"2023-12-19T16:13:23.972134+0000","last_became_active":"2023-12-19T16:13:23.971794+0000","last_became_peered":"2023-12-19T16:13:23.971794+0000","last_unstale":"2023-12-19T16:13:23.972134+0000","last_undegraded":"2023-12-19T16:13:23.972134+0000","last_fullsized":"2023-12-19T16:13:23.972134+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:10:40.239434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973230+0000","last_change":"2023-12-19T16:13:23.973230+0000","last_active":"2023-12-19T16:13:23.973230+0000","last_peered":"2023-12-19T16:13:23.973230+0000","last_clean":"2023-12-19T16:13:23.973230+0000","last_became_active":"2023-12-19T16:13:23.972890+0000","last_became_peered":"2023-12-19T16:13:23.972890+0000","last_unstale":"2023-12-19T16:13:23.973230+0000","last_undegraded":"2023-12-19T16:13:23.973230+0000","last_fullsized":"2023-12-19T16:13:23.973230+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972074+0000","last_change":"2023-12-19T16:13:23.972074+0000","last_active":"2023-12-19T16:13:23.972074+0000","last_peered":"2023-12-19T16:13:23.972074+0000","last_clean":"2023-12-19T16:13:23.972074+0000","last_became_active":"2023-12-19T16:13:23.971730+0000","last_became_peered":"2023-12-19T16:13:23.971730+0000","last_unstale":"2023-12-19T16:13:23.972074+0000","last_undegraded":"2023-12-19T16:13:23.972074+0000","last_fullsized":"2023-12-19T16:13:23.972074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.973455+0000","last_change":"2023-12-19T16:13:23.973455+0000","last_active":"2023-12-19T16:13:23.973455+0000","last_peered":"2023-12-19T16:13:23.973455+0000","last_clean":"2023-12-19T16:13:23.973455+0000","last_became_active":"2023-12-19T16:13:23.973172+0000","last_became_peered":"2023-12-19T16:13:23.973172+0000","last_unstale":"2023-12-19T16:13:23.973455+0000","last_undegraded":"2023-12-19T16:13:23.973455+0000","last_fullsized":"2023-12-19T16:13:23.973455+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:08:12.544870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974712+0000","last_change":"2023-12-19T16:13:23.974712+0000","last_active":"2023-12-19T16:13:23.974712+0000","last_peered":"2023-12-19T16:13:23.974712+0000","last_clean":"2023-12-19T16:13:23.974712+0000","last_became_active":"2023-12-19T16:13:23.974313+0000","last_became_peered":"2023-12-19T16:13:23.974313+0000","last_unstale":"2023-12-19T16:13:23.974712+0000","last_undegraded":"2023-12-19T16:13:23.974712+0000","last_fullsized":"2023-12-19T16:13:23.974712+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:27:42.410785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975220+0000","last_change":"2023-12-19T16:13:23.975220+0000","last_active":"2023-12-19T16:13:23.975220+0000","last_peered":"2023-12-19T16:13:23.975220+0000","last_clean":"2023-12-19T16:13:23.975220+0000","last_became_active":"2023-12-19T16:13:23.974852+0000","last_became_peered":"2023-12-19T16:13:23.974852+0000","last_unstale":"2023-12-19T16:13:23.975220+0000","last_undegraded":"2023-12-19T16:13:23.975220+0000","last_fullsized":"2023-12-19T16:13:23.975220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:50:51.076791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970695+0000","last_change":"2023-12-19T16:13:23.970695+0000","last_active":"2023-12-19T16:13:23.970695+0000","last_peered":"2023-12-19T16:13:23.970695+0000","last_clean":"2023-12-19T16:13:23.970695+0000","last_became_active":"2023-12-19T16:13:23.970298+0000","last_became_peered":"2023-12-19T16:13:23.970298+0000","last_unstale":"2023-12-19T16:13:23.970695+0000","last_undegraded":"2023-12-19T16:13:23.970695+0000","last_fullsized":"2023-12-19T16:13:23.970695+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:40:57.360787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.971241+0000","last_change":"2023-12-19T16:13:23.971241+0000","last_active":"2023-12-19T16:13:23.971241+0000","last_peered":"2023-12-19T16:13:23.971241+0000","last_clean":"2023-12-19T16:13:23.971241+0000","last_became_active":"2023-12-19T16:13:23.970830+0000","last_became_peered":"2023-12-19T16:13:23.970830+0000","last_unstale":"2023-12-19T16:13:23.971241+0000","last_undegraded":"2023-12-19T16:13:23.971241+0000","last_fullsized":"2023-12-19T16:13:23.971241+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:35.816448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970418+0000","last_change":"2023-12-19T16:13:23.970418+0000","last_active":"2023-12-19T16:13:23.970418+0000","last_peered":"2023-12-19T16:13:23.970418+0000","last_clean":"2023-12-19T16:13:23.970418+0000","last_became_active":"2023-12-19T16:13:23.970017+0000","last_became_peered":"2023-12-19T16:13:23.970017+0000","last_unstale":"2023-12-19T16:13:23.970418+0000","last_undegraded":"2023-12-19T16:13:23.970418+0000","last_fullsized":"2023-12-19T16:13:23.970418+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"212'4","reported_seq":13,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.993588+0000","last_change":"2023-12-19T16:13:22.985592+0000","last_active":"2023-12-19T16:13:23.993588+0000","last_peered":"2023-12-19T16:13:23.993588+0000","last_clean":"2023-12-19T16:13:23.993588+0000","last_became_active":"2023-12-19T16:13:22.985029+0000","last_became_peered":"2023-12-19T16:13:22.985029+0000","last_unstale":"2023-12-19T16:13:23.993588+0000","last_undegraded":"2023-12-19T16:13:23.993588+0000","last_fullsized":"2023-12-19T16:13:23.993588+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:27:10.353292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.965087+0000","last_change":"2023-12-19T16:13:23.965087+0000","last_active":"2023-12-19T16:13:23.965087+0000","last_peered":"2023-12-19T16:13:23.965087+0000","last_clean":"2023-12-19T16:13:23.965087+0000","last_became_active":"2023-12-19T16:13:23.963517+0000","last_became_peered":"2023-12-19T16:13:23.963517+0000","last_unstale":"2023-12-19T16:13:23.965087+0000","last_undegraded":"2023-12-19T16:13:23.965087+0000","last_fullsized":"2023-12-19T16:13:23.965087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:53.856584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966263+0000","last_change":"2023-12-19T16:13:22.971877+0000","last_active":"2023-12-19T16:13:23.966263+0000","last_peered":"2023-12-19T16:13:23.966263+0000","last_clean":"2023-12-19T16:13:23.966263+0000","last_became_active":"2023-12-19T16:13:22.971379+0000","last_became_peered":"2023-12-19T16:13:22.971379+0000","last_unstale":"2023-12-19T16:13:23.966263+0000","last_undegraded":"2023-12-19T16:13:23.966263+0000","last_fullsized":"2023-12-19T16:13:23.966263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:15.133576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970699+0000","last_change":"2023-12-19T16:13:23.970699+0000","last_active":"2023-12-19T16:13:23.970699+0000","last_peered":"2023-12-19T16:13:23.970699+0000","last_clean":"2023-12-19T16:13:23.970699+0000","last_became_active":"2023-12-19T16:13:23.970380+0000","last_became_peered":"2023-12-19T16:13:23.970380+0000","last_unstale":"2023-12-19T16:13:23.970699+0000","last_undegraded":"2023-12-19T16:13:23.970699+0000","last_fullsized":"2023-12-19T16:13:23.970699+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:40:49.211205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.989216+0000","last_change":"2023-12-19T16:13:22.968588+0000","last_active":"2023-12-19T16:13:23.989216+0000","last_peered":"2023-12-19T16:13:23.989216+0000","last_clean":"2023-12-19T16:13:23.989216+0000","last_became_active":"2023-12-19T16:13:22.968279+0000","last_became_peered":"2023-12-19T16:13:22.968279+0000","last_unstale":"2023-12-19T16:13:23.989216+0000","last_undegraded":"2023-12-19T16:13:23.989216+0000","last_fullsized":"2023-12-19T16:13:23.989216+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:23.922688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.969754+0000","last_change":"2023-12-19T16:13:23.969754+0000","last_active":"2023-12-19T16:13:23.969754+0000","last_peered":"2023-12-19T16:13:23.969754+0000","last_clean":"2023-12-19T16:13:23.969754+0000","last_became_active":"2023-12-19T16:13:23.969457+0000","last_became_peered":"2023-12-19T16:13:23.969457+0000","last_unstale":"2023-12-19T16:13:23.969754+0000","last_undegraded":"2023-12-19T16:13:23.969754+0000","last_fullsized":"2023-12-19T16:13:23.969754+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:34:28.820521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.986316+0000","last_change":"2023-12-19T16:13:22.975829+0000","last_active":"2023-12-19T16:13:23.986316+0000","last_peered":"2023-12-19T16:13:23.986316+0000","last_clean":"2023-12-19T16:13:23.986316+0000","last_became_active":"2023-12-19T16:13:22.975534+0000","last_became_peered":"2023-12-19T16:13:22.975534+0000","last_unstale":"2023-12-19T16:13:23.986316+0000","last_undegraded":"2023-12-19T16:13:23.986316+0000","last_fullsized":"2023-12-19T16:13:23.986316+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:38.021765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975561+0000","last_change":"2023-12-19T16:13:23.975561+0000","last_active":"2023-12-19T16:13:23.975561+0000","last_peered":"2023-12-19T16:13:23.975561+0000","last_clean":"2023-12-19T16:13:23.975561+0000","last_became_active":"2023-12-19T16:13:23.972636+0000","last_became_peered":"2023-12-19T16:13:23.972636+0000","last_unstale":"2023-12-19T16:13:23.975561+0000","last_undegraded":"2023-12-19T16:13:23.975561+0000","last_fullsized":"2023-12-19T16:13:23.975561+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:49.958263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.957587+0000","last_change":"2023-12-19T16:13:22.969848+0000","last_active":"2023-12-19T16:13:23.957587+0000","last_peered":"2023-12-19T16:13:23.957587+0000","last_clean":"2023-12-19T16:13:23.957587+0000","last_became_active":"2023-12-19T16:13:22.969550+0000","last_became_peered":"2023-12-19T16:13:22.969550+0000","last_unstale":"2023-12-19T16:13:23.957587+0000","last_undegraded":"2023-12-19T16:13:23.957587+0000","last_fullsized":"2023-12-19T16:13:23.957587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:46:30.011014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968982+0000","last_change":"2023-12-19T16:13:23.968982+0000","last_active":"2023-12-19T16:13:23.968982+0000","last_peered":"2023-12-19T16:13:23.968982+0000","last_clean":"2023-12-19T16:13:23.968982+0000","last_became_active":"2023-12-19T16:13:23.967337+0000","last_became_peered":"2023-12-19T16:13:23.967337+0000","last_unstale":"2023-12-19T16:13:23.968982+0000","last_undegraded":"2023-12-19T16:13:23.968982+0000","last_fullsized":"2023-12-19T16:13:23.968982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:51.122504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.950587+0000","last_change":"2023-12-19T16:13:22.978661+0000","last_active":"2023-12-19T16:13:23.950587+0000","last_peered":"2023-12-19T16:13:23.950587+0000","last_clean":"2023-12-19T16:13:23.950587+0000","last_became_active":"2023-12-19T16:13:22.978363+0000","last_became_peered":"2023-12-19T16:13:22.978363+0000","last_unstale":"2023-12-19T16:13:23.950587+0000","last_undegraded":"2023-12-19T16:13:23.950587+0000","last_fullsized":"2023-12-19T16:13:23.950587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:16.553027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974682+0000","last_change":"2023-12-19T16:13:23.974682+0000","last_active":"2023-12-19T16:13:23.974682+0000","last_peered":"2023-12-19T16:13:23.974682+0000","last_clean":"2023-12-19T16:13:23.974682+0000","last_became_active":"2023-12-19T16:13:23.972382+0000","last_became_peered":"2023-12-19T16:13:23.972382+0000","last_unstale":"2023-12-19T16:13:23.974682+0000","last_undegraded":"2023-12-19T16:13:23.974682+0000","last_fullsized":"2023-12-19T16:13:23.974682+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.975997+0000","last_change":"2023-12-19T16:13:22.974491+0000","last_active":"2023-12-19T16:13:22.975997+0000","last_peered":"2023-12-19T16:13:22.975997+0000","last_clean":"2023-12-19T16:13:22.975997+0000","last_became_active":"2023-12-19T16:13:22.974174+0000","last_became_peered":"2023-12-19T16:13:22.974174+0000","last_unstale":"2023-12-19T16:13:22.975997+0000","last_undegraded":"2023-12-19T16:13:22.975997+0000","last_fullsized":"2023-12-19T16:13:22.975997+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:34:27.805742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974528+0000","last_change":"2023-12-19T16:13:23.974528+0000","last_active":"2023-12-19T16:13:23.974528+0000","last_peered":"2023-12-19T16:13:23.974528+0000","last_clean":"2023-12-19T16:13:23.974528+0000","last_became_active":"2023-12-19T16:13:23.974253+0000","last_became_peered":"2023-12-19T16:13:23.974253+0000","last_unstale":"2023-12-19T16:13:23.974528+0000","last_undegraded":"2023-12-19T16:13:23.974528+0000","last_fullsized":"2023-12-19T16:13:23.974528+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.983031+0000","last_change":"2023-12-19T16:13:22.977080+0000","last_active":"2023-12-19T16:13:22.983031+0000","last_peered":"2023-12-19T16:13:22.983031+0000","last_clean":"2023-12-19T16:13:22.983031+0000","last_became_active":"2023-12-19T16:13:22.976810+0000","last_became_peered":"2023-12-19T16:13:22.976810+0000","last_unstale":"2023-12-19T16:13:22.983031+0000","last_undegraded":"2023-12-19T16:13:22.983031+0000","last_fullsized":"2023-12-19T16:13:22.983031+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:51:50.107725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976284+0000","last_change":"2023-12-19T16:13:23.976284+0000","last_active":"2023-12-19T16:13:23.976284+0000","last_peered":"2023-12-19T16:13:23.976284+0000","last_clean":"2023-12-19T16:13:23.976284+0000","last_became_active":"2023-12-19T16:13:23.976005+0000","last_became_peered":"2023-12-19T16:13:23.976005+0000","last_unstale":"2023-12-19T16:13:23.976284+0000","last_undegraded":"2023-12-19T16:13:23.976284+0000","last_fullsized":"2023-12-19T16:13:23.976284+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:42:05.874463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981143+0000","last_change":"2023-12-19T16:13:22.977002+0000","last_active":"2023-12-19T16:13:22.981143+0000","last_peered":"2023-12-19T16:13:22.981143+0000","last_clean":"2023-12-19T16:13:22.981143+0000","last_became_active":"2023-12-19T16:13:22.976711+0000","last_became_peered":"2023-12-19T16:13:22.976711+0000","last_unstale":"2023-12-19T16:13:22.981143+0000","last_undegraded":"2023-12-19T16:13:22.981143+0000","last_fullsized":"2023-12-19T16:13:22.981143+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:53.278721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972786+0000","last_change":"2023-12-19T16:13:23.972786+0000","last_active":"2023-12-19T16:13:23.972786+0000","last_peered":"2023-12-19T16:13:23.972786+0000","last_clean":"2023-12-19T16:13:23.972786+0000","last_became_active":"2023-12-19T16:13:23.972371+0000","last_became_peered":"2023-12-19T16:13:23.972371+0000","last_unstale":"2023-12-19T16:13:23.972786+0000","last_undegraded":"2023-12-19T16:13:23.972786+0000","last_fullsized":"2023-12-19T16:13:23.972786+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"212'3","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.992763+0000","last_change":"2023-12-19T16:13:22.982586+0000","last_active":"2023-12-19T16:13:23.992763+0000","last_peered":"2023-12-19T16:13:23.992763+0000","last_clean":"2023-12-19T16:13:23.992763+0000","last_became_active":"2023-12-19T16:13:22.982288+0000","last_became_peered":"2023-12-19T16:13:22.982288+0000","last_unstale":"2023-12-19T16:13:23.992763+0000","last_undegraded":"2023-12-19T16:13:23.992763+0000","last_fullsized":"2023-12-19T16:13:23.992763+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:18:38.074071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.982962+0000","last_change":"2023-12-19T16:13:23.982962+0000","last_active":"2023-12-19T16:13:23.982962+0000","last_peered":"2023-12-19T16:13:23.982962+0000","last_clean":"2023-12-19T16:13:23.982962+0000","last_became_active":"2023-12-19T16:13:23.980078+0000","last_became_peered":"2023-12-19T16:13:23.980078+0000","last_unstale":"2023-12-19T16:13:23.982962+0000","last_undegraded":"2023-12-19T16:13:23.982962+0000","last_fullsized":"2023-12-19T16:13:23.982962+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:17.856062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"212'6","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.990016+0000","last_change":"2023-12-19T16:13:22.971936+0000","last_active":"2023-12-19T16:13:23.990016+0000","last_peered":"2023-12-19T16:13:23.990016+0000","last_clean":"2023-12-19T16:13:23.990016+0000","last_became_active":"2023-12-19T16:13:22.971262+0000","last_became_peered":"2023-12-19T16:13:22.971262+0000","last_unstale":"2023-12-19T16:13:23.990016+0000","last_undegraded":"2023-12-19T16:13:23.990016+0000","last_fullsized":"2023-12-19T16:13:23.990016+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:42:45.282864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.972842+0000","last_change":"2023-12-19T16:13:23.972842+0000","last_active":"2023-12-19T16:13:23.972842+0000","last_peered":"2023-12-19T16:13:23.972842+0000","last_clean":"2023-12-19T16:13:23.972842+0000","last_became_active":"2023-12-19T16:13:23.972572+0000","last_became_peered":"2023-12-19T16:13:23.972572+0000","last_unstale":"2023-12-19T16:13:23.972842+0000","last_undegraded":"2023-12-19T16:13:23.972842+0000","last_fullsized":"2023-12-19T16:13:23.972842+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T22:35:17.640963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991660+0000","last_change":"2023-12-19T16:13:22.980227+0000","last_active":"2023-12-19T16:13:23.991660+0000","last_peered":"2023-12-19T16:13:23.991660+0000","last_clean":"2023-12-19T16:13:23.991660+0000","last_became_active":"2023-12-19T16:13:22.979897+0000","last_became_peered":"2023-12-19T16:13:22.979897+0000","last_unstale":"2023-12-19T16:13:23.991660+0000","last_undegraded":"2023-12-19T16:13:23.991660+0000","last_fullsized":"2023-12-19T16:13:23.991660+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:15.967328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977431+0000","last_change":"2023-12-19T16:13:23.977431+0000","last_active":"2023-12-19T16:13:23.977431+0000","last_peered":"2023-12-19T16:13:23.977431+0000","last_clean":"2023-12-19T16:13:23.977431+0000","last_became_active":"2023-12-19T16:13:23.972608+0000","last_became_peered":"2023-12-19T16:13:23.972608+0000","last_unstale":"2023-12-19T16:13:23.977431+0000","last_undegraded":"2023-12-19T16:13:23.977431+0000","last_fullsized":"2023-12-19T16:13:23.977431+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T23:29:27.253172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986563+0000","last_change":"2023-12-19T16:13:22.982663+0000","last_active":"2023-12-19T16:13:22.986563+0000","last_peered":"2023-12-19T16:13:22.986563+0000","last_clean":"2023-12-19T16:13:22.986563+0000","last_became_active":"2023-12-19T16:13:22.982391+0000","last_became_peered":"2023-12-19T16:13:22.982391+0000","last_unstale":"2023-12-19T16:13:22.986563+0000","last_undegraded":"2023-12-19T16:13:22.986563+0000","last_fullsized":"2023-12-19T16:13:22.986563+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:49.526251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.964024+0000","last_change":"2023-12-19T16:13:23.964024+0000","last_active":"2023-12-19T16:13:23.964024+0000","last_peered":"2023-12-19T16:13:23.964024+0000","last_clean":"2023-12-19T16:13:23.964024+0000","last_became_active":"2023-12-19T16:13:23.963649+0000","last_became_peered":"2023-12-19T16:13:23.963649+0000","last_unstale":"2023-12-19T16:13:23.964024+0000","last_undegraded":"2023-12-19T16:13:23.964024+0000","last_fullsized":"2023-12-19T16:13:23.964024+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:03.353242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.987287+0000","last_change":"2023-12-19T16:13:22.981232+0000","last_active":"2023-12-19T16:13:22.987287+0000","last_peered":"2023-12-19T16:13:22.987287+0000","last_clean":"2023-12-19T16:13:22.987287+0000","last_became_active":"2023-12-19T16:13:22.980929+0000","last_became_peered":"2023-12-19T16:13:22.980929+0000","last_unstale":"2023-12-19T16:13:22.987287+0000","last_undegraded":"2023-12-19T16:13:22.987287+0000","last_fullsized":"2023-12-19T16:13:22.987287+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:49:14.707493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.976276+0000","last_change":"2023-12-19T16:13:23.976276+0000","last_active":"2023-12-19T16:13:23.976276+0000","last_peered":"2023-12-19T16:13:23.976276+0000","last_clean":"2023-12-19T16:13:23.976276+0000","last_became_active":"2023-12-19T16:13:23.973860+0000","last_became_peered":"2023-12-19T16:13:23.973860+0000","last_unstale":"2023-12-19T16:13:23.976276+0000","last_undegraded":"2023-12-19T16:13:23.976276+0000","last_fullsized":"2023-12-19T16:13:23.976276+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:09.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"212'8","reported_seq":17,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991327+0000","last_change":"2023-12-19T16:13:22.980604+0000","last_active":"2023-12-19T16:13:23.991327+0000","last_peered":"2023-12-19T16:13:23.991327+0000","last_clean":"2023-12-19T16:13:23.991327+0000","last_became_active":"2023-12-19T16:13:22.979693+0000","last_became_peered":"2023-12-19T16:13:22.979693+0000","last_unstale":"2023-12-19T16:13:23.991327+0000","last_undegraded":"2023-12-19T16:13:23.991327+0000","last_fullsized":"2023-12-19T16:13:23.991327+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:53:20.656471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970531+0000","last_change":"2023-12-19T16:13:23.970531+0000","last_active":"2023-12-19T16:13:23.970531+0000","last_peered":"2023-12-19T16:13:23.970531+0000","last_clean":"2023-12-19T16:13:23.970531+0000","last_became_active":"2023-12-19T16:13:23.970218+0000","last_became_peered":"2023-12-19T16:13:23.970218+0000","last_unstale":"2023-12-19T16:13:23.970531+0000","last_undegraded":"2023-12-19T16:13:23.970531+0000","last_fullsized":"2023-12-19T16:13:23.970531+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:44:38.534471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.961675+0000","last_change":"2023-12-19T16:13:22.973700+0000","last_active":"2023-12-19T16:13:23.961675+0000","last_peered":"2023-12-19T16:13:23.961675+0000","last_clean":"2023-12-19T16:13:23.961675+0000","last_became_active":"2023-12-19T16:13:22.973412+0000","last_became_peered":"2023-12-19T16:13:22.973412+0000","last_unstale":"2023-12-19T16:13:23.961675+0000","last_undegraded":"2023-12-19T16:13:23.961675+0000","last_fullsized":"2023-12-19T16:13:23.961675+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:29.812164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.985046+0000","last_change":"2023-12-19T16:13:23.985046+0000","last_active":"2023-12-19T16:13:23.985046+0000","last_peered":"2023-12-19T16:13:23.985046+0000","last_clean":"2023-12-19T16:13:23.985046+0000","last_became_active":"2023-12-19T16:13:23.984698+0000","last_became_peered":"2023-12-19T16:13:23.984698+0000","last_unstale":"2023-12-19T16:13:23.985046+0000","last_undegraded":"2023-12-19T16:13:23.985046+0000","last_fullsized":"2023-12-19T16:13:23.985046+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:36.715851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.992008+0000","last_change":"2023-12-19T16:13:22.982473+0000","last_active":"2023-12-19T16:13:23.992008+0000","last_peered":"2023-12-19T16:13:23.992008+0000","last_clean":"2023-12-19T16:13:23.992008+0000","last_became_active":"2023-12-19T16:13:22.982170+0000","last_became_peered":"2023-12-19T16:13:22.982170+0000","last_unstale":"2023-12-19T16:13:23.992008+0000","last_undegraded":"2023-12-19T16:13:23.992008+0000","last_fullsized":"2023-12-19T16:13:23.992008+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:29.812164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.966316+0000","last_change":"2023-12-19T16:13:23.966316+0000","last_active":"2023-12-19T16:13:23.966316+0000","last_peered":"2023-12-19T16:13:23.966316+0000","last_clean":"2023-12-19T16:13:23.966316+0000","last_became_active":"2023-12-19T16:13:23.966025+0000","last_became_peered":"2023-12-19T16:13:23.966025+0000","last_unstale":"2023-12-19T16:13:23.966316+0000","last_undegraded":"2023-12-19T16:13:23.966316+0000","last_fullsized":"2023-12-19T16:13:23.966316+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:49:04.026807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"212'1","reported_seq":10,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.951594+0000","last_change":"2023-12-19T16:13:22.980710+0000","last_active":"2023-12-19T16:13:23.951594+0000","last_peered":"2023-12-19T16:13:23.951594+0000","last_clean":"2023-12-19T16:13:23.951594+0000","last_became_active":"2023-12-19T16:13:22.980054+0000","last_became_peered":"2023-12-19T16:13:22.980054+0000","last_unstale":"2023-12-19T16:13:23.951594+0000","last_undegraded":"2023-12-19T16:13:23.951594+0000","last_fullsized":"2023-12-19T16:13:23.951594+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:10:49.814105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.977111+0000","last_change":"2023-12-19T16:13:23.977111+0000","last_active":"2023-12-19T16:13:23.977111+0000","last_peered":"2023-12-19T16:13:23.977111+0000","last_clean":"2023-12-19T16:13:23.977111+0000","last_became_active":"2023-12-19T16:13:23.976577+0000","last_became_peered":"2023-12-19T16:13:23.976577+0000","last_unstale":"2023-12-19T16:13:23.977111+0000","last_undegraded":"2023-12-19T16:13:23.977111+0000","last_fullsized":"2023-12-19T16:13:23.977111+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.972767+0000","last_change":"2023-12-19T16:13:22.969403+0000","last_active":"2023-12-19T16:13:22.972767+0000","last_peered":"2023-12-19T16:13:22.972767+0000","last_clean":"2023-12-19T16:13:22.972767+0000","last_became_active":"2023-12-19T16:13:22.969060+0000","last_became_peered":"2023-12-19T16:13:22.969060+0000","last_unstale":"2023-12-19T16:13:22.972767+0000","last_undegraded":"2023-12-19T16:13:22.972767+0000","last_fullsized":"2023-12-19T16:13:22.972767+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:16.553027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.967488+0000","last_change":"2023-12-19T16:13:23.967488+0000","last_active":"2023-12-19T16:13:23.967488+0000","last_peered":"2023-12-19T16:13:23.967488+0000","last_clean":"2023-12-19T16:13:23.967488+0000","last_became_active":"2023-12-19T16:13:23.967173+0000","last_became_peered":"2023-12-19T16:13:23.967173+0000","last_unstale":"2023-12-19T16:13:23.967488+0000","last_undegraded":"2023-12-19T16:13:23.967488+0000","last_fullsized":"2023-12-19T16:13:23.967488+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:39.158672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.986952+0000","last_change":"2023-12-19T16:13:22.978603+0000","last_active":"2023-12-19T16:13:23.986952+0000","last_peered":"2023-12-19T16:13:23.986952+0000","last_clean":"2023-12-19T16:13:23.986952+0000","last_became_active":"2023-12-19T16:13:22.978271+0000","last_became_peered":"2023-12-19T16:13:22.978271+0000","last_unstale":"2023-12-19T16:13:23.986952+0000","last_undegraded":"2023-12-19T16:13:23.986952+0000","last_fullsized":"2023-12-19T16:13:23.986952+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:23.923216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.968520+0000","last_change":"2023-12-19T16:13:23.968520+0000","last_active":"2023-12-19T16:13:23.968520+0000","last_peered":"2023-12-19T16:13:23.968520+0000","last_clean":"2023-12-19T16:13:23.968520+0000","last_became_active":"2023-12-19T16:13:23.968173+0000","last_became_peered":"2023-12-19T16:13:23.968173+0000","last_unstale":"2023-12-19T16:13:23.968520+0000","last_undegraded":"2023-12-19T16:13:23.968520+0000","last_fullsized":"2023-12-19T16:13:23.968520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:35.816448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.987970+0000","last_change":"2023-12-19T16:13:22.986081+0000","last_active":"2023-12-19T16:13:22.987970+0000","last_peered":"2023-12-19T16:13:22.987970+0000","last_clean":"2023-12-19T16:13:22.987970+0000","last_became_active":"2023-12-19T16:13:22.985780+0000","last_became_peered":"2023-12-19T16:13:22.985780+0000","last_unstale":"2023-12-19T16:13:22.987970+0000","last_undegraded":"2023-12-19T16:13:22.987970+0000","last_fullsized":"2023-12-19T16:13:22.987970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T02:19:19.690490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974372+0000","last_change":"2023-12-19T16:13:23.974372+0000","last_active":"2023-12-19T16:13:23.974372+0000","last_peered":"2023-12-19T16:13:23.974372+0000","last_clean":"2023-12-19T16:13:23.974372+0000","last_became_active":"2023-12-19T16:13:23.974093+0000","last_became_peered":"2023-12-19T16:13:23.974093+0000","last_unstale":"2023-12-19T16:13:23.974372+0000","last_undegraded":"2023-12-19T16:13:23.974372+0000","last_fullsized":"2023-12-19T16:13:23.974372+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:20.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986709+0000","last_change":"2023-12-19T16:13:22.977533+0000","last_active":"2023-12-19T16:13:22.986709+0000","last_peered":"2023-12-19T16:13:22.986709+0000","last_clean":"2023-12-19T16:13:22.986709+0000","last_became_active":"2023-12-19T16:13:22.977218+0000","last_became_peered":"2023-12-19T16:13:22.977218+0000","last_unstale":"2023-12-19T16:13:22.986709+0000","last_undegraded":"2023-12-19T16:13:22.986709+0000","last_fullsized":"2023-12-19T16:13:22.986709+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:20.030474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.964250+0000","last_change":"2023-12-19T16:13:23.964250+0000","last_active":"2023-12-19T16:13:23.964250+0000","last_peered":"2023-12-19T16:13:23.964250+0000","last_clean":"2023-12-19T16:13:23.964250+0000","last_became_active":"2023-12-19T16:13:23.961324+0000","last_became_peered":"2023-12-19T16:13:23.961324+0000","last_unstale":"2023-12-19T16:13:23.964250+0000","last_undegraded":"2023-12-19T16:13:23.964250+0000","last_fullsized":"2023-12-19T16:13:23.964250+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:39.572048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.970724+0000","last_change":"2023-12-19T16:13:22.969473+0000","last_active":"2023-12-19T16:13:22.970724+0000","last_peered":"2023-12-19T16:13:22.970724+0000","last_clean":"2023-12-19T16:13:22.970724+0000","last_became_active":"2023-12-19T16:13:22.969128+0000","last_became_peered":"2023-12-19T16:13:22.969128+0000","last_unstale":"2023-12-19T16:13:22.970724+0000","last_undegraded":"2023-12-19T16:13:22.970724+0000","last_fullsized":"2023-12-19T16:13:22.970724+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:17:49.588817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.975141+0000","last_change":"2023-12-19T16:13:23.975141+0000","last_active":"2023-12-19T16:13:23.975141+0000","last_peered":"2023-12-19T16:13:23.975141+0000","last_clean":"2023-12-19T16:13:23.975141+0000","last_became_active":"2023-12-19T16:13:23.974801+0000","last_became_peered":"2023-12-19T16:13:23.974801+0000","last_unstale":"2023-12-19T16:13:23.975141+0000","last_undegraded":"2023-12-19T16:13:23.975141+0000","last_fullsized":"2023-12-19T16:13:23.975141+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:37:09.093365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"212'2","reported_seq":11,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.991585+0000","last_change":"2023-12-19T16:13:22.980663+0000","last_active":"2023-12-19T16:13:23.991585+0000","last_peered":"2023-12-19T16:13:23.991585+0000","last_clean":"2023-12-19T16:13:23.991585+0000","last_became_active":"2023-12-19T16:13:22.980227+0000","last_became_peered":"2023-12-19T16:13:22.980227+0000","last_unstale":"2023-12-19T16:13:23.991585+0000","last_undegraded":"2023-12-19T16:13:23.991585+0000","last_fullsized":"2023-12-19T16:13:23.991585+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:40:18.559957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970393+0000","last_change":"2023-12-19T16:13:23.970393+0000","last_active":"2023-12-19T16:13:23.970393+0000","last_peered":"2023-12-19T16:13:23.970393+0000","last_clean":"2023-12-19T16:13:23.970393+0000","last_became_active":"2023-12-19T16:13:23.969809+0000","last_became_peered":"2023-12-19T16:13:23.969809+0000","last_unstale":"2023-12-19T16:13:23.970393+0000","last_undegraded":"2023-12-19T16:13:23.970393+0000","last_fullsized":"2023-12-19T16:13:23.970393+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:12:46.205371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986901+0000","last_change":"2023-12-19T16:13:22.980021+0000","last_active":"2023-12-19T16:13:22.986901+0000","last_peered":"2023-12-19T16:13:22.986901+0000","last_clean":"2023-12-19T16:13:22.986901+0000","last_became_active":"2023-12-19T16:13:22.979727+0000","last_became_peered":"2023-12-19T16:13:22.979727+0000","last_unstale":"2023-12-19T16:13:22.986901+0000","last_undegraded":"2023-12-19T16:13:22.986901+0000","last_fullsized":"2023-12-19T16:13:22.986901+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:19:50.395415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.978349+0000","last_change":"2023-12-19T16:13:23.978349+0000","last_active":"2023-12-19T16:13:23.978349+0000","last_peered":"2023-12-19T16:13:23.978349+0000","last_clean":"2023-12-19T16:13:23.978349+0000","last_became_active":"2023-12-19T16:13:23.975361+0000","last_became_peered":"2023-12-19T16:13:23.975361+0000","last_unstale":"2023-12-19T16:13:23.978349+0000","last_undegraded":"2023-12-19T16:13:23.978349+0000","last_fullsized":"2023-12-19T16:13:23.978349+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:39:57.022153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981097+0000","last_change":"2023-12-19T16:13:22.972652+0000","last_active":"2023-12-19T16:13:22.981097+0000","last_peered":"2023-12-19T16:13:22.981097+0000","last_clean":"2023-12-19T16:13:22.981097+0000","last_became_active":"2023-12-19T16:13:22.971701+0000","last_became_peered":"2023-12-19T16:13:22.971701+0000","last_unstale":"2023-12-19T16:13:22.981097+0000","last_undegraded":"2023-12-19T16:13:22.981097+0000","last_fullsized":"2023-12-19T16:13:22.981097+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:52:00.342528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974813+0000","last_change":"2023-12-19T16:13:23.974813+0000","last_active":"2023-12-19T16:13:23.974813+0000","last_peered":"2023-12-19T16:13:23.974813+0000","last_clean":"2023-12-19T16:13:23.974813+0000","last_became_active":"2023-12-19T16:13:23.974504+0000","last_became_peered":"2023-12-19T16:13:23.974504+0000","last_unstale":"2023-12-19T16:13:23.974813+0000","last_undegraded":"2023-12-19T16:13:23.974813+0000","last_fullsized":"2023-12-19T16:13:23.974813+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:38:49.842475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.977917+0000","last_change":"2023-12-19T16:13:22.969654+0000","last_active":"2023-12-19T16:13:22.977917+0000","last_peered":"2023-12-19T16:13:22.977917+0000","last_clean":"2023-12-19T16:13:22.977917+0000","last_became_active":"2023-12-19T16:13:22.968326+0000","last_became_peered":"2023-12-19T16:13:22.968326+0000","last_unstale":"2023-12-19T16:13:22.977917+0000","last_undegraded":"2023-12-19T16:13:22.977917+0000","last_fullsized":"2023-12-19T16:13:22.977917+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:16.841283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.967299+0000","last_change":"2023-12-19T16:13:23.967299+0000","last_active":"2023-12-19T16:13:23.967299+0000","last_peered":"2023-12-19T16:13:23.967299+0000","last_clean":"2023-12-19T16:13:23.967299+0000","last_became_active":"2023-12-19T16:13:23.965278+0000","last_became_peered":"2023-12-19T16:13:23.965278+0000","last_unstale":"2023-12-19T16:13:23.967299+0000","last_undegraded":"2023-12-19T16:13:23.967299+0000","last_fullsized":"2023-12-19T16:13:23.967299+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:54.293501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.975200+0000","last_change":"2023-12-19T16:13:22.973975+0000","last_active":"2023-12-19T16:13:22.975200+0000","last_peered":"2023-12-19T16:13:22.975200+0000","last_clean":"2023-12-19T16:13:22.975200+0000","last_became_active":"2023-12-19T16:13:22.973655+0000","last_became_peered":"2023-12-19T16:13:22.973655+0000","last_unstale":"2023-12-19T16:13:22.975200+0000","last_undegraded":"2023-12-19T16:13:22.975200+0000","last_fullsized":"2023-12-19T16:13:22.975200+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:42:45.282864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.970698+0000","last_change":"2023-12-19T16:13:23.970698+0000","last_active":"2023-12-19T16:13:23.970698+0000","last_peered":"2023-12-19T16:13:23.970698+0000","last_clean":"2023-12-19T16:13:23.970698+0000","last_became_active":"2023-12-19T16:13:23.970371+0000","last_became_peered":"2023-12-19T16:13:23.970371+0000","last_unstale":"2023-12-19T16:13:23.970698+0000","last_undegraded":"2023-12-19T16:13:23.970698+0000","last_fullsized":"2023-12-19T16:13:23.970698+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:08.222744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.971704+0000","last_change":"2023-12-19T16:13:22.969865+0000","last_active":"2023-12-19T16:13:22.971704+0000","last_peered":"2023-12-19T16:13:22.971704+0000","last_clean":"2023-12-19T16:13:22.971704+0000","last_became_active":"2023-12-19T16:13:22.969507+0000","last_became_peered":"2023-12-19T16:13:22.969507+0000","last_unstale":"2023-12-19T16:13:22.971704+0000","last_undegraded":"2023-12-19T16:13:22.971704+0000","last_fullsized":"2023-12-19T16:13:22.971704+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:02:38.021765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.982269+0000","last_change":"2023-12-19T16:13:23.982269+0000","last_active":"2023-12-19T16:13:23.982269+0000","last_peered":"2023-12-19T16:13:23.982269+0000","last_clean":"2023-12-19T16:13:23.982269+0000","last_became_active":"2023-12-19T16:13:23.971527+0000","last_became_peered":"2023-12-19T16:13:23.971527+0000","last_unstale":"2023-12-19T16:13:23.982269+0000","last_undegraded":"2023-12-19T16:13:23.982269+0000","last_fullsized":"2023-12-19T16:13:23.982269+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T01:34:52.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.986766+0000","last_change":"2023-12-19T16:13:22.977822+0000","last_active":"2023-12-19T16:13:22.986766+0000","last_peered":"2023-12-19T16:13:22.986766+0000","last_clean":"2023-12-19T16:13:22.986766+0000","last_became_active":"2023-12-19T16:13:22.977507+0000","last_became_peered":"2023-12-19T16:13:22.977507+0000","last_unstale":"2023-12-19T16:13:22.986766+0000","last_undegraded":"2023-12-19T16:13:22.986766+0000","last_fullsized":"2023-12-19T16:13:22.986766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:08:24.790765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.974327+0000","last_change":"2023-12-19T16:13:23.974327+0000","last_active":"2023-12-19T16:13:23.974327+0000","last_peered":"2023-12-19T16:13:23.974327+0000","last_clean":"2023-12-19T16:13:23.974327+0000","last_became_active":"2023-12-19T16:13:23.974000+0000","last_became_peered":"2023-12-19T16:13:23.974000+0000","last_unstale":"2023-12-19T16:13:23.974327+0000","last_undegraded":"2023-12-19T16:13:23.974327+0000","last_fullsized":"2023-12-19T16:13:23.974327+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:53:09.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.976527+0000","last_change":"2023-12-19T16:13:22.974512+0000","last_active":"2023-12-19T16:13:22.976527+0000","last_peered":"2023-12-19T16:13:22.976527+0000","last_clean":"2023-12-19T16:13:22.976527+0000","last_became_active":"2023-12-19T16:13:22.974207+0000","last_became_peered":"2023-12-19T16:13:22.974207+0000","last_unstale":"2023-12-19T16:13:22.976527+0000","last_undegraded":"2023-12-19T16:13:22.976527+0000","last_fullsized":"2023-12-19T16:13:22.976527+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:18:46.942157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":8,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-12-19T16:13:23.978238+0000","last_change":"2023-12-19T16:13:23.978238+0000","last_active":"2023-12-19T16:13:23.978238+0000","last_peered":"2023-12-19T16:13:23.978238+0000","last_clean":"2023-12-19T16:13:23.978238+0000","last_became_active":"2023-12-19T16:13:23.977509+0000","last_became_peered":"2023-12-19T16:13:23.977509+0000","last_unstale":"2023-12-19T16:13:23.978238+0000","last_undegraded":"2023-12-19T16:13:23.978238+0000","last_fullsized":"2023-12-19T16:13:23.978238+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:22.931065+0000","last_clean_scrub_stamp":"2023-12-19T16:13:22.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:09:16.982107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.981753+0000","last_change":"2023-12-19T16:13:22.975500+0000","last_active":"2023-12-19T16:13:22.981753+0000","last_peered":"2023-12-19T16:13:22.981753+0000","last_clean":"2023-12-19T16:13:22.981753+0000","last_became_active":"2023-12-19T16:13:22.975228+0000","last_became_peered":"2023-12-19T16:13:22.975228+0000","last_unstale":"2023-12-19T16:13:22.981753+0000","last_undegraded":"2023-12-19T16:13:22.981753+0000","last_fullsized":"2023-12-19T16:13:22.981753+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T16:13:21.916286+0000","last_clean_scrub_stamp":"2023-12-19T16:13:21.916286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:38:45.917623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"202'348","reported_seq":1138,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T16:13:22.959701+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:22.959701+0000","last_peered":"2023-12-19T16:13:22.959701+0000","last_clean":"2023-12-19T16:13:22.959701+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:22.959701+0000","last_undegraded":"2023-12-19T16:13:22.959701+0000","last_fullsized":"2023-12-19T16:13:22.959701+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575972,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304560,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067280,"statfs":{"total":96636764160,"available":96324894720,"internally_reserved":0,"allocated":3252224,"data_stored":1949500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.052},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":0.98799999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":1.081},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.4670000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":1.2529999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.339}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.3029999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":1.2},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.6739999999999999}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.732},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.2270000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575972,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312176,"kb_used_data":2600,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059664,"statfs":{"total":96636764160,"available":96317095936,"internally_reserved":0,"allocated":2662400,"data_stored":1359202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.163,"15min":1.179},"min":{"1min":0.877,"5min":0.69999999999999996,"15min":0.60099999999999998},"max":{"1min":1.488,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.224,"15min":1.2390000000000001},"min":{"1min":0.89700000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.597,"5min":1.74,"15min":1.7909999999999999},"last":1.1739999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.76500000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5509999999999999,"5min":1.643,"15min":1.7010000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2609999999999999,"15min":1.274},"min":{"1min":0.89400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.8080000000000001,"15min":1.827},"last":1.075}]},{"osd":2,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.202,"15min":1.23},"min":{"1min":0.878,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.504,"5min":1.651,"15min":1.8080000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.28,"15min":1.288},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.599,"5min":1.732,"15min":1.732},"last":1.266}]},{"osd":3,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.042,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5840000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.3280000000000001},"min":{"1min":0.80200000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0309999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.212,"15min":1.2050000000000001},"min":{"1min":1.0289999999999999,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.4710000000000001,"5min":1.708,"15min":1.774},"last":1.361},{"interface":"front","average":{"1min":1.375,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":1.02,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.8140000000000001,"5min":1.966,"15min":1.966},"last":1.3400000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.296,"15min":1.3129999999999999},"min":{"1min":0.88200000000000001,"5min":0.83299999999999996,"15min":0.68000000000000005},"max":{"1min":1.7509999999999999,"5min":1.788,"15min":1.923},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.3620000000000001,"15min":1.3720000000000001},"min":{"1min":1.1519999999999999,"5min":0.94099999999999995,"15min":0.68899999999999995},"max":{"1min":1.593,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.1279999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4219999999999999,"15min":1.4350000000000001},"min":{"1min":1.101,"5min":0.92900000000000005,"15min":0.73799999999999999},"max":{"1min":1.706,"5min":1.8720000000000001,"15min":2.169},"last":1.238},{"interface":"front","average":{"1min":1.478,"5min":1.4430000000000001,"15min":1.46},"min":{"1min":0.97499999999999998,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.855,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.111}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308092,"kb_used_data":2612,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063748,"statfs":{"total":96636764160,"available":96321277952,"internally_reserved":0,"allocated":2674688,"data_stored":1360690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.101,"15min":1.1060000000000001},"min":{"1min":0.622,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.337,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.746,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.5600000000000001,"5min":1.6619999999999999,"15min":1.907},"last":0.58699999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2689999999999999,"15min":1.3100000000000001},"min":{"1min":0.86099999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6559999999999999,"5min":1.7350000000000001,"15min":1.7490000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2509999999999999,"15min":1.276},"min":{"1min":0.88400000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.552,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.92800000000000005}]},{"osd":2,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2769999999999999,"15min":1.2749999999999999},"min":{"1min":0.92900000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.641,"5min":1.9259999999999999,"15min":2.0249999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.92200000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.6639999999999999,"5min":1.7769999999999999,"15min":1.831},"last":0.88700000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2809999999999999,"15min":1.306},"min":{"1min":1.0049999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.575,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.284,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":1.024,"5min":0.89300000000000002,"15min":0.85799999999999998},"max":{"1min":1.585,"5min":1.734,"15min":2.0449999999999999},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.73999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.8},"last":1.01},{"interface":"front","average":{"1min":1.306,"5min":1.2969999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7869999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.83499999999999996}]},{"osd":6,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.385,"15min":1.3779999999999999},"min":{"1min":1.208,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.8680000000000001,"15min":1.968},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.3999999999999999,"15min":1.401},"min":{"1min":0.93999999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.724,"5min":1.7929999999999999,"15min":1.794},"last":1.0980000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4419999999999999,"15min":1.4179999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.71,"5min":2.395,"15min":2.395},"last":1.139},{"interface":"front","average":{"1min":1.429,"5min":1.411,"15min":1.4179999999999999},"min":{"1min":1.018,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8240000000000001,"5min":1.9770000000000001,"15min":1.99},"last":1.169}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1359132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.355},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.4139999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":1.012}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.244},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":1.2090000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":1.157},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":0.81699999999999995}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.484}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.391},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.4490000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575971,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299892,"kb_used_data":2604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071948,"statfs":{"total":96636764160,"available":96329674752,"internally_reserved":0,"allocated":2666496,"data_stored":1359292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.90100000000000002,"15min":0.90800000000000003},"min":{"1min":0.66700000000000004,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.5660000000000001,"5min":1.633,"15min":1.7729999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.97599999999999998,"15min":1.0089999999999999},"min":{"1min":0.59599999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.367,"5min":1.5600000000000001,"15min":1.9370000000000001},"last":0.67200000000000004}]},{"osd":1,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.94099999999999995,"15min":0.96599999999999997},"min":{"1min":0.69499999999999995,"5min":0.67200000000000004,"15min":0.53900000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":2.0350000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.1459999999999999,"15min":1.125},"min":{"1min":0.75900000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5760000000000001,"5min":1.724,"15min":2.2269999999999999},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.151,"15min":1.169},"min":{"1min":0.89500000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6020000000000001,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.24,"15min":1.26},"min":{"1min":0.84599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.7370000000000001,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.302,"15min":1.333},"min":{"1min":1.0820000000000001,"5min":0.754,"15min":0.754},"max":{"1min":1.6080000000000001,"5min":1.7270000000000001,"15min":2.5099999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.3200000000000001,"15min":1.3180000000000001},"min":{"1min":1.052,"5min":0.96199999999999997,"15min":0.72199999999999998},"max":{"1min":1.788,"5min":1.788,"15min":2.3450000000000002},"last":1.3660000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.367,"15min":1.3640000000000001},"min":{"1min":1.117,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.133},{"interface":"front","average":{"1min":1.5,"5min":1.4470000000000001,"15min":1.4379999999999999},"min":{"1min":1.105,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.1909999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":0.85299999999999998}]},{"osd":6,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.508,"15min":1.516},"min":{"1min":1.194,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.04},{"interface":"front","average":{"1min":1.516,"5min":1.4419999999999999,"15min":1.4670000000000001},"min":{"1min":1.054,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9970000000000001,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.5409999999999999,"15min":1.5600000000000001},"min":{"1min":1.234,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":0.87},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.494,"15min":1.512},"min":{"1min":1.248,"5min":1.018,"15min":1.018},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.244}]}]},{"osd":2,"up_from":13,"seq":55834575973,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1358675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.95499999999999996,"15min":0.96499999999999997},"min":{"1min":0.73899999999999999,"5min":0.61899999999999999,"15min":0.54600000000000004},"max":{"1min":1.2070000000000001,"5min":1.4510000000000001,"15min":1.581},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.157,"15min":1.151},"min":{"1min":0.874,"5min":0.79900000000000004,"15min":0.66200000000000003},"max":{"1min":1.339,"5min":1.629,"15min":1.796},"last":0.97099999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.194,"15min":1.1870000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7090000000000001,"5min":2.327,"15min":2.327},"last":1.274},{"interface":"front","average":{"1min":1.234,"5min":1.169,"15min":1.143},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.621},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0289999999999999,"15min":1.0409999999999999},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.284,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.141,"15min":1.149},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5169999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.66500000000000004}]},{"osd":4,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4550000000000001,"15min":1.444},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.6839999999999999,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4670000000000001,"15min":1.4510000000000001},"min":{"1min":1.002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.6839999999999999,"5min":2.218,"15min":2.218},"last":1.377}]},{"osd":5,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.4399999999999999,"15min":1.425},"min":{"1min":1.1000000000000001,"5min":1.0660000000000001,"15min":0.91600000000000004},"max":{"1min":1.901,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.431,"15min":1.411},"min":{"1min":0.96099999999999997,"5min":0.879,"15min":0.879},"max":{"1min":1.841,"5min":2.0139999999999998,"15min":2.4340000000000002},"last":1.234}]},{"osd":6,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4159999999999999,"15min":1.407},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5669999999999999,"15min":1.5389999999999999},"min":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.071},"max":{"1min":1.8029999999999999,"5min":2.609,"15min":2.609},"last":1.351}]},{"osd":7,"last update":"Tue Dec 19 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.5820000000000001,"15min":1.5609999999999999},"min":{"1min":1.2350000000000001,"5min":1.1479999999999999,"15min":0.89600000000000002},"max":{"1min":1.8440000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.323},{"interface":"front","average":{"1min":1.643,"5min":1.5669999999999999,"15min":1.548},"min":{"1min":1.2729999999999999,"5min":1.028,"15min":1.028},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.1899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575971,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312172,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059668,"statfs":{"total":96636764160,"available":96317100032,"internally_reserved":0,"allocated":2658304,"data_stored":1359132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.90800000000000003},"min":{"1min":0.55800000000000005,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.351,"5min":1.6559999999999999,"15min":1.6579999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0129999999999999,"15min":1.0369999999999999},"min":{"1min":0.77600000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.46,"5min":1.548,"15min":1.9119999999999999},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.05,"15min":1.069},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.149,"5min":1.1240000000000001,"15min":1.161},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.63900000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2809999999999999,"15min":1.2869999999999999},"min":{"1min":0.88800000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7330000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.107,"15min":1.1060000000000001},"min":{"1min":0.72899999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4099999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.93500000000000005}]},{"osd":4,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.306,"15min":1.331},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.903,"5min":1.903,"15min":2.1459999999999999},"last":1.214},{"interface":"front","average":{"1min":1.423,"5min":1.397,"15min":1.427},"min":{"1min":1.1659999999999999,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.0110000000000001},"last":1.014}]},{"osd":5,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.325,"15min":1.345},"min":{"1min":1.0129999999999999,"5min":0.86399999999999999,"15min":0.82099999999999995},"max":{"1min":1.5109999999999999,"5min":1.833,"15min":1.9159999999999999},"last":1.167},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4430000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.323},"last":1.0700000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3799999999999999,"15min":1.4039999999999999},"min":{"1min":1.109,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5920000000000001,"5min":1.972,"15min":2.1139999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4950000000000001,"15min":1.518},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2400000000000002},"last":1.327}]},{"osd":7,"last update":"Tue Dec 19 16:12:33 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.52,"15min":1.5349999999999999},"min":{"1min":1.0900000000000001,"5min":1.026,"15min":1.026},"max":{"1min":1.905,"5min":2.2010000000000001,"15min":2.2229999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.468,"15min":1.496},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":1.7150000000000001,"5min":2.0670000000000002,"15min":2.3170000000000002},"last":1.383}]}]},{"osd":0,"up_from":13,"seq":55834575971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304572,"kb_used_data":3188,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067268,"statfs":{"total":96636764160,"available":96324882432,"internally_reserved":0,"allocated":3264512,"data_stored":1949189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90600000000000003,"15min":0.91200000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.439},"max":{"1min":1.3500000000000001,"5min":1.4530000000000001,"15min":1.6419999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0049999999999999,"15min":1.012},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.58999999999999997},"max":{"1min":1.4850000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.165,"15min":1.145},"min":{"1min":0.72299999999999998,"5min":0.68300000000000005,"15min":0.57099999999999995},"max":{"1min":1.536,"5min":1.748,"15min":1.8400000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.244,"15min":1.272},"min":{"1min":0.75800000000000001,"5min":0.66200000000000003,"15min":0.54300000000000004},"max":{"1min":1.633,"5min":1.655,"15min":1.8500000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.036,"15min":1.046},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1919999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.873},{"interface":"front","average":{"1min":1.175,"5min":1.2490000000000001,"15min":1.2549999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.55300000000000005},"max":{"1min":1.609,"5min":1.974,"15min":1.974},"last":1.385}]},{"osd":4,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3859999999999999,"15min":1.3899999999999999},"min":{"1min":1.01,"5min":1.01,"15min":0.69499999999999995},"max":{"1min":1.6699999999999999,"5min":1.885,"15min":2.0209999999999999},"last":1},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.367,"15min":1.3360000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.71999999999999997},"max":{"1min":1.6599999999999999,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.103}]},{"osd":5,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4510000000000001,"15min":1.4390000000000001},"min":{"1min":1.103,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.8839999999999999,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.548},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4530000000000001,"15min":1.4590000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.919,"5min":1.919,"15min":1.9810000000000001},"last":1.238}]},{"osd":6,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.4119999999999999,"15min":1.417},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.577,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.254},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.5329999999999999,"15min":1.516},"min":{"1min":1.1180000000000001,"5min":1.056,"15min":0.93899999999999995},"max":{"1min":1.998,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.6459999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:12:52 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.54,"15min":1.5309999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.776,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.21},{"interface":"front","average":{"1min":1.363,"5min":1.5,"15min":1.5049999999999999},"min":{"1min":1.113,"5min":1.1080000000000001,"15min":1.008},"max":{"1min":1.734,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.298}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:13:28.714 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T16:13:28.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T16:13:28.924 INFO:tasks.ceph.mds.c.smithi050.stderr:2023-12-19T16:13:28.919+0000 7f03665d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T16:13:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.931+0000 7f4a1984c700 1 Processor -- start 2023-12-19T16:13:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.931+0000 7f4a1984c700 1 -- start start 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1984c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 0x7f4a1406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1984c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1406b730 con 0x7f4a140b1440 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1984c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a140ac390 con 0x7f4a140ab6d0 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1984c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140ac670 con 0x7f4a1406ae20 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1904b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f4a140b1440 0x7f4a140b18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41248/0 (socket says 172.21.15.50:41248) 2023-12-19T16:13:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1904b700 1 -- 172.21.15.50:0/1840428138 learned_addr learned my addr 172.21.15.50:0/1840428138 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a1884a700 1 --2- 172.21.15.50:0/1840428138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 0x7f4a1406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.935+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538243112 0 0) 0x7f4a1406b730 con 0x7f4a140b1440 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49fc003570 con 0x7f4a140b1440 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056083333 0 0) 0x7f4a140ac390 con 0x7f4a140ab6d0 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a1406b730 con 0x7f4a140ab6d0 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1884a700 1 -- 172.21.15.50:0/1840428138 >> v1:172.21.15.125:6789/0 conn(0x7f4a140ab6d0 legacy=0x7f4a140abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1884a700 1 -- 172.21.15.50:0/1840428138 >> v1:172.21.15.50:6789/0 conn(0x7f4a140b1440 legacy=0x7f4a140b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1884a700 1 -- 172.21.15.50:0/1840428138 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a140ac950 con 0x7f4a1406ae20 2023-12-19T16:13:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1884a700 1 --2- 172.21.15.50:0/1840428138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 0x7f4a1406b1f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009910 tx=0x7f4a040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c382928c54cf3fc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04011630 con 0x7f4a1406ae20 2023-12-19T16:13:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04011790 con 0x7f4a1406ae20 2023-12-19T16:13:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a137fe700 1 -- 172.21.15.50:0/1840428138 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04011b30 con 0x7f4a1406ae20 2023-12-19T16:13:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 -- 172.21.15.50:0/1840428138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 msgr2=0x7f4a1406b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 --2- 172.21.15.50:0/1840428138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 0x7f4a1406b1f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009910 tx=0x7f4a040091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 -- 172.21.15.50:0/1840428138 shutdown_connections 2023-12-19T16:13:28.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 --2- 172.21.15.50:0/1840428138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a1406ae20 0x7f4a1406b1f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 -- 172.21.15.50:0/1840428138 >> 172.21.15.50:0/1840428138 conn(0x7f4a140c8cd0 msgr2=0x7f4a140c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 -- 172.21.15.50:0/1840428138 shutdown_connections 2023-12-19T16:13:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.939+0000 7f4a1984c700 1 -- 172.21.15.50:0/1840428138 wait complete. 2023-12-19T16:13:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 Processor -- start 2023-12-19T16:13:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 -- start start 2023-12-19T16:13:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 0x7f4a140c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a140b1440 0x7f4a140c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14157d50 con 0x7f4a1406ae20 2023-12-19T16:13:28.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140c56c0 con 0x7f4a140b1440 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1984c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140c59a0 con 0x7f4a140ab6d0 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1884a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1884a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60706/0 (socket says 172.21.15.50:60706) 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.943+0000 7f4a1884a700 1 -- 172.21.15.50:0/3032823446 learned_addr learned my addr 172.21.15.50:0/3032823446 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1904b700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a140b1440 0x7f4a140c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a13fff700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 0x7f4a140c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 -- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 msgr2=0x7f4a140c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 0x7f4a140c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 -- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a140b1440 msgr2=0x7f4a140c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a140b1440 0x7f4a140c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 -- 172.21.15.50:0/3032823446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a04009420 con 0x7f4a1406ae20 2023-12-19T16:13:28.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a13fff700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 0x7f4a140c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1884a700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009f00 tx=0x7f4a04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0401f070 con 0x7f4a1406ae20 2023-12-19T16:13:28.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04000bc0 con 0x7f4a1406ae20 2023-12-19T16:13:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14157700 con 0x7f4a1406ae20 2023-12-19T16:13:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04025420 con 0x7f4a1406ae20 2023-12-19T16:13:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.947+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1414de40 con 0x7f4a1406ae20 2023-12-19T16:13:28.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.951+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a140c6550 con 0x7f4a1406ae20 2023-12-19T16:13:28.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.951+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4a04023070 con 0x7f4a1406ae20 2023-12-19T16:13:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.951+0000 7f4a10ff9700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 0x7f4a00062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:28.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.951+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f4a0407ddc0 con 0x7f4a1406ae20 2023-12-19T16:13:28.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.951+0000 7f4a13fff700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 0x7f4a00062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:28.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.955+0000 7f4a13fff700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 0x7f4a00062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a08007fd0 tx=0x7f4a08009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:28.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:28.955+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4a140c6550 con 0x7f4a1406ae20 2023-12-19T16:13:29.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.183+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4a140c6550 con 0x7f4a1406ae20 2023-12-19T16:13:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.195+0000 7f4a10ff9700 1 -- 172.21.15.50:0/3032823446 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4a140c6550 con 0x7f4a1406ae20 2023-12-19T16:13:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 msgr2=0x7f4a00062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 0x7f4a00062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a08007fd0 tx=0x7f4a08009040 comp rx=0 tx=0).stop 2023-12-19T16:13:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 msgr2=0x7f4a1414d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.204 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009f00 tx=0x7f4a04017000 comp rx=0 tx=0).stop 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 shutdown_connections 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4a00060670 0x7f4a00062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a140b1440 0x7f4a140c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4a140ab6d0 0x7f4a140c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 --2- 172.21.15.50:0/3032823446 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a1406ae20 0x7f4a1414d900 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 >> 172.21.15.50:0/3032823446 conn(0x7f4a140c8cd0 msgr2=0x7f4a140ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 shutdown_connections 2023-12-19T16:13:29.205 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.199+0000 7f4a1984c700 1 -- 172.21.15.50:0/3032823446 wait complete. 2023-12-19T16:13:29.207 INFO:teuthology.orchestra.run.smithi050.stderr:updated caps for client.0 2023-12-19T16:13:29.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:29.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.387+0000 7fbf14eae700 1 Processor -- start 2023-12-19T16:13:29.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 -- start start 2023-12-19T16:13:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf10156c90 0x7fbf100cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.396 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf1015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100cd5a0 con 0x7fbf100cda30 2023-12-19T16:13:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf10157470 con 0x7fbf10156c90 2023-12-19T16:13:29.397 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf14eae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100cc990 con 0x7fbf10157060 2023-12-19T16:13:29.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf10156c90 0x7fbf100cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbf10156c90 0x7fbf100cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60718/0 (socket says 172.21.15.50:60718) 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 -- 172.21.15.50:0/4083330236 learned_addr learned my addr 172.21.15.50:0/4083330236 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0ed9d700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf1015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 -- 172.21.15.50:0/4083330236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 msgr2=0x7fbf1015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf1015ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 -- 172.21.15.50:0/4083330236 >> v1:172.21.15.125:6789/0 conn(0x7fbf100cda30 legacy=0x7fbf100c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.391+0000 7fbf0e59c700 1 -- 172.21.15.50:0/4083330236 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100ccc70 con 0x7fbf10156c90 2023-12-19T16:13:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf100cc3b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fbf000099b0 tx=0x7fbf00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664c1d3d84f89d74 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf0dd9b700 1 -- 172.21.15.50:0/4083330236 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000b820 con 0x7fbf10156c90 2023-12-19T16:13:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf0dd9b700 1 -- 172.21.15.50:0/4083330236 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0000b980 con 0x7fbf10156c90 2023-12-19T16:13:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf0dd9b700 1 -- 172.21.15.50:0/4083330236 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000bd20 con 0x7fbf10156c90 2023-12-19T16:13:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- 172.21.15.50:0/4083330236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 msgr2=0x7fbf100cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf100cc3b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fbf000099b0 tx=0x7fbf00009250 comp rx=0 tx=0).stop 2023-12-19T16:13:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- 172.21.15.50:0/4083330236 shutdown_connections 2023-12-19T16:13:29.405 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf1015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 --2- 172.21.15.50:0/4083330236 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf100cc3b0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- 172.21.15.50:0/4083330236 >> 172.21.15.50:0/4083330236 conn(0x7fbf1006c7c0 msgr2=0x7fbf1006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- 172.21.15.50:0/4083330236 shutdown_connections 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- 172.21.15.50:0/4083330236 wait complete. 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 Processor -- start 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.395+0000 7fbf14eae700 1 -- start start 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 0x7fbf101575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf10158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c81a0 con 0x7fbf10156c90 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10068550 con 0x7fbf100cda30 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10067c20 con 0x7fbf10157060 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60730/0 (socket says 172.21.15.50:60730) 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 -- 172.21.15.50:0/3294749298 learned_addr learned my addr 172.21.15.50:0/3294749298 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf0ed9d700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf10158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 0x7fbf101575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 -- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 msgr2=0x7fbf10158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf10158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 -- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 msgr2=0x7fbf101575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 0x7fbf101575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 -- 172.21.15.50:0/3294749298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000094f0 con 0x7fbf10156c90 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf0ed9d700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf10158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:29.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 0x7fbf101575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf07fff700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c600 tx=0x7fbef8007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8011590 con 0x7fbf10156c90 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8011710 con 0x7fbf10156c90 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf10067f00 con 0x7fbf10156c90 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8011ad0 con 0x7fbf10156c90 2023-12-19T16:13:29.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.399+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf100689e0 con 0x7fbf10156c90 2023-12-19T16:13:29.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.403+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf100686e0 con 0x7fbf10156c90 2023-12-19T16:13:29.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.403+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbef801c460 con 0x7fbf10156c90 2023-12-19T16:13:29.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.403+0000 7fbf067fc700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 0x7fbef0062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:29.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.407+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fbef807c820 con 0x7fbf10156c90 2023-12-19T16:13:29.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.407+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 0x7fbef0062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:29.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.407+0000 7fbf0e59c700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 0x7fbef0062bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf000099b0 tx=0x7fbf00016000 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:29.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.411+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbef804cbd0 con 0x7fbf10156c90 2023-12-19T16:13:29.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:13:29.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:13:29.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.615+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf10068380 con 0x7fbf10156c90 2023-12-19T16:13:29.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf067fc700 1 -- 172.21.15.50:0/3294749298 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fbef80501f0 con 0x7fbf10156c90 2023-12-19T16:13:29.623 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:29.623 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:29.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 msgr2=0x7fbef0062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 0x7fbef0062bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbf000099b0 tx=0x7fbf00016000 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 msgr2=0x7fbf10157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c600 tx=0x7fbef8007f60 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 shutdown_connections 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbef0060740 0x7fbef0062bf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbf10157060 0x7fbf10158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbf10156c90 0x7fbf10157af0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 --2- 172.21.15.50:0/3294749298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100cda30 0x7fbf101575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 >> 172.21.15.50:0/3294749298 conn(0x7fbf1006c7c0 msgr2=0x7fbf100b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 shutdown_connections 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:29.619+0000 7fbf14eae700 1 -- 172.21.15.50:0/3294749298 wait complete. 2023-12-19T16:13:29.626 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:30.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T16:13:23.400827+0000', 'modified': '2023-12-19T16:13:26.977563+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10389}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10389': {'gid': 10389, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6835/4273555224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6833', 'nonce': 4273555224}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 4273555224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T16:13:30.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T16:13:30.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T16:13:30.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.211+0000 7fdd2ee6e700 1 Processor -- start 2023-12-19T16:13:30.218 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.215+0000 7fdd2ee6e700 1 -- start start 2023-12-19T16:13:30.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2ee6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 0x7fdd2813f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2ee6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2806dc00 con 0x7fdd280c7d90 2023-12-19T16:13:30.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2ee6e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2813fb30 con 0x7fdd2813f6b0 2023-12-19T16:13:30.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2ee6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2806e090 con 0x7fdd280c8160 2023-12-19T16:13:30.222 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2cc0a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdd280c7d90 0x7fdd2806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41250/0 (socket says 172.21.15.50:41250) 2023-12-19T16:13:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/1800185032 learned_addr learned my addr 172.21.15.50:0/1800185032 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd27fff700 1 --2- 172.21.15.50:0/1800185032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 0x7fdd2813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815669736 0 0) 0x7fdd2806dc00 con 0x7fdd280c7d90 2023-12-19T16:13:30.223 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd18003570 con 0x7fdd280c7d90 2023-12-19T16:13:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877797887 0 0) 0x7fdd18003570 con 0x7fdd280c7d90 2023-12-19T16:13:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd27fff700 1 -- 172.21.15.50:0/1800185032 >> v1:172.21.15.125:6789/0 conn(0x7fdd2813f6b0 legacy=0x7fdd280ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd27fff700 1 -- 172.21.15.50:0/1800185032 >> v1:172.21.15.50:6789/0 conn(0x7fdd280c7d90 legacy=0x7fdd2806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd27fff700 1 -- 172.21.15.50:0/1800185032 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2806cff0 con 0x7fdd280c8160 2023-12-19T16:13:30.224 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd27fff700 1 --2- 172.21.15.50:0/1800185032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 0x7fdd2813f170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009940 tx=0x7fdd140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0b20f84b1a78ab server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c001470 con 0x7fdd280c8160 2023-12-19T16:13:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd14011420 con 0x7fdd280c8160 2023-12-19T16:13:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.219+0000 7fdd277fe700 1 -- 172.21.15.50:0/1800185032 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd140117c0 con 0x7fdd280c8160 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/1800185032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 msgr2=0x7fdd2813f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/1800185032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 0x7fdd2813f170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009940 tx=0x7fdd140091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/1800185032 shutdown_connections 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/1800185032 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c8160 0x7fdd2813f170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/1800185032 >> 172.21.15.50:0/1800185032 conn(0x7fdd28067b60 msgr2=0x7fdd280ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/1800185032 shutdown_connections 2023-12-19T16:13:30.226 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/1800185032 wait complete. 2023-12-19T16:13:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 Processor -- start 2023-12-19T16:13:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.223+0000 7fdd2ee6e700 1 -- start start 2023-12-19T16:13:30.230 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c7d90 0x7fdd280c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 0x7fdd28151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2813ad70 con 0x7fdd2813f6b0 2023-12-19T16:13:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28150df0 con 0x7fdd280c8160 2023-12-19T16:13:30.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280ca410 con 0x7fdd280c7d90 2023-12-19T16:13:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60744/0 (socket says 172.21.15.50:60744) 2023-12-19T16:13:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 -- 172.21.15.50:0/3347890467 learned_addr learned my addr 172.21.15.50:0/3347890467 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c7d90 0x7fdd280c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd27fff700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 0x7fdd28151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 -- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c7d90 msgr2=0x7fdd280c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c7d90 0x7fdd280c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 -- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 msgr2=0x7fdd28151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 0x7fdd28151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 -- 172.21.15.50:0/3347890467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14009420 con 0x7fdd2813f6b0 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd27fff700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 0x7fdd28151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000c960 tx=0x7fdd20001740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd24ff9700 1 -- 172.21.15.50:0/3347890467 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20012630 con 0x7fdd2813f6b0 2023-12-19T16:13:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd24ff9700 1 -- 172.21.15.50:0/3347890467 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20012790 con 0x7fdd2813f6b0 2023-12-19T16:13:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd24ff9700 1 -- 172.21.15.50:0/3347890467 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20012b50 con 0x7fdd2813f6b0 2023-12-19T16:13:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.227+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd281501e0 con 0x7fdd2813f6b0 2023-12-19T16:13:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.231+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd281507a0 con 0x7fdd2813f6b0 2023-12-19T16:13:30.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.231+0000 7fdd24ff9700 1 -- 172.21.15.50:0/3347890467 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdd20003030 con 0x7fdd2813f6b0 2023-12-19T16:13:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd24ff9700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 0x7fdd0c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 Processor -- start 2023-12-19T16:13:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd24ff9700 1 -- 172.21.15.50:0/3347890467 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fdd2007cac0 con 0x7fdd2813f6b0 2023-12-19T16:13:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 -- start start 2023-12-19T16:13:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 0x7fdd0c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a37440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a37980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1964f30 con 0x17b9ae0 2023-12-19T16:13:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1960ed0 con 0x1804790 2023-12-19T16:13:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x19270b0 con 0x17b9710 2023-12-19T16:13:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 0x7fdd0c062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c001ed0 tx=0x7fdd1c001a20 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60756/0 (socket says 172.21.15.50:60756) 2023-12-19T16:13:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2975939791 learned_addr learned my addr 172.21.15.50:0/2975939791 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd27fff700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a37440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a37980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 msgr2=0x1a37440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a37440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 msgr2=0x1a37980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a37980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2975939791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x18059e0 con 0x17b9ae0 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.235+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a37980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00fd80 tx=0x7fdd1c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ea901ff190709f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd11ffb700 1 -- 172.21.15.50:0/2975939791 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c015b00 con 0x17b9ae0 2023-12-19T16:13:30.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd11ffb700 1 -- 172.21.15.50:0/2975939791 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c015c60 con 0x17b9ae0 2023-12-19T16:13:30.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd11ffb700 1 -- 172.21.15.50:0/2975939791 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0133d0 con 0x17b9ae0 2023-12-19T16:13:30.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- 172.21.15.50:0/2975939791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 msgr2=0x1a36f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00fd80 tx=0x7fdd1c016000 comp rx=0 tx=0).stop 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- 172.21.15.50:0/2975939791 shutdown_connections 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a37980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a37440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 --2- 172.21.15.50:0/2975939791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a36f00 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- 172.21.15.50:0/2975939791 >> 172.21.15.50:0/2975939791 conn(0x17b9eb0 msgr2=0x19f3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- 172.21.15.50:0/2975939791 shutdown_connections 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- 172.21.15.50:0/2975939791 wait complete. 2023-12-19T16:13:30.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 -- start start 2023-12-19T16:13:30.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.239+0000 7fdd31107740 1 Processor -- start 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a3b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1926d50 con 0x17b9ae0 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x18ff970 con 0x1804790 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x19606e0 con 0x17b9710 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 10 client.0 _ll_register_callbacks cb 0x19aa6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60764/0 (socket says 172.21.15.50:60764) 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 -- 172.21.15.50:0/167381138 learned_addr learned my addr 172.21.15.50:0/167381138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a3b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 -- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 msgr2=0x1a3b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a3b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 -- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 msgr2=0x1a1ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 -- 172.21.15.50:0/167381138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c00fa30 con 0x17b9ae0 2023-12-19T16:13:30.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd27fff700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009860 tx=0x7fdd14017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9f3cc8ac24eabd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T16:13:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14024070 con 0x17b9ae0 2023-12-19T16:13:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fdd1402ac90 con 0x17b9ae0 2023-12-19T16:13:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x18ffcc0 con 0x17b9ae0 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd1402ae50 con 0x17b9ae0 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 10 client.10533 fetch_fsmap learned FSMap version 99 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) v3 -- 0x192f370 con 0x17b9ae0 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x18ffcc0 con 0x17b9ae0 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 99) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdd14018030 con 0x17b9ae0 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.243+0000 7fdd31107740 10 client.10533 fetch_fsmap finished waiting for FSMap version 99 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 10 client.10533 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T16:13:30.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 20 client.10533 populate_metadata read hostname 'smithi050' 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a446a0 0x1a46a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 4 client.10533 mds_command: new command op to 10389 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 --> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x19033e0 con 0x1a446a0 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a446a0 0x1a46a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd06ffd700 10 client.10533 ms_handle_connect on v2:172.21.15.50:6833/4273555224 2023-12-19T16:13:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a446a0 0x1a46a80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21748e5250ba92f server_cookie=5c7d93707ec0957d in_seq=0 out_seq=0 2023-12-19T16:13:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd06ffd700 1 -- 172.21.15.50:0/167381138 <== mds.0 v2:172.21.15.50:6833/4273555224 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x19033e0 con 0x1a446a0 2023-12-19T16:13:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd06ffd700 10 client.10533 handle_command_reply: tid=0 2023-12-19T16:13:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 client.10533 shutdown 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 msgr2=0x1a3bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009860 tx=0x7fdd14017000 comp rx=0 tx=0).stop 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 shutdown_connections 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a446a0 0x1a46a80 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9ae0 0x1a3bad0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 --2- 172.21.15.50:0/167381138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9710 0x1a3b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd06ffd700 0 client.10533 ms_handle_reset on v2:172.21.15.50:6833/4273555224 2023-12-19T16:13:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 >> 172.21.15.50:0/167381138 conn(0x17b9eb0 msgr2=0x19ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 shutdown_connections 2023-12-19T16:13:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 1 -- 172.21.15.50:0/167381138 wait complete. 2023-12-19T16:13:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.247+0000 7fdd31107740 20 client.10533 trim_cache size 0 max 16384 2023-12-19T16:13:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 Processor -- start 2023-12-19T16:13:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- start start 2023-12-19T16:13:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a1c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1965ef0 con 0x17b9710 2023-12-19T16:13:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x1934f40 con 0x1804790 2023-12-19T16:13:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x1936550 con 0x17b9ae0 2023-12-19T16:13:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60778/0 (socket says 172.21.15.50:60778) 2023-12-19T16:13:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/1205148451 learned_addr learned my addr 172.21.15.50:0/1205148451 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 msgr2=0x1a1c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a1c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 msgr2=0x1a1cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/1205148451 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1915d80 con 0x17b9710 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c017010 tx=0x7fdd1c022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89c2534242f172c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd127fc700 1 -- 172.21.15.50:0/1205148451 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c033070 con 0x17b9710 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd127fc700 1 -- 172.21.15.50:0/1205148451 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c0016e0 con 0x17b9710 2023-12-19T16:13:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd127fc700 1 -- 172.21.15.50:0/1205148451 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c015b00 con 0x17b9710 2023-12-19T16:13:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- 172.21.15.50:0/1205148451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 msgr2=0x1a700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c017010 tx=0x7fdd1c022040 comp rx=0 tx=0).stop 2023-12-19T16:13:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- 172.21.15.50:0/1205148451 shutdown_connections 2023-12-19T16:13:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a1cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a1c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 --2- 172.21.15.50:0/1205148451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a700d0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.271+0000 7fdd31107740 1 -- 172.21.15.50:0/1205148451 >> 172.21.15.50:0/1205148451 conn(0x17b9eb0 msgr2=0x19ed1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.275+0000 7fdd31107740 1 -- 172.21.15.50:0/1205148451 shutdown_connections 2023-12-19T16:13:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.275+0000 7fdd31107740 1 -- 172.21.15.50:0/1205148451 wait complete. 2023-12-19T16:13:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.275+0000 7fdd31107740 1 -- start start 2023-12-19T16:13:30.280 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.275+0000 7fdd31107740 1 Processor -- start 2023-12-19T16:13:30.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a18030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a18570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x1836650 con 0x17b9710 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x19649d0 con 0x1804790 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x195e890 con 0x17b9ae0 2023-12-19T16:13:30.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd31107740 10 client.0 _ll_register_callbacks cb 0x17859d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60784/0 (socket says 172.21.15.50:60784) 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2216687828 learned_addr learned my addr 172.21.15.50:0/2216687828 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd27fff700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a18030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2d40b700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a18570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 msgr2=0x1a18030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a18030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 msgr2=0x1a18570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a18570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 -- 172.21.15.50:0/2216687828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x18059e0 con 0x17b9710 2023-12-19T16:13:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd2cc0a700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0011e0 tx=0x7fdd1c033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e9f3650d2e68eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.279+0000 7fdd06ffd700 10 client.0 ms_handle_connect on v2:172.21.15.50:3300/0 2023-12-19T16:13:30.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.283+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c013a20 con 0x17b9710 2023-12-19T16:13:30.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.283+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mon.0 v2:172.21.15.50:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7fdd1c013b80 con 0x17b9710 2023-12-19T16:13:30.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd1c015c60 con 0x17b9710 2023-12-19T16:13:30.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd1c03f410 con 0x17b9710 2023-12-19T16:13:30.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 10 client.10539 fetch_fsmap learned FSMap version 99 2023-12-19T16:13:30.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) v3 -- 0x190a130 con 0x17b9710 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mon.0 v2:172.21.15.50:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c015c60 con 0x17b9710 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mon.0 v2:172.21.15.50:3300/0 5 ==== fsmap(e 99) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdd1c03f570 con 0x17b9710 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 10 client.10539 fetch_fsmap finished waiting for FSMap version 99 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 10 client.10539 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 20 client.10539 populate_metadata read hostname 'smithi050' 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a18e70 0x1a98730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 4 client.10539 mds_command: new command op to 10389 tid=0[{"prefix": "status"}] 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 --> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1935410 con 0x1a18e70 2023-12-19T16:13:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd27fff700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a18e70 0x1a98730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 10 client.10539 ms_handle_connect on v2:172.21.15.50:6833/4273555224 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd27fff700 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a18e70 0x1a98730 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68edacae7412e9d1 server_cookie=d05cdc9f3fc0cad3 in_seq=0 out_seq=0 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 1 -- 172.21.15.50:0/2216687828 <== mds.0 v2:172.21.15.50:6833/4273555224 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x1935410 con 0x1a18e70 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 10 client.10539 handle_command_reply: tid=0 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 client.10539 shutdown 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 msgr2=0x1a17af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0011e0 tx=0x7fdd1c033000 comp rx=0 tx=0).stop 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 shutdown_connections 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:6833/4273555224,v1:172.21.15.50:6835/4273555224] conn(0x1a18e70 0x1a98730 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x1804790 0x1a18570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x17b9ae0 0x1a18030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 --2- 172.21.15.50:0/2216687828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x17b9710 0x1a17af0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd06ffd700 0 client.10539 ms_handle_reset on v2:172.21.15.50:6833/4273555224 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 >> 172.21.15.50:0/2216687828 conn(0x17b9eb0 msgr2=0x19edad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 shutdown_connections 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.287+0000 7fdd31107740 1 -- 172.21.15.50:0/2216687828 wait complete. 2023-12-19T16:13:30.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd31107740 20 client.10539 trim_cache size 0 max 16384 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_fsid": "8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d", 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "whoami": 0, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 10389, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "want_state": "up:active", 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "up:active", 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs", 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "rank_uptime": 6.3293367009999999, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "mdsmap_epoch": 99, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch": 213, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap_epoch_barrier": 213, 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout: "uptime": 12.25799752 2023-12-19T16:13:30.296 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:13:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 msgr2=0x7fdd0c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 0x7fdd0c062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c001ed0 tx=0x7fdd1c001a20 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 msgr2=0x7fdd28156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000c960 tx=0x7fdd20001740 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 shutdown_connections 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdd0c0606a0 0x7fdd0c062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd280c8160 0x7fdd28151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd2813f6b0 0x7fdd28156f60 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 --2- 172.21.15.50:0/3347890467 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdd280c7d90 0x7fdd280c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 >> 172.21.15.50:0/3347890467 conn(0x7fdd28067b60 msgr2=0x7fdd2815ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 shutdown_connections 2023-12-19T16:13:30.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.291+0000 7fdd2ee6e700 1 -- 172.21.15.50:0/3347890467 wait complete. 2023-12-19T16:13:30.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T16:13:30.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.471+0000 7febd682e700 1 Processor -- start 2023-12-19T16:13:30.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.471+0000 7febd682e700 1 -- start start 2023-12-19T16:13:30.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd682e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7febd0140ae0 0x7febd00680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd682e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0068630 0x7febd00ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.479 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd682e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd006e150 con 0x7febd0140710 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd682e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd0068ab0 con 0x7febd0140ae0 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd682e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd006d540 con 0x7febd0068630 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febcffff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7febd0140710 0x7febd00c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41254/0 (socket says 172.21.15.50:41254) 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febcffff700 1 -- 172.21.15.50:0/3371224289 learned_addr learned my addr 172.21.15.50:0/3371224289 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febc7fff700 1 --2- 172.21.15.50:0/3371224289 >> v2:172.21.15.125:3300/0 conn(0x7febd0140ae0 0x7febd00680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febd4dcb700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0068630 0x7febd00ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febcf7fe700 1 -- 172.21.15.50:0/3371224289 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630027049 0 0) 0x7febd006e150 con 0x7febd0140710 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febcf7fe700 1 -- 172.21.15.50:0/3371224289 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc003570 con 0x7febd0140710 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febc7fff700 1 -- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0068630 msgr2=0x7febd00ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febc7fff700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0068630 0x7febd00ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febc7fff700 1 -- 172.21.15.50:0/3371224289 >> v1:172.21.15.50:6789/0 conn(0x7febd0140710 legacy=0x7febd00c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.481 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.475+0000 7febc7fff700 1 -- 172.21.15.50:0/3371224289 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd006d820 con 0x7febd0140ae0 2023-12-19T16:13:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febc7fff700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0140ae0 0x7febd00680f0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7febb80098e0 tx=0x7febb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a9d3b9ea6f8031a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febcf7fe700 1 -- 172.21.15.50:0/3371224289 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8011630 con 0x7febd0140ae0 2023-12-19T16:13:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febcf7fe700 1 -- 172.21.15.50:0/3371224289 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb8011790 con 0x7febd0140ae0 2023-12-19T16:13:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febcf7fe700 1 -- 172.21.15.50:0/3371224289 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8011b30 con 0x7febd0140ae0 2023-12-19T16:13:30.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 -- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0140ae0 msgr2=0x7febd00680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.483 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0140ae0 0x7febd00680f0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7febb80098e0 tx=0x7febb80091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 -- 172.21.15.50:0/3371224289 shutdown_connections 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0068630 0x7febd00ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 --2- 172.21.15.50:0/3371224289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0140ae0 0x7febd00680f0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 -- 172.21.15.50:0/3371224289 >> 172.21.15.50:0/3371224289 conn(0x7febd00c8850 msgr2=0x7febd00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 -- 172.21.15.50:0/3371224289 shutdown_connections 2023-12-19T16:13:30.484 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.479+0000 7febd682e700 1 -- 172.21.15.50:0/3371224289 wait complete. 2023-12-19T16:13:30.485 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 Processor -- start 2023-12-19T16:13:30.486 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 -- start start 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0068630 0x7febd00b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0140ae0 0x7febd00c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00c0b80 con 0x7febd0140710 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd00c2a60 con 0x7febd0068630 2023-12-19T16:13:30.487 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd682e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd00a8bc0 con 0x7febd0140ae0 2023-12-19T16:13:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60792/0 (socket says 172.21.15.50:60792) 2023-12-19T16:13:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 -- 172.21.15.50:0/3227866140 learned_addr learned my addr 172.21.15.50:0/3227866140 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febcffff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0068630 0x7febd00b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febd4dcb700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0140ae0 0x7febd00c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 -- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0140ae0 msgr2=0x7febd00c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0140ae0 0x7febd00c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 -- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0068630 msgr2=0x7febd00b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0068630 0x7febd00b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 -- 172.21.15.50:0/3227866140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8009420 con 0x7febd0140710 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.483+0000 7febc7fff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7febb8009ed0 tx=0x7febb8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.489 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8000d10 con 0x7febd0140710 2023-12-19T16:13:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb8011900 con 0x7febd0140710 2023-12-19T16:13:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb8025670 con 0x7febd0140710 2023-12-19T16:13:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd00a8ea0 con 0x7febd0140710 2023-12-19T16:13:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd00c0dd0 con 0x7febd0140710 2023-12-19T16:13:30.492 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd00c0ff0 con 0x7febd0140710 2023-12-19T16:13:30.493 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.487+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7febb8023070 con 0x7febd0140710 2023-12-19T16:13:30.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.491+0000 7febccff9700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 0x7febb0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:30.494 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.491+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7febb807dfc0 con 0x7febd0140710 2023-12-19T16:13:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.491+0000 7febcffff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 0x7febb0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.491+0000 7febcffff700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 0x7febb0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febd00b03f0 tx=0x7febc0009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:30.501 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.495+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7febb804e250 con 0x7febd0140710 2023-12-19T16:13:30.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.707+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7febd0071120 con 0x7febd0140710 2023-12-19T16:13:30.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.707+0000 7febccff9700 1 -- 172.21.15.50:0/3227866140 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) v1 ==== 53+0+73 (secure 0 0 0) 0x7febb8015020 con 0x7febd0140710 2023-12-19T16:13:30.712 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 msgr2=0x7febb0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 0x7febb0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febd00b03f0 tx=0x7febc0009040 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 msgr2=0x7febd013dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7febb8009ed0 tx=0x7febb8017000 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 shutdown_connections 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7febb0060670 0x7febb0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7febd0140ae0 0x7febd00c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febd0140710 0x7febd013dac0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 --2- 172.21.15.50:0/3227866140 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febd0068630 0x7febd00b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 >> 172.21.15.50:0/3227866140 conn(0x7febd00c8850 msgr2=0x7febd00c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:30.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 shutdown_connections 2023-12-19T16:13:30.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:30.711+0000 7febd682e700 1 -- 172.21.15.50:0/3227866140 wait complete. 2023-12-19T16:13:30.737 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T16:13:30.737 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T16:13:30.737 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T16:13:30.737 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T16:13:30.737 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:30.738 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T16:13:30.738 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T16:13:30.738 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T16:13:30.738 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:30.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:30.738 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T16:13:30.757 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.758 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 36739sec preferred_lft 36739sec 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T16:13:30.759 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T16:13:30.760 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:30.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:30.761 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T16:13:30.843 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T16:13:30.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:30.847 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T16:13:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T16:13:30.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:30.912 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T16:13:30.913 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T16:13:30.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T16:13:30.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T16:13:30.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T16:13:30.913 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T16:13:31.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:31.049 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T16:13:31.067 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T16:13:31.067 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T16:13:31.068 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:31.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:31.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T16:13:31.153 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T16:13:31.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:13:31.154 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:31.219 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:31.219 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:13:31.219 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:31.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:13:31.379 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:13:31.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:13:31.461 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:13:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:13:31.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:31.464 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:13:31.524 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T16:13:31.525 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T16:13:31.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:13:31.619 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:13:31.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:13:31.696 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:13:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:13:31.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:31.699 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:13:31.714 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T16:13:31.711+0000 7ff7729c1380 -1 init, newargv = 0x55e1a6409e30 newargc=15 2023-12-19T16:13:31.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62050]: starting ceph client 2023-12-19T16:13:31.727 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62050]: starting fuse 2023-12-19T16:13:31.729 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T16:13:31.730 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T16:13:31.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T16:13:31.730 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T16:13:31.730 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T16:13:31.731 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T16:13:31.732 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T16:13:31.893 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.62050.asok 2023-12-19T16:13:32.078 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62050.asok 2023-12-19T16:13:32.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:32.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62050.asok status 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "62050", 2023-12-19T16:13:32.270 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 10551, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 10551 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3060282734 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:32.271 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "192.168.0.1:0", 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3060282734 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.10551 192.168.0.1:0/3060282734", 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "192.168.0.1:0/3060282734", 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 99, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 213, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T16:13:32.272 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:13:32.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:32.284 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T16:13:32.284 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:32.307 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T16:13:32.307 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:32.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:32.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:32.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T16:13:32.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.599+0000 7f0931ef0700 1 Processor -- start 2023-12-19T16:13:32.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.599+0000 7f0931ef0700 1 -- start start 2023-12-19T16:13:32.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f0931ef0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f092c06ae50 0x7f092c163bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:32.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f0931ef0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c06e160 con 0x7f092c158e50 2023-12-19T16:13:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f0931ef0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092c06d0c0 con 0x7f092c159220 2023-12-19T16:13:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f0931ef0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f092c06ced0 con 0x7f092c06ae50 2023-12-19T16:13:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092b7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f092c158e50 0x7f092c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41264/0 (socket says 172.21.15.50:41264) 2023-12-19T16:13:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092b7fe700 1 -- 172.21.15.50:0/2421674875 learned_addr learned my addr 172.21.15.50:0/2421674875 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:32.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092bfff700 1 --2- 172.21.15.50:0/2421674875 >> v2:172.21.15.125:3300/0 conn(0x7f092c06ae50 0x7f092c163bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:32.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092a7fc700 1 -- 172.21.15.50:0/2421674875 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256597564 0 0) 0x7f092c06e160 con 0x7f092c158e50 2023-12-19T16:13:32.608 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092a7fc700 1 -- 172.21.15.50:0/2421674875 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914003570 con 0x7f092c158e50 2023-12-19T16:13:32.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092bfff700 1 -- 172.21.15.50:0/2421674875 >> v1:172.21.15.125:6789/0 conn(0x7f092c159220 legacy=0x7f092c06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092bfff700 1 -- 172.21.15.50:0/2421674875 >> v1:172.21.15.50:6789/0 conn(0x7f092c158e50 legacy=0x7f092c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.609 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092bfff700 1 -- 172.21.15.50:0/2421674875 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c06d3a0 con 0x7f092c06ae50 2023-12-19T16:13:32.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092bfff700 1 --2- 172.21.15.50:0/2421674875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c06ae50 0x7f092c163bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0924009e90 tx=0x7f0924009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d876bb3b5435282 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:32.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.603+0000 7f092a7fc700 1 -- 172.21.15.50:0/2421674875 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400e070 con 0x7f092c06ae50 2023-12-19T16:13:32.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f092a7fc700 1 -- 172.21.15.50:0/2421674875 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092400bc40 con 0x7f092c06ae50 2023-12-19T16:13:32.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f092a7fc700 1 -- 172.21.15.50:0/2421674875 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924012430 con 0x7f092c06ae50 2023-12-19T16:13:32.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 -- 172.21.15.50:0/2421674875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c06ae50 msgr2=0x7f092c163bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 --2- 172.21.15.50:0/2421674875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c06ae50 0x7f092c163bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0924009e90 tx=0x7f0924009780 comp rx=0 tx=0).stop 2023-12-19T16:13:32.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 -- 172.21.15.50:0/2421674875 shutdown_connections 2023-12-19T16:13:32.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 --2- 172.21.15.50:0/2421674875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c06ae50 0x7f092c163bc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 -- 172.21.15.50:0/2421674875 >> 172.21.15.50:0/2421674875 conn(0x7f092c067b60 msgr2=0x7f092c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:32.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 -- 172.21.15.50:0/2421674875 shutdown_connections 2023-12-19T16:13:32.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.607+0000 7f0931ef0700 1 -- 172.21.15.50:0/2421674875 wait complete. 2023-12-19T16:13:32.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 Processor -- start 2023-12-19T16:13:32.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 -- start start 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 0x7f092c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c159220 0x7f092c15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c16d2e0 con 0x7f092c158e50 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0ca5a0 con 0x7f092c159220 2023-12-19T16:13:32.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f0931ef0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0c9c70 con 0x7f092c06ae50 2023-12-19T16:13:32.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:32.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60806/0 (socket says 172.21.15.50:60806) 2023-12-19T16:13:32.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 -- 172.21.15.50:0/1904290924 learned_addr learned my addr 172.21.15.50:0/1904290924 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:32.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092b7fe700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 0x7f092c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:32.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092bfff700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c159220 0x7f092c15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 -- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 msgr2=0x7f092c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 0x7f092c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 -- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c159220 msgr2=0x7f092c15bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c159220 0x7f092c15bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.611+0000 7f092affd700 1 -- 172.21.15.50:0/1904290924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009a20 con 0x7f092c158e50 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f092b7fe700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 0x7f092c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f092affd700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f092000c590 tx=0x7f0920007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920011590 con 0x7f092c158e50 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920011710 con 0x7f092c158e50 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c0c9f50 con 0x7f092c158e50 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920011ad0 con 0x7f092c158e50 2023-12-19T16:13:32.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c1539f0 con 0x7f092c158e50 2023-12-19T16:13:32.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.615+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c0ca750 con 0x7f092c158e50 2023-12-19T16:13:32.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.619+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f092001b3b0 con 0x7f092c158e50 2023-12-19T16:13:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.619+0000 7f09137fe700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 0x7f0918062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.619+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f092007bee0 con 0x7f092c158e50 2023-12-19T16:13:32.624 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.619+0000 7f092b7fe700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 0x7f0918062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:32.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.623+0000 7f092b7fe700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 0x7f0918062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f091c0058d0 tx=0x7f091c009040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:32.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.627+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f092004c170 con 0x7f092c158e50 2023-12-19T16:13:32.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.823+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f092c071120 con 0x7f09180606a0 2023-12-19T16:13:32.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.827+0000 7f09137fe700 1 -- 172.21.15.50:0/1904290924 <== mgr.10012 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f092c071120 con 0x7f09180606a0 2023-12-19T16:13:32.832 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2023-12-19T16:13:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2023-12-19T16:13:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "cephfs" 2023-12-19T16:13:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:32.832 INFO:teuthology.orchestra.run.smithi050.stdout:] 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 msgr2=0x7f0918062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 0x7f0918062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f091c0058d0 tx=0x7f091c009040 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 msgr2=0x7f092c15b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f092000c590 tx=0x7f0920007f50 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 shutdown_connections 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f09180606a0 0x7f0918062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f092c159220 0x7f092c15bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f092c158e50 0x7f092c15b9b0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 --2- 172.21.15.50:0/1904290924 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f092c06ae50 0x7f092c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 >> 172.21.15.50:0/1904290924 conn(0x7f092c067b60 msgr2=0x7f092c0b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:32.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 shutdown_connections 2023-12-19T16:13:32.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:32.831+0000 7f0931ef0700 1 -- 172.21.15.50:0/1904290924 wait complete. 2023-12-19T16:13:32.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T16:13:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 Processor -- start 2023-12-19T16:13:33.027 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 -- start start 2023-12-19T16:13:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f166406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640b0280 con 0x7f166406cd00 2023-12-19T16:13:33.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166406d130 con 0x7f1664158fa0 2023-12-19T16:13:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f166a878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166406b4e0 con 0x7f1664159370 2023-12-19T16:13:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f1668e15700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f166406cd00 0x7f16640b06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41280/0 (socket says 172.21.15.50:41280) 2023-12-19T16:13:33.029 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f1668e15700 1 -- 172.21.15.50:0/4261999394 learned_addr learned my addr 172.21.15.50:0/4261999394 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.023+0000 7f165bfff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f166406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817749919 0 0) 0x7f16640b0280 con 0x7f166406cd00 2023-12-19T16:13:33.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1650003570 con 0x7f166406cd00 2023-12-19T16:13:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610138107 0 0) 0x7f166406d130 con 0x7f1664158fa0 2023-12-19T16:13:33.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640b0280 con 0x7f1664158fa0 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3752012437 0 0) 0x7f1650003570 con 0x7f166406cd00 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 msgr2=0x7f166406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f166406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 >> v1:172.21.15.125:6789/0 conn(0x7f1664158fa0 legacy=0x7f166406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166406b7c0 con 0x7f166406cd00 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f165c0017f0 con 0x7f166406cd00 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 >> v1:172.21.15.50:6789/0 conn(0x7f166406cd00 legacy=0x7f16640b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f1650004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1650004810 0x7f1650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f16500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166406b4e0 con 0x7f1650003c20 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165000bb40 con 0x7f1650004810 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165000bd00 con 0x7f1664159370 2023-12-19T16:13:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f1650004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1668e15700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f16500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f165bfff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1650004810 0x7f1650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 -- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 msgr2=0x7f16500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664159370 0x7f16500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 -- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1650004810 msgr2=0x7f1650004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1650004810 0x7f1650004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.034 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 -- 172.21.15.50:0/4261999394 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1650003570 con 0x7f1650003c20 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f165bfff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1650004810 0x7f1650004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f1663fff700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f1650004010 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1654001a10 tx=0x7f1654007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c62fd255c4ffef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165400c5d0 con 0x7f1650003c20 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165400c730 con 0x7f1650003c20 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.027+0000 7f16637fe700 1 -- 172.21.15.50:0/4261999394 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165400cad0 con 0x7f1650003c20 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- 172.21.15.50:0/4261999394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 msgr2=0x7f1650004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 --2- 172.21.15.50:0/4261999394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f1650004010 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1654001a10 tx=0x7f1654007d30 comp rx=0 tx=0).stop 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f1668e15700 1 -- 172.21.15.50:0/4261999394 reap_dead start 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- 172.21.15.50:0/4261999394 shutdown_connections 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- 172.21.15.50:0/4261999394 >> 172.21.15.50:0/4261999394 conn(0x7f16640d0b90 msgr2=0x7f16640ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- 172.21.15.50:0/4261999394 shutdown_connections 2023-12-19T16:13:33.035 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- 172.21.15.50:0/4261999394 wait complete. 2023-12-19T16:13:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 Processor -- start 2023-12-19T16:13:33.036 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.031+0000 7f166a878700 1 -- start start 2023-12-19T16:13:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664169130 0x7f1664157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16640c9130 0x7f16640c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640cbc80 con 0x7f1650003c20 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c72f0 con 0x7f16640c9130 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640cb3e0 con 0x7f1664169130 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60838/0 (socket says 172.21.15.50:60838) 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 -- 172.21.15.50:0/2167174161 learned_addr learned my addr 172.21.15.50:0/2167174161 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f165bfff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664169130 0x7f1664157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1668e15700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16640c9130 0x7f16640c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 -- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664169130 msgr2=0x7f1664157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664169130 0x7f1664157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 -- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16640c9130 msgr2=0x7f16640c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16640c9130 0x7f16640c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 -- 172.21.15.50:0/2167174161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640cb6c0 con 0x7f1650003c20 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1663fff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f165000bb10 tx=0x7f1654015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165400c5d0 con 0x7f1650003c20 2023-12-19T16:13:33.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165400cab0 con 0x7f1650003c20 2023-12-19T16:13:33.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654006a90 con 0x7f1650003c20 2023-12-19T16:13:33.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640cb9a0 con 0x7f1650003c20 2023-12-19T16:13:33.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.035+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1664157d80 con 0x7f1650003c20 2023-12-19T16:13:33.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.039+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1664071120 con 0x7f1650003c20 2023-12-19T16:13:33.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.039+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 100) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1654019070 con 0x7f1650003c20 2023-12-19T16:13:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.039+0000 7f1660ff9700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.044 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.039+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f165407e2d0 con 0x7f1650003c20 2023-12-19T16:13:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.039+0000 7f165bfff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.043+0000 7f165bfff700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 0x7f1644062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16641662e0 tx=0x7f164c008040 comp rx=0 tx=0).ready entity=mgr.10012 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.047+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f165404e560 con 0x7f1650003c20 2023-12-19T16:13:33.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.259+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f166406e310 con 0x7f1650003c20 2023-12-19T16:13:33.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.315+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 101) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f165404e930 con 0x7f1650003c20 2023-12-19T16:13:33.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.315+0000 7f1660ff9700 1 -- 172.21.15.50:0/2167174161 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v101) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1654063ac0 con 0x7f1650003c20 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 msgr2=0x7f1644062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 0x7f1644062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16641662e0 tx=0x7f164c008040 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 msgr2=0x7f16640cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f165000bb10 tx=0x7f1654015000 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 shutdown_connections 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16640c9130 0x7f16640c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1664169130 0x7f1664157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 --2- 172.21.15.50:0/2167174161 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1650003c20 0x7f16640cace0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 >> 172.21.15.50:0/2167174161 conn(0x7f16640d0b90 msgr2=0x7f16640ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 shutdown_connections 2023-12-19T16:13:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.319+0000 7f166a878700 1 -- 172.21.15.50:0/2167174161 wait complete. 2023-12-19T16:13:33.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T16:13:33.509 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:33.497+0000 7efe149d0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:33.509 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:33.503+0000 7faaddaf3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 Processor -- start 2023-12-19T16:13:33.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 -- start start 2023-12-19T16:13:33.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cfed0 0x7f2f6c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c06cca0 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c06cf80 con 0x7f2f6c0d02a0 2023-12-19T16:13:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f71678700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b3510 con 0x7f2f6c0cfed0 2023-12-19T16:13:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f6a7fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2f6c06c7c0 0x7f2f6c0b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41284/0 (socket says 172.21.15.50:41284) 2023-12-19T16:13:33.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f6a7fc700 1 -- 172.21.15.50:0/3331752817 learned_addr learned my addr 172.21.15.50:0/3331752817 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154585892 0 0) 0x7f2f6c06cca0 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f6affd700 1 --2- 172.21.15.50:0/3331752817 >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cfed0 0x7f2f6c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.527+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f54003570 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866953121 0 0) 0x7f2f54003570 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cfed0 msgr2=0x7f2f6c06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 --2- 172.21.15.50:0/3331752817 >> v2:172.21.15.125:3300/0 conn(0x7f2f6c0cfed0 0x7f2f6c06dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 >> v1:172.21.15.125:6789/0 conn(0x7f2f6c0d02a0 legacy=0x7f2f6c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c06d260 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f2f60001470 con 0x7f2f6c06c7c0 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 >> v1:172.21.15.50:6789/0 conn(0x7f2f6c06c7c0 legacy=0x7f2f6c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f54003a60 0x7f2f54003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f54004810 0x7f2f54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 0x7f2f5400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0b3510 con 0x7f2f540052e0 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5400c230 con 0x7f2f54003a60 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5400c3f0 con 0x7f2f54004810 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 0x7f2f5400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6affd700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f54003a60 0x7f2f54003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.536 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6a7fc700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f54004810 0x7f2f54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f54004810 msgr2=0x7f2f54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f54004810 0x7f2f54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f54003a60 msgr2=0x7f2f54003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f54003a60 0x7f2f54003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3331752817 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f54003570 con 0x7f2f540052e0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3331752817 reap_dead start 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 0x7f2f5400ba60 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f2f64001fc0 tx=0x7f2f64001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708567e3cabb6f92 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64001950 con 0x7f2f540052e0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400c8b0 con 0x7f2f540052e0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f69ffb700 1 -- 172.21.15.50:0/3331752817 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400cc50 con 0x7f2f540052e0 2023-12-19T16:13:33.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 -- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 msgr2=0x7f2f5400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 0x7f2f5400ba60 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f2f64001fc0 tx=0x7f2f64001640 comp rx=0 tx=0).stop 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 -- 172.21.15.50:0/3331752817 shutdown_connections 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 --2- 172.21.15.50:0/3331752817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f540052e0 0x7f2f5400ba60 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 -- 172.21.15.50:0/3331752817 >> 172.21.15.50:0/3331752817 conn(0x7f2f6c0670e0 msgr2=0x7f2f6c0674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.531+0000 7f2f71678700 1 -- 172.21.15.50:0/3331752817 shutdown_connections 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 -- 172.21.15.50:0/3331752817 wait complete. 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 Processor -- start 2023-12-19T16:13:33.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 -- start start 2023-12-19T16:13:33.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f540052e0 0x7f2f6c169b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c151cb0 0x7f2f6c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.541 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c157960 con 0x7f2f6c15d120 2023-12-19T16:13:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c156ea0 con 0x7f2f6c151cb0 2023-12-19T16:13:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.535+0000 7f2f71678700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c15dad0 con 0x7f2f540052e0 2023-12-19T16:13:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60850/0 (socket says 172.21.15.50:60850) 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3124590596 learned_addr learned my addr 172.21.15.50:0/3124590596 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6a7fc700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c151cb0 0x7f2f6c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6affd700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f540052e0 0x7f2f6c169b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f540052e0 msgr2=0x7f2f6c169b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f540052e0 0x7f2f6c169b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c151cb0 msgr2=0x7f2f6c156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c151cb0 0x7f2f6c156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 -- 172.21.15.50:0/3124590596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c16a3c0 con 0x7f2f6c15d120 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f6b7fe700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c6a0 tx=0x7f2f64015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400c6d0 con 0x7f2f6c15d120 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64012e00 con 0x7f2f6c15d120 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64006dd0 con 0x7f2f6c15d120 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c152120 con 0x7f2f6c15d120 2023-12-19T16:13:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c1573a0 con 0x7f2f6c15d120 2023-12-19T16:13:33.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.539+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c16a190 con 0x7f2f6c15d120 2023-12-19T16:13:33.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.543+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 101) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f2f64007030 con 0x7f2f6c15d120 2023-12-19T16:13:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.543+0000 7f2f52ffd700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 0x7f2f58062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.543+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f2f6407d880 con 0x7f2f6c15d120 2023-12-19T16:13:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.543+0000 7f2f6affd700 1 -- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 msgr2=0x7f2f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.543+0000 7f2f6affd700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 0x7f2f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:33.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.551+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f6c16a190 con 0x7f2f6c15d120 2023-12-19T16:13:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.723+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2f6c16a190 con 0x7f2f6c15d120 2023-12-19T16:13:33.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f52ffd700 1 -- 172.21.15.50:0/3124590596 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2f6c16a190 con 0x7f2f6c15d120 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 msgr2=0x7f2f58062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 0x7f2f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 msgr2=0x7f2f6c15d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c6a0 tx=0x7f2f64015000 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 shutdown_connections 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f580606c0 0x7f2f58062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c151cb0 0x7f2f6c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f6c15d120 0x7f2f6c15d590 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 --2- 172.21.15.50:0/3124590596 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f540052e0 0x7f2f6c169b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.731+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 >> 172.21.15.50:0/3124590596 conn(0x7f2f6c0670e0 msgr2=0x7f2f6c154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.735+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 shutdown_connections 2023-12-19T16:13:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.735+0000 7f2f71678700 1 -- 172.21.15.50:0/3124590596 wait complete. 2023-12-19T16:13:33.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T16:13:33.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 Processor -- start 2023-12-19T16:13:33.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 -- start start 2023-12-19T16:13:33.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf040acf30 0x7faf041557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf04156780 con 0x7faf0406d1d0 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf04158f90 con 0x7faf040ad300 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0b24c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf041532a0 con 0x7faf040acf30 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf040acf30 0x7faf041557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faf040acf30 0x7faf041557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60858/0 (socket says 172.21.15.50:60858) 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 -- 172.21.15.50:0/1128495182 learned_addr learned my addr 172.21.15.50:0/1128495182 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0aa4b700 1 --1- 172.21.15.50:0/1128495182 >> v1:172.21.15.50:6789/0 conn(0x7faf0406d1d0 0x7faf040ac1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41298/0 (socket says 172.21.15.50:41298) 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 -- 172.21.15.50:0/1128495182 >> v1:172.21.15.125:6789/0 conn(0x7faf040ad300 legacy=0x7faf04159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 -- 172.21.15.50:0/1128495182 >> v1:172.21.15.50:6789/0 conn(0x7faf0406d1d0 legacy=0x7faf040ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.979+0000 7faf0a24a700 1 -- 172.21.15.50:0/1128495182 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0406d630 con 0x7faf040acf30 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0a24a700 1 --2- 172.21.15.50:0/1128495182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf040acf30 0x7faf041557d0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7faf000099b0 tx=0x7faf00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d87c99225917f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf09248700 1 -- 172.21.15.50:0/1128495182 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000b860 con 0x7faf040acf30 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf09248700 1 -- 172.21.15.50:0/1128495182 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0000b9c0 con 0x7faf040acf30 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf09248700 1 -- 172.21.15.50:0/1128495182 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000bd60 con 0x7faf040acf30 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 -- 172.21.15.50:0/1128495182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf040acf30 msgr2=0x7faf041557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.987 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 --2- 172.21.15.50:0/1128495182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf040acf30 0x7faf041557d0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7faf000099b0 tx=0x7faf00009250 comp rx=0 tx=0).stop 2023-12-19T16:13:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 -- 172.21.15.50:0/1128495182 shutdown_connections 2023-12-19T16:13:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 --2- 172.21.15.50:0/1128495182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf040acf30 0x7faf041557d0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 -- 172.21.15.50:0/1128495182 >> 172.21.15.50:0/1128495182 conn(0x7faf04067b50 msgr2=0x7faf040cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 -- 172.21.15.50:0/1128495182 shutdown_connections 2023-12-19T16:13:33.988 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.983+0000 7faf0b24c700 1 -- 172.21.15.50:0/1128495182 wait complete. 2023-12-19T16:13:33.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 Processor -- start 2023-12-19T16:13:33.991 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 -- start start 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf040acf30 0x7faf041473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf040ad300 0x7faf040ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0414ea50 con 0x7faf0406d1d0 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04147930 con 0x7faf040acf30 2023-12-19T16:13:33.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0b24c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0416d120 con 0x7faf040ad300 2023-12-19T16:13:33.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0a24a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0a24a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60866/0 (socket says 172.21.15.50:60866) 2023-12-19T16:13:33.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0a24a700 1 -- 172.21.15.50:0/2269934734 learned_addr learned my addr 172.21.15.50:0/2269934734 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:33.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf0aa4b700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf040ad300 0x7faf040ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.995 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.987+0000 7faf09a49700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf040acf30 0x7faf041473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 -- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf040ad300 msgr2=0x7faf040ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf040ad300 0x7faf040ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 -- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf040acf30 msgr2=0x7faf041473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf040acf30 0x7faf041473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 -- 172.21.15.50:0/2269934734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000094f0 con 0x7faf0406d1d0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0a24a700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7faf04069d20 tx=0x7faf00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00024440 con 0x7faf0406d1d0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf000245a0 con 0x7faf0406d1d0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0416d400 con 0x7faf0406d1d0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00024940 con 0x7faf0406d1d0 2023-12-19T16:13:33.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0416d9c0 con 0x7faf0406d1d0 2023-12-19T16:13:33.997 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.991+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0404be40 con 0x7faf0406d1d0 2023-12-19T16:13:33.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.995+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 101) v1 ==== 80080+0+0 (secure 0 0 0) 0x7faf0001a420 con 0x7faf0406d1d0 2023-12-19T16:13:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.995+0000 7faefa7fc700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 0x7faee8062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:33.999 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.995+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7faf0007e8b0 con 0x7faf0406d1d0 2023-12-19T16:13:34.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.995+0000 7faf09a49700 1 -- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 msgr2=0x7faee8062bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:34.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.995+0000 7faf09a49700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 0x7faee8062bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:34.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:33.999+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faf0004ec70 con 0x7faf0406d1d0 2023-12-19T16:13:34.063 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.053+0000 7efe149d0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:34.093 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.087+0000 7faaddaf3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:34.141 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.133+0000 7efe149d0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:34.170 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.163+0000 7faaddaf3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:34.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.175+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7faf040cb770 con 0x7faf0406d1d0 2023-12-19T16:13:34.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.179+0000 7faefa7fc700 1 -- 172.21.15.50:0/2269934734 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7faf00052290 con 0x7faf0406d1d0 2023-12-19T16:13:34.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 msgr2=0x7faee8062bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 0x7faee8062bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 msgr2=0x7faf0416cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7faf04069d20 tx=0x7faf00017000 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 shutdown_connections 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7faee8060710 0x7faee8062bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf040acf30 0x7faf041473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7faf040ad300 0x7faf040ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 --2- 172.21.15.50:0/2269934734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faf0406d1d0 0x7faf0416cb00 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 >> 172.21.15.50:0/2269934734 conn(0x7faf04067b50 msgr2=0x7faf040c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 shutdown_connections 2023-12-19T16:13:34.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.183+0000 7faf0b24c700 1 -- 172.21.15.50:0/2269934734 wait complete. 2023-12-19T16:13:34.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:34.210 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T16:13:34.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T16:13:34.293 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.285+0000 7efe149d0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:34.318 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.315+0000 7faaddaf3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 Processor -- start 2023-12-19T16:13:34.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 -- start start 2023-12-19T16:13:34.456 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f2406dea0 0x7f8f2406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240cbb80 con 0x7f8f240cb700 2023-12-19T16:13:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240cb270 con 0x7f8f2406dad0 2023-12-19T16:13:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f2968e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f240ceb00 con 0x7f8f2406dea0 2023-12-19T16:13:34.457 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f28e8d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8f240cb700 0x7f8f240cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41312/0 (socket says 172.21.15.50:41312) 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.451+0000 7f8f28e8d700 1 -- 172.21.15.50:0/3890973250 learned_addr learned my addr 172.21.15.50:0/3890973250 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f237fe700 1 --2- 172.21.15.50:0/3890973250 >> v2:172.21.15.125:3300/0 conn(0x7f8f2406dea0 0x7f8f2406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f22ffd700 1 -- 172.21.15.50:0/3890973250 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053339096 0 0) 0x7f8f1c001490 con 0x7f8f240cb700 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f22ffd700 1 -- 172.21.15.50:0/3890973250 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f0c003570 con 0x7f8f240cb700 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f237fe700 1 -- 172.21.15.50:0/3890973250 >> v1:172.21.15.125:6789/0 conn(0x7f8f2406dad0 legacy=0x7f8f240ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f237fe700 1 -- 172.21.15.50:0/3890973250 >> v1:172.21.15.50:6789/0 conn(0x7f8f240cb700 legacy=0x7f8f240cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f237fe700 1 -- 172.21.15.50:0/3890973250 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240ca6a0 con 0x7f8f2406dea0 2023-12-19T16:13:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f237fe700 1 --2- 172.21.15.50:0/3890973250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dea0 0x7f8f2406b0f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f180098e0 tx=0x7f8f180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4d30ef0e5bc6f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f22ffd700 1 -- 172.21.15.50:0/3890973250 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800b640 con 0x7f8f2406dea0 2023-12-19T16:13:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f22ffd700 1 -- 172.21.15.50:0/3890973250 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1800b7a0 con 0x7f8f2406dea0 2023-12-19T16:13:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f22ffd700 1 -- 172.21.15.50:0/3890973250 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800bb40 con 0x7f8f2406dea0 2023-12-19T16:13:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 -- 172.21.15.50:0/3890973250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dea0 msgr2=0x7f8f2406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 --2- 172.21.15.50:0/3890973250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dea0 0x7f8f2406b0f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f180098e0 tx=0x7f8f180091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 -- 172.21.15.50:0/3890973250 shutdown_connections 2023-12-19T16:13:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 --2- 172.21.15.50:0/3890973250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dea0 0x7f8f2406b0f0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 -- 172.21.15.50:0/3890973250 >> 172.21.15.50:0/3890973250 conn(0x7f8f240670e0 msgr2=0x7f8f240674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 -- 172.21.15.50:0/3890973250 shutdown_connections 2023-12-19T16:13:34.461 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.455+0000 7f8f2968e700 1 -- 172.21.15.50:0/3890973250 wait complete. 2023-12-19T16:13:34.463 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.459+0000 7f8f2968e700 1 Processor -- start 2023-12-19T16:13:34.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.459+0000 7f8f2968e700 1 -- start start 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dad0 0x7f8f240b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f240cb700 0x7f8f24155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f240b44b0 con 0x7f8f2406dea0 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f240b3a00 con 0x7f8f2406dad0 2023-12-19T16:13:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f240c7020 con 0x7f8f240cb700 2023-12-19T16:13:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:34.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f28e8d700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f240cb700 0x7f8f24155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f23fff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dad0 0x7f8f240b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60872/0 (socket says 172.21.15.50:60872) 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 -- 172.21.15.50:0/1899289816 learned_addr learned my addr 172.21.15.50:0/1899289816 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f240cb700 msgr2=0x7f8f24155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f240cb700 0x7f8f24155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dad0 msgr2=0x7f8f240b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dad0 0x7f8f240b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:34.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f18009420 con 0x7f8f2406dea0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f237fe700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f8f2406bb60 tx=0x7f8f18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18000d10 con 0x7f8f2406dea0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1800bb20 con 0x7f8f2406dea0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18011650 con 0x7f8f2406dea0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f240b3c10 con 0x7f8f2406dea0 2023-12-19T16:13:34.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.463+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f240b41d0 con 0x7f8f2406dea0 2023-12-19T16:13:34.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.467+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f240c6dd0 con 0x7f8f2406dea0 2023-12-19T16:13:34.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.467+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 101) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f8f180119c0 con 0x7f8f2406dea0 2023-12-19T16:13:34.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.467+0000 7f8f0bfff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.467+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6137+0+0 (secure 0 0 0) 0x7f8f1807e0f0 con 0x7f8f2406dea0 2023-12-19T16:13:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.471+0000 7f8f23fff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.471+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:34.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.475+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f1804e380 con 0x7f8f2406dea0 2023-12-19T16:13:34.567 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.557+0000 7efe149d0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:34.643 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.633+0000 7efe149d0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:34.650 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.647+0000 7faaddaf3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:34.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.675+0000 7f8f23fff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:34.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.675+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:34.719+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f2404be40 con 0x7f8f100606c0 2023-12-19T16:13:34.733 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.727+0000 7faaddaf3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:34.864 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.853+0000 7efe149d0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:34.949 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:34.937+0000 7efe149d0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:34.967 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:34.963+0000 7faaddaf3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:35.036 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:35.025+0000 7efe149d0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:35.059 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:35.055+0000 7faaddaf3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:35.079+0000 7f8f23fff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:35.079+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T16:13:35.150 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:35.143+0000 7faaddaf3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:35.440 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:35.429+0000 7efe149d0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:35.575 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:35.571+0000 7faaddaf3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:35.612 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:35.601+0000 7efe149d0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:35.755 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:35.751+0000 7faaddaf3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:35.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:35.883+0000 7f8f23fff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:35.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:35.883+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T16:13:35.987 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:35.977+0000 7efe149d0000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:36.143 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.139+0000 7faaddaf3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:36.202 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.193+0000 7efe149d0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:36.292 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.281+0000 7efe149d0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:36.366 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.357+0000 7efe149d0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:36.371 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.367+0000 7faaddaf3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:36.448 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.437+0000 7efe149d0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:36.465 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.459+0000 7faaddaf3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:36.547 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.543+0000 7faaddaf3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:36.635 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.631+0000 7faaddaf3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:36.665 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.653+0000 7efe149d0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:36.738 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.729+0000 7efe149d0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:36.828 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:36.817+0000 7efe149d0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:36.857 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.851+0000 7faaddaf3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:36.938 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:36.935+0000 7faaddaf3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:37.011 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:37.001+0000 7efe149d0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:37.028 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:37.023+0000 7faaddaf3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:37.094 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:37.085+0000 7efe149d0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:37.227 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:37.223+0000 7faaddaf3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:37.320 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:37.315+0000 7faaddaf3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:37.487+0000 7f8f23fff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:37.490 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:37.487+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T16:13:37.808 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:37.797+0000 7efe149d0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:37.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:37.823+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 102) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f8f1804e750 con 0x7f8f2406dea0 2023-12-19T16:13:37.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:37.823+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:37.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:37.823+0000 7f8f0bfff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.031 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd50a7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.031 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd50a7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.031 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd50a7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.031 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd50a7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd50a7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd80ad700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd80ad700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd80ad700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd80ad700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.032 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:38.021+0000 7efdd80ad700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:38.085 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:38.079+0000 7faaddaf3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:38.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.835+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 103) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f8f1807c3b0 con 0x7f8f2406dea0 2023-12-19T16:13:38.840 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.835+0000 7f8f0bfff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:38.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.835+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f2404be40 con 0x7f8f100606c0 2023-12-19T16:13:38.841 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.835+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.839+0000 7f8f23fff700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f14001200 tx=0x7f8f14007ec0 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.951+0000 7f8f0bfff700 1 -- 172.21.15.50:0/1899289816 <== mgr.10378 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8f2404be40 con 0x7f8f100606c0 2023-12-19T16:13:38.958 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule set for path snap_test_dir1 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 msgr2=0x7f8f10062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f14001200 tx=0x7f8f14007ec0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 msgr2=0x7f8f240b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f8f2406bb60 tx=0x7f8f18017000 comp rx=0 tx=0).stop 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 shutdown_connections 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f8f100606c0 0x7f8f10062b70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f8f240cb700 0x7f8f24155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f2406dea0 0x7f8f240b34c0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 --2- 172.21.15.50:0/1899289816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f2406dad0 0x7f8f240b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 >> 172.21.15.50:0/1899289816 conn(0x7f8f240670e0 msgr2=0x7f8f240cfc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 shutdown_connections 2023-12-19T16:13:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:38.959+0000 7f8f2968e700 1 -- 172.21.15.50:0/1899289816 wait complete. 2023-12-19T16:13:38.988 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T16:13:38.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-12-19T16:13:39.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.227+0000 7f0e79a2e700 1 Processor -- start 2023-12-19T16:13:39.231 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.227+0000 7f0e79a2e700 1 -- start start 2023-12-19T16:13:39.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e79a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e740670e0 0x7f0e74067530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e79a2e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0e7406d3d0 0x7f0e7406d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e79a2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7406dd90 con 0x7f0e74158c60 2023-12-19T16:13:39.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e79a2e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e740cb550 con 0x7f0e7406d3d0 2023-12-19T16:13:39.235 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e79a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7406cf40 con 0x7f0e740670e0 2023-12-19T16:13:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e72ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0e74158c60 0x7f0e740afb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:43282/0 (socket says 172.21.15.50:43282) 2023-12-19T16:13:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e72ffd700 1 -- 172.21.15.50:0/1949541633 learned_addr learned my addr 172.21.15.50:0/1949541633 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0e7406d3d0 0x7f0e7406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e727fc700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e740670e0 0x7f0e74067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e71ffb700 1 -- 172.21.15.50:0/1949541633 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820550723 0 0) 0x7f0e64001110 con 0x7f0e74158c60 2023-12-19T16:13:39.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 -- 172.21.15.50:0/1949541633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e740670e0 msgr2=0x7f0e74067530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:39.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e740670e0 0x7f0e74067530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 -- 172.21.15.50:0/1949541633 >> v1:172.21.15.125:6789/0 conn(0x7f0e74158c60 legacy=0x7f0e740afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 -- 172.21.15.50:0/1949541633 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74067a70 con 0x7f0e7406d3d0 2023-12-19T16:13:39.237 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.231+0000 7f0e737fe700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7406d3d0 0x7f0e7406d850 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009e60 tx=0x7f0e6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0b297127718a4d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:39.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e71ffb700 1 -- 172.21.15.50:0/1949541633 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c00e070 con 0x7f0e7406d3d0 2023-12-19T16:13:39.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e71ffb700 1 -- 172.21.15.50:0/1949541633 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c00bb60 con 0x7f0e7406d3d0 2023-12-19T16:13:39.238 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e71ffb700 1 -- 172.21.15.50:0/1949541633 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c012430 con 0x7f0e7406d3d0 2023-12-19T16:13:39.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- 172.21.15.50:0/1949541633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7406d3d0 msgr2=0x7f0e7406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7406d3d0 0x7f0e7406d850 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009e60 tx=0x7f0e6c009780 comp rx=0 tx=0).stop 2023-12-19T16:13:39.239 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- 172.21.15.50:0/1949541633 shutdown_connections 2023-12-19T16:13:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e7406d3d0 0x7f0e7406d850 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/1949541633 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e740670e0 0x7f0e74067530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- 172.21.15.50:0/1949541633 >> 172.21.15.50:0/1949541633 conn(0x7f0e740cfde0 msgr2=0x7f0e740d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- 172.21.15.50:0/1949541633 shutdown_connections 2023-12-19T16:13:39.240 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- 172.21.15.50:0/1949541633 wait complete. 2023-12-19T16:13:39.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 Processor -- start 2023-12-19T16:13:39.241 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.235+0000 7f0e79a2e700 1 -- start start 2023-12-19T16:13:39.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.242 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e7406d3d0 0x7f0e740b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 0x7f0e74169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7415c3d0 con 0x7f0e740670e0 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e740b54d0 con 0x7f0e74158c60 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e740b57b0 con 0x7f0e7406d3d0 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.243 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e737fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 0x7f0e74169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43126/0 (socket says 172.21.15.50:43126) 2023-12-19T16:13:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 -- 172.21.15.50:0/862249882 learned_addr learned my addr 172.21.15.50:0/862249882 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e727fc700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e7406d3d0 0x7f0e740b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 -- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e7406d3d0 msgr2=0x7f0e740b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e7406d3d0 0x7f0e740b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 -- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 msgr2=0x7f0e74169310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 0x7f0e74169310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 -- 172.21.15.50:0/862249882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c009a20 con 0x7f0e740670e0 2023-12-19T16:13:39.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e737fe700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 0x7f0e74169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:39.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e72ffd700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400c560 tx=0x7f0e64000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:39.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64005b30 con 0x7f0e740670e0 2023-12-19T16:13:39.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64005c90 con 0x7f0e740670e0 2023-12-19T16:13:39.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e740b5a90 con 0x7f0e740670e0 2023-12-19T16:13:39.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.239+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64007320 con 0x7f0e740670e0 2023-12-19T16:13:39.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.243+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e74168df0 con 0x7f0e740670e0 2023-12-19T16:13:39.247 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.243+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7404be40 con 0x7f0e740670e0 2023-12-19T16:13:39.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.243+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 103) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f0e6401b3b0 con 0x7f0e740670e0 2023-12-19T16:13:39.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.247+0000 7f0e5affd700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 0x7f0e600628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:39.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.247+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f0e6407bab0 con 0x7f0e740670e0 2023-12-19T16:13:39.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.247+0000 7f0e727fc700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 0x7f0e600628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:39.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.247+0000 7f0e727fc700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 0x7f0e600628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e7416bf60 tx=0x7f0e680091a0 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:39.256 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.251+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e6404bf70 con 0x7f0e740670e0 2023-12-19T16:13:39.449 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.443+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e7406dd90 con 0x7f0e600603f0 2023-12-19T16:13:39.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.559+0000 7f0e5affd700 1 -- 172.21.15.50:0/862249882 <== mgr.10378 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f0e7406dd90 con 0x7f0e600603f0 2023-12-19T16:13:39.565 INFO:teuthology.orchestra.run.smithi050.stdout:Retention added to path snap_test_dir1 2023-12-19T16:13:39.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 msgr2=0x7f0e600628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 0x7f0e600628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e7416bf60 tx=0x7f0e680091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 msgr2=0x7f0e740b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400c560 tx=0x7f0e64000fe0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 shutdown_connections 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0e600603f0 0x7f0e600628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e74158c60 0x7f0e74169310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0e7406d3d0 0x7f0e740b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 --2- 172.21.15.50:0/862249882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0e740670e0 0x7f0e740b4c60 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 >> 172.21.15.50:0/862249882 conn(0x7f0e740cfde0 msgr2=0x7f0e740b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 shutdown_connections 2023-12-19T16:13:39.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:39.563+0000 7f0e79a2e700 1 -- 172.21.15.50:0/862249882 wait complete. 2023-12-19T16:13:39.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-12-19T16:13:39.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T16:13:39.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T16:13:39.678 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-12-19T16:13:39.678 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.679 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-12-19T16:13:39.680 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.50,172.21.15.125,[v2:172.21.15.125:3301,v1:172.21.15.125:6790]" 2023-12-19T16:13:39.681 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-12-19T16:13:39.682 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.683 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: osd op complaint time = 180 2023-12-19T16:13:39.684 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug client = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug finisher = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-12-19T16:13:39.685 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T16:13:39.686 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: mon op complaint time = 120 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T16:13:39.687 INFO:teuthology.orchestra.run.smithi050.stdout: client mount timeout = 600 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: debug client = 20 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = "" 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: rados mon op timeout = 900 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: rados osd op timeout = 900 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: client snapdir = .snap 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.a] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.c] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.b] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mds.d] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout:[mds] 2023-12-19T16:13:39.688 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds = 20 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: mds debug frag = True 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: mds debug scatterstat = True 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: mds op complaint time = 180 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: mds verify scatter = True 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: osd op complaint time = 180 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: rados mon op timeout = 900 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi050.stdout: rados osd op timeout = 900 2023-12-19T16:13:39.689 INFO:teuthology.orchestra.run.smithi125.stdout:[global] 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: chdir = "" 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: auth supported = cephx 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: filestore xattr use omap = true 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: mon clock drift allowed = 1.000 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: osd crush chooseleaf type = 0 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: auth debug = true 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on old message = true 2023-12-19T16:13:39.690 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on bug = true 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon pg warn max object skew = 0 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default size = 2 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow primary affinity = true 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow pg remap = true 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on legacy crush tunables = false 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on crush straw calc version zero = false 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on no sortbitwise = false 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on osd down out interval zero = false 2023-12-19T16:13:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on too few osds = false 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon_allow_pool_size_one = true 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: osd default data pool replay window = 5 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon allow pool delete = true 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon cluster log file level = debug 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: debug asserts on shutdown = true 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon health detail to clog = false 2023-12-19T16:13:39.692 INFO:teuthology.orchestra.run.smithi125.stdout: mon host = "172.21.15.50,172.21.15.125,[v2:172.21.15.125:3301,v1:172.21.15.125:6790]" 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: fsid = 8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout:[osd] 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd journal size = 100 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub load threshold = 5.0 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub max interval = 600 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd mclock profile = high_recovery_ops 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd recover clone overlap = true 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd recovery max chunk = 1048576 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug shutdown = true 2023-12-19T16:13:39.693 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug op order = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug verify stray on activate = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd open classes on start = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug pg log writeout = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd deep scrub update digest min age = 30 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd map max advance = 10 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: journal zero on create = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: filestore ondisk finisher threads = 3 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: filestore apply finisher threads = 3 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: bdev debug aio = true 2023-12-19T16:13:39.694 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug misdirected ops = true 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore block size = 96636764160 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore fsck on mount = True 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluefs = 20 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluestore = 20 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: debug osd = 20 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: debug rocksdb = 10 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd full ratio = 0.9 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd nearfull ratio = 0.8 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: osd failsafe full ratio = 0.95 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: osd objectstore = bluestore 2023-12-19T16:13:39.695 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr] 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug mgr = 20 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: mgr/telemetry/nag = false 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug finisher = 20 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout:[mon] 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T16:13:39.696 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: debug paxos = 20 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon data avail warn = 5 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon mgr mkfs grace = 240 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd reporter subtree level = osd 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd prime pg temp = true 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.697 INFO:teuthology.orchestra.run.smithi125.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: # 1m isn't quite enough 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: mon_down_mkfs_grace = 2m 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_filestore_osds = false 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: mon op complaint time = 120 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout:[client] 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: rgw cache enabled = true 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable ops log = true 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable usage log = true 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T16:13:39.698 INFO:teuthology.orchestra.run.smithi125.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: client mount timeout = 600 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = "" 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout: client snapdir = .snap 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.a] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.b] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.c] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2023-12-19T16:13:39.699 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout:[mds] 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds = 20 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug frag = True 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug scatterstat = True 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: mds op complaint time = 180 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: mds verify scatter = True 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-12-19T16:13:39.700 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-12-19T16:13:39.735 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T16:13:39.735 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T16:13:39.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:39.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62050]: fuse finished with error 0 and tester_r 0 2023-12-19T16:13:39.786 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T16:13:45.888 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:13:45.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:45.888 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T16:13:45.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T16:13:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 Processor -- start 2023-12-19T16:13:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 -- start start 2023-12-19T16:13:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f1c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c06a850 con 0x7f3f1c06ace0 2023-12-19T16:13:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c069cb0 con 0x7f3f1c159710 2023-12-19T16:13:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f21745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c069f20 con 0x7f3f1c15d320 2023-12-19T16:13:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f127fc700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3f1c06ace0 0x7f3f1c159000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54060/0 (socket says 172.21.15.50:54060) 2023-12-19T16:13:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f127fc700 1 -- 172.21.15.50:0/1791179131 learned_addr learned my addr 172.21.15.50:0/1791179131 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1affd700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f1c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.129 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874097917 0 0) 0x7f3f1c06a850 con 0x7f3f1c06ace0 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f08003570 con 0x7f3f1c06ace0 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741124364 0 0) 0x7f3f1c069cb0 con 0x7f3f1c159710 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c06a850 con 0x7f3f1c159710 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259105009 0 0) 0x7f3f08003570 con 0x7f3f1c06ace0 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 msgr2=0x7f3f1c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f1c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 >> v1:172.21.15.125:6789/0 conn(0x7f3f1c159710 legacy=0x7f3f1c067a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c06a200 con 0x7f3f1c06ace0 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f3f04001470 con 0x7f3f1c06ace0 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 >> v1:172.21.15.50:6789/0 conn(0x7f3f1c06ace0 legacy=0x7f3f1c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f08003c20 0x7f3f08004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08004810 0x7f3f08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c069f20 con 0x7f3f08004810 2023-12-19T16:13:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0800bb40 con 0x7f3f08003c20 2023-12-19T16:13:46.131 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.123+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0800bd00 con 0x7f3f1c15d320 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f127fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08004810 0x7f3f08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1affd700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f08003c20 0x7f3f08004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 -- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f08003c20 msgr2=0x7f3f08004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f08003c20 0x7f3f08004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 -- 172.21.15.50:0/1791179131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08004810 msgr2=0x7f3f08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08004810 0x7f3f08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 -- 172.21.15.50:0/1791179131 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f08003570 con 0x7f3f1c15d320 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1affd700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f08003c20 0x7f3f08004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f127fc700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f08004810 0x7f3f08004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f080051c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3f14002120 tx=0x7f3f140017a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17fc5ff894104196 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14001ab0 con 0x7f3f1c15d320 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400c980 con 0x7f3f1c15d320 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1a7fc700 1 -- 172.21.15.50:0/1791179131 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400cd20 con 0x7f3f1c15d320 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 -- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 msgr2=0x7f3f080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 --2- 172.21.15.50:0/1791179131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c15d320 0x7f3f080051c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3f14002120 tx=0x7f3f140017a0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f1b7fe700 1 -- 172.21.15.50:0/1791179131 reap_dead start 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 -- 172.21.15.50:0/1791179131 shutdown_connections 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 -- 172.21.15.50:0/1791179131 >> 172.21.15.50:0/1791179131 conn(0x7f3f1c0cc690 msgr2=0x7f3f1c0c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 -- 172.21.15.50:0/1791179131 shutdown_connections 2023-12-19T16:13:46.134 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.127+0000 7f3f21745700 1 -- 172.21.15.50:0/1791179131 wait complete. 2023-12-19T16:13:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 Processor -- start 2023-12-19T16:13:46.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 -- start start 2023-12-19T16:13:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f1c168ce0 0x7f3f1c15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 0x7f3f1c1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c140bc0 con 0x7f3f1c15d320 2023-12-19T16:13:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c15ccf0 con 0x7f3f1c168ce0 2023-12-19T16:13:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.131+0000 7f3f21745700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c15c1b0 con 0x7f3f1c156370 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43146/0 (socket says 172.21.15.50:43146) 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 -- 172.21.15.50:0/2360164788 learned_addr learned my addr 172.21.15.50:0/2360164788 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f127fc700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f1c168ce0 0x7f3f1c15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 0x7f3f1c1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 -- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 msgr2=0x7f3f1c1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 0x7f3f1c1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 -- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f1c168ce0 msgr2=0x7f3f1c15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f1c168ce0 0x7f3f1c15bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 -- 172.21.15.50:0/2360164788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c15c450 con 0x7f3f1c15d320 2023-12-19T16:13:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1b7fe700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 0x7f3f1c1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f1affd700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006e50 tx=0x7f3f0c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c011ec0 con 0x7f3f1c15d320 2023-12-19T16:13:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00fcd0 con 0x7f3f1c15d320 2023-12-19T16:13:46.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c15c730 con 0x7f3f1c15d320 2023-12-19T16:13:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c005c30 con 0x7f3f1c15d320 2023-12-19T16:13:46.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.135+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c156d10 con 0x7f3f1c15d320 2023-12-19T16:13:46.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.139+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c071110 con 0x7f3f1c15d320 2023-12-19T16:13:46.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.139+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 104) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f3f0c006260 con 0x7f3f1c15d320 2023-12-19T16:13:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.139+0000 7f3f137fe700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 0x7f3efc062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.139+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3f0c07d7c0 con 0x7f3f1c15d320 2023-12-19T16:13:46.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.143+0000 7f3f127fc700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 0x7f3efc062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:46.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.143+0000 7f3f127fc700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 0x7f3efc062c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0cbb00 tx=0x7f3f04009040 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:46.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.147+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3f0c04d820 con 0x7f3f1c15d320 2023-12-19T16:13:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.355+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f1c0c9370 con 0x7f3f1c15d320 2023-12-19T16:13:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.355+0000 7f3f137fe700 1 -- 172.21.15.50:0/2360164788 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3f0c021060 con 0x7f3f1c15d320 2023-12-19T16:13:46.361 INFO:teuthology.orchestra.run.smithi050.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T16:13:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 msgr2=0x7f3efc062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 0x7f3efc062c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0cbb00 tx=0x7f3f04009040 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 msgr2=0x7f3f1c157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006e50 tx=0x7f3f0c009120 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 shutdown_connections 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3efc060780 0x7f3efc062c30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3f1c168ce0 0x7f3f1c15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3f1c156370 0x7f3f1c1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 --2- 172.21.15.50:0/2360164788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f1c15d320 0x7f3f1c157900 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:46.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 >> 172.21.15.50:0/2360164788 conn(0x7f3f1c0cc690 msgr2=0x7f3f1c0c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 shutdown_connections 2023-12-19T16:13:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:46.359+0000 7f3f21745700 1 -- 172.21.15.50:0/2360164788 wait complete. 2023-12-19T16:13:46.388 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T16:13:46.388 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T16:13:46.388 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi050.front.sepia.ceph.com 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T16:13:46.389 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:46.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.389 DEBUG:teuthology.orchestra.run.smithi050:> ip addr 2023-12-19T16:13:46.404 INFO:teuthology.orchestra.run.smithi050.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 ::1/128 scope host 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f8 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f0 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:6b:fd:f9 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp7s0f1 2023-12-19T16:13:46.405 INFO:teuthology.orchestra.run.smithi050.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:84 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f0 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.21.15.50/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft 36724sec preferred_lft 36724sec 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::ec4:7aff:febd:1584/64 scope link 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 0c:c4:7a:bd:15:85 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: altname enp1s0f1 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 02:42:74:f6:2c:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.406 INFO:teuthology.orchestra.run.smithi050.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::3cca:6dff:fe36:2631/64 scope link 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 6e:f2:bb:07:3a:7a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::6cf2:bbff:fe07:3a7a/64 scope link 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: link/ether 96:2a:55:0e:0f:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: inet6 fe80::942a:55ff:fe0e:fc4/64 scope link 2023-12-19T16:13:46.407 INFO:teuthology.orchestra.run.smithi050.stdout: valid_lft forever preferred_lft forever 2023-12-19T16:13:46.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.408 DEBUG:teuthology.orchestra.run.smithi050:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T16:13:46.490 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-12-19T16:13:46.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.494 DEBUG:teuthology.orchestra.run.smithi050:> route 2023-12-19T16:13:46.561 INFO:teuthology.orchestra.run.smithi050.stdout:Kernel IP routing table 2023-12-19T16:13:46.561 INFO:teuthology.orchestra.run.smithi050.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T16:13:46.561 INFO:teuthology.orchestra.run.smithi050.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-19T16:13:46.562 INFO:teuthology.orchestra.run.smithi050.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T16:13:46.562 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-19T16:13:46.562 INFO:teuthology.orchestra.run.smithi050.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T16:13:46.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> (cd / && exec bash -c ' 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> set -e 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T16:13:46.562 DEBUG:teuthology.orchestra.run.smithi050:> ') 2023-12-19T16:13:46.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.709 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T16:13:46.726 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T16:13:46.726 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T16:13:46.727 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:46.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:46.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link set brx.0 up 2023-12-19T16:13:46.812 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T16:13:46.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:13:46.813 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:46.879 INFO:teuthology.orchestra.run.smithi050.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T16:13:46.880 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T16:13:46.880 DEBUG:teuthology.orchestra.run.smithi050:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:46.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:13:47.015 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:13:47.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:13:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:13:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:13:47.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:47.097 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:13:47.164 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T16:13:47.164 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T16:13:47.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:13:47.257 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:13:47.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:13:47.336 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:13:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:13:47.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:47.340 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:13:47.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:2023-12-19T16:13:47.351+0000 7f2ba6fc5380 -1 init, newargv = 0x5574a2c54950 newargc=15 2023-12-19T16:13:47.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62385]: starting ceph client 2023-12-19T16:13:47.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62385]: starting fuse 2023-12-19T16:13:48.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo modprobe fuse 2023-12-19T16:13:48.514 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T16:13:48.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T16:13:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T16:13:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T16:13:48.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:48.600 DEBUG:teuthology.orchestra.run.smithi050:> ls /sys/fs/fuse/connections 2023-12-19T16:13:48.667 INFO:teuthology.orchestra.run.smithi050.stdout:49 2023-12-19T16:13:48.667 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> import glob 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> import re 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> import os 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> import subprocess 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> def _find_admin_socket(client_name): 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> files = glob.glob(asok_path) 2023-12-19T16:13:48.668 DEBUG:teuthology.orchestra.run.smithi050:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> # Given a non-glob path, it better be there 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> if "*" not in asok_path: 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> assert(len(files) == 1) 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> return files[0] 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> for f in files: 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> contents = proc_f.read() 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> if mountpoint in contents: 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> return f 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T16:13:48.669 DEBUG:teuthology.orchestra.run.smithi050:> 2023-12-19T16:13:48.670 DEBUG:teuthology.orchestra.run.smithi050:> print(_find_admin_socket("client.0")) 2023-12-19T16:13:48.670 DEBUG:teuthology.orchestra.run.smithi050:> ' 2023-12-19T16:13:48.840 INFO:teuthology.orchestra.run.smithi050.stdout:/var/run/ceph/ceph-client.0.62385.asok 2023-12-19T16:13:49.024 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62385.asok 2023-12-19T16:13:49.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:49.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62385.asok status 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "metadata": { 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "entity_id": "0", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "hostname": "smithi050", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pid": "62385", 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "root": "/" 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_count": 0, 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "dentry_pinned_count": 0, 2023-12-19T16:13:49.216 INFO:teuthology.orchestra.run.smithi050.stdout: "id": 9125, 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "inst": { 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "name": { 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "client", 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 9125 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 576284643 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": { 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2023-12-19T16:13:49.217 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:0", 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 576284643 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "inst_str": "client.9125 172.21.15.50:0/576284643", 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "addr_str": "172.21.15.50:0/576284643", 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "inode_count": 1, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_epoch": 99, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch": 214, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_epoch_barrier": 0, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklisted": false, 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout: "fs_name": "cephfs" 2023-12-19T16:13:49.218 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:13:49.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:49.229 DEBUG:teuthology.orchestra.run.smithi050:> stat --file-system '--printf=%T 2023-12-19T16:13:49.230 DEBUG:teuthology.orchestra.run.smithi050:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:49.252 INFO:teuthology.orchestra.run.smithi050.stdout:fuseblk 2023-12-19T16:13:49.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:49.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:49.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:49.337 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-16_14_00 in ~21s... 2023-12-19T16:13:49.338 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-12-19T16:13:49.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T16:13:49.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.551+0000 7f2accdc5700 1 Processor -- start 2023-12-19T16:13:49.557 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.551+0000 7f2accdc5700 1 -- start start 2023-12-19T16:13:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2accdc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2accdc5700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2ac8158010 0x7f2ac806c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2accdc5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8067d00 con 0x7f2ac806c8b0 2023-12-19T16:13:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2accdc5700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac806cd30 con 0x7f2ac8158010 2023-12-19T16:13:49.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2accdc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8158480 con 0x7f2ac8151550 2023-12-19T16:13:49.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2ac6d9d700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f2ac806c8b0 0x7f2ac8068190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52684/0 (socket says 172.21.15.50:52684) 2023-12-19T16:13:49.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/3579114061 learned_addr learned my addr 172.21.15.50:0/3579114061 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:49.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2ac659c700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2abffff700 1 --2- 172.21.15.50:0/3579114061 >> v2:172.21.15.125:3300/0 conn(0x7f2ac8158010 0x7f2ac806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105060008 0 0) 0x7f2ac8067d00 con 0x7f2ac806c8b0 2023-12-19T16:13:49.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.555+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab4003570 con 0x7f2ac806c8b0 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684868061 0 0) 0x7f2ab4003570 con 0x7f2ac806c8b0 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 msgr2=0x7f2ac80c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 >> v2:172.21.15.125:3300/0 conn(0x7f2ac8158010 msgr2=0x7f2ac806c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 --2- 172.21.15.50:0/3579114061 >> v2:172.21.15.125:3300/0 conn(0x7f2ac8158010 0x7f2ac806c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80c8540 con 0x7f2ac806c8b0 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f2ac00017f0 con 0x7f2ac806c8b0 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 >> v1:172.21.15.50:6789/0 conn(0x7f2ac806c8b0 legacy=0x7f2ac8068190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ab4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab4003fe0 0x7f2ab4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ab4004d50 0x7f2ab40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac806cd30 con 0x7f2ab4003fe0 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8158480 con 0x7f2ab4004d50 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab400ba90 con 0x7f2ac8151550 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2abffff700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab4003fe0 0x7f2ab4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ab4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac6d9d700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ab4004d50 0x7f2ab40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 -- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ab4004d50 msgr2=0x7f2ab40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ab4004d50 0x7f2ab40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 -- 172.21.15.50:0/3579114061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab4003fe0 msgr2=0x7f2ab4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ab4003fe0 0x7f2ab4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 -- 172.21.15.50:0/3579114061 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4003570 con 0x7f2ac8151550 2023-12-19T16:13:49.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac659c700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ab4003a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009c00 tx=0x7f2ab80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae4f8761155ea00 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800e6b0 con 0x7f2ac8151550 2023-12-19T16:13:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800e810 con 0x7f2ac8151550 2023-12-19T16:13:49.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.559+0000 7f2ac5d9b700 1 -- 172.21.15.50:0/3579114061 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800ebd0 con 0x7f2ac8151550 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 msgr2=0x7f2ab4003a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 --2- 172.21.15.50:0/3579114061 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ab4003a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009c00 tx=0x7f2ab80091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/3579114061 reap_dead start 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- 172.21.15.50:0/3579114061 shutdown_connections 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- 172.21.15.50:0/3579114061 >> 172.21.15.50:0/3579114061 conn(0x7f2ac80cce60 msgr2=0x7f2ac80cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- 172.21.15.50:0/3579114061 shutdown_connections 2023-12-19T16:13:49.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- 172.21.15.50:0/3579114061 wait complete. 2023-12-19T16:13:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 Processor -- start 2023-12-19T16:13:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.563+0000 7f2accdc5700 1 -- start start 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ac80c5b60 0x7f2ac80c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80ae420 con 0x7f2ac80c6510 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80ae700 con 0x7f2ac80c5b60 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80ae9e0 con 0x7f2ac8151550 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41412/0 (socket says 172.21.15.50:41412) 2023-12-19T16:13:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/311371504 learned_addr learned my addr 172.21.15.50:0/311371504 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:49.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2abffff700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ac80c5b60 0x7f2ac80c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac659c700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 msgr2=0x7f2ac80c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ac80c5b60 msgr2=0x7f2ac80c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ac80c5b60 0x7f2ac80c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 -- 172.21.15.50:0/311371504 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80aecc0 con 0x7f2ac80c6510 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2ac6d9d700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80b2e60 tx=0x7f2ac8067c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0007b50 con 0x7f2ac80c6510 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000eb10 con 0x7f2ac80c6510 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000eeb0 con 0x7f2ac80c6510 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac80aefa0 con 0x7f2ac80c6510 2023-12-19T16:13:49.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.567+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac80c4450 con 0x7f2ac80c6510 2023-12-19T16:13:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.571+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80af190 con 0x7f2ac80c6510 2023-12-19T16:13:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.571+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 104) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f2ac0013630 con 0x7f2ac80c6510 2023-12-19T16:13:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.575+0000 7f2abe7fc700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 0x7f2aa8062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:49.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.575+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f2ac007d2e0 con 0x7f2ac80c6510 2023-12-19T16:13:49.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.575+0000 7f2ac659c700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 0x7f2aa8062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:49.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.575+0000 7f2ac659c700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 0x7f2aa8062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80091e0 tx=0x7f2ab801a000 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:49.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.579+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ac004d2a0 con 0x7f2ac80c6510 2023-12-19T16:13:49.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.771+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ac80c4670 con 0x7f2aa8060780 2023-12-19T16:13:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.787+0000 7f2abe7fc700 1 -- 172.21.15.50:0/311371504 <== mgr.10378 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f2ac80c4670 con 0x7f2aa8060780 2023-12-19T16:13:49.791 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:49.792 INFO:teuthology.orchestra.run.smithi050.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T16:13:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 msgr2=0x7f2aa8062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 0x7f2aa8062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80091e0 tx=0x7f2ab801a000 comp rx=0 tx=0).stop 2023-12-19T16:13:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 msgr2=0x7f2ac80caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:49.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80b2e60 tx=0x7f2ac8067c70 comp rx=0 tx=0).stop 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 shutdown_connections 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2aa8060780 0x7f2aa8062c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ac80c5b60 0x7f2ac80c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ac80c6510 0x7f2ac80caa30 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 --2- 172.21.15.50:0/311371504 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2ac8151550 0x7f2ac80c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 >> 172.21.15.50:0/311371504 conn(0x7f2ac80cce60 msgr2=0x7f2ac8068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:49.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 shutdown_connections 2023-12-19T16:13:49.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.791+0000 7f2accdc5700 1 -- 172.21.15.50:0/311371504 wait complete. 2023-12-19T16:13:49.816 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T16:13:49.817 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-12-19T16:13:49.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T16:13:49.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.991+0000 7fcb50145700 1 Processor -- start 2023-12-19T16:13:49.996 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.991+0000 7fcb50145700 1 -- start start 2023-12-19T16:13:50.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb50145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb48159030 0x7fcb480cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb50145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb480ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.000 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb50145700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb48067da0 con 0x7fcb48159400 2023-12-19T16:13:50.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb50145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb48068940 con 0x7fcb48159030 2023-12-19T16:13:50.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb50145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48067190 con 0x7fcb480cfcb0 2023-12-19T16:13:50.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb48159030 0x7fcb480cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcb48159030 0x7fcb480cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41416/0 (socket says 172.21.15.50:41416) 2023-12-19T16:13:50.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 -- 172.21.15.50:0/1560194811 learned_addr learned my addr 172.21.15.50:0/1560194811 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4e6e2700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb480ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 -- 172.21.15.50:0/1560194811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 msgr2=0x7fcb480ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb480ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4dee1700 1 -- 172.21.15.50:0/1560194811 >> v1:172.21.15.125:6789/0 conn(0x7fcb48159400 legacy=0x7fcb48068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4dee1700 1 -- 172.21.15.50:0/1560194811 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48067470 con 0x7fcb48159030 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.995+0000 7fcb4cedf700 1 -- 172.21.15.50:0/1560194811 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80256964 0 0) 0x7fcb48067da0 con 0x7fcb48159400 2023-12-19T16:13:50.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4e6e2700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb480ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:50.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4dee1700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb480cb170 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fcb440099b0 tx=0x7fcb44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca47d5ff3b4bcc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4cedf700 1 -- 172.21.15.50:0/1560194811 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400b860 con 0x7fcb48159030 2023-12-19T16:13:50.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4cedf700 1 -- 172.21.15.50:0/1560194811 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4400b9c0 con 0x7fcb48159030 2023-12-19T16:13:50.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb4cedf700 1 -- 172.21.15.50:0/1560194811 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400bd60 con 0x7fcb48159030 2023-12-19T16:13:50.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 -- 172.21.15.50:0/1560194811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 msgr2=0x7fcb480cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb480cb170 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fcb440099b0 tx=0x7fcb44009250 comp rx=0 tx=0).stop 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 -- 172.21.15.50:0/1560194811 shutdown_connections 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb480ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 --2- 172.21.15.50:0/1560194811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb480cb170 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 -- 172.21.15.50:0/1560194811 >> 172.21.15.50:0/1560194811 conn(0x7fcb480d0170 msgr2=0x7fcb480d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 -- 172.21.15.50:0/1560194811 shutdown_connections 2023-12-19T16:13:50.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:49.999+0000 7fcb50145700 1 -- 172.21.15.50:0/1560194811 wait complete. 2023-12-19T16:13:50.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.003+0000 7fcb50145700 1 Processor -- start 2023-12-19T16:13:50.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.003+0000 7fcb50145700 1 -- start start 2023-12-19T16:13:50.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb4816d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb48159400 0x7fcb481528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4816dec0 con 0x7fcb48159030 2023-12-19T16:13:50.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb481699f0 con 0x7fcb48159400 2023-12-19T16:13:50.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480ca000 con 0x7fcb480cfcb0 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41418/0 (socket says 172.21.15.50:41418) 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 -- 172.21.15.50:0/622261144 learned_addr learned my addr 172.21.15.50:0/622261144 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4e6e2700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb48159400 0x7fcb481528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4dee1700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb4816d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 -- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 msgr2=0x7fcb4816d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb4816d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 -- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb48159400 msgr2=0x7fcb481528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb48159400 0x7fcb481528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 -- 172.21.15.50:0/622261144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440094f0 con 0x7fcb48159030 2023-12-19T16:13:50.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb4d6e0700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c560 tx=0x7fcb38007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38011590 con 0x7fcb48159030 2023-12-19T16:13:50.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38011710 con 0x7fcb48159030 2023-12-19T16:13:50.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb480c8ab0 con 0x7fcb48159030 2023-12-19T16:13:50.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38011ad0 con 0x7fcb48159030 2023-12-19T16:13:50.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.007+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb480c94a0 con 0x7fcb48159030 2023-12-19T16:13:50.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.011+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480708a0 con 0x7fcb48159030 2023-12-19T16:13:50.015 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.011+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 104) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fcb3801b3b0 con 0x7fcb48159030 2023-12-19T16:13:50.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.011+0000 7fcb3dffb700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 0x7fcb2c062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.011+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fcb3807c240 con 0x7fcb48159030 2023-12-19T16:13:50.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.011+0000 7fcb4dee1700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 0x7fcb2c062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.015+0000 7fcb4dee1700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 0x7fcb2c062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb440099b0 tx=0x7fcb44017000 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.019+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcb3804c200 con 0x7fcb48159030 2023-12-19T16:13:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.211+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb480ca1e0 con 0x7fcb2c060750 2023-12-19T16:13:50.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.343+0000 7fcb3dffb700 1 -- 172.21.15.50:0/622261144 <== mgr.10378 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcb480ca1e0 con 0x7fcb2c060750 2023-12-19T16:13:50.347 INFO:teuthology.orchestra.run.smithi050.stdout:Schedule removed for path snap_test_dir1 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 msgr2=0x7fcb2c062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 0x7fcb2c062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb440099b0 tx=0x7fcb44017000 comp rx=0 tx=0).stop 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 msgr2=0x7fcb4815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c560 tx=0x7fcb38007f50 comp rx=0 tx=0).stop 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 shutdown_connections 2023-12-19T16:13:50.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fcb2c060750 0x7fcb2c062c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb48159400 0x7fcb481528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb48159030 0x7fcb4815b1d0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 --2- 172.21.15.50:0/622261144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fcb480cfcb0 0x7fcb4816d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 >> 172.21.15.50:0/622261144 conn(0x7fcb480d0170 msgr2=0x7fcb480afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 shutdown_connections 2023-12-19T16:13:50.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.347+0000 7fcb50145700 1 -- 172.21.15.50:0/622261144 wait complete. 2023-12-19T16:13:50.373 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T16:13:50.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:50.373 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T16:13:50.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T16:13:50.403 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T16:13:50.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T16:13:50.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.691+0000 7f2f7a46e700 1 Processor -- start 2023-12-19T16:13:50.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.691+0000 7f2f7a46e700 1 -- start start 2023-12-19T16:13:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f7a46e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ac140 0x7f2f740ac510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f7a46e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2f7406b000 0x7f2f740adda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f7a46e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740ab740 con 0x7f2f740ae2e0 2023-12-19T16:13:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f7a46e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7415aa90 con 0x7f2f7406b000 2023-12-19T16:13:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f7a46e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7406a560 con 0x7f2f740ac140 2023-12-19T16:13:50.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2f7406b000 0x7f2f740adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2f7406b000 0x7f2f740adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41422/0 (socket says 172.21.15.50:41422) 2023-12-19T16:13:50.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 -- 172.21.15.50:0/3194639105 learned_addr learned my addr 172.21.15.50:0/3194639105 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:50.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f78a0b700 1 --1- 172.21.15.50:0/3194639105 >> v1:172.21.15.50:6789/0 conn(0x7f2f740ae2e0 0x7f2f740aa830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52686/0 (socket says 172.21.15.50:52686) 2023-12-19T16:13:50.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f73fff700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ac140 0x7f2f740ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 -- 172.21.15.50:0/3194639105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ac140 msgr2=0x7f2f740ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:50.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ac140 0x7f2f740ac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 -- 172.21.15.50:0/3194639105 >> v1:172.21.15.50:6789/0 conn(0x7f2f740ae2e0 legacy=0x7f2f740aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 -- 172.21.15.50:0/3194639105 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f74068280 con 0x7f2f7406b000 2023-12-19T16:13:50.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f737fe700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f7406b000 0x7f2f740adda0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0099b0 tx=0x7f2f5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ff62b4175652d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f72ffd700 1 -- 172.21.15.50:0/3194639105 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c009ea0 con 0x7f2f7406b000 2023-12-19T16:13:50.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f72ffd700 1 -- 172.21.15.50:0/3194639105 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c00b860 con 0x7f2f7406b000 2023-12-19T16:13:50.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.695+0000 7f2f72ffd700 1 -- 172.21.15.50:0/3194639105 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c00bc00 con 0x7f2f7406b000 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 -- 172.21.15.50:0/3194639105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f7406b000 msgr2=0x7f2f740adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f7406b000 0x7f2f740adda0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0099b0 tx=0x7f2f5c009250 comp rx=0 tx=0).stop 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 -- 172.21.15.50:0/3194639105 shutdown_connections 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f7406b000 0x7f2f740adda0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/3194639105 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ac140 0x7f2f740ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 -- 172.21.15.50:0/3194639105 >> 172.21.15.50:0/3194639105 conn(0x7f2f740cc680 msgr2=0x7f2f740c8620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 -- 172.21.15.50:0/3194639105 shutdown_connections 2023-12-19T16:13:50.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 -- 172.21.15.50:0/3194639105 wait complete. 2023-12-19T16:13:50.705 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.699+0000 7f2f7a46e700 1 Processor -- start 2023-12-19T16:13:50.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 -- start start 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7406b000 0x7f2f7415a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ae2e0 0x7f2f7415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7415a5c0 con 0x7f2f740ac140 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740a44a0 con 0x7f2f7406b000 2023-12-19T16:13:50.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f7a46e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740a4780 con 0x7f2f740ae2e0 2023-12-19T16:13:50.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41432/0 (socket says 172.21.15.50:41432) 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 -- 172.21.15.50:0/967089163 learned_addr learned my addr 172.21.15.50:0/967089163 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f73fff700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7406b000 0x7f2f7415a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f78a0b700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ae2e0 0x7f2f7415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 -- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ae2e0 msgr2=0x7f2f7415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ae2e0 0x7f2f7415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 -- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7406b000 msgr2=0x7f2f7415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7406b000 0x7f2f7415a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:50.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.703+0000 7f2f737fe700 1 -- 172.21.15.50:0/967089163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0094f0 con 0x7f2f740ac140 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f737fe700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f2f740c83b0 tx=0x7f2f5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c000d10 con 0x7f2f740ac140 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f740a7f50 con 0x7f2f740ac140 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c00bbe0 con 0x7f2f740ac140 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c011670 con 0x7f2f740ac140 2023-12-19T16:13:50.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f740a8510 con 0x7f2f740ac140 2023-12-19T16:13:50.712 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.707+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f740a48f0 con 0x7f2f740ac140 2023-12-19T16:13:50.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.711+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 104) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f2f5c011aa0 con 0x7f2f740ac140 2023-12-19T16:13:50.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.711+0000 7f2f6bfff700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 0x7f2f54062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:50.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.711+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f2f5c07e740 con 0x7f2f740ac140 2023-12-19T16:13:50.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.711+0000 7f2f73fff700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 0x7f2f54062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:50.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.711+0000 7f2f73fff700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 0x7f2f54062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009db0 tx=0x7f2f640091a0 comp rx=0 tx=0).ready entity=mgr.10378 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:50.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.719+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2f5c04e700 con 0x7f2f740ac140 2023-12-19T16:13:50.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:50.931+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f2f74071110 con 0x7f2f740ac140 2023-12-19T16:13:51.333 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.327+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f2f5c011d50 con 0x7f2f740ac140 2023-12-19T16:13:51.336 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.331+0000 7f2f6bfff700 1 -- 172.21.15.50:0/967089163 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v105) v1 ==== 93+0+0 (secure 0 0 0) 0x7f2f5c052190 con 0x7f2f740ac140 2023-12-19T16:13:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 msgr2=0x7f2f54062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 0x7f2f54062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009db0 tx=0x7f2f640091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 msgr2=0x7f2f740a4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f2f740c83b0 tx=0x7f2f5c017000 comp rx=0 tx=0).stop 2023-12-19T16:13:51.340 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 shutdown_connections 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f2f54060750 0x7f2f54062c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f2f740ae2e0 0x7f2f7415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2f740ac140 0x7f2f740a4c10 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 --2- 172.21.15.50:0/967089163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f7406b000 0x7f2f7415a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 >> 172.21.15.50:0/967089163 conn(0x7f2f740cc680 msgr2=0x7f2f740c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 shutdown_connections 2023-12-19T16:13:51.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.335+0000 7f2f7a46e700 1 -- 172.21.15.50:0/967089163 wait complete. 2023-12-19T16:13:51.366 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T16:13:51.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:51.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T16:13:51.441 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T16:13:51.441 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi050.front.sepia.ceph.com... 2023-12-19T16:13:51.441 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T16:13:51.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T16:13:51.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi050.stderr:ceph-fuse[62385]: fuse finished with error 0 and tester_r 0 2023-12-19T16:13:51.483 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:13:51.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:51.484 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T16:13:51.537 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:51.525+0000 7f14ecef3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:51.547 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:13:51.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:51.547 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T16:13:51.550 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:51.547+0000 7f0b80723000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T16:13:51.566 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T16:13:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:51.567 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:13:51.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:51.568 DEBUG:teuthology.orchestra.run.smithi050:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T16:13:51.611 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T16:13:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:51.613 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:13:51.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:13:51.613 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T16:13:51.632 INFO:teuthology.orchestra.run.smithi125.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T16:13:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:13:51.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:51.822 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 Processor -- start 2023-12-19T16:13:51.823 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 -- start start 2023-12-19T16:13:51.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1d440ce2f0 0x7f1d440c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:51.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d44069140 con 0x7f1d440cdf20 2023-12-19T16:13:51.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4415d040 con 0x7f1d440c88d0 2023-12-19T16:13:51.825 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d4a145700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d440ae1b0 con 0x7f1d440ce2f0 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d3bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1d440ce2f0 0x7f1d440c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d3bfff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1d440ce2f0 0x7f1d440c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41458/0 (socket says 172.21.15.50:41458) 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3400433500 learned_addr learned my addr 172.21.15.50:0/3400433500 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.819+0000 7f1d437fe700 1 --1- 172.21.15.50:0/3400433500 >> v1:172.21.15.50:6789/0 conn(0x7f1d440cdf20 0x7f1d440adaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52702/0 (socket says 172.21.15.50:52702) 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3400433500 >> v1:172.21.15.125:6789/0 conn(0x7f1d440c88d0 legacy=0x7f1d44158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3400433500 >> v1:172.21.15.50:6789/0 conn(0x7f1d440cdf20 legacy=0x7f1d440adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.826 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3400433500 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440c8ee0 con 0x7f1d440ce2f0 2023-12-19T16:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d3bfff700 1 --2- 172.21.15.50:0/3400433500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440ce2f0 0x7f1d440c8390 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0099b0 tx=0x7f1d2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67388c77c63e870 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d42ffd700 1 -- 172.21.15.50:0/3400433500 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00b8f0 con 0x7f1d440ce2f0 2023-12-19T16:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d42ffd700 1 -- 172.21.15.50:0/3400433500 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00ba50 con 0x7f1d440ce2f0 2023-12-19T16:13:51.827 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d42ffd700 1 -- 172.21.15.50:0/3400433500 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00bdf0 con 0x7f1d440ce2f0 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 -- 172.21.15.50:0/3400433500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440ce2f0 msgr2=0x7f1d440c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3400433500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440ce2f0 0x7f1d440c8390 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0099b0 tx=0x7f1d2c009250 comp rx=0 tx=0).stop 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 -- 172.21.15.50:0/3400433500 shutdown_connections 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3400433500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440ce2f0 0x7f1d440c8390 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 -- 172.21.15.50:0/3400433500 >> 172.21.15.50:0/3400433500 conn(0x7f1d4406ac10 msgr2=0x7f1d4406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 -- 172.21.15.50:0/3400433500 shutdown_connections 2023-12-19T16:13:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.823+0000 7f1d4a145700 1 -- 172.21.15.50:0/3400433500 wait complete. 2023-12-19T16:13:51.829 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 Processor -- start 2023-12-19T16:13:51.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 -- start start 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 0x7f1d440b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d440ce2f0 0x7f1d441490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4416e020 con 0x7f1d440cdf20 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4416aaa0 con 0x7f1d440c88d0 2023-12-19T16:13:51.831 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440b26e0 con 0x7f1d440ce2f0 2023-12-19T16:13:51.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:51.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:57284/0 (socket says 172.21.15.50:57284) 2023-12-19T16:13:51.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 -- 172.21.15.50:0/3881114917 learned_addr learned my addr 172.21.15.50:0/3881114917 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:51.832 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d3bfff700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 0x7f1d440b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d43fff700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d440ce2f0 0x7f1d441490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d440ce2f0 msgr2=0x7f1d441490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d440ce2f0 0x7f1d441490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 msgr2=0x7f1d440b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 0x7f1d440b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 -- 172.21.15.50:0/3881114917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0094f0 con 0x7f1d440c88d0 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d3bfff700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 0x7f1d440b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:51.833 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d437fe700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400c7f0 tx=0x7f1d34007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34005a70 con 0x7f1d440c88d0 2023-12-19T16:13:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d34005bf0 con 0x7f1d440c88d0 2023-12-19T16:13:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d440c6090 con 0x7f1d440c88d0 2023-12-19T16:13:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.827+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340070d0 con 0x7f1d440c88d0 2023-12-19T16:13:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.831+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4416a200 con 0x7f1d440c88d0 2023-12-19T16:13:51.835 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.831+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d440b0c30 con 0x7f1d440c88d0 2023-12-19T16:13:51.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.831+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f1d34007480 con 0x7f1d440c88d0 2023-12-19T16:13:51.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.835+0000 7f1d3b7fe700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 0x7f1d24062c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:51.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.835+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f1d3407c320 con 0x7f1d440c88d0 2023-12-19T16:13:51.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.835+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 msgr2=0x7f1d24062c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:51.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.835+0000 7f1d3bfff700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 0x7f1d24062c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:51.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:51.843+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1d3404c360 con 0x7f1d440c88d0 2023-12-19T16:13:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.039+0000 7f1d3bfff700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 msgr2=0x7f1d24062c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:52.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.039+0000 7f1d3bfff700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 0x7f1d24062c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:52.067 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.057+0000 7f14ecef3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:52.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.067+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d44071110 con 0x7f1d440c88d0 2023-12-19T16:13:52.071 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.067+0000 7f1d3b7fe700 1 -- 172.21.15.50:0/3881114917 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f1d3404f980 con 0x7f1d440c88d0 2023-12-19T16:13:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:52.072 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:52.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 msgr2=0x7f1d24062c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:52.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 0x7f1d24062c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 msgr2=0x7f1d4416dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400c7f0 tx=0x7f1d34007f70 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 shutdown_connections 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1d240607d0 0x7f1d24062c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1d440ce2f0 0x7f1d441490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1d440cdf20 0x7f1d440b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 --2- 172.21.15.50:0/3881114917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d440c88d0 0x7f1d4416dae0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 >> 172.21.15.50:0/3881114917 conn(0x7f1d4406ac10 msgr2=0x7f1d44157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 shutdown_connections 2023-12-19T16:13:52.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.071+0000 7f1d4a145700 1 -- 172.21.15.50:0/3881114917 wait complete. 2023-12-19T16:13:52.079 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:52.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:52.138 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.135+0000 7f0b80723000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T16:13:52.144 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.133+0000 7f14ecef3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:52.223 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.219+0000 7f0b80723000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T16:13:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 Processor -- start 2023-12-19T16:13:52.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 -- start start 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ae8159010 0x7f5ae80b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ae80b3300 0x7f5ae80cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae80cbab0 con 0x7f5ae80b3300 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae80caea0 con 0x7f5ae8159010 2023-12-19T16:13:52.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5aec8cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80cb180 con 0x7f5ae81593e0 2023-12-19T16:13:52.262 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5ae6d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ae80b3300 0x7f5ae80cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5ae6d9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5ae80b3300 0x7f5ae80cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41478/0 (socket says 172.21.15.50:41478) 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.255+0000 7f5ae6d9d700 1 -- 172.21.15.50:0/839089138 learned_addr learned my addr 172.21.15.50:0/839089138 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae659c700 1 --2- 172.21.15.50:0/839089138 >> v2:172.21.15.125:3300/0 conn(0x7f5ae8159010 0x7f5ae80b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae5d9b700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 -- 172.21.15.50:0/839089138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 msgr2=0x7f5ae80cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 -- 172.21.15.50:0/839089138 >> v2:172.21.15.125:3300/0 conn(0x7f5ae8159010 msgr2=0x7f5ae80b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 --2- 172.21.15.50:0/839089138 >> v2:172.21.15.125:3300/0 conn(0x7f5ae8159010 0x7f5ae80b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 -- 172.21.15.50:0/839089138 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80cb460 con 0x7f5ae80b3300 2023-12-19T16:13:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae6d9d700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae80b3300 0x7f5ae80cc480 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f5ae806a0e0 tx=0x7f5ae0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f60283ae66285f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae559a700 1 -- 172.21.15.50:0/839089138 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000e070 con 0x7f5ae80b3300 2023-12-19T16:13:52.264 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae559a700 1 -- 172.21.15.50:0/839089138 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae000ba20 con 0x7f5ae80b3300 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5ae559a700 1 -- 172.21.15.50:0/839089138 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00092e0 con 0x7f5ae80b3300 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 -- 172.21.15.50:0/839089138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae80b3300 msgr2=0x7f5ae80cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae80b3300 0x7f5ae80cc480 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f5ae806a0e0 tx=0x7f5ae0009840 comp rx=0 tx=0).stop 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 -- 172.21.15.50:0/839089138 shutdown_connections 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae80b3300 0x7f5ae80cc480 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/839089138 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/839089138 >> v2:172.21.15.125:3300/0 conn(0x7f5ae8159010 0x7f5ae80b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 -- 172.21.15.50:0/839089138 >> 172.21.15.50:0/839089138 conn(0x7f5ae80d0ac0 msgr2=0x7f5ae80cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 -- 172.21.15.50:0/839089138 shutdown_connections 2023-12-19T16:13:52.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.259+0000 7f5aec8cd700 1 -- 172.21.15.50:0/839089138 wait complete. 2023-12-19T16:13:52.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 Processor -- start 2023-12-19T16:13:52.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 -- start start 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ae80b3300 0x7f5ae8154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8154e50 con 0x7f5ae8159010 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80c9b70 con 0x7f5ae80b3300 2023-12-19T16:13:52.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5aec8cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80c8f60 con 0x7f5ae81593e0 2023-12-19T16:13:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5ae5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5ae5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41486/0 (socket says 172.21.15.50:41486) 2023-12-19T16:13:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5ae5d9b700 1 -- 172.21.15.50:0/1912051130 learned_addr learned my addr 172.21.15.50:0/1912051130 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.263+0000 7f5ae6d9d700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae659c700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ae80b3300 0x7f5ae8154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 -- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 msgr2=0x7f5ae80b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 -- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ae80b3300 msgr2=0x7f5ae8154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ae80b3300 0x7f5ae8154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 -- 172.21.15.50:0/1912051130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0009ae0 con 0x7f5ae8159010 2023-12-19T16:13:52.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ae5d9b700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00d860 tx=0x7f5adc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc0125f0 con 0x7f5ae8159010 2023-12-19T16:13:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc012750 con 0x7f5ae8159010 2023-12-19T16:13:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae80c9240 con 0x7f5ae8159010 2023-12-19T16:13:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc012af0 con 0x7f5ae8159010 2023-12-19T16:13:52.272 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae80c9800 con 0x7f5ae8159010 2023-12-19T16:13:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.267+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae8071110 con 0x7f5ae8159010 2023-12-19T16:13:52.275 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.271+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f5adc013480 con 0x7f5ae8159010 2023-12-19T16:13:52.276 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.271+0000 7f5ad67fc700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 0x7f5ac8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.271+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f5adc07d0f0 con 0x7f5ae8159010 2023-12-19T16:13:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.271+0000 7f5ae659c700 1 -- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 msgr2=0x7f5ac8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:52.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.271+0000 7f5ae659c700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 0x7f5ac8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:52.278 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.269+0000 7f14ecef3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:52.283 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.279+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5adc050be0 con 0x7f5ae8159010 2023-12-19T16:13:52.371 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.367+0000 7f0b80723000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:52.458 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.455+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae80b19f0 con 0x7f5ae8159010 2023-12-19T16:13:52.460 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.455+0000 7f5ad67fc700 1 -- 172.21.15.50:0/1912051130 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+16245 (secure 0 0 0) 0x7f5adc050770 con 0x7f5ae8159010 2023-12-19T16:13:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":214,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:37.811943+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/651582691":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2769533890":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/878125710":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2905251978":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/3734826841":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2299774917":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6833/44063":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6832/44063":"2023-12-20T16:13:37.811876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:52.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 msgr2=0x7f5ac8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:52.465 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 0x7f5ac8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 msgr2=0x7f5ae80b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00d860 tx=0x7f5adc008040 comp rx=0 tx=0).stop 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 shutdown_connections 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5ac8060700 0x7f5ac8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f5ae81593e0 0x7f5ae80b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ae8159010 0x7f5ae80b4850 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 --2- 172.21.15.50:0/1912051130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ae80b3300 0x7f5ae8154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 >> 172.21.15.50:0/1912051130 conn(0x7f5ae80d0ac0 msgr2=0x7f5ae80b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:52.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.459+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 shutdown_connections 2023-12-19T16:13:52.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.463+0000 7f5aec8cd700 1 -- 172.21.15.50:0/1912051130 wait complete. 2023-12-19T16:13:52.486 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T14:38:56.213601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T16:13:21.069471+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T16:13:22.410710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T16:13:52.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T16:13:52.535 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.525+0000 7f14ecef3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:52.605 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.593+0000 7f14ecef3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:52.664 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.659+0000 7f0b80723000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T16:13:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.699+0000 7fea89523700 1 Processor -- start 2023-12-19T16:13:52.704 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.699+0000 7fea89523700 1 -- start start 2023-12-19T16:13:52.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea89523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea89523700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fea840cef90 0x7fea8415ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea89523700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea84067060 con 0x7fea84158c60 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea89523700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea84067900 con 0x7fea840cef90 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea89523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840cf750 con 0x7fea8415a040 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:40116/0 (socket says 172.21.15.50:40116) 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 -- 172.21.15.50:0/3037966887 learned_addr learned my addr 172.21.15.50:0/3037966887 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:52.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea88d22700 1 --2- 172.21.15.50:0/3037966887 >> v2:172.21.15.125:3300/0 conn(0x7fea840cef90 0x7fea8415ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 -- 172.21.15.50:0/3037966887 >> v2:172.21.15.125:3300/0 conn(0x7fea840cef90 msgr2=0x7fea8415ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 --2- 172.21.15.50:0/3037966887 >> v2:172.21.15.125:3300/0 conn(0x7fea840cef90 0x7fea8415ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 -- 172.21.15.50:0/3037966887 >> v1:172.21.15.125:6789/0 conn(0x7fea84158c60 legacy=0x7fea840b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 -- 172.21.15.50:0/3037966887 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840672b0 con 0x7fea8415a040 2023-12-19T16:13:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.703+0000 7fea837fe700 1 --2- 172.21.15.50:0/3037966887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fea780099b0 tx=0x7fea78009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83f6bb0f670d123d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:52.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea82ffd700 1 -- 172.21.15.50:0/3037966887 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78009ea0 con 0x7fea8415a040 2023-12-19T16:13:52.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea82ffd700 1 -- 172.21.15.50:0/3037966887 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7800b860 con 0x7fea8415a040 2023-12-19T16:13:52.710 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea82ffd700 1 -- 172.21.15.50:0/3037966887 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7800bc00 con 0x7fea8415a040 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 -- 172.21.15.50:0/3037966887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 msgr2=0x7fea840b2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 --2- 172.21.15.50:0/3037966887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fea780099b0 tx=0x7fea78009250 comp rx=0 tx=0).stop 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 -- 172.21.15.50:0/3037966887 shutdown_connections 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 --2- 172.21.15.50:0/3037966887 >> v2:172.21.15.125:3300/0 conn(0x7fea840cef90 0x7fea8415ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 --2- 172.21.15.50:0/3037966887 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea8415a040 0x7fea840b2cd0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 -- 172.21.15.50:0/3037966887 >> 172.21.15.50:0/3037966887 conn(0x7fea84068f80 msgr2=0x7fea84069380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 -- 172.21.15.50:0/3037966887 shutdown_connections 2023-12-19T16:13:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.707+0000 7fea89523700 1 -- 172.21.15.50:0/3037966887 wait complete. 2023-12-19T16:13:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 Processor -- start 2023-12-19T16:13:52.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 -- start start 2023-12-19T16:13:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea840cef90 0x7fea8416d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 0x7fea8414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea840b0810 con 0x7fea8415a040 2023-12-19T16:13:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8414ec20 con 0x7fea840cef90 2023-12-19T16:13:52.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea89523700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840af920 con 0x7fea84158c60 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41494/0 (socket says 172.21.15.50:41494) 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 -- 172.21.15.50:0/2057861549 learned_addr learned my addr 172.21.15.50:0/2057861549 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea83fff700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea840cef90 0x7fea8416d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea837fe700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 0x7fea8414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 -- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 msgr2=0x7fea8414f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 0x7fea8414f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 -- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea840cef90 msgr2=0x7fea8416d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea840cef90 0x7fea8416d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea88d22700 1 -- 172.21.15.50:0/2057861549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea780094f0 con 0x7fea8415a040 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.711+0000 7fea837fe700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 0x7fea8414f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea88d22700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009df0 tx=0x7fea7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:52.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00cb50 con 0x7fea8415a040 2023-12-19T16:13:52.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c00f040 con 0x7fea8415a040 2023-12-19T16:13:52.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea840afc00 con 0x7fea8415a040 2023-12-19T16:13:52.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c014480 con 0x7fea8415a040 2023-12-19T16:13:52.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea840b01c0 con 0x7fea8415a040 2023-12-19T16:13:52.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea840b0980 con 0x7fea8415a040 2023-12-19T16:13:52.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.715+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fea7c014830 con 0x7fea8415a040 2023-12-19T16:13:52.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.719+0000 7fea6bfff700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 0x7fea70062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:52.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.719+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7fea7c07d900 con 0x7fea8415a040 2023-12-19T16:13:52.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.719+0000 7fea83fff700 1 -- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 msgr2=0x7fea70062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:52.722 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.719+0000 7fea83fff700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 0x7fea70062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:52.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.723+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fea7c04d7b0 con 0x7fea8415a040 2023-12-19T16:13:52.742 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.739+0000 7f0b80723000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:52.807 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.797+0000 7f14ecef3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:52.890 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.881+0000 7f14ecef3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:52.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.895+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea8404be40 con 0x7fea8415a040 2023-12-19T16:13:52.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.899+0000 7fea6bfff700 1 -- 172.21.15.50:0/2057861549 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea7c018020 con 0x7fea8415a040 2023-12-19T16:13:52.902 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 msgr2=0x7fea70062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 0x7fea70062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 msgr2=0x7fea841566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009df0 tx=0x7fea7c00b040 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 shutdown_connections 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fea70060700 0x7fea70062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea8415a040 0x7fea841566e0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fea84158c60 0x7fea8414f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 --2- 172.21.15.50:0/2057861549 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea840cef90 0x7fea8416d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:52.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 >> 172.21.15.50:0/2057861549 conn(0x7fea84068f80 msgr2=0x7fea8414c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:52.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 shutdown_connections 2023-12-19T16:13:52.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:52.903+0000 7fea89523700 1 -- 172.21.15.50:0/2057861549 wait complete. 2023-12-19T16:13:52.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T16:13:52.972 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:52.961+0000 7f14ecef3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:52.980 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:52.975+0000 7f0b80723000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T16:13:53.070 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:53.067+0000 7f0b80723000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T16:13:53.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.099+0000 7f3ad0cf0700 1 Processor -- start 2023-12-19T16:13:53.102 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.099+0000 7f3ad0cf0700 1 -- start start 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.099+0000 7f3ad0cf0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3acc158ec0 0x7f3acc0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3acc15a800 con 0x7f3acc1597d0 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3acc067470 con 0x7f3acc158ec0 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0cf660 con 0x7f3acc0ceef0 2023-12-19T16:13:53.106 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3aca59c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3acc158ec0 0x7f3acc0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:40146/0 (socket says 172.21.15.50:40146) 2023-12-19T16:13:53.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/2740184353 learned_addr learned my addr 172.21.15.50:0/2740184353 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:53.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/2740184353 >> v2:172.21.15.125:3300/0 conn(0x7f3acc158ec0 msgr2=0x7f3acc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:53.107 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 --2- 172.21.15.50:0/2740184353 >> v2:172.21.15.125:3300/0 conn(0x7f3acc158ec0 0x7f3acc0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/2740184353 >> v1:172.21.15.125:6789/0 conn(0x7f3acc1597d0 legacy=0x7f3acc15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/2740184353 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0cf940 con 0x7f3acc0ceef0 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac9d9b700 1 --2- 172.21.15.50:0/2740184353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00099b0 tx=0x7f3ac0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=229423c24d1f42d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac959a700 1 -- 172.21.15.50:0/2740184353 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0009ea0 con 0x7f3acc0ceef0 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac959a700 1 -- 172.21.15.50:0/2740184353 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac000b860 con 0x7f3acc0ceef0 2023-12-19T16:13:53.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ac959a700 1 -- 172.21.15.50:0/2740184353 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000bc00 con 0x7f3acc0ceef0 2023-12-19T16:13:53.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/2740184353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 msgr2=0x7f3acc159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.103+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/2740184353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00099b0 tx=0x7f3ac0009250 comp rx=0 tx=0).stop 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/2740184353 shutdown_connections 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/2740184353 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc0ceef0 0x7f3acc159290 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/2740184353 >> v2:172.21.15.125:3300/0 conn(0x7f3acc158ec0 0x7f3acc0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/2740184353 >> 172.21.15.50:0/2740184353 conn(0x7f3acc0d0000 msgr2=0x7f3acc0d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/2740184353 shutdown_connections 2023-12-19T16:13:53.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/2740184353 wait complete. 2023-12-19T16:13:53.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 Processor -- start 2023-12-19T16:13:53.112 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.107+0000 7f3ad0cf0700 1 -- start start 2023-12-19T16:13:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3acc0ceef0 0x7f3acc15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc1597d0 0x7f3acc1578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc16da20 con 0x7f3acc158ec0 2023-12-19T16:13:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc16ce10 con 0x7f3acc0ceef0 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc16d0f0 con 0x7f3acc1597d0 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41500/0 (socket says 172.21.15.50:41500) 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/1263308432 learned_addr learned my addr 172.21.15.50:0/1263308432 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3acad9d700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc1597d0 0x7f3acc1578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.115 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3aca59c700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3acc0ceef0 0x7f3acc15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc1597d0 msgr2=0x7f3acc1578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc1597d0 0x7f3acc1578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3acc0ceef0 msgr2=0x7f3acc15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3acc0ceef0 0x7f3acc15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 -- 172.21.15.50:0/1263308432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00094f0 con 0x7f3acc158ec0 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ac9d9b700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ca5f0 tx=0x7f3ac0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0000d10 con 0x7f3acc158ec0 2023-12-19T16:13:53.116 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac000bbe0 con 0x7f3acc158ec0 2023-12-19T16:13:53.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0011670 con 0x7f3acc158ec0 2023-12-19T16:13:53.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3acc16d3d0 con 0x7f3acc158ec0 2023-12-19T16:13:53.117 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.111+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc157e10 con 0x7f3acc158ec0 2023-12-19T16:13:53.118 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc071110 con 0x7f3acc158ec0 2023-12-19T16:13:53.119 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f3ac0011aa0 con 0x7f3acc158ec0 2023-12-19T16:13:53.120 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3aba7fc700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 0x7f3aac062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f3ac007e710 con 0x7f3acc158ec0 2023-12-19T16:13:53.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3aca59c700 1 -- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 msgr2=0x7f3aac062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:53.121 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.115+0000 7f3aca59c700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 0x7f3aac062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:53.124 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.119+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3ac004e6d0 con 0x7f3acc158ec0 2023-12-19T16:13:53.167 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:53.163+0000 7f0b80723000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T16:13:53.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3acc0cfc20 con 0x7f3acc158ec0 2023-12-19T16:13:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3aba7fc700 1 -- 172.21.15.50:0/1263308432 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v214) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ac0051cf0 con 0x7f3acc158ec0 2023-12-19T16:13:53.315 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 32 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 msgr2=0x7f3aac062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 0x7f3aac062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 msgr2=0x7f3acc151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.311+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ca5f0 tx=0x7f3ac0017000 comp rx=0 tx=0).stop 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 shutdown_connections 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f3aac060700 0x7f3aac062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f3acc1597d0 0x7f3acc1578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3acc158ec0 0x7f3acc151a90 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 --2- 172.21.15.50:0/1263308432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3acc0ceef0 0x7f3acc15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 >> 172.21.15.50:0/1263308432 conn(0x7f3acc0d0000 msgr2=0x7f3acc154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 shutdown_connections 2023-12-19T16:13:53.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.315+0000 7f3ad0cf0700 1 -- 172.21.15.50:0/1263308432 wait complete. 2023-12-19T16:13:53.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T16:13:53.345 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:53.337+0000 7f14ecef3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:53.507 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:53.497+0000 7f14ecef3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:53.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 Processor -- start 2023-12-19T16:13:53.519 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 -- start start 2023-12-19T16:13:53.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f093015d730 0x7f09300c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.521 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093006a530 con 0x7f0930158f80 2023-12-19T16:13:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0930069c00 con 0x7f0930159350 2023-12-19T16:13:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.515+0000 7f093582e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300bfd80 con 0x7f093015d730 2023-12-19T16:13:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092effd700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0930158f80 0x7f09300bf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52716/0 (socket says 172.21.15.50:52716) 2023-12-19T16:13:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092effd700 1 -- 172.21.15.50:0/2292183856 learned_addr learned my addr 172.21.15.50:0/2292183856 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092e7fc700 1 --2- 172.21.15.50:0/2292183856 >> v2:172.21.15.125:3300/0 conn(0x7f093015d730 0x7f09300c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092dffb700 1 -- 172.21.15.50:0/2292183856 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765614025 0 0) 0x7f093006a530 con 0x7f0930158f80 2023-12-19T16:13:53.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092dffb700 1 -- 172.21.15.50:0/2292183856 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0918003570 con 0x7f0930158f80 2023-12-19T16:13:53.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092e7fc700 1 -- 172.21.15.50:0/2292183856 >> v1:172.21.15.125:6789/0 conn(0x7f0930159350 legacy=0x7f093006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092e7fc700 1 -- 172.21.15.50:0/2292183856 >> v1:172.21.15.50:6789/0 conn(0x7f0930158f80 legacy=0x7f09300bf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.523 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092e7fc700 1 -- 172.21.15.50:0/2292183856 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930069ee0 con 0x7f093015d730 2023-12-19T16:13:53.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092e7fc700 1 --2- 172.21.15.50:0/2292183856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f093015d730 0x7f09300c6aa0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f09240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1dca4ac320e50f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092dffb700 1 -- 172.21.15.50:0/2292183856 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400b640 con 0x7f093015d730 2023-12-19T16:13:53.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092dffb700 1 -- 172.21.15.50:0/2292183856 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092400b7a0 con 0x7f093015d730 2023-12-19T16:13:53.524 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f092dffb700 1 -- 172.21.15.50:0/2292183856 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092400bb40 con 0x7f093015d730 2023-12-19T16:13:53.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 -- 172.21.15.50:0/2292183856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f093015d730 msgr2=0x7f09300c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 --2- 172.21.15.50:0/2292183856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f093015d730 0x7f09300c6aa0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f09240098e0 tx=0x7f09240091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.525 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 -- 172.21.15.50:0/2292183856 shutdown_connections 2023-12-19T16:13:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 --2- 172.21.15.50:0/2292183856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f093015d730 0x7f09300c6aa0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 -- 172.21.15.50:0/2292183856 >> 172.21.15.50:0/2292183856 conn(0x7f09300cc6c0 msgr2=0x7f09300c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 -- 172.21.15.50:0/2292183856 shutdown_connections 2023-12-19T16:13:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.519+0000 7f093582e700 1 -- 172.21.15.50:0/2292183856 wait complete. 2023-12-19T16:13:53.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.523+0000 7f093582e700 1 Processor -- start 2023-12-19T16:13:53.528 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.523+0000 7f093582e700 1 -- start start 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0930159350 0x7f09300c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 0x7f09300c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300af1d0 con 0x7f0930158f80 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300af450 con 0x7f0930159350 2023-12-19T16:13:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300af730 con 0x7f093015d730 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41514/0 (socket says 172.21.15.50:41514) 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 -- 172.21.15.50:0/1045997261 learned_addr learned my addr 172.21.15.50:0/1045997261 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092e7fc700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0930159350 0x7f09300c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092f7fe700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 0x7f09300c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 -- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 msgr2=0x7f09300c13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 0x7f09300c13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 -- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0930159350 msgr2=0x7f09300c0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0930159350 0x7f09300c0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 -- 172.21.15.50:0/1045997261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009420 con 0x7f0930158f80 2023-12-19T16:13:53.532 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092f7fe700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 0x7f09300c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f092effd700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f092000c7c0 tx=0x7f0920007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920005b30 con 0x7f0930158f80 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920005c90 con 0x7f0930158f80 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09300afa10 con 0x7f0930158f80 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920007340 con 0x7f0930158f80 2023-12-19T16:13:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0930169bc0 con 0x7f0930158f80 2023-12-19T16:13:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.527+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930071110 con 0x7f0930158f80 2023-12-19T16:13:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.531+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f092001b400 con 0x7f0930158f80 2023-12-19T16:13:53.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.531+0000 7f0916ffd700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 0x7f091c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.535+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f092007c8e0 con 0x7f0930158f80 2023-12-19T16:13:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.535+0000 7f092e7fc700 1 -- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 msgr2=0x7f091c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.535+0000 7f092e7fc700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 0x7f091c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:53.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.539+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f092004c9c0 con 0x7f0930158f80 2023-12-19T16:13:53.599 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:53.595+0000 7f0b80723000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T16:13:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.711+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09300c5c30 con 0x7f0930158f80 2023-12-19T16:13:53.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.711+0000 7f0916ffd700 1 -- 172.21.15.50:0/1045997261 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v214) v1 ==== 100+0+11 (secure 0 0 0) 0x7f092004ffe0 con 0x7f0930158f80 2023-12-19T16:13:53.717 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 64 2023-12-19T16:13:53.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 msgr2=0x7f091c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:53.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 0x7f091c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 msgr2=0x7f09300aec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f092000c7c0 tx=0x7f0920007e00 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 shutdown_connections 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f091c060700 0x7f091c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0930159350 0x7f09300c0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f093015d730 0x7f09300c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 --2- 172.21.15.50:0/1045997261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0930158f80 0x7f09300aec90 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 >> 172.21.15.50:0/1045997261 conn(0x7f09300cc6c0 msgr2=0x7f0930157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 shutdown_connections 2023-12-19T16:13:53.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.715+0000 7f093582e700 1 -- 172.21.15.50:0/1045997261 wait complete. 2023-12-19T16:13:53.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:53.776 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:53.771+0000 7f0b80723000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T16:13:53.844 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:53.833+0000 7f14ecef3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:53.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 Processor -- start 2023-12-19T16:13:53.907 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 -- start start 2023-12-19T16:13:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74ec069f50 0x7f74ec0673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ec0abd00 con 0x7f74ec0b1080 2023-12-19T16:13:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ec0abb10 con 0x7f74ec069f50 2023-12-19T16:13:53.908 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f13cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0abfe0 con 0x7f74ec0b0cb0 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74ebfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f0bcb700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74ec069f50 0x7f74ec0673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f0bcb700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74ec069f50 0x7f74ec0673d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:57350/0 (socket says 172.21.15.50:57350) 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74f0bcb700 1 -- 172.21.15.50:0/3546007992 learned_addr learned my addr 172.21.15.50:0/3546007992 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74ebfff700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:40180/0 (socket says 172.21.15.50:40180) 2023-12-19T16:13:53.909 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.903+0000 7f74eb7fe700 1 --1- 172.21.15.50:0/3546007992 >> v1:172.21.15.125:6789/0 conn(0x7f74ec0b1080 0x7f74ec069840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:39178/0 (socket says 172.21.15.50:39178) 2023-12-19T16:13:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f0bcb700 1 -- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 msgr2=0x7f74ec06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f0bcb700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f0bcb700 1 -- 172.21.15.50:0/3546007992 >> v1:172.21.15.125:6789/0 conn(0x7f74ec0b1080 legacy=0x7f74ec069840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f0bcb700 1 -- 172.21.15.50:0/3546007992 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec0ac2c0 con 0x7f74ec069f50 2023-12-19T16:13:53.910 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74ebfff700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f0bcb700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec069f50 0x7f74ec0673d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009f20 tx=0x7f74e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c441083082008951 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74eaffd700 1 -- 172.21.15.50:0/3546007992 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e400e070 con 0x7f74ec069f50 2023-12-19T16:13:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74eaffd700 1 -- 172.21.15.50:0/3546007992 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e400b9d0 con 0x7f74ec069f50 2023-12-19T16:13:53.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74eaffd700 1 -- 172.21.15.50:0/3546007992 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e40092e0 con 0x7f74ec069f50 2023-12-19T16:13:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 -- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec069f50 msgr2=0x7f74ec0673d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.912 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec069f50 0x7f74ec0673d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009f20 tx=0x7f74e4009840 comp rx=0 tx=0).stop 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 -- 172.21.15.50:0/3546007992 shutdown_connections 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec069f50 0x7f74ec0673d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 --2- 172.21.15.50:0/3546007992 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec0b0cb0 0x7f74ec06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 -- 172.21.15.50:0/3546007992 >> 172.21.15.50:0/3546007992 conn(0x7f74ec0cc530 msgr2=0x7f74ec0c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 -- 172.21.15.50:0/3546007992 shutdown_connections 2023-12-19T16:13:53.913 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.907+0000 7f74f13cc700 1 -- 172.21.15.50:0/3546007992 wait complete. 2023-12-19T16:13:53.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.911+0000 7f74f13cc700 1 Processor -- start 2023-12-19T16:13:53.916 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.911+0000 7f74f13cc700 1 -- start start 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec069f50 0x7f74ec157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 0x7f74ec0c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec164e00 con 0x7f74ec0b1080 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0c14c0 con 0x7f74ec0b0cb0 2023-12-19T16:13:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0c35e0 con 0x7f74ec069f50 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41524/0 (socket says 172.21.15.50:41524) 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 -- 172.21.15.50:0/2746420111 learned_addr learned my addr 172.21.15.50:0/2746420111 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74eb7fe700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 0x7f74ec0c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74ebfff700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec069f50 0x7f74ec157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec069f50 msgr2=0x7f74ec157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec069f50 0x7f74ec157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 msgr2=0x7f74ec0c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 0x7f74ec0c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 -- 172.21.15.50:0/2746420111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc009420 con 0x7f74ec0b1080 2023-12-19T16:13:53.920 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74eb7fe700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 0x7f74ec0c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f0bcb700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f74e400b610 tx=0x7f74e4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e40070a0 con 0x7f74ec0b1080 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e4007b30 con 0x7f74ec0b1080 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e4009ae0 con 0x7f74ec0b1080 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e401d420 con 0x7f74ec0b1080 2023-12-19T16:13:53.921 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.915+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ec0c3ba0 con 0x7f74ec0b1080 2023-12-19T16:13:53.922 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.919+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ec164f70 con 0x7f74ec0b1080 2023-12-19T16:13:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.919+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f74e4003050 con 0x7f74ec0b1080 2023-12-19T16:13:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.923+0000 7f74d3fff700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 0x7f74d8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.923+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f74e407eba0 con 0x7f74ec0b1080 2023-12-19T16:13:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.923+0000 7f74ebfff700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 msgr2=0x7f74d8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:53.926 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.923+0000 7f74ebfff700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 0x7f74d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:53.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:53.927+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f74e404ea50 con 0x7f74ec0b1080 2023-12-19T16:13:54.073 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.061+0000 7f14ecef3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:54.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.127+0000 7f74ebfff700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 msgr2=0x7f74d8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:54.130 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.127+0000 7f74ebfff700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 0x7f74d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:54.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.135+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74ec0c3f40 con 0x7f74ec0b1080 2023-12-19T16:13:54.142 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74d3fff700 1 -- 172.21.15.50:0/2746420111 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f74e4052070 con 0x7f74ec0b1080 2023-12-19T16:13:54.143 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:54.143 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 msgr2=0x7f74d8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 0x7f74d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 msgr2=0x7f74ec0c2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f74e400b610 tx=0x7f74e4018000 comp rx=0 tx=0).stop 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 shutdown_connections 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f74d8060700 0x7f74d8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74ec0b0cb0 0x7f74ec0c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74ec0b1080 0x7f74ec0c2fb0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 --2- 172.21.15.50:0/2746420111 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f74ec069f50 0x7f74ec157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.139+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 >> 172.21.15.50:0/2746420111 conn(0x7f74ec0cc530 msgr2=0x7f74ec0aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:54.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.143+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 shutdown_connections 2023-12-19T16:13:54.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.143+0000 7f74f13cc700 1 -- 172.21.15.50:0/2746420111 wait complete. 2023-12-19T16:13:54.148 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:54.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:54.171 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.161+0000 7f14ecef3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:54.182 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.179+0000 7f0b80723000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T16:13:54.244 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.233+0000 7f14ecef3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:54.318 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.309+0000 7f14ecef3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.331+0000 7f041d82e700 1 Processor -- start 2023-12-19T16:13:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 -- start start 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04181547d0 0x7f041815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 0x7f0418157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f041806bf30 0x7f041806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0418158420 con 0x7f041806bf30 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041815b990 con 0x7f0418154ba0 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f041d82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041806c890 con 0x7f04181547d0 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f041806bf30 0x7f041806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f041806bf30 0x7f041806c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41540/0 (socket says 172.21.15.50:41540) 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 -- 172.21.15.50:0/1880503650 learned_addr learned my addr 172.21.15.50:0/1880503650 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04167fc700 1 --2- 172.21.15.50:0/1880503650 >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 0x7f0418157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 -- 172.21.15.50:0/1880503650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04181547d0 msgr2=0x7f041815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:54.341 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 --2- 172.21.15.50:0/1880503650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04181547d0 0x7f041815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 -- 172.21.15.50:0/1880503650 >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 msgr2=0x7f0418157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 --2- 172.21.15.50:0/1880503650 >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 0x7f0418157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 -- 172.21.15.50:0/1880503650 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041815b0f0 con 0x7f041806bf30 2023-12-19T16:13:54.342 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04167fc700 1 --2- 172.21.15.50:0/1880503650 >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 0x7f0418157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f04177fe700 1 --2- 172.21.15.50:0/1880503650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f041806bf30 0x7f041806c350 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f04180cb8a0 tx=0x7f0410009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2341037c75945ef3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.335+0000 7f0415ffb700 1 -- 172.21.15.50:0/1880503650 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041000e070 con 0x7f041806bf30 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f0415ffb700 1 -- 172.21.15.50:0/1880503650 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000ba50 con 0x7f041806bf30 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f0415ffb700 1 -- 172.21.15.50:0/1880503650 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04100092e0 con 0x7f041806bf30 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 -- 172.21.15.50:0/1880503650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f041806bf30 msgr2=0x7f041806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 --2- 172.21.15.50:0/1880503650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f041806bf30 0x7f041806c350 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f04180cb8a0 tx=0x7f0410009840 comp rx=0 tx=0).stop 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 -- 172.21.15.50:0/1880503650 shutdown_connections 2023-12-19T16:13:54.343 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 --2- 172.21.15.50:0/1880503650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f041806bf30 0x7f041806c350 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 --2- 172.21.15.50:0/1880503650 >> v2:172.21.15.125:3300/0 conn(0x7f0418154ba0 0x7f0418157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 --2- 172.21.15.50:0/1880503650 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f04181547d0 0x7f041815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 -- 172.21.15.50:0/1880503650 >> 172.21.15.50:0/1880503650 conn(0x7f04180cde20 msgr2=0x7f04180ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 -- 172.21.15.50:0/1880503650 shutdown_connections 2023-12-19T16:13:54.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 -- 172.21.15.50:0/1880503650 wait complete. 2023-12-19T16:13:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.339+0000 7f041d82e700 1 Processor -- start 2023-12-19T16:13:54.345 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 -- start start 2023-12-19T16:13:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 0x7f04180c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04181547d0 0x7f04180b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04180b1570 con 0x7f04181547d0 2023-12-19T16:13:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0418155700 con 0x7f0418154ba0 2023-12-19T16:13:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f041d82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04180b0e70 con 0x7f041806bf30 2023-12-19T16:13:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:57386/0 (socket says 172.21.15.50:57386) 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f0416ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 0x7f04180c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04167fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04181547d0 0x7f04180b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 -- 172.21.15.50:0/3196457044 learned_addr learned my addr 172.21.15.50:0/3196457044 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 -- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 msgr2=0x7f04180c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 0x7f04180c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 -- 172.21.15.50:0/3196457044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04181547d0 msgr2=0x7f04180b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04181547d0 0x7f04180b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 -- 172.21.15.50:0/3196457044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410009ae0 con 0x7f0418154ba0 2023-12-19T16:13:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f0416ffd700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 0x7f04180c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.343+0000 7f04177fe700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0410009fd0 tx=0x7f0410018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04100032f0 con 0x7f0418154ba0 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410008ce0 con 0x7f0418154ba0 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04180b17c0 con 0x7f0418154ba0 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410026690 con 0x7f0418154ba0 2023-12-19T16:13:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181550b0 con 0x7f0418154ba0 2023-12-19T16:13:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04181558b0 con 0x7f0418154ba0 2023-12-19T16:13:54.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.347+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f0410003040 con 0x7f0418154ba0 2023-12-19T16:13:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.351+0000 7f03feffd700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 0x7f0404062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.351+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f041007e890 con 0x7f0418154ba0 2023-12-19T16:13:54.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.351+0000 7f0416ffd700 1 -- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 msgr2=0x7f0404062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.351+0000 7f0416ffd700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 0x7f0404062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:54.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.355+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f041004e740 con 0x7f0418154ba0 2023-12-19T16:13:54.429 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.423+0000 7f0b80723000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T16:13:54.532 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.527+0000 7f0b80723000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T16:13:54.535 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.531+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0418071110 con 0x7f0418154ba0 2023-12-19T16:13:54.537 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.531+0000 7f03feffd700 1 -- 172.21.15.50:0/3196457044 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+16245 (secure 0 0 0) 0x7f0410051d60 con 0x7f0418154ba0 2023-12-19T16:13:54.538 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:54.538 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":214,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:37.811943+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/651582691":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2769533890":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/878125710":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2905251978":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/3734826841":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2299774917":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6833/44063":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6832/44063":"2023-12-20T16:13:37.811876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:54.541 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.529+0000 7f14ecef3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 msgr2=0x7f0404062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 0x7f0404062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 msgr2=0x7f0418155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0410009fd0 tx=0x7f0410018000 comp rx=0 tx=0).stop 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 shutdown_connections 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0404060730 0x7f0404062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0418154ba0 0x7f0418155b90 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04181547d0 0x7f04180b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 --2- 172.21.15.50:0/3196457044 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f041806bf30 0x7f04180c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.535+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 >> 172.21.15.50:0/3196457044 conn(0x7f04180cde20 msgr2=0x7f0418158650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.539+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 shutdown_connections 2023-12-19T16:13:54.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.539+0000 7f041d82e700 1 -- 172.21.15.50:0/3196457044 wait complete. 2023-12-19T16:13:54.565 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T16:13:54.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:54.614 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.605+0000 7f14ecef3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:54.615 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.611+0000 7f0b80723000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T16:13:54.700 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.689+0000 7f14ecef3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:54.708 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.703+0000 7f0b80723000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T16:13:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.723+0000 7f97e7e2e700 1 Processor -- start 2023-12-19T16:13:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.723+0000 7f97e7e2e700 1 -- start start 2023-12-19T16:13:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e7e2e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f97e00c7080 0x7f97e00bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e7e2e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e0067e10 con 0x7f97e00682a0 2023-12-19T16:13:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e7e2e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e0068740 con 0x7f97e00bec60 2023-12-19T16:13:54.731 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e7e2e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e0067200 con 0x7f97e00c7080 2023-12-19T16:13:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e63cb700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f97e00682a0 0x7f97e006a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52722/0 (socket says 172.21.15.50:52722) 2023-12-19T16:13:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e63cb700 1 -- 172.21.15.50:0/3374944538 learned_addr learned my addr 172.21.15.50:0/3374944538 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:54.732 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e53c9700 1 --2- 172.21.15.50:0/3374944538 >> v2:172.21.15.125:3300/0 conn(0x7f97e00c7080 0x7f97e00bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e4bc8700 1 -- 172.21.15.50:0/3374944538 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397394683 0 0) 0x7f97e0067e10 con 0x7f97e00682a0 2023-12-19T16:13:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e4bc8700 1 -- 172.21.15.50:0/3374944538 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97cc003570 con 0x7f97e00682a0 2023-12-19T16:13:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e53c9700 1 -- 172.21.15.50:0/3374944538 >> v1:172.21.15.125:6789/0 conn(0x7f97e00bec60 legacy=0x7f97e00c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e53c9700 1 -- 172.21.15.50:0/3374944538 >> v1:172.21.15.50:6789/0 conn(0x7f97e00682a0 legacy=0x7f97e006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.733 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e53c9700 1 -- 172.21.15.50:0/3374944538 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00674e0 con 0x7f97e00c7080 2023-12-19T16:13:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.727+0000 7f97e53c9700 1 --2- 172.21.15.50:0/3374944538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00c7080 0x7f97e00bf030 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f97d00098e0 tx=0x7f97d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a12baf67f1d25a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e4bc8700 1 -- 172.21.15.50:0/3374944538 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d000b640 con 0x7f97e00c7080 2023-12-19T16:13:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e4bc8700 1 -- 172.21.15.50:0/3374944538 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d000b7a0 con 0x7f97e00c7080 2023-12-19T16:13:54.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e4bc8700 1 -- 172.21.15.50:0/3374944538 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d000bb40 con 0x7f97e00c7080 2023-12-19T16:13:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 -- 172.21.15.50:0/3374944538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00c7080 msgr2=0x7f97e00bf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/3374944538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00c7080 0x7f97e00bf030 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f97d00098e0 tx=0x7f97d00091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 -- 172.21.15.50:0/3374944538 shutdown_connections 2023-12-19T16:13:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/3374944538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00c7080 0x7f97e00bf030 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.735 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 -- 172.21.15.50:0/3374944538 >> 172.21.15.50:0/3374944538 conn(0x7f97e00cc690 msgr2=0x7f97e00c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 -- 172.21.15.50:0/3374944538 shutdown_connections 2023-12-19T16:13:54.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 -- 172.21.15.50:0/3374944538 wait complete. 2023-12-19T16:13:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.731+0000 7f97e7e2e700 1 Processor -- start 2023-12-19T16:13:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 -- start start 2023-12-19T16:13:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 0x7f97e0162830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00bec60 0x7f97e00c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e016d9a0 con 0x7f97e00c7080 2023-12-19T16:13:54.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c5030 con 0x7f97e00bec60 2023-12-19T16:13:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e00c5310 con 0x7f97e00682a0 2023-12-19T16:13:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41550/0 (socket says 172.21.15.50:41550) 2023-12-19T16:13:54.740 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 -- 172.21.15.50:0/1008897144 learned_addr learned my addr 172.21.15.50:0/1008897144 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e53c9700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00bec60 0x7f97e00c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e5bca700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 0x7f97e0162830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 msgr2=0x7f97e0162830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 0x7f97e0162830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00bec60 msgr2=0x7f97e00c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00bec60 0x7f97e00c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 -- 172.21.15.50:0/1008897144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0009420 con 0x7f97e00c7080 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e5bca700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 0x7f97e0162830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:54.741 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e63cb700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f97d8001460 tx=0x7f97d8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d800a8a0 con 0x7f97e00c7080 2023-12-19T16:13:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d800aa00 con 0x7f97e00c7080 2023-12-19T16:13:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e00c55f0 con 0x7f97e00c7080 2023-12-19T16:13:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.735+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d800ad80 con 0x7f97e00c7080 2023-12-19T16:13:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.739+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e015b3b0 con 0x7f97e00c7080 2023-12-19T16:13:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.739+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e016db10 con 0x7f97e00c7080 2023-12-19T16:13:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.739+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f97d8007300 con 0x7f97e00c7080 2023-12-19T16:13:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.743+0000 7f97d5ffb700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 0x7f97c4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.743+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f97d807d1e0 con 0x7f97e00c7080 2023-12-19T16:13:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.743+0000 7f97e5bca700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 msgr2=0x7f97c4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.743+0000 7f97e5bca700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 0x7f97c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.751+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f97e016db10 con 0x7f97e00c7080 2023-12-19T16:13:54.889 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.877+0000 7f14ecef3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:54.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.947+0000 7f97e5bca700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 msgr2=0x7f97c4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:54.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.947+0000 7f97e5bca700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 0x7f97c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:54.953 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:54.947+0000 7f0b80723000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T16:13:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.959+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97e016dae0 con 0x7f97e00c7080 2023-12-19T16:13:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.959+0000 7f97d5ffb700 1 -- 172.21.15.50:0/1008897144 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f97d804d220 con 0x7f97e00c7080 2023-12-19T16:13:54.964 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:54.964 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:54.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 msgr2=0x7f97c4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 0x7f97c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 msgr2=0x7f97e0069b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f97d8001460 tx=0x7f97d8001a40 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 shutdown_connections 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f97c4060700 0x7f97c4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97e00bec60 0x7f97e00c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97e00c7080 0x7f97e0069b10 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 --2- 172.21.15.50:0/1008897144 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f97e00682a0 0x7f97e0162830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 >> 172.21.15.50:0/1008897144 conn(0x7f97e00cc690 msgr2=0x7f97e015d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 shutdown_connections 2023-12-19T16:13:54.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:54.963+0000 7f97e7e2e700 1 -- 172.21.15.50:0/1008897144 wait complete. 2023-12-19T16:13:54.970 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:54.971 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:54.961+0000 7f14ecef3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:54.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:13:55.044 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:55.039+0000 7f0b80723000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T16:13:55.145 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:55.139+0000 7f0b80723000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T16:13:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 Processor -- start 2023-12-19T16:13:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 -- start start 2023-12-19T16:13:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44c4142b00 0x7f44c40cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44c41594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c4067fc0 con 0x7f44c4142730 2023-12-19T16:13:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c40673b0 con 0x7f44c4142b00 2023-12-19T16:13:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4067690 con 0x7f44c40cf500 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c259c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f44c4142730 0x7f44c4068450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52724/0 (socket says 172.21.15.50:52724) 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c259c700 1 -- 172.21.15.50:0/3503007393 learned_addr learned my addr 172.21.15.50:0/3503007393 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44c41594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c1d9b700 1 --2- 172.21.15.50:0/3503007393 >> v2:172.21.15.125:3300/0 conn(0x7f44c4142b00 0x7f44c40cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397273587 0 0) 0x7f44c4067fc0 con 0x7f44c4142730 2023-12-19T16:13:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.151+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ac003570 con 0x7f44c4142730 2023-12-19T16:13:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031829879 0 0) 0x7f44ac003570 con 0x7f44c4142730 2023-12-19T16:13:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 msgr2=0x7f44c41594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44c41594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 >> v2:172.21.15.125:3300/0 conn(0x7f44c4142b00 msgr2=0x7f44c40cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:55.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 --2- 172.21.15.50:0/3503007393 >> v2:172.21.15.125:3300/0 conn(0x7f44c4142b00 0x7f44c40cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4067970 con 0x7f44c4142730 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f44b40015f0 con 0x7f44c4142730 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 >> v1:172.21.15.50:6789/0 conn(0x7f44c4142730 legacy=0x7f44c4068450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44ac003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac003fe0 0x7f44ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ac004d50 0x7f44ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40673b0 con 0x7f44ac004d50 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4067690 con 0x7f44ac003fe0 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac00ba90 con 0x7f44c40cf500 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ac004d50 0x7f44ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c259c700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44ac003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c1d9b700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac003fe0 0x7f44ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 msgr2=0x7f44ac003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:55.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c40cf500 0x7f44ac003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac003fe0 msgr2=0x7f44ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac003fe0 0x7f44ac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3503007393 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac003570 con 0x7f44ac004d50 2023-12-19T16:13:55.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ac004d50 0x7f44ac0051c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00e210 tx=0x7f44bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe3cd6409f5d9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc00ea60 con 0x7f44ac004d50 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc00f040 con 0x7f44ac004d50 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c159a700 1 -- 172.21.15.50:0/3503007393 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc00ee10 con 0x7f44ac004d50 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 -- 172.21.15.50:0/3503007393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ac004d50 msgr2=0x7f44ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 --2- 172.21.15.50:0/3503007393 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44ac004d50 0x7f44ac0051c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00e210 tx=0x7f44bc00b040 comp rx=0 tx=0).stop 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3503007393 reap_dead start 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 -- 172.21.15.50:0/3503007393 shutdown_connections 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 -- 172.21.15.50:0/3503007393 >> 172.21.15.50:0/3503007393 conn(0x7f44c40cfea0 msgr2=0x7f44c40d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 -- 172.21.15.50:0/3503007393 shutdown_connections 2023-12-19T16:13:55.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.155+0000 7f44c359e700 1 -- 172.21.15.50:0/3503007393 wait complete. 2023-12-19T16:13:55.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 Processor -- start 2023-12-19T16:13:55.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 -- start start 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac004d50 0x7f44c4158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 0x7f44c413eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41376b0 con 0x7f44c4137140 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4137990 con 0x7f44ac004d50 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4137c70 con 0x7f44c4136cd0 2023-12-19T16:13:55.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41566/0 (socket says 172.21.15.50:41566) 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3376987660 learned_addr learned my addr 172.21.15.50:0/3376987660 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c259c700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac004d50 0x7f44c4158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c1d9b700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 0x7f44c413eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 msgr2=0x7f44c413eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 0x7f44c413eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac004d50 msgr2=0x7f44c4158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac004d50 0x7f44c4158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c2d9d700 1 -- 172.21.15.50:0/3376987660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4137f50 con 0x7f44c4137140 2023-12-19T16:13:55.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.159+0000 7f44c1d9b700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 0x7f44c413eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44c2d9d700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00afd0 tx=0x7f44bc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc01e640 con 0x7f44c4137140 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc022070 con 0x7f44c4137140 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc01e9d0 con 0x7f44c4137140 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c4138230 con 0x7f44c4137140 2023-12-19T16:13:55.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c4138bb0 con 0x7f44c4137140 2023-12-19T16:13:55.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.163+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c404be40 con 0x7f44c4137140 2023-12-19T16:13:55.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.167+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f44bc003040 con 0x7f44c4137140 2023-12-19T16:13:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.167+0000 7f44b27fc700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 0x7f44a4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.167+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7f44bc024070 con 0x7f44c4137140 2023-12-19T16:13:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.167+0000 7f44c259c700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 msgr2=0x7f44a4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.167+0000 7f44c259c700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 0x7f44a4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:55.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.171+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44bc04f010 con 0x7f44c4137140 2023-12-19T16:13:55.363 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:55.359+0000 7f0b80723000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T16:13:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.371+0000 7f44c259c700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 msgr2=0x7f44a4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.371+0000 7f44c259c700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 0x7f44a4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T16:13:55.393 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.387+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c4067c50 con 0x7f44c4137140 2023-12-19T16:13:55.394 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.391+0000 7f44b27fc700 1 -- 172.21.15.50:0/3376987660 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f44bc018090 con 0x7f44c4137140 2023-12-19T16:13:55.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:55.395 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T16:13:23.400827+0000","modified":"2023-12-19T16:13:26.977563+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10389":{"gid":10389,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6835/4273555224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":4273555224},{"type":"v1","addr":"172.21.15.50:6835","nonce":4273555224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.391+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 msgr2=0x7f44a4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.391+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 0x7f44a4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.391+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 msgr2=0x7f44c413eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.391+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00afd0 tx=0x7f44bc01a000 comp rx=0 tx=0).stop 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 shutdown_connections 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f44a4060700 0x7f44a4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44c4137140 0x7f44c413eff0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f44c4136cd0 0x7f44c413eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 --2- 172.21.15.50:0/3376987660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44ac004d50 0x7f44c4158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 >> 172.21.15.50:0/3376987660 conn(0x7f44c40cfea0 msgr2=0x7f44c40b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 shutdown_connections 2023-12-19T16:13:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.395+0000 7f44c359e700 1 -- 172.21.15.50:0/3376987660 wait complete. 2023-12-19T16:13:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 99 2023-12-19T16:13:55.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:13:55.464 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:55.459+0000 7f0b80723000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T16:13:55.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 Processor -- start 2023-12-19T16:13:55.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 -- start start 2023-12-19T16:13:55.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc0afc30 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc0b1310 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbd0aae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc156fe0 con 0x7ffbcc0cc2f0 2023-12-19T16:13:55.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.587+0000 7ffbc9d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41578/0 (socket says 172.21.15.50:41578) 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 -- 172.21.15.50:0/1252152024 learned_addr learned my addr 172.21.15.50:0/1252152024 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbca59c700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 -- 172.21.15.50:0/1252152024 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 msgr2=0x7ffbcc0af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 -- 172.21.15.50:0/1252152024 >> v1:172.21.15.125:6789/0 conn(0x7ffbcc15cbd0 legacy=0x7ffbcc15d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 -- 172.21.15.50:0/1252152024 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc067060 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc9d9b700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00099b0 tx=0x7ffbc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dfadb7608fe130 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:55.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc959a700 1 -- 172.21.15.50:0/1252152024 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000b860 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc959a700 1 -- 172.21.15.50:0/1252152024 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc000b9c0 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbc959a700 1 -- 172.21.15.50:0/1252152024 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000bd60 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 -- 172.21.15.50:0/1252152024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc0b1dd0 msgr2=0x7ffbcc0b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00099b0 tx=0x7ffbc0009250 comp rx=0 tx=0).stop 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 -- 172.21.15.50:0/1252152024 shutdown_connections 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0b2220 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/1252152024 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 -- 172.21.15.50:0/1252152024 >> 172.21.15.50:0/1252152024 conn(0x7ffbcc0d0a70 msgr2=0x7ffbcc0cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 -- 172.21.15.50:0/1252152024 shutdown_connections 2023-12-19T16:13:55.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.591+0000 7ffbd0aae700 1 -- 172.21.15.50:0/1252152024 wait complete. 2023-12-19T16:13:55.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 Processor -- start 2023-12-19T16:13:55.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 -- start start 2023-12-19T16:13:55.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.600 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 0x7ffbcc0c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc0c3a00 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc15eb00 con 0x7ffbcc0b1dd0 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbd0aae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc0ca160 con 0x7ffbcc0cc2f0 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 0x7ffbcc0c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbc9d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbc9d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3301/0 says I am v2:172.21.15.50:40248/0 (socket says 172.21.15.50:40248) 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbc9d9b700 1 -- 172.21.15.50:0/4194483255 learned_addr learned my addr 172.21.15.50:0/4194483255 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:55.601 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbca59c700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 -- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 msgr2=0x7ffbcc0c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 -- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbcc0b1dd0 msgr2=0x7ffbcc0c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.602 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.595+0000 7ffbcad9d700 1 -- 172.21.15.50:0/4194483255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00094f0 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbcad9d700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 0x7ffbcc0c8f70 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c790 tx=0x7ffbc40015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4012630 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc40127b0 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4012b70 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc15e260 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc15e820 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.603 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc0cf840 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.605 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.599+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 105) v1 ==== 80120+0+0 (secure 0 0 0) 0x7ffbc4003030 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.603+0000 7ffbba7fc700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 0x7ffbac062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:55.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.603+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6481+0+0 (secure 0 0 0) 0x7ffbc407ca80 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.603+0000 7ffbca59c700 1 -- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 msgr2=0x7ffbac062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.125:6832/44063 2023-12-19T16:13:55.607 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.603+0000 7ffbca59c700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 0x7ffbac062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T16:13:55.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.611+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffbcc0cf840 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.676 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.665+0000 7f14ecef3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:55.693 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.687+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 106) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ffbc400edb0 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.691+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 msgr2=0x7ffbac062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T16:13:55.694 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.691+0000 7ffbba7fc700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 0x7ffbac062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.780 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b95e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.780 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b95e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.780 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b95e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b95e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b95e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b65e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b65e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b65e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.781 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b65e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.782 INFO:tasks.ceph.mgr.x.smithi125.stderr:2023-12-19T16:13:55.769+0000 7f14b65e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T16:13:55.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.799+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbcc0b29e0 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.799+0000 7ffbba7fc700 1 -- 172.21.15.50:0/4194483255 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+16683 (secure 0 0 0) 0x7ffbcc0b29e0 con 0x7ffbcc15cbd0 2023-12-19T16:13:55.806 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:13:55.806 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":215,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:55.677621+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T16:13:21.069471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T16:13:22.410710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/44652760":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/194513819":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1837833525":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1534107098":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/4136708096":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/651582691":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2769533890":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/878125710":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2905251978":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/3734826841":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/1362444107":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/3321035538":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/2299774917":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6833/44063":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6832/44063":"2023-12-20T16:13:37.811876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 msgr2=0x7ffbcc0c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 0x7ffbcc0c8f70 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c790 tx=0x7ffbc40015f0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 shutdown_connections 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7ffbac060700 0x7ffbac062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffbcc15cbd0 0x7ffbcc0c8f70 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7ffbcc0cc2f0 0x7ffbcc0c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 --2- 172.21.15.50:0/4194483255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffbcc0b1dd0 0x7ffbcc0c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 >> 172.21.15.50:0/4194483255 conn(0x7ffbcc0d0a70 msgr2=0x7ffbcc0b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 shutdown_connections 2023-12-19T16:13:55.810 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:55.803+0000 7ffbd0aae700 1 -- 172.21.15.50:0/4194483255 wait complete. 2023-12-19T16:13:55.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T16:13:56.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 Processor -- start 2023-12-19T16:13:56.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 -- start start 2023-12-19T16:13:56.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f47100b1450 0x7f47100b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4710069dc0 con 0x7f47101590b0 2023-12-19T16:13:56.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100691b0 con 0x7f4710158ce0 2023-12-19T16:13:56.045 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.039+0000 7f471762e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100b1e10 con 0x7f47100b1450 2023-12-19T16:13:56.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4714bc9700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f47101590b0 0x7f471015d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52730/0 (socket says 172.21.15.50:52730) 2023-12-19T16:13:56.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4714bc9700 1 -- 172.21.15.50:0/2927888119 learned_addr learned my addr 172.21.15.50:0/2927888119 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:56.046 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4715bcb700 1 --2- 172.21.15.50:0/2927888119 >> v2:172.21.15.50:3300/0 conn(0x7f47100b1450 0x7f47100b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697645212 0 0) 0x7f4710069dc0 con 0x7f47101590b0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc003570 con 0x7f47101590b0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962712900 0 0) 0x7f47100691b0 con 0x7f4710158ce0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4710069dc0 con 0x7f4710158ce0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700199766 0 0) 0x7f46fc003570 con 0x7f47101590b0 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4715bcb700 1 -- 172.21.15.50:0/2927888119 >> v1:172.21.15.125:6789/0 conn(0x7f4710158ce0 legacy=0x7f471006a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4715bcb700 1 -- 172.21.15.50:0/2927888119 >> v1:172.21.15.50:6789/0 conn(0x7f47101590b0 legacy=0x7f471015d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.047 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4715bcb700 1 -- 172.21.15.50:0/2927888119 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710069490 con 0x7f47100b1450 2023-12-19T16:13:56.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4715bcb700 1 --2- 172.21.15.50:0/2927888119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47100b1450 0x7f47100b18d0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f4708009e60 tx=0x7f4708009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d08e2f2b0b5bbbe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:56.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470800e070 con 0x7f47100b1450 2023-12-19T16:13:56.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470800b7a0 con 0x7f47100b1450 2023-12-19T16:13:56.048 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f4707fff700 1 -- 172.21.15.50:0/2927888119 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700001470 con 0x7f47100b1450 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 -- 172.21.15.50:0/2927888119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47100b1450 msgr2=0x7f47100b18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 --2- 172.21.15.50:0/2927888119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47100b1450 0x7f47100b18d0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f4708009e60 tx=0x7f4708009780 comp rx=0 tx=0).stop 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 -- 172.21.15.50:0/2927888119 shutdown_connections 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 --2- 172.21.15.50:0/2927888119 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47100b1450 0x7f47100b18d0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 -- 172.21.15.50:0/2927888119 >> 172.21.15.50:0/2927888119 conn(0x7f471006c0f0 msgr2=0x7f471006c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 -- 172.21.15.50:0/2927888119 shutdown_connections 2023-12-19T16:13:56.049 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.043+0000 7f471762e700 1 -- 172.21.15.50:0/2927888119 wait complete. 2023-12-19T16:13:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 Processor -- start 2023-12-19T16:13:56.051 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 -- start start 2023-12-19T16:13:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47100b1450 0x7f47101553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.052 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47101590b0 0x7f47100ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4710159600 con 0x7f4710158ce0 2023-12-19T16:13:56.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47101598e0 con 0x7f47100b1450 2023-12-19T16:13:56.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f471762e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4710155900 con 0x7f47101590b0 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f4714bc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f4714bc9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:41604/0 (socket says 172.21.15.50:41604) 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f4714bc9700 1 -- 172.21.15.50:0/2140599122 learned_addr learned my addr 172.21.15.50:0/2140599122 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f4715bcb700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47101590b0 0x7f47100ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.047+0000 7f47153ca700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47100b1450 0x7f47101553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 -- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47101590b0 msgr2=0x7f47100ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47101590b0 0x7f47100ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 -- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47100b1450 msgr2=0x7f47101553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47100b1450 0x7f47101553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 -- 172.21.15.50:0/2140599122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4708009a20 con 0x7f4710158ce0 2023-12-19T16:13:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f4714bc9700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f470000cba0 tx=0x7f4700001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700007360 con 0x7f4710158ce0 2023-12-19T16:13:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000074e0 con 0x7f4710158ce0 2023-12-19T16:13:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47100cca10 con 0x7f4710158ce0 2023-12-19T16:13:56.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700004830 con 0x7f4710158ce0 2023-12-19T16:13:56.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47100ccf90 con 0x7f4710158ce0 2023-12-19T16:13:56.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.051+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4710071110 con 0x7f4710158ce0 2023-12-19T16:13:56.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.055+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 106) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f4700004ba0 con 0x7f4710158ce0 2023-12-19T16:13:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.055+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6782+0+0 (secure 0 0 0) 0x7f470007bad0 con 0x7f4710158ce0 2023-12-19T16:13:56.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.059+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f470004c0d0 con 0x7f4710158ce0 2023-12-19T16:13:56.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.263+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4710068c60 con 0x7f4710158ce0 2023-12-19T16:13:56.272 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-12-19T16:13:56.267+0000 7f40d6969700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T16:13:56.277 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.271+0000 7f47057fa700 1 -- 172.21.15.50:0/2140599122 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) v1 ==== 161+0+0 (secure 0 0 0) 0x7f470004f6f0 con 0x7f4710158ce0 2023-12-19T16:13:56.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 msgr2=0x7f471015bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.281 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f470000cba0 tx=0x7f4700001b50 comp rx=0 tx=0).stop 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 shutdown_connections 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47101590b0 0x7f47100ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4710158ce0 0x7f471015bce0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 --2- 172.21.15.50:0/2140599122 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47100b1450 0x7f47101553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 >> 172.21.15.50:0/2140599122 conn(0x7f471006c0f0 msgr2=0x7f47100c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.275+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 shutdown_connections 2023-12-19T16:13:56.282 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.279+0000 7f471762e700 1 -- 172.21.15.50:0/2140599122 wait complete. 2023-12-19T16:13:56.283 INFO:teuthology.orchestra.run.smithi050.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T16:13:56.291 INFO:tasks.ceph.mds.c.smithi050.stderr: -3634> 2023-12-19T16:13:28.919+0000 7f03665d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T16:13:56.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T16:13:56.318 INFO:tasks.ceph.mgr.y.smithi050.stderr:2023-12-19T16:13:56.315+0000 7f0b80723000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T16:13:56.344 INFO:tasks.ceph.mds.c.smithi050.stdout:starting mds.c at 2023-12-19T16:13:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.459+0000 7f7c5759e700 1 Processor -- start 2023-12-19T16:13:56.464 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.459+0000 7f7c5759e700 1 -- start start 2023-12-19T16:13:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c5815aff0 0x7f7c5815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c58069780 con 0x7f7c5815b970 2023-12-19T16:13:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c58067960 con 0x7f7c58159d00 2023-12-19T16:13:56.466 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c58067770 con 0x7f7c5815aff0 2023-12-19T16:13:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c5815aff0 0x7f7c5815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7c5815aff0 0x7f7c5815b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51516/0 (socket says 172.21.15.50:51516) 2023-12-19T16:13:56.467 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 -- 172.21.15.50:0/3292836199 learned_addr learned my addr 172.21.15.50:0/3292836199 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:56.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 -- 172.21.15.50:0/3292836199 >> v1:172.21.15.125:6789/0 conn(0x7f7c58159d00 legacy=0x7f7c5815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5559a700 1 -- 172.21.15.50:0/3292836199 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422233222 0 0) 0x7f7c58067960 con 0x7f7c58159d00 2023-12-19T16:13:56.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 -- 172.21.15.50:0/3292836199 >> v1:172.21.15.50:6789/0 conn(0x7f7c5815b970 legacy=0x7f7c58067060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 -- 172.21.15.50:0/3292836199 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58067c40 con 0x7f7c5815aff0 2023-12-19T16:13:56.468 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5659c700 1 --2- 172.21.15.50:0/3292836199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5815aff0 0x7f7c5815b3c0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009f20 tx=0x7f7c48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209d1eaad41cc189 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5559a700 1 -- 172.21.15.50:0/3292836199 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800e070 con 0x7f7c5815aff0 2023-12-19T16:13:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5559a700 1 -- 172.21.15.50:0/3292836199 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4800ba80 con 0x7f7c5815aff0 2023-12-19T16:13:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5559a700 1 -- 172.21.15.50:0/3292836199 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c480092e0 con 0x7f7c5815aff0 2023-12-19T16:13:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 -- 172.21.15.50:0/3292836199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5815aff0 msgr2=0x7f7c5815b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.469 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.463+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3292836199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5815aff0 0x7f7c5815b3c0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009f20 tx=0x7f7c48009840 comp rx=0 tx=0).stop 2023-12-19T16:13:56.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 -- 172.21.15.50:0/3292836199 shutdown_connections 2023-12-19T16:13:56.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3292836199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c5815aff0 0x7f7c5815b3c0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 -- 172.21.15.50:0/3292836199 >> 172.21.15.50:0/3292836199 conn(0x7f7c580cc530 msgr2=0x7f7c580c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:56.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 -- 172.21.15.50:0/3292836199 shutdown_connections 2023-12-19T16:13:56.470 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 -- 172.21.15.50:0/3292836199 wait complete. 2023-12-19T16:13:56.472 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 Processor -- start 2023-12-19T16:13:56.473 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.467+0000 7f7c5759e700 1 -- start start 2023-12-19T16:13:56.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c5815aff0 0x7f7c581543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c5815b970 0x7f7c580c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.474 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c58154910 con 0x7f7c58159d00 2023-12-19T16:13:56.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580c4eb0 con 0x7f7c5815b970 2023-12-19T16:13:56.475 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5816a5c0 con 0x7f7c5815aff0 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c55d9b700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c5815aff0 0x7f7c581543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51524/0 (socket says 172.21.15.50:51524) 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 -- 172.21.15.50:0/3274505243 learned_addr learned my addr 172.21.15.50:0/3274505243 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c56d9d700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c5815b970 0x7f7c580c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 -- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c5815aff0 msgr2=0x7f7c581543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c5815aff0 0x7f7c581543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 -- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c5815b970 msgr2=0x7f7c580c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c5815b970 0x7f7c580c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 -- 172.21.15.50:0/3274505243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48009ae0 con 0x7f7c58159d00 2023-12-19T16:13:56.476 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5659c700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009880 tx=0x7f7c48018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c48007af0 con 0x7f7c58159d00 2023-12-19T16:13:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c48007c50 con 0x7f7c58159d00 2023-12-19T16:13:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c581696d0 con 0x7f7c58159d00 2023-12-19T16:13:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c48026460 con 0x7f7c58159d00 2023-12-19T16:13:56.477 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.471+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c58169c90 con 0x7f7c58159d00 2023-12-19T16:13:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.475+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5804be40 con 0x7f7c58159d00 2023-12-19T16:13:56.480 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.475+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 106) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f7c48026810 con 0x7f7c58159d00 2023-12-19T16:13:56.482 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.479+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f7c4807e620 con 0x7f7c58159d00 2023-12-19T16:13:56.488 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.483+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c4804eb40 con 0x7f7c58159d00 2023-12-19T16:13:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.691+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7c58067f20 con 0x7f7c58159d00 2023-12-19T16:13:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.695+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 107) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f7c4804e5f0 con 0x7f7c58159d00 2023-12-19T16:13:56.702 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.699+0000 7f7c467fc700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 0x7f7c38068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:56.703 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.699+0000 7f7c55d9b700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 0x7f7c38068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:56.706 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:56.703+0000 7f7c55d9b700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 0x7f7c38068930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c580c2910 tx=0x7f7c4c007ec0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:57.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.347+0000 7f7c467fc700 1 -- 172.21.15.50:0/3274505243 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v101) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c48055440 con 0x7f7c58159d00 2023-12-19T16:13:57.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 msgr2=0x7f7c38068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 0x7f7c38068930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7c580c2910 tx=0x7f7c4c007ec0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 msgr2=0x7f7c5816a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009880 tx=0x7f7c48018000 comp rx=0 tx=0).stop 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 shutdown_connections 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7c38066550 0x7f7c38068930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c5815b970 0x7f7c580c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f7c5815aff0 0x7f7c581543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 --2- 172.21.15.50:0/3274505243 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c58159d00 0x7f7c5816a810 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 >> 172.21.15.50:0/3274505243 conn(0x7f7c580cc530 msgr2=0x7f7c58154bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 shutdown_connections 2023-12-19T16:13:57.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.351+0000 7f7c5759e700 1 -- 172.21.15.50:0/3274505243 wait complete. 2023-12-19T16:13:57.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T16:13:57.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.603+0000 7f907ad23700 1 Processor -- start 2023-12-19T16:13:57.606 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.603+0000 7f907ad23700 1 -- start start 2023-12-19T16:13:57.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907ad23700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f907415cee0 0x7f907415d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907ad23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9074158b20 0x7f9074158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907ad23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90740cea80 con 0x7f90740cb3a0 2023-12-19T16:13:57.610 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907ad23700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740b0b40 con 0x7f907415cee0 2023-12-19T16:13:57.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907ad23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90740ced60 con 0x7f9074158b20 2023-12-19T16:13:57.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f907415cee0 0x7f907415d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f907a522700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9074158b20 0x7f9074158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.611 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079520700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f90740cb3a0 0x7f90740ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50768/0 (socket says 172.21.15.50:50768) 2023-12-19T16:13:57.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079520700 1 -- 172.21.15.50:0/1998506771 learned_addr learned my addr 172.21.15.50:0/1998506771 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:57.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9078d1f700 1 -- 172.21.15.50:0/1998506771 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610173432 0 0) 0x7f90740cea80 con 0x7f90740cb3a0 2023-12-19T16:13:57.612 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9078d1f700 1 -- 172.21.15.50:0/1998506771 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9060003570 con 0x7f90740cb3a0 2023-12-19T16:13:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 -- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9074158b20 msgr2=0x7f9074158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 --2- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9074158b20 0x7f9074158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 -- 172.21.15.50:0/1998506771 >> v1:172.21.15.125:6789/0 conn(0x7f90740cb3a0 legacy=0x7f90740ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 -- 172.21.15.50:0/1998506771 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740cf040 con 0x7f907415cee0 2023-12-19T16:13:57.613 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.607+0000 7f9079d21700 1 --2- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907415cee0 0x7f907415d2b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f90700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a395f72486d3d30a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:57.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f9078d1f700 1 -- 172.21.15.50:0/1998506771 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070011630 con 0x7f907415cee0 2023-12-19T16:13:57.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f9078d1f700 1 -- 172.21.15.50:0/1998506771 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070011790 con 0x7f907415cee0 2023-12-19T16:13:57.614 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f9078d1f700 1 -- 172.21.15.50:0/1998506771 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070011b30 con 0x7f907415cee0 2023-12-19T16:13:57.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 -- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907415cee0 msgr2=0x7f907415d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 --2- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907415cee0 0x7f907415d2b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f90700091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 -- 172.21.15.50:0/1998506771 shutdown_connections 2023-12-19T16:13:57.615 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 --2- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f9074158b20 0x7f9074158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 --2- 172.21.15.50:0/1998506771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907415cee0 0x7f907415d2b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 -- 172.21.15.50:0/1998506771 >> 172.21.15.50:0/1998506771 conn(0x7f90740cff20 msgr2=0x7f90740d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:57.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 -- 172.21.15.50:0/1998506771 shutdown_connections 2023-12-19T16:13:57.616 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.611+0000 7f907ad23700 1 -- 172.21.15.50:0/1998506771 wait complete. 2023-12-19T16:13:57.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 Processor -- start 2023-12-19T16:13:57.618 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 -- start start 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9074158b20 0x7f90740b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 0x7f907414d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90741567e0 con 0x7f90740cb3a0 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074155bd0 con 0x7f9074158b20 2023-12-19T16:13:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907ad23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9074155eb0 con 0x7f907415cee0 2023-12-19T16:13:57.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.621 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51526/0 (socket says 172.21.15.50:51526) 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 -- 172.21.15.50:0/2216065637 learned_addr learned my addr 172.21.15.50:0/2216065637 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907a522700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 0x7f907414d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079520700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9074158b20 0x7f90740b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 -- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 msgr2=0x7f907414d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 0x7f907414d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 -- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9074158b20 msgr2=0x7f90740b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9074158b20 0x7f90740b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f9079d21700 1 -- 172.21.15.50:0/2216065637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070009420 con 0x7f90740cb3a0 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.615+0000 7f907a522700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 0x7f907414d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f9079d21700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9074064370 tx=0x7f9070017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070011b10 con 0x7f90740cb3a0 2023-12-19T16:13:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070026450 con 0x7f90740cb3a0 2023-12-19T16:13:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700267d0 con 0x7f90740cb3a0 2023-12-19T16:13:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9074156190 con 0x7f90740cb3a0 2023-12-19T16:13:57.623 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074156c70 con 0x7f90740cb3a0 2023-12-19T16:13:57.625 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.619+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907404be40 con 0x7f90740cb3a0 2023-12-19T16:13:57.626 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.623+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 107) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f9070026bc0 con 0x7f90740cb3a0 2023-12-19T16:13:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.623+0000 7f9069ffb700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 0x7f9058062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:57.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.623+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f907007e2f0 con 0x7f90740cb3a0 2023-12-19T16:13:57.628 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.623+0000 7f9079520700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 0x7f9058062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:57.629 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.623+0000 7f9079520700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 0x7f9058062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f90741574d0 tx=0x7f9064009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:57.636 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.631+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f907004e630 con 0x7f90740cb3a0 2023-12-19T16:13:57.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:57.807+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f90740cf320 con 0x7f90740cb3a0 2023-12-19T16:13:58.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.355+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f907004ea00 con 0x7f90740cb3a0 2023-12-19T16:13:58.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.723+0000 7f9069ffb700 1 -- 172.21.15.50:0/2216065637 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v217) v1 ==== 181+0+0 (secure 0 0 0) 0x7f90700520c0 con 0x7f90740cb3a0 2023-12-19T16:13:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 msgr2=0x7f9058062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 0x7f9058062880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f90741574d0 tx=0x7f9064009040 comp rx=0 tx=0).stop 2023-12-19T16:13:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 msgr2=0x7f9074155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:58.736 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9074064370 tx=0x7f9070017000 comp rx=0 tx=0).stop 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 shutdown_connections 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f90580603d0 0x7f9058062880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9074158b20 0x7f90740b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f907415cee0 0x7f907414d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.731+0000 7f907ad23700 1 --2- 172.21.15.50:0/2216065637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90740cb3a0 0x7f9074155310 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.735+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 >> 172.21.15.50:0/2216065637 conn(0x7f90740cff20 msgr2=0x7f90740d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:58.738 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.735+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 shutdown_connections 2023-12-19T16:13:58.739 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.735+0000 7f907ad23700 1 -- 172.21.15.50:0/2216065637 wait complete. 2023-12-19T16:13:58.741 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_metadata' removed 2023-12-19T16:13:58.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T16:13:58.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.935+0000 7fe4a1785700 1 Processor -- start 2023-12-19T16:13:59.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.935+0000 7fe4a1785700 1 -- start start 2023-12-19T16:13:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe49c0c67f0 0x7fe49c067060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe49c159400 0x7fe49c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0675a0 0x7fe49c0c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c0c7640 con 0x7fe49c159400 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c06a2c0 con 0x7fe49c0c67f0 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c067b20 con 0x7fe49c0675a0 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe49c159400 0x7fe49c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe49c159400 0x7fe49c0c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51528/0 (socket says 172.21.15.50:51528) 2023-12-19T16:13:59.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 -- 172.21.15.50:0/1826875305 learned_addr learned my addr 172.21.15.50:0/1826875305 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49b7fe700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0675a0 0x7fe49c0c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49affd700 1 --2- 172.21.15.50:0/1826875305 >> v2:172.21.15.125:3300/0 conn(0x7fe49c0c67f0 0x7fe49c067060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 -- 172.21.15.50:0/1826875305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0675a0 msgr2=0x7fe49c0c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0675a0 0x7fe49c0c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 -- 172.21.15.50:0/1826875305 >> v2:172.21.15.125:3300/0 conn(0x7fe49c0c67f0 msgr2=0x7fe49c067060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/1826875305 >> v2:172.21.15.125:3300/0 conn(0x7fe49c0c67f0 0x7fe49c067060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 -- 172.21.15.50:0/1826875305 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c067e00 con 0x7fe49c159400 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c159400 0x7fe49c0c6bc0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe49c070770 tx=0x7fe490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2c409b33dbb608 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe499ffb700 1 -- 172.21.15.50:0/1826875305 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49000b860 con 0x7fe49c159400 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe499ffb700 1 -- 172.21.15.50:0/1826875305 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49000b9c0 con 0x7fe49c159400 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe499ffb700 1 -- 172.21.15.50:0/1826875305 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49000bd60 con 0x7fe49c159400 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- 172.21.15.50:0/1826875305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c159400 msgr2=0x7fe49c0c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c159400 0x7fe49c0c6bc0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe49c070770 tx=0x7fe490009250 comp rx=0 tx=0).stop 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- 172.21.15.50:0/1826875305 shutdown_connections 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c159400 0x7fe49c0c6bc0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- 172.21.15.50:0/1826875305 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0675a0 0x7fe49c0c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 --2- 172.21.15.50:0/1826875305 >> v2:172.21.15.125:3300/0 conn(0x7fe49c0c67f0 0x7fe49c067060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- 172.21.15.50:0/1826875305 >> 172.21.15.50:0/1826875305 conn(0x7fe49c0cc7e0 msgr2=0x7fe49c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- 172.21.15.50:0/1826875305 shutdown_connections 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.939+0000 7fe4a1785700 1 -- 172.21.15.50:0/1826875305 wait complete. 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 Processor -- start 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 -- start start 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe49c0c67f0 0x7fe49c0af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 0x7fe49c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0afd10 con 0x7fe49c0675a0 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0c3130 con 0x7fe49c0c67f0 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.943+0000 7fe4a1785700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c157050 con 0x7fe49c0af890 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51536/0 (socket says 172.21.15.50:51536) 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 -- 172.21.15.50:0/4007634181 learned_addr learned my addr 172.21.15.50:0/4007634181 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe49c0c67f0 0x7fe49c0af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49b7fe700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 0x7fe49c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 -- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 msgr2=0x7fe49c156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 0x7fe49c156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 -- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe49c0c67f0 msgr2=0x7fe49c0af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe49c0c67f0 0x7fe49c0af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 -- 172.21.15.50:0/4007634181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900094f0 con 0x7fe49c0675a0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49b7fe700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 0x7fe49c156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe49affd700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00d860 tx=0x7fe48c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0125f0 con 0x7fe49c0675a0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c012750 con 0x7fe49c0675a0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c157330 con 0x7fe49c0675a0 2023-12-19T16:13:59.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c012b10 con 0x7fe49c0675a0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.947+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c0c2dc0 con 0x7fe49c0675a0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.951+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c0708a0 con 0x7fe49c0675a0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.951+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe48c01e460 con 0x7fe49c0675a0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.951+0000 7fe482ffd700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 0x7fe488062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.951+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6430+0+0 (secure 0 0 0) 0x7fe48c07d140 con 0x7fe49c0675a0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.951+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 0x7fe488062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.955+0000 7fe49a7fc700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 0x7fe488062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe49c070770 tx=0x7fe490017000 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:58.959+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe48c04d350 con 0x7fe49c0675a0 2023-12-19T16:13:59.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.139+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe49c15faf0 con 0x7fe49c0675a0 2023-12-19T16:13:59.742 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.739+0000 7fe482ffd700 1 -- 172.21.15.50:0/4007634181 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v218) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe48c050970 con 0x7fe49c0675a0 2023-12-19T16:13:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 msgr2=0x7fe488062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 0x7fe488062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe49c070770 tx=0x7fe490017000 comp rx=0 tx=0).stop 2023-12-19T16:13:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 msgr2=0x7fe49c1506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00d860 tx=0x7fe48c008040 comp rx=0 tx=0).stop 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 shutdown_connections 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe488060670 0x7fe488062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe49c0c67f0 0x7fe49c0af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe49c0af890 0x7fe49c156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 --2- 172.21.15.50:0/4007634181 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe49c0675a0 0x7fe49c1506a0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 >> 172.21.15.50:0/4007634181 conn(0x7fe49c0cc7e0 msgr2=0x7fe49c151350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 shutdown_connections 2023-12-19T16:13:59.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.743+0000 7fe4a1785700 1 -- 172.21.15.50:0/4007634181 wait complete. 2023-12-19T16:13:59.749 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'cephfs_data' removed 2023-12-19T16:13:59.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T16:13:59.930 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.927+0000 7f1dca616700 1 Processor -- start 2023-12-19T16:13:59.931 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.927+0000 7f1dca616700 1 -- start start 2023-12-19T16:13:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dca616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 0x7f1dc415d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dca616700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1dc40b23d0 0x7f1dc4067a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dca616700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40caf70 con 0x7f1dc415bca0 2023-12-19T16:13:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dca616700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc415d550 con 0x7f1dc40b23d0 2023-12-19T16:13:59.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dca616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4068350 con 0x7f1dc415cc40 2023-12-19T16:13:59.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc9e15700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1dc415bca0 0x7f1dc415c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59168/0 (socket says 172.21.15.50:59168) 2023-12-19T16:13:59.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc9e15700 1 -- 172.21.15.50:0/4290651213 learned_addr learned my addr 172.21.15.50:0/4290651213 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:13:59.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/4290651213 >> v2:172.21.15.125:3300/0 conn(0x7f1dc40b23d0 0x7f1dc4067a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc9614700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 0x7f1dc415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dbbfff700 1 -- 172.21.15.50:0/4290651213 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905307647 0 0) 0x7f1dc40caf70 con 0x7f1dc415bca0 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dbbfff700 1 -- 172.21.15.50:0/4290651213 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0003570 con 0x7f1dc415bca0 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 -- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 msgr2=0x7f1dc415d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 0x7f1dc415d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 -- 172.21.15.50:0/4290651213 >> v1:172.21.15.50:6789/0 conn(0x7f1dc415bca0 legacy=0x7f1dc415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 -- 172.21.15.50:0/4290651213 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4068630 con 0x7f1dc40b23d0 2023-12-19T16:13:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc9614700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 0x7f1dc415d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:13:59.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.931+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc40b23d0 0x7f1dc4067a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1db40098e0 tx=0x7f1db40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cfd06271b4bc65f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dbbfff700 1 -- 172.21.15.50:0/4290651213 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011630 con 0x7f1dc40b23d0 2023-12-19T16:13:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dbbfff700 1 -- 172.21.15.50:0/4290651213 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4011790 con 0x7f1dc40b23d0 2023-12-19T16:13:59.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dbbfff700 1 -- 172.21.15.50:0/4290651213 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011b30 con 0x7f1dc40b23d0 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc40b23d0 msgr2=0x7f1dc4067a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc40b23d0 0x7f1dc4067a10 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1db40098e0 tx=0x7f1db40091a0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- 172.21.15.50:0/4290651213 shutdown_connections 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc40b23d0 0x7f1dc4067a10 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 --2- 172.21.15.50:0/4290651213 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc415cc40 0x7f1dc415d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- 172.21.15.50:0/4290651213 >> 172.21.15.50:0/4290651213 conn(0x7f1dc4069b90 msgr2=0x7f1dc40cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:13:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- 172.21.15.50:0/4290651213 shutdown_connections 2023-12-19T16:13:59.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- 172.21.15.50:0/4290651213 wait complete. 2023-12-19T16:13:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 Processor -- start 2023-12-19T16:13:59.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.935+0000 7f1dca616700 1 -- start start 2023-12-19T16:13:59.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc40b23d0 0x7f1dc41567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 0x7f1dc4148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4156310 con 0x7f1dc415bca0 2023-12-19T16:13:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc41554a0 con 0x7f1dc415cc40 2023-12-19T16:13:59.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4155700 con 0x7f1dc40b23d0 2023-12-19T16:13:59.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51538/0 (socket says 172.21.15.50:51538) 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 -- 172.21.15.50:0/291392022 learned_addr learned my addr 172.21.15.50:0/291392022 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc9614700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc40b23d0 0x7f1dc41567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc9e15700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 0x7f1dc4148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 -- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc40b23d0 msgr2=0x7f1dc41567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc40b23d0 0x7f1dc41567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 -- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 msgr2=0x7f1dc4148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 0x7f1dc4148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 -- 172.21.15.50:0/291392022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc0009420 con 0x7f1dc415bca0 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc9e15700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 0x7f1dc4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dc8e13700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1db4013000 tx=0x7f1db4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011b10 con 0x7f1dc415bca0 2023-12-19T16:13:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db40253f0 con 0x7f1dc415bca0 2023-12-19T16:13:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db40257b0 con 0x7f1dc415bca0 2023-12-19T16:13:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db4009420 con 0x7f1dc415bca0 2023-12-19T16:13:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.939+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4155cc0 con 0x7f1dc415bca0 2023-12-19T16:13:59.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.943+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc41564c0 con 0x7f1dc415bca0 2023-12-19T16:13:59.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.943+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1db4025ba0 con 0x7f1dc415bca0 2023-12-19T16:13:59.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.943+0000 7f1db97fa700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 0x7f1da8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:13:59.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.943+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f1db407e170 con 0x7f1dc415bca0 2023-12-19T16:13:59.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.947+0000 7f1dc9614700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 0x7f1da8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:13:59.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.947+0000 7f1dc9614700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 0x7f1da8062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009eb0 tx=0x7f1dc00091a0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:13:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:13:59.951+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1dc41564c0 con 0x7f1dc415bca0 2023-12-19T16:13:59.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:13:59.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:14:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.131+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1dc404be40 con 0x7f1dc415bca0 2023-12-19T16:14:00.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.143+0000 7f1db97fa700 1 -- 172.21.15.50:0/291392022 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1db404e520 con 0x7f1dc415bca0 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 msgr2=0x7f1da8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 0x7f1da8062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009eb0 tx=0x7f1dc00091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 msgr2=0x7f1dc40cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1db4013000 tx=0x7f1db4017000 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 shutdown_connections 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f1da80606a0 0x7f1da8062b50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1dc415cc40 0x7f1dc4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1dc415bca0 0x7f1dc40cb930 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 --2- 172.21.15.50:0/291392022 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f1dc40b23d0 0x7f1dc41567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 >> 172.21.15.50:0/291392022 conn(0x7f1dc4069b90 msgr2=0x7f1dc40cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:00.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 shutdown_connections 2023-12-19T16:14:00.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.147+0000 7f1dca616700 1 -- 172.21.15.50:0/291392022 wait complete. 2023-12-19T16:14:00.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T16:14:00.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.339+0000 7f2445d85700 1 Processor -- start 2023-12-19T16:14:00.344 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.339+0000 7f2445d85700 1 -- start start 2023-12-19T16:14:00.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f2445d85700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f244015d4d0 0x7f24400c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f2445d85700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400c6cb0 con 0x7f244015d100 2023-12-19T16:14:00.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f2445d85700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400c6f90 con 0x7f24400c6750 2023-12-19T16:14:00.346 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f2445d85700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24400c2af0 con 0x7f244015d4d0 2023-12-19T16:14:00.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243f7fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f244015d100 0x7f24400c23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59182/0 (socket says 172.21.15.50:59182) 2023-12-19T16:14:00.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243f7fe700 1 -- 172.21.15.50:0/2754346113 learned_addr learned my addr 172.21.15.50:0/2754346113 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:00.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243effd700 1 --2- 172.21.15.50:0/2754346113 >> v2:172.21.15.125:3300/0 conn(0x7f244015d4d0 0x7f24400c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366831508 0 0) 0x7f24400c6cb0 con 0x7f244015d100 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2428003570 con 0x7f244015d100 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272785177 0 0) 0x7f24400c6f90 con 0x7f24400c6750 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24400c6cb0 con 0x7f24400c6750 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243effd700 1 -- 172.21.15.50:0/2754346113 >> v1:172.21.15.125:6789/0 conn(0x7f24400c6750 legacy=0x7f24400c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243effd700 1 -- 172.21.15.50:0/2754346113 >> v1:172.21.15.50:6789/0 conn(0x7f244015d100 legacy=0x7f24400c23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243effd700 1 -- 172.21.15.50:0/2754346113 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400c7270 con 0x7f244015d4d0 2023-12-19T16:14:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243effd700 1 --2- 172.21.15.50:0/2754346113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d4d0 0x7f24400c7d50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f24340098e0 tx=0x7f24340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63aa54d678a7f7fb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2434011630 con 0x7f244015d4d0 2023-12-19T16:14:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2434011790 con 0x7f244015d4d0 2023-12-19T16:14:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.343+0000 7f243e7fc700 1 -- 172.21.15.50:0/2754346113 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2434011b30 con 0x7f244015d4d0 2023-12-19T16:14:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- 172.21.15.50:0/2754346113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d4d0 msgr2=0x7f24400c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 --2- 172.21.15.50:0/2754346113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d4d0 0x7f24400c7d50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f24340098e0 tx=0x7f24340091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- 172.21.15.50:0/2754346113 shutdown_connections 2023-12-19T16:14:00.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 --2- 172.21.15.50:0/2754346113 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d4d0 0x7f24400c7d50 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- 172.21.15.50:0/2754346113 >> 172.21.15.50:0/2754346113 conn(0x7f24400c8630 msgr2=0x7f24400c8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:00.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- 172.21.15.50:0/2754346113 shutdown_connections 2023-12-19T16:14:00.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- 172.21.15.50:0/2754346113 wait complete. 2023-12-19T16:14:00.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 Processor -- start 2023-12-19T16:14:00.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.347+0000 7f2445d85700 1 -- start start 2023-12-19T16:14:00.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f24400c6750 0x7f244014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d100 0x7f24400c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440155180 con 0x7f244015d4d0 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244014e4b0 con 0x7f244015d100 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2440155c90 con 0x7f24400c6750 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51548/0 (socket says 172.21.15.50:51548) 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 -- 172.21.15.50:0/177679656 learned_addr learned my addr 172.21.15.50:0/177679656 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243f7fe700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f24400c6750 0x7f244014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243effd700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d100 0x7f24400c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 -- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f24400c6750 msgr2=0x7f244014df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f24400c6750 0x7f244014df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 -- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d100 msgr2=0x7f24400c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d100 0x7f24400c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 -- 172.21.15.50:0/177679656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434009420 con 0x7f244015d4d0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f243ffff700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f243800c940 tx=0x7f24380027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438012630 con 0x7f244015d4d0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24380127b0 con 0x7f244015d4d0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438012b70 con 0x7f244015d4d0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2440155360 con 0x7f244015d4d0 2023-12-19T16:14:00.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.351+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2440155920 con 0x7f244015d4d0 2023-12-19T16:14:00.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.355+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2440071110 con 0x7f244015d4d0 2023-12-19T16:14:00.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.355+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f243801b400 con 0x7f244015d4d0 2023-12-19T16:14:00.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.355+0000 7f24277fe700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 0x7f242c062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.355+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f243807cdc0 con 0x7f244015d4d0 2023-12-19T16:14:00.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.355+0000 7f243f7fe700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 0x7f242c062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.359+0000 7f243f7fe700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 0x7f242c062bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2430007fd0 tx=0x7f2430009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.363+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f243804d0f0 con 0x7f244015d4d0 2023-12-19T16:14:00.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.539+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f24400c7550 con 0x7f244015d4d0 2023-12-19T16:14:00.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.555+0000 7f24277fe700 1 -- 172.21.15.50:0/177679656 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v74) v1 ==== 115+0+0 (secure 0 0 0) 0x7f24380160b0 con 0x7f244015d4d0 2023-12-19T16:14:00.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 msgr2=0x7f242c062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 0x7f242c062bd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2430007fd0 tx=0x7f2430009040 comp rx=0 tx=0).stop 2023-12-19T16:14:00.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 msgr2=0x7f24400c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f243800c940 tx=0x7f24380027a0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 shutdown_connections 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f242c060720 0x7f242c062bd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f244015d100 0x7f24400c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f244015d4d0 0x7f24400c9a00 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 --2- 172.21.15.50:0/177679656 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f24400c6750 0x7f244014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 >> 172.21.15.50:0/177679656 conn(0x7f24400c8630 msgr2=0x7f244015b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:00.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 shutdown_connections 2023-12-19T16:14:00.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.563+0000 7f2445d85700 1 -- 172.21.15.50:0/177679656 wait complete. 2023-12-19T16:14:00.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T16:14:00.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.759+0000 7fa51ee8e700 1 Processor -- start 2023-12-19T16:14:00.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.759+0000 7fa51ee8e700 1 -- start start 2023-12-19T16:14:00.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ee8e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa51806c860 0x7fa518156f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ee8e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51815a410 con 0x7fa5181561d0 2023-12-19T16:14:00.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ee8e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51815a6f0 con 0x7fa518155e00 2023-12-19T16:14:00.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ee8e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa518156700 con 0x7fa51806c860 2023-12-19T16:14:00.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa51806c860 0x7fa518156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa51806c860 0x7fa518156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51560/0 (socket says 172.21.15.50:51560) 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 -- 172.21.15.50:0/1768525003 learned_addr learned my addr 172.21.15.50:0/1768525003 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 -- 172.21.15.50:0/1768525003 >> v1:172.21.15.125:6789/0 conn(0x7fa518155e00 legacy=0x7fa5181599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 -- 172.21.15.50:0/1768525003 >> v1:172.21.15.50:6789/0 conn(0x7fa5181561d0 legacy=0x7fa51806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ce8a700 1 -- 172.21.15.50:0/1768525003 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822624418 0 0) 0x7fa51815a410 con 0x7fa5181561d0 2023-12-19T16:14:00.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 -- 172.21.15.50:0/1768525003 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5181569e0 con 0x7fa51806c860 2023-12-19T16:14:00.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51e68d700 1 --2- 172.21.15.50:0/1768525003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa51806c860 0x7fa518156f00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f20 tx=0x7fa510009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8820bdb71a8dd313 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ce8a700 1 -- 172.21.15.50:0/1768525003 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51000e070 con 0x7fa51806c860 2023-12-19T16:14:00.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ce8a700 1 -- 172.21.15.50:0/1768525003 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51000ba80 con 0x7fa51806c860 2023-12-19T16:14:00.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.763+0000 7fa51ce8a700 1 -- 172.21.15.50:0/1768525003 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100092e0 con 0x7fa51806c860 2023-12-19T16:14:00.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- 172.21.15.50:0/1768525003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa51806c860 msgr2=0x7fa518156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/1768525003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa51806c860 0x7fa518156f00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f20 tx=0x7fa510009840 comp rx=0 tx=0).stop 2023-12-19T16:14:00.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- 172.21.15.50:0/1768525003 shutdown_connections 2023-12-19T16:14:00.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/1768525003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa51806c860 0x7fa518156f00 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- 172.21.15.50:0/1768525003 >> 172.21.15.50:0/1768525003 conn(0x7fa5180ccb70 msgr2=0x7fa5180ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:00.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- 172.21.15.50:0/1768525003 shutdown_connections 2023-12-19T16:14:00.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- 172.21.15.50:0/1768525003 wait complete. 2023-12-19T16:14:00.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 Processor -- start 2023-12-19T16:14:00.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- start start 2023-12-19T16:14:00.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 0x7fa5180c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5181561d0 0x7fa51814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180678f0 con 0x7fa518155e00 2023-12-19T16:14:00.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518068710 con 0x7fa5181561d0 2023-12-19T16:14:00.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.767+0000 7fa51ee8e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa518067b00 con 0x7fa51806c860 2023-12-19T16:14:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51570/0 (socket says 172.21.15.50:51570) 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 -- 172.21.15.50:0/2508594131 learned_addr learned my addr 172.21.15.50:0/2508594131 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51e68d700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5181561d0 0x7fa51814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51de8c700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 0x7fa5180c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 -- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 msgr2=0x7fa5180c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 0x7fa5180c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 -- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5181561d0 msgr2=0x7fa51814f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5181561d0 0x7fa51814f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 -- 172.21.15.50:0/2508594131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510009ae0 con 0x7fa518155e00 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51de8c700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 0x7fa5180c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51d68b700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fa50800c5b0 tx=0x7fa508007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508011590 con 0x7fa518155e00 2023-12-19T16:14:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa508011710 con 0x7fa518155e00 2023-12-19T16:14:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518067de0 con 0x7fa518155e00 2023-12-19T16:14:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508011ad0 con 0x7fa518155e00 2023-12-19T16:14:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5180683a0 con 0x7fa518155e00 2023-12-19T16:14:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.771+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51804be40 con 0x7fa518155e00 2023-12-19T16:14:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.775+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa50801b3b0 con 0x7fa518155e00 2023-12-19T16:14:00.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.775+0000 7fa50dffb700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 0x7fa4fc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:00.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.775+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fa50807bf20 con 0x7fa518155e00 2023-12-19T16:14:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.775+0000 7fa51de8c700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 0x7fa4fc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.779+0000 7fa51de8c700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 0x7fa4fc062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa514007fd0 tx=0x7fa514009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:00.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.783+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa50804c250 con 0x7fa518155e00 2023-12-19T16:14:00.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.955+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa518068850 con 0x7fa518155e00 2023-12-19T16:14:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.963+0000 7fa50dffb700 1 -- 172.21.15.50:0/2508594131 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa5080170b0 con 0x7fa518155e00 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 msgr2=0x7fa4fc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 0x7fa4fc062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa514007fd0 tx=0x7fa514009040 comp rx=0 tx=0).stop 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 msgr2=0x7fa518068ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fa50800c5b0 tx=0x7fa508007f60 comp rx=0 tx=0).stop 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 shutdown_connections 2023-12-19T16:14:00.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fa4fc060670 0x7fa4fc062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5181561d0 0x7fa51814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa518155e00 0x7fa518068ba0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 --2- 172.21.15.50:0/2508594131 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fa51806c860 0x7fa5180c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 >> 172.21.15.50:0/2508594131 conn(0x7fa5180ccb70 msgr2=0x7fa518151710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 shutdown_connections 2023-12-19T16:14:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:00.967+0000 7fa51ee8e700 1 -- 172.21.15.50:0/2508594131 wait complete. 2023-12-19T16:14:00.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-12-19T16:14:01.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.155+0000 7fafd40f0700 1 Processor -- start 2023-12-19T16:14:01.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 -- start start 2023-12-19T16:14:01.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0bec60 0x7fafcc0bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafcc0c7080 0x7fafcc06a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.164 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc067e10 con 0x7fafcc0682a0 2023-12-19T16:14:01.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc067200 con 0x7fafcc0c7080 2023-12-19T16:14:01.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd40f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0674e0 con 0x7fafcc0bec60 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd168b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fafcc0682a0 0x7fafcc0c6970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59196/0 (socket says 172.21.15.50:59196) 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd168b700 1 -- 172.21.15.50:0/3898740500 learned_addr learned my addr 172.21.15.50:0/3898740500 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd268d700 1 --2- 172.21.15.50:0/3898740500 >> v2:172.21.15.125:3300/0 conn(0x7fafcc0c7080 0x7fafcc06a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.159+0000 7fafd1e8c700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0bec60 0x7fafcc0bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826374960 0 0) 0x7fafcc067e10 con 0x7fafcc0682a0 2023-12-19T16:14:01.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb8003570 con 0x7fafcc0682a0 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984887176 0 0) 0x7fafb8003570 con 0x7fafcc0682a0 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd268d700 1 -- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0bec60 msgr2=0x7fafcc0bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd268d700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0bec60 0x7fafcc0bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd268d700 1 -- 172.21.15.50:0/3898740500 >> v1:172.21.15.50:6789/0 conn(0x7fafcc0682a0 legacy=0x7fafcc0c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd268d700 1 -- 172.21.15.50:0/3898740500 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc0677c0 con 0x7fafcc0c7080 2023-12-19T16:14:01.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd268d700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0c7080 0x7fafcc06a860 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009e60 tx=0x7fafc4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb78a8f76f6d459 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:01.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc400e070 con 0x7fafcc0c7080 2023-12-19T16:14:01.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc400b7a0 con 0x7fafcc0c7080 2023-12-19T16:14:01.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd0e8a700 1 -- 172.21.15.50:0/3898740500 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc001470 con 0x7fafcc0c7080 2023-12-19T16:14:01.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 -- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0c7080 msgr2=0x7fafcc06a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0c7080 0x7fafcc06a860 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009e60 tx=0x7fafc4009780 comp rx=0 tx=0).stop 2023-12-19T16:14:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 -- 172.21.15.50:0/3898740500 shutdown_connections 2023-12-19T16:14:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0c7080 0x7fafcc06a860 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 --2- 172.21.15.50:0/3898740500 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0bec60 0x7fafcc0bf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.163+0000 7fafd40f0700 1 -- 172.21.15.50:0/3898740500 >> 172.21.15.50:0/3898740500 conn(0x7fafcc0cc690 msgr2=0x7fafcc0c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:01.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.167+0000 7fafd40f0700 1 -- 172.21.15.50:0/3898740500 shutdown_connections 2023-12-19T16:14:01.170 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.167+0000 7fafd40f0700 1 -- 172.21.15.50:0/3898740500 wait complete. 2023-12-19T16:14:01.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.167+0000 7fafd40f0700 1 Processor -- start 2023-12-19T16:14:01.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.167+0000 7fafd40f0700 1 -- start start 2023-12-19T16:14:01.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0682a0 0x7fafcc1605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0bec60 0x7fafcc16df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0c5bd0 con 0x7fafcc0c7080 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc160130 con 0x7fafcc0bec60 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc16e440 con 0x7fafcc0682a0 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51574/0 (socket says 172.21.15.50:51574) 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 -- 172.21.15.50:0/1315295872 learned_addr learned my addr 172.21.15.50:0/1315295872 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd168b700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0bec60 0x7fafcc16df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd1e8c700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0682a0 0x7fafcc1605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 -- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0682a0 msgr2=0x7fafcc1605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0682a0 0x7fafcc1605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 -- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0bec60 msgr2=0x7fafcc16df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0bec60 0x7fafcc16df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 -- 172.21.15.50:0/1315295872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4009a20 con 0x7fafcc0c7080 2023-12-19T16:14:01.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd268d700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fafc40097c0 tx=0x7fafc4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4012ad0 con 0x7fafcc0c7080 2023-12-19T16:14:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc4012c50 con 0x7fafcc0c7080 2023-12-19T16:14:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc15f520 con 0x7fafcc0c7080 2023-12-19T16:14:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4027470 con 0x7fafcc0c7080 2023-12-19T16:14:01.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.171+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc15fae0 con 0x7fafcc0c7080 2023-12-19T16:14:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.175+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc1602c0 con 0x7fafcc0c7080 2023-12-19T16:14:01.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.175+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fafc4024070 con 0x7fafcc0c7080 2023-12-19T16:14:01.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.179+0000 7fafc1ffb700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 0x7fafb0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:01.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.179+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fafc407eef0 con 0x7fafcc0c7080 2023-12-19T16:14:01.183 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.179+0000 7fafd1e8c700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 0x7fafb0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:01.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.179+0000 7fafd1e8c700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 0x7fafb0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009da0 tx=0x7fafc8009120 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:01.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.183+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fafc404f220 con 0x7fafcc0c7080 2023-12-19T16:14:01.426 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.423+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fafcc071170 con 0x7fafcc0c7080 2023-12-19T16:14:01.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.742+0000 7fafc1ffb700 1 -- 172.21.15.50:0/1315295872 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3091) v1 ==== 155+0+0 (secure 0 0 0) 0x7fafc4052840 con 0x7fafcc0c7080 2023-12-19T16:14:01.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 msgr2=0x7fafb0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 0x7fafb0062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009da0 tx=0x7fafc8009120 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 msgr2=0x7fafcc15d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fafc40097c0 tx=0x7fafc4018000 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 shutdown_connections 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fafb00606a0 0x7fafb0062b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafcc0bec60 0x7fafcc16df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fafcc0c7080 0x7fafcc15d020 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 --2- 172.21.15.50:0/1315295872 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fafcc0682a0 0x7fafcc1605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 >> 172.21.15.50:0/1315295872 conn(0x7fafcc0cc690 msgr2=0x7fafcc0ac8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 shutdown_connections 2023-12-19T16:14:01.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:01.746+0000 7fafd40f0700 1 -- 172.21.15.50:0/1315295872 wait complete. 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner: 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner:Ran 11 tests in 5646.989s 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner: 2023-12-19T16:14:01.774 INFO:tasks.cephfs_test_runner:OK 2023-12-19T16:14:01.774 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-12-19T16:14:01.784 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-19T16:14:01.794 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-19T16:14:01.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:01.795 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-12-19T16:14:01.814 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-19T16:14:01.815 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-12-19T16:14:01.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:01.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T16:14:01.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:01.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-19T16:14:01.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:01.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2023-12-19T16:14:02.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:02.211 DEBUG:teuthology.orchestra.run.smithi050:> ip netns list 2023-12-19T16:14:02.231 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T16:14:02.231 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T16:14:02.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:02.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T16:14:02.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:02.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T16:14:02.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T16:14:02.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo ip link delete ceph-brx 2023-12-19T16:14:02.617 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-19T16:14:02.626 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-19T16:14:02.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:14:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 Processor -- start 2023-12-19T16:14:02.791 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 -- start start 2023-12-19T16:14:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4758067060 0x7f4758067430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f47580c8e60 0x7f47580ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475806b1a0 con 0x7f47580c92b0 2023-12-19T16:14:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580ae350 con 0x7f47580c8e60 2023-12-19T16:14:02.793 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.786+0000 7f475e665700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4758067970 con 0x7f4758067060 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f47580c8e60 0x7f47580ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f47580c8e60 0x7f47580ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51580/0 (socket says 172.21.15.50:51580) 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 -- 172.21.15.50:0/4138636027 learned_addr learned my addr 172.21.15.50:0/4138636027 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f4757fff700 1 --2- 172.21.15.50:0/4138636027 >> v2:172.21.15.125:3300/0 conn(0x7f4758067060 0x7f4758067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 -- 172.21.15.50:0/4138636027 >> v2:172.21.15.125:3300/0 conn(0x7f4758067060 msgr2=0x7f4758067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:02.794 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 --2- 172.21.15.50:0/4138636027 >> v2:172.21.15.125:3300/0 conn(0x7f4758067060 0x7f4758067430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 -- 172.21.15.50:0/4138636027 >> v1:172.21.15.50:6789/0 conn(0x7f47580c92b0 legacy=0x7f47580abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 -- 172.21.15.50:0/4138636027 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475806a900 con 0x7f47580c8e60 2023-12-19T16:14:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f47577fe700 1 --2- 172.21.15.50:0/4138636027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47580c8e60 0x7f47580ab920 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f4758069500 tx=0x7f474c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f873a89b00ca06b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f4756ffd700 1 -- 172.21.15.50:0/4138636027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00b820 con 0x7f47580c8e60 2023-12-19T16:14:02.795 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f4756ffd700 1 -- 172.21.15.50:0/4138636027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c00b980 con 0x7f47580c8e60 2023-12-19T16:14:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f4756ffd700 1 -- 172.21.15.50:0/4138636027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00bd20 con 0x7f47580c8e60 2023-12-19T16:14:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 -- 172.21.15.50:0/4138636027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47580c8e60 msgr2=0x7f47580ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 --2- 172.21.15.50:0/4138636027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47580c8e60 0x7f47580ab920 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f4758069500 tx=0x7f474c009250 comp rx=0 tx=0).stop 2023-12-19T16:14:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 -- 172.21.15.50:0/4138636027 shutdown_connections 2023-12-19T16:14:02.796 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 --2- 172.21.15.50:0/4138636027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47580c8e60 0x7f47580ab920 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 --2- 172.21.15.50:0/4138636027 >> v2:172.21.15.125:3300/0 conn(0x7f4758067060 0x7f4758067430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 -- 172.21.15.50:0/4138636027 >> 172.21.15.50:0/4138636027 conn(0x7f47580cc690 msgr2=0x7f47580c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 -- 172.21.15.50:0/4138636027 shutdown_connections 2023-12-19T16:14:02.797 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.790+0000 7f475e665700 1 -- 172.21.15.50:0/4138636027 wait complete. 2023-12-19T16:14:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 Processor -- start 2023-12-19T16:14:02.798 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 -- start start 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47580c92b0 0x7f4758153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47580c1e20 0x7f4758154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758154750 con 0x7f4758067060 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580c23c0 con 0x7f47580c92b0 2023-12-19T16:14:02.800 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f475e665700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4758158cd0 con 0x7f47580c1e20 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51588/0 (socket says 172.21.15.50:51588) 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 -- 172.21.15.50:0/64199289 learned_addr learned my addr 172.21.15.50:0/64199289 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f47577fe700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47580c92b0 0x7f4758153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 -- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47580c1e20 msgr2=0x7f4758154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47580c1e20 0x7f4758154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 -- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47580c92b0 msgr2=0x7f4758153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47580c92b0 0x7f4758153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 -- 172.21.15.50:0/64199289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0094f0 con 0x7f4758067060 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.794+0000 7f4757fff700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f474800d810 tx=0x7f4748008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480125f0 con 0x7f4758067060 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748012750 con 0x7f4758067060 2023-12-19T16:14:02.802 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748012b10 con 0x7f4758067060 2023-12-19T16:14:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4758158fb0 con 0x7f4758067060 2023-12-19T16:14:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4758159570 con 0x7f4758067060 2023-12-19T16:14:02.803 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4758071110 con 0x7f4758067060 2023-12-19T16:14:02.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.798+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f474801e460 con 0x7f4758067060 2023-12-19T16:14:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.802+0000 7f473ffff700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 0x7f4744062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:02.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.802+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f474807d1d0 con 0x7f4758067060 2023-12-19T16:14:02.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.802+0000 7f47577fe700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 0x7f4744062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:02.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.802+0000 7f47577fe700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 0x7f4744062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4758069500 tx=0x7f474c016000 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.806+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f474804d500 con 0x7f4758067060 2023-12-19T16:14:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.978+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47580bf2a0 con 0x7f4758067060 2023-12-19T16:14:02.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.978+0000 7f473ffff700 1 -- 172.21.15.50:0/64199289 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+13593 (secure 0 0 0) 0x7f4748050b20 con 0x7f4758067060 2023-12-19T16:14:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:02.986 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":218,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:59.722350+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6833/4273555224":"2023-12-20T16:13:56.268975+0000","172.21.15.125:0/44652760":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/194513819":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1837833525":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1534107098":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/4136708096":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/651582691":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2769533890":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/878125710":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2905251978":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/3734826841":"2023-12-20T16:13:37.811876+0000","172.21.15.50:6835/4273555224":"2023-12-20T16:13:56.268975+0000","172.21.15.125:0/1362444107":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/3321035538":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/2299774917":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6833/44063":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6832/44063":"2023-12-20T16:13:37.811876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 msgr2=0x7f4744062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 0x7f4744062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4758069500 tx=0x7f474c016000 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 msgr2=0x7f47580aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f474800d810 tx=0x7f4748008040 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 shutdown_connections 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4744060670 0x7f4744062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47580c92b0 0x7f4758153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f47580c1e20 0x7f4758154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 --2- 172.21.15.50:0/64199289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4758067060 0x7f47580aeed0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 >> 172.21.15.50:0/64199289 conn(0x7f47580cc690 msgr2=0x7f47580ab5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 shutdown_connections 2023-12-19T16:14:02.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:02.982+0000 7f475e665700 1 -- 172.21.15.50:0/64199289 wait complete. 2023-12-19T16:14:03.011 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-19T16:14:03.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:14:03.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.166+0000 7f0abb4cd700 1 Processor -- start 2023-12-19T16:14:03.173 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.166+0000 7f0abb4cd700 1 -- start start 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0abb4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab40bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0abb4cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab4067e10 con 0x7f0ab40682a0 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0abb4cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab4068740 con 0x7f0ab40bec60 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0abb4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab4067200 con 0x7f0ab40c7080 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab9a6a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0ab40682a0 0x7f0ab406a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59214/0 (socket says 172.21.15.50:59214) 2023-12-19T16:14:03.175 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab9a6a700 1 -- 172.21.15.50:0/3742321921 learned_addr learned my addr 172.21.15.50:0/3742321921 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab8a68700 1 --2- 172.21.15.50:0/3742321921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab40bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3742321921 >> v1:172.21.15.125:6789/0 conn(0x7f0ab40bec60 legacy=0x7f0ab40c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.176 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3742321921 >> v1:172.21.15.50:6789/0 conn(0x7f0ab40682a0 legacy=0x7f0ab406a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0aabfff700 1 -- 172.21.15.50:0/3742321921 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627859939 0 0) 0x7f0ab4068740 con 0x7f0ab40bec60 2023-12-19T16:14:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3742321921 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab40674e0 con 0x7f0ab40c7080 2023-12-19T16:14:03.177 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.170+0000 7f0ab8a68700 1 --2- 172.21.15.50:0/3742321921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab40bf030 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009910 tx=0x7f0aa40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77207a3c0df1cbc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0aabfff700 1 -- 172.21.15.50:0/3742321921 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400b760 con 0x7f0ab40c7080 2023-12-19T16:14:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0aabfff700 1 -- 172.21.15.50:0/3742321921 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400b8c0 con 0x7f0ab40c7080 2023-12-19T16:14:03.178 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0aabfff700 1 -- 172.21.15.50:0/3742321921 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400bc60 con 0x7f0ab40c7080 2023-12-19T16:14:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3742321921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 msgr2=0x7f0ab40bf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3742321921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab40bf030 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009910 tx=0x7f0aa40091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.179 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3742321921 shutdown_connections 2023-12-19T16:14:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3742321921 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab40bf030 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3742321921 >> 172.21.15.50:0/3742321921 conn(0x7f0ab40cc690 msgr2=0x7f0ab40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3742321921 shutdown_connections 2023-12-19T16:14:03.180 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.174+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3742321921 wait complete. 2023-12-19T16:14:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 Processor -- start 2023-12-19T16:14:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 -- start start 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40682a0 0x7f0ab40aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40b04b0 con 0x7f0ab40bec60 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40aef50 con 0x7f0ab40682a0 2023-12-19T16:14:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0abb4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40afa20 con 0x7f0ab40c7080 2023-12-19T16:14:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51590/0 (socket says 172.21.15.50:51590) 2023-12-19T16:14:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3356693337 learned_addr learned my addr 172.21.15.50:0/3356693337 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab9a6a700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.185 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab9269700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40682a0 0x7f0ab40aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 msgr2=0x7f0ab415e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab415e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40682a0 msgr2=0x7f0ab40aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40682a0 0x7f0ab40aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 -- 172.21.15.50:0/3356693337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4009420 con 0x7f0ab40bec60 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab9a6a700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.178+0000 7f0ab8a68700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40093f0 tx=0x7f0aa4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa401c480 con 0x7f0ab40bec60 2023-12-19T16:14:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa401c600 con 0x7f0ab40bec60 2023-12-19T16:14:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa401c9c0 con 0x7f0ab40bec60 2023-12-19T16:14:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab415a900 con 0x7f0ab40bec60 2023-12-19T16:14:03.187 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab40af460 con 0x7f0ab40bec60 2023-12-19T16:14:03.188 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab4071110 con 0x7f0ab40bec60 2023-12-19T16:14:03.189 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.182+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0aa401a450 con 0x7f0ab40bec60 2023-12-19T16:14:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.186+0000 7f0aa97fa700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 0x7f0a98062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.190 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.186+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f0aa407ec70 con 0x7f0ab40bec60 2023-12-19T16:14:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.186+0000 7f0ab9269700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 0x7f0a98062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.186+0000 7f0ab9269700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 0x7f0a98062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0001230 tx=0x7f0ab0009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.198 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.194+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0aa404efa0 con 0x7f0ab40bec60 2023-12-19T16:14:03.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.362+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ab40677c0 con 0x7f0a98060670 2023-12-19T16:14:03.376 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.370+0000 7f0aa97fa700 1 -- 172.21.15.50:0/3356693337 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47893 (secure 0 0 0) 0x7f0ab40677c0 con 0x7f0a98060670 2023-12-19T16:14:03.377 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 msgr2=0x7f0a98062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 0x7f0a98062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0001230 tx=0x7f0ab0009040 comp rx=0 tx=0).stop 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 msgr2=0x7f0ab415a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40093f0 tx=0x7f0aa4016000 comp rx=0 tx=0).stop 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 shutdown_connections 2023-12-19T16:14:03.381 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a98060670 0x7f0a98062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0ab40c7080 0x7f0ab415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0ab40bec60 0x7f0ab415a3c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 --2- 172.21.15.50:0/3356693337 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab40682a0 0x7f0ab40aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 >> 172.21.15.50:0/3356693337 conn(0x7f0ab40cc690 msgr2=0x7f0ab415d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 shutdown_connections 2023-12-19T16:14:03.382 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.374+0000 7f0abb4cd700 1 -- 172.21.15.50:0/3356693337 wait complete. 2023-12-19T16:14:03.384 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:14:03.403 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-19T16:14:01.703379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":2470948,"kb_used_data":23012,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752503772,"statfs":{"total":773094113280,"available":770563862528,"internally_reserved":0,"allocated":23564288,"data_stored":12933306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667618,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.989596"},"pg_stats":[{"pgid":"1.0","version":"215'370","reported_seq":1219,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-12-19T16:13:55.796474+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:55.796474+0000","last_peered":"2023-12-19T16:13:55.796474+0000","last_clean":"2023-12-19T16:13:55.796474+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:55.796474+0000","last_undegraded":"2023-12-19T16:13:55.796474+0000","last_fullsized":"2023-12-19T16:13:55.796474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575977,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304596,"kb_used_data":3212,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067244,"statfs":{"total":96636764160,"available":96324857856,"internally_reserved":0,"allocated":3289088,"data_stored":1970598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.173},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":1.0009999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.0900000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":0.754}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.2290000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.401}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.357}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.272},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.476}]}]},{"osd":6,"up_from":13,"seq":55834575977,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312528,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059312,"statfs":{"total":96636764160,"available":96316735488,"internally_reserved":0,"allocated":3022848,"data_stored":1701272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.163,"15min":1.179},"min":{"1min":0.877,"5min":0.69999999999999996,"15min":0.60099999999999998},"max":{"1min":1.488,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":1.274},{"interface":"front","average":{"1min":1.292,"5min":1.224,"15min":1.2390000000000001},"min":{"1min":0.89700000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.597,"5min":1.74,"15min":1.7909999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.76500000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5509999999999999,"5min":1.643,"15min":1.7010000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2609999999999999,"15min":1.274},"min":{"1min":0.89400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.8080000000000001,"15min":1.827},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.202,"15min":1.23},"min":{"1min":0.878,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.504,"5min":1.651,"15min":1.8080000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.28,"15min":1.288},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.599,"5min":1.732,"15min":1.732},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.042,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5840000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.296},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.3280000000000001},"min":{"1min":0.80200000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.337}]},{"osd":4,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.212,"15min":1.2050000000000001},"min":{"1min":1.0289999999999999,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.4710000000000001,"5min":1.708,"15min":1.774},"last":1.034},{"interface":"front","average":{"1min":1.375,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":1.02,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.8140000000000001,"5min":1.966,"15min":1.966},"last":0.96699999999999997}]},{"osd":5,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.296,"15min":1.3129999999999999},"min":{"1min":0.88200000000000001,"5min":0.83299999999999996,"15min":0.68000000000000005},"max":{"1min":1.7509999999999999,"5min":1.788,"15min":1.923},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.3620000000000001,"15min":1.3720000000000001},"min":{"1min":1.1519999999999999,"5min":0.94099999999999995,"15min":0.68899999999999995},"max":{"1min":1.593,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.202}]},{"osd":7,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4219999999999999,"15min":1.4350000000000001},"min":{"1min":1.101,"5min":0.92900000000000005,"15min":0.73799999999999999},"max":{"1min":1.706,"5min":1.8720000000000001,"15min":2.169},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4430000000000001,"15min":1.46},"min":{"1min":0.97499999999999998,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.855,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.4510000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575977,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308136,"kb_used_data":2656,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063704,"statfs":{"total":96636764160,"available":96321232896,"internally_reserved":0,"allocated":2719744,"data_stored":1390291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.101,"15min":1.1060000000000001},"min":{"1min":0.622,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.337,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.746,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.5600000000000001,"5min":1.6619999999999999,"15min":1.907},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2689999999999999,"15min":1.3100000000000001},"min":{"1min":0.86099999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.6559999999999999,"5min":1.7350000000000001,"15min":1.7490000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2509999999999999,"15min":1.276},"min":{"1min":0.88400000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.552,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.77800000000000002}]},{"osd":2,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2769999999999999,"15min":1.2749999999999999},"min":{"1min":0.92900000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.641,"5min":1.9259999999999999,"15min":2.0249999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.22,"15min":1.2210000000000001},"min":{"1min":0.92200000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.6639999999999999,"5min":1.7769999999999999,"15min":1.831},"last":0.82199999999999995}]},{"osd":3,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2809999999999999,"15min":1.306},"min":{"1min":1.0049999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.575,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":1.024,"5min":0.89300000000000002,"15min":0.85799999999999998},"max":{"1min":1.585,"5min":1.734,"15min":2.0449999999999999},"last":0.754}]},{"osd":4,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.1419999999999999,"15min":1.1439999999999999},"min":{"1min":0.73999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.8},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.2969999999999999,"15min":1.28},"min":{"1min":0.79700000000000004,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7869999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.173}]},{"osd":6,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.385,"15min":1.3779999999999999},"min":{"1min":1.208,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.8680000000000001,"15min":1.968},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.381,"5min":1.3999999999999999,"15min":1.401},"min":{"1min":0.93999999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.724,"5min":1.7929999999999999,"15min":1.794},"last":1.254}]},{"osd":7,"last update":"Tue Dec 19 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4419999999999999,"15min":1.4179999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.71,"5min":2.395,"15min":2.395},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.411,"15min":1.4179999999999999},"min":{"1min":1.018,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8240000000000001,"5min":1.9770000000000001,"15min":1.99},"last":1.095}]}]},{"osd":4,"up_from":13,"seq":55834575977,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312232,"kb_used_data":2656,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059608,"statfs":{"total":96636764160,"available":96317038592,"internally_reserved":0,"allocated":2719744,"data_stored":1391099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.046},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.3120000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.1779999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":0.90700000000000003}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.877}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":1.127}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.4670000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.087}]}]},{"osd":3,"up_from":13,"seq":55834575976,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300252,"kb_used_data":2964,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071588,"statfs":{"total":96636764160,"available":96329306112,"internally_reserved":0,"allocated":3035136,"data_stored":1708622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":0.90600000000000003},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2609999999999999,"5min":1.633,"15min":1.7729999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96899999999999997,"15min":1.0009999999999999},"min":{"1min":0.66000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.9370000000000001},"last":1.014}]},{"osd":1,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.96299999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.53900000000000003},"max":{"1min":1.716,"5min":1.716,"15min":2.0350000000000001},"last":1.23},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.147,"15min":1.119},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.379,"5min":1.724,"15min":2.2269999999999999},"last":0.95999999999999996}]},{"osd":2,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1679999999999999},"min":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.726,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":1.321},{"interface":"front","average":{"1min":1.196,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.71599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.643,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":1.204}]},{"osd":4,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.81599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.631,"5min":1.718,"15min":2.5099999999999998},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.876,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.788,"15min":2.3450000000000002},"last":1.427}]},{"osd":5,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3600000000000001,"15min":1.355},"min":{"1min":0.93300000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.6679999999999999,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":0.85299999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0179999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":1.8999999999999999}]},{"osd":6,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.04,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.9830000000000001,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.744},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.448,"15min":1.458},"min":{"1min":0.879,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.798,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":1.48}]},{"osd":7,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5449999999999999,"15min":1.55},"min":{"1min":0.87,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8720000000000001,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.8720000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312248,"kb_used_data":2672,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059592,"statfs":{"total":96636764160,"available":96317022208,"internally_reserved":0,"allocated":2736128,"data_stored":1400063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94599999999999995,"15min":0.96099999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.1160000000000001,"15min":1.1439999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.3520000000000001,"5min":1.629,"15min":1.796},"last":0.90200000000000002}]},{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4059999999999999,"5min":2.327,"15min":2.327},"last":1.091},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.143,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.399,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.351}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0069999999999999,"15min":1.044},"min":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.5489999999999999,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.105,"15min":1.137},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3979999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":1.0620000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.569,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.89300000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.786,"5min":2.218,"15min":2.218},"last":1.9099999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.395,"15min":1.4099999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6910000000000001,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.377,"15min":1.4039999999999999},"min":{"1min":0.91600000000000004,"5min":0.879,"15min":0.879},"max":{"1min":1.5940000000000001,"5min":2.0049999999999999,"15min":2.4340000000000002},"last":1.55}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.0099999999999998,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8500000000000001,"5min":2.609,"15min":2.609},"last":1.488}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5349999999999999,"15min":1.5509999999999999},"min":{"1min":1.054,"5min":1.054,"15min":0.89600000000000002},"max":{"1min":1.833,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.5269999999999999,"15min":1.5409999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.9319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.5980000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575977,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316348,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055492,"statfs":{"total":96636764160,"available":96312823808,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.874,"15min":0.90000000000000002},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.4690000000000001,"15min":1.6579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.97399999999999998,"15min":1.032},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.454,"5min":1.51,"15min":1.9119999999999999},"last":1.3340000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0449999999999999,"15min":1.0669999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.643,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1,"5min":1.093,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.641,"5min":1.823,"15min":1.823},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.242,"15min":1.2789999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.665,"5min":1.778,"15min":1.8180000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0960000000000001,"15min":1.105},"min":{"1min":0.65600000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.264}]},{"osd":4,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3120000000000001,"15min":1.3260000000000001},"min":{"1min":1,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.302},{"interface":"front","average":{"1min":1.395,"5min":1.391,"15min":1.423},"min":{"1min":1.014,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":2.0110000000000001},"last":1.204}]},{"osd":5,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3009999999999999,"15min":1.337},"min":{"1min":1.0049999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.597,"5min":1.833,"15min":1.897},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.395,"15min":1.4370000000000001},"min":{"1min":1.038,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.919,"5min":2.0579999999999998,"15min":2.165},"last":1.5600000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3460000000000001,"15min":1.397},"min":{"1min":0.97299999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5820000000000001,"5min":1.9590000000000001,"15min":2.1139999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4770000000000001,"15min":1.512},"min":{"1min":1.1180000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.4710000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.494,"15min":1.53},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4510000000000001,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":1.752}]}]},{"osd":0,"up_from":13,"seq":55834575976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304608,"kb_used_data":3224,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067232,"statfs":{"total":96636764160,"available":96324845568,"internally_reserved":0,"allocated":3301376,"data_stored":1970287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.439},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.6419999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0069999999999999,"15min":1.008},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4259999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.141,"15min":1.147},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.57099999999999995},"max":{"1min":1.605,"5min":1.748,"15min":1.8400000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.143,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.54300000000000004},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8500000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.02,"15min":1.0409999999999999},"min":{"1min":0.60499999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.448,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.448},{"interface":"front","average":{"1min":1.121,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.55300000000000005},"max":{"1min":1.7290000000000001,"5min":1.974,"15min":1.974},"last":0.73399999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.873,"5min":0.873,"15min":0.69499999999999995},"max":{"1min":1.8120000000000001,"5min":1.885,"15min":2.0209999999999999},"last":0.873},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.339,"15min":1.331},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71999999999999997},"max":{"1min":1.762,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.0720000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.071,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.6870000000000001,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.331},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4419999999999999,"15min":1.456},"min":{"1min":1.056,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.712,"5min":1.919,"15min":1.9810000000000001},"last":1.351}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.738,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.23},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.93899999999999995},"max":{"1min":1.714,"5min":2.0009999999999999,"15min":2.1600000000000001},"last":1.421}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5249999999999999,"15min":1.528},"min":{"1min":1.103,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.431,"5min":1.486,"15min":1.502},"min":{"1min":1.01,"5min":1.01,"15min":1.008},"max":{"1min":1.8620000000000001,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:14:03.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:14:03.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.566+0000 7f5567b45700 1 Processor -- start 2023-12-19T16:14:03.574 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 -- start start 2023-12-19T16:14:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55600bec60 0x7f55600bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55600c6eb0 0x7f556006a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600c7330 con 0x7f55600c6eb0 2023-12-19T16:14:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5560067e10 con 0x7f55600bec60 2023-12-19T16:14:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f5567b45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5560067200 con 0x7f55600682a0 2023-12-19T16:14:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55600c6eb0 0x7f556006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f55600c6eb0 0x7f556006a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51600/0 (socket says 172.21.15.50:51600) 2023-12-19T16:14:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 -- 172.21.15.50:0/3596298799 learned_addr learned my addr 172.21.15.50:0/3596298799 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55650e0700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55658e1700 1 --2- 172.21.15.50:0/3596298799 >> v2:172.21.15.125:3300/0 conn(0x7f55600bec60 0x7f55600bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 -- 172.21.15.50:0/3596298799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 msgr2=0x7f55600c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 -- 172.21.15.50:0/3596298799 >> v2:172.21.15.125:3300/0 conn(0x7f55600bec60 msgr2=0x7f55600bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 --2- 172.21.15.50:0/3596298799 >> v2:172.21.15.125:3300/0 conn(0x7f55600bec60 0x7f55600bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.570+0000 7f55660e2700 1 -- 172.21.15.50:0/3596298799 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55600674e0 con 0x7f55600c6eb0 2023-12-19T16:14:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f55660e2700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f556006a860 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f55600694b0 tx=0x7f5558009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798d288c5e8f630a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f55648df700 1 -- 172.21.15.50:0/3596298799 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800b640 con 0x7f55600c6eb0 2023-12-19T16:14:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f55648df700 1 -- 172.21.15.50:0/3596298799 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555800b7a0 con 0x7f55600c6eb0 2023-12-19T16:14:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f55648df700 1 -- 172.21.15.50:0/3596298799 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800bb60 con 0x7f55600c6eb0 2023-12-19T16:14:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 -- 172.21.15.50:0/3596298799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 msgr2=0x7f556006a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f556006a860 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f55600694b0 tx=0x7f5558009250 comp rx=0 tx=0).stop 2023-12-19T16:14:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 -- 172.21.15.50:0/3596298799 shutdown_connections 2023-12-19T16:14:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f556006a860 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 --2- 172.21.15.50:0/3596298799 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 --2- 172.21.15.50:0/3596298799 >> v2:172.21.15.125:3300/0 conn(0x7f55600bec60 0x7f55600bf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 -- 172.21.15.50:0/3596298799 >> 172.21.15.50:0/3596298799 conn(0x7f55600cc690 msgr2=0x7f55600c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 -- 172.21.15.50:0/3596298799 shutdown_connections 2023-12-19T16:14:03.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.574+0000 7f5567b45700 1 -- 172.21.15.50:0/3596298799 wait complete. 2023-12-19T16:14:03.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 Processor -- start 2023-12-19T16:14:03.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 -- start start 2023-12-19T16:14:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55600bec60 0x7f55600bf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c00a0 con 0x7f55600c6eb0 2023-12-19T16:14:03.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c40f0 con 0x7f55600bec60 2023-12-19T16:14:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f5567b45700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c43d0 con 0x7f55600682a0 2023-12-19T16:14:03.585 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51616/0 (socket says 172.21.15.50:51616) 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 -- 172.21.15.50:0/1515231714 learned_addr learned my addr 172.21.15.50:0/1515231714 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55650e0700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55600bec60 0x7f55600bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55658e1700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 -- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 msgr2=0x7f55600c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 -- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55600bec60 msgr2=0x7f55600bf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55600bec60 0x7f55600bf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55660e2700 1 -- 172.21.15.50:0/1515231714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580094f0 con 0x7f55600c6eb0 2023-12-19T16:14:03.586 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.578+0000 7f55658e1700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f55660e2700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f55580094c0 tx=0x7f5558017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555801a450 con 0x7f55600c6eb0 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555801a5d0 con 0x7f55600c6eb0 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555801a990 con 0x7f55600c6eb0 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55600c46b0 con 0x7f55600c6eb0 2023-12-19T16:14:03.587 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55600c4c70 con 0x7f55600c6eb0 2023-12-19T16:14:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.582+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55600c4e60 con 0x7f55600c6eb0 2023-12-19T16:14:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.586+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5558006d00 con 0x7f55600c6eb0 2023-12-19T16:14:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.586+0000 7f5555ffb700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 0x7f5544062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.586+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f555807e640 con 0x7f55600c6eb0 2023-12-19T16:14:03.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.586+0000 7f55658e1700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 0x7f5544062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.586+0000 7f55658e1700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 0x7f5544062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f555c009fd0 tx=0x7f555c00b040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.590+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f555804e9f0 con 0x7f55600c6eb0 2023-12-19T16:14:03.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.762+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5560071110 con 0x7f5544060670 2023-12-19T16:14:03.773 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.766+0000 7f5555ffb700 1 -- 172.21.15.50:0/1515231714 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47794 (secure 0 0 0) 0x7f5560071110 con 0x7f5544060670 2023-12-19T16:14:03.774 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 msgr2=0x7f5544062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 0x7f5544062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f555c009fd0 tx=0x7f555c00b040 comp rx=0 tx=0).stop 2023-12-19T16:14:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 msgr2=0x7f55600bfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f55580094c0 tx=0x7f5558017000 comp rx=0 tx=0).stop 2023-12-19T16:14:03.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 shutdown_connections 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5544060670 0x7f5544062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55600bec60 0x7f55600bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55600c6eb0 0x7f55600bfb60 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 --2- 172.21.15.50:0/1515231714 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f55600682a0 0x7f55600c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 >> 172.21.15.50:0/1515231714 conn(0x7f55600cc690 msgr2=0x7f55600cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 shutdown_connections 2023-12-19T16:14:03.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.770+0000 7f5567b45700 1 -- 172.21.15.50:0/1515231714 wait complete. 2023-12-19T16:14:03.780 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:14:03.799 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-19T16:14:03.704088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470496,"kb_used_data":22560,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752504224,"statfs":{"total":773094113280,"available":770564325376,"internally_reserved":0,"allocated":23101440,"data_stored":12387306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667618,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.990305"},"pg_stats":[{"pgid":"1.0","version":"215'370","reported_seq":1221,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-12-19T16:13:56.770416+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:56.770416+0000","last_peered":"2023-12-19T16:13:56.770416+0000","last_clean":"2023-12-19T16:13:56.770416+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:56.770416+0000","last_undegraded":"2023-12-19T16:13:56.770416+0000","last_fullsized":"2023-12-19T16:13:56.770416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575978,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304640,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067200,"statfs":{"total":96636764160,"available":96324812800,"internally_reserved":0,"allocated":3334144,"data_stored":1991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.325},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":1.196}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":1.351},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.2989999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":1.266}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.155}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.6000000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.077}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.546}]}]},{"osd":6,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.163,"15min":1.179},"min":{"1min":0.877,"5min":0.69999999999999996,"15min":0.60099999999999998},"max":{"1min":1.488,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":1.099},{"interface":"front","average":{"1min":1.292,"5min":1.224,"15min":1.2390000000000001},"min":{"1min":0.89700000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.597,"5min":1.74,"15min":1.7909999999999999},"last":1.0860000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.76500000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5509999999999999,"5min":1.643,"15min":1.7010000000000001},"last":1.141},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2609999999999999,"15min":1.274},"min":{"1min":0.89400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.8080000000000001,"15min":1.827},"last":0.998}]},{"osd":2,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.202,"15min":1.23},"min":{"1min":0.878,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.504,"5min":1.651,"15min":1.8080000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.28,"15min":1.288},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.599,"5min":1.732,"15min":1.732},"last":1.018}]},{"osd":3,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.042,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5840000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.3280000000000001},"min":{"1min":0.80200000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.212,"15min":1.2050000000000001},"min":{"1min":1.0289999999999999,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.4710000000000001,"5min":1.708,"15min":1.774},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":1.02,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.8140000000000001,"5min":1.966,"15min":1.966},"last":1.044}]},{"osd":5,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.296,"15min":1.3129999999999999},"min":{"1min":0.88200000000000001,"5min":0.83299999999999996,"15min":0.68000000000000005},"max":{"1min":1.7509999999999999,"5min":1.788,"15min":1.923},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.3620000000000001,"15min":1.3720000000000001},"min":{"1min":1.1519999999999999,"5min":0.94099999999999995,"15min":0.68899999999999995},"max":{"1min":1.593,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.1899999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4219999999999999,"15min":1.4350000000000001},"min":{"1min":1.101,"5min":0.92900000000000005,"15min":0.73799999999999999},"max":{"1min":1.706,"5min":1.8720000000000001,"15min":2.169},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.478,"5min":1.4430000000000001,"15min":1.46},"min":{"1min":0.97499999999999998,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.855,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.2869999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575978,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308156,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063684,"statfs":{"total":96636764160,"available":96321212416,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0800000000000001,"15min":1.1000000000000001},"min":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.714,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.089},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.143,"15min":1.1879999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.544,"5min":1.6619999999999999,"15min":1.907},"last":0.997}]},{"osd":1,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.228,"15min":1.2989999999999999},"min":{"1min":0.73999999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.4670000000000001,"5min":1.6759999999999999,"15min":1.7490000000000001},"last":1.135},{"interface":"front","average":{"1min":1.139,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.77800000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.5549999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.228}]},{"osd":2,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.83599999999999997,"15min":0.66300000000000003},"max":{"1min":1.454,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.3},{"interface":"front","average":{"1min":1.153,"5min":1.1890000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4379999999999999,"5min":1.7769999999999999,"15min":1.831},"last":1.2749999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2589999999999999,"15min":1.296},"min":{"1min":0.83099999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5529999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.55,"5min":1.734,"15min":2.0449999999999999},"last":1.4019999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1419999999999999,"15min":1.141},"min":{"1min":0.74199999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.278,"15min":1.2709999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.167}]},{"osd":6,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.355,"15min":1.3680000000000001},"min":{"1min":0.92500000000000004,"5min":0.878,"15min":0.49099999999999999},"max":{"1min":1.625,"5min":1.843,"15min":1.968},"last":1.512},{"interface":"front","average":{"1min":1.369,"5min":1.393,"15min":1.3999999999999999},"min":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.60599999999999998},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4239999999999999,"15min":1.413},"min":{"1min":1.0609999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7589999999999999,"5min":2.395,"15min":2.395},"last":1.556},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":0.92400000000000004,"5min":0.80400000000000005,"15min":0.55500000000000005},"max":{"1min":1.7609999999999999,"5min":1.8440000000000001,"15min":1.9770000000000001},"last":1.335}]}]},{"osd":4,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.1739999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.135},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.2110000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":1.24},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":1.0429999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.274},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.995}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":1.083}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.52},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.496},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.3500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575977,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299964,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071876,"statfs":{"total":96636764160,"available":96329601024,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":0.90600000000000003},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2609999999999999,"5min":1.633,"15min":1.7729999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96899999999999997,"15min":1.0009999999999999},"min":{"1min":0.66000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.9370000000000001},"last":0.86599999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.96299999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.53900000000000003},"max":{"1min":1.716,"5min":1.716,"15min":2.0350000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.147,"15min":1.119},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.379,"5min":1.724,"15min":2.2269999999999999},"last":1.482}]},{"osd":2,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1679999999999999},"min":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.726,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":1.306},{"interface":"front","average":{"1min":1.196,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.71599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.643,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":1.7370000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.81599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.631,"5min":1.718,"15min":2.5099999999999998},"last":1.234},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.876,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.788,"15min":2.3450000000000002},"last":1.2030000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3600000000000001,"15min":1.355},"min":{"1min":0.93300000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.6679999999999999,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.319},{"interface":"front","average":{"1min":1.393,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":0.85299999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0179999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":1.623}]},{"osd":6,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.04,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.9830000000000001,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.448,"15min":1.458},"min":{"1min":0.879,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.798,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":1.367}]},{"osd":7,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5449999999999999,"15min":1.55},"min":{"1min":0.87,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8720000000000001,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.4430000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312248,"kb_used_data":2672,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059592,"statfs":{"total":96636764160,"available":96317022208,"internally_reserved":0,"allocated":2736128,"data_stored":1400063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94599999999999995,"15min":0.96099999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.1160000000000001,"15min":1.1439999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.3520000000000001,"5min":1.629,"15min":1.796},"last":0.90200000000000002}]},{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4059999999999999,"5min":2.327,"15min":2.327},"last":1.091},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.143,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.399,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.351}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0069999999999999,"15min":1.044},"min":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.5489999999999999,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.105,"15min":1.137},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3979999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":1.0620000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.569,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.89300000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.786,"5min":2.218,"15min":2.218},"last":1.9099999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.395,"15min":1.4099999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6910000000000001,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.377,"15min":1.4039999999999999},"min":{"1min":0.91600000000000004,"5min":0.879,"15min":0.879},"max":{"1min":1.5940000000000001,"5min":2.0049999999999999,"15min":2.4340000000000002},"last":1.55}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.0099999999999998,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8500000000000001,"5min":2.609,"15min":2.609},"last":1.488}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5349999999999999,"15min":1.5509999999999999},"min":{"1min":1.054,"5min":1.054,"15min":0.89600000000000002},"max":{"1min":1.833,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.5269999999999999,"15min":1.5409999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.9319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.5980000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575977,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316348,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055492,"statfs":{"total":96636764160,"available":96312823808,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.874,"15min":0.90000000000000002},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.4690000000000001,"15min":1.6579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.97399999999999998,"15min":1.032},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.454,"5min":1.51,"15min":1.9119999999999999},"last":1.3340000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0449999999999999,"15min":1.0669999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.643,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1,"5min":1.093,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.641,"5min":1.823,"15min":1.823},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.242,"15min":1.2789999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.665,"5min":1.778,"15min":1.8180000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0960000000000001,"15min":1.105},"min":{"1min":0.65600000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.264}]},{"osd":4,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3120000000000001,"15min":1.3260000000000001},"min":{"1min":1,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.302},{"interface":"front","average":{"1min":1.395,"5min":1.391,"15min":1.423},"min":{"1min":1.014,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":2.0110000000000001},"last":1.204}]},{"osd":5,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3009999999999999,"15min":1.337},"min":{"1min":1.0049999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.597,"5min":1.833,"15min":1.897},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.395,"15min":1.4370000000000001},"min":{"1min":1.038,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.919,"5min":2.0579999999999998,"15min":2.165},"last":1.5600000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3460000000000001,"15min":1.397},"min":{"1min":0.97299999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5820000000000001,"5min":1.9590000000000001,"15min":2.1139999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4770000000000001,"15min":1.512},"min":{"1min":1.1180000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.4710000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.494,"15min":1.53},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4510000000000001,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":1.752}]}]},{"osd":0,"up_from":13,"seq":55834575977,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304636,"kb_used_data":3252,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067204,"statfs":{"total":96636764160,"available":96324816896,"internally_reserved":0,"allocated":3330048,"data_stored":1990431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.439},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.6419999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0069999999999999,"15min":1.008},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4259999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.78100000000000003}]},{"osd":2,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.141,"15min":1.147},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.57099999999999995},"max":{"1min":1.605,"5min":1.748,"15min":1.8400000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.54300000000000004},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8500000000000001},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.02,"15min":1.0409999999999999},"min":{"1min":0.60499999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.448,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.55300000000000005},"max":{"1min":1.7290000000000001,"5min":1.974,"15min":1.974},"last":1.198}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.873,"5min":0.873,"15min":0.69499999999999995},"max":{"1min":1.8120000000000001,"5min":1.885,"15min":2.0209999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.339,"15min":1.331},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71999999999999997},"max":{"1min":1.762,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.367}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.071,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.6870000000000001,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.306},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4419999999999999,"15min":1.456},"min":{"1min":1.056,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.712,"5min":1.919,"15min":1.9810000000000001},"last":1.28}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.738,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.454},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.93899999999999995},"max":{"1min":1.714,"5min":2.0009999999999999,"15min":2.1600000000000001},"last":1.421}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5249999999999999,"15min":1.528},"min":{"1min":1.103,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.486,"15min":1.502},"min":{"1min":1.01,"5min":1.01,"15min":1.008},"max":{"1min":1.8620000000000001,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.5509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:14:03.800 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-19T16:14:03.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:14:03.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 Processor -- start 2023-12-19T16:14:03.966 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 -- start start 2023-12-19T16:14:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbff40c92b0 0x7fbff40af510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff406b1a0 con 0x7fbff4067060 2023-12-19T16:14:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff40ae350 con 0x7fbff40c92b0 2023-12-19T16:14:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbffa4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff406a900 con 0x7fbff40c8e60 2023-12-19T16:14:03.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbff3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbff4067060 0x7fbff4067430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59224/0 (socket says 172.21.15.50:59224) 2023-12-19T16:14:03.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbff3fff700 1 -- 172.21.15.50:0/3037341015 learned_addr learned my addr 172.21.15.50:0/3037341015 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:03.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbff8a6a700 1 --2- 172.21.15.50:0/3037341015 >> v2:172.21.15.125:3300/0 conn(0x7fbff40c92b0 0x7fbff40af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.969 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbff37fe700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.962+0000 7fbff2ffd700 1 -- 172.21.15.50:0/3037341015 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188486669 0 0) 0x7fbff406b1a0 con 0x7fbff4067060 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff2ffd700 1 -- 172.21.15.50:0/3037341015 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4003570 con 0x7fbff4067060 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff8a6a700 1 -- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 msgr2=0x7fbff40aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff8a6a700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff8a6a700 1 -- 172.21.15.50:0/3037341015 >> v1:172.21.15.50:6789/0 conn(0x7fbff4067060 legacy=0x7fbff4067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff8a6a700 1 -- 172.21.15.50:0/3037341015 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff406abe0 con 0x7fbff40c92b0 2023-12-19T16:14:03.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff8a6a700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40af510 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0098e0 tx=0x7fbfec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=268b737d8069459d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff2ffd700 1 -- 172.21.15.50:0/3037341015 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec011630 con 0x7fbff40c92b0 2023-12-19T16:14:03.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff2ffd700 1 -- 172.21.15.50:0/3037341015 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec011790 con 0x7fbff40c92b0 2023-12-19T16:14:03.971 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbff2ffd700 1 -- 172.21.15.50:0/3037341015 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec011b30 con 0x7fbff40c92b0 2023-12-19T16:14:03.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 -- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 msgr2=0x7fbff40af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40af510 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0098e0 tx=0x7fbfec0091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 -- 172.21.15.50:0/3037341015 shutdown_connections 2023-12-19T16:14:03.972 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40af510 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/3037341015 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 -- 172.21.15.50:0/3037341015 >> 172.21.15.50:0/3037341015 conn(0x7fbff40cc690 msgr2=0x7fbff40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:03.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 -- 172.21.15.50:0/3037341015 shutdown_connections 2023-12-19T16:14:03.973 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.966+0000 7fbffa4cd700 1 -- 172.21.15.50:0/3037341015 wait complete. 2023-12-19T16:14:03.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 Processor -- start 2023-12-19T16:14:03.974 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 -- start start 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40bb3a0 con 0x7fbff4067060 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40bb830 con 0x7fbff40c92b0 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbffa4cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40ba7d0 con 0x7fbff40c8e60 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.977 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51626/0 (socket says 172.21.15.50:51626) 2023-12-19T16:14:03.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 -- 172.21.15.50:0/4173861482 learned_addr learned my addr 172.21.15.50:0/4173861482 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:03.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff37fe700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.978 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff8a6a700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 -- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 msgr2=0x7fbff40c13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40c13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 -- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 msgr2=0x7fbff40bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40bbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff3fff700 1 -- 172.21.15.50:0/4173861482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec009420 con 0x7fbff4067060 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff8a6a700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:03.979 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.970+0000 7fbff37fe700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbff3fff700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c790 tx=0x7fbfe4007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4007300 con 0x7fbff4067060 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4007460 con 0x7fbff4067060 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff40baa70 con 0x7fbff4067060 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40057f0 con 0x7fbff4067060 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff40bb030 con 0x7fbff4067060 2023-12-19T16:14:03.980 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.974+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff4071110 con 0x7fbff4067060 2023-12-19T16:14:03.982 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.978+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbfe4005c20 con 0x7fbff4067060 2023-12-19T16:14:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.978+0000 7fbfebfff700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 0x7fbfd8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.978+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fbfe407c630 con 0x7fbff4067060 2023-12-19T16:14:03.984 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.978+0000 7fbff37fe700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 0x7fbfd8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:03.985 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.978+0000 7fbff37fe700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 0x7fbfd8062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbff40c8270 tx=0x7fbfdc00b040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:03.989 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:03.982+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbfe404c9e0 con 0x7fbff4067060 2023-12-19T16:14:04.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.146+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbff40bb4e0 con 0x7fbfd8060670 2023-12-19T16:14:04.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.154+0000 7fbfebfff700 1 -- 172.21.15.50:0/4173861482 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47794 (secure 0 0 0) 0x7fbfdc00f050 con 0x7fbfd8060670 2023-12-19T16:14:04.160 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 msgr2=0x7fbfd8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 0x7fbfd8062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbff40c8270 tx=0x7fbfdc00b040 comp rx=0 tx=0).stop 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 msgr2=0x7fbff4151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c790 tx=0x7fbfe4007c80 comp rx=0 tx=0).stop 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 shutdown_connections 2023-12-19T16:14:04.162 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fbfd8060670 0x7fbfd8062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbff40c92b0 0x7fbff40bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fbff40c8e60 0x7fbff40c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 --2- 172.21.15.50:0/4173861482 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbff4067060 0x7fbff4151c40 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 >> 172.21.15.50:0/4173861482 conn(0x7fbff40cc690 msgr2=0x7fbff40c9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 shutdown_connections 2023-12-19T16:14:04.163 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.158+0000 7fbffa4cd700 1 -- 172.21.15.50:0/4173861482 wait complete. 2023-12-19T16:14:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:14:04.184 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-19T16:14:03.704088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470496,"kb_used_data":22560,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752504224,"statfs":{"total":773094113280,"available":770564325376,"internally_reserved":0,"allocated":23101440,"data_stored":12387306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667618,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-287,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.990305"},"pg_stats":[{"pgid":"1.0","version":"215'370","reported_seq":1221,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-12-19T16:13:56.770416+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:56.770416+0000","last_peered":"2023-12-19T16:13:56.770416+0000","last_clean":"2023-12-19T16:13:56.770416+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:56.770416+0000","last_undegraded":"2023-12-19T16:13:56.770416+0000","last_fullsized":"2023-12-19T16:13:56.770416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575978,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304640,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067200,"statfs":{"total":96636764160,"available":96324812800,"internally_reserved":0,"allocated":3334144,"data_stored":1991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.325},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":1.196}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":1.351},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.2989999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":1.266}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.155}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.6000000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.077}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.546}]}]},{"osd":6,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.163,"15min":1.179},"min":{"1min":0.877,"5min":0.69999999999999996,"15min":0.60099999999999998},"max":{"1min":1.488,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":1.099},{"interface":"front","average":{"1min":1.292,"5min":1.224,"15min":1.2390000000000001},"min":{"1min":0.89700000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.597,"5min":1.74,"15min":1.7909999999999999},"last":1.0860000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.76500000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5509999999999999,"5min":1.643,"15min":1.7010000000000001},"last":1.141},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2609999999999999,"15min":1.274},"min":{"1min":0.89400000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.8080000000000001,"15min":1.827},"last":0.998}]},{"osd":2,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.202,"15min":1.23},"min":{"1min":0.878,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.504,"5min":1.651,"15min":1.8080000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.28,"15min":1.288},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.599,"5min":1.732,"15min":1.732},"last":1.018}]},{"osd":3,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":1.042,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5840000000000001,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.3280000000000001},"min":{"1min":0.80200000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.212,"15min":1.2050000000000001},"min":{"1min":1.0289999999999999,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.4710000000000001,"5min":1.708,"15min":1.774},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.2629999999999999,"15min":1.304},"min":{"1min":1.02,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.8140000000000001,"5min":1.966,"15min":1.966},"last":1.044}]},{"osd":5,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.296,"15min":1.3129999999999999},"min":{"1min":0.88200000000000001,"5min":0.83299999999999996,"15min":0.68000000000000005},"max":{"1min":1.7509999999999999,"5min":1.788,"15min":1.923},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.3620000000000001,"15min":1.3720000000000001},"min":{"1min":1.1519999999999999,"5min":0.94099999999999995,"15min":0.68899999999999995},"max":{"1min":1.593,"5min":1.8200000000000001,"15min":2.1179999999999999},"last":1.1899999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:13:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4219999999999999,"15min":1.4350000000000001},"min":{"1min":1.101,"5min":0.92900000000000005,"15min":0.73799999999999999},"max":{"1min":1.706,"5min":1.8720000000000001,"15min":2.169},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.478,"5min":1.4430000000000001,"15min":1.46},"min":{"1min":0.97499999999999998,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.855,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.2869999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575978,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308156,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063684,"statfs":{"total":96636764160,"available":96321212416,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0800000000000001,"15min":1.1000000000000001},"min":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.714,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.089},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.143,"15min":1.1879999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.544,"5min":1.6619999999999999,"15min":1.907},"last":0.997}]},{"osd":1,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.228,"15min":1.2989999999999999},"min":{"1min":0.73999999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.4670000000000001,"5min":1.6759999999999999,"15min":1.7490000000000001},"last":1.135},{"interface":"front","average":{"1min":1.139,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.77800000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.5549999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.228}]},{"osd":2,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.83599999999999997,"15min":0.66300000000000003},"max":{"1min":1.454,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.3},{"interface":"front","average":{"1min":1.153,"5min":1.1890000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4379999999999999,"5min":1.7769999999999999,"15min":1.831},"last":1.2749999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2589999999999999,"15min":1.296},"min":{"1min":0.83099999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5529999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.55,"5min":1.734,"15min":2.0449999999999999},"last":1.4019999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1419999999999999,"15min":1.141},"min":{"1min":0.74199999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.278,"15min":1.2709999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.167}]},{"osd":6,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.355,"15min":1.3680000000000001},"min":{"1min":0.92500000000000004,"5min":0.878,"15min":0.49099999999999999},"max":{"1min":1.625,"5min":1.843,"15min":1.968},"last":1.512},{"interface":"front","average":{"1min":1.369,"5min":1.393,"15min":1.3999999999999999},"min":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.60599999999999998},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4239999999999999,"15min":1.413},"min":{"1min":1.0609999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7589999999999999,"5min":2.395,"15min":2.395},"last":1.556},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":0.92400000000000004,"5min":0.80400000000000005,"15min":0.55500000000000005},"max":{"1min":1.7609999999999999,"5min":1.8440000000000001,"15min":1.9770000000000001},"last":1.335}]}]},{"osd":4,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.1739999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.135},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.2110000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":1.24},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":1.0429999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.274},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.995}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":1.083}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.52},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.496},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.3500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575977,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299964,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071876,"statfs":{"total":96636764160,"available":96329601024,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":0.90600000000000003},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2609999999999999,"5min":1.633,"15min":1.7729999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96899999999999997,"15min":1.0009999999999999},"min":{"1min":0.66000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.9370000000000001},"last":0.86599999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.96299999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.53900000000000003},"max":{"1min":1.716,"5min":1.716,"15min":2.0350000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.147,"15min":1.119},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.379,"5min":1.724,"15min":2.2269999999999999},"last":1.482}]},{"osd":2,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1679999999999999},"min":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.726,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":1.306},{"interface":"front","average":{"1min":1.196,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.71599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.643,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":1.7370000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.81599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.631,"5min":1.718,"15min":2.5099999999999998},"last":1.234},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.876,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.788,"15min":2.3450000000000002},"last":1.2030000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3600000000000001,"15min":1.355},"min":{"1min":0.93300000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.6679999999999999,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.319},{"interface":"front","average":{"1min":1.393,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":0.85299999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0179999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":1.623}]},{"osd":6,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.04,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.9830000000000001,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.448,"15min":1.458},"min":{"1min":0.879,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.798,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":1.367}]},{"osd":7,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5449999999999999,"15min":1.55},"min":{"1min":0.87,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8720000000000001,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.4430000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312248,"kb_used_data":2672,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059592,"statfs":{"total":96636764160,"available":96317022208,"internally_reserved":0,"allocated":2736128,"data_stored":1400063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94599999999999995,"15min":0.96099999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.1160000000000001,"15min":1.1439999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.3520000000000001,"5min":1.629,"15min":1.796},"last":0.90200000000000002}]},{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4059999999999999,"5min":2.327,"15min":2.327},"last":1.091},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.143,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.399,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.351}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0069999999999999,"15min":1.044},"min":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.5489999999999999,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.105,"15min":1.137},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3979999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":1.0620000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.569,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.89300000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.786,"5min":2.218,"15min":2.218},"last":1.9099999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.395,"15min":1.4099999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6910000000000001,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.377,"15min":1.4039999999999999},"min":{"1min":0.91600000000000004,"5min":0.879,"15min":0.879},"max":{"1min":1.5940000000000001,"5min":2.0049999999999999,"15min":2.4340000000000002},"last":1.55}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.0099999999999998,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8500000000000001,"5min":2.609,"15min":2.609},"last":1.488}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5349999999999999,"15min":1.5509999999999999},"min":{"1min":1.054,"5min":1.054,"15min":0.89600000000000002},"max":{"1min":1.833,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.5269999999999999,"15min":1.5409999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.9319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.5980000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575977,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316348,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055492,"statfs":{"total":96636764160,"available":96312823808,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.874,"15min":0.90000000000000002},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.4690000000000001,"15min":1.6579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.97399999999999998,"15min":1.032},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.454,"5min":1.51,"15min":1.9119999999999999},"last":1.3340000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0449999999999999,"15min":1.0669999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.643,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1,"5min":1.093,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.641,"5min":1.823,"15min":1.823},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.242,"15min":1.2789999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.665,"5min":1.778,"15min":1.8180000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0960000000000001,"15min":1.105},"min":{"1min":0.65600000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.264}]},{"osd":4,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3120000000000001,"15min":1.3260000000000001},"min":{"1min":1,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.302},{"interface":"front","average":{"1min":1.395,"5min":1.391,"15min":1.423},"min":{"1min":1.014,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":2.0110000000000001},"last":1.204}]},{"osd":5,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3009999999999999,"15min":1.337},"min":{"1min":1.0049999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.597,"5min":1.833,"15min":1.897},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.395,"15min":1.4370000000000001},"min":{"1min":1.038,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.919,"5min":2.0579999999999998,"15min":2.165},"last":1.5600000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3460000000000001,"15min":1.397},"min":{"1min":0.97299999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5820000000000001,"5min":1.9590000000000001,"15min":2.1139999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4770000000000001,"15min":1.512},"min":{"1min":1.1180000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.4710000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.494,"15min":1.53},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4510000000000001,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":1.752}]}]},{"osd":0,"up_from":13,"seq":55834575977,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304636,"kb_used_data":3252,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067204,"statfs":{"total":96636764160,"available":96324816896,"internally_reserved":0,"allocated":3330048,"data_stored":1990431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.439},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.6419999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0069999999999999,"15min":1.008},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4259999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.78100000000000003}]},{"osd":2,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.141,"15min":1.147},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.57099999999999995},"max":{"1min":1.605,"5min":1.748,"15min":1.8400000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.54300000000000004},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8500000000000001},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.02,"15min":1.0409999999999999},"min":{"1min":0.60499999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.448,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.55300000000000005},"max":{"1min":1.7290000000000001,"5min":1.974,"15min":1.974},"last":1.198}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.873,"5min":0.873,"15min":0.69499999999999995},"max":{"1min":1.8120000000000001,"5min":1.885,"15min":2.0209999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.339,"15min":1.331},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71999999999999997},"max":{"1min":1.762,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.367}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.071,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.6870000000000001,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.306},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4419999999999999,"15min":1.456},"min":{"1min":1.056,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.712,"5min":1.919,"15min":1.9810000000000001},"last":1.28}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.738,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.454},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.93899999999999995},"max":{"1min":1.714,"5min":2.0009999999999999,"15min":2.1600000000000001},"last":1.421}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5249999999999999,"15min":1.528},"min":{"1min":1.103,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.486,"15min":1.502},"min":{"1min":1.01,"5min":1.01,"15min":1.008},"max":{"1min":1.8620000000000001,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.5509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:14:04.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T16:14:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.346+0000 7fc3d8e0c700 1 Processor -- start 2023-12-19T16:14:04.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.346+0000 7fc3d8e0c700 1 -- start start 2023-12-19T16:14:04.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d8e0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 0x7fc3d406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d8e0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d40abd00 con 0x7fc3d4069f50 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d8e0c700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d40abfe0 con 0x7fc3d40b1080 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d8e0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d40abb10 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d3fff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc3d4069f50 0x7fc3d40673d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59234/0 (socket says 172.21.15.50:59234) 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d3fff700 1 -- 172.21.15.50:0/1318565269 learned_addr learned my addr 172.21.15.50:0/1318565269 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:04.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.350+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/1318565269 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 0x7fc3d406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066645596 0 0) 0x7fc3d40abd00 con 0x7fc3d4069f50 2023-12-19T16:14:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0003570 con 0x7fc3d4069f50 2023-12-19T16:14:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581336788 0 0) 0x7fc3d40abfe0 con 0x7fc3d40b1080 2023-12-19T16:14:04.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d40abd00 con 0x7fc3d40b1080 2023-12-19T16:14:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d37fe700 1 -- 172.21.15.50:0/1318565269 >> v1:172.21.15.125:6789/0 conn(0x7fc3d40b1080 legacy=0x7fc3d4069840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d37fe700 1 -- 172.21.15.50:0/1318565269 >> v1:172.21.15.50:6789/0 conn(0x7fc3d4069f50 legacy=0x7fc3d40673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d37fe700 1 -- 172.21.15.50:0/1318565269 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40ac2c0 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/1318565269 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 0x7fc3d406a730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009910 tx=0x7fc3c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338f71f24ec6bfc9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4011630 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c4011790 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d2ffd700 1 -- 172.21.15.50:0/1318565269 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4011b30 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/1318565269 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 msgr2=0x7fc3d406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/1318565269 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 0x7fc3d406a730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009910 tx=0x7fc3c40091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/1318565269 shutdown_connections 2023-12-19T16:14:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/1318565269 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b0cb0 0x7fc3d406a730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/1318565269 >> 172.21.15.50:0/1318565269 conn(0x7fc3d40cc530 msgr2=0x7fc3d40c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/1318565269 shutdown_connections 2023-12-19T16:14:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.354+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/1318565269 wait complete. 2023-12-19T16:14:04.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 Processor -- start 2023-12-19T16:14:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 -- start start 2023-12-19T16:14:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d40b0cb0 0x7fc3d40a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b1080 0x7fc3d40a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d40a5270 con 0x7fc3d4069f50 2023-12-19T16:14:04.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4168af0 con 0x7fc3d40b0cb0 2023-12-19T16:14:04.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.358+0000 7fc3d8e0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41681c0 con 0x7fc3d40b1080 2023-12-19T16:14:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51630/0 (socket says 172.21.15.50:51630) 2023-12-19T16:14:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 -- 172.21.15.50:0/825264496 learned_addr learned my addr 172.21.15.50:0/825264496 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d3fff700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b1080 0x7fc3d40a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:04.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3cbfff700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d40b0cb0 0x7fc3d40a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 -- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b1080 msgr2=0x7fc3d40a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b1080 0x7fc3d40a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 -- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d40b0cb0 msgr2=0x7fc3d40a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d40b0cb0 0x7fc3d40a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 -- 172.21.15.50:0/825264496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4009420 con 0x7fc3d4069f50 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009f00 tx=0x7fc3c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c401f070 con 0x7fc3d4069f50 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c4002f70 con 0x7fc3d4069f50 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4011900 con 0x7fc3d4069f50 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d41684a0 con 0x7fc3d4069f50 2023-12-19T16:14:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d41478a0 con 0x7fc3d4069f50 2023-12-19T16:14:04.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.362+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d404be40 con 0x7fc3d4069f50 2023-12-19T16:14:04.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.366+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc3c4023070 con 0x7fc3d4069f50 2023-12-19T16:14:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.366+0000 7fc3cb7fe700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 0x7fc3b4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:04.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.366+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fc3c407d840 con 0x7fc3d4069f50 2023-12-19T16:14:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.366+0000 7fc3cbfff700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 0x7fc3b4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:04.374 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.370+0000 7fc3cbfff700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 0x7fc3b4062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40cb660 tx=0x7fc3bc009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:04.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.374+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc3c404dca0 con 0x7fc3d4069f50 2023-12-19T16:14:04.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.546+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d406ac70 con 0x7fc3d4069f50 2023-12-19T16:14:04.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.546+0000 7fc3cb7fe700 1 -- 172.21.15.50:0/825264496 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+13593 (secure 0 0 0) 0x7fc3c4051240 con 0x7fc3d4069f50 2023-12-19T16:14:04.552 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:04.552 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":218,"fsid":"8a3e066f-0ac0-4da9-a59b-c4b3f1e2d97d","created":"2023-12-19T14:38:42.714191+0000","modified":"2023-12-19T16:13:59.722350+0000","last_up_change":"2023-12-19T14:38:55.033335+0000","last_in_change":"2023-12-19T14:38:45.726769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T14:38:56.213601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b2790f77-b8c8-4d2d-ba35-18140a10a9ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":33839},{"type":"v1","addr":"172.21.15.50:6801","nonce":33839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":33839},{"type":"v1","addr":"172.21.15.50:6803","nonce":33839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":33839},{"type":"v1","addr":"172.21.15.50:6807","nonce":33839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":33839},{"type":"v1","addr":"172.21.15.50:6805","nonce":33839}]},"public_addr":"172.21.15.50:6801/33839","cluster_addr":"172.21.15.50:6803/33839","heartbeat_back_addr":"172.21.15.50:6807/33839","heartbeat_front_addr":"172.21.15.50:6805/33839","state":["exists","up"]},{"osd":1,"uuid":"e0075d32-ebb0-4a3a-b24d-81059f4b0775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":33841},{"type":"v1","addr":"172.21.15.50:6819","nonce":33841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":33841},{"type":"v1","addr":"172.21.15.50:6823","nonce":33841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":33841},{"type":"v1","addr":"172.21.15.50:6831","nonce":33841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":33841},{"type":"v1","addr":"172.21.15.50:6828","nonce":33841}]},"public_addr":"172.21.15.50:6819/33841","cluster_addr":"172.21.15.50:6823/33841","heartbeat_back_addr":"172.21.15.50:6831/33841","heartbeat_front_addr":"172.21.15.50:6828/33841","state":["exists","up"]},{"osd":2,"uuid":"10ce176d-07e0-4d6e-adb0-d6ea4c52b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":33838},{"type":"v1","addr":"172.21.15.50:6809","nonce":33838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":33838},{"type":"v1","addr":"172.21.15.50:6811","nonce":33838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":33838},{"type":"v1","addr":"172.21.15.50:6815","nonce":33838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":33838},{"type":"v1","addr":"172.21.15.50:6813","nonce":33838}]},"public_addr":"172.21.15.50:6809/33838","cluster_addr":"172.21.15.50:6811/33838","heartbeat_back_addr":"172.21.15.50:6815/33838","heartbeat_front_addr":"172.21.15.50:6813/33838","state":["exists","up"]},{"osd":3,"uuid":"b7522fc3-0859-4ba8-8d9e-f60d66a999ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":33840},{"type":"v1","addr":"172.21.15.50:6818","nonce":33840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":33840},{"type":"v1","addr":"172.21.15.50:6822","nonce":33840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6827","nonce":33840},{"type":"v1","addr":"172.21.15.50:6829","nonce":33840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":33840},{"type":"v1","addr":"172.21.15.50:6826","nonce":33840}]},"public_addr":"172.21.15.50:6818/33840","cluster_addr":"172.21.15.50:6822/33840","heartbeat_back_addr":"172.21.15.50:6829/33840","heartbeat_front_addr":"172.21.15.50:6826/33840","state":["exists","up"]},{"osd":4,"uuid":"2a72128a-b8f2-4dc3-994b-651ffc33b923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":34166},{"type":"v1","addr":"172.21.15.125:6825","nonce":34166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":34166},{"type":"v1","addr":"172.21.15.125:6827","nonce":34166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":34166},{"type":"v1","addr":"172.21.15.125:6831","nonce":34166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":34166},{"type":"v1","addr":"172.21.15.125:6829","nonce":34166}]},"public_addr":"172.21.15.125:6825/34166","cluster_addr":"172.21.15.125:6827/34166","heartbeat_back_addr":"172.21.15.125:6831/34166","heartbeat_front_addr":"172.21.15.125:6829/34166","state":["exists","up"]},{"osd":5,"uuid":"23091188-4eb7-4031-acbf-f650873398da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":34168},{"type":"v1","addr":"172.21.15.125:6810","nonce":34168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":34168},{"type":"v1","addr":"172.21.15.125:6815","nonce":34168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":34168},{"type":"v1","addr":"172.21.15.125:6823","nonce":34168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6817","nonce":34168},{"type":"v1","addr":"172.21.15.125:6820","nonce":34168}]},"public_addr":"172.21.15.125:6810/34168","cluster_addr":"172.21.15.125:6815/34168","heartbeat_back_addr":"172.21.15.125:6823/34168","heartbeat_front_addr":"172.21.15.125:6820/34168","state":["exists","up"]},{"osd":6,"uuid":"ff2c80ac-fb63-4c00-af3d-11845bb54a35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":34165},{"type":"v1","addr":"172.21.15.125:6801","nonce":34165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":34165},{"type":"v1","addr":"172.21.15.125:6803","nonce":34165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":34165},{"type":"v1","addr":"172.21.15.125:6807","nonce":34165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":34165},{"type":"v1","addr":"172.21.15.125:6805","nonce":34165}]},"public_addr":"172.21.15.125:6801/34165","cluster_addr":"172.21.15.125:6803/34165","heartbeat_back_addr":"172.21.15.125:6807/34165","heartbeat_front_addr":"172.21.15.125:6805/34165","state":["exists","up"]},{"osd":7,"uuid":"fc3490c7-db4f-483d-b20d-d7bbda886522","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6809","nonce":34167},{"type":"v1","addr":"172.21.15.125:6811","nonce":34167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6813","nonce":34167},{"type":"v1","addr":"172.21.15.125:6814","nonce":34167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6819","nonce":34167},{"type":"v1","addr":"172.21.15.125:6821","nonce":34167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":34167},{"type":"v1","addr":"172.21.15.125:6818","nonce":34167}]},"public_addr":"172.21.15.125:6811/34167","cluster_addr":"172.21.15.125:6814/34167","heartbeat_back_addr":"172.21.15.125:6821/34167","heartbeat_front_addr":"172.21.15.125:6818/34167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.530151+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.400416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.395368+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.736761+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.717654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.635106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.594821+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T14:38:51.752170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6833/4273555224":"2023-12-20T16:13:56.268975+0000","172.21.15.125:0/44652760":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/194513819":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1837833525":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/1534107098":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/4136708096":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/651582691":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2769533890":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/878125710":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/2905251978":"2023-12-20T16:13:37.811876+0000","172.21.15.125:0/3734826841":"2023-12-20T16:13:37.811876+0000","172.21.15.50:6835/4273555224":"2023-12-20T16:13:56.268975+0000","172.21.15.125:0/1362444107":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/3321035538":"2023-12-20T16:13:55.677553+0000","172.21.15.125:0/2299774917":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6833/44063":"2023-12-20T16:13:37.811876+0000","172.21.15.125:6832/44063":"2023-12-20T16:13:37.811876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T16:14:04.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 msgr2=0x7fc3b4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 0x7fc3b4062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40cb660 tx=0x7fc3bc009040 comp rx=0 tx=0).stop 2023-12-19T16:14:04.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 msgr2=0x7fc3d4147360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009f00 tx=0x7fc3c4017000 comp rx=0 tx=0).stop 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 shutdown_connections 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc3b4060670 0x7fc3b4062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3d40b0cb0 0x7fc3d40a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc3d40b1080 0x7fc3d40a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 --2- 172.21.15.50:0/825264496 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3d4069f50 0x7fc3d4147360 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 >> 172.21.15.50:0/825264496 conn(0x7fc3d40cc530 msgr2=0x7fc3d40aa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 shutdown_connections 2023-12-19T16:14:04.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:04.550+0000 7fc3d8e0c700 1 -- 172.21.15.50:0/825264496 wait complete. 2023-12-19T16:14:05.579 INFO:tasks.ceph:Scrubbing osd.0 2023-12-19T16:14:05.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.758+0000 7f95b359e700 1 Processor -- start 2023-12-19T16:14:05.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.758+0000 7f95b359e700 1 -- start start 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b359e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95b40b0d30 0x7f95b406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 0x7f95b40abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b40ade90 con 0x7f95b40b1100 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b359e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b40adb80 con 0x7f95b40b0d30 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b40ab3b0 con 0x7f95b406acd0 2023-12-19T16:14:05.766 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b1d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f95b40b1100 0x7f95b4067a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59242/0 (socket says 172.21.15.50:59242) 2023-12-19T16:14:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b1d9b700 1 -- 172.21.15.50:0/1975964833 learned_addr learned my addr 172.21.15.50:0/1975964833 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/1975964833 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 0x7f95b40abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b259c700 1 --2- 172.21.15.50:0/1975964833 >> v2:172.21.15.125:3300/0 conn(0x7f95b40b0d30 0x7f95b406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b159a700 1 -- 172.21.15.50:0/1975964833 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220947821 0 0) 0x7f95b40ade90 con 0x7f95b40b1100 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b159a700 1 -- 172.21.15.50:0/1975964833 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959c003570 con 0x7f95b40b1100 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 -- 172.21.15.50:0/1975964833 >> v2:172.21.15.125:3300/0 conn(0x7f95b40b0d30 msgr2=0x7f95b406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/1975964833 >> v2:172.21.15.125:3300/0 conn(0x7f95b40b0d30 0x7f95b406b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 -- 172.21.15.50:0/1975964833 >> v1:172.21.15.50:6789/0 conn(0x7f95b40b1100 legacy=0x7f95b4067a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 -- 172.21.15.50:0/1975964833 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b40ab690 con 0x7f95b406acd0 2023-12-19T16:14:05.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/1975964833 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 0x7f95b40abe70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009e60 tx=0x7f95ac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0dc4c0a5aa14b5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b159a700 1 -- 172.21.15.50:0/1975964833 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00e070 con 0x7f95b406acd0 2023-12-19T16:14:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b159a700 1 -- 172.21.15.50:0/1975964833 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac00bab0 con 0x7f95b406acd0 2023-12-19T16:14:05.769 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.762+0000 7f95b159a700 1 -- 172.21.15.50:0/1975964833 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00be70 con 0x7f95b406acd0 2023-12-19T16:14:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 -- 172.21.15.50:0/1975964833 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 msgr2=0x7f95b40abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 --2- 172.21.15.50:0/1975964833 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 0x7f95b40abe70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009e60 tx=0x7f95ac009780 comp rx=0 tx=0).stop 2023-12-19T16:14:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 -- 172.21.15.50:0/1975964833 shutdown_connections 2023-12-19T16:14:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 --2- 172.21.15.50:0/1975964833 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b406acd0 0x7f95b40abe70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 --2- 172.21.15.50:0/1975964833 >> v2:172.21.15.125:3300/0 conn(0x7f95b40b0d30 0x7f95b406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 -- 172.21.15.50:0/1975964833 >> 172.21.15.50:0/1975964833 conn(0x7f95b40c8560 msgr2=0x7f95b40c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:05.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 -- 172.21.15.50:0/1975964833 shutdown_connections 2023-12-19T16:14:05.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.766+0000 7f95b359e700 1 -- 172.21.15.50:0/1975964833 wait complete. 2023-12-19T16:14:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 Processor -- start 2023-12-19T16:14:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 -- start start 2023-12-19T16:14:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95b406acd0 0x7f95b415dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b40b0d30 0x7f95b416d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 0x7f95b4157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b40a4850 con 0x7f95b40b1100 2023-12-19T16:14:05.776 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b416de50 con 0x7f95b406acd0 2023-12-19T16:14:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b359e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b415ee80 con 0x7f95b40b0d30 2023-12-19T16:14:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 0x7f95b4157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 0x7f95b4157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51646/0 (socket says 172.21.15.50:51646) 2023-12-19T16:14:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 -- 172.21.15.50:0/4189612172 learned_addr learned my addr 172.21.15.50:0/4189612172 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:05.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b1d9b700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b40b0d30 0x7f95b416d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b259c700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95b406acd0 0x7f95b415dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 -- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b40b0d30 msgr2=0x7f95b416d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f95b40b0d30 0x7f95b416d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 -- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95b406acd0 msgr2=0x7f95b415dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95b406acd0 0x7f95b415dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac009a20 con 0x7f95b40b1100 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.770+0000 7f95b2d9d700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 0x7f95b4157d10 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f95b40ab4f0 tx=0x7f95ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac012e50 con 0x7f95b40b1100 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac027470 con 0x7f95b40b1100 2023-12-19T16:14:05.779 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b415e270 con 0x7f95b40b1100 2023-12-19T16:14:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac027830 con 0x7f95b40b1100 2023-12-19T16:14:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b415e830 con 0x7f95b40b1100 2023-12-19T16:14:05.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95b40b1a40 con 0x7f95b40b1100 2023-12-19T16:14:05.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.774+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f95ac024070 con 0x7f95b40b1100 2023-12-19T16:14:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95a27fc700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9594060670 0x7f9594062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.783 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f95ac07f160 con 0x7f95b40b1100 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95a27fc700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f9594066fe0 0x7f9594069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9594069b40 con 0x7f9594066fe0 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95ac0816c0 con 0x7f95b40b1100 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95b259c700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9594060670 0x7f9594062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95b1d9b700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f9594066fe0 0x7f9594069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:05.784 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95b1d9b700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f9594066fe0 0x7f9594069470 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:05.785 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.778+0000 7f95b259c700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9594060670 0x7f9594062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009db0 tx=0x7f95a40091a0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:05.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.782+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== osd.0 v2:172.21.15.50:6800/33839 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9594069b40 con 0x7f9594066fe0 2023-12-19T16:14:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.798+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 --> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f95b404be40 con 0x7f9594066fe0 2023-12-19T16:14:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95a27fc700 1 -- 172.21.15.50:0/4189612172 <== osd.0 v2:172.21.15.50:6800/33839 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f95b404be40 con 0x7f9594066fe0 2023-12-19T16:14:05.806 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:05.806 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:05.807 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f9594066fe0 msgr2=0x7f9594069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:6800/33839,v1:172.21.15.50:6801/33839] conn(0x7f9594066fe0 0x7f9594069470 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9594060670 msgr2=0x7f9594062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f9594060670 0x7f9594062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009db0 tx=0x7f95a40091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 msgr2=0x7f95b4157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 --2- 172.21.15.50:0/4189612172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95b40b1100 0x7f95b4157d10 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f95b40ab4f0 tx=0x7f95ac019000 comp rx=0 tx=0).stop 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b2d9d700 1 -- 172.21.15.50:0/4189612172 reap_dead start 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 shutdown_connections 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 >> 172.21.15.50:0/4189612172 conn(0x7f95b40c8560 msgr2=0x7f95b406e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 shutdown_connections 2023-12-19T16:14:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.802+0000 7f95b359e700 1 -- 172.21.15.50:0/4189612172 wait complete. 2023-12-19T16:14:05.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-19T16:14:06.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.998+0000 7f497a278700 1 Processor -- start 2023-12-19T16:14:06.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:05.998+0000 7f497a278700 1 -- start start 2023-12-19T16:14:06.007 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f497a278700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4974159090 0x7f49740b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f497a278700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49740cb710 con 0x7f49740b24c0 2023-12-19T16:14:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f497a278700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49740cade0 con 0x7f49740af620 2023-12-19T16:14:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f497a278700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49740af490 con 0x7f4974159090 2023-12-19T16:14:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4978815700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f49740b24c0 0x7f49740cbba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59246/0 (socket says 172.21.15.50:59246) 2023-12-19T16:14:06.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4978815700 1 -- 172.21.15.50:0/83964291 learned_addr learned my addr 172.21.15.50:0/83964291 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4973fff700 1 --2- 172.21.15.50:0/83964291 >> v2:172.21.15.125:3300/0 conn(0x7f4974159090 0x7f49740b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112051518 0 0) 0x7f49740cb710 con 0x7f49740b24c0 2023-12-19T16:14:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f495c003570 con 0x7f49740b24c0 2023-12-19T16:14:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509629819 0 0) 0x7f49740cade0 con 0x7f49740af620 2023-12-19T16:14:06.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.002+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49740cb710 con 0x7f49740af620 2023-12-19T16:14:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193052207 0 0) 0x7f495c003570 con 0x7f49740b24c0 2023-12-19T16:14:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 >> v2:172.21.15.125:3300/0 conn(0x7f4974159090 msgr2=0x7f49740b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 --2- 172.21.15.50:0/83964291 >> v2:172.21.15.125:3300/0 conn(0x7f4974159090 0x7f49740b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 >> v1:172.21.15.125:6789/0 conn(0x7f49740af620 legacy=0x7f49740b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49740cb0c0 con 0x7f49740b24c0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2615441310 0 0) 0x7f496c0017f0 con 0x7f49740b24c0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 >> v1:172.21.15.50:6789/0 conn(0x7f49740b24c0 legacy=0x7f49740cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 0x7f495c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c004810 0x7f495c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f495c0051c0 0x7f495c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49740af490 con 0x7f495c003bf0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c00c0b0 con 0x7f495c0051c0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c00c270 con 0x7f495c004810 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 0x7f495c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4978815700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f495c0051c0 0x7f495c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f49737fe700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c004810 0x7f495c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 -- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c004810 msgr2=0x7f495c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c004810 0x7f495c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 -- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f495c0051c0 msgr2=0x7f495c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:06.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f495c0051c0 0x7f495c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 -- 172.21.15.50:0/83964291 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c003570 con 0x7f495c003bf0 2023-12-19T16:14:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4978815700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f495c0051c0 0x7f495c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:06.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4978815700 1 -- 172.21.15.50:0/83964291 reap_dead start 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4973fff700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 0x7f495c003fe0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4964009c00 tx=0x7f49640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8f069511b7ddab server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496400e710 con 0x7f495c003bf0 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496400e870 con 0x7f495c003bf0 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f4972ffd700 1 -- 172.21.15.50:0/83964291 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496400ec30 con 0x7f495c003bf0 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 -- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 msgr2=0x7f495c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 0x7f495c003fe0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4964009c00 tx=0x7f49640091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 -- 172.21.15.50:0/83964291 shutdown_connections 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 --2- 172.21.15.50:0/83964291 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f495c003bf0 0x7f495c003fe0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 -- 172.21.15.50:0/83964291 >> 172.21.15.50:0/83964291 conn(0x7f49740d0ab0 msgr2=0x7f49740cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 -- 172.21.15.50:0/83964291 shutdown_connections 2023-12-19T16:14:06.014 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.006+0000 7f497a278700 1 -- 172.21.15.50:0/83964291 wait complete. 2023-12-19T16:14:06.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.010+0000 7f497a278700 1 Processor -- start 2023-12-19T16:14:06.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.010+0000 7f497a278700 1 -- start start 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c003bf0 0x7f497415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497415ac10 0x7f497414bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497415bdd0 con 0x7f4974156430 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497415b1c0 con 0x7f497415ac10 2023-12-19T16:14:06.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497415b4a0 con 0x7f495c003bf0 2023-12-19T16:14:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51670/0 (socket says 172.21.15.50:51670) 2023-12-19T16:14:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 -- 172.21.15.50:0/2858436772 learned_addr learned my addr 172.21.15.50:0/2858436772 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:06.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f4978815700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497415ac10 0x7f497414bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f4973fff700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c003bf0 0x7f497415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 -- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c003bf0 msgr2=0x7f497415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c003bf0 0x7f497415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 -- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497415ac10 msgr2=0x7f497414bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497415ac10 0x7f497414bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.020 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 -- 172.21.15.50:0/2858436772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497415b780 con 0x7f4974156430 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f49737fe700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f496800e5c0 tx=0x7f4968009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968018070 con 0x7f4974156430 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968006870 con 0x7f4974156430 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497415ba60 con 0x7f4974156430 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968006c30 con 0x7f4974156430 2023-12-19T16:14:06.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.014+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49741568a0 con 0x7f4974156430 2023-12-19T16:14:06.022 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.018+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4974071110 con 0x7f4974156430 2023-12-19T16:14:06.023 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.018+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4968007240 con 0x7f4974156430 2023-12-19T16:14:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.018+0000 7f495bfff700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 0x7f4960062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.025 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.022+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f496807c780 con 0x7f4974156430 2023-12-19T16:14:06.026 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.022+0000 7f4973fff700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 0x7f4960062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.028 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.022+0000 7f4973fff700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 0x7f4960062b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f49640091e0 tx=0x7f496401a000 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.032 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.026+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f496804cb70 con 0x7f4974156430 2023-12-19T16:14:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.198+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f49740cb3a0 con 0x7f4960060670 2023-12-19T16:14:06.203 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.198+0000 7f495bfff700 1 -- 172.21.15.50:0/2858436772 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f49740cb3a0 con 0x7f4960060670 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 msgr2=0x7f4960062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 0x7f4960062b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f49640091e0 tx=0x7f496401a000 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 msgr2=0x7f497415a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f496800e5c0 tx=0x7f4968009040 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 shutdown_connections 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4960060670 0x7f4960062b20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497415ac10 0x7f497414bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4974156430 0x7f497415a6d0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 --2- 172.21.15.50:0/2858436772 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f495c003bf0 0x7f497415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.208 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 >> 172.21.15.50:0/2858436772 conn(0x7f49740d0ab0 msgr2=0x7f4974156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 shutdown_connections 2023-12-19T16:14:06.209 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.202+0000 7f497a278700 1 -- 172.21.15.50:0/2858436772 wait complete. 2023-12-19T16:14:06.211 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2023-12-19T16:14:06.231 INFO:tasks.ceph:Scrubbing osd.1 2023-12-19T16:14:06.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:06.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.398+0000 7fdf08d23700 1 Processor -- start 2023-12-19T16:14:06.402 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.398+0000 7fdf08d23700 1 -- start start 2023-12-19T16:14:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf08d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 0x7fdf0415d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf08d23700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040b0b40 con 0x7fdf0415cee0 2023-12-19T16:14:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf08d23700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040cea80 con 0x7fdf040cb3a0 2023-12-19T16:14:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf08d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040ced60 con 0x7fdf04159230 2023-12-19T16:14:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf02ffd700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdf040cb3a0 0x7fdf04158b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:50814/0 (socket says 172.21.15.50:50814) 2023-12-19T16:14:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf02ffd700 1 -- 172.21.15.50:0/63329533 learned_addr learned my addr 172.21.15.50:0/63329533 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf037fe700 1 --1- 172.21.15.50:0/63329533 >> v1:172.21.15.50:6789/0 conn(0x7fdf0415cee0 0x7fdf040ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59260/0 (socket says 172.21.15.50:59260) 2023-12-19T16:14:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf03fff700 1 --2- 172.21.15.50:0/63329533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 0x7fdf0415d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312755144 0 0) 0x7fdf040cea80 con 0x7fdf040cb3a0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec003570 con 0x7fdf040cb3a0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183128631 0 0) 0x7fdf040b0b40 con 0x7fdf0415cee0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf040cea80 con 0x7fdf0415cee0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366295278 0 0) 0x7fdf040cea80 con 0x7fdf0415cee0 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf03fff700 1 -- 172.21.15.50:0/63329533 >> v1:172.21.15.125:6789/0 conn(0x7fdf040cb3a0 legacy=0x7fdf04158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf03fff700 1 -- 172.21.15.50:0/63329533 >> v1:172.21.15.50:6789/0 conn(0x7fdf0415cee0 legacy=0x7fdf040ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf03fff700 1 -- 172.21.15.50:0/63329533 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040cf040 con 0x7fdf04159230 2023-12-19T16:14:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf03fff700 1 --2- 172.21.15.50:0/63329533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 0x7fdf0415d2b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009e90 tx=0x7fdefc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b907b1ca701fb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.409 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.402+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00e070 con 0x7fdf04159230 2023-12-19T16:14:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc00b870 con 0x7fdf04159230 2023-12-19T16:14:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf027fc700 1 -- 172.21.15.50:0/63329533 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4001470 con 0x7fdf04159230 2023-12-19T16:14:06.410 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- 172.21.15.50:0/63329533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 msgr2=0x7fdf0415d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 --2- 172.21.15.50:0/63329533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 0x7fdf0415d2b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009e90 tx=0x7fdefc009780 comp rx=0 tx=0).stop 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- 172.21.15.50:0/63329533 shutdown_connections 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 --2- 172.21.15.50:0/63329533 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf04159230 0x7fdf0415d2b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- 172.21.15.50:0/63329533 >> 172.21.15.50:0/63329533 conn(0x7fdf040cff20 msgr2=0x7fdf040d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- 172.21.15.50:0/63329533 shutdown_connections 2023-12-19T16:14:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- 172.21.15.50:0/63329533 wait complete. 2023-12-19T16:14:06.412 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 Processor -- start 2023-12-19T16:14:06.413 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.406+0000 7fdf08d23700 1 -- start start 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 0x7fdf0415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf04159230 0x7fdf04154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf0415cee0 0x7fdf041571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415aa30 con 0x7fdf040cb3a0 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415ad10 con 0x7fdf04159230 2023-12-19T16:14:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040c8760 con 0x7fdf0415cee0 2023-12-19T16:14:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 0x7fdf0415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 0x7fdf0415a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51674/0 (socket says 172.21.15.50:51674) 2023-12-19T16:14:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 -- 172.21.15.50:0/1402938313 learned_addr learned my addr 172.21.15.50:0/1402938313 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf03fff700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf0415cee0 0x7fdf041571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf02ffd700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf04159230 0x7fdf04154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 -- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf0415cee0 msgr2=0x7fdf041571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdf0415cee0 0x7fdf041571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 -- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf04159230 msgr2=0x7fdf04154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf04159230 0x7fdf04154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc009a20 con 0x7fdf040cb3a0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf037fe700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 0x7fdf0415a1c0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fdef400cb90 tx=0x7fdef4001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4007360 con 0x7fdf040cb3a0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef40074e0 con 0x7fdf040cb3a0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4004830 con 0x7fdf040cb3a0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040c8a40 con 0x7fdf040cb3a0 2023-12-19T16:14:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.410+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04154780 con 0x7fdf040cb3a0 2023-12-19T16:14:06.418 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf0404be40 con 0x7fdf040cb3a0 2023-12-19T16:14:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdef4004bc0 con 0x7fdf040cb3a0 2023-12-19T16:14:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdef0060670 0x7fdef0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fdef407c2e0 con 0x7fdf040cb3a0 2023-12-19T16:14:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7fdef0066fe0 0x7fdef0069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef0069b40 con 0x7fdef0066fe0 2023-12-19T16:14:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdef407e900 con 0x7fdf040cb3a0 2023-12-19T16:14:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdf02ffd700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdef0060670 0x7fdef0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.414+0000 7fdf03fff700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7fdef0066fe0 0x7fdef0069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.418+0000 7fdf03fff700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7fdef0066fe0 0x7fdef0069470 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.422 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.418+0000 7fdf02ffd700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdef0060670 0x7fdef0062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf040b5f20 tx=0x7fdef8009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.423 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.418+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== osd.1 v2:172.21.15.50:6817/33841 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fdef0069b40 con 0x7fdef0066fe0 2023-12-19T16:14:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.434+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 --> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdf040cdd50 con 0x7fdef0066fe0 2023-12-19T16:14:06.441 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdeeb7fe700 1 -- 172.21.15.50:0/1402938313 <== osd.1 v2:172.21.15.50:6817/33841 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fdf040cdd50 con 0x7fdef0066fe0 2023-12-19T16:14:06.442 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:06.442 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:06.442 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7fdef0066fe0 msgr2=0x7fdef0069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.442 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:6817/33841,v1:172.21.15.50:6819/33841] conn(0x7fdef0066fe0 0x7fdef0069470 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdef0060670 msgr2=0x7fdef0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdef0060670 0x7fdef0062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf040b5f20 tx=0x7fdef8009040 comp rx=0 tx=0).stop 2023-12-19T16:14:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 msgr2=0x7fdf0415a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 --2- 172.21.15.50:0/1402938313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf040cb3a0 0x7fdf0415a1c0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fdef400cb90 tx=0x7fdef4001b50 comp rx=0 tx=0).stop 2023-12-19T16:14:06.443 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf03fff700 1 -- 172.21.15.50:0/1402938313 reap_dead start 2023-12-19T16:14:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 shutdown_connections 2023-12-19T16:14:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 >> 172.21.15.50:0/1402938313 conn(0x7fdf040cff20 msgr2=0x7fdf040ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 shutdown_connections 2023-12-19T16:14:06.444 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.438+0000 7fdf08d23700 1 -- 172.21.15.50:0/1402938313 wait complete. 2023-12-19T16:14:06.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-19T16:14:06.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.646+0000 7fdca02cd700 1 Processor -- start 2023-12-19T16:14:06.653 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.646+0000 7fdca02cd700 1 -- start start 2023-12-19T16:14:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdca02cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdc980c7350 0x7fdc981582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdca02cd700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc98069d40 con 0x7fdc9815a9d0 2023-12-19T16:14:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdca02cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc98157a20 con 0x7fdc98156fc0 2023-12-19T16:14:06.655 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdca02cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc98157d00 con 0x7fdc980c7350 2023-12-19T16:14:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9e069700 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdc98156fc0 0x7fdc980c6c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.50:34872/0 (socket says 172.21.15.50:34872) 2023-12-19T16:14:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9e069700 1 -- 172.21.15.50:0/3801059055 learned_addr learned my addr 172.21.15.50:0/3801059055 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9d868700 1 --2- 172.21.15.50:0/3801059055 >> v2:172.21.15.50:3300/0 conn(0x7fdc980c7350 0x7fdc981582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9e86a700 1 --1- 172.21.15.50:0/3801059055 >> v1:172.21.15.50:6789/0 conn(0x7fdc9815a9d0 0x7fdc9806a1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44912/0 (socket says 172.21.15.50:44912) 2023-12-19T16:14:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9d067700 1 -- 172.21.15.50:0/3801059055 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999392129 0 0) 0x7fdc98157a20 con 0x7fdc98156fc0 2023-12-19T16:14:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9d868700 1 -- 172.21.15.50:0/3801059055 >> v1:172.21.15.125:6789/0 conn(0x7fdc98156fc0 legacy=0x7fdc980c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9d868700 1 -- 172.21.15.50:0/3801059055 >> v1:172.21.15.50:6789/0 conn(0x7fdc9815a9d0 legacy=0x7fdc9806a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.650+0000 7fdc9d868700 1 -- 172.21.15.50:0/3801059055 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98157fe0 con 0x7fdc980c7350 2023-12-19T16:14:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdc9d868700 1 --2- 172.21.15.50:0/3801059055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc980c7350 0x7fdc981582c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fdc880098e0 tx=0x7fdc880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c7965c3ba54d1d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.658 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdc9d067700 1 -- 172.21.15.50:0/3801059055 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88009e80 con 0x7fdc980c7350 2023-12-19T16:14:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdc9d067700 1 -- 172.21.15.50:0/3801059055 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8800b760 con 0x7fdc980c7350 2023-12-19T16:14:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdc9d067700 1 -- 172.21.15.50:0/3801059055 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800bae0 con 0x7fdc980c7350 2023-12-19T16:14:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 -- 172.21.15.50:0/3801059055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc980c7350 msgr2=0x7fdc981582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3801059055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc980c7350 0x7fdc981582c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fdc880098e0 tx=0x7fdc880091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 -- 172.21.15.50:0/3801059055 shutdown_connections 2023-12-19T16:14:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3801059055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc980c7350 0x7fdc981582c0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 -- 172.21.15.50:0/3801059055 >> 172.21.15.50:0/3801059055 conn(0x7fdc980cc6c0 msgr2=0x7fdc980c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 -- 172.21.15.50:0/3801059055 shutdown_connections 2023-12-19T16:14:06.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.654+0000 7fdca02cd700 1 -- 172.21.15.50:0/3801059055 wait complete. 2023-12-19T16:14:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 Processor -- start 2023-12-19T16:14:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 -- start start 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc980c7350 0x7fdc980b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 0x7fdc981691e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980c3480 con 0x7fdc9815a9d0 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980af190 con 0x7fdc98156fc0 2023-12-19T16:14:06.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdca02cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980c31a0 con 0x7fdc980c7350 2023-12-19T16:14:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdc9e86a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdc9e86a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59866/0 (socket says 172.21.15.50:59866) 2023-12-19T16:14:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdc9e86a700 1 -- 172.21.15.50:0/3535587591 learned_addr learned my addr 172.21.15.50:0/3535587591 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdc9d868700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 0x7fdc981691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.658+0000 7fdc9e069700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc980c7350 0x7fdc980b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 -- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc980c7350 msgr2=0x7fdc980b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc980c7350 0x7fdc980b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 -- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 msgr2=0x7fdc981691e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 0x7fdc981691e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 -- 172.21.15.50:0/3535587591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88009420 con 0x7fdc9815a9d0 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9d868700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 0x7fdc981691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:06.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc9e86a700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000ce30 tx=0x7fdc90001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90005df0 con 0x7fdc9815a9d0 2023-12-19T16:14:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9000a9b0 con 0x7fdc9815a9d0 2023-12-19T16:14:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc980aeb40 con 0x7fdc9815a9d0 2023-12-19T16:14:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9000ad70 con 0x7fdc9815a9d0 2023-12-19T16:14:06.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc980b1860 con 0x7fdc9815a9d0 2023-12-19T16:14:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.662+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc980af320 con 0x7fdc9815a9d0 2023-12-19T16:14:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.666+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdc90007300 con 0x7fdc9815a9d0 2023-12-19T16:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.666+0000 7fdc8e7fc700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 0x7fdc7c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.666+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fdc9007cd40 con 0x7fdc9815a9d0 2023-12-19T16:14:06.672 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.666+0000 7fdc9e069700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 0x7fdc7c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:06.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.666+0000 7fdc9e069700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 0x7fdc7c062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc940011e0 tx=0x7fdc94009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:06.677 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.674+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc9004d0f0 con 0x7fdc9815a9d0 2023-12-19T16:14:06.869 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.862+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc98071110 con 0x7fdc7c060670 2023-12-19T16:14:06.870 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.866+0000 7fdc8e7fc700 1 -- 172.21.15.50:0/3535587591 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdc98071110 con 0x7fdc7c060670 2023-12-19T16:14:06.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 msgr2=0x7fdc7c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 0x7fdc7c062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc940011e0 tx=0x7fdc94009040 comp rx=0 tx=0).stop 2023-12-19T16:14:06.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 msgr2=0x7fdc980c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:06.874 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000ce30 tx=0x7fdc90001b80 comp rx=0 tx=0).stop 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 shutdown_connections 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdc7c060670 0x7fdc7c062b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdc98156fc0 0x7fdc981691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc9815a9d0 0x7fdc980c4980 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 --2- 172.21.15.50:0/3535587591 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdc980c7350 0x7fdc980b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 >> 172.21.15.50:0/3535587591 conn(0x7fdc980cc6c0 msgr2=0x7fdc98159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:06.875 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 shutdown_connections 2023-12-19T16:14:06.876 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:06.870+0000 7fdca02cd700 1 -- 172.21.15.50:0/3535587591 wait complete. 2023-12-19T16:14:06.877 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2023-12-19T16:14:06.899 INFO:tasks.ceph:Scrubbing osd.2 2023-12-19T16:14:06.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.066+0000 7f4d4faf0700 1 Processor -- start 2023-12-19T16:14:07.072 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.066+0000 7f4d4faf0700 1 -- start start 2023-12-19T16:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4d480c7440 0x7f4d480ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4d48067060 0x7f4d480acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d480abd50 con 0x7f4d480bf030 2023-12-19T16:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4815db90 con 0x7f4d480c7440 2023-12-19T16:14:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d480684f0 con 0x7f4d48067060 2023-12-19T16:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4d480c7440 0x7f4d480ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4d480c7440 0x7f4d480ac7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59868/0 (socket says 172.21.15.50:59868) 2023-12-19T16:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 -- 172.21.15.50:0/3451102401 learned_addr learned my addr 172.21.15.50:0/3451102401 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4e08d700 1 --2- 172.21.15.50:0/3451102401 >> v2:172.21.15.125:3300/0 conn(0x7f4d48067060 0x7f4d480acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 -- 172.21.15.50:0/3451102401 >> v2:172.21.15.125:3300/0 conn(0x7f4d48067060 msgr2=0x7f4d480acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 --2- 172.21.15.50:0/3451102401 >> v2:172.21.15.125:3300/0 conn(0x7f4d48067060 0x7f4d480acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 -- 172.21.15.50:0/3451102401 >> v1:172.21.15.50:6789/0 conn(0x7f4d480bf030 legacy=0x7f4d480c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 -- 172.21.15.50:0/3451102401 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d480687d0 con 0x7f4d480c7440 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4c88a700 1 -- 172.21.15.50:0/3451102401 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237242624 0 0) 0x7f4d480abd50 con 0x7f4d480bf030 2023-12-19T16:14:07.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4d08b700 1 --2- 172.21.15.50:0/3451102401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d480c7440 0x7f4d480ac7f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f4d48070630 tx=0x7f4d38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b283f2a11f4f49d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4c88a700 1 -- 172.21.15.50:0/3451102401 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800b860 con 0x7f4d480c7440 2023-12-19T16:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4c88a700 1 -- 172.21.15.50:0/3451102401 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3800b9c0 con 0x7f4d480c7440 2023-12-19T16:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4c88a700 1 -- 172.21.15.50:0/3451102401 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800bd60 con 0x7f4d480c7440 2023-12-19T16:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- 172.21.15.50:0/3451102401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d480c7440 msgr2=0x7f4d480ac7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/3451102401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d480c7440 0x7f4d480ac7f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f4d48070630 tx=0x7f4d38009250 comp rx=0 tx=0).stop 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- 172.21.15.50:0/3451102401 shutdown_connections 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/3451102401 >> v2:172.21.15.125:3300/0 conn(0x7f4d48067060 0x7f4d480acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/3451102401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d480c7440 0x7f4d480ac7f0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.070+0000 7f4d4faf0700 1 -- 172.21.15.50:0/3451102401 >> 172.21.15.50:0/3451102401 conn(0x7f4d480cc7e0 msgr2=0x7f4d480c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- 172.21.15.50:0/3451102401 shutdown_connections 2023-12-19T16:14:07.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- 172.21.15.50:0/3451102401 wait complete. 2023-12-19T16:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 Processor -- start 2023-12-19T16:14:07.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- start start 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 0x7f4d4815a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d480bf030 0x7f4d48157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4d480c7440 0x7f4d480c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48159e80 con 0x7f4d48067060 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4815a440 con 0x7f4d480bf030 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.074+0000 7f4d4faf0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d480c5920 con 0x7f4d480c7440 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 0x7f4d4815a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 0x7f4d4815a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59880/0 (socket says 172.21.15.50:59880) 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 -- 172.21.15.50:0/2939565383 learned_addr learned my addr 172.21.15.50:0/2939565383 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d08b700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d480bf030 0x7f4d48157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4e08d700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4d480c7440 0x7f4d480c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 -- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4d480c7440 msgr2=0x7f4d480c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4d480c7440 0x7f4d480c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 -- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d480bf030 msgr2=0x7f4d48157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4d480bf030 0x7f4d48157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380094f0 con 0x7f4d48067060 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4d88c700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 0x7f4d4815a8d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c810 tx=0x7f4d44007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44011590 con 0x7f4d48067060 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d44011710 con 0x7f4d48067060 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44011ad0 con 0x7f4d48067060 2023-12-19T16:14:07.084 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d480c5c00 con 0x7f4d48067060 2023-12-19T16:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4815ae10 con 0x7f4d48067060 2023-12-19T16:14:07.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.078+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d480708a0 con 0x7f4d48067060 2023-12-19T16:14:07.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4d4401b3b0 con 0x7f4d48067060 2023-12-19T16:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4d2c060670 0x7f4d2c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f4d4407c060 con 0x7f4d48067060 2023-12-19T16:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7f4d2c066fe0 0x7f4d2c069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d2c069b40 con 0x7f4d2c066fe0 2023-12-19T16:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d4407e560 con 0x7f4d48067060 2023-12-19T16:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d4d08b700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4d2c060670 0x7f4d2c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.082+0000 7f4d4e08d700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7f4d2c066fe0 0x7f4d2c069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.086+0000 7f4d4e08d700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7f4d2c066fe0 0x7f4d2c069470 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.086+0000 7f4d4d08b700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4d2c060670 0x7f4d2c062b20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d48070630 tx=0x7f4d38017000 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.086+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== osd.2 v2:172.21.15.50:6808/33838 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4d2c069b40 con 0x7f4d2c066fe0 2023-12-19T16:14:07.108 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.102+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 --> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4d480abff0 con 0x7f4d2c066fe0 2023-12-19T16:14:07.109 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.102+0000 7f4d3dffb700 1 -- 172.21.15.50:0/2939565383 <== osd.2 v2:172.21.15.50:6808/33838 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4d480abff0 con 0x7f4d2c066fe0 2023-12-19T16:14:07.109 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:07.109 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:07.109 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7f4d2c066fe0 msgr2=0x7f4d2c069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:6808/33838,v1:172.21.15.50:6809/33838] conn(0x7f4d2c066fe0 0x7f4d2c069470 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4d2c060670 msgr2=0x7f4d2c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4d2c060670 0x7f4d2c062b20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d48070630 tx=0x7f4d38017000 comp rx=0 tx=0).stop 2023-12-19T16:14:07.110 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 msgr2=0x7f4d4815a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 --2- 172.21.15.50:0/2939565383 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4d48067060 0x7f4d4815a8d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c810 tx=0x7f4d44007df0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4e08d700 1 -- 172.21.15.50:0/2939565383 reap_dead start 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 shutdown_connections 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 >> 172.21.15.50:0/2939565383 conn(0x7f4d480cc7e0 msgr2=0x7f4d480c9840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 shutdown_connections 2023-12-19T16:14:07.111 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.106+0000 7f4d4faf0700 1 -- 172.21.15.50:0/2939565383 wait complete. 2023-12-19T16:14:07.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-19T16:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.302+0000 7f96f6aae700 1 Processor -- start 2023-12-19T16:14:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.302+0000 7f96f6aae700 1 -- start start 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96f0159090 0x7f96f00b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f00afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96f00b1db0 0x7f96f00b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f00af490 con 0x7f96f00b1db0 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f00cb710 con 0x7f96f0159090 2023-12-19T16:14:07.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cade0 con 0x7f96f00af620 2023-12-19T16:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96f00b1db0 0x7f96f00b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f96f00b1db0 0x7f96f00b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59888/0 (socket says 172.21.15.50:59888) 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 -- 172.21.15.50:0/3639021366 learned_addr learned my addr 172.21.15.50:0/3639021366 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f484a700 1 --2- 172.21.15.50:0/3639021366 >> v2:172.21.15.125:3300/0 conn(0x7f96f0159090 0x7f96f00b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96effff700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f00afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 -- 172.21.15.50:0/3639021366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 msgr2=0x7f96f00afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f00afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 -- 172.21.15.50:0/3639021366 >> v2:172.21.15.125:3300/0 conn(0x7f96f0159090 msgr2=0x7f96f00b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 --2- 172.21.15.50:0/3639021366 >> v2:172.21.15.125:3300/0 conn(0x7f96f0159090 0x7f96f00b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 -- 172.21.15.50:0/3639021366 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00cb0c0 con 0x7f96f00b1db0 2023-12-19T16:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f504b700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f00b1db0 0x7f96f00b2230 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f96f0088ea0 tx=0x7f96e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b94894add0bc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96ef7fe700 1 -- 172.21.15.50:0/3639021366 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e800e070 con 0x7f96f00b1db0 2023-12-19T16:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96ef7fe700 1 -- 172.21.15.50:0/3639021366 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e800b970 con 0x7f96f00b1db0 2023-12-19T16:14:07.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96ef7fe700 1 -- 172.21.15.50:0/3639021366 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e80092e0 con 0x7f96f00b1db0 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 -- 172.21.15.50:0/3639021366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f00b1db0 msgr2=0x7f96f00b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.306+0000 7f96f6aae700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f00b1db0 0x7f96f00b2230 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f96f0088ea0 tx=0x7f96e8009840 comp rx=0 tx=0).stop 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 -- 172.21.15.50:0/3639021366 shutdown_connections 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f00b1db0 0x7f96f00b2230 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 --2- 172.21.15.50:0/3639021366 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f00afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 --2- 172.21.15.50:0/3639021366 >> v2:172.21.15.125:3300/0 conn(0x7f96f0159090 0x7f96f00b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 -- 172.21.15.50:0/3639021366 >> 172.21.15.50:0/3639021366 conn(0x7f96f00d0ab0 msgr2=0x7f96f00cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 -- 172.21.15.50:0/3639021366 shutdown_connections 2023-12-19T16:14:07.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 -- 172.21.15.50:0/3639021366 wait complete. 2023-12-19T16:14:07.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 Processor -- start 2023-12-19T16:14:07.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.310+0000 7f96f6aae700 1 -- start start 2023-12-19T16:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f0169760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 0x7f96f015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f0169ca0 con 0x7f96f0159090 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f015e090 con 0x7f96f00b1db0 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f0159a20 con 0x7f96f00af620 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f504b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 0x7f96f015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:50664/0 (socket says 172.21.15.50:50664) 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f484a700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f0169760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 -- 172.21.15.50:0/2619563452 learned_addr learned my addr 172.21.15.50:0/2619563452 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 -- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 msgr2=0x7f96f0169760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f0169760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 -- 172.21.15.50:0/2619563452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 msgr2=0x7f96f015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 0x7f96f015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 -- 172.21.15.50:0/2619563452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8009ae0 con 0x7f96f00b1db0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f504b700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 0x7f96f015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96effff700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ec80 tx=0x7f96dc009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc00c850 con 0x7f96f00b1db0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc00c9b0 con 0x7f96f00b1db0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc00cd30 con 0x7f96f00b1db0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f015db10 con 0x7f96f00b1db0 2023-12-19T16:14:07.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.314+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f0159740 con 0x7f96f00b1db0 2023-12-19T16:14:07.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.318+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f0071110 con 0x7f96f00b1db0 2023-12-19T16:14:07.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.318+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f96dc013680 con 0x7f96f00b1db0 2023-12-19T16:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.318+0000 7f96ecff9700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 0x7f96d4062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.322+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f96dc07cfe0 con 0x7f96f00b1db0 2023-12-19T16:14:07.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.322+0000 7f96f484a700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 0x7f96d4062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.322+0000 7f96f484a700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 0x7f96d4062bf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009d10 tx=0x7f96e40091a0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.326+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f96dc04d3b0 con 0x7f96f00b1db0 2023-12-19T16:14:07.508 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.502+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f96f015de70 con 0x7f96d4060740 2023-12-19T16:14:07.509 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.502+0000 7f96ecff9700 1 -- 172.21.15.50:0/2619563452 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f96f015de70 con 0x7f96d4060740 2023-12-19T16:14:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 msgr2=0x7f96d4062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 0x7f96d4062bf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009d10 tx=0x7f96e40091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 msgr2=0x7f96f0156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.513 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ec80 tx=0x7f96dc009120 comp rx=0 tx=0).stop 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 shutdown_connections 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f96d4060740 0x7f96d4062bf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96f00b1db0 0x7f96f0156bc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96f0159090 0x7f96f015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 --2- 172.21.15.50:0/2619563452 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f96f00af620 0x7f96f0169760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.506+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 >> 172.21.15.50:0/2619563452 conn(0x7f96f00d0ab0 msgr2=0x7f96f0067060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.510+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 shutdown_connections 2023-12-19T16:14:07.514 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.510+0000 7f96f6aae700 1 -- 172.21.15.50:0/2619563452 wait complete. 2023-12-19T16:14:07.517 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2023-12-19T16:14:07.536 INFO:tasks.ceph:Scrubbing osd.3 2023-12-19T16:14:07.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:07.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 Processor -- start 2023-12-19T16:14:07.711 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 -- start start 2023-12-19T16:14:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdad4156dd0 0x7fdad40ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad415b290 con 0x7fdad41571a0 2023-12-19T16:14:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad415ac80 con 0x7fdad415b720 2023-12-19T16:14:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad8aa5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad40ae4a0 con 0x7fdad4156dd0 2023-12-19T16:14:07.713 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad1d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdad41571a0 0x7fdad40adc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44922/0 (socket says 172.21.15.50:44922) 2023-12-19T16:14:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad1d9b700 1 -- 172.21.15.50:0/520276989 learned_addr learned my addr 172.21.15.50:0/520276989 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.706+0000 7fdad259c700 1 --2- 172.21.15.50:0/520276989 >> v2:172.21.15.50:3300/0 conn(0x7fdad4156dd0 0x7fdad40ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161922124 0 0) 0x7fdad415b290 con 0x7fdad41571a0 2023-12-19T16:14:07.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdabc003570 con 0x7fdad41571a0 2023-12-19T16:14:07.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249586197 0 0) 0x7fdad415ac80 con 0x7fdad415b720 2023-12-19T16:14:07.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad259c700 1 -- 172.21.15.50:0/520276989 >> v1:172.21.15.125:6789/0 conn(0x7fdad415b720 legacy=0x7fdad40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad259c700 1 -- 172.21.15.50:0/520276989 >> v1:172.21.15.50:6789/0 conn(0x7fdad41571a0 legacy=0x7fdad40adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.715 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad259c700 1 -- 172.21.15.50:0/520276989 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad415af20 con 0x7fdad4156dd0 2023-12-19T16:14:07.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad259c700 1 --2- 172.21.15.50:0/520276989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad40ca3e0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdac40098e0 tx=0x7fdac40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfeb81997d67d4ec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400b640 con 0x7fdad4156dd0 2023-12-19T16:14:07.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400b7a0 con 0x7fdad4156dd0 2023-12-19T16:14:07.716 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad159a700 1 -- 172.21.15.50:0/520276989 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac400bb40 con 0x7fdad4156dd0 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 -- 172.21.15.50:0/520276989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 msgr2=0x7fdad40ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 --2- 172.21.15.50:0/520276989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad40ca3e0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fdac40098e0 tx=0x7fdac40091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 -- 172.21.15.50:0/520276989 shutdown_connections 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 --2- 172.21.15.50:0/520276989 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad40ca3e0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 -- 172.21.15.50:0/520276989 >> 172.21.15.50:0/520276989 conn(0x7fdad4067060 msgr2=0x7fdad4067460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 -- 172.21.15.50:0/520276989 shutdown_connections 2023-12-19T16:14:07.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.710+0000 7fdad8aa5700 1 -- 172.21.15.50:0/520276989 wait complete. 2023-12-19T16:14:07.718 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 Processor -- start 2023-12-19T16:14:07.719 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 -- start start 2023-12-19T16:14:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad4168d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.720 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad41571a0 0x7fdad40cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad415b720 0x7fdad40cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad41580c0 con 0x7fdad4156dd0 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4158360 con 0x7fdad415b720 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad8aa5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4158640 con 0x7fdad41571a0 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad4168d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad4168d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59908/0 (socket says 172.21.15.50:59908) 2023-12-19T16:14:07.721 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 -- 172.21.15.50:0/337251426 learned_addr learned my addr 172.21.15.50:0/337251426 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad2d9d700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad415b720 0x7fdad40cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad1d9b700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad41571a0 0x7fdad40cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 -- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad41571a0 msgr2=0x7fdad40cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdad41571a0 0x7fdad40cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 -- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad415b720 msgr2=0x7fdad40cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdad415b720 0x7fdad40cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.714+0000 7fdad259c700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009420 con 0x7fdad4156dd0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdad259c700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad4168d90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fdad40cef90 tx=0x7fdac4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac401f070 con 0x7fdad4156dd0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400bb20 con 0x7fdad4156dd0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4011670 con 0x7fdad4156dd0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad40b1710 con 0x7fdad4156dd0 2023-12-19T16:14:07.723 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad415c200 con 0x7fdad4156dd0 2023-12-19T16:14:07.724 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.718+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdad40708a0 con 0x7fdad4156dd0 2023-12-19T16:14:07.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdac4011a60 con 0x7fdad4156dd0 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdab4060670 0x7fdab4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fdac407df70 con 0x7fdad4156dd0 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7fdab4066fe0 0x7fdab4069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdab4069b40 con 0x7fdab4066fe0 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdac40804d0 con 0x7fdad4156dd0 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdad2d9d700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7fdab4066fe0 0x7fdab4069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.727 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdad1d9b700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdab4060670 0x7fdab4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdad2d9d700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7fdab4066fe0 0x7fdab4069470 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.728 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.722+0000 7fdad1d9b700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdab4060670 0x7fdab4062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdac80058d0 tx=0x7fdac8009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.729 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.726+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== osd.3 v2:172.21.15.50:6816/33840 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fdab4069b40 con 0x7fdab4066fe0 2023-12-19T16:14:07.746 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 --> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdad40ae780 con 0x7fdab4066fe0 2023-12-19T16:14:07.747 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdac27fc700 1 -- 172.21.15.50:0/337251426 <== osd.3 v2:172.21.15.50:6816/33840 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fdad40ae780 con 0x7fdab4066fe0 2023-12-19T16:14:07.747 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:07.747 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:07.747 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:07.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7fdab4066fe0 msgr2=0x7fdab4069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:6816/33840,v1:172.21.15.50:6818/33840] conn(0x7fdab4066fe0 0x7fdab4069470 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdab4060670 msgr2=0x7fdab4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdab4060670 0x7fdab4062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdac80058d0 tx=0x7fdac8009040 comp rx=0 tx=0).stop 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 msgr2=0x7fdad4168d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 --2- 172.21.15.50:0/337251426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdad4156dd0 0x7fdad4168d90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fdad40cef90 tx=0x7fdac4017000 comp rx=0 tx=0).stop 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad2d9d700 1 -- 172.21.15.50:0/337251426 reap_dead start 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 shutdown_connections 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 >> 172.21.15.50:0/337251426 conn(0x7fdad4067060 msgr2=0x7fdad4155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 shutdown_connections 2023-12-19T16:14:07.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.742+0000 7fdad8aa5700 1 -- 172.21.15.50:0/337251426 wait complete. 2023-12-19T16:14:07.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-19T16:14:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.930+0000 7f0a6f66e700 1 Processor -- start 2023-12-19T16:14:07.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.930+0000 7f0a6f66e700 1 -- start start 2023-12-19T16:14:07.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6f66e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680ca390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6f66e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680ca8d0 con 0x7f0a681606c0 2023-12-19T16:14:07.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6f66e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6806a850 con 0x7f0a68160a90 2023-12-19T16:14:07.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6f66e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68069cb0 con 0x7f0a680c9f30 2023-12-19T16:14:07.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6d40a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f0a681606c0 0x7f0a6806ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44930/0 (socket says 172.21.15.50:44930) 2023-12-19T16:14:07.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6d40a700 1 -- 172.21.15.50:0/1200142307 learned_addr learned my addr 172.21.15.50:0/1200142307 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:07.940 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6dc0b700 1 --2- 172.21.15.50:0/1200142307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811524154 0 0) 0x7f0a64001110 con 0x7f0a681606c0 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a54003570 con 0x7f0a681606c0 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081458493 0 0) 0x7f0a6806a850 con 0x7f0a68160a90 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a64001110 con 0x7f0a68160a90 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1200142307 >> v1:172.21.15.125:6789/0 conn(0x7f0a68160a90 legacy=0x7f0a68067a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.941 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1200142307 >> v1:172.21.15.50:6789/0 conn(0x7f0a681606c0 legacy=0x7f0a6806ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263525450 0 0) 0x7f0a54003570 con 0x7f0a681606c0 2023-12-19T16:14:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.934+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1200142307 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a68069f20 con 0x7f0a680c9f30 2023-12-19T16:14:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6dc0b700 1 --2- 172.21.15.50:0/1200142307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680ca390 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009e90 tx=0x7f0a60009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a372c710fb27145 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000e070 con 0x7f0a680c9f30 2023-12-19T16:14:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6000b870 con 0x7f0a680c9f30 2023-12-19T16:14:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a5ffff700 1 -- 172.21.15.50:0/1200142307 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64001620 con 0x7f0a680c9f30 2023-12-19T16:14:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1200142307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 msgr2=0x7f0a680ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1200142307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680ca390 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009e90 tx=0x7f0a60009780 comp rx=0 tx=0).stop 2023-12-19T16:14:07.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1200142307 shutdown_connections 2023-12-19T16:14:07.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1200142307 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680ca390 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1200142307 >> 172.21.15.50:0/1200142307 conn(0x7f0a680cc6c0 msgr2=0x7f0a680c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:07.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1200142307 shutdown_connections 2023-12-19T16:14:07.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.938+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1200142307 wait complete. 2023-12-19T16:14:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 Processor -- start 2023-12-19T16:14:07.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 -- start start 2023-12-19T16:14:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 0x7f0a681562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6815c1c0 con 0x7f0a68160a90 2023-12-19T16:14:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680aef50 con 0x7f0a681606c0 2023-12-19T16:14:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6f66e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680afd70 con 0x7f0a680c9f30 2023-12-19T16:14:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59920/0 (socket says 172.21.15.50:59920) 2023-12-19T16:14:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1728767632 learned_addr learned my addr 172.21.15.50:0/1728767632 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6d40a700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6cc09700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 0x7f0a681562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 msgr2=0x7f0a680aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 msgr2=0x7f0a681562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 0x7f0a681562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 -- 172.21.15.50:0/1728767632 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60009a20 con 0x7f0a68160a90 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6cc09700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 0x7f0a681562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.942+0000 7f0a6dc0b700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009de0 tx=0x7f0a60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60012d80 con 0x7f0a68160a90 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6001e450 con 0x7f0a68160a90 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a680af160 con 0x7f0a68160a90 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6001e810 con 0x7f0a68160a90 2023-12-19T16:14:07.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a680af720 con 0x7f0a68160a90 2023-12-19T16:14:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a68071110 con 0x7f0a68160a90 2023-12-19T16:14:07.953 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.946+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0a6001ec40 con 0x7f0a68160a90 2023-12-19T16:14:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.950+0000 7f0a5d7fa700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 0x7f0a4c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:07.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.950+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f0a6007ea90 con 0x7f0a68160a90 2023-12-19T16:14:07.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.950+0000 7f0a6d40a700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 0x7f0a4c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:07.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.950+0000 7f0a6d40a700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 0x7f0a4c062b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a64000c00 tx=0x7f0a64001b40 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:07.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:07.954+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a6004ecb0 con 0x7f0a68160a90 2023-12-19T16:14:08.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.130+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a680c9350 con 0x7f0a4c0606a0 2023-12-19T16:14:08.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.130+0000 7f0a5d7fa700 1 -- 172.21.15.50:0/1728767632 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0a680c9350 con 0x7f0a4c0606a0 2023-12-19T16:14:08.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 msgr2=0x7f0a4c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 0x7f0a4c062b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a64000c00 tx=0x7f0a64001b40 comp rx=0 tx=0).stop 2023-12-19T16:14:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 msgr2=0x7f0a68158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009de0 tx=0x7f0a60017000 comp rx=0 tx=0).stop 2023-12-19T16:14:08.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 shutdown_connections 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f0a4c0606a0 0x7f0a4c062b50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a681606c0 0x7f0a681562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a68160a90 0x7f0a68158060 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 --2- 172.21.15.50:0/1728767632 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f0a680c9f30 0x7f0a680aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 >> 172.21.15.50:0/1728767632 conn(0x7f0a680cc6c0 msgr2=0x7f0a68068250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 shutdown_connections 2023-12-19T16:14:08.141 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.134+0000 7f0a6f66e700 1 -- 172.21.15.50:0/1728767632 wait complete. 2023-12-19T16:14:08.142 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 3 to deep-scrub 2023-12-19T16:14:08.163 INFO:tasks.ceph:Scrubbing osd.4 2023-12-19T16:14:08.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:08.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.342+0000 7fc831e03700 1 Processor -- start 2023-12-19T16:14:08.347 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.342+0000 7fc831e03700 1 -- start start 2023-12-19T16:14:08.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831e03700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc82c159230 0x7fc82c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831e03700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c0b0b40 con 0x7fc82c0cb3a0 2023-12-19T16:14:08.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831e03700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c0cea80 con 0x7fc82c15cee0 2023-12-19T16:14:08.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831e03700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82c1596b0 con 0x7fc82c159230 2023-12-19T16:14:08.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82bfff700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc82c0cb3a0 0x7fc82c158b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44932/0 (socket says 172.21.15.50:44932) 2023-12-19T16:14:08.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82bfff700 1 -- 172.21.15.50:0/1622296372 learned_addr learned my addr 172.21.15.50:0/1622296372 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:08.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831602700 1 --2- 172.21.15.50:0/1622296372 >> v2:172.21.15.125:3300/0 conn(0x7fc82c159230 0x7fc82c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503301632 0 0) 0x7fc82c0b0b40 con 0x7fc82c0cb3a0 2023-12-19T16:14:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc81c003570 con 0x7fc82c0cb3a0 2023-12-19T16:14:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277311426 0 0) 0x7fc82c0cea80 con 0x7fc82c15cee0 2023-12-19T16:14:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c0b0b40 con 0x7fc82c15cee0 2023-12-19T16:14:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831602700 1 -- 172.21.15.50:0/1622296372 >> v1:172.21.15.125:6789/0 conn(0x7fc82c15cee0 legacy=0x7fc82c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831602700 1 -- 172.21.15.50:0/1622296372 >> v1:172.21.15.50:6789/0 conn(0x7fc82c0cb3a0 legacy=0x7fc82c158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.346+0000 7fc831602700 1 -- 172.21.15.50:0/1622296372 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c0ced60 con 0x7fc82c159230 2023-12-19T16:14:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831602700 1 --2- 172.21.15.50:0/1622296372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c159230 0x7fc82c15d2b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc824009e60 tx=0x7fc824009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=271d485b4a7e26b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82400e070 con 0x7fc82c159230 2023-12-19T16:14:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82400bab0 con 0x7fc82c159230 2023-12-19T16:14:08.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc82b7fe700 1 -- 172.21.15.50:0/1622296372 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82400be70 con 0x7fc82c159230 2023-12-19T16:14:08.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 -- 172.21.15.50:0/1622296372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c159230 msgr2=0x7fc82c15d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 --2- 172.21.15.50:0/1622296372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c159230 0x7fc82c15d2b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc824009e60 tx=0x7fc824009780 comp rx=0 tx=0).stop 2023-12-19T16:14:08.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 -- 172.21.15.50:0/1622296372 shutdown_connections 2023-12-19T16:14:08.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 --2- 172.21.15.50:0/1622296372 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c159230 0x7fc82c15d2b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 -- 172.21.15.50:0/1622296372 >> 172.21.15.50:0/1622296372 conn(0x7fc82c0cff20 msgr2=0x7fc82c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:08.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 -- 172.21.15.50:0/1622296372 shutdown_connections 2023-12-19T16:14:08.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.350+0000 7fc831e03700 1 -- 172.21.15.50:0/1622296372 wait complete. 2023-12-19T16:14:08.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 Processor -- start 2023-12-19T16:14:08.359 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 -- start start 2023-12-19T16:14:08.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c0cb3a0 0x7fc82c145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.360 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc82c159230 0x7fc82c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 0x7fc82c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c153ab0 con 0x7fc82c15cee0 2023-12-19T16:14:08.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c154310 con 0x7fc82c0cb3a0 2023-12-19T16:14:08.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831e03700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0b5ad0 con 0x7fc82c159230 2023-12-19T16:14:08.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831602700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 0x7fc82c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831602700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 0x7fc82c156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59922/0 (socket says 172.21.15.50:59922) 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc831602700 1 -- 172.21.15.50:0/129805902 learned_addr learned my addr 172.21.15.50:0/129805902 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc830e01700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c0cb3a0 0x7fc82c145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.354+0000 7fc82bfff700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc82c159230 0x7fc82c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 -- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc82c159230 msgr2=0x7fc82c155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.362 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fc82c159230 0x7fc82c155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 -- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c0cb3a0 msgr2=0x7fc82c145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc82c0cb3a0 0x7fc82c145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc824009a20 con 0x7fc82c15cee0 2023-12-19T16:14:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831602700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 0x7fc82c156440 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0cb8c0 tx=0x7fc824019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824012db0 con 0x7fc82c15cee0 2023-12-19T16:14:08.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc824026480 con 0x7fc82c15cee0 2023-12-19T16:14:08.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824026840 con 0x7fc82c15cee0 2023-12-19T16:14:08.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c153cc0 con 0x7fc82c15cee0 2023-12-19T16:14:08.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c146270 con 0x7fc82c15cee0 2023-12-19T16:14:08.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.358+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc82c04be40 con 0x7fc82c15cee0 2023-12-19T16:14:08.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc82401c400 con 0x7fc82c15cee0 2023-12-19T16:14:08.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc810060670 0x7fc810062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fc82407eb10 con 0x7fc82c15cee0 2023-12-19T16:14:08.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7fc810066fe0 0x7fc810069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc810069b40 con 0x7fc810066fe0 2023-12-19T16:14:08.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc824083220 con 0x7fc82c15cee0 2023-12-19T16:14:08.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc830e01700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc810060670 0x7fc810062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.362+0000 7fc82bfff700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7fc810066fe0 0x7fc810069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.369 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.366+0000 7fc830e01700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc810060670 0x7fc810062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc820007fd0 tx=0x7fc820007d50 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.366+0000 7fc82bfff700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7fc810066fe0 0x7fc810069470 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.371 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.366+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== osd.4 v2:172.21.15.125:6824/34166 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc810069b40 con 0x7fc810066fe0 2023-12-19T16:14:08.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.382+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 --> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc82c0cf040 con 0x7fc810066fe0 2023-12-19T16:14:08.389 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc828ff9700 1 -- 172.21.15.50:0/129805902 <== osd.4 v2:172.21.15.125:6824/34166 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc82c0cf040 con 0x7fc810066fe0 2023-12-19T16:14:08.390 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:08.390 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:08.390 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:08.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7fc810066fe0 msgr2=0x7fc810069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.390 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6824/34166,v1:172.21.15.125:6825/34166] conn(0x7fc810066fe0 0x7fc810069470 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc810060670 msgr2=0x7fc810062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fc810060670 0x7fc810062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc820007fd0 tx=0x7fc820007d50 comp rx=0 tx=0).stop 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 msgr2=0x7fc82c156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 --2- 172.21.15.50:0/129805902 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc82c15cee0 0x7fc82c156440 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0cb8c0 tx=0x7fc824019000 comp rx=0 tx=0).stop 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831602700 1 -- 172.21.15.50:0/129805902 reap_dead start 2023-12-19T16:14:08.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 shutdown_connections 2023-12-19T16:14:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 >> 172.21.15.50:0/129805902 conn(0x7fc82c0cff20 msgr2=0x7fc82c0ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 shutdown_connections 2023-12-19T16:14:08.392 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.386+0000 7fc831e03700 1 -- 172.21.15.50:0/129805902 wait complete. 2023-12-19T16:14:08.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-19T16:14:08.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.570+0000 7f4c4b82e700 1 Processor -- start 2023-12-19T16:14:08.577 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.570+0000 7f4c4b82e700 1 -- start start 2023-12-19T16:14:08.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c44159090 0x7f4c440b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.579 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4c440af620 0x7f4c440afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4c440b1db0 0x7f4c440b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c440af490 con 0x7f4c440b1db0 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c440cb710 con 0x7f4c440af620 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c4b82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c440cade0 con 0x7f4c44159090 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4c440b1db0 0x7f4c440b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.580 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4c440b1db0 0x7f4c440b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59932/0 (socket says 172.21.15.50:59932) 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 -- 172.21.15.50:0/3755183659 learned_addr learned my addr 172.21.15.50:0/3755183659 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c48dc9700 1 --2- 172.21.15.50:0/3755183659 >> v2:172.21.15.125:3300/0 conn(0x7f4c440af620 0x7f4c440afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c495ca700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c44159090 0x7f4c440b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 -- 172.21.15.50:0/3755183659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c44159090 msgr2=0x7f4c440b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c44159090 0x7f4c440b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.581 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 -- 172.21.15.50:0/3755183659 >> v2:172.21.15.125:3300/0 conn(0x7f4c440af620 msgr2=0x7f4c440afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 --2- 172.21.15.50:0/3755183659 >> v2:172.21.15.125:3300/0 conn(0x7f4c440af620 0x7f4c440afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 -- 172.21.15.50:0/3755183659 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440cb0c0 con 0x7f4c440b1db0 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.574+0000 7f4c49dcb700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440b1db0 0x7f4c440b2230 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406a190 tx=0x7f4c3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79060ce4945a6bef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c3bfff700 1 -- 172.21.15.50:0/3755183659 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c00e070 con 0x7f4c440b1db0 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c3bfff700 1 -- 172.21.15.50:0/3755183659 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c00b970 con 0x7f4c440b1db0 2023-12-19T16:14:08.582 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c3bfff700 1 -- 172.21.15.50:0/3755183659 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0092e0 con 0x7f4c440b1db0 2023-12-19T16:14:08.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 -- 172.21.15.50:0/3755183659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440b1db0 msgr2=0x7f4c440b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440b1db0 0x7f4c440b2230 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f4c4406a190 tx=0x7f4c3c009840 comp rx=0 tx=0).stop 2023-12-19T16:14:08.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 -- 172.21.15.50:0/3755183659 shutdown_connections 2023-12-19T16:14:08.583 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440b1db0 0x7f4c440b2230 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/3755183659 >> v2:172.21.15.125:3300/0 conn(0x7f4c440af620 0x7f4c440afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/3755183659 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c44159090 0x7f4c440b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 -- 172.21.15.50:0/3755183659 >> 172.21.15.50:0/3755183659 conn(0x7f4c440d0ab0 msgr2=0x7f4c440cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:08.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 -- 172.21.15.50:0/3755183659 shutdown_connections 2023-12-19T16:14:08.584 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.578+0000 7f4c4b82e700 1 -- 172.21.15.50:0/3755183659 wait complete. 2023-12-19T16:14:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 Processor -- start 2023-12-19T16:14:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 -- start start 2023-12-19T16:14:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c440b1db0 0x7f4c440c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c44159090 0x7f4c441565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c440c76b0 con 0x7f4c440af620 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c44168fe0 con 0x7f4c44159090 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c4b82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c44169830 con 0x7f4c440b1db0 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c495ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.593 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c495ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59936/0 (socket says 172.21.15.50:59936) 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.582+0000 7f4c495ca700 1 -- 172.21.15.50:0/1010463287 learned_addr learned my addr 172.21.15.50:0/1010463287 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c48dc9700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c440b1db0 0x7f4c440c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c49dcb700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c44159090 0x7f4c441565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 -- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c440b1db0 msgr2=0x7f4c440c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c440b1db0 0x7f4c440c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.594 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 -- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c44159090 msgr2=0x7f4c441565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c44159090 0x7f4c441565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 -- 172.21.15.50:0/1010463287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c009ae0 con 0x7f4c440af620 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c495ca700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000e960 tx=0x7f4c400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40013680 con 0x7f4c440af620 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c400137e0 con 0x7f4c440af620 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4415dcc0 con 0x7f4c440af620 2023-12-19T16:14:08.595 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40013b80 con 0x7f4c440af620 2023-12-19T16:14:08.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.586+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c440c7450 con 0x7f4c440af620 2023-12-19T16:14:08.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.590+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c441695e0 con 0x7f4c440af620 2023-12-19T16:14:08.596 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.590+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4c400123d0 con 0x7f4c440af620 2023-12-19T16:14:08.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.590+0000 7f4c397fa700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 0x7f4c28062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:08.597 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.590+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f4c4007cb30 con 0x7f4c440af620 2023-12-19T16:14:08.598 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.594+0000 7f4c48dc9700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 0x7f4c28062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:08.599 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.594+0000 7f4c48dc9700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 0x7f4c28062bd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c440c9130 tx=0x7f4c34009120 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:08.604 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.598+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4c4004ce60 con 0x7f4c440af620 2023-12-19T16:14:08.780 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.774+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c44169970 con 0x7f4c28060720 2023-12-19T16:14:08.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.774+0000 7f4c397fa700 1 -- 172.21.15.50:0/1010463287 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c34011a50 con 0x7f4c28060720 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.778+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 msgr2=0x7f4c28062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.778+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 0x7f4c28062bd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c440c9130 tx=0x7f4c34009120 comp rx=0 tx=0).stop 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.778+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 msgr2=0x7f4c440c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.778+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000e960 tx=0x7f4c400091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 shutdown_connections 2023-12-19T16:14:08.786 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4c28060720 0x7f4c28062bd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c44159090 0x7f4c441565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4c440b1db0 0x7f4c440c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 --2- 172.21.15.50:0/1010463287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c440af620 0x7f4c440c7930 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 >> 172.21.15.50:0/1010463287 conn(0x7f4c440d0ab0 msgr2=0x7f4c441575a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 shutdown_connections 2023-12-19T16:14:08.787 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.782+0000 7f4c4b82e700 1 -- 172.21.15.50:0/1010463287 wait complete. 2023-12-19T16:14:08.789 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 4 to deep-scrub 2023-12-19T16:14:08.811 INFO:tasks.ceph:Scrubbing osd.5 2023-12-19T16:14:08.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:08.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.990+0000 7f4ff22f0700 1 Processor -- start 2023-12-19T16:14:08.998 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.990+0000 7f4ff22f0700 1 -- start start 2023-12-19T16:14:09.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff22f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec0abdc0 0x7f4fec0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff22f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4fec069980 0x7f4fec0b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff22f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0678d0 con 0x7f4fec156f80 2023-12-19T16:14:09.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff22f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec067bb0 con 0x7f4fec069980 2023-12-19T16:14:09.001 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff22f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec067e90 con 0x7f4fec0abdc0 2023-12-19T16:14:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.994+0000 7f4ff088d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4fec069980 0x7f4fec0b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f4fec069980 0x7f4fec0b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59948/0 (socket says 172.21.15.50:59948) 2023-12-19T16:14:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 -- 172.21.15.50:0/416348762 learned_addr learned my addr 172.21.15.50:0/416348762 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4fe3fff700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec0abdc0 0x7f4fec0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 -- 172.21.15.50:0/416348762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec0abdc0 msgr2=0x7f4fec0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec0abdc0 0x7f4fec0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 -- 172.21.15.50:0/416348762 >> v1:172.21.15.125:6789/0 conn(0x7f4fec156f80 legacy=0x7f4fec0ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 -- 172.21.15.50:0/416348762 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec068170 con 0x7f4fec069980 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff088d700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec069980 0x7f4fec0b2360 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009f20 tx=0x7f4fe4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e3a4de792ded36 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4feb7fe700 1 -- 172.21.15.50:0/416348762 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe400e070 con 0x7f4fec069980 2023-12-19T16:14:09.003 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4feb7fe700 1 -- 172.21.15.50:0/416348762 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe400ba80 con 0x7f4fec069980 2023-12-19T16:14:09.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4feb7fe700 1 -- 172.21.15.50:0/416348762 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40092e0 con 0x7f4fec069980 2023-12-19T16:14:09.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 -- 172.21.15.50:0/416348762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec069980 msgr2=0x7f4fec0b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.004 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec069980 0x7f4fec0b2360 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009f20 tx=0x7f4fe4009840 comp rx=0 tx=0).stop 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 -- 172.21.15.50:0/416348762 shutdown_connections 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec069980 0x7f4fec0b2360 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/416348762 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec0abdc0 0x7f4fec0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 -- 172.21.15.50:0/416348762 >> 172.21.15.50:0/416348762 conn(0x7f4fec096190 msgr2=0x7f4fec0afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 -- 172.21.15.50:0/416348762 shutdown_connections 2023-12-19T16:14:09.005 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:08.998+0000 7f4ff22f0700 1 -- 172.21.15.50:0/416348762 wait complete. 2023-12-19T16:14:09.008 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.002+0000 7f4ff22f0700 1 Processor -- start 2023-12-19T16:14:09.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.002+0000 7f4ff22f0700 1 -- start start 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec069980 0x7f4fec1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 0x7f4fec159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fec156f80 0x7f4fec158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec096d10 con 0x7f4fec0abdc0 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1580c0 con 0x7f4fec156f80 2023-12-19T16:14:09.010 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1574f0 con 0x7f4fec069980 2023-12-19T16:14:09.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 0x7f4fec159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 0x7f4fec159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59950/0 (socket says 172.21.15.50:59950) 2023-12-19T16:14:09.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 -- 172.21.15.50:0/1885573744 learned_addr learned my addr 172.21.15.50:0/1885573744 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4febfff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec069980 0x7f4fec1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.011 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff088d700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fec156f80 0x7f4fec158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 -- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec069980 msgr2=0x7f4fec1564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f4fec069980 0x7f4fec1564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 -- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fec156f80 msgr2=0x7f4fec158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fec156f80 0x7f4fec158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4009ae0 con 0x7f4fec0abdc0 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe3fff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 0x7f4fec159490 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400d890 tx=0x7f4fd4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40125f0 con 0x7f4fec0abdc0 2023-12-19T16:14:09.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd4012750 con 0x7f4fec0abdc0 2023-12-19T16:14:09.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4012af0 con 0x7f4fec0abdc0 2023-12-19T16:14:09.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec157790 con 0x7f4fec0abdc0 2023-12-19T16:14:09.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.006+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec157d50 con 0x7f4fec0abdc0 2023-12-19T16:14:09.013 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.010+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4fec0708a0 con 0x7f4fec0abdc0 2023-12-19T16:14:09.016 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.010+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4fd401e460 con 0x7f4fec0abdc0 2023-12-19T16:14:09.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.010+0000 7f4fe8ff9700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4fcc0606a0 0x7f4fcc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.017 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.010+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f4fd407d0b0 con 0x7f4fec0abdc0 2023-12-19T16:14:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.010+0000 7f4fe8ff9700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7f4fcc067010 0x7f4fcc0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fcc069b70 con 0x7f4fcc067010 2023-12-19T16:14:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fd407f550 con 0x7f4fec0abdc0 2023-12-19T16:14:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4febfff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4fcc0606a0 0x7f4fcc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4ff088d700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7f4fcc067010 0x7f4fcc0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4febfff700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4fcc0606a0 0x7f4fcc062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc003a20 tx=0x7f4fdc007f10 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.019 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4ff088d700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7f4fcc067010 0x7f4fcc0694a0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.021 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.014+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== osd.5 v2:172.21.15.125:6808/34168 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4fcc069b70 con 0x7f4fcc067010 2023-12-19T16:14:09.038 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 --> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4fec068450 con 0x7f4fcc067010 2023-12-19T16:14:09.039 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4fe8ff9700 1 -- 172.21.15.50:0/1885573744 <== osd.5 v2:172.21.15.125:6808/34168 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4fec068450 con 0x7f4fcc067010 2023-12-19T16:14:09.040 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:09.040 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:09.040 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7f4fcc067010 msgr2=0x7f4fcc0694a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.040 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6808/34168,v1:172.21.15.125:6810/34168] conn(0x7f4fcc067010 0x7f4fcc0694a0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4fcc0606a0 msgr2=0x7f4fcc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f4fcc0606a0 0x7f4fcc062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc003a20 tx=0x7f4fdc007f10 comp rx=0 tx=0).stop 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 msgr2=0x7f4fec159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 --2- 172.21.15.50:0/1885573744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4fec0abdc0 0x7f4fec159490 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400d890 tx=0x7f4fd4008040 comp rx=0 tx=0).stop 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff088d700 1 -- 172.21.15.50:0/1885573744 reap_dead start 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 shutdown_connections 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 >> 172.21.15.50:0/1885573744 conn(0x7f4fec096190 msgr2=0x7f4fec0ab6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 shutdown_connections 2023-12-19T16:14:09.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.034+0000 7f4ff22f0700 1 -- 172.21.15.50:0/1885573744 wait complete. 2023-12-19T16:14:09.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-19T16:14:09.244 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.238+0000 7f5762763700 1 Processor -- start 2023-12-19T16:14:09.245 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.238+0000 7f5762763700 1 -- start start 2023-12-19T16:14:09.248 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5762763700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 0x7f575c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5762763700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f575c0b1be0 0x7f575c0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5762763700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0cb3e0 con 0x7f575c159210 2023-12-19T16:14:09.249 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5762763700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c068960 con 0x7f575c0b1be0 2023-12-19T16:14:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5762763700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0caab0 con 0x7f575c158e40 2023-12-19T16:14:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5760f60700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f575c159210 0x7f575c067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44938/0 (socket says 172.21.15.50:44938) 2023-12-19T16:14:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5760f60700 1 -- 172.21.15.50:0/1128417837 learned_addr learned my addr 172.21.15.50:0/1128417837 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5761761700 1 --2- 172.21.15.50:0/1128417837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 0x7f575c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.250 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.242+0000 7f5761f62700 1 --2- 172.21.15.50:0/1128417837 >> v2:172.21.15.125:3300/0 conn(0x7f575c0b1be0 0x7f575c0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5753fff700 1 -- 172.21.15.50:0/1128417837 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107514899 0 0) 0x7f575c0cb3e0 con 0x7f575c159210 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5753fff700 1 -- 172.21.15.50:0/1128417837 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5748003570 con 0x7f575c159210 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5761761700 1 -- 172.21.15.50:0/1128417837 >> v2:172.21.15.125:3300/0 conn(0x7f575c0b1be0 msgr2=0x7f575c0b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5761761700 1 --2- 172.21.15.50:0/1128417837 >> v2:172.21.15.125:3300/0 conn(0x7f575c0b1be0 0x7f575c0b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5761761700 1 -- 172.21.15.50:0/1128417837 >> v1:172.21.15.50:6789/0 conn(0x7f575c159210 legacy=0x7f575c067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5761761700 1 -- 172.21.15.50:0/1128417837 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0cad90 con 0x7f575c158e40 2023-12-19T16:14:09.251 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5761761700 1 --2- 172.21.15.50:0/1128417837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 0x7f575c0b2c00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f57580098e0 tx=0x7f57580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a8af5d987fb2350 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5753fff700 1 -- 172.21.15.50:0/1128417837 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758011630 con 0x7f575c158e40 2023-12-19T16:14:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5753fff700 1 -- 172.21.15.50:0/1128417837 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758011790 con 0x7f575c158e40 2023-12-19T16:14:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5753fff700 1 -- 172.21.15.50:0/1128417837 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758011b30 con 0x7f575c158e40 2023-12-19T16:14:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 -- 172.21.15.50:0/1128417837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 msgr2=0x7f575c0b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.252 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 --2- 172.21.15.50:0/1128417837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 0x7f575c0b2c00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f57580098e0 tx=0x7f57580091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 -- 172.21.15.50:0/1128417837 shutdown_connections 2023-12-19T16:14:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 --2- 172.21.15.50:0/1128417837 >> v2:172.21.15.125:3300/0 conn(0x7f575c0b1be0 0x7f575c0b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 --2- 172.21.15.50:0/1128417837 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c158e40 0x7f575c0b2c00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.253 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 -- 172.21.15.50:0/1128417837 >> 172.21.15.50:0/1128417837 conn(0x7f575c0d0900 msgr2=0x7f575c0cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 -- 172.21.15.50:0/1128417837 shutdown_connections 2023-12-19T16:14:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.246+0000 7f5762763700 1 -- 172.21.15.50:0/1128417837 wait complete. 2023-12-19T16:14:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 Processor -- start 2023-12-19T16:14:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 -- start start 2023-12-19T16:14:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f575c158e40 0x7f575c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c159210 0x7f575c15e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.257 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0c6310 con 0x7f575c0b1be0 2023-12-19T16:14:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0c5270 con 0x7f575c158e40 2023-12-19T16:14:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5762763700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0c5550 con 0x7f575c159210 2023-12-19T16:14:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5761761700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5761761700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59954/0 (socket says 172.21.15.50:59954) 2023-12-19T16:14:09.258 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.250+0000 7f5761761700 1 -- 172.21.15.50:0/1543037828 learned_addr learned my addr 172.21.15.50:0/1543037828 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5760f60700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f575c158e40 0x7f575c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761f62700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c159210 0x7f575c15e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 -- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c159210 msgr2=0x7f575c15e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c159210 0x7f575c15e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 -- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f575c158e40 msgr2=0x7f575c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f575c158e40 0x7f575c15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 -- 172.21.15.50:0/1543037828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5758009420 con 0x7f575c0b1be0 2023-12-19T16:14:09.259 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5761761700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cabf0 tx=0x7f5758017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758011b10 con 0x7f575c0b1be0 2023-12-19T16:14:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580253f0 con 0x7f575c0b1be0 2023-12-19T16:14:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758025790 con 0x7f575c0b1be0 2023-12-19T16:14:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0c5830 con 0x7f575c0b1be0 2023-12-19T16:14:09.260 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c0c5df0 con 0x7f575c0b1be0 2023-12-19T16:14:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.254+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c04be40 con 0x7f575c0b1be0 2023-12-19T16:14:09.263 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.258+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5758025b80 con 0x7f575c0b1be0 2023-12-19T16:14:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.258+0000 7f57517fa700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 0x7f5740062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.262+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f575807e2b0 con 0x7f575c0b1be0 2023-12-19T16:14:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.262+0000 7f5760f60700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 0x7f5740062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.262+0000 7f5760f60700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 0x7f5740062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f575c16ca60 tx=0x7f574c009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.271 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.266+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f575804e660 con 0x7f575c0b1be0 2023-12-19T16:14:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.442+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f575c0cb070 con 0x7f5740060670 2023-12-19T16:14:09.448 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.442+0000 7f57517fa700 1 -- 172.21.15.50:0/1543037828 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f575c0cb070 con 0x7f5740060670 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 msgr2=0x7f5740062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 0x7f5740062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f575c16ca60 tx=0x7f574c009040 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 msgr2=0x7f575c0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cabf0 tx=0x7f5758017000 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 shutdown_connections 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f5740060670 0x7f5740062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f575c158e40 0x7f575c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f575c159210 0x7f575c15e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 --2- 172.21.15.50:0/1543037828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f575c0b1be0 0x7f575c0c4c90 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 >> 172.21.15.50:0/1543037828 conn(0x7f575c0d0900 msgr2=0x7f575c1536f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 shutdown_connections 2023-12-19T16:14:09.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.446+0000 7f5762763700 1 -- 172.21.15.50:0/1543037828 wait complete. 2023-12-19T16:14:09.454 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 5 to deep-scrub 2023-12-19T16:14:09.473 INFO:tasks.ceph:Scrubbing osd.6 2023-12-19T16:14:09.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:09.656 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.650+0000 7f03d92cd700 1 Processor -- start 2023-12-19T16:14:09.657 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.650+0000 7f03d92cd700 1 -- start start 2023-12-19T16:14:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d92cd700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03d4159400 0x7f03d4068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d92cd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03d4068770 0x7f03d40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d92cd700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d4067da0 con 0x7f03d4159030 2023-12-19T16:14:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d92cd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40cb880 con 0x7f03d4068770 2023-12-19T16:14:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d92cd700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d4067190 con 0x7f03d4159400 2023-12-19T16:14:09.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03d4068770 0x7f03d40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f03d4068770 0x7f03d40ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59964/0 (socket says 172.21.15.50:59964) 2023-12-19T16:14:09.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 -- 172.21.15.50:0/3334283899 learned_addr learned my addr 172.21.15.50:0/3334283899 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.660 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d27fc700 1 --2- 172.21.15.50:0/3334283899 >> v2:172.21.15.125:3300/0 conn(0x7f03d4159400 0x7f03d4068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 -- 172.21.15.50:0/3334283899 >> v2:172.21.15.125:3300/0 conn(0x7f03d4159400 msgr2=0x7f03d4068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 --2- 172.21.15.50:0/3334283899 >> v2:172.21.15.125:3300/0 conn(0x7f03d4159400 0x7f03d4068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 -- 172.21.15.50:0/3334283899 >> v1:172.21.15.125:6789/0 conn(0x7f03d4159030 legacy=0x7f03d40cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.654+0000 7f03d37fe700 1 -- 172.21.15.50:0/3334283899 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4067470 con 0x7f03d4068770 2023-12-19T16:14:09.661 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d37fe700 1 --2- 172.21.15.50:0/3334283899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4068770 0x7f03d40ce0d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f03d406abe0 tx=0x7f03cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09c0e8ae8901787 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d1ffb700 1 -- 172.21.15.50:0/3334283899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00e070 con 0x7f03d4068770 2023-12-19T16:14:09.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d1ffb700 1 -- 172.21.15.50:0/3334283899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00ba20 con 0x7f03d4068770 2023-12-19T16:14:09.662 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d1ffb700 1 -- 172.21.15.50:0/3334283899 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0092e0 con 0x7f03d4068770 2023-12-19T16:14:09.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 -- 172.21.15.50:0/3334283899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4068770 msgr2=0x7f03d40ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 --2- 172.21.15.50:0/3334283899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4068770 0x7f03d40ce0d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f03d406abe0 tx=0x7f03cc009840 comp rx=0 tx=0).stop 2023-12-19T16:14:09.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 -- 172.21.15.50:0/3334283899 shutdown_connections 2023-12-19T16:14:09.663 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 --2- 172.21.15.50:0/3334283899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4068770 0x7f03d40ce0d0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 --2- 172.21.15.50:0/3334283899 >> v2:172.21.15.125:3300/0 conn(0x7f03d4159400 0x7f03d4068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 -- 172.21.15.50:0/3334283899 >> 172.21.15.50:0/3334283899 conn(0x7f03d40d0170 msgr2=0x7f03d40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 -- 172.21.15.50:0/3334283899 shutdown_connections 2023-12-19T16:14:09.664 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.658+0000 7f03d92cd700 1 -- 172.21.15.50:0/3334283899 wait complete. 2023-12-19T16:14:09.665 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 Processor -- start 2023-12-19T16:14:09.666 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 -- start start 2023-12-19T16:14:09.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d4068770 0x7f03d4140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 0x7f03d40c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d4159400 0x7f03d40b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40b4940 con 0x7f03d4159030 2023-12-19T16:14:09.667 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41539f0 con 0x7f03d4068770 2023-12-19T16:14:09.668 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40b3a60 con 0x7f03d4159400 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 0x7f03d40c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 0x7f03d40c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59974/0 (socket says 172.21.15.50:59974) 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d2ffd700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d4068770 0x7f03d4140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 -- 172.21.15.50:0/1656107906 learned_addr learned my addr 172.21.15.50:0/1656107906 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d37fe700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d4159400 0x7f03d40b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 -- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d4159400 msgr2=0x7f03d40b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f03d4159400 0x7f03d40b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 -- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d4068770 msgr2=0x7f03d4140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03d4068770 0x7f03d4140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc009ae0 con 0x7f03d4159030 2023-12-19T16:14:09.669 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d27fc700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 0x7f03d40c7220 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e900 tx=0x7f03c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8013680 con 0x7f03d4159030 2023-12-19T16:14:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c80137e0 con 0x7f03d4159030 2023-12-19T16:14:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.662+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d40b3d40 con 0x7f03d4159030 2023-12-19T16:14:09.670 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.666+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8013b80 con 0x7f03d4159030 2023-12-19T16:14:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.666+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d40b4e70 con 0x7f03d4159030 2023-12-19T16:14:09.671 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.666+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03d40708a0 con 0x7f03d4159030 2023-12-19T16:14:09.673 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.666+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f03c80123d0 con 0x7f03d4159030 2023-12-19T16:14:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03baffd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03c0060670 0x7f03c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.674 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f03c807cb00 con 0x7f03d4159030 2023-12-19T16:14:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03baffd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f03c0066fe0 0x7f03c0069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03c0069b40 con 0x7f03c0066fe0 2023-12-19T16:14:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03c8081220 con 0x7f03d4159030 2023-12-19T16:14:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03d2ffd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03c0060670 0x7f03c0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.675 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03d37fe700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f03c0066fe0 0x7f03c0069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03d2ffd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03c0060670 0x7f03c0062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03c4003a80 tx=0x7f03c4007ec0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.676 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.670+0000 7f03d37fe700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f03c0066fe0 0x7f03c0069470 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.678 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.674+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== osd.6 v2:172.21.15.125:6800/34165 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f03c0069b40 con 0x7f03c0066fe0 2023-12-19T16:14:09.696 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.690+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 --> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f03d4067750 con 0x7f03c0066fe0 2023-12-19T16:14:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.690+0000 7f03baffd700 1 -- 172.21.15.50:0/1656107906 <== osd.6 v2:172.21.15.125:6800/34165 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f03d4067750 con 0x7f03c0066fe0 2023-12-19T16:14:09.697 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:09.697 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:09.697 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.690+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f03c0066fe0 msgr2=0x7f03c0069470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.690+0000 7f03d92cd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6800/34165,v1:172.21.15.125:6801/34165] conn(0x7f03c0066fe0 0x7f03c0069470 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.698 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03c0060670 msgr2=0x7f03c0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f03c0060670 0x7f03c0062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f03c4003a80 tx=0x7f03c4007ec0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 msgr2=0x7f03d40c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 --2- 172.21.15.50:0/1656107906 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4159030 0x7f03d40c7220 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f03c800e900 tx=0x7f03c80091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d37fe700 1 -- 172.21.15.50:0/1656107906 reap_dead start 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 shutdown_connections 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 >> 172.21.15.50:0/1656107906 conn(0x7f03d40d0170 msgr2=0x7f03d415ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 shutdown_connections 2023-12-19T16:14:09.699 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.694+0000 7f03d92cd700 1 -- 172.21.15.50:0/1656107906 wait complete. 2023-12-19T16:14:09.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-19T16:14:09.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.878+0000 7f58456f0700 1 Processor -- start 2023-12-19T16:14:09.882 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.878+0000 7f58456f0700 1 -- start start 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f58456f0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 0x7f58400c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f58456f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 0x7f58400c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f58456f0700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400ca820 con 0x7f58400c9d00 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f58456f0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400689a0 con 0x7f584013f490 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f58456f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5840067500 con 0x7f584013f860 2023-12-19T16:14:09.886 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583effd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 0x7f58400c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583effd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 0x7f58400c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59986/0 (socket says 172.21.15.50:59986) 2023-12-19T16:14:09.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 0x7f58400c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583effd700 1 -- 172.21.15.50:0/2020120475 learned_addr learned my addr 172.21.15.50:0/2020120475 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.887 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 -- 172.21.15.50:0/2020120475 >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 msgr2=0x7f58400c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:09.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 --2- 172.21.15.50:0/2020120475 >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 0x7f58400c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 -- 172.21.15.50:0/2020120475 >> v1:172.21.15.125:6789/0 conn(0x7f58400c9d00 legacy=0x7f58400c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 -- 172.21.15.50:0/2020120475 --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400ca260 con 0x7f584013f860 2023-12-19T16:14:09.888 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583f7fe700 1 --2- 172.21.15.50:0/2020120475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 0x7f58400c80b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5838009e60 tx=0x7f5838009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60a346c97ca18e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583dffb700 1 -- 172.21.15.50:0/2020120475 <== mon.2 v2:172.21.15.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583800e070 con 0x7f584013f860 2023-12-19T16:14:09.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583dffb700 1 -- 172.21.15.50:0/2020120475 <== mon.2 v2:172.21.15.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583800bb60 con 0x7f584013f860 2023-12-19T16:14:09.889 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.882+0000 7f583dffb700 1 -- 172.21.15.50:0/2020120475 <== mon.2 v2:172.21.15.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838012430 con 0x7f584013f860 2023-12-19T16:14:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 -- 172.21.15.50:0/2020120475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 msgr2=0x7f58400c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 --2- 172.21.15.50:0/2020120475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 0x7f58400c80b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5838009e60 tx=0x7f5838009780 comp rx=0 tx=0).stop 2023-12-19T16:14:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 -- 172.21.15.50:0/2020120475 shutdown_connections 2023-12-19T16:14:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 --2- 172.21.15.50:0/2020120475 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f584013f860 0x7f58400c80b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.890 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 --2- 172.21.15.50:0/2020120475 >> v2:172.21.15.50:3300/0 conn(0x7f584013f490 0x7f58400c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 -- 172.21.15.50:0/2020120475 >> 172.21.15.50:0/2020120475 conn(0x7f58400cc6c0 msgr2=0x7f58400c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 -- 172.21.15.50:0/2020120475 shutdown_connections 2023-12-19T16:14:09.891 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.886+0000 7f58456f0700 1 -- 172.21.15.50:0/2020120475 wait complete. 2023-12-19T16:14:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 Processor -- start 2023-12-19T16:14:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 -- start start 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58400c9d00 0x7f5840158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f584013f490 0x7f58400afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840157f80 con 0x7f584013f860 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840158950 con 0x7f584013f490 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f58456f0700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584015ba20 con 0x7f58400c9d00 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59990/0 (socket says 172.21.15.50:59990) 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f583f7fe700 1 -- 172.21.15.50:0/1828639528 learned_addr learned my addr 172.21.15.50:0/1828639528 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f583effd700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58400c9d00 0x7f5840158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.897 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.890+0000 7f583e7fc700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f584013f490 0x7f58400afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 -- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58400c9d00 msgr2=0x7f5840158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58400c9d00 0x7f5840158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 -- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f584013f490 msgr2=0x7f58400afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:09.898 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f584013f490 0x7f58400afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 -- 172.21.15.50:0/1828639528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009a20 con 0x7f584013f860 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f583f7fe700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f58400c2a90 tx=0x7f5838019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838009420 con 0x7f584013f860 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838027470 con 0x7f584013f860 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838027810 con 0x7f584013f860 2023-12-19T16:14:09.899 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5840157650 con 0x7f584013f860 2023-12-19T16:14:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5840157c10 con 0x7f584013f860 2023-12-19T16:14:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.894+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5840158130 con 0x7f584013f860 2023-12-19T16:14:09.902 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.898+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5838024070 con 0x7f584013f860 2023-12-19T16:14:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.898+0000 7f5826ffd700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 0x7f582c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:09.904 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.898+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f583807f050 con 0x7f584013f860 2023-12-19T16:14:09.905 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.898+0000 7f583effd700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 0x7f582c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:09.906 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.902+0000 7f583effd700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 0x7f582c062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5840170c30 tx=0x7f5830009250 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:09.911 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:09.906+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f58380aebe0 con 0x7f584013f860 2023-12-19T16:14:10.085 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.078+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f5840071110 con 0x7f582c0606a0 2023-12-19T16:14:10.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.082+0000 7f5826ffd700 1 -- 172.21.15.50:0/1828639528 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5840071110 con 0x7f582c0606a0 2023-12-19T16:14:10.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 msgr2=0x7f582c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 0x7f582c062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5840170c30 tx=0x7f5830009250 comp rx=0 tx=0).stop 2023-12-19T16:14:10.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 msgr2=0x7f58400c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f58400c2a90 tx=0x7f5838019000 comp rx=0 tx=0).stop 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 shutdown_connections 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f582c0606a0 0x7f582c062b50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f584013f490 0x7f58400afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f584013f860 0x7f58400c50b0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 --2- 172.21.15.50:0/1828639528 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f58400c9d00 0x7f5840158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 >> 172.21.15.50:0/1828639528 conn(0x7f58400cc6c0 msgr2=0x7f584006a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 shutdown_connections 2023-12-19T16:14:10.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.086+0000 7f58456f0700 1 -- 172.21.15.50:0/1828639528 wait complete. 2023-12-19T16:14:10.094 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 6 to deep-scrub 2023-12-19T16:14:10.112 INFO:tasks.ceph:Scrubbing osd.7 2023-12-19T16:14:10.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-19T16:14:10.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.290+0000 7f999ec0c700 1 Processor -- start 2023-12-19T16:14:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.294+0000 7f999ec0c700 1 -- start start 2023-12-19T16:14:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999ec0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 0x7f999815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999ec0c700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9998158b20 0x7f9998158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999ec0c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980cea80 con 0x7f999815cee0 2023-12-19T16:14:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999ec0c700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980b0b40 con 0x7f9998158b20 2023-12-19T16:14:10.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999ec0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99980ced60 con 0x7f99980cb3a0 2023-12-19T16:14:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999dc0a700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f999815cee0 0x7f99980ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44946/0 (socket says 172.21.15.50:44946) 2023-12-19T16:14:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999dc0a700 1 -- 172.21.15.50:0/2557507926 learned_addr learned my addr 172.21.15.50:0/2557507926 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 --2- 172.21.15.50:0/2557507926 >> v2:172.21.15.125:3300/0 conn(0x7f9998158b20 0x7f9998158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999d409700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 0x7f999815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 -- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 msgr2=0x7f999815d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 0x7f999815d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 -- 172.21.15.50:0/2557507926 >> v1:172.21.15.50:6789/0 conn(0x7f999815cee0 legacy=0x7f99980ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 -- 172.21.15.50:0/2557507926 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980cf040 con 0x7f9998158b20 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999d409700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 0x7f999815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:10.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.298+0000 7f999e40b700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9998158b20 0x7f9998158fc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9990009e60 tx=0x7f9990009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=169ae1be4a4a1568 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999cc08700 1 -- 172.21.15.50:0/2557507926 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999000e070 con 0x7f9998158b20 2023-12-19T16:14:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999cc08700 1 -- 172.21.15.50:0/2557507926 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999000bc30 con 0x7f9998158b20 2023-12-19T16:14:10.310 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999cc08700 1 -- 172.21.15.50:0/2557507926 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990012430 con 0x7f9998158b20 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 -- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9998158b20 msgr2=0x7f9998158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9998158b20 0x7f9998158fc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9990009e60 tx=0x7f9990009780 comp rx=0 tx=0).stop 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 -- 172.21.15.50:0/2557507926 shutdown_connections 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9998158b20 0x7f9998158fc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 --2- 172.21.15.50:0/2557507926 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f99980cb3a0 0x7f999815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 -- 172.21.15.50:0/2557507926 >> 172.21.15.50:0/2557507926 conn(0x7f99980cff20 msgr2=0x7f99980d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 -- 172.21.15.50:0/2557507926 shutdown_connections 2023-12-19T16:14:10.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 -- 172.21.15.50:0/2557507926 wait complete. 2023-12-19T16:14:10.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.306+0000 7f999ec0c700 1 Processor -- start 2023-12-19T16:14:10.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 -- start start 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 0x7f999815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9998158b20 0x7f999815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f999815cee0 0x7f99980c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998156d90 con 0x7f9998158b20 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999815afa0 con 0x7f99980cb3a0 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999ec0c700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999815b200 con 0x7f999815cee0 2023-12-19T16:14:10.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 0x7f999815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999d409700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9998158b20 0x7f999815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 0x7f999815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.50:50816/0 (socket says 172.21.15.50:50816) 2023-12-19T16:14:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 -- 172.21.15.50:0/375305479 learned_addr learned my addr 172.21.15.50:0/375305479 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999e40b700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f999815cee0 0x7f99980c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 -- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f999815cee0 msgr2=0x7f99980c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f999815cee0 0x7f99980c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 -- 172.21.15.50:0/375305479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9998158b20 msgr2=0x7f999815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9998158b20 0x7f999815aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990009a20 con 0x7f99980cb3a0 2023-12-19T16:14:10.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.310+0000 7f999dc0a700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 0x7f999815a520 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f999400c560 tx=0x7f9994001020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9994005b30 con 0x7f99980cb3a0 2023-12-19T16:14:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9994005c90 con 0x7f99980cb3a0 2023-12-19T16:14:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9994007320 con 0x7f99980cb3a0 2023-12-19T16:14:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999815b4e0 con 0x7f99980cb3a0 2023-12-19T16:14:10.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999815bb20 con 0x7f99980cb3a0 2023-12-19T16:14:10.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.314+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f999804be40 con 0x7f99980cb3a0 2023-12-19T16:14:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f999401b3b0 con 0x7f99980cb3a0 2023-12-19T16:14:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f998dffb700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f997c060740 0x7f997c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f999407be50 con 0x7f99980cb3a0 2023-12-19T16:14:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f999d409700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f997c060740 0x7f997c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f998dffb700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7f997c0670b0 0x7f997c069540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.318+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c069c10 con 0x7f997c0670b0 2023-12-19T16:14:10.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.322+0000 7f999e40b700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7f997c0670b0 0x7f997c069540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.322+0000 7f999d409700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f997c060740 0x7f997c062bf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99980c9fc0 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.322+0000 7f999e40b700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7f997c0670b0 0x7f997c069540 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.329 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.322+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== osd.7 v2:172.21.15.125:6809/34167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f997c069c10 con 0x7f997c0670b0 2023-12-19T16:14:10.330 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.326+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=218) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9994001160 con 0x7f99980cb3a0 2023-12-19T16:14:10.348 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.342+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 --> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f999815be40 con 0x7f997c0670b0 2023-12-19T16:14:10.349 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.342+0000 7f998dffb700 1 -- 172.21.15.50:0/375305479 <== osd.7 v2:172.21.15.125:6809/34167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f999815be40 con 0x7f997c0670b0 2023-12-19T16:14:10.350 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2023-12-19T16:14:10.350 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T16:14:10.350 INFO:teuthology.orchestra.run.smithi050.stdout:} 2023-12-19T16:14:10.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7f997c0670b0 msgr2=0x7f997c069540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.350 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6809/34167,v1:172.21.15.125:6811/34167] conn(0x7f997c0670b0 0x7f997c069540 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f997c060740 msgr2=0x7f997c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.351 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f997c060740 0x7f997c062bf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99980c9fc0 tx=0x7f9988008040 comp rx=0 tx=0).stop 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 msgr2=0x7f999815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 --2- 172.21.15.50:0/375305479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99980cb3a0 0x7f999815a520 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f999400c560 tx=0x7f9994001020 comp rx=0 tx=0).stop 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999e40b700 1 -- 172.21.15.50:0/375305479 reap_dead start 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 shutdown_connections 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 >> 172.21.15.50:0/375305479 conn(0x7f99980cff20 msgr2=0x7f99980ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 shutdown_connections 2023-12-19T16:14:10.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.346+0000 7f999ec0c700 1 -- 172.21.15.50:0/375305479 wait complete. 2023-12-19T16:14:10.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-19T16:14:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.546+0000 7fe2745c5700 1 Processor -- start 2023-12-19T16:14:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.546+0000 7fe2745c5700 1 -- start start 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe26c15a9d0 0x7fe26c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c069d40 con 0x7fe26c156fc0 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c157a20 con 0x7fe26c0c7350 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c157d00 con 0x7fe26c15a9d0 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe26c15a9d0 0x7fe26c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe26c15a9d0 0x7fe26c1582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59998/0 (socket says 172.21.15.50:59998) 2023-12-19T16:14:10.554 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 -- 172.21.15.50:0/804469965 learned_addr learned my addr 172.21.15.50:0/804469965 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 -- 172.21.15.50:0/804469965 >> v1:172.21.15.125:6789/0 conn(0x7fe26c0c7350 legacy=0x7fe26c06a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 -- 172.21.15.50:0/804469965 >> v1:172.21.15.50:6789/0 conn(0x7fe26c156fc0 legacy=0x7fe26c0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 -- 172.21.15.50:0/804469965 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c157fe0 con 0x7fe26c15a9d0 2023-12-19T16:14:10.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe272b62700 1 --2- 172.21.15.50:0/804469965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c15a9d0 0x7fe26c1582c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe264009f20 tx=0x7fe264009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cda9be0b52b6b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe27135f700 1 -- 172.21.15.50:0/804469965 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26400e070 con 0x7fe26c15a9d0 2023-12-19T16:14:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe27135f700 1 -- 172.21.15.50:0/804469965 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26400ba80 con 0x7fe26c15a9d0 2023-12-19T16:14:10.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe27135f700 1 -- 172.21.15.50:0/804469965 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2640092e0 con 0x7fe26c15a9d0 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- 172.21.15.50:0/804469965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c15a9d0 msgr2=0x7fe26c1582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 --2- 172.21.15.50:0/804469965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c15a9d0 0x7fe26c1582c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe264009f20 tx=0x7fe264009840 comp rx=0 tx=0).stop 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- 172.21.15.50:0/804469965 shutdown_connections 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 --2- 172.21.15.50:0/804469965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c15a9d0 0x7fe26c1582c0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- 172.21.15.50:0/804469965 >> 172.21.15.50:0/804469965 conn(0x7fe26c0cc6c0 msgr2=0x7fe26c0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- 172.21.15.50:0/804469965 shutdown_connections 2023-12-19T16:14:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.550+0000 7fe2745c5700 1 -- 172.21.15.50:0/804469965 wait complete. 2023-12-19T16:14:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 Processor -- start 2023-12-19T16:14:10.559 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 -- start start 2023-12-19T16:14:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 0x7fe26c0ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe26c15a9d0 0x7fe26c0c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0c5db0 con 0x7fe26c0c7350 2023-12-19T16:14:10.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0af0f0 con 0x7fe26c156fc0 2023-12-19T16:14:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe2745c5700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0af3d0 con 0x7fe26c15a9d0 2023-12-19T16:14:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.554+0000 7fe272361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe271b60700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 0x7fe26c0ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60014/0 (socket says 172.21.15.50:60014) 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 -- 172.21.15.50:0/595245790 learned_addr learned my addr 172.21.15.50:0/595245790 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272b62700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe26c15a9d0 0x7fe26c0c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 -- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe26c15a9d0 msgr2=0x7fe26c0c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe26c15a9d0 0x7fe26c0c5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 -- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 msgr2=0x7fe26c0ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 0x7fe26c0ada50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 -- 172.21.15.50:0/595245790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264009ae0 con 0x7fe26c0c7350 2023-12-19T16:14:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe271b60700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 0x7fe26c0ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe272361700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c590 tx=0x7fe26800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268011590 con 0x7fe26c0c7350 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268011710 con 0x7fe26c0c7350 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268011ad0 con 0x7fe26c0c7350 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c0af6b0 con 0x7fe26c0c7350 2023-12-19T16:14:10.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c0afc70 con 0x7fe26c0c7350 2023-12-19T16:14:10.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.558+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c071110 con 0x7fe26c0c7350 2023-12-19T16:14:10.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.562+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe26801b3b0 con 0x7fe26c0c7350 2023-12-19T16:14:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.562+0000 7fe2627fc700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 0x7fe250062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.562+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fe26807bf50 con 0x7fe26c0c7350 2023-12-19T16:14:10.568 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.562+0000 7fe271b60700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 0x7fe250062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.566+0000 7fe271b60700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 0x7fe250062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe26c16c9e0 tx=0x7fe25c009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.575 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.570+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe26804c280 con 0x7fe26c0c7350 2023-12-19T16:14:10.748 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.742+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fe26c0af540 con 0x7fe250060670 2023-12-19T16:14:10.749 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.742+0000 7fe2627fc700 1 -- 172.21.15.50:0/595245790 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe26c0af540 con 0x7fe250060670 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 msgr2=0x7fe250062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 0x7fe250062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe26c16c9e0 tx=0x7fe25c009040 comp rx=0 tx=0).stop 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 msgr2=0x7fe26c0aea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c590 tx=0x7fe26800c8a0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 shutdown_connections 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fe250060670 0x7fe250062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c156fc0 0x7fe26c0ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fe26c15a9d0 0x7fe26c0c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 --2- 172.21.15.50:0/595245790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe26c0c7350 0x7fe26c0aea10 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 >> 172.21.15.50:0/595245790 conn(0x7fe26c0cc6c0 msgr2=0x7fe26c0c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 shutdown_connections 2023-12-19T16:14:10.752 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.746+0000 7fe2745c5700 1 -- 172.21.15.50:0/595245790 wait complete. 2023-12-19T16:14:10.754 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 7 to deep-scrub 2023-12-19T16:14:10.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:14:10.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 Processor -- start 2023-12-19T16:14:10.942 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 -- start start 2023-12-19T16:14:10.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f79580ad980 0x7f79580ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79580c78e0 0x7f79580c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958067da0 con 0x7f7958155f20 2023-12-19T16:14:10.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79580c8210 con 0x7f79580ad980 2023-12-19T16:14:10.945 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795ca6e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7958068940 con 0x7f79580c78e0 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795659c700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7958155f20 0x7f7958068230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44966/0 (socket says 172.21.15.50:44966) 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f795659c700 1 -- 172.21.15.50:0/286114219 learned_addr learned my addr 172.21.15.50:0/286114219 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f7955d9b700 1 --2- 172.21.15.50:0/286114219 >> v2:172.21.15.50:3300/0 conn(0x7f79580ad980 0x7f79580ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.938+0000 7f7956d9d700 1 --2- 172.21.15.50:0/286114219 >> v2:172.21.15.125:3300/0 conn(0x7f79580c78e0 0x7f79580c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795559a700 1 -- 172.21.15.50:0/286114219 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707179505 0 0) 0x7f7958067da0 con 0x7f7958155f20 2023-12-19T16:14:10.946 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795559a700 1 -- 172.21.15.50:0/286114219 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940003570 con 0x7f7958155f20 2023-12-19T16:14:10.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f7955d9b700 1 -- 172.21.15.50:0/286114219 >> v2:172.21.15.125:3300/0 conn(0x7f79580c78e0 msgr2=0x7f79580c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:10.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f7955d9b700 1 --2- 172.21.15.50:0/286114219 >> v2:172.21.15.125:3300/0 conn(0x7f79580c78e0 0x7f79580c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f7955d9b700 1 -- 172.21.15.50:0/286114219 >> v1:172.21.15.50:6789/0 conn(0x7f7958155f20 legacy=0x7f7958068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f7955d9b700 1 -- 172.21.15.50:0/286114219 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958067190 con 0x7f79580ad980 2023-12-19T16:14:10.947 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f7955d9b700 1 --2- 172.21.15.50:0/286114219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79580ad980 0x7f79580ade00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f795806d740 tx=0x7f794c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ae335b3ca67b25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795559a700 1 -- 172.21.15.50:0/286114219 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c011630 con 0x7f79580ad980 2023-12-19T16:14:10.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795559a700 1 -- 172.21.15.50:0/286114219 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c0117b0 con 0x7f79580ad980 2023-12-19T16:14:10.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795559a700 1 -- 172.21.15.50:0/286114219 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c011b70 con 0x7f79580ad980 2023-12-19T16:14:10.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 -- 172.21.15.50:0/286114219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79580ad980 msgr2=0x7f79580ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.948 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 --2- 172.21.15.50:0/286114219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79580ad980 0x7f79580ade00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f795806d740 tx=0x7f794c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 -- 172.21.15.50:0/286114219 shutdown_connections 2023-12-19T16:14:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 --2- 172.21.15.50:0/286114219 >> v2:172.21.15.125:3300/0 conn(0x7f79580c78e0 0x7f79580c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 --2- 172.21.15.50:0/286114219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79580ad980 0x7f79580ade00 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.949 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 -- 172.21.15.50:0/286114219 >> 172.21.15.50:0/286114219 conn(0x7f79580cc690 msgr2=0x7f79580c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 -- 172.21.15.50:0/286114219 shutdown_connections 2023-12-19T16:14:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.942+0000 7f795ca6e700 1 -- 172.21.15.50:0/286114219 wait complete. 2023-12-19T16:14:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 Processor -- start 2023-12-19T16:14:10.951 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 -- start start 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79580ad980 0x7f7958154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 0x7f7958157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580c2140 con 0x7f7958155f20 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958157180 con 0x7f79580c78e0 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f795ca6e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79580c2950 con 0x7f79580ad980 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.946+0000 7f7956d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60028/0 (socket says 172.21.15.50:60028) 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 -- 172.21.15.50:0/246952661 learned_addr learned my addr 172.21.15.50:0/246952661 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7955d9b700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 0x7f7958157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f795659c700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79580ad980 0x7f7958154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 -- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79580ad980 msgr2=0x7f7958154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79580ad980 0x7f7958154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 -- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 msgr2=0x7f7958157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 0x7f7958157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 -- 172.21.15.50:0/246952661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c009420 con 0x7f7958155f20 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7955d9b700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 0x7f7958157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f7956d9d700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7950009ec0 tx=0x7f795000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795000cc20 con 0x7f7958155f20 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795000f040 con 0x7f7958155f20 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950014480 con 0x7f7958155f20 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79580c2390 con 0x7f7958155f20 2023-12-19T16:14:10.956 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7958156e10 con 0x7f7958155f20 2023-12-19T16:14:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.950+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79581572f0 con 0x7f7958155f20 2023-12-19T16:14:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.954+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7950014830 con 0x7f7958155f20 2023-12-19T16:14:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.954+0000 7f79467fc700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 0x7f7938062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.954+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f795007d600 con 0x7f7958155f20 2023-12-19T16:14:10.960 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.954+0000 7f795659c700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 0x7f7938062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.954+0000 7f795659c700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 0x7f7938062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7948007fd0 tx=0x7f7948009040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:10.962+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f79500512b0 con 0x7f7958155f20 2023-12-19T16:14:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.142+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7958071110 con 0x7f7938060670 2023-12-19T16:14:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f79467fc700 1 -- 172.21.15.50:0/246952661 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47695 (secure 0 0 0) 0x7f7958071110 con 0x7f7938060670 2023-12-19T16:14:11.154 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 msgr2=0x7f7938062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 0x7f7938062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7948007fd0 tx=0x7f7948009040 comp rx=0 tx=0).stop 2023-12-19T16:14:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 msgr2=0x7f79580bf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7950009ec0 tx=0x7f795000b040 comp rx=0 tx=0).stop 2023-12-19T16:14:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 shutdown_connections 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f7938060670 0x7f7938062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79580c78e0 0x7f7958157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958155f20 0x7f79580bf850 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 --2- 172.21.15.50:0/246952661 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f79580ad980 0x7f7958154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 >> 172.21.15.50:0/246952661 conn(0x7f79580cc690 msgr2=0x7f795806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 shutdown_connections 2023-12-19T16:14:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:11.150+0000 7f795ca6e700 1 -- 172.21.15.50:0/246952661 wait complete. 2023-12-19T16:14:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:14:11.177 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-19T16:14:09.705857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470496,"kb_used_data":22560,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752504224,"statfs":{"total":773094113280,"available":770564325376,"internally_reserved":0,"allocated":23101440,"data_stored":12387306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003061"},"pg_stats":[{"pgid":"1.0","version":"215'370","reported_seq":1221,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-12-19T16:13:56.770416+0000","last_change":"2023-12-19T14:38:57.644568+0000","last_active":"2023-12-19T16:13:56.770416+0000","last_peered":"2023-12-19T16:13:56.770416+0000","last_clean":"2023-12-19T16:13:56.770416+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:13:56.770416+0000","last_undegraded":"2023-12-19T16:13:56.770416+0000","last_fullsized":"2023-12-19T16:13:56.770416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T14:38:56.213864+0000","last_clean_scrub_stamp":"2023-12-19T14:38:56.213864+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:03:09.380886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575979,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304640,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067200,"statfs":{"total":96636764160,"available":96324812800,"internally_reserved":0,"allocated":3334144,"data_stored":1991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0900000000000001,"15min":1.115},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.3340000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.69499999999999995,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.389,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.125,"15min":1.135},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.329,"5min":1.623,"15min":1.903},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.2230000000000001,"15min":1.232},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.49,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":1.034}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.70299999999999996},"max":{"1min":1.5960000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.464},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.306,"15min":1.319},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.76300000000000001},"max":{"1min":1.5269999999999999,"5min":1.738,"15min":1.7709999999999999},"last":1.276}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.298,"15min":1.331},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.6659999999999999,"5min":1.736,"15min":1.839},"last":1.708},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3440000000000001,"15min":1.3620000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.6020000000000001,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.389}]},{"osd":4,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2230000000000001,"15min":1.179},"min":{"1min":0.92000000000000004,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.597,"5min":1.8280000000000001,"15min":1.849},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.2370000000000001,"15min":1.2330000000000001},"min":{"1min":0.89900000000000002,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":1.716,"5min":1.716,"15min":1.784},"last":1.637}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2549999999999999,"15min":1.25},"min":{"1min":0.95299999999999996,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":1.821,"15min":1.825},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.377,"15min":1.3480000000000001},"min":{"1min":1.119,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.7290000000000001,"5min":1.9199999999999999,"15min":2.0030000000000001},"last":1.425}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4390000000000001,"15min":1.431},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.8280000000000001,"5min":1.994,"15min":1.994},"last":1.506},{"interface":"front","average":{"1min":1.415,"5min":1.4099999999999999,"15min":1.3999999999999999},"min":{"1min":0.96599999999999997,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7829999999999999,"5min":1.9410000000000001,"15min":2.1059999999999999},"last":1.7629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.153,"15min":1.1699999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.60099999999999998},"max":{"1min":1.5580000000000001,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":1.488},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.216,"15min":1.236},"min":{"1min":0.84099999999999997,"5min":0.435,"15min":0.435},"max":{"1min":1.637,"5min":1.74,"15min":1.7909999999999999},"last":1.181}]},{"osd":1,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1899999999999999,"15min":1.194},"min":{"1min":0.86399999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5469999999999999,"5min":1.643,"15min":1.7010000000000001},"last":1.266},{"interface":"front","average":{"1min":1.179,"5min":1.2410000000000001,"15min":1.2689999999999999},"min":{"1min":0.73399999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5449999999999999,"5min":1.806,"15min":1.827},"last":1.4059999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1779999999999999,"15min":1.22},"min":{"1min":0.73999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4530000000000001,"5min":1.651,"15min":1.772},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.93999999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.5369999999999999,"5min":1.732,"15min":1.732},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2909999999999999,"15min":1.3280000000000001},"min":{"1min":0.81299999999999994,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6659999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2889999999999999,"15min":1.3220000000000001},"min":{"1min":0.76300000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7370000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2889999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.208,"15min":1.2050000000000001},"min":{"1min":0.92100000000000004,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.6379999999999999,"5min":1.708,"15min":1.774},"last":1.371},{"interface":"front","average":{"1min":1.292,"5min":1.2729999999999999,"15min":1.3069999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.7729999999999999,"5min":1.966,"15min":1.966},"last":1.056}]},{"osd":5,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3200000000000001,"15min":1.3109999999999999},"min":{"1min":0.91300000000000003,"5min":0.88200000000000001,"15min":0.71499999999999997},"max":{"1min":1.7869999999999999,"5min":1.788,"15min":1.923},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.363,"15min":1.3640000000000001},"min":{"1min":1.0840000000000001,"5min":0.97899999999999998,"15min":0.68899999999999995},"max":{"1min":1.7430000000000001,"5min":1.8100000000000001,"15min":2.1179999999999999},"last":1.5229999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.4219999999999999,"15min":1.4319999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.752,"5min":1.839,"15min":2.169},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.454,"15min":1.456},"min":{"1min":0.96699999999999997,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.9419999999999999,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":0.96699999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308156,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063684,"statfs":{"total":96636764160,"available":96321212416,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0800000000000001,"15min":1.1000000000000001},"min":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.714,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.143,"15min":1.1879999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.544,"5min":1.6619999999999999,"15min":1.907},"last":1.29}]},{"osd":1,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.228,"15min":1.2989999999999999},"min":{"1min":0.73999999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.4670000000000001,"5min":1.6759999999999999,"15min":1.7490000000000001},"last":1.27},{"interface":"front","average":{"1min":1.139,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.77800000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.5549999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.429}]},{"osd":2,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.83599999999999997,"15min":0.66300000000000003},"max":{"1min":1.454,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.1890000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4379999999999999,"5min":1.7769999999999999,"15min":1.831},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2589999999999999,"15min":1.296},"min":{"1min":0.83099999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5529999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.365},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.55,"5min":1.734,"15min":2.0449999999999999},"last":1.599}]},{"osd":4,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1419999999999999,"15min":1.141},"min":{"1min":0.74199999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.278,"15min":1.2709999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.6220000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.355,"15min":1.3680000000000001},"min":{"1min":0.92500000000000004,"5min":0.878,"15min":0.49099999999999999},"max":{"1min":1.625,"5min":1.843,"15min":1.968},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.393,"15min":1.3999999999999999},"min":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.60599999999999998},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":1.333}]},{"osd":7,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4239999999999999,"15min":1.413},"min":{"1min":1.0609999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7589999999999999,"5min":2.395,"15min":2.395},"last":1.504},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":0.92400000000000004,"5min":0.80400000000000005,"15min":0.55500000000000005},"max":{"1min":1.7609999999999999,"5min":1.8440000000000001,"15min":1.9770000000000001},"last":1.105}]}]},{"osd":4,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.109,"15min":1.1160000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5820000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.133,"5min":1.1399999999999999,"15min":1.153},"min":{"1min":0.90400000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.022}]},{"osd":1,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3089999999999999,"15min":1.306},"min":{"1min":0.97999999999999998,"5min":0.81999999999999995,"15min":0.81899999999999995},"max":{"1min":1.796,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.395},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.94399999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.615,"15min":1.7969999999999999},"last":1.361}]},{"osd":2,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.232},"min":{"1min":0.86599999999999999,"5min":0.67400000000000004,"15min":0.67000000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.73},"last":1.073},{"interface":"front","average":{"1min":1.212,"5min":1.2410000000000001,"15min":1.252},"min":{"1min":0.86399999999999999,"5min":0.78300000000000003,"15min":0.504},"max":{"1min":1.5800000000000001,"5min":1.73,"15min":1.764},"last":1.496}]},{"osd":3,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2150000000000001,"15min":1.226},"min":{"1min":0.92300000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.621,"5min":1.804,"15min":1.804},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.337,"15min":1.3180000000000001},"min":{"1min":1.0840000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":1.5569999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.81100000000000005,"5min":0.79700000000000004,"15min":0.53700000000000003},"max":{"1min":1.6559999999999999,"5min":1.744,"15min":1.855},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.2270000000000001,"15min":1.1970000000000001},"min":{"1min":0.85199999999999998,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.653,"5min":2.089,"15min":2.089},"last":1.8049999999999999}]},{"osd":6,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.389,"15min":1.321},"min":{"1min":0.997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.4710000000000001,"15min":1.448},"min":{"1min":1.2210000000000001,"5min":0.85299999999999998,"15min":0.59199999999999997},"max":{"1min":1.7789999999999999,"5min":1.97,"15min":1.982},"last":1.8420000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.407,"15min":1.3620000000000001},"min":{"1min":1.103,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.7450000000000001,"5min":1.982,"15min":1.982},"last":1.752},{"interface":"front","average":{"1min":1.609,"5min":1.5529999999999999,"15min":1.4990000000000001},"min":{"1min":1.3660000000000001,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":1.9890000000000001,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.9219999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299964,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071876,"statfs":{"total":96636764160,"available":96329601024,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":0.90600000000000003},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2609999999999999,"5min":1.633,"15min":1.7729999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96899999999999997,"15min":1.0009999999999999},"min":{"1min":0.66000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.9370000000000001},"last":1.137}]},{"osd":1,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.96299999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.53900000000000003},"max":{"1min":1.716,"5min":1.716,"15min":2.0350000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.147,"15min":1.119},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.379,"5min":1.724,"15min":2.2269999999999999},"last":0.95099999999999996}]},{"osd":2,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1679999999999999},"min":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.726,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.71599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.643,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":1.165}]},{"osd":4,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.81599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.631,"5min":1.718,"15min":2.5099999999999998},"last":1.224},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.876,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.788,"15min":2.3450000000000002},"last":1.004}]},{"osd":5,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3600000000000001,"15min":1.355},"min":{"1min":0.93300000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.6679999999999999,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.085},{"interface":"front","average":{"1min":1.393,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":0.85299999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0179999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":1.286}]},{"osd":6,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.04,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.9830000000000001,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.448,"15min":1.458},"min":{"1min":0.879,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.798,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":1.3260000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5449999999999999,"15min":1.55},"min":{"1min":0.87,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8720000000000001,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.208}]}]},{"osd":2,"up_from":13,"seq":55834575980,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312248,"kb_used_data":2672,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059592,"statfs":{"total":96636764160,"available":96317022208,"internally_reserved":0,"allocated":2736128,"data_stored":1400063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94599999999999995,"15min":0.96099999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.1160000000000001,"15min":1.1439999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.3520000000000001,"5min":1.629,"15min":1.796},"last":0.96199999999999997}]},{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4059999999999999,"5min":2.327,"15min":2.327},"last":1.458},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.143,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.399,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.86099999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0069999999999999,"15min":1.044},"min":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.5489999999999999,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.105,"15min":1.137},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3979999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.0800000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":1.0620000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.569,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.89300000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.786,"5min":2.218,"15min":2.218},"last":1.655}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.395,"15min":1.4099999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6910000000000001,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.722},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.377,"15min":1.4039999999999999},"min":{"1min":0.91600000000000004,"5min":0.879,"15min":0.879},"max":{"1min":1.5940000000000001,"5min":2.0049999999999999,"15min":2.4340000000000002},"last":1.369}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.0099999999999998,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.425},{"interface":"front","average":{"1min":1.333,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8500000000000001,"5min":2.609,"15min":2.609},"last":1.147}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5349999999999999,"15min":1.5509999999999999},"min":{"1min":1.054,"5min":1.054,"15min":0.89600000000000002},"max":{"1min":1.833,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.5269999999999999,"15min":1.5409999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.9319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.335}]}]},{"osd":1,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316348,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055492,"statfs":{"total":96636764160,"available":96312823808,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.874,"15min":0.90000000000000002},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.4690000000000001,"15min":1.6579999999999999},"last":0.754},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.97399999999999998,"15min":1.032},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.454,"5min":1.51,"15min":1.9119999999999999},"last":1.026}]},{"osd":2,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0449999999999999,"15min":1.0669999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.643,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1,"5min":1.093,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.641,"5min":1.823,"15min":1.823},"last":1.2709999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.242,"15min":1.2789999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.665,"5min":1.778,"15min":1.8180000000000001},"last":1.446},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0960000000000001,"15min":1.105},"min":{"1min":0.65600000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.0820000000000001}]},{"osd":4,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3120000000000001,"15min":1.3260000000000001},"min":{"1min":1,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.395,"5min":1.391,"15min":1.423},"min":{"1min":1.014,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":2.0110000000000001},"last":1.355}]},{"osd":5,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3009999999999999,"15min":1.337},"min":{"1min":1.0049999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.597,"5min":1.833,"15min":1.897},"last":1.405},{"interface":"front","average":{"1min":1.397,"5min":1.395,"15min":1.4370000000000001},"min":{"1min":1.038,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.919,"5min":2.0579999999999998,"15min":2.165},"last":1.381}]},{"osd":6,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3460000000000001,"15min":1.397},"min":{"1min":0.97299999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5820000000000001,"5min":1.9590000000000001,"15min":2.1139999999999999},"last":1.323},{"interface":"front","average":{"1min":1.464,"5min":1.4770000000000001,"15min":1.512},"min":{"1min":1.1180000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.302}]},{"osd":7,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.494,"15min":1.53},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":1.548},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4510000000000001,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":1.2370000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575978,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304636,"kb_used_data":3252,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067204,"statfs":{"total":96636764160,"available":96324816896,"internally_reserved":0,"allocated":3330048,"data_stored":1990431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.439},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.6419999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0069999999999999,"15min":1.008},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4259999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.78100000000000003}]},{"osd":2,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.141,"15min":1.147},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.57099999999999995},"max":{"1min":1.605,"5min":1.748,"15min":1.8400000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.54300000000000004},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8500000000000001},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.02,"15min":1.0409999999999999},"min":{"1min":0.60499999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.448,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.55300000000000005},"max":{"1min":1.7290000000000001,"5min":1.974,"15min":1.974},"last":1.198}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.873,"5min":0.873,"15min":0.69499999999999995},"max":{"1min":1.8120000000000001,"5min":1.885,"15min":2.0209999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.339,"15min":1.331},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71999999999999997},"max":{"1min":1.762,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.367}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.071,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.6870000000000001,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.306},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4419999999999999,"15min":1.456},"min":{"1min":1.056,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.712,"5min":1.919,"15min":1.9810000000000001},"last":1.28}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.738,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.454},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.93899999999999995},"max":{"1min":1.714,"5min":2.0009999999999999,"15min":2.1600000000000001},"last":1.421}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5249999999999999,"15min":1.528},"min":{"1min":1.103,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.486,"15min":1.502},"min":{"1min":1.01,"5min":1.01,"15min":1.008},"max":{"1min":1.8620000000000001,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.5509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:14:11.181 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-19T14:38:56.213864+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=19, tm_hour=14, tm_min=38, tm_sec=56, tm_wday=1, tm_yday=353, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=19, tm_hour=16, tm_min=14, tm_sec=4, tm_wday=1, tm_yday=353, tm_isdst=0) 2023-12-19T16:14:11.181 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-19T16:14:30.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:14:30.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:14:31.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T16:14:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.346+0000 7fdb9a82e700 1 Processor -- start 2023-12-19T16:14:31.352 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.346+0000 7fdb9a82e700 1 -- start start 2023-12-19T16:14:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb9a82e700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdb940cc2f0 0x7fdb940af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb9a82e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb940674f0 0x7fdb94067910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb9a82e700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb940afc30 con 0x7fdb940b1dd0 2023-12-19T16:14:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb9a82e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb940b1310 con 0x7fdb940674f0 2023-12-19T16:14:31.354 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb9a82e700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb94156fe0 con 0x7fdb940cc2f0 2023-12-19T16:14:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb940674f0 0x7fdb94067910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fdb940674f0 0x7fdb94067910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52658/0 (socket says 172.21.15.50:52658) 2023-12-19T16:14:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 -- 172.21.15.50:0/161135075 learned_addr learned my addr 172.21.15.50:0/161135075 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:31.355 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb93fff700 1 --2- 172.21.15.50:0/161135075 >> v2:172.21.15.125:3300/0 conn(0x7fdb940cc2f0 0x7fdb940af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 -- 172.21.15.50:0/161135075 >> v2:172.21.15.125:3300/0 conn(0x7fdb940cc2f0 msgr2=0x7fdb940af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 --2- 172.21.15.50:0/161135075 >> v2:172.21.15.125:3300/0 conn(0x7fdb940cc2f0 0x7fdb940af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 -- 172.21.15.50:0/161135075 >> v1:172.21.15.125:6789/0 conn(0x7fdb940b1dd0 legacy=0x7fdb940b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 -- 172.21.15.50:0/161135075 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb94067060 con 0x7fdb940674f0 2023-12-19T16:14:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb98dcb700 1 --2- 172.21.15.50:0/161135075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940674f0 0x7fdb94067910 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fdb940cad50 tx=0x7fdb8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce861ee75498c9c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb937fe700 1 -- 172.21.15.50:0/161135075 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c00b860 con 0x7fdb940674f0 2023-12-19T16:14:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb937fe700 1 -- 172.21.15.50:0/161135075 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c00b9c0 con 0x7fdb940674f0 2023-12-19T16:14:31.357 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.350+0000 7fdb937fe700 1 -- 172.21.15.50:0/161135075 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c00bd60 con 0x7fdb940674f0 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 -- 172.21.15.50:0/161135075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940674f0 msgr2=0x7fdb94067910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/161135075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940674f0 0x7fdb94067910 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fdb940cad50 tx=0x7fdb8c009250 comp rx=0 tx=0).stop 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 -- 172.21.15.50:0/161135075 shutdown_connections 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/161135075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940674f0 0x7fdb94067910 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/161135075 >> v2:172.21.15.125:3300/0 conn(0x7fdb940cc2f0 0x7fdb940af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 -- 172.21.15.50:0/161135075 >> 172.21.15.50:0/161135075 conn(0x7fdb940d0a70 msgr2=0x7fdb940cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 -- 172.21.15.50:0/161135075 shutdown_connections 2023-12-19T16:14:31.358 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 -- 172.21.15.50:0/161135075 wait complete. 2023-12-19T16:14:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.354+0000 7fdb9a82e700 1 Processor -- start 2023-12-19T16:14:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 -- start start 2023-12-19T16:14:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb940674f0 0x7fdb94155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb940b1dd0 0x7fdb94169a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9415e600 con 0x7fdb940cc2f0 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb94169090 con 0x7fdb940674f0 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb9a82e700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9415e0c0 con 0x7fdb940b1dd0 2023-12-19T16:14:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52674/0 (socket says 172.21.15.50:52674) 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 -- 172.21.15.50:0/3235593259 learned_addr learned my addr 172.21.15.50:0/3235593259 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb93fff700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb940674f0 0x7fdb94155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb8b7fe700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb940b1dd0 0x7fdb94169a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 -- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb940b1dd0 msgr2=0x7fdb94169a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb940b1dd0 0x7fdb94169a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 -- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb940674f0 msgr2=0x7fdb94155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb940674f0 0x7fdb94155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.358+0000 7fdb98dcb700 1 -- 172.21.15.50:0/3235593259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0094f0 con 0x7fdb940cc2f0 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb98dcb700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009290 tx=0x7fdb8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c00b860 con 0x7fdb940cc2f0 2023-12-19T16:14:31.366 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c01a630 con 0x7fdb940cc2f0 2023-12-19T16:14:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9415e380 con 0x7fdb940cc2f0 2023-12-19T16:14:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c01a9f0 con 0x7fdb940cc2f0 2023-12-19T16:14:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb94158c70 con 0x7fdb940cc2f0 2023-12-19T16:14:31.368 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.362+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb941693b0 con 0x7fdb940cc2f0 2023-12-19T16:14:31.370 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.366+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdb8c02d440 con 0x7fdb940cc2f0 2023-12-19T16:14:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.366+0000 7fdb90ff9700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 0x7fdb74062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.366+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fdb8c07e790 con 0x7fdb940cc2f0 2023-12-19T16:14:31.373 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.366+0000 7fdb93fff700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 0x7fdb74062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.370+0000 7fdb93fff700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 0x7fdb74062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb840075a0 tx=0x7fdb840097e0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.380 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.374+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdb941693b0 con 0x7fdb940cc2f0 2023-12-19T16:14:31.560 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.554+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 --> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb941693b0 con 0x7fdb74060670 2023-12-19T16:14:31.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb90ff9700 1 -- 172.21.15.50:0/3235593259 <== mgr.10692 v2:172.21.15.125:6832/44063 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48061 (secure 0 0 0) 0x7fdb941693b0 con 0x7fdb74060670 2023-12-19T16:14:31.567 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 msgr2=0x7fdb74062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 0x7fdb74062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb840075a0 tx=0x7fdb840097e0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 msgr2=0x7fdb94159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009290 tx=0x7fdb8c017000 comp rx=0 tx=0).stop 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 shutdown_connections 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fdb74060670 0x7fdb74062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb940cc2f0 0x7fdb94159030 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fdb940b1dd0 0x7fdb94169a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 --2- 172.21.15.50:0/3235593259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb940674f0 0x7fdb94155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.562+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 >> 172.21.15.50:0/3235593259 conn(0x7fdb940d0a70 msgr2=0x7fdb94156160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:31.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.566+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 shutdown_connections 2023-12-19T16:14:31.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.566+0000 7fdb9a82e700 1 -- 172.21.15.50:0/3235593259 wait complete. 2023-12-19T16:14:31.573 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-12-19T16:14:31.593 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T16:14:29.711757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2470496,"kb_used_data":22560,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752504224,"statfs":{"total":773094113280,"available":770564325376,"internally_reserved":0,"allocated":23101440,"data_stored":12387306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003275"},"pg_stats":[{"pgid":"1.0","version":"218'371","reported_seq":1233,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-12-19T16:14:10.793619+0000","last_change":"2023-12-19T16:14:10.793575+0000","last_active":"2023-12-19T16:14:10.793619+0000","last_peered":"2023-12-19T16:14:10.793619+0000","last_clean":"2023-12-19T16:14:10.793619+0000","last_became_active":"2023-12-19T14:38:57.642347+0000","last_became_peered":"2023-12-19T14:38:57.642347+0000","last_unstale":"2023-12-19T16:14:10.793619+0000","last_undegraded":"2023-12-19T16:14:10.793619+0000","last_fullsized":"2023-12-19T16:14:10.793619+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'371","last_scrub_stamp":"2023-12-19T16:14:10.793453+0000","last_deep_scrub":"218'371","last_deep_scrub_stamp":"2023-12-19T16:14:10.793453+0000","last_clean_scrub_stamp":"2023-12-19T16:14:10.793453+0000","objects_scrubbed":2,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T03:54:06.557176+0000","scrub_duration":0.01159147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575983,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304640,"kb_used_data":3256,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067200,"statfs":{"total":96636764160,"available":96324812800,"internally_reserved":0,"allocated":3334144,"data_stored":1991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.079,"15min":1.107},"min":{"1min":0.69599999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"max":{"1min":1.5720000000000001,"5min":1.661,"15min":1.6919999999999999},"last":1.085},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.157,"15min":1.1639999999999999},"min":{"1min":0.65600000000000003,"5min":0.61699999999999999,"15min":0.496},"max":{"1min":1.4359999999999999,"5min":1.6220000000000001,"15min":1.9379999999999999},"last":1.2190000000000001}]},{"osd":1,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.117,"15min":1.1339999999999999},"min":{"1min":0.78500000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4199999999999999,"5min":1.6160000000000001,"15min":1.903},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.2290000000000001,"15min":1.2350000000000001},"min":{"1min":0.94099999999999995,"5min":0.76900000000000002,"15min":0.73299999999999998},"max":{"1min":1.54,"5min":1.7230000000000001,"15min":2.0619999999999998},"last":0.96699999999999997}]},{"osd":2,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.2829999999999999,"15min":1.2929999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.671,"5min":1.671,"15min":1.8160000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.2849999999999999,"15min":1.3160000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.524,"5min":1.738,"15min":1.7709999999999999},"last":1.151}]},{"osd":3,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2829999999999999,"15min":1.329},"min":{"1min":0.93799999999999994,"5min":0.82899999999999996,"15min":0.52500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.839},"last":1.425},{"interface":"front","average":{"1min":1.329,"5min":1.3260000000000001,"15min":1.3620000000000001},"min":{"1min":1.0920000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.607,"5min":1.7509999999999999,"15min":1.9630000000000001},"last":1.0349999999999999}]},{"osd":4,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2490000000000001,"15min":1.1930000000000001},"min":{"1min":0.96099999999999997,"5min":0.64300000000000002,"15min":0.55600000000000005},"max":{"1min":1.6080000000000001,"5min":1.8280000000000001,"15min":1.849},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.2509999999999999,"15min":1.2430000000000001},"min":{"1min":1.0029999999999999,"5min":0.70799999999999996,"15min":0.47699999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.1739999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.238,"15min":1.2549999999999999},"min":{"1min":0.90800000000000003,"5min":0.85899999999999999,"15min":0.53600000000000003},"max":{"1min":1.593,"5min":1.679,"15min":1.825},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3879999999999999,"15min":1.357},"min":{"1min":0.97799999999999998,"5min":0.77800000000000002,"15min":0.45000000000000001},"max":{"1min":1.6830000000000001,"5min":1.823,"15min":2.0030000000000001},"last":1.117}]},{"osd":6,"last update":"Tue Dec 19 16:14:11 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.458,"15min":1.4390000000000001},"min":{"1min":1.2529999999999999,"5min":0.96199999999999997,"15min":0.54200000000000004},"max":{"1min":1.7729999999999999,"5min":1.909,"15min":1.994},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.403,"15min":1.403},"min":{"1min":1.0549999999999999,"5min":0.83699999999999997,"15min":0.63100000000000001},"max":{"1min":1.7629999999999999,"5min":1.8640000000000001,"15min":2.1059999999999999},"last":1.3169999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.153,"15min":1.1699999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.60099999999999998},"max":{"1min":1.5580000000000001,"5min":1.5820000000000001,"15min":1.7150000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.216,"15min":1.236},"min":{"1min":0.84099999999999997,"5min":0.435,"15min":0.435},"max":{"1min":1.637,"5min":1.74,"15min":1.7909999999999999},"last":1.425}]},{"osd":1,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1899999999999999,"15min":1.194},"min":{"1min":0.86399999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5469999999999999,"5min":1.643,"15min":1.7010000000000001},"last":1.101},{"interface":"front","average":{"1min":1.179,"5min":1.2410000000000001,"15min":1.2689999999999999},"min":{"1min":0.73399999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5449999999999999,"5min":1.806,"15min":1.827},"last":1.4710000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1779999999999999,"15min":1.22},"min":{"1min":0.73999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4530000000000001,"5min":1.651,"15min":1.772},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.2809999999999999,"15min":1.286},"min":{"1min":0.93999999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.5369999999999999,"5min":1.732,"15min":1.732},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2909999999999999,"15min":1.3280000000000001},"min":{"1min":0.81299999999999994,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.6659999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2889999999999999,"15min":1.3220000000000001},"min":{"1min":0.76300000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.7370000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.228}]},{"osd":4,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.208,"15min":1.2050000000000001},"min":{"1min":0.92100000000000004,"5min":0.74099999999999999,"15min":0.54300000000000004},"max":{"1min":1.6379999999999999,"5min":1.708,"15min":1.774},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.292,"5min":1.2729999999999999,"15min":1.3069999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.70999999999999996},"max":{"1min":1.7729999999999999,"5min":1.966,"15min":1.966},"last":1.367}]},{"osd":5,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3200000000000001,"15min":1.3109999999999999},"min":{"1min":0.91300000000000003,"5min":0.88200000000000001,"15min":0.71499999999999997},"max":{"1min":1.7869999999999999,"5min":1.788,"15min":1.923},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.321,"5min":1.363,"15min":1.3640000000000001},"min":{"1min":1.0840000000000001,"5min":0.97899999999999998,"15min":0.68899999999999995},"max":{"1min":1.7430000000000001,"5min":1.8100000000000001,"15min":2.1179999999999999},"last":1.1559999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:14:04 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.4219999999999999,"15min":1.4319999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76600000000000001},"max":{"1min":1.752,"5min":1.839,"15min":2.169},"last":1.534},{"interface":"front","average":{"1min":1.411,"5min":1.454,"15min":1.456},"min":{"1min":0.96699999999999997,"5min":0.83999999999999997,"15min":0.77200000000000002},"max":{"1min":1.9419999999999999,"5min":1.9890000000000001,"15min":2.1749999999999998},"last":1.1759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308156,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063684,"statfs":{"total":96636764160,"available":96321212416,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0800000000000001,"15min":1.1000000000000001},"min":{"1min":0.58299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.714,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.143,"15min":1.1879999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.544,"5min":1.6619999999999999,"15min":1.907},"last":1.103}]},{"osd":1,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.228,"15min":1.2989999999999999},"min":{"1min":0.73999999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.4670000000000001,"5min":1.6759999999999999,"15min":1.7490000000000001},"last":1.645},{"interface":"front","average":{"1min":1.139,"5min":1.2230000000000001,"15min":1.2649999999999999},"min":{"1min":0.77800000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.5549999999999999,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.2030000000000001}]},{"osd":2,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.85399999999999998,"5min":0.83599999999999997,"15min":0.66300000000000003},"max":{"1min":1.454,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.153,"5min":1.1890000000000001,"15min":1.2170000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4379999999999999,"5min":1.7769999999999999,"15min":1.831},"last":1.4490000000000001}]},{"osd":3,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.2589999999999999,"15min":1.296},"min":{"1min":0.83099999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.5529999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.55,"5min":1.734,"15min":2.0449999999999999},"last":1.579}]},{"osd":4,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1419999999999999,"15min":1.141},"min":{"1min":0.74199999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.829,"5min":1.829,"15min":1.829},"last":1.536},{"interface":"front","average":{"1min":1.268,"5min":1.278,"15min":1.2709999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.4199999999999999}]},{"osd":6,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.355,"15min":1.3680000000000001},"min":{"1min":0.92500000000000004,"5min":0.878,"15min":0.49099999999999999},"max":{"1min":1.625,"5min":1.843,"15min":1.968},"last":1.766},{"interface":"front","average":{"1min":1.369,"5min":1.393,"15min":1.3999999999999999},"min":{"1min":0.95799999999999996,"5min":0.64200000000000002,"15min":0.60599999999999998},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":1.6279999999999999}]},{"osd":7,"last update":"Tue Dec 19 16:13:59 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4239999999999999,"15min":1.413},"min":{"1min":1.0609999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7589999999999999,"5min":2.395,"15min":2.395},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4119999999999999},"min":{"1min":0.92400000000000004,"5min":0.80400000000000005,"15min":0.55500000000000005},"max":{"1min":1.7609999999999999,"5min":1.8440000000000001,"15min":1.9770000000000001},"last":1.7110000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312252,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059588,"statfs":{"total":96636764160,"available":96317018112,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1319999999999999,"15min":1.117},"min":{"1min":0.746,"5min":0.71999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5900000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.196,"5min":1.155,"15min":1.1579999999999999},"min":{"1min":0.85799999999999998,"5min":0.70999999999999996,"15min":0.61399999999999999},"max":{"1min":1.5189999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.0569999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3129999999999999,"15min":1.3089999999999999},"min":{"1min":0.83599999999999997,"5min":0.83499999999999996,"15min":0.81899999999999995},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.123},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.93999999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7969999999999999},"last":1.401}]},{"osd":2,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.21,"15min":1.2230000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.244,"15min":1.2529999999999999},"min":{"1min":0.90700000000000003,"5min":0.84999999999999998,"15min":0.504},"max":{"1min":1.5720000000000001,"5min":1.7030000000000001,"15min":1.764},"last":1.3149999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2250000000000001,"15min":1.224},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.7210000000000001,"5min":1.804,"15min":1.804},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3560000000000001,"15min":1.327},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.76800000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.24}]},{"osd":5,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2110000000000001,"15min":1.1759999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.53700000000000003},"max":{"1min":1.6899999999999999,"5min":1.744,"15min":1.855},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.256,"15min":1.2070000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68200000000000005},"max":{"1min":1.819,"5min":2.089,"15min":2.089},"last":0.90300000000000002}]},{"osd":6,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.411,"15min":1.333},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.59399999999999997},"max":{"1min":2.012,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5009999999999999,"15min":1.454},"min":{"1min":1.1419999999999999,"5min":0.85799999999999998,"15min":0.59199999999999997},"max":{"1min":1.8799999999999999,"5min":1.97,"15min":1.982},"last":1.4950000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.4510000000000001,"15min":1.375},"min":{"1min":1.1240000000000001,"5min":0.73699999999999999,"15min":0.61299999999999999},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.381},{"interface":"front","average":{"1min":1.609,"5min":1.5780000000000001,"15min":1.506},"min":{"1min":1.087,"5min":0.872,"15min":0.60199999999999998},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":1.446}]}]},{"osd":3,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":299964,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071876,"statfs":{"total":96636764160,"available":96329601024,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":0.90600000000000003},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2609999999999999,"5min":1.633,"15min":1.7729999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96899999999999997,"15min":1.0009999999999999},"min":{"1min":0.66000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.9370000000000001},"last":1.032}]},{"osd":1,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.96299999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.53900000000000003},"max":{"1min":1.716,"5min":1.716,"15min":2.0350000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.147,"15min":1.119},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.379,"5min":1.724,"15min":2.2269999999999999},"last":1.1259999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.151,"15min":1.1679999999999999},"min":{"1min":0.71099999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.726,"5min":1.7350000000000001,"15min":2.9470000000000001},"last":1.704},{"interface":"front","average":{"1min":1.196,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.71599999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.643,"5min":1.8300000000000001,"15min":1.9970000000000001},"last":1.629}]},{"osd":4,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3280000000000001},"min":{"1min":0.81599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.631,"5min":1.718,"15min":2.5099999999999998},"last":1.427},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.876,"15min":0.72199999999999998},"max":{"1min":1.6419999999999999,"5min":1.788,"15min":2.3450000000000002},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3600000000000001,"15min":1.355},"min":{"1min":0.93300000000000005,"5min":0.90200000000000002,"15min":0.90200000000000002},"max":{"1min":1.6679999999999999,"5min":2.0470000000000002,"15min":2.1909999999999998},"last":1.526},{"interface":"front","average":{"1min":1.393,"5min":1.454,"15min":1.4359999999999999},"min":{"1min":0.85299999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0179999999999998,"5min":2.1909999999999998,"15min":2.4689999999999999},"last":1.4930000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":1.04,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.9830000000000001,"5min":2.2549999999999999,"15min":2.3740000000000001},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.448,"15min":1.458},"min":{"1min":0.879,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.798,"5min":2.0270000000000001,"15min":2.6459999999999999},"last":1.1950000000000001}]},{"osd":7,"last update":"Tue Dec 19 16:13:56 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5449999999999999,"15min":1.55},"min":{"1min":0.87,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8779999999999999,"5min":2.1640000000000001,"15min":2.7589999999999999},"last":1.609},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5,"15min":1.508},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":1.8720000000000001,"5min":1.9199999999999999,"15min":2.2709999999999999},"last":1.6619999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312248,"kb_used_data":2672,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059592,"statfs":{"total":96636764160,"available":96317022208,"internally_reserved":0,"allocated":2736128,"data_stored":1400063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94599999999999995,"15min":0.96099999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.1160000000000001,"15min":1.1439999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.3520000000000001,"5min":1.629,"15min":1.796},"last":1.1559999999999999}]},{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.149,"15min":1.1739999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4059999999999999,"5min":2.327,"15min":2.327},"last":1.093},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.143,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.399,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.92500000000000004}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0069999999999999,"15min":1.044},"min":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.5489999999999999,"5min":1.9299999999999999,"15min":2.1190000000000002},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.105,"15min":1.137},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3979999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.55}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":1.0620000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.569,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.89300000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.786,"5min":2.218,"15min":2.218},"last":1.597}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.395,"15min":1.4099999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6910000000000001,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":1.512},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.377,"15min":1.4039999999999999},"min":{"1min":0.91600000000000004,"5min":0.879,"15min":0.879},"max":{"1min":1.5940000000000001,"5min":2.0049999999999999,"15min":2.4340000000000002},"last":1.427}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.397,"15min":1.4039999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.0099999999999998,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.514,"15min":1.5249999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8500000000000001,"5min":2.609,"15min":2.609},"last":1.641}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5349999999999999,"15min":1.5509999999999999},"min":{"1min":1.054,"5min":1.054,"15min":0.89600000000000002},"max":{"1min":1.833,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.679},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.5269999999999999,"15min":1.5409999999999999},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.9319999999999999,"5min":2.3319999999999999,"15min":2.4830000000000001},"last":1.6990000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":316348,"kb_used_data":2676,"kb_used_omap":0,"kb_used_meta":313664,"kb_avail":94055492,"statfs":{"total":96636764160,"available":96312823808,"internally_reserved":0,"allocated":2740224,"data_stored":1401074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321191936},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.874,"15min":0.90000000000000002},"min":{"1min":0.52500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3300000000000001,"5min":1.4690000000000001,"15min":1.6579999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.97399999999999998,"15min":1.032},"min":{"1min":0.58199999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.454,"5min":1.51,"15min":1.9119999999999999},"last":1.0309999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0449999999999999,"15min":1.0669999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.643,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1,"5min":1.093,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.641,"5min":1.823,"15min":1.823},"last":1.5109999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.242,"15min":1.2789999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.665,"5min":1.778,"15min":1.8180000000000001},"last":1.367},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0960000000000001,"15min":1.105},"min":{"1min":0.65600000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":1.325}]},{"osd":4,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3120000000000001,"15min":1.3260000000000001},"min":{"1min":1,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.163},{"interface":"front","average":{"1min":1.395,"5min":1.391,"15min":1.423},"min":{"1min":1.014,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":2.0110000000000001},"last":1.6519999999999999}]},{"osd":5,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3009999999999999,"15min":1.337},"min":{"1min":1.0049999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":1.597,"5min":1.833,"15min":1.897},"last":1.556},{"interface":"front","average":{"1min":1.397,"5min":1.395,"15min":1.4370000000000001},"min":{"1min":1.038,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.919,"5min":2.0579999999999998,"15min":2.165},"last":1.4670000000000001}]},{"osd":6,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3460000000000001,"15min":1.397},"min":{"1min":0.97299999999999998,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":1.5820000000000001,"5min":1.9590000000000001,"15min":2.1139999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.4770000000000001,"15min":1.512},"min":{"1min":1.1180000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.597}]},{"osd":7,"last update":"Tue Dec 19 16:13:33 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.494,"15min":1.53},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":1.736},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4510000000000001,"15min":1.4910000000000001},"min":{"1min":1.0289999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":2.7080000000000002,"5min":2.7080000000000002,"15min":2.7080000000000002},"last":1.6359999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304636,"kb_used_data":3252,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067204,"statfs":{"total":96636764160,"available":96324816896,"internally_reserved":0,"allocated":3330048,"data_stored":1990431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.439},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.6419999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0069999999999999,"15min":1.008},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4259999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2889999999999999}]},{"osd":2,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.141,"15min":1.147},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.57099999999999995},"max":{"1min":1.605,"5min":1.748,"15min":1.8400000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.143,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.54300000000000004},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8500000000000001},"last":1.1559999999999999}]},{"osd":3,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.02,"15min":1.0409999999999999},"min":{"1min":0.60499999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.448,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.121,"5min":1.2230000000000001,"15min":1.2470000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.55300000000000005},"max":{"1min":1.7290000000000001,"5min":1.974,"15min":1.974},"last":0.93600000000000005}]},{"osd":4,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3799999999999999,"15min":1.393},"min":{"1min":0.873,"5min":0.873,"15min":0.69499999999999995},"max":{"1min":1.8120000000000001,"5min":1.885,"15min":2.0209999999999999},"last":1.397},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.339,"15min":1.331},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.71999999999999997},"max":{"1min":1.762,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.198}]},{"osd":5,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.431,"15min":1.4350000000000001},"min":{"1min":1.071,"5min":0.91200000000000003,"15min":0.90300000000000002},"max":{"1min":1.6870000000000001,"5min":1.8859999999999999,"15min":2.0779999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4419999999999999,"15min":1.456},"min":{"1min":1.056,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.712,"5min":1.919,"15min":1.9810000000000001},"last":1.514}]},{"osd":6,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3979999999999999,"15min":1.4079999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.738,"5min":1.8779999999999999,"15min":2.2349999999999999},"last":1.454},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":0.93899999999999995},"max":{"1min":1.714,"5min":2.0009999999999999,"15min":2.1600000000000001},"last":1.113}]},{"osd":7,"last update":"Tue Dec 19 16:13:55 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5249999999999999,"15min":1.528},"min":{"1min":1.103,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.9390000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.486,"15min":1.502},"min":{"1min":1.01,"5min":1.01,"15min":1.008},"max":{"1min":1.8620000000000001,"5min":2.0219999999999998,"15min":2.3250000000000002},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T16:14:31.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-19T16:14:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.746+0000 7f35ce323700 1 Processor -- start 2023-12-19T16:14:31.751 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.746+0000 7f35ce323700 1 -- start start 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35c8158ed0 0x7f35c806b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c80c88e0 0x7f35c806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35c81592a0 0x7f35c815cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c806c5c0 con 0x7f35c81592a0 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c8067060 con 0x7f35c8158ed0 2023-12-19T16:14:31.755 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ce323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c8067320 con 0x7f35c80c88e0 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35c81592a0 0x7f35c815cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f35c81592a0 0x7f35c815cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52676/0 (socket says 172.21.15.50:52676) 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 -- 172.21.15.50:0/3013555139 learned_addr learned my addr 172.21.15.50:0/3013555139 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cd321700 1 --2- 172.21.15.50:0/3013555139 >> v2:172.21.15.125:3300/0 conn(0x7f35c8158ed0 0x7f35c806b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35ccb20700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c80c88e0 0x7f35c806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.756 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 -- 172.21.15.50:0/3013555139 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c80c88e0 msgr2=0x7f35c806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c80c88e0 0x7f35c806bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 -- 172.21.15.50:0/3013555139 >> v2:172.21.15.125:3300/0 conn(0x7f35c8158ed0 msgr2=0x7f35c806b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 --2- 172.21.15.50:0/3013555139 >> v2:172.21.15.125:3300/0 conn(0x7f35c8158ed0 0x7f35c806b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 -- 172.21.15.50:0/3013555139 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80c68d0 con 0x7f35c81592a0 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35cdb22700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c81592a0 0x7f35c815cc70 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f35c806dc80 tx=0x7f35c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed64e04f3e17a81 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.757 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35bffff700 1 -- 172.21.15.50:0/3013555139 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c000e070 con 0x7f35c81592a0 2023-12-19T16:14:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35bffff700 1 -- 172.21.15.50:0/3013555139 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c000ba50 con 0x7f35c81592a0 2023-12-19T16:14:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.750+0000 7f35bffff700 1 -- 172.21.15.50:0/3013555139 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00092e0 con 0x7f35c81592a0 2023-12-19T16:14:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- 172.21.15.50:0/3013555139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c81592a0 msgr2=0x7f35c815cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c81592a0 0x7f35c815cc70 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f35c806dc80 tx=0x7f35c0009840 comp rx=0 tx=0).stop 2023-12-19T16:14:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- 172.21.15.50:0/3013555139 shutdown_connections 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c81592a0 0x7f35c815cc70 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 --2- 172.21.15.50:0/3013555139 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c80c88e0 0x7f35c806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 --2- 172.21.15.50:0/3013555139 >> v2:172.21.15.125:3300/0 conn(0x7f35c8158ed0 0x7f35c806b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- 172.21.15.50:0/3013555139 >> 172.21.15.50:0/3013555139 conn(0x7f35c80c8d00 msgr2=0x7f35c80c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- 172.21.15.50:0/3013555139 shutdown_connections 2023-12-19T16:14:31.759 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- 172.21.15.50:0/3013555139 wait complete. 2023-12-19T16:14:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 Processor -- start 2023-12-19T16:14:31.761 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.754+0000 7f35ce323700 1 -- start start 2023-12-19T16:14:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35c80c88e0 0x7f35c80aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c81592a0 0x7f35c815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.762 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80ac350 con 0x7f35c8158ed0 2023-12-19T16:14:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80ac630 con 0x7f35c80c88e0 2023-12-19T16:14:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80ac910 con 0x7f35c81592a0 2023-12-19T16:14:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52682/0 (socket says 172.21.15.50:52682) 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 -- 172.21.15.50:0/3675956361 learned_addr learned my addr 172.21.15.50:0/3675956361 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35cd321700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35c80c88e0 0x7f35c80aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35cdb22700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c81592a0 0x7f35c815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 -- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c81592a0 msgr2=0x7f35c815ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c81592a0 0x7f35c815ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 -- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35c80c88e0 msgr2=0x7f35c80aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35c80c88e0 0x7f35c80aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.764 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 -- 172.21.15.50:0/3675956361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c0009ae0 con 0x7f35c8158ed0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ccb20700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f35b800d7e0 tx=0x7f35b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b80125f0 con 0x7f35c8158ed0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8012750 con 0x7f35c8158ed0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c80acbf0 con 0x7f35c8158ed0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8012b10 con 0x7f35c8158ed0 2023-12-19T16:14:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.758+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c80adaa0 con 0x7f35c8158ed0 2023-12-19T16:14:31.767 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.762+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c804be40 con 0x7f35c8158ed0 2023-12-19T16:14:31.768 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.762+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f35b8013480 con 0x7f35c8158ed0 2023-12-19T16:14:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.766+0000 7f35bd7fa700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 0x7f35ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:31.770 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.766+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7f35b807ce00 con 0x7f35c8158ed0 2023-12-19T16:14:31.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.766+0000 7f35cd321700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 0x7f35ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.766+0000 7f35cd321700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 0x7f35ac062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006fd0 tx=0x7f35c4008040 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:31.777 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.770+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f35b8050bc0 con 0x7f35c8158ed0 2023-12-19T16:14:31.944 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.938+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f35c806c180 con 0x7f35c8158ed0 2023-12-19T16:14:31.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.946+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f35b8050bc0 con 0x7f35c8158ed0 2023-12-19T16:14:31.950 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.946+0000 7f35bd7fa700 1 -- 172.21.15.50:0/3675956361 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v76)=0 v76) v1 ==== 123+0+0 (secure 0 0 0) 0x7f35b8050750 con 0x7f35c8158ed0 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 msgr2=0x7f35ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 0x7f35ac062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006fd0 tx=0x7f35c4008040 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 msgr2=0x7f35c8152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f35b800d7e0 tx=0x7f35b8008040 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 shutdown_connections 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7f35ac060670 0x7f35ac062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7f35c81592a0 0x7f35c815ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f35c8158ed0 0x7f35c8152980 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.950+0000 7f35ce323700 1 --2- 172.21.15.50:0/3675956361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35c80c88e0 0x7f35c80aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.954+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 >> 172.21.15.50:0/3675956361 conn(0x7f35c80c8d00 msgr2=0x7f35c8154900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:31.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.954+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 shutdown_connections 2023-12-19T16:14:31.959 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:31.954+0000 7f35ce323700 1 -- 172.21.15.50:0/3675956361 wait complete. 2023-12-19T16:14:31.978 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T16:14:31.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T16:14:32.139 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.134+0000 7fef9cbc3700 1 Processor -- start 2023-12-19T16:14:32.140 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.134+0000 7fef9cbc3700 1 -- start start 2023-12-19T16:14:32.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef9cbc3700 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fef98159210 0x7fef980b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:32.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef9cbc3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef980cb3e0 con 0x7fef98158e40 2023-12-19T16:14:32.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef9cbc3700 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef98068960 con 0x7fef980b1be0 2023-12-19T16:14:32.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef9cbc3700 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef980caab0 con 0x7fef98159210 2023-12-19T16:14:32.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef977fe700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fef98158e40 0x7fef98067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53794/0 (socket says 172.21.15.50:53794) 2023-12-19T16:14:32.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef977fe700 1 -- 172.21.15.50:0/3328382548 learned_addr learned my addr 172.21.15.50:0/3328382548 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-12-19T16:14:32.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef96ffd700 1 --2- 172.21.15.50:0/3328382548 >> v2:172.21.15.125:3300/0 conn(0x7fef98159210 0x7fef980b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57614806 0 0) 0x7fef980cb3e0 con 0x7fef98158e40 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef80003570 con 0x7fef98158e40 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 <== mon.1 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571518561 0 0) 0x7fef98068960 con 0x7fef980b1be0 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef980cb3e0 con 0x7fef980b1be0 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef96ffd700 1 -- 172.21.15.50:0/3328382548 >> v1:172.21.15.125:6789/0 conn(0x7fef980b1be0 legacy=0x7fef980b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef96ffd700 1 -- 172.21.15.50:0/3328382548 >> v1:172.21.15.50:6789/0 conn(0x7fef98158e40 legacy=0x7fef98067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.145 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef96ffd700 1 -- 172.21.15.50:0/3328382548 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980cad90 con 0x7fef98159210 2023-12-19T16:14:32.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.138+0000 7fef96ffd700 1 --2- 172.21.15.50:0/3328382548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef98159210 0x7fef980b2c00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0098e0 tx=0x7fef8c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa57bf3c244c20d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:32.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011630 con 0x7fef98159210 2023-12-19T16:14:32.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fef8c011790 con 0x7fef98159210 2023-12-19T16:14:32.146 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef967fc700 1 -- 172.21.15.50:0/3328382548 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011b50 con 0x7fef98159210 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 -- 172.21.15.50:0/3328382548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef98159210 msgr2=0x7fef980b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/3328382548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef98159210 0x7fef980b2c00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0098e0 tx=0x7fef8c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 -- 172.21.15.50:0/3328382548 shutdown_connections 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/3328382548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef98159210 0x7fef980b2c00 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 -- 172.21.15.50:0/3328382548 >> 172.21.15.50:0/3328382548 conn(0x7fef980d0900 msgr2=0x7fef980cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 -- 172.21.15.50:0/3328382548 shutdown_connections 2023-12-19T16:14:32.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 -- 172.21.15.50:0/3328382548 wait complete. 2023-12-19T16:14:32.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.142+0000 7fef9cbc3700 1 Processor -- start 2023-12-19T16:14:32.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 -- start start 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 0x7fef980c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef98158e40 0x7fef9815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980c65c0 con 0x7fef98159210 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980c78a0 con 0x7fef980b1be0 2023-12-19T16:14:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef9cbc3700 1 -- --> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef980c6f70 con 0x7fef98158e40 2023-12-19T16:14:32.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef97fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:32.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef96ffd700 1 --2- >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef98158e40 0x7fef9815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:32.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 0x7fef980c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:32.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef97fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52692/0 (socket says 172.21.15.50:52692) 2023-12-19T16:14:32.152 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef97fff700 1 -- 172.21.15.50:0/554495534 learned_addr learned my addr 172.21.15.50:0/554495534 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-12-19T16:14:32.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 -- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef98158e40 msgr2=0x7fef9815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef98158e40 0x7fef9815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 -- 172.21.15.50:0/554495534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 msgr2=0x7fef9815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.153 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef977fe700 1 -- 172.21.15.50:0/554495534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c009420 con 0x7fef980b1be0 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.146+0000 7fef97fff700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef977fe700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 0x7fef980c9790 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c580 tx=0x7fef88007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88005b30 con 0x7fef980b1be0 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fef88005c90 con 0x7fef980b1be0 2023-12-19T16:14:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88007340 con 0x7fef980b1be0 2023-12-19T16:14:32.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef980c7250 con 0x7fef980b1be0 2023-12-19T16:14:32.155 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef980c9cd0 con 0x7fef980b1be0 2023-12-19T16:14:32.156 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.150+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9804be40 con 0x7fef980b1be0 2023-12-19T16:14:32.158 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.154+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 108) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fef8801b3b0 con 0x7fef980b1be0 2023-12-19T16:14:32.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.154+0000 7fef7f7fe700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 0x7fef84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T16:14:32.159 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.154+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6016+0+0 (secure 0 0 0) 0x7fef8807bf60 con 0x7fef980b1be0 2023-12-19T16:14:32.160 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.154+0000 7fef96ffd700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 0x7fef84062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T16:14:32.161 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.154+0000 7fef96ffd700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 0x7fef84062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0093f0 tx=0x7fef8c0091a0 comp rx=0 tx=0).ready entity=mgr.10692 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T16:14:32.168 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.162+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fef8804c310 con 0x7fef980b1be0 2023-12-19T16:14:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.370+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef980cb070 con 0x7fef980b1be0 2023-12-19T16:14:32.379 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.374+0000 7fef7f7fe700 1 -- 172.21.15.50:0/554495534 <== mon.1 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3533 (secure 0 0 0) 0x7fef88017090 con 0x7fef980b1be0 2023-12-19T16:14:32.379 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-19T16:14:32.380 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":101,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/274543537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":274543537},{"type":"v1","addr":"172.21.15.125:6836","nonce":274543537}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":8933,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2831702896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6835","nonce":2831702896},{"type":"v1","addr":"172.21.15.125:6837","nonce":2831702896}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6834/888097858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":888097858},{"type":"v1","addr":"172.21.15.50:6834","nonce":888097858}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":94},{"gid":10803,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/1808870801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6833","nonce":1808870801},{"type":"v1","addr":"172.21.15.50:6835","nonce":1808870801}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":101}],"filesystems":[]} 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 msgr2=0x7fef84062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 0x7fef84062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0093f0 tx=0x7fef8c0091a0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 msgr2=0x7fef980c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 0x7fef980c9790 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c580 tx=0x7fef88007d70 comp rx=0 tx=0).stop 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 shutdown_connections 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:6832/44063,v1:172.21.15.125:6833/44063] conn(0x7fef84060670 0x7fef84062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef98159210 0x7fef9815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.384 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3301/0,v1:172.21.15.125:6790/0] conn(0x7fef98158e40 0x7fef9815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 --2- 172.21.15.50:0/554495534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef980b1be0 0x7fef980c9790 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T16:14:32.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 >> 172.21.15.50:0/554495534 conn(0x7fef980d0900 msgr2=0x7fef980b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T16:14:32.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 shutdown_connections 2023-12-19T16:14:32.385 INFO:teuthology.orchestra.run.smithi050.stderr:2023-12-19T16:14:32.378+0000 7fef9cbc3700 1 -- 172.21.15.50:0/554495534 wait complete. 2023-12-19T16:14:32.387 INFO:teuthology.orchestra.run.smithi050.stderr:dumped fsmap epoch 101 2023-12-19T16:14:32.411 DEBUG:tasks.cephfs.filesystem:already dead... 2023-12-19T16:14:32.411 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-19T16:14:32.411 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T16:14:32.412 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:32.432 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T16:14:32.432 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T16:14:32.433 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:32.498 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T16:14:32.499 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T16:14:32.499 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:32.523 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T16:14:32.523 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T16:14:32.523 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:32.588 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T16:14:32.588 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-19T16:14:32.589 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-19T16:14:32.589 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:32.674 INFO:tasks.ceph.osd.0:Stopped 2023-12-19T16:14:32.675 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-19T16:14:32.675 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:36.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-19T16:14:38.776 INFO:tasks.ceph.osd.1:Stopped 2023-12-19T16:14:38.777 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-19T16:14:38.777 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:42.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-19T16:14:42.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-19T16:14:44.879 INFO:tasks.ceph.osd.2:Stopped 2023-12-19T16:14:44.879 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-19T16:14:44.879 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:48.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-19T16:14:48.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-19T16:14:50.981 INFO:tasks.ceph.osd.3:Stopped 2023-12-19T16:14:50.981 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-19T16:14:50.981 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:51.063 INFO:tasks.ceph.osd.4:Stopped 2023-12-19T16:14:51.063 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-19T16:14:51.063 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:54.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-19T16:14:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-19T16:14:57.165 INFO:tasks.ceph.osd.5:Stopped 2023-12-19T16:14:57.166 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-19T16:14:57.166 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:57.251 INFO:tasks.ceph.osd.6:Stopped 2023-12-19T16:14:57.251 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-19T16:14:57.251 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:14:59.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-19T16:14:59.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-19T16:15:00.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:00.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:03.352 INFO:tasks.ceph.osd.7:Stopped 2023-12-19T16:15:03.353 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-19T16:15:03.353 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-19T16:15:03.353 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:15:05.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2023-12-19T16:15:05.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-19T16:15:09.455 INFO:tasks.ceph.mgr.y:Stopped 2023-12-19T16:15:09.456 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-19T16:15:09.456 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:15:09.521 INFO:tasks.ceph.mgr.x:Stopped 2023-12-19T16:15:09.521 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-19T16:15:09.521 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-19T16:15:09.521 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:15:09.572 INFO:tasks.ceph.mon.a:Stopped 2023-12-19T16:15:09.573 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-19T16:15:09.573 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:15:09.614 INFO:tasks.ceph.mon.b:Stopped 2023-12-19T16:15:09.615 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-19T16:15:09.615 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T16:15:09.652 INFO:tasks.ceph.mon.c:Stopped 2023-12-19T16:15:09.652 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-19T16:15:09.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T16:15:09.704 INFO:teuthology.orchestra.run.smithi050.stdout:1702996897.632042 mon.a (mon.0) 358 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-19T16:15:09.705 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-12-19T16:15:09.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T16:15:09.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T16:15:09.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T16:15:09.934 INFO:teuthology.orchestra.run.smithi050.stdout:1702996897.632042 mon.a (mon.0) 358 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-19T16:15:09.935 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:15:09.936 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-19T16:15:10.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:15:10.197 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-19T16:15:10.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2023-12-19T16:15:10.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:15:10.557 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-19T16:15:10.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:15:10.777 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-19T16:15:11.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:15:11.017 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-19T16:15:11.256 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:15:11.257 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-19T16:15:11.467 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:15:11.467 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-19T16:15:11.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:15:11.678 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-19T16:15:11.881 INFO:tasks.ceph:Archiving mon data... 2023-12-19T16:15:11.883 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/data/mon.a.tgz 2023-12-19T16:15:11.883 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-12-19T16:15:11.901 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.93uUTFaFle 2023-12-19T16:15:11.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.93uUTFaFle 2023-12-19T16:15:12.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.93uUTFaFle 2023-12-19T16:15:12.476 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.93uUTFaFle is 2MB 2023-12-19T16:15:12.598 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.93uUTFaFle 2023-12-19T16:15:12.618 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/data/mon.b.tgz 2023-12-19T16:15:12.619 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2023-12-19T16:15:12.637 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.vzbygkHeWd 2023-12-19T16:15:12.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vzbygkHeWd 2023-12-19T16:15:13.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.vzbygkHeWd 2023-12-19T16:15:13.181 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.vzbygkHeWd is 2MB 2023-12-19T16:15:13.293 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.vzbygkHeWd 2023-12-19T16:15:13.312 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/data/mon.c.tgz 2023-12-19T16:15:13.312 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2023-12-19T16:15:13.369 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.nZKFh223vy 2023-12-19T16:15:13.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.nZKFh223vy 2023-12-19T16:15:13.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.nZKFh223vy 2023-12-19T16:15:13.917 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.nZKFh223vy is 2MB 2023-12-19T16:15:14.033 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.nZKFh223vy 2023-12-19T16:15:14.053 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-19T16:15:14.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-19T16:15:14.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-19T16:15:14.137 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-19T16:15:14.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-19T16:15:14.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-19T16:15:14.193 INFO:teuthology.orchestra.run.smithi050.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-19T16:15:14.242 INFO:teuthology.orchestra.run.smithi125.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-19T16:15:14.256 INFO:tasks.ceph:Archiving crash dumps... 2023-12-19T16:15:14.258 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi050/crash 2023-12-19T16:15:14.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-19T16:15:14.296 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi125/crash 2023-12-19T16:15:14.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-19T16:15:14.335 INFO:tasks.ceph:Shutting down logrotate 2023-12-19T16:15:14.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:14.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:14.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:14.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-19T16:15:14.519 INFO:tasks.ceph:Compressing logs... 2023-12-19T16:15:14.520 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T16:15:14.532 DEBUG:teuthology.orchestra.run.smithi125:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T16:15:14.568 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46474.log 2023-12-19T16:15:14.569 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54609.log 2023-12-19T16:15:14.572 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62561.log 2023-12-19T16:15:14.572 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46474.log.gz 2023-12-19T16:15:14.574 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54609.log: gzip -5 --verbose -- 88.1%/var/log/ceph/ceph-client.admin.53335.log -- replaced with /var/log/ceph/ceph-client.admin.54609.log.gz 2023-12-19T16:15:14.574 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:14.576 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2023-12-19T16:15:14.577 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62561.log.gz 2023-12-19T16:15:14.578 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52341.log 2023-12-19T16:15:14.578 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53335.log.gz 2023-12-19T16:15:14.579 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40002.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2023-12-19T16:15:14.580 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44855.log 2023-12-19T16:15:14.582 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57811.log 2023-12-19T16:15:14.583 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52341.log: /var/log/ceph/ceph-client.admin.44855.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44855.log.gz 2023-12-19T16:15:14.583 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52341.log.gz 2023-12-19T16:15:14.584 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62799.log 2023-12-19T16:15:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61120.log 2023-12-19T16:15:14.586 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62799.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57811.log.gz 2023-12-19T16:15:14.587 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62799.log.gz 2023-12-19T16:15:14.588 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38022.log 2023-12-19T16:15:14.589 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2023-12-19T16:15:14.590 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61120.log.gz 2023-12-19T16:15:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38022.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38022.log.gz 2023-12-19T16:15:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57571.log 2023-12-19T16:15:14.592 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47121.log 2023-12-19T16:15:14.593 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2023-12-19T16:15:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57571.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57571.log.gz 2023-12-19T16:15:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44587.log 2023-12-19T16:15:14.596 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57355.log 2023-12-19T16:15:14.596 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47121.log.gz 2023-12-19T16:15:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62157.log 2023-12-19T16:15:14.598 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44587.log.gz 2023-12-19T16:15:14.598 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42196.log 2023-12-19T16:15:14.598 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57355.log.gz 2023-12-19T16:15:14.600 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58360.log 2023-12-19T16:15:14.601 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62157.log.gz 2023-12-19T16:15:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42196.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42196.log.gz 2023-12-19T16:15:14.602 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45590.log 2023-12-19T16:15:14.603 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58178.log 2023-12-19T16:15:14.603 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58360.log.gz 2023-12-19T16:15:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45590.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45590.log.gz 2023-12-19T16:15:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61752.log 2023-12-19T16:15:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46409.log 2023-12-19T16:15:14.607 INFO:teuthology.orchestra.run.smithi050.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.58178.log.gz 2023-12-19T16:15:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-12-19T16:15:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61752.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61752.log.gz 2023-12-19T16:15:14.608 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48585.log 2023-12-19T16:15:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46409.log.gz 2023-12-19T16:15:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51933.log 2023-12-19T16:15:14.609 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-19T16:15:14.610 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50272.log 2023-12-19T16:15:14.610 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48585.log.gz 2023-12-19T16:15:14.611 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.39518.log 2023-12-19T16:15:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2023-12-19T16:15:14.612 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51933.log.gz 2023-12-19T16:15:14.613 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50272.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50272.log.gz 2023-12-19T16:15:14.613 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34100.log 2023-12-19T16:15:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60352.log 2023-12-19T16:15:14.614 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-19T16:15:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38787.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2023-12-19T16:15:14.615 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2023-12-19T16:15:14.616 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.1.39518.log: /var/log/ceph/ceph-client.admin.34100.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34100.log.gz 2023-12-19T16:15:14.616 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33939.log 2023-12-19T16:15:14.616 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60352.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60352.log.gz 2023-12-19T16:15:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60872.log 2023-12-19T16:15:14.618 INFO:teuthology.orchestra.run.smithi125.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.1.39518.log.gz 2023-12-19T16:15:14.618 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-19T16:15:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52993.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2023-12-19T16:15:14.619 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38202.log 2023-12-19T16:15:14.620 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.33939.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33939.log.gz 2023-12-19T16:15:14.620 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-19T16:15:14.620 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39975.log 2023-12-19T16:15:14.621 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60872.log.gz 2023-12-19T16:15:14.622 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-19T16:15:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38202.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38202.log.gz 2023-12-19T16:15:14.622 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43561.log 2023-12-19T16:15:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39975.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39975.log.gz 2023-12-19T16:15:14.623 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42649.log 2023-12-19T16:15:14.624 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33966.log 2023-12-19T16:15:14.625 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59845.log 2023-12-19T16:15:14.625 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43561.log.gz 2023-12-19T16:15:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57002.log 2023-12-19T16:15:14.626 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2023-12-19T16:15:14.627 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59845.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59845.log.gz 2023-12-19T16:15:14.628 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51987.log 2023-12-19T16:15:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.39632.log 2023-12-19T16:15:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.33966.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33966.log.gz 2023-12-19T16:15:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57002.log.gz 2023-12-19T16:15:14.630 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-19T16:15:14.631 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.3.39632.log: 89.6% -- replaced with /var/log/ceph/ceph-client.3.39632.log.gz 2023-12-19T16:15:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51987.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51987.log.gz 2023-12-19T16:15:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61723.log 2023-12-19T16:15:14.632 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-19T16:15:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2023-12-19T16:15:14.633 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34021.log 2023-12-19T16:15:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61723.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61723.log.gz 2023-12-19T16:15:14.635 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2023-12-19T16:15:14.636 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2023-12-19T16:15:14.636 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2023-12-19T16:15:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59982.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2023-12-19T16:15:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43320.log 2023-12-19T16:15:14.639 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34021.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34021.log.gz 2023-12-19T16:15:14.639 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34075.log 2023-12-19T16:15:14.639 INFO:teuthology.orchestra.run.smithi125.stderr: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-19T16:15:14.640 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61467.log 2023-12-19T16:15:14.640 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2023-12-19T16:15:14.640 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43320.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43320.log.gz 2023-12-19T16:15:14.640 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63305.log 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.34075.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34075.log.gz 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61467.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61467.log.gz 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63114.log 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63305.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63305.log.gz 2023-12-19T16:15:14.643 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61019.log 2023-12-19T16:15:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63114.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63114.log.gz 2023-12-19T16:15:14.644 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.61669.log 2023-12-19T16:15:14.645 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33993.log 2023-12-19T16:15:14.646 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61019.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61019.log.gz 2023-12-19T16:15:14.646 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.43480.log 2023-12-19T16:15:14.646 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61669.log.gz 2023-12-19T16:15:14.647 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61309.log 2023-12-19T16:15:14.647 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34048.log 2023-12-19T16:15:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43480.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43480.log.gz 2023-12-19T16:15:14.648 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55668.log 2023-12-19T16:15:14.648 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-19T16:15:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61309.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61309.log.gz 2023-12-19T16:15:14.649 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58805.log 2023-12-19T16:15:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55668.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55668.log.gz 2023-12-19T16:15:14.651 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40643.log 2023-12-19T16:15:14.652 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.34048.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-19T16:15:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58805.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58805.log.gz 2023-12-19T16:15:14.652 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56344.log 2023-12-19T16:15:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40643.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40643.log.gz 2023-12-19T16:15:14.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58715.log 2023-12-19T16:15:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.33993.log: 87.6% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34048.log.gz 2023-12-19T16:15:14.655 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33993.log.gz 2023-12-19T16:15:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34128.log 2023-12-19T16:15:14.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56344.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56344.log.gz 2023-12-19T16:15:14.655 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61833.log 2023-12-19T16:15:14.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57931.log 2023-12-19T16:15:14.656 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58715.log.gz 2023-12-19T16:15:14.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61833.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61833.log.gz 2023-12-19T16:15:14.658 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48403.log 2023-12-19T16:15:14.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57931.log: gzip -5 --verbose -- 88.4%/var/log/ceph/ceph-client.admin.33965.log 2023-12-19T16:15:14.659 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57931.log.gz 2023-12-19T16:15:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48403.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.48403.log.gz 2023-12-19T16:15:14.660 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2023-12-19T16:15:14.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.33965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2023-12-19T16:15:14.662 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33965.log.gz 2023-12-19T16:15:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51547.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2023-12-19T16:15:14.663 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51879.log 2023-12-19T16:15:14.664 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62958.log 2023-12-19T16:15:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42517.log: /var/log/ceph/ceph-client.admin.51879.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2023-12-19T16:15:14.666 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.34128.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-19T16:15:14.666 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34128.log.gz 2023-12-19T16:15:14.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2023-12-19T16:15:14.666 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51879.log.gz 2023-12-19T16:15:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49156.log 2023-12-19T16:15:14.667 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62958.log.gz 2023-12-19T16:15:14.669 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61201.log 2023-12-19T16:15:14.669 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2023-12-19T16:15:14.670 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2023-12-19T16:15:14.670 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49156.log.gz 2023-12-19T16:15:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61201.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61201.log.gz 2023-12-19T16:15:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54502.log 2023-12-19T16:15:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2023-12-19T16:15:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42022.log: /var/log/ceph/ceph-client.0.54502.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz 2023-12-19T16:15:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42466.log 2023-12-19T16:15:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52312.log 2023-12-19T16:15:14.677 INFO:teuthology.orchestra.run.smithi050.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz 2023-12-19T16:15:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55108.log 2023-12-19T16:15:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.7.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-12-19T16:15:14.678 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz 2023-12-19T16:15:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52312.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52312.log.gz 2023-12-19T16:15:14.679 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54693.log 2023-12-19T16:15:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55108.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55108.log.gz 2023-12-19T16:15:14.680 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62125.log 2023-12-19T16:15:14.682 INFO:teuthology.orchestra.run.smithi125.stderr: 93.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-19T16:15:14.682 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2023-12-19T16:15:14.682 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54693.log.gz 2023-12-19T16:15:14.683 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43077.log 89.2% 2023-12-19T16:15:14.684 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62125.log.gz 2023-12-19T16:15:14.685 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59698.log 2023-12-19T16:15:14.685 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2023-12-19T16:15:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43077.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43077.log.gz 2023-12-19T16:15:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2023-12-19T16:15:14.687 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2023-12-19T16:15:14.687 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59698.log.gz 2023-12-19T16:15:14.688 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49791.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2023-12-19T16:15:14.689 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2023-12-19T16:15:14.690 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43875.log 2023-12-19T16:15:14.690 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2023-12-19T16:15:14.691 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40832.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2023-12-19T16:15:14.691 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52594.log 2023-12-19T16:15:14.692 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37995.log 2023-12-19T16:15:14.692 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43875.log.gz 2023-12-19T16:15:14.693 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52594.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52594.log.gz 2023-12-19T16:15:14.694 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2023-12-19T16:15:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37995.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37995.log.gz 2023-12-19T16:15:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63737.log 2023-12-19T16:15:14.696 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52073.log 2023-12-19T16:15:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50539.log: /var/log/ceph/ceph-client.admin.63737.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2023-12-19T16:15:14.698 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 89.4%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63737.log.gz/var/log/ceph/ceph-client.admin.57840.log 2023-12-19T16:15:14.698 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:14.699 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56041.log 2023-12-19T16:15:14.699 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52073.log.gz 2023-12-19T16:15:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57840.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57840.log.gz 2023-12-19T16:15:14.700 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2023-12-19T16:15:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56041.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56041.log.gz 2023-12-19T16:15:14.702 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2023-12-19T16:15:14.703 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56909.log 2023-12-19T16:15:14.703 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2023-12-19T16:15:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39813.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2023-12-19T16:15:14.704 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-12-19T16:15:14.706 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61494.log 2023-12-19T16:15:14.706 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56909.log.gz 2023-12-19T16:15:14.707 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58929.log 2023-12-19T16:15:14.708 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61494.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61494.log.gz 2023-12-19T16:15:14.709 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57757.log 2023-12-19T16:15:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59725.log 2023-12-19T16:15:14.710 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58929.log.gz 2023-12-19T16:15:14.711 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52014.log 2023-12-19T16:15:14.712 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57757.log.gz 2023-12-19T16:15:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59725.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59725.log.gz 2023-12-19T16:15:14.713 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48612.log 2023-12-19T16:15:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52014.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52014.log.gz 2023-12-19T16:15:14.715 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42729.log 2023-12-19T16:15:14.717 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53584.log 2023-12-19T16:15:14.718 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48612.log: /var/log/ceph/ceph-client.admin.42729.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48612.log.gz 2023-12-19T16:15:14.718 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42729.log.gz 2023-12-19T16:15:14.718 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53692.log 2023-12-19T16:15:14.720 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53584.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53584.log.gz 2023-12-19T16:15:14.720 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63251.log 2023-12-19T16:15:14.722 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43453.log 2023-12-19T16:15:14.722 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53692.log.gz 2023-12-19T16:15:14.723 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57436.log 2023-12-19T16:15:14.723 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63251.log.gz 2023-12-19T16:15:14.724 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50825.log 2023-12-19T16:15:14.725 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43453.log.gz 2023-12-19T16:15:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57436.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57436.log.gz 2023-12-19T16:15:14.726 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47559.log 2023-12-19T16:15:14.727 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54206.log 2023-12-19T16:15:14.728 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50825.log.gz 2023-12-19T16:15:14.729 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50391.log 2023-12-19T16:15:14.730 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47559.log.gz 2023-12-19T16:15:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54206.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54206.log.gz 2023-12-19T16:15:14.731 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53800.log 2023-12-19T16:15:14.732 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60752.log 2023-12-19T16:15:14.732 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50391.log.gz 2023-12-19T16:15:14.734 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55436.log 2023-12-19T16:15:14.734 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53800.log.gz 2023-12-19T16:15:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60752.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60752.log.gz 2023-12-19T16:15:14.735 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48166.log 2023-12-19T16:15:14.736 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2023-12-19T16:15:14.737 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55436.log.gz 2023-12-19T16:15:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48166.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48166.log.gz 2023-12-19T16:15:14.738 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43266.log 2023-12-19T16:15:14.739 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2023-12-19T16:15:14.740 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2023-12-19T16:15:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57096.log 87.8% 2023-12-19T16:15:14.741 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43266.log.gz 2023-12-19T16:15:14.743 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56715.log 2023-12-19T16:15:14.743 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2023-12-19T16:15:14.744 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2023-12-19T16:15:14.744 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57096.log.gz 2023-12-19T16:15:14.746 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58526.log 2023-12-19T16:15:14.746 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56715.log.gz 2023-12-19T16:15:14.747 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41075.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-12-19T16:15:14.748 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2023-12-19T16:15:14.748 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47876.log 2023-12-19T16:15:14.748 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58526.log.gz 2023-12-19T16:15:14.750 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56613.log 2023-12-19T16:15:14.750 INFO:teuthology.orchestra.run.smithi050.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.0.54502.log.gz 2023-12-19T16:15:14.752 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2023-12-19T16:15:14.752 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47876.log.gz 2023-12-19T16:15:14.753 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.56613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57730.log 2023-12-19T16:15:14.755 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41749.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2023-12-19T16:15:14.755 INFO:teuthology.orchestra.run.smithi050.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.56613.log.gz 2023-12-19T16:15:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60197.log 2023-12-19T16:15:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57730.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57730.log.gz 2023-12-19T16:15:14.757 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59783.log 2023-12-19T16:15:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.60197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52857.log 2023-12-19T16:15:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47221.log 2023-12-19T16:15:14.760 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59783.log.gz 2023-12-19T16:15:14.762 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62650.log 2023-12-19T16:15:14.762 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52857.log.gz 2023-12-19T16:15:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49971.log 2023-12-19T16:15:14.764 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47221.log.gz 2023-12-19T16:15:14.765 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62650.log: gzip -5 --verbose 88.3%-- /var/log/ceph/ceph-client.admin.61860.log 2023-12-19T16:15:14.765 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62650.log.gz 2023-12-19T16:15:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49971.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49971.log.gz-5 2023-12-19T16:15:14.767 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40616.log 2023-12-19T16:15:14.769 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47248.log 2023-12-19T16:15:14.769 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61860.log.gz 2023-12-19T16:15:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40616.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2023-12-19T16:15:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47956.log 2023-12-19T16:15:14.771 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2023-12-19T16:15:14.771 INFO:teuthology.orchestra.run.smithi050.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47248.log.gz 2023-12-19T16:15:14.772 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47956.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47956.log.gz 2023-12-19T16:15:14.773 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39381.log 2023-12-19T16:15:14.774 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50511.log 2023-12-19T16:15:14.774 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2023-12-19T16:15:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39381.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39381.log.gz 2023-12-19T16:15:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47302.log 2023-12-19T16:15:14.776 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54446.log 2023-12-19T16:15:14.777 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50511.log.gz 2023-12-19T16:15:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47302.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47302.log.gz 2023-12-19T16:15:14.778 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42385.log 2023-12-19T16:15:14.779 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2023-12-19T16:15:14.779 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54446.log.gz 2023-12-19T16:15:14.780 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42385.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42385.log.gz 2023-12-19T16:15:14.781 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2023-12-19T16:15:14.782 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53364.log 2023-12-19T16:15:14.782 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2023-12-19T16:15:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40859.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2023-12-19T16:15:14.783 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42439.log 2023-12-19T16:15:14.784 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58960.log 2023-12-19T16:15:14.785 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53364.log.gz 2023-12-19T16:15:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57409.log 2023-12-19T16:15:14.786 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42439.log.gz 2023-12-19T16:15:14.787 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58960.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58960.log.gz 2023-12-19T16:15:14.788 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53773.log 2023-12-19T16:15:14.789 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39786.log 2023-12-19T16:15:14.789 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57409.log.gz 2023-12-19T16:15:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55586.log 2023-12-19T16:15:14.791 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53773.log.gz 2023-12-19T16:15:14.792 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60625.log 2023-12-19T16:15:14.792 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39786.log.gz 2023-12-19T16:15:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55586.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55586.log.gz 2023-12-19T16:15:14.793 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60842.log 2023-12-19T16:15:14.794 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49318.log 2023-12-19T16:15:14.795 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60625.log.gz 2023-12-19T16:15:14.796 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44648.log 2023-12-19T16:15:14.797 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60842.log.gz 2023-12-19T16:15:14.798 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63014.log 2023-12-19T16:15:14.798 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49318.log.gz 2023-12-19T16:15:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44929.log 2023-12-19T16:15:14.799 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44648.log.gz 2023-12-19T16:15:14.801 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63278.log 2023-12-19T16:15:14.801 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63014.log.gz 2023-12-19T16:15:14.803 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42250.log 2023-12-19T16:15:14.803 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44929.log.gz 2023-12-19T16:15:14.804 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59020.log 2023-12-19T16:15:14.805 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63278.log.gz 2023-12-19T16:15:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42250.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.42250.log.gz 2023-12-19T16:15:14.806 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2023-12-19T16:15:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59020.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59020.log.gz 2023-12-19T16:15:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2023-12-19T16:15:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59550.log 2023-12-19T16:15:14.808 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2023-12-19T16:15:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39705.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2023-12-19T16:15:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2023-12-19T16:15:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58014.log 2023-12-19T16:15:14.811 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59550.log.gz 2023-12-19T16:15:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44736.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2023-12-19T16:15:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53959.log 2023-12-19T16:15:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2023-12-19T16:15:14.814 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58014.log.gz 2023-12-19T16:15:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53959.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53959.log.gz 2023-12-19T16:15:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62680.log 2023-12-19T16:15:14.815 INFO:teuthology.orchestra.run.smithi050.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.60197.log.gz 2023-12-19T16:15:14.816 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47062.log 2023-12-19T16:15:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62680.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2023-12-19T16:15:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2023-12-19T16:15:14.818 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62680.log.gz 2023-12-19T16:15:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47062.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47062.log.gz 2023-12-19T16:15:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2023-12-19T16:15:14.820 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45725.log 2023-12-19T16:15:14.821 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2023-12-19T16:15:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43399.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2023-12-19T16:15:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55556.log 2023-12-19T16:15:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41623.log 2023-12-19T16:15:14.824 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45725.log.gz 2023-12-19T16:15:14.825 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55556.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55556.log.gz 2023-12-19T16:15:14.825 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58742.log 2023-12-19T16:15:14.827 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41623.log: gzip -5 --verbose -- 88.6%/var/log/ceph/ceph-client.admin.61605.log 2023-12-19T16:15:14.827 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41623.log.gz 2023-12-19T16:15:14.828 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37940.log 2023-12-19T16:15:14.829 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58742.log.gz 2023-12-19T16:15:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48447.log 2023-12-19T16:15:14.830 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61605.log.gz 2023-12-19T16:15:14.831 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62288.log 2023-12-19T16:15:14.831 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37940.log.gz 2023-12-19T16:15:14.833 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48447.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48447.log.gz 2023-12-19T16:15:14.833 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60299.log 2023-12-19T16:15:14.834 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61055.log 2023-12-19T16:15:14.835 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62288.log.gz 2023-12-19T16:15:14.836 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2023-12-19T16:15:14.836 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60299.log.gz 2023-12-19T16:15:14.837 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61055.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61055.log.gz 2023-12-19T16:15:14.837 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53305.log 2023-12-19T16:15:14.839 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49851.log 2023-12-19T16:15:14.839 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2023-12-19T16:15:14.840 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60900.log 2023-12-19T16:15:14.841 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53305.log.gz 2023-12-19T16:15:14.842 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49851.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49851.log.gz 2023-12-19T16:15:14.842 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52621.log 2023-12-19T16:15:14.844 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52376.log 2023-12-19T16:15:14.844 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60900.log.gz 2023-12-19T16:15:14.845 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52621.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52621.log.gz 2023-12-19T16:15:14.845 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-19T16:15:14.846 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2023-12-19T16:15:14.847 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52376.log.gz 2023-12-19T16:15:14.848 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39624.log 2023-12-19T16:15:14.849 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47849.log 2023-12-19T16:15:14.850 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2023-12-19T16:15:14.851 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2023-12-19T16:15:14.851 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39624.log.gz 2023-12-19T16:15:14.853 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2023-12-19T16:15:14.853 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47849.log.gz 2023-12-19T16:15:14.854 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61914.log 2023-12-19T16:15:14.854 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2023-12-19T16:15:14.856 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63197.log 2023-12-19T16:15:14.856 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2023-12-19T16:15:14.857 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56974.log 2023-12-19T16:15:14.857 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61914.log.gz 2023-12-19T16:15:14.859 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2023-12-19T16:15:14.859 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63197.log.gz 2023-12-19T16:15:14.860 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50659.log 2023-12-19T16:15:14.860 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56974.log.gz 2023-12-19T16:15:14.862 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38310.log 2023-12-19T16:15:14.862 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2023-12-19T16:15:14.863 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57063.log 2023-12-19T16:15:14.863 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50659.log.gz 2023-12-19T16:15:14.864 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2023-12-19T16:15:14.865 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38310.log.gz 2023-12-19T16:15:14.866 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60961.log 2023-12-19T16:15:14.866 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57063.log.gz 2023-12-19T16:15:14.868 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52202.log 2023-12-19T16:15:14.868 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2023-12-19T16:15:14.869 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63494.log 2023-12-19T16:15:14.869 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60961.log.gz 2023-12-19T16:15:14.870 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52202.log: gzip -5 87.7%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.52202.log.gz/var/log/ceph/ceph-client.admin.55884.log 2023-12-19T16:15:14.870 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:14.872 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61634.log 2023-12-19T16:15:14.872 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63494.log.gz 2023-12-19T16:15:14.874 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2023-12-19T16:15:14.874 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2023-12-19T16:15:14.875 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48197.log 2023-12-19T16:15:14.875 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61634.log.gz 2023-12-19T16:15:14.877 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43426.log 2023-12-19T16:15:14.877 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2023-12-19T16:15:14.878 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47329.log 2023-12-19T16:15:14.878 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48197.log.gz 2023-12-19T16:15:14.879 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49264.log 2023-12-19T16:15:14.879 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43426.log.gz 2023-12-19T16:15:14.881 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43788.log 2023-12-19T16:15:14.881 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47329.log.gz 2023-12-19T16:15:14.882 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49264.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49264.log.gz 2023-12-19T16:15:14.883 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59139.log 2023-12-19T16:15:14.884 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50083.log 2023-12-19T16:15:14.884 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43788.log.gz 2023-12-19T16:15:14.886 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56124.log 2023-12-19T16:15:14.886 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59139.log.gz 2023-12-19T16:15:14.887 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62589.log 2023-12-19T16:15:14.887 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50083.log.gz 2023-12-19T16:15:14.888 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2023-12-19T16:15:14.889 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56124.log.gz 2023-12-19T16:15:14.890 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62589.log: gzip 89.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.62589.log.gz--verbose 2023-12-19T16:15:14.890 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.56768.log 2023-12-19T16:15:14.891 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40535.log 2023-12-19T16:15:14.892 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2023-12-19T16:15:14.893 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50798.log 2023-12-19T16:15:14.893 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56768.log.gz 2023-12-19T16:15:14.895 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59296.log 2023-12-19T16:15:14.895 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2023-12-19T16:15:14.896 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46593.log 2023-12-19T16:15:14.896 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50798.log.gz 2023-12-19T16:15:14.898 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59296.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59296.log.gz 2023-12-19T16:15:14.898 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61228.log 2023-12-19T16:15:14.899 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2023-12-19T16:15:14.900 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46593.log.gz 2023-12-19T16:15:14.901 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61228.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61228.log.gz 2023-12-19T16:15:14.901 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2023-12-19T16:15:14.903 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46818.log 2023-12-19T16:15:14.903 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2023-12-19T16:15:14.905 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53746.log 2023-12-19T16:15:14.905 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2023-12-19T16:15:14.906 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63548.log 2023-12-19T16:15:14.906 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46818.log.gz 2023-12-19T16:15:14.907 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60723.log 2023-12-19T16:15:14.907 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53746.log.gz 2023-12-19T16:15:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45752.log 2023-12-19T16:15:14.909 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63548.log.gz 2023-12-19T16:15:14.910 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57155.log 2023-12-19T16:15:14.911 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60723.log.gz 2023-12-19T16:15:14.912 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.40350.log 2023-12-19T16:15:14.912 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45752.log.gz 2023-12-19T16:15:14.913 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50624.log 2023-12-19T16:15:14.914 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57155.log.gz 2023-12-19T16:15:14.915 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.2.40350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2023-12-19T16:15:14.915 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.2.40350.log.gz 2023-12-19T16:15:14.916 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50624.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51724.log 2023-12-19T16:15:14.917 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50624.log.gz 2023-12-19T16:15:14.918 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2023-12-19T16:15:14.918 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2023-12-19T16:15:14.920 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62829.log 2023-12-19T16:15:14.920 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51724.log.gz 2023-12-19T16:15:14.921 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54069.log 2023-12-19T16:15:14.921 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2023-12-19T16:15:14.923 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48639.log 2023-12-19T16:15:14.923 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62829.log.gz 2023-12-19T16:15:14.924 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43050.log 2023-12-19T16:15:14.924 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54069.log.gz 2023-12-19T16:15:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2023-12-19T16:15:14.926 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48639.log.gz 2023-12-19T16:15:14.927 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59671.log 2023-12-19T16:15:14.927 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43050.log.gz 2023-12-19T16:15:14.929 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43293.log 2023-12-19T16:15:14.929 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2023-12-19T16:15:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59671.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59671.log.gz 2023-12-19T16:15:14.930 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60812.log 2023-12-19T16:15:14.932 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44441.log 2023-12-19T16:15:14.932 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43293.log.gz 2023-12-19T16:15:14.933 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54152.log 2023-12-19T16:15:14.933 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60812.log.gz 2023-12-19T16:15:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2023-12-19T16:15:14.935 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44441.log.gz 2023-12-19T16:15:14.936 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48852.log 2023-12-19T16:15:14.936 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54152.log.gz 2023-12-19T16:15:14.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 2023-12-19T16:15:14.938 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2023-12-19T16:15:14.939 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46041.log 2023-12-19T16:15:14.939 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48852.log.gz 2023-12-19T16:15:14.940 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47613.log 2023-12-19T16:15:14.940 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gz 2023-12-19T16:15:14.942 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2023-12-19T16:15:14.942 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46041.log.gz 2023-12-19T16:15:14.943 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43131.log 2023-12-19T16:15:14.943 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47613.log.gz 2023-12-19T16:15:14.945 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56881.log 2023-12-19T16:15:14.945 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2023-12-19T16:15:14.946 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2023-12-19T16:15:14.947 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43131.log.gz 2023-12-19T16:15:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51960.log 2023-12-19T16:15:14.948 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56881.log.gz 2023-12-19T16:15:14.948 INFO:teuthology.orchestra.run.smithi050.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-19T16:15:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2023-12-19T16:15:14.949 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2023-12-19T16:15:14.951 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46846.log 2023-12-19T16:15:14.951 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51960.log.gz 2023-12-19T16:15:14.952 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37756.log 2023-12-19T16:15:14.953 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2023-12-19T16:15:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46846.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46846.log.gz 2023-12-19T16:15:14.954 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44494.log 2023-12-19T16:15:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52229.log 2023-12-19T16:15:14.956 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37756.log.gz 2023-12-19T16:15:14.956 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44494.log.gz 2023-12-19T16:15:14.957 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61696.log 2023-12-19T16:15:14.957 INFO:teuthology.orchestra.run.smithi050.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-12-19T16:15:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2023-12-19T16:15:14.959 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52229.log.gz 2023-12-19T16:15:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39732.log 2023-12-19T16:15:14.960 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61696.log.gz 2023-12-19T16:15:14.960 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46125.log 2023-12-19T16:15:14.961 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2023-12-19T16:15:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39732.log: gzip -5 --verbose -- 89.5%/var/log/ceph/ceph-client.admin.43021.log 2023-12-19T16:15:14.962 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39732.log.gz 2023-12-19T16:15:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46125.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46125.log.gz 2023-12-19T16:15:14.963 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63712.log 2023-12-19T16:15:14.964 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2023-12-19T16:15:14.964 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43021.log.gz 2023-12-19T16:15:14.966 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52256.log 2023-12-19T16:15:14.966 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63712.log.gz 2023-12-19T16:15:14.967 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53120.log 2023-12-19T16:15:14.967 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz 2023-12-19T16:15:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52256.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52256.log.gz 2023-12-19T16:15:14.968 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59872.log 2023-12-19T16:15:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53120.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.53120.log.gz 2023-12-19T16:15:14.969 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.63332.log 2023-12-19T16:15:14.970 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59872.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59872.log.gz 2023-12-19T16:15:14.971 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52283.log 2023-12-19T16:15:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63332.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63332.log.gz 2023-12-19T16:15:14.972 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45118.log 2023-12-19T16:15:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52283.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52283.log.gz 2023-12-19T16:15:14.973 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-19T16:15:14.974 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46682.log 2023-12-19T16:15:14.974 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45118.log.gz 2023-12-19T16:15:14.975 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59617.log 2023-12-19T16:15:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46682.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46682.log.gz 2023-12-19T16:15:14.977 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60272.log 2023-12-19T16:15:14.979 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51115.log 2023-12-19T16:15:14.979 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59617.log.gz 2023-12-19T16:15:14.980 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48693.log 2023-12-19T16:15:14.980 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60272.log.gz 2023-12-19T16:15:14.981 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-19T16:15:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51115.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51115.log.gz 2023-12-19T16:15:14.982 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48960.log 2023-12-19T16:15:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48693.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48693.log.gz 2023-12-19T16:15:14.984 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38868.log 2023-12-19T16:15:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2023-12-19T16:15:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48960.log: /var/log/ceph/ceph-client.admin.38868.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48960.log.gz 2023-12-19T16:15:14.987 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38868.log.gz 2023-12-19T16:15:14.987 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43239.log 2023-12-19T16:15:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44559.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2023-12-19T16:15:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2023-12-19T16:15:14.989 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2023-12-19T16:15:14.989 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43239.log.gz 2023-12-19T16:15:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38943.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2023-12-19T16:15:14.991 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57126.log 2023-12-19T16:15:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39678.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz-5 2023-12-19T16:15:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47449.log 2023-12-19T16:15:14.994 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54662.log 2023-12-19T16:15:14.994 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57126.log.gz 2023-12-19T16:15:14.996 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59323.log 2023-12-19T16:15:14.996 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47449.log.gz 2023-12-19T16:15:14.997 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2023-12-19T16:15:14.997 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54662.log.gz 2023-12-19T16:15:14.998 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47586.log 2023-12-19T16:15:14.999 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59323.log.gz 2023-12-19T16:15:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49587.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gz 2023-12-19T16:15:15.000 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62769.log 2023-12-19T16:15:15.002 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47586.log.gz 2023-12-19T16:15:15.002 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44066.log 2023-12-19T16:15:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62769.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62769.log.gz 2023-12-19T16:15:15.003 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42847.log 2023-12-19T16:15:15.005 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43642.log 2023-12-19T16:15:15.007 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44066.log: /var/log/ceph/ceph-client.admin.42847.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44066.log.gz 2023-12-19T16:15:15.007 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42847.log.gz 2023-12-19T16:15:15.007 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57987.log 2023-12-19T16:15:15.008 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55405.log 2023-12-19T16:15:15.008 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43642.log.gz 2023-12-19T16:15:15.009 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60992.log 2023-12-19T16:15:15.009 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57987.log.gz 2023-12-19T16:15:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61147.log 2023-12-19T16:15:15.011 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55405.log.gz 2023-12-19T16:15:15.012 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62620.log 2023-12-19T16:15:15.012 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60992.log.gz 2023-12-19T16:15:15.013 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59377.log 2023-12-19T16:15:15.013 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61147.log.gz 2023-12-19T16:15:15.014 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62620.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62620.log.gz 2023-12-19T16:15:15.014 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48989.log 2023-12-19T16:15:15.016 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56854.log 2023-12-19T16:15:15.016 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59377.log.gz 2023-12-19T16:15:15.017 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50028.log 2023-12-19T16:15:15.017 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48989.log.gz 2023-12-19T16:15:15.018 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2023-12-19T16:15:15.018 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56854.log.gz 2023-12-19T16:15:15.019 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60782.log 2023-12-19T16:15:15.020 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50028.log.gz 2023-12-19T16:15:15.021 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51088.log 2023-12-19T16:15:15.021 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2023-12-19T16:15:15.022 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57960.log 2023-12-19T16:15:15.022 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60782.log.gz 2023-12-19T16:15:15.023 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39160.log 2023-12-19T16:15:15.023 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51088.log.gz 2023-12-19T16:15:15.024 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57703.log 2023-12-19T16:15:15.024 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57960.log.gz 2023-12-19T16:15:15.025 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2023-12-19T16:15:15.026 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39160.log.gz 2023-12-19T16:15:15.027 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57703.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57703.log.gz 2023-12-19T16:15:15.027 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63602.log 2023-12-19T16:15:15.028 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63440.log 2023-12-19T16:15:15.029 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2023-12-19T16:15:15.033 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63602.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63602.log.gz 2023-12-19T16:15:15.033 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58095.log 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63440.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63440.log.gz 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57904.log 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58095.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58095.log.gz 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57904.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57904.log.gz 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41804.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2023-12-19T16:15:15.035 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45671.log 2023-12-19T16:15:15.036 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40805.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2023-12-19T16:15:15.036 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48107.log 2023-12-19T16:15:15.036 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45671.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45671.log.gz 2023-12-19T16:15:15.037 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57463.log 2023-12-19T16:15:15.038 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57544.log 2023-12-19T16:15:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48107.log: /var/log/ceph/ceph-client.admin.57463.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48107.log.gz 2023-12-19T16:15:15.040 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57463.log.gz 2023-12-19T16:15:15.040 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56371.log 2023-12-19T16:15:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57544.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57544.log.gz 2023-12-19T16:15:15.041 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44334.log 2023-12-19T16:15:15.042 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49534.log 2023-12-19T16:15:15.043 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56371.log.gz 2023-12-19T16:15:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44334.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44334.log.gz 2023-12-19T16:15:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48558.log 2023-12-19T16:15:15.045 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2023-12-19T16:15:15.046 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49534.log.gz 2023-12-19T16:15:15.047 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2023-12-19T16:15:15.047 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48558.log.gz 2023-12-19T16:15:15.048 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45358.log 2023-12-19T16:15:15.048 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2023-12-19T16:15:15.049 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58608.log 2023-12-19T16:15:15.049 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2023-12-19T16:15:15.050 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2023-12-19T16:15:15.050 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45358.log.gz 2023-12-19T16:15:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58608.log: gzip 87.6%-5 -- replaced with /var/log/ceph/ceph-client.admin.58608.log.gz--verbose 2023-12-19T16:15:15.052 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.44175.log 2023-12-19T16:15:15.053 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54040.log 2023-12-19T16:15:15.053 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2023-12-19T16:15:15.055 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40994.log 2023-12-19T16:15:15.055 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44175.log.gz 2023-12-19T16:15:15.056 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42622.log 2023-12-19T16:15:15.056 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54040.log.gz 2023-12-19T16:15:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40994.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40994.log.gz 2023-12-19T16:15:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2023-12-19T16:15:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42622.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42622.log.gz 2023-12-19T16:15:15.058 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62050.log 2023-12-19T16:15:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41332.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2023-12-19T16:15:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47387.log 2023-12-19T16:15:15.060 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.62050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39142.log 2023-12-19T16:15:15.061 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.0.62050.log.gz 2023-12-19T16:15:15.062 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2023-12-19T16:15:15.062 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47387.log.gz 2023-12-19T16:15:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39142.log: gzip -5 --verbose -- 88.3%/var/log/ceph/ceph-client.admin.59079.log -- replaced with /var/log/ceph/ceph-client.admin.39142.log.gz 2023-12-19T16:15:15.063 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41722.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2023-12-19T16:15:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59928.log 2023-12-19T16:15:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59079.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59079.log.gz 2023-12-19T16:15:15.065 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56557.log 2023-12-19T16:15:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59928.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59928.log.gz 2023-12-19T16:15:15.067 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63143.log 2023-12-19T16:15:15.068 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2023-12-19T16:15:15.069 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56557.log: /var/log/ceph/ceph-client.admin.63143.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56557.log.gz 2023-12-19T16:15:15.070 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63143.log.gz 2023-12-19T16:15:15.070 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51309.log 2023-12-19T16:15:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51487.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2023-12-19T16:15:15.071 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55857.log 2023-12-19T16:15:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51309.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51309.log.gz 2023-12-19T16:15:15.072 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39166.log 2023-12-19T16:15:15.074 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42331.log 2023-12-19T16:15:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39166.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz 2023-12-19T16:15:15.076 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39166.log.gz 2023-12-19T16:15:15.076 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51517.log 2023-12-19T16:15:15.077 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47903.log 2023-12-19T16:15:15.077 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42331.log.gz 2023-12-19T16:15:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51517.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51517.log.gz 2023-12-19T16:15:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56236.log 2023-12-19T16:15:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50218.log 2023-12-19T16:15:15.080 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47903.log.gz 2023-12-19T16:15:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56236.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56236.log.gz 2023-12-19T16:15:15.081 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53456.log 2023-12-19T16:15:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50218.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.50218.log.gz 2023-12-19T16:15:15.082 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.62710.log 2023-12-19T16:15:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53456.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53456.log.gz 2023-12-19T16:15:15.083 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54775.log 2023-12-19T16:15:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62710.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62710.log.gz 2023-12-19T16:15:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.55344.log 2023-12-19T16:15:15.085 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54775.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54775.log.gz-5 2023-12-19T16:15:15.086 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.49822.log 2023-12-19T16:15:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55344.log.gz 2023-12-19T16:15:15.087 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45644.log 2023-12-19T16:15:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49822.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49822.log.gz 2023-12-19T16:15:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2023-12-19T16:15:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45644.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45644.log.gz 2023-12-19T16:15:15.089 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50299.log 2023-12-19T16:15:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41413.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2023-12-19T16:15:15.091 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61549.log 2023-12-19T16:15:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50299.log: gzip 89.3%-5 -- replaced with /var/log/ceph/ceph-client.admin.50299.log.gz--verbose 2023-12-19T16:15:15.092 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.53986.log 2023-12-19T16:15:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61549.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61549.log.gz 2023-12-19T16:15:15.093 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50771.log 2023-12-19T16:15:15.094 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53986.log: gzip -5 --verbose -- 88.7%/var/log/ceph/ceph-client.admin.47358.log 2023-12-19T16:15:15.094 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53986.log.gz 2023-12-19T16:15:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50771.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50771.log.gz 2023-12-19T16:15:15.095 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63359.log 2023-12-19T16:15:15.096 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47358.log: gzip -5 --verbose -- 87.5%/var/log/ceph/ceph-client.admin.42277.log -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2023-12-19T16:15:15.096 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63359.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63359.log.gz 2023-12-19T16:15:15.097 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45830.log 2023-12-19T16:15:15.098 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52513.log 2023-12-19T16:15:15.099 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2023-12-19T16:15:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45830.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45830.log.gz 2023-12-19T16:15:15.100 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54395.log 2023-12-19T16:15:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52513.log.gz 2023-12-19T16:15:15.101 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52803.log 2023-12-19T16:15:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44883.log 2023-12-19T16:15:15.102 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54395.log.gz 2023-12-19T16:15:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52803.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52803.log.gz 2023-12-19T16:15:15.103 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45064.log 2023-12-19T16:15:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33837.log 2023-12-19T16:15:15.105 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44883.log.gz 2023-12-19T16:15:15.106 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48285.log 2023-12-19T16:15:15.106 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45064.log.gz 2023-12-19T16:15:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.33837.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33837.log.gz 2023-12-19T16:15:15.107 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59517.log 2023-12-19T16:15:15.108 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2023-12-19T16:15:15.108 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48285.log.gz 2023-12-19T16:15:15.109 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59517.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59517.log.gz 2023-12-19T16:15:15.110 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2023-12-19T16:15:15.111 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53185.log 2023-12-19T16:15:15.111 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2023-12-19T16:15:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52459.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2023-12-19T16:15:15.112 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54555.log 2023-12-19T16:15:15.114 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40214.log 2023-12-19T16:15:15.114 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53185.log.gz 2023-12-19T16:15:15.115 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54555.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54555.log.gz 2023-12-19T16:15:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40967.log 2023-12-19T16:15:15.117 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.40214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56095.log 2023-12-19T16:15:15.118 INFO:teuthology.orchestra.run.smithi050.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.0.40214.log.gz 2023-12-19T16:15:15.119 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2023-12-19T16:15:15.119 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40967.log.gz 2023-12-19T16:15:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56095.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56095.log.gz 2023-12-19T16:15:15.120 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50879.log 2023-12-19T16:15:15.121 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42358.log 2023-12-19T16:15:15.121 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz 2023-12-19T16:15:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50879.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50879.log.gz 2023-12-19T16:15:15.123 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47640.log 2023-12-19T16:15:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57246.log 2023-12-19T16:15:15.125 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42358.log.gz 2023-12-19T16:15:15.125 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47640.log.gz 2023-12-19T16:15:15.126 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49045.log 2023-12-19T16:15:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57246.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57246.log.gz 2023-12-19T16:15:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49237.log 2023-12-19T16:15:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49045.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49045.log.gz 2023-12-19T16:15:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45204.log 2023-12-19T16:15:15.129 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63170.log 2023-12-19T16:15:15.130 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49237.log.gz 2023-12-19T16:15:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45204.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45204.log.gz 2023-12-19T16:15:15.131 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46533.log 2023-12-19T16:15:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63170.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63170.log.gz 2023-12-19T16:15:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60435.log 2023-12-19T16:15:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46533.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46533.log.gz 2023-12-19T16:15:15.134 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47476.log 2023-12-19T16:15:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60435.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60435.log.gz 2023-12-19T16:15:15.135 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59431.log 2023-12-19T16:15:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47476.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47476.log.gz 2023-12-19T16:15:15.136 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2023-12-19T16:15:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49882.log 2023-12-19T16:15:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59431.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59431.log.gz 2023-12-19T16:15:15.139 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58068.log 2023-12-19T16:15:15.140 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2023-12-19T16:15:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49882.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49882.log.gz 2023-12-19T16:15:15.141 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39371.log 2023-12-19T16:15:15.142 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 88.1% 2023-12-19T16:15:15.142 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58068.log.gz 2023-12-19T16:15:15.143 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39371.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39371.log.gz 2023-12-19T16:15:15.143 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2023-12-19T16:15:15.144 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61576.log 2023-12-19T16:15:15.144 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2023-12-19T16:15:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43817.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2023-12-19T16:15:15.145 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53395.log 2023-12-19T16:15:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2023-12-19T16:15:15.146 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61576.log.gz 2023-12-19T16:15:15.147 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53395.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53395.log.gz 2023-12-19T16:15:15.148 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2023-12-19T16:15:15.149 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51142.log 88.5% 2023-12-19T16:15:15.149 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2023-12-19T16:15:15.150 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42169.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2023-12-19T16:15:15.150 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57185.log 2023-12-19T16:15:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51142.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51142.log.gz 2023-12-19T16:15:15.151 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2023-12-19T16:15:15.152 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44827.log 2023-12-19T16:15:15.152 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57185.log.gz 2023-12-19T16:15:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41858.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2023-12-19T16:15:15.153 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2023-12-19T16:15:15.154 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44827.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44827.log.gz 2023-12-19T16:15:15.155 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2023-12-19T16:15:15.156 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38173.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2023-12-19T16:15:15.156 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49072.log 2023-12-19T16:15:15.157 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46900.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gz 2023-12-19T16:15:15.157 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.54179.log 2023-12-19T16:15:15.158 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49072.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49072.log.gz 2023-12-19T16:15:15.158 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2023-12-19T16:15:15.159 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54179.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54179.log.gz 2023-12-19T16:15:15.159 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2023-12-19T16:15:15.160 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38364.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2023-12-19T16:15:15.160 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2023-12-19T16:15:15.162 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45961.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2023-12-19T16:15:15.162 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2023-12-19T16:15:15.163 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41156.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2023-12-19T16:15:15.163 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43846.log 2023-12-19T16:15:15.164 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40454.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2023-12-19T16:15:15.164 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50566.log 2023-12-19T16:15:15.165 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43846.log: gzip 88.7%-5 -- replaced with /var/log/ceph/ceph-client.admin.43846.log.gz--verbose 2023-12-19T16:15:15.165 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.49616.log 2023-12-19T16:15:15.166 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50566.log: gzip 88.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.50566.log.gz 2023-12-19T16:15:15.166 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.56180.log 2023-12-19T16:15:15.167 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49616.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2023-12-19T16:15:15.167 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52041.log 2023-12-19T16:15:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56180.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56180.log.gz 2023-12-19T16:15:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58899.log 2023-12-19T16:15:15.169 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52041.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52041.log.gz 2023-12-19T16:15:15.170 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56398.log 2023-12-19T16:15:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58899.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58899.log.gz 2023-12-19T16:15:15.171 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2023-12-19T16:15:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56398.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56398.log.gz 2023-12-19T16:15:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45172.log 2023-12-19T16:15:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47089.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2023-12-19T16:15:15.173 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38937.log 2023-12-19T16:15:15.174 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45172.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45172.log.gz 2023-12-19T16:15:15.174 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45443.log 2023-12-19T16:15:15.175 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38937.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38937.log.gz 2023-12-19T16:15:15.175 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50001.log 2023-12-19T16:15:15.176 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45443.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45443.log.gz 2023-12-19T16:15:15.177 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46074.log 2023-12-19T16:15:15.178 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55830.log 2023-12-19T16:15:15.179 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50001.log.gz 2023-12-19T16:15:15.179 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46074.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46074.log.gz 2023-12-19T16:15:15.179 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2023-12-19T16:15:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55830.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55830.log.gz 2023-12-19T16:15:15.181 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58286.log 2023-12-19T16:15:15.182 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2023-12-19T16:15:15.183 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2023-12-19T16:15:15.184 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.58286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38080.log 2023-12-19T16:15:15.186 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39921.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2023-12-19T16:15:15.186 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55526.log 2023-12-19T16:15:15.187 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2023-12-19T16:15:15.187 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38080.log.gz 2023-12-19T16:15:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55526.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-19T16:15:15.190 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55526.log.gz 2023-12-19T16:15:15.191 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56688.log 2023-12-19T16:15:15.191 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2023-12-19T16:15:15.192 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61093.log 2023-12-19T16:15:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56688.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56688.log.gz 2023-12-19T16:15:15.194 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2023-12-19T16:15:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45385.log 2023-12-19T16:15:15.195 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61093.log.gz 2023-12-19T16:15:15.197 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2023-12-19T16:15:15.197 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2023-12-19T16:15:15.198 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59240.log 2023-12-19T16:15:15.198 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45385.log.gz 2023-12-19T16:15:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40589.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2023-12-19T16:15:15.200 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38942.log 2023-12-19T16:15:15.201 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48375.log 2023-12-19T16:15:15.202 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59240.log.gz 2023-12-19T16:15:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49210.log 2023-12-19T16:15:15.202 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38942.log.gz 2023-12-19T16:15:15.203 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2023-12-19T16:15:15.204 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48375.log.gz 2023-12-19T16:15:15.205 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49210.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49210.log.gz 2023-12-19T16:15:15.205 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2023-12-19T16:15:15.206 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2023-12-19T16:15:15.206 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2023-12-19T16:15:15.207 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39597.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz-5 2023-12-19T16:15:15.208 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.43933.log 2023-12-19T16:15:15.209 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51456.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2023-12-19T16:15:15.209 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51666.log 2023-12-19T16:15:15.210 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62496.log 2023-12-19T16:15:15.211 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43933.log.gz 2023-12-19T16:15:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51666.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51666.log.gz 2023-12-19T16:15:15.212 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44228.log 2023-12-19T16:15:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62496.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62496.log.gz 2023-12-19T16:15:15.213 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43904.log 2023-12-19T16:15:15.214 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2023-12-19T16:15:15.215 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44228.log.gz 2023-12-19T16:15:15.215 INFO:teuthology.orchestra.run.smithi050.stderr: 95.2% -- replaced with /var/log/ceph/ceph-client.0.58286.log.gz 2023-12-19T16:15:15.216 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62893.log 2023-12-19T16:15:15.216 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43904.log.gz 2023-12-19T16:15:15.217 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63764.log 2023-12-19T16:15:15.218 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2023-12-19T16:15:15.219 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59955.log 2023-12-19T16:15:15.219 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62893.log.gz 2023-12-19T16:15:15.220 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63764.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63764.log.gz 2023-12-19T16:15:15.220 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48720.log 2023-12-19T16:15:15.221 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49667.log 2023-12-19T16:15:15.221 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59955.log.gz 2023-12-19T16:15:15.223 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48720.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48720.log.gz 2023-12-19T16:15:15.223 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55976.log 2023-12-19T16:15:15.224 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51428.log 2023-12-19T16:15:15.224 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49667.log.gz 2023-12-19T16:15:15.225 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55976.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55976.log.gz 2023-12-19T16:15:15.226 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56799.log 2023-12-19T16:15:15.227 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2023-12-19T16:15:15.227 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51428.log.gz 2023-12-19T16:15:15.228 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56799.log: gzip -5 --verbose 88.7%-- /var/log/ceph/ceph-client.admin.61255.log -- replaced with /var/log/ceph/ceph-client.admin.56799.log.gz 2023-12-19T16:15:15.228 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.229 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43701.log 2023-12-19T16:15:15.229 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2023-12-19T16:15:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61255.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61255.log.gz 2023-12-19T16:15:15.230 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60141.log 2023-12-19T16:15:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58333.log 2023-12-19T16:15:15.232 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43701.log.gz 2023-12-19T16:15:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60141.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60141.log.gz 2023-12-19T16:15:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55803.log 2023-12-19T16:15:15.234 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2023-12-19T16:15:15.234 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58333.log.gz 2023-12-19T16:15:15.235 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55613.log 2023-12-19T16:15:15.235 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55803.log.gz 2023-12-19T16:15:15.236 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2023-12-19T16:15:15.236 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2023-12-19T16:15:15.238 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55613.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55613.log.gz 2023-12-19T16:15:15.238 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2023-12-19T16:15:15.239 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39354.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2023-12-19T16:15:15.239 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58229.log 2023-12-19T16:15:15.240 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39382.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2023-12-19T16:15:15.240 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53638.log 2023-12-19T16:15:15.241 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47745.log 2023-12-19T16:15:15.241 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58229.log.gz 2023-12-19T16:15:15.242 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47035.log 2023-12-19T16:15:15.243 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53638.log.gz 2023-12-19T16:15:15.244 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47745.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47745.log.gz 2023-12-19T16:15:15.244 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60383.log 2023-12-19T16:15:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47035.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47035.log.gz 2023-12-19T16:15:15.245 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55315.log 2023-12-19T16:15:15.247 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63575.log 2023-12-19T16:15:15.247 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60383.log.gz 2023-12-19T16:15:15.248 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38391.log 2023-12-19T16:15:15.248 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55315.log.gz 2023-12-19T16:15:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63575.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63575.log.gz 2023-12-19T16:15:15.249 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61401.log 2023-12-19T16:15:15.250 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50056.log 2023-12-19T16:15:15.250 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38391.log.gz 2023-12-19T16:15:15.251 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61401.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61401.log.gz 2023-12-19T16:15:15.252 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51906.log 2023-12-19T16:15:15.253 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2023-12-19T16:15:15.253 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50056.log.gz 2023-12-19T16:15:15.254 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51906.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.51906.log.gz 2023-12-19T16:15:15.254 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.38144.log 2023-12-19T16:15:15.255 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58149.log 2023-12-19T16:15:15.256 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2023-12-19T16:15:15.256 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47694.log 2023-12-19T16:15:15.257 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38144.log.gz 2023-12-19T16:15:15.258 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46651.log 2023-12-19T16:15:15.258 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58149.log.gz 2023-12-19T16:15:15.259 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59109.log 2023-12-19T16:15:15.260 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.47694.log.gz 2023-12-19T16:15:15.260 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63467.log 2023-12-19T16:15:15.261 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46651.log.gz 2023-12-19T16:15:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59109.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59109.log.gz 2023-12-19T16:15:15.262 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50137.log 2023-12-19T16:15:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63467.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63467.log.gz 2023-12-19T16:15:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51825.log 2023-12-19T16:15:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50137.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50137.log.gz 2023-12-19T16:15:15.265 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47154.log 2023-12-19T16:15:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51825.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.51825.log.gz 2023-12-19T16:15:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.54314.log 2023-12-19T16:15:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47154.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47154.log.gz 2023-12-19T16:15:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2023-12-19T16:15:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54314.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54314.log.gz 2023-12-19T16:15:15.269 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52403.log 2023-12-19T16:15:15.270 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2023-12-19T16:15:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52403.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2023-12-19T16:15:15.271 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52403.log.gz 2023-12-19T16:15:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60245.log 2023-12-19T16:15:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59267.log 2023-12-19T16:15:15.274 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2023-12-19T16:15:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60245.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60245.log.gz 2023-12-19T16:15:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2023-12-19T16:15:15.276 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48345.log 2023-12-19T16:15:15.276 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59267.log.gz 2023-12-19T16:15:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44618.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz-5 2023-12-19T16:15:15.277 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.57328.log 2023-12-19T16:15:15.278 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2023-12-19T16:15:15.279 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48345.log.gz 2023-12-19T16:15:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61368.log 2023-12-19T16:15:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38679.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57328.log.gz 2023-12-19T16:15:15.281 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2023-12-19T16:15:15.281 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57636.log 2023-12-19T16:15:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43588.log 2023-12-19T16:15:15.283 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61368.log.gz 2023-12-19T16:15:15.284 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57636.log: gzip 87.5%-5 -- replaced with /var/log/ceph/ceph-client.admin.57636.log.gz 2023-12-19T16:15:15.285 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.39165.log 2023-12-19T16:15:15.286 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43588.log: gzip -5 88.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43588.log.gz/var/log/ceph/ceph-client.admin.41831.log 2023-12-19T16:15:15.286 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39165.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2023-12-19T16:15:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42905.log 2023-12-19T16:15:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41831.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2023-12-19T16:15:15.289 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47802.log 2023-12-19T16:15:15.290 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57273.log 2023-12-19T16:15:15.290 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42905.log.gz 2023-12-19T16:15:15.292 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.47802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52567.log 2023-12-19T16:15:15.293 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57273.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.47802.log.gz 2023-12-19T16:15:15.294 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57273.log.gz 2023-12-19T16:15:15.294 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46772.log 2023-12-19T16:15:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52567.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52567.log.gz 2023-12-19T16:15:15.296 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2023-12-19T16:15:15.297 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60090.log 2023-12-19T16:15:15.297 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46772.log.gz 2023-12-19T16:15:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49507.log 2023-12-19T16:15:15.301 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2023-12-19T16:15:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53215.log 2023-12-19T16:15:15.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49507.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60090.log.gz 2023-12-19T16:15:15.302 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49507.log.gz 2023-12-19T16:15:15.302 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2023-12-19T16:15:15.303 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51752.log 2023-12-19T16:15:15.303 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53215.log.gz 2023-12-19T16:15:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40751.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2023-12-19T16:15:15.304 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45304.log 2023-12-19T16:15:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51276.log 2023-12-19T16:15:15.306 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51752.log.gz 2023-12-19T16:15:15.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45886.log 2023-12-19T16:15:15.307 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45304.log.gz 2023-12-19T16:15:15.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51276.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51276.log.gz 2023-12-19T16:15:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50713.log 2023-12-19T16:15:15.309 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.45886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38949.log 2023-12-19T16:15:15.311 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42934.log 2023-12-19T16:15:15.311 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50713.log.gz 2023-12-19T16:15:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38949.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38949.log.gz 2023-12-19T16:15:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2023-12-19T16:15:15.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51605.log 2023-12-19T16:15:15.315 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42934.log.gz 2023-12-19T16:15:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55081.log 2023-12-19T16:15:15.316 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz 2023-12-19T16:15:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47987.log 2023-12-19T16:15:15.318 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51605.log.gz 2023-12-19T16:15:15.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2023-12-19T16:15:15.320 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55081.log.gz 2023-12-19T16:15:15.321 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52430.log 2023-12-19T16:15:15.321 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47987.log.gz 2023-12-19T16:15:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49940.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2023-12-19T16:15:15.322 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58686.log 2023-12-19T16:15:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46744.log 2023-12-19T16:15:15.324 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52430.log.gz 2023-12-19T16:15:15.325 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48752.log 2023-12-19T16:15:15.325 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58686.log.gz 2023-12-19T16:15:15.326 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46744.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46744.log.gz 2023-12-19T16:15:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49183.log 2023-12-19T16:15:15.327 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54098.log 2023-12-19T16:15:15.328 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48752.log.gz 2023-12-19T16:15:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50906.log 2023-12-19T16:15:15.330 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49183.log.gz 2023-12-19T16:15:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54098.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54098.log.gz 2023-12-19T16:15:15.331 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54582.log 2023-12-19T16:15:15.332 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2023-12-19T16:15:15.332 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50906.log.gz 2023-12-19T16:15:15.334 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38706.log 2023-12-19T16:15:15.334 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54582.log.gz 2023-12-19T16:15:15.335 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56317.log 2023-12-19T16:15:15.335 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2023-12-19T16:15:15.336 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56209.log 2023-12-19T16:15:15.336 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38706.log.gz 2023-12-19T16:15:15.338 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56317.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56317.log.gz 2023-12-19T16:15:15.338 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44708.log 2023-12-19T16:15:15.339 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62985.log 2023-12-19T16:15:15.339 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56209.log.gz 2023-12-19T16:15:15.341 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52540.log 2023-12-19T16:15:15.341 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44708.log.gz 2023-12-19T16:15:15.342 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62985.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62985.log.gz-5 2023-12-19T16:15:15.342 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.37813.log 2023-12-19T16:15:15.343 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39759.log 2023-12-19T16:15:15.344 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52540.log.gz 2023-12-19T16:15:15.347 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37813.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2023-12-19T16:15:15.347 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50457.log 2023-12-19T16:15:15.347 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39759.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39759.log.gz 2023-12-19T16:15:15.347 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42992.log 2023-12-19T16:15:15.348 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50457.log.gz 2023-12-19T16:15:15.348 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56506.log 2023-12-19T16:15:15.349 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56068.log 2023-12-19T16:15:15.349 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42992.log.gz 2023-12-19T16:15:15.351 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2023-12-19T16:15:15.352 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56506.log.gz 2023-12-19T16:15:15.352 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57490.log 2023-12-19T16:15:15.353 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56068.log.gz 2023-12-19T16:15:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42676.log 2023-12-19T16:15:15.354 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2023-12-19T16:15:15.356 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46152.log 2023-12-19T16:15:15.356 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57490.log.gz 2023-12-19T16:15:15.357 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42676.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42676.log.gz 2023-12-19T16:15:15.357 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57517.log 2023-12-19T16:15:15.358 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2023-12-19T16:15:15.358 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46152.log.gz 2023-12-19T16:15:15.360 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55641.log 2023-12-19T16:15:15.360 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57517.log.gz 2023-12-19T16:15:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48226.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2023-12-19T16:15:15.361 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2023-12-19T16:15:15.362 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2023-12-19T16:15:15.363 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55641.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55641.log.gz 2023-12-19T16:15:15.364 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45010.log 2023-12-19T16:15:15.364 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2023-12-19T16:15:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60009.log 2023-12-19T16:15:15.365 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2023-12-19T16:15:15.366 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2023-12-19T16:15:15.367 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45010.log.gz 2023-12-19T16:15:15.369 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63629.log 2023-12-19T16:15:15.369 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60009.log.gz 2023-12-19T16:15:15.370 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48785.log 2023-12-19T16:15:15.371 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2023-12-19T16:15:15.371 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53665.log 2023-12-19T16:15:15.372 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63629.log.gz 2023-12-19T16:15:15.373 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48315.log 2023-12-19T16:15:15.373 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48785.log.gz 2023-12-19T16:15:15.374 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51336.log 2023-12-19T16:15:15.374 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53665.log.gz 2023-12-19T16:15:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48315.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48315.log.gz 2023-12-19T16:15:15.376 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2023-12-19T16:15:15.377 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53483.log 2023-12-19T16:15:15.377 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51336.log.gz 2023-12-19T16:15:15.379 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44387.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2023-12-19T16:15:15.379 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62098.log 2023-12-19T16:15:15.380 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53719.log 2023-12-19T16:15:15.380 INFO:teuthology.orchestra.run.smithi050.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53483.log.gz 2023-12-19T16:15:15.382 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62098.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62098.log.gz 2023-12-19T16:15:15.382 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38814.log 2023-12-19T16:15:15.382 INFO:teuthology.orchestra.run.smithi050.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.45886.log.gz 2023-12-19T16:15:15.383 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53719.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53719.log.gz 2023-12-19T16:15:15.383 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58441.log 2023-12-19T16:15:15.385 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46873.log 2023-12-19T16:15:15.385 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38814.log.gz 2023-12-19T16:15:15.385 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53087.log 2023-12-19T16:15:15.386 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58441.log.gz 2023-12-19T16:15:15.387 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46873.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46873.log.gz 2023-12-19T16:15:15.387 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55776.log 2023-12-19T16:15:15.388 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45145.log 2023-12-19T16:15:15.388 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53087.log.gz 2023-12-19T16:15:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55776.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55776.log.gzgzip 2023-12-19T16:15:15.389 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.51224.log 2023-12-19T16:15:15.391 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61282.log 2023-12-19T16:15:15.391 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45145.log.gz 2023-12-19T16:15:15.392 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47422.log 2023-12-19T16:15:15.392 INFO:teuthology.orchestra.run.smithi050.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51224.log.gz 2023-12-19T16:15:15.393 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61282.log: gzip -5 --verbose 88.8%-- -- replaced with /var/log/ceph/ceph-client.admin.61282.log.gz/var/log/ceph/ceph-client.admin.59754.log 2023-12-19T16:15:15.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.394 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47422.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47422.log.gz 2023-12-19T16:15:15.395 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50245.log 2023-12-19T16:15:15.396 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2023-12-19T16:15:15.396 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59754.log.gz 2023-12-19T16:15:15.397 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50245.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50245.log.gz 2023-12-19T16:15:15.397 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63041.log 2023-12-19T16:15:15.398 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63656.log 2023-12-19T16:15:15.399 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2023-12-19T16:15:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41440.log 2023-12-19T16:15:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63656.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63041.log.gz 2023-12-19T16:15:15.402 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63656.log.gz 2023-12-19T16:15:15.402 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53245.log 2023-12-19T16:15:15.403 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59350.log 2023-12-19T16:15:15.403 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41440.log.gz 2023-12-19T16:15:15.404 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58122.log 2023-12-19T16:15:15.404 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2023-12-19T16:15:15.406 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60063.log 2023-12-19T16:15:15.406 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59350.log.gz 2023-12-19T16:15:15.407 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59644.log 2023-12-19T16:15:15.407 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58122.log.gz 2023-12-19T16:15:15.409 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60063.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60063.log.gz 2023-12-19T16:15:15.409 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63386.log 2023-12-19T16:15:15.410 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43759.log 2023-12-19T16:15:15.411 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59644.log.gz 2023-12-19T16:15:15.412 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63386.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63386.log.gz 2023-12-19T16:15:15.412 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2023-12-19T16:15:15.413 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2023-12-19T16:15:15.413 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43759.log.gz 2023-12-19T16:15:15.414 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39651.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2023-12-19T16:15:15.415 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50933.log 2023-12-19T16:15:15.416 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53557.log 2023-12-19T16:15:15.416 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2023-12-19T16:15:15.417 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2023-12-19T16:15:15.418 INFO:teuthology.orchestra.run.smithi050.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.50933.log.gz 2023-12-19T16:15:15.418 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2023-12-19T16:15:15.419 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53557.log.gz 2023-12-19T16:15:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40562.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2023-12-19T16:15:15.420 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59818.log 2023-12-19T16:15:15.422 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33708.log 2023-12-19T16:15:15.422 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz 2023-12-19T16:15:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59818.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59818.log.gz 2023-12-19T16:15:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2023-12-19T16:15:15.424 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.33708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2023-12-19T16:15:15.425 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33708.log.gz 2023-12-19T16:15:15.426 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49099.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2023-12-19T16:15:15.426 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2023-12-19T16:15:15.427 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-19T16:15:15.427 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2023-12-19T16:15:15.428 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40697.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2023-12-19T16:15:15.429 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42223.log 2023-12-19T16:15:15.430 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63224.log 2023-12-19T16:15:15.432 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42223.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42223.log.gz 2023-12-19T16:15:15.432 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45037.log 2023-12-19T16:15:15.433 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63224.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63224.log.gz 2023-12-19T16:15:15.433 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58388.log 2023-12-19T16:15:15.434 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45037.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45037.log.gz 2023-12-19T16:15:15.435 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48477.log 2023-12-19T16:15:15.436 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62205.log 2023-12-19T16:15:15.436 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58388.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58388.log.gz 2023-12-19T16:15:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48879.log 2023-12-19T16:15:15.438 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48477.log.gz 2023-12-19T16:15:15.438 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62205.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62205.log.gz 2023-12-19T16:15:15.439 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62465.log 2023-12-19T16:15:15.440 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-19T16:15:15.440 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2023-12-19T16:15:15.441 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48879.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48879.log.gz 2023-12-19T16:15:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2023-12-19T16:15:15.446 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62465.log.gz 2023-12-19T16:15:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50110.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2023-12-19T16:15:15.446 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51363.log 2023-12-19T16:15:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60036.log 2023-12-19T16:15:15.447 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2023-12-19T16:15:15.447 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41529.log 2023-12-19T16:15:15.447 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51363.log.gz 2023-12-19T16:15:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59202.log 2023-12-19T16:15:15.448 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60036.log.gz 2023-12-19T16:15:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41529.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41529.log.gz 2023-12-19T16:15:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54260.log 2023-12-19T16:15:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59202.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59202.log.gz 2023-12-19T16:15:15.450 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54341.log 2023-12-19T16:15:15.451 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61994.log 2023-12-19T16:15:15.451 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54260.log.gz 2023-12-19T16:15:15.452 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37968.log 2023-12-19T16:15:15.452 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54341.log.gz 2023-12-19T16:15:15.453 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38256.log 2023-12-19T16:15:15.453 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61994.log.gz 2023-12-19T16:15:15.454 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55374.log 2023-12-19T16:15:15.454 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37968.log.gz 2023-12-19T16:15:15.455 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42876.log 2023-12-19T16:15:15.455 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2023-12-19T16:15:15.456 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55374.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-19T16:15:15.457 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55374.log.gz 2023-12-19T16:15:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42876.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42876.log.gzgzip 2023-12-19T16:15:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.57784.log 2023-12-19T16:15:15.459 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57032.log 2023-12-19T16:15:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57784.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57784.log.gz 2023-12-19T16:15:15.460 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2023-12-19T16:15:15.461 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42574.log 2023-12-19T16:15:15.462 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57032.log.gz 2023-12-19T16:15:15.463 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz-5 2023-12-19T16:15:15.463 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.44956.log 2023-12-19T16:15:15.465 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.42574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57382.log 2023-12-19T16:15:15.466 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43185.log 2023-12-19T16:15:15.466 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44956.log.gz 2023-12-19T16:15:15.468 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2023-12-19T16:15:15.468 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57382.log.gz 2023-12-19T16:15:15.469 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55722.log 2023-12-19T16:15:15.469 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43185.log.gz 2023-12-19T16:15:15.471 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51041.log 2023-12-19T16:15:15.471 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2023-12-19T16:15:15.472 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55722.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55722.log.gz 2023-12-19T16:15:15.473 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.49761.log 2023-12-19T16:15:15.474 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.51041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46502.log 2023-12-19T16:15:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49761.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49761.log.gz 94.1% -- replaced with /var/log/ceph/ceph-client.0.51041.log.gz 2023-12-19T16:15:15.476 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.476 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44122.log 2023-12-19T16:15:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46502.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46502.log.gz 2023-12-19T16:15:15.477 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61779.log 2023-12-19T16:15:15.479 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42140.log 2023-12-19T16:15:15.479 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44122.log.gz 2023-12-19T16:15:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63521.log 2023-12-19T16:15:15.480 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61779.log.gz 2023-12-19T16:15:15.481 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42140.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42140.log.gz 2023-12-19T16:15:15.482 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53865.log 2023-12-19T16:15:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52175.log 2023-12-19T16:15:15.484 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63521.log.gz 2023-12-19T16:15:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53865.log: gzip -5 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53865.log.gz 2023-12-19T16:15:15.485 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.57869.log 2023-12-19T16:15:15.486 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52175.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55749.log 2023-12-19T16:15:15.486 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52175.log.gz 2023-12-19T16:15:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50326.log 2023-12-19T16:15:15.488 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57869.log.gz 2023-12-19T16:15:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55749.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55749.log.gz 2023-12-19T16:15:15.489 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61336.log 2023-12-19T16:15:15.491 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43212.log 2023-12-19T16:15:15.491 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50326.log.gz 2023-12-19T16:15:15.492 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57215.log 2023-12-19T16:15:15.492 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61336.log.gz 2023-12-19T16:15:15.493 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43212.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43212.log.gz 2023-12-19T16:15:15.494 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61806.log 2023-12-19T16:15:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57215.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57215.log.gz 2023-12-19T16:15:15.495 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49425.log 2023-12-19T16:15:15.497 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50164.log 2023-12-19T16:15:15.497 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61806.log.gz 2023-12-19T16:15:15.498 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.49425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2023-12-19T16:15:15.499 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.0.49425.log.gz 2023-12-19T16:15:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62738.log 2023-12-19T16:15:15.500 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50164.log.gz 2023-12-19T16:15:15.501 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39375.log 2023-12-19T16:15:15.501 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2023-12-19T16:15:15.503 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59901.log 2023-12-19T16:15:15.503 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62738.log.gz 2023-12-19T16:15:15.504 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49291.log 2023-12-19T16:15:15.504 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39375.log.gz 2023-12-19T16:15:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45934.log 2023-12-19T16:15:15.506 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59901.log.gz 2023-12-19T16:15:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49291.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49291.log.gz 2023-12-19T16:15:15.507 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2023-12-19T16:15:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57301.log 2023-12-19T16:15:15.508 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45934.log.gz 2023-12-19T16:15:15.510 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56661.log 2023-12-19T16:15:15.510 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2023-12-19T16:15:15.511 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59049.log 2023-12-19T16:15:15.511 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57301.log.gz 2023-12-19T16:15:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2023-12-19T16:15:15.513 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56661.log.gz 2023-12-19T16:15:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59049.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59049.log.gz 2023-12-19T16:15:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43158.log 2023-12-19T16:15:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38841.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2023-12-19T16:15:15.516 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63413.log 2023-12-19T16:15:15.518 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52699.log 2023-12-19T16:15:15.518 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43158.log.gz 2023-12-19T16:15:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2023-12-19T16:15:15.520 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63413.log.gz 2023-12-19T16:15:15.521 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47008.log 2023-12-19T16:15:15.521 INFO:teuthology.orchestra.run.smithi050.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52699.log.gz 2023-12-19T16:15:15.522 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52106.log 2023-12-19T16:15:15.522 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2023-12-19T16:15:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63068.log 2023-12-19T16:15:15.524 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47008.log.gz 2023-12-19T16:15:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52106.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52106.log.gz 2023-12-19T16:15:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58475.log 2023-12-19T16:15:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63068.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63068.log.gz 2023-12-19T16:15:15.526 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62438.log 2023-12-19T16:15:15.527 INFO:teuthology.orchestra.run.smithi125.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-12-19T16:15:15.528 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43672.log 2023-12-19T16:15:15.528 INFO:teuthology.orchestra.run.smithi050.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.58475.log.gz 2023-12-19T16:15:15.529 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38283.log 2023-12-19T16:15:15.530 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62438.log.gz 2023-12-19T16:15:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43672.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43672.log.gz 2023-12-19T16:15:15.531 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62857.log 2023-12-19T16:15:15.533 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41305.log 2023-12-19T16:15:15.533 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38283.log.gz 2023-12-19T16:15:15.534 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62385.log 2023-12-19T16:15:15.534 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.62857.log.gz 2023-12-19T16:15:15.535 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41305.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41305.log.gz 2023-12-19T16:15:15.536 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55911.log 2023-12-19T16:15:15.537 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.62385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43730.log 2023-12-19T16:15:15.538 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55911.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55911.log.gz 2023-12-19T16:15:15.539 INFO:teuthology.orchestra.run.smithi050.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.0.62385.log.gz 2023-12-19T16:15:15.540 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43730.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43730.log.gz 2023-12-19T16:15:15.540 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53832.log 2023-12-19T16:15:15.541 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38625.log 2023-12-19T16:15:15.543 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2023-12-19T16:15:15.543 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53832.log.gz 2023-12-19T16:15:15.545 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2023-12-19T16:15:15.545 INFO:teuthology.orchestra.run.smithi050.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38625.log.gz 2023-12-19T16:15:15.546 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50742.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2023-12-19T16:15:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56263.log 2023-12-19T16:15:15.547 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40508.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2023-12-19T16:15:15.548 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38337.log 2023-12-19T16:15:15.550 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59458.log 2023-12-19T16:15:15.550 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56263.log.gz 2023-12-19T16:15:15.552 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50484.log 2023-12-19T16:15:15.552 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38337.log.gz 2023-12-19T16:15:15.552 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60931.log 2023-12-19T16:15:15.552 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59458.log.gz 2023-12-19T16:15:15.554 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50484.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50484.log.gz-5 2023-12-19T16:15:15.554 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.56425.log 2023-12-19T16:15:15.555 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60931.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60931.log.gz 2023-12-19T16:15:15.555 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47667.log 2023-12-19T16:15:15.557 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58870.log 2023-12-19T16:15:15.557 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2023-12-19T16:15:15.558 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47667.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47667.log.gz 2023-12-19T16:15:15.559 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58870.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58870.log.gz 2023-12-19T16:15:15.560 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2023-12-19T16:15:15.561 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59485.log 2023-12-19T16:15:15.563 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62237.log 2023-12-19T16:15:15.563 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2023-12-19T16:15:15.564 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59485.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59485.log.gz 2023-12-19T16:15:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62237.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62237.log.gz 2023-12-19T16:15:15.566 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38733.log 2023-12-19T16:15:15.568 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50191.log 2023-12-19T16:15:15.569 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48257.log 2023-12-19T16:15:15.570 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38733.log.gz 2023-12-19T16:15:15.571 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2023-12-19T16:15:15.571 INFO:teuthology.orchestra.run.smithi050.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50191.log.gz 2023-12-19T16:15:15.572 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48257.log: gzip -5 --verbose 89.4%-- -- replaced with /var/log/ceph/ceph-client.admin.48257.log.gz/var/log/ceph/ceph-client.admin.45412.log 2023-12-19T16:15:15.572 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:15.574 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63683.log 2023-12-19T16:15:15.574 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2023-12-19T16:15:15.575 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2023-12-19T16:15:15.575 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2023-12-19T16:15:15.577 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.63683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62329.log 2023-12-19T16:15:15.577 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63683.log.gz 2023-12-19T16:15:15.578 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55467.log 2023-12-19T16:15:15.578 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2023-12-19T16:15:15.580 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42818.log 2023-12-19T16:15:15.580 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62329.log.gz 2023-12-19T16:15:15.580 INFO:teuthology.orchestra.run.smithi050.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.0.42574.log.gz 2023-12-19T16:15:15.581 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2023-12-19T16:15:15.581 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55467.log.gz 2023-12-19T16:15:15.582 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2023-12-19T16:15:15.582 INFO:teuthology.orchestra.run.smithi050.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42818.log.gz 2023-12-19T16:15:15.583 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58041.log 2023-12-19T16:15:15.584 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2023-12-19T16:15:15.584 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42086.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2023-12-19T16:15:15.585 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60462.log 2023-12-19T16:15:15.586 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60658.log 2023-12-19T16:15:15.586 INFO:teuthology.orchestra.run.smithi050.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58041.log.gz 2023-12-19T16:15:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60462.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60462.log.gz 2023-12-19T16:15:15.587 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2023-12-19T16:15:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.60658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55943.log 2023-12-19T16:15:15.589 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60658.log.gz 2023-12-19T16:15:15.589 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41885.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2023-12-19T16:15:15.590 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55496.log 2023-12-19T16:15:15.591 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54748.log 2023-12-19T16:15:15.591 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55943.log.gz 2023-12-19T16:15:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55496.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55496.log.gz 2023-12-19T16:15:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42763.log 2023-12-19T16:15:15.593 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2023-12-19T16:15:15.594 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54748.log.gz 2023-12-19T16:15:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42763.log: 90.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.42763.log.gz-5 2023-12-19T16:15:15.595 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.52910.log 2023-12-19T16:15:15.596 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-19T16:15:15.596 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2023-12-19T16:15:15.597 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.52910.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52910.log.gz 2023-12-19T16:15:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41966.log 2023-12-19T16:15:15.599 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47505.log 2023-12-19T16:15:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41966.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41966.log.gz 2023-12-19T16:15:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2023-12-19T16:15:15.602 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46954.log 2023-12-19T16:15:15.602 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47505.log.gz 2023-12-19T16:15:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2023-12-19T16:15:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.45617.log 2023-12-19T16:15:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46954.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46954.log.gz 2023-12-19T16:15:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38933.log 2023-12-19T16:15:15.607 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49126.log 2023-12-19T16:15:15.607 INFO:teuthology.orchestra.run.smithi050.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45617.log.gz 2023-12-19T16:15:15.608 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38933.log: gzip 88.9%-5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log 2023-12-19T16:15:15.608 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38933.log.gz 2023-12-19T16:15:15.609 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2023-12-19T16:15:15.609 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49126.log.gz 2023-12-19T16:15:15.611 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56479.log 2023-12-19T16:15:15.611 INFO:teuthology.orchestra.run.smithi050.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz 2023-12-19T16:15:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41021.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2023-12-19T16:15:15.612 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54013.log 2023-12-19T16:15:15.614 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2023-12-19T16:15:15.614 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56479.log.gz 2023-12-19T16:15:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54013.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54013.log.gz 2023-12-19T16:15:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39570.log 2023-12-19T16:15:15.616 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2023-12-19T16:15:15.617 INFO:teuthology.orchestra.run.smithi050.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2023-12-19T16:15:15.618 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39570.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39570.log.gz 2023-12-19T16:15:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.59166.log 2023-12-19T16:15:15.620 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59404.log 2023-12-19T16:15:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59166.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2023-12-19T16:15:15.622 INFO:teuthology.orchestra.run.smithi050.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59166.log.gz 2023-12-19T16:15:15.622 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.59404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43615.log 2023-12-19T16:15:15.623 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59404.log.gz 2023-12-19T16:15:15.623 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2023-12-19T16:15:15.624 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43615.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43615.log.gz 2023-12-19T16:15:15.625 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40940.log 2023-12-19T16:15:15.626 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54233.log 2023-12-19T16:15:15.626 INFO:teuthology.orchestra.run.smithi050.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2023-12-19T16:15:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40940.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40940.log.gz 2023-12-19T16:15:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:--verbose -- /var/log/ceph/ceph-client.admin.50852.log 2023-12-19T16:15:15.629 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54233.log: gzip 88.9%-5 -- replaced with /var/log/ceph/ceph-client.admin.54233.log.gz--verbose 2023-12-19T16:15:15.629 INFO:teuthology.orchestra.run.smithi050.stderr:-- /var/log/ceph/ceph-client.admin.43960.log 2023-12-19T16:15:15.631 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50852.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46623.log 2023-12-19T16:15:15.631 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50852.log.gz 2023-12-19T16:15:15.632 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43960.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43960.log.gz 2023-12-19T16:15:15.632 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54125.log 2023-12-19T16:15:15.633 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43507.log 2023-12-19T16:15:15.634 INFO:teuthology.orchestra.run.smithi050.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46623.log.gz 2023-12-19T16:15:15.635 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45561.log 2023-12-19T16:15:15.635 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54125.log.gz 2023-12-19T16:15:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43507.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43507.log.gz 2023-12-19T16:15:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50686.log 2023-12-19T16:15:15.638 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51636.log 2023-12-19T16:15:15.638 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45561.log.gz 2023-12-19T16:15:15.640 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50686.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50686.log.gz 2023-12-19T16:15:15.640 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44281.log 2023-12-19T16:15:15.640 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51636.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51636.log.gz 2023-12-19T16:15:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39840.log 2023-12-19T16:15:15.642 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.44281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55695.log 2023-12-19T16:15:15.642 INFO:teuthology.orchestra.run.smithi050.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44281.log.gz 2023-12-19T16:15:15.643 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39840.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2023-12-19T16:15:15.644 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46563.log 2023-12-19T16:15:15.645 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.55695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46981.log 2023-12-19T16:15:15.645 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55695.log.gz 2023-12-19T16:15:15.651 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46563.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46563.log.gz 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53425.log 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37913.log 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46981.log.gz 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53425.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53425.log.gz 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43104.log 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58635.log 2023-12-19T16:15:15.652 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37913.log.gz 2023-12-19T16:15:15.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43104.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43104.log.gz 2023-12-19T16:15:15.655 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2023-12-19T16:15:15.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38939.log 2023-12-19T16:15:15.656 INFO:teuthology.orchestra.run.smithi050.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58635.log.gz 2023-12-19T16:15:15.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61943.log 2023-12-19T16:15:15.656 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2023-12-19T16:15:15.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38939.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38939.log.gz 2023-12-19T16:15:15.657 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2023-12-19T16:15:15.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61887.log 2023-12-19T16:15:15.659 INFO:teuthology.orchestra.run.smithi050.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61943.log.gz 2023-12-19T16:15:15.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40778.log 2023-12-19T16:15:15.660 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2023-12-19T16:15:15.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61887.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61887.log.gz 2023-12-19T16:15:15.661 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56290.log 2023-12-19T16:15:15.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40778.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40778.log.gz 2023-12-19T16:15:15.663 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53611.log 2023-12-19T16:15:15.664 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56290.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56290.log.gz 2023-12-19T16:15:15.665 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2023-12-19T16:15:15.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48933.log 2023-12-19T16:15:15.666 INFO:teuthology.orchestra.run.smithi050.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53611.log.gz 2023-12-19T16:15:15.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41197.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2023-12-19T16:15:15.667 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52755.log 2023-12-19T16:15:15.668 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52486.log 2023-12-19T16:15:15.669 INFO:teuthology.orchestra.run.smithi050.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48933.log.gz 2023-12-19T16:15:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.0.52755.log: /var/log/ceph/ceph-client.admin.52486.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52486.log.gz 2023-12-19T16:15:15.672 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28860.log 2023-12-19T16:15:15.674 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38760.log 2023-12-19T16:15:15.675 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.28860.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28860.log.gz 2023-12-19T16:15:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38760.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38760.log.gz 2023-12-19T16:15:15.677 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41129.log 2023-12-19T16:15:15.679 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49369.log 2023-12-19T16:15:15.680 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41129.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41129.log.gz 2023-12-19T16:15:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.49369.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49369.log.gz 2023-12-19T16:15:15.682 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56452.log 2023-12-19T16:15:15.684 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62931.log 2023-12-19T16:15:15.686 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.56452.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56452.log.gz 2023-12-19T16:15:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.62931.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62931.log.gz 2023-12-19T16:15:15.688 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2023-12-19T16:15:15.689 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61174.log 2023-12-19T16:15:15.690 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42051.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2023-12-19T16:15:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61174.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61174.log.gz 2023-12-19T16:15:15.692 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2023-12-19T16:15:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2023-12-19T16:15:15.695 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42113.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2023-12-19T16:15:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40258.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2023-12-19T16:15:15.696 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2023-12-19T16:15:15.698 INFO:teuthology.orchestra.run.smithi050.stderr: 94.6% -- replaced with /var/log/ceph/ceph-client.0.52755.log.gz 2023-12-19T16:15:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2023-12-19T16:15:15.700 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.45698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48906.log 2023-12-19T16:15:15.700 INFO:teuthology.orchestra.run.smithi050.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2023-12-19T16:15:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.61521.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2023-12-19T16:15:15.701 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47275.log 2023-12-19T16:15:15.702 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48906.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48906.log.gz 2023-12-19T16:15:15.703 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54287.log 2023-12-19T16:15:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.47275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2023-12-19T16:15:15.704 INFO:teuthology.orchestra.run.smithi050.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47275.log.gz 2023-12-19T16:15:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.54287.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54287.log.gz 2023-12-19T16:15:15.706 INFO:teuthology.orchestra.run.smithi050.stderr:-5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-19T16:15:15.707 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41048.log: gzip 88.6%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz-- 2023-12-19T16:15:15.708 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53276.log 2023-12-19T16:15:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57603.log 2023-12-19T16:15:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.53276.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53276.log.gz 2023-12-19T16:15:15.712 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.57603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39379.log 2023-12-19T16:15:15.712 INFO:teuthology.orchestra.run.smithi050.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57603.log.gz 2023-12-19T16:15:15.713 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2023-12-19T16:15:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39379.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39379.log.gz 2023-12-19T16:15:15.715 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58989.log 2023-12-19T16:15:15.716 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39155.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2023-12-19T16:15:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50358.log 2023-12-19T16:15:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.58989.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58989.log.gz 2023-12-19T16:15:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46714.log 2023-12-19T16:15:15.720 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.50358.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50358.log.gz 2023-12-19T16:15:15.721 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51852.log 2023-12-19T16:15:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.46714.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2023-12-19T16:15:15.723 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.51852.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51852.log.gz 2023-12-19T16:15:15.724 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48531.log 2023-12-19T16:15:15.727 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.48531.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48531.log.gz 2023-12-19T16:15:15.786 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-12-19T16:15:16.016 INFO:teuthology.orchestra.run.smithi125.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-19T16:15:16.677 INFO:teuthology.orchestra.run.smithi125.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-19T16:15:16.712 INFO:teuthology.orchestra.run.smithi125.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-19T16:15:18.066 INFO:teuthology.orchestra.run.smithi050.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-19T16:15:19.012 INFO:teuthology.orchestra.run.smithi125.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-19T16:15:19.110 INFO:teuthology.orchestra.run.smithi050.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-19T16:15:19.138 INFO:teuthology.orchestra.run.smithi125.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-19T16:15:19.815 INFO:teuthology.orchestra.run.smithi125.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-19T16:15:19.840 INFO:teuthology.orchestra.run.smithi125.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-19T16:15:19.845 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-12-19T16:15:19.845 INFO:teuthology.orchestra.run.smithi125.stderr:real 0m5.265s 2023-12-19T16:15:19.845 INFO:teuthology.orchestra.run.smithi125.stderr:user 0m23.820s 2023-12-19T16:15:19.845 INFO:teuthology.orchestra.run.smithi125.stderr:sys 0m1.994s 2023-12-19T16:15:20.373 INFO:teuthology.orchestra.run.smithi050.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-19T16:15:20.572 INFO:teuthology.orchestra.run.smithi050.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-19T16:15:21.266 INFO:teuthology.orchestra.run.smithi050.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-19T16:15:21.270 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-12-19T16:15:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m6.728s 2023-12-19T16:15:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m23.668s 2023-12-19T16:15:21.270 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m3.501s 2023-12-19T16:15:21.271 INFO:tasks.ceph:Archiving logs... 2023-12-19T16:15:21.272 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi050/log 2023-12-19T16:15:21.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-19T16:15:25.166 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi125/log 2023-12-19T16:15:25.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-19T16:15:28.229 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-19T16:15:28.239 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-19T16:15:28.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T16:15:28.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T16:15:28.343 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-19T16:15:28.352 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-19T16:15:28.353 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T16:15:28.360 DEBUG:teuthology.orchestra.run.smithi125:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T16:15:28.401 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T16:15:28.401 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi050.stdout:*hv01.front.sepi 44.4.53.2 2 u 263 256 377 0.214 -0.501 0.186 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi050.stdout:+hv02.front.sepi 63.231.80.2 3 u 134 256 377 0.215 -0.406 0.149 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi050.stdout:+hv03.front.sepi 74.6.168.72 3 u 185 256 377 0.235 -1.336 0.229 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi125.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================== 2023-12-19T16:15:28.402 INFO:teuthology.orchestra.run.smithi125.stdout:*hv01.front.sepi 44.4.53.2 2 u 236 256 377 0.234 1.820 0.597 2023-12-19T16:15:28.403 INFO:teuthology.orchestra.run.smithi125.stdout:+hv02.front.sepi 63.231.80.2 3 u 105 256 377 0.164 2.285 0.586 2023-12-19T16:15:28.403 INFO:teuthology.orchestra.run.smithi125.stdout:+hv03.front.sepi 74.6.168.72 3 u 192 256 377 0.228 0.443 1.085 2023-12-19T16:15:28.403 INFO:teuthology.orchestra.run.smithi125.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-19T16:15:28.404 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-19T16:15:28.414 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-19T16:15:28.414 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-19T16:15:28.423 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-19T16:15:28.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-19T16:15:28.440 INFO:teuthology.task.internal:Duration was 6549.647185 seconds 2023-12-19T16:15:28.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-19T16:15:28.449 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-19T16:15:28.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T16:15:28.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T16:15:28.611 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-19T16:15:28.611 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-12-19T16:15:28.611 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-19T16:15:28.648 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi125.front.sepia.ceph.com 2023-12-19T16:15:28.648 DEBUG:teuthology.orchestra.run.smithi125:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-19T16:15:28.689 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-19T16:15:28.690 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-19T16:15:28.700 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-19T16:15:28.802 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-19T16:15:28.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-19T16:15:28.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-19T16:15:28.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-19T16:15:28.942 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-19T16:15:28.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T16:15:28.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T16:15:29.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-19T16:15:29.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T16:15:29.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T16:15:29.067 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-12-19T16:15:29.103 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = core 2023-12-19T16:15:29.136 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-19T16:15:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:15:29.163 DEBUG:teuthology.orchestra.run.smithi125:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-19T16:15:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T16:15:29.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-19T16:15:29.209 INFO:teuthology.task.internal:Transferring archived files... 2023-12-19T16:15:29.210 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi050 2023-12-19T16:15:29.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T16:15:29.255 DEBUG:teuthology.misc:Transferring archived files from smithi125:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876/remote/smithi125 2023-12-19T16:15:29.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T16:15:29.301 INFO:teuthology.task.internal:Removing archive directory... 2023-12-19T16:15:29.302 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-19T16:15:29.309 DEBUG:teuthology.orchestra.run.smithi125:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-19T16:15:29.362 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-19T16:15:29.374 INFO:teuthology.task.internal:Not uploading archives. 2023-12-19T16:15:29.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-19T16:15:29.384 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-19T16:15:29.384 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T16:15:29.391 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T16:15:29.409 INFO:teuthology.orchestra.run.smithi050.stdout: 1048609 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 16:15 /home/ubuntu/cephtest 2023-12-19T16:15:29.429 INFO:teuthology.orchestra.run.smithi125.stdout: 1048600 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 16:15 /home/ubuntu/cephtest 2023-12-19T16:15:29.433 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-19T16:15:29.444 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-19T16:15:29.467 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T16:15:29.490 DEBUG:teuthology.nuke:shortname: smithi050 2023-12-19T16:15:29.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T16:15:29.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 14:16:21.564741', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-19T16:15:29.516 DEBUG:teuthology.nuke:shortname: smithi125 2023-12-19T16:15:29.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T16:15:29.535 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_12:25:28-fs-rishabh-mdscap-quincy-testing-default-smithi/7496876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 14:16:21.565958', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-19T16:15:29.567 INFO:teuthology.orchestra.console.smithi050:Power off 2023-12-19T16:15:29.568 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T16:15:29.590 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-12-19T16:15:29.591 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:29.606 INFO:teuthology.orchestra.console.smithi125:Power off 2023-12-19T16:15:29.607 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T16:15:29.617 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-19T16:15:29.629 DEBUG:teuthology.orchestra.console.smithi125:power off output: Chassis Power Control: Down/Off 2023-12-19T16:15:29.629 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:29.658 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2023-12-19T16:15:33.618 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:33.659 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:33.729 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-19T16:15:33.770 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2023-12-19T16:15:37.733 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:37.771 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:37.844 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-19T16:15:37.882 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-12-19T16:15:37.982 INFO:teuthology.orchestra.console.smithi125:Power off completed 2023-12-19T16:15:41.845 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T16:15:41.955 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-12-19T16:15:42.055 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-12-19T16:15:42.161 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6549.647184848785 failure_reason: '"1702996897.632042 mon.a (mon.0) 358 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default mon.a-kernel-sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c mon.b-kernel-sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c owner: scheduled_rishabh@teuthology success: false 2023-12-19T16:15:42.161 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T16:15:42.237 INFO:teuthology.run:FAIL